Taking the stock market as a whole object, we assume that prior losses and gains are two different factors that can influence risk preference separately. The two factors are introduced as separate explanatory variables into the time-varying GARCH-M (TVRA-GARCH-M) model. Then, we redefine prior losses and gains by selecting different reference point to study investors’ time-varying risk preference. The empirical evidence shows that investors’ risk preference is time varying and is influenced by previous outcomes; the stock market as a whole exhibits house money effect; that is, prior gains can decrease investors’ risk aversion while prior losses increase their risk aversion. Besides, different reference points selected by investors will cause different valuation of prior losses and gains, thus affecting investors’ risk preference.