Hierarchical Bayesian Modeling of Context-Dependent Learning Rates in Reward Prediction

Abstract

In reward learning, the difference between the expected value of an outcome and the observed outcome is the prediction error. Reinforcement learning models use a learning rate parameter to discount the prediction error when the model updates the expected value of a stimulus. Previous work has demonstrated that learning agents adapt their learning rates according to the uncertainty of the context. In this thesis, I analyzed data from a task in which participants learned reward values in two risk contexts with different degrees of uncertainty. I computed and compared empirical learning rates, learning rates fitted by reinforcement learning models using conventional log likelihood maximization, and learning rates estimated by hierarchical Bayesian modeling. The results from all three analyses suggested that participants had higher learning rates in the low-risk context. Hierarchical Bayesian modeling estimated learning rates more robustly than the other two methods. I used hierarchical Bayesian modeling to examine participant learning rates, which decayed over time within an uncertainty context. Finally, I demonstrated that positive prediction errors were associated with higher learning rates in the low-risk context but not the high-risk context. This novel finding suggests that high uncertainty might interfere with optimistic over-attention to positive outcomes.