The Successor Representation Model in Human Reinforcement Learning: Using Cross-Stimulus Suppression as an Indicator of Neural Expectation to Detect Signatures of State Evaluation Strategies

Abstract

Theories of reinforcement learning have focused on two families of algorithms representing habitual and goal-directed decision-making respectively: model-free and model-based algorithms. These algorithms are commonly interpreted as opposite extremes in a tradeoff between behavioral flexibility and computational efficiency. In this study, we evaluate neural evidence for an intermediate class of algorithms, the successor representation (SR), that blends the efficiency of model-free algorithms with the flexibility of model-based algorithms. This model caches and reuses aggregate multi-step future state occupancies, allowing for a flexible yet subtler, more cognitive notion of habit. A key behavioral signature of the SR model is differential sensitivity to reward than transition revaluations, due to non-updated predictions in the latter. A recent behavioral study has shown human learning characterized by errors following distal transition changes suggesting reliance of the SR. This study aims to detect neural counterparts of these inferred non-updated predictions using cross-stimulus suppression as a signature of neural expectation. We assess whether suppression indicating non-updated predictions occur more often on transition revaluations when subjects make incorrect choices. We find such preliminary evidence in the prefrontal cortex, orbitofrontal cortex and parietal lobe for the successor representation in human reinforcement learning.