Figure 4

Reinforcement of action values (schematic). Each object is associated with 12 action values. For the object in trial t, 4 action values inform the response of the current trial t, 4 values concern the response of the next trial t + 1, and the remaining 4 values contribute to the response of the second next trial t + 2. Correspondingly, the response of trial t is based on 12 actions values: 4 values of the current object t, 4 values of the previous object t - 1, and 4 values of the pre-previous object t - 2. Temporal context determines which action values are reinforced consistently. (a) In the absence of temporal context, only the current object's action values are reinforced consistently and come to reflect the correct choice. In this case, the decision in trial t is based on 4 action values of object t. (b) In the presence of temporal context, both the current and the previous object's action values are reinforced consistently. Thus, the decision in trial t is based on 4 action values of object t and 4 action values of object t - 1.