Skip to main content

Advertisement

Properties of synaptic plasticity rules implementing actor-critic temporal-difference learning

Article metrics

  • 1226 Accesses

There is considerable interest in establishing a link between system-level learning and synaptic plasticity [13]. In a previous study [4] we presented a specific set of biologically plausible synaptic plasticity rules implementing temporal-difference (TD) learning in a spiking neuronal network inspired by the actor-critic architecture [5]. We showed the equivalence between the plasticity rules and the traditional discrete-time TD(0) algorithm and demonstrated that the network learns a complex task with a similar speed to its discrete time counterpart and attains the same equilibrium performance. However, the set of learning rules represents only one possible way in which actor-critic TD learning could be implemented in the brain, and so the model has only limited predictive power for experimental work.

Here, we extract properties of synaptic plasticity rules that suffice to implement actor-critic TD(0) learning, under the assumption that states are represented by elevated rates in disjunct sets of neurons. On this basis we define generalized classes of continuous time synaptic plasticity rules that implement value function and policy updates. The main property is that the amount and sign of the weight update depends on a characteristic change in the activity of the critic module combined with a global reward signal. We present concrete examples belonging to the defined class and demonstrate that they are able to solve a non-trivial task. We further analyze to what extent the defined class of plasticity rules are compatible with experimental findings of synaptic plasticity [6, 7].

References

  1. 1.

    Izhikevich EM: Solving the distal reward problem through linkage of STDP and dopamine signaling. Cerebral Cortex. 2007, 17 (10): 2443-2452. 10.1093/cercor/bhl152.

  2. 2.

    Baras D, Meir R: Reinforcement Learning, Spike-Time-Dependent Plasticity, and the BCM Rule. Neural Computation. 2007, 19: 2245-2279. 10.1162/neco.2007.19.8.2245.

  3. 3.

    Florian RV: Reinforcement learning Through Modulation of Spike-Timing – Dependent Synaptic Plasticity. Neural Computation. 2007, 19: 1468-1502. 10.1162/neco.2007.19.6.1468.

  4. 4.

    Potjans W, Morrison A, Diesmann M: A spiking neural network model for the actor-critic temporal-difference learning algorithm. 342.6. 37th SFN meeting, San Diego, USA.

  5. 5.

    Sutton RS, Barto AG: Reinforcement learning, An Introduction. 1998, The MIT press

  6. 6.

    Kirkwood A, Rioult MG, Bear MF: Experience-dependent modification of synaptic plasticity in visual cortex. Nature. 1996, 381: 526-528. 10.1038/381526a0.

  7. 7.

    Reynolds JNJ, Wickens JR: Dopamine-dependent plasticity of corticostriatal synapses. Neural Networks. 2002, 15: 507-521. 10.1016/S0893-6080(02)00045-X.

Download references

Acknowledgements

Partially funded by DIP F1.2, BMBF Grant 01GQ0420 to the Bernstein Center for Computational Neuroscience Freiburg, and EU Grant 15879 (FACETS).

Author information

Correspondence to Wiebke Potjans.

Rights and permissions

Open Access This article is published under license to BioMed Central Ltd. This is an Open Access article is distributed under the terms of the Creative Commons Attribution 2.0 International License (https://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and Permissions

About this article

Cite this article

Potjans, W., Morrison, A. & Diesmann, M. Properties of synaptic plasticity rules implementing actor-critic temporal-difference learning. BMC Neurosci 9, P69 (2008) doi:10.1186/1471-2202-9-S1-P69

Download citation

Keywords

  • Synaptic Plasticity
  • Neuronal Network
  • Critic Module
  • Elevated Rate
  • Reward Signal