This title appears in the Scientific Report :
2011
Please use the identifier:
http://dx.doi.org/10.1371/journal.pcbi.1001133 in citations.
An Imperfect Dopaminergic Error Signal Can Drive Temporal-Difference Learning.
An Imperfect Dopaminergic Error Signal Can Drive Temporal-Difference Learning.
An open problem in the field of computational neuroscience is how to link synaptic plasticity to system-level learning. A promising framework in this context is temporal-difference (TD) learning. Experimental evidence that supports the hypothesis that the mammalian brain performs temporal-difference...
Saved in:
Personal Name(s): | Potjans, W |
---|---|
Diesmann, M. / Morrison, A. | |
Contributing Institute: |
Computational and Systems Neuroscience; INM-6 |
Published in: | PLoS Computational Biology, 7 (2011) S. e1001133 |
Imprint: |
San Francisco, Calif.
Public Library of Science
2011
|
Physical Description: |
e1001133 |
PubMed ID: |
21589888 |
DOI: |
10.1371/journal.pcbi.1001133 |
Document Type: |
Journal Article |
Research Program: |
Brain-inspired multiscale computation in neuromorphic hybrid systems Pathophysiological Mechanisms of Neurological and Psychiatric Diseases Neurowissenschaften |
Series Title: |
PLoS Computational Biology
7 |
Subject (ZB): | |
Publikationsportal JuSER |
An open problem in the field of computational neuroscience is how to link synaptic plasticity to system-level learning. A promising framework in this context is temporal-difference (TD) learning. Experimental evidence that supports the hypothesis that the mammalian brain performs temporal-difference learning includes the resemblance of the phasic activity of the midbrain dopaminergic neurons to the TD error and the discovery that cortico-striatal synaptic plasticity is modulated by dopamine. However, as the phasic dopaminergic signal does not reproduce all the properties of the theoretical TD error, it is unclear whether it is capable of driving behavior adaptation in complex tasks. Here, we present a spiking temporal-difference learning model based on the actor-critic architecture. The model dynamically generates a dopaminergic signal with realistic firing rates and exploits this signal to modulate the plasticity of synapses as a third factor. The predictions of our proposed plasticity dynamics are in good agreement with experimental results with respect to dopamine, pre- and post-synaptic activity. An analytical mapping from the parameters of our proposed plasticity dynamics to those of the classical discrete-time TD algorithm reveals that the biological constraints of the dopaminergic signal entail a modified TD algorithm with self-adapting learning parameters and an adapting offset. We show that the neuronal network is able to learn a task with sparse positive rewards as fast as the corresponding classical discrete-time TD algorithm. However, the performance of the neuronal network is impaired with respect to the traditional algorithm on a task with both positive and negative rewards and breaks down entirely on a task with purely negative rewards. Our model demonstrates that the asymmetry of a realistic dopaminergic signal enables TD learning when learning is driven by positive rewards but not when driven by negative rewards. |