acceptodds
Withdrawn from ICLR 2027

Representational advantages of distributional reinforcement learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Distributional reinforcement learning (dRL) agents, which learn the full distribution of returns, often outperform classical RL (cRL) agents and account for neural activity in mammalian reward-learning circuits. These findings suggest that dRL is a computationally advantageous learning strategy, although the mechanisms behind this advantage remain unclear. We develop an exact theory of learning under cRL and dRL objectives in a minimal model—a two-layer linear network—and characterize how the expectile-based dRL objective shapes state representations throughout learning. Unlike cRL, dRL learning is driven by higher-order moments of the reward distribution, which are captured by the input–output covariance matrix. As a result, dRL networks learn at least as fast as cRL networks, and differences in moments beyond the mean speed up learning further. Moments are learned sequentially, in an order set by the reward statistics, which also fully determine the geometry of the hidden representation. This representation supports faster transfer learning in dRL, whereas pre-training with the cRL objective can impede transfer. We validate our results in simulation in linear and non-linear networks. This work lays an analytic foundation for understanding the performance gains observed in more complex dRL models and tasks, and yields testable predictions for how biological circuits represent and acquire distributional information.

Primary area:
applications to neuroscience & cognitive science
Keywords:
reinforcement learning, distributional reinforcement learning, exact learning dynamics, linear neural networks, representation learning theory, computational neuroscience, transfer learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_U1vzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the learning dynamics of distribution matching against expectation matching. The goal is to gain understanding of the representation learnt in distributional reinforcement learning compared to traditional value function training. The work simplifies the policy evaluation setting by considering a set of supervised learning setups as a proxy for policy evaluation.

The paper extends prior works that studied the learning dynamics of networks with 2 linear layers for mean regression. They do so by formalizing distribution matching via expectile regression / matching.

The paper studies in detail a two-state setup that allows both obtaining closed-form solutions for the MSE objectives and performing simulations. The authors discuss the learning speed and the hidden-state geometry. Both agree and show that distributional targets allow learning embeddings that preserve distinctions between inputs with different moments beyond their means. In the case of equal means, the usual mean regression network collapses to a single embedding.

Critical strengths

The idea of borrowing from the litterature on supervised learning learning dynamics and extend it to distribution matching seems novel.

Critical weaknesses

In my opinion, the primary issue with this work is its exposition and lack of motivation. Despite spending several days reading it and familiarizing myself with the literature on learning dynamics, I still fail to grasp what the motivation for this work was.

  • This work is not about distributional reinforcement learning.
    • As far as I understand, it studies supervised learning as a proxy for policy evaluation. It never motivates this choice or connects it back to policy evaluation.
    • Empirical distributional policy evaluation relies on temporal differences, whose dynamics might be very different from the supervised case.
    • There is no empirical connection made with any policy evaluation setup. I could not find any evidence that the representational advantage exposed for the supervised learning case should bring any benefit for policy evaluation and then for control. In my opinion, this work is poorly motivated.
  • There is no red line / story to follow. This work is a collection of facts tied together.
    • After several days, I still cannot tell why I am reading what I am reading for almost every section of this work.
    • Section 2.3, which in my opinion is the most important one for the background, compresses too much. I had to spend several hours going through each sentence step by step. The derivations take several pages. I am not sure the average reader would get any insight after reading the first two paragraphs. Yet they seem essential to the rest of the paper.
    • Most experiments are scattered in the appendix, which makes it very hard to follow.
    • Starting from Section 3.2, the problem gets very severe. The prose becomes very hard to follow. I would not blame any LLM usage if the text conveyed an intent that was clear to follow, but here this is not the case.
  • The work does address the lack of control discussion but does not emphasize the lack of actual policy evaluation.
  • The discussion section is a collection of assertions and facts, many of which are not tied to the story. As an average reader with expertise in distributional RL, I could not get any insights from “Predictions for neuroscience” or “Predictions for deep RL,” as the exposition is very cryptic.
Questions for authors
  • In the discussion, you say, "We have developed an exact theory of learning for classical and distributional RL objectives in two layer linear networks." Did you? What was your contribution for the classical RL setting? Where did you go beyond simple expectation/distributional regression? You studied two losses used in RL based on MSE and expectiles. Put differently, what does your analysis establish regarding actual RL using these objectives?
Reviewer_b5wSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies why distributional RL (dRL) may have a representation-learning advantage over classical RL (cRL). The authors consider two-layer linear networks trained to predict either mean rewards or reward-distribution expectiles. Using existing results on deep linear network dynamics, they relate learning speed and hidden representations to the singular values of an input-output covariance matrix. They argue that differences in reward variance and shape can speed up the learning of distributional targets, determine the order in which representation modes are learned, and preserve state information that may help with a later task. The analysis is mainly derived for an MSE surrogate with oracle expectile targets and is supported by simulations with linear and ReLU networks.

Critical strengths
  1. I think the representation analysis is interesting. The paper goes beyond the general intuition that distributional prediction provides richer training signals and gives a concrete connection between reward statistics, singular modes, and hidden representation geometry. This provides a useful way to understand what information the network learns.

  2. The transfer experiment is also interesting. It shows a setting where two states have the same mean reward but different reward distributions, so dRL preserves a state distinction that becomes useful after the reward structure changes. The authors provide an analytical treatment and qualitative validation with ReLU networks.

Critical weaknesses

1. I have a major concern about the connection between the theory and the actual dRL objective. The exact learning dynamics are derived for dRL-MSE, which regresses on oracle state-conditioned expectiles, rather than the asymmetric expectile loss used by dRL. The paper argues that these two objectives share the same optimum, but Appendix C makes this conditional on the expectile map being exactly representable. It is unclear why this condition holds in the experiments, where the inputs are sampled from overlapping state-dependent Gaussian distributions. In general, the expectile of the conditional reward distribution given an input is not equal to the conditional average of the state-wise expectiles. Therefore, the two losses need not have the same optimal prediction function, even before considering the learning dynamics. The simulations show similar behavior, but I do not think they establish the theoretical equivalence needed to transfer the main results from dRL-MSE to dRL.

2. The main motivation is not well aligned with what the theory actually establishes. The Introduction motivates the work using the faster learning and higher performance of dRL agents in sequential RL tasks. However, the theoretical setting is a single-step reward prediction problem without actions, Bellman updates, or policy learning. I understand the motivation for studying a minimal model, but it is unclear how the results explain the performance advantage discussed in the Introduction. The discussion also moves from RL performance to dopamine RPE evidence without clearly explaining how that motivates the particular representation theory. There is also a problem with the learning-speed comparison. Appendix F.1 acknowledges that dRL can have a larger singular value simply because it has more output channels, even when those channels contain no additional distributional information. The main two-state experiment has identical mean rewards, so cRL does not need to distinguish the states to solve its prediction task. I therefore find the connection between faster representation learning and faster learning on a common RL objective insufficiently established. The transfer result is more direct, but it uses a task switch specifically aligned with the state distinction preserved by dRL, so its generality remains unclear.

Overall, I think the paper contains an interesting analysis of representation learning under distributional prediction targets. However, the theoretical connection to actual dRL is not sufficiently justified, and the results do not yet support the broader learning and performance claims used to motivate the paper. These are my main reasons for not supporting acceptance in its current form.

Questions for authors
  1. Clarification of the theoretical assumption: Under what conditions do dRL and dRL-MSE have the same population optimum and hidden representation? In particular, how is the exact-representability assumption in Appendix C satisfied when the state-dependent input distributions overlap? Can the authors justify the equivalence for the experimental setting, rather than only under an additional realizability assumption?

  2. Learning-speed comparison: Can the authors compare dRL with a cRL baseline having the same number of output heads, with controlled loss normalization and gradient scale? More importantly, does dRL learn faster on a common mean-reward prediction objective, rather than only showing faster growth of a distribution-specific singular mode?

  3. Scope and transfer: What evidence supports extending the proposed explanation to sequential RL learning or policy performance? For Section 5, does the transfer advantage persist when the second task requires different input directions, rather than exactly the distinction preserved during the first phase?

Reviewer_vQQvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper applies established SVD techniques for analyzing learning dynamics in linear networks to supervised expectile estimation. This provides an analysis perspective on state representation based on reward distributions.

Critical strengths

None

Critical weaknesses
  • main concern is the limited novelty and potential impact, both for machine learning and neuroscience. The analysis follows a standard approach based on input-output covariance in linear networks, applied to a relatively narrow setting.

  • analysis relies on strong assumptions such as symmetry (line 147), focuses exclusively on expectile estimation, and does not address multi-step dynamics or decision-making, which are central to RL, nor adaptive dopamine dynamics on the neuroscience side.

  • difficult to distinguish predictions specific to dRL from general properties of deep linear networks or existing distributional RL theory.

  • a key claimed result that higher-order reward moments accelerate learning (line 445) is supported by a minuscule, apparently insignificant difference (Fig. 2C), and essentially follows from standard deep linear network theory. The second key claimed result (line 446/447) is already established.

  • dRL and dRL-MSE exhibit different dynamics (Fig. 7B). Analyzing the simplified dRL-MSE model and using dRL networks primarily for validation seems insufficient, especially given the potential importance of baseline offsets in practice.

  • analysis provides no clear practical improvements for dRL or validated novel neuroscience predictions.

Questions for authors

What concrete practical guidance does the analysis provide for implementing dRL?

Can the analysis generate and validate novel predictions about dopamine dynamics during reward learning?

What does this analysis tell us about (multi-step) dRL?

I acknowledge that these questions are primarily suggestions for future work. Perhaps the first question is somewhat addressable at this stage.

Minor: the colored markers in Fig. 2A are difficult to distinguish.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.