- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper combines distributional and relative value modelling. It proposes to learn distributions of return differences between state–action pairs. As the natural relative Bellman recursion is not closed when the two transitions have different continuations, the authors introduce a modified MDP via a process called "uniformization". This MDP preserves the expected action values but not the return distributions. The work instantiates this idea using C51's categorical representation. The authors report moderate empirical improvements over C51 on several aggregate Atari metrics.
Critical strengths
None
Critical weaknesses
The primary issues, in my opinion, are writing, exposition and motivation. I would invite the authors to rewrite most of the sections from the main part.
- The motivation section does not provide any motivations ... Why would relative value modelling help ? Why would distributional modelling help ? More importantly : why combining the two ideas be a fruitful idea ? What problems of each does the combination of the two address ?
- The related works section is very brief and does not provide answers to the questions raised by the motivation section
- Section 5 is very difficult to read despite the ideas being somewhat simple. Section 5 is not constructive. It is a collection of facts without a red line.
- The need for self-loops and uniformization needs to be clarified. The alternative using absorbing states seems reasonable. (See question below)
- The equation 11 drops out of nowhere and hides the Bellman operator that it implicitly defines
- It is not clear to me, even after several readings, to what extent uniformization modifies the original MDP and its return distributions. Although the expected values are preserved, the distributions are not and the implications are insufficiently discussed. For traditional control where only the expectation matters this should not be an issue but this removes a lot of the interest of distributional modelling with risk-aware policy optimization. If the distributions are distorted, I am not sure the risk in the two MDPs can be interpreted the same way.
- Overall, the mathematical exposition would be greatly improved by using Definition, Lemma and Proposition blocks.
- The contraction advertised in the abstract isn't discussed in the main part (despite available space). It is not clear from the main part on what Bellman operator it is proved.
- The contraction proof is in Wasserstein while the objective used by C51 is based on cross-entropy. This is not a major blocker as this is a standard move. However, showing contraction under the objective that is used in practice would strengthen the contribution.
- Lack of baseline : all the baselines are variants of C51. There is no other distributional baseline. The core idea of the work is to combine distributional and relative modelling. Yet I could find no ablation removing the distributional component. As far as I understand, you could reuse the ideas you exposed and make a "deterministic"/expectation version. Other potential baselines were mentioned in the related works yet not discussed (i.e dueling networks).
Questions for authors
- What are the motivations for combining distributional and relative value modelling ?
- What is the issue with absorbing states ? I could not understand the argument in "A zero-return absorbing reference state would restore closure, but comparisons against it are absolute return distributions.".
3 comments
The contraction proof is in Wasserstein while the objective used by C51 is based on cross-entropy.
We have the same type of guarantee as the original C51 paper. If you criticize our guarantees you also criticize C51's guarantees, which is a highly established paper in the literature. Further, Rowland et al. (2018) analyse the categorical projection via the Cramér distance.
The related works section is very brief and does not provide answers to the questions raised by the motivation section
Yes, we originally put some related work in the appendix. In the revised version, we moved the additional parts back into the main text.
Lack of baseline : all the baselines are variants of C51. There is no other distributional baseline. [...].
C51 is the first established distributional RL algorithm where we add one additional component, namely the relative perspective. By using other distributional baselines, we would introduce additional degrees of freedom w.r.t. design choices and methodology. This is open for future work, but we added new distributional baselines.
References:
- Relative Value Learning: Höftmann, Marc, Jan Robine, and Stefan Harmeling. "Relative Value Learning." arXiv preprint arXiv:2607.21120 (2026).
Questions:
What are the motivations for combining distributional and relative value modelling ?
See our answers above.
What is the issue with absorbing states ? I could not understand the argument in [...].
When you introduce an additional absorbing state-action pair , you can compare every other pair against it via which would recover the absolute distribution directly. From a relative perspective, this makes no sense, because you would learn the absolute distribution implicitly and bring back the gauge freedom.
Best regards.
I thank the authors for their reply, which I address point by point below.
First, I would suggest we keep this conversation civil. I have carefully read your work, and I strongly believe it is flawed in many aspects that I have reconsidered. This is not a personal opinion of the authors.
-
Why no strengths:
Atari is an extremely noisy benchmark. A 21% jump is not impressive in light of the added complexity of the method. The theoretical contribution is not a strength if poorly presented or misaligned.
-
Motivation:
I acknowledge the new arguments. My point is that this "motivation" section is very poorly written and does not start with a proper introduction, which is traditional in 99% of papers at these kinds of venues. You do not take proper distance from the problem. You do not invite the reader to accept why your ideas are relevant. Motivating is not ticking a checklist; it is engaging the reader. The link with biology would help had you properly hinted at why this has been a fruitful motivation in the past. At an introductory level, it is not clear, in my opinion.
-
Clarity:
I acknowledge the other two reviewers' opinions. I would argue that they might have understood your work because the ideas it discusses are simple, even when poorly presented. I am taking the perspective of an average reader, and I maintain my opinion that the presentation is not constructive and does not introduce the concepts in a pedagogical order.
-
Equation 11:
My concern was not addressed.
-
Uniformization - traditional control and distortion:
You have not addressed my concerns, which were explicit. I maintain that this is not a limitation but a flaw.
-
Contraction presentation:
There is a difference between giving a proof in the main part and discussing a result in the main part. It is a core expectation in any new distributional RL work that this is discussed decently in the main part.
-
Contraction in Wasserstein:
I can criticize C51; many, many works have implicitly done so. The field of distributional RL has matured a lot since its introduction. Even if I did implicitly criticize C51 (according to you), this is a twisted rhetorical argument to say that since it is a respected work and you do something similar, I cannot criticize that aspect of your work. The work as it stands is incomplete. It is missing a proper contraction proof for the divergence used in practice. This is expected of every new paper in this field whose scope is similar to yours. The C51 work had the luxury of not having this expectation placed on it because it was the first and foundational work in this field.
-
Baselines:
The performance improvement brought by your method should always be compared to similar improvements from other such small modifications. Not testing your idea on at least one different distributional method does not allow you to show that your method could generalize.
Dear reviewer.
Why none? Look at Figure 4 where our empirical results on Atari-57 surpass C51 by approximately 21% in median score, 19% in interquartile mean, and 12% in optimality gap. Further, what about our theoretical contributions? Is nothing novel in our work?
We do provide motivation and show there is evidence that biological systems (in particular mammals) are (1) estimating situations relative to each other and (2) represent dopamine-based quantities in a distributional way. Both phenomena appear in brains. In our revised version we additionally added computational motivations like action gaps, zero centering and gauge freedom. E.g. considering gauge freedom, this means the insight that the magnitude of Q-values is irrelevant for decision making (e.g. Q(s1,a1)=100,Q(s1,a2)=101 or Q(s1,a1)=10,Q(s1,a2)=11 chooses both times a2) is directly exploited by relative learning.
Well, the other two reviewers find that our paper is clear and easy to follow (Presentation 3). Further, how is Section 5 a collection of facts? It slowly derives the relative Bellman recursion step by step.
The need for self-loops and uniformization is not motivated in Section 5, but already in Section 4. Then in Equation 9, self loops are characterized (Motivation: terminal states are a problem, remove them from the MDP). And in Equation 10, the same is done for uniformization (Motivation: same as before). Following that, the new Bellman operator is derived.
Equation 11 is just a Bellman equation in its residual form, as we state in the text before. Rearranging the residual directly exposes self-loops.
Both other reviewers find our paper easy to follow. What is not clear to you regarding uniformization and its return distributions? Our paper also includes two figures that directly address your concerns. Figure 1 shows a regular episodic MDP which becomes uniformized. Thereby, terminal states are removed and self-loops are added around these locations. Rewards and transition probabilities are changed to keep expected values. Figure 2 shows what uniformization does to a distribution with multiple modes. What is unclear?
We already discussed this in our limitations section.
There is no available space, since the proof takes 1/2 page. The proof is put into the appendix for curious minds and completeness. It gives only marginal additive value in the main text. On top of that, our empirical results show that the new Bellman equation works reliably over all 57 Atari games.
It's mentioned in line 321-323. The contraction in Thm. A.3 is for the relative operator in Eq. 23 with help of Proposition A.2.