acceptodds
Under review as a conference paper at ICLR 2027

A Relative Perspective on Distributional Reinforcement Learning

Abstract

How much better is one action than another? We introduce Relative C51 (R-C51), a distributional reinforcement learning method that directly learns distributions of return differences between state–action pairs. A central obstacle is episode termination. When only one of two sampled transitions terminates, subtracting their Bellman targets requires absolute continuation information unavailable from relative comparisons alone. We address this with Bellman uniformization, a transformation that introduces self-loops and rescales rewards to get a common discount factor while preserving expected action values. For a fixed policy, the resulting unprojected operator is contractive, and its unique fixed point recovers the original action-value differences in expectation, although the full distributions can differ from episodic return differences. R-C51 implements this with a categorical relative critic, a learned continuation head, and an explicit mixture over successor comparisons. On the full 57-game Atari benchmark, R-C51 improves on C51 by approximately % in median score, % in interquartile mean, and % in optimality gap. These results support relative distributional prediction as a promising approach to value-based reinforcement learning.

Primary area:
reinforcement learning
Keywords:
distributional reinforcement learning, relative value learning, atari

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6mDVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper combines distributional and relative value modelling. It proposes to learn distributions of return differences between state–action pairs. As the natural relative Bellman recursion is not closed when the two transitions have different continuations, the authors introduce a modified MDP via a process called "uniformization". This MDP preserves the expected action values but not the return distributions. The work instantiates this idea using C51's categorical representation. The authors report moderate empirical improvements over C51 on several aggregate Atari metrics.

Critical strengths

None

Critical weaknesses

The primary issues, in my opinion, are writing, exposition and motivation. I would invite the authors to rewrite most of the sections from the main part.

  • The motivation section does not provide any motivations ... Why would relative value modelling help ? Why would distributional modelling help ? More importantly : why combining the two ideas be a fruitful idea ? What problems of each does the combination of the two address ?
  • The related works section is very brief and does not provide answers to the questions raised by the motivation section
  • Section 5 is very difficult to read despite the ideas being somewhat simple. Section 5 is not constructive. It is a collection of facts without a red line.
    • The need for self-loops and uniformization needs to be clarified. The alternative using absorbing states seems reasonable. (See question below)
    • The equation 11 drops out of nowhere and hides the Bellman operator that it implicitly defines
    • It is not clear to me, even after several readings, to what extent uniformization modifies the original MDP and its return distributions. Although the expected values are preserved, the distributions are not and the implications are insufficiently discussed. For traditional control where only the expectation matters this should not be an issue but this removes a lot of the interest of distributional modelling with risk-aware policy optimization. If the distributions are distorted, I am not sure the risk in the two MDPs can be interpreted the same way.
    • Overall, the mathematical exposition would be greatly improved by using Definition, Lemma and Proposition blocks.
  • The contraction advertised in the abstract isn't discussed in the main part (despite available space). It is not clear from the main part on what Bellman operator it is proved.
  • The contraction proof is in Wasserstein while the objective used by C51 is based on cross-entropy. This is not a major blocker as this is a standard move. However, showing contraction under the objective that is used in practice would strengthen the contribution.
  • Lack of baseline : all the baselines are variants of C51. There is no other distributional baseline. The core idea of the work is to combine distributional and relative modelling. Yet I could find no ablation removing the distributional component. As far as I understand, you could reuse the ideas you exposed and make a "deterministic"/expectation version. Other potential baselines were mentioned in the related works yet not discussed (i.e dueling networks).
Questions for authors
  • What are the motivations for combining distributional and relative value modelling ?
  • What is the issue with absorbing states ? I could not understand the argument in "A zero-return absorbing reference state would restore closure, but comparisons against it are absolute return distributions.".
3 comments

Dear reviewer.

Critical Strengths:

None

Why none? Look at Figure 4 where our empirical results on Atari-57 surpass C51 by approximately 2121% in median score, 1919% in interquartile mean, and 1212% in optimality gap. Further, what about our theoretical contributions? Is nothing novel in our work?

The motivation section does not provide any motivations ... Why would relative value modelling help ? More importantly : why combining the two ideas be a fruitful idea ? What problems of each does the combination of the two address ?.

We do provide motivation and show there is evidence that biological systems (in particular mammals) are (1) estimating situations relative to each other and (2) represent dopamine-based quantities in a distributional way. Both phenomena appear in brains. In our revised version we additionally added computational motivations like action gaps, zero centering and gauge freedom. E.g. considering gauge freedom, this means the insight that the magnitude of Q-values is irrelevant for decision making (e.g. Q(s1,a1)=100,Q(s1,a2)=101Q(s_1, a_1)=100, Q(s_1, a_2)=101 or Q(s1,a1)=10,Q(s1,a2)=11Q(s_1, a_1)=10, Q(s_1, a_2)=11 chooses both times a2a_2) is directly exploited by relative learning.

I would invite the authors to rewrite most of the sections from the main part [...] Section 5 is very difficult to read despite the ideas being somewhat simple. Section 5 is not constructive. It is a collection of facts without a red line. [...]

Well, the other two reviewers find that our paper is clear and easy to follow (Presentation 3). Further, how is Section 5 a collection of facts? It slowly derives the relative Bellman recursion step by step.

Section 5 is very difficult to read [...] The need for self-loops and uniformization needs to be clarified.

The need for self-loops and uniformization is not motivated in Section 5, but already in Section 4. Then in Equation 9, self loops are characterized (Motivation: terminal states are a problem, remove them from the MDP). And in Equation 10, the same is done for uniformization (Motivation: same as before). Following that, the new Bellman operator is derived.

The equation 11 drops out of nowhere and hides the Bellman operator that it implicitly defines.

Equation 11 is just a Bellman equation in its residual form, as we state in the text before. Rearranging the residual directly exposes self-loops.

It is not clear to me, even after several readings, to what extent uniformization modifies the original MDP and its return distributions.

Both other reviewers find our paper easy to follow. What is not clear to you regarding uniformization and its return distributions? Our paper also includes two figures that directly address your concerns. Figure 1 shows a regular episodic MDP which becomes uniformized. Thereby, terminal states are removed and self-loops are added around these locations. Rewards and transition probabilities are changed to keep expected values. Figure 2 shows what uniformization does to a distribution with multiple modes. What is unclear?

For traditional control where only the expectation matters this should not be an issue but this removes a lot of the interest of distributional modelling with risk-aware policy optimization.

We already discussed this in our limitations section.

The contraction advertised in the abstract isn't discussed in the main part (despite available space).

There is no available space, since the proof takes 1/2 page. The proof is put into the appendix for curious minds and completeness. It gives only marginal additive value in the main text. On top of that, our empirical results show that the new Bellman equation works reliably over all 57 Atari games.

It is not clear from the main part on what Bellman operator it is proved.

It's mentioned in line 321-323. The contraction in Thm. A.3 is for the relative operator in Eq. 23 with help of Proposition A.2.

The contraction proof is in Wasserstein while the objective used by C51 is based on cross-entropy.

We have the same type of guarantee as the original C51 paper. If you criticize our guarantees you also criticize C51's guarantees, which is a highly established paper in the literature. Further, Rowland et al. (2018) analyse the categorical projection via the Cramér distance.

The related works section is very brief and does not provide answers to the questions raised by the motivation section

Yes, we originally put some related work in the appendix. In the revised version, we moved the additional parts back into the main text.

Lack of baseline : all the baselines are variants of C51. There is no other distributional baseline. [...].

C51 is the first established distributional RL algorithm where we add one additional component, namely the relative perspective. By using other distributional baselines, we would introduce additional degrees of freedom w.r.t. design choices and methodology. This is open for future work, but we added new distributional baselines.

References:

  • Relative Value Learning: Höftmann, Marc, Jan Robine, and Stefan Harmeling. "Relative Value Learning." arXiv preprint arXiv:2607.21120 (2026).

Questions:

What are the motivations for combining distributional and relative value modelling ?

See our answers above.

What is the issue with absorbing states ? I could not understand the argument in [...].

When you introduce an additional absorbing state-action pair xabsx_\text{abs}, you can compare every other pair xx against it via Δ(x,xabs)=Gx−0\Delta(x, x_\text{abs}) = G_x - 0 which would recover the absolute distribution directly. From a relative perspective, this makes no sense, because you would learn the absolute distribution implicitly and bring back the gauge freedom.

Best regards.

I thank the authors for their reply, which I address point by point below.

First, I would suggest we keep this conversation civil. I have carefully read your work, and I strongly believe it is flawed in many aspects that I have reconsidered. This is not a personal opinion of the authors.

  • Why no strengths:

    Atari is an extremely noisy benchmark. A 21% jump is not impressive in light of the added complexity of the method. The theoretical contribution is not a strength if poorly presented or misaligned.

  • Motivation:

    I acknowledge the new arguments. My point is that this "motivation" section is very poorly written and does not start with a proper introduction, which is traditional in 99% of papers at these kinds of venues. You do not take proper distance from the problem. You do not invite the reader to accept why your ideas are relevant. Motivating is not ticking a checklist; it is engaging the reader. The link with biology would help had you properly hinted at why this has been a fruitful motivation in the past. At an introductory level, it is not clear, in my opinion.

  • Clarity:

    I acknowledge the other two reviewers' opinions. I would argue that they might have understood your work because the ideas it discusses are simple, even when poorly presented. I am taking the perspective of an average reader, and I maintain my opinion that the presentation is not constructive and does not introduce the concepts in a pedagogical order.

  • Equation 11:

    My concern was not addressed.

  • Uniformization - traditional control and distortion:

    You have not addressed my concerns, which were explicit. I maintain that this is not a limitation but a flaw.

  • Contraction presentation:

    There is a difference between giving a proof in the main part and discussing a result in the main part. It is a core expectation in any new distributional RL work that this is discussed decently in the main part.

  • Contraction in Wasserstein:

    I can criticize C51; many, many works have implicitly done so. The field of distributional RL has matured a lot since its introduction. Even if I did implicitly criticize C51 (according to you), this is a twisted rhetorical argument to say that since it is a respected work and you do something similar, I cannot criticize that aspect of your work. The work as it stands is incomplete. It is missing a proper contraction proof for the divergence used in practice. This is expected of every new paper in this field whose scope is similar to yours. The C51 work had the luxury of not having this expectation placed on it because it was the first and foundational work in this field.

  • Baselines:

    The performance improvement brought by your method should always be compared to similar improvements from other such small modifications. Not testing your idea on at least one different distributional method does not allow you to show that your method could generalize.

Reviewer_YAdQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes R-C51, which directly learns distributions of return differences between state and action pairs. Bellman uniformization handles unequal termination by introducing a common discount while preserving expected value differences. The authors establish contraction for the unprojected operator under a fixed policy and evaluate the method on Atari.

Critical strengths

The idea is clear. The evaluation across all 57 Atari games, with five seeds for the main comparisons, is a strength. The capacity, update frequency, and uniformized C51 ablations are useful. The improvements in median score and IQM are encouraging, although the mean score is slightly lower.

Critical weaknesses

My main concern is why learning difference distributions should be better. The paper explains how to make the relative recursion work, but gives less insight into why this should improve estimation or action ranking. The contraction result establishes convergence of the idealized operator, not an advantage over learning absolute distributions. Since action selection ultimately uses only predicted means, employing a scalar relative critic with the same representation and target construction would be particularly useful for understanding what the distributional objective adds.

The additional computation makes this question more important. With AA actions, a standard critic predicts AA return distributions, whereas Eq. (31) requires O(A2)O(A^2) pair evaluations. I understand that parameters are shared, so this does not mean training a separate model for every pair. Still, comparisons against a common reference would already recover the correct ranking with exact predictions. Averaging over all references may help with approximation errors, but the paper does not show how much it helps. Appendix C.3 reports about 50% longer training, without establishing whether the advantage remains under equal runtime or how action selection costs scale with the number of actions.

The role of the categorical support remains unclear. Figure 8 shows less endpoint mass for R-C51 on [−10,10][-10,10], suggesting that part of the gain could come from reduced support saturation. This is a useful observation, but it leaves open whether the advantage would persist with better support choices for the absolute critics. Wider supports with enough atoms to maintain resolution would provide a more informative comparison.

The continuation model introduces another source of approximation error. Exact value preservation relies on the true continuation expectations, while Eq. (26) uses learned estimates in the denominator 1−c^x1-\hat c_x.

Questions for authors

How do runtime and action selection latency scale with the number of actions?

Does R-C51 improve value difference estimates or action rankings, compared with a scalar relative critic?

How sensitive are the results to errors in the continuation predictions, especially near states where continuation is almost certain?

4 comments

We thank you for the constructive feedback. Based on your positive feedback on presentation (3), informativeness (2) and soundness (3), we are surprised about your chosen rating of 2. Please consider the following points:

The paper explains how to make the relative recursion work, but gives less insight into why this should improve estimation or action ranking.

While we do motivate with biological findings, we now added computational arguments (action gaps and zero centering) for the distributional perspective and reiterate the relative learning motivation regarding gauge freedom. Gauge freedom means in our case: For action rankings the magnitude of Q-values is irrelevant for decision making (e.g. Q(s1,a1)=100,Q(s1,a2)=101Q(s_1, a_1)=100, Q(s_1, a_2)=101 or Q(s1,a1)=10,Q(s1,a2)=11Q(s_1, a_1)=10, Q(s_1, a_2)=11 chooses both times a2a_2). The insight that only the relationship between actions matters is exploited by the relative recursion. Thanks, we added this to the paper.

Since action selection ultimately uses only predicted means, employing a scalar relative critic with the same representation and target construction would be particularly useful for understanding what the distributional objective adds.

This is a good point. In fact, we started working on the scalar relative critic (which we call "R-DQN") before working on R-C51. Sadly, R-DQN does not work better than a DQN baseline, which we hypothesize is due to the more challenging regression task and variance. Our Atari-57 results show that this problem does not occur when considering R-C51.

The contraction result establishes convergence of the idealized operator, not an advantage over learning absolute distributions.

A contraction proof is about convergence properties. Its goal is not to show an advantage over learning absolute distributions, but rather that the algorithm actually does what it should do. More generally, algorithms like DQN and PPO only guarantee convergence of the idealized operator as well. Using a neural network as function approximator breaks this for DQN, PPO, C51 and R-C51.

Still, comparisons against a common reference would already recover the correct ranking with exact predictions. Averaging over all references may help with approximation errors, but the paper does not show how much it helps.

Fair point. We did run a three-seed ablation over all 57 games where we use only one action (always the action with index 0) as the only reference xb=(s,a0)x_b = (s,a_0). The performance drops noticeably. In terms of aggregated metrics, the ablation reaches 0.8480.848 (−23.7-23.7%) in Median, 0.9840.984 (−14.8-14.8%) in IQM, 2.792.79 (−15.2-15.2%) in Mean and 0.3460.346 (+10.2+10.2%) in optimality gap. We also added a figure (see Figure 7) to the main text for this ablation.

The role of the categorical support remains unclear. Figure 8 shows less endpoint mass for R-C51 on [−10,10][-10,10], suggesting that part of the gain could come from reduced support saturation. [...].

Note that we did not fine-tune the support range for our algorithm. Instead, the interval [−10,10][-10,10] is chosen by Bellemare et al. (C51) themselves in their original work and our R-C51 sticks to this choice for fair comparison. For this rebuttal, we ablate C51 with a wider support range of [−15,15][-15, 15] and did not observe any improvements (see Appendix, Figure 10). All aggregated metrics are lower.

The continuation model introduces another source of approximation error. Exact value preservation relies on the true continuation expectations, while Eq. (26) uses learned estimates in the denominator.

Correct, and we mentioned this already in our limitations section. But remember, this is ALWAYS the case when using neural networks as function approximators. By raising this argument, you also criticize the complete model-based RL Dreamer paper series, most notably DreamerV3. DreamerV3 also learns episode termination with a neural network (see continue predictor in Equation 1 of the DreamerV3 paper) and then uses these estimates to learn a value function (actor-critic style) using rollouts in imagination. The continue predictor is exactly what we are doing as well. Yet, DreamerV3 is considered state-of-the-art and a highly influential paper in the literature. Despite "another source of approximation error", our algorithm improves on C51 and the operator stays a γˉ\bar{\gamma}-contraction whatever the error in c^\hat{c} is.

References:

  • C51: Bellemare, Marc G., Will Dabney, and Rémi Munos. "A distributional perspective on reinforcement learning." International conference on machine learning. PMLR, 2017.
  • DQN: Mnih, Volodymyr, et al. "Human-level control through deep reinforcement learning." Nature 518.7540 (2015): 529-533.
  • PPO: Schulman, John, et al. "Proximal policy optimization algorithms." arXiv preprint arXiv:1707.06347 (2017).
  • DreamerV3: Hafner, Danijar, et al. "Mastering diverse domains through world models." arXiv preprint arXiv:2301.04104 (2023).
  • Relative Value Learning: Höftmann, Marc, Jan Robine, and Stefan Harmeling. "Relative Value Learning." arXiv preprint arXiv:2607.21120 (2026).

Questions

Does R-C51 improve value difference estimates or action rankings, compared with a scalar relative critic?

R-DQN (scalar relative critic) did not work well compared to DQN, so we abandoned it. Our Atari-57 experiments for R-C51 show a good performance.

How sensitive are the results to errors in the continuation predictions, especially near states where continuation is almost certain?

The Atari-57 experiments with 5 seeds for R-C51 show better performance than C51. Therefore the continuation predictions are reliable since C51 does not learn these predictions. In addition, C51-U uses the same learned continuation head and still improves over C51. The operator stays a γˉ\bar{\gamma}-contraction whatever the error in c^\hat{c} is.

Thank you for your time. We hope that we properly solved your concerns. Please reconsider your score. Best regards.

Thanks for the response and the extra efforts. I curious about that whether learning pairwise distributions offers enough benefit over other ways of learning return distributions. Could you include a comparison with QR-DQN or IQN under the same interaction budget and comparable tuning? These methods do not rely on C51’s fixed support, so they would be useful baselines given the discussion about support saturation. I’m not suggesting that R-C51 has to beat every existing method, but comparing only with C51 leaves its broader practical benefit unclear.

I’m also still wondering about larger action spaces. As I read Eq. (31), computing all action scores involves comparing every action with every other action. I understand that these predictions can share features and be computed in a batch, but how much overhead does this add as the number of actions grows?

Thanks again for your reply!

Could you include a comparison with QR-DQN or IQN under the same interaction budget and comparable tuning? [...]

Yes. We added QR-DQN and IQN to the main figure (Figure 4) using official Google DeepMind results (see reference below), where the authors describe their implementations as "best-effort replication of the corresponding paper implementation". The interaction budget (40M frames, eval with ϵ=0.001\epsilon=0.001) is the same as ours. While R-C51 is competitive with QR-DQN, IQN is better. Note that C51 is still the closest baseline to our algorithm.

I’m also still wondering about larger action spaces. As I read Eq. (31), computing all action scores involves comparing every action with every other action. [...]

Sure. For all following measurements, we consider an H200 MIG slice (1g.18gb, $\approx$1/7 of an H200).

Correct. Eq. (31) is currently quadratic in A (action space), but only in the two-layer MLP head at the end. The CNN encoder runs once per state and the action embeddings once per action. E.g., on Atari ∣A∣≤18|A| \leq 18 (324324 pairs) this adds about 0.03 ms per acting step and makes a training update ~1.6x slower than using a single reference.

The cost becomes significant beyond roughly 64 actions. Going from ∣A∣=64|A|=64 to ∣A∣=128|A|=128 multiplies the update time by 3.7, so a training update becomes 8x slower than with a single reference at ∣A∣=64|A|=64 (17.617.6 vs 2.22.2 ms) and 25x slower at ∣A∣=128|A|=128 (64.664.6 vs 2.72.7 ms). For a batch with ∣A∣=256|A|=256, we run out of memory on 18 GB VRAM.

Regarding your suggested ablation on action selection: the overhead is mostly a design choice. Eq. (31) does not need to use all actions of A. One or k≤∣A∣k\leq|A| sampled references work too. E.g. for ∣A∣=1024|A|=1024, 1212 ms per update with one action, or 6565 ms with k=12k=12.


References:

Google DeepMind algorithm baselines: https://github.com/google-deepmind/dqn_zoo

Best regards.

Reviewer_a6ASOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies relative action-value estimation in distributional reinforcement learning. It proposes Relative C51 (R-C51), which directly learns distributions of return differences between state–action pairs, rather than the absolute return distributions modeled by standard C51.

Critical strengths

The paper addresses a meaningful problem. Directly learning distributions of return differences between state–action pairs offers an alternative to the absolute return distributions used by standard C51. The authors also introduce Bellman uniformization to handle relative Bellman updates when the two transitions have different termination outcomes, supported by theoretical analysis. The experimental evaluation is fairly comprehensive, covering all 57 Atari games.

Critical weaknesses
  • Errors in continuation probability estimates may affect the accuracy of relative action values, but the paper lacks sufficient analysis and experimental validation.

  • R-C51 introduces an additional network to predict continuation probabilities in order to handle episode termination in the relative Bellman recursion. These predictions are used to compute reward scaling factors and successor transition probabilities. However, the theoretical analysis assumes exact expected transition discounts, while the implementation relies on estimates from a neural network. There is therefore a gap between the theoretical assumptions and the actual implementation.

Questions for authors
  • The theoretical analysis assumes exact expected transition discounts, but R-C51 estimates continuation probabilities using a neural network. Could the authors clarify whether errors in these estimates affect relative value estimates and, consequently, the performance of the final policy?
3 comments

Dear reviewer, thank you for the assessment. Based on your review, the only criticism we can identify is the prediction head that estimates continuation probabilities and its properties. This is only a part of our paper and we are surprised that you choose rating 2 for that.

While you acknowledge that our "[...] experimental evaluation is fairly comprehensive, covering all 57 Atari games.", we want to highlight again that our empirical results surpass C51 by approximately 2121% in median score, 1919% in interquartile mean, and 1212% in optimality gap.

Errors in continuation probability estimates may affect the accuracy of relative action values, but the paper lacks sufficient analysis and experimental validation.

Our experiments are done on the complete 57-game Atari benchmark with 5 seeds. During these experiments, the continuation head is obviously learned and our algorithm is still improving compared to the C51 baseline. We find it hard to understand why you call this an insufficient experimental validation while stating that "[...] experimental evaluation is fairly comprehensive".

R-C51 introduces an additional network to predict continuation probabilities in order to handle episode termination in the relative Bellman recursion. These predictions are used to compute reward scaling factors and successor transition probabilities. However, the theoretical analysis assumes exact expected transition discounts, while the implementation relies on estimates from a neural network. There is therefore a gap between the theoretical assumptions and the actual implementation.

Correct, and we mentioned this already in our limitations. But remember, this is ALWAYS the case when using neural networks as function approximators. By raising this argument, you also criticize the complete model-based RL Dreamer paper series, most notably DreamerV3. DreamerV3 also learns episode termination with a neural network (see continue predictor in Equation 1 of the DreamerV3 paper) and then uses these estimates to learn the value function (actor-critic style) using rollouts in imagination. The continue predictor is exactly what we are doing as well. Yet, DreamerV3 is considered state-of-the-art and a highly influential paper in the literature. Further, the operator stays a γˉ\bar{\gamma}-contraction whatever the error in c^\hat{c} is.

References:

  • DreamerV3: Hafner, Danijar, et al. "Mastering diverse domains through world models." arXiv preprint arXiv:2301.04104 (2023).
  • DQN: Mnih, Volodymyr, et al. "Human-level control through deep reinforcement learning." Nature 518.7540 (2015): 529-533.
  • PPO: Schulman, John, et al. "Proximal policy optimization algorithms." arXiv preprint arXiv:1707.06347 (2017).

Questions

The theoretical analysis assumes exact expected transition discounts, but R-C51 estimates continuation probabilities using a neural network. Could the authors clarify whether errors in these estimates affect relative value estimates and, consequently, the performance of the final policy?

We think that we addressed your question with our comments above. The relative value estimates are robust since we improve over the C51 baseline on the 57-game Atari benchmark. In addition, C51-U uses the same learned continuation head and still improves over C51. The operator stays a γˉ\bar{\gamma}-contraction whatever the error in c^\hat{c} is.

Thanks for your time. We hope that your concerns have been properly addressed and would be happy to see a raise in score. Best regards.

Thanks for the response and the extra efforts. I would like to further clarify that my concern is not that neural networks inevitably have prediction errors and therefore cannot be used in reinforcement learning. My concern is that the theoretical analysis relies on the exact expected transition discount, while the actual implementation uses a continuation probability predicted by a neural network to estimate this quantity.

The response addresses this concern by showing the overall performance improvement on Atari-57. However, I do not think this resolves my concern. The improved overall performance only shows that the complete algorithm achieves good results in the experiments. It does not show that prediction errors from the Continuation Head have no effect on value estimation. It also does not guarantee that the original value-preservation property still holds when the exact value is replaced by an approximation.

Similarly, I find it puzzling to interpret this concern as a criticism of the general use of neural network approximation in reinforcement learning. The paper first derives the value-preservation property of Bellman Uniformization using the exact expected transition discount. This exact quantity is then used to determine reward scaling and successor-state weights, so that the required mathematical relationship is satisfied. However, in the actual implementation, this exact quantity is replaced with a neural network prediction. The paper also states that the learned continuation model and function approximation lie beyond the scope of the original theoretical guarantee.

Without further analysis of the errors introduced by this approximation and their effects, how can the original value-preservation property be shown to still apply to the actual algorithm? Is it reasonable to rely on performance improvements from an approximate implementation that falls outside the scope of the theoretical guarantee? Since the actual implementation falls outside the scope of the original theoretical guarantee, at least further analysis is needed to examine the errors introduced by this approximation and their effects on the original value-preservation relationship and actual value estimation.

I would like to see an error analysis of this specific approximation, together with experiments that examine how these errors affect policy performance. Without such analysis and experiments, I currently have no reason to increase my score.

Thanks for the clarification. We have started to work on the theoretical analysis + experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.