- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies the non-convergence of return-distribution estimates in distributional RL control. When multiple optimal actions have the same expected return but different return distributions, small value-estimation errors can cause greedy action selection to keep switching, even when the estimated action values converge. The paper proposes double distributional value iteration (DDVI). An auxiliary distributional estimator provides action-value estimates for a softmax reference policy with a decreasing temperature, while the main estimator performs distributional policy evaluation under that policy. The authors prove convergence to the return law of the tie-uniform optimal policy in tabular planning and stochastic approximation settings. Experiments on tabular MDPs and three MuJoCo tasks evaluate policy selection, distributional stability, and return performance.
Critical strengths
-
The problem is meaningful, and the proposed separation is reasonably motivated. The paper addresses a known difficulty of distributional control: convergence of expected values does not necessarily imply convergence of return distributions. Using a smooth reference policy for the main distributional update is a natural way to avoid abrupt greedy action switches.
-
The tabular planning result is interesting. The method selects a specific optimal policy without requiring a unique optimal action, and the analysis connects the cooling schedule, value-estimation error, and distributional convergence. The tabular experiments also show that DDVI can approach the return law of the tie-uniform optimal policy.
Critical weaknesses
1. I have concerns about the proof of the stochastic approximation result. Lemma 8 in Appendix B appears false as stated. In its recursion, an additive perturbation satisfying only is not sufficient for convergence. For example, , , and satisfy the stated conditions but do not give . The argument may be repairable if the perturbation is scaled by the stepsize, but the current lemma does not justify Theorem 2. I also find Eq. (11) unclear. Algorithm 1 updates the auxiliary estimator with the current transition before constructing the reference policy. Therefore, the reference policy depends on the same sample used in the main update, and the stated conditional expectation does not seem to follow from the given filtration. These issues directly affect the claimed almost-sure convergence guarantee.
2. The experiments do not sufficiently establish the claimed non-convergence resolution or justify the extra estimator. In the tabular experiments, the Wasserstein error is measured against the return law of the tie-uniform optimal policy. A greedy method with deterministic tie-breaking may instead converge to a different optimal policy and its return law. In particular, the nonzero error reported for the greedy baseline in Appendix D.5 does not by itself demonstrate persistent distributional oscillation. I think an experiment showing actual non-convergence of greedy distributional updates would be more convincing.
I also find the baseline comparison insufficient. The auxiliary distributional estimator is only used through its mean, which follows standard Q-learning in the tabular setting. It is unclear why a scalar Q auxiliary estimator, or a simpler annealed-softmax C51 variant, would not provide the same benefit. The closest related convergence approach using entropy regularization [1] is discussed but not compared experimentally. DDVI also requires an additional categorical network, yet no computational-cost comparison is reported. The MuJoCo results are mixed across only three seeds: Hopper has a lower mean return and higher standard deviation than C51, while the improvement on Reacher is small. Without stronger baselines and a comparison under matched computational budgets, it is difficult to assess whether the additional complexity is justified.
[1] Yash Jhaveri, et al. Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning. NeurIPS, 2025.
Questions for authors
Please see Critical Weaknesses 1 and 2. I think resolving the proof issues and providing more direct evidence of non-convergence resolution, together with the simpler baselines and computational costs, could materially affect my assessment.
5 comments
We now continue addressing your remaining questions as follows.
3.
The experiments do not sufficiently establish the claimed non-convergence resolution or justify the extra estimator. In the tabular experiments, the Wasserstein error is measured against the return law of the tie-uniform optimal policy. A greedy method with deterministic tie-breaking may instead converge to a different optimal policy and its return law. In particular, the nonzero error reported for the greedy baseline in Appendix D.5 does not by itself demonstrate persistent distributional oscillation. I think an experiment showing actual non-convergence of greedy distributional updates would be more convincing.
We agree that the metric in Appendix D.5 is unsuitable for diagnosing non-convergence: a nonzero Wasserstein distance to the tie-uniform optimal policy's return law may simply reflect convergence to another optimal policy and its return law. We will replace this diagnostic with changes between consecutive policies and consecutive return-distribution estimates.
Our supplementary experiment uses a toy MDP with two optimal actions that have equal expected returns but different return distributions. We compare C51-style DVI and DDVI using 2,000 exact model-based updates and 20 paired initializations. Policy TV measures changes between consecutive reference policies at the decision state, and adjacent measures changes between consecutive return-distribution estimates at the start state. Both metrics are averaged over the final 300 iterations and across initializations.
| Method | Return | Policy TV | Adjacent |
|---|---|---|---|
| C51-style DVI | 49.005 | 1.000 | 4.65 |
| DDVI | 49.005 |
Both methods attain the same expected return, but C51-style DVI continues to switch policies and return-distribution estimates over the observed window, whereas DDVI is much more stable. In this toy MDP, switching between equally good optimal actions preserves expected return. With function approximation, however, fitting a changed return-distribution target can also change estimates at other state–action pairs through shared parameters. This can alter action choices elsewhere, so the effects need not remain confined to equally good optimal actions and may also affect returns. Our motivation is therefore to stabilize policy selection and return-distribution learning while learning an optimal policy.
In deep RL, Schaul et al. (2022) observed frequent changes in greedy actions during training. Tang and Berseth (2024) found that limiting changes to predictions outside the current training batch improved DoubleDQN's returns on MinAtar. We will add deep RL experiments measuring policy changes, changes in the learned return distributions, and control returns to examine these effects with function approximation.
4.
I also find the baseline comparison insufficient. The auxiliary distributional estimator is only used through its mean, which follows standard Q-learning in the tabular setting. It is unclear why a scalar Q auxiliary estimator, or a simpler annealed-softmax C51 variant, would not provide the same benefit.
Scalar Q auxiliary. You are correct that a scalar Q-function can replace the auxiliary distribution in the tabular setting, with the same convergence argument under our assumptions. In deep RL, we keep both branches distributional so that they use the same representation and loss. This is intended to keep their learning dynamics similar as the auxiliary branch guides the main branch.
Single-distribution annealed softmax. This alternative uses the main estimator's own values to select actions with a gradually cooled softmax. We retain the auxiliary branch because proving convergence of this alternative with sampled updates is technically very challenging. An unusually high sampled target can make an action look better and increase its softmax weight. This changes the targets in later updates, feeding sampling fluctuations back into the same estimator. Cooling can amplify small random value differences into large policy changes. Controlling this coupled randomness throughout cooling is the central theoretical difficulty.
We compare both alternatives with DDVI in the same supplementary toy MDP experiment, using 2,000 outer iterations, 20 paired initializations, and the final-300-iteration metrics defined above.
| Method | Return | Policy TV | Adjacent |
|---|---|---|---|
| C51-style DVI | 49.005 | 1.000 | 4.65 |
| Scalar-auxiliary DDVI | 49.005 | ||
| Single-distribution softmax | 49.005 | ||
| DDVI | 49.005 |
Both alternatives achieve the same return and similar stability in this tabular example. We will include these baselines in our additional deep RL experiments and compare their performance with DDVI.
5.
The closest related convergence approach using entropy regularization [1] is discussed but not compared experimentally.
Jhaveri et al. (2025) first run entropy-regularized control updates at a fixed temperature, then construct a policy with a decoupled temperature and hold it fixed during distributional evaluation. DDVI updates the reference policy and return distributions together throughout training, using unregularized backups while gradually reducing the policy temperature. We will consider adding a comparison with their two-stage implementation in Section 4.3 on the same tabular MDPs, counting the updates in both of their stages and matching the total number of Bellman updates.
6.
DDVI also requires an additional categorical network, yet no computational-cost comparison is reported.
We recorded elapsed execution times during our experiments but did not include them in the submitted manuscript. For 500k environment steps, DDVI's mean runtime across three seeds was 1.58 times that of C51 on Swimmer, 1.54 times on Hopper, and 1.62 times on Reacher. The additional estimator thus incurs a constant-factor overhead in these runs, with total runtime remaining below twice that of C51. We will add these timing results and the measurement protocol to the revised manuscript.
7.
The MuJoCo results are mixed across only three seeds: Hopper has a lower mean return and higher standard deviation than C51, while the improvement on Reacher is small.
The return difference on Hopper changes sign across seeds. We will extend the comparison with more seeds and include the scalar-auxiliary and single-distribution softmax baselines, using the same evaluation protocol and reporting confidence intervals for the return differences. We will also measure changes between consecutive policies and return-distribution estimates to examine whether improved distributional stability translates into better control performance.
Thank you for acknowledging the motivation and theoretical contribution of our work, and for your constructive feedback. We respond to each point below.
1.
Thank you for carefully checking this step of the proof. We agree that convergence to zero alone would not suffice for an unscaled additive perturbation. However, the required stepsize scaling is already present in Lemma 8 of the submitted manuscript. Specifically, Eq. (74) reads:
(1-\alpha_t(s,a))d^{(t)}(s,a) +\alpha_t(s,a)\bigl[\gamma D^{(t)}+r_t\bigr].$$ Thus, the perturbation enters as $\alpha_t(s,a)r_t$, and this scaling is also retained in Eq. (86), where Lemma 8 is applied. We will make the placement of this factor clearer in both statements. ### 2. > I also find Eq. (11) unclear. Algorithm 1 updates the auxiliary estimator with the current transition before constructing the reference policy. Therefore, the reference policy depends on the same sample used in the main update, and the stated conditional expectation does not seem to follow from the given filtration. These issues directly affect the claimed almost-sure convergence guarantee. In the current ordering of Algorithm 1, the reference policy depends on the transition used in the main update. We agree that Eq. (11) therefore does not follow from the stated filtration. We will revise Algorithm 1 to construct the reference policy from the auxiliary law $\nu_t$ before sampling the current transition, and keep this policy fixed throughout the update: $$\begin{aligned} q_t(s,a) &=\mathbb{E}_{Z\sim\nu_t(s,a)}[Z], \\\\ \pi^{(t+1)}(a\mid s) &=\frac{\exp(q_t(s,a)/\lambda_t)}{\sum_b\exp(q_t(s,b)/\lambda_t)}. \end{aligned}$$ We then sample $(R_t,S'_t)$. Both branches may still share this transition, with both targets computed from $\nu_t$ and $\eta_t$ before either law is updated. The main target action is sampled from $\pi^{(t+1)}(\cdot\mid S'_t)$. Let $\mathcal F_t$ contain the information available before sampling the transition, including $S_t,A_t,\nu_t,\eta_t,\lambda_t$, and the stepsizes. The reference policy $\pi^{(t+1)}$ is then $\mathcal F_t$-measurable. For $$\begin{aligned} \widehat{\eta}^{(t+1)} &=(b_{R_t,\gamma})_{\\#}\eta_t(S'_t,B_t). \end{aligned}$$ the tower property gives $$\begin{aligned} \mathbb{E}[\widehat{\eta}^{(t+1)}\mid\mathcal{F}_t] &= \mathbb{E}\\!\left[ \sum_b\pi^{(t+1)}(b\mid S'_t) (b_{R_t,\gamma})_{\\#}\eta_t(S'_t,b) \\,\middle|\\,\mathcal{F}_t \right] \\\\ &=(\mathcal{T}^{\pi^{(t+1)}}\eta_t)(S_t,A_t). \end{aligned}$$ This establishes Eq. (11) as an equality of measures and justifies the zero conditional mean of the sampling noise used in Appendix B. With this correction, the remainder of the convergence proof applies unchanged, yielding the same almost-sure convergence guarantee under the stated assumptions. In particular, Eq. (59) already uses $Q^{(t)}$ together with $\lambda_t$, consistent with Assumption 3 and the revised policy definition. The reported experimental results are also unaffected: the tabular experiments use model-based planning updates, which do not require this correction, while Algorithm 2, used for the deep RL experiments, already constructs the reference policy and both targets before either network is updated. We will update Algorithm 1, the stochastic approximation description in Section 3.2, Assumption 1, and the corresponding statements in Appendix B to make this ordering explicit and consistent throughout. Due to the length limitation of each comment, we will address the remaining questions in the following comment.