acceptodds
Under review as a conference paper at ICLR 2027

Resolving Non-Convergence in Distributional Control: A Homotopy Approach with Double Distributional Value Iteration

Abstract

Distributional reinforcement learning models the full probability distribution of cumulative returns rather than just their expected values. In control settings, standard algorithms select actions using a greedy policy based on the mean returns of the current distributional estimates. However, when multiple actions have the same optimal expected return but different return distributions, small estimation errors in their means cause the greedy policy to abruptly jump back and forth between these actions. Because the policy keeps switching, the target return distributions for Bellman updates oscillate continuously, preventing the estimated return distributions from converging even after value errors become negligible. To resolve this chattering issue, we provide an algorithm,at each step, an auxiliary update estimates action values and turns them into a softmax reference policy, and a main update learns the return distribution by distributional Bellman evaluation under that policy. The softmax is evaluated at a positive temperature λt. Cooling the temperature to zero throughout training brings the policy back to the optimal actions and, on a tie, shares its mass uniformly over them, while the objective and the return target stay unchanged. Theoretically, for tabular MDPs, we prove for the planning version of our algorithm that the softmax policy converges to the uniform distribution over optimal actions and that the return distributions converge to its law in maximal 1-Wasserstein distance. For its stochastic approximation version, we prove almost-sure convergence to the same policy and return law. We provide a practical implementation using categorical networks. Experiments in tabular environments confirm that algorithm eliminates distributional oscillation and guarantees convergence, while continuous-control benchmarks in MuJoCo demonstrate improved distributional stability during training.

Primary area:
reinforcement learning
Keywords:
Distributional Reinforcement Learning, Double Distributional Value Iteration, Convergence Guarantees, Homotopic Policy Optimization, Reference Policies, Vanishing Regularization, Non-Unique Optimal Policies, Return Distribution Convergence, Policy Stabilization, Markov Decision Processes

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LSmYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies the non-convergence of return-distribution estimates in distributional RL control. When multiple optimal actions have the same expected return but different return distributions, small value-estimation errors can cause greedy action selection to keep switching, even when the estimated action values converge. The paper proposes double distributional value iteration (DDVI). An auxiliary distributional estimator provides action-value estimates for a softmax reference policy with a decreasing temperature, while the main estimator performs distributional policy evaluation under that policy. The authors prove convergence to the return law of the tie-uniform optimal policy in tabular planning and stochastic approximation settings. Experiments on tabular MDPs and three MuJoCo tasks evaluate policy selection, distributional stability, and return performance.

Critical strengths
  1. The problem is meaningful, and the proposed separation is reasonably motivated. The paper addresses a known difficulty of distributional control: convergence of expected values does not necessarily imply convergence of return distributions. Using a smooth reference policy for the main distributional update is a natural way to avoid abrupt greedy action switches.

  2. The tabular planning result is interesting. The method selects a specific optimal policy without requiring a unique optimal action, and the analysis connects the cooling schedule, value-estimation error, and distributional convergence. The tabular experiments also show that DDVI can approach the return law of the tie-uniform optimal policy.

Critical weaknesses

1. I have concerns about the proof of the stochastic approximation result. Lemma 8 in Appendix B appears false as stated. In its recursion, an additive perturbation satisfying only rt→0r_t \to 0 is not sufficient for convergence. For example, dt=1d_t=1, αt=1/(t+1)\alpha_t=1/(t+1), and rt=(1−γ)/(t+1)r_t=(1-\gamma)/(t+1) satisfy the stated conditions but do not give dt→0d_t\to0. The argument may be repairable if the perturbation is scaled by the stepsize, but the current lemma does not justify Theorem 2. I also find Eq. (11) unclear. Algorithm 1 updates the auxiliary estimator with the current transition before constructing the reference policy. Therefore, the reference policy depends on the same sample used in the main update, and the stated conditional expectation does not seem to follow from the given filtration. These issues directly affect the claimed almost-sure convergence guarantee.

2. The experiments do not sufficiently establish the claimed non-convergence resolution or justify the extra estimator. In the tabular experiments, the Wasserstein error is measured against the return law of the tie-uniform optimal policy. A greedy method with deterministic tie-breaking may instead converge to a different optimal policy and its return law. In particular, the nonzero error reported for the greedy baseline in Appendix D.5 does not by itself demonstrate persistent distributional oscillation. I think an experiment showing actual non-convergence of greedy distributional updates would be more convincing.

I also find the baseline comparison insufficient. The auxiliary distributional estimator is only used through its mean, which follows standard Q-learning in the tabular setting. It is unclear why a scalar Q auxiliary estimator, or a simpler annealed-softmax C51 variant, would not provide the same benefit. The closest related convergence approach using entropy regularization [1] is discussed but not compared experimentally. DDVI also requires an additional categorical network, yet no computational-cost comparison is reported. The MuJoCo results are mixed across only three seeds: Hopper has a lower mean return and higher standard deviation than C51, while the improvement on Reacher is small. Without stronger baselines and a comparison under matched computational budgets, it is difficult to assess whether the additional complexity is justified.

[1] Yash Jhaveri, et al. Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning. NeurIPS, 2025.

Questions for authors

Please see Critical Weaknesses 1 and 2. I think resolving the proof issues and providing more direct evidence of non-convergence resolution, together with the simpler baselines and computational costs, could materially affect my assessment.

5 comments

Thank you for acknowledging the motivation and theoretical contribution of our work, and for your constructive feedback. We respond to each point below.

1.

Lemma 8 in Appendix B appears false as stated. In its recursion, an additive perturbation satisfying only is not sufficient for convergence. For example, dt≡1d_t\equiv1, αt=1/(t+1)\alpha_t=1/(t+1), and rt=(1−γ)/(t+1)r_t=(1-\gamma)/(t+1) satisfy the stated conditions but do not giv dt→0d_t\to0 . The argument may be repairable if the perturbation is scaled by the stepsize, but the current lemma does not justify Theorem 2.

Thank you for carefully checking this step of the proof. We agree that convergence to zero alone would not suffice for an unscaled additive perturbation. However, the required stepsize scaling is already present in Lemma 8 of the submitted manuscript. Specifically, Eq. (74) reads:

(1-\alpha_t(s,a))d^{(t)}(s,a) +\alpha_t(s,a)\bigl[\gamma D^{(t)}+r_t\bigr].$$ Thus, the perturbation enters as $\alpha_t(s,a)r_t$, and this scaling is also retained in Eq. (86), where Lemma 8 is applied. We will make the placement of this factor clearer in both statements. ### 2. > I also find Eq. (11) unclear. Algorithm 1 updates the auxiliary estimator with the current transition before constructing the reference policy. Therefore, the reference policy depends on the same sample used in the main update, and the stated conditional expectation does not seem to follow from the given filtration. These issues directly affect the claimed almost-sure convergence guarantee. In the current ordering of Algorithm 1, the reference policy depends on the transition used in the main update. We agree that Eq. (11) therefore does not follow from the stated filtration. We will revise Algorithm 1 to construct the reference policy from the auxiliary law $\nu_t$ before sampling the current transition, and keep this policy fixed throughout the update: $$\begin{aligned} q_t(s,a) &=\mathbb{E}_{Z\sim\nu_t(s,a)}[Z], \\\\ \pi^{(t+1)}(a\mid s) &=\frac{\exp(q_t(s,a)/\lambda_t)}{\sum_b\exp(q_t(s,b)/\lambda_t)}. \end{aligned}$$ We then sample $(R_t,S'_t)$. Both branches may still share this transition, with both targets computed from $\nu_t$ and $\eta_t$ before either law is updated. The main target action is sampled from $\pi^{(t+1)}(\cdot\mid S'_t)$. Let $\mathcal F_t$ contain the information available before sampling the transition, including $S_t,A_t,\nu_t,\eta_t,\lambda_t$, and the stepsizes. The reference policy $\pi^{(t+1)}$ is then $\mathcal F_t$-measurable. For $$\begin{aligned} \widehat{\eta}^{(t+1)} &=(b_{R_t,\gamma})_{\\#}\eta_t(S'_t,B_t). \end{aligned}$$ the tower property gives $$\begin{aligned} \mathbb{E}[\widehat{\eta}^{(t+1)}\mid\mathcal{F}_t] &= \mathbb{E}\\!\left[ \sum_b\pi^{(t+1)}(b\mid S'_t) (b_{R_t,\gamma})_{\\#}\eta_t(S'_t,b) \\,\middle|\\,\mathcal{F}_t \right] \\\\ &=(\mathcal{T}^{\pi^{(t+1)}}\eta_t)(S_t,A_t). \end{aligned}$$ This establishes Eq. (11) as an equality of measures and justifies the zero conditional mean of the sampling noise used in Appendix B. With this correction, the remainder of the convergence proof applies unchanged, yielding the same almost-sure convergence guarantee under the stated assumptions. In particular, Eq. (59) already uses $Q^{(t)}$ together with $\lambda_t$, consistent with Assumption 3 and the revised policy definition. The reported experimental results are also unaffected: the tabular experiments use model-based planning updates, which do not require this correction, while Algorithm 2, used for the deep RL experiments, already constructs the reference policy and both targets before either network is updated. We will update Algorithm 1, the stochastic approximation description in Section 3.2, Assumption 1, and the corresponding statements in Appendix B to make this ordering explicit and consistent throughout. Due to the length limitation of each comment, we will address the remaining questions in the following comment.

We now continue addressing your remaining questions as follows.

3.

The experiments do not sufficiently establish the claimed non-convergence resolution or justify the extra estimator. In the tabular experiments, the Wasserstein error is measured against the return law of the tie-uniform optimal policy. A greedy method with deterministic tie-breaking may instead converge to a different optimal policy and its return law. In particular, the nonzero error reported for the greedy baseline in Appendix D.5 does not by itself demonstrate persistent distributional oscillation. I think an experiment showing actual non-convergence of greedy distributional updates would be more convincing.

We agree that the metric in Appendix D.5 is unsuitable for diagnosing non-convergence: a nonzero Wasserstein distance to the tie-uniform optimal policy's return law may simply reflect convergence to another optimal policy and its return law. We will replace this diagnostic with changes between consecutive policies and consecutive return-distribution estimates.

Our supplementary experiment uses a toy MDP with two optimal actions that have equal expected returns but different return distributions. We compare C51-style DVI and DDVI using 2,000 exact model-based updates and 20 paired initializations. Policy TV measures changes between consecutive reference policies at the decision state, and adjacent W1W_1 measures changes between consecutive return-distribution estimates at the start state. Both metrics are averaged over the final 300 iterations and across initializations.

MethodReturnPolicy TVAdjacent W1W_1
C51-style DVI49.0051.0004.65
DDVI49.005<0.001<0.001<0.001<0.001

Both methods attain the same expected return, but C51-style DVI continues to switch policies and return-distribution estimates over the observed window, whereas DDVI is much more stable. In this toy MDP, switching between equally good optimal actions preserves expected return. With function approximation, however, fitting a changed return-distribution target can also change estimates at other state–action pairs through shared parameters. This can alter action choices elsewhere, so the effects need not remain confined to equally good optimal actions and may also affect returns. Our motivation is therefore to stabilize policy selection and return-distribution learning while learning an optimal policy.

In deep RL, Schaul et al. (2022) observed frequent changes in greedy actions during training. Tang and Berseth (2024) found that limiting changes to predictions outside the current training batch improved DoubleDQN's returns on MinAtar. We will add deep RL experiments measuring policy changes, changes in the learned return distributions, and control returns to examine these effects with function approximation.

4.

I also find the baseline comparison insufficient. The auxiliary distributional estimator is only used through its mean, which follows standard Q-learning in the tabular setting. It is unclear why a scalar Q auxiliary estimator, or a simpler annealed-softmax C51 variant, would not provide the same benefit.

Scalar Q auxiliary. You are correct that a scalar Q-function can replace the auxiliary distribution in the tabular setting, with the same convergence argument under our assumptions. In deep RL, we keep both branches distributional so that they use the same representation and loss. This is intended to keep their learning dynamics similar as the auxiliary branch guides the main branch.

Single-distribution annealed softmax. This alternative uses the main estimator's own values to select actions with a gradually cooled softmax. We retain the auxiliary branch because proving convergence of this alternative with sampled updates is technically very challenging. An unusually high sampled target can make an action look better and increase its softmax weight. This changes the targets in later updates, feeding sampling fluctuations back into the same estimator. Cooling can amplify small random value differences into large policy changes. Controlling this coupled randomness throughout cooling is the central theoretical difficulty.

We compare both alternatives with DDVI in the same supplementary toy MDP experiment, using 2,000 outer iterations, 20 paired initializations, and the final-300-iteration metrics defined above.

MethodReturnPolicy TVAdjacent W1W_1
C51-style DVI49.0051.0004.65
Scalar-auxiliary DDVI49.005<0.001<0.001<0.001<0.001
Single-distribution softmax49.005<0.001<0.001<0.001<0.001
DDVI49.005<0.001<0.001<0.001<0.001

Both alternatives achieve the same return and similar stability in this tabular example. We will include these baselines in our additional deep RL experiments and compare their performance with DDVI.

5.

The closest related convergence approach using entropy regularization [1] is discussed but not compared experimentally.

Jhaveri et al. (2025) first run entropy-regularized control updates at a fixed temperature, then construct a policy with a decoupled temperature and hold it fixed during distributional evaluation. DDVI updates the reference policy and return distributions together throughout training, using unregularized backups while gradually reducing the policy temperature. We will consider adding a comparison with their two-stage implementation in Section 4.3 on the same tabular MDPs, counting the updates in both of their stages and matching the total number of Bellman updates.

6.

DDVI also requires an additional categorical network, yet no computational-cost comparison is reported.

We recorded elapsed execution times during our experiments but did not include them in the submitted manuscript. For 500k environment steps, DDVI's mean runtime across three seeds was 1.58 times that of C51 on Swimmer, 1.54 times on Hopper, and 1.62 times on Reacher. The additional estimator thus incurs a constant-factor overhead in these runs, with total runtime remaining below twice that of C51. We will add these timing results and the measurement protocol to the revised manuscript.

7.

The MuJoCo results are mixed across only three seeds: Hopper has a lower mean return and higher standard deviation than C51, while the improvement on Reacher is small.

The return difference on Hopper changes sign across seeds. We will extend the comparison with more seeds and include the scalar-auxiliary and single-distribution softmax baselines, using the same evaluation protocol and reporting confidence intervals for the return differences. We will also measure changes between consecutive policies and return-distribution estimates to examine whether improved distributional stability translates into better control performance.

Reviewer_2acGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper tries to tackle the non-convergence of estimated return distributions during control. It introduces a new method, Double Distributional Value Iteration (DDVI), that uses two critics : the first builds its Bellman targets using the action with the highest estimated mean return (usual greedy action). The first critic’s mean returns are passed through a softmax to obtain action probabilities. The second critic uses these probabilities to sample the next action in its Bellman target. The temperature decreases during training. MuJoCo experiments report more stable distribution estimates, with mixed improvements in expected return.

Critical strengths

This work tries to tackle an issue that has not received, as far as I know, a lot of attention : the non-convergence of the estimated return distributions when using greedy action selection for control.

Critical weaknesses

In my opinion, the primary issues with this work are motivations and presentation. Some parts contain long cryptic prose that is difficult to decipher. Some parts use a lot of inline math in a very dense way which does not make them human friendly to read. Given my substantial concerns with the main part, I have not verified the proofs in the appendix.

  • The motivation for this work is weak. The non-convergence of the control operator is known to exist in the provided references (Bellemare et al., 2017, 2023). However, why is it a practical issue ? Are there documented examples in the literature showing that this issue is relevant ? Are you the first to try to tackle this specific issue ?.
  • The word "homotopy" is pushed very heavily but never properly introduced.
  • The related work section mixes prior work and the definition of the methods proposed by the work.
  • Section 4.2 contains no insights on the results. Many symbols are not introduced (in Assumption 1). Corollary 1 is not accompanied by any insight that would be useful for the average reader.
  • The work does not isolate the issue in a control setting or show whether policy learning suffers when the estimated return distributions fail to converge.
  • The control experiments on MuJoCo : (1) do not illustrate the point of the paper as they are in deterministic environments that do not contain the randomness needed for the non-convergence issue to appear (2) do not show a reliable improvement in terms of control on 2 out of 3 environments (3) do not contain enough environments

The main part contains several statements that an average reader would have a very hard time deciphering as they (1) use weird idioms (2) use idioms that have not been introduced (3) lack a clear connection with the prior content (using linking words for example) :

  • "Our homotopy approach is motivated by this softmax,the Gibbs measure that converges,as the temperature vanishes,to the uniform distribution on the maximizers of its argument (Hwang, 1980);a cooling schedule makes that limit algorithmic and tracks a smooth path back to greedy selection (Geman & Geman, 1984; Hajek, 1988; Rose, 1998; Allgower & Georg, 1990)."
  • "The proof uses a common argument: value errors that vanish faster than the temperature equalize the softmax weights on tied optimal actions and suppress suboptimal actions. The resulting policy error gives a vanishing perturbation of evaluation under π ∗ , whose Wasserstein contraction yields distributional convergence. For the stochastic approximation version, martingale-noise bounds and the Robbins–Monro stepsizes additionally control the sampling noise and yield almost-sure convergence."
  • "We use the path only to form the reference policy inside an otherwise unregularized iteration. The temperature rescales the estimated means; it does not enter the reward or the return distribution being learned."
  • This is a very convoluted way to introduce entropy regularization / soft-MDP : "The policy is rewarded for expected return and for its own randomness, and at a fixed positive temperature the optimal policy is a softmax".
  • Lines 151–171 (and 174–185) are almost unreadable.
  • Table 1 : displays very small numbers that could perhaps be rounded to 0

The main part contains several weird statements that, unless I misunderstood, seem factually wrong or not precise enough:

  • Line 47 : "the greedy action is the bootstrap target of the distributional Bellman update" (how can an action be a target value ?)
  • Line 55 : "entropy-regularized methods replace greedy selection by a regularization term" (they turn the MDP into a soft-MDP. How could a regularization term output actions ?)
  • Line 105 : "is a contraction" (a contraction in what ? (i.e the Wasserstein distance))
  • Line 106 : "control does not inherit this guarantee" (you need to be more specific, this is one of the most important statements of the whole work)
  • Line 127 : "The regularized Bellman operator is then a contraction" (as a consequence of what ? something that was just said ?)
  • Line 336 : "three additional environments" (I see two)
Questions for authors
  • A prior work is mentioned (Jhaveri et al. (2025)) that, it seems, tried to address a similar issue. Did that work introduce a competing algorithm ? If yes, can it be tested in the same settings as your experiment ? If yes, how does it perform ?
  • The non-convergence to a single policy is a mathematical failure mode, but why would it be an empirical one that matters ? Does it prevent the induced policy to be optimal ?
6 comments

Thank you for your careful reading and detailed comments. We will simplify the presentation and explain the motivation, algorithm, and proof ideas more directly. We respond to each point below.

1.

The motivation for this work is weak. The non-convergence of the control operator is known to exist in the provided references (Bellemare et al., 2017, 2023). However, why is it a practical issue ? Are there documented examples in the literature showing that this issue is relevant ? Are you the first to try to tackle this specific issue ?.

Our motivation is to stabilize policy selection and return-distribution learning during control. Greedy updates can keep switching between actions with different return distributions even when their expected returns converge. In a tabular MDP, switching between equally good optimal actions can preserve expected return. With function approximation, fitting a changed distributional target can also change estimates at other state–action pairs through shared parameters. This can alter action choices elsewhere, so the effects need not remain confined to equally good optimal actions and may also affect control returns.

In deep RL, Schaul et al. (2022) observed frequent changes in greedy actions during training. Tang and Berseth (2024) found that limiting changes to predictions outside the current training batch improved DoubleDQN's returns on MinAtar. These observations support the practical relevance of policy stability. Our supplementary toy experiment directly examines policy switching and changes in the learned return distributions.

Earlier work has addressed this issue. Jhaveri et al. (2025) study policy and return-distribution convergence through entropy regularization and decoupled temperatures. Their Theorem 4.7 uses separate control and evaluation stages. DDVI provides a practical, single-stage learning algorithm that updates both estimators together using unregularized backups while cooling the policy temperature, with convergence guarantees for tabular planning and stochastic approximation. We will make this relationship clearer in the paper.

2.

The word "homotopy" is pushed very heavily but never properly introduced.

A homotopy between continuous maps f0,f1:X→Yf_0,f_1:X\to Y is a continuous map H:X×[0,1]→YH:X\times[0,1]\to Y satisfying H(x,0)=f0(x)H(x,0)=f_0(x) and H(x,1)=f1(x)H(x,1)=f_1(x).

We use the term in the numerical-continuation sense: for fixed action values, varying the temperature continuously deforms a softmax policy toward its greedy limit. DDVI follows this temperature-indexed family by gradually lowering the temperature during learning. We will introduce this meaning before using the term.

3.

The related work section mixes prior work and the definition of the methods proposed by the work.

We will keep Section 2 focused on prior work and its relation to DDVI, and move the description of our policy construction and updates to Section 3.

4.

Section 4.2 contains no insights on the results. Many symbols are not introduced (in Assumption 1). Corollary 1 is not accompanied by any insight that would be useful for the average reader.

We agree that Section 4.2 needs to explain the role of its assumptions and the meaning of its convergence result. Cooling too quickly can turn small value-estimation errors into large changes in action probabilities, especially when optimal actions are tied. Requiring these errors to vanish faster than the temperature ensures that the reference policy converges to the uniform policy over optimal actions. Under the stated sampling and stepsize conditions, the main distribution then converges almost surely to that policy's return law, even if the auxiliary distributions do not converge. We will add this explanation to Section 4.2.

For Assumption 1, we will introduce the filtration, the two target actions, and the sampled distributional target before using them, and align their notation with Algorithm 1. We will also specify that the reference policy is constructed from the pre-update auxiliary estimate before sampling the current transition.

Corollary 1 shows how the temperature condition in Theorem 2 can be satisfied when the auxiliary means follow standard tabular Q-learning. Under the stated sampling and stepsize conditions, the cooling schedule λt=(t+1)−p\lambda_t=(t+1)^{-p}, with 0<p<1/20<p<1/2, satisfies Assumption 3. We will retain its position after Theorem 2 and rename it as a proposition to explain how the theorem applies in this setting.

Due to the length limitation of each comment, we will address the remaining questions in the following comment.

We now continue addressing your remaining questions as follows.

5.

The work does not isolate the issue in a control setting or show whether policy learning suffers when the estimated return distributions fail to converge.

The non-convergence to a single policy is a mathematical failure mode, but why would it be an empirical one that matters? Does it prevent the induced policy to be optimal?

Failure to converge to a single policy does not by itself imply suboptimal control: switching between equally good optimal actions preserves expected return. Our supplementary toy MDP illustrates this distinction with two optimal actions that have equal expected returns but different return distributions. We use 2,000 exact model-based updates and 20 paired initializations, recomputing the reference policy from the current estimates.

Policy TV measures changes between consecutive reference policies at the decision state, and adjacent W1W_1 measures changes between consecutive return-distribution estimates at the start state. Both metrics are averaged over the final 300 iterations and across initializations.

MethodReturnPolicy TVAdjacent W1W_1
C51-style DVI49.0051.0004.65
DDVI49.005<0.001<0.001<0.001<0.001

Both methods attain the same expected return, while C51-style DVI continues to switch policies and distributional estimates over the observed window. With function approximation, however, fitting a changed distributional target can also change estimates and action choices elsewhere through shared parameters. These changes need not stay within the optimal action set and may affect control performance.

Frequent policy switching is already a known phenomenon in deep RL: Schaul et al. (2022) observed frequent changes in greedy actions during training, and Tang and Berseth (2024) found that limiting changes to predictions outside the current training batch improved DoubleDQN's returns on MinAtar. These findings support the practical importance of policy stability. We will add deep RL experiments measuring policy changes, changes in return distributions, and control returns to examine these effects in our setting.

6.

The control experiments on MuJoCo : (1) do not illustrate the point of the paper as they are in deterministic environments that do not contain the randomness needed for the non-convergence issue to appear (2) do not show a reliable improvement in terms of control on 2 out of 3 environments (3) do not contain enough environments

Even with deterministic rewards and transitions, a stochastic policy can produce different action sequences and returns. DDVI's main estimator evaluates a softmax policy, so at positive temperatures its target return distribution can include randomness from action selection. The MuJoCo experiments evaluate DDVI with function approximation; the toy MDP separately isolates switching between optimal actions with different return distributions.

We agree that the current results on Hopper and Reacher do not show a clear improvement over C51. We will extend the comparison with more seeds and environments, including tasks with stochastic rewards or transitions, and add the scalar-auxiliary and single-distribution softmax baselines. We will report return differences with confidence intervals, together with changes between consecutive policies and return-distribution estimates.

We now continue addressing your remaining questions as follows.

7.

The main part contains several statements that an average reader would have a very hard time deciphering as they (1) use weird idioms (2) use idioms that have not been introduced (3) lack a clear connection with the prior content (using linking words for example) :

"Our homotopy approach is motivated by this softmax,the Gibbs measure that converges,as the temperature vanishes,to the uniform distribution on the maximizers of its argument (Hwang, 1980);a cooling schedule makes that limit algorithmic and tracks a smooth path back to greedy selection (Geman & Geman, 1984; Hajek, 1988; Rose, 1998; Allgower & Georg, 1990)."

"The proof uses a common argument: value errors that vanish faster than the temperature equalize the softmax weights on tied optimal actions and suppress suboptimal actions. The resulting policy error gives a vanishing perturbation of evaluation under π∗\pi^* , whose Wasserstein contraction yields distributional convergence. For the stochastic approximation version, martingale-noise bounds and the Robbins–Monro stepsizes additionally control the sampling noise and yield almost-sure convergence."

"We use the path only to form the reference policy inside an otherwise unregularized iteration. The temperature rescales the estimated means; it does not enter the reward or the return distribution being learned."

This is a very convoluted way to introduce entropy regularization / soft-MDP : "The policy is rewarded for expected return and for its own randomness, and at a fixed positive temperature the optimal policy is a softmax".

Lines 151–171 (and 174–185) are almost unreadable.

Table 1 : displays very small numbers that could perhaps be rounded to 0

We will rewrite the quoted passages as follows.

Homotopy: “For fixed action values q(a)q(a), softmax assigns action probabilities proportional to exp⁡(q(a)/λ)\exp(q(a)/\lambda), where λ>0\lambda>0 is the temperature. As λ\lambda tends to zero, this policy approaches the uniform distribution over maximizing actions (Hwang, 1980). This limit motivates our homotopy approach: at iteration tt, we use a positive temperature λt\lambda_t from a sequence that decreases to zero (Geman and Geman, 1984; Hajek, 1988; Rose, 1998; Allgower and Georg, 1990).”

Proof intuition: “When value-estimation errors vanish faster than the temperature, the softmax probabilities of tied optimal actions become equal in the limit. As the temperature goes to zero, the probabilities of suboptimal actions vanish. The reference policy therefore converges, and the main update approaches evaluation under a fixed optimal policy. Contraction in Wasserstein distance then gives convergence of the return-distribution estimates. For sampled updates, bounds on the sampling noise and the stepsize conditions give the same convergence almost surely.”

Temperature and rewards: “We compute the reference policy by applying softmax to the auxiliary action values divided by the temperature. Both Bellman targets use the original reward plus the discounted next-state return, with no entropy bonus.”

Entropy regularization: “Entropy-regularized control adds an entropy bonus, weighted by the temperature, to the expected-return objective. At a fixed positive temperature, the optimal policy is a softmax of the action values for this regularized problem.”

For Lines 151–171, we will introduce the return and its mean before the return distribution, explain the transformation R+γZR+\gamma Z before introducing push-forward notation, and display the Bellman equations separately. For Lines 174–185, we will first explain ordinary value iteration, then show that taking means in distributional value iteration recovers it, while convergence of the means does not guarantee convergence of the return distributions. We will also correct grammar and punctuation throughout the paper.

For Table 1, we will report errors below 10−810^{-8} as “<10−8<10^{-8}” and state this convention in the caption.

Reviewer_eyviOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies the problem of convergent estimation of return distributions in the control setting, which has notably been elusive since the seminal work in distributional RL. The paper introduces a new algorithmic framework leveraging two return distribution estimates which achieves convergent estimation of the return distribution associated to the policy that is uniform on optimal actions---in the tabular setting. Convergence proofs are given for both nonparametric distributional dynamic programming and nonparametric distributional TD-learning / stochastic approximation. Finally, a practical instantiation of the proposed framework is implemented and tested on certain RL environments, both tabular and with neural networks, demonstrating stable policy and return distribution estimation.

Critical strengths

This paper tackles an important and difficult problem, and provides a simple solution. The mathematical results appear to be correct, and I believe the algorithmic framework provides lots of value for further work in both theory and practice.

Critical weaknesses

The main concern I have is that this paper tackles a very similar problem to that of Jhaveri et al., 2025, and the connection is not discussed very thoroughly. Indeed, Theorem 2.5 of Jhaveri et al., 2025 seems to provide a similar result, at least in the dynamic programming setting. As far as I can tell, that paper focuses more on continuous MDPs (while this paper focuses on tabular only), and the price paid for that is a more complicated / convoluted algorithmic framework / analysis, as well as an algorithm that requires multiple stages instead of a single dynamic programming scheme. I don't mean to suggest that this paper isn't novel enough; indeed, I enjoyed this paper, I just think framing it relative to Jhaveri et al., 2025 more precisely would be helpful.

Also, it is not clear to me that the "double-distributional" part is actually necessary (see Questions), though this is not a major issue, more again about presentation.

While the purpose of this paper was to establish the theory and was not about empirical results, the empirical results are not particularly easy to interpret, since it seems like C51 does not suffer from the problems that motivated the design of the new algorithm / theory. So, overall, I'm not entirely clear on what the takeaway should be from the experimental results.

Lastly, there are many areas where the paper is difficult to read, whether that be due to structural issues (see Questions), or grammatical / punctuation errors.

Questions for authors

It's not actually clear to me why any of the methods in this work require the "double distributional" piece. Throughout all the algorithms and analysis, as far as I can tell, the ν\nu distributions are used only to extract means, which are then used to derive a softmax policy. So it seems ν\nu can be replaced with a standard Q-function. Is that correct?

On that note, do you think it's possible to forego the auxiliary return distribution function / Q-function altogether, and just use the extracted means from η\eta from the previous iteration? This is less clear to me, but it seems possible since the Boltzmann map is known to be a policy improvement operator.

In the experiments, why does C51 seem to track the π⋆\pi^\star almost perfectly from step (in Figure 1)?

Not a question, but I almost concluded that the proof of Theorem 4.1 is entirely incomplete – the phrasing of the paper suggests that the proof is given in section 4.1, but in reality the proof is in the appendix, though this isn't mentioned anywhere. Likewise, even in the appendix, there are lemmas stated with no indication of where their proofs will be.

Again not a question and just a suggestion to make the content easier to follow – Corollary 1 is framed and positioned oddly, which made it difficult for me to parse. Calling it a corollary makes it sound like it is a downstream consequence of Theorem 2, which is not the case – Theorem 2 is still the "final" theorem. As I understand it, Corollary 1 gives a sufficient condition under which one of the assumptions for Theorem 2 holds. I would call this a lemma / proposition instead of a corollary, and the text should clarify where it stands relative to Theorem 2.

3 comments

Thank you for recognizing the importance of the problem and the simplicity of our approach, and for your constructive suggestions. We respond to each point below.

1.

The main concern I have is that this paper tackles a very similar problem to that of Jhaveri et al., 2025, and the connection is not discussed very thoroughly. Indeed, Theorem 2.5 of Jhaveri et al., 2025 seems to provide a similar result, at least in the dynamic programming setting. As far as I can tell, that paper focuses more on continuous MDPs (while this paper focuses on tabular only), and the price paid for that is a more complicated / convoluted algorithmic framework / analysis, as well as an algorithm that requires multiple stages instead of a single dynamic programming scheme. I don't mean to suggest that this paper isn't novel enough; Indeed, I enjoyed this paper, I just think framing it relative to Jhaveri et al., 2025 more precisely would be helpful.

We agree that Theorem 2.5 of Jhaveri et al. (2025) is closely related to our tabular results. Under their Assumption 2.4, it establishes convergence of the exact optimal occupancy measures of entropy-regularized problems as the temperature vanishes. Their result describes the limiting behavior of exact solutions across temperatures. Our results establish convergence of the learning iterates while updating estimates and cooling the temperature together.

Our method provides a practical, single-stage learning algorithm with convergence guarantees. It updates value estimates and return distributions together while cooling the policy temperature. At each iteration, the auxiliary estimator supplies action values for the reference policy, and the main estimator updates the return distribution under that policy; both use unregularized Bellman backups. Their Theorem 4.7 provides a control-then-evaluation procedure with suitable fixed temperatures and separate iteration budgets for a prescribed accuracy; DDVI performs control and evaluation updates together throughout training. Under the stated conditions, we establish convergence of these iterates in tabular planning and almost-sure convergence of the stochastic approximation version.

2.

.While the purpose of this paper was to establish the theory and was not about empirical results, the empirical results are not particularly easy to interpret, since it seems like C51 does not suffer from the problems that motivated the design of the new algorithm / theory. So, overall, I'm not entirely clear on what the takeaway should be from the experimental results.

Our original experiments did not clearly illustrate the instability motivating DDVI. We therefore conducted an additional experiment in a toy MDP where two optimal actions have equal expected returns but different return distributions. We compare C51-style DVI and DDVI using exact model-based updates over 2,000 iterations with 20 paired initializations.

Policy TV measures the total variation distance between consecutive reference policies at the decision state, and adjacent W1W_1 measures the Wasserstein-1 distance between consecutive return-distribution estimates at the start state. Both metrics are averaged over the final 300 iterations and across initializations.

MethodReturnPolicy TVAdjacent W1W_1
C51-style DVI49.0051.0004.65
DDVI49.005<0.001<0.001<0.001<0.001

Both methods obtain the same expected return. Over the final 300 observed iterations, however, C51-style DVI continues to switch its target policy and exhibits substantial changes in its estimated return distribution, whereas DDVI is much more stable. This illustrates the intended benefit: stabilizing policy selection and distributional estimates while preserving expected-return performance.

In this toy MDP, switching between equally good optimal actions leaves expected returns unchanged. With function approximation, fitting a changed return-distribution target can also change estimates at other state–action pairs through shared parameters. These changes can affect action choices elsewhere, so the effects of switching need not remain confined to equally good optimal actions and may also affect returns. Schaul et al. (2022) document frequent greedy-action changes in deep RL. Tang and Berseth (2024) study how updates change predictions outside the training batch and report that reducing these changes improves DoubleDQN's returns on MinAtar. These observations motivate studying the broader effects of policy switching in deep RL.

Due to the length limitation of each comment, we will address the remaining questions in the following comment.

We now continue addressing your remaining questions as follows.

3.

.It's not actually clear to me why any of the methods in this work require the "double distributional" piece. Throughout all the algorithms and analysis, as far as I can tell, the distributions are used only to extract means, which are then used to derive a softmax policy. So it seems can be replaced with a standard Q-function. Is that correct?

You are correct: in the tabular setting, a scalar Q-function can replace the auxiliary distribution, and our convergence proof applies under the same assumptions.

We formulate the method as double distributional to give the auxiliary and main branches a common distributional representation and update structure. This choice is motivated by the aim of keeping their learning dynamics similar as the auxiliary branch guides the main branch. With function approximation, both branches can then use the same distributional loss, which we expect to help stabilize their coupled learning. We will add a remark early in the paper explaining this design rationale and the scalar alternative, and conduct additional experiments comparing the two choices to assess this expected benefit.

4.

On that note, do you think it's possible to forego the auxiliary return distribution function / Q-function altogether, and just use the extracted means from from the previous iteration? This is less clear to me, but it seems possible since the Boltzmann map is known to be a policy improvement operator.

Algorithmically, yes: we can construct the softmax policy from the main distribution's means and update that same distribution under the resulting policy, without an auxiliary estimator.

The difficulty is proving convergence with sampled updates. An unusually high sampled target can raise an action's estimated mean and its weight in the next softmax policy. That policy then changes subsequent targets for the same distribution, feeding sampling fluctuations back into its updates. Near tied optimal actions, cooling can amplify small random differences in the means into large changes in action probabilities. A proof must control this feedback and show that the resulting value errors vanish faster than the temperature. We retain an auxiliary branch whose means follow standard Q-learning, allowing us to use existing error bounds under the stated assumptions.

We will include this single-distribution variant as a baseline in our additional deep RL experiments and compare its performance with DDVI.

5.

In the experiments, why does C51 seem to track the almost perfectly from step (in Figure 1)?

We agree that the policy-distance metric in Figure 1 is not suitable for showing policy instability. For C51, it compares the target policy with a greedy policy derived from the same value estimates, so it can remain zero even when the selected action keeps changing. We will revise Figure 1 using the metrics from our supplementary experiment: the total variation distance between consecutive reference policies and the Wasserstein-1 distance between consecutive return-distribution estimates. These directly measure changes in the policy and the estimated return distribution over time.

6.

Not a question, but I almost concluded that the proof of Theorem 4.1 is entirely incomplete – the phrasing of the paper suggests that the proof is given in section 4.1, but in reality the proof is in the appendix, though this isn't mentioned anywhere. Likewise, even in the appendix, there are lemmas stated with no indication of where their proofs will be.

The proof of the planning theorem and the supporting lemmas appear in Appendices A.1 and A.2, respectively. We will add explicit proof references after their statements.

7.

Again not a question and just a suggestion to make the content easier to follow – Corollary 1 is framed and positioned oddly, which made it difficult for me to parse. Calling it a corollary makes it sound like it is a downstream consequence of Theorem 2, which is not the case – Theorem 2 is still the "final" theorem. As I understand it, Corollary 1 gives a sufficient condition under which one of the assumptions for Theorem 2 holds. I would call this a lemma / proposition instead of a corollary, and the text should clarify where it stands relative to Theorem 2.

Corollary 1 is intended to show how the temperature condition in Theorem 2 can be satisfied when the auxiliary means follow standard tabular Q-learning. Under the stated sampling and stepsize conditions, it gives an explicit cooling schedule, λt=(t+1)−p\lambda_t=(t+1)^{-p} with 0<p<1/20<p<1/2, that satisfies Assumption 3. We will retain its position after Theorem 2 to explain how the theorem applies in this setting, while renaming it as a proposition and clarifying this supporting role.

Thanks to the authors for the clarifications; I have no further questions or concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.