- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how independently trained RL policies can become both high-performing and distributionally similar across training runs. The authors bound the KL divergence between two Boltzmann policies using their Q-value difference and a shared temperature. Based on this result, Q-value Expectile Disagreement (QED) is proposed to adjust the Boltzmann temperature using disagreement between two critics within one run. Experiments on eighteen continuous-control tasks report lower policy differences across runs and competitive returns.
Critical strengths
The motivation on policy similarity is interesting. Enforcing consistency across runs can be important to real-world high stake applications, and conventional average may run the risk of averaging particularly bad runs as similar rewards can be obtained through very different behaviors. Studying this difference is useful for understanding the reliability of RL algorithms. QED also provides a simple rule for adjusting entropy. The current version includes direct policy comparisons, rollout measurements, and comparisons with fixed temperatures across ten random seeds.
Critical weaknesses
My concerns are the overall theoretical contribution is limited.
- Theorem 4.1 follows from standard bounds on Boltzmann policies. Theorem 4.2 uses a standard value-iteration argument. More importantly, both results require a shared temperature computed from two runs. The practical method instead uses double-critic disagreement as a “single-run proxy for cross-run disagreement.” These critics share training data and may agree even when independent runs learn different policies.
- Correlation plot compares averages across tasks but it does not seem to prove that this estimate works for individual (critical) states or throughout training.
- While the toy example shows how high entropy can lead to poor value estimates outside the available data. It does not directly test consistency across runs or validate the proposed temperature rule.
- I think there is a missing piece of the paper: is there empirical evidence that increasing similarity across runs translates to improved robustness?
Questions for authors
- What is the main new insight of Theorems 4.1 and 4.2? How does it support the practical temperature rule? The theory uses a shared temperature across two runs, but the practical method chooses the temperature within each run.
- Does double-critic disagreement track differences between independent runs at individual states and throughout training? Both critics could agree at a state even when independently trained critics disagree?
- Does greater similarity across runs lead to improved robustness? Tests with observation noise or small changes to the environment would help establish this benefit. Similar policies may still share the same failure modes.
Dear Reviewer 7dWT,
Thank you for your feedback and for highlighting the importance of the problem we study.
My concerns are the overall theoretical contribution is limited.
Our submission is not a learning theory submission. Learning theoretic results on replication exist (see, Eaton et al 2023, 2026; Karbasi et al 2023; Hopkins et al. 2025). However, we are not aware of any empirical algorithms for behavior consistency. The paper’s main contribution is an empirical approach to behavior consistency, which we evaluate extensively.
The practical method instead uses double-critic disagreement as a “single-run proxy for cross-run disagreement.
It is unclear to us why approximation of the theory is listed as a weakness in a deep learning paper, where it is quite common to use approximations. There are very few empirical deep learning methods that implement an exact theoretical algorithm. For more detail see response to Q2.
While the toy example shows how high entropy can lead to poor value estimates outside the available data. It does not directly test consistency across runs or validate the proposed temperature rule.
That is correct. That intention of this experiment is to highlight why naively increasing entropy can lead to unstable training, not to show consistency results. It motivates the fact that we need algorithms that are able to train stably at high entropy, such as MAD-TD. Consistency is tested across a wide range of experiments in section 6.
I think there is a missing piece of the paper: is there empirical evidence that increasing similarity across runs translates to improved robustness?
The topic of robustness is already widely studied in the literature. We are not studying the topic of robustness; we study the problem of consistency. It is unclear to us why this is listed as a weakness.
Q1 - What is the main new insight of Theorems 4.1 and 4.2?
The main insight is summarized in lines 236-238 in the text. The theory tells us that two Q-functions from different runs can be used to bound the KL divergence between runs and that this coupling does not prevent convergence. Note that unlike traditional convergence bounds, our bound contains a non-monotonic term arising from disagreement-dependent temperature. This is very non-traditional and the reason we set out to prove Theorem 4.2 in the first place.
The arguments largely follow traditional analysis and we do not make any claims in the paper about the novelty of analysis. In times where AI can generate highly complex proofs, proof complexity is not something we strive for. The Theorems are supposed to build intuition and show how a desirable goal (in this case across-run consistency) can be achieved.
How does it support the practical temperature rule?
We directly implement this rule via approximation.
The theory uses a shared temperature across two runs, but the practical method chooses the temperature within each run.
Yes, this is the correct understanding. We use an approximation and test empirically whether it is a good approximation. We find that our method consistently reduces cross-run divergence and leads to similar behavior which validates this approximation.
Q2 - Does double-critic disagreement track differences between independent runs at individual states and throughout training? Both critics could agree at a state even when independently trained critics disagree?
Yes, this is the correct understanding. There is a failure mode where both critics agree on something out-of-distribution and thus, temperature is low. Given our extensive experiments we do not believe that this is an issue as our method works across a wide range of environments. We have added a new experiment in now E.2 that shows how critics disagree on out-of-distribution states and how this correlates with cross-run Q disagreement.
Q3 - Does greater similarity across runs lead to improved robustness?
We do not study robustness in our paper and we see no clear theoretical reason to expect that higher entropy would improve robustness. To answer the question though, we took the walker_walk task and evaluated agents trained with SAC and SAC-QED using Gaussian state perturbation ranging from σ 0.1 to 0.5. As expected, we do not see a difference in performance between the methods. Robustness is orthogonal to cross-run consistency.