acceptodds
Under review as a conference paper at ICLR 2027

Behavior-Consistent Deep Reinforcement Learning

Abstract

Reinforcement learning (RL) often exhibits high variance across training runs, leading to unreliable performance and posing a major challenge to deployment in real-world domains. In this work, we address the challenge of cross-run policy divergence by formalizing the problem of \em behavior-consistent RL, where the objective is to obtain policies that are both high-performing and distributionally similar across training runs. Our key observation is that maximum-entropy RL provides a direct mechanism for controlling behavioral divergence by anchoring runs to a common (uniform) prior. We prove that, for Boltzmann policies, choosing the temperature proportional to -function disagreement bounds the pairwise KL divergence between the induced policies. However, we also show that naïvely increasing entropy might impair policy optimization while amplifying off-policy error. Building upon these observations, we propose -value Expectile Disagreement (QED), a state-dependent temperature schedule that uses double-critic disagreement as a single-run proxy for cross-run disagreement. Empirically, we demonstrate that across 18 continuous-control tasks, QED reduces across-run divergence by two orders of magnitude without sacrificing performance, resulting in a considerable reduction in return variance at modest sample-efficiency costs.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, replicability, reproducibility, stability, behavioral consistency, policy consistency, policy divergence, continuous control, KL regularization, maximum entropy reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7dWTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how independently trained RL policies can become both high-performing and distributionally similar across training runs. The authors bound the KL divergence between two Boltzmann policies using their Q-value difference and a shared temperature. Based on this result, Q-value Expectile Disagreement (QED) is proposed to adjust the Boltzmann temperature using disagreement between two critics within one run. Experiments on eighteen continuous-control tasks report lower policy differences across runs and competitive returns.

Critical strengths

The motivation on policy similarity is interesting. Enforcing consistency across runs can be important to real-world high stake applications, and conventional average may run the risk of averaging particularly bad runs as similar rewards can be obtained through very different behaviors. Studying this difference is useful for understanding the reliability of RL algorithms. QED also provides a simple rule for adjusting entropy. The current version includes direct policy comparisons, rollout measurements, and comparisons with fixed temperatures across ten random seeds.

Critical weaknesses

My concerns are the overall theoretical contribution is limited.

  • Theorem 4.1 follows from standard bounds on Boltzmann policies. Theorem 4.2 uses a standard value-iteration argument. More importantly, both results require a shared temperature computed from two runs. The practical method instead uses double-critic disagreement as a “single-run proxy for cross-run disagreement.” These critics share training data and may agree even when independent runs learn different policies.
  • Correlation plot compares averages across tasks but it does not seem to prove that this estimate works for individual (critical) states or throughout training.
  • While the toy example shows how high entropy can lead to poor value estimates outside the available data. It does not directly test consistency across runs or validate the proposed temperature rule.
  • I think there is a missing piece of the paper: is there empirical evidence that increasing similarity across runs translates to improved robustness?
Questions for authors
  • What is the main new insight of Theorems 4.1 and 4.2? How does it support the practical temperature rule? The theory uses a shared temperature across two runs, but the practical method chooses the temperature within each run.
  • Does double-critic disagreement track differences between independent runs at individual states and throughout training? Both critics could agree at a state even when independently trained critics disagree?
  • Does greater similarity across runs lead to improved robustness? Tests with observation noise or small changes to the environment would help establish this benefit. Similar policies may still share the same failure modes.
1 comment

Dear Reviewer 7dWT,

Thank you for your feedback and for highlighting the importance of the problem we study.

My concerns are the overall theoretical contribution is limited.
Our submission is not a learning theory submission. Learning theoretic results on replication exist (see, Eaton et al 2023, 2026; Karbasi et al 2023; Hopkins et al. 2025). However, we are not aware of any empirical algorithms for behavior consistency. The paper’s main contribution is an empirical approach to behavior consistency, which we evaluate extensively.

The practical method instead uses double-critic disagreement as a “single-run proxy for cross-run disagreement.
It is unclear to us why approximation of the theory is listed as a weakness in a deep learning paper, where it is quite common to use approximations. There are very few empirical deep learning methods that implement an exact theoretical algorithm. For more detail see response to Q2.

While the toy example shows how high entropy can lead to poor value estimates outside the available data. It does not directly test consistency across runs or validate the proposed temperature rule.
That is correct. That intention of this experiment is to highlight why naively increasing entropy can lead to unstable training, not to show consistency results. It motivates the fact that we need algorithms that are able to train stably at high entropy, such as MAD-TD. Consistency is tested across a wide range of experiments in section 6.

I think there is a missing piece of the paper: is there empirical evidence that increasing similarity across runs translates to improved robustness?
The topic of robustness is already widely studied in the literature. We are not studying the topic of robustness; we study the problem of consistency. It is unclear to us why this is listed as a weakness.

Q1 - What is the main new insight of Theorems 4.1 and 4.2?
The main insight is summarized in lines 236-238 in the text. The theory tells us that two Q-functions from different runs can be used to bound the KL divergence between runs and that this coupling does not prevent convergence. Note that unlike traditional convergence bounds, our bound contains a non-monotonic term arising from disagreement-dependent temperature. This is very non-traditional and the reason we set out to prove Theorem 4.2 in the first place.

The arguments largely follow traditional analysis and we do not make any claims in the paper about the novelty of analysis. In times where AI can generate highly complex proofs, proof complexity is not something we strive for. The Theorems are supposed to build intuition and show how a desirable goal (in this case across-run consistency) can be achieved.

How does it support the practical temperature rule?
We directly implement this rule via approximation.

The theory uses a shared temperature across two runs, but the practical method chooses the temperature within each run.
Yes, this is the correct understanding. We use an approximation and test empirically whether it is a good approximation. We find that our method consistently reduces cross-run divergence and leads to similar behavior which validates this approximation.

Q2 - Does double-critic disagreement track differences between independent runs at individual states and throughout training? Both critics could agree at a state even when independently trained critics disagree?
Yes, this is the correct understanding. There is a failure mode where both critics agree on something out-of-distribution and thus, temperature is low. Given our extensive experiments we do not believe that this is an issue as our method works across a wide range of environments. We have added a new experiment in now E.2 that shows how critics disagree on out-of-distribution states and how this correlates with cross-run Q disagreement.

Q3 - Does greater similarity across runs lead to improved robustness?
We do not study robustness in our paper and we see no clear theoretical reason to expect that higher entropy would improve robustness. To answer the question though, we took the walker_walk task and evaluated agents trained with SAC and SAC-QED using Gaussian state perturbation ranging from σ\sigma 0.1 to 0.5. As expected, we do not see a difference in performance between the methods. Robustness is orthogonal to cross-run consistency.

Reviewer_ZBQaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies behavioral consistency in deep reinforcement learning. The goal is to train policies that achieve high returns while behaving similarly across independent training runs. The authors show that scaling the entropy temperature with Q-value disagreement can bound the KL divergence between Boltzmann policies. Based on this observation, they propose Q-value Expectile Disagreement (QED), which uses disagreement between two critics to adapt the entropy temperature. Experiments on 18 continuous-control tasks show that QED substantially reduces cross-run policy divergence while maintaining competitive returns in many settings.

Critical strengths
  1. This paper studies behavioral differences across independent RL training runs, beyond the commonly reported variance in returns. This is relevant to reproducibility and reliable deployment.

  2. The experiments cover 18 continuous-control tasks and two base algorithms. The reported reduction in cross-run policy divergence is substantial. The authors also include action-distance and qualitative trajectory demonstrations, which provide additional evidence of behavioral similarity.

Critical weaknesses
  1. If different training runs achieve similar rewards but exhibit different behaviors, and some behaviors are less safe than others, this may indicate reward misspecification. Making the training runs more consistent does not address this underlying problem.

  2. I agree that reducing variability across training runs is useful for scientific reproducibility. However, diversity is also important for exploration. For example, classical RL often collects trajectories from multiple environments, while methods such as GRPO rely on sampling multiple responses to obtain diverse behaviors and learning signals. It is unclear how the proposed method would fit into these settings or whether encouraging consistency could limit exploration. It would be interesting to see experiments on exploration tasks such as PointMaze and AntMaze to understand whether QED causes more substantial performance degradation in these settings.

Questions for authors

How does QED help reduce unsafe behavior, as motivated in the introduction? Unsafe behavior may result from reward misspecification, while QED only encourages behavioral similarity without guaranteeing that the learned behavior is safe. Could the authors clarify how behavioral consistency addresses this issue?

How does QED affect exploration? While reducing cross-run variability is useful for reproducibility, diverse behaviors are often important for exploration. For example, classical RL uses multiple environment rollouts, and GRPO relies on sampling multiple responses. How would QED apply to these settings? It would be helpful to understand this tradeoff from both theoretical and empirical perspectives, particularly whether QED leads to larger performance drops on sparse-reward tasks such as PointMaze and AntMaze.

Given that RL training for LLMs can already be relatively stable, would applying QED to GRPO on reasoning tasks such as MATH-500 provide meaningful variance reduction, or would it primarily slow convergence by limiting exploration?

2 comments

Dear Reviewer ZBQa

Thank you for for your feedback and for highlighting the substantial improvement in cross-run policy divergence.

If different training runs achieve similar rewards but exhibit different behaviors, and some behaviors are less safe than others, this may indicate reward misspecification. Making the training runs more consistent does not address this underlying problem.
Correct. Our paper does not study reward misspecification. It is unclear to us why this is listed as a weakness. In fact, if the reward is indeed misspecified, a behaviorally consistent agent might make it easier to identify this misspecification.

Exploration
There seems to be a misunderstanding. Having similar action distributions does not necessarily mean that these distributions are low variance. Larger entropy means more exploration not less exploration and QED explicitly increases temperature when the estimator is uncertain. QED does not limit exploration. In fact, it increases exploration. We already have experiments on a very hard exploration task, namely hopper_hop (see Appendix E.3). We show that our method improves performance on this exploration task. The key thing that can cause confusion is that across training runs our method explores in the same way; it does not explore less.

How does QED help reduce unsafe behavior
We do not claim it helps reduce unsafe behavior. The text states it helps with safety validation. If we have trained a policy that we think is safe, and we have to retrain it at any point, we know that the new policy is also safe. We will change this wording to behavior validation to avoid future confusion.

GRPO relies on sampling multiple responses. How would QED apply to these settings?
QED can be extended to the base GRPO estimator, if we can get two state-dependent estimates for the return to estimate uncertainty. That should be relatively easy to do. However, our paper is not about language modeling so it is unclear why including GRPO would strengthen the paper or what additional findings we could have from it.

Given that RL training for LLMs can already be relatively stable, would applying QED to GRPO on reasoning tasks such as MATH-500 provide meaningful variance reduction, or would it primarily slow convergence by limiting exploration?
As mentioned above, QED does not decrease exploration. Variance across training runs is different from variance within training runs. Two policies trained with different runs that have very high entropy are very similar. That is true independent of domain. We would expect similar across-run results to emerge for language modeling tasks.

Thank you for the detailed response and clarifications. I appreciate the distinction between cross-run consistency and within-run exploration.

My main concern remains that higher action entropy does not necessarily guarantee more effective exploration, particularly in sparse-reward environments. While the hopper hop results are encouraging, they do not fully address this concern. I also remain concerned about the trade-off between behavioral consistency and sample efficiency.

Overall, I appreciate the contributions, but my concerns remain, so I will maintain my original score.

Reviewer_J248OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses behavioral consistency across independent training runs of RL, an interesting topic. As illustrated in Figure 1, independently trained policies can converge to different behaviors, and I expect this to occur more often in larger environments than in smaller ones. To address this, the paper relates the divergence between Boltzmann policies to the disagreement between their critics, and based on this relationship, adjusts the strength of entropy regularization in a state-dependent manner (QED), using the disagreement between the two critics within a single run as a proxy for cross-run disagreement. The paper reports that this reduces the pairwise policy divergence across runs and, for MAD-SAC, the return variance.

Critical strengths

The paper introduces the issue of behavioral consistency to the reinforcement learning problem. As the paper states, this matters significantly when identifying the cause of a behavior shift after changing the reward, that is, whether the shift is due to the change in the reward or to randomness across training runs, and also when deploying LLMs. In addition, the theoretical connection the paper establishes between policy divergence and critic disagreement in maximum-entropy RL is meaningful.

Critical weaknesses

No experiments in the settings that motivate behavioral consistency (most critical): The paper motivates behavioral consistency mainly through reward design, debugging, and repeated retraining, where one needs to attribute a behavior shift to a change in the reward rather than to randomness across runs. However, all experiments use a fixed reward on dm control tasks, and none of these motivating settings is evaluated. As a result, the practical impact of the method is not sufficiently demonstrated. I consider this the most critical weakness.

Trade-off between performance and consistency: I acknowledge the value of behavioral consistency, but it should be achieved without excessively sacrificing final performance. The cost of QED is not consistent across environments. For MAD-SAC, QED improves both the mean return and the dispersion on some tasks, reduces the mean return on others, and nearly fails on pendulum swingup, with similar failures for SAC-LN (Figure 8).

No analysis of the state-dependent temperature: Algorithmically, the main difference from SAC is that α\alpha is determined per state. An analysis of how α(s)\alpha(s) varies across states, and which states receive a large α\alpha, would demonstrate the significance of the state-dependent temperature for behavioral consistency, but the paper provides none. Without it, it remains unclear whether the state dependence contributes beyond a global temperature schedule.

Questions for authors

Q1. The per-task results suggest that the relationship between mean return and return variance under QED differs across tasks. For MAD-SAC (Figure 9), QED improves the mean return while reducing the dispersion on tasks such as finger turn hard, finger turn easy, hopper hop, and fish swim, whereas it reduces the dispersion at the cost of mean return on tasks such as quadruped run, walker run, and cheetah run, and nearly fails on pendulum swingup. Is this difference simply due to the environments, and if so, which properties of a task determine whether QED improves both or trades one for the other?

Q2. In a setting where the reward is fixed, one could run several seeds and deploy the best-performing policy, which reduces the practical need for behavioral consistency. In the setting the authors highlight, where the reward is modified during reward design, does QED still keep the cross-run divergence small after the reward changes, or does its effect depend on the reward function and the environment?

Q3. Does the state-dependent temperature α(s)\alpha(s) converge by the end of training, or does it keep changing? Since α(s)\alpha(s) depends on the critics, a continuously changing α\alpha would make the soft-optimal policy a moving target. Could the authors report the evolution of α(s)\alpha(s) over training, for example its mean and spread across states, and whether it reaches the floor?

If α(s)\alpha(s) reaches the floor, the difference from the baseline would arise only from the early high-temperature phase. In that case, could the authors compare QED with SAC trained with α\alpha fixed at the value QED reaches by the end of training, and with SAC following QED’s average temperature schedule without state dependence? The latter would show whether the state dependence of α(s)\alpha(s) contributes beyond a global annealing schedule.

2 comments

Dear Reviewer J248,

Thank you for your feedback and for highlighting the importance of the problem we study.

No experiments in the settings that motivate behavioral consistency (most critical):
It is unclear to us which experiment is requested. There may be a misunderstanding around this point. See response to Q2. We do not agree that something critical is missing here to make the claims we make.

Trade-off between performance and consistency
Inconsistent performance changes are not uncommon in the RL literature because there are generally lots of moving components when one changes a piece of an algorithm. As such they are not only present in our work but can be found consistently throughout the literature where they have been accepted. For example, the recent XQC algorithm [3] provides state-of-the-art results comparable with that of SimbaV2 [4]. However, if we look at individual tasks such as dog_walk and pendulum_swingup, which algorithm dominates is task dependent. Such artifacts are usually averaged out by providing aggregate comparisons over a large number of environments so that one does not get stuck in over-optimizing for individual tasks. We do the same thing by averaging over 18 different tasks with different properties including difficult and easy, sparse and dense rewards, etc.. Moreover, our goal is to show that QED reduces cross-run behavioral divergence without systematically sacrificing performance, which our evidence consistently supports. Our paper does not claim to fully resolve the problem of BRL but it is the first one to study it. We expect future algorithms to improve on our approach.

No analysis of the state-dependent temperature
States in which the Q-function is uncertain receive high α\alpha. We already provide a comparison against a fixed set of α\alpha values in prev Appendix E.2, now section 6.3. We have also added a new experiment in Appendix now E.2 that shows how within-run critics disagree on individual out-of-distribution states and how this correlates with cross-run Q disagreement.

Q1: What determines performance impact
There are three things that are already outlined in the paper and that affect the performance of the base algorithm when using QED: 1.) Section 5.2 extensively illustrates how overestimation can happen if data is not collected sufficiently fast. 2.) In our limitations section E.4 we acknowledge that sometimes disagreement can take a long time to vanish. The model might only start learning once temperature has decreased on some tasks because it requires a lot of precision. 3.) QED increases exploration as we outline in E.3. As a result, tasks that have sparser reward such as hopper tend to see performance gains.

Q2: Run several seeds and deploy the best performing policy.
We have demonstrated in Figure 1 on page 2 that individual policies independent of their performance can have different behavior even if the performance is very high and similar.

Reward changes:
There seems to be a misunderstanding. We do not argue that the policy stays the same if the reward changes. The reward should determine the behavior and if the reward changes, clearly the behavior should change. The point that L044 is making is the following: Suppose we are doing iterative reward design where we adjust our reward function until we find a specific behavior. We pick a reward, examine the behavior, and if we don’t like the behavior, we iterate. If now in the second iteration the policy looks different, with classical algorithms we cannot determine whether this is due to randomness or because the reward change we made was effective. This is not critical to any of our claims.

Q3: Does the temperature converge.
This is task dependent as we highlight the limitations in E.4. We added the plot of all environments average α(s)\alpha(s) over time in Appendix H. For many environments, α\alpha is high early and decreases as expected. However, we don’t necessarily expect it to converge since it is measured over the whole buffer which includes states that the robot might not see again. If the robot does not see these states, we expect uncertainty to stay high which may not be a problem for our approach. Since we don’t know which states are out-of-distribution for the agent, we cannot easily determine a better metric but we are open to suggestions.

Show that state dependence contributes
We already have an experiment that illustrates this in prev E.2, now 6.3. We agree that this result is important and have used the space in the rebuttal submission to move it to the main text. We test against various fixed values of α\alpha. A comparison against tuned alphas per environment would be unusual and unfair since our approach is not tuned per environment. Similar to before, the important point is aggregate performance over tasks not performance within individual environments.

Thank you for the response.

Regarding W1 and Q2, while I think my point was misunderstood, I also acknowledge that I overstated it by calling it the most critical weakness. I did not suggest that the policy should remain the same after the reward changes. Rather, I find the reward-design motivation novel and would like to see it demonstrated. When the reward is modified, the behavior of existing methods can vary widely across runs, and the degree of this variation can itself depend on the reward function. Currently, the effect of QED is confirmed only under the given fixed rewards, and an experiment showing that QED remains consistent under each modified reward, while existing methods do not, would make the claim of the paper much stronger.

The answer based on Figure 1 is clear, and Appendix E.2 supports the validity of the proxy. However, what I asked in Q3 included the standard deviation of α(s)\alpha(s) across states during training, beyond its mean, which neither the paper nor the response reports. If α\alpha varies little across states at a given point in training, QED is effectively equivalent to a global temperature schedule.

Reviewer ZBQa’s first weakness seems to reflect a related concern: behavioral consistency depends not only on the RL algorithm but also on how the reward is defined. Since this interplay is not examined, I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.