acceptodds
Under review as a conference paper at ICLR 2027

In-Distribution Ensemble Variance Suppression Critic for Offline Reinforcement Learning

15 views

Abstract

Offline reinforcement learning (RL) enables learning policies from fixed datasets without costly or unsafe environment interactions. To mitigate value overestimation in this setting, ensemble critics are widely used and have demonstrated remarkable effectiveness. However, we reveal a critical flaw in existing methods: the ensemble variance of Q-values often exceeds the action-value variance conditioned on the same state. This over-dispersion obscures true action advantages and degrades the identification capability for out-of-distribution (OOD) actions, causing policies to chase ensemble noise and resulting in sub-optimal policy optimization. To address this, we propose an In-Distribution Variance Suppression Critic. Specifically, we introduce a variance suppression term that explicitly penalizes ensemble disagreement on in-distribution dataset points, thereby aligning the ensemble uncertainty with the intrinsic action-value variation. Combined with EDAC, a strong ensemble-based offline RL baseline, our method consistently achieve state-of-the-art performance among model-free offline RL algorithms across multiple diverse benchmarks.

Primary area:
reinforcement learning
Keywords:
Offline Reinforcement Learning, Uncertainty, Ensemble Critic, Variance Suppression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4u8cOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a new auxiliary objective for ensemble-based offline RL. It claims that ensemble-based offline RL methods such as EDAC suffer from excessive ensemble disagreement. Specifically, even on in-distribution dataset actions, the variance across Q-ensemble members exceeds the variation of Q-values across actions at the same state. To address this, the paper proposes EVA, a regularizer that penalizes differences between the Q-values of ensemble members on in-distribution state-action pairs. EVA is added on top of EDAC's objective. The paper reports the highest average score compared against state-of-the-art model-free and scalable offline RL methods on D4RL MuJoCo, AntMaze, and OGBench.

Critical strengths
  • The proposed objective is simple, easy to add to existing ensemble-based offline RL methods, and adds negligible overhead.
  • The OGBench results are impressive. Adding an ensemble critic with EVA to FQL substantially improves performance on non-saturated tasks, suggesting that ensemble critics are underexplored in scalable offline RL.
Critical weaknesses
  • It seems that the central claim is only partially supported. Figure 4 shows that, without EVA, ensemble disagreement is similar on dataset and random actions, which supports the claim that EDAC's disagreement does not separate ID from OOD actions. However, the paper's main claim, that ensemble variance exceeds the intrinsic action-value variance at the same state and thereby makes the policy chase noise, is shown only for a single state.
  • The OGBench evaluation covers only the default task of three environments. Results on all five tasks per environment or on additional environments would better support the claim of broad applicability.
  • There are some citation errors in Related Work. Double Q-learning is attributed to Mnih et al. (2013), which is the DQN paper. The correct references should be van Hasselt et al. (2016) [1]. Decision Transformer is attributed to Janner et al. (2021), which is the Trajectory Transformer paper. Decision Transformer is Chen et al. (2021) [2].

[1] van Hasselt et al., Deep Reinforcement Learning with Double Q-learning, AAAI 2016
[2] Chen et al., Decision Transformer: Reinforcement Learning via Sequence Modeling, NeurIPS 2021

Questions for authors
  • Figure 1 qualitatively compares ensemble disagreement with the variation of Q-values along the line between the dataset and policy actions, but only for a single state. Can the authors quantify both quantities (e.g., averaged over many states and tasks), with and without EVA, to support the claim that ensemble variance systematically exceeds the intrinsic action-value variance?
  • Since all critics share in EDAC the same TD target under min aggregation, why do they disagree so substantially on the dataset action in Figure 1? Is this mainly due to EDAC's gradient-diversity term?
1 comment

Thank you for the detailed response and the additional OGBench experiments. Most of my concerns have been addressed, and Figure 2 is a good addition. Therefore, I will raise my score from 2 to 3. One suggestion: rerunning humanoidmaze with different settings seems unnecessary, as a slightly lower score would not weaken the main claims. I encourage keeping the experimental setup consistent across tasks.

Reviewer_b3WNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this manuscript, the authors reveal a critical claw existin in current ensemble critics methods in offline RL: the Q-value ensemble variance is much larger than action variance even condition on the same state. To address this issue, the authors propose In-Distribution Variance Suppression Critic, specifically the EVA (Ensemble VAriance suppression), a variance-suppression term could be easily plugged in existing methods. Experiments across diverse benchmarks demonstrate the superiority of the proposed method.

Critical strengths
  • The studied problem is interesting and critical. The in‑distribution over‑dispersion issue for ensemble critics is indeed a critical problem for offline RL methods. The observation provides general insight for ensemble‑based offline RL.

  • The proposed method is simple and straightforward. The EVA loss can be easily plugged in EDAC and other baseline methods.

  • The experiment studies are comprehensive. The authors consider classic D4RL (MuJoCo, AntMaze) and challenging goal‑conditioned OGBench. The ablation studies are also well designed.

Critical weaknesses
  • Few theorectical analysis: although the studies problem is critical and the proposed solution sounds appealing, there are few theoretical studies explaining why in‑distribution over‑dispersion arises. The authors are also suggested to provide some theorectical analysis on EDA.

  • The potential benefits of the proposed method may come from two sources,one is the suppressing in distribution ensemble variance as the authors claim, the other is EVA acts purely as Q‑output regularizer that stabilizes critic training. Are there any further studies to disentangle these two aspects?

  • Though EVA targets uncertainty quality, real Q-value over‑estimation / under‑estimation is not measured. We cannot confirm whether EVA mitigates value over‑estimation or only changes inter‑critic spread.

Questions for authors
  • An interesting question is that the authors apply EDA on offline RL scenarios, and behavior under online fine‑tuning is unstudied. Could the authors consider offline to online (O2O) scenario in RL? The ensemble uncertainty dynamics remains unknown when new samples come in.

  • It seems the performance of the proposed methods depends on the hyper-parameter β\beta. Is there any guidelines for selecting β\beta, or offline observable metric to pre‑dict β\beta

  • The considers datasets, such as D4RL/ORBenchmark are classic datasets. Do the authors consider other extreme datasets, such as extreme sparse dataset, or datasets with large noise. Is the EDA method still effective under such circumstances?

Reviewer_57FWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EVA, an in-distribution variance regularizer for ensemble critics in offline reinforcement learning. EVA penalizes disagreement between randomly paired critics on dataset state–action pairs, aiming to reduce in-distribution disagreement while retaining useful diversity outside the dataset. The method is incorporated into EDAC-based training and an FQL-based framework, and evaluated on D4RL MuJoCo, AntMaze, and three OGBench tasks. The paper reports improved returns with little additional computational cost, alongside analyses of ensemble disagreement, regularization strength, and ensemble size.

The central claim is that excessive in-distribution ensemble disagreement harms value learning and policy optimization, and that suppressing this disagreement provides a simple remedy.

The paper presents a simple regularizer with promising empirical benefits, particularly in the matched OGBench experiments. However, its main explanatory claims are not adequately established, the claimed anti-collapse benefit of random pairing is not demonstrated, and important experimental attribution questions remain unresolved. My recommendation is driven by these gaps, rather than by the method’s simplicity or the absence of difficult theory.

Critical strengths

The experiments provide meaningful evidence of practical utility.

The matched FQL-EDAC versus FQL-EDAC-EVA comparison increases the reported average score from 22.5 to 47.0 across the three evaluated OGBench tasks. Although the evaluation remains limited in scope, this comparison directly supports the usefulness of adding EVA to the tested ensemble configuration and provides the strongest evidence supporting acceptance. The D4RL locomotion results provide additional positive evidence, with the reported average score increasing from 92.9 for EDAC to 96.2 for EVA, using a common EVA coefficient across the nine tasks.

The method is inexpensive and easy to incorporate into existing ensemble training.

EVA operates on critic predictions at dataset actions and introduces little additional computational overhead. The distinction between encouraging agreement on observed actions and retaining diversity elsewhere is also a useful design motivation. A sufficiently validated regularizer of this simplicity could be practically valuable without requiring a technically elaborate algorithm.

Critical weaknesses

The central explanation of “over-dispersion” is insufficiently defined and validated, and the distinctive role of random pairing remains unclear.

The paper motivates EVA by suggesting that ensemble variance exceeds “intrinsic action-value variance,” but does not clearly define the latter quantity or provide an estimator for it. Disagreement across critics at a fixed state–action pair and variation in true values across actions are different statistical objects. Figures 1 and 4 illustrate substantial in-distribution disagreement, and Figure 4 provides evidence that moderate regularization can improve both returns and the separation between dataset-action and random-action disagreement. These observations support the design motivation. However, they do not establish that disagreement is excessive relative to a well-defined reference, that EVA restores uncertainty calibration, or that reduced disagreement improves value accuracy, action rankings, or policy gradients.

Furthermore, under the stated random pairing scheme, the expected EVA penalty is proportional to the sample variance, as correctly expressed by the paper’s variance identity. Complete agreement gives zero EVA loss and zero EVA gradient for every pairing. Random pairing therefore does not itself exclude this solution or establish an anti-collapse mechanism. The other training objectives may preserve useful diversity, but this needs to be distinguished from a property of EVA.

These issues matter because the paper attributes its benefits to a particular statistical explanation and an anti-collapse effect of random pairing. The current evidence supports empirical regularization benefits more convincingly than these specific claims. A useful variance regularizer would remain a valid contribution even without a distinctive pairing mechanism, but its contribution should be characterized accordingly.

The experimental evidence does not fully isolate EVA’s contribution or support the breadth of the generality claims.

AntMaze lacks a matched no-EVA baseline using the same target construction, actor objective, ensemble size, and training settings. Consequently, comparison with other methods does not establish how much of the improvement comes from EVA rather than the surrounding training recipe.

The OGBench comparison is stronger, and the regularization-strength and ensemble-size ablations provide useful sensitivity evidence. However, evaluation covers only three tasks, and the selection protocol for task-specific coefficients and the tuning budgets for the corresponding baselines are not reported. Clarifying these choices would help assess robustness and the fairness of the comparison.

Algorithmic ambiguity further limits interpretation. Equation (9) describes dataset-action cloning, whereas the OGBench experiments state that the FQL actor is unchanged. The task-specific actor objectives and the treatment of entropy in the EDAC-based actor and TD targets should be explicit. Finally, the broad claim about being the first to demonstrate ensemble benefits should be narrowed: the strongest controlled evidence here concerns EVA added to an ensemble configuration.

Together, these gaps leave uncertainty about the contribution’s scope and reproducibility, despite the promising reported gains.

Questions for authors

Definition and mechanism: What precisely is “intrinsic action-value variance,” including its source of randomness or reference distribution? What evidence establishes excessive disagreement relative to this quantity, and connects its reduction to improved value accuracy, action rankings, or policy gradients?

Random pairing: Have you compared EVA with a direct sample-variance penalty under matched expected loss scaling and identical training settings? If so, does random pairing provide a reproducible advantage? This would clarify whether pairing offers a benefit beyond implementing variance regularization stochastically, and help assess the claimed anti-collapse mechanism.

AntMaze attribution: What are the results when EVA is removed by setting β=0\beta=0, while retaining the same target construction, actor objective, ensemble size, and training budget? This comparison could materially change my assessment of the evidence beyond OGBench.

Implementation and model selection: Please specify the actual actor objectives and entropy treatment for each experimental suite, and explain how the regularization coefficients were selected, including the selection criterion, search budget, and whether the no-EVA baselines received comparable tuning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.