acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Ensemble Aggregation for Actor-Critics

Abstract

Ensembles are standard in off-policy actor-critic learning, but their efficacy depends critically on how they are aggregated. Many methods construct two ensemble-based reference estimators: a critic reference for bootstrapping and an actor reference for policy optimization. Existing approaches typically control these references through prescribed aggregation rules or externally specified hyperparameters. We introduce Adaptive Ensemble Aggregation (AEA), which calibrates both references based on actor-critic training dynamics. Extensive empirical evaluations study AEA across three settings: interactive learning, sample-efficient learning, and learning with external data collected by other behavior policies. Across multiple continuous-control tasks and external datasets of varying behavioral quality and coverage, AEA performs competitively against strong baselines while using the same AEA update throughout.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Actor-Critic Methods, Ensemble

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AyVUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AEA, an adaptive ensemble aggregation method for actor-critic reinforcement learning. It learns separate aggregation coefficients for actor optimization and critic bootstrapping using a disagreement-normalized majority-vote update. Experiments across MuJoCo and DMC tasks, including high-UTD and external-data settings, show competitive performance against several ensemble actor-critic baselines.

Critical strengths
  1. The method is simple and practical, allowing actor and critic aggregation coefficients to adapt during training with little additional computational overhead.
  2. The experiments cover a broad range of tasks and training settings, including standard online RL, high-UTD learning, and learning with external data. The results show promising improvements over several relevant baselines.
Critical weaknesses

All critics are trained on the same replay minibatches and Bellman targets, which may cause their predictions to become highly similar despite different initializations. This is particularly concerning in high-UTD settings. Since AEA relies on disagreement to adjust both actor and critic updates, it is unclear whether the learned coefficients continue to have a meaningful effect throughout training.

The fixed-coefficient ablation shows mixed results, with fixed coefficients outperforming adaptive AEA in several external-data tasks. The paper also lacks comparisons with alternative coefficient adaptation rules, making it difficult to assess the specific benefit of the proposed majority-vote update.

Questions for authors
  1. Since all critics share the same replay data and Bellman targets, how does ensemble disagreement evolve during training? Could the authors report both the disagreement magnitude and the effective aggregation terms over time, particularly in high-UTD settings?

  2. Given that fixed coefficients perform better in several external-data tasks, what is the main advantage of continuous adaptation? How does the majority-vote update compare with simpler alternatives such as L2- or Huber-based calibration?

3 comments

We thank the reviewer for the constructive feedback and for recognizing the simplicity, practicality, and minimal computational overhead of AEA across diverse training settings.

1. Evolution of Ensemble Disagreement Under Shared Targets in High-UTD Settings

Shared replay minibatches and Bellman targets are standard in SAC and REDQ, so this design is not specific to AEA. They do not force independently initialized neural critics to make identical predictions during finite training. In AEA, the critics are fitted only approximately while the policy and replay distribution continue to evolve. Empirically, we observe that ensemble disagreement does not collapse during high-UTD training. We can include the corresponding disagreement curves in the revised appendix if the reviewer finds them useful.

2. Main Advantage of Continuous Adaptation vs. Hindsight Fixed Coefficients

Please refer to our general response on adaptive versus fixed coefficients. A hindsight-selected pair is only known after training. Beforehand, one must either gamble on a static choice or run a separate per-task search. AEA instead calibrates the coefficients within the training run, providing a practical solution without an additional tuning stage.

3. Why Majority-Vote Over L2L_2- or Huber-Based Calibration

We kindly ask the reviewer to refer to our general response on majority-vote calibration. In particular, both L2L_2 and Huber let residual magnitude influence the update: L2L_2 gradients scale with the residual, while Huber retains this dependence within its quadratic region and clips larger gradients. We instead take a majority vote of the residual signs so that the calibration of the scalars is more stable; we do not claim this rule is optimal, but it is simple and effective.

Thank you for the clarifications. I appreciate the explanation regarding hindsight-selected fixed coefficients and the motivation for majority-vote calibration.

My main concern remains the effectiveness of disagreement-based adaptation during training. I agree that shared replay and Bellman targets are standard in SAC and REDQ. However, the key question is whether ensemble disagreement remains sufficiently large for the learned coefficients to meaningfully affect the actor and critic updates, particularly under high UTD.

Since the authors mention that disagreement does not collapse empirically, it would be helpful to include the disagreement magnitude and effective aggregation terms (∣κδ∣|\kappa\delta| and ∣κˉδˉ∣|\bar{\kappa}\bar{\delta}|) over training. These would provide more direct evidence of the proposed adaptation's practical impact.

I would be happy to reconsider my assessment based on these results.

We thank the reviewer for suggesting these diagnostics. Since we cannot upload figures during rebuttal, we summarize the observations below and will include disagreement and effective aggregation trajectories in the revision.

As a general pattern, Q-values and disagreement are initially both small. On many tasks, disagreement rises rapidly within the first 0.1M interactions, as critics begin fitting and generalizing from the limited available experience, followed by further adjustment over approximately 0.1–0.3M. It may then decline (around 50%), persist with fluctuations (around 30%), or increase again (around 20%), depending on the setting and environment. These patterns vary across settings and environments; however, all three can accompany successful learning, depending on what the tasks need. In most cases, we also observe approximately coincident transitions in disagreement and the learned Q-values.

Lastly, we would like to emphasize that AEA’s practical effectiveness is supported by concrete learning results. Given how sensitive deep RL can be to its update rules, it is meaningful that the same calibration method based on disagreement performs well across a wide variety of settings and outperforms strong baselines on many tasks. In the high-UTD setting alone, AEA achieves the highest reported mean AULC on 10 of the 14 tasks, reflecting fast, effective learning and strong overall performance over the training budget.

Reviewer_tbM1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Adaptive Ensemble Aggregation (AEA), a method that adaptively learns how to aggregate ensemble critics in off-policy actor-critic RL. Instead of using fixed aggregation rules as in REDQ, AEA learns two coefficients to control the level of pessimism and optimism for critic and actor updates separately. The coefficients are updated using a majority-vote mechanism based on the sign of prediction errors, which avoids large errors dominating the updates. The paper also provides convergence analysis under tabular settings. Experiments are conducted on MuJoCo and DMC benchmarks under three settings: interactive learning, sample-efficient learning, and learning with external data. The results show that AEA achieves competitive performance compared to REDQ in the sample-efficient setting and RLPD in the learning with external data setting.

Critical strengths
  1. This paper is well-motivated and clearly written. The overall structure is well organized and easy to follow. The proposed method itself is also simple and intuitive
  2. The paper provides extensive experimental evaluation across differnet learning settings, including interactive learning, sample-efficient learning, and learning with external data. The additional analysis, such as the learned coefficient trajectories, fixed-coefficient ablations, and initialization sensitivity, are also useful. They help answer some natural questions that arise when reading the paper, and provide more insights into how the proposed method behaves during training.
Critical weaknesses
  1. The benefit of adaptively learn the aggregation coefficients is not clearly. In Appendix E.2, the fixed-coefficient version sometimes performs better than AEA. Although these fixed coefficients are selected in hindsight, it still raises questions about whether continuous adaptation is necessary. Also, the actor coefficient seems to change very little during training on many tasks
  2. The empirical improvements over strong baselines are not very consistent. In the sample-efficient setting, AEA only outperforms REDQ on 2 out of 5 MuJoCo tasks, and the improvements on several DMC tasks are relatively small. Similarly, in the learning with external data setting, AEA shows strong improvements on Humanoid and HalfCheetah, but performs worse than RLPD on all Ant datasets and most Hopper datasets. Although the experiments are extensive, the overall performance gains seems relatively limited and highly dependent on the environment.
Questions for authors

One question I have is about the hyperparameter tuning. The paper motivates AEA as a way to avoid manually tuning the aggregation coefficients. However, learning these coefficients also introduces a new coefficient learning rate hyperparameter. Since REDQ and RLPD already use relatively fixed aggregation rules across environments, it is not very clear how much hyperparameter tuning AEA actually saves. I feel the practical benefit seems rather marginal.

1 comment

We thank the reviewer for the positive assessment of our paper's motivation, clarity, and extensive experimental and diagnostic evaluations across all three regimes. We address each concern below.

1. Necessity of Continuous Adaptation vs. Hindsight Fixed Coefficients

We appreciate the reviewer highlighting the relatively small changes in the actor coefficient. We see this finding as a strength of our study, not a disadvantage. Indeed, it is an interesting finding of AEA’s calibration: the learned trajectories reveal that the actor coefficient can remain relatively steady (but is still not fixed) on many tasks while the critic coefficient continues to change. Making these distinct dynamics observable provides insight into how actor and critic aggregation evolve during training. Regarding fixed coefficients, please refer to our general response. In particular, when an AEA-informed fixed pair performs well, this shows that calibration can identify useful coefficients; when freezing it reduces performance, continued adaptation provides further benefits relative to that pair.

2. Consistency of Empirical Improvements Across Regimes and Suites

A key empirical result is that AEA works across three learning regimes (interactive, sample-efficient high-UTD, and learning with external data) and 2 suites (MuJoCo and DMC) using the same calibration rule and hyperparameters. We note that no method in our comparisons performs best on every task. Nevertheless, AEA achieves higher reported IQM than REDQ on 10 of 14 sample-efficient tasks, including 8 of 9 DMC tasks, and substantial gains on Humanoid with medium and expert external data, although RLPD remains stronger on some environments. This breadth of applicability, without per-task coefficient search, is a strong practical result.

3. Does AEA Actually Save Hyperparameter Tuning?

REDQ’s fixed aggregation rule still encodes specific design choices: the minimum of M=2M=2 randomly selected critics for bootstrapping and the ensemble mean for the actor. Its original ablations compare different subset sizes and favor M=2M=2. These choices are simple, but leave open whether different aggregation tendencies would be useful across tasks and training stages. AEA investigates this question by learning those tendencies and making their evolution observable.

USAC explores similar flexibility through a 5×35\times 3 coefficient grid per task. GPL-SAC retains task-specific target entropies: HalfCheetah −3-3 (usual −6-6), Hopper −1-1 (−3-3), Walker2d −3-3 (−6-6), Ant −4-4 (−8-8), and Humanoid −2-2 (−17-17). Its pixel-based DMC variant also changes Walker-run’s return horizon from 3 to 1 and batch size from 256 to 512, and reduces Quadruped-run’s replay capacity from 10610^6 to 10510^5. These task-specific configurations remain even within online-only evaluation, without evaluating learning with external data (GPL, Tables 2–3).

AEA performs calibration within the training run, without a separate coefficient search. Using a learning rate to optimize learnable parameters is standard practice; its presence alone does not imply additional per-task tuning. We keep ηκˉ=ηκ=0.1\eta_{\bar{\kappa}}=\eta_{\kappa}=0.1, along with the same entropy rule, unchanged across all evaluated tasks and settings. The practical benefit is therefore learning and understanding task-dependent aggregation without additional per-task calibration tuning.

Reviewer_sP7pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Adaptive Ensemble Aggregation (AEA) for off-policy actor-critic reinforcement learning. AEA constructs separate critic and actor reference estimators by shifting the corresponding ensemble mean by a learned multiple of pairwise critic disagreement. The critic reference determines bootstrapping targets, while the actor reference guides policy optimization.

The main methodological contribution is a calibration rule derived from disagreement-normalized absolute losses. Each transition contributes the sign of a regression residual, preventing large residual magnitudes from dominating coefficient updates. The paper establishes connections to SAC and REDQ, characterizes the calibration losses through empirical medians, and proves convergence under idealized exact tabular evaluation and policy iteration.

Experiments cover interactive learning, high-update-to-data learning, and online learning augmented with external datasets. AEA performs particularly well in the high-reuse setting and on several external-data conditions, while remaining competitive in standard interactive learning.

Critical strengths
  1. A simple aggregation mechanism with a clear derivation and limited additional machinery.

    AEA adds two scalar coefficients without auxiliary networks or separate return-estimation buffers. Its normalized absolute losses yield bounded per-transition contributions, providing a concrete mechanism for limiting the influence of unusually large regression residuals. Proposition 1 also places SAC and the conditional expected REDQ reference within the same parameterization, making the relationship to established methods precise.

  2. Broad empirical evaluation with meaningful gains in demanding settings.

    The paper evaluates both limited and heavy data reuse, as well as external datasets of different behavioral quality. Comparisons with REDQ and SAC-N match ensemble size and update-to-data ratio in the sample-efficient setting. Several improvements are substantial: for example, external-data Humanoid-expert IQM increases from 5,375 for RLPD to 9,933 for AEA. The inclusion of final return, IQM, AULC, coefficient trajectories, and fixed-pair experiments makes the empirical evidence more informative than a single headline comparison.

Critical weaknesses
  1. The evaluation does not sufficiently isolate the proposed calibration rule or compare it with the closest adaptive alternatives.

    Appendix A identifies GPL, OPL, VPL, TOP, and USAC as closely related approaches, but the reported performance comparisons primarily involve fixed aggregation rules or different uncertainty-estimation mechanisms. These comparisons establish practical competitiveness but do not directly demonstrate an advantage over existing adaptive aggregation methods.

    Likewise, the ablations do not separate actor-only calibration, critic-only calibration, simultaneous calibration, and residual-magnitude-based alternatives. The hindsight fixed-pair comparison is useful, but it cannot establish whether the transition-wise voting rule itself accounts for the improvements. This is consequential because the paper’s central novelty lies in how aggregation is learned.

  2. The interpretation and validation of the two-coefficient dynamics remain incomplete.

    Equations (8) and (10) use the same residual-sign statistic with opposite update directions. If both coefficients use the same pre-update residual and direct fixed-step updates, their changes satisfy

    = -\gamma \frac{\eta_{\bar{\kappa}}}{\eta_\kappa} \Delta\kappa$$ before clipping. Thus, under this implementation, the coefficients do not move independently in the interior of the feasible region. Sequential residual recomputation or clipping can change this relationship, making the precise implementation important. Moreover, agreement between two learned references is a self-consistency criterion, rather than direct evidence of reduced value-estimation error. The tabular theorem holds for arbitrary bounded coefficient sequences, so it does not specifically justify the proposed voting rule. Direct measurements of estimation bias and targeted coefficient ablations would strengthen the methodological argument considerably.
Questions for authors
  1. Coefficient-update implementation: Are the residuals in the two calibration updates evaluated from the same pre-update quantities, or recomputed after updating κˉ\bar{\kappa}? If they share the same residual, how should the coupling above affect the interpretation of separately learned actor and critic aggregation?

  2. Closest adaptive baselines: How does AEA compare with GPL or another closely related adaptive-pessimism method under the same backbone and evaluation protocol? Such a comparison would materially affect my assessment of the methodological contribution.

  3. Mechanism isolation: What happens when only the critic coefficient or only the actor coefficient is learned? How does transition-wise voting compare with an appropriately stabilized residual-magnitude-based update?

  4. Robustness and numerical handling: How are zero or near-zero disagreement values handled? Are the simplified sign updates implemented directly, or are the normalized losses differentiated numerically? How sensitive are results to the coefficient step size of 0.1 and the clipping interval? The initialization study currently uses only one seed.

  5. External-data comparison: Can the AEA–RLPD comparison be repeated with entropy initialization, target entropy, and other backbone choices matched? Appendix C.3 specifies different entropy configurations, which leaves uncertainty about how much of the improvement comes from aggregation.

3 comments

We thank the reviewer for the insightful review and highlighting our strenghts. We address each concern below.

1. Coefficient-Update Implementation

The updates are sequential: we first update κˉ\bar{\kappa}, then recompute the target and residual before updating κ\kappa. Thus, the shared-pre-update-residual premise does not describe our implementation. The coefficients serve distinct actor and critic roles but we do not claim independent dynamics. As with actor–critic updates motivated by policy iteration, coupling between sequentially updated components does not undermine their distinct roles. We will make the update order and residual recomputation explicit in Algorithm 1.

2. Comparison with Closest Adaptive Baselines

Most of the mentioned methods rely on very specific designs and even substantial task-specific configurations.

GPL focuses its main SAC comparisons on sample-efficient online learning with UTD=20, one of our three settings. Its UTD=1 ablation reports instabilities on Ant and Humanoid (Appendix D). The task-specific configurations include, but are not limited to, the following. Instead of the usual target-entropy rule −dim⁡(A)-\dim(\mathcal{A}), GPL-SAC uses −1-1 for Hopper (usual −3-3), −3-3 for HalfCheetah (−6-6), −3-3 for Walker2d (−6-6), −4-4 for Ant (−8-8), and −2-2 for Humanoid (−17-17). In pixel-based DMC, GPL-DrQ changes Walker-run’s return horizon from 3 to 1 and batch size from 256 to 512, and reduces Quadruped-run’s replay capacity from 10610^6 to 10510^5. GPL-DrQ-Anneal additionally schedules optimism from 0.5 to 0 over 500K steps. These experiments cover online interaction only; external-data learning is not evaluated, and offline RL extensions are left to future work.

USAC evaluates only five MuJoCo tasks under N=2,G=1N=2, G=1, covering one of our two benchmark suites and one of our three learning settings. Even within this scope, it selects coefficients through a 3×53\times 5 grid search for each task, for which the authors explicitly acknowledge as limitations. Extending its 15-configuration search to every task condition in our full benchmark would constitute a extensive scale of separate tuning study.

TOP selects from the prescribed set β∈{−1,0}\beta\in\{-1,0\}, applies the same selected coefficient to actor and critic, and updates the selection using episodic return differences. Its pixel-based DMC experiments also retain explicit environment-specific configurations: encoder, actor, and critic learning rates are 2×10−42\times 10^{-4} for Cheetah-run but 10−310^{-3} for other tasks; Walker-walk uses cropping to 84×8484\times 84, whereas other tasks use translation with 108×108108\times 108 images; action repeat is 2 for Finger-spin and Walker-walk, 8 for Cartpole-swingup, and 4 otherwise.

Our experiments evaluate the practical value of one calibration mechanism across three learning settings, with shared calibration hyperparameters and no separate per-task coefficient search. Extending each related method to every task condition would require additional implementation, configuration choices, and validation. We did not undertake these separate extension and tuning studies, and therefore discuss these methods as related approaches rather than empirical baselines. We do not consider completing all such studies a prerequisite for establishing AEA’s practical contribution.

3. Calibration Mechanism

Please refer to our general response on majority-vote calibration. We have not separately evaluated these variants. However, the joint calibration rule is evaluated across three learning regimes using the same calibration hyperparameters, alongside fixed-pair ablations with the backbone held fixed. These results support the practical effectiveness of the proposed mechanism.

4. Robustness and Numerical Handling of Near-Zero Disagreement

We use the same calibration rule to learn the parameters throughout our experiments: on 2 suites and 3 learning regimes, where within the hybrid setting, there are also 3 types of offline datasets. We believe that these results provide broad empirical evidence of robustness across tasks, update regimes, and external data qualities, without task-specific calibration tuning. To handle near-zero disagreement, we use a numerical floor εδ=10−4\varepsilon_{\delta}=10^{-4}. Using a small numerical floor to avoid division by zero is a common practice in deep learning implementations.

5. External-Data Comparison and Entropy Configuration

RLPD is also based on SAC, but uses its own entropy settings: α0=1.0\alpha_0=1.0 and target entropy −dim⁡(A)/2-\dim(\mathcal{A})/2. For AEA, we keep the usual SAC target entropy of −dim⁡(A)-\dim(\mathcal{A}) and the same initialization, α0=0.2\alpha_0=0.2, across all three settings. We would like to emphasize that, with or without external data, and whatever the quality of the external data, AEA does not require additional entropy tuning. More specifically, AEA’s entropy initialization and target entropy are unchanged across all three regimes.

I sincerely thank the authors for their feedback. However, I still have reservations regarding the advantages of the proposed method over other baseline approaches. Furthermore, I have not yet seen the updated manuscript incorporating the revisions. Based on the above, I will maintain my initial score.

Reviewer_sP7pOpenReview ↗

Response

I referred to the author's response. The author's reply addressed some of my concerns, but regarding External-data and External-experiment, I still haven't received a satisfactory response. Therefore, I will maintain my original score.

Comments on the paper

3 comments
~Haque_M_Ishfaq1OpenReview ↗

Comparison With Langevin Soft Actor Critic (LSAC)

Hi authors,

We had a highly related work from ICLR 2025 : Ishfaq, H., Wang, G., Islam, S., & Precup, D. (2025, May). Langevin soft actor-critic: Efficient exploration through uncertainty-driven critic learning. In International Conference on Learning Representations.

In this paper, we consider Langevin Monte Carlo based Thompson Sampling for uncertainty estimation in critic in SAC and DSAC for sample efficient RL with improved exploration. Many of the motivations in your paper aligns with ours. So, we would appreciate if you could consider citing our work as well as use LSAC as a baseline for your experiments.

We open sourced our code as well: https://github.com/hmishfaq/LSAC

We thank all reviewers for their constructive feedback and for recognizing the simplicity and practicality of AEA, its low computational overhead, the clarity of our presentation, and the breadth of our empirical evaluation across extensive challenging settings: interactive learning (N=2,G=1N=2, G=1), sample-efficient high-UTD learning (N=10,G=20N=10, G=20), and hybrid learning with external datasets of varying behavioral quality (simple, medium, expert). Before addressing each review point-by-point, we summarize two main questions:

  • Why continuous online adaptation rather than fixed coefficients (Reviewers sP7p, tbM1, AyVU): The fixed-pair ablation is not intended to claim that online adaptation always outperforms any fixed pair. In fact, we deliberately use a strong comparator: a pair selected in hindsight from AEA’s learned coefficients, and thus informed by the data. USAC (Tasdighi et al., 2025) already examined pre-selected fixed pairs and found that suitable values vary across environments. The difficulty without online calibration is choosing such a pair before training: one must either gamble on a static choice or run a costly two-dimensional grid search for each task. USAC evaluates a 5×35\times 3 grid per task, without establishing an optimal pair. On the other hand, AEA offers a simple, effective way to calibrate coefficients during training.

  • Why majority-vote (sign) calibration (Reviewers sP7p, AyVU): This is a design choice. We do not expect it to be optimal, but it is simple and effective. One might naturally try to calibrate the coefficients using squared TD errors, but a few very large errors could then make these scalar parameters swing sharply. We want them to guide learning more steadily. Our calibration mechanism instead lets each transition vote on which way to move a coefficient. The majority vote gives a useful direction without letting the magnitude of an extreme error dominate the update.

Dear Reviewers,

The authors have provided rebuttals to your review. Could you please check their responses and provide any further feedback or questions for the authors to address at your convenience?

Thank you for your time and contribution.

Regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.