- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes AEA, an adaptive ensemble aggregation method for actor-critic reinforcement learning. It learns separate aggregation coefficients for actor optimization and critic bootstrapping using a disagreement-normalized majority-vote update. Experiments across MuJoCo and DMC tasks, including high-UTD and external-data settings, show competitive performance against several ensemble actor-critic baselines.
Critical strengths
- The method is simple and practical, allowing actor and critic aggregation coefficients to adapt during training with little additional computational overhead.
- The experiments cover a broad range of tasks and training settings, including standard online RL, high-UTD learning, and learning with external data. The results show promising improvements over several relevant baselines.
Critical weaknesses
All critics are trained on the same replay minibatches and Bellman targets, which may cause their predictions to become highly similar despite different initializations. This is particularly concerning in high-UTD settings. Since AEA relies on disagreement to adjust both actor and critic updates, it is unclear whether the learned coefficients continue to have a meaningful effect throughout training.
The fixed-coefficient ablation shows mixed results, with fixed coefficients outperforming adaptive AEA in several external-data tasks. The paper also lacks comparisons with alternative coefficient adaptation rules, making it difficult to assess the specific benefit of the proposed majority-vote update.
Questions for authors
-
Since all critics share the same replay data and Bellman targets, how does ensemble disagreement evolve during training? Could the authors report both the disagreement magnitude and the effective aggregation terms over time, particularly in high-UTD settings?
-
Given that fixed coefficients perform better in several external-data tasks, what is the main advantage of continuous adaptation? How does the majority-vote update compare with simpler alternatives such as L2- or Huber-based calibration?
3 comments
Thank you for the clarifications. I appreciate the explanation regarding hindsight-selected fixed coefficients and the motivation for majority-vote calibration.
My main concern remains the effectiveness of disagreement-based adaptation during training. I agree that shared replay and Bellman targets are standard in SAC and REDQ. However, the key question is whether ensemble disagreement remains sufficiently large for the learned coefficients to meaningfully affect the actor and critic updates, particularly under high UTD.
Since the authors mention that disagreement does not collapse empirically, it would be helpful to include the disagreement magnitude and effective aggregation terms ( and ) over training. These would provide more direct evidence of the proposed adaptation's practical impact.
I would be happy to reconsider my assessment based on these results.
We thank the reviewer for suggesting these diagnostics. Since we cannot upload figures during rebuttal, we summarize the observations below and will include disagreement and effective aggregation trajectories in the revision.
As a general pattern, Q-values and disagreement are initially both small. On many tasks, disagreement rises rapidly within the first 0.1M interactions, as critics begin fitting and generalizing from the limited available experience, followed by further adjustment over approximately 0.1–0.3M. It may then decline (around 50%), persist with fluctuations (around 30%), or increase again (around 20%), depending on the setting and environment. These patterns vary across settings and environments; however, all three can accompany successful learning, depending on what the tasks need. In most cases, we also observe approximately coincident transitions in disagreement and the learned Q-values.
Lastly, we would like to emphasize that AEA’s practical effectiveness is supported by concrete learning results. Given how sensitive deep RL can be to its update rules, it is meaningful that the same calibration method based on disagreement performs well across a wide variety of settings and outperforms strong baselines on many tasks. In the high-UTD setting alone, AEA achieves the highest reported mean AULC on 10 of the 14 tasks, reflecting fast, effective learning and strong overall performance over the training budget.
We thank the reviewer for the constructive feedback and for recognizing the simplicity, practicality, and minimal computational overhead of AEA across diverse training settings.
1. Evolution of Ensemble Disagreement Under Shared Targets in High-UTD Settings
Shared replay minibatches and Bellman targets are standard in SAC and REDQ, so this design is not specific to AEA. They do not force independently initialized neural critics to make identical predictions during finite training. In AEA, the critics are fitted only approximately while the policy and replay distribution continue to evolve. Empirically, we observe that ensemble disagreement does not collapse during high-UTD training. We can include the corresponding disagreement curves in the revised appendix if the reviewer finds them useful.
2. Main Advantage of Continuous Adaptation vs. Hindsight Fixed Coefficients
Please refer to our general response on adaptive versus fixed coefficients. A hindsight-selected pair is only known after training. Beforehand, one must either gamble on a static choice or run a separate per-task search. AEA instead calibrates the coefficients within the training run, providing a practical solution without an additional tuning stage.
3. Why Majority-Vote Over L2- or Huber-Based Calibration
We kindly ask the reviewer to refer to our general response on majority-vote calibration. In particular, both L2 and Huber let residual magnitude influence the update: L2 gradients scale with the residual, while Huber retains this dependence within its quadratic region and clips larger gradients. We instead take a majority vote of the residual signs so that the calibration of the scalars is more stable; we do not claim this rule is optimal, but it is simple and effective.