- Rating
- 3
- Confidence
- 1(−1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a unified view of instability in deep reinforcement learning. Rather than treating exploration, policy improvement, and neural optimization as a simple sequential pipeline, the authors model them as multiple concurrent processes operating on a shared learner state.
The learner state is written as
$ S_t = (\theta_t, D_t, L_t), $
and the training process is decomposed into three operators:
$ T^{\mathrm{exp}}, \qquad T^{\mathrm{pol}}, \qquad T^{\mathrm{opt}}, $
corresponding to exploration/data collection, policy improvement, and neural optimization.
The central concept introduced by the paper is hyperparameter collusion: a suboptimal design choice in one component can be accidentally compensated for by side effects from another component, leading to fragile and highly coupled hyperparameter dependencies. For example, small batch sizes may appear necessary not because they optimize the objective better, but because optimizer noise unintentionally contributes to exploration; similarly, overly conservative PPO clipping may partly compensate for trust-region violations caused by stale optimizer statistics or momentum. 51867_Design_Principles_for_Ro
Based on this perspective, the paper proposes three design principles:
- maintain high behavioral diversity early in training;
- calibrate the magnitude of policy improvement steps;
- use nonstationarity-aware optimizers.
Practical implementations include entropy scheduling, PPO clipping or SAC target-update scheduling, optimizer-state resets, and learning-rate warmup.
The paper validates these ideas across Atari and Brax experiments with PPO and SAC, showing reduced variance, fewer catastrophic failures, better architecture scaling, and improved hyperparameter robustness.
Critical strengths
-
The conceptual contribution is strong and potentially long-lasting.
The notion of hyperparameter collusion is memorable and useful. It captures a phenomenon that many RL practitioners encounter in practice: a seemingly indispensable hyperparameter may only be compensating for a defect elsewhere in the system. This framing could become a useful vocabulary for analyzing RL systems beyond this specific paper.
-
The paper goes beyond a slogan and provides a structured formalization.
The decomposition into gives the “side effects” and “race dynamics” narrative a concrete operational meaning rather than leaving it as an analogy.
-
The PPO trust-region analysis is particularly compelling.
The paper shows that nominal PPO clipping bounds can substantially underestimate realized policy movement. It further identifies stale optimizer statistics and momentum as important causes of overshooting, and demonstrates that optimizer reset plus learning-rate warmup can improve adherence to the intended policy-update constraint.
In my view, this is one of the strongest and most self-contained scientific contributions of the paper.
-
The Atari batch-size analysis is diagnostically interesting.
Instead of simply repeating the empirical observation that “small batches work better,” the authors distinguish the direct optimization effect from the indirect exploration effect. Their observation that larger batches can be beneficial for the actual training objective provides concrete support for the broader claim that standard RL settings may rely on cross-component compensation.
-
The experiments focus on robustness rather than only peak performance.
The paper studies seed variability, catastrophic failures, architecture scaling, and hyperparameter perturbations. These evaluations are well aligned with the paper’s central claims.
Critical weaknesses
-
The core formalism is still primarily explanatory rather than predictive.
Concepts such as side effects, component integrity, and hyperparameter collusion are intuitive, but they currently lack quantitative definitions. As a result, the framework is powerful as a post-hoc explanation, yet difficult to falsify experimentally.
-
The full recipe changes many factors simultaneously.
The final recipe modifies entropy scheduling, clipping or target-update schedules, learning rate, Adam state, warmup, data volume, optimization steps, value-loss weighting, and network architecture. 51867_Design_Principles_for_Ro
Although the appendix contains step-wise ablations, it is still difficult to attribute the final robustness gains specifically to the proposed principles rather than to the increased training budget or stronger architecture.
-
The “algorithm-agnostic” claim may be somewhat too broad.
The empirical evidence mainly covers value-based Atari, PPO, and SAC. It remains unclear whether the same design principles transfer cleanly to offline RL, model-based RL, multi-agent RL, or RLHF/RLVR.
-
Robustness and sample efficiency are not fully disentangled.
The authors explicitly acknowledge that greater stability often requires more data or more conservative optimization.
A stricter equal-sample or equal-FLOP comparison would make the practical trade-off clearer.
-
Some mechanistic explanations remain non-exclusive.
For instance, entropy collapse, optimizer noise, critic error, representation conditioning, and local optima may interact. The proposed explanations are plausible, but some alternative causal accounts are not ruled out.
Questions for authors
- Can the authors define a quantitative measure of hyperparameter collusion, e.g. via cross-component intervention sensitivity or second-order effects?
- Can the framework produce a falsifiable prediction—for example, a known hyperparameter sensitivity that should disappear once two components are functionally isolated?
- How much of the full-recipe gain comes from increased data and additional gradient steps? Could the authors provide strictly matched comparisons under equal environment steps and equal FLOPs?
- Does optimizer-state reset remain effective with RMSProp, AdamW, or SGD with momentum?
- Can at least one of the proposed principles be validated in a substantially different setting such as LLM RL, offline RL, or model-based RL?
2 comments
Part 2
3. Disentangling Sample/FLOP Budget from Algorithmic Stability (Q3)
"How much of the full-recipe gain comes from increased data and additional gradient steps? Could the authors provide strictly matched comparisons under equal environment steps and equal FLOPs?"
We agree that budget-matched evaluations provide essential insight into the recipe, and we have added standalone (single-factor), leave-one-out, and strictly budget-matched ablations in Appendix C.6 (Figures 23 and 24). Under strictly equal environment steps and equal FLOPs (1x data, 1x grad updates, condition C4), applying our operator-isolation principles (entropy schedule + trust-region/target schedule + Adam reset/warmup) substantially increases aggregate Brax IQM return over the 1x baseline (Figure 23a) and reduces the catastrophic seed failure rate to (Figure 23b), while Figure 24 shows the corresponding learning curves aligned against both environment steps and gradient updates.
We also highlight two important subtleties:
- Operator isolation is a prerequisite for scaling compute/data: Adding 2x gradient updates in isolation (
C3in Figure 23) increases catastrophic failures and degrades return on several tasks because extra optimization epochs exacerbate trust-region overshoot when is unisolated. Our recipe is what allows the 2 data and update budget (C5) to yield clean, monotonic improvements. - Role of network width: Network width and depth in Section 5.2 (Figure 7 and Appendix C.3, Figure 20) were varied as a robustness stress-test (showing that baseline configurations collapse as depth/width scale, whereas the recipe scales stably), rather than used as a hidden crutch for the small-MLP recipe.
4. Generalization Across Optimizers: RMSProp, AdamW, SGD+M, and Muon (Q4)
"Does optimizer-state reset remain effective with RMSProp, AdamW, or SGD with momentum?"
We have added these experiments in Appendix E.1 (Figures 26 and 27), evaluating per-iteration state resets and LR warmup across Adam, AdamW, RMSProp (momentum ), SGD+Momentum (), and Muon (Newton–Schulz orthogonalized updates) on four Brax environments (Humanoid, Ant, Walker2d, and HalfCheetah).
Across all standard optimizers (Adam, AdamW, SGD+M), resetting the optimizer state and warming up the learning rate sharply reduces maximum ratio overshoot and approximate KL divergence while improving IQM return. Comparing SGD+M and RMSProp cleanly isolates the two mechanisms: SGD+M demonstrates that carrying over first-order momentum across iteration boundaries directly drives trust-region overshoot, while RMSProp shows that resetting/warming up the second-order preconditioner statistics prevents step-size spikes at the start of each optimization round.
5. Scope of "Algorithm-Agnostic" & Validation in New Paradigms (Q5)
"Can at least one of the proposed principles be validated in a substantially different setting such as LLM RL, offline RL, or model-based RL?"
- Scoping & Cross-Paradigm Connections: We have revised the Abstract, Section 3.1, and Section 6 to clarify that our concurrency framework models online RL (where Texp, Tpol, and Topt interact in a closed loop), and noted in Section 6 how the framework specializes when moving to regimes with fewer or additional moving parts.
- New Empirical Validations (Offline-to-Online RL & Sequence-Level PPO/GRPO):
- In Appendix B.1 (Figures 10 and 11c), we evaluate offline-to-online RL on Brax, showing that an unregularized policy improvement operator diverges unless paired with an artificially tiny learning rate, whereas properly regularizing Tpol restores a positive learning-rate gradient.
- In Appendix E.2 (Figures 28–30), we evaluate multi-epoch PPO and online GRPO on a causal Transformer architecture across autoregressive sequence reversal (both sparse exact-match and competing copy-shortcut rewards) and modular addition (). We find the exact same pathologies and remedies: standard Adam suffers from compounding sequence-level ratio overshoot and premature token-entropy collapse (deg<hit), whereas applying our operator-isolation principles (entropy/clipping decay and optimizer moment resets/damping) eliminates ratio spikes and enables reliable convergence to the global optimum across seeds.
Note on Ethics Flag
We noticed "Flag For Ethics Review: Yes" was selected on the review form despite no ethical concerns being mentioned in the review text. Could you confirm if this was indeed the case?
Part 1
We thank Reviewer Z6k3 for their thoughtful review and for highlighting the potential impact of the paper’s contribution—in particular our identification of hyperparameter collusion and its utility in diagnosing DQN batch-size and PPO trust-region sensitivity. We also appreciate your recognition of our focus on algorithmic robustness rather than cherry-picked peak performance.
We address your questions and concerns below (note, due to length limitations on openreview we have had to split our response over two comments).
1. Quantitative Definition of Hyperparameter Collusion (Q1)
It is completely reasonable to ask how hyperparameter collusion can be quantified in practice. Because hyperparameter collusion is an umbrella term for cross-process interference across diverse algorithm components, a single universal scalar cannot capture every physical realization (for instance, collusion between batch size and exploration stochasticity manifests in different state statistics than collusion between PPO clipping and optimizer momentum). However, we have revised Section 3.2.1 to provide both a general geometric characterization and a concrete operational procedure:
2. Falsifiable Predictions Tested by the Framework (Q2)
Two such predictions were implicitly tested in Section 4; based on your feedback, we have restructured Sections 3.2.1 and 4 to make these falsifiable predictions explicit, and added dedicated 2D grid evaluations in Appendices B.1, D, and E.1 to test them thoroughly: