acceptodds
Under review as a conference paper at ICLR 2027

Design Principles for Robust Reinforcement Learning Systems

Abstract

Deep reinforcement learning (RL) algorithms must concurrently coordinate exploration, policy improvement, and neural network optimization. While prior work has seen great success in optimizing these components in isolation, this paper argues that the asynchronous interactions between them are the primary driver of sensitivity and instability in modern deep RL systems. We demonstrate that well-known vulnerabilities, such as batch size limits in DQN, clipping bounds in PPO, and entropy tuning in continuous control, are often symptoms of hyperparameter collusion, where a sub-optimal choice in one component is required to mask inadequacies in another. Drawing an analogy to concurrent software systems, we propose a set of algorithm-agnostic design patterns aimed at ensuring the correctness of each component. We empirically show that these principles significantly improve the robustness and peak performance of widely-used deep RL agents, including PPO and SAC. We hope this work facilitates greater scientific rigor by simplifying standard baselines and streamlining the evaluation of new algorithms

Primary area:
reinforcement learning
Keywords:
reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Z6k3OpenReview ↗
Rating
3
Confidence
1(−1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a unified view of instability in deep reinforcement learning. Rather than treating exploration, policy improvement, and neural optimization as a simple sequential pipeline, the authors model them as multiple concurrent processes operating on a shared learner state.

The learner state is written as

$ S_t = (\theta_t, D_t, L_t), $

and the training process is decomposed into three operators:

$ T^{\mathrm{exp}}, \qquad T^{\mathrm{pol}}, \qquad T^{\mathrm{opt}}, $

corresponding to exploration/data collection, policy improvement, and neural optimization.

The central concept introduced by the paper is hyperparameter collusion: a suboptimal design choice in one component can be accidentally compensated for by side effects from another component, leading to fragile and highly coupled hyperparameter dependencies. For example, small batch sizes may appear necessary not because they optimize the objective better, but because optimizer noise unintentionally contributes to exploration; similarly, overly conservative PPO clipping may partly compensate for trust-region violations caused by stale optimizer statistics or momentum. 51867_Design_Principles_for_Ro

Based on this perspective, the paper proposes three design principles:

  1. maintain high behavioral diversity early in training;
  2. calibrate the magnitude of policy improvement steps;
  3. use nonstationarity-aware optimizers.

Practical implementations include entropy scheduling, PPO clipping or SAC target-update scheduling, optimizer-state resets, and learning-rate warmup.
The paper validates these ideas across Atari and Brax experiments with PPO and SAC, showing reduced variance, fewer catastrophic failures, better architecture scaling, and improved hyperparameter robustness.

Critical strengths
  1. The conceptual contribution is strong and potentially long-lasting.

    The notion of hyperparameter collusion is memorable and useful. It captures a phenomenon that many RL practitioners encounter in practice: a seemingly indispensable hyperparameter may only be compensating for a defect elsewhere in the system. This framing could become a useful vocabulary for analyzing RL systems beyond this specific paper.

  2. The paper goes beyond a slogan and provides a structured formalization.

    The decomposition into �0��0� gives the “side effects” and “race dynamics” narrative a concrete operational meaning rather than leaving it as an analogy.

  3. The PPO trust-region analysis is particularly compelling.

    The paper shows that nominal PPO clipping bounds can substantially underestimate realized policy movement. It further identifies stale optimizer statistics and momentum as important causes of overshooting, and demonstrates that optimizer reset plus learning-rate warmup can improve adherence to the intended policy-update constraint.

    In my view, this is one of the strongest and most self-contained scientific contributions of the paper.

  4. The Atari batch-size analysis is diagnostically interesting.

    Instead of simply repeating the empirical observation that “small batches work better,” the authors distinguish the direct optimization effect from the indirect exploration effect. Their observation that larger batches can be beneficial for the actual training objective provides concrete support for the broader claim that standard RL settings may rely on cross-component compensation.

  5. The experiments focus on robustness rather than only peak performance.

    The paper studies seed variability, catastrophic failures, architecture scaling, and hyperparameter perturbations. These evaluations are well aligned with the paper’s central claims.

Critical weaknesses
  1. The core formalism is still primarily explanatory rather than predictive.

    Concepts such as side effects, component integrity, and hyperparameter collusion are intuitive, but they currently lack quantitative definitions. As a result, the framework is powerful as a post-hoc explanation, yet difficult to falsify experimentally.

  2. The full recipe changes many factors simultaneously.

    The final recipe modifies entropy scheduling, clipping or target-update schedules, learning rate, Adam state, warmup, data volume, optimization steps, value-loss weighting, and network architecture. 51867_Design_Principles_for_Ro

    Although the appendix contains step-wise ablations, it is still difficult to attribute the final robustness gains specifically to the proposed principles rather than to the increased training budget or stronger architecture.

  3. The “algorithm-agnostic” claim may be somewhat too broad.

    The empirical evidence mainly covers value-based Atari, PPO, and SAC. It remains unclear whether the same design principles transfer cleanly to offline RL, model-based RL, multi-agent RL, or RLHF/RLVR.

  4. Robustness and sample efficiency are not fully disentangled.

    The authors explicitly acknowledge that greater stability often requires more data or more conservative optimization.

    A stricter equal-sample or equal-FLOP comparison would make the practical trade-off clearer.

  5. Some mechanistic explanations remain non-exclusive.

    For instance, entropy collapse, optimizer noise, critic error, representation conditioning, and local optima may interact. The proposed explanations are plausible, but some alternative causal accounts are not ruled out.

Questions for authors
  1. Can the authors define a quantitative measure of hyperparameter collusion, e.g. via cross-component intervention sensitivity or second-order effects?
  2. Can the framework produce a falsifiable prediction—for example, a known hyperparameter sensitivity that should disappear once two components are functionally isolated?
  3. How much of the full-recipe gain comes from increased data and additional gradient steps? Could the authors provide strictly matched comparisons under equal environment steps and equal FLOPs?
  4. Does optimizer-state reset remain effective with RMSProp, AdamW, or SGD with momentum?
  5. Can at least one of the proposed principles be validated in a substantially different setting such as LLM RL, offline RL, or model-based RL?
2 comments

We thank Reviewer Z6k3 for their thoughtful review and for highlighting the potential impact of the paper’s contribution—in particular our identification of hyperparameter collusion and its utility in diagnosing DQN batch-size and PPO trust-region sensitivity. We also appreciate your recognition of our focus on algorithmic robustness rather than cherry-picked peak performance.

We address your questions and concerns below (note, due to length limitations on openreview we have had to split our response over two comments).

1. Quantitative Definition of Hyperparameter Collusion (Q1)

"Can the authors define a quantitative measure of hyperparameter collusion, e.g. via cross-component intervention sensitivity or second-order effects?"

It is completely reasonable to ask how hyperparameter collusion can be quantified in practice. Because hyperparameter collusion is an umbrella term for cross-process interference across diverse algorithm components, a single universal scalar cannot capture every physical realization (for instance, collusion between batch size and exploration stochasticity manifests in different state statistics than collusion between PPO clipping and optimizer momentum). However, we have revised Section 3.2.1 to provide both a general geometric characterization and a concrete operational procedure:

  • 2D Hyperparameter Landscape Signature: At the level of the performance landscape, collusion is characterized by coupled optima where the improvement direction of one hyperparameter flips in sign depending on the value of another (Figure 2 and Appendix B.1, Figures 8–11).
  • Three-Step Diagnostic Template: In Section 3.2.1, we lay out a general three-step procedure for quantifying specific instances of collusion: (1) measure the intermediate quantity in the coupled process TjT^{\mathrm{j}} perturbed by hyperparameter hih_i (such as empirical policy entropy in TexpT^{\mathrm{exp}} or realized step size / trust-region violation rate in TpolT^{\mathrm{pol}}); (2) verify that varying hih_i does not degrade TiT^{\mathrm{i}}'s own local objective; and (3) intervene on TjT^{\mathrm{j}} to control that intermediate quantity directly. We have revised Section 4 to explicitly tie each empirical case study back to this template so practitioners can apply the same diagnostic workflow to their own systems.

2. Falsifiable Predictions Tested by the Framework (Q2)

"Can the framework produce a falsifiable prediction—for example, a known hyperparameter sensitivity that should disappear once two components are functionally isolated?"

Two such predictions were implicitly tested in Section 4; based on your feedback, we have restructured Sections 3.2.1 and 4 to make these falsifiable predictions explicit, and added dedicated 2D grid evaluations in Appendices B.1, D, and E.1 to test them thoroughly:

  1. Prediction 1 (PPO ϵ×η\epsilon \times \eta Isolation): If conservative PPO clipping thresholds and small learning rates are compensating for stale Adam momentum drift ($T^{\mathrm{opt}} \rightarrow T^{\mathrm{pol}} $ side effect), then: (a) performance across disparate (ϵ\epsilon, epochs, η\eta) settings should collapse onto a single curve when plotted against realized policy update magnitude (Figure 4d and Appendix B.2, Figure 13); and (b) isolating ToptT^{\mathrm{opt}} via per-iteration Adam resets and LR warmup should eliminate trust-region overshoot and flatten performance sensitivity across the 2D (ϵη\epsilon \eta)grid—which is directly confirmed in Appendix B.1 (Figure 9) and Appendix E.1 (Figure 27).
  2. Prediction 2 (Atari Batch-Size Isolation): If large batch sizes (B>64B>64) fail in DQN/C51/Rainbow because lower gradient noise starves exploration and alters effective step sizes ($T^{\mathrm{opt}} \rightarrow T^{\mathrm{exp}} $ side effect) rather than hurting optimization, then: (a) larger batch sizes should perform monotonically better on the true discounted, clipped training objective (Figure 3d, falsifying the hypothesis that large batches hurt Q-function optimization); and (b) directly controlling for update magnitude and exploration via co-scaled optimizer hyperparameters (η∝1/B\eta \propto 1/\sqrt{B}, scaled Adam ϵ\epsilon, and LR schedules) should close the large-batch performance gap on unclipped game scores—which we confirm in Appendix D (Figure 25).

3. Disentangling Sample/FLOP Budget from Algorithmic Stability (Q3)

"How much of the full-recipe gain comes from increased data and additional gradient steps? Could the authors provide strictly matched comparisons under equal environment steps and equal FLOPs?"

We agree that budget-matched evaluations provide essential insight into the recipe, and we have added standalone (single-factor), leave-one-out, and strictly budget-matched ablations in Appendix C.6 (Figures 23 and 24). Under strictly equal environment steps and equal FLOPs (1x data, 1x grad updates, condition C4), applying our operator-isolation principles (entropy schedule + trust-region/target schedule + Adam reset/warmup) substantially increases aggregate Brax IQM return over the 1x baseline (Figure 23a) and reduces the catastrophic seed failure rate to <2%< 2\% (Figure 23b), while Figure 24 shows the corresponding learning curves aligned against both environment steps and gradient updates.

We also highlight two important subtleties:

  1. Operator isolation is a prerequisite for scaling compute/data: Adding 2x gradient updates in isolation (C3 in Figure 23) increases catastrophic failures and degrades return on several tasks because extra optimization epochs exacerbate trust-region overshoot when ToptT^{\mathrm{opt}} is unisolated. Our recipe is what allows the 2 data and update budget (C5) to yield clean, monotonic improvements.
  2. Role of network width: Network width and depth in Section 5.2 (Figure 7 and Appendix C.3, Figure 20) were varied as a robustness stress-test (showing that baseline configurations collapse as depth/width scale, whereas the recipe scales stably), rather than used as a hidden crutch for the small-MLP recipe.

4. Generalization Across Optimizers: RMSProp, AdamW, SGD+M, and Muon (Q4)

"Does optimizer-state reset remain effective with RMSProp, AdamW, or SGD with momentum?"

We have added these experiments in Appendix E.1 (Figures 26 and 27), evaluating per-iteration state resets and LR warmup across Adam, AdamW, RMSProp (momentum β1=0\beta_1=0), SGD+Momentum (β=0.9\beta=0.9), and Muon (Newton–Schulz orthogonalized updates) on four Brax environments (Humanoid, Ant, Walker2d, and HalfCheetah).

Across all standard optimizers (Adam, AdamW, SGD+M), resetting the optimizer state and warming up the learning rate sharply reduces maximum ratio overshoot and approximate KL divergence while improving IQM return. Comparing SGD+M and RMSProp cleanly isolates the two mechanisms: SGD+M demonstrates that carrying over first-order momentum μt\mu_t across iteration boundaries directly drives trust-region overshoot, while RMSProp shows that resetting/warming up the second-order preconditioner statistics νt\nu_t prevents step-size spikes at the start of each optimization round.


5. Scope of "Algorithm-Agnostic" & Validation in New Paradigms (Q5)

"Can at least one of the proposed principles be validated in a substantially different setting such as LLM RL, offline RL, or model-based RL?"

  • Scoping & Cross-Paradigm Connections: We have revised the Abstract, Section 3.1, and Section 6 to clarify that our concurrency framework models online RL (where Texp, Tpol, and Topt interact in a closed loop), and noted in Section 6 how the framework specializes when moving to regimes with fewer or additional moving parts.
  • New Empirical Validations (Offline-to-Online RL & Sequence-Level PPO/GRPO):
    1. In Appendix B.1 (Figures 10 and 11c), we evaluate offline-to-online RL on Brax, showing that an unregularized policy improvement operator diverges unless paired with an artificially tiny learning rate, whereas properly regularizing Tpol restores a positive learning-rate gradient.
    2. In Appendix E.2 (Figures 28–30), we evaluate multi-epoch PPO and online GRPO on a causal Transformer architecture across autoregressive sequence reversal (both sparse exact-match and competing copy-shortcut rewards) and modular addition ( mod  4\bmod\, 4). We find the exact same pathologies and remedies: standard Adam suffers from compounding sequence-level ratio overshoot and premature token-entropy collapse (deg<hit), whereas applying our operator-isolation principles (entropy/clipping decay and optimizer moment resets/damping) eliminates ratio spikes and enables reliable convergence to the global optimum across seeds.

Note on Ethics Flag

We noticed "Flag For Ethics Review: Yes" was selected on the review form despite no ethical concerns being mentioned in the review text. Could you confirm if this was indeed the case?

Reviewer_xYX1OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

During the years, the learning dynamics of the most famous deep reinforcement learning algorithm (PPO, SAC, ..) are mostly completely analized. It's common for such kind of algorithm to use common range values from their respective main hyperparameters. The paper formalizes and validates a novelty formalization of such dinamics, identifying three different operators, that drive the learning impacting on the final performance at different seeds. Evaluating respectively how the \emph{exploration}, \emph{policy optimization} and \emph{optimizer dynamics} impact on deep rl algorithm learning dealing with continuous control environments, the author model everything as a concurrent system, where hyperparameter collusion could happen due to the influence that each parameter has on the others.

Critical strengths

I found the paper very clear and easy to follow. While the argument of the paper are of common domain, i found that the formalization of the problem as a concurrent system is novel and understandable. Most of all I consider valuable tackling the problems independently while considering a possible race conditions among the hyperparameters. The experimental section on mainly continuous control task is reasonable for evaluating the improvement in perfomance of the recipe formulated, as well as running PPO and SAC for the demonstrative purposes. The design recipe formalized is practical and easy to follow.

Critical weaknesses

While i found the paper very easy and clear to follow, i had some problem looking at the figure, due to their misleading labels and their dimensions. Spcifically Fig.2 where the label stating the color code for batch sizes, but not all of them are then present in all the subfigures, without any consideration. Even if the recipe for hyperparameter tuning is easy to follow, maybe more complex strategy could be involved to increase the novelty of the paper itself, not relying on common strategy like simple entropy bonus or update strategies for target networks. I would expect a minimal formalization on how specific recipe to handle race conditions impact on the τhit,andτext\tau_{hit}, and \tau_{ext}

Questions for authors
  • Can you clarify how the batch size are considered in Fig.2 ?
  • How do your findings on resetting Adam moments in PPO relate to prior work on optimizer resets in value-based RL?
2 comments

We thank Reviewer xYX1 for their positive assessment of the paper's clarity, the novelty of our concurrent-system formalization, and the practical utility of our stabilization recipe. Below we address your specific questions and suggestions.

1. Clarifying Batch Sizes in Figure 3 (formerly Figure 2, Q1)

"Can you clarify how the batch size are considered in Fig.2?"

In the original submission, the shared legend listed all seven batch sizes (B∈{8,16,32,64,128,256,512{)B \in \{8, 16, 32, 64, 128, 256, 512\{), while subplots (b) and (c) displayed only a subset of curves (and subplots (a) and (d) plotted batch size directly along the x-axis). In the revised manuscript (now Figure 3), we have regenerated the figure with larger panel dimensions and included all seven batch sizes from the legend in subplots (b) and (c) (noting that B=8 in Rainbow was run for 50 iterations due to wall-clock cost), while subplots (a) and (d) report final aggregate performance as a function of batch size along the horizontal axis.


2. Formalizing Race Dynamics & Why Simple Mechanisms Suffice

"Even if the recipe for hyperparameter tuning is easy to follow, maybe more complex strategy could be involved to increase the novelty of the paper itself... I would expect a minimal formalization on how specific recipe to handle race conditions impact on the [learning dynamics]."

This comment raises two distinct points: the complexity of the recipe, and the formal connection between our theoretical discussion of race conditions and concrete hyperparameter changes.

On the first point—that we could introduce more complex strategies to increase novelty—we respectfully push back. A central goal of our work is to improve the clarity, simplicity, and reproducibility of baseline RL algorithms; introducing complex new machinery simply for the sake of novelty would run counter to this objective, when standard mechanisms already succeed once their cross-process interactions are properly coordinated.

We completely agree with your second point, and we have expanded Section 3.2.2 (Equation 1) with a formal worked example showing how operator hyperparameters directly bound the ordering of the progress stopping time τhit\tau_{\mathrm{hit}} and the degeneracy/entropy-extinction stopping time τdeg\tau_{\mathrm{deg}}. This analysis directly grounds our empirical results in Section 4.3, which applies this rate-balancing strategy to environments where exploration and exploitation are in tension (as well as our 2D hyperparameter landscape visualizations in Figure 2 and Appendix B.1). We believe this tighter connection between Sections 3 and 4 has significantly improved the presentation, and we thank you for the suggestion.


3. Relation to Prior Work on Optimizer Resets in Value-Based RL (Q2)

"How do your findings on resetting Adam moments in PPO relate to prior work on optimizer resets in value-based RL?"

You are right to point out the rich literature on resets in deep RL. The high-level motivation for resetting optimizer states in PPO is the same as in value-based RL: when the local optimization problem shifts, stale optimizer statistics no longer provide a well-calibrated inductive bias for the new objective.

The nature of the nonstationarity, however, differs slightly between the two settings:

  • In value-based RL (DQN), the agent samples transitions continuously while the bootstrap regression target updates on a fixed schedule (Lt→Lt+1L_t \rightarrow L_{t+1}).
  • In PPO, the agent runs multiple optimization epochs on a fixed rollout batch, allowing the optimizer to converge into the clipped plateau where gradients vanish ∣ρ(θ)−1∣≥ε|\rho(\theta) - 1| \ge \varepsilon, deflating Adam's second-order moment νt\nu_t) while carrying over first-order momentum μt\mu_t. This makes PPO uniquely vulnerable to immediate trust-region overshooting at the start of the next rollout iteration (when importance ratios reset to ρ(θ)=1\rho(\theta) = 1), motivating our combination of state resets with an intra-iteration learning-rate warmup period.

We have expanded Section 4.2 to explicitly contrast these two mechanisms and cite prior work on optimizer resets in value-based RL (Asadi et al., 2023; Lyle et al., 2023), and we further isolate the effects of first-order momentum vs. second-order preconditioning across five optimizers (Adam, AdamW, RMSProp, SGD+M, and Muon) in Appendix E.1 (Figures 26–27).

Reviewer_xYX1OpenReview ↗

Final Score

Thank you for the thorough rebuttal and updated manuscript; i appreciate the update in Section~3.2.2, and the expanded literature positioning on optimizer resets directly address my initial concerns, so I am happy to maintain my positive assessment and keep my score.

Reviewer_G4h4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes guidelines on stabilizing RL training. The authors analyze the behavior of RL pipelines through decomposing the state update operator into three components: the exploration operator which updates the data buffer, the policy improvement operator which specifies the local optimization objective, and the network parameter optimizer update. They show that design choices that control one of the components can have side-effects on other components, or the ordering of exploration and exploitation can give rise to race dynamics and thus cause instabilities in RL training. The authors propose a recipe using explicit exploration schedules favoring high exploration early in training, smaller policy improvement steps in later stages of training, and optimizer state resets and learning rate warmup to ensure that each component is well-specified and robust. Through empirical results using PPO and SAC on Brax the authors show that their recipe improves evaluation performance and robustness.

Critical strengths
  • Recipe improves performance of SAC, and PPO compared to baseline configuration.
  • The proposed recipe makes sense and the results are robust across PPO and SAC on Brax environments.
  • The authors empirically justify the components of their recipe through cumulative ablations.
Critical weaknesses
  • The plots are hard to read. Increasing their size would help improve readability.
  • A Figure below Figure 9 is missing caption.
  • The Figures above Figure 10 are missing caption.
  • One of the Figures above Figure 10 replicates Figure 5
  • In the Appendix, the recipe is sometimes referred to as "Golden". It would be good to be consistent with nomenclature.
  • The abstract says the paper demonstrates DQN batch-size limits are often symptoms of collusion, but the fix of the robustness to batch size for DQN through applying the proposed design principles is not shown.
  • Figure 4 title says "Inverted double pendulum", but the caption mentions humanoid environment. Further, the left subplot runs for half the timesteps budget compared to the other two subplots.
  • Missing ablation of the 2xdata and Adam state reset components of the recipe by showing their performance on their own each to assess their contribution to performance.
Questions for authors
  • In the ablations of the components, what is the contribution of more data, more gradient updates per round and resetting Adam state evaluated on their own?
  • How does applying your proposed design principles to C51 and Rainbow change their performance and sensitivity to larger batches on Atari?
2 comments

We thank Reviewer G4h4 for their detailed review. We appreciated the assessment that our proposed recipe makes intuitive sense, is empirically justified through ablations, and consistently improves evaluation performance and robustness across PPO and SAC on Brax. We have addressed each concern below.

1. Standalone & Leave-One-Out Ablations (2x Data, More Grad Updates, Adam State Reset)

"In the ablations of the components, what is the contribution of more data, more gradient updates per round and resetting Adam state evaluated on their own?"

We have added add-one-in as well as leave-one-out ablations to address this comment in Appendix C.6, Figure 23, and added matched 1x-budget learning curves in Figure 24. In summary:

  • Increasing the optimizer or data budget alone does not translate to significant improvements—in fact, doubling the number of gradient updates per round on its own (C3) increases the catastrophic failure rate and induces instability on several tasks.
  • Each algorithmic component of the recipe has a net positive effect on performance, with the distribution of gains across environments reflecting the nature of the intervention (e.g., hard-exploration tasks benefit more from the exploration schedule than from optimizer resets).
  • The recipe substantially improves IQM return and failure rate even in strictly matched 1x iso-budget evaluations (C4). Importantly, isolating the operators is what allows the 2x data and optimizer budget (C5) to scale cleanly rather than hurting performance.

2. Applying the Design Principles to C51 & Rainbow on Atari (Q2 & Abstract Alignment)

"How does applying your proposed design principles to C51 and Rainbow change their performance and sensitivity to larger batches on Atari?"

We preface our response by noting that value-based Atari agents present two practical evaluation challenges: first, running the full 200M-frame budget across the benchmark takes roughly 5 GPU-days per seed; second, as shown in Figure 3d (formerly Figure 2d), Atari results are confounded by the mismatch between the discounted, clipped objective being optimized and the undiscounted, unclipped game score used for evaluation. We have also updated the Abstract to clearly reflect that our Atari experiments serve as a diagnostic analysis of this collusion. In Appendix D (Figure 25), we include evaluations on Asterix examining how C51 and Rainbow respond when we apply our operator-isolation principles across batch sizes B∈{8,…,1024}B \in \{8, \dots, 1024\}:

  • Controlling optimizer update magnitude: Correctly co-scaling the optimizer hyperparameters with batch size (B/32\sqrt{B/32}, scaled Adam , and cosine/linear learning-rate schedules) to keep the effective update norm calibrated already closes much of the performance gap at large batch sizes (B=256--512).
  • Controlling exploration collapse: Because Rainbow relies on NoisyNets rather than -greedy exploration, its learned noise scale collapses under the lower gradient variance of large batches (Figure 3c); explicitly maintaining exploration alongside step-size calibration mitigates this large-batch degradation.

3. Figure Readability, Captions, & Nomenclature Fixes

We thank the reviewer for catching these formatting issues, all of which have been corrected in the revised PDF:

  • Figure 5 (formerly Figure 4) Title, Caption, & Horizon: Corrected the environment label mismatch between the title and caption, and aligned all three subplots to a uniform training-budget horizon (the left panel previously displayed the unscaled 1x step count).
  • Appendix Captions & Duplicate Figure: Added full descriptive captions and figure numbers to the previously uncaptioned plots (Figures 16 and 17, formerly adjacent to Figures 9 and 10), and removed the duplicate bar chart of Figure 6 (formerly Figure 5).
  • Plot Sizing & Nomenclature: Enlarged multi-panel figures across the main text and Appendix (including splitting the diagnostic grid into Figures 14 and 15 for legibility) and replaced all legacy "Golden" labels in the Appendix with uniform terminology.

Thank you for the detailed response and for the substantial revisions to the manuscript. The new ablations in Appendix C.6 answer my main question about the contribution of each component of the recipe, and I appreciate the added Atari experiments in Appendix D and the presentation fixes. I am raising my score to 3 and recommend acceptance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment to all reviewers

We thank all three reviewers for their constructive feedback and for recognizing the value of modeling deep RL training as a concurrent system (Texp,Topt,Tpol)(\mathcal{T}^{\mathrm{exp}},\mathcal{T}^{\mathrm{opt}}, \mathcal{T}^{\mathrm{pol}}) governed by side effects, race dynamics, and hyperparameter collusion. We have taken the feedback into account and conducted extensive ablations and follow-up experiments as requested by the reviewers. We have also improved the clarity of the text, in particular explaining how the framework can be leveraged to form falsifiable hypotheses and improving the rigor of the discussion of race conditions in Section 3.2.2. We include a more detailed list of changes below:

Standalone, leave-one-out, and fixed budget brax ablations (G4h4, Z6k3): We extended the ablations from Figure 6 (formerly Figure 5) into Appendix C.6 (Figures 23–24), which now includes both add-one-in and leave-one-out evaluations. When comparing strictly matched sample and FLOP budgets (1x data, 1x grad), following our operator-isolation principles substantially improves IQM return and reduces catastrophic failure rates. Notably, brax’s default hyperparameter setting often underperforms its default budget when gradient updates are increased in isolation, whereas our recipe more consistently benefits from increased budget in any form.

Hyperparameter collusion definition and falsifiable hypotheses (Z6k3, xYX1): We revised Section 3.2.1 to give a more precise characterization of hyperparameter collusion in the performance landscape, and added visualizations of its 2D grid signature in Figure 2 and Appendix B.1 (Figures 8–11). We also laid out an explicit three-step template for generating falsifiable hypotheses about the causes of hyperparameter sensitivity, which we apply throughout Sections 4 and 5, and added a concrete stopping-time bound illustrating how operator pacing governs race conditions in Section 3.2.2.

Optimizer generalization and new RL paradigms (Z6k3, xYX1): In Appendix E.1 (Figures 26–27), we evaluate per-iteration optimizer state resets and warmup across Adam, AdamW, RMSProp, SGD with momentum, and Muon, isolating the roles of first-order momentum drift vs. second-order preconditioner staleness and visualizing how resets decouple the 2D (,) landscape. In addition, we validate our framework in offline-to-online RL (Appendix B.1, Figures 10–11c) and autoregressive sequence-level PPO and GRPO (Appendix E.2, Figures 28–30).

Atari batch-size stabilization (G4h4, Z6k3): In Appendix D (Figure 25), we include full 200M-frame evaluations examining the role of co-scaling learning rate, Adam , and learning-rate schedules with batch size to control update magnitude, showing that isolating the optimizer step size already closes much of the performance gap across batch sizes in C51 and Rainbow.

Figure and text polishing (G4h4, xYX1): We enlarged figures across the main paper and Appendix, plotted all seven batch sizes in Figure 3 (formerly Figure 2), fixed the Figure 5 (formerly Figure 4) title/caption/horizon discrepancy, added captions to Figures 16–17 (formerly uncaptioned near Figures 9–10), removed the duplicate bar plot, and unified the recipe terminology throughout the Appendix.


We once again thank the reviewers for their careful analysis of the paper and helpful recommendations. We believe that incorporating this feedback has significantly strengthened the manuscript, and we are happy to respond to any additional questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.