acceptodds
Under review as a conference paper at ICLR 2027

An Empirical Study of Reinforcement Learning Scaling Laws

Abstract

The discovery of scaling laws enables the prediction of _the performance_, _the optimal resource allocation strategy_, and _the best hyperparameter configuration_ of a large-scale training job based on inexpensive experiments at smaller scales. Guided by these empirical scaling laws, the deep learning community has achieved great success in training large (self-)supervised models on Internet-scale datasets. In contrast, despite its potential and wide application, reinforcement learning (RL) has yet to find a guiding principle for predictable scaling. In this work, we conduct an empirical study of RL scaling laws of an actor-critic algorithm in two procedurally generated environments, Starpilot and Craftax. On the one hand, we discovered empirical scaling laws with strong predictive power in both environments. The discovered scaling laws accurately predict the aforementioned three quantities for larger-scale training jobs up to two orders of magnitude in total training compute. In particular, we employ a variant of Elo scores for the single-agent RL paradigm as an alternative performance metric than the commonly used mean episode return. Our results show that the Elo score is a much more predictable metric than the mean episode return, making it a better choice for fitting scaling laws. By applying our discovered scaling laws to increase training compute, we established a new state of the art in the challenging Craftax environment. On the other hand, beneath the surface of the global scaling trend, we observed a non-monotonic scaling pattern of the compute-optimal model size in Craftax. We connect this observation to a pathology known as Ray interference.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, scaling laws

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HWSgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

As training jobs grow, scaling laws have become invaluable for predicting performance, resource allocation, and optimal hyperparameter configuration with only inexpensive experiments. While the deep learning community has successfully leveraged scaling laws, the corresponding reinforcement learning (RL) community has lagged. This paper aims to discover fundamental scaling laws for RL training. To operationalize this, the authors studied the A2C algorithm in two environments, Starpilot and Craftax. The discovered scaling laws predicted the performance, resource allocation, and hyperparameter configuration of larger training runs based on runs requiring two orders of magnitude less compute. During analysis, the authors found that a derived Elo score was more predictive than the commonly used mean episode reward. Moreover, while the analysis found exploitable scaling laws, a non-monotonic scaling law for model size was discovered in the Craftax setting, highlighting the complexity of deriving global scaling laws in RL settings. The authors attribute this unfortunate observation to Ray interference.

Critical strengths

The authors bring up a critical, underexplored research area: scaling laws for general reinforcement learning. In particular, they argue the importance of determining the scaling laws for performance, model size, and hyperparameters, then leveraging them to predict performance for larger training jobs. To operationalize this, experiments leverage a novel application of Elo scoring to learn reliable scaling laws and show these laws can be used to achieve a new SOTA performance on one of the provided testing environments, Craftax. ​ In the Craftax environment, the authors also recognize an interesting inverse scaling phenomenon where sometimes the larger model does worse than a smaller model under the same training budget. Analysis of the environment dynamics and corresponding performance trends shows that the inverse relationship likely arises from a shift in reward signals, aka Ray interference.

Critical weaknesses

The paper positions itself as a first-of-its-kind empirical study of scaling laws in RL for general sequential decision-making (Line 126). However, the experimental setup does not seem to match this characterization. First, the authors restrict analysis to StarPilot and Craftax environments without providing principled justification for why these environments were chosen and whether they represent reasonably diverse decision-making setups. I am particularly concerned about the limited scope of environments because of how the analysis changes depending on the environment (Line 472). Second, the choice to focus on A2C is not motivated. In particular, I am curious why this algorithm was chosen over other relevant model-free algorithms. ​ I also had a few concerns about the Elo metric and the proposed advantages of using derived scaling laws to predict downstream performance. Specifically, unlike mean episode return, Elo scores require bootstrapping over a pool of training runs for a given training configuration. Given this, I believe an experiment determining how sensitive Elo’s predictive power is to the number of other training runs (Line 299) is warranted. A version of Figure 4 with different pool sizes would be a useful way to show this. If larger pool sizes are needed, I would expect a more in-depth analysis of the total compute required to derive the best-performing scaling laws at a target large training compute, i.e., 1e19 training FLOPs. In particular, how much training compute is saved when using a scaling law approach instead of other standard parameter search methods. This comparison would help motivate the claim that discovering these scaling laws can achieve near-optimal performance for larger training runs without using too much compute.

Questions for authors

I had a few clarification questions. ​

Where is the result referenced in Line 149-150 that led to a more restricted hyperparameter search space?

​ How exactly can optimal data size be derived from optimal model size (Line 202-203)?

​ When considering which hyperparameters to focus on (Line 149-152), did the authors keep any hyperparameters besides entropy regularization constant?

​ Given the authors posit Ray interference as the main reason for the inverse scaling trend seen in Craftax, is the assumption that the Starpilot environment does not have this shift in reward signal? I believe a discussion of the environments and their relative differences would have been useful in the Appendix or main text.

​ What was the total training compute used in the prior SOTA performance result on Craftax (Line 372)?

​ How does the paper’s derived performance on Starpilot compare with the current SOTA result?

​ Could the observed inverse scaling come from reward hacking instead of Ray interference? Do these environments have multiple reasonable reward formulations to check this?

​ What exactly do the authors mean by “a single, universal scaling law may not apply across all arbitrary RL environments” (Line 483)? Is this more a statement about how A2C handles different environments or something more fundamental about whether scaling laws even make sense for RL?

1 comment
AuthorsOpenReview ↗

Response to Reviewer HWSg

We thank Reviewer HWSg for the thorough review and detailed questions. Below we directly address both main concerns followed by your eight clarification questions.

W1 - Motivation for Environment and Algorithm Selections

For the environments, we chose Starpilot because it was used by Hilton et al. (2023) to highlight the difficulty of fitting performance scaling laws in RL. We chose Craftax because it is open-ended and the prior state of the art is far from the performance ceiling, so there is substantial room to demonstrate the practical usage and benefits of scaling laws.

For the algorithm, we chose A2C for its simplicity and straightforward compute accounting: total training compute factors directly as C=M⋅DC = M \cdot D, without additional axes such as an off-policy replay ratio or model-based planning steps.

W2 - Elo Pool Size and Compute Savings Over Direct Tuning at 1e19 FLOPs

Fitting scaling laws in Elo requires 0 extra training FLOPs compared to fitting scaling laws in episode return, as both reuse the exact same isoFLOPs checkpoints across [1e15, 1e17] FLOPs. The Elo pool size is 320 trained runs in Starpilot and 460 trained runs in Craftax. Summing across [1e15, 1e17] FLOPs, these runs consume 1.01e19 FLOPs in Starpilot and 1.39e19 FLOPs in Craftax, which is only 1x and 1.4x a single 1e19 FLOPs run.

Directly searching at 1e19 FLOPs will cost significantly more training compute. For instance, sweeping 5 model sizes directly at 1e19 FLOPs would require 5e19 FLOPs with 1 seed and 5e20 FLOPs with 10 seeds, costing approximately 5x–50x more compute. Moreover, we only tested up to 1e19 FLOPs due to resource and time constraints, but the fitted scaling laws could in principle extrapolate to even higher compute regimes where compute savings would be even more significant.

Answers to Clarification Questions

  • Q1 - Where is the result that led to a more restricted hyperparameter search space? This decision was based on informal preliminary experiments conducted early in the project to scope the search space, for which we do not have presentable sweeps to include in the paper. Across those early runs, fixing the entropy coefficient (0.01 in Starpilot and 0.001 in Craftax) performed consistently across small compute budgets, so we kept it fixed to make the main grid search over batch size, learning rate, and model size computationally tractable.
  • Q2 - How can optimal data size be derived from optimal model size? There is a typo on Line 203, which printed N∗N^{\ast} instead of D∗D^{\ast} (now corrected in Section 3.2). Since total training compute factors as C=M⋅DC = M \cdot D, substituting M∗(C)=aMCdMM^{\ast}(C) = a_M C^{d_M} directly yields the optimal data size: D∗(C)=C/M∗(C)=(1/aM)C1−dMD^{\ast}(C) = C / M^{\ast}(C) = (1 / a_M) C^{1 - d_M}.
  • Q3 - Were any hyperparameters besides entropy regularization kept constant? Yes, all hyperparameters other than batch size and learning rate are kept constant across compute budgets.
  • Q4 - Is the assumption that Starpilot does not have a shift in reward signal? Yes. Starpilot has a stationary, homogeneous reward structure throughout every episode (+1 per enemy or turret destroyed) with no discrete progression gates or competing sub-goals, so additional compute strictly accumulates the same reward signal without shifting between competing objectives.
  • Q5 - What was the total training compute used in the prior Craftax SOTA? The prior 20.6% SOTA is from Hamon (2024) (transformerXL_PPO_JAX), which trained a 5M-parameter Transformer-XL agent with PPO (4 update epochs per batch) for 4e9 environment frames. Under standard 6ND accounting, total training compute is 6 x 5e6 x 4 x 4e9 = 4.8e17 FLOPs.
  • Q6 - How does our Starpilot performance compare with the current SOTA? In ProcGen's unlimited-level Starpilot (Hard) setting, Cobbe et al. (2020) establish an asymptotic benchmark ceiling of 35 using large-batch PPO trained for 8 billion frames. Our scaled A2C agent reaches 43.13 at 1e19 FLOPs (7.8 billion frames), which exceeds the 35 ceiling at a comparable frame budget.
  • Q7 - Could the observed inverse scaling come from reward hacking instead of Ray interference? No, it is not reward hacking, as the per-achievement metrics in Figure 5 directly confirm: at the 3e14 FLOPs crossover, the smaller model achieves higher return by unlocking genuine downstream progression achievements (such as enter_dungeon), not by exploiting a degenerate reward loop.
  • Q8 - What is meant by "a single, universal scaling law may not apply across all arbitrary RL environments"? The scaling law can depend on both the environment dynamics and the algorithm's learning dynamics, and therefore a single universal scaling law that applies to all environments may not exist. It is a statement on the dependency of scaling laws on the environment and learning dynamics.
Reviewer_JJHBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how hyperparameters, model size, and performance scale with training compute for A2C in Starpilot and Craftax. The authors combine bootstrap-based hyperparameter selection with isoFLOPs experiments and introduce a single-agent Elo evaluation procedure based on pairwise comparisons of returns on shared environment instances. They evaluate extrapolation to larger training budgets, report strong Craftax performance, and identify a local non-monotonic relationship between training compute and the compute-optimal model size.

Critical strengths
  1. The paper evaluates extrapolation rather than only fitting observed results. Separately examining hyperparameter, allocation, and performance predictions makes the study relevant to practical training decisions.
  2. It identifies local departures from otherwise useful global scaling trends. The non-monotonic behavior cautions against applying a global power law indiscriminately across training regimes and constitutes a useful empirical contribution.
Critical weaknesses
  1. The independence of the Elo extrapolation protocol is insufficiently specified. If high-compute agents participate in jointly calibrating the Elo scores used for small-compute fitting, those fitting targets may not be available independently before the large-scale experiments. Anchoring to a random policy removes an additive ambiguity but does not generally make fitted ratings invariant to league composition. This is a clarification concern, not an established instance of leakage.
  2. The largest-scale experiments provide limited evidence of configuration optimality. A successful run using the predicted configuration demonstrates practical feasibility, but not proximity to the optimum at that budget. Prediction intervals and a decision-relevant interpretation of the reported errors would also help substantiate the claim of accurate extrapolation.
Questions for authors
  1. Do the high-compute agents participate in the league used to estimate the small-compute Elo scores? Does extrapolation remain accurate when the reference agents and their ratings are fixed beforehand?
  2. Can you report uncertainty intervals for the fitted laws and high-compute predictions, incorporating training variability and Elo estimation uncertainty?
  3. At the existing higher-compute budgets, how much does the predicted configuration improve over simply retaining the best small-compute configuration?
1 comment
AuthorsOpenReview ↗

Response to Reviewer JJHB

We thank Reviewer JJHB for their constructive feedback on the methodology. Below we first clarify the role of the 1e19 FLOPs experiments and then directly answer all three of your questions.

On Configuration Optimality at the Largest Scale (1e19 FLOPs)

As noted in Section 3, the largest-scale experiments (1e19 FLOPs) were not meant to verify the optimality of the predicted configuration; they were meant to demonstrate the practical usage of scaling laws in the exact regime where verifying optimality via sweeps is computationally prohibitive. The predictive power and optimality of our hyperparameter and allocation scaling laws are verified independently at 1e17 FLOPs (Figure 2) and 1e18 FLOPs (Figure 3 and Table 2), where measuring empirical ground-truth optima remains feasible.

Q1 - Do High-Compute Agents Participate in the Small-Compute Elo League?

No. High-compute agents (1e18 and 1e19 FLOPs) do not participate in estimating the small-compute (1e15–1e17 FLOPs) Elo ratings, and all reported extrapolations already keep the reference agents and their ratings fixed beforehand. Our evaluation uses a two-stage protocol (now written out explicitly in the Appendix):

  • Stage 1 (Small-Compute League): We form a pairwise tournament exclusively among checkpoints in [1e15, 1e17] FLOPs and the random policy, fit Bradley-Terry Elo ratings anchored so the random policy has 0 Elo, and fit all scaling laws strictly to these Stage-1 ratings.
  • Stage 2 (Frozen-Reference Evaluation): To score 1e18 and 1e19 checkpoints on the same scale, we evaluate them against the Stage-1 pool while freezing all Stage-1 ratings at their Stage-1 values, so only the held-out 1e18 and 1e19 ratings are fitted.

Q2 - Uncertainty Intervals Incorporating Training Variability and Elo Estimation Uncertainty

To propagate both RL training variability and Elo estimation uncertainty simultaneously, we run a 100-replicate non-parametric seed bootstrap over [1e15, 1e17] FLOPs (resampling the 10 seeds with replacement, rebuilding the pairwise win matrix, re-estimating anchored Bradley-Terry Elo, and re-fitting the isoFLOPs parabolas and scaling lines in each replicate). Fitted scaling-law slopes are shown below:

EnvironmentParameterPoint FitBootstrap Median95% CI
StarpilotdMd_M0.4420.444[0.407, 0.503]
StarpilotdElod_{\mathrm{Elo}}241.0236.7[228.8, 243.2]
CraftaxdMd_M0.4280.428[0.200, 0.684]
CraftaxdElod_{\mathrm{Elo}}223.9216.0[200.9, 230.2]

Extrapolating these bootstrap fits to 1e18 and 1e19 FLOPs yields the following prediction intervals:

EnvironmentQuantityPoint PredBootstrap Median95% CIRealized
StarpilotM∗M^{\ast} @ 1e184.85e84.88e8[4.46e8, 6.09e8]3.69e8
StarpilotElo @ 1e181151.51141.7[1127.9, 1154.0]1110.7
StarpilotM∗M^{\ast} @ 1e191.32e91.36e9[1.15e9, 1.92e9]—
StarpilotElo @ 1e191392.51378.4[1358.9, 1396.2]1312.3
CraftaxM∗M^{\ast} @ 1e182.73e72.84e7[1.28e7, 7.79e7]2.65e7
CraftaxElo @ 1e182043.02022.6[1963.2, 2074.5]2087.5
CraftaxM∗M^{\ast} @ 1e197.31e77.62e7[2.20e7, 3.74e8]—
CraftaxElo @ 1e192266.82238.6[2170.5, 2300.8]2186.0

Q3 - Improvement Over Retaining Small-Compute Configurations

We do not have access to enough compute resources at the moment to test small-compute configurations at 1e18 FLOPs. Instead, we reuse our hyperparameter sweep data (Figure 2): we choose the empirically best batch size and learning rate at 1e15 FLOPs (B=29,α=2−9B = 2^9, \alpha = 2^{-9} in Starpilot; B=212,α=2−10B = 2^{12}, \alpha = 2^{-10} in Craftax) and read their IQM episode return at 1e17 FLOPs alongside our scaling-law predicted hyperparameters and the optimal hyperparameters at 1e17 FLOPs:

Environment1e15 Best Hypers @ 1e17Predicted Hypers @ 1e17Optimal Hypers @ 1e17
Starpilot30.4832.0232.31
Craftax58.5463.4664.76

Retaining the best hyperparameters at 1e15 FLOPs performs significantly worse than the predicted hyperparameters and the optimal hyperparameters at 1e17 FLOPs, highlighting the necessity of predicting hyperparameters via scaling laws.

Reviewer_wofBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper report on a systematic study of scaling laws in reinforcement learning, i.e. it investigates whether smaller experiments can guide the configuration and predict the performance of larger RL training runs.

Their contribution builds on an existing RL scaling literature, with particular emphasis on a single-agent Elo metric and validation beyond the compute range used for fitting.

The authors consider three mains aspects: (i) achievable performance, (ii) optimal model-data allocation, and (iii) optimal hyperparameters. Using a two-stage framework on A2C in Starpilot and Craftax environments, the authors discover power laws for hyperparameters and optimal FLOPsper-frame. The experiments demonstrate that the single-agent Elo scores scale log-linearly with compute. Scaling laws predict performance up to 100× larger scales. Guided by these laws, the authors new impressive SOTA of 52.1% on Craftax at 1e19 FLOPs. The paper also reports non-monotonic scaling of compute-optimal model size in Craftax. The authors connects this to "Ray interference" pathology.

Critical strengths
  1. Comprehensive three-question framework with validation at multiple scales. The authors report a systematic study of performance, allocation, and hyperparameter questions with validation of prediction fidelity at 10× compute frontier (comparing predictions against empirical ground truth) and practical demonstration at 100× scale. Within the focused/narrow investigation of the single agent and the Elo metric, the authors established that it enables reliable extrapolation. The limitations of the study are discussed appropriately.

  2. The paper demonstrates extrapolation accuracy, establishing scaling laws as practical engineering tools for RL rather than just descriptive regularities.

  3. Guided by scaling laws, the authors train to 1e19 FLOPs and achieve 52.1% on Craftax, establishing new SOTA. Achieving SOTA through scaling-law-guided training demonstrates that the approach has practical utility for resource allocation in real RL projects

Critical weaknesses

Assuming the novelty claims are cross-checked and corrected if needed I do not see major flaws.

Questions for authors
  1. The authors may need to relate their study to Rybkin et al. (ICML 2025) https://proceedings.mlr.press/v267/rybkin25a.html that seem to address performance predictability, resource allocation, and hyperparameter relationships, with explicit extrapolation experiments. The novelty claim may need some adjustment.

  2. Is the connection to Ray interference convincing / not speculative?

1 comment
AuthorsOpenReview ↗

Response to Reviewer wofB

We thank Reviewer wofB for their positive assessment of our work. Below we address both of your questions.

Q1 - Relation to Rybkin et al

Thank you for pointing out Rybkin et al. (2025). In Section 2 of the revised manuscript, we cite and relate to Rybkin et al. Two key differences distinguish our study:

  • Performance prediction at larger FLOPs: Rybkin et al. address the allocation and hyperparameter questions for reaching pre-specified return thresholds, whereas they do not address the performance question: their scaling laws do not predict how well an agent performs at a larger training FLOPs budget and cannot predict a performance level that has never been attained before. By introducing single-agent Elo, our scaling laws directly predict previously unattained performance levels up to 100× larger compute scales, as demonstrated by our new Craftax SOTA result.
  • RL paradigm and compute allocation axes: Rybkin et al. focus on off-policy value-based methods at a fixed model size, where training compute is allocated between environment data and the update-to-data (UTD) ratio. In contrast, we focus on on-policy actor-critic methods, where training compute is allocated between environment data and model size (C=M⋅DC = M \cdot D).

Q2 - Is the Connection to Ray Interference Convincing or Speculative?

It is a speculative explanation. On a behavior level, Figure 5 clearly exhibits the characteristic symptom of Ray interference: at 3e14 FLOPs, learning the higher-reward enter_dungeon skill coincides with a sharp drop in optional defeat_monsters achievements, as one reward source suppresses another and extra model capacity goes temporarily unused. However, we did not conduct a gradient-level study to confirm that Ray interference is indeed the underlying root cause.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all three reviewers for their thoughtful and constructive feedback. We summarize here the concrete additions and revisions in the updated manuscript and answer each reviewer's specific questions directly in separate comments below.

  1. Full 10-Seed Completion of the 1e19 FLOPs Craftax Run. At initial submission, our 1e19 FLOPs Craftax hero run had partially completed (52.1% score). All 10 independent seeds have now completed, reaching a new SOTA Craftax score of 54.0% and reducing the prediction error in win rate from +15.9% to +11.4%.
  2. Related Work Positioning. We revised the related work section and added a discussion of Rybkin et al. (ICML 2025) alongside Fu et al. (2026).
  3. Bootstrap Uncertainty Intervals. In the Appendix, we added 95% confidence and prediction intervals from a 100-replicate non-parametric seed bootstrap over [1e15, 1e17] FLOPs that jointly propagates RL training variability and Elo estimation uncertainty.
  4. Explicit Two-Stage Frozen-League Elo Protocol. In the Appendix, we wrote out our two-stage Bradley-Terry Elo procedure explicitly and clarified in Section 3.3 that Elo reuses the exact isoFLOPs checkpoints with zero extra training FLOPs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.