- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
As training jobs grow, scaling laws have become invaluable for predicting performance, resource allocation, and optimal hyperparameter configuration with only inexpensive experiments. While the deep learning community has successfully leveraged scaling laws, the corresponding reinforcement learning (RL) community has lagged. This paper aims to discover fundamental scaling laws for RL training. To operationalize this, the authors studied the A2C algorithm in two environments, Starpilot and Craftax. The discovered scaling laws predicted the performance, resource allocation, and hyperparameter configuration of larger training runs based on runs requiring two orders of magnitude less compute. During analysis, the authors found that a derived Elo score was more predictive than the commonly used mean episode reward. Moreover, while the analysis found exploitable scaling laws, a non-monotonic scaling law for model size was discovered in the Craftax setting, highlighting the complexity of deriving global scaling laws in RL settings. The authors attribute this unfortunate observation to Ray interference.
Critical strengths
The authors bring up a critical, underexplored research area: scaling laws for general reinforcement learning. In particular, they argue the importance of determining the scaling laws for performance, model size, and hyperparameters, then leveraging them to predict performance for larger training jobs. To operationalize this, experiments leverage a novel application of Elo scoring to learn reliable scaling laws and show these laws can be used to achieve a new SOTA performance on one of the provided testing environments, Craftax. In the Craftax environment, the authors also recognize an interesting inverse scaling phenomenon where sometimes the larger model does worse than a smaller model under the same training budget. Analysis of the environment dynamics and corresponding performance trends shows that the inverse relationship likely arises from a shift in reward signals, aka Ray interference.
Critical weaknesses
The paper positions itself as a first-of-its-kind empirical study of scaling laws in RL for general sequential decision-making (Line 126). However, the experimental setup does not seem to match this characterization. First, the authors restrict analysis to StarPilot and Craftax environments without providing principled justification for why these environments were chosen and whether they represent reasonably diverse decision-making setups. I am particularly concerned about the limited scope of environments because of how the analysis changes depending on the environment (Line 472). Second, the choice to focus on A2C is not motivated. In particular, I am curious why this algorithm was chosen over other relevant model-free algorithms. I also had a few concerns about the Elo metric and the proposed advantages of using derived scaling laws to predict downstream performance. Specifically, unlike mean episode return, Elo scores require bootstrapping over a pool of training runs for a given training configuration. Given this, I believe an experiment determining how sensitive Elo’s predictive power is to the number of other training runs (Line 299) is warranted. A version of Figure 4 with different pool sizes would be a useful way to show this. If larger pool sizes are needed, I would expect a more in-depth analysis of the total compute required to derive the best-performing scaling laws at a target large training compute, i.e., 1e19 training FLOPs. In particular, how much training compute is saved when using a scaling law approach instead of other standard parameter search methods. This comparison would help motivate the claim that discovering these scaling laws can achieve near-optimal performance for larger training runs without using too much compute.
Questions for authors
I had a few clarification questions.
Where is the result referenced in Line 149-150 that led to a more restricted hyperparameter search space?
How exactly can optimal data size be derived from optimal model size (Line 202-203)?
When considering which hyperparameters to focus on (Line 149-152), did the authors keep any hyperparameters besides entropy regularization constant?
Given the authors posit Ray interference as the main reason for the inverse scaling trend seen in Craftax, is the assumption that the Starpilot environment does not have this shift in reward signal? I believe a discussion of the environments and their relative differences would have been useful in the Appendix or main text.
What was the total training compute used in the prior SOTA performance result on Craftax (Line 372)?
How does the paper’s derived performance on Starpilot compare with the current SOTA result?
Could the observed inverse scaling come from reward hacking instead of Ray interference? Do these environments have multiple reasonable reward formulations to check this?
What exactly do the authors mean by “a single, universal scaling law may not apply across all arbitrary RL environments” (Line 483)? Is this more a statement about how A2C handles different environments or something more fundamental about whether scaling laws even make sense for RL?
Response to Reviewer HWSg
We thank Reviewer HWSg for the thorough review and detailed questions. Below we directly address both main concerns followed by your eight clarification questions.
W1 - Motivation for Environment and Algorithm Selections
For the environments, we chose Starpilot because it was used by Hilton et al. (2023) to highlight the difficulty of fitting performance scaling laws in RL. We chose Craftax because it is open-ended and the prior state of the art is far from the performance ceiling, so there is substantial room to demonstrate the practical usage and benefits of scaling laws.
For the algorithm, we chose A2C for its simplicity and straightforward compute accounting: total training compute factors directly as C=M⋅D, without additional axes such as an off-policy replay ratio or model-based planning steps.
W2 - Elo Pool Size and Compute Savings Over Direct Tuning at 1e19 FLOPs
Fitting scaling laws in Elo requires 0 extra training FLOPs compared to fitting scaling laws in episode return, as both reuse the exact same isoFLOPs checkpoints across [1e15, 1e17] FLOPs. The Elo pool size is 320 trained runs in Starpilot and 460 trained runs in Craftax. Summing across [1e15, 1e17] FLOPs, these runs consume 1.01e19 FLOPs in Starpilot and 1.39e19 FLOPs in Craftax, which is only 1x and 1.4x a single 1e19 FLOPs run.
Directly searching at 1e19 FLOPs will cost significantly more training compute. For instance, sweeping 5 model sizes directly at 1e19 FLOPs would require 5e19 FLOPs with 1 seed and 5e20 FLOPs with 10 seeds, costing approximately 5x–50x more compute. Moreover, we only tested up to 1e19 FLOPs due to resource and time constraints, but the fitted scaling laws could in principle extrapolate to even higher compute regimes where compute savings would be even more significant.
Answers to Clarification Questions
transformerXL_PPO_JAX), which trained a 5M-parameter Transformer-XL agent with PPO (4 update epochs per batch) for 4e9 environment frames. Under standard 6ND accounting, total training compute is 6 x 5e6 x 4 x 4e9 = 4.8e17 FLOPs.enter_dungeon), not by exploiting a degenerate reward loop.