- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper discusses the issue of identifying strong reinforcement learning (RL) baselines for long-horizon agents, with the motivation to standardize the baselines in the literature and avoid people from using under-tuned baselines. To that end, the authors propose to use overfitting on a small scale dataset as a way to reveal the failure mode, and their findings suggest a list of modifications that can be made to improve its performance, such as gated reward on valid completion, matching training and inference numerics, and scheduling useful recent experiences.
=== post rebuttal ===
Thank you for the response. I feel my concern for (W1) remains since the paper in its current state feels like putting together a lot of things and sharing its observations, rather than exploring scientific insights. Thus, I maintain my score.
Critical strengths
The paper explores multiple ways to combine together different engineering practices together to form a better RL baseline, which could be informative to the community.
Critical weaknesses
- The paper is very confusing to read, and I feel its presentation is more like listing a group of things the authors attempted at the problem without connecting on why they are done this way. The first four pages are easy to follow, but the remaining is hard to read.
- As the authors have acknowledged in Section 6, most experiments in this paper only uses one seed over one harness only. Given that the claim of the paper is on a strong RL baseline that people can use to avoid from using an under-tuned weaker baseline, I think more experiments and some report in terms of variance should at least be attempted. It is understandable that compute budgets may restrain people from conducting larger scale experiments, but the authors can at least try two seeds instead of one.
- On section 3.1, the authors claim that “If an RL pipeline cannot reliably memorize a small, solvable subset of tasks, it cannot be expected to generalize over a large, noisy corpus.” While I agree with this statement, what I do not agree with is the inverse, which the authors implicitly assumed in the method’s design. It seems the authors mostly relied on using performance obtained from overfitting a model on a small testbed to establish whether an RL algorithm is usable or not, but it could also be the case that “if an RL pipeline reliably memorize a small solvable subset of tasks, it cannot generalize.” This scenario is possible for the authors’ proposed approach. Thus, I believe the study’s methodology needs more evidence, explanation, and clarification.
Questions for authors
Similar as Critical Weakness (3), I wonder what is the reason behind choosing overfitting as a stress test for the proposed approach?
Response to Reviewer oo56
Thank you for taking the time to read our work.The soundness rating rests on a reading of the method that the paper does not support. We address it first, then presentation and seeds.
Soundness (necessary vs. sufficient). The review argues that fitting a small pool does not imply generalization. The paper never claims it does. Its logic has two steps, in opposite directions.
Holding the pool is necessary for full training to succeed, not sufficient: the submitted text opens the stress-test section with "If an RL pipeline cannot reliably memorize a small, solvable subset of tasks, it cannot be expected to generalize over a large, noisy corpus."
Our corrections are sufficient for the recipe to hold the pool, not necessary: other remedies may also work, and Appendix K notes that removing recovery or enforcing our particular format "is not a universal prescription". So the screen only rejects recipes, passing it is never taken as evidence of generalization, and every headline number comes from held-out SWE-bench Verified and Pro, none from the training pool. The gain–decay model in Appendix B.1 formalizes the first step: a recipe that keeps losing ground on the pool carries a permanent deficit into full training. The paper also reports the case the review raises, where training reward and held-out accuracy separate: past step 250 the 4B degrades on held-out data while its training reward barely moves, and the 9B, trained on the same deck under the same eviction rule, does not (Fig. 6). The revision adds an explicit statement of this screen-then-validate logic to Sections 3.1 and 6.
Presentation. Each component in Section 3 is introduced by the failure that motivates it, in the order in which the diagnostic exposes those failures:
The revision collects this chain in a roadmap paragraph and Table 1, which lists each failure with its fix and measured effect.
Each fix is an established technique, chosen for a failure the testbed exposed and weighed against the published alternatives that Appendix K compares with ours. For invalid turns, SkyRL-Agent continues with recovery hints, NeMo assigns negative advantages to invalid calls, and SWE-Master forces a submission; FrogNano and MiMo-V2.6 penalize length, while we use no length penalty. The testbed also tests alternatives directly: masking invalid turns instead of ending the rollout, run for this rebuttal, collapses three times and then plateaus about 15 points lower (see our reply to QkkX).
Seeds. Two new seeds of the proposed recipe agree with the submitted run within one point in every reported window. The stability claims of Section 3 concern training dynamics on the testbed, where repeated runs fit in the rebuttal period, so we reran the recipe there changing only the random seeds:
The table uses the paper's metric and windows, and seed 42 reproduces the submitted numbers. Values are post-gate rewards averaged over completed rollouts, with peaks taken within the 240 plotted epochs; revised Fig. 3 shows the three seeds as a mean with a min–max band.
We did not rerun the failing recipes, because a single collapse already makes a recipe unreliable. Seeds on which it happens to survive would not make it safe to use. Gated reward alone has in fact collapsed in two independent runs, near epoch 42 (Fig. 1) and near epoch 200 (faint line in Fig. 3).
Since the soundness concern rests on a claim the paper does not make, and the proposed recipe is stable across seeds, we ask the reviewer to reconsider the rating.