- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper asks whether measurements of a starting policy can predict which prompt distribution deserves RL compute. It constructs four environments, fits training gains using the density and coverage of usable advantage signals, and tests prediction across model scales. Environment choice matters a lot, but the predictor does not beat simple selection rules and fails on a uniform mixture.
Critical strengths
-
The question is useful. The authors match budgets by generated tokens, freeze predictions in advance, and report the 8B failure and competitive simple baselines openly.
-
The environment effects are substantial: the best and worst 4B environments differ by 11.4 points at matched token budget. Figure 3's finding that updates can concentrate on a small fraction of prompts is also informative.
Critical weaknesses
-
I don't see a practical gain in choosing environments. In Table 1, mean regret is 0.48, the same as always choosing HL and worse than H alone at 0.33. Simple rules also make the same choices at 14B. More concerning, the only candidate outside the four constructed environments—the uniform mixture—is predicted to lose utility and ranked last at all three budgets. It actually gains at all three and beats every individual environment at the largest budget (Table 4C). This matters more for selection than reducing curve RMSE from 2.40 to 2.09. Measurement is also costly: Stage D alone uses 30% of the tokens of the eight 14B RL runs, and regret does not charge this cost. The results show that environment choice matters, but do not yet show that this predictor helps. Independent candidate tests or a narrower framing are needed.
-
The explanation of usable signal needs more work. In the frozen 14B model, rho has coefficient -0.077 (Appendix D): at fixed H, more signal density predicts less gain. H is almost identical to the fraction of groups with mixed outcomes, with correlation 0.999. What does the extra coordinate explain? The setup truncates 38%–75% of responses, disables dynamic sampling, and couples response length with truncation, so their effects cannot be separated. Please explain the negative coefficient and whether the pattern holds with less truncation before presenting this as a general signal-survival principle.
Questions for authors
-
On the frozen 14B test, how do budget/scale-only, H-only, and cost/admission predictors compare in RMSE and ranking? Does rho–H still help once measurement costs are included?
-
What are the uniform mixture's rho and H? Are they outside the fitting range, and why is the prediction reversed? Please include uniform mixing as a simple selection baseline.
-
How do you interpret the negative rho coefficient? Is there an analysis controlling for response length, or evidence under a higher length cap, that separates length from truncation?
Also, please clarify whether the main conclusions use token- or step-matched budgets; Table 2 and Figure 2 give different orderings. The drop in pass@4 for every 14B arm also needs to qualify what “utility” means here.
3 comments
Author rebuttal to Reviewer q7mg - Part 2/2
W2/Q1 summary: The negative coefficient conflicts with the paper's monotone interpretation of usable signal.
W2/Q1 response: Revised Section 3.2 calls admitted advantage density, and Section 6 separates measurement from causal interpretation. It is admitted leave-one-out coefficient mass per generated token, not a claim that increasing it must improve utility.
The negative fitted coefficient is conditional on , scale, and budget in a small correlated design. Admission, response cost, mixedness, , and are not independently varied. The current design therefore does not isolate truncation masking, longer reasoning, or admission as the cause of the low-admission endpoint wins. The revision removes language implying that separate causal result. The supported mechanism statement remains consequential: reward variation, admitted coefficient mass, and its concentration are distinct under the fixed loss contract, while raw reward variation alone does not order held-out utility.
The predictive comparison remains strong. Removing environment information lowers same-fold prospective 14B rank accuracy from 77.8% to 50.0% and raises regret from 0.288 to 1.358 points under the fixed tie-break.
W2/Q4 summary: The truncation-heavy contract, disabled dynamic sampling, and small warm start may determine the observed ordering.
W2/Q4 response: Revised Section 4 and Appendix A specify the 12,288-token cap and overlong filtering as part of the controlled contract. NeMo treats overlong filtering and dynamic sampling separately: the former masks capped responses, while the latter replaces zero-variance groups. We disabled dynamic sampling because replacement would change the realized environment distribution. This does not imply invariance to DAPO-style replacement or a higher cap.
The existing trajectories provide a stronger diagnostic. Cumulative truncation is not uniformly increasing during RL. From step 16 to 128 it falls from 65.5% to 53.2% in 1.7B HL, 58.3% to 40.4% in 4B HL, 54.8% to 48.0% in 8B HL, and 57.4% to 35.7% in 14B HL.
The degraded 1.7B LH seed is different. Degeneration begins around steps 49–56. At step 56, mean output length is 11,310 tokens, 87.5% of responses are capped, and only 4/32 responses are loss-admissible. By step 72 its reward is zero under sustained cap pressure. We found no coincident KL, importance-ratio, checkpoint, tokenizer, or infrastructure anomaly. This supports an association between cap pressure and one learned failure mode, but not a separate causal effect of truncation.
The 2,048-example SFT warm start is fixed across all environments, so its size cannot explain the within-policy treatment spread. It does define the policy-relative measurements, however. A different warm start could change which prompts are mixed, admitted, or costly, and the revised scope statement now says so.
Q5 summary: Are comparisons aligned by actual generated tokens, and does the result extend beyond pass@1?
Q5 response: Sections 3.1 and 5 make generated-action-token matching primary. Step-128 values are descriptive because environments consume different tokens per step. The 11.4-point spread and Equation 7 use generated-token budgets.
Pass@1 is primary. Revised Section 7.2 reports the independent native- audit over 36,000 responses. Its per-response pass@1 agrees with the surface, , and HL again leads at 14B. Native pass@4 remains below the warm start for every environment. The benefit is improved single-rollout reliability, not expanded four-sample coverage.
The protected candidate test directly addresses a central request for independently constructed environments. Candidate identities, forecasts, and scoring rules were fixed before any RL outcome. The selected model lowers allocation regret relative to the environment-agnostic and simple feature baselines, while the later crossover defines the current claim boundary. In light of this direct test and the revised analysis, we would appreciate the reviewer's reassessment of the paper and overall score.
Reply to author's rebuttal
Thank you for the clarification. After considering your response, I believe the manuscript still requires substantial revision. I will therefore maintain my original score.
Author rebuttal to Reviewer q7mg - Part 1/2
We thank the reviewer for separating the controlled effect, curve prediction, allocation, and composition transfer. The result is substantially larger than a 0.31-point regression improvement.
Under one warm start and trainer, environment choice produces an 11.4-point token-matched utility spread at 4B. The four effects replicate on Polaris with correlation 0.991. The untouched 14B curves are predicted with 2.292-point RMSE and 0.288-point mean regret. Finite-budget RL returns therefore depend materially on the environment and can be forecast within the controlled family. The uniform-mixture result identifies the boundary beyond it.
W1/Q2 response: Revised Section 7.2 fits every requested baseline on the same 1.7B, 4B, and 8B rows and scores 14B without refitting. These are post-hoc comparators on the original prospective fold, not additional prospective forecasts.
The direct conclusion is that environment information improves prospective ranking over a model that knows only scale and budget. Under the fixed tie-break needed for the environment-agnostic model, mean regret falls by 1.07 points. Cost/admission has slightly better RMSE than ρ-H and equal rank/regret, so the evidence supports environment-aware prediction rather than the unique superiority of one representation.
The task is not reducible to retaining the four labels. In a leave-one-environment-type-out audit, each fold removes one complete HH, HL, LH, or LL type at all four scales. ρ-H obtains 2.45-point RMSE and 0.48-point regret, compared with 2.73 and 4.20 for budget/scale only. Cost/admission obtains 2.58 and 0.48. This remains a within-family test, but the held-out label is absent from fitting.
W1/Q3 response: Revised Section 7.3 reports the 14B uniform-mixture miss and protected candidate test. The mixture has ρ=13.600 and H=0.418, within the 16-condition ranges and inside their convex hull in (logρ,H). The miss is therefore not simple range extrapolation. It identifies a composition-transfer boundary. The revised title, How Training Environments Shape RL Utility Across Model Scale, makes this scope explicit.
Before review, we also froze a protected candidate-selection experiment with two identity-disjoint uniform prompt banks and one nonuniform mixture, all disjoint from the evaluation panel. Candidate identities, two seeds per candidate, forecasts, and scoring rules were fixed before any RL outcome. All six runs and 50 evaluation shards passed integrity, covering 25 policies and 25,000 responses.
The selected cost/admission model predicts uniform B>uniform A>nonuniform. It selects the observed best candidate at the first two of four budgets, then misses crossovers to nonuniform and uniform A. Mean regret is 0.740 points, compared with 1.042 for budget/scale only, 1.250 for composition-only rules, and 1.760 for raw mixedness or starting success. Curve RMSE is 1.131 versus 1.045 for budget/scale only, and both reach 50.0% rank accuracy. Thus the measurements reduce decision regret on independently constructed candidates but do not improve full-curve calibration or recover later crossovers. This extends the evidence beyond the four named factorial environments while preserving a clear composition-transfer boundary.
W1 response: Submitted regret measures RL tokens and does not include probe cost. Appendix E now reports it separately. A 64-prompt-per-environment probe uses one quarter of the submitted measurement and preserves all three frozen 14B decisions in 99.54% of 5,000 resamples, with 0.950 mean pairwise-order agreement. At 128 prompts, decision preservation is 100%, pairwise agreement is 0.980, and median curve-deviation RMSE is 0.311 points. This is retrospective sensitivity, not independent validation.
Continued in Part 2/2.