- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates how evaluation protocols and sampling noise affect claims about whether RLVR expands a model’s reasoning boundary. By holding a base/RLVR model pair fixed and varying sampling temperature, prompting, and scoring rules, the authors examine changes in the pass@k crossover point. They introduce empirical noise estimates and a decision procedure that distinguishes finite-budget RLVR advantages, base-model catch-up, and inconclusive comparisons. I recommend for acceptance because it provides a new angle to understand the RLVR's boundaries and limitations.
Critical strengths
Important and well-motivated research question. The paper addresses a central measurement issue in the RLVR boundary debate: whether apparently conflicting conclusions reflect differences in model capabilities, evaluation protocols, or statistical uncertainty.
Controlled experimental design. Holding the model pair fixed while changing individual protocol components provides direct evidence that evaluation choices can materially affect the observed crossover. The temperature comparison is particularly informative.
-
Explicit treatment of measurement uncertainty. Rather than treating every change in the crossover point as meaningful, the authors compare observed movements against empirically measured noise. Distinguishing inconclusive evidence from a negative result is a valuable contribution.
-
Transparency about limitations. The paper reports censoring, truncation, protocol validity concerns, and cases where its instruments cannot resolve a comparison. It also acknowledges the exploratory nature of the proposed mechanism. Also, it cites many very closely related studies such as Yue Yang's and Wu's papers.
-
Useful practical contribution. The evaluation harness, decision procedure, and reporting recommendations could improve the reproducibility and interpretation of future RLVR studies.
-
Careful distinction between performance and support. The authors explicitly frame their verdicts as finite-budget comparisons rather than claims about theoretical support. This distinction helps prevent overinterpretation of pass@k results.
Critical weaknesses
The strongest noise-clearing temperature results are concentrated in Minerva comparisons using a chat template on the base model. Although this limitation is clearly acknowledged, further evidence under appropriate base-model prompting would strengthen generalizability.
Multiple scoring rules reuse the same generated completions, so the reported comparison counts should not be interpreted as independent replications. Organizing the main findings around distinct generation settings would make the evidence easier to assess.
The proposed pass@1 asymmetry mechanism is suggestive rather than conclusive. It would benefit from further cases that distinguish its predictions from the simple baseline of always predicting a later crossover.
Questions for authors
Can the main temperature findings be replicated under base-appropriate prompting with sufficient sampling and token budgets?
How sensitive are the final verdicts to the gate threshold and the aggregation rule?
3 comments
Thanks for the response from the authors. I have read through the point-by-point reply and found it successfully addressed my concern. I hope the authors can incorporate these new results into the revised version.
I will maintain my original score and recommend clear acceptance.
The new results are in the revised PDF
Thank you for confirming, and for the clear recommendation. The revised PDF, uploaded on 13 November, already contains the new results. Section 4 reports both registered follow-up experiments: the four-example-prompt replication on Minerva (page 6) and the test of the g = 0.15 gate on new same-model runs (page 8). Appendix P gives the full read-outs: P.1 the replication (Tables 14 and 15), P.2 the gate test (Table 16), P.3 the two dated amendments to the registration, and P.4 the post-review analyses of gates, aggregation rules, judgeable shares and truncation (Tables 17 to 23).
Response to Reviewer DBwD
Thank you for the review and for the two precise questions. A registered experiment answers Q1, and a second registered experiment validates the gate (Appendix P). The threshold and aggregation sensitivity uses post-review analyses of the stored outputs.
Q1. Replication under base-appropriate prompting. With the same four-example prompt on both arms (Qwen2.5-7B and SimpleRL-Zoo, top-p 1, 128 samples per question, 4,096-token cap), three of four Minerva seeds change from "base caught up" at T=0.6 to "RLVR still ahead" at T=1.0, and the fourth changes from unjudgeable to ahead. This meets the first registered criterion. The four seeds are the replication units; the 12 scoring rules reuse their generations and are not counted as replications. Appendix P also regroups the original rule-level comparisons by generation setting (a post-review table). On the token budget: truncation at the 4,096-token cap is at most 7.64% on Minerva and 2.39% on MATH-500. Appendix P.1 (Tables 14 and 15) reports both benchmarks and every registered test.
Q2. Gate threshold and aggregation. In the post-review analyses, Llama and Mistral change under the all-rule vote at each of the seven gates we tested, from g=0.01 to g=0.25, and at g=0.15 also when at least 18 of 36 labels must be judgeable. Tables 19 to 21 in Appendix P list the changes over the gates, aggregation rules and judgeable-share requirements. With the case-study rule alone at g=0.15, Llama keeps its change and Mistral becomes unjudgeable at T=1.0. Under unanimity at g=0.15, both pairs are unjudgeable at both temperatures. For the gate itself, we fixed g=0.15 and tested it on new same-model runs: 11 cells across four base models, both formats and three benchmarks. Of 220 ordered comparisons, 1 is AHEAD (0.5%), 177 are CAUGHT and 42 are unjudgeable; on the 22 prespecified disjoint seed pairs, 0 are AHEAD (exact one-sided 95% upper bound 12.7%). The registered criterion is met.
The pass@1 asymmetry. Before the new runs, we registered how the pass@1 ratios predict the direction of a verdict shift, and the prediction matched on Minerva: pooled base pass@1 falls from 8.69% to 5.49% and RLVR pass@1 from 9.50% to 7.09%, and the verdict moves from CAUGHT to AHEAD. In the original grid, the asymmetry matches the observed direction in 12 of 12 cases and the constant rule in 11 of 12. The replication agrees with both predictors, so the case that separates them is the one in the original grid.