acceptodds
Under review as a conference paper at ICLR 2027

Judging RLVR Boundary Comparisons Against Their Own Measured Noise

Abstract

Whether reinforcement learning with verifiable rewards (RLVR) expands a base model's reasoning boundary is disputed in print. We show that both published answers can be read off one fixed (base, RLVR) pair at every sampling budget we test by moving only sampling temperature and scoring rule; the raw grid counts behind this are not floor-judged, and the base-ahead ones concentrate in chat-template cells at low temperature. Every movement is then judged against measured noise. Of the 30 registered comparisons between the two temperatures this debate uses, 11 are unjudgeable, moving less than the noise floor of the cells they compare; the rest sit on two Minerva cell pairs, and the 11 of them reseeded all clear again. In the sharpest instance the crossover point k⋆ moves from 10 to 48 on the same problems, scoring rule and seed. A post hoc observation at k=1 gives the movement a mechanism: temperature lowers the base arm's pass@1 by a larger factor than the RLVR arm's, and the sign of that asymmetry matches the direction k⋆ moves in every cell where it is defined, though a constant rule matches all but one. We turn the measurement into a decision procedure that returns AHEAD, CAUGHT or UNJUDGEABLE for a pair under a protocol, with its false-AHEAD rate set by a null control of two draws from one arm; neither the measured noise floor nor this null-calibrated gate has a counterpart we found in this literature. On seven released pairs the verdict changes with temperature for two of them at the null-calibrated gate: the answer belongs to the protocol as much as to the pair, and it names a finite-budget advantage rather than a support relation. The harness, the noise-floor and lineage instruments, and judge_boundary.py are in the supplementary material.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning with verifiable rewards, reasoning boundary, pass@k, evaluation protocol, seed noise, LLM post-training

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_DBwDOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how evaluation protocols and sampling noise affect claims about whether RLVR expands a model’s reasoning boundary. By holding a base/RLVR model pair fixed and varying sampling temperature, prompting, and scoring rules, the authors examine changes in the pass@k crossover point. They introduce empirical noise estimates and a decision procedure that distinguishes finite-budget RLVR advantages, base-model catch-up, and inconclusive comparisons. I recommend for acceptance because it provides a new angle to understand the RLVR's boundaries and limitations.

Critical strengths

Important and well-motivated research question. The paper addresses a central measurement issue in the RLVR boundary debate: whether apparently conflicting conclusions reflect differences in model capabilities, evaluation protocols, or statistical uncertainty.

Controlled experimental design. Holding the model pair fixed while changing individual protocol components provides direct evidence that evaluation choices can materially affect the observed crossover. The temperature comparison is particularly informative.

  1. Explicit treatment of measurement uncertainty. Rather than treating every change in the crossover point as meaningful, the authors compare observed movements against empirically measured noise. Distinguishing inconclusive evidence from a negative result is a valuable contribution.

  2. Transparency about limitations. The paper reports censoring, truncation, protocol validity concerns, and cases where its instruments cannot resolve a comparison. It also acknowledges the exploratory nature of the proposed mechanism. Also, it cites many very closely related studies such as Yue Yang's and Wu's papers.

  3. Useful practical contribution. The evaluation harness, decision procedure, and reporting recommendations could improve the reproducibility and interpretation of future RLVR studies.

  4. Careful distinction between performance and support. The authors explicitly frame their verdicts as finite-budget comparisons rather than claims about theoretical support. This distinction helps prevent overinterpretation of pass@k results.

Critical weaknesses

The strongest noise-clearing temperature results are concentrated in Minerva comparisons using a chat template on the base model. Although this limitation is clearly acknowledged, further evidence under appropriate base-model prompting would strengthen generalizability.

Multiple scoring rules reuse the same generated completions, so the reported comparison counts should not be interpreted as independent replications. Organizing the main findings around distinct generation settings would make the evidence easier to assess.

The proposed pass@1 asymmetry mechanism is suggestive rather than conclusive. It would benefit from further cases that distinguish its predictions from the simple baseline of always predicting a later crossover.

Questions for authors

Can the main temperature findings be replicated under base-appropriate prompting with sufficient sampling and token budgets?

How sensitive are the final verdicts to the gate threshold and the aggregation rule?

3 comments
AuthorsOpenReview ↗

Response to Reviewer DBwD

Thank you for the review and for the two precise questions. A registered experiment answers Q1, and a second registered experiment validates the gate (Appendix P). The threshold and aggregation sensitivity uses post-review analyses of the stored outputs.

Q1. Replication under base-appropriate prompting. With the same four-example prompt on both arms (Qwen2.5-7B and SimpleRL-Zoo, top-p 1, 128 samples per question, 4,096-token cap), three of four Minerva seeds change from "base caught up" at T=0.6 to "RLVR still ahead" at T=1.0, and the fourth changes from unjudgeable to ahead. This meets the first registered criterion. The four seeds are the replication units; the 12 scoring rules reuse their generations and are not counted as replications. Appendix P also regroups the original rule-level comparisons by generation setting (a post-review table). On the token budget: truncation at the 4,096-token cap is at most 7.64% on Minerva and 2.39% on MATH-500. Appendix P.1 (Tables 14 and 15) reports both benchmarks and every registered test.

Q2. Gate threshold and aggregation. In the post-review analyses, Llama and Mistral change under the all-rule vote at each of the seven gates we tested, from g=0.01 to g=0.25, and at g=0.15 also when at least 18 of 36 labels must be judgeable. Tables 19 to 21 in Appendix P list the changes over the gates, aggregation rules and judgeable-share requirements. With the case-study rule alone at g=0.15, Llama keeps its change and Mistral becomes unjudgeable at T=1.0. Under unanimity at g=0.15, both pairs are unjudgeable at both temperatures. For the gate itself, we fixed g=0.15 and tested it on new same-model runs: 11 cells across four base models, both formats and three benchmarks. Of 220 ordered comparisons, 1 is AHEAD (0.5%), 177 are CAUGHT and 42 are unjudgeable; on the 22 prespecified disjoint seed pairs, 0 are AHEAD (exact one-sided 95% upper bound 12.7%). The registered criterion is met.

The pass@1 asymmetry. Before the new runs, we registered how the pass@1 ratios predict the direction of a verdict shift, and the prediction matched on Minerva: pooled base pass@1 falls from 8.69% to 5.49% and RLVR pass@1 from 9.50% to 7.09%, and the verdict moves from CAUGHT to AHEAD. In the original grid, the asymmetry matches the observed direction in 12 of 12 cases and the constant rule in 11 of 12. The replication agrees with both predictors, so the case that separates them is the one in the original grid.

Thanks for the response from the authors. I have read through the point-by-point reply and found it successfully addressed my concern. I hope the authors can incorporate these new results into the revised version.

I will maintain my original score and recommend clear acceptance.

AuthorsOpenReview ↗

The new results are in the revised PDF

Thank you for confirming, and for the clear recommendation. The revised PDF, uploaded on 13 November, already contains the new results. Section 4 reports both registered follow-up experiments: the four-example-prompt replication on Minerva (page 6) and the test of the g = 0.15 gate on new same-model runs (page 8). Appendix P gives the full read-outs: P.1 the replication (Tables 14 and 15), P.2 the gate test (Table 16), P.3 the two dated amendments to the registration, and P.4 the post-review analyses of gates, aggregation rules, judgeable shares and truncation (Tables 17 to 23).

Reviewer_Z961OpenReview ↗
Rating
2
Confidence
1
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether conclusions about RLVR's impact on reasoning boundaries are sensitive to evaluation protocols and sampling noise. The authors systematically examine how sampling temperature, prompt formatting, and scoring rules affect the crossover point in pass@k between base and RLVR-trained models. They propose a noise-aware evaluation framework that measures the variability of crossover points and provides calibrated judgments of AHEAD, CAUGHT, or UNJUDGEABLE. Experiments across multiple mathematical reasoning benchmarks and seven publicly released base/RLVR model pairs show that evaluation choices can substantially alter the observed crossover points and, in some cases, reverse the conclusions about RLVR's advantage. The paper highlights the importance of accounting for sampling noise and evaluation protocols when interpreting finite-budget reasoning performance.

Critical strengths
  1. Important insights into RLVR boundary evaluation. The paper addresses an important and ongoing debate about whether RLVR expands the reasoning capabilities of base models. By demonstrating that different evaluation protocols can lead to substantially different crossover points, even for the same model pair, the work provides valuable insights into the reliability of existing RLVR boundary conclusions.

  2. Systematic and noise-aware evaluation. The authors introduce a measurement framework that accounts for sampling variability through repeated seeds, resampling-based noise estimation, and calibrated decision rules. The evaluation across multiple mathematical benchmarks and seven released model pairs further strengthens the empirical analysis. This framework provides a useful methodology for assessing whether observed differences in reasoning boundaries are statistically meaningful.

Critical weaknesses
  1. Limited generalizability of the main empirical findings. Although the paper evaluates multiple benchmarks and model pairs, the statistically distinguishable temperature effects in the main 7B experiment are concentrated in only two Minerva configurations. None of the MATH-500 comparisons exceeds its measured noise floor. Moreover, the strongest evidence is obtained under the chat-template setting. This concentration limits how broadly the observed protocol sensitivity can be generalized across benchmarks and evaluation settings.

  2. Sensitivity and validation of the proposed decision procedure. Several important aspects of the evaluation framework, including the null-calibrated decision threshold, were adjusted after observing experimental results. The paper also reports cases where the final verdict depends on the calibration threshold. While these adjustments are disclosed, further independent validation would strengthen confidence in the robustness and general applicability of the proposed AHEAD/CAUGHT/UNJUDGEABLE decision procedure.

Questions for authors
  1. Generalizability of the observed temperature effects. The paper reports that 19 of the 30 temperature comparisons exceed their measured noise floors, but all 19 are concentrated in two Minerva cell pairs, while none of the MATH-500 comparisons clears its noise floor. Could the authors clarify to what extent this concentration limits the generality of the main conclusions? In particular, is there additional evidence that the observed temperature sensitivity is not specific to these configurations? A convincing demonstration of robustness across benchmarks or model pairs would substantially strengthen my assessment.

  2. Robustness of the null-calibrated decision procedure. The decision threshold was adjusted from the registered value of g=0.25 to g=0.15 based on the null-control results. While the motivation is understandable, the paper also reports that some model-level verdicts are sensitive to this choice. Could the authors clarify how stable the AHEAD/CAUGHT/UNJUDGEABLE conclusions are under alternative thresholds and aggregation rules? In particular, would the main findings remain unchanged under an independently calibrated threshold? This is important for evaluating the reliability of the proposed methodology.

  3. Validity of the chat-template evaluation setting. The paper acknowledges that applying a chat template to a base model that was not trained with one may constitute an evaluation error. However, the strongest temperature-dependent crossover effects are observed under this setting. Could the authors clarify whether the main conclusion about protocol sensitivity remains supported when both models are evaluated under an appropriate and defensible prompting protocol? This distinction is important for determining whether the observed effects reflect genuine evaluation sensitivity or are primarily driven by an unsuitable prompting configuration.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Z961

Thank you for the review. Two experiments that we registered before any new generation answer Q1, Q3 and the validation part of Q2 (Appendix P). The sensitivity part of Q2 uses post-review analyses of the stored outputs.

Q1 and Q3. Generality, and a defensible prompt for the base model. The registered replication uses the same four-example prompt on both arms of Qwen2.5-7B and SimpleRL-Zoo (four paired seeds, top-p 1, 128 samples per question, 4,096-token cap). On Minerva, three of four seeds change from "base caught up" at T=0.6 to "RLVR still ahead" at T=1.0, and the fourth changes from unjudgeable to ahead. This meets the first registered criterion, a verdict shift. The registered direction check from pass@1 also matches. So on Minerva the verdict moves with temperature under base-appropriate prompting, with no chat template on either arm. On MATH-500, one seed moves from caught to ahead and none moves the other way. Appendix P.1 (Tables 14 and 15) reports both benchmarks and every registered test.

Evidence beyond this pair is already in the paper. On Qwen2.5-14B, both measurable contrasts move strictly later at T=1.0 under every rule with both endpoints (8 on MATH-500, 12 on Minerva), and every rule with a floor clears it (5 of 5 and 12 of 12; Table 9). Across the seven released pairs, the Llama-3.1-8B and Mistral-7B verdicts change with temperature.

Q2. Independent validation of the fixed threshold. We fixed g=0.15 in the registration and tested it on new same-model runs that played no part in choosing it: 11 cells across four base models, both prompt formats and three benchmarks. Of 220 ordered comparisons, 1 is AHEAD (0.5%), 177 are CAUGHT and 42 are unjudgeable. On the 22 prespecified disjoint seed pairs, 0 are AHEAD (exact one-sided 95% upper bound 12.7%). The registered criterion is met.

On alternative thresholds and aggregation (post-review analyses): Llama and Mistral change from "caught" to "ahead" at each of the seven gates we tested, from g=0.01 to g=0.25, and at g=0.15 both changes persist when at least half of the labels must be judgeable. With the case-study rule alone at g=0.15, Llama keeps its change and Mistral becomes unjudgeable at T=1.0. Under unanimity at g=0.15, both pairs are unjudgeable at both temperatures. Tables 19 to 21 in Appendix P list the changes over the gates, aggregation rules and judgeable-share requirements.

Thanks for running the follow-up experiments. I like the idea of judging changes in the crossover against measured sampling noise. The four-shot replication on Minerva addresses my concern about using a chat template for the base model, and the new same-model controls give me more confidence in the threshold.

My remaining concern is the scope of the conclusion. The main comparisons use earlier-generation checkpoints up to 14B, mostly from one RLVR training recipe, and the benchmarks all focus on quantitative reasoning. How much of the observed sensitivity depends on the particular base model and RLVR checkpoint? The effect could be smaller, larger, or less consistent with newer or larger models, more recent benchmarks, or tasks with a different kind of verifier, such as code generation. I’d encourage you to narrow the abstract to what the experiments establish: evaluation choices can change the finite-budget verdict for the model pairs and tasks studied here.

I’ve spent a fair amount of time studying Qwen2.5 on math reasoning tasks, and I’m familiar with these benchmarks. I find the measurement idea valuable. What I’m still unsure about is whether the current evidence supports the broader contribution as stated.

The new evidence is enough for me to raise my score from 2 to 3 (Weak Accept). I’ll leave it to the AC to weigh the value of the measurement approach alongside the remaining uncertainty about how broadly the findings apply.

AuthorsOpenReview ↗

Scope of the conclusion

Thank you for the follow-up and for raising your score.

How much depends on the pair. Within the studied set, the paper measures this dependence directly. Section 4 (RQ3, Table 3) holds the protocol fixed and varies the pair over seven released SimpleRL-Zoo pairs, from 1.5B to 14B, on general, specialised and few-shot bases. Moving only the temperature from 0.6 to 1.0 changes the verdict for two of the seven pairs, Llama-3.1-8B and Mistral-7B-v0.1, and leaves it unchanged for five. Two further recipes appear in the appendices: Oat-Zero on Qwen2.5-Math-1.5B (Appendix J) and Open-Reasoner-Zero-7B on the 7B base (Appendix O). The procedure does not assume that the size of the effect transfers. Each comparison is judged against its own resampling noise, at a gate calibrated on a null control, so a new pair receives a verdict or UNJUDGEABLE under its own measured noise.

The abstract. We will narrow the abstract as you suggest: evaluation choices can change the finite-budget verdict for the model pairs and tasks studied here. The limitations will state the scope: released pairs up to 14B, mostly from one RLVR recipe, on quantitative reasoning with answer-matching verifiers.

AuthorsOpenReview ↗

Revision uploaded with the narrowed scope

Thank you again for the follow-up and for the concrete suggestion on scope.

The revised PDF now makes both changes:

  • Abstract. It now ends on what the experiments establish: for the model pairs and tasks studied here, evaluation choices can change the finite-budget verdict.
  • Section 7. It states the scope: the released pairs reach 14B parameters and mostly use one RLVR recipe, and the tasks test quantitative reasoning with answer-matching verifiers.

One small note: your comment mentions raising the score from 2 to 3, but the rating on your review still shows 2. In case the edit was not saved, we wanted to let you know.

Thank you for your time and for the constructive discussion.

Reviewer_RMBBOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies whether conclusions about RLVR extending a base model's reasoning boundary are sensitive to evaluation choices. It focuses on the first budget k* at which estimated base-model pass@k reaches the RLVR model's pass@k. For a fixed Qwen2.5-7B/SimpleRL-Zoo pair, the authors vary sampling, prompting, and scoring, and estimate crossover variability using repeated seeds and half-splits of sampled completions. The strongest example moves k* from 10 to 48 when temperature changes from 0.6 to 1.0. Nineteen of 30 standard-temperature comparisons exceed their reported noise floors, all from two Minerva chat-template cell pairs. The paper also proposes a bootstrap-based AHEAD/CAUGHT/UNJUDGEABLE procedure, calibrated using same-model null comparisons; across seven released pairs, two aggregate verdicts change with temperature. Its principal contribution is a measurement framework and empirical warning about finite-budget comparisons. These results do not establish whether RLVR expands true reasoning support.

Critical strengths
  1. A convincing controlled example of protocol dependence. The Minerva comparison holds the models, problems, scoring rule, and seed fixed while varying temperature. The 10-to-48 crossover shift is supported by a paired problem bootstrap, with a 95% ratio interval of [2.31, 9.80], and by consistency across scoring rules and added sampling seeds (Section 4, p.6; Appendices B-C). Thus the main finding is supported independently of the particular proposed noise-floor threshold. This is consequential because it directly demonstrates that a finite-budget comparison cannot be interpreted as a protocol-independent property of the model pair.

  2. Useful measurement discipline and reporting. The paper distinguishes finite-budget advantage from support expansion, makes censoring and seed variability explicit, and reports negative axes, truncation, and post hoc methodological changes. It also acknowledges that rule-level counts overstate replication and reports an inconclusive second-recipe test. These practices and the reporting recommendations are useful contributions to a debate in which single crossover estimates can otherwise be overinterpreted.

  3. Value to the community. The AI research community needs work that critically examines how evaluation and reporting choices shape scientific conclusions. I appreciate the authors’ initiative in bringing these issues into focus and encouraging more careful, transparent reporting. The paper’s potential to improve evaluation practices and help researchers interpret results more responsibly is an important strength that weighs in favor of acceptance.

Critical weaknesses
  1. The proposed reliability procedure has insufficient independent validation. The half-split floor measures noise at n/2 while judging movement at n; its conservativeness for a censored, nonlinear statistic is acknowledged rather than established. Appendix F reports that 8/30 comparisons obtain a floor from a wider group even though neither directly compared cell has a valid one. Separately, g=0.15 is selected after examining the same 45 null comparisons from three cells used to report the 2/45 false-AHEAD rate (Appendix M). This is useful empirical calibration, but independent validation and uncertainty estimates are needed to justify reuse across other protocols and models. Majority aggregation adds sensitivity: tightening the gate changes DeepSeek-Math's verdict after 24/36 judgments become unjudgeable. Since a reusable decision procedure is a central claimed contribution, these limitations materially affect my recommendation. They are not a rebuttal of the separately bootstrapped Minerva effect.

  2. The broader validation is substantially weaker than the main case study. All 19 floor-clearing main-grid temperature comparisons come from two Minerva chat-template cell pairs, a prompt setting the authors themselves regard as an evaluation error for an untrained base. Few-shot results do not provide two measurable crossover endpoints, and the second-recipe transfer test is inconclusive. The 14B evidence is useful corroboration, but several broader pair verdicts remain difficult to interpret: Mistral's RLVR truncation is 35.6-72.6%, versus 2.1-14.0% for its base, and the verdict procedure does not gate on this (Table 3). These facts do not erase the narrow protocol-sensitivity result; they limit the demonstrated reliability of the broader framework. In addition, p.6 and Appendix A infer too much from unbiased pass@k estimation: unbiasedness does not prevent estimated crossings from disappearing with additional samples, so the claimed definitive refutation of a budget explanation should be weakened.

Questions for authors
  1. Validation that could change my recommendation: With g=0.15 fixed before evaluation, is there an independent set of same-model null comparisons on which the false-AHEAD rate and its uncertainty can be reported? In particular, does calibration hold across different model pairs, prompt formats, and degrees of crossover censoring? Such evidence would directly address the first weakness.

  2. Clarification and sensitivity of the reported verdicts: How do the seven pair verdicts, especially the two temperature-induced changes, behave under the primary scoring rule, per-benchmark reporting, and a minimum requirement on the fraction of judgeable judgments? Which conclusions remain when severe truncation is addressed? Appendix M compares aggregation rules on the null; I am asking for the corresponding sensitivity of the substantive model-pair conclusions.

  3. Evidence and interpretation of the prompt-format result: Is there a noise-resolved temperature shift under appropriate base-model prompting with adequate generation length? Please also clarify the inference in Appendix A: the larger-budget observation is informative, but unbiasedness alone does not establish that the population curves never cross. A corrected interpretation or additional evidence could affect how broadly I view the validated contribution.

3 comments
AuthorsOpenReview ↗

Response to Reviewer RMBB

Thank you for the review. We ran the two experiments your questions ask for (Q1 and Q3). We registered both before any new generation, and Appendix P reports the registered read-outs. The answer to Q2 uses post-review analyses of the stored outputs (Appendix P).

Q1. Independent validation of the gate. We fixed g=0.15 and majority aggregation in the registration, then generated new same-model runs: 11 cells with five new seeds each, across four base models (Qwen2.5-7B, Qwen2.5-Math-7B, Llama-3.1-8B, DeepSeek-Math-7B), both prompt formats and three benchmarks. None of these runs is among the 45 comparisons used to choose g.

  • Of 220 ordered comparisons, 1 is AHEAD (0.5%), 177 are CAUGHT and 42 are unjudgeable.
  • Under the primary rule alone and under unanimity, 0 of 220 are AHEAD.
  • On the 22 prespecified disjoint seed pairs, 0 are AHEAD. The exact one-sided 95% upper bound is 12.7%, the smallest bound this design can give.
  • The single AHEAD is a Qwen2.5-7B four-example MATH-500 comparison. Every other model, format and benchmark stratum has 0. The two new Minerva cells, chat and four-example, have 0 of 40; all earlier false-AHEAD calls were on Minerva with the chat format.

The registered criterion (rate at most 5% and no disjoint AHEAD) is met. R2 did not vary the censoring level of a real model pair; the models, formats and benchmarks above are the range it covers.

On the half-split floor (post-review checks, Appendix P): on the three cells where both floors can be computed, the half-split floor is at least the seed-repeat floor for all 27 pairs of cell and rule (Table 26). Restricting the audit to the 22 comparisons with a floor measured on at least one of their own two cells keeps all 19 floor-clearing comparisons. The 8 comparisons whose two cells have no valid floor of their own are all on MATH-500, and none of them is among the 19.

Q2. Sensitivity of the seven pair verdicts. At g=0.15, the first two columns give the all-rule verdict on AIME 2025, MATH-500 and Minerva, in that order (C: base caught up within budget; A: RLVR still ahead; U: unjudgeable). The last column uses the case-study rule alone and a majority of judgeable benchmark labels.

PairT=0.6T=1.0Primary rule, 0.6 → 1.0
Qwen2.5-7BCCCACCC → U
DeepSeek-Math-7BUACCAAA → U
Llama-3.1-8BUCCUAAC → A
Qwen2.5-Math-7BACAAAAA → A
Mistral-7BCCCCAAC → U
Qwen2.5-14BCCCACCC → C
Qwen2.5-1.5BUAAAAAA → A

Llama and Mistral change from C to A under the all-rule vote at each of the seven gates we tested, from g=0.01 to g=0.25. At g=0.15, both changes persist when at least half of the 36 labels must be judgeable. At g=0.15 with a 0.75 requirement, or under unanimity, each of the two pairs is unjudgeable at one temperature or both, and neither changes between C and A. Tables 19 to 21 in Appendix P list the changes over the gates, aggregation rules and judgeable-share requirements. Per benchmark, all seven pairs change at least one label between the two temperatures, and six of them have a benchmark that changes from C to A. Under the primary rule alone, Llama keeps its C → A change. On truncation, Appendix P applies 13 screens, including the paper's own cross-temperature screen. Llama keeps its change under all 13. Mistral has a cell left under 2 of the 13 and keeps its change under both; the other 11 leave no Mistral cell.

Q3. Base-appropriate prompting. The registered replication uses Qwen2.5-7B and SimpleRL-Zoo with the same four-example prompt on both arms, four paired seeds, top-p 1, 128 samples per question and a 4,096-token generation cap.

On Minerva (272 questions), three seeds change from CAUGHT at T=0.6 to AHEAD at T=1.0, and the fourth changes from unjudgeable to AHEAD. This meets the first registered criterion, a verdict shift. The registered direction check also matches. Pooled base pass@1 falls from 8.69% to 5.49% and RLVR pass@1 from 9.50% to 7.09%, so the base loses the larger factor, which is the direction of the observed shift from CAUGHT to AHEAD. Truncation at this cap is at most 7.64% on Minerva. Appendix P.1 (Tables 14 and 15) reports both benchmarks and every registered test.

Appendix A. We removed the inference from unbiasedness. Appendix A now reports the crossover counts at both sample sizes.

Thank you for the response and for running the two registered experiments. The independent null set and the four-example replication are what I asked for, and the Appendix A fix is fine. I also appreciate that you reported the full sensitivity grid, including the rows that go against you.

I am raising my score to 4. I believe studies like this work help the community and enrich the literature.

A few things I would like fixed in the final version:

  1. Two of seven pairs. Only Llama's change looks robust. Mistral's is unjudgeable under the primary rule and survives 2 of 13 truncation screens, with its RLVR arm truncating 36–75% of samples. Please say this in the abstract.
  2. R1. The shift shows on Minerva only, and MATH-500 is inconclusive. Which scoring rule is Table 15, and which gate and aggregation is Table 14? Seed 2 at T=0.6 is CAUGHT in Table 14, but RLVR is ahead at every listed k in Table 15.
  3. Prompting. With the four-example prompt the RLVR arm's Minerva pass@1 at T=1.0 is 7.09%, against 22.6% with the chat template. Please state this as a limitation.
  4. R2. The Minerva chat-template stratum, where all earlier false-AHEAD calls were, has only 2 disjoint pairs, and the nulls are base arms at T=0.6 only. Please scope the calibration claim to that.

Minor: the reseeding claim should read 53 of 53 measurable out of 55 possible (Table 27), and Table 22 seems to screen the absolute truncation gap.

AuthorsOpenReview ↗

Tables 14 and 15, and the final version

Thank you for re-reading the revision and for raising your score.

Tables 14 and 15. The two tables use different scoring. Table 15 uses one scoring rule, the paper's case-study rule (boxed-then-marker extraction, exact string match). Table 14 uses the registered verdict procedure: for each of the 12 scoring rules, the gate g = 0.15 over 20,000 problem resamples, then the majority over the judgeable rules. Every verdict in Table 14 is the same under all three bootstrap RNG seeds. For Minerva seed 2 at T = 0.6, the case-study rule does not cross by k = 128; this is the +0.74-point gap in Table 15. Seven of the 12 rules cross before k = 128, at k from 32 to 100, and more than half of the judgeable rules read CAUGHT. We will name the rule in the Table 15 caption and the gate and aggregation in the Table 14 caption.

Final version. We will make the four changes and the two minor fixes you list: the abstract statement on the Llama-3.1-8B and Mistral-7B changes, the Minerva-only scope of R1, the four-example prompt as a limitation, the scope of the calibration claim to the strata R2 covers, the reseeding count (53 of 53 measurable replicate comparisons out of 55 possible, Table 27), and the Table 22 definitions, which do screen the absolute truncation gap.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.