acceptodds
Under review as a conference paper at ICLR 2027

Where Does RLVR Act? A Conserved Token Ledger for What Post-Training Adds to a Base Model

Abstract

Most token-level accounts of what reinforcement learning with verifiable rewards (RLVR) changes in a base model are statistics of the two policies' next-token distributions, which have no total in accuracy and cannot say what a token is worth. Prefix transplantation with one permanent handoff (the trained policy writes tokens, the base the rest) is a conserved ledger: by the performance-difference lemma its increments are the trained tokens' expected advantages under the base's own value function and sum to the measured gain; a dense check at the first four positions finds no over-count on any cell and a small pooled under-count (+0.014), as action-set truncation predicts. Across 13 cells (0.5B–8B), where front-loading is largest (0.73 of the gain in four tokens), the policy's most frequent opening, grafted into every problem, reproduces the whole early gain, and a better template on the base supplies about half (46–66%) of the early lift of every Qwen-Math RLVR descendant tested: the early credit is mostly format, relative to the base's template. GRPO on a random reward delivers 0.80 of its gain in four tokens, all of it format. Each advantage is a room, set by the token's surprisal under the base (its rate, in nats), times a worth, its value over the base's alternatives. Divergence statistics track the room; none predicts the worth's size beyond weakly (rank correlation at most 0.26 in magnitude). Code, raw records and scripts regenerating every table are provided as supplementary material.

Primary area:
reinforcement learning
Keywords:
post-training, rlvr, token-level attribution, performance difference lemma, prefix transplant, kl budget, base models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nznLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper studies performance gains from RLVR training, and their distribution along a generation trace. It introduces a prefix transplantation analysis in which the RL-trained model generates the first k tokens and the base model completes the remaining response. Using the performance-difference lemma, the resulting accuracy gains can be decomposed into token-level advantages under the base policy. Experiments across several model pairs suggest that early gains, especially for Qwen-Math models, can often be reproduced by generic opening prefixes or stronger prompting. The paper further studies how token-level divergence relates to estimated downstream value.

Critical strengths

The paper includes extensive experiments and ablations across different models and prompting conditions. The authors also report negative and inconclusive findings, and provide substantial supplementary materials for reproducibility.

Critical weaknesses
  1. Extremely poor presentation and organization. The paper is genuinely exhausting to read. There is no Conclusion section, and the Abstract and Introduction are overloaded with unnecessary details (e.g., "13 cells", "+0.014", "m=8", "m=32") before the experimental setup is even introduced. The text is often unnaturally compressed, with awkward sentences such as "whether it grades the trained token our nulls leave open" (Line 106). Many paragraphs lack a clear logical flow, making the main contributions and takeaways very difficult to identify.

  2. Limited novelty and unclear significance of the findings. The core mathematical framework is a direct application of the well-known performance-difference lemma [1,2]. Closely related token-level interventions and analyses of where RLVR changes model behavior have already been explored [3,4]. The central observation that RLVR gains often reflect elicitation of existing base-model capabilities, with strong sensitivity to prompting and even reward quality, is also well established [5,6]. I struggle to identify a sufficiently novel insight beyond these works.

The empirical findings are also heavily dependent on the choice of baseline. The strongest front-loading result comes from a post-hoc selected Qwen-Math setting, and much of the reported early gain disappears under stronger base-model prompts (Tables 43–45). Even the interpretation of "format" versus "content" changes substantially with the baseline. Moreover, the generic-prefix intervention does not cleanly isolate formatting effects, and the empirical conservation checks have limited statistical power. Although the authors acknowledge the strong dependence on the baseline prompt, I find that this substantially limits the significance of reported results. It remains unclear what generalizable insight the proposed "ledger" provides beyond the already established sensitivity of RLVR gains to prompting and elicitation.

Overall, the paper contains a substantial amount of analysis, but I find th scientific contribution rather limited and the presentation far below the expected standard.

  1. Kakade, S., and Langford, J. Approximately Optimal Approximate Reinforcement Learning. ICML, 2002.
  2. Setlur, A., et al. Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning. ICLR, 2025.
  3. Meng, H., et al. Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs. ICLR, 2026.
  4. Venhoff, C., et al. Base Models Know How to Reason, Thinking Models Learn When. ICML, 2026.
  5. Liu, Z., et al. Understanding R1-Zero-Like Training: A Critical Perspective. COLM, 2025.
  6. Shao, R., et al. Spurious Rewards: Rethinking Training Signals in RLVR. ICML, 2026.
Questions for authors
  1. What is the main new scientific insight enabled by the proposed ledger that could not be obtained from prior prefix transplantation or token-level intervention studies?
  2. Given the strong dependence on prompting and baseline selection, which of the main empirical conclusions should be considered general properties of RLVR, rather than artifacts of the specific experimental setup?
3 comments
AuthorsOpenReview ↗

Response to Reviewer nznL

We thank Reviewer nznL for a frank review. The criticism of the presentation was justified, and we rewrote the paper accordingly.

Presentation. The revised PDF rewrites the abstract and introduction in plain language without implementation details, defines each coined term where it first appears, states the contribution against prior work in one paragraph, reorganises the results into seven subsections with one claim each, moves the key tables into the main text and the frontier proposition into Appendix B, and adds a Conclusion. Compressed sentences were rewritten; the one you quote now reads: "Our nulls cannot tell whether dispersion singles out the trained token or only its position."

Q1. What new insight does the ledger enable?

Token-level studies (Meng et al.; Akgül et al.; Venhoff et al.) locate where the trained and base models differ, or how much accuracy an intervention recovers. The ledger prices how much accuracy each trained token buys, in units that add up to the gain, so a localisation claim can be checked against controls in the same units. Three results follow that a divergence statistic cannot give:

  1. Rate concentration does not determine gain concentration. The first four tokens carry about 11% of the trace's rate (surprisal under the base) on both Qwen2.5-Math-7B/GSM8K and Llama-3.1-8B/GSM8K, yet deliver 73% and 1% of the gain (Table 27). On Qwen2.5-7B with SimpleRL-Zoo and Open-Reasoner-Zero (checkpoint families Akgül et al. analyse; Venhoff et al. also study Open-Reasoner-Zero-7B), ρ(4) is 0.09 and 0.07.
  2. Strengthening the base prompt removes most early credit while leaving a measurable residual gap. On the 32B SimpleRL-Zoo checkpoint that Meng et al. analyse, four trained tokens buy +0.163 against the base under the shared template and −0.059 against the base with no template, on the same problems (Table 2); on every front-loaded cell re-scored that way, the largest point estimate is 0.048. On the correct-reward descendants of Qwen2.5-Math-7B, the residual gap against the no-template base is dominated by the within-mode accuracy part of a descriptive decomposition (Table 60), and the same GRPO recipe keeps 0.141 with the correct reward and 0.021 with a random one (§4.4).
  3. Divergence statistics measure room, not worth. Proposition 2 shows that no statistic of the two next-token distributions determines the worth; empirically |r_s| ≤ 0.24 against the worth factor, and under a single decoding policy at m=32 rate and the base's entropy stay within −0.21 to −0.02 (§4.6).

Whole-trace studies (Liu et al.; Shao et al.) show that templates and rewards shape the total gain. The ledger shows which tokens carry the template-dependent part, and that it is the part a random reward also buys. The identity is classical (Kakade & Langford; Setlur et al.); we claim a measurement framework, not a new lemma.

Q2. Which conclusions are general, and which are setup-specific?

The revised Conclusion separates them.

  • Held on every pair we measured, now including 14B, 32B and multiple-choice QA: the identity; its adding up at the first four positions, and over the first 32 on the five cells checked (§4.1); divergence statistics tracking room, not worth.
  • Setup-specific, and reported as such: how early the gain sits, which depends on the base and on the baseline's prompt, and what the opening controls say about mechanism, which differs between GSM8K, MATH-500 and the 32B pair.

A token-level claim about "what RLVR adds" is incomplete without its baseline; the ledger measures how much the baseline matters.

Other points.

  • Post-hoc primary cell. Every cell's curve is reported (Figure 2; Tables 12–13; new pairs in Table 3). The primary cell only determines where the controls run deepest, and we say that it was chosen after the curves were measured. The 14B and 32B pairs were chosen before any of their results were known.
  • The generic prefix does not isolate formatting. Agreed. New §4.3 grafts seven fixed openings and splits each lift into the response's mode and its quality. On the GSM8K cells of the Qwen2.5-Math-7B descendants, neither the trained model's word order nor "Let's think step by step." is needed to recover most of the early lift, and most of it is the base no longer leaving the problem; on MATH-500 and at 32B part of it is quality, so we state the mechanism per pair.
  • Power of the conservation check. It now runs over the first 32 positions on 5 cells, with intervals (Tables 61–62), and detects over-counts of 0.06–0.10 when the per-position weights depart from the sampling policy.

Thank you again for your review. Before the discussion period closes on 18 November, we wanted to check whether the revision addresses your concerns. The paper was rewritten for readability (a plain-language abstract and introduction, one claim per results subsection, and a Conclusion), and our reply answers your two questions directly: what the ledger shows that divergence statistics or intervention studies cannot, with the 32B checkpoint of Meng et al. as an example, and which conclusions held on every pair versus which depend on the setup. We would welcome any further questions, and would be grateful if you would consider whether the revision changes your assessment.

I thank the authors for their response and the substantial revision. The clearer presentation and additional controls address some of my concerns. I have raised Presentation and Informativeness to 2, but am inclined to maintain my original rating, as I still have reservations about the incremental contribution and the strong dependence of the findings on the baseline.

Reviewer_9mVrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work try to decompose post-training accuracy gains into token-wise value contributions relative to the base model to re-examine the origins of RLVR improvements, and identifies a counterintuitive finding: for some Qwen-Math models, RLVR benefits seemingly concentrated on early tokens can be largely recovered through fixed answer prefixes, prompt templates, or few-shot prompting. The authors further emphasize three rigorously separable claims: token-level credit assignment with additivity and conservation can be mathematically constructed from a base value function; experimental evidence strongly supports that early-token gains are substantially reproducible with fixed prefixes under specific Qwen-Math setups; and two broader conjectures — that RLVR essentially learns formatting instead of new reasoning ability, or that divergence-based methods generally cannot pinpoint truly important tokens — remain insufficiently validated.

Critical strengths

S1. The research question is well chosen. The paper identifies a genuine issue in token-level RLVR analysis: distributional difference does not equate to performance contribution. Compared with analyses based solely on entropy, KL divergence or attention patterns, this work establishes a more direct link to downstream task performance.

S2. The theoretical formulation is clearly defined. Rather than claiming to propose a novel Performance Difference Lemma, the authors adapt this classical result into a practical measurement framework. Its core feature lies in additive, signed, accuracy-valued attribution.

S3. The experimental controls are more comprehensive than those in typical mechanistic analysis. Beyond the native RLVR prefix, the authors conduct comparisons across generic prefixes, foreign prefixes, base-own openings, prompt templates, few-shot settings, random-reward GRPO, and reverse handoff. These controls enable the paper to rule out several simplistic explanations proactively, instead of merely observing correlations post hoc.

Critical weaknesses

W1: Attributing the effect of the fixed opening to “format” suffers from insufficient causal identification. This issue is critical and undermines one of the paper’s central empirical claims. The problematic claims appear in the Abstract (p.1, lines 20–25), Section 4.2 (p.6–7, lines 314–346), and Section 4.2 (p.7, lines 347–367). The authors observe that inserting the RLVR model’s frequent, problem-agnostic fixed opening into the base model recovers a large fraction of the top-four token performance gain. However, a generic prefix may carry not only formatting signals but also reasoning mode triggers, implicit task instructions, targeted subsequent generation distributions, and reasoning patterns already learned during the base model’s pretraining. The experiment only demonstrates that Generic prefix can reproduce early gain, not that Early gain is caused by formatting. Notably, the authors themselves acknowledge in Section 3 (p.5, lines 232–236) that the generic prefix is not entirely free of problem-type content. This flaw is severe because the claim represents one of the paper’s most prominent empirical takeaways, yet the authors lack experiments that independently manipulate format and reasoning instructions. As such, “format” serves merely as an interpretation of the observed results rather than a rigorously identified mechanism. A more conservative phrasing would state that substantial early gain can be reproduced by fixed, problem-agnostic prefixes, while the underlying mechanism remains undetermined.

W2: The base model in the main experiments uses a template that disadvantages itself, which may inflate the RLVR performance gap. Relevant discussion appears in Section 3 (p.4, lines 210–215), Section 4.2 (p.7, lines 351–367), and Section 6 (p.9, lines 471–477). The authors observe the base model prefers Question/Answer formatting whereas the trained model favors chat templates. Section 6 shows the raw ~0.398 performance gap falls to 0.049, 0.234, or 0.119 when the base’s template is adjusted. The main experiment computes J(πr;τr)−J(πb;τr)J(\pi_r;\tau_r)-J(\pi_b;\tau_r), instead of evaluating both models under their individually optimized inference settings. Since taurtau_r is selected for the trained model, this comparison systematically handicaps the base. Fixed-template comparison is methodologically valid and does not invalidate results, yet interpreting this gap as post-training-induced reasoning capability constitutes overreach. This artifact distorts overall gains and the central metric ρ(4)\rho(4) (the early-token gain fraction), as baselines change denominators and prefix effects. The safer conclusion: RLVR gains are strongly template-sensitive, and the gap measured under a base-disadvantaging template cannot represent RLVR’s full standalone benefit.

W3: Token-level value estimation lacks sufficient precision: only 8 rollouts per action and truncated action space induce non-negligible bias and selection noise. With uneliminable 10−310^{-3} estimation error comparable to search margins and weak conservation checks, the empirical A^t\hat A_t cannot reliably support fine-grained worth/efficiency conclusions in Sec 4.3–4.4; results are exploratory only, not evidence of theoretical defects.

Questions for authors

First, can the authors provide experimental evidence demonstrating that the gains from the generic opening stem from format rather than reasoning-mode elicitation? This directly challenges the paper’s most important mechanistic explanation.

Second, can the authors validate the token-level conclusions in Sections 4.3–4.4 under a unified decoding policy and with higher rollout fidelity? This is critical for verifying whether the paper’s new measurement framework can reliably yield fine-grained empirical findings.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 9mVr (1/2)

We thank Reviewer 9mVr for a detailed and fair review. Your summary of what the paper does and does not establish is accurate, and we revised along the lines you suggest.

W1 / Q1. Is the generic opening's effect format, or reasoning-mode elicitation?

We agree that the generic-prefix result shows that a fixed, problem-agnostic opening reproduces the early gain, not why. We adopted your phrasing: §4.2 is now "Where front-loading is largest, a problem-agnostic opening reproduces the early gain", and the paper no longer calls the early credit "format".

To address the question we added an experiment that varies the candidate mechanisms separately (new §4.3, Table 1; all 18 cells in Table 58). The base continues m=8 times per problem from seven fixed openings of three to seven tokens:

openingintended comparison
nonebaseline R(0)
policy's most frequent openingthe paper's generic arm
base's own most frequent openingplacebo
paraphrase of the policy's openingwording vs. meaning
policy's tokens in reverse orderthe same words, no word order
"Let's think step by step."explicit reasoning instruction
"The answer is"encourages an immediate answer

An LLM judge (GPT-5.4-mini) labels a uniform sample of completions as an on-task derivation, a bare answer, off-task text, or unfinished; a second judge from another family (Claude Sonnet 4.6) agrees on 93% (κ = 0.81). On that sample each lift splits exactly into a mode part (how often the base writes each kind of response) and a quality part (how accurate each kind is). The split is descriptive, not a causal mediation analysis, and either mechanism could move both parts, so we read it together with the controls.

Five RLVR descendants of Qwen2.5-Math-7B, GSM8K (Tables 1, 58, 59):

  • Openings that start a written solution buy about the same lift: the policy's +0.206 to +0.262, the base's own +0.196 to +0.251, "Let's think step by step." +0.198 to +0.246, the policy's tokens reversed +0.218 to +0.258. A paraphrase buys less (+0.141 to +0.221). On four of the five cells the policy's most frequent opening is the base's own ("To solve this problem"), so the placebo separates them only on the primary cell (+0.247 for the base's, +0.262 for the policy's).
  • With no opening the base leaves the problem in 24–34% of samples (another exercise, a dialogue turn); after the policy's opening, in 0–1%. Mode carries most of the lift on four of the five cells (+0.175 to +0.228); on the primary cell mode is +0.106 and quality +0.142, an interval that includes zero. For the reasoning instruction, mode is +0.158 to +0.234 and quality −0.051 to +0.125 (resolved from zero on 1 of 5 cells).
  • Openings that do not start a written solution fail: "The answer is" costs −0.144 to −0.091, and the distilled model's "Okay, so I" buys +0.009, leaving the base off-task in 18% of samples.

The pattern is not universal. On MATH-500, where the base leaves the problem less often (11%), part of the lift is quality (+0.152 [+0.029, +0.241] for the policy's opening, +0.092 [+0.024, +0.159] for the reasoning instruction). On the 32B pair the policy opens with the base's own "Step 1:", which buys +0.131, with a point estimate of more than half through quality (+0.069 [−0.012, +0.146]), while the reversed tokens and the reasoning instruction hurt (−0.106, −0.189).

So we scope the conclusion. On the GSM8K cells, neither the trained model's word order nor an explicit reasoning instruction is needed to recover most of the early lift, and the drop in off-task continuations points to a large response-initiation component: the base committing to write a solution rather than continue the prompt. The controls do not exclude an implicit elicitation of reasoning, which the quality parts on MATH-500 and at 32B suggest. §4.3 and §4.5 state this per pair.

AuthorsOpenReview ↗

Response to Reviewer 9mVr (2/2)

W2. The base's template disadvantages it and inflates the gap and ρ(4).

We agree. New §4.4 opens by stating that the gap under a shared template is not the trained model's standalone benefit, and the paper no longer reads it as reasoning capability added by post-training.

New Table 2 re-scores each base under the best-performing of the prompts evaluated (the pair's template, Question/Answer, none; the first and last on MATH-500), chosen on the problems scored, so the comparison is descriptive. It recomputes the gap and the four-token lift against that baseline on 17 cells:

  • The pair's template is the best-performing prompt on only 1 of them.
  • Primary cell: the base with no template scores 0.703 instead of 0.424; the gap falls from 0.388 to 0.109, and four trained tokens add −0.003. (Appendix F's 0.398 → 0.119 is computed on the traces longer than 128 tokens, as ρ(128) needs; Table 2's gap is the two models' accuracy difference on the 99 problems every prompt shares. The distilled model's template, not among those evaluated in Table 2, is better still: gap 0.049, App. D.)
  • On every cell that front-loads under the shared template and was re-scored with no template, the largest point estimate of the four-token lift is 0.048, including MATH-500 and the new pairs (32B: base 0.898 instead of 0.605, gap 0.344 → 0.051, four tokens −0.059).

Against the no-template baseline a residual gap remains. The ledger places little of it in the first four tokens: on the primary cell ρ(4) ≈ 0 and the curve reaches only 0.43 by k=128 (App. F). The response-mode analysis attributes most of it to the within-mode accuracy (quality) part (Table 60): under rule-based labels on every sample, at least 95% of each of the four correct-reward descendants' gaps (0.109–0.186); on the two also judged, 89% or more of the judged-sample gaps (0.158, 0.204). On the primary cell mode is −0.023 and quality +0.131 [+0.091, +0.171], and on-task derivations are +0.126 [+0.083, +0.165] more accurate on the same problems (judge: +0.164 [+0.111, +0.224]). The same GRPO recipe keeps 0.141 with the correct reward and 0.021 [−0.023, +0.067] with a random one, whose positive quality part (+0.043) is partly offset by a negative mode part. The template's own lift splits less robustly: Question/Answer's +0.176 is mostly mode under the judge (+0.154 mode, +0.022 quality) but +0.063 mode and +0.100 quality under the rule-based labels; §4.4 and App. H.1 report both.

So your safer conclusion is the one we now state: RLVR gains measured under a shared template are strongly template-sensitive. The ledger adds which part is which: the early credit, which a stronger prompt largely removes and a random reward also produces, and a residual gap against that prompt that sits later in the trace, is mostly within-mode accuracy, and is much smaller under a random reward.

W3 / Q2. Token-level precision; a unified decoding policy and higher fidelity.

Agreed. §4.6–4.7 (formerly §4.3–4.4) are now labelled exploratory. We re-ran the whole-trace ledger on 8 cells with the base sampling at temperature 1 without nucleus (so weights and values belong to one policy), m=32 rollouts per action instead of 8, and the base's top six tokens plus the realised one instead of four. The conclusions hold (Table 63):

  • Room. The rate tracks |A_t| (r_s = 0.72–0.89).
  • Worth. Rate and the base's entropy correlate with the worth's magnitude at only −0.21 to −0.02. (The two-policy divergences were not re-run under this policy.)
  • Value dispersion. The base's dispersion predicts worth beyond a shared-rate null by +0.141 to +0.294, above all 300 null draws on every run (p = 1/301).
  • Wasted rate. 52–61%; it falls as the estimator sharpens, so we report it as an estimate, not a bound.
  • Search margin. The cross-fitted upper end is within its 95% interval of zero on 7 of 8 runs and below zero on the other; §4.7 calls this "a statement of resolution, not of equivalence".

Thank you again for the detailed review. As the discussion period closes on 18 November, we wanted to check whether our responses address your concerns. In brief: the new opening arms (§4.3, Table 1) separate response initiation from an explicit reasoning instruction, and the mechanism is now stated per pair; each base is re-scored under the best-performing of the prompts evaluated (§4.4, Table 2), with the residual gap split into mode and quality in Table 60; and §4.6–4.7 are re-run under a single decoding policy at m=32 (Table 63) and labelled exploratory. If anything remains unclear or unconvincing, we would be glad to clarify before the deadline, and we would be grateful if you would consider whether the revision changes your assessment.

Reviewer_beyDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies what RLVR actually adds to a base model. Many previous works said the change concentrates in the first few tokens based on statistics between the models' next-token logits. which cannot say how much accuracy each token is worth. The authors use prefix transplantation, first use trained model predicts the first k tokens, base model predicts rests to finish, and the first k tokens' contributions sum to the accuracy gain. Massive experiments shows that

  1. the ledger conserves;
  2. In Qwen-Math series, the first four tokens' gain is almost all answering format
  3. divergence statistics show where things changed but not what the change is worth
Critical strengths

1.Solid experiments and ablation. Lots of model pairs. In ablation, GRPO random reward results in 80% front loading, and all are formats, the result is insteresting.

2.Propose a new method to measure the token's value with very good reproducibility.

Critical weaknesses

The conclusion is tied to the QwenMath family and math tasks, and models <= 8B, whether it holds on other base models and tasks?

Questions for authors

1.Can the conservation check go beyond 4?

2.On other task like IF\QA etc, or on larger models, does the ledger still conserve, and is front-loading still mostly format?

3 comments
AuthorsOpenReview ↗

Response to Reviewer beyD

We thank Reviewer beyD for the positive assessment and for two questions that led directly to new experiments. Both are answered in the revised PDF.

Q1. Can the conservation check go beyond 4 tokens?

Yes. We extended the dense ledger to each of the first 32 positions on 5 cells where the gain keeps accruing after k=4: QM7B/GSM8K, QM7B/MATH-500, and the Qwen2.5-0.5B, Qwen2.5-1.5B and Llama-3.1-8B pairs on GSM8K. At each position we measure the advantage directly (m=8 rollouts per action) and compare the summed advantages with R(k)−R(0) from the transplant curve on the same traces, at k=4, 16 and 32 (§4.1; Tables 61–62 in App. H).

  • With restricted-action reweighting toward the rollout sampler (temperature 0.8 and nucleus 0.95 applied within the measured actions, an approximation to the full-vocabulary sampler), every cell's shortfall interval overlaps the truncation band at every depth (Table 62). On the 1.5B cell this requires scoring the base's top 12 tokens rather than its top 4, because its first token leaves half of its probability outside the top 4. Examples at k=32 (shortfall, 95% interval): QM7B/GSM8K +0.088 [+0.009, +0.168] against a band of [0, 0.58]; Llama-8B −0.058 [−0.133, +0.020]; 1.5B with top 12 +0.021 [−0.060, +0.099].
  • With the main runs' weights (the base's temperature-1 probabilities), three of the five cells overlap the band at every depth (Table 61). The two flattest bases, 1.5B and Llama-8B, over-count at one depth, by 0.065 and 0.099. Widening Llama's action set to 12 tokens does not remove the discrepancy; restricted-action reweighting reduces it.

The revised §4.1 adds two caveats: the reweighting is a sensitivity diagnostic rather than an exact matched-policy check, and beyond k=4 the truncation band widens faster than the lift (T = 0.58 against a lift of 0.27 at k=32 on the primary cell), so at k=16–32 the check guards against over-counting, not under-counting. The pooled two-sided check at k=4 is unchanged: an under-count of +0.014 [+0.002, +0.025] over 13 cells, as action-set truncation predicts.

Q2 / Weakness. Other tasks and larger models: does the ledger still conserve, and is front-loading still mostly format?

We added 6 cells (new §4.5, Table 3) and one further pair:

  • Larger models: SimpleRL-Zoo on Qwen2.5-14B and Qwen2.5-32B, on GSM8K and MATH-500. The 32B checkpoint is one that Meng et al. (2026) analyse with token-level divergence statistics.
  • Multiple-choice QA: General-Reasoner, zero-RL from Qwen2.5-7B and Qwen2.5-14B, scored on the 13 non-math categories of MMLU-Pro.
  • Instruction following: Tulu 3 8B, from its DPO checkpoint to its RLVR checkpoint, on IFEval.

Findings:

  • Conservation. At k≤4 the dense check is consistent on all six new cells.
  • How early the gain sits depends on the base. On GSM8K the 14B and 32B pairs front-load (ρ(4) = 0.48 for both) while Qwen2.5-7B under the same recipe does not (0.09); on MATH-500 the three are alike (0.28–0.29). General-Reasoner front-loads little on MMLU-Pro (0.14 at 7B, 0.09 at 14B).
  • A fixed opening carries much of the early lift. The trained model's single most frequent opening, grafted into every problem, buys +0.131 of the 32B pair's +0.154 on GSM8K, +0.108 and +0.096 of +0.094 and +0.098 on MATH-500 (14B, 32B), and +0.051 of the 14B pair's +0.119 on GSM8K (Tables 1, 58). Pooled with the next two openings, as in the main protocol, it buys less on three of the four cells, because those openings buy little or hurt (Table 3).
  • The early credit depends on the baseline's prompt. With no template, the best-performing of the prompts evaluated for both bases, the 14B and 32B bases score 0.876 and 0.898 on GSM8K instead of 0.710 and 0.605; the gap shrinks to 0.073 and 0.051, and the first four trained tokens add −0.009 and −0.059 (MATH-500: +0.021 and −0.001). On every front-loaded cell re-scored this way, the largest point estimate of what four trained tokens buy is 0.048 (§4.4, Table 2).
  • Instruction following. RLVR adds only +0.028 [+0.012, +0.045] to the strict prompt-level IFEval accuracy of Tulu 3's DPO checkpoint, too small to decompose, so we report it as such.

On scope: the identity holds exactly for any pair of policies. What depends on the pairing is where the gain sits; the new runs show that this depends on the base and on the baseline's prompt, and the revised Conclusion separates what held on every pair from what depends on the setup.

Thank you again for the supportive review. We hope the new runs answer both of your questions: the conservation check over the first 32 positions (§4.1, Tables 61–62), and the 14B and 32B, multiple-choice QA and instruction-following pairs (§4.5, Table 3). If anything is unclear, we are happy to clarify before the discussion period closes on 18 November.

Thank you for the detailed response and additional experiments. Extending the conservation checks to 32 tokens and including 14B/32B models and non-math QA tasks substantially addresses my concerns about experimental coverage. I decide to retain my initial positive rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: summary of the revision

We thank the three reviewers for careful and constructive reviews. They asked for four things: a readable paper (nznL, 9mVr); generality beyond Qwen-Math, mathematics and 8B (beyD, nznL); a test of whether the generic-prefix result is format or reasoning-mode elicitation (9mVr, nznL); and token-level estimates under one decoding policy at higher resolution (9mVr). Changes are in blue. New runs follow Section 3 unless stated; their scripts, records and result files are in the updated supplement.

1. Presentation. Abstract and introduction rewritten in plain language; coined terms defined at first use; results in seven subsections, one claim each; key tables in the main text; the frontier proposition moved to Appendix B; a Conclusion that separates what holds on every pair from what depends on the setup.

2. Generality (§4.5, Table 3). SimpleRL-Zoo on Qwen2.5-14B and -32B (GSM8K, MATH-500): on GSM8K both front-load (ρ(4) = 0.48) while Qwen2.5-7B under the same recipe does not (0.09). General-Reasoner on the non-math categories of MMLU-Pro: little front-loading (0.14, 0.09). Tulu 3 8B, DPO → RLVR on IFEval: +0.028 [+0.012, +0.045] strict prompt-level accuracy, too small to decompose.

3. Conservation over 32 positions (§4.1, Tables 61–62). On 5 cells, with the measured actions reweighted toward the rollout sampler (an approximation to the full-vocabulary sampler), every shortfall interval overlaps the truncation band at k = 4, 16 and 32 (the 1.5B base with its top 12 tokens scored). Beyond k = 4 the check guards against over-counting.

4. Opening vs. reasoning (§4.3, Table 1). The base continues from seven fixed openings; an LLM judge labels each response, and each lift splits descriptively into mode and quality. On GSM8K neither the trained model's word order nor an explicit reasoning instruction is needed for most of the early lift, and most of it is the base no longer leaving the problem; on MATH-500 and at 32B part of it is quality.

5. Baseline prompt (§4.4, Table 2). Every base is re-scored under the best-performing of the prompts evaluated. On every front-loaded cell re-scored with no template, the largest point estimate of the four-token lift is 0.048. Against that base, the remaining gaps of the four correct-reward Qwen2.5-Math-7B descendants (0.109–0.186) are dominated by the quality part (Table 60), and the same GRPO recipe keeps 0.141 with the correct reward and 0.021 with a random one.

6. One decoding policy, m = 32 (§4.6–4.7, Table 63, 8 cells). The rate tracks |A_t| (r_s = 0.72–0.89); rate and the base's entropy correlate with the worth's magnitude at −0.21 to −0.02; the base's value dispersion beats a shared-rate null by +0.141 to +0.294 (above all 300 null draws on every run, p = 1/301); no run resolves a positive search margin. §4.6–4.7 are labelled exploratory.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.