acceptodds
Under review as a conference paper at ICLR 2027

Accepted but Unfinished: The Salvage Value of Truncated Rollouts in RLVR

Abstract

A policy trained with verifiable rewards can learn to place an answer its verifier accepts and then keep generating until the response cap. The reason is the default scoring path: the four open RLVR stacks we inspected grade whatever text is retained at the cap, so a truncated rollout is not scored zero but earns its salvage value, the mean reward paid to rollouts that hit the cap. This one unreported quantity governs what the policy learns, what the pass rate measures, and what the update does. Learning: in one default-grader run all 848 easy-stratum rollouts are accepted while 11 complete; in another, completed-and-correct output falls from 0.70 to 0.015 in a single update while two thirds of truncated outputs still earn full credit. Measurement: salvage value is exactly recoverable from three logged scalars, verified per difficulty stratum across five policy families; since it enters the pass rate as , the same policy is ranked differently at different caps without training (44.7 pp swing, rank correlation 0.60). Update: salvage is a distinct credit term, the covariance of reward with policy score within truncated rollouts; a preregistered matched fork that pins truncated reward at the checkpoint's own salvage value finds the default-grader arm truncating more in four of six seeds. The published treatments of overlong outputs act on different terms of the same decomposition: a fixed high reward costs completion in all five seeds, masking costs accuracy in nine of twelve matched cells, and neither substitutes for the other. Salvage value is the coordinate along which acceptance and completion separate; it is readable from existing logs and belongs next to the pass rate.

Primary area:
reinforcement learning
Keywords:
RLVR, truncation, reward design, response length budget, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cenoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how reinforcement learning with verifiable rewards handles responses that reach the generation limit without terminating. It defines “salvage value” as the mean reward assigned to truncated responses and shows how this quantity relates verifier acceptance, completion, and reported reward. It also decomposes an idealized policy gradient into completed-answer credit, truncation-status credit, and reward-dependent credit within truncated responses.

The empirical study includes budget sweeps, examples of accepted answers followed by continued generation, fixed truncation-reward interventions, token-loss masking, and matched checkpoint forks. The principal finding is that high verifier acceptance can coexist with very low completed-and-correct accuracy. The paper recommends reporting these outcomes separately and specifying truncation rewards, loss masks, and evaluation budgets explicitly.

Critical strengths

The paper documents a practically important discrepancy between accepted answers and completed responses. The reported cases are striking: in one training batch, all 848 easy-stratum responses are accepted, while only 11 complete. Reloaded-checkpoint probes provide additional evidence that answers can appear well before generation stops at the cap. This motivates reporting completion and acceptance separately and makes the contribution more concrete than an abstract observation about reward specification.

The analysis usefully distinguishes interventions that are often grouped together. Assigning a constant reward to truncated outputs, masking their token losses, retaining their rewards in the group baseline, and changing the generation budget are different operations. The decomposition clarifies these distinctions. The authors also report contrary results, failed predictions, and limitations, including the absence of saturation in the later default-grader cohort. This transparency helps establish the scope of the findings.

Critical weaknesses

The matched-fork experiment does not isolate the claimed within-truncation mechanism. Appendix S15.3 replaces truncated rewards with a historical, batch-level constant. This removes within-truncation reward variation, but it does not preserve each prompt’s conditional mean reward, even initially, and does not preserve the evolving status contrast during subsequent updates. The appendix explicitly acknowledges the additional term (b−bset)∇τ(b-b_{\mathrm{set}})\nabla\tau, which conflicts with the statement that the design isolates the proposed term “and nothing else.” Moreover, the predicted truncation direction appears in only four of six seeds, the reported interval includes zero, and strict-accuracy effects split three versus three. These results support an exploratory intervention comparison, but do not establish that the within-truncation covariance drives increased truncation.

The headline intervention findings have limited robustness across configurations. Nine of the twelve masking comparisons use Qwen3.5-4B-Base. The policy-equal estimate reported in Appendix S16 is −2.8 percentage points, with a 95% interval of [−7.7, +2.2], providing weaker support for a general masking penalty than the pooled twelve-cell interval suggests. The two terminal default-grader collapses also occurred under earlier launcher revisions, while none of the fourteen later default runs saturated. The authors acknowledge these limitations, but the broad explanatory framing remains stronger than the evidence. Because the central accounting identities are elementary, the contribution depends particularly on robust empirical evidence and careful causal interpretation.

Questions for authors

Can you separate removal of within-truncation reward variation from changes in the truncation-versus-completion reward contrast? For example, a same-checkpoint, per-prompt gradient comparison on identical sampled trajectories could assess the proposed component without relying on an eight-update intervention with drifting reward means.

How should the four-of-six fork result and its interval containing zero be interpreted? Would you revise the causal claim to an inconclusive result, or is there additional evidence directly connecting the within-truncation gradient component to subsequent termination behavior?

What conclusion about masking survives policy-equal weighting and a consistently defined inclusion rule across all completed comparisons? Please distinguish a descriptive average over the selected cells from an effect expected across policy families.

5 comments
AuthorsOpenReview ↗

Same-trajectory controls with prompt means held fixed (1/3)

Thank you for your careful review and the concrete suggestion for separating the reward components. We implemented the same-checkpoint, per-prompt comparison you proposed and replicated it at a second checkpoint. On identical trajectories, the controls hold the sampled reward-mean contrast fixed and separate opposing local stopping contributions. Full-gradient identities, finite parameter interventions and half-dose checks connect the algebra to measured parameter effects (Section 6; S31).

1. The requested identical-trajectory intervention

Let rr be actual reward, uu the score through the last complete box, and brb_r and bub_u their truncated means within the current prompt. TT is the truncation indicator, and τg\tau_g is the truncated fraction in that prompt group. Completed rewards are unchanged. D/F/R assign rr, brb_r and br+u−bub_r+u-b_u to truncated outputs; P assigns r−0.25r-0.25.

Let AA denote the group-centered advantage. With group-std normalization disabled and a common response-token-mean score Jacobian:

AD−AF=T(r−br)A_D-A_F = T(r-b_r)

AD−AR=T[(r−br)−(u−bu)]A_D-A_R = T[(r-b_r)-(u-b_u)]

AR−AF=T(u−bu)A_R-A_F = T(u-b_u)

D/F/R preserve the sampled prompt mean and completion/truncation mean contrast. Each first-step comparison shares trajectories within its seed; subsequent branches recompute means on their own current batches.

Each checkpoint supplies 1,024 fresh trajectories and frozen correct dev prefixes (230/240 questions). Common-step pure SGD gives EOS log-probability contrasts:

Contrasts47 measured / predicteds56 measured / predicted
D−FD-F-.001043 / -.001044-.004310 / -.004581
D−RD-R-.001593 / -.001588-.013531 / -.013763
R−FR-F+.000550 / +.000544+.009221 / +.009181

Predictions use actual FP32 parameter differences. Full-gradient residuals are 2.39×10−62.39\times 10^{-6} and 2.66×10−72.66\times 10^{-7}; repeated D is exact. Half the internal component gives 50.31%/48.98% of its full effect. All four D/F/R/P task steps raise the readout; the contrasts identify their relative contributions.

2. Controlling the entire reward distribution, then isolating the mean change

Four fixed permutations preserve each prompt's entire truncated reward multiset, completed rewards and baseline. Their S-F effects are +.000589/+.001411/+.000594/-.001049. All S arms share D's mean and variance; F is their common reference. Credit-to-trajectory assignment therefore changes the local response with those reward statistics held fixed.

Direct complete-box scoring U differs from R by the following advantage term:

AU−AR=(bu−br)(T−τg)A_U-A_R = (b_u-b_r)(T-\tau_g)

The full-gradient residual is 9.02×10−89.02\times 10^{-8}. Measured U-R EOS difference is -.042861 [question interval -.048186,-.037776], versus prediction -.042360. Its status projection, expressed per unit step before multiplication by the learning rate, has a 64-training-prompt interval [-17.9666,-3.0212]. This isolates the extra reward-level effect introduced by direct parser repair.

The reference sequence-sum, temperature-1 score repeats all four D/F/R/P signs on the retained batch. S31.10 reports both score conventions and inference units: dev-question intervals condition on the intervention; training-prompt intervals span zero for the internal projections. The finite effects are identified for the fixed checkpoint, batch and SGD step.

3. Historical fork attribution

We corrected "isolates that term and nothing else": the old global constant changes both internal credit and the status contrast. Its six-seed truncation difference is +1.416 pp [-3.687,+6.519], positive in 4/6, with completed-correct signs split 3/3. It measures the combined change in these two terms. The new per-prompt experiment supplies the component identification requested here. It preserves the sampled contrast exactly, and the fixed-multiset control also preserves variance across assignments. Our second response connects these local interventions to actual optimizer and training outcomes.

AuthorsOpenReview ↗

Local effects, Adam updates, and training through H64 (2/3)

The new evidence follows the component through three measured stages: a controlled parameter intervention, the actual optimizer step, and subsequent generation. This makes the connection to termination explicit at each stage. We added eight-seed mathematical training, both prespecified continuations through H64, and four-seed executable-code training (Section 6; S31.6 and S32).

1. Actual Adam and eight-seed outcomes

R-F's raw stopping-gradient projection is positive in 6/8 first batches; its actual first-Adam task-step projection is negative in 8/8. Analytical and actual projections agree within 8.3×10−78.3\times 10^{-7} absolute error. This measures the optimizer's effect on how the credit aligns with stopping.

The H8 study uses eight paired seeds from one checkpoint and 96,000 held-out generations. All 256 updates preserve each F/R branch's current prompt means to 3.34×10−163.34\times 10^{-16}. R-D completed correctness is +1.77 pp [95% paired-seed t interval .57,2.97], 8/8 positive (planned secondary metric; Holm p=.041). The primary D-F truncation contrast is +.25 pp [-.81,1.31], Holm p=.739.

2. Longer training and the stopping readout

Seeds 101/102 continue with model and Adam state retained; a runtime/capacity rule fixes the reached horizons. Their mean R-D completed-correct differences are +2.38/+3.88/-7.47/+1.60 pp at H8/H16/H32/H64. H32 seed effects are +.13/-15.07 pp. At H64, completed correctness is 9.95/0.00/11.55/34.77% for D/F/R/P: all four arm averages have lost accuracy. S32.1 retains every seed, arm, reached horizon and termination outcome. Its question intervals condition on two trained models; the original H8 intervals use eight training seeds.

At H32, R-D fixed-prefix EOS log probability is +1.09828, while free-generation truncation is +8.83 pp and stopping within the next 128 tokens changes by -.49 pp. The fixed-prefix readout and the evolving generation distribution consequently provide different information. We state the local component effect and the full treatment's horizon-dependent behavior separately.

3. Cross-domain zero-component validation

In code seed 201's first batch, one truncated response receives reward, but truncated reward is constant within every prompt. D/F coefficients coincide. All 7,615,616,512 FP32 master coordinates, BF16 weights and 1,017 Adam tensors agree exactly. In an all-correct group with one truncated response, Z instead assigns -.875 to it and +.125 to each completed response.

The actual code H16 study yields F-D completed correctness +1.88 pp [-1.45,5.21] (primary, Holm p=.340) and Z-D +2.61 pp [1.32,3.90], 4/4 positive (planned secondary, Holm p=.023). This is a complementary, falsifiable check of the same decomposition: positive aggregate payment with constant within-prompt rewards predicts a zero internal component, and the full parameter comparison verifies that prediction. Zeroing reward changes the status comparison instead. All code seeds and H1/H8/H16 are in S32.2.

Together, the component controls identify the local stopping effect; the Adam comparison measures how the optimizer transforms the signal; and the paired forks measure the full reward treatment during generation. This connects each termination claim to a directly observed quantity and specifies which experimental conditions it describes.

AuthorsOpenReview ↗

Masking: consistent inclusion and policy-equal estimates (3/3)

Policy-equal weighting and a consistent inclusion rule give a configuration-specific masking result. We now state that result directly and retain both weightings. The operation-level conclusion is exact: reward replacement changes advantages; the audited masking implementation removes truncated token losses while retaining their rewards in the group baseline (Section 6; S16, S26 and S31.8).

1. Estimates and inclusion sensitivity

SetNegative differencesCell mean (pp)Policy-equal mean (pp)
Submitted 12 cells, four policies9/12-5.272-2.757
Audited 14 completed lineages, five policies11/14-5.652-3.866
Including the early-ended pair11/15+.658+4.996

The submitted policy-equal 95% interval is [-7.678,+2.163] pp. These are descriptive averages over the stated configurations. The early-ended E12/M42 pair is a separate stratum: at common evaluation index 7, completed correctness is .045 versus .935 (+89.0 pp). Both strata and both weightings appear in the revised ledger.

2. One rule applied across the ledger

We use explicit completed-correct accuracy, one endpoint per lineage, and the latest common post-training evaluation, marking earlier-than-final endpoints. Resumed duplicate rollout IDs and ancestor/continuation pairs count once. The old Qwen 3 +.9 pp pair has different training-file, launcher and monitor hashes and leaves the matched set. Three completed comparisons enter: -13.167, -2.183 and +.383 pp. S26 gives their provenance.

The revised text reports these configuration-specific estimates and inclusion sensitivity. The per-prompt controls and code parameter check explain the operational distinction: reward replacement changes advantages, while masking changes the token losses receiving updates and retains a baseline effect.

Your three questions now have corresponding revisions: same-trajectory component controls, separate optimizer and behavioral measurements, and an explicit masking inclusion/weighting table. We would appreciate your reassessment of soundness and the explanatory contribution on this basis.

Reviewer_YoTROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper highlights that the default scoring in many open-source RLVR stacks grades the response generated regardless of whether they have been truncated. The authors call the average reward given to cut-off responses the “salvage value” and show it can be calculated from numbers training logs. They argue it can teach models to write the answer early and then keep going instead of stopping, and that it distorts reported accuracy, since changing only the length limit changes the score. In experiments with small math models (3–7B), they show a few training runs where nearly every answer was accepted but almost none were finished. They also test what happens when cut-off responses get a fixed reward instead, run a controlled comparison where the only difference between two training runs is whether cut-off responses get this credit, and compare a common fix of excluding cut-off responses from training, which they find usually lowers accuracy.

Critical strengths
  1. How truncated outputs are graded is an implementation detail that rarely gets reported, it changes the training objective and the reported score. The framework audit (Table S2) and the reporting recommendation are directly useful to practitioners.
  2. The proposed salvage value needs no extra runs or annotation. The demonstration that one checkpoint scores 0.162 or 0.440 on the hardest problems depending only on the evaluation cap is a valuable caution about comparing training runs at different budgets.
Critical weaknesses
  1. The paper assumes that graders credit answers anywhere in the text, as in boxed-answer math with lenient extraction. In common setups where the answer comes in a structured format at the end (e.g., a final JSON), or where graders read only the end of the response (as the DAPO reference grader does), truncated outputs typically earn no credit and salvage is near zero. The paper’s own data shows strong model dependence: salvage is paid in 77% to 3% of cells across model families. Experiments are also narrow: 3–7B models, math only, one stack, one verifier family, and 10–16 training updates, far shorter than typical RLVR runs where these dynamics would play out. The paper should characterize which formats, graders, and training regimes are affected.

  2. The paper argues that credit for cut-off responses drives policies to answer early and then keep generating until the cap. The evidence shows this can co-occur, not that salvage causes it or that it is common. The severe collapses come from earlier code revisions and don’t reproduce on the current setup across 14 runs. In the remaining case, salvage was about zero before the collapse, so it cannot have driven it. The one direct causal test, the preregistered fork, yields an effect statistically indistinguishable from zero. This does not support any causal relation.

  3. I think the novelty of the paper is limited. The fact that truncated outputs can receive credit, and should be handled explicitly, is already recognized: overlong masking, length penalties, and explicit zero-at-truncation rules exist for exactly this reason.

Questions for authors
  1. Can you strengthen the causal test with more seeds, longer training or forks from a high-truncation regime
  2. Beyond the reporting recommendation, what do you see as the main new insight relative to existing overlong-handling methods?
  3. Do the findings hold over longer training, larger models and other domains?
5 comments
AuthorsOpenReview ↗

New insights beyond existing overlong treatments (1/3)

Thank you for your careful review and your questions about novelty, causality, and applicability beyond mathematics. The new evidence shows why strict answer verification can reward an unfinished response, and how retained credit can contain opposing local stopping contributions at the same prompt mean. This extends the original learning/measurement/update account through controlled parameter interventions and executable-code training. Existing zeroing, penalty and masking methods modify different parts of that account: the reward level, its within-prompt allocation and the token losses receiving updates. Three results make those distinctions experimentally concrete (Sections 3 and 6; S31-S32).

1. Strict answer verification can still reward unfinished responses

The code verifier requires a complete Python block passing every supplied test. Nevertheless, 154 of 157 rewarded truncated outputs in the baseline/H1 audit continue with Explanation after the program. Four-seed actual training under this verifier extends the accepted-but-unfinished phenomenon beyond boxed-answer mathematics.

The same code study supplies a sharp parameter control: a batch has a rewarded truncated output but constant truncated reward within each prompt. Flattening to each prompt's truncated mean then produces exactly the same parameters and Adam state as default grading. Positive aggregate salvage and an available within-prompt learning component are separate quantities.

2. The same rewards can produce different local stopping updates

On identical mathematical trajectories, D uses paid reward r; F assigns the current prompt's truncated mean b_r; R assigns b_r+u-b_u, with u scoring the last complete box and b_u its mean among that prompt's truncated outputs. P subtracts 0.25 from truncated rewards. Completed rewards are unchanged. D/F/R preserve sampled prompt means and the completed/truncated reward contrast.

At two checkpoints (1,024 trajectories each; 230/240 fixed correct dev prefixes), pure-SGD D-R EOS log-probability effects are -.001593/-.013531, and R-F effects are +.000550/+.009221. The opposing relative contributions match parameter-gradient predictions; half the internal component gives 50.31%/48.98% of the full effect.

Four fixed permutations then preserve each prompt's entire truncated reward multiset, completed rewards and baseline. Their effects relative to F are +.000589/+.001411/+.000594/-.001049. The four S arms share D's mean and variance; F is a common reference. This measures the effect of assigning the same rewards to different trajectories.

3. Parser repair also changes the incentive to finish

Direct complete-box scoring U differs from mean-preserving R by exactly (b_u-b_r)(T-tau_g). T indicates truncation, and tau_g is the truncated fraction in the current prompt group. The full-gradient residual is 9.02e-8; measured local U-R EOS difference is -.042861 [question interval -.048186,-.037776], matching prediction -.042360. Here the additional status term opposes stopping. Answer selection and the reward level of truncation are therefore independently measurable parts of a parser change.

A sequence-sum, temperature-1 reference score repeats the four D/F/R/P signs on the same retained batch. S31.10 states the fixed-checkpoint/batch intervention and both inference units: the 64-training-prompt intervals span zero for internal projections, while the U-R status projection remains negative.

The empirical finding is the measured opposition between complete-answer and parser-boundary credit on the same accepted prefixes, together with the extra status effect of parser repair. The fixed-multiset experiment controls for reward-distribution changes; the code parameter identity checks the predicted zero-component case. Each test therefore has a specific role in evaluating the decomposition.

This gives a practical use for the original scalar and the new controls. The three-log identity measures the paid salvage value; current within-prompt variation identifies where flattening changes sampled advantages; the component comparison measures its local effect. Existing zeroing, flattening and masking rules alter different parts of this account. The next two responses give the grader/domain evidence and the full training outcomes.

AuthorsOpenReview ↗

Code training and the conditions for truncation credit (2/3)

A complete-answer requirement and a natural-termination requirement impose different contracts. We tested your end-of-response example with the unmodified DAPO grader and added actual code training requiring complete programs to pass every supplied test. Both tests locate the conditions under which accepted credit survives truncation (S31.2 and S32.2).

1. Four-seed executable-code training

DeepSeek-R1-Distill-Qwen-7B trains on 256 tasks and evaluates 256 disjoint tasks, with four seeds and 16 full-parameter updates per arm (192 updates). Tasks use TACO/APPS/CodeContests/Codeforces sources in verifiable-coding-problems-python. Splits group duplicate statement/source identities, and supplied reference programs must pass all tests before model sampling.

The verifier selects the last complete Python block and requires every supplied test to pass. D retains reward, F assigns the current prompt's truncated mean, and Z assigns zero at truncation. All arms share the evaluation grader.

H16 armCompleted correct (%)Truncation (%)
D29.6463.60
F31.5258.64
Z32.2560.60

Z-D completed correctness is +2.61 pp [95% paired-seed interval 1.32,3.90], positive in 4/4 seeds (planned secondary; Holm p=.023). The primary F-D contrast is +1.88 pp [-1.45,5.21], Holm p=.340. F preserves each current prompt mean; Z also changes the status contrast. All seeds and H1/H8/H16 appear in S32.2. D's conditional salvage is 1.19%, and token-aligned continuations identify the passing program followed by explanation.

2. End-of-response grading and format compatibility

The unmodified DAPO reference function (commit 33fe3176, default official evaluation mode) accepts 0/1,024 truncated outputs in the old 4K checkmark batch, 125/366 in later s47, 149/398 in fresh s47, and 7/231 in fresh s56. Tail scoring removes credit from distant checkmark tails, while repeated answers near the cap retain credit. S31.2 gives all twelve cohorts.

Fresh s47 has similar original/DAPO counts, 150/398 versus 149/398, yet 125 labels differ and only 87 outputs are jointly accepted. This is an empirical reason to measure allocation alongside acceptance: nearly equal pass counts correspond to different labels on 125 trajectories. Acceptance is compared on a common binary scale here; DAPO's native reward is +1/-1, so its truncated mean reward is twice the acceptance fraction minus one.

Pinned DAPO reference implementation.

The pinned upstream slime marker gate accepts neither truncated nor completed outputs in our 12,288 marker-free responses. We label this a format-compatibility ablation; the DAPO test uses its full original checker. With binary correctness rewards, an explicit completion gate or zero-at-truncation override makes salvage zero by design.

3. Larger-model evaluation and complementary selection evidence

The 27B evaluation pilot has 576 outputs, with accepted-truncated counts 0/143, 19/59 and 15/39 at caps 1,024/4,096/12,288. Training coverage remains 3-7B mathematics and 7B code.

A separate fixed-text audit selects an earlier unchanged, fully passing program in 191/388 preselected entry-point errors using a fixed interface rule. All responses naturally stop. It tests which artifact a continuation causes the grader to select; the four-seed code experiment above supplies the cross-domain training evidence.

The scope is therefore specified by the extraction rule, reward scale and termination contract together. The old checkmark case shows tail grading removing credit; repeated answers and complete executable programs show how credit survives under other concrete contracts. This explains the observed model/grader dependence within the paper's original framework.

AuthorsOpenReview ↗

Causal controls, seed replication, and longer training (3/3)

We added the local controls, seed replication, longer trajectories and non-mathematical training requested in your review. Two-checkpoint interventions identify the effects of specific credit components on a fixed stopping readout. Eight paired math seeds, both prespecified continuations through H64, and four code seeds measure the full treatments under actual optimization (Section 6; S31.6 and S32).

1. Eight-seed training and the actual optimizer

The H8 experiment uses eight paired seeds from one checkpoint, four arms and 96,000 held-out generations under a common grader. F/R preserve current sampled prompt means in all 256 updates, to 3.34e-16.

R-D completed correctness is +1.77 pp [95% paired-seed t interval .57,2.97], positive in 8/8 seeds (planned secondary; Holm p=.041). The primary D-F truncation contrast is +.25 pp [-.81,1.31], Holm p=.739.

The optimizer measurably changes alignment with stopping: R-F's raw projection is positive in 6/8 first batches, while its actual first-Adam task-step projection is negative in 8/8. Analytical and actual projections agree within 8.3e-7 absolute error. We therefore measure the deployed update alongside the raw component.

2. Both continuations through H64

The two seeds retain model and Adam state; a time/runtime rule fixes the reached horizons. Their mean R-D completed-correct differences are +2.38/+3.88/-7.47/+1.60 pp at H8/H16/H32/H64. The gain changes with the horizon; at H64 all four arm averages lose accuracy, with D/F/R/P completed correctness 9.95/0.00/11.55/34.77%.

S32.1 reports every arm, seed, reached horizon and termination outcome, including the divergent seed effects. Continuation intervals resample questions conditional on two trained models; H8 inference uses eight training seeds. These trajectories establish the measured time window alongside the controlled local effects.

3. What the historical cases and fork contribute

The original cases document accepted-but-unfinished behavior. The two terminal cases used earlier launchers; fourteen later default runs finish without saturation. In the transient case, salvage is zero before the excursion and then reaches .612 while completed-correct mass falls from .696 to .0146. The new same-trajectory interventions supply the local causal identification.

The old global-pin fork changes both the reward-mean contrast and internal variation. Its six-seed truncation difference is +1.42 pp [-3.69,+6.52], with completed-correct signs split 3/3. The revised attribution identifies both terms changed by this combined intervention.

The resulting insight is transferable across the two studied domains: verifying an answer artifact, retaining reward at the cap and producing a group-relative learning signal are separate operations that can be measured and controlled. Mathematics and code test different components of this same account; their R and Z treatments retain their distinct definitions. We would appreciate your reassessment of novelty and scope in light of the controlled mechanisms and actual executable-code training. These results connect the paid salvage value to identifiable update components and measured termination outcomes.

Reviewer_3A4FOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies mathematical RLVR runs in which responses receive correct-answer rewards but continue generating until the length limit. It introduces the average reward of truncated responses as a diagnostic and examines how generation budgets, reward replacement, and loss masking affect training and evaluation.

Critical strengths

The logs and checkpoint evaluations show that high reward can coexist with poor stopping behavior, making a useful case for reporting correctness and completion separately. The distinction between reward replacement and loss masking also helps clarify how different treatments of truncated responses affect training.

Critical weaknesses
  1. The stopping failure is not sufficiently explained. The paper does not clearly show what models generate after producing a correct answer. An early boxed answer is not necessarily the answer accepted by the verifier. This makes it difficult to assess whether the proposed reward mechanism explains the observed behavior.

  2. The results provide limited new understanding beyond practical diagnostics. Decomposing evaluation metrics and controlling generation budgets are useful, but their research contribution is modest. The fixed-reward intervention changes both the mean reward of truncated responses and the reward differences between them, while its results across seeds are mixed. I find the problem worth studying, but the current explanation and evidence insufficient for acceptance.

Questions for authors
  1. How much text is generated after the answer accepted by the verifier, and what does it contain? Please provide representative continuations and the corresponding stopping settings.
  2. Does stopping behavior change because the average reward of truncated responses changes, or because correct and incorrect truncated responses are now rewarded equally? Can the authors distinguish these two effects?
4 comments
AuthorsOpenReview ↗

Verifier-selected answers, continuations, and stopping settings (1/3)

Thank you for your careful review and for asking us to trace the verifier-selected answer and distinguish reward level from reward allocation. The accepted answer can be followed by thousands of tokens, and the continuation depends on the regime: checkmarks in the old runs, repeated answers in later mathematics, and explanation after test-passing code. Replaying 10,240 mathematical responses reproduces all 2,176 accepted-truncated labels (Section 5; S31.1 and S32.2).

1. What follows the answer actually accepted?

We separately locate the answer selected from the retained response and the earliest complete boxed prefix accepted by the same checker. Four selected probes are shown below; accepted-truncated counts for all ten saved batches appear in the original-grader column of the table in S31.2.

ProbeAccepted / truncatedMedian tokens after selected answerAfter earliest accepted prefix
Old cap 4,096940 / 1,0243,5983,616
Old cap 12,288886 / 95010,56610,902.5
Later s56, step 15165 / 209224,292
Later s47, step 15131 / 366111,406

The old runs append checkmarks: 935/940 and 858/886 accepted-truncated responses have checkmark-dominated tails. One gives the correct boxed 4*sqrt(2), then repeatedly emits checkmarks. Later runs repeatedly restate answers: 146/165 and 131/131 contain at least ten boxes. An s56 example repeats a boxed 3, "I think that's correct", and "Final answer". The later checker often selects a repeated answer near the cap although an earlier prefix was already accepted. Table 2 and S31.1 distinguish these two boundaries.

Settings are temperature .6, top-p .95, top-k 20, prompt limit 2,048, and response/context caps 4,096/6,144 or 12,288/14,336. All 2,743 truncated responses reach the cap without EOS; all 7,497 completed responses contain EOS.

2. A complete program can also belong to an unfinished response

The code verifier selects the last complete Python block and requires every supplied test to pass. Among 157 rewarded truncated baseline/H1 outputs, 154 continue with an Explanation section, two reach its heading prefix, and one closes the passing code fence at the cap. Median tokens after the selected fence are 71/88/62/86.5 for baseline/D/F/Z. The selected programs are aligned to original tokens and their executed test results.

The distinction is therefore between the accepted answer artifact and the end of its response. A checker requiring a complete, test-passing program can still pay reward at forced termination. Our second response separates the reward-level and reward-allocation effects that you ask about.

AuthorsOpenReview ↗

Separating reward level from within-prompt credit allocation (2/3)

We can separate reward level from reward allocation on the same trajectories. Under controlled SGD steps, allocation changes the EOS readout on fixed correct-answer prefixes while each prompt's mean is held fixed; the effect also varies across assignments of the same reward multiset. Direct parser repair supplies a separately measured reward-level effect. These controls address the two explanations in your question (Section 6; S31.4-S31.6 and S31.10).

1. Holding each prompt's mean fixed

On identical trajectories, let rr be paid reward, uu the score through the last complete box, and brb_r and bub_u their truncated means within the current prompt. TT indicates truncation, and τg\tau_g is the truncated fraction in the current prompt group. Completed rewards stay unchanged. Truncated rewards are:

D=r,F=br,R=br+u−bu,P=r−0.25D=r,\quad F=b_r,\quad R=b_r+u-b_u,\quad P=r-0.25

D/F/R preserve each sampled prompt mean and its completion/truncation reward contrast. Let AA denote the group-centered advantage. With group centering, no group-std normalization and a common score Jacobian:

AD−AF=T(r−br)A_D-A_F = T(r-b_r)

AD−AR=T[(r−br)−(u−bu)]A_D-A_R = T[(r-b_r)-(u-b_u)]

AR−AF=T(u−bu)A_R-A_F = T(u-b_u)

These isolate actual internal, parser-boundary and complete-answer credit. P changes the status contrast. Each training branch recomputes the means on its own current batch.

2. Measured local response at two checkpoints

Each checkpoint supplies 1,024 fresh trajectories and frozen correct dev prefixes (230/240 questions). Common-step FP32 SGD yields differences in mean EOS log probability:

Contrasts47s56
D-F: actual internal credit-.001043-.004310
D-R: parser-boundary credit-.001593-.013531
R-F: complete-answer credit+.000550+.009221

The effects match parameter-gradient predictions; full-gradient identity residuals are at most 2.39×10−62.39\times 10^{-6} and 2.66×10−72.66\times 10^{-7}, respectively. Halving only the internal component gives 50.31%/48.98% of its full effect. All D/F/R/P full task steps raise this readout; the contrasts identify opposing relative contributions.

3. The same rewards, assigned differently; then a separate level change

Four fixed within-prompt permutations share D's entire truncated reward multiset, completed rewards and baseline. Their EOS effects relative to common reference F are +.000589/+.001411/+.000594/-.001049. Assigning the same rewards to different trajectories changes the local effect at identical mean and variance.

Direct complete-box scoring U differs from R by exactly:

AU−AR=(bu−br)(T−τg)A_U-A_R = (b_u-b_r)(T-\tau_g)

The full-gradient residual is 9.02×10−89.02\times 10^{-8}; measured U-R EOS difference is -.042861 (question interval [-.048186,-.037776]). Here parser repair adds a status incentive opposing stopping. A sequence-sum, temperature-1 reference score repeats the four D/F/R/P contrast signs on the same retained batch.

These finite interventions identify effects at the fixed checkpoint, batch and SGD step. S31.10 separates dev-question intervals from the 64-training-prompt intervals, which span zero for internal projections and remain negative for the U-R status projection. The historical global-pin fork is now described as a combined intervention (+1.42 pp truncation, interval [-3.69,+6.52]); the prompt-specific controls provide the requested separation.

The practical consequence is a controlled choice: changing which truncated answers receive credit and changing the amount paid at truncation are different interventions. The revised decomposition specifies both, and the parameter interventions measure their local effects.

AuthorsOpenReview ↗

Training evidence and new insights across mathematics and code (3/3)

The key new finding is that complete-answer and parser-boundary credit produce opposing local stopping effects at fixed prompt means. The salvage mean records payment at the cap; within-prompt allocation supplies a distinct update component. Mathematics gives controlled nonzero components. Executable code tests when that component vanishes and extends training to a verifier requiring complete, test-passing programs.

1. Actual training in both domains

The mathematical experiment uses eight paired seeds from one checkpoint, D/F/R/P for eight Adam updates, and 750 held-out questions with four responses each. The code experiment uses a 7B model, 256 training and 256 disjoint held-out tasks, four seeds and D/F/Z for 16 full-parameter updates. All arms within each study use the same evaluation grader.

ComparisonOutcomeDifference and 95% paired-seed intervalPlanned role; Holm p
Math R-D, H8completed correct+1.77 pp [.57,2.97], 8/8 positivesecondary; .041
Math D-F, H8truncation+.25 pp [-.81,1.31]primary; .739
Code Z-D, H16completed correct+2.61 pp [1.32,3.90], 4/4 positivesecondary; .023
Code F-D, H16completed correct+1.88 pp [-1.45,5.21]primary; .340

R reallocates complete-answer credit at fixed current means; Z sets truncated reward to zero. These are distinct operations in the same decomposition. Code D/F/Z achieve 29.64/31.52/32.25% completed correctness. The verifier requires a complete program passing every test, yet D retains 1.19% conditional salvage.

2. Why the component and the full behavior are both measured

In one code batch, truncated reward is positive in aggregate but constant within each prompt. D/F then yield identical parameters and Adam state. Thus group structure determines whether the internal component is available. In mathematics, R-F's raw stopping projection is positive in 6/8 first batches, while its actual first-Adam projection is negative in 8/8: the optimizer transforms the available signal.

Both prespecified math continuation seeds reach H64. Their R-D completed-correct differences average +2.38/+3.88/-7.47/+1.60 pp at H8/H16/H32/H64. All four arm averages lose accuracy by H64, reaching 9.95/0.00/11.55/34.77% for D/F/R/P. S32.1 gives every seed, arm and reached horizon; continuation intervals condition on the two trained models.

3. What this adds to existing overlong treatments

The decomposition gives testable current-batch consequences:

OperationComponent changedConsequence
Flatten to the prompt's truncated meaninternal reward allocationIdentical sampled advantages when truncated rewards are constant within each prompt; verified by identical code parameters and Adam state
Apply a common truncation penaltycompleted/truncated contrastThe centered penalty vanishes in entirely completed or entirely truncated groups
Mask truncated token losses while retaining their rewards in the baselineloss supportTruncated samples still affect the advantages of completed samples

Together with the fixed-multiset test, this turns the original learning/measurement/update account into an experimentally checked explanation of the available signals. Sections 5-7 and S31-S32 connect these controls to the measured training outcomes. We would appreciate your reassessment of the mechanism and informativeness in light of these additions.

AuthorsOpenReview ↗

Follow-up on answer continuations and reward controls

Dear Reviewer 3A4F,

Thank you again for your review. As the discussion draws to a close, we wanted to highlight two additions responding to your questions:

  • Our audit distinguishes the verifier-selected answer from the earliest accepted prefix: old continuations contain checkmarks; later ones repeat answers. It reports tail lengths and stopping settings (S31.1).
  • On identical trajectories, controls hold each prompt's sampled reward mean fixed and separate opposing local EOS contributions from complete-answer and parser-boundary credit (SGD on fixed correct-answer prefixes; Section 6, S31).

Our first two responses give the details. We would appreciate your view on whether these additions address your questions, and any remaining concerns we can clarify.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overview of the revision and additional evidence

We thank the reviewers for their detailed questions. We have revised the manuscript in response. Revised passages and the headings of newly added appendix sections are marked in dark blue.

Across mathematics and executable code, a complete accepted answer can belong to an unfinished response. The new controls show that reward level and within-prompt credit allocation have distinct local update effects. For RLVR practitioners, these findings inform two routine decisions: how to interpret a pass-rate gain and which learning signals to change when handling overlong outputs. Salvage value is recoverable from three logged scalars, allowing the measurement audit to start from existing logs without retraining. The added evidence develops the original learning/measurement/update account.

1. Acceptance and completion separate even under complete-program verification. We replay 10,240 mathematical responses to locate the answer actually selected by the verifier, its continuation and the stopping settings. We also complete four-seed executable-code training, with 16 full-parameter updates per arm and a verifier requiring every supplied test to pass. A complete passing program can be followed by explanation until the cap. Thus the boundary between an accepted answer artifact and a finished response also arises in an executable domain. (Sections 5-7; S31.1, S32.2.)

2. The amount paid and the allocation of credit have distinct update effects. At two checkpoints, same-trajectory, per-prompt controls hold reward means fixed. Controlled SGD steps reveal opposite local EOS effects from complete-answer and parser-boundary credit on fixed correct-answer prefixes. Further controls preserve the entire within-prompt truncated-reward multiset while changing its assignment, and separately measure the status-reward effect introduced by directly repairing answer extraction. These experiments connect verifier decisions to identifiable components of the update. (Section 6; S31.)

3. The decomposition yields testable conditions for when a component is active. A code batch has positive aggregate truncation credit but constant truncated rewards within each prompt. Default grading and per-prompt mean replacement then produce exactly identical parameters and Adam state, as predicted. Mathematical interventions provide the complementary nonzero components. Eight-seed mathematical forks, two prespecified continuations through 64 updates, and the code training comparisons separately measure full-training outcomes; their complete trajectories and comparison roles appear in the reviewer-specific replies. (Sections 6-7; S31-S32.)

Practical use. For binary verifier rewards, truncation rate times salvage value measures the accepted-but-unfinished share of reported success. Recording within-prompt truncated-reward variation and completion-status composition helps identify which credit contrasts an overlong treatment changes. These checks guide evaluation and reward design according to whether success requires an accepted artifact, a completed response, or both.

The revision preserves the title, research question, central definition and decomposition, and the original section order. New evidence is concentrated in Sections 5-7 and S31-S32; the revision guide also identifies corrections to the historical fork attribution, aligned-budget analysis and masking comparisons. Below we address each reviewer's questions with the relevant settings, results and locations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.