- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates how reinforcement learning with verifiable rewards handles responses that reach the generation limit without terminating. It defines “salvage value” as the mean reward assigned to truncated responses and shows how this quantity relates verifier acceptance, completion, and reported reward. It also decomposes an idealized policy gradient into completed-answer credit, truncation-status credit, and reward-dependent credit within truncated responses.
The empirical study includes budget sweeps, examples of accepted answers followed by continued generation, fixed truncation-reward interventions, token-loss masking, and matched checkpoint forks. The principal finding is that high verifier acceptance can coexist with very low completed-and-correct accuracy. The paper recommends reporting these outcomes separately and specifying truncation rewards, loss masks, and evaluation budgets explicitly.
Critical strengths
The paper documents a practically important discrepancy between accepted answers and completed responses. The reported cases are striking: in one training batch, all 848 easy-stratum responses are accepted, while only 11 complete. Reloaded-checkpoint probes provide additional evidence that answers can appear well before generation stops at the cap. This motivates reporting completion and acceptance separately and makes the contribution more concrete than an abstract observation about reward specification.
The analysis usefully distinguishes interventions that are often grouped together. Assigning a constant reward to truncated outputs, masking their token losses, retaining their rewards in the group baseline, and changing the generation budget are different operations. The decomposition clarifies these distinctions. The authors also report contrary results, failed predictions, and limitations, including the absence of saturation in the later default-grader cohort. This transparency helps establish the scope of the findings.
Critical weaknesses
The matched-fork experiment does not isolate the claimed within-truncation mechanism. Appendix S15.3 replaces truncated rewards with a historical, batch-level constant. This removes within-truncation reward variation, but it does not preserve each prompt’s conditional mean reward, even initially, and does not preserve the evolving status contrast during subsequent updates. The appendix explicitly acknowledges the additional term , which conflicts with the statement that the design isolates the proposed term “and nothing else.” Moreover, the predicted truncation direction appears in only four of six seeds, the reported interval includes zero, and strict-accuracy effects split three versus three. These results support an exploratory intervention comparison, but do not establish that the within-truncation covariance drives increased truncation.
The headline intervention findings have limited robustness across configurations. Nine of the twelve masking comparisons use Qwen3.5-4B-Base. The policy-equal estimate reported in Appendix S16 is −2.8 percentage points, with a 95% interval of [−7.7, +2.2], providing weaker support for a general masking penalty than the pooled twelve-cell interval suggests. The two terminal default-grader collapses also occurred under earlier launcher revisions, while none of the fourteen later default runs saturated. The authors acknowledge these limitations, but the broad explanatory framing remains stronger than the evidence. Because the central accounting identities are elementary, the contribution depends particularly on robust empirical evidence and careful causal interpretation.
Questions for authors
Can you separate removal of within-truncation reward variation from changes in the truncation-versus-completion reward contrast? For example, a same-checkpoint, per-prompt gradient comparison on identical sampled trajectories could assess the proposed component without relying on an eight-update intervention with drifting reward means.
How should the four-of-six fork result and its interval containing zero be interpreted? Would you revise the causal claim to an inconclusive result, or is there additional evidence directly connecting the within-truncation gradient component to subsequent termination behavior?
What conclusion about masking survives policy-equal weighting and a consistently defined inclusion rule across all completed comparisons? Please distinguish a descriptive average over the selected cells from an effect expected across policy families.
5 comments
Local effects, Adam updates, and training through H64 (2/3)
The new evidence follows the component through three measured stages: a controlled parameter intervention, the actual optimizer step, and subsequent generation. This makes the connection to termination explicit at each stage. We added eight-seed mathematical training, both prespecified continuations through H64, and four-seed executable-code training (Section 6; S31.6 and S32).
1. Actual Adam and eight-seed outcomes
R-F's raw stopping-gradient projection is positive in 6/8 first batches; its actual first-Adam task-step projection is negative in 8/8. Analytical and actual projections agree within absolute error. This measures the optimizer's effect on how the credit aligns with stopping.
The H8 study uses eight paired seeds from one checkpoint and 96,000 held-out generations. All 256 updates preserve each F/R branch's current prompt means to . R-D completed correctness is +1.77 pp [95% paired-seed t interval .57,2.97], 8/8 positive (planned secondary metric; Holm p=.041). The primary D-F truncation contrast is +.25 pp [-.81,1.31], Holm p=.739.
2. Longer training and the stopping readout
Seeds 101/102 continue with model and Adam state retained; a runtime/capacity rule fixes the reached horizons. Their mean R-D completed-correct differences are +2.38/+3.88/-7.47/+1.60 pp at H8/H16/H32/H64. H32 seed effects are +.13/-15.07 pp. At H64, completed correctness is 9.95/0.00/11.55/34.77% for D/F/R/P: all four arm averages have lost accuracy. S32.1 retains every seed, arm, reached horizon and termination outcome. Its question intervals condition on two trained models; the original H8 intervals use eight training seeds.
At H32, R-D fixed-prefix EOS log probability is +1.09828, while free-generation truncation is +8.83 pp and stopping within the next 128 tokens changes by -.49 pp. The fixed-prefix readout and the evolving generation distribution consequently provide different information. We state the local component effect and the full treatment's horizon-dependent behavior separately.
3. Cross-domain zero-component validation
In code seed 201's first batch, one truncated response receives reward, but truncated reward is constant within every prompt. D/F coefficients coincide. All 7,615,616,512 FP32 master coordinates, BF16 weights and 1,017 Adam tensors agree exactly. In an all-correct group with one truncated response, Z instead assigns -.875 to it and +.125 to each completed response.
The actual code H16 study yields F-D completed correctness +1.88 pp [-1.45,5.21] (primary, Holm p=.340) and Z-D +2.61 pp [1.32,3.90], 4/4 positive (planned secondary, Holm p=.023). This is a complementary, falsifiable check of the same decomposition: positive aggregate payment with constant within-prompt rewards predicts a zero internal component, and the full parameter comparison verifies that prediction. Zeroing reward changes the status comparison instead. All code seeds and H1/H8/H16 are in S32.2.
Together, the component controls identify the local stopping effect; the Adam comparison measures how the optimizer transforms the signal; and the paired forks measure the full reward treatment during generation. This connects each termination claim to a directly observed quantity and specifies which experimental conditions it describes.
Masking: consistent inclusion and policy-equal estimates (3/3)
Policy-equal weighting and a consistent inclusion rule give a configuration-specific masking result. We now state that result directly and retain both weightings. The operation-level conclusion is exact: reward replacement changes advantages; the audited masking implementation removes truncated token losses while retaining their rewards in the group baseline (Section 6; S16, S26 and S31.8).
1. Estimates and inclusion sensitivity
| Set | Negative differences | Cell mean (pp) | Policy-equal mean (pp) |
|---|---|---|---|
| Submitted 12 cells, four policies | 9/12 | -5.272 | -2.757 |
| Audited 14 completed lineages, five policies | 11/14 | -5.652 | -3.866 |
| Including the early-ended pair | 11/15 | +.658 | +4.996 |
The submitted policy-equal 95% interval is [-7.678,+2.163] pp. These are descriptive averages over the stated configurations. The early-ended E12/M42 pair is a separate stratum: at common evaluation index 7, completed correctness is .045 versus .935 (+89.0 pp). Both strata and both weightings appear in the revised ledger.
2. One rule applied across the ledger
We use explicit completed-correct accuracy, one endpoint per lineage, and the latest common post-training evaluation, marking earlier-than-final endpoints. Resumed duplicate rollout IDs and ancestor/continuation pairs count once. The old Qwen 3 +.9 pp pair has different training-file, launcher and monitor hashes and leaves the matched set. Three completed comparisons enter: -13.167, -2.183 and +.383 pp. S26 gives their provenance.
The revised text reports these configuration-specific estimates and inclusion sensitivity. The per-prompt controls and code parameter check explain the operational distinction: reward replacement changes advantages, while masking changes the token losses receiving updates and retains a baseline effect.
Your three questions now have corresponding revisions: same-trajectory component controls, separate optimizer and behavioral measurements, and an explicit masking inclusion/weighting table. We would appreciate your reassessment of soundness and the explanatory contribution on this basis.
Same-trajectory controls with prompt means held fixed (1/3)
Thank you for your careful review and the concrete suggestion for separating the reward components. We implemented the same-checkpoint, per-prompt comparison you proposed and replicated it at a second checkpoint. On identical trajectories, the controls hold the sampled reward-mean contrast fixed and separate opposing local stopping contributions. Full-gradient identities, finite parameter interventions and half-dose checks connect the algebra to measured parameter effects (Section 6; S31).
1. The requested identical-trajectory intervention
Let r be actual reward, u the score through the last complete box, and br and bu their truncated means within the current prompt. T is the truncation indicator, and τg is the truncated fraction in that prompt group. Completed rewards are unchanged. D/F/R assign r, br and br+u−bu to truncated outputs; P assigns r−0.25.
Let A denote the group-centered advantage. With group-std normalization disabled and a common response-token-mean score Jacobian:
AD−AF=T(r−br)
AD−AR=T[(r−br)−(u−bu)]
AR−AF=T(u−bu)
D/F/R preserve the sampled prompt mean and completion/truncation mean contrast. Each first-step comparison shares trajectories within its seed; subsequent branches recompute means on their own current batches.
Each checkpoint supplies 1,024 fresh trajectories and frozen correct dev prefixes (230/240 questions). Common-step pure SGD gives EOS log-probability contrasts:
Predictions use actual FP32 parameter differences. Full-gradient residuals are 2.39×10−6 and 2.66×10−7; repeated D is exact. Half the internal component gives 50.31%/48.98% of its full effect. All four D/F/R/P task steps raise the readout; the contrasts identify their relative contributions.
2. Controlling the entire reward distribution, then isolating the mean change
Four fixed permutations preserve each prompt's entire truncated reward multiset, completed rewards and baseline. Their S-F effects are +.000589/+.001411/+.000594/-.001049. All S arms share D's mean and variance; F is their common reference. Credit-to-trajectory assignment therefore changes the local response with those reward statistics held fixed.
Direct complete-box scoring U differs from R by the following advantage term:
AU−AR=(bu−br)(T−τg)
The full-gradient residual is 9.02×10−8. Measured U-R EOS difference is -.042861 [question interval -.048186,-.037776], versus prediction -.042360. Its status projection, expressed per unit step before multiplication by the learning rate, has a 64-training-prompt interval [-17.9666,-3.0212]. This isolates the extra reward-level effect introduced by direct parser repair.
The reference sequence-sum, temperature-1 score repeats all four D/F/R/P signs on the retained batch. S31.10 reports both score conventions and inference units: dev-question intervals condition on the intervention; training-prompt intervals span zero for the internal projections. The finite effects are identified for the fixed checkpoint, batch and SGD step.
3. Historical fork attribution
We corrected "isolates that term and nothing else": the old global constant changes both internal credit and the status contrast. Its six-seed truncation difference is +1.416 pp [-3.687,+6.519], positive in 4/6, with completed-correct signs split 3/3. It measures the combined change in these two terms. The new per-prompt experiment supplies the component identification requested here. It preserves the sampled contrast exactly, and the fixed-multiset control also preserves variance across assignments. Our second response connects these local interventions to actual optimizer and training outcomes.