- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper investigates how reasoning traces used for supervised fine-tuning interact with the fixed generation budget, or “canvas,” of diffusion language models. It distinguishes two potential sources of overrun: learning to produce reasoning that exceeds the available space, and learning not to terminate because training examples were truncated before their answers and ending tokens. Using LLaDA-8B-Base and LLaDA-8B-Instruct, the authors compare teacher-generated corpora and the s1K-based d1 pipeline. Their strongest controlled comparison uses the same 14B teacher and the same 5,661 problems: terse traces yield a 15.8-percentage-point GSM8K improvement over verbose traces at a 512-token budget, with 14.3 points assigned to overrun by their descriptive decomposition. They also find that the 4,096-token training cap removes the closed answer block from approximately 91% of s1K-1.1 examples. Replacing this corpus improves the Base student, while results on the Instruct student depend strongly on the task: s1K remains better on GSM8K, but the alternative corpus transfers better to Countdown, including after GSM8K reinforcement learning.
Critical strengths
The paper addresses a practically important evaluation problem: accuracy alone can conceal whether a model produces an incorrect answer or fails to deliver an answer within its budget. Its inspection of saved generations, separation of failure categories, and audit of training-time answer loss make the findings concrete and actionable. The same-teacher experiment is substantially stronger than simply comparing teachers of different sizes, and the persistence of its accuracy gap across checkpoints increases confidence that the observed effect is substantial. The paper also deserves credit for reporting unfavorable results: the proposed corpus does not consistently improve the instruction-tuned student, and mathematical fine-tuning harms broader capabilities. The statistical reporting is unusually transparent, with individual seeds, common-support comparisons, checkpoint sensitivity, and explicit limitations. The novelty is principally diagnostic and analytical. IGPO already proposed concise, length-aligned SFT traces, so this paper’s distinctive contribution is measuring the associated completion failures, quantifying answer-stripping preprocessing, and investigating when these problems influence downstream performance.
Critical weaknesses
My main concern is that the evidence establishes the observed failure pattern more convincingly than its causal explanation. The per-problem decomposition in Section 3.2 is valid accounting, but assigning a discordant example to overrun does not establish how much accuracy would be recovered by eliminating overrun. Likewise, similar accuracy on problems both students answer cannot exclude reasoning-quality differences on problems where one student overruns. The authors acknowledge this in Appendix C.1, but Section 5.3 still argues too strongly that an advantage in reasoning quality would necessarily appear as fewer wrong answers. The cleanest intervention also changes prompting, reasoning style, and intermediate content together with length; it has only one training run per arm and is evaluated at one budget. It therefore demonstrates that terse versus verbose supervision changes behavior, while isolating length itself remains incomplete. The s1K comparison has additional confounds: domain coverage, example count, system prompt, training schedule, and answer preservation all differ. Finally, the recommended corpus screen is insufficiently validated. Its 16-of-19 ordering statistic reuses a small collection of corpora, mean length produces identical rankings, and the suggested answer-retention threshold is not empirically established. Indeed, the screen would reject s1K even though it produces the better seeded Instruct GSM8K results.
Questions for authors
MATH500 uses a different budget-failure criterion from GSM8K because its parser can extract answers from unfinished outputs; this distinction should be visible wherever the attribution results are compared. The procedure for deciding that a generation is dominated by repetition, and for detecting a correct value inside an overrun, should be specified sufficiently for reproduction. Appendix D also shows that the strict correctness filter disproportionately discards correct 14B and 32B outputs containing monetary or thousands-separator formatting; the resulting topic-selection bias should be highlighted alongside cross-teacher comparisons. Checkpoint selection initially uses test performance, and the later robustness checks mitigate rather than fully replace the need for independent validation across seeds.
3 comments
Comments on author response
Thank you for the detailed response and additional experiments. The three-seed repaired-s1K control and answer-forcing results substantially address my main concerns about confounding and the interpretation of overrun. The Dream and AR experiments also strengthen the scope of the findings, while the revised discussion appropriately acknowledges the remaining entanglement between length, style, and nontermination. I encourage the authors to maintain this distinction throughout the paper: answer forcing recovers approximately 45% of the Base comparison gap and 70% of the same-teacher gap, supporting a substantial recoverable component without establishing that the entire descriptive attribution is causal. Please clarify the denominator behind the reported 84% recovery from repairing s1K, and present the relevant absolute accuracies under matched evaluation and checkpoint-selection protocols.
Thank you. Each point is now addressed in the revised PDF.
1. The denominator of the 84%. It is the budget-fit corpus's paired gain over s1K-1.1 on the same 1,063 GSM8K problems (those not used for step selection), at L = 1024, over the same three seed pairs. The repaired s1K gains 7.09 points and the budget-fit corpus 8.47 (CI 6.6–10.3), so the share is 84% (CI 65.5–104.5%, resampling problems jointly for the three arms). In that comparison only the repaired runs had their steps chosen on the 256-problem screen; s1K-1.1 and the budget-fit corpus were scored at their reference steps.
2. Matched checkpoint selection. Every run's step is now chosen by one rule: highest accuracy on the fixed 256-problem GSM8K screen at L = 512, earliest on ties, computed from the saved full-test generations. One evaluation was missing (budget-fit, seed 2, at its newly selected step), and we ran it.
Selected steps:
| Corpus | Seed 42 | Seed 1 | Seed 2 |
|---|---|---|---|
| s1K-1.1 | 500 | 500 | 500 |
| Repaired s1K | 1,500 | 500 | 2,460 |
| Budget-fit | 3,500 | 2,000 | 1,500 |
For s1K-1.1 the rule picks the reference step in every seed; for the budget-fit corpus it changes seeds 1 and 2.
GSM8K accuracy at L = 1024 on the 1,063 problems (mean ± s.d. over three seeds; the repaired s1K is screen-selected in both columns):
| Corpus | Matched selection | Reference steps |
|---|---|---|
| s1K-1.1 | 64.4 ± 4.4 | 64.4 ± 4.4 |
| Repaired s1K | 71.5 ± 2.9 | 71.5 ± 2.9 |
| Budget-fit | 72.0 ± 1.2 | 72.9 ± 2.4 |
Under matched selection, paired on the same problems and seeds:
- Repaired s1K vs s1K-1.1: +7.1 points (CI 5.2–9.1), unchanged because the s1K-1.1 steps do not change: overrun +6.5, stopping +0.3, wrong answers +0.3.
- Budget-fit vs s1K-1.1: +7.6 points (CI 5.8–9.3).
- Recovered share: 94% (7.09 of 7.56, CI 74.0–118.0%), against 84% (7.09 of 8.47, CI 65.5–104.5%) at the reference steps.
- Repaired s1K vs budget-fit: 0.5 points behind (CI −1.2 to 2.1): overrun costs it 2.4 (CI 1.6–3.2) and stopping 0.1, while wrong answers favour it by 2.0 (CI 0.6–3.4).
Under either protocol, keeping s1K's answers recovers most of the gap, and neither interval excludes full recovery. What remains in the point estimate is overrun from traces that still exceed the canvas (ρ = 99.9%). The repair still covers only the base student on GSM8K (Appendix R).
3. Recoverable versus descriptive. We agree and keep the two apart. Section 3.2 now states that answer forcing closes about 45% of the base-student gap (8.8 to 4.8 points) and 70% of the same-teacher gap (15.8 to 4.8), a substantial recoverable component, while the attribution itself stays descriptive. The limitations say the same and add that forcing recovers nothing on Countdown.
Changes: Section 3.2; Section 5.3 (matched share next to the 84%); Appendix L.1 and Table 26 (both protocols, each run's step in parentheses); limitations (Section 6).
Thank you.
1. The attribution is accounting, not causation. Agreed. To bound what overrun hides, we added answer forcing (Section 3.2, Appendix C.2, Table 7).
We also corrected Section 5.3: a reasoning advantage now "should show at least partly as fewer wrong answers among answered problems"; the section states that both measures condition on answering, and that forcing cannot tell whether the remaining 4.8 points reflect weaker reasoning or loops that used up the canvas.
2. The isolation experiment: style, one run per arm, one budget.
3. The s1K confounds. We ran the control (Section 5.3, Appendix L.1).
With size, prompt and schedule ruled out on seed 42, domain and trace length remain. The repaired s1K is 1.4 points behind: overrun costs it 3.1, offset by wrong answers (1.5) and stopping (0.2); its traces still exceed the canvas.
4. Validating the corpus screen. We agree with all four points:
Questions.
We agree with your reading of the contribution: it is diagnostic and analytical. The introduction and Related Work state its relation to IGPO.