- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies redundancy in long CoT traces generated by reasoning models and proposes SCRATCH, a two-stage SFT framework for learning to produce shorter reasoning traces. In the first stage, Scratch Trace Construction (STC), the target model generates solutions to training problems, and only responses with verified correct final answers are retained. Each accepted CoT is divided into units at paragraph and numbered-step boundaries. GPT-4o-mini receives the question, the verified answer, and all numbered units, and selects those intended to preserve the key calculations, conditions, and logical connections while removing redundant explanations and unnecessary branches. In the second stage, Progressive Compression Curriculum (PCC), the model is fine-tuned on the compressed traces, starting with less-compressed examples, progressively introducing more-compressed ones, and ending with a consolidation stage focused on highly compressed examples. Across three models and three mathematical benchmarks, the authors report an average reduction of 24.87% in generated tokens and an average accuracy change of −0.36 percentage points relative to the original models.
Critical strengths
S1. SCRATCH reduces output length across multiple models and math benchmarks with small average accuracy changes. The shorter-generation behavior also transfers to non-math tasks without additional fine-tuning.
S2. Configurations E and F hold training targets, batch contents, per-example exposure, and the final consolidation stage fixed, changing only the order of earlier batches. This provides a relatively clean test of progressive ordering.
S3. SCRATCH builds compressed supervision offline and uses standard LoRA SFT. Inference requires no architectural changes or additional compression model, simplifying integration into existing pipelines.
Critical weaknesses
W1. The paper reports mean results from five decoding samples per question but provides no standard errors or confidence intervals. Given the modest gains in the PCC ordering ablation, paired confidence intervals or bootstrap estimates based on the existing evaluation outputs would help assess the reliability of these differences without additional model training.
W2. A straightforward alternative is to sample multiple solutions per problem, select the shortest correct response, and use it for SFT, as explored by Munkhbat et al. (2025). The paper does not compare STC with this baseline, leaving its incremental benefit unclear.
W3. SPIRIT (Cui et al., 2025) already prunes reasoning steps to construct concise supervision, while Train Long, Think Short (Hammoud et al., 2025) uses a curriculum to encourage progressively shorter reasoning. Although their selection criteria and training algorithms differ from SCRATCH, the paper should clarify what is novel beyond combining these established ideas and why its specific design matters.
Questions for authors
Q1. The original MATH dataset (2021) contains historical AIME problems, whereas evaluation covers AIME 2020–2025. Have the authors checked the 1,000 selected MATH training problems for exact or near-duplicate overlaps with evaluation questions, particularly AIME 2020? Please report the results.
Q2. Steps unnecessary for a particular correct solution may still be useful for learning, such as attempts that illustrate errors or revisions. How do the authors view this potential difference between what is needed for the final answer and what may benefit the model during training?
Q3. Could removing unsuccessful attempts reduce strategy diversity or backtracking ability? Pass@5 measures solution coverage rather than path diversity. Could the authors analyze alternative solution strategies and error recovery before and after training?
Q4. The reported mean token length includes both correct and incorrect responses. Could the authors report token lengths and reductions separately for these two groups, to clarify whether the overall savings come mainly from correct or incorrect responses?
3 comments
Response 2 of 2
Q1. Overlap with AIME
We verified that the entire selected pool of 2,000 questions used for training and development (1,000 GSM8K and 1,000 MATH) has no overlap with the 180 AIME 2020–2025 evaluation problems, including all 30 AIME 2020 problems.
Q2. What the answer needs versus what helps learning
Thank you for highlighting the learning value of errors and revisions. We distinguish a complete response that ends in a wrong answer from useful unsuccessful attempts within a verified-correct response. SCRATCH excludes the former from positive SFT targets. Using such responses as rejected examples in direct preference optimization (DPO) is a direction for future work.
Within a verified-correct response, STC asks the annotator to retain unsuccessful attempts and corrections that supply useful assumptions, conditions or information needed later. In an expert audit of 50 native traces across the three backbones, the default annotator retained 60 of 62 useful error-correction units and 157 of 160 critical-condition units. Q3 examines the related question directly: whether the trained model still finds different correct strategies and recovers from errors.
Q3. Strategy diversity and error recovery
We tested two abilities on 50 questions (25 from MATH and 25 from AIME). Question selection did not depend on either model's outputs, and evaluators were not told which model produced each answer.
- Diversity of correct strategies. For each question, we examined five answers from each model and grouped the correct ones by mathematical approach. Rewordings of the same approach count as one strategy. The table reports the number of questions with at least two distinct correct strategies.
- Recovery from errors. For each question, we prepared two partial solutions that contain a recoverable error, for 100 in total. Baseline and SCRATCH received the same 100 starts. A recovery counts as successful only if the model corrects the error, completes a valid derivation, and reaches the correct answer.
Table entries show Baseline → SCRATCH.
| Model | Questions with ≥2 correct strategies (of 50) | Successful recoveries (of 100) |
|---|---|---|
| DeepSeek-1.5B | 11 → 10 | 61 → 60 |
| DeepScaleR-1.5B | 17 → 16 | 69 → 66 |
| Qwen3-14B | 30 → 29 | 83 → 82 |
On this sample, the counts remain close to Baseline, differing by one multi-strategy question and one to three successful recoveries per model.
Q4. Correct and incorrect response lengths
Both correct and incorrect responses become shorter in all nine model–benchmark settings, with reductions for correct responses ranging from 14.50% to 36.35%. DeepSeek-1.5B on MATH-500 has the largest reduction for correct responses:
| Response group | Baseline tokens | SCRATCH tokens | Reduction |
|---|---|---|---|
| Correct | 3582.5 | 2280.1 | 36.35% |
| Incorrect | 7493.9 | 4431.1 | 40.87% |
Averaged over the nine settings, shorter correct responses account for about 65% of the overall token saving. Correct responses contribute more in seven of the nine settings; the exceptions are the two 1.5B models on AIME.
The new comparisons help explain how intact reasoning units and PCC contribute to SCRATCH. Your questions also led us to examine useful corrections and the model's behavior after compression, beyond average accuracy and length. Please let me know if any additional experiments or analyses might help you reconsider our manuscript.
I appreciate the authors’ response, which addresses some of my concerns, and I am maintaining my original rating.
Response 1 of 2
We appreciate your recognition of SCRATCH's efficiency, controlled ordering comparison, and simple deployment. Your questions about its contribution beyond prior work and the learning value of intermediate attempts helped us focus the additional experiments. We have added the suggested comparisons and looked more closely at which reasoning units are retained, as well as strategy diversity and error recovery after training.
W1. Stability across five evaluation runs
Each cell gives Baseline / SCRATCH over the five evaluation runs in Table 1. Accuracy standard deviation (SD) is expressed in percentage points. For tokens, relative SD (RSD) is the SD of the five run-level means divided by that method's own mean, expressed as a percentage.
Accuracy SD (percentage points).
Token RSD (%).
Across all nine settings, every SCRATCH run uses fewer tokens on average than every Baseline run.
We also assessed training variation with three independent trainings per configuration on DeepSeek-1.5B. Averaging the trainings and benchmarks equally, Full SCRATCH has 0.84 points higher accuracy and 4.43% fewer tokens than STC + prefix shuffle + consolidation, and 18.71% fewer tokens than STC + Standard SFT with a −0.08-point accuracy change. On DeepScaleR-1.5B and Qwen3-14B, it is more accurate and shorter than the prefix-shuffle control in all six settings. These results agree with PCC's intended role: stronger supervision for compact reasoning, with accuracy largely preserved.
W2/W3. Other methods
We compared SCRATCH with shortest-correct SFT (Munkhbat et al., 2025) and adaptations of SPIRIT-FT (Cui et al., 2025) and TLTS (Hammoud et al., 2025) on DeepSeek-1.5B, given the compute needed for GRPO. All methods use the same 1,800-question pool, evaluation settings, and output caps. Optimizer updates are matched, while TLTS also generates responses during online reinforcement learning. Each row is one trained model evaluated with five answers per question. Cells give accuracy (%) / mean tokens.
Full SCRATCH is shortest on all three benchmarks, more accurate than both adaptations throughout, and most accurate on MATH-500 and AIME. Our shortest-correct control is the Naive BoN variant of Munkhbat et al. (2025): it keeps the shortest verified-correct response among 16 samples, covering 1,769 of 1,800 questions. Their full recipe also adds few-shot conditioning and sample augmentation. Whole-response selection leaves any redundancy inside the chosen answer; STC selects within a response. Under the same Standard SFT, STC targets are more accurate on all three benchmarks, with somewhat longer outputs. With PCC, Full SCRATCH averages 1.49 points higher accuracy and 16.29% fewer tokens than the shortest-correct SFT.
What distinguishes SCRATCH. SPIRIT-FT uses the target model's perplexity to select removals and merges steps with an LLM when needed. STC instead selects computations, conditions, and dependencies while preserving the retained units verbatim. A matched control uses GPT-4o-mini to rewrite the same native traces concisely. STC + Standard SFT is 2.39 points more accurate on average than Direct Rewrite + Standard SFT. This suggests a benefit from keeping the model's own expressions, rather than constructing new ones.
TLTS applies a decreasing absolute token-budget target during online GRPO, with rewards for correctness, length, and format. PCC places the curriculum in offline supervised training: targets stay fixed, while their frequency and ordering change according to relative compression, followed by consolidation. SCRATCH pairs verbatim native-unit selection with this offline curriculum. The comparisons and ablations support both choices. In the final version, we will clarify these distinctions in Related Work.