- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ROAST, a reinforcement-learning framework for reducing excessive reasoning. ROAST uses an online suffix automaton during training rollouts to detect severe structural repetition and terminate the reasoning phase, and combines this with token-level advantage reshaping that favors efficient correct responses while selectively down-weighting detected redundant suffixes. The method is evaluated on several mathematical reasoning benchmarks across multiple Qwen3 and DeepSeek-R1-Distill models, with additional OOD evaluation on GPQA. Importantly, the SAM intervention is disabled at evaluation time, so the reported test-time shortening reflects behavior learned by the post-trained policy rather than inference-time truncation.
Critical strengths
The paper identifies a real and potentially expensive failure mode in reasoning models: some trajectories enter severe repetitive loops and consume many tokens after useful reasoning has effectively finished. The idea of localizing the efficiency penalty to a detected redundant suffix is more compelling than applying a generic length penalty to all reasoning, since legitimate difficult problems may genuinely require long solutions.
I also appreciate that SAM is disabled during evaluation while the trained models remain shorter. This provides useful evidence that the method changes the learned policy rather than obtaining its gains simply through a test-time truncation rule.
Critical weaknesses
My main concern is that the headline compression relative to the base model obscures the smaller incremental gain over the more relevant GRPO baseline. For example, on Qwen3-4B, GRPO already substantially reduces reasoning length, and ROAST provides roughly a further 18--20% reduction with only a small accuracy change. This can still be practically useful, but it makes the incremental contribution considerably more modest than the base-model-relative compression numbers suggest.
More importantly, the ablation raises a question about what actually drives this gain. SAM alone provides only a small average length reduction, while the efficiency anchor accounts for much more of the compression. This suggests that the novel structural-repetition mechanism may function mainly as a stabilizer for more conventional efficiency shaping rather than being the primary source of the efficiency improvement. In addition, SAM detects a relatively narrow form of overthinking based on structural repetition, and the intervention rate appears lower for the stronger models in the study. Since the largest evaluated model is only 7B, I am not yet convinced that this failure mode remains sufficiently prevalent at larger scales or that the method covers overthinking more generally.
Questions for authors
-
Could the authors report results on substantially larger reasoning models and analyze SAM intervention frequency as a function of model capability and problem difficulty?
-
Could the authors clarify the causal contribution of SAM relative to the efficiency anchors? In particular, given the relatively small compression from SAM alone, is the main role of SAM better understood as stabilizing efficiency-oriented training rather than directly removing most redundant reasoning?
-
Do the authors have an estimate of SAM precision and recall against human-labeled overthinking, including semantically redundant but non-repetitive reasoning? This would help establish how much of the broader overthinking problem is actually captured by the proposed detector.
5 comments
Response to Reviewer 2RPf (2/2)
Q2: Contributions of SAM and the efficiency anchors
SAM makes training with the efficiency anchors more effective. It stops repetitive rollouts and identifies the suffix to penalize, while the anchors favor correct responses that are shorter or use fewer reflection steps.
Training with and without SAM.
Both Qwen3-4B runs enable both anchors; only Full ROAST uses SAM during training rollouts. Validation uses SAM off, one response per question, and an 8k cap. We report checkpoints through step 320 on a doubling grid; cells show accuracy (%) / mean reasoning tokens.
| Step | AIME24 w/o SAM | AIME24 Full ROAST | GSM8K w/o SAM | GSM8K Full ROAST |
|---|---|---|---|---|
| 0 | 6.67 / 7,304.2 | 3.33 / 6,818.0 | 90.37 / 1,020.9 | 90.07 / 904.3 |
| 40 | 50.00 / 6,144.6 | 53.33 / 6,112.7 | 92.87 / 673.3 | 93.78 / 665.7 |
| 80 | 53.33 / 5,838.8 | 53.33 / 5,206.3 | 92.27 / 515.5 | 93.33 / 457.6 |
| 160 | 53.33 / 5,244.0 | 53.33 / 4,867.1 | 92.80 / 473.3 | 94.62 / 392.2 |
| 320 | 50.00 / 5,468.0 | 50.00 / 4,897.6 | 94.62 / 455.9 | 94.62 / 455.9 |
On AIME24, SAM saves 7.19–10.83% of reasoning tokens at equal accuracy at steps 80, 160, and 320. On GSM8K, step 80 saves 11.23% with a 1.06 pp accuracy gain; step 160 saves 17.14% with a 1.82 pp gain. Both GSM8K metrics match at step 320.
Contribution of the suffix penalty.
The w/o Penalty control keeps SAM and both anchors on (1TaD W1). At step 100, adding the penalty saves 17.95% of GSM8K reasoning tokens with an accuracy change of −0.08 pp, and 1.18% of AIME24 tokens at equal accuracy.
Single-anchor ablations.
In Table 3, adding SAM and its suffix penalty to either anchor improves accuracy and shortens outputs. SAM Only includes the penalty and neither anchor.
| Qwen3-4B comparison | Mean token change | Accuracy Δ (pp) |
|---|---|---|
| Length anchor vs GRPO-8k | −633 | −0.96 |
| SAM Only vs GRPO-8k | −87 | +0.48 |
| Length anchor + SAM vs length anchor | −190 | +1.28 |
| Step+SAM vs Step | −327 | +1.25 |
Q3: SAM precision and recall
Human annotations.
We use three sets:
- AMC23 and MinervaMath: 1,872 responses from six base models, manually annotated by two experts for redundant steps, to evaluate threshold sensitivity.
- DeltaBench: 1,235 error-selected responses with human annotations of unhelpful reasoning; 755 have positive or negative labels and 480 are undetermined.
- ReasoningFlow: 31 traces manually annotated by two experts for restatements, including nonverbatim redundancy.
Threshold sensitivity.
The default threshold of 120 gives the highest precision in this sweep, 93.14%–96.08%. Lower thresholds increase recall and F1. We vary only the repeat threshold on the 1,872 math responses, keeping the window at 200 tokens and density at 0.85. A response is positive if it contains at least one human-labeled redundant step. The ranges account for 24 undetermined labels (Appendix J.5).
| Repeat threshold | Responses with an alarm | Sample precision | Sample recall | Sample F1 |
|---|---|---|---|---|
| 120 | 102 | 93.14%–96.08% | 8.32%–8.72% | 0.153–0.160 |
| 90 | 158 | 91.14%–93.67% | 12.62%–13.16% | 0.222–0.231 |
| 60 | 359 | 87.74%–90.25% | 27.73%–28.67% | 0.421–0.435 |
| 45 | 617 | 83.95%–86.71% | 45.92%–47.01% | 0.594–0.610 |
| 30 | 1,030 | 74.56%–76.60% | 68.33%–69.09% | 0.713–0.727 |
On 240 AIME26 pairs from two 14B models (window 200), threshold 120 changed 2 correct answers to wrong, versus 19 at threshold 30. This supports our default's emphasis on precision and avoiding premature termination. The paper's Table 4 shows the same trade-off: (120, 200) cuts 19.5% of responses with 0 correct-to-wrong changes; (60, 100) cuts 58.1% with 41.
DeltaBench and nonverbatim redundancy.
At threshold 120, DeltaBench gives 84.3–85.2% response-level precision and 7.7–9.3% recall on the 755 labeled responses. Both tokenizers process the same traces; about half of the triggered responses have undetermined labels. On ReasoningFlow, SAM detects 2 of 22 positive traces. Among 18 traces with nonverbatim restatements of at least 11 tokens, its first window overlaps none of those spans. Appendix J.1 reports the full detection and window-overlap results.
References
[1] He, Y., et al. (2025). Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning? arXiv:2502.19361.
[2] Lee, J., et al. (2026). ReasoningFlow: Discourse Structures for Understanding LLM Reasoning Traces. arXiv:2606.05402.
Inquiry Regarding Discussion Phase Follow-Up
Dear Reviewer 2RPf,
Thank you again for your constructive feedback. As the discussion period draws to a close, we would be grateful if you could revisit our responses.
To address your concerns, we provided preliminary Qwen3-14B training results at step 60, additional controls examining the contributions of SAM and the suffix penalty with both efficiency anchors enabled, and sample-level precision, recall, and F1 measured against human annotations across repetition thresholds. We also clarified the incremental gains over GRPO-8k and SAM’s limited coverage of nonverbatim redundancy.
We would particularly appreciate your assessment of the additional evidence on model scale, SAM’s contribution during training, and detector reliability. Please let us know if any concerns remain or require further clarification.
Thank you again for your time and consideration.
Best regards,
The Authors of Paper 25398
Response to Reviewer 2RPf (1/2)
W1: Additional gains over GRPO-8k
ROAST improves accuracy over GRPO-8k while using substantially fewer tokens: mean math output length fell by 21.82% on Qwen3-1.7B and 18.50% on Qwen3-4B, with accuracy gains of 4.46 and 0.60 pp, respectively. Both trained models used standard decoding with SAM off (Table 1).
The math results come from Table 1 and the GPQA-Diamond results from Appendix D.
On Qwen3-4B, ROAST reduced end-to-end training time on two A800 GPUs from 66.5 h (GRPO-8k) to 59.8 h (10.1% less).
Q1: Model scale and problem difficulty
Preliminary 14B training results.
Qwen3-14B training has reached step 60. At this checkpoint, ROAST used 22.43–40.72% fewer tokens than the base model and achieved higher accuracy on all four math benchmarks. Both were evaluated using standard decoding with SAM off.
The early training trend is similar at 4B and 14B. From step 0 to 60, AIME24 reasoning length fell by 10.68% and 12.13%, with accuracy gains of 36.67 and 40.00 pp, respectively.
Trigger frequency depends on the task.
Larger models triggered less often on GPQA, while Qwen3-4B and 14B triggered similarly on code. We compare counts over equally long reasoning prefixes for each question within a model family.
On the 81 GPQA questions with expert difficulty ratings, trigger frequency showed no consistent trend with difficulty.
SAM applied to 14B base models.
SAM also shortened outputs when applied directly to the Qwen3-14B and DeepSeek-R1-Distill-Qwen-14B base checkpoints. The table combines both models and compares standard decoding with SAM enabled. Code uses 142 tasks with validated unit tests per model and threshold 120; math uses threshold 60 for higher coverage (Q3). Both keep the window at 200 tokens and density at 0.85.
Fixed-cap comparison.
Against an 8,192-token reasoning cap on the same math pairs, SAM used 2.55% more output tokens and confirmed correct counts went from 904 to 906.