acceptodds
Under review as a conference paper at ICLR 2027

Not All Short Chains Are Equal: How CoT Compression Affects SFT and RL

Abstract

Chain-of-thought (CoT) compression has attracted growing interest as a way to reduce training and inference costs. However, the compression ratio and accuracy may not fully reflect the value of compressed trajectories as training data. Similar compression ratios may lead to different outcomes in supervised fine-tuning (SFT) and subsequent reinforcement learning (RL). To investigate this, we conduct an end-to-end study on mathematical reasoning, comparing uncompressed data, prompt-based Chain-of-Draft compression, and self-distillation-based CRISP compression. On Qwen3-8B, SFT on Chain-of-Draft data yields 12.78 percentage points higher AIME accuracy than SFT on CRISP data. Its AIME advantage is concentrated on problems that the starting model can solve occasionally but inconsistently. In subsequent RL, Chain-of-Draft retains higher final accuracy than CRISP. These findings suggest that the value of short-CoT training data depends not only on response length, but also on how well it preserves the model's ability to solve problems it initially solves inconsistently. Since SFT also provides the initialization for RL, evaluating this training value should consider both immediate SFT outcomes and final performance after RL.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Chain-of-Thought Compression, Reasoning Data Quality, Supervised Fine-Tuning, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HqWfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how different CoT compression methods affect training outcomes in SFT and subsequent RL. It compares uncompressed trajectories with Chain-of-Draft and CRISP trajectories of similar average length. Chain-of-Draft achieves higher SFT and post-RL accuracy than CRISP, particularly on problems that the base model solves inconsistently, and the paper further analyzes differences in the reasoning content retained by the two methods.

Critical strengths
  1. CoD outperforms CRISP by 12.78 percentage points on AIME after SFT with Qwen3-8B, despite the two datasets having similar average response lengths.

  2. The paper evaluates the effects of compressed CoT data through both SFT and subsequent RL, and CoD maintains higher post-RL accuracy than CRISP on Qwen3-8B and Qwen3-4B.

Critical weaknesses

The CoD and CRISP datasets are not fully matched. They differ in the generating model, candidate-selection procedure, target correctness, and final dataset membership. Consequently, these factors may also contribute to the observed performance differences.

Questions for authors
  1. CoD and CRISP differ in their generating models, candidate-selection procedures, target correctness, and final dataset membership. How can the authors determine how much of the performance gap is caused specifically by the reasoning content retained during compression?

  2. The main SFT and RL experiments use only one training seed. Are the reported differences between CoD and CRISP stable across multiple training runs, particularly the 2.08-point post-RL gap on Qwen3-8B?

  3. All SFT variants underperform the corresponding base models, and the SFT-initialized RL models underperform Direct RL in the reported Qwen experiments. How should the practical benefit of compressed-CoT SFT be interpreted under these results?

3 comments

Response (Part 1/2)

Thank you for your careful review and comments. We address your questions below.

1. Differences between the CoD and CRISP datasets

Both methods start from Qwen3-8B. CoD uses concise prompting, whereas CRISP first updates the model through OPSD and then generates SFT data. The change in weights is part of CRISP, not a difference in the initial backbone.

Rejection sampling with answer verification retains 59,101 verifier-approved, non-truncated CRISP responses. Since the CRISP data have already been filtered for correctness, we construct CoD data on these same questions. The goal is to match target correctness as closely as possible while preserving common question coverage. In each round, we retain correct, non-truncated responses. Only questions without an accepted response proceed to the next round. Generation stops for a question as soon as an accepted response is obtained.

For questions still without a verifier-approved response after four rounds, we retain the shortest non-truncated response as a fallback. This avoids changing the question set by removing these questions. Only 224 questions with all candidates truncated are excluded (0.38%). CoD retains 99.62% of the common question set, keeping dataset sizes closely matched.

CoD has a lower proportion of verifier-approved targets but better mathematical SFT and subsequent RL results. Its advantage therefore cannot be explained simply by a higher proportion of verified targets.

The common initial backbone and core training settings, together with closely matched question coverage and dataset sizes, aim to reduce influences other than reasoning content.

Section 5.2 and Appendix B.3 provide additional evidence through a deletion-position ablation (Figure 4b, Table 8). Both conditions use the same 4,096 Raw responses, initial model, and SFT settings, with matched deletion amounts per example. Removing different parts still yields different training outcomes, supporting the importance of retained content.

Response (Part 2/2)

2. Robustness across random seeds

We use one training seed and do not yet have multi-seed RL results. Repeating the full pipeline is costly, so we prioritized comparisons across models and SFT training depths under limited sources.

Although we do not use multiple training seeds, we broadened the evaluation along several dimensions to examine consistency and generality. The mathematical SFT experiments cover Qwen3-8B, Qwen3-4B, and Olmo-3-7B-Think. CoD outperforms CRISP by 12.78, 8.06, and 7.22 percentage points, respectively. Both Qwen models retain this ordering after RL, as do all four tested SFT initialization depths on Qwen3-8B. These provide consistency checks across models and initializations. Eight responses per AIME problem also reduce single-generation variability across the 90 problems.

3. Practical benefit relative to Base and Direct RL

The current results do not establish that these SFT-to-RL configurations outperform Direct RL. Our main goal is to analyze how different compressed datasets affect SFT and subsequent RL. Their benefits should be interpreted in terms of capability preservation, response length, and final training outcomes.

One possible explanation is that offline SFT trajectories do not fully match the strong base models' existing reasoning behavior or the evaluation demands. Fitting these trajectories may alter that behavior. Subsequent RL starts from different policies, and a shared budget need not remove the effects of SFT initialization. These explanations require targeted validation.

CoD nevertheless preserves capabilities better than Raw supervision in this setting. On Qwen3-8B, it reduces the accuracy drop relative to Base from 18.20 percentage points with Raw-SFT to 4.03 points. Thus, while CoD does not exceed Base, it substantially mitigates the decline seen with Raw supervision under the current setup.

The results also show a trade-off between accuracy and response length. As reported in Appendix Table 9, Qwen3-8B CoD-SFT generates an average of 12,618 tokens on the evaluation set, compared with 15,672 for Base. It uses fewer output tokens while retaining accuracy relatively close to Base. This offers a potential inference-efficiency benefit, despite the modest loss in accuracy.

Thank you again for your suggestions. We will strengthen the controlled comparisons and stability check in future work.

Thanks for the detailed response. The clarification on dataset construction and the deletion-position ablation address most of my concern about the CoD–CRISP comparison. I still think multi-seed RL results would be useful, but the consistent trends across models and SFT depths provide some reassurance. The practical role of compressed-CoT SFT is also clearer now. I will raise my rating.

Reviewer_tZpoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores how Chain-of-Thought (CoT) compression affects Supervised Fine-Tuning (SFT) and subsequent Reinforcement Learning (RL). By comparing several approaches—"Raw," "Chain-of-Drafts" (CoD), and CRISP—across multiple model families, the authors demonstrate that even when response lengths are similar, the training outcomes can differ significantly. Further analysis reveals that preserving intermediate reasoning steps is essential for maintaining a model's ability to solve complex, challenging problems.

Critical strengths

Relevant research questions. This paper goes beyond focusing solely on compression ratios; it also evaluates the downstream utility of compressed reasoning trajectories within the contexts of supervised fine-tuning (SFT) and reinforcement learning (RL).

Comprehensive empirical analysis. The experiments encompass a variety of models and tasks, supplemented by in-depth analyses of specific issues. Furthermore, ablation studies—in which the positions of removed elements are controlled—are conducted to investigate the underlying causes of the observed differences.

Critical weaknesses

The training targets have limited comparability. While all targets in CRISP are human-verified, CoD includes 9,395 "fallback" targets that have not undergone human verification. Due to differences in the candidate selection process and the distribution of target correctness, it is difficult to determine whether performance gaps stem from the compression strategy itself or from the specific reasoning content that has been retained.

Questions for authors
  1. To what extent is the performance gap between CoD and CRISP attributable to the compression strategy itself, rather than differences in the correctness of the targets or the selection of candidates? Would it be possible to present the results of a controlled experiment comparing both methods under the condition that they share the same set of targets—those deemed valid by a verifier?

  2. How ​​robust is the reported performance advantage across different random seeds? Specifically, would it be possible to provide reinforcement learning (RL) results using multiple random seeds, along with an assessment of the associated uncertainty, to verify that the observed performance differences and the ranking of the checkpoints remain consistent?

3 comments

Response (Part 1/2)

Thank you for your careful review and for recognizing our work. We will answer your questions below.

1. Target correctness and candidate selection

First, we would like to clarify that CRISP targets are verified with MathVerify, an evaluating system, not by humans. CoD uses the same verifier.

We generate CRISP responses with Qwen3-8B after OPSD training. Rejection sampling with answer verification retains 59,101 verifier-approved, non-truncated responses.

Since the CRISP data have already been filtered for correctness, we construct CoD data on these same questions. The goal is to match target correctness as closely as possible while preserving common question coverage. In each round, we retain correct, non-truncated responses. Only questions without an accepted response proceed to the next round. Generation stops for a question as soon as an accepted response is obtained.

For questions still without a verifier-approved response after four rounds, we retain the shortest non-truncated response as a fallback. This avoids changing the question set by removing these questions. Only 224 questions with all candidates truncated are excluded (0.38%). CoD retains 99.62% of the common question set, keeping dataset sizes closely matched.

CoD has a lower proportion of verifier-approved targets but better mathematical SFT and subsequent RL results. Its advantage therefore cannot be explained simply by more verified targets. The shared questions, similar dataset sizes, and common core training settings improve comparability.

Your suggested controlled comparison on a common verifier-approved subset would help examine the effects of target correctness and candidate selection further. Our current experiments prioritize common question coverage and do not include training on this subset. We plan to consider it as an extension of the controlled experiments to further study how these factors relate to training outcomes.

Response (Part 2/2)

2. Robustness across random seeds

We use one training seed and do not yet have multi-seed RL results. Repeating the full pipeline is costly, so we prioritized comparisons across models and SFT training depths under limited resources.

Although we do not use multiple training seeds, we broadened the evaluation along several dimensions to examine consistency and generality. The mathematical SFT experiments cover Qwen3-8B, Qwen3-4B, and Olmo-3-7B-Think. CoD outperforms CRISP by 12.78, 8.06, and 7.22 percentage points, respectively. Both Qwen models retain this ordering after RL, as do all four tested SFT initialization depths on Qwen3-8B. These provide consistency checks across models and initializations. Eight responses per AIME problem also reduce single-generation variability across the 90 problems.

We plan further controlled comparisons and stability checks to test the scope of these observations. Thank you again for your suggestions.

Thanks for your rebuttal, and I will maintain my positive score.

Reviewer_7gmSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares uncompressed Raw trajectories, prompt-based Chain-of-Draft (CoD), and self-distillation-based CRISP as training data for supervised fine-tuning (SFT) and subsequent reinforcement learning (RL). Despite similar average lengths, CoD substantially outperforms CRISP after SFT on Qwen3-8B and retains a smaller advantage after RL. The gap is mainly associated with problems that the base model solves inconsistently, while a deletion ablation suggests that retaining intermediate reasoning may be more useful than retaining post-thinking exposition. The paper argues that short-CoT data should therefore be evaluated by the reasoning content and capabilities they preserve, rather than by compression ratio alone.

Critical strengths
  1. The paper distinguishes the quality of a compressed trajectory as a generated response from its value as training supervision. Evaluating compressed CoT data through both SFT and subsequent RL is a useful perspective, since response length and correctness alone do not determine downstream training outcomes.

  2. The problem-level grouping and state-transition analysis help locate the CoD–CRISP gap on problems that the base model solves inconsistently. Tracking multiple SFT initializations through a fixed RL budget also shows that immediate SFT accuracy does not fully predict the final RL outcome. These analyses provide useful empirical evidence, although they do not establish the proposed mechanism causally.

Critical weaknesses
  1. The main evidence comes from only two compression methods and models within a narrow 4B–8B scale range, with most analyses centered on Qwen3 and mathematical reasoning. There is no validation on substantially larger models, and the ordering changes across models on code generation. CoD also does not consistently outperform Raw, while none of the SFT-to-RL pipelines outperforms Direct RL. The results therefore establish a setting-specific CoD–CRISP difference rather than a general property of short-CoT training data.

  2. CoD and CRISP differ in their generation models, prompts, number of candidates, selection procedures, and target-correctness distributions. Consequently, the performance gap cannot be cleanly attributed to the compression strategy or to retaining more intermediate reasoning. A comparison using matched generators, correctness, candidate selection, length distributions, and training budgets would be needed to isolate the claimed effect.

  3. The claim that equally short trajectories can have different value, or that useful intermediate reasoning should be retained instead of repetitive exposition, is relatively intuitive. The proposed preservation of success on uncertain problems is a retrospective diagnostic that requires repeated sampling and downstream training, rather than a usable data-selection criterion. It is also inferred indirectly from response length and eight-sample success states. With one training seed and only 90 AIME problems, the paper does not establish either the proposed exploration mechanism or the robustness of the observed transitions.

Questions for authors
  1. Do the conclusions remain valid across a broader set of CoT-compression baselines, substantially larger model scales, and additional reasoning domains under matched response lengths and training budgets?

  2. Can the authors construct a more tightly controlled comparison in which the generation model, target correctness, number of candidates, selection procedure, and length distribution are matched? This would help isolate the effect of the compression strategy and the location of retained content.

  3. Can the proposed notion of preserving success on uncertain problems be converted into a data-quality metric or selection rule that is available before full SFT and RL? How stable is this signal across training seeds, independent grouping samples, larger evaluation sets, model families, and downstream tasks?

3 comments

Response (Part 1/2)

Thank you for your careful review and comments. We will respond to your questions below.

1. Comparability of the experimental settings

(i) Generating model. Raw uses the original responses. Both compression methods start from Qwen3-8B: CoD generates concise responses through prompting, while CRISP first updates the model through OPSD and then generates SFT data. The change in generator weights is part of CRISP, not a difference in the initial backbone.

(ii) Prompts. The CRISP prompts in the appendix are the teacher and student prompts used during OPSD training. The CoD prompt is used for compressed response generation. Their differences follow from the methods themselves, since prompting is CoD's compression mechanism.

(iii) Target correctness, number of candidates, and selection procedure. We generate CRISP responses after OPSD training and use rejection sampling with answer verification to retain 59,101 correct, non-truncated responses.

On the same question set, CoD uses up to four verifier-guided generation rounds to match target correctness as closely as possible while preserving question coverage. We retain the first verifier-approved, non-truncated response and retry only questions without one.

If all four rounds fail verification, we retain the shortest non-truncated response as a fallback to avoid changing the question composition. Only 224 questions with all candidates truncated are excluded (0.38%). CoD retains 99.62% of the common question set, keeping dataset sizes closely matched.

CoD has a lower proportion of verifier-approved targets than CRISP but achieves better mathematical SFT and subsequent RL results. Its advantage therefore cannot be explained simply by a higher proportion of verified targets. The shared question set, verifier-guided retries, similar dataset sizes, and common core training settings improve comparability.

(iv) Length distribution. Different compression strategies naturally produce different response-length distributions, which may affect training. Our main comparison evaluates the data produced by each method, so we retain these differences and report their distributions in Appendix B.1.

Response (Part 2/2)

We continue with the scope of our findings and the stability of the analysis.

2. Compression baselines model scale and reasoning domains

Compression baselines. CoD and CRISP represent two distinct routes to compressed trajectories: prompting and self-distillation, respectively. We prioritized an in-depth study of tracing their effects through SFT and subsequent RL, including different SFT initializations, problem-level diagnostics, and controlled ablations. This provides depth, although coverage of additional baselines remains limited.

Model scale. Our mathematical SFT experiments cover Qwen3-8B, Qwen3-4B, and Olmo-3-7B-Think. CoD exceeds CRISP by 12.78, 8.06, and 7.22 percentage points on this models, respectively. This supports consistency across the tested scales and families. Given the cost of repeated SFT and RL runs and the widespread use of these models, we prioritized these capable 4B-8B models. Substantially larger models remain to be tested.

Reasoning domains. Our study focuses on mathematical reasoning, and all training uses mathematical data. Code generation serves as an additional evaluation of cross-domain transfer; no code data are used for training. Section 4.1 and Appendix E report that this transfer is model-dependent: the mathematical advantage does not consistently extend to code generation across models.

3. Selection metrics and stability of the analysis

Selection metrics. This study focuses on analyzing how compressed CoT data affect SFT and subsequent RL. It does not aim to introduce a data-quality metric or selection algorithm available before training. "Preserving success on uncertain problems" is a diagnostic: we group problems by the initial model and track accuracy and sampled success states after SFT.

It requires SFT evaluation, but not completed RL. We agree that turning these findings into a metric that predicts training value in advance would be useful. We plan to investigate this through further experiments.

Stability. We use one training seed. Appendix C.4 nevertheless shows fewer mixed-to-0/8 transitions with CoD than CRISP on Qwen3-4B and Olmo-3-7B-Think, providing support across models.

We also draw eight responses for each of the 90 AIME problems to reduce single-generation variability. Further stability checks remain future work.

Thank you again for your detailed comments. We hope these responses clarify our experimental design, data construction, and analytical goals. We also plan to test more compression methods, larger models, and additional reasoning tasks, and explore data-quality metrics available before training in future work.

We would like to thank the reviewer again for the thoughtful and constructive comments. In our responses, we clarify the CoD and CRISP data construction procedures, explain the controls used in the comparison, and discuss the scope and stability of our findings.

If time permits, we would appreciate your feedback on whether these explanations address your main concerns. We welcome any remaining questions and would be happy to clarify them during the discussion period. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.