- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Absorbability-Guided CoT Compression (AGC²), which selects compressed reasoning supervision according to whether the current model can learn from it while preserving accuracy. For each training batch, the method evaluates different token budgets through short local SFT updates and retains the shortest acceptable candidate. Experiments on three model scales and three mathematical benchmarks show substantial reductions in response length while maintaining or improving accuracy in most settings.
Critical strengths
-
The paper introduces a meaningful model-dependent view of CoT compression. Instead of selecting compression levels only from trajectory length or token importance, AGC² tests whether the current model can actually learn from each compressed candidate through short local adaptation.
-
AGC² achieves a strong accuracy–efficiency trade-off across 7B, 14B, and 32B models. It reduces response length by 19.2%–67.3% while matching or exceeding the original model’s accuracy in seven of nine model–benchmark settings.
Critical weaknesses
-
The absorbability score is evaluated on the same batch from which the answer-verified SFT examples are drawn. This partial train-evaluation overlap may favor candidates that fit the current batch rather than measuring generalizable absorbability.
-
The computational overhead of AGC² is not sufficiently evaluated. For each training batch, the method may construct multiple compressed candidates, perform separate SFT updates from fresh model copies, and evaluate each adapted model on the full batch. However, the paper does not report the total training time, GPU cost, or comparison with the training cost of the baselines.
Questions for authors
-
Since the answer-verified training subset overlaps with the batch used to evaluate each adapted candidate, how does AGC² distinguish generalizable absorbability from short-term fitting to the current batch? Could the authors evaluate candidate budgets on a disjoint held-out subset?
-
What is the total training cost of AGC² compared with the baselines, given the repeated compression, adaptation, and evaluation required for each batch?
-
When the reference batch is unreliable, incorrect trajectories are excluded from SFT and AGC² only lengthens the compressed trajectories from the correct subset. Why should increasing their token budget recover performance on the unanswered examples, and how often does this branch actually find an acceptable candidate?
-
The empirical evaluation is limited to three datasets and different scales of the same DeepSeek-R1-Distill-Qwen family. GSM8K and MATH-500 may not fully represent challenging reasoning tasks, while AIME2024 contains only 30 problems, making its reported improvements sensitive to a small number of examples. Could the authors evaluate AGC² on larger and more challenging benchmarks, such as OlympiadBench or College Math, and on other model families, such as original Qwen or Llama-based models, to better demonstrate its robustness and generalizability?
3 comments
Thank you for the detailed response and additional experiments. The reported training cost, unreliable-branch statistics, and expanded evaluations clarify several of my questions. However, I remain concerned about same-batch adaptation and evaluation, the substantial computational overhead, and the mixed cross-family results. I therefore maintain my current rating.
We thank the reviewer for the follow-up and for acknowledging the clarifications on training cost, the unreliable branch, and the expanded evaluations. We address the three remaining concerns directly.
Same-batch adaptation and evaluation. This score is a per-batch absorbability test, not a held-out generalization estimate, and it is used only to select a budget. After the model is updated on the compressed targets for a batch, we evaluate it by free generation from the questions alone, not by teacher-forcing on those targets. The evaluation batch is also a superset of the SFT subset : it includes problems whose reference CoTs were excluded. A candidate passes only if the compressed reasoning is learnable within that limited update. The overlap is intentional: the candidate targets are the training data for that batch, so the relevant test is whether the model can absorb them. A disjoint held-out set would instead measure transfer to other problems, which is a different question and is noisy after two local steps. Generalization is checked separately on the held-out test sets; on the Qwen family, the downstream MATH-500 and AIME2024 gains indicate that the per-batch signal is not merely fitting the batch.
Computational overhead. We agree that the measured overhead is substantial relative to static SFT. For 7B model, the end-to-end allocated cost is 21.383 h and 85.531 GPU-hours versus 0.825 h and 2.139 GPU-hours for TokenSkip, an additional 83.4 GPU-hours. These allocated hours include compressor waiting; the measured student SFT and evaluation execution is 13.748 GPU-hours, versus 1.552 for TokenSkip, and most of it (13.256) is in-batch generation for evaluation. The search is bounded: 1.57 candidates and 3.13 local steps per scheduled batch on average. This is a one-time offline cost, not a per-query cost, and peak GPU memory is slightly lower (45.48 vs 49.49 GB). At inference, reduces response length by 67.3%, 49.7%, and 23.2% on GSM8K, MATH-500, and AIME2024, so the cost is amortized after roughly – queries. Static SFT is cheaper but weaker on harder tasks; we will state this trade-off explicitly and explore cheaper proxy evaluation in future works.
Cross-family results. On DeepSeek-R1-Distill-Llama-8B, is 20.34% shorter on GSM8K but 8.42 points lower; on MATH-500 and AIME2024, it does not shorten responses and remains 1.6 and 3.3 points below Original. The method avoids CoT-Valve's catastrophic AIME collapse (40.00 vs 20.00), but it does not improve over Original on this family. The Qwen results show the intended regime: MATH-500 gains at 7B/14B/32B and AIME2024 gains at 14B/32B with shorter responses. We will supplement the results and discuss the performance degradation in the revise version.
We thank the reviewer again for the continued engagement. These comments have helped us sharpen the absorbability test, the cost analysis, and the cross-family evaluation, and we would be grateful for any further suggestions that can improve the paper.
Response to Reviewer yuf3
We thank Reviewer yuf3 for the questions on absorbability, cost, the unreliable branch, and scope.
W1 & Q1: This test assesses the compressed supervision for the current batch, not held-out accuracy.
W2 & Q2: We measured 7B training costs for AGC2 and TokenSkip on the same GPU. We use TokenSkip because its code and configurations are public. The costs are below.
Peak GPU memory is slightly lower than TokenSkip; the additional training time comes from candidate compression, local adaptation, and in-batch evaluation over up to K=3 candidate budgets per batch. This is a one-time offline cost: at inference, AGC2 saves 711 tokens on GSM8K (-67.3%), 1,419 on MATH-500 (-49.7%), and 2,453 on AIME2024 (-23.2%). The extra 83.4 GPU-hours are amortized after roughly 2×105–4×105 queries. We acknowledge the higher overhead than single-pass SFT and will explore faster proxy evaluations.
Q3: We thank you for identifying the ambiguity in the unreliable-reference branch. We will correct Sections 4.2–4.3 to distinguish reference accuracy, the supervision subset, and post-adaptation accuracy, and fix the inequality in Figure 2: acceptance requires at,b≥τ; at,b<τ means the search must continue if trials remain.
The initial at,0 is the correctness rate of the stored references, not model accuracy under a short cap. When at,0<τ, incorrect references stay excluded from SFT; a larger budget compresses only the correct subset Btref, not the incorrect examples. Evaluation is still on the full batch Btlong.
Longer targets preserve more reasoning from the correct subset but do not guarantee that the model solves the remaining questions. Our earlier description implied assured recovery and was too strong. The branch accepts a candidate only if full-batch accuracy reaches τ; otherwise, it keeps the pre-batch parameters after at most K=3 trials. Batches are grouped by reference-CoT length, not by a verified difficulty label; we will correct this too.
In the 7B experiment, 78.9% of scored batches enter the reliable branch and 21.1% the unreliable branch; three scheduled batches are skipped without a reference score. Of the unreliable batches, 16.1% find an accepted candidate in 3 trials, while 83.9% do not.
Q4: We appreciate the suggestion to test harder benchmarks and another model family. On OlympiadBench with DeepSeek-R1-Distill-Qwen-7B, baselines are from Appendix Table 6 of Liu et al., ACL 2026.
AGC2 reduces mean length by 15.08% while increasing accuracy by 2.27% over Original. These results demonstrate the effectiveness of our method on a larger and more challenging benchmark.
We also evaluate DeepSeek-R1-Distill-Llama-8B on GSM8K, MATH-500, and AIME2024:
Because the TH2T appendix results appear problematic and the appendix lacks generation hyperparameters for DeepSeek-R1-Distill-Llama-8B, we compare with CoT-Valve (Table 3 of Ma et al., ACL 2025). On GSM8K, AGC2 shortens responses by 20.34% while accuracy changes from 90.30% to 81.88%. On MATH-500 and AIME2024, CoT-Valve's aggressive compression drops AIME2024 accuracy from 43.33% to 20.00%. In contrast, AGC2 detects that this violates local absorbability on these trajectories, so it preserves or slightly expands the token budget and maintains much stronger accuracy. We will include these results in the revision.