acceptodds
Desk rejected at ICLR 2027

CRISP: Interpretable and Adaptive Chain-of-Thought Compression via Small-Model Collaboration

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Chain-of-Thought (CoT) reasoning substantially improves the performance of large language models on complex tasks, but often produces lengthy and redundant intermediate steps. Existing approaches delete historical tokens, terminate reasoning early, or compress context into latent representations, but they may require additional training, discard useful information, or produce model-dependent states that are difficult to interpret and reuse. To address these limitations, we propose CRISP, a fine-grained framework for step-level CoT management. CRISP assigns each newly generated reasoning step one of five actions: KEEP, LIGHT, HEAVY, SKIP, and STOP, enabling it to preserve, compress, remove, or terminate reasoning content dynamically. To select the appropriate action, CRISP introduces a lightweight Multi-Agent Controller (MAC), which applies role-specific prompts to a shared small language model and extracts action-aware semantic representations for lightweight probe-based prediction, without retraining the large reasoning model. The same small model also serves as a step compressor: LIGHT removes superficial redundancy, whereas HEAVY distills future-relevant information into compact natural-language states. Unlike latent representations, these compressed states are interpretable, reusable, and transferable across reasoning models. Experiments across multiple benchmarks and model families show that CRISP achieves a favorable accuracy–efficiency trade-off, demonstrating the effectiveness of semantic step-level management for long-chain reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
CoT, Small-Model Collaboration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WLMwOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

CRISP proposes a training-free, step-level framework for managing long Chain-of-Thought reasoning, motivated by the observation that intermediate steps vary widely in how useful they remain. After each reasoning step, a lightweight Multi-Agent Controller built on a small model selects one of five actions: KEEP the step, LIGHT or HEAVY compression, SKIP, or STOP reasoning. For compressible steps, the same auxiliary model rewrites the step into concise natural language. The reasoning model re-encodes this text, which replaces the original KV entries, so the compressed states stay interpretable and can be transferred across models. Experiments with DeepSeek-R1-Distill-Qwen-7B, DeepSeek-R1-Distill-Llama-8B, and Qwen3-14B-Thinking on several benchmarks show that CRISP largely preserves baseline accuracy and reduces inference time and peak memory.

Critical strengths
  1. Unified action space. Keeping, compressing, deleting, and stopping are handled in one coherent framework, where prior work usually addresses only one of these.
  2. Interpretable, transferable states. Compressed reasoning stays in readable natural language, so it can be inspected, verified, and reused across model families, which latent or gist-token methods cannot offer.
  3. Breadth of analysis. The paper covers latency, memory, compression quality, cross-model transfer, and case studies.
Critical weaknesses
  1. No realistic serving evaluation. There are no throughput or tail-latency results under continuous batching or frameworks like vLLM or SGLang. The latency studies are isolated micro-benchmarks with fixed short contexts, and the hardware and inference framework are not specified. In batched serving, per-sequence asynchronous KV edits, roughly 1-second compression stalls, and the cost of re-encoding or invalidating downstream KV are not addressed.
  2. Heuristic step segmentation. Splitting on \n\n may not match true semantic step boundaries across models or tasks. Some newer models, such as the Qwen3.5 series, use bullet points to split sentences. Using \ n\n to split sentences makes cross-model compatibility difficult.
  3. Dependence on labeled data and judges. Controller training relies on a custom redundancy-labeled dataset, and compression quality is scored by an LLM judge. Both raise questions about label quality and generalization to new domains.
Questions for authors
  1. What inference framework was the experiment based on? How does it perform in real-world batch inference serving scenarios? This question is very important.
  2. The training data and benchmark domains are basically the same. Is there a risk of overfitting in the method? It would be best to add out-of-domain tests.
  3. AIME24 is a bit old. It is recommended to add AIME25 and AIME26 tests.
5 comments
AuthorsOpenReview ↗

Response to Reviewer

We thank the reviewer for recognizing CRISP’s unified action space, readable and transferable states, and breadth of analysis. Your concerns about serving performance, step segmentation, and supervision quality help us identify where the evaluation and implementation details need to be clearer. We address these issues and the requested additional evaluations below.


(R3W1/R3Q1) Inference backend, parallel batching, and serving integration: Thank you for highlighting the execution and serving details. Our main experiments use Hugging Face Transformers for generation and state updates, with lm-evaluation-harness for task construction and evaluation. The main-table results are not obtained with vLLM or SGLang. End-to-end timing includes controller calls, LIGHT/HEAVY rewriting, history encoding, replacement re-encoding, KV management, and final-answer generation; the component microbenchmarks further explain the overhead.

We have additionally implemented batched execution and vLLM/SGLang integration. The LRM and auxiliary controller/compressor are managed independently and can be placed on separate devices. Each request pauses at its own reasoning boundary; boundary requests are routed and compressed in batches, and pending compression jobs enter an asynchronous queue. SLM compression runs in parallel with LRM decoding for other requests in the same batch that do not require compression, so those requests are not blocked by the compression operation. The affected request bears its own waiting cost and rejoins the runnable batch after processing, without requiring the entire batch to wait synchronously.

KV updates retain unchanged prefix blocks/pages and release and recompute only the affected suffix, allowing the same request to continue without aborting and re-adding it. CRISP intervenes after the latest step, before later steps have been generated, so no already-generated downstream steps need to be invalidated. HEAVY uses the global compressed history, recent raw steps, and the current step; accepted summaries are re-encoded into reasoning-model tokens, validated, and committed. These scheduling and cache-update mechanisms are documented in the appendix.

We have also completed batched functional and performance validation of both integrations on AIME26 using A800-80GB GPUs and batch size 16, with implementation and validation settings added to the appendix. CRISP’s main contribution is step-level semantic state management without training the reasoning model. These implementations establish integration with batched serving engines; further optimization of high-concurrency throughput, tail latency, and engine-specific scheduling is future systems work.


(R3W2) Configurable step boundaries and cross-format evaluation: We use double newlines (\n\n) as an online-detectable step boundary, following common operational definitions in related work [1–3]. Boundary detection is performed by the generator; MAC receives an already segmented step and predicts its action. The boundary rule is configurable through generate(seperater='\n\n'), allowing a custom delimiter without changing MAC’s decision interface.

To evaluate an alternative output format, we added experiments with Qwen3.5-9B using bullet-point boundaries. On GSM8K, CRISP-0.5B improves accuracy from 73.77% to 78.92% while reducing tokens from 1275.47 to 678.64 and time from 3.83 to 2.53. On MATH-500, accuracy increases from 48.80% to 52.80%, with reductions in both tokens and time. The complete results are shown below.

MethodSLMGSM8KMATH 500
Acc.tokenstimeAcc.tokenstime
vanilla-73.771275.473.8348.802093.596.97
CRISPQwen2.5-0.5B78.92678.642.5352.801735.505.83
CRISPQwen2.5-1.5B74.071146.873.3249.002000.346.51

These results demonstrate that CRISP can be adapted to the evaluated delimiter format while maintaining a useful accuracy–efficiency trade-off. The state-management interface is separate from boundary detection, and alternative formatting conventions can be supported through configurable rules.

[1] Hogwild! Inference: Parallel LLM Generation via Concurrent Attention

[2] The Lessons of Developing Process Reward Models in Mathematical Reasoning

[3] GlimpRouter: Efficient Collaborative Inference by Glimpsing One Token of Thoughts

(R3W3/R3Q2) Supervision, compression quality, and domain generalization: Thank you for the comments. We address these concerns through label construction, compression evaluation, and domain transfer.

First, the supervision has explicit construction criteria and is evaluated on held-out data. Qwen3-30B-Instruct labels KEEP/LIGHT/HEAVY/SKIP using a shared semantic rubric over the question, history, and current step. Strong-model semantic evaluation follows established approaches such as LLM-as-a-Judge [1] and G-Eval [2]. STOP labels additionally use an actual intervention: thinking is ended at the candidate prefix, the reasoning model generates an answer, and correctness is checked against the reference. Related outcome-based intermediate supervision is used by Math-Shepherd [3] and OmegaPRM [4]; our STOP labels specifically test whether ending thinking yields a correct answer under the stated decoding protocol.

We report validation results on data that do not overlap with training, using an 85%/15% training–validation split. Overall five-way accuracy is 80.54%/79.79% for the two auxiliary sizes, and STOP accuracy is 96.40%/97.70%. The table also reports precision, recall, and F1 for the high-risk actions. These held-out results show that the learned policy generalizes beyond its training examples.

Auxiliary modelActionAccuracy (%) ↑Precision (%) ↑Recall (%) ↑F1 (%) ↑
Qwen2.5-0.5BAll80.5480.5480.5480.54
Qwen2.5-1.5BAll79.7979.7979.7979.79
Qwen2.5-0.5BSTOP96.4076.9096.3885.54
Qwen2.5-0.5BSKIP95.3580.3678.6079.47
Qwen2.5-1.5BSTOP97.7084.3197.2990.34
Qwen2.5-1.5BSKIP93.8073.9770.7472.32

Second, compression quality is assessed through model scores, downstream answers, and external benchmarks. Strong-model evaluation with fixed criteria follows [1,2]. Table 3 uses a 10-point scale, with overall quality of 8.23–9.48 and faithfulness of 8.67–9.78. In Table 4, forced compression reduces accuracy by 1.06–3.64 percentage points, mostly around two points, supporting retention of information needed by the downstream task.

We additionally evaluate reference-based quality and end-to-end cost on CNN/DailyMail and Google Sentence Compression. On CNN/DailyMail, SLM ROUGE-L is 0.228/0.219 versus 0.227–0.236 for the LRMs, while total time is 0.29/0.58 h versus 3.10–5.50 h. On Google Sentence Compression, SLM time is 0.09/0.10 h versus 2.01–3.96 h for the LRMs. Together with the quality metrics below, these results support the SLM compressor’s practical quality–cost trade-off.

CompressorBenchmarkROUGE-1 ↑ROUGE-2 ↑ROUGE-L ↑Token-F1 ↑Wall time (h) ↓Tokens/s ↑
Qwen2.5-0.5B (SLM)CNN/DailyMail0.3420.1240.2280.3140.2956.1
Qwen2.5-1.5B (SLM)CNN/DailyMail0.3450.1180.2190.3150.5846.6
R1-Qwen-7B (LRM)CNN/DailyMail0.3680.1260.2310.3423.135.2
R1-Llama-8B (LRM)CNN/DailyMail0.3590.1250.2270.3345.529.8
Qwen3-14B (LRM)CNN/DailyMail0.380.1310.2360.354.7622
Qwen2.5-0.5B (SLM)Google Sentence Compression0.4520.2410.4270.4020.0956.7
Qwen2.5-1.5B (SLM)Google Sentence Compression0.4430.220.4160.3920.147.6
R1-Qwen-7B (LRM)Google Sentence Compression0.4410.2610.4280.394334.3
R1-Llama-8B (LRM)Google Sentence Compression0.4940.2860.4810.4462.0144.7
Qwen3-14B (LRM)Google Sentence Compression0.430.2620.4190.3873.9627.5

Third, we add end-to-end transfer evaluations in finance and law. Our supervision covers mathematics, code, and science. Learning compression from reasoning data and evaluating on downstream benchmarks is also the experimental paradigm used by LightThinker [5]. To extend beyond these training domains, we apply the current MAC to FinQA [6] and LegalBench’s insurance_policy_interpretation task [7].

BenchmarkMethodnAccuracy (%) ↑Closing rate (%) ↑Average time (s/case) ↓
FinQAVanilla114766.6199.39100.2
FinQACRISP-0.5B114762.0799.1370.3
FinQACRISP-1.5B114764.2599.2276.9
LegalBench insurance_policy_interpretationVanilla13342.86100.0016.2
LegalBench insurance_policy_interpretationCRISP-0.5B13339.10100.0010.4
LegalBench insurance_policy_interpretationCRISP-1.5B13339.85100.0014.9

On FinQA, CRISP-0.5B/1.5B reduce average time from 100.2 s to 70.3/76.9 s (29.8%/23.3%), with accuracy changing from 66.61% to 62.07%/64.25%. On the legal task, the 0.5B configuration reduces time from 16.2 s to 10.4 s (35.8%), with accuracy changing from 42.86% to 39.10%; all three configurations have a 100% closing rate. These results demonstrate accuracy–efficiency trade-offs for the learned state-management policy in new domains. They provide end-to-end task evidence, while action-classification performance is reported separately in the held-out validation table. Both model backbones remain frozen, and training is limited to the lightweight control heads and Arbiter.

[1] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

[2] G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment

[3] Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations

[4] Improve Mathematical Reasoning in Language Models by Automated Process Supervision

[5] LightThinker: Thinking Step-by-Step Compression

[6] FinQA: A Dataset of Numerical Reasoning over Financial Data

[7] LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models


(R3Q3) Additional AIME26 evaluation: Thank you for the suggestion. We have completed AIME26 comparisons with the baselines and both CRISP auxiliary-model sizes under a 32K-token generation budget. Closing rate measures answer completion within that budget, and time is end-to-end evaluation time in hours.

Qwen2.5-7B

MethodAuxiliary modelAccuracy (%) ↑Closing rate (%) ↑Time (h) ↓
Vanilla—53.3393.334.60
H2O—26.6733.3317.90
SnapKV—46.6790.004.60
SepLLM—16.6770.007.40
DEER—36.67100.004.40
CRISPQwen2.5-0.5B40.00100.003.30
CRISPQwen2.5-1.5B46.6796.674.20

Llama3-8B

MethodAuxiliary modelAccuracy (%) ↑Closing rate (%) ↑Time (h) ↓
Vanilla—43.3393.334.30
H2O—26.6760.0017.50
SnapKV—43.3393.334.30
SepLLM—13.3373.338.60
DEER—33.33100.004.60
CRISPQwen2.5-0.5B43.33100.003.90
CRISPQwen2.5-1.5B33.3393.334.00

Qwen3-14B

MethodAuxiliary modelAccuracy (%) ↑Closing rate (%) ↑Time (h) ↓
Vanilla—63.3393.338.40
H2O—63.3396.676.70
SnapKV—53.3383.339.20
SepLLM—16.6743.3315.80
DEER—63.33100.008.50
CRISPQwen2.5-0.5B63.3396.676.70
CRISPQwen2.5-1.5B63.3393.338.20

CRISP-0.5B matches Vanilla accuracy on Llama-8B and Qwen3-14B (43.33% and 63.33%) while reducing time by 9.3% and 20.2%; closing rates reach 100% and 96.67%. Some eviction baselines struggle to complete answers on the smaller backbones, while DEER completes reliably but has lower accuracy there. The full results below and in the appendix show the resulting backbone-dependent accuracy–speed trade-offs.

Reviewer_eCtNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents CRISP, a step-level framework for CoT as a semantic reasoning-state management. CRISP operates without updating/training the reasoning model. It consists of Multi-agent controller (MAC) to determine if the reasoning is sufficient for answering, to delete thoughts, or to remove redundancy. It does it with 5 executable actions: KEEP, LIGHT, HEAVY, SKIP and STOP. An auxiliary model replaces reasoning with compact natural-language states without updating the reasoing model.

Critical strengths

None

Critical weaknesses
  1. The novelty of paper is limited, as the whole study is just asking a smaller LLM to summarize the context with some extra steps to either KEEP, STOP, or, SKIP. The results from Table 2 do not indicate any meaningful improvements of the proposed method.
  2. The quality of step-level compression is low in quality, as shown in Table 3. The study uses an LLM to determine the quality of the compressed step.
Questions for authors
  1. What is a step in the chain of thought? The prompts used for agents in MAC also do not define what a step is.
  2. For compression, how does the CRISP perform as compared to asking the LRM itself to generate short and compact CoTs?
  3. Is a small LLM better than the LRM to determine that the current CoT is enough to answer the question?
  4. GSM8k and MATH-500 are light on reasoning and do not require producing long CoTs. Why CRISP is evaluated on these datasets?
  5. What is the temperature used for LRM? If greedy decoding is not used, then pass@k is needed to make results more concrete.
3 comments
AuthorsOpenReview ↗

Response to Reviewer

We thank the reviewer for the careful reading and the direct questions about CRISP’s contribution, compression quality, step definition, and evaluation protocol. These comments help us explain the motivation for semantic state management and identify the comparisons needed to assess its value. We respond to each concern below.


(R2W1) Contributions of semantic state management: Thank you for the comment. CRISP’s contribution is to manage the evolving reasoning state without updating the reasoning model. Alongside keeping, deleting, or stopping, it identifies useful steps that can be rewritten into shorter, readable text. Table 1 compares these capabilities. Both backbones remain frozen; only the lightweight control heads are trained.

The five actions address different semantic questions. KEEP preserves necessary information, LIGHT/HEAVY concern shortening useful content, SKIP asks whether a step adds information beyond the history, and STOP asks whether the reasoning is sufficient to answer. Local verbosity, cross-step redundancy, and answer sufficiency therefore need different evidence. Our preliminary results in Section 2.2 and Figure 1 show that simple statistics and shallow MLPs struggle to separate these categories, and that different roles favor different representation layers. This is why we use specialists to provide complementary evidence for the Arbiter.

The risk-aware design also allows the Arbiter to reject a risky specialist suggestion. The added AppendixB.2 cases show both sides: a 427-token polynomial step is skipped with the recorded answer still correct, while f(-2)=2 is retained despite deletion/stopping signals because it is needed later.

Action configuration0.5B Acc. (%)0.5B Wall (h)1.5B Acc. (%)1.5B Wall (h)
SKIP-Only86.018.2891.016.54
STOP-Only81.65.3188.47.40
Single-Controller81.04.3082.06.50
COMPRESS-Only92.813.5292.415.52
SKIP+COMPRESS89.818.7690.019.91
COMPRESS+STOP85.24.6687.28.24
CRISP82.45.7685.010.20

We also narrow the efficiency claims to configuration-dependent accuracy–cost trade-offs. The newly added MATH-500 ablation gives quantitative evidence for rewriting. With 0.5B, COMPRESS+STOP raises accuracy from 81.6% to 85.2% over STOP-Only and reduces time from 5.31 h to 4.66 h. COMPRESS-Only also exceeds SKIP-Only at both sizes. We have therefore focused the conclusion on interpretable semantic control and the measured accuracy–cost trade-offs, whose benefits depend on the task and model configuration.


(R2W2/R2Q2) Compression quality and choice of compressor: Thank you for the suggestions. Table 3 uses a 10-point evaluation following LLM-as-a-Judge [3] and G-Eval [4]. Overall quality ranges from 8.23 to 9.48 and faithfulness from 8.67 to 9.78, so the results do not suggest generally low compression quality. Table 4 also checks whether compressed states remain useful for answering: forced compression reduces accuracy by 1.06–3.64 percentage points, mostly around two points.

Enabling LRM generates shorter reasoning needs to train it specifically with SFT/RL, which adds reasoning-model training cost. To address the comparison with the LRM itself, we evaluated both small and large compressors under the same rewriting and scoring settings, aggregating three compression modes. The results below show that Qwen2.5-1.5B approaches the overall quality of the 7B and 8B LRMs. Qwen3-14B scores higher, but frequent rewriting with a larger model costs more computation.

CompressorSemantic quality ↑Information retention ↑Readability ↑Reasoning continuity ↑Faithfulness ↑Overall quality ↑
Qwen2.5-0.5B (SLM)8.968.89.228.889.428.72
Qwen2.5-1.5B (SLM)9.229.079.539.129.389.07
R1-Qwen-7B (LRM)9.299.139.419.169.359.11
R1-Llama-8B (LRM)9.399.269.419.39.589.17
Qwen3-14B (LRM)9.849.819.849.819.919.73

We also evaluated CNN/DailyMail [1] and Google Sentence Compression [2] to check quality against reference texts and measure the total time required. The small models achieve similar ROUGE-L scores in several comparisons while taking substantially less time.

CompressorBenchmarkROUGE-1 ↑ROUGE-2 ↑ROUGE-L ↑Token-F1 ↑Wall time (h) ↓Tokens/s ↑
Qwen2.5-0.5B (SLM)CNN/DailyMail0.3420.1240.2280.3140.2956.1
Qwen2.5-1.5B (SLM)CNN/DailyMail0.3450.1180.2190.3150.5846.6
R1-Qwen-7B (LRM)CNN/DailyMail0.3680.1260.2310.3423.135.2
R1-Llama-8B (LRM)CNN/DailyMail0.3590.1250.2270.3345.529.8
Qwen3-14B (LRM)CNN/DailyMail0.380.1310.2360.354.7622
Qwen2.5-0.5B (SLM)Google Sentence Compression0.4520.2410.4270.4020.0956.7
Qwen2.5-1.5B (SLM)Google Sentence Compression0.4430.220.4160.3920.147.6
R1-Qwen-7B (LRM)Google Sentence Compression0.4410.2610.4280.394334.3
R1-Llama-8B (LRM)Google Sentence Compression0.4940.2860.4810.4462.0144.7
Qwen3-14B (LRM)Google Sentence Compression0.430.2620.4190.3873.9627.5

These results explain our choice of a dedicated small compressor for frequent rewriting. It keeps the reasoning backbone frozen, avoids the additional training required by an SFT/RL approach, and keeps rewriting instructions outside the main reasoning history. Our direct LRM comparison concerns rewriting the same inputs; prompting an LRM to generate a shorter original CoT is a different intervention.

[1] Get To The Point: Summarization with Pointer-Generator Networks

[2] Overcoming the Lack of Parallel Data in Sentence Compression

[3] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

[4] G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment


(R2Q1) Definition of a reasoning step: As defined in Section 2.1, we treat each reasoning segment delimited by consecutive line breaks (\n\n) as a step. The boundary is detected online by the LRM generation procedure. Only after a segment is completed are the question, effective history, and current segment passed to MAC. MAC therefore selects an action for an already segmented step; it does not identify step boundaries itself. We will explicitly mark the current step in the input template.

This follows similar operational definitions in Hogwild! Inference [1], the Qwen process-reward-model study [2], and GlimpRouter [3]. We do not assume that every segment is an indivisible logical unit. Since boundary detection is separate from MAC, other formats can use configurable delimiters; we will document the parameters and examples.

[1] Hogwild! Inference: Parallel LLM Generation via Concurrent Attention

[2] The Lessons of Developing Process Reward Models in Mathematical Reasoning

[3] GlimpRouter: Efficient Collaborative Inference by Glimpsing One Token of Thoughts


(R2Q3) Learning cost-effective stopping decisions: Thank you for the question. Our aim is to learn a stopping decision that is inexpensive enough to use frequently during reasoning. An LRM can learn stopping behavior through SFT/RL, or use signals already available during generation. We examine the latter approach in Section 2.2 and Figure 1(a), using entropy, information scores, and length-related features.

To answer the stopping question directly, we added STOP-specific results. The table below compares CRISP with minimum entropy, the strongest tested statistical criterion by F1.

MethodAccuracy (%) ↑Precision (%) ↑Recall (%) ↑F1 (%) ↑
info_score (entropy x log len)46.785.643.257.4
entropy mean40.385.833.748.4
entropy median42.485.436.951.6
entropy max53.084.653.165.2
entropy rank vs prefix30.186.219.031.1
step length (tokens)41.284.635.750.2
log step index25.287.911.520.4
entropy min61.883.767.274.5
entropy spread (max - mean)39.784.233.848.3
CRISP-0.5B96.4076.9096.3885.54
CRISP-1.5B97.7084.3197.2990.34

CRISP identifies more stoppable prefixes, with higher recall and F1. Some statistical criteria have higher precision, but miss many opportunities to stop, which is why we report all three metrics together.

Our STOP supervision comes from actually ending thinking, generating an answer, and checking correctness. The lightweight specialists and Arbiter learn from those outcomes while both backbones remain frozen. We also include the trial-answer-based DEER [1] in Table 2. This is a comparison of specific stopping methods and their costs, rather than a claim that a smaller model is generally a better judge than the LRM.

[1] Dynamic Early Exit in Reasoning Models

(R2Q4) Benchmark choice and observed reasoning lengths: We chose GSM8K and MATH-500 because they are widely used in KV compression and efficient reasoning, including LightThinker [1], DEER [2], and TokenSkip [3]. In our experiments, we use gsm8k_cot_zeroshot with R1-Qwen-7B, R1-Llama-8B, and Qwen3-14B.

Besides, we use gsm8k_cot_zeroshot rather than the standard 8-shot GSM8K setting. With the relatively compact R1-Qwen-7B, R1-Llama-8B, and Qwen3-14B reasoning backbones, these tasks elicit substantial multi-step reasoning in our experiments. The observed distributions are summarized below: mean trace lengths range from 10.0 to 106.2 steps across the evaluated settings, and the maximum lengths on MATH-500 range from 581 to 949 steps. These trajectories provide a relevant setting for evaluating step-level CoT compression and inference efficiency.

DatasetBackboneMean stepsMedian stepsMaximum stepsMean CoT tokensTraces with ≥20 steps (%)
GSM8K-CoT-0shotR1-Qwen-7B10.0785500.5711
GSM8K-CoT-0shotR1-Llama-8B30.022254990.2753
GSM8K-CoT-0shotQwen3-14B29.321248143353
MATH-500R1-Qwen-7B71.932815355968
MATH-500R1-Llama-8B95.036949313365
MATH-500Qwen3-14B106.2785814456100

We have also added the more challenging AIME26 evaluation in Appendix Table [X]. With a 32K-token budget, CRISP-0.5B retains Vanilla accuracy on Llama-8B and Qwen3-14B while reducing time by 9.3% and 20.2%, respectively. On Qwen-7B, time decreases by 28.3%, but accuracy falls from 53.33% to 40.00%.

[1] LightThinker: Thinking Step-by-Step Compression

[2] Dynamic Early Exit in Reasoning Models

[3] TokenSkip: Controllable Chain-of-Thought Compression in LLMs


(R2Q5) Decoding settings and pass@k: The LRM in our main experiments uses greedy decoding with temperature=0 and seed=1234. We will explicitly report the sampling configuration, generation budget, and stopping conditions, together with the auxiliary compressor’s decoding settings.

Reviewer_soroOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CRISP manages a reasoning model’s active chain of thought at step boundaries. A controller chooses KEEP, LIGHT, HEAVY, SKIP, or STOP. LIGHT and HEAVY replace the current step with a natural-language rewrite from an auxiliary model; SKIP removes it; STOP ends thinking. The controller uses role-conditioned representations from a frozen auxiliary backbone, trained probes, and an Arbiter to combine answer sufficiency, deletion safety, redundancy, and compression-strength signals. Risk thresholds are calibrated on validation data.

The method is an inference-time intervention on the reasoning model, rather than an entirely training-free system: controller probes and the Arbiter are trained. Appendix B.1 specifies that rewriting rolls back the current-step cache and re-encodes the replacement while reusing the unchanged prefix. Experiments cover three reasoning backbones and five benchmarks, with further controller, overhead, compression-quality, and cross-model transfer studies.

Critical strengths
  1. The intervention has a useful semantic distinction and a concrete implementation. A step can be useful without needing to remain verbatim, and disposable without implying that reasoning should stop. Treating these cases separately is sensible. Appendix B.1 also addresses the important implementation issue: replacing the newest step does not require re-encoding the entire preceding history.

  2. There is evidence that readable compressed states can remain useful. The Llama-8B MATH-500 result improves from 76.60% for Vanilla to 83.80% with the 0.5B auxiliary model, with lower reported time. Table 5 also shows a useful transfer setting: CRISP-1.5B achieves 80.0% on the Llama-8B target, compared with 70.0% for Self-CoT. The overhead breakdown and premature-STOP failure in the appendix are welcome; they show that the authors have examined costs and failure modes, not just successful examples.

Critical weaknesses
  1. The evidence for a reliable risk-aware controller is difficult to reconcile. The paragraph accompanying Figure 6 reports training-set MAC results, including overall accuracy of 0.88 and macro-F1 of 0.76. Table 8 reports substantially different policy scores, while the best five-way Arbiter in Table 10 has macro-F1 0.3581, accuracy 0.4318, and LIGHT F1 0.0211. These could describe different stages or evaluation populations; I do not assume they are contradictory. However, the manuscript does not explain which results characterize the deployed policy. Appendix B.1 also permits “specialist or Arbiter probabilities” for inference without specifying the actual choice. Since STOP and SKIP can irreversibly remove useful reasoning, held-out precision and the final decision procedure are central to the soundness of the method.

  2. The main table supports a narrower accuracy–efficiency trade-off than the paper’s conclusion suggests. On Qwen3-14B, average accuracy falls from 79.17 to 74.60 with the 0.5B auxiliary model, while reported average time changes only from 10.24 to 9.88. DEER obtains 77.42 at 9.89, effectively the same time with higher accuracy. The Qwen-7B AIME result also drops from 56.67% to 30.00% in the 0.5B setting. Meanwhile, several eviction baselines nearly fail completely on individual tasks, making their precise cache settings and adaptations important. Appendix C.1 gives a reproduction description, and C.3 gives overhead measurements, but neither replaces a clearly specified matched operating point or a STOP-only control. It remains hard to tell when rewriting is preferable to simpler interventions.

Questions for authors
  1. Which controller, checkpoint, and evaluation split produced Table 2? Please reconcile Figure 6 and Tables 8–10, specify whether train/validation splits are problem-disjoint, and report the actual inference thresholds and held-out STOP/SKIP precision. How are specialist and Arbiter probabilities used in the final policy?

  2. Can the main accuracy–latency results be compared with STOP-only and a comparable single-controller variant? Please specify baseline cache budgets, generation limits, hardware/batching, and whether reported total time includes replacement re-encoding and full-context HEAVY rewriting. The existing overhead studies are helpful, but do not isolate the benefit of the full decision system.

  3. Which values in Table 4 are correct? For the 1.5B LIGHT row, the displayed accuracies imply changes of −1.06 points on Llama-8B and −3.64 on Qwen-7B, rather than the listed −1.90 and −2.05. The latter also affects the claim that all forced-compression settings stay within approximately 2.5 points of Vanilla.

2 comments
AuthorsOpenReview ↗

Response to Reviewer

We thank the reviewer for the careful assessment and for recognizing the semantic distinction between CRISP’s actions, its concrete cache implementation, and the usefulness of readable reasoning states. Your comments help us clarify the deployed controller, strengthen the comparison with simpler interventions, and correct the reporting errors. We address these points below.


(R1W1/R1Q1) Controller checkpoints, validation results, and final decisions: We use non-overlapping 85%/15% training–validation splits (test_ratio=0.15), preserving label proportions as closely as possible. All controllers use Adam for 60 epochs: batch size 16, learning rate 4×10⁻⁴, 10% linear warmup, weight decay 10⁻², dropout 0.22, and gradient clipping 1.0. The deployed checkpoint and component ablations are distinguished below.

ResultsPurposeDataset sizeTraining splitValidation splitCheckpoint
Figure 6, Table 2, and downstream experimentsFinal-controller diagnostics and downstream evaluation3,0002,550450mac_final
Table 8History-window ablation1,000850150window_final
Table 9Specialist-probe and layer-selection analysis1,000850150role_final
Table 10Arbiter component and layer-selection ablation1,000850150arbiter_final

We additionally report validation metrics for the final checkpoint used in Figure 6 and Table 2, including separate STOP and SKIP results. Overall five-way accuracy is 80.54%/79.79% for the 0.5B/1.5B controllers. STOP accuracy is 96.40%/97.70%, with recall of 96.38%/97.29%; SKIP accuracy is 95.35%/93.80%. Precision, recall, and F1 are reported separately below to characterize these high-risk actions.

Auxiliary modelActionAccuracy (%) ↑Precision (%) ↑Recall (%) ↑F1 (%) ↑
Qwen2.5-0.5BAll80.5480.5480.5480.54
Qwen2.5-1.5BAll79.7979.7979.7979.79
Qwen2.5-0.5BSTOP96.4076.9096.3885.54
Qwen2.5-0.5BSKIP95.3580.3678.6079.47
Qwen2.5-1.5BSTOP97.7084.3197.2990.34
Qwen2.5-1.5BSKIP93.8073.9770.7472.32

The Arbiter integrates specialist probabilities, evidence, and context to select the final action. In the added Appendix B.2 examples, SKIP removes a 427-token polynomial step without changing the recorded correct answer; in function evaluation, the Arbiter preserves f(-2)=2 with KEEP=0.862 despite SkipProbe SKIP=0.541 and StopProbe STOP=0.864. These examples make specialist acceptance/overruling inspectable. We will clarify probability usage in Appendix B.1.


(R1W2/R1Q2) End-to-end evaluation and action ablations: Our experiments use eight A800 GPUs, BF16, batch size 1, a 32K-token output limit, and 2,048-token budgets for KV-eviction baselines. Table 2 timing includes routing, rewriting, history encoding for LIGHT/HEAVY, replacement re-encoding, KV management, and answer generation. We have added these settings to the appendix. Batch size 1 is used for mechanism evaluation. Our batched execution and vLLM/SGLang integration are described in the responses to R3W1/R3Q1 and in the Appendix.

In Table 2, we already shows accuracy–time benefits in several configurations. Qwen-7B CRISP-1.5B achieves average accuracy/time of 57.93/3.80 versus DEER’s 54.08/4.04; Llama-8B CRISP-0.5B achieves 53.04/4.98 versus 51.22/8.31. Gains are smaller on Qwen3-14B. Some eviction baselines exhaust the output budget, while premature stopping can reduce DEER’s accuracy.

Besides, we additionally compare action variants on MATH-500, as shown below. COMPRESS-Only achieves the highest accuracy, supporting the value of retaining useful information through rewriting. The Single-Controller variant has lower runtime but also lower accuracy, which may reflect premature stopping. Full CRISP combines preservation, rewriting, deletion, and stopping to provide a performance–efficiency trade-off.

Action configuration0.5B Acc. (%)0.5B Wall (h)1.5B Acc. (%)1.5B Wall (h)
SKIP-Only86.018.2891.016.54
STOP-Only81.65.3188.47.40
Single-Controller81.04.3082.06.50
COMPRESS-Only92.813.5292.415.52
SKIP+COMPRESS89.818.7690.019.91
COMPRESS+STOP85.24.6687.28.24
CRISP82.45.7685.010.20

Moreover, the case studies in Figure 5 and Appendix C.6 illustrate how compression retains important numerical deductions while rewriting verbose steps into readable states. We also quantify controller and compressor overhead in Section 5.1 and Figure 9, complementing the end-to-end comparisons above.

(R1Q3) Corrections to Table 4: Thank you for identifying the errors in the reported accuracy differences. For the 1.5B LIGHT configuration, the correct changes are −1.06 percentage points on Llama-8B and −3.64 percentage points on Qwen-7B. Across all configurations, the accuracy drops range from 1.06 to 3.64 percentage points, so the statement that all settings remain within approximately 2.5 points of Vanilla was inaccurate. The corrected differences are shown below, and we have updated the corresponding values and statements in the manuscript.

Auxiliary modelCompression policyLlama-8B Δ (pp)Qwen-7B Δ (pp)
0.5BLIGHT−1.97−2.43
0.5BHEAVY−1.97−2.35
0.5BLIGHT + HEAVY−1.97−2.43
1.5BLIGHT−1.06−3.64
1.5BHEAVY−2.05−2.35
1.5BLIGHT + HEAVY−1.97−2.12

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.