- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
CRISP proposes a training-free, step-level framework for managing long Chain-of-Thought reasoning, motivated by the observation that intermediate steps vary widely in how useful they remain. After each reasoning step, a lightweight Multi-Agent Controller built on a small model selects one of five actions: KEEP the step, LIGHT or HEAVY compression, SKIP, or STOP reasoning. For compressible steps, the same auxiliary model rewrites the step into concise natural language. The reasoning model re-encodes this text, which replaces the original KV entries, so the compressed states stay interpretable and can be transferred across models. Experiments with DeepSeek-R1-Distill-Qwen-7B, DeepSeek-R1-Distill-Llama-8B, and Qwen3-14B-Thinking on several benchmarks show that CRISP largely preserves baseline accuracy and reduces inference time and peak memory.
Critical strengths
- Unified action space. Keeping, compressing, deleting, and stopping are handled in one coherent framework, where prior work usually addresses only one of these.
- Interpretable, transferable states. Compressed reasoning stays in readable natural language, so it can be inspected, verified, and reused across model families, which latent or gist-token methods cannot offer.
- Breadth of analysis. The paper covers latency, memory, compression quality, cross-model transfer, and case studies.
Critical weaknesses
- No realistic serving evaluation. There are no throughput or tail-latency results under continuous batching or frameworks like vLLM or SGLang. The latency studies are isolated micro-benchmarks with fixed short contexts, and the hardware and inference framework are not specified. In batched serving, per-sequence asynchronous KV edits, roughly 1-second compression stalls, and the cost of re-encoding or invalidating downstream KV are not addressed.
- Heuristic step segmentation. Splitting on \n\n may not match true semantic step boundaries across models or tasks. Some newer models, such as the Qwen3.5 series, use bullet points to split sentences. Using \ n\n to split sentences makes cross-model compatibility difficult.
- Dependence on labeled data and judges. Controller training relies on a custom redundancy-labeled dataset, and compression quality is scored by an LLM judge. Both raise questions about label quality and generalization to new domains.
Questions for authors
- What inference framework was the experiment based on? How does it perform in real-world batch inference serving scenarios? This question is very important.
- The training data and benchmark domains are basically the same. Is there a risk of overfitting in the method? It would be best to add out-of-domain tests.
- AIME24 is a bit old. It is recommended to add AIME25 and AIME26 tests.
5 comments
(R3W3/R3Q2) Supervision, compression quality, and domain generalization: Thank you for the comments. We address these concerns through label construction, compression evaluation, and domain transfer.
First, the supervision has explicit construction criteria and is evaluated on held-out data. Qwen3-30B-Instruct labels KEEP/LIGHT/HEAVY/SKIP using a shared semantic rubric over the question, history, and current step. Strong-model semantic evaluation follows established approaches such as LLM-as-a-Judge [1] and G-Eval [2]. STOP labels additionally use an actual intervention: thinking is ended at the candidate prefix, the reasoning model generates an answer, and correctness is checked against the reference. Related outcome-based intermediate supervision is used by Math-Shepherd [3] and OmegaPRM [4]; our STOP labels specifically test whether ending thinking yields a correct answer under the stated decoding protocol.
We report validation results on data that do not overlap with training, using an 85%/15% training–validation split. Overall five-way accuracy is 80.54%/79.79% for the two auxiliary sizes, and STOP accuracy is 96.40%/97.70%. The table also reports precision, recall, and F1 for the high-risk actions. These held-out results show that the learned policy generalizes beyond its training examples.
| Auxiliary model | Action | Accuracy (%) ↑ | Precision (%) ↑ | Recall (%) ↑ | F1 (%) ↑ |
|---|---|---|---|---|---|
| Qwen2.5-0.5B | All | 80.54 | 80.54 | 80.54 | 80.54 |
| Qwen2.5-1.5B | All | 79.79 | 79.79 | 79.79 | 79.79 |
| Qwen2.5-0.5B | STOP | 96.40 | 76.90 | 96.38 | 85.54 |
| Qwen2.5-0.5B | SKIP | 95.35 | 80.36 | 78.60 | 79.47 |
| Qwen2.5-1.5B | STOP | 97.70 | 84.31 | 97.29 | 90.34 |
| Qwen2.5-1.5B | SKIP | 93.80 | 73.97 | 70.74 | 72.32 |
Second, compression quality is assessed through model scores, downstream answers, and external benchmarks. Strong-model evaluation with fixed criteria follows [1,2]. Table 3 uses a 10-point scale, with overall quality of 8.23–9.48 and faithfulness of 8.67–9.78. In Table 4, forced compression reduces accuracy by 1.06–3.64 percentage points, mostly around two points, supporting retention of information needed by the downstream task.
We additionally evaluate reference-based quality and end-to-end cost on CNN/DailyMail and Google Sentence Compression. On CNN/DailyMail, SLM ROUGE-L is 0.228/0.219 versus 0.227–0.236 for the LRMs, while total time is 0.29/0.58 h versus 3.10–5.50 h. On Google Sentence Compression, SLM time is 0.09/0.10 h versus 2.01–3.96 h for the LRMs. Together with the quality metrics below, these results support the SLM compressor’s practical quality–cost trade-off.
| Compressor | Benchmark | ROUGE-1 ↑ | ROUGE-2 ↑ | ROUGE-L ↑ | Token-F1 ↑ | Wall time (h) ↓ | Tokens/s ↑ |
|---|---|---|---|---|---|---|---|
| Qwen2.5-0.5B (SLM) | CNN/DailyMail | 0.342 | 0.124 | 0.228 | 0.314 | 0.29 | 56.1 |
| Qwen2.5-1.5B (SLM) | CNN/DailyMail | 0.345 | 0.118 | 0.219 | 0.315 | 0.58 | 46.6 |
| R1-Qwen-7B (LRM) | CNN/DailyMail | 0.368 | 0.126 | 0.231 | 0.342 | 3.1 | 35.2 |
| R1-Llama-8B (LRM) | CNN/DailyMail | 0.359 | 0.125 | 0.227 | 0.334 | 5.5 | 29.8 |
| Qwen3-14B (LRM) | CNN/DailyMail | 0.38 | 0.131 | 0.236 | 0.35 | 4.76 | 22 |
| Qwen2.5-0.5B (SLM) | Google Sentence Compression | 0.452 | 0.241 | 0.427 | 0.402 | 0.09 | 56.7 |
| Qwen2.5-1.5B (SLM) | Google Sentence Compression | 0.443 | 0.22 | 0.416 | 0.392 | 0.1 | 47.6 |
| R1-Qwen-7B (LRM) | Google Sentence Compression | 0.441 | 0.261 | 0.428 | 0.394 | 3 | 34.3 |
| R1-Llama-8B (LRM) | Google Sentence Compression | 0.494 | 0.286 | 0.481 | 0.446 | 2.01 | 44.7 |
| Qwen3-14B (LRM) | Google Sentence Compression | 0.43 | 0.262 | 0.419 | 0.387 | 3.96 | 27.5 |
Third, we add end-to-end transfer evaluations in finance and law. Our supervision covers mathematics, code, and science. Learning compression from reasoning data and evaluating on downstream benchmarks is also the experimental paradigm used by LightThinker [5]. To extend beyond these training domains, we apply the current MAC to FinQA [6] and LegalBench’s insurance_policy_interpretation task [7].
| Benchmark | Method | n | Accuracy (%) ↑ | Closing rate (%) ↑ | Average time (s/case) ↓ |
|---|---|---|---|---|---|
| FinQA | Vanilla | 1147 | 66.61 | 99.39 | 100.2 |
| FinQA | CRISP-0.5B | 1147 | 62.07 | 99.13 | 70.3 |
| FinQA | CRISP-1.5B | 1147 | 64.25 | 99.22 | 76.9 |
| LegalBench insurance_policy_interpretation | Vanilla | 133 | 42.86 | 100.00 | 16.2 |
| LegalBench insurance_policy_interpretation | CRISP-0.5B | 133 | 39.10 | 100.00 | 10.4 |
| LegalBench insurance_policy_interpretation | CRISP-1.5B | 133 | 39.85 | 100.00 | 14.9 |
On FinQA, CRISP-0.5B/1.5B reduce average time from 100.2 s to 70.3/76.9 s (29.8%/23.3%), with accuracy changing from 66.61% to 62.07%/64.25%. On the legal task, the 0.5B configuration reduces time from 16.2 s to 10.4 s (35.8%), with accuracy changing from 42.86% to 39.10%; all three configurations have a 100% closing rate. These results demonstrate accuracy–efficiency trade-offs for the learned state-management policy in new domains. They provide end-to-end task evidence, while action-classification performance is reported separately in the held-out validation table. Both model backbones remain frozen, and training is limited to the lightweight control heads and Arbiter.
[1] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
[2] G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
[3] Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations
[4] Improve Mathematical Reasoning in Language Models by Automated Process Supervision
[5] LightThinker: Thinking Step-by-Step Compression
[6] FinQA: A Dataset of Numerical Reasoning over Financial Data
[7] LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
(R3Q3) Additional AIME26 evaluation: Thank you for the suggestion. We have completed AIME26 comparisons with the baselines and both CRISP auxiliary-model sizes under a 32K-token generation budget. Closing rate measures answer completion within that budget, and time is end-to-end evaluation time in hours.
Qwen2.5-7B
| Method | Auxiliary model | Accuracy (%) ↑ | Closing rate (%) ↑ | Time (h) ↓ |
|---|---|---|---|---|
| Vanilla | — | 53.33 | 93.33 | 4.60 |
| H2O | — | 26.67 | 33.33 | 17.90 |
| SnapKV | — | 46.67 | 90.00 | 4.60 |
| SepLLM | — | 16.67 | 70.00 | 7.40 |
| DEER | — | 36.67 | 100.00 | 4.40 |
| CRISP | Qwen2.5-0.5B | 40.00 | 100.00 | 3.30 |
| CRISP | Qwen2.5-1.5B | 46.67 | 96.67 | 4.20 |
Llama3-8B
| Method | Auxiliary model | Accuracy (%) ↑ | Closing rate (%) ↑ | Time (h) ↓ |
|---|---|---|---|---|
| Vanilla | — | 43.33 | 93.33 | 4.30 |
| H2O | — | 26.67 | 60.00 | 17.50 |
| SnapKV | — | 43.33 | 93.33 | 4.30 |
| SepLLM | — | 13.33 | 73.33 | 8.60 |
| DEER | — | 33.33 | 100.00 | 4.60 |
| CRISP | Qwen2.5-0.5B | 43.33 | 100.00 | 3.90 |
| CRISP | Qwen2.5-1.5B | 33.33 | 93.33 | 4.00 |
Qwen3-14B
| Method | Auxiliary model | Accuracy (%) ↑ | Closing rate (%) ↑ | Time (h) ↓ |
|---|---|---|---|---|
| Vanilla | — | 63.33 | 93.33 | 8.40 |
| H2O | — | 63.33 | 96.67 | 6.70 |
| SnapKV | — | 53.33 | 83.33 | 9.20 |
| SepLLM | — | 16.67 | 43.33 | 15.80 |
| DEER | — | 63.33 | 100.00 | 8.50 |
| CRISP | Qwen2.5-0.5B | 63.33 | 96.67 | 6.70 |
| CRISP | Qwen2.5-1.5B | 63.33 | 93.33 | 8.20 |
CRISP-0.5B matches Vanilla accuracy on Llama-8B and Qwen3-14B (43.33% and 63.33%) while reducing time by 9.3% and 20.2%; closing rates reach 100% and 96.67%. Some eviction baselines struggle to complete answers on the smaller backbones, while DEER completes reliably but has lower accuracy there. The full results below and in the appendix show the resulting backbone-dependent accuracy–speed trade-offs.
Response to Reviewer
We thank the reviewer for recognizing CRISP’s unified action space, readable and transferable states, and breadth of analysis. Your concerns about serving performance, step segmentation, and supervision quality help us identify where the evaluation and implementation details need to be clearer. We address these issues and the requested additional evaluations below.
(R3W1/R3Q1) Inference backend, parallel batching, and serving integration: Thank you for highlighting the execution and serving details. Our main experiments use Hugging Face Transformers for generation and state updates, with lm-evaluation-harness for task construction and evaluation. The main-table results are not obtained with vLLM or SGLang. End-to-end timing includes controller calls, LIGHT/HEAVY rewriting, history encoding, replacement re-encoding, KV management, and final-answer generation; the component microbenchmarks further explain the overhead.
We have additionally implemented batched execution and vLLM/SGLang integration. The LRM and auxiliary controller/compressor are managed independently and can be placed on separate devices. Each request pauses at its own reasoning boundary; boundary requests are routed and compressed in batches, and pending compression jobs enter an asynchronous queue. SLM compression runs in parallel with LRM decoding for other requests in the same batch that do not require compression, so those requests are not blocked by the compression operation. The affected request bears its own waiting cost and rejoins the runnable batch after processing, without requiring the entire batch to wait synchronously.
KV updates retain unchanged prefix blocks/pages and release and recompute only the affected suffix, allowing the same request to continue without aborting and re-adding it. CRISP intervenes after the latest step, before later steps have been generated, so no already-generated downstream steps need to be invalidated. HEAVY uses the global compressed history, recent raw steps, and the current step; accepted summaries are re-encoded into reasoning-model tokens, validated, and committed. These scheduling and cache-update mechanisms are documented in the appendix.
We have also completed batched functional and performance validation of both integrations on AIME26 using A800-80GB GPUs and batch size 16, with implementation and validation settings added to the appendix. CRISP’s main contribution is step-level semantic state management without training the reasoning model. These implementations establish integration with batched serving engines; further optimization of high-concurrency throughput, tail latency, and engine-specific scheduling is future systems work.
(R3W2) Configurable step boundaries and cross-format evaluation: We use double newlines (
\n\n) as an online-detectable step boundary, following common operational definitions in related work [1–3]. Boundary detection is performed by the generator; MAC receives an already segmented step and predicts its action. The boundary rule is configurable throughgenerate(seperater='\n\n'), allowing a custom delimiter without changing MAC’s decision interface.To evaluate an alternative output format, we added experiments with Qwen3.5-9B using bullet-point boundaries. On GSM8K, CRISP-0.5B improves accuracy from 73.77% to 78.92% while reducing tokens from 1275.47 to 678.64 and time from 3.83 to 2.53. On MATH-500, accuracy increases from 48.80% to 52.80%, with reductions in both tokens and time. The complete results are shown below.
These results demonstrate that CRISP can be adapted to the evaluated delimiter format while maintaining a useful accuracy–efficiency trade-off. The state-management interface is separate from boundary detection, and alternative formatting conventions can be supported through configurable rules.
[1] Hogwild! Inference: Parallel LLM Generation via Concurrent Attention
[2] The Lessons of Developing Process Reward Models in Mathematical Reasoning
[3] GlimpRouter: Efficient Collaborative Inference by Glimpsing One Token of Thoughts