- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors address the limitations of token-level correction in small-large model collaborative decoding, where a small language model (SLM) generates most reasoning steps while a large language model (LLM) is invoked at local disagreements. Existing methods merely replace or rerank the next token, leaving the committed prefix and its internal key-value (KV) state intact, and state repair alone does not guarantee more reliable subsequent reasoning. To this end, the authors formulate small-large model collaboration as a post-injection recovery problem under a finite budget and propose the Solution-Domain Topology Tree (SDTT). When a disagreement is detected, the system rolls back to an earlier reasoning anchor, employs a learnable Cache-to-Cache (C2C) projection to construct a repaired state while retaining the unrepaired state as a paired reference, and lets a frozen SLM expand recovery trees from both states under a matched budget. Terminal trajectories with identical predictions are grouped into answer basins, and a lineage-aware selector determines the final output based on basin structure and generation information, with the original output retained as a fallback. Extensive experiments on five mathematical reasoning benchmarks demonstrate that SDTT improves the Math Avg. of the Qwen3-1.7B SLM by 34.64% relative to greedy SLM decoding and outperforms token-level baselines, confirming the value of state-level intervention and solution-domain evaluation in collaborative decoding.
Critical strengths
The authors identify a critical limitation of token-level collaborative decoding—existing methods replace or rerank the next token without revoking the committed prefix and its internal KV state—and reformulate small-large model collaboration as a post-injection recovery problem, offering a novel and well-grounded perspective on when and how local assistance should intervene.
Critical weaknesses
-
The most relevant baselines are missing. The method is fundamentally a draft and verify scheme between a small model and a large model, so it should be compared against speculative decoding methods, especially EAGLE-3[1], which also operates at the feature level of the target model.
-
Latency is also measured only in the authors' own setup, not under practical serving frameworks like vLLM or SGLang. This makes it difficult to evaluate the claimed accuracy efficiency tradeoff against the actual state of the art.
-
The central claim is not directly tested. The paper argues that token level correction cannot undo the committed prefix and that state level repair is necessary, but no baseline where the LLM generates a repair prefix and the SLM re-prefills from that text is evaluated. The ablations only show that cache intervention helps over extra sampling (95.71 vs 95.10 and 95.28 on GSM8K), and the gain over the strongest token level baseline is small (Math Avg 50.22 vs 49.50). Since all mechanism evidence is on GSM8K, it remains unclear whether the gain really comes from state level intervention.
[1] EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test, NeurIPS 2025
Questions for authors
-
Can the reported 4.83x latency be reproduced under vLLM or SGLang? How does SDTT compare with EAGLE under an equal latency or throughput budget?
-
How sensitive are the results to the chosen rollback length, tree widths, and threshold? Are the mechanism ablations consistent on the other four benchmarks, or specific to GSM8K?
-
The evaluation is limited to math benchmarks, and testing on other domains such as code generation (e.g., HumanEval, LiveCodeBench) would better demonstrate its general applicability.