acceptodds
Under review as a conference paper at ICLR 2027

When Divergence Emerges, Look Back: Exploring Solution-Domain Topology in Collaborative Decoding

Abstract

Token-level small-large model collaborative decoding has a small language model (SLM) generate most reasoning steps and invokes a large language model (LLM) at local disagreements. Existing token-level correction methods mainly alter the current output, but cannot revoke a committed prefix and its internal state; state repair alone also does not establish whether the repaired state can produce more reliable subsequent reasoning. We therefore formulate small–large model collaboration as a post-injection recovery problem under a finite budget and propose the Solution-Domain Topology Tree. When a disagreement is detected, the system rolls back to an earlier reasoning anchor and uses learnable Cache-to-Cache (C2C) projection to construct a repaired state while retaining the unrepaired state as a paired reference. A frozen SLM then evaluates the future solution domains induced by both states under a matched budget and determines whether to adopt the intervened reasoning result. On five mathematical reasoning benchmarks, SDTT improves the Qwen3-1.7B SLM's Math Avg. by 34.64% relative to greedy SLM decoding. This result indicates that effective small-large model collaboration depends not on optimizing the current token in isolation, but on whether the intervention changes the state on which the SLM relies for continued reasoning in a way that improves access to reliable solution domains within a finite budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
collaborative decoding, LLM reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bRUHOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors address the limitations of token-level correction in small-large model collaborative decoding, where a small language model (SLM) generates most reasoning steps while a large language model (LLM) is invoked at local disagreements. Existing methods merely replace or rerank the next token, leaving the committed prefix and its internal key-value (KV) state intact, and state repair alone does not guarantee more reliable subsequent reasoning. To this end, the authors formulate small-large model collaboration as a post-injection recovery problem under a finite budget and propose the Solution-Domain Topology Tree (SDTT). When a disagreement is detected, the system rolls back to an earlier reasoning anchor, employs a learnable Cache-to-Cache (C2C) projection to construct a repaired state while retaining the unrepaired state as a paired reference, and lets a frozen SLM expand recovery trees from both states under a matched budget. Terminal trajectories with identical predictions are grouped into answer basins, and a lineage-aware selector determines the final output based on basin structure and generation information, with the original output retained as a fallback. Extensive experiments on five mathematical reasoning benchmarks demonstrate that SDTT improves the Math Avg. of the Qwen3-1.7B SLM by 34.64% relative to greedy SLM decoding and outperforms token-level baselines, confirming the value of state-level intervention and solution-domain evaluation in collaborative decoding.

Critical strengths

The authors identify a critical limitation of token-level collaborative decoding—existing methods replace or rerank the next token without revoking the committed prefix and its internal KV state—and reformulate small-large model collaboration as a post-injection recovery problem, offering a novel and well-grounded perspective on when and how local assistance should intervene.

Critical weaknesses
  1. The most relevant baselines are missing. The method is fundamentally a draft and verify scheme between a small model and a large model, so it should be compared against speculative decoding methods, especially EAGLE-3[1], which also operates at the feature level of the target model.

  2. Latency is also measured only in the authors' own setup, not under practical serving frameworks like vLLM or SGLang. This makes it difficult to evaluate the claimed accuracy efficiency tradeoff against the actual state of the art.

  3. The central claim is not directly tested. The paper argues that token level correction cannot undo the committed prefix and that state level repair is necessary, but no baseline where the LLM generates a repair prefix and the SLM re-prefills from that text is evaluated. The ablations only show that cache intervention helps over extra sampling (95.71 vs 95.10 and 95.28 on GSM8K), and the gain over the strongest token level baseline is small (Math Avg 50.22 vs 49.50). Since all mechanism evidence is on GSM8K, it remains unclear whether the gain really comes from state level intervention.

[1] EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test, NeurIPS 2025

Questions for authors
  1. Can the reported 4.83x latency be reproduced under vLLM or SGLang? How does SDTT compare with EAGLE under an equal latency or throughput budget?

  2. How sensitive are the results to the chosen rollback length, tree widths, and threshold? Are the mechanism ablations consistent on the other four benchmarks, or specific to GSM8K?

  3. The evaluation is limited to math benchmarks, and testing on other domains such as code generation (e.g., HumanEval, LiveCodeBench) would better demonstrate its general applicability.

Reviewer_zideOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets the problem of collaborative decoding, when a small LLM and a large LLM collaborate to generate tokens. Different from previous works that mainly consider and replace the current token, the paper proposes SDTT, which can rollback 7 tokens and use a trained projector to map the large LLM's KV cache to the small model's KV cache. SDTT then uses a tree search method to determine the answer. SDTT improves the performance of collaborative decoding by higher than 30%, comparing to greedy search methods.

Critical strengths

The paper addresses an interesting problem in collaborative decoding. Instead of only correcting the current token when the SLM and LLM disagree, it considers whether rolling back and modifying the KV cache can improve subsequent reasoning. This is a reasonable direction to explore. The proposed method combines KV cache mapping with tree-based candidate selection. Comparing the continuations from the original and repaired states is an interesting design choice, as it allows the method to consider the effect of an intervention on subsequent generation.

The experiments cover five mathematical reasoning benchmarks and two SLM sizes. The paper also includes comparisons with recent methods, inference cost measurements, and component-level ablations, which help evaluate the proposed approach.

Critical weaknesses

The motivation for small-large model collaboration is unclear. In traditional speculative decoding, the small model is mainly used to accelerate inference while preserving the performance of the large model. However, this paper focuses on improving reasoning accuracy rather than efficiency. I am not sure what the motivation is for using a small-large model pair in this setting, especially since the proposed method introduces considerable inference overhead.

If the goal is to improve accuracy beyond what the large model can achieve, I would expect a clearer explanation of why combining these two models helps. For example, combining models with complementary strengths is a more natural motivation for accuracy-oriented collaboration, as explored in "Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding" (Wang et al., 2025). This closely related work is also missing from the references.

More importantly, the paper does not report the performance of the standalone Qwen3-32B model. Without this baseline, it is difficult to understand the actual benefit of the proposed collaboration. The method is also considerably slower than existing collaborative decoding methods, while the accuracy improvements are relatively small. Overall, I find the motivation and practical benefits of this small-large collaboration framework insufficiently justified.

Questions for authors

What is the main goal of small-large model collaboration in this paper, accuracy or efficiency? Could the authors provide the standalone Qwen3-32B results for comparison?

In Table 3, No-op+No-op achieves higher candidate reachability than SDTT. Does this suggest that the improvement mainly comes from the selector rather than KV cache repair?

The accuracy improvement over FutureBridge is small despite substantially higher latency. Is this improvement statistically significant?

Reviewer_EFiQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that token-level SLM–LLM collaboration only fixes the next token and cannot undo the committed prefix or its KV state.

SDTT works as follows:

  1. At the first S2T-rule disagreement, roll back rrb=7r_{\text{rb}}=7 tokens.
  2. A learned C2C projector fuses the LLM's KV cache into the SLM's, giving a repaired root s1s_1. The unrepaired root s0s_0 is kept.
  3. The frozen SLM expands a 4×24\times2 tree from each root.
  4. An 18-feature MLP ranks the resulting answer basins and returns the top basin's medoid, or the original output if the score is below a threshold.

With Qwen3-1.7B/0.6B (SLM) and Qwen3-32B (LLM) on five math benchmarks, it reports Math Avg. 50.22 at 1.7B (+34.64% over greedy) and 37.52 at 0.6B. It also includes GSM8K ablations, reachability/selection theory and two exploratory audits.

Critical strengths

S1. The paper asks the right question: does an intervention change which futures the SLM can reach? It also supplies the right tools to answer it: the reachability vs. selection split (App. B.2) and a budget-matched second no-op tree (Prop. 2). This could separate "the LLM helped" from "more sampling helped."

S2. Exploratory audits are reported transparently with exact paired tests (App. C.2–C.3), and the reported pp-values and CIs match the counts. This makes the claims checkable.

Critical weaknesses

W1. The main comparison is unreliable.

  • All Greedy/Maj@8/S2T-Local/R2R/Takeover/S2T entries in Table 1 exactly match FutureBridge (arXiv:2608.06819, Table 1). That paper uses the same models, benchmarks and token limit, and the submission gives no attribution.
  • FutureBridge's own column is reported lower than in its source:
    • Math Avg.: 49.5049.50 vs. 50.3850.38 (1.7B) and 34.9034.90 vs. 43.6443.64 (0.6B).
    • GSM8K: 96.096.0 vs. 96.896.8.
    • Latency: 3.80×3.80\times vs. 3.30×3.30\times, although the Table 2 caption says its cost isn't reported.
  • With the source's numbers, SDTT loses at both scales (50.22<50.3850.22<50.38; 37.52<43.6437.52<43.64) and on GSM8K (96.2<96.896.2<96.8), where all ablations are run. This is not a missing-comparison complaint: the authors chose to report this baseline.
  • Over S2T, the gain is only +1.08+1.08 / +1.04+1.04 Math Avg. It costs +52%+52\% latency (4.83×4.83\times vs. 3.18×3.18\times) and 5.3 vs. 4.8 LLM calls, despite the "matched collaboration budget" heading.
  • No seeds or CIs are reported, and one AIME problem is worth 3.33.3 points.
  • The headline +34.64%+34.64\% is measured against greedy, a comparison Sec. 5.2 itself says doesn't isolate the method.

W2. The mechanism is unsupported and the diagnostics are inconsistent.

  • Repair lowers reachability. Pr⁡(E0∪E1)−Pr⁡(E0∪E0′)=94.63−95.42<0\Pr(E_0\cup E_1)-\Pr(E_0\cup E_0')=94.63-95.42<0. By the paper's own Prop. 2, the repaired tree adds fewer correct candidates than a second no-op tree.
  • The audits are null or negative. Repaired topology vs. no-op topology is 55.65%55.65\% vs. 60.48%60.48\% (8 wins / 14 losses, p=0.29p=0.29). The projector holdout is net −3-3.
  • Resampling explains most of the gain. No-op16 (no repair) already reaches 95.10%95.10\% on GSM8K vs. 82.0%82.0\% greedy, which is about 13 of SDTT's ~14 points.
  • Table 4 violates Eq. (8). Because yorigy_{\text{orig}} is shared, E[Yπ−Y0]=W2C−C2W\mathbb{E}[Y_\pi-Y_0]=\mathrm{W2C}-\mathrm{C2W} forces Acc−(W2C−C2W)\mathrm{Acc}-(\mathrm{W2C}-\mathrm{C2W}) to be constant across conditions. The table gives 91.83/90.54/90.75/93.9791.83/90.54/90.75/93.97 instead. SDTT has the highest accuracy but the lowest net transition (+2.23+2.23).
  • Other impossibilities:
    • No-op16's correct-basin mass (98.4598.45) exceeds its correct-candidate rate (94.8794.87).
    • Paired C2C and SDTT differ only in selection, yet report different reachability (94.1194.11 vs. 94.6394.63), so the selector gain is confounded with sampling.
    • SDTT's ablation entries are taken from Table 1, not from the ablation run (Secs. 5.3–5.4).

Secondary

  • Prop. 1 is vacuous. Successors of s1s_1 and s0s_0 never coincide, so ν1⊥ν0\nu_1\perp\nu_0, and the hypothesis then requires every successor of s1s_1 to beat every successor of s0s_0. An immediate-EOS successor (qh−1=0q_{h-1}=0) already breaks this.
  • The trigger daudd_{\text{aud}} and the training data are unspecified. Takeover and S2T-Local are undefined, and R2R is uncited.
Questions for authors

Clarifications

  1. Which Table 1–2 numbers come from arXiv:2608.06819, and which were re-run? Why does the FutureBridge column differ from its source? What remains of the claim with the source's numbers?
  2. Over which states are W2C, C2W and Fallback computed? Why does Acc−(W2C−C2W)\mathrm{Acc}-(\mathrm{W2C}-\mathrm{C2W}) vary across conditions? Please re-run all ablations on the same trees, with seeds and CIs, ideally on MATH-500 or OlympiadBench.
  3. Is yorigy_{\text{orig}} the greedy path or the S2T path? How do one trigger and one injection produce 5.3 LLM calls? How does No-op16 reach 95.10%95.10\%?
  4. Please specify:
    • the exact daud(t)d_{\text{aud}}(t), including whether it uses LLM outputs;
    • whether thinking mode is on;
    • decoding parameters and bridge length;
    • the projector/selector training data, and any overlap with evaluation states (Sec. 4.4 says they "remain in evaluation");
    • the number of runs per Table 1 entry (79.7%≠k/50079.7\%\neq k/500, and greedy cannot score 5.6%5.6\% on 30 problems).

Suggestions

  • Add matched-compute baselines: LLM-only, speculative decoding, and Maj@16, PRM best-of-NN or confidence-weighted voting over the same ~16 samples.
  • Report tokens, FLOPs and memory alongside latency.
  • Ablate the tree shape. By App. B.3, Phit(4×2)=1−(E[(1−Q)2])4≤1−(1−EQ)8=Phit(8×1)P_{\text{hit}}(4\times2)=1-(\mathbb{E}[(1-Q)^2])^4\le1-(1-\mathbb{E}Q)^8=P_{\text{hit}}(8\times1).
  • Discuss prior work on rollback on disagreement, e.g. BiLD (NeurIPS 2023).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.