acceptodds
Under review as a conference paper at ICLR 2027

Divergence Decoding: Training-Free Capability Fusion

Abstract

While large language models excel in reasoning, these generalists often lack knowledge for specialized scientific domains. Conversely, domain models (specialists), while knowledgeable, suffer from specialization side-effects including diminished logic and reduced robustness. To address this dilemma, we introduce Divergence Decoding, a training-free framework for capability fusion. It reconstructs the "draft-and-verify" skeleton of speculative decoding into an adaptive routing mechanism. The core is using Jensen-Shannon divergence to monitor the distributional disagreement between the two models at each token. When the specialist exhibits significant divergence, our method identifies it as a potential reasoning risk and instantaneously routes control to the generalist. This allows the dynamic injection of general reasoning while preserving domain expertise, achieving inference-time policy composition of the generalist and the specialist. We evaluate Divergence Decoding across diverse model families (Qwen and Llama series) on challenging scientific benchmarks (GPQA, ChemBench, and ChemCoTBench). Experimental results demonstrate that Divergence Decoding outperforms both the domain-specialized and general-purpose models, effectively surpassing the performance of most single-model baseline. This suggests that Divergence Decoding provides a general, training-free paradigm for fusing diverse LLM capabilities through adaptive inference-time collaboration.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
LLM, reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_95BQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a training-free decoding method that combines a domain-specialized model with a general reasoning model using token-level routing based on JS divergence. The method is simple and achieves strong improvements across several scientific reasoning benchmarks. However, the evaluation is limited to scientific specialist-generalist settings, the theoretical guarantees rely on strong assumptions about when each model should be preferred, and the current baselines do not fully demonstrate that the gains come specifically from the proposed JS-divergence routing mechanism. Overall, the paper presents a practical and effective capability-fusion approach, but its central claims would benefit from stronger theoretical and empirical support.

Critical strengths

The paper presents a simple and practical approach for combining specialist and general-purpose language models at inference time without requiring additional training. The proposed divergence-based routing mechanism is model-agnostic and can be applied to different specialist-generalist pairs with minimal changes to the underlying models. Empirically, the method demonstrates strong and consistent improvements across multiple scientific reasoning benchmarks, often outperforming both constituent models.

Critical weaknesses
  1. The theoretical results rely on assumptions that are not directly validated in the paper. In particular, the guarantees require that low-divergence states favor the specialist model and high-divergence states favor the generalist model. Since the effectiveness of the method depends on these assumptions, the theory provides limited evidence that JS divergence is a reliable routing signal beyond the studied settings.

  2. The empirical evaluation is restricted to scientific specialist-generalist model pairs. It is therefore unclear whether divergence-based routing would remain effective for other forms of specialization, such as coding, mathematics, or legal reasoning.

  3. The paper does not fully isolate the contribution of JS-divergence routing. While comparisons are provided against speculative decoding, self-resampling, and CITER, stronger training-free routing or model-selection baselines are absent. As a result, it is difficult to determine whether the gains arise specifically from the proposed divergence criterion or more generally from combining complementary models.

  4. The routing policy depends on a calibrated divergence threshold. The paper proposes a threshold-selection procedure, but provides limited analysis of how sensitive the method is to threshold choice across different benchmarks and model pairs.

  5. The proposed approach requires inference with both models throughout decoding. This increases computational cost and latency relative to single-model generation, and the paper provides only limited analysis of the resulting cost-performance tradeoff.

  6. The paper mainly analyzes successful capability fusion settings. A more detailed characterization of cases where routing fails, or where one model consistently dominates the other, would help clarify the limitations of the proposed approach.

  7. There are a few minor presentation issues. For example, a LaTeX expression appears to be incorrectly rendered around line 819, and there is a quotation-mark formatting issue around line 928. A careful proofreading pass would improve the overall presentation quality

Questions for authors

Chek my comments in Weaknesses

Reviewer_YdSVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper comes up with a novel method called Divergence Decoding which is a training-free way to combine a domain-specialist LLM and a general reasoning LLM at inference time. The specialist drafts a short block of tokens, and the generalist reads it in one forward pass. At the first position where the two models' Jensen–Shannon (JS) divergence exceeds a threshold, the rest of the draft is dropped, one token is sampled from the generalist, and the specialist takes over again. Unlike speculative decoding, the method does not preserve either model's distribution. A theorem in Appendix A states that, under assumptions on both models, this policy beats either model alone. Experiments on several science benchmarks report that the combination beats both models on most tasks, while only a small fraction of tokens come from the generalist.

Critical strengths
  1. The method is simple, training-free and cheap. No router training or fine-tuning is needed, and verifying a whole block in one pass keeps latency close to one model. If the gains hold, the method would be easy to adopt.
  2. The results are convincing. On GPQA-Chemistry, taking generalist tokens at random positions at the same rate barely helps, while the divergence gate helps a lot. The per-question analysis shows the method fixes many questions and breaks almost none. This is the strongest evidence that where the method intervenes matters. The observation that a single connective ("Next" → "Wait") can redirect a long reasoning chain is also interesting. My regret is that this is run on only one benchmark, the one with the weakest baselines.
Critical weaknesses
  1. High divergence does not mean the generalist is right, but the method and the theory assume it does.

--> JS divergence does not mean B's distribution has an advantage over A. It says the models disagree, not which one is right, yet the method always defers to the generalist. Where the generalist lacks domain knowledge, this replaces a correct specialist token with a wrong one. Fig. 4c shows a non-negligible share of interventions land on chemical and symbolic tokens, exactly where the generalist should be weakest.

--> The theory assumes this problem away. Assumptions A.1 and A.2 turn the gate into a perfect switch that always picks the better model, so the theorem holds almost by definition. These assumptions are never checked, and the paper's own results contradict them. At the threshold used, which sits close to JS's maximum, the generalist would need to match the ideal distribution almost exactly everywhere. Yet it performs at chance on GPQA-Chemistry. The predicted strict improvement also fails on ChemBench Toxicity and Safety (Qwen pair). The theorem also bounds the wrong quantity. Its losses are averaged over the states the combined method visits, not the states each model visits alone. So it says nothing about final accuracy compared with each model alone, which is what the experiments measure.

  1. The comparison is not compute-fair, and the key ensembling baselines are missing. --> The method uses the compute of two models but is compared only with single models. The natural baseline is a weighted mixture of the two models' probabilities, which is the smooth version of the hard gate. Sampling from this mixture is the same as routing tokens to the generalist at random, so the paper's random control is already one point of this baseline. We need a sweep over weights. A soft gate, whose weight on the generalist grows with the disagreement, would also show whether the hard switch is needed or whether it is too abrupt and could degrade downstream generations. Also why not combine the distributions of both models.

--> The gains may not come from the gate:

  1. Self-Resample, which never uses the specialist, nearly matches the full method on several ChemCoTBench tasks (Table 4), and Self-Resample is never defined.
  2. On GPQA-Chemistry, both single-model baselines are at or below chance and far below published numbers. This suggests cut-off outputs or failed answer extraction. Methods appear to be scored on different GPQA question sets, and the gains are close to run-to-run noise.
Questions for authors

Experiments that could change my assessment:

  1. Accuracy vs. intervention rate, against a smooth ensemble. Please sweep τ over its full range (0 to ln 2) and plot accuracy against the share of tokens from the generalist. On the same plot, add the weighted mixture of the two models, sweeping its weight, with both tuned by the same calibration procedure. If the hard gate sits above the mixture, where the method intervenes matters. If not, the gain comes from simple mixing. This plot also shows how robust the method is to τ, which the current narrow, single-run sensitivity analysis cannot.

  2. Does high divergence mean the generalist is right? At each intervention, continue once with the specialist's token and once with the generalist's, and compare final accuracy, separately for connective and domain-specific tokens. This directly tests the core assumption and A.1.

  3. Direction-aware and cheaper baselines. How does the method compare with (a) a lightweight learned router trained for accuracy, e.g., a linear probe on both models' hidden states or Co-LLM, and (b) simply inserting "Wait" into the specialist's output, as in budget forcing (s1)? Since most interventions are early connectives (Fig. 4a), (b) tests whether the generalist is needed at all.

  4. Which model pairs and tokenizers does the method need? Each pair changes family, size and benchmark at once, and every generalist is R1-distilled. The only clear cross-tokenizer pair is tested on a very small set. Holding the specialist and benchmark fixed, could you vary only the generalist (same vs. different family and tokenizer, reasoning-distilled vs. instruction-tuned), using the plot from Q1?

Reviewer_rgYcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Divergence Decoding combines a scientific specialist and a general reasoning model without training a router. The specialist drafts token blocks; the generalist evaluates their prefixes, and a Jensen–Shannon divergence threshold determines whether to accept specialist tokens or sample a generalist replacement and restart drafting. The implementation uses top-10 distributions and describes an alignment procedure for different tokenizers. Experiments cover Qwen- and Llama-based chemistry pairs and additional GPQA domain experts, with alternative routing/sampling comparisons, a matched-rate random-intervention control, and latency measurements. A theoretical result gives sufficient conditions for improving expected next-token distributional risk over both models.

Critical strengths
  1. The routing rule is simple and potentially useful when both constituent models already exist. Tables 1–3 show improvements over the stronger constituent on many chemistry tasks across two model families, while also exposing exceptions. The blockwise implementation makes the intervention mechanism concrete without claiming distribution preservation.

  2. The GPQA-Chemistry control is more convincing than isolated qualitative examples. At approximately matched intervention frequency, random intervention achieves 12/72 correct, compared with 27/72 for divergence gating; the specialist alone obtains 11/72. The paired specialist comparison reports 17 corrections and one regression. These observations support the relevance of intervention location, although they remain a small evaluation (Appendix C).

Critical weaknesses
  1. The operative routing and calibration procedures are underspecified at points that matter for reproduction. Section 3 defines JS between next-token distributions, but Appendix B uses truncated supports and sequence-level alignment across tokenizers. It does not fully specify the shared outcome space, probability aggregation for one-to-many token alignments, or the treatment of unaligned mass. The label-free calibration also leaves the precise predictive-support and self-consistency estimators insufficiently defined, and reports an estimated ChemCoTBench threshold of 0.64 versus an adopted 0.65. These details determine which tokens are routed, especially for heterogeneous GPQA pairs.

  2. The evidence does not yet establish a general capability-fusion advantage at comparable resources. Table 4 adds useful Self-Resample and CITER controls, but does not specify their sampling/reranking budgets, router-training protocol, or repeated-run uncertainty. Table 6 gives latency against an ambiguously identified Single LLM rather than a full resource-matched comparison. Meanwhile, Theorem A.1 assumes a uniformly accurate generalist and a separated specialist-advantage region; it compares risk under a common state distribution and does not establish end-task accuracy after each policy induces different prefixes. It also does not directly cover the implemented truncated/aligned statistic. The empirical gains are promising, but these gaps limit the breadth of the explanation and comparison.

Questions for authors
  1. Implementation clarification: Can you provide executable pseudocode and a worked example for one-to-many tokenizer alignment, including normalization, unaligned probability mass, and exactly which distributions are used before or after temperature/top-p/top-k processing?

  2. Calibration clarification: Please define the predictive-support and self-consistency estimators, calibration sample/prefix counts, and thresholds for every model pair, including GPQA Biology and Physics. Were their calibration examples excluded from final scoring, and how was the adopted 0.65 selected relative to the estimated 0.64?

  3. Decisive comparison: Under matched generation/verification resources, do the improvements over Self-Resample and CITER persist across seeds? Please identify the timing baseline and report paired correctness and uncertainty against the generalist, not only the specialist, on GPQA. The theorem’s state-distribution and truncation scope should also be stated explicitly.

Reviewer_aProOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Divergence Decoding, a training-free framework for combining domain-specialized and general-purpose language models. The method uses Jensen–Shannon (JS) divergence between their next-token distributions to determine whether to accept the specialist's output or fall back to the generalist. A blockwise draft-and-verify procedure implements this routing strategy. Experiments on ChemBench, ChemCoTBench, and GPQA report improvements over individual models in several scientific tasks.

Critical strengths

1.The paper addresses a practically relevant problem. Combining domain-specific knowledge with general reasoning capabilities is a meaningful direction for improving scientific language models.

2.The proposed framework is simple and shows empirical improvements. The training-free approach is straightforward to implement, and experiments across multiple scientific benchmarks demonstrate potential benefits from adaptive model collaboration.

Critical weaknesses

1.Limited methodological novelty and insufficient differentiation from existing routing approaches.

The proposed method essentially replaces a learned token-level router with a fixed JS-divergence threshold to select between two language models. Token-level expert collaboration has already been investigated by ETR, CITER, and FusionRoute. Although avoiding router training is convenient, the paper does not establish a substantial algorithmic advance beyond existing routing frameworks. The comparison with CITER is restricted to one model pair on ChemCoTBench, and the proposed method does not consistently outperform alternative strategies across all evaluated metrics. Therefore, the incremental methodological contribution remains limited.

2.Accuracy improvements come at the cost of inconsistent and sometimes substantially worse inference efficiency.

Despite improvements in accuracy, Table 6 shows that overall generation time increases from 200.16 s to 209.19 s, while molecule-understanding latency increases from 189.02 s to 233.04 s. Appendix E further reports substantial slowdowns on individual tasks. The method also requires running two large language models, increasing memory and computational requirements. Consequently, the claimed efficiency and scalability advantages are not convincingly supported. Without a demonstrated improvement in the accuracy–latency–memory trade-off over existing routing approaches, the practical significance of the framework remains questionable.

3.The core assumption that JS divergence indicates model reliability is not sufficiently validated.

High predictive disagreement does not necessarily mean that the specialist is incorrect or that the generalist provides a better prediction. The theoretical analysis relies on strong assumptions about the generalist's proximity to the unknown target distribution and the specialist's in-domain advantage. These assumptions are not empirically established. Although the authors provide divergence ablations and random-intervention controls, these experiments do not directly demonstrate that JS divergence reliably identifies which model should be trusted. Different benchmarks also require different routing thresholds, further limiting the generality of the proposed mechanism.

Questions for authors

1.What is the fundamental algorithmic contribution beyond using a fixed divergence threshold for existing token-level model routing? Can the authors establish a consistent advantage over alternative routing methods under matched computational budgets?

2.Given that overall inference time increases in Table 6, what practical efficiency advantage does Divergence Decoding provide? Can the authors report accuracy–latency–memory trade-offs against existing collaboration approaches?

3.Can the authors empirically demonstrate that JS divergence predicts the relative reliability of the specialist and generalist? How often do high-divergence interventions improve or degrade final accuracy?

4.Can a single threshold generalize across benchmarks and model combinations, and do the theoretical assumptions hold in the evaluated settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.