acceptodds
Under review as a conference paper at ICLR 2027

Model Priming: Unverified Handoffs for Efficient Decoding

Abstract

The fast adoption of large language models is straining global compute, making inference efficiency both an economic and environmental imperative. We investigate primed decoding: a scheduled handoff policy in which a small model M_q continues a generation initiated by a larger, more capable model M_p. Unlike speculative decoding, primed decoding performs no verification and requires no router, retraining, or shared serving infrastructure. We evaluate it on five model pairs from two model families and six reasoning, mathematics, and code benchmarks, measuring accuracy together with throughput, cost, and energy on H200 GPUs. For instance, generating only the first 1024 tokens with Qwen3.5-122B-A10B and the rest with Qwen3.5-4B recovers 92% of Qwen3.5-122B-A10B's performance and reduces single-request latency by 1.6–1.8× on long generations. At the same time, it cuts cost by 60% and energy consumption by 68%. These results suggest that a short high-capability prefix can establish a reasoning trajectory that a smaller model often sustains, enabling substantial offloading of generation to cheaper models without token-level coordination.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
decoding, efficient, fast, llm, generation, handoff

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gGLxOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Model Priming, a simple method that utilizes small model and large model together to save computation. Specifically, the large model simply generates a fixed number of tokens first, and then the small model continues to generate after that point. The performance evaluations showcase that the overall performance interpolates between the two models’ performance with varying amount of tokens written by the large model.

Critical strengths

The approach is extremely simple and straightforward, and empirically shows some performance preservation with notable cost reduction.

Critical weaknesses

The paper lacks comparisons with relevant baselines. There is a large body of prior work on utilizing small and large models together to improve efficiency, but the paper compares against only a few, primitive approaches. For example, COPE [1] uses a large model as a planner, and a small model as executor. This closely matches with the paper’s setup of a large model generating a small amount of text and the small model continues generating, as the large model can generate a concise plan and a small model could execute it. Other examples may include works like SpecReason [2], where large and small models collaborate to improve efficiency while preserving or even improving performance. Comparisons against existing state-of-the-art methods are necessary in order to claim that the given performance preservation at the given cost is meaningful.

Furthermore, the paper lacks analysis on why priming works, only providing speculative explanations. Without them, the paper would not provide sufficient insights to add to the community.

[1] Efficient LLM Collaboration via Planning, TMLR 2026

[2] SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoning, NeurIPS 2025

Questions for authors

Could the authors provide comparisons with other state-of-the-art baselines, and a deeper analysis about why model priming works well?

Reviewer_Zjn2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

A large model generates a fixed-length prefix, after which a smaller model completes the response without verification. Experiments across five model pairs and six benchmarks characterize accuracy, latency, cost, and energy tradeoffs. One reported configuration retains 92% of the large model's accuracy at roughly 40% of its cost, with additional schedule and cross-family ablations.

Critical strengths
  1. The approach is easy to deploy and achieves substantial savings in favorable model/task configurations.
  2. Multiple model pairs, recent reasoning benchmarks, and schedule ablations give a useful empirical picture of the tradeoff.
Critical weaknesses
  1. A fixed large-to-small handoff is a restricted multi-model decoding schedule. The ablations do not fully isolate the special value of early reasoning from generic benefits of conditioning on a stronger model's output.

  2. The robustness of the accuracy–cost advantage is uncertain. Useful prefix lengths and accuracy retention vary across tasks and model pairs. Handoff-selection transfer and matched-cost routing comparisons are insufficiently established, limiting confidence in deployment on unseen task mixtures.

Questions for authors
  1. What distinct empirical or conceptual contribution does fixed-prefix priming add to existing cascading and token-level routing approaches?
  2. What evidence separates the value of early reasoning from problem restatement, planning, or generic conditioning on a stronger model's output?
  3. How is the prefix length selected without evaluation-label feedback, and does a length chosen on validation tasks transfer to unseen tasks and model pairs?
  4. How does the method compare with feasible token-level routing at matched total inference cost, including prefill, switching overhead, and completion length?
  5. What trace-level failure patterns explain weaker configurations, and can harmful handoffs be anticipated from the prefix or model pair?
Reviewer_repcOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies primed decoding, in which a large model MPM_P generates the first kk tokens of a response and a small model MQM_Q generates the rest, without verification, a router, or retraining. The authors evaluate five model pairs from the Qwen3.5 and Gemma-4 families on six benchmarks (GPQA, AIME'26, MATH-500, HumanEval, MBPP, and HLE), using k∈{256,…,2048}k \in \{256, \dots, 2048\} and five seeds. They compare priming with MPM_P or MQM_Q alone, a confidence cascade, and speculative decoding. They measure accuracy, single-request latency, monetary cost, and GPU energy on H200 GPUs. For Qwen3.5-122B-A10B ⇒ Qwen3.5-4B at k=1024k=1024, priming retains 92% of MPM_P's accuracy, runs 1.6–1.8× faster, and reduces cost by 60% and energy use by 68%. Ablations show that interleaved and reverse schedules are less cost-effective, prefixes shorter than 256 tokens provide no gain, and handoff between model families (Qwen ⇒ Gemma) also works.

Critical strengths

S1. A simple, deployable method with a careful cost study. Priming requires no verifier, router, or retraining, and it can operate through APIs across model families without a shared serving stack. The paper measures accuracy, latency, monetary cost, and energy, uses five seeds, and provides a closed-form cost condition (Eq. 4). This makes the trade-off clear for practitioners.

S2. Useful ablations on when to use the large model. The paper tests reverse priming, interleaving, and very short prefixes. It finds that prefixes shorter than 256 tokens do not help and that using the same token budget at the start is more effective than using it later. This is a clear, practical observation.

Critical weaknesses

W1. The latency results use a different workload from the accuracy evaluation and may overstate the gains. The speedups are measured with greedy decoding on fixed-length outputs that ignore EOS (Appendix H). The authors acknowledge that these results are an upper bound. Table 10 shows that, for Gemma-4-31B ⇒ E4B at k=2048k=2048, MPM_P generates 55% of GPQA tokens and 82–94% of HumanEval/MBPP tokens. Thus, actual latency gains on these tasks are likely much smaller than 1.5×. Even under the optimistic workload, speculative decoding is lossless and faster for two model pairs (e.g., Gemma: 1.96× speedup at 100% accuracy retention, compared with 1.50× for priming at 90%). The latency claims in the abstract and Fig. 1 should be qualified.

W2. The accuracy–cost trade-off is modest, and important baselines are missing. For Qwen3.5-122B ⇒ 4B, MQM_Q alone reaches 83.8% of MPM_P's accuracy at 4.4% of the cost. Priming at k=256k=256 reaches 86.4% at 18.8% of the cost—about four times the cost for a 2.6-point gain. At k=2048k=2048, it reaches 92.3% at 48.1% of the cost, close to the cascade with 40% deferral (92.1% at 51.6%). The average accuracy also includes tasks where MPM_P generates most of the answer, so some of the reported "recovery" may come from MPM_P answering directly. The paper lacks comparisons at matched cost with two natural baselines: (i) a mid-size model alone (e.g., Qwen3.5-9B or 27B) and (ii) additional test-time compute for MQM_Q, such as majority voting. Learned query routers (e.g., RouteLLM) are also missing.

W3. Limited novelty and analysis. The main idea—that a strong prefix can steer a weaker model—follows from prior work cited in the paper (Sun et al., 2025). The method uses a single, fixed handoff, with kk selected by a sweep. The paper does not explain why priming succeeds or fails. For example, it does not analyze the content of the prefix at the handoff point, and the authors state that they "did not verify" their explanation for why short prefixes fail. The paper also provides no adaptive rule for selecting kk per query. On difficult tasks (HLE and AIME with Gemma), accuracy retention falls to 60–79%, which limits the method's usefulness.

Questions for authors
  1. Can you report latency and energy per query on sampled responses with EOS, rather than per token on fixed-length outputs?
  2. How does priming compare at matched cost with a mid-size model alone and with majority voting across several MQM_Q samples?
  3. How much of the accuracy gain remains when measured only on queries that actually require a handoff?
  4. After priming, does MQM_Q generate longer or shorter continuations than it does alone, and how does this affect total cost?
  5. Most closed APIs hide reasoning tokens. How practical is an API-level handoff in this setting?
  6. Could a simple signal (e.g., the end of the planning phase or entropy) be used to select kk for each query?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.