acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Target: From Imitation to Collaboration in Speculative Decoding

Abstract

Speculative decoding accelerates large language model inference by letting a small draft model propose tokens for parallel verification by a larger target. For tasks evaluated by answer correctness, however, agreement with the target is an imperfect criterion: a weaker draft can occasionally offer a continuation that leads to a better final answer. Exploiting these disagreements requires learning from delayed outcomes while retaining the efficiency of parallel verification. We present Collaborative Speculative Decoding (CoSpec), a framework that learns arbitration within speculative decoding from the outcomes of complete rollouts. An arbitrator jointly observes the context, draft block, and target verification tokens through a hybrid attention mask, and decides whether to retain the draft or fall back to the target at each mismatch. Reinforcement learning combines final-answer correctness with accepted-length shaping to train these decisions. CoSpec retains parallel target verification while optimizing task utility rather than preserving the target distribution. We evaluate two model families on reasoning, code, and knowledge benchmarks. With LLaMA-3.3-70B as the target, CoSpec achieves a mean speedup of 3.96× over target-only decoding and raises the mean score on GSM8K, HumanEval, and MBPP from 90.05 to 91.72. These results support learning from downstream outcomes as an effective way to exploit draft–target complementarity within speculative decoding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, LLM Agent, LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_b7BcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Collaborative Speculative Decoding (CoSpec), which learns to arbitrate disagreements between draft and target models to improve generation quality and inference efficiency jointly. An arbitrator uses a hybrid attention mask to access the context, draft block, and target verification tokens, and is trained through supervised warm-up followed by reinforcement learning on complete generation outcomes.

Critical strengths
  1. The paper provides evidence of improved task quality while retaining parallel verification.
  2. In this paper, SFT-only training, judge-free RL, reward ablations, target-model RL, and counterfactual branch analysis provide complementary evidence about the source of the gains.
Critical weaknesses
  1. There is no direct experimental comparison with DIVERSED, and comparisons with CoSD and AutoJudge have limited coverage.
  2. The causal-mask ablation removes both bidirectional interaction and target-token visibility, preventing isolation of the hybrid attention contribution. CoSpec and the target-RL baseline also use different training compute budgets.
Questions for authors
  1. How much improvement does hybrid attention provide over a causal or interleaved arbitrator when target verification tokens remain visible?
  2. Does CoSpec still win on both score and speed against CoSD, AutoJudge, and DIVERSED under one protocol?
  3. Does the 78% versus 20% acceptance gap in Table 3 remain under a different continuation policy?
Reviewer_1RMfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on speculative decoding, where we have a small draft model propose tokens for verification by a larger target language model, accelerating large language model inference. Instead of maintaining the response distribution of larger target model, this paper aims to improve the downstream task utility by leveraging the draft-target complementarity, achieving better task performance than target model or draft model solely. They propose to learn an arbitrator to decide whether to keep the draft or fall bak to the target through RL training. Experiments on GSM8K, HumanEval and MBPP demonstrate the improvement and speedup brought by the proposed method.

Critical strengths
  • The main problem is well explained, and the motivation is clear and sensible.
  • Comprehensive experiments across several different benchmarks show the effectiveness of the proposed method. Speedup is reported to demonstrate the benefits of the method more clearly.
Critical weaknesses
  • From the experiments, the evidence seems to be weak to say the accuracy gains come from complmentarity-aware arbitration. The improvement on some benchmarks may be not significant.
  • Some works that focus on the same problem is discussed in this paper but not compared. For example, speculative cascades, DIVERSED, etc.
Questions for authors
  • How is the proposed method compared to other works on the similar problems, like speculative cascades, DIVERSED, etc.
  • Is there any standard error reported for each model and benchmark? Would like to see whether the improvment is within noise.
Reviewer_KVyFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CoSpec learns whether to retain a draft token or use the target model's token at speculative-decoding mismatches, optimizing final-answer utility rather than preserving the target distribution. A separate LoRA-adapted arbitrator jointly reads the context, draft block, and target verification tokens through a hybrid attention mask. Judge-supervised warm-up is followed by reinforcement learning on complete speculative rollouts using correctness-conditioned span rewards and failure penalties. Experiments on LLaMA and Qwen pairs report simultaneous throughput and task-score improvements, transfer from math training to other tasks, and counterfactual branch analyses. Additional controls examine training signals, related verifiers, statistical significance, and serving overhead.

Critical strengths
  1. The method targets a useful speed-quality operating point and measures its extra inference cost. The arbitration rule in Section 3 preserves parallel verification while allowing useful draft disagreements to survive. Table 1 reports a LLaMA greedy mean score increase from 90.05 to 91.72 with 3.96x throughput relative to target-only decoding, and gains with the Qwen pair as well. Unlike an accepted-length-only evaluation, the reported speed includes arbitration. Appendix J.1 further quantifies latency and memory and shows the decline in benefit under higher concurrency, making the practical claim more informative.

  2. The controls provide meaningful evidence for outcome-based arbitration. Tables 5-6 compare SFT-only, no-judge RL, terminal-only rewards, reward components, and alternative uses of added capacity or supervision. No-judge RL still improves over the target, while full rollout training improves over judge SFT alone. The unequal-utility counterfactual cases in Table 3 are more informative than aggregate agreement: the policy accepts the draft much more often when its branch is better. Appendix G also gives paired discordant counts and confirms a net gain of 70 on full MATH after correction, while acknowledging nonsignificance on the smaller main benchmarks.

Critical weaknesses
  1. The evidence does not fully isolate the contribution relative to the closest outcome-trained methods and information access. Sections 1-2 correctly identify DIVERSED as already learning draft-target mixtures from sequence-level rewards and an acceptance regularizer, but the experiments do not compare against it. The CoSD and AutoJudge comparisons are useful but do not substitute for that comparison. In addition, the causal-mask ablation in Appendix D removes access to all target verification tokens at the readout positions as well as bidirectional interaction. It therefore cannot determine whether the proposed whole-round architecture improves over a simpler arbitrator with the same target evidence. These gaps limit the attribution of the gains to the specific arbitration design.

  2. The speed-quality claims need a more reproducible and complete operating-point comparison. Appendices C and J report aggregate throughput, latency components, and memory, but do not specify enough of the inference configuration to reproduce these results, including serving GPU type/count, precision, parallelism, engine configuration, and concrete length distributions. High-concurrency results compare only with target-only decoding, although EAGLE-3 is already close to CoSpec at the main LLaMA operating point (3.79x versus 3.96x mean speed). Training variability is also not captured by three evaluation runs of a fixed policy, and the strongest paired significance result is on full MATH rather than the three headline benchmarks. This limits confidence in the breadth and size of the advantage over an efficient competing system.

Questions for authors
  1. Under a shared model pair, training data, and comparable optimization budget, how does CoSpec compare with DIVERSED on the speed-quality frontier? This is the closest test of the benefit of explicit round-level arbitration over outcome-trained mixing.
  2. If a simpler causal or per-position arbitrator is given the same relevant target verification evidence, how much improvement remains from bidirectional whole-block interaction? The current mask ablation changes both information access and attention structure.
  3. Please provide the complete inference configuration and compare CoSpec with EAGLE-3 under the same higher-concurrency workloads, with latency/throughput uncertainty. Do the task-score gains also persist across independently trained arbitrators? These results would clarify whether the reported advantage is robust enough to justify the extra model and training cost.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.