acceptodds
Under review as a conference paper at ICLR 2027

Compass: Repairing When Target-Model Guidance Hurts in Parallel Speculative Decoding

Abstract

Target-model features are increasingly used to improve parallel speculative drafting, but stronger guidance does not necessarily translate into faster decoding. We reveal that target-model guidance can hurt it in two ways: feature injection adds drafting overhead that can limit the speed benefits of improved acceptance, while target-guided adaptation can introduce new errors in prefixes already accepted by the original drafter, undermining the benefit of target-model guidance. We introduce *Compass*, a lightweight target-guided parallel speculative decoding framework that combines compact layerwise target guidance with reference-aware guidance optimization. The guidance module compresses multi-layer target states into a shared low-dimensional source and injects separate gate and up input shifts into each draft MLP, reusing native projections to limit drafting overhead. The optimization module uses the initial drafter as a frozen reference model to protect previously accepted prefixes and encourage recovery beyond its acceptance boundary. We construct a margin-based prefix objective aligned with greedy verification and derive a per-block lower bound linking the designed loss to acceptance gain over the reference model. Implemented in vLLM, *Compass* achieves mean AR-relative speedups of 5.27, 4.25, and 2.39 on Qwen3-8B, Qwen3-14B, and Qwen3-32B across six benchmarks, exceeding the strongest available baseline for each target by 7.0%, 8.1%, and 12.4%. Ablations show that compact guidance retains acceptance benefits at lower drafting cost, while reference-aware optimization extends accepted prefixes and accelerates decoding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, Large Language Models, Acceptance-Aware Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_19hvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a parallel speculative decoding framework called Compass. The main contribution is to combine with two components: 1) Compact Layerwise Target Guidance compresses multiple target hidden states into a shared low-dimensional representation and injects separate shifts into the gate and up inputs of each draft MLP; 2) Reference-Aware Guidance Optimization uses a frozen initial drafter to distinguish previously accepted prefixes from prefixes requiring repair, applying different losses to these two groups.

Theoretically, the paper also derives a per-block lower bound on acceptance improvement under greedy verification. Experiments on vLLM report improvements in acceptance and decoding speed across three Qwen3 target models and six benchmarks.

Critical strengths

In general, the paper provides some practical findings while running parallel SD:

  • Some new findings ignored by previous works: improvements in draft acceptance can be offset by the computational overhead of target conditioning. The diagnosis are well-motivated with empirical results.

  • Proposal of Compact Layerwise Target Guidance and Reference-Aware Guidance Optimization to adress the new problems.

  • Extensive evaluations based on vLLM framework with existing schemes such as DSpark, Dflash, Domino, etc.

Critical weaknesses

While I appreciate the new findings and the proposed method, I think the paper has the following weaknesses:

  1. Overall, the paper builds on the existing technique/architecture and the contribution on the architectural side is rather incremental. Multilevel target-feature fusion, target-dependent MLP steering, and low-dimensional projections are well-established ingredients. The usage of MLP projection is expected and does not conceptually differentiate this work from the existing ones. To this end, I don't see much that is fundamentally new in this work.

  2. The design of acceptance-aware training objectives is not new either. The related-work section identifies several acceptance-aware objectives, including prefix-product objectives and adaptive position weighting. The main ablation compares RGO with KL+CE, but does not compare it against these closer alternatives.

Conceptually, RGO is similar to the existing re-weighting schemes -- I see "reference-aware protection/repair split" as a good packaging of the method rather than a fundamental novelty. Specifically, it should compare RGO against: (a) a plain KL regularizer toward the frozen reference, and (b) a uniform margin loss with triangular weights but no repair branch.

  1. The theoretical part looks a bit forced, or AI-generated. I don't get much from it, neither by reading the proofs.

  2. The cost of the proposed method is not analyzed or evaluated. While the paper argues the proposed method as low-cost, the inishgts from guidance latency, peak memory, total training cost, and end-to-end throughput against a matched baseline is missing in the manuscript.

  3. Results other than Qwen are prefered.

Questions for authors

Please check the weaknesses part.

Reviewer_bPe7OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies target-guided parallel speculative decoding and argues that stronger target-model guidance can hurt decoding speed in two ways: by increasing draft-time overhead and by damaging prefixes that were already accepted by the original drafter. To address this, the authors propose Compass, which combines: (1) Compact Layerwise Target Guidance (CLTG) compresses multi-layer target features into a low-dimensional source and injects per-layer gate/up shifts into each draft MLP, aiming to preserve the benefits of layerwise guidance at lower drafting cost. (2) Reference-Aware Guidance Optimization (RGO) uses the initial drafter as a frozen reference model and optimizes a margin-based prefix objective that separates “protection” of already accepted prefixes from “repair” beyond the reference acceptance boundary.

Critical strengths
  1. The paper focuses on an important systems-level failure mode of target-guided speculative decoding. The distinction between improving proposal quality and improving end-to-end speed is important, and the paper articulates this clearly. The identification of guidance overhead and acceptance damage as separate failure modes is useful and practically relevant.
  2. The method is reasonably well motivated and supported by targeted ablations. The compact guidance design is intuitive, and the CLTG/RGO ablations do suggest that both components contribute. The implementation in vLLM and evaluation across multiple targets and benchmarks also strengthen the paper’s practical relevance.
Critical weaknesses
  1. The empirical evidence is weaker than the headline results suggest. The main table reports no uncertainty estimates, repeated-run variation, or significance analysis, while several claimed gains over baselines are relatively modest. Baseline coverage is also uneven across targets, making the “best overall” conclusion less convincing than it first appears.
  2. The paper does not cleanly establish how much of the gain comes from the proposed optimization objective versus stronger target guidance and additional training capacity. Although the ablations are useful, the evidence still falls short of isolating the specific contribution of the reference-aware objective. In particular, the practical importance of the lower-bound analysis remains unclear, since the paper does not show that the bound is predictive of training progress or model selection, beyond motivating the loss construction.
Questions for authors
  1. Please report uncertainty estimates or repeated-run variability for the main results in Table 1. Some reported improvements over the strongest baseline appear relatively small.
  2. Please clarify what is established specifically about the benefit of the reference-aware objective beyond the effect of adding target guidance and further training. A more explicit attribution would strengthen the main methodological claim.
  3. Please clarify the practical role of the acceptance-gain bound in the paper. Is it mainly a motivation for the loss design, or does it provide actionable guidance for optimization, model selection, or interpretation of training behavior?
  4. The study is restricted to greedy decoding and a single drafter family on Qwen3 targets. Please clarify the intended scope of the conclusions and which claims should be interpreted as setting-specific.
Reviewer_1X7cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper highlights a good observation that the hidden states from the target model contains lots of information, but injecting such info to the drafter does not necessarily lead to improvement of speed and could even decrease the acc len. Compass added two components, which are Compact Layerwise Target Guidance (CLTG) and Reference-Aware Guidance Optimization (RGO) to reduce the impact of target guidance. Experiments on Qwen3 and 6 benchmarks have demonstrated that COMPASS can achieve up to 5.27× speedup.

Critical strengths
  1. The authors have identified target-guidance as a inherent limitation in previous methods and have conducted analysis on the reparied and damaged acceptance endpoints to provide useful evidence.
  2. The design of CLTG is simple and efficient. Reusing the existing MLP projections reduces the parameter and latency overhead associated with expanded-space guidance.
  3. Compass achieves consistent improvements across the evaluated target models and benchmarks. The component ablations and runtime breakdowns provide useful evidence for the complementary benefits of CLTG and RGO.
Critical weaknesses
  1. The authors have mentioned other optim methods such as D-PACE and SpecDiff-2, but there is no experiment to compare Compass with these methods to highlight the necessity of Compass's RGO.
  2. All main experiments use greedy decoding, while practical speculative decoding frequently involves stochastic sampling. It is unclear whether the same training objective improves acceptance under stochastic verification.
Questions for authors
  1. Why is K set to 8 for training but 16 for inference in Table 4 and 5.
  2. Can the authors provide empirical measurements of the tightness of the acceptance-gain bound throughout training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.