acceptodds
Under review as a conference paper at ICLR 2027

HAWK: Rethinking Multimodal Drafting for Speculative Decoding

Abstract

Speculative decoding has achieved substantial lossless speedups for LLMs, but remains less effective for large vision-language models (LVLMs), where lightweight drafters struggle to use rich multimodal information. Additionally, standard distillation supervises the drafter only along the original training trajectory, without modeling how target predictions shift after the drafter's own proposals. As drafting moves away from this trajectory, the drafter can increasingly disagree with the target, reducing acceptance in later steps. We propose HAWK to address both limitations. HAWK uses representation similarity to select informative target layers and learns how to combine their hidden states. For visual information, it directly provides the drafter with compressed visual hidden states from the target model instead of raw visual tokens, making the visual information easier for a shallow drafter to use. HAWK also trains the drafter to capture how target predictions change after its own proposals, improving its agreement with the target during multi-step drafting. On SmolVLM-256M across ten multimodal benchmarks, HAWK raises average acceptance length from 3.32 to 4.08 and speedup from 2.19× to 2.60× over EAGLE-3 under greedy decoding, and from 2.89 to 3.41 and 1.92× to 2.19× under sampling.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
vision-language models, speculative decoding

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_i5ofOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies two limitations of speculative decoding for vision-language models: how to provide informative multimodal representations, and how to reduce the mismatch between the drafter’s training trajectory and its own multi-step predictions at inference time. To address the first issue, the authors propose representation-based target-layer selection and learned layer fusion, with pooling of visual hidden states. To address the second issue, they introduce prediction-shift supervision, which trains the drafter to match how the target model’s logits change when the original token is replaced by the drafter’s own proposal. The proposed method improves average acceptance length and inference speedup over EAGLE-3 and several multimodal speculative decoding baselines on multiple benchmarks.

Critical strengths

1.The information-injection design is well motivated.

The paper observes that fixed-depth layer selection may not align with where representations actually change across different VLMs, and that a shallow drafter may have difficulty extracting useful information directly from a long sequence of visual tokens. Using representation similarity to select target layers and reusing target-processed visual hidden states therefore provides a reasonable way to expose the drafter to richer and more compact multimodal information. Plus, using pooling to reduce visual token is also a common but efficient method.

2. Prediction-shift supervision is an interesting and relatively novel training objective.

This method explicitly models how the target prediction changes when the drafter’s own token alters the context. This is a more structured form of supervision than simply adding off-trajectory examples, and the ablation against direct distillation suggests that the shift-based objective provides additional benefit beyond using substituted contexts alone.

Critical weaknesses

1.The novelty and attribution of the layer-fusion component are somewhat limited.

CKA-based layer selection and learned feature fusion are straightforward extensions of existing representation-selection techniques. More importantly, HAWK reads substantially more target layers than EAGLE-3 and then fuses them into three streams. Therefore, part of the gain may come from exposing the drafter to more target features rather than from the specific representation-based selection strategy itself. A stricter comparison using the same number of target layers would make the contribution easier to isolate.

2. The motivation of prediction-shift supervision under greedy decoding is not yet fully convincing.

The substituted token is explicitly chosen to differ from the target model’s top-1 prediction. Under greedy speculative decoding, such a token would be rejected at that position, and predictions conditioned on the rejected token would not directly contribute to the accepted prefix. It is therefore unclear why accurately modeling the target’s behavior after such a substitution should improve deeper greedy acceptance, beyond acting as a form of perturbation-based regularization or context augmentation. The empirical gains are clear, but the causal explanation requires further clarification. The fact that direct distillation on the same substituted contexts already improves acceptance also suggests that part of the gain may come simply from training on drafter-induced contexts.

3. The visual pooling strategy is heuristic and its behavior across different visual resolutions is unclear.

The method uses a fixed 4x pooling factor, while the amount and structure of visual information can vary substantially with image resolution and token count. Although the paper compares no pooling, 4x and 16x pooling, it remains unclear whether a fixed compression factor is appropriate across different resolutions or whether an adaptive strategy based on image/token complexity and task categories would be more effective.

4.The scalability of the method to stronger contemporary VLMs remains insufficiently demonstrated.

The larger-scale experiments are limited to LLaVA-1.6-7B and Qwen2.5-VL-7B, and even these models are trained with a reduced data budget. These targets are relatively small and no longer representative of the strongest modern VLMs. It is therefore unclear whether the reported gains persist at substantially larger scales.

Questions for authors
  1. Could the authors further clarify the mechanism behind prediction-shift supervision, especially under greedy decoding? Since the substituted token is deliberately different from the target’s top-1 token, that branch would be rejected immediately during greedy verification. Why should accurately matching the target’s prediction shift after such a rejected token improve acceptance along the actual greedy trajectory? Is the main benefit better understood as on-policy context augmentation, local regularization of the drafter, or something else? Relatedly, the results appear sensitive to the weight of the shift loss: λshift=0.1\lambda_{\text{shift}}=0.1 performs better than a larger weight of 0.3. Could the authors explain why over-weighting the shift objective hurts performance, and how this hyperparameter interacts with the standard distillation objective?
  2. How should visual-state pooling behave across different image resolutions and visual-token lengths? The current method uses a fixed 4x pooling factor. Is the same compression ratio optimal for low-resolution and high-resolution inputs? Have the authors considered an adaptive pooling strategy based on resolution, token count, attention statistics, or visual complexity?
1 comment

W1 — Layer selection and fusion

HAWK uses CKA to choose which target states to pass to the drafter and learns how to combine them into three streams. Holding the ten CKA layers fixed in Table 4, learning the weights raises τ from 3.21 to 3.24 and speedup from 2.09× to 2.11×. The uniform grid uses nine layers and learned mixing, so comparing its τ of 3.17 with 3.21 changes both the layer count and the mixing rule.

The target computes these states in its usual forward pass. The measured speedup includes the cost of reading and mixing them during generation. We will add a layer-count sweep and compare CKA, uniform and random selections at the same number of layers per region with the same mixing rule.

W2 / Q1 — Prediction-shift supervision

Under greedy decoding, verification discards the substituted token and its descendants. We use these substitutions during training to teach the drafter how the target’s next-token predictions change after a likely drafting error. The reviewer’s direct-distillation comparison tests whether seeing the same altered contexts is enough.

In Table 3, direct distillation on those contexts raises τ from 3.05 to 3.10. With the same substitutions, prediction-shift supervision reaches 3.17 at weight 0.1 and 3.11 at weight 0.3. The two objectives see the same contexts, so the difference measures the effect of changing the training objective in this ablation.

The gap is larger deeper in the draft. Relative to standard distillation alone, the shift objective raises the probability of accepting the first k draft tokens by 3% at k=1 and 23% at k=5. Direct distillation improves it by at most 10% (Figure 3b). We will add runs across seeds and compare drafter proposals with other token substitutions.

W3 / Q2 — Pooling across resolutions

Pooling acts within each 64-row image tile. At 4×, the drafter keeps 16 rows per tile, so SmolVLM’s 9-, 13- and 17-tile inputs use 144, 208 and 272 rows, respectively. The target receives the full visual sequence for verification.

Across the six benchmarks in Table 2, 4× gives τ=3.10 and SR=2.10×, compared with τ=3.05 and SR=2.07× without pooling. At 16×, τ falls to 3.04. We will report τ and speedup by tile count and test an adaptive pooling factor.

W4 — Evaluation on larger targets

At the same 30K-example, 10-epoch budget, HAWK improves accepted length and speedup over EAGLE-3 on LLaVA-1.6-7B and Qwen2.5-VL-7B at both temperatures. Under greedy decoding, τ rises from 3.73 to 4.02 and from 3.19 to 3.40, respectively.

We will add a comparison on a newer target with HAWK and EAGLE-3 trained at the same budget.

Reviewer_VHh1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HAWK, a framework for improving speculative decoding in large vision-language models (LVLMs). HAWK selects target model layers based on representation similarity and combines their hidden states through learnable fusion weights. It also compresses visual hidden states computed by the target model and provides them to a lightweight drafter. Additionally, the authors introduce prediction-shift supervision, which trains the drafter to capture how target predictions change when its own proposed tokens are substituted into the context.

The authors evaluate HAWK primarily on SmolVLM-256M, comparing it against EAGLE-3, ViSpec, and HiViS, and extend the evaluation to LLaVA-1.6-7B and Qwen2.5-VL-7B. Improvements in average acceptance length and inference speedup are reported.

Critical strengths
  1. Practical improvements in multimodal speculative decoding.

HAWK improves the target representations and training supervision used by lightweight drafters while retaining the speculative decoding framework. On SmolVLM-256M, it achieves higher acceptance lengths and speedups than EAGLE-3, ViSpec, and HiViS under both greedy decoding and sampling. These results demonstrate the potential practical value of the proposed approach.

  1. An interesting approach to prediction-shift supervision.

Training the drafter to capture how target predictions change in response to its own token proposals is a potentially useful complement to conventional distribution matching. The observation that acceptance improvements become more pronounced at deeper drafting steps provides some support for this motivation.

Critical weaknesses
  1. Insufficient experimental evaluation, particularly on larger vision-language models.

Although the paper evaluates three target models, the main experiments and ablation studies are primarily conducted on SmolVLM-256M, a relatively small model. The evaluation on the two 7B-scale models is limited to comparisons with EAGLE-3, without including other relevant multimodal speculative decoding baselines such as ViSpec and HiViS.

Furthermore, each benchmark is evaluated on only 80 prompts, making the overall evaluation rather limited despite the use of ten benchmarks. The performance improvements on the 7B-scale models are also considerably smaller than those observed on SmolVLM-256M.

Given that the paper aims to improve speculative decoding for large vision-language models, I find the experimental evidence insufficient to convincingly demonstrate the effectiveness and generalizability of the proposed method at a meaningful scale.

  1. The independent contributions of the proposed components are not sufficiently established.

HAWK combines CKA-based layer selection, learnable feature fusion, visual-state pooling, and prediction-shift supervision. Although the paper provides ablation studies, the independent necessity and contribution of some components remain unclear.

In particular, HAWK uses more target layers than EAGLE-3. While Table 4 compares uniform and CKA-based layer selection, it remains unclear how much of the improvement comes from representation-aware selection rather than simply accessing more target layers. Moreover, the main HAWK configuration and the CKA ablation use different sets of selected layers, and the reason for this difference is not sufficiently explained.

Prediction-shift supervision also applies multiple token substitutions simultaneously to compute changes in target predictions. As a result, the measured prediction shifts may reflect the combined effects of multiple substitutions rather than the effect of an individual token replacement. The relationship between this procedure and the claimed token-level prediction-shift supervision requires further justification.

The contribution of visual-state pooling is also relatively modest in terms of acceptance length and speedup, and its advantages over other straightforward visual compression strategies are not convincingly established.

Overall, while the complete framework improves speculative decoding performance, the current experiments do not sufficiently establish the necessity or individual contributions of its components.

Questions for authors

1.Why are the main ablation studies and comparisons with ViSpec and HiViS restricted to SmolVLM-256M? Can the authors provide stronger baseline comparisons and component-level analyses on the two 7B-scale models?

2.Given that only 80 prompts are evaluated per benchmark, how representative are the reported improvements? Can the authors provide evaluations on larger subsets of the benchmarks?

3.Why are the speedup improvements over EAGLE-3 considerably smaller on LLaVA-1.6-7B and Qwen2.5-VL-7B? What practical advantages justify the additional complexity of HAWK at larger scales?

4.How can the benefits of CKA-based layer selection be distinguished from those of simply using more target layers? Why do the main HAWK configuration and the CKA ablation use different selected layers?

5.In prediction-shift supervision, multiple token substitutions are applied simultaneously. How do the authors distinguish the effect of an individual substitution from the accumulated effects of other substitutions?

1 comment

W1 / Q1 / Q2 / Q3 — Evaluation scale

HAWK improves accepted length and speedup on both 7B targets at both temperatures under the same training budget as EAGLE-3 (30K examples, 10 epochs). At T=0, mean τ and speedup rise from 3.73 to 4.02 and 2.37× to 2.47× on LLaVA-1.6-7B, and from 3.19 to 3.40 and 2.17× to 2.25× on Qwen2.5-VL-7B (Table 5).

On SmolVLM-256M, we expanded the evaluation to 160 prompts on each of 13 benchmarks, including MMStar, COCO-Caption and OmniDocBench. With all prompts included, HAWK improves τ on every benchmark at both temperatures.

TemperatureMean τ (EAGLE-3 → HAWK)Mean speedup (EAGLE-3 → HAWK)
T=03.51 → 4.112.28× → 2.56×
T=12.75 → 3.201.77× → 1.99×

We will add ViSpec and HiViS at the same 7B training budget and run component ablations on a 7B target.

W2 / Q4 / Q5 — Component ablations

(i) Layer selection and mixing. The nine-layer uniform grid with learned mixing gives τ=3.17, and the ten-layer CKA set with uniform mixing gives τ=3.21. This comparison changes both the layer count and the mixing rule. With the ten CKA layers held fixed, learning the weights raises τ from 3.21 to 3.24. We will add CKA, uniform and random selections with the same number of layers per region and the same mixing rule.

(ii) Layer indices. The SmolVLM-256M HAWK row for Table 1 in Table 6 contains a typo. We trained the main HAWK model and the CKA variants in Table 4 using the same ten CKA-selected blocks listed in Table 7. The early, middle and late sets are [1,2,5,9], [14,18,21] and [23,25,27], respectively, using zero-based decoder block indices. We will correct Table 6 and update Figure 4 and Appendix E with the weights from the trained checkpoint.

(iii) Prediction-shift supervision. We substitute the selected draft tokens in one sequence and score it in one extra target pass during training. We apply the loss at each substituted position. A later position can see earlier substitutions, so the target shift at that position can include their effects. We will add a control that scores each substitution in isolation.

(iv) Pooling. At four epochs, 4× pooling reduces the drafter’s visual rows from 832 to 208 for a 13-tile input, with τ rising from 3.05 to 3.10 and SR from 2.07× to 2.10×. At 10 epochs, the full model reaches τ=3.82 and SR=2.45×, compared with τ=3.40 and SR=2.19× without pooling. We will add strided and random row selection at the same visual-token budget.

Comments on the paper

1 comment

W1 / Q2 — Ablation stability

Tables 2–4 use one training seed. This follows the reporting practice in EAGLE-3, ViSpec and HiViS, whose main tables give point estimates for acceptance length and speedup. We will add means and standard deviations across three seeds for Tables 2–4.

(i) More prompts. We doubled the prompts per benchmark from 80 to 160 and added MMStar, COCO-Caption and OmniDocBench. On SmolVLM-256M, HAWK improves τ on each of the 13 benchmarks at both temperatures with every prompt included.

TemperatureMean τ (EAGLE-3 → HAWK)Mean speedup (EAGLE-3 → HAWK)
T=03.51 → 4.112.28× → 2.56×
T=12.75 → 3.201.77× → 1.99×

(ii) Longer training. After 10 epochs, the full model reaches τ=3.82 and SR=2.45×. Dropping pooling from that configuration gives τ=3.40 and SR=2.19×.

W2 / Q1 / Q3 — 7B evaluation and exact-output filtering

At 7B, HAWK and EAGLE-3 train on the same 30K examples for 10 epochs. At T=0, HAWK raises τ from 3.73 to 4.02 and speedup from 2.37× to 2.47× on LLaVA-1.6-7B. On Qwen2.5-VL-7B, τ rises from 3.19 to 3.40 and speedup from 2.17× to 2.25×. At T=1, mean τ rises from 2.98 to 3.18 and from 2.87 to 3.02, respectively. We will add ViSpec and HiViS at the same 7B training budget.

Exact-output filtering. In a separate greedy SmolVLM-256M run with 80 prompts per benchmark, both EAGLE-3 and HAWK matched the autoregressive output on 682 of 800 prompts. The counts out of 80 were 48 (MMMU), 70 (SEED-Bench), 70 (MME), 74 (VQAv2), 65 (MM-Vet), 67 (MathVista), 76 (ScienceQA), 75 (GQA), 76 (TextVQA) and 61 (DocVQA). Across all 800 prompts, HAWK raises τ from 3.37 to 4.07 and speedup from 2.22× to 2.55×.

For comparison, Table 1 reports τ from 3.32 to 4.08 and speedup from 2.19× to 2.60× on the greedy subset where all four methods matched the autoregressive output. The expanded 13-benchmark results use all 160 prompts per benchmark. We will report exact-match counts separately for Table 1 and the expanded evaluation.

Q4 — CKA sensitivity

With the paper’s 32-dimension setting, the three SmolVLM regions are 1–10, 11–22 and 23–28. The selected layers are [1,2,5,9], [14,18,21] and [23,25,27], respectively (Table 7). With all dimensions included, one high-variance dimension drives adjacent-layer CKA to 1.000 across blocks 11–28.

The 16/32/64-dimension sweep used a different calibration set, which shifted the boundary at 32 dimensions. We will add a sweep on the paper’s 48 conversations and report how the boundaries and selected layers change as we vary the number of dimensions removed.

Q5 — Learned fusion

The pairs 2.33/2.07 and 2.38/2.05 are the speedups on SEED-Bench and MM-Vet. Holding the CKA layers fixed in Table 4, learned mixing raises mean τ from 3.21 to 3.24 and mean speedup from 2.09× to 2.11×.

We will update Figure 4 and Appendix E with the learned weights exported from the checkpoint for the ten CKA layers used in training. Uniform mixing assigns 1/4 to each early layer and 1/3 to each middle or late layer.

Q6 — Timing and drafter size

Yes. We measure generation tokens per second against autoregressive decoding. The timed generation includes drafting and target verification, including the feature fusion performed during generation (Appendix B).

HAWK uses the same single-layer drafter core as EAGLE-3. It learns scalar weights to mix the selected states into the same three input streams. We will add parameter counts for both drafters and break down the time spent on feature fusion, drafting and verification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.