- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies two limitations of speculative decoding for vision-language models: how to provide informative multimodal representations, and how to reduce the mismatch between the drafter’s training trajectory and its own multi-step predictions at inference time. To address the first issue, the authors propose representation-based target-layer selection and learned layer fusion, with pooling of visual hidden states. To address the second issue, they introduce prediction-shift supervision, which trains the drafter to match how the target model’s logits change when the original token is replaced by the drafter’s own proposal. The proposed method improves average acceptance length and inference speedup over EAGLE-3 and several multimodal speculative decoding baselines on multiple benchmarks.
Critical strengths
1.The information-injection design is well motivated.
The paper observes that fixed-depth layer selection may not align with where representations actually change across different VLMs, and that a shallow drafter may have difficulty extracting useful information directly from a long sequence of visual tokens. Using representation similarity to select target layers and reusing target-processed visual hidden states therefore provides a reasonable way to expose the drafter to richer and more compact multimodal information. Plus, using pooling to reduce visual token is also a common but efficient method.
2. Prediction-shift supervision is an interesting and relatively novel training objective.
This method explicitly models how the target prediction changes when the drafter’s own token alters the context. This is a more structured form of supervision than simply adding off-trajectory examples, and the ablation against direct distillation suggests that the shift-based objective provides additional benefit beyond using substituted contexts alone.
Critical weaknesses
1.The novelty and attribution of the layer-fusion component are somewhat limited.
CKA-based layer selection and learned feature fusion are straightforward extensions of existing representation-selection techniques. More importantly, HAWK reads substantially more target layers than EAGLE-3 and then fuses them into three streams. Therefore, part of the gain may come from exposing the drafter to more target features rather than from the specific representation-based selection strategy itself. A stricter comparison using the same number of target layers would make the contribution easier to isolate.
2. The motivation of prediction-shift supervision under greedy decoding is not yet fully convincing.
The substituted token is explicitly chosen to differ from the target model’s top-1 prediction. Under greedy speculative decoding, such a token would be rejected at that position, and predictions conditioned on the rejected token would not directly contribute to the accepted prefix. It is therefore unclear why accurately modeling the target’s behavior after such a substitution should improve deeper greedy acceptance, beyond acting as a form of perturbation-based regularization or context augmentation. The empirical gains are clear, but the causal explanation requires further clarification. The fact that direct distillation on the same substituted contexts already improves acceptance also suggests that part of the gain may come simply from training on drafter-induced contexts.
3. The visual pooling strategy is heuristic and its behavior across different visual resolutions is unclear.
The method uses a fixed 4x pooling factor, while the amount and structure of visual information can vary substantially with image resolution and token count. Although the paper compares no pooling, 4x and 16x pooling, it remains unclear whether a fixed compression factor is appropriate across different resolutions or whether an adaptive strategy based on image/token complexity and task categories would be more effective.
4.The scalability of the method to stronger contemporary VLMs remains insufficiently demonstrated.
The larger-scale experiments are limited to LLaVA-1.6-7B and Qwen2.5-VL-7B, and even these models are trained with a reduced data budget. These targets are relatively small and no longer representative of the strongest modern VLMs. It is therefore unclear whether the reported gains persist at substantially larger scales.
Questions for authors
- Could the authors further clarify the mechanism behind prediction-shift supervision, especially under greedy decoding? Since the substituted token is deliberately different from the target’s top-1 token, that branch would be rejected immediately during greedy verification. Why should accurately matching the target’s prediction shift after such a rejected token improve acceptance along the actual greedy trajectory? Is the main benefit better understood as on-policy context augmentation, local regularization of the drafter, or something else? Relatedly, the results appear sensitive to the weight of the shift loss: performs better than a larger weight of 0.3. Could the authors explain why over-weighting the shift objective hurts performance, and how this hyperparameter interacts with the standard distillation objective?
- How should visual-state pooling behave across different image resolutions and visual-token lengths? The current method uses a fixed 4x pooling factor. Is the same compression ratio optimal for low-resolution and high-resolution inputs? Have the authors considered an adaptive pooling strategy based on resolution, token count, attention statistics, or visual complexity?
W1 — Layer selection and fusion
HAWK uses CKA to choose which target states to pass to the drafter and learns how to combine them into three streams. Holding the ten CKA layers fixed in Table 4, learning the weights raises τ from 3.21 to 3.24 and speedup from 2.09× to 2.11×. The uniform grid uses nine layers and learned mixing, so comparing its τ of 3.17 with 3.21 changes both the layer count and the mixing rule.
The target computes these states in its usual forward pass. The measured speedup includes the cost of reading and mixing them during generation. We will add a layer-count sweep and compare CKA, uniform and random selections at the same number of layers per region with the same mixing rule.
W2 / Q1 — Prediction-shift supervision
Under greedy decoding, verification discards the substituted token and its descendants. We use these substitutions during training to teach the drafter how the target’s next-token predictions change after a likely drafting error. The reviewer’s direct-distillation comparison tests whether seeing the same altered contexts is enough.
In Table 3, direct distillation on those contexts raises τ from 3.05 to 3.10. With the same substitutions, prediction-shift supervision reaches 3.17 at weight 0.1 and 3.11 at weight 0.3. The two objectives see the same contexts, so the difference measures the effect of changing the training objective in this ablation.
The gap is larger deeper in the draft. Relative to standard distillation alone, the shift objective raises the probability of accepting the first k draft tokens by 3% at k=1 and 23% at k=5. Direct distillation improves it by at most 10% (Figure 3b). We will add runs across seeds and compare drafter proposals with other token substitutions.
W3 / Q2 — Pooling across resolutions
Pooling acts within each 64-row image tile. At 4×, the drafter keeps 16 rows per tile, so SmolVLM’s 9-, 13- and 17-tile inputs use 144, 208 and 272 rows, respectively. The target receives the full visual sequence for verification.
Across the six benchmarks in Table 2, 4× gives τ=3.10 and SR=2.10×, compared with τ=3.05 and SR=2.07× without pooling. At 16×, τ falls to 3.04. We will report τ and speedup by tile count and test an adaptive pooling factor.
W4 — Evaluation on larger targets
At the same 30K-example, 10-epoch budget, HAWK improves accepted length and speedup over EAGLE-3 on LLaVA-1.6-7B and Qwen2.5-VL-7B at both temperatures. Under greedy decoding, τ rises from 3.73 to 4.02 and from 3.19 to 3.40, respectively.
We will add a comparison on a newer target with HAWK and EAGLE-3 trained at the same budget.