acceptodds
Under review as a conference paper at ICLR 2027

HARP: Horizon-Aware Representation Preference for Parallel Speculative Decoding

Abstract

Speculative decoding (SD) accelerates large language model (LLM) inference by letting a lightweight drafter propose tokens that the target model verifies in parallel. Parallel drafters push this paradigm further by predicting an entire block of future tokens from a single target computation at the verified prefix, yet they treat all positions in the block uniformly, fusing target hidden states into one shared representation and ranking candidates by token confidence. We identify prediction horizon, the distance of a position from the verified prefix, as a key factor in both drafting and verification. On the drafting side, the target layer with the strongest predictive signal shifts systematically from upper layers toward intermediate layers as the horizon grows, so different future positions favor different target representations. On the verification side, we prove that for repair trees preserving the main draft chain, the additional accepted length decomposes exactly at the first mismatch into prefix survival, alternative-token probability, and recoverable continuation, showing that the value of a repair branch depends on more than per-token confidence. Based on these results, we propose HARP (Horizon-Aware Representation Preference), a training-free inference framework that assigns each horizon its preferred combination of already-computed target layers and allocates repair branches by this reachable value. On Qwen3-4B with a frozen DFlash drafter, HARP improves end-to-end throughput by 19.7% over chain verification, compared with 9.7% for confidence-based allocation, and achieves higher throughput than recent draft-tree methods while verifying fewer nodes. A factorial study shows that the two components provide complementary gains, and HARP improves throughput by 10.8–14.3% across six target–drafter pairs. Prediction horizon thus provides a principled axis for exploiting target representations and verification structure in training-free SD acceleration.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Speculative Decoding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xb9ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HARP combines horizon-dependent target-layer fusion with repair-branch allocation based on prefix survival, alternative-token probability, and recoverable continuation. It keeps model weights frozen and reports a 19.7% throughput improvement over chain verification on Qwen3-4B/DFlash.

Critical strengths

1.Interesting motivation. Horizon-dependent representation preferences provide a useful perspective on parallel drafting, supported by probes and source-layer interventions. 2. Clear repair-value analysis. The first-mismatch decomposition explains why alternative-token confidence alone is insufficient for allocating repair branches. 3. Relevant empirical evaluation. The paper includes end-to-end timing, a representation-by-allocation factorial study, and comparisons with recent tree methods.

Critical weaknesses
  1. The central representation intervention is underspecified. DFlash conditions on target features through cached K/V representations. Equation (4) does not explain how different draft horizons access differently fused context features within one frozen drafter pass. The attention routing, cache organization, actual fusion coefficients, and associated overhead are needed to reproduce this contribution.

  2. The representation-regret argument is not valid as stated. Equation (1) compares a multi-layer composition against the best individual layer, so the difference need not be nonnegative. Different single-layer preferences also do not imply that shared fusion necessarily discards useful information. These claims require correction, although the empirical observations may remain valid.

  3. Experimental scope and reporting need clarification. The full 19.7% gain is demonstrated on 100 prompts for one target–drafter pair. Cross-system results use shared representations and therefore primarily validate branch allocation. Additionally, Table 3 reports different draft accuracies under “identical proposals”; this requires clarification regarding whether statistics use fixed prefixes or method-specific online trajectories.

  4. Numerical divergence is insufficiently controlled. Only 42% of 1024-token outputs match sequential decoding exactly in BF16. This does not itself establish quality degradation, but observing divergence across all methods does not demonstrate that comparisons are unbiased. Per-method output agreement, task quality, and output-length statistics would help establish the reliability of the speed comparisons.

Questions for authors
  1. How is horizon-specific conditioning implemented through DFlash’s attention and KV cache without additional drafter passes?
  2. Which reported statistics use identical fixed prefixes, and which use independently visited online states?
  3. Do the throughput gains persist under stronger numerical-equivalence controls, and are downstream task scores comparable?
Reviewer_VskYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that parallel (block) speculative drafters such as DFlash should treat block positions differently according to their prediction horizon. This paper proposes HARP (Horizon-Aware Representation Preference), a training-free inference framework that assigns each horizon its preferred combination of already-computed target layers and allocates repair branches by this reachable value.

Critical strengths
  1. It provides a useful empirical observation. Figures 1 and 6 show a shift in layer preference with horizon. Mapping it per horizon on a parallel drafter's input layers is useful for designing feature-conditioned drafters.
  2. This method is training-free and cheap to achieve.
Critical weaknesses
  1. There are some internal numerical inconsistencies.

(a) Table 1 gives overall Top-2@K = 0.314 (shared), while Section 5.2 and Table 5 give 0.78 at near horizons and 0.31 at far horizons. An overall value of 0.314 would require that fewer than 1% of first mismatches fall at k ≤ 3. Yet the chain's P(A ≥ 1) = 0.82 already implies P(K = 1) ≈ 0.18, so the overall value should be at least about 0.40.

(b) The overall repair hit of HARP (0.341 → 0.412) is higher than both group-level hits in Table 5 and Fig. 4c (near 0.31 → 0.37, far 0.18 → 0.25), which is impossible for a weighted average unless the two tables use different allocations or states that are not identified.

(c) In Table 3, "Draft Acc." (macro top-1 before verification) changes from 0.58 to 0.61 across tree policies that consume "identical DFlash proposals", and Table 4 reports 0.617 for the same configuration.

  1. The proposed findings overlap with existing work that is not cited. Future Lens[1] already showed that middle layers predict future tokens better than the final layer. Similar findings have been provided by [2].

  2. Unfair comparison. Appendix F says the shared control is the unmodified fusion (α\alpha = 1), which contradicts Section 4.4's claim that "the shared control receives the same selection budget". Without a control that tunes a single global set of layer scales with the same search budget, the gain cannot be attributed to horizon specificity rather than to "any re-weighting of DFlash's input layers". The chosen grouping, the α\alpha values, their bounds, and the size of the search space are also never reported.

  3. The best hyperparameters differ among different models. This requires a validation set and selection for each model, limiting the generalizability of the method.

[1] Future Lens: Anticipating Subsequent Tokens from a Single Hidden State. CONLL 2023.

[2] Do language models plan ahead for future tokens? COLM 2024.

Questions for authors
  1. Can the hyperparameters of one model generalize in different domains? For example, if you select a set of hyperparameters in the math domain, does it work well on code or other domains?
Reviewer_CAGbOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes HARP, a training-free framework for parallel speculative decoding. It combines horizon-aware hidden-state scaling with reachability-aware tree allocation to improve draft quality and verification efficiency. Experiments show up to 19.7% higher throughput than chain verification.

Critical strengths
  1. The paper identifies horizon-dependent preferences for target-model hidden layers and proposes a training-free scaling strategy to improve draft quality.

  2. The paper derives a first-mismatch decomposition and proposes an efficient repair-branch allocation strategy to improve verification efficiency.

Critical weaknesses

Unfair baseline comparisons. DDTree, CaDDTree, and BASTION are evaluated at approximately 40 verification nodes, substantially below their reported optimal operating budgets (often 128–512 nodes). In particular, fixing the budget disables the adaptive budget selection mechanisms of CaDDTree and BASTION. Without budget sweeps or comparisons against properly tuned baselines, the claim that HARP outperforms existing tree-based methods is not sufficiently supported.

Questions for authors

The horizon grouping and scales α are tuned on a development split drawn from the same five tasks as the test set. How do the authors rule out that the gain comes from this tuning rather than from a transferable effect? Specifically: (1) what search space and selection criterion were used; (2) how sensitive is throughput to α; and (3) does an α tuned on some tasks still help on held-out tasks without re-tuning (e.g., leave-one-task-out)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.