acceptodds
Under review as a conference paper at ICLR 2027

Trajectory-Retrieval Speculative Decoding: When Does a Model's Own History Help?

Abstract

Long chain-of-thought reasoning increases sequential decoding cost while creating a growing history of potentially reusable continuations. We investigate when this history supplies useful drafts and complements an existing drafter. Controlled source comparisons reveal trajectory-specific reuse, motivating our method Trajectory-Local Adaptive Retrieval (TLAR). TLAR retrieves approximately matched continuations from the current trajectory and uses recent verification outcomes to adapt retrieval activation and candidate width. TLAR combines retrieved continuations with model-generated drafts in a shared candidate tree, preserving the target model's output distribution through exact verification. Across code debugging, mathematics, and open-ended writing, our evaluation connects source reuse, incremental acceptance, and execution cost. Combining TLAR with strong retrieval baselines improves token acceptance under matched verification budgets and increases end-to-end throughput over the draft-model baseline. These findings support generated trajectories as runtime memory for adaptive inference. See implementation details at: https://anonymous.4open.science/r/TLR-DE77.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, Chain-of-Thought Reasoning, Trajectory Retrieval, Retrieval-Based Drafting, Large Language Models, Inference Acceleration, Test-Time Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WKtsOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Trajectory-Local Adaptive Retrieval, a speculative decoding method that augment existing stand-alone drafters with fuzzy-matched patterns retrieved from the ongoing generation trajectory. The paper provides comprehensive analysis of the approach, including it's incremental improvements on top of existing baselines and different trends in diverse domains. Experiments also showcase improvements over the baselines.

Critical strengths

The paper is easy to read and the motivation is clear. It introduces multiple components to maximally utilize the within-trajectory information for improved speculative decoding, and is composible with existing methods (which is useful for easy adoption).

Critical weaknesses

The novelty of the paper is somewhat limited, as previous work (including STAND) already showcased that drafting with self-trajectory retrieval can improve the speculative decoding performance, which seems to be the core idea of this paper.

The empirical usefulness is not fully addressed. While the paper reports performance by combining smalldraft (stand-alone drafters) with retrieval-based drafts, there are much faster and performant drafting methods available nowadays, including Eagle, DFlash, and DSpark. To show that the provided approach has practical value, the paper should demonstrate that the given approach can improve the performance even when combined with strong drafters, rather than relatively weak, standalone drafters.

For acceptance, I would expect the paper to either be conceptually novel or show strong practicality. The current version of the paper does not satisfy this bad.

Questions for authors

This is a suggestion for improving the paper in the future, rather than a question. So addressing this question is not crucial for my assessment of the paper in the current round.

Having that said, I think the paper could be substantially stronger if it is positioned for long-running agentic systems. As suggested in Table 2 (and as expected), the gain of this work gets stronger with longer context. The gain may seem rather marginal for the context lengths tested, which is up to 4k. However for long-running agent systems, context easily accumulates to multiple millions of tokens, providing rich information for trajectory-based speculators to use. If the authors explicitly suggest this as the research motivation and build the evaluations around this setup, the paper could become much more timely, and empirically powerful.

Reviewer_b6jHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the limitation of current speculative decoding methods, which generally rely on trained draft components, by investigating when a model's own generated trajectory can serve as a reusable draft source during long chain-of-thought reasoning. To answer this question, the authors first conduct controlled source comparisons that reveal trajectory-specific reuse, then propose Trajectory-Local Adaptive Retrieval (TLAR). Specifically, TLAR retrieves approximately matched continuations from the current trajectory, adapts retrieval activation and candidate width based on recent verification outcomes through a feedback controller, and combines retrieved continuations with small-draft-model candidates in a shared prefix-sharing tree, preserving the target model's output distribution through exact verification. Extensive experiments across code debugging, mathematics, and open-ended writing demonstrate that TLAR adds accepted tokens beyond the draft model, complements strong retrieval baselines under matched verification budgets, and improves end-to-end throughput, with gains scaling as output length increases.

Critical strengths
  1. The paper provides a rigorous source-level analysis demonstrating that a model's own generated trajectory serves as a distinct, reusable draft source for speculative decoding, and reveals that such trajectory-specific reuse is contingent on both the model and the task.

  2. The proposed TLAR method is training-free and lossless: it adaptively allocates retrieval and verification effort based on recent verification feedback, consistently increases token acceptance beyond a draft model, complements strong retrieval baselines under matched verification budgets, and improves end-to-end throughput in long-output reasoning.

Critical weaknesses
  1. The paper does not directly compare TLAR with trained drafters such as EAGLE-3. The archived reference (Appendix G.5) uses a mismatched configuration (256 output tokens vs. 1,024 to 4,096 in the main evaluation) and shows EAGLE-3 reaching +56.5% over vanilla decoding on Qwen, far above any TLAR gain (8.3% to 12.4%), with an unexplained −14.0% on Llama. The concurrent AgentSpec [1], which also uses the model's own history as a lossless draft source, is neither cited nor compared. Without these comparisons, the relative value of the method cannot be assessed, which undermines the central claim.

  2. The efficiency claim is not validated under standard serving conditions. Experiments run on a custom, undisclosed pipeline on a single 2. NVIDIA B200 rather than vLLM or SGLang, and only batch sizes 1 and 32 are tested, with gains already shrinking from 8.3% to 7.6% for Qwen. No experiment covers larger batches (64 to 256), where retrieval-based methods typically degrade, although the authors themselves note that evaluation should cross batch size. Since throughput is the primary claimed benefit, this gap is consequential.

[1] AgentSpec: Speculative Decoding for Batch Inference of LLM Agents. EMNLP, 2026.

Questions for authors
  1. In the archived EAGLE-3 reference (Appendix G.5), throughput on Llama decreases from 19,436.63 to 16,721.52 tokens/s. Is this due to a checkpoint mismatch or configuration issue, and does it limit the interpretability of the archive?

  2. What is the serving engine used in the end-to-end experiments (Appendix G)? Was there a specific reason not to integrate the method into vLLM or SGLang, and how would the reported throughput transfer to those systems?

  3. The half-life and probe-interval sweeps (Table 22) show that the defaults H=32 and P=32 are not optimal for Llama, where H=8 to 16 and P=8 yield higher gains. How were the defaults selected, and is there a principled way to set them per model?

  4. Since the reported gains decline between batch 1 and 32 (8.3% to 7.6% for Qwen), has the method been tested at larger batch sizes (64 to 256), where draft-based methods typically degrade?

Reviewer_Gk9bOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Trajectory-Local Adaptive Retrieval (TLAR), a retrieval-based speculative decoding method designed to accelerate long chain-of-thought reasoning in large language models. TLAR retrieves candidate continuations from previously generated tokens using approximate context matching and combines them with predictions from a small draft model for target-model verification. An adaptive controller further adjusts retrieval activation and candidate width based on historical verification outcomes to balance token acceptance and computational cost. Experiments across code debugging, mathematical reasoning, and open-ended writing demonstrate improved token acceptance and end-to-end decoding throughput.

Critical strengths
  • The paper addresses a practically important problem of improving inference efficiency for long chain-of-thought reasoning without requiring additional model training.

  • The proposed method is simple and easy to integrate with existing speculative decoding frameworks. Its adaptive retrieval mechanism provides a practical way to balance additional token acceptance against verification overhead.

Critical weaknesses
  • TLAR relies on local token-level matching rather than semantic similarity or reasoning-state consistency. Similar local contexts may correspond to different reasoning states, particularly in mathematical reasoning and code debugging. The authors should provide stronger evidence that the retrieved continuations capture meaningful reasoning patterns rather than merely recurring tokens or formatting patterns.

  • TLAR ranks retrieved candidates solely by recency, assuming that more recent historical matches provide more useful continuations. However, recency does not necessarily indicate prediction quality, especially when multiple approximate matches exist. The authors should justify this design choice and compare recency-based ranking with alternatives such as context similarity and historical acceptance-based ranking.

  • The method fixes the retrieved continuation length to four tokens without sufficient justification. While the authors analyze the effects of context length and candidate width, the impact of continuation length on token acceptance and verification cost remains unexplored. An ablation study with different continuation lengths would help justify this design choice and clarify its efficiency trade-offs.

  • The methodological novelty appears somewhat limited, as several key components, including history-based token retrieval, continuation reuse, and fusion with model-generated drafts, have already been explored in prior works such as STAND, SuffixDecoding, and RASD. The main distinction seems to lie in the adaptive retrieval controller and trajectory-specific analysis. The authors should more clearly establish the novelty of these contributions and demonstrate their advantages over existing retrieval-based and adaptive speculative decoding methods.

Questions for authors

Refer to Critical Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.