acceptodds
Under review as a conference paper at ICLR 2027

Speculative Multi-Trajectory Decoding for Efficient Multimodal LLM Planning

Abstract

Multimodal large language model (MLLM) planners typically serialize trajectories into token sequences and decode them autoregressively, making sequential generation a major inference bottleneck that becomes more severe when multiple candidates are required. To address this issue, we propose Speculative Multi-Trajectory Decoding (SMTD), a target-verified framework that accelerates trajectory generation through a unified draft–verify–select process while keeping the pretrained OneVL target model frozen. Lightweight Medusa and anchored-GRU drafters first propose blocks of future trajectory tokens, which are verified in parallel by the target model to reduce sequential evaluations while preserving target-consistent generation. Building on this mechanism, SMTD extends speculative decoding to multi-trajectory planning via speculative multi-sampling, efficiently producing multiple verified candidates. A Unified Online Candidate Selector (UOCS) then scores these candidates and selects the final trajectory at inference time. Experiments on NAVSIM, ROADWork, and Impromptu demonstrate consistent latency reductions with competitive planning quality. On NAVSIM, SMTD reduces mean decoding latency from 5.221 s to 1.349 s. The remaining gap between selected and oracle candidates further identifies online selection as a key bottleneck after generation is accelerated.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Trajectory Planning, Speculative Decoding, Multi-Trajectory Generation, Multi-Candidate Planning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_KGETOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SMTD, a speculative decoding framework for efficient multi-trajectory generation in VLA-based driving. Experiments demonstrate substantial VLA inference speedups with considerable planning performance.

However, I have the following concerns:

C1. Insufficient ablation studies. The contributions of individual components to the overall speedup are not clearly isolated, making it difficult to assess the necessity of each design choice.

C2. Unclear source of PDMS improvements. Although speculative sampling preserves the target distribution, SMTD achieves higher Oracle PDMS than standard autoregressive sampling. Further analysis is needed to explain these gains and assess their statistical reliability.

C3. Incomplete end-to-end efficiency evaluation. The reported latency excludes online candidate selection. Measuring the full generation-and-selection pipeline would provide stronger evidence of the method's practical efficiency.

Critical strengths

see above

Critical weaknesses

see above

Questions for authors

see above

1 comment
AuthorsOpenReview ↗

Response to Reviewer KGET: Component Effectiveness, PDMS Improvements, and End-to-End Latency

Thank you for highlighting the importance of component ablations, the source of PDMS improvements, and complete pipeline timing. Our additional experiments demonstrate three complementary roles: speculative decoding improves generation efficiency, multiple candidates increase access to high-quality trajectories, and UOCS improves the final choice. All experiments below evaluate the complete NAVSIM test split, with three repetitions and mean results reported.

C1. Effectiveness of the three components

We evaluated all eight combinations of speculative decoding (SD), multiple candidates (MC), and UOCS. Configurations share the target model and sampling settings, using the same frozen UOCS wherever enabled. With SD off, AR shares prefill and batches branches. MC off permits one sampled trajectory; MC on permits four sampled trajectories plus the reference. Without UOCS, selection uses a fixed independent uniform draw. With one eligible trajectory, the selector returns that trajectory. Every configuration includes fresh reference generation.

SDMCUOCSE2E (s)Selected PDMSOracle PDMS
OffOffOff9.10187.6987.69
OffOffOn9.10587.6987.69
OffOnOff9.31491.0795.17
OffOnOn9.34291.8995.17
OnOffOff5.83786.8286.82
OnOffOn5.86286.8286.82
OnOnOff8.75890.3694.92
OnOnOn8.78492.1794.92

SD improves efficiency. With MC and UOCS enabled, SD reduces end-to-end latency from 9.342 to 8.784 s, a 6.0% reduction. The fully enabled configuration takes 8.784 s in the ablation and 8.895 s in the separate pipeline-timing experiment in C3.

MC improves candidate quality. With SD enabled, expanding from one sampled trajectory to four candidates plus the reference raises Oracle PDMS from 86.82 to 94.92. Excluding the reference still yields 94.91, demonstrating the contribution of generated candidates.

UOCS improves the final choice. On the same N=4 SMTD pool, UOCS raises Selected PDMS from 90.36 to 92.17 while Oracle remains 94.92. It also exceeds the expected score of uniform selection, 90.67. Feature extraction and selection together cost approximately 26 ms. These results show how the three components combine efficient generation, stronger candidate coverage, and quality-aware selection.

C2. Why multiple candidates improve PDMS while preserving the target distribution

The key is repeated sampling: more draws increase the chance that the candidate pool contains a high-quality trajectory. Let q be the probability that one trajectory exceeds a quality threshold for a given scene. For N independent draws,

$ P(\text{at least one high-quality trajectory})=1-(1-q)^N. $

This probability increases with N without changing any individual trajectory's distribution. Multi-candidate planning exploits this opportunity; speculative decoding improves generation efficiency, and UOCS selects the final output.

The results support this mechanism. As N increases from 2 to 4 to 8, SMTD's Oracle PDMS rises from 93.14 to 94.92 to 95.64, while Selected PDMS increases from 89.66 to 92.17 to 92.96. The ablation further shows that expanding from one to four generated candidates, excluding the reference, raises Oracle PDMS by 8.09 points.

At the same N, AR and SMTD can produce different realized candidate pools, so their measured Oracle scores need not coincide. Increasing N explains the benefit over single-trajectory planning; differences between decoders at fixed N can reflect finite sampling variation. SMTD's effectiveness comes from efficiently generating useful alternatives from the target distribution and selecting among them, without requiring a change to that distribution.

C3. Complete end-to-end latency

The original table measures candidate-generation latency. We additionally measured the complete planning pipeline, including greedy reference generation, candidate generation, and online selection:

$ T_{\mathrm{E2E}}=T_{\mathrm{pre/post}}+T_{\mathrm{reference}}+T_{\mathrm{candidates}}+T_{\mathrm{UOCS}}. $

The reference is freshly generated by ordinary autoregressive greedy decoding of the frozen OneVL target, using token-wise argmax. UOCS time includes feature extraction, prediction, and final selection.

NReference (s)Candidates (s)UOCS (s)Pre/post (s)E2E (s)
24.5532.3010.0240.0626.940
44.5514.2560.0260.0638.895
84.5467.6340.0290.06212.272

Online selection adds approximately 24-29 ms, while reference generation contributes approximately 4.55 s.

Reviewer_xwS8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Speculative Multi-Trajectory Decoding (SMTD) to accelerate generating multiple trajectory candidates for multimodal driving. It uses shared scene prefill, batched target verification, and speculative decoding to generate multiple candidate sequences, followed by a learned UOCS module to select the final sequence.

Critical strengths
  1. The framework is simple and practical, and can generate multiple candidates more efficiently than sequential autoregressive sampling.
  2. The UOCS selector helps reduce the gap between the available candidate quality and the final selected trajectory.
Critical weaknesses
  1. Limited novelty in speculative decoding. The SD component mainly applies existing Medusa/Hydra-style methods to multi-trajectory generation, without a clear new SD mechanism.
  2. Weak comparison with recent SD methods. The paper does not compare against recent methods such as EAGLE-3, making it unclear whether the approach is competitive with current SOTA SD methods.
  3. Limited novelty in token budget selection. The cost–acceptance based draft-length selection is closely related to existing adaptive speculation methods such as AdaEDL, DISCO.
  4. Limited novelty of UOCS. The candidate selector mainly uses handcrafted trajectory/scene features with an ExtraTrees model, and its algorithmic novelty is unclear.
Questions for authors
  1. What is the key technical novelty of SMTD beyond applying existing speculative decoding methods to multiple trajectory branches?
  2. How does the proposed draft-length/token-budget selection differ from existing adaptive speculation methods such as DISCO and AdaEDL? What is the advantage in this setting?
  3. How does SMTD's single trajectory performance compare with recent SOTA speculative decoding methods such as EAGLE-3, DFlash?
2 comments
AuthorsOpenReview ↗

Response to Reviewer xwS8: Technical Contribution, Draft-Length Selection, and Recent Baselines

Thank you for recognizing SMTD's practical value. Below, we clarify its technical contribution, distinguish offline draft-length calibration from adaptive speculation, and present single-trajectory comparisons with EAGLE-3 and DFlash.

1. Technical contribution

SMTD coordinates multi-candidate generation and selection under a planning latency budget. Branches share scene prefill and batch verification across compatible states, while maintaining separate accepted prefixes and corrections. The anchored-GRU conditions each draft token on both the target hidden anchor and preceding draft tokens to capture within-trajectory dependencies.

UOCS combines trajectory/scene features with candidate-relative and set-level information, enabling one frozen selector to rank candidates across generators and candidate counts. Our contribution is this coordinated generation-and-selection design; ExtraTrees and the distribution-preserving speculative sampling rule build on established methods.

Our new 64-scene ablation separates speculative decoding, multiple candidates, and selection. At N=4, SMTD's pool of four sampled trajectories plus the reference achieves Oracle PDMS 94.92. UOCS achieves Selected PDMS 92.17, compared with 90.67 for expected uniform selection from the same pool. This illustrates the distinct roles of candidate generation and selection, while speculative decoding addresses computational cost.

2. Draft-length selection versus DISCO and AdaEDL

DISCO dynamically adjusts speculation lookahead; AdaEDL uses draft-distribution entropy for early stopping. We instead calibrate a fixed draft length offline for each drafter-dataset pair:

$ C(K)=1+\sum_{j=1}^{K}\prod_{i=1}^{j}\alpha_i,\qquad K^\star=\arg\max_{K\in\mathcal{K}}\frac{C(K)}{1+rK}. $

Here, αi\alpha_i is the conditional acceptance probability at position ii, and rr is the normalized marginal drafting/verification cost. Statistics and costs are estimated on the training partition, and KK is fixed before testing.

This applies established cost-acceptance analysis to our planning setting. It balances committed tokens against longer-block costs, keeps the planned draft length uniform across branches, and avoids per-round stopping decisions. The paper's KK sweep shows why more accepted tokens do not necessarily yield lower latency.

3. Single-trajectory comparison with EAGLE-3 and DFlash

We evaluated all methods on the complete evaluation splits of NAVSIM, ROADWork, and Impromptu, repeating evaluation three times and reporting mean latency and quality. Within each dataset, methods share the frozen OneVL target, inputs, greedy decoding setting, and timing boundary on an RTX 5090D. Latency includes preprocessing, prefill, decoding, and trajectory parsing, excluding model loading and file I/O. Single-trajectory inference uses neither UOCS nor an additional reference trajectory.

MethodNAVSIM (s)ROADWork (s)Impromptu (s)
AR4.6034.8294.345
SMTD / anchored-GRU1.2611.4061.856
EAGLE-33.2792.8921.911
DFlash4.6134.3593.217

Quality is measured by NAVSIM PDMS (higher is better), and ROADWork/Impromptu ADE/FDE in pixels/meters, respectively (lower is better).

MethodNAVSIM PDMSROADWork ADE/FDEImpromptu ADE/FDE
AR90.8930.73 / 65.901.711 / 5.023
SMTD / anchored-GRU90.8828.51 / 60.261.689 / 4.948
EAGLE-392.1328.90 / 61.471.693 / 4.989
DFlash90.9028.54 / 60.211.730 / 5.085

Anchored-GRU achieves the lowest latency on all three datasets, with 3.65x, 3.44x, and 2.34x speedups over AR. EAGLE-3 takes 2.60x and 2.06x as long on NAVSIM and ROADWork, but is close on Impromptu (1.911 versus 1.856 s). DFlash's speedups over AR are approximately 1.00x, 1.11x, and 1.35x, showing limited latency gains in these implementations.

The quality-latency trade-off is also informative: EAGLE-3 achieves higher NAVSIM PDMS (92.13 versus 90.88), but takes 3.279 s versus 1.261 s. Anchored-GRU achieves the lowest ADE/FDE on ROADWork and Impromptu alongside the lowest latency. These results support its favorable quality-latency balance, consistent with the motivation for a shared recurrent drafter that models token dependencies with low overhead.

Thank you for the response. The reason for improvement in single trajectory compared to Eagle3 and DFlash is not clear. Slight drop in quality does not sufficiently explain the 2x - 3x speedup over Eagle3 and DFlash. The accept length is not reported. DFlash has very limited speedup compared to AR, indicating unoptimized implementation, so the results are not convincing. I will keep the existing score.

Reviewer_ajXVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SMTD accelerates tokenized trajectory generation for a frozen OneVL planner using Medusa or anchored-GRU drafting and target verification. Its multi-candidate implementation combines anchored-GRU, shared scene prefill, batched verification, and UOCS selection. Experiments cover single-trajectory decoding on NAVSIM, ROADWork, and Impromptu, and multi-candidate planning on NAVSIM. At eight candidates, it reports 8.10 s generation latency and 90.26/94.11 Selected/Oracle PDMS.

Critical strengths
  1. Tables 3–5 evaluate the same frozen target across three complete test sets. NAVSIM latency decreases from 5.221 to 1.349 s, and ROADWork latency from 5.524 to 1.427 s, with quality broadly close to Greedy. These are useful practical gains. On Impromptu, the best proposed latency essentially ties Token Recycling.

  2. Tables 1–2 include several search and sampling baselines, including parallel Arithmetic Sampling, and use a shared frozen selector. Reporting Oracle and Selected PDMS separately makes the gap between candidate potential and realized quality explicit. The BF16 audit also improves transparency.

Critical weaknesses
  1. Spec-VLA already applies speculation to VLA action tokens, although it relaxes acceptance for robotic actions. SMTD uses strict verification of driving trajectories and retains multiple complete candidates. In Eq. (2), the main change from Hydra is a shared GRU with repeated anchor input in place of separate MLP heads, and conditioning on the target state and earlier draft tokens is already present in Hydra.

  2. Table 1 does not include ordinary AR sampling with the same shared prefill and branch batching. Arithmetic Sampling is a useful parallel baseline, but it uses a different candidate-generation procedure. The current results therefore do not show how much of the multi-candidate speedup comes from speculation itself.

  3. There are also some missing details in Appendix A. It lists UOCS’s features and training split, but the ΔPDM training label, training candidate pool, and source of agent-context inputs are not fully specified.

  4. The text in Figures 1–2, especially Figure 1, is too small to read comfortably at normal page size.

Questions for authors
  1. At N=8, how much of the latency reduction comes from speculation when target-only AR uses the same shared prefill, branch batching, and sampling settings? A matched timing or existing profiling would help. Is Greedy-reference generation included consistently in the reported timings?

  2. At N=4, SMTD has higher Oracle PDMS than Arithmetic Sampling (93.13 vs. 92.43) but lower Selected PDMS (89.75 vs. 89.90). Can the existing candidate scores and UOCS predictions explain this reversal? Please clarify how UOCS's training pool covers the different generators and candidate budgets.

  3. Are the completed trajectories independent samples from the target distribution, or are random draws coupled across branches? Eq. (1) addresses each candidate's marginal distribution. Please clarify the joint sampling policy and how it differs from Arithmetic Sampling when interpreting the fixed-N Oracle results.

  4. Could you specify the exact formula for UOCS’s ΔPDM training label, the source of the agent-context features at inference time, and the sampling settings (temperature, top-p and top-k) used for the target and drafter?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ajXV: Matched Timing, Candidate Selection, and Sampling Details

Thank you. Our matched experiments show an SMTD latency advantage at N=2,4, while the Oracle/Selected reversal exposes a selector limitation. We clarify these findings, joint sampling, and training details below.

1. Matched AR comparison and reference timing

Both methods use the same 64 NAVSIM scenes, three timing repetitions, target sampling settings, frozen target, and UOCS. Both share prefill across candidates and batch branch execution. “Shared-prefill AR” includes branch batching. Times are seconds per scene:

NShared-prefill AR generationSMTD generationShared-prefill AR E2ESMTD E2E
24.7842.3019.4226.940
44.8194.2569.4608.895
84.7957.6349.43912.272

SMTD remains faster at N=2 and N=4, with end-to-end speedups of 1.36x and 1.06x. These gains persist with both shared prefill and branch batching enabled for AR. Shared-prefill AR scales favorably and is faster at N=8, where speculation provides no net latency reduction.

Every E2E measurement includes a freshly generated AR-greedy reference (approximately 4.55 s), candidates, UOCS feature extraction/scoring, and pre/post-processing. Original Table 1 excludes the separate reference and selector costs; we will clarify its caption. Timing repetitions reuse the same scene seed.

2. Oracle/Selected reversal and training-pool coverage

Generating a high-scoring trajectory does not guarantee its selection. At N=4, SMTD achieves higher Oracle PDMS (93.13 versus 92.43), but lower Selected PDMS (89.75 versus 89.90). UOCS ranks candidates using predicted scores, so its ranking can miss the strongest available trajectory.

Our supplementary optimized-SMTD N=4 audit illustrates this: in scene 7561, UOCS chooses a candidate with predicted delta 0.0318 and actual PDMS 53.27 over one with predicted delta 0.0281 and actual PDMS 100. This is supplementary evidence of a ranking error, not a recovered original-table decision.

Multi-candidate generation supplies alternatives, speculative decoding targets their generation cost, and UOCS determines the executed trajectory. The reversal motivates improving selection; speculation's efficiency contribution is measured separately above.

The supplementary UOCS is trained on 8,718 verified navtrain scenes, using eight shared-prefill AR samples plus a reference. Nested prefixes form N=2,4,8 pools; set-dependent features are recomputed for each N. One ExtraTrees model is fitted jointly and frozen across generators and budgets. Training uses an anchored-GRU greedy reference; evaluation uses fresh AR greedy. The pool covers multiple budgets, not multiple generators. This is an audited reconstruction, not a recovered original-table selector.

3. Joint sampling and Arithmetic Sampling

SMTD branches have separate histories/KV caches and distinct proposal, acceptance, and correction draws from a shared CUDA generator. Draws are not broadcast across branches, and duplicate trajectories are allowed. Shared prefill reuses deterministic computation.

With exact rejection sampling and independent draws, the intended joint law is:

$ p(Y^{(1)},\ldots,Y^{(N)}\mid x) =\prod_{i=1}^{N}p(Y^{(i)}\mid x). $

Here p is the target distribution after temperature/top-p processing. The deterministic reference and fallback copies are excluded from this statement. Scheduling changes random-number consumption; BF16 execution can also change probabilities and fixed-seed outputs.

Arithmetic Sampling instead couples branches through a shared random offset, stratified arithmetic codes, and shared vocabulary permutations. Equal marginals need not yield equal Oracle distributions. Against matched iid AR, exact speculative sampling predicts the same Oracle distribution; a higher observed Oracle score alone does not establish a quality improvement from speculation.

4. Labels, context, and sampling settings

Supplementary UOCS labels use official NAVSIM v1.1 PDMResults.score, S in [0,1]:

$ S(\tau)=\mathrm{NC}(\tau)\mathrm{DA}(\tau) \frac{5\mathrm{EP}(\tau)+5\mathrm{TTC}(\tau)+2\mathrm{C}(\tau)}{12}, \qquad y_i=S(\tau_i)-S(\tau_{\mathrm{ref}}). $

NC denotes no-at-fault-collision compliance, DA drivable-area compliance, EP evaluator-normalized progress, TTC time-to-collision compliance, and C comfort. Reported PDMS is 100S. UOCS predicts the score difference from the reference and selects the largest prediction.

Agent features use current-frame ground-truth positions/velocities of up to eight nearest vehicles/pedestrians/bicycles within 40 m and x > -10 m. They are not VLM predictions; future annotations and offline PDMS are not online inputs. Timing excludes upstream perception and file I/O.

Target and draft sampling use temperature=1, top-p=0.95, and no top-k truncation. The draft distribution uses its trajectory-token vocabulary. The reference uses argmax; matched SMTD uses K=13.

Thank you for adding the matched batched AR comparison. SMTD speeds up generation and selection at N=2 and N=4, but takes about 30% longer at N=8, the setting highlighted in the paper. The new matched comparison reports only timing on the 64 NAVSIM scenes, leaving the quality–latency trade-off against batched AR unclear. I maintain my rating of 2.

Reviewer_8ohjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper looks at the latency of MLLM driving planners that output trajectories as tokens decoded autoregressively, which gets even more demanding when you want several candidate trajectories. They keep the OneVL planner frozen and propose Speculative Multi-Trajectory Decoding (SMTD). A small drafter (Medusa, or their anchored-GRU) guesses the next K trajectory tokens, and OneVL checks them in one pass as a batch, so several tokens get committed per OneVL step. For multiple candidates, the branches share the scene encoding and are verified together. Then a Unified Online Candidate Selector (UOCS) picks the final trajectory from the candidates plus the greedy one. They report 2.4–3.9× faster single-trajectory decoding on NAVSIM, ROADWork and Impromptu with about the same quality, and they additionally provide Oracle numbers that show an upper bound of performance.

Critical strengths
  1. I like the core idea and the intuition. A cheap drafter guesses ahead, and the big MLLM only verifies in a batch. The easier, finer details of the trajectory don't need the MLLM to generate them token by token, but the MLLM still checks every token. Doing this batch-wise across candidates, to take advantage of the GPU's batch capability, is also a nice idea. Applying Medusa-style drafting to driving trajectories is a nice contribution in itself.
  2. The way they frame multi-candidate planning is good: candidate-set potential, candidate-generation latency, and the quality of the final choice. I also like that they report Selected and Oracle PDMS separately, so we can see how much of the candidates' potential the selector actually gets.
Critical weaknesses
  1. The efficiency, which is the main promise of the paper, is not shown for the full pipeline.
    • No end-to-end latency. The reported multi-candidate latency is not end-to-end, and does not show the latency of each part of the design. As well, it is not clear how the greedy trajectory in the selection pool is generated or whether its time is counted. This is simple to measure, and without it we can't tell whether the proposed solution is useful in a full driving pipeline. It feels like incomplete presentation, and the claims should be revised based on those numbers.
    • No optimized inference. Since the work targets practical latency reduction, it can benefit from showing its integration with optimized inference (vLLM, SGLang, ...) to check that the speedups still matter there.
    • Little gain over a simple baseline. Compared to Arithmetic Sampling, a simple existing method, speculative multi-sampling doesn't offer much. Selected PDMS is 89.55 vs 89.65 at N=2, 89.75 vs 89.90 at N=4, and 90.26 vs 90.06 at N=8.
    • Small gain over a single trajectory. The gain over a single trajectory is small (+0.65 to +1.36 PDMS) for several times the latency, and this trade-off should be shown side by side and discussed.
  2. The proposed parts and the quality claims are not justified by the comparisons.
    • Medusa vs anchored-GRU. The anchored-GRU is basically the same as Medusa in terms of performance and latency: on NAVSIM, 1.349 s vs 1.428 s with exactly the same PDMS (88.9523). On ROADWork and Impromptu it is slower. The anchored-GRU agrees more with OneVL in Figure 4, but what that leads to in terms of performance is not supported. Also, only the anchored-GRU is used in the multi-candidate system; Medusa could be tested too. If SMTD with Medusa gives the same results, the introduction of the anchored-GRU is not justified.
    • Planning quality. The comparison is very narrow, only to OneVL. "Competitive planning quality" is never put in numbers in the abstract or compared against other planners. It is not necessary to achieve state of the art, but it should show relative numbers to other methods, to compare both latency and performance gaps.

Overall, I think the core idea and intuition are interesting, but the current framing does not show its advantages quantitatively, or qualitatively, as the paper lacks any qualitative examples.

Questions for authors
  1. What is the end-to-end latency (candidates + greedy trajectory + selector)? How is the greedy trajectory generated, and is it included in Table 1's time?
  2. What inference framework did you use, and do the speedups hold on optimized inference (vLLM, SGLang)?
  3. Can you show SMTD with Medusa, and clarify what the anchored-GRU design brings over Medusa?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 8ohj: End-to-End Latency, Inference Framework, and Drafter Comparison

Thank you for highlighting the need to clarify the complete planning cost, inference framework, and drafter choice. We have measured the full generate-and-select pipeline and added a matched Medusa-based SMTD comparison. These experiments quantify the reference/selection overhead and the benefit of anchored-GRU in multi-candidate decoding.

1. End-to-end latency and greedy reference

Table 1 reports candidate-generation latency, including scene prefill and candidate decoding. It excludes both the separately generated greedy reference and UOCS selection. We will state these exclusions explicitly in the caption. The reference is generated by ordinary autoregressive greedy decoding of the frozen OneVL target, using token-wise argmax. In the supplementary experiment, it is generated afresh and timed for every scene and configuration:

$ T_{\mathrm{E2E}} = T_{\mathrm{pre/post}} + T_{\mathrm{greedy}} + T_{\mathrm{candidates}} + T_{\mathrm{UOCS}}. $

Here, UOCS time includes feature extraction, scoring, and selection. The following results use optimized SMTD on 64 prespecified NAVSIM scenes, with three timing repetitions. All times are seconds per scene.

NGreedy referenceCandidatesUOCSPre/postEnd-to-end
24.5532.3010.0240.0626.940
44.5514.2560.0260.0638.895
84.5467.6340.0290.06212.272

Selection adds approximately 24–29 ms; the greedy reference contributes approximately 4.55 s. All configurations share the target, sampling settings, and frozen UOCS, reconstructed on 8,718 available navtrain scenes.

Timing starts with the RGB image and current-frame agent context available in memory and ends with the selected CPU trajectory. Agent context is supplied from current-frame ground-truth annotations. Model loading, file I/O, upstream perception, and offline PDMS evaluation are excluded. We will distinguish this planning-pipeline latency from candidate-generation latency.

2. Inference framework and vLLM/SGLang

Our implementation uses PyTorch and Hugging Face Transformers with a custom decoding controller. The supplementary runs use PyTorch 2.10.0, Transformers 4.57.0, and BF16 target inference on one RTX 5090D.

This implementation provides direct control over the target hidden states used for anchored drafting, per-trajectory KV caches, branch regrouping for batched verification, and state updates after acceptance or residual correction. It also allows the compared decoding methods to share the same target-model execution path and evaluation settings.

Integrating SMTD into vLLM or SGLang requires adapting this controller to the engine's cache management and speculative-decoding scheduler, then validating its sampling and verification behavior. We have not benchmarked SMTD in either engine; the reported speedups apply to our PyTorch/Transformers implementation.

3. SMTD with Medusa and the benefit of anchored-GRU

Our single-trajectory experiments already show that anchored-GRU is faster than Medusa on NAVSIM: 1.349 s versus 1.428 s. This finding motivated our choice of anchored-GRU for multi-candidate SMTD on this dataset.

Since multi-candidate decoding retains the draft-and-verify mechanism, we expected this advantage to extend to the complete generate-and-select pipeline. To test this expectation and address your suggestion, we additionally evaluated SMTD with Medusa.

Both drafters use the same optimized controller, frozen target, AR reference, UOCS, and target/draft sampling settings: temperature 1, top-p 0.95, and top-k disabled. At matched draft length K=13, the completed first timing pass covers the same 256 NAVSIM scenes:

NMedusa E2E (s)GRU E2E (s)Medusa acc. lengthGRU acc. length
27.3336.9343.5394.520
49.5608.8553.5844.499
813.58212.2903.5724.486

GRU denotes anchored-GRU. Accepted length is the mean number of accepted draft tokens per branch verification round, excluding the confirmed root token.

The results confirm the same latency ordering as the NAVSIM single-trajectory experiment: anchored-GRU reduces end-to-end latency by 5.4%, 7.4%, and 9.5% at N=2,4,8. Medusa at its paper-selected K=14 yields 7.311, 9.615, and 13.656 s, preserving this ordering.

Medusa predicts token offsets through parallel heads from a common target state. Anchored-GRU conditions each proposal on the preceding draft token and a fixed projection of the verified target state:

$ s_{t,0} = P(h_t), \qquad s_{t,j} = \mathrm{GRU}(s_{t,j-1}, [E(d_{t,j-1}); A(h_t)]). $

This models dependencies within the draft block. The longer accepted prefixes observed above are consistent with this design motivation and accompany lower end-to-end latency in the tested multi-candidate configuration.

Thanks to the authors for the response. My concerns are partially addressed. But the weakness still exists. I prefer to keep my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.