acceptodds
Under review as a conference paper at ICLR 2027

VTD: Visible-Text Drafting for Speculative Decoding in Vision-Language Models

Abstract

Generating HTML or Markdown from images often requires vision-language models to produce long sequences, making autoregressive decoding a major source of latency. Retrieval drafters can accelerate generation without training, but their text sources may omit content visible in the image. At steps where dynamic retrieval accepts no draft descendant, only of the target's next three-token spans have a matching occurrence beginning at an earlier position in the same output. This limited recurrence motivates an additional retrieval source. We introduce VTD (Visible-Text Drafting), a speculative decoding method that uses OCR-recovered text as a retrieval source for drafting. We run OCR once, index the recognized text in a suffix automaton, and fuse its draft tree with a tree from a dynamic automaton over the prompt and accepted output. VTD requires no drafter training, external text corpus, or changes to target model weights. On five benchmarks, VTD achieves mean end-to-end speedups of on Qwen2.5-VL-3B, on Qwen2.5-VL-7B, and on InternVL2.5-8B. VTD outperforms every evaluated training-free retrieval baseline in speedup on every benchmark and target, and improves average speedup over the trained ViSpec drafter by on Qwen2.5-VL-7B. These results show that visible text is a complementary source for retrieval drafting.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language model, speculative decoding, retrieval-based drafter, markup language generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_okXtOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Visible-Text Drafting (VTD), a training-free speculative decoding method designed to accelerate Vision-Language Model (VLM) inference on image-to-markup transcription tasks such as webpage-to-HTML and document-to-Markdown generation. The authors argue that conventional retrieval-based speculative drafters, which rely exclusively on prompt context or past generated text, fail to predict content that is visible in the input image prior to its first emission. To overcome this, VTD adopts a dual-path retrieval strategy: it runs an off-the-shelf OCR model once on the input image to build a fixed visible-text suffix automaton, alongside a dynamic suffix automaton that continuously indexes the generated sequence history. At each decoding step, both automata propose candidate continuation trees rooted at the target model's previous greedy token. These two trees are merged into a single verified tree and scored by the target VLM in a single forward pass with a tree-attention mask. Across five benchmarks and three target models from the Qwen-VL and InternVL families, the paper reports end-to-end speedups ranging between 1.62x and 1.97x, outperforming both training-free retrieval baselines and the trained ViSpec drafter.

Critical strengths
  1. The paper provides a thorough empirical diagnosis of generation misses in multimodal transcription - only 2.3% of 3-token miss spans have appeared in the prior output sequence - convincingly exposing the fundamental recall ceiling of history-only retrieval drafters in multimodal tasks.
  2. The authors include the serial execution latency of the OCR inference pass in their primary end-to-end wall-clock throughput and speedup measurements.
  3. VTD requires no auxiliary training of draft models or specialized prediction heads, allowing it to be deployed across different open-source VLM architectures without target-specific fine-tuning.
  4. The target-side tree verification strictly preserves the target model's greedy autoregressive output distribution, ensuring no degradation in transcription fidelity.
Critical weaknesses
  1. By reintroducing an external OCR model into the inference pipeline, VTD creates a conceptual paradox: If a lightweight, highly accurate OCR tool is already available in the system, why use an expensive billion-parameter VLM to autoregressively decode document content in the first place? A conventional two-stage pipeline (OCR + text LLM) is typically faster and less resource-intensive.

  2. Table 3 demonstrates that when switching from EasyOCR to a modern, more accurate OCR engine (PaddleOCR / PP-OCRv5), the Mean Accepted Tokens (MAT) improves, but wall-clock speedup drops dramatically across all five benchmarks, collapsing to 0.97x on OCR-Md-Dense (a net slowdown). This reveals that VTD’s wall-clock advantage is extremely fragile: the entire acceleration benefit exists only within a narrow window where the OCR tool is small enough, the image is simple enough, and decoding is long enough.

  3. Section 4.3 admits that tokenization and suffix-automaton construction of the OCR text are excluded from the reported latency. Furthermore, speculative decoding in production serving systems (e.g., vLLM, SGLang) relies heavily on batched operations and continuous scheduling. Integrating an unbatched, external Python/PyTorch OCR pipeline into the request critical path introduces substantial memory fragmentation, CUDA context overhead, and pipeline bubbles that cannot easily scale to batched multi-tenant environments.

  4. Section 4.2 motivates tree fusion by asserting that selecting between the two sources would discard viable branches. Yet, the paper never benchmarks a simple "Source Selection / Switching" baseline (e.g., choose whichever automaton has a longer suffix match, or prioritize OCR when match length exceeds a threshold). Without this ablation, it is impossible to determine whether the complex tree-merging and tree-attention masking add genuine algorithmic value, or if a trivial switching heuristic would achieve the same speedup with lower verification width.

  5. Table 2 presents Dynamic-only, Dynamic + Static Corpus, and Dynamic + Visible Text, but omits Visible-only (drafting solely from the OCR automaton). Figure 3(c) shows that on dense documents, the visible source provides up to 69% of accepted tokens while the dynamic source is negligible. If the visible source alone provides almost all the speedup, the dynamic sequence automaton (and the dual-path machinery) may be largely redundant.

  6. SAM-D is evaluated with a draft budget of 40 nodes. In contrast, VTD merges two 40-node trees, yielding a combined tree of up to 79 candidate nodes. It remains unproven whether VTD’s higher acceptance rate is driven by its fusion strategy or simply by doubling the target model's speculative candidate budget per forward pass.

  7. As documented in Table 5, VTD yields negligible acceleration on benchmarks where output is not directly transcribed from visible text (e.g., ChartMimic MAT gain +0.07; Chart2Code MAT gain +0.09). Consequently, VTD is not a general-purpose speculative decoding method for VLMs, but rather a domain-specific speedup trick restricted strictly to literal image transcription. This narrows the scope of the work's contribution.

Questions for authors
  1. In Section 4.2, tree merging is motivated by avoiding the loss of valid branches from either source, but Figure 4(a) shows that evaluating the merged tree increases per-step verification latency by up to 15.0%. What are the Mean Accepted Tokens (MAT), throughput, and wall-clock speedup when using a simple single-source selection heuristic (e.g., dynamically choosing either the OCR or dynamic branch based on match length or a threshold) instead of tree merging? Does the modest acceptance gain of tree fusion justify the extra tree-attention verification overhead over greedy source selection?

  2. The baseline SAM-D is restricted to a draft budget of N=40N = 40 nodes, whereas VTD verifies a merged tree containing up to 2N−1=792N - 1 = 79 candidate nodes. What is the performance when: (1) SAM-D is allowed an equal budget of N=79N = 79 nodes on its dynamic sequence index? (2) VTD is strictly constrained to a total combined budget of N=40N = 40 nodes (e.g., dynamic and visible sources split the 40-node ceiling)? Additionally, is VTD's acceptance gain an algorithmic achievement of dual-source retrieval, or largely an artifact of doubling the target model's speculative candidate pool per forward pass?

  3. Table 2 reports Dynamic-only and Dynamic + Visible, but omits Visible-only - drafting solely from the fixed OCR automaton Avis\mathcal{A}_{vis}. Given that visible-text drafting accounts for up to 69% of accepted tokens on dense document benchmarks (Figure 3c), what is the standalone performance of Avis\mathcal{A}_{vis} across all benchmarks? If visible-only drafting achieves comparable speedups on document tasks, does maintaining the dynamic automaton Aseq\mathcal{A}_{seq} - which requires online graph expansion after every accepted token—provide sufficient marginal utility to justify its algorithmic and memory complexity?

  4. Section 4.3 states that tokenization of the OCR text and construction of the visible suffix automaton are excluded from reported latency. Furthermore, Table 3 shows that replacing EasyOCR with PaddleOCR (PP-OCRv5) flips the method into a net slowdown (0.97×0.97\times on OCR-Md-Dense). What is the exact wall-clock overhead (in milliseconds) of tokenizing the OCR output and constructing \mathcal{A}_{vis}, and how does including these steps affect the end-to-end speedups in Table 1? Analytically or empirically, what is the maximum permissible OCR latency threshold (TOCRmax⁡T_{\text{OCR}}^{\max}) as a function of output sequence length, beyond which VTD fails to provide any wall-clock acceleration?

  5. If a deployment pipeline is already willing to pay the computational and latency cost of running an external OCR pass prior to generation, how does VTD compare against a standard OCR-augmented prompting baseline (i.e., concatenating the OCR-extracted text directly into the prompt context of the VLM or a faster text-only LLM)? Does VTD offer higher wall-clock throughput, lower peak memory, or better formatting fidelity than simply providing the OCR text upfront?

5 comments

[1/3] We thank the reviewer for recognizing our miss diagnosis (only 2.3% of 3-token miss spans recur in the prior output), the end-to-end accounting of OCR latency, and VTD's training-free portability. We ran every requested experiment on all five benchmarks with Table 1's protocol (Qwen2.5-VL-7B, same pages and AR baseline) and will add all new results to the revised manuscript. This comment covers W1a and W1b/Q4; [2/3] covers W1c and W2/Q1–Q3; [3/3] covers W3 and Q5.

W1a. Why run a VLM if OCR is available?

We agree that a two-stage pipeline (OCR + text-only LLM) can be cheaper. VTD does not compete with it, because the two produce different outputs. VTD never outputs OCR text. It uses it only as a source of draft guesses, and the VLM writes and verifies every token.

  1. OCR covers the text, not the structure. Even on OCR-Md-Dense, the most text-heavy benchmark, OCR supplies only 42% of the output tokens; the target's own token at each verification step supplies 40% and the in-sequence index 19%. On webpages the output is HTML/CSS layout, which a reader does not produce. OCR plus a text-only LLM loses layout, styling, tables and reading order.
  2. OCR errors cannot reach the output. A drafted token is kept only if the target would have produced it, so an OCR error costs at most one rejected guess. In a two-stage pipeline it goes straight into the output.
  3. The reader is small and replaceable. EasyOCR has 24.6M parameters, against 7–8B for the target. What matters is acceptance per unit of OCR latency on the deployment hardware; on a GPU, the more accurate PaddleOCR is also faster and gives more speedup (W1b).

W1b / Q4. OCR latency and excluded overheads

  • Excluded overheads, now measured. Tokenizing the OCR text and building Avis\mathcal{A}_{vis} take a median 2.4 ms per page (p95 6.3 ms), 0.03% of VTD's end-to-end time.
  • Maximum tolerable OCR latency. VTD beats AR exactly when TOCR<TAR−TVTD-decT_{OCR} < T_{AR} - T_{VTD\text{-}dec}, so for output length LL, TOCRmax⁡≈L⋅tAR(1−1/sdec)T_{OCR}^{\max} \approx L \cdot t_{AR}(1 - 1/s_{dec}), where tARt_{AR} is AR time per token and sdecs_{dec} is VTD's speedup without OCR. From the Table 1 runs:
BenchmarkMean LLSaving / tokenTOCRmax⁡T_{OCR}^{\max}EasyOCR (mean)Headroom
WebCode2M1,31213.8 ms18.1 s0.67 s27×
Design2Code1,01213.2 ms13.3 s0.36 s37×
OmniDocBench76811.8 ms9.1 s1.08 s8.4×
OCR-Md-Dense56014.9 ms8.3 s1.20 s6.9×
OmniAI-OCR95116.1 ms15.3 s1.03 s15×

Headroom is 4.7–26× on Qwen2.5-VL-3B and 3.3–26× on InternVL2.5-8B (tightest: InternVL on OmniDocBench, 484-token mean output). OCR costs more than it saves on 14 of 475 Qwen2.5-VL-7B pages, all with outputs of at most 407 tokens (median page: 773).

  • The Table 3 slowdown is a device effect. Table 3 ran PaddleOCR (PP-OCRv5 mobile) on the CPU and EasyOCR on the GPU, which the submission did not state. We reran PaddleOCR on the GPU with the same models (serial speedup, OCR on the critical path; one common page set per benchmark, so cells can differ from Table 3 in the second decimal):
VTD + reader (device)Median OCR / pageWebCode2M (mid)Design2CodeOmniDocBenchOCR-Md-DenseOmniAI-OCR
PaddleOCR (CPU), as in Table 33.9–9.9 s1.69×1.62×1.06×0.97×1.29×
EasyOCR (GPU), default0.29–1.16 s2.28×2.14×1.58×1.78×2.15×
PaddleOCR (GPU)0.11–0.28 s2.76×2.30×1.98×2.43×2.49×

On the GPU, PaddleOCR keeps its MAT and beats EasyOCR's serial speedup on all five benchmarks, by 7–37%. The 0.97× in Table 3 therefore came from the CPU, and on a GPU the more accurate reader is also the faster choice. We will add the GPU row to Table 3, and we have rewritten §5.3 so that the comparison no longer reads as an engine ranking.

  • Where VTD loses. A slow enough reader does erase the gain, but OCR would have to be 3.3–37× slower than EasyOCR on the GPU (Headroom above) for VTD to fall below AR, so we do not consider the operating window narrow.

[2/3]

W1c. Serving systems

We agree that we evaluate batch size 1 in an HF-based harness, as is common for lossless speculative decoding, and we do not claim results for large batches; the revision will state this limitation explicitly. Two properties suggest the cost is manageable in serving:

  1. OCR runs once per request, outside the decode loop. It can be batched across requests with off-the-shelf tooling (EasyOCR exposes batched recognition), and it needs only the image, so it can overlap with image encoding or with other requests' decoding.
  2. The rest of VTD is a CPU-side suffix-automaton lookup, the same kind of drafter vLLM already ships for n-gram and suffix speculation. It needs no extra GPU model and no extra KV cache.

As with all speculative decoding, gains shrink as batch size grows and decoding becomes compute-bound.

W2 / Q1–Q3. Fusion vs. selection, equal budget, visible-only

We ran all three ablations on all five benchmarks with Table 1's protocol. All rows share one page set per benchmark (pages where all six configurations stop at a natural end of sequence), so the two reference rows can differ slightly from Table 1. Cells are MAT / speedup; the OCR pass is included wherever it is used.

ConfigurationWebCode2MDesign2CodeOmniDocBenchOCR-Md-DenseOmniAI-OCR
SAM-D (dynamic only, N = 40)1.72 / 1.60×1.82 / 1.74×1.36 / 1.31×1.32 / 1.25×1.67 / 1.56×
SAM-D, N = 801.75 / 1.62×1.82 / 1.75×1.37 / 1.33×1.32 / 1.24×1.67 / 1.56×
Visible-only (OCR index alone)1.39 / 1.26×1.22 / 1.15×1.61 / 1.20×1.93 / 1.44×1.86 / 1.60×
Selection (longer match wins)2.49 / 2.19×2.35 / 2.13×1.97 / 1.49×2.53 / 1.74×2.57 / 2.08×
VTD, merged tree capped at 402.53 / 2.21×2.37 / 2.14×2.03 / 1.52×2.60 / 1.77×2.70 / 2.17×
VTD (fusion, default)2.53 / 2.22×2.37 / 2.14×2.03 / 1.52×2.60 / 1.78×2.70 / 2.17×
  • Equal budget (Q2). Doubling SAM-D's budget to N = 80 changes its MAT by at most 0.03. VTD capped at 40 nodes keeps its full gain and, with half the budget, is 1.14–1.43× faster than SAM-D at N = 80. The gain does not come from verifying more nodes.
  • Visible-only (Q3). The OCR index alone beats the dynamic index alone on the three document benchmarks (OCR-Md-Dense 1.93 vs. 1.32 MAT) and loses on the two webpage benchmarks (Design2Code 1.22 vs. 1.82), where markup repeats within the output. Neither source alone reaches fusion on any benchmark, so both are needed.
  • Fusion vs. selection (Q1). Selection captures most of the gain, which confirms that the main contribution is the visible-text source. Fusion is at least as good on all five benchmarks, by 0.02–0.13 MAT (largest on OmniAI-OCR, 2.70 vs. 2.57), at no extra drafting cost.

Two further results point the same way:

  • The merged tree is far below its bound. The bound is 2N − 1 = 79 nodes, but the merged tree averages 8.3–14.0 nodes per step (p95 25–40), because each source's budget scales with its match length (Eq. 1).
  • Extra nodes alone give no speedup. In the wrong-language control (Table 9), the English reader's text does not match Arabic pages. The tree is built exactly as in VTD and MAT rises from 1.32 to 1.51, yet speedup is unchanged (1.283× → 1.284×).

On the marginal value of the dynamic index (Q3): it supplies 31–66% of drafted tokens (Fig. 3c), most on webpages, at one amortized O(1) CPU automaton extension per accepted token. Appendix B.2 measures fusion's whole extra verification work at 3–15% per step.

[3/3]

W3. Applicability boundary

We agree that VTD is not a general-purpose VLM accelerator, and the paper says so (Appendix B.4). It targets long outputs that transcribe visible text, which covers webpage-to-code and document-to-Markdown, both common deployed uses of VLMs in which decoding dominates latency. Domain-targeted, training-free drafters are an established design point; SuffixDecoding, for example, targets repetitive agentic workloads. Where image text is not transcribed (charts), VTD costs little and speedup still rises slightly: ChartMimic 1.477× → 1.508×, Chart2Code 1.848× → 1.890× (Table 5). Users therefore do not need to detect the task in advance.

Q5. OCR-augmented prompting

We ran this baseline on Qwen2.5-VL-7B with the same EasyOCR text VTD uses, appended to the instruction, on the same pages. The OCR pass is charged to every row; speedup is over AR on the original prompt.

WebCode2M (mid)Design2CodeOmniDocBenchOCR-Md-DenseOmniAI-OCR
AR, OCR text in prompt0.94×0.97×0.89×0.91×0.90×
SAM-D, OCR text in prompt2.14×2.06×1.44×1.73×2.00×
VTD2.23×2.12×1.57×1.79×2.19×
Outputs changed by the OCR prompt98%100%94%82%98%
  • Throughput. The OCR text adds 213–669 prompt tokens (prefill +0.03–0.10 s) and slows AR itself. Even with SAM-D retrieving from that text, the method stays below VTD on every benchmark.
  • Peak memory. It grows by 0.1–1.8 GB with the longer prompt. VTD adds no prompt tokens.
  • Fidelity. The OCR prompt changes 82–100% of outputs. Character edit distance to the ground truth rises on four of five benchmarks (e.g., OCR-Md-Dense 0.137 → 0.160) and is unchanged on OmniAI-OCR. On Design2Code, CLIP similarity of the rendered page to the reference moves by −0.013 (95% CI −0.039 to +0.014): pages change with no consistent gain. VTD's rendered pages are identical to the original on 56 of 67 pages; the rest differ only where AR's top two logits are tied at fp16 precision, which any change of kernel or batch shape can flip.

The two approaches also differ in kind. VTD leaves the model's input unchanged and preserves its greedy output up to fp16 ties. OCR prompting changes the input, so it needs its own quality evaluation. A text-only LLM fed OCR text cannot recover layout, styling or tables from the image, so it is not an alternative way of solving image-to-markup.

Reviewer_CB6nOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Visible-Text Drafting (VTD), a training-free speculative decoding method for accelerating long-form generation in vision-language models (VLMs). The authors identify that existing retrieval-based drafters are limited by their reliance on previously available text, leaving image-specific content inaccessible before it is generated. To address this limitation, VTD extracts visible text from the input image using OCR and builds a per-image suffix automaton, which is combined with a dynamic suffix automaton over the prompt and generated output. Candidate trees from both sources are merged and jointly verified by the target model. Experiments across five benchmarks and multiple VLM architectures demonstrate consistent speedup improvements over existing training-free retrieval-based methods, without requiring additional drafter training or modifications to the target model.

Critical strengths

1.The paper is well motivated by an empirical analysis of the limitations of existing retrieval-based speculative decoding methods. The observations provide a clear rationale for introducing visible image text as an additional drafting source.

2.The proposed method is simple, intuitive, and practical. It effectively combines complementary retrieval sources without requiring additional training or modifications to the target model, making it readily applicable to different VLM architectures.

  1. The experiments are comprehensive, covering multiple VLMs and benchmarks. The ablation studies, source attribution analysis, and OCR latency evaluation provide convincing evidence of the method's effectiveness and help explain its performance gains.
Critical weaknesses

I did not identify any critical weaknesses that fundamentally undermine the proposed method or its main conclusions. The motivation, methodological design, and experimental evidence are generally convincing. My remaining concerns are mainly related to the consistency of experimental configurations and the clarity of reporting, which can be addressable through clarification.

Questions for authors
  1. Section 4.2 sets b_seq = 0 for VTD, while Appendix A.3 reports b=5 for the SAM-D baseline. Could the authors clarify whether the dynamic retrieval component uses the same match-length offset in both methods?

  2. Appendix B.6 states that (K=8) is used throughout the experiments. However, for Qwen2.5-VL-7B on Design2Code, Table 1 reports a MAT of 2.37, while Table 7 reports 2.639 under (K=8). Could the authors clarify why these results differ despite the stated configuration, and whether any other experimental settings were changed?

3 comments

We thank the reviewer for the careful reading and for finding the motivation, the method and the experiments convincing. Both questions point to real reporting problems in the submission, and we have fixed them in the revised PDF.

Q1. Does the dynamic index use the same match-length offset in VTD and SAM-D?

Yes. In both methods the dynamic index uses an effective offset of 0.

  • b = 5 applies only to SAM-D's external static corpus. Both the official SAM-Decoding implementation and our port subtract it only from static-corpus match lengths, never from the dynamic suffix automaton over the prompt and generated output.
  • Our SAM-D baseline is dynamic-only (the static corpus is disabled), so b = 5 has no effect on it. Its dynamic index is configured exactly as VTD's dynamic source (N = 40, α = 4, b_seq = 0). VTD and SAM-D therefore differ only in the added visible-text source.

Appendix A.3 listed b = 5 among the SAM-D settings without saying that it applies only to the static corpus. We have rewritten that sentence and added a note in §4.2.

The offset still matters for the visible-text source. Applying the corpus-scale b = 5 to it removes most short matches and lowers VTD's MAT by 7.8–34.2% across the five benchmarks (Table 8), which is why VTD uses b_vis = 0.

Q2. Why does Table 7 report MAT 2.639 at K = 8 on Design2Code while Table 1 reports 2.37?

Two errors in Table 7 combined:

  1. Wrong source run. Table 7's K = 8 row came from an earlier run that also loaded the external static corpus, so it measured a three-source configuration (dynamic + static corpus + visible text), not VTD. The K = 4/16/32 rows came from two-source runs.
  2. Different page filter. Table 7 dropped only pages where a decoder hits the generation cap, keeping 74 Design2Code pages. Table 1 also drops "runaway" pages, where a decoder's output exceeds 1.5× the autoregressive length, and keeps only pages on which every method ends naturally (68 pages). One of the extra pages reaches MAT ≈ 17 because the decoder keeps repeating the page's own markup, and that page alone raises the mean.

Re-evaluated with two-source runs on exactly Table 1's pages (68 Design2Code and 97 OCR-Md-Dense):

KDesign2CodeOCR-Md-Dense
MATSpeedupMATSpeedup
42.3712.140×2.5981.784×
8 (default)2.3712.140×2.6001.778×
162.3712.141×2.6001.784×
322.3712.140×2.6001.785×
  • K has no measurable effect. MAT is identical to two decimals and speedup varies by at most 0.007× (0.4%). The cap rarely binds because a single page's OCR index branches little: weighted by occurrence, only 3.3% (Design2Code) and 7.0% (OCR-Md-Dense) of automaton states have more than eight distinct continuations. K = 8 is therefore not tuned; any value in 4–32 gives the same result.
  • We corrected our conclusion. The submission said K = 8 "reaches the useful branch-coverage plateau". Appendix B.6 now says that VTD is insensitive to K.

Other experimental settings

We re-checked every table and figure. All configuration settings are identical throughout.

  • Appendix B.2 / Fig. 4 used the same three-source Design2Code run as Table 7. They now show MAT 1.82 → 2.37 (previously 2.06 → 2.59), 2.135× serial and 2.206× with OCR hidden (previously 2.263× and 2.338×), and a 2.9% verification-cost increase per step (previously 3.0%). The OmniAI-OCR and OCR-Md-Dense entries were already correct, and Fig. 1 reads only the dynamic-only arm, so it is unaffected.

  • Tables 2, 3, 5, 7, 8 and 9 apply same filter rule (natural end of sequence, no runaway) to the runs each table compares, and their WebCode2M column is the mid split, on which these ablations were run. A few cells therefore differ slightly from Table 1 (e.g., WebCode2M VTD MAT 2.52 vs. 2.54). The captions now say so.

We thank the reviewer again; these checks made the appendix consistent with the main results.

Thank you for the detailed response. My concerns have been adequately addressed, and I will maintain my original decision.

We appreciate your careful review and are glad that our responses clarified the technical details. Thank you for your engagement throughout this revision process.

Reviewer_3v1vOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a method called VTD, which uses OCR to read the results from images and then uses them as candidate inputs during speculative decoding. In this way, the candidate generation process can be accelerated.

Critical strengths

The idea is simple and easy to implement. It can be readily integrated into existing speculative decoding frameworks.

Critical weaknesses

The reported speedups are relatively limited, and the choice of baselines is overly narrow. The paper mainly compares VTD with training-free retrieval-based speculative sampling methods, such as PLD, REST, SuffixD, and SAM-D. However, it does not compare against conventional speculative decoding methods, such as EAGLE. The reported speedups are 1.84× for Qwen2.5-VL-3B, 1.97× for Qwen2.5-VL-7B, and 1.62× for InternVL2.5-8B. All of these improvements are below 2×, making the practical benefit relatively modest. The selected baselines appear too weak, and the paper does not provide a convincing explanation for this choice.

Questions for authors
  1. Based on my experience, EAGLE-3 can often achieve 2–3× speedups in downstream domains after fine-tuning, which is substantially better than the 1.5–2× speedups reported for the methods discussed in this paper. If training a small drafter model can achieve comparable or better performance, why should we use the proposed method instead? If the authors consider training to be a major overhead, standard speculative decoding with a small model—sometimes only a fraction of the size of the target model—can also achieve speedups exceeding 1.5×.

  2. Please provide a clearer explanation of why retrieval-based methods are necessary for inference in vision-language model (VLM) scenarios. More comprehensive experimental results are also needed to demonstrate whether native speculative decoding genuinely performs poorly in this setting. Based on the current results, I instead find that retrieval-based methods may not be particularly suitable for VLMs.

  3. Please include comparisons with conventional speculative decoding methods, such as EAGLE. The paper should also report relevant tree parameters, including the tree depth and width.

4 comments

We thank the reviewer for noting that VTD is simple and easy to integrate into existing speculative decoding frameworks. We added a comparison with a public EAGLE-3 drafter, a combination of EAGLE-3 with VTD, an analysis of a small draft model, and the tree parameters, and will add all of them to the revised manuscript.

W1. Narrow baselines and modest speedups

We agree that EAGLE-style drafters are the standard baseline, and we now compare against EAGLE-3 directly (Q1). On the two other points:

  1. Table 1 already includes a trained VLM drafter. ViSpec reaches the highest MAT on Qwen2.5-VL-7B (2.73 on average) but a lower speedup (1.44×), because its draft model runs at every decoding step. VTD reaches 1.97× on average with no training.
  2. The speedups are end-to-end. They include the OCR pass on the critical path and are measured on long outputs at batch size 1. VTD is also not limited to EasyOCR: with PaddleOCR on the GPU it reaches 2.04–2.77× on the five benchmarks (Table 3) (reply to Reviewer okXt, W1b).

Q1. Would a trained EAGLE-3 or a small draft model do better?

EAGLE-3. We evaluated the public EAGLE-3 checkpoint for Qwen2.5-VL-7B (from SpecForge, trained on 95K ALLaVA-4V samples) with Table 1's settings (greedy decoding, batch size 1) and a 30-token tree (depth 3, top-k 8), the fastest of the three trees we tried (Q3).

MethodWebCode2MDesign2CodeOmniDocBenchOCR-Md-DenseOmniAI-OCRAverage
MATSpeedupMATSpeedupMATSpeedupMATSpeedupMATSpeedupMATSpeedup
EAGLE-31.511.13×1.411.05×1.841.27×1.901.35×2.171.65×1.771.29×
VTD2.542.22×2.372.14×2.081.56×2.601.78×2.702.17×2.461.97×
EAGLE-3 + VTD2.982.05×2.631.87×2.761.54×3.331.82×3.352.18×3.011.89×
  • VTD is faster than EAGLE-3 on all five benchmarks, by 1.23–2.04× (1.97× vs. 1.29× on average). The public checkpoint accepts few tokens on long markup outputs.
  • VTD composes with a trained drafter. Merging VTD's dynamic and OCR trees into EAGLE-3's tree raises MAT from 1.41–2.17 to 2.63–3.35 (1.77 → 3.01 on average), and the combination is 1.21–1.81× faster than EAGLE-3 alone.
  • The combination is not always faster than VTD alone. On our GPU it is 8–14% slower than VTD on the two webpage benchmarks and between 1% slower and 2% faster on the three document benchmarks (OmniDocBench 1.54× vs. 1.56×, OCR-Md-Dense 1.82× vs. 1.78×, OmniAI-OCR 2.18× vs. 2.17×). Each EAGLE-3 step adds 12–16 ms over an autoregressive step (three sequential draft-head passes plus a 30-node verification), against 5–10 ms for VTD.
  • Domain fine-tuning. We did not train a domain-specific EAGLE-3, so we cannot rule out that one would outperform VTD. VTD needs no training and, as the last row shows, adds to a trained drafter rather than competing with it.

A small draft model. On our harness, Qwen2.5-VL-3B costs 0.69–0.74 of Qwen2.5-VL-7B's time per token. With a draft model of relative cost c, standard speculative decoding is bounded by (γ+1)/(γc+1) < 1/c even when every drafted token is accepted. This gives at most 1.35–1.45× for 3B drafting for 7B, below VTD's 1.56–2.22× on all five benchmarks (Table 1). The 3B drafter would also need its own image prefill and about 7 GB of extra GPU memory.

Q2. Why retrieval-based drafting for VLMs?

We agree that history-only retrieval suits VLMs poorly, and the submission makes the same diagnosis (§3.2): only 2.3% of 3-token miss spans have already appeared in the output, so text that comes from the image is not yet in the history when it is first needed. This is why SAM-D reaches only 1.25–1.75× on Qwen2.5-VL-7B. VTD adds the source that history lacks, the visible text on the image, and that raises the speedup to 1.56–2.22× with no training. Model-based drafting is not ruled out either, since VTD's trees merge into EAGLE-3's tree (Q1).

Q3. Tree parameters

  • VTD. Each source s builds a tree of at most N = 40 nodes, sized by its suffix-match length ℓ (Eq. 1: n_s = min(N, 1 + ⌊α · max(0, ℓ − b_s)⌋), α = 4, b_s = 0 for both sources). The visible-text source admits at most K = 8 nodes per depth. The two trees share the root, so the merged tree has at most 2N − 1 = 79 nodes. Measured per decoding step on Qwen2.5-VL-7B, the merged tree has 7.7–14.2 nodes on average (95th percentile 25–40), mean depth 4.0–7.4 and mean maximum width 2.1–3.5.
  • EAGLE-3. 30 draft tokens, depth 3, top-k 8. On Design2Code and OmniAI-OCR benchmark, the larger trees 60/5/10 and 64/4/10 (tokens/depth/top-k) raised MAT by at most 0.11 but cut speedup by 3–9%, so we use 30/3/8.

The revision adds the EAGLE-3 comparison to §5.2 (Table 1b) and the rest to Appendix B.10–B.11.

AuthorsOpenReview ↗

Gentle Reminder to Reviewer 3v1v

We would be grateful for any additional feedback you might have following our rebuttal. We remain available to address questions or concerns and would welcome your perspective whenever convenient.

I still have some concerns about the feasibility of your approach. Your additional experiments reveal an interesting pattern: EAGLE-3 achieves a 1.51× speedup, VTD achieves 2.54×, and EAGLE-3 + VTD achieves 2.98×. The individual gains over the baseline are nearly additive: 0.51 + 1.54 = 2.05, which is close to the combined gain of 1.98. This suggests that EAGLE-3 and VTD may offer complementary benefits.

However, my understanding is that VLMs are still fundamentally generating text tokens, and their output token distributions may be more concentrated. It is therefore unclear to me why a standard method such as EAGLE-3 would perform poorly in this setting. Your results seem to suggest that the two methods are complementary in terms of which tokens are accepted.

Could you analyze which types of tokens are more likely to be accepted with EAGLE-3 and which with VTD? What properties of the VLM setting give rise to this complementarity?

We thank the reviewer for the follow-up. In short, the two drafters win at different steps: VTD copies text that already exists verbatim, on the image or earlier in the output, and EAGLE-3 predicts the rest. The measurements below show this directly.

One clarification on the numbers: 1.51, 2.54 and 2.98 are the MAT columns for WebCode2M. The corresponding speedups are 1.13×, 2.22× and 2.05× (1.29×, 1.97× and 1.89× on average). The reviewer's additivity observation does hold for MAT: on the same pages in one harness, EAGLE-3 + VTD keeps 82–95% of the sum of the two methods' MAT gains on all five benchmarks. Speedups add less, because a combined step costs more than either step alone (Q2).

Q1. Which tokens does each method draft?

At every decoding step of the EAGLE-3 + VTD runs, we recorded which source drafted the accepted path. The table gives each source's share of accepted draft tokens. Ties are credited to EAGLE-3, so VTD's shares are lower bounds.

SourceWebCode2MDesign2CodeOmniDocBenchOCR-Md-DenseOmniAI-OCR
EAGLE-319%18%35%31%37%
VTD (dynamic + OCR trees)81%82%65%69%63%
OCR tree alone30%16%31%46%28%
Output tokens drafted by VTD54%52%37%48%45%
  • VTD takes the copied tokens. The OCR tree proposes only continuations of text read from the image, and the dynamic tree only continuations of strings already in the prompt or output. Every VTD token is therefore a verbatim copy. Even with ties going to EAGLE-3, VTD drafts 63–82% of the accepted draft tokens and 37–54% of all output tokens.
  • EAGLE-3 takes the predicted tokens. It drafts from the target's hidden states and is not limited to copies. Its share rises to 31–37% on the document benchmarks, where its solo MAT is also higher (1.84–2.17 vs. 1.41–1.51 on webpages).

Q2. Why does EAGLE-3 do poorly here, and why are the two complementary?

  1. The outputs are predictable, and a trained VLM drafter does predict them. ViSpec reaches the highest MAT on Qwen2.5-VL-7B in Table 1 (2.73, vs. 2.46 for VTD). We attribute the public EAGLE-3 checkpoint's lower MAT (1.41–2.17) to its training data, 95K general-purpose ALLaVA-4V caption and instruction samples, not to model-based drafting itself.
  2. The bottleneck is the cost per step. On our GPU, an EAGLE-3 step takes 1.58–1.67× as long as an autoregressive step, and a VTD step 1.23–1.42×, because VTD drafts with a suffix-automaton lookup instead of three draft-model passes and verifies a smaller tree (7.7–14.2 nodes vs. 30; Q3 of our first reply). ViSpec's draft model also runs at every step, so its top MAT yields 1.44× against VTD's 1.97×. Step cost also explains the combination: it is on par with VTD on the three document benchmarks (1.54× vs. 1.56×, 1.82× vs. 1.78×, 2.18× vs. 2.17×), where EAGLE-3 adds the most tokens, and slower on the two webpage benchmarks.
  3. The VLM setting makes copying unusually effective. Much of the output transcribes text visible in the image or repeats strings written earlier. History alone cannot supply the image text: at steps that accept no draft token, only 2.3% of the next three-token spans had appeared earlier in the output (§3.2). The OCR tree fills this gap and supplies 16–46% of the accepted draft tokens in the combination. A learned drafter must predict these spans token by token; a copy source drafts them at once and needs no training.

On feasibility. Without any training, VTD has the highest speedup in all 15 model–benchmark cells of Table 1 (three VLMs, five benchmarks). Added to a trained drafter, it raises MAT by 0.92–1.47 tokens per step on every benchmark. It can therefore be added to an existing drafter without retraining.

We thank the reviewer again for the careful reading and the thoughtful follow-up, which helped us explain more clearly where VTD's gains come from.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.