- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Visible-Text Drafting (VTD), a training-free speculative decoding method designed to accelerate Vision-Language Model (VLM) inference on image-to-markup transcription tasks such as webpage-to-HTML and document-to-Markdown generation. The authors argue that conventional retrieval-based speculative drafters, which rely exclusively on prompt context or past generated text, fail to predict content that is visible in the input image prior to its first emission. To overcome this, VTD adopts a dual-path retrieval strategy: it runs an off-the-shelf OCR model once on the input image to build a fixed visible-text suffix automaton, alongside a dynamic suffix automaton that continuously indexes the generated sequence history. At each decoding step, both automata propose candidate continuation trees rooted at the target model's previous greedy token. These two trees are merged into a single verified tree and scored by the target VLM in a single forward pass with a tree-attention mask. Across five benchmarks and three target models from the Qwen-VL and InternVL families, the paper reports end-to-end speedups ranging between 1.62x and 1.97x, outperforming both training-free retrieval baselines and the trained ViSpec drafter.
Critical strengths
- The paper provides a thorough empirical diagnosis of generation misses in multimodal transcription - only 2.3% of 3-token miss spans have appeared in the prior output sequence - convincingly exposing the fundamental recall ceiling of history-only retrieval drafters in multimodal tasks.
- The authors include the serial execution latency of the OCR inference pass in their primary end-to-end wall-clock throughput and speedup measurements.
- VTD requires no auxiliary training of draft models or specialized prediction heads, allowing it to be deployed across different open-source VLM architectures without target-specific fine-tuning.
- The target-side tree verification strictly preserves the target model's greedy autoregressive output distribution, ensuring no degradation in transcription fidelity.
Critical weaknesses
-
By reintroducing an external OCR model into the inference pipeline, VTD creates a conceptual paradox: If a lightweight, highly accurate OCR tool is already available in the system, why use an expensive billion-parameter VLM to autoregressively decode document content in the first place? A conventional two-stage pipeline (OCR + text LLM) is typically faster and less resource-intensive.
-
Table 3 demonstrates that when switching from EasyOCR to a modern, more accurate OCR engine (PaddleOCR / PP-OCRv5), the Mean Accepted Tokens (MAT) improves, but wall-clock speedup drops dramatically across all five benchmarks, collapsing to 0.97x on OCR-Md-Dense (a net slowdown). This reveals that VTD’s wall-clock advantage is extremely fragile: the entire acceleration benefit exists only within a narrow window where the OCR tool is small enough, the image is simple enough, and decoding is long enough.
-
Section 4.3 admits that tokenization and suffix-automaton construction of the OCR text are excluded from the reported latency. Furthermore, speculative decoding in production serving systems (e.g., vLLM, SGLang) relies heavily on batched operations and continuous scheduling. Integrating an unbatched, external Python/PyTorch OCR pipeline into the request critical path introduces substantial memory fragmentation, CUDA context overhead, and pipeline bubbles that cannot easily scale to batched multi-tenant environments.
-
Section 4.2 motivates tree fusion by asserting that selecting between the two sources would discard viable branches. Yet, the paper never benchmarks a simple "Source Selection / Switching" baseline (e.g., choose whichever automaton has a longer suffix match, or prioritize OCR when match length exceeds a threshold). Without this ablation, it is impossible to determine whether the complex tree-merging and tree-attention masking add genuine algorithmic value, or if a trivial switching heuristic would achieve the same speedup with lower verification width.
-
Table 2 presents Dynamic-only, Dynamic + Static Corpus, and Dynamic + Visible Text, but omits Visible-only (drafting solely from the OCR automaton). Figure 3(c) shows that on dense documents, the visible source provides up to 69% of accepted tokens while the dynamic source is negligible. If the visible source alone provides almost all the speedup, the dynamic sequence automaton (and the dual-path machinery) may be largely redundant.
-
SAM-D is evaluated with a draft budget of 40 nodes. In contrast, VTD merges two 40-node trees, yielding a combined tree of up to 79 candidate nodes. It remains unproven whether VTD’s higher acceptance rate is driven by its fusion strategy or simply by doubling the target model's speculative candidate budget per forward pass.
-
As documented in Table 5, VTD yields negligible acceleration on benchmarks where output is not directly transcribed from visible text (e.g., ChartMimic MAT gain +0.07; Chart2Code MAT gain +0.09). Consequently, VTD is not a general-purpose speculative decoding method for VLMs, but rather a domain-specific speedup trick restricted strictly to literal image transcription. This narrows the scope of the work's contribution.
Questions for authors
-
In Section 4.2, tree merging is motivated by avoiding the loss of valid branches from either source, but Figure 4(a) shows that evaluating the merged tree increases per-step verification latency by up to 15.0%. What are the Mean Accepted Tokens (MAT), throughput, and wall-clock speedup when using a simple single-source selection heuristic (e.g., dynamically choosing either the OCR or dynamic branch based on match length or a threshold) instead of tree merging? Does the modest acceptance gain of tree fusion justify the extra tree-attention verification overhead over greedy source selection?
-
The baseline SAM-D is restricted to a draft budget of nodes, whereas VTD verifies a merged tree containing up to candidate nodes. What is the performance when: (1) SAM-D is allowed an equal budget of nodes on its dynamic sequence index? (2) VTD is strictly constrained to a total combined budget of nodes (e.g., dynamic and visible sources split the 40-node ceiling)? Additionally, is VTD's acceptance gain an algorithmic achievement of dual-source retrieval, or largely an artifact of doubling the target model's speculative candidate pool per forward pass?
-
Table 2 reports Dynamic-only and Dynamic + Visible, but omits Visible-only - drafting solely from the fixed OCR automaton . Given that visible-text drafting accounts for up to 69% of accepted tokens on dense document benchmarks (Figure 3c), what is the standalone performance of across all benchmarks? If visible-only drafting achieves comparable speedups on document tasks, does maintaining the dynamic automaton - which requires online graph expansion after every accepted token—provide sufficient marginal utility to justify its algorithmic and memory complexity?
-
Section 4.3 states that tokenization of the OCR text and construction of the visible suffix automaton are excluded from reported latency. Furthermore, Table 3 shows that replacing EasyOCR with PaddleOCR (PP-OCRv5) flips the method into a net slowdown ( on OCR-Md-Dense). What is the exact wall-clock overhead (in milliseconds) of tokenizing the OCR output and constructing \mathcal{A}_{vis}, and how does including these steps affect the end-to-end speedups in Table 1? Analytically or empirically, what is the maximum permissible OCR latency threshold () as a function of output sequence length, beyond which VTD fails to provide any wall-clock acceleration?
-
If a deployment pipeline is already willing to pay the computational and latency cost of running an external OCR pass prior to generation, how does VTD compare against a standard OCR-augmented prompting baseline (i.e., concatenating the OCR-extracted text directly into the prompt context of the VLM or a faster text-only LLM)? Does VTD offer higher wall-clock throughput, lower peak memory, or better formatting fidelity than simply providing the OCR text upfront?
5 comments
[2/3]
W1c. Serving systems
We agree that we evaluate batch size 1 in an HF-based harness, as is common for lossless speculative decoding, and we do not claim results for large batches; the revision will state this limitation explicitly. Two properties suggest the cost is manageable in serving:
- OCR runs once per request, outside the decode loop. It can be batched across requests with off-the-shelf tooling (EasyOCR exposes batched recognition), and it needs only the image, so it can overlap with image encoding or with other requests' decoding.
- The rest of VTD is a CPU-side suffix-automaton lookup, the same kind of drafter vLLM already ships for n-gram and suffix speculation. It needs no extra GPU model and no extra KV cache.
As with all speculative decoding, gains shrink as batch size grows and decoding becomes compute-bound.
W2 / Q1–Q3. Fusion vs. selection, equal budget, visible-only
We ran all three ablations on all five benchmarks with Table 1's protocol. All rows share one page set per benchmark (pages where all six configurations stop at a natural end of sequence), so the two reference rows can differ slightly from Table 1. Cells are MAT / speedup; the OCR pass is included wherever it is used.
| Configuration | WebCode2M | Design2Code | OmniDocBench | OCR-Md-Dense | OmniAI-OCR |
|---|---|---|---|---|---|
| SAM-D (dynamic only, N = 40) | 1.72 / 1.60× | 1.82 / 1.74× | 1.36 / 1.31× | 1.32 / 1.25× | 1.67 / 1.56× |
| SAM-D, N = 80 | 1.75 / 1.62× | 1.82 / 1.75× | 1.37 / 1.33× | 1.32 / 1.24× | 1.67 / 1.56× |
| Visible-only (OCR index alone) | 1.39 / 1.26× | 1.22 / 1.15× | 1.61 / 1.20× | 1.93 / 1.44× | 1.86 / 1.60× |
| Selection (longer match wins) | 2.49 / 2.19× | 2.35 / 2.13× | 1.97 / 1.49× | 2.53 / 1.74× | 2.57 / 2.08× |
| VTD, merged tree capped at 40 | 2.53 / 2.21× | 2.37 / 2.14× | 2.03 / 1.52× | 2.60 / 1.77× | 2.70 / 2.17× |
| VTD (fusion, default) | 2.53 / 2.22× | 2.37 / 2.14× | 2.03 / 1.52× | 2.60 / 1.78× | 2.70 / 2.17× |
- Equal budget (Q2). Doubling SAM-D's budget to N = 80 changes its MAT by at most 0.03. VTD capped at 40 nodes keeps its full gain and, with half the budget, is 1.14–1.43× faster than SAM-D at N = 80. The gain does not come from verifying more nodes.
- Visible-only (Q3). The OCR index alone beats the dynamic index alone on the three document benchmarks (OCR-Md-Dense 1.93 vs. 1.32 MAT) and loses on the two webpage benchmarks (Design2Code 1.22 vs. 1.82), where markup repeats within the output. Neither source alone reaches fusion on any benchmark, so both are needed.
- Fusion vs. selection (Q1). Selection captures most of the gain, which confirms that the main contribution is the visible-text source. Fusion is at least as good on all five benchmarks, by 0.02–0.13 MAT (largest on OmniAI-OCR, 2.70 vs. 2.57), at no extra drafting cost.
Two further results point the same way:
- The merged tree is far below its bound. The bound is 2N − 1 = 79 nodes, but the merged tree averages 8.3–14.0 nodes per step (p95 25–40), because each source's budget scales with its match length (Eq. 1).
- Extra nodes alone give no speedup. In the wrong-language control (Table 9), the English reader's text does not match Arabic pages. The tree is built exactly as in VTD and MAT rises from 1.32 to 1.51, yet speedup is unchanged (1.283× → 1.284×).
On the marginal value of the dynamic index (Q3): it supplies 31–66% of drafted tokens (Fig. 3c), most on webpages, at one amortized O(1) CPU automaton extension per accepted token. Appendix B.2 measures fusion's whole extra verification work at 3–15% per step.
[3/3]
W3. Applicability boundary
We agree that VTD is not a general-purpose VLM accelerator, and the paper says so (Appendix B.4). It targets long outputs that transcribe visible text, which covers webpage-to-code and document-to-Markdown, both common deployed uses of VLMs in which decoding dominates latency. Domain-targeted, training-free drafters are an established design point; SuffixDecoding, for example, targets repetitive agentic workloads. Where image text is not transcribed (charts), VTD costs little and speedup still rises slightly: ChartMimic 1.477× → 1.508×, Chart2Code 1.848× → 1.890× (Table 5). Users therefore do not need to detect the task in advance.
Q5. OCR-augmented prompting
We ran this baseline on Qwen2.5-VL-7B with the same EasyOCR text VTD uses, appended to the instruction, on the same pages. The OCR pass is charged to every row; speedup is over AR on the original prompt.
| WebCode2M (mid) | Design2Code | OmniDocBench | OCR-Md-Dense | OmniAI-OCR | |
|---|---|---|---|---|---|
| AR, OCR text in prompt | 0.94× | 0.97× | 0.89× | 0.91× | 0.90× |
| SAM-D, OCR text in prompt | 2.14× | 2.06× | 1.44× | 1.73× | 2.00× |
| VTD | 2.23× | 2.12× | 1.57× | 1.79× | 2.19× |
| Outputs changed by the OCR prompt | 98% | 100% | 94% | 82% | 98% |
- Throughput. The OCR text adds 213–669 prompt tokens (prefill +0.03–0.10 s) and slows AR itself. Even with SAM-D retrieving from that text, the method stays below VTD on every benchmark.
- Peak memory. It grows by 0.1–1.8 GB with the longer prompt. VTD adds no prompt tokens.
- Fidelity. The OCR prompt changes 82–100% of outputs. Character edit distance to the ground truth rises on four of five benchmarks (e.g., OCR-Md-Dense 0.137 → 0.160) and is unchanged on OmniAI-OCR. On Design2Code, CLIP similarity of the rendered page to the reference moves by −0.013 (95% CI −0.039 to +0.014): pages change with no consistent gain. VTD's rendered pages are identical to the original on 56 of 67 pages; the rest differ only where AR's top two logits are tied at fp16 precision, which any change of kernel or batch shape can flip.
The two approaches also differ in kind. VTD leaves the model's input unchanged and preserves its greedy output up to fp16 ties. OCR prompting changes the input, so it needs its own quality evaluation. A text-only LLM fed OCR text cannot recover layout, styling or tables from the image, so it is not an alternative way of solving image-to-markup.
[1/3] We thank the reviewer for recognizing our miss diagnosis (only 2.3% of 3-token miss spans recur in the prior output), the end-to-end accounting of OCR latency, and VTD's training-free portability. We ran every requested experiment on all five benchmarks with Table 1's protocol (Qwen2.5-VL-7B, same pages and AR baseline) and will add all new results to the revised manuscript. This comment covers W1a and W1b/Q4; [2/3] covers W1c and W2/Q1–Q3; [3/3] covers W3 and Q5.
W1a. Why run a VLM if OCR is available?
We agree that a two-stage pipeline (OCR + text-only LLM) can be cheaper. VTD does not compete with it, because the two produce different outputs. VTD never outputs OCR text. It uses it only as a source of draft guesses, and the VLM writes and verifies every token.
W1b / Q4. OCR latency and excluded overheads
Headroom is 4.7–26× on Qwen2.5-VL-3B and 3.3–26× on InternVL2.5-8B (tightest: InternVL on OmniDocBench, 484-token mean output). OCR costs more than it saves on 14 of 475 Qwen2.5-VL-7B pages, all with outputs of at most 407 tokens (median page: 773).
On the GPU, PaddleOCR keeps its MAT and beats EasyOCR's serial speedup on all five benchmarks, by 7–37%. The 0.97× in Table 3 therefore came from the CPU, and on a GPU the more accurate reader is also the faster choice. We will add the GPU row to Table 3, and we have rewritten §5.3 so that the comparison no longer reads as an engine ranking.