- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Length-Agnostic Sparse Attention (LASA) for continuous parsing of long documents. It combines page-constrained visual access and Page-RoPE with a router that selects a fixed budget of visual blocks and passes routing probabilities to token-level attention. The parser is adapted from UnlimitedOCR. The paper also introduces MarathonDocBench, containing 100 documents of 103-300 pages. Experiments on this benchmark and MPDocBench show improved overall parsing scores, gains on cross-page text and tables, and nearly constant per-step decoding latency as input length increases. A sliding-window strategy bounds the active KV cache for longer inputs.
Critical strengths
S1. LASA improves the cross-page text and table parsing that motivates the work. It achieves SOTA performance on MPDocBench and MarathonDocBench, compared with PaddleOCR-VL-1.6 (Tables 1 and 2). On MarathonDocBench, cross-page text edit distance and cross-page table TEDS also improves. MarathonDocBench extends evaluation to hundred-page documents with more cross-page text and table continuations per document than MPDocBench.
S2. The controlled ablations show gains beyond continued training. LASA outperforms the continued-training baseline with full visual access, while the proposed router improves over MSA routing with page constraints and Page-RoPE held fixed, at the same reported visual access fraction (Table 3). LASA also substantially reduces per-step decoding latency compared with full attention and R-SWA under the same input and output settings (Figure 5).
Critical weaknesses
W1. The additional value of learned sparse routing over a simpler page-restricted design is not fully isolated. Table 3 lacks a baseline that attends to all page-compatible visual blocks using Page-RoPE and the same local attention window, without a sparse router. With bounded tokens per page, this baseline should also have a visual attention scope independent of total page count. Comparing its accuracy and latency would clarify the additional benefit of Token Bounding. The final model also differs from the original UnlimitedOCR setup in training scale, context length, and visual representations. Although the controlled ablations demonstrate an architectural gain, the full improvement over the original checkpoint cannot be attributed solely to LASA.
W2. The efficiency and cross-page evaluations leave aspects of full-document processing unclear. Figure 5 uses a fixed output length and therefore does not measure complete transcription of long documents. The stages included in its end-to-end throughput are also unspecified. Since the attention mechanism restricts direct remote visual access to blocks overlapping the current page, the paper should explain how information from other pages reaches the current query. The reported gains in cross-page continuity do not establish how performance varies with the span of cross-page structures or with streaming boundaries. Table 4 also omits the dataset, window width, and stride used for its quality comparison.
Questions for authors
Q1. Under the same 200K-sample, 32K-context ablation setting, how does LASA compare with attention to all page-compatible visual blocks in Equation (3), using Page-RoPE and the same local attention window, without a sparse router? Please report accuracy and latency under matched inference settings. If feasible, please also report a continued-training R-SWA baseline under the final 400K / 64K / Gundam configuration.
Q2. Which stages are included in the E2E throughput in Figure 5? For complete transcription of a small representative subset of hundred-page documents, could you report total processing time, output token counts, completion rates, and peak GPU memory under matched hardware? If available, a breakdown of visual encoding, prefill, and decoding would help explain the full-task speedup.
Q3. Section 3.1 uses <PAGE> tokens to establish page associations. How is g(q) updated during inference, and how does cross-page information reach a query under the direct remote visual access restriction in Equation (3)? Please specify the dataset, window width W, and stride S used in Table 4, and clarify whether its quality evaluation includes structures crossing streaming boundaries.
Rebuttal
Thank you for the thoughtful review and for recognizing the merits of our work.
1. Isolating the architectural contribution of LASA (W1 & Q1)
A1: Thank you for pointing out the potential confounding of data distillation and architectural improvements. The preliminary ablations used only 200K samples, so we ran controlled comparisons under the final setting (400K samples, 64K context, Gundam representation), now added as Appendix C of the revision.
On MPDocBench, LASA achieves 83.13 against 79.36 for the retrained R-SWA baseline (+3.77), the same conclusion as under the 200K ablation configuration (Table 3).
Page-Restricted Dense Attention keeps LASA's Page Canonicalization but attends to all visual blocks in each page instead of the Top-K blocks. LASA achieves comparable accuracy (83.13 vs. 84.12) while attending to only 384 of the 1,913 visual tokens a page carries under the 4 × 4 Gundam configuration used for timing, i.e. 79.9% fewer per decoding step. Since the attended blocks are read once per query, the saving grows with concurrency: measured under the protocol of Appendix A.5, it brings little at batch size 3, where weight loading and kernel launches dominate each step, but reaches 3,683 against 3,090 tokens/s at batch size 32 (+19%), the regime in which a document parser is typically served (Table 7). The fixed budget also makes the per-step cost independent of how many visual tokens a page carries, whereas the dense variant scales with it.
2. Details of the speed evaluation in Figure 5 (W2 & Q2)
A2: All methods use identical inputs and an output budget of 1,024 tokens per page (N × 1,024 for an N-page document), using batch size 3 and greedy decoding, with Sliding-Window Streaming disabled for LASA. All methods complete 100% of requests within the 80 GB memory limit, with comparable peak GPU memory usage.
For a 100-page document (102,400 output tokens), LASA completes in 402.4 s, compared with 2,316.6 s for R-SWA and 2,824.0 s for Full Attention. E2E time covers queueing, image preprocessing, prefill (which includes visual encoding) and decoding (full breakdown in Appendix A.5).
3. Details of Sliding-Window Streaming (W2 & Q3)
A3: Sliding-Window Streaming enables long-document inference under bounded GPU memory. When the window advances, our implementation retains the context a cross-page query needs, so the field each query can attend to is the same as under single-pass decoding; Algorithm 1 in Appendix B spells this out.
Table 5 in the revision (Table 4 in the submission) reports our setting, W = 1 with stride 1, the most demanding configuration for streaming. Each window then holds a single page, so every page break is also a window boundary and every cross-page structure in the evaluation has to be carried across one. Parsing quality is nonetheless unchanged (83.13 vs. 83.18 on MPDocBench, 71.80 vs. 71.73 on MarathonDocBench).
Because peak visual KV cache is bounded by W rather than document length D (Figure 7), far more requests stay resident at the same concurrency: at concurrency 32 on MarathonDocBench, streaming keeps all 32 resident at 23% peak KV usage and reaches 3.933 pages/s against 1.044 for full-context decoding, a 3.77× gap (Table 6).
4. Performance vs. cross-page span (W2)
A4: We grouped MarathonDocBench tables by span, excluding matching errors.
Table. T-TableTEDS by cross-page span on MarathonDocBench.
Both degrade as span grows, including the pipeline with multi-round refinement by a proprietary LLM, which suggests the difficulty comes largely from the task.
5. Page assignment and cross-page information flow under Eq. (3) (W2 & Q3)
A5: During inference, g(q) is the running count of emitted
<PAGE>tokens, so its value identifies the page of each query without any additional mechanism.Eq. (3) does not imply strict page-level visual isolation: since visual block boundaries are independent of page boundaries, selected blocks may straddle adjacent pages, letting the query access some visual information from preceding pages. The local text window additionally preserves previously decoded text, giving both visual and textual cross-page context. Appendix D gives a visualized example, including how g(q) is updated.
We sincerely thank you again for your thoughtful feedback, which has helped us separate the architectural contribution from the training configuration and state our speed results more precisely. We are glad to clarify any remaining point, and would be very grateful if you could reconsider your assessment.