acceptodds
Under review as a conference paper at ICLR 2027

Length-Agnostic Sparse Attention for Unlimited Document Parsing

Abstract

Real-world applications such as financial analysis and contract review often involve documents spanning tens or hundreds of pages, making ultra-long document parsing increasingly important. Existing solutions face a fundamental trade-off between scalability and cross-page modeling, as page-by-page parsing loses cross-page dependencies while full-document modeling faces both prohibitive computation and an overwhelming visual context. Fixed-budget sparse attention provides a natural way to bound the context accessed by each query, but reliably selecting the few relevant blocks from an expanding multi-page context remains challenging. Crucially, we observe that document evidence exhibits a hierarchical sparsity pattern, with relevant evidence sparse across pages but locally concentrated within relevant pages. Leveraging this structure, we propose Length-Agnostic Sparse Attention (LASA), which enables reliable fixed-budget routing with an attention scope independent of document length, providing a scalable foundation for unlimited document parsing. LASA combines Page Canonicalization to exploit page structure, constrain the visual search space, and stabilize page-level positional modeling, with Token Bounding to reliably select a fixed budget of query-relevant visual blocks. Together, these designs enable efficient end-to-end cross-page modeling with stable performance as document length increases. We further introduce MarathonDocBench, a benchmark for hundred-page document parsing with substantially longer documents and richer cross-page structures than existing benchmarks. The resulting LASA-based parser consistently outperforms strong page-by-page pipelines and general-purpose large-scale VLMs on both MPDocBench and MarathonDocBench. By bounding the effective attention scope of each query, LASA keeps per-step decoding time approximately constant as document length increases and achieves up to speedup over full attention at the hundred-page scale.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
long context modeling, sparse attention, document parsing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9J3vOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Length-Agnostic Sparse Attention (LASA) for continuous parsing of long documents. It combines page-constrained visual access and Page-RoPE with a router that selects a fixed budget of visual blocks and passes routing probabilities to token-level attention. The parser is adapted from UnlimitedOCR. The paper also introduces MarathonDocBench, containing 100 documents of 103-300 pages. Experiments on this benchmark and MPDocBench show improved overall parsing scores, gains on cross-page text and tables, and nearly constant per-step decoding latency as input length increases. A sliding-window strategy bounds the active KV cache for longer inputs.

Critical strengths

S1. LASA improves the cross-page text and table parsing that motivates the work. It achieves SOTA performance on MPDocBench and MarathonDocBench, compared with PaddleOCR-VL-1.6 (Tables 1 and 2). On MarathonDocBench, cross-page text edit distance and cross-page table TEDS also improves. MarathonDocBench extends evaluation to hundred-page documents with more cross-page text and table continuations per document than MPDocBench.

S2. The controlled ablations show gains beyond continued training. LASA outperforms the continued-training baseline with full visual access, while the proposed router improves over MSA routing with page constraints and Page-RoPE held fixed, at the same reported visual access fraction (Table 3). LASA also substantially reduces per-step decoding latency compared with full attention and R-SWA under the same input and output settings (Figure 5).

Critical weaknesses

W1. The additional value of learned sparse routing over a simpler page-restricted design is not fully isolated. Table 3 lacks a baseline that attends to all page-compatible visual blocks using Page-RoPE and the same local attention window, without a sparse router. With bounded tokens per page, this baseline should also have a visual attention scope independent of total page count. Comparing its accuracy and latency would clarify the additional benefit of Token Bounding. The final model also differs from the original UnlimitedOCR setup in training scale, context length, and visual representations. Although the controlled ablations demonstrate an architectural gain, the full improvement over the original checkpoint cannot be attributed solely to LASA.

W2. The efficiency and cross-page evaluations leave aspects of full-document processing unclear. Figure 5 uses a fixed output length and therefore does not measure complete transcription of long documents. The stages included in its end-to-end throughput are also unspecified. Since the attention mechanism restricts direct remote visual access to blocks overlapping the current page, the paper should explain how information from other pages reaches the current query. The reported gains in cross-page continuity do not establish how performance varies with the span of cross-page structures or with streaming boundaries. Table 4 also omits the dataset, window width, and stride used for its quality comparison.

Questions for authors

Q1. Under the same 200K-sample, 32K-context ablation setting, how does LASA compare with attention to all page-compatible visual blocks in Equation (3), using Page-RoPE and the same local attention window, without a sparse router? Please report accuracy and latency under matched inference settings. If feasible, please also report a continued-training R-SWA baseline under the final 400K / 64K / Gundam configuration.

Q2. Which stages are included in the E2E throughput in Figure 5? For complete transcription of a small representative subset of hundred-page documents, could you report total processing time, output token counts, completion rates, and peak GPU memory under matched hardware? If available, a breakdown of visual encoding, prefill, and decoding would help explain the full-task speedup.

Q3. Section 3.1 uses <PAGE> tokens to establish page associations. How is g(q) updated during inference, and how does cross-page information reach a query under the direct remote visual access restriction in Equation (3)? Please specify the dataset, window width W, and stride S used in Table 4, and clarify whether its quality evaluation includes structures crossing streaming boundaries.

1 comment

Thank you for the thoughtful review and for recognizing the merits of our work.

1. Isolating the architectural contribution of LASA (W1 & Q1)

A1: Thank you for pointing out the potential confounding of data distillation and architectural improvements. The preliminary ablations used only 200K samples, so we ran controlled comparisons under the final setting (400K samples, 64K context, Gundam representation), now added as Appendix C of the revision.

"a continued-training R-SWA baseline under the final ... configuration" (Q1)

On MPDocBench, LASA achieves 83.13 against 79.36 for the retrained R-SWA baseline (+3.77), the same conclusion as under the 200K ablation configuration (Table 3).

"attention to all page-compatible visual blocks ... without a sparse router" (Q1)

Page-Restricted Dense Attention keeps LASA's Page Canonicalization but attends to all visual blocks in each page instead of the Top-K blocks. LASA achieves comparable accuracy (83.13 vs. 84.12) while attending to only 384 of the 1,913 visual tokens a page carries under the 4 × 4 Gundam configuration used for timing, i.e. 79.9% fewer per decoding step. Since the attended blocks are read once per query, the saving grows with concurrency: measured under the protocol of Appendix A.5, it brings little at batch size 3, where weight loading and kernel launches dominate each step, but reaches 3,683 against 3,090 tokens/s at batch size 32 (+19%), the regime in which a document parser is typically served (Table 7). The fixed budget also makes the per-step cost independent of how many visual tokens a page carries, whereas the dense variant scales with it.

2. Details of the speed evaluation in Figure 5 (W2 & Q2)

A2: All methods use identical inputs and an output budget of 1,024 tokens per page (N × 1,024 for an N-page document), using batch size 3 and greedy decoding, with Sliding-Window Streaming disabled for LASA. All methods complete 100% of requests within the 80 GB memory limit, with comparable peak GPU memory usage.

For a 100-page document (102,400 output tokens), LASA completes in 402.4 s, compared with 2,316.6 s for R-SWA and 2,824.0 s for Full Attention. E2E time covers queueing, image preprocessing, prefill (which includes visual encoding) and decoding (full breakdown in Appendix A.5).

3. Details of Sliding-Window Streaming (W2 & Q3)

A3: Sliding-Window Streaming enables long-document inference under bounded GPU memory. When the window advances, our implementation retains the context a cross-page query needs, so the field each query can attend to is the same as under single-pass decoding; Algorithm 1 in Appendix B spells this out.

Table 5 in the revision (Table 4 in the submission) reports our setting, W = 1 with stride 1, the most demanding configuration for streaming. Each window then holds a single page, so every page break is also a window boundary and every cross-page structure in the evaluation has to be carried across one. Parsing quality is nonetheless unchanged (83.13 vs. 83.18 on MPDocBench, 71.80 vs. 71.73 on MarathonDocBench).

Because peak visual KV cache is bounded by W rather than document length D (Figure 7), far more requests stay resident at the same concurrency: at concurrency 32 on MarathonDocBench, streaming keeps all 32 resident at 23% peak KV usage and reaches 3.933 pages/s against 1.044 for full-context decoding, a 3.77× gap (Table 6).

4. Performance vs. cross-page span (W2)

A4: We grouped MarathonDocBench tables by span, excluding matching errors.

Table. T-TableTEDS by cross-page span on MarathonDocBench.

Span (pages)2345–7≥8
# Tables910159635627
Pseudo-labeling pipeline0.8300.8490.7810.6150.463
Ours0.7220.7170.5970.5460.475

Both degrade as span grows, including the pipeline with multi-round refinement by a proprietary LLM, which suggests the difficulty comes largely from the task.

5. Page assignment and cross-page information flow under Eq. (3) (W2 & Q3)

A5: During inference, g(q) is the running count of emitted <PAGE> tokens, so its value identifies the page of each query without any additional mechanism.

Eq. (3) does not imply strict page-level visual isolation: since visual block boundaries are independent of page boundaries, selected blocks may straddle adjacent pages, letting the query access some visual information from preceding pages. The local text window additionally preserves previously decoded text, giving both visual and textual cross-page context. Appendix D gives a visualized example, including how g(q) is updated.

We sincerely thank you again for your thoughtful feedback, which has helped us separate the architectural contribution from the training configuration and state our speed results more precisely. We are glad to clarify any remaining point, and would be very grateful if you could reconsider your assessment.

Reviewer_mBxZOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LASA for long-document parsing, combining page-constrained visual access, Page-RoPE, and a learned router with a fixed visual-token budget. It builds on UnlimitedOCR and introduces MarathonDocBench, containing 100 documents of 103–300 pages. The parser achieves the best reported overall scores on both benchmarks and nearly constant per-step decoding latency, with up to an 8.2× speedup over full attention in the evaluated setting.

Critical strengths
  1. Using page structure to constrain visual attention is a sensible design. At 100 pages, Figure 5 reports 5.8 ms per decoding step, compared with 47.8 ms for full attention. Table 3 also shows a 3.61-point gain over MSA with the same page constraints and Page-RoPE, supporting the value of the proposed router.

  2. MarathonDocBench extends evaluation to documents with hundreds of pages and richer cross-page structures. LASA reaches an overall score of 71.80, compared with 69.47 for the reported PaddleOCR-VL-1.6 baseline, with gains in cross-page text and table metrics. The benchmark and these results make the paper a useful contribution to long-document parsing.

Critical weaknesses
  1. The practical comparison with strong page-wise pipelines is still incomplete. Appendix A.1 uses PaddleOCR-VL-1.6 + MinerU-Popo + supplementary merging for training labels, but its evaluation results are not clearly identified. This leaves some uncertainty about LASA’s advantage over parallel page-wise parsing with capable cross-page post-processing.

  2. Some experimental details are missing. Baseline input settings, output limits, and failure handling are not sufficiently described, making the long-document results harder to interpret. The timing study also uses only 1,024 output tokens, so the cost of parsing complete documents remains unclear.

  3. Table 3 supports the gain over MSA, but the benefit beyond page restriction is less clear. With bounded per-page input and a fixed local text window, dense attention within the current page already has a scope independent of total page count. A comparison with this simpler variant, and clearer positioning relative to DashAttention and HiLS, would help establish the router’s contribution.

Questions for authors
  1. Which results in Tables 1–2, if any, include the PaddleOCR-VL-1.6 + MinerU-Popo (Xu et al., 2026) + supplementary merging pipeline from Appendix A.1? For the general VLMs, were full-document outputs obtained, and how were truncated or failed responses scored?

  2. Could you walk through how LASA handles a cross-page table? In particular, how is g(q) updated, and how does information from earlier pages reach the current prediction given the visual attention mask and local text window?

  3. Do you have a comparison that keeps page constraints, Page-RoPE, and training settings fixed, but replaces sparse routing with dense within-page attention? Accuracy and latency on both short and hundred-page documents would be informative.

  4. What stages are included in the reported E2E throughput, and which experiments use streaming? Please clarify Table 4’s dataset, window, and stride, and share full-document timing if available.

3 comments

1. Page-wise pipeline vs. LASA; failed/truncated outputs of general VLMs (W1 & Q1)

A1: Thank you for these important questions.

"LASA's advantage over parallel page-wise parsing" (W1)

Following your suggestion, we evaluated the Appendix A.1 pipeline on MPDocBench. Its annotation stage (PaddleOCR-VL-1.6 + MinerU-Popo) scores 80.99, and merging plus multiple rounds of refinement by a closed-source LLM (GPT-6-Sol) raises it to 85.44. Applying this to long documents needs repeated LLM calls, adding cost and latency. LASA instead reaches comparable accuracy (83.13) end-to-end without such an LLM.

"For the general VLMs, were full-document outputs obtained, and how were truncated or failed responses scored?" (Q1)

Full-document outputs were requested in a single pass. For requests whose input exceeds a model's context length, we assign a score of 0, since no output covers the complete input. Responses truncated by output-length limits are evaluated normally on their partial outputs.

2. Details of the speed evaluation in Figure 5 (W2 & Q4)

A2: The output budget is 1,024 tokens per page (N × 1,024 for an N-page document), so the reported E2E time covers parsing the complete document. All methods are evaluated with identical inputs and output budgets at batch size 3, with Sliding-Window Streaming disabled for LASA. E2E time covers queueing, image preprocessing, prefill and decoding.

For a 100-page document (102,400 output tokens), LASA completes in 402.4 s, compared with 2,316.6 s for R-SWA and 2,824.0 s for Full Attention. The full setup and per-stage breakdown are in Appendix A.5.

3. Details of Sliding-Window Streaming (W2 & Q3)

A3: Sliding-Window Streaming enables long-document inference under bounded GPU memory: it keeps the same effective context as single-pass decoding across window boundaries, and peak visual KV cache scales with the window width W instead of the document length D. Algorithm 1 is expanded in Appendix B.

In the extreme setting we use, W = 1 with stride 1, parsing quality is unchanged (83.13 vs. 83.18 on MPDocBench, 71.80 vs. 71.73 on MarathonDocBench; Table 5 in the revision, Table 4 in the submission). At concurrency 32 on MarathonDocBench, streaming reaches 3.933 pages/s against 1.044 for full-context decoding (3.77×) while occupying at most 23% of the KV cache (Table 6).

4. Contribution of sparse routing and positioning vs. DashAttention and HiLS (W3 & Q3)

"replaces sparse routing with dense within-page attention" (Q3)

Page-Restricted Dense Attention keeps LASA's Page Canonicalization but attends to all visual blocks in each page instead of the Top-K blocks. It achieves 84.12 / 72.69 on MPDocBench / MarathonDocBench against 83.13 / 71.80 for LASA, while LASA attends to only 384 of the 1,913 visual tokens a page carries under the 4 × 4 Gundam timing configuration, i.e. 79.9% fewer per decoding step. Measured under the protocol of Appendix A.5, this brings little at batch size 3, where weight loading and kernel launches dominate each step, but grows with concurrency, reaching 3,683 against 3,090 tokens/s at batch size 32 (+19%), the regime in which a document parser is typically served (Appendix C).

"clearer positioning relative to DashAttention and HiLS" (W3)

DashAttention and HiLS both select chunks by content alone, whereas LASA constrains selection by page structure. DashAttention uses α-entmax to pick a variable number of chunks per query. Substituting it for Token Bounding under our training setting reaches 83.02 on MPDocBench, close to LASA's 83.13, but its data-dependent budget makes KV gathering irregular and harder to serve than a fixed Top-K. HiLS also keeps a fixed budget, but learns chunk selection end-to-end from landmark-token summaries, and its routing is trained during pretraining or continued pretraining on billions of tokens. Our single SFT pass over document data gives such a router far less signal, so we expect it to be less suited to our setting.

5. Example of LASA handling a cross-page table (Q2)

A5: Each page's part is transcribed as a self-contained HTML table, the continuation fragment labeled append_table. g(q) counts emitted <PAGE> tokens, tying each query to its page.

While generating page k+1, remote visual attention covers page-(k+1) blocks, and boundary-straddling blocks also carry a few page-k visual tokens (Sec. 3.1). It predicts append_table from the page-(k+1) layout, these boundary tokens and the page-k table in the local text window. Fragments are then merged deterministically; Appendix D gives a visualized example.

We sincerely thank you for these detailed questions, which have helped us strengthen both the empirical comparisons and the presentation of our speed evaluation. We hope the revisions address your remaining concerns, and are glad to clarify any further point.

Thank you for the thorough rebuttal and additional experiments. I will maintain my current rating.

AuthorsOpenReview ↗

Response to Reviewer mBxZ

Thanks for your time reviewing our paper and all the constructive suggestions.

Reviewer_9hZSOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper works on end-to-end document parsing for very long documents (tens to hundreds of pages). Right now, page-by-page OCR pipelines lose cross-page tables and paragraphs, while full-attention VLMs get way too slow as the page count grows. The authors first show in Fig. 3 that if you just take a standard sparse router (like MSA) trained with KL distillation and run it on multi-page docs, the score recall drops from 96.5% on single pages down to 47.7%. This happens because pages in the same document look very similar visually, and full attention weights get too spread out over long contexts.

To solve this, the paper proposes LASA, which has two main pieces. First, Page Canonicalization restricts remote visual block selection to the current page (tracked with <PAGE> tokens) and uses an alternating 0-1 page index in RoPE (Page-RoPE) so positional embeddings stay bounded no matter how many pages there are. Second, Token Bounding uses an end-to-end router to pick a fixed budget of kk visual blocks and adds their log-probabilities as a bias into the main attention, so the router can be trained directly with the LM loss. The authors also build MarathonDocBench, a new benchmark of 100 long documents (103 to 300 pages) with lots of cross-page text and tables. Fine-tuning from UnlimitedOCR, their model beats strong pipelines like PaddleOCR-VL-1.6 and general VLMs on both MPDocBench and MarathonDocBench, while keeping per-step decoding latency flat and running up to 8.2x faster than full attention at 10

Critical strengths
  1. The motivation and empirical analysis in Section 2.3 (Fig. 3) are very clear. Instead of blindly applying existing sparse attention to long documents, the authors show why KL-distilled routing fails across multiple pages. Restricting the visual search space by page and training the top-kk router end-to-end through an attention bias is a simple and practical solution that keeps per-step decoding time nearly constant.

  2. MarathonDocBench is a very useful benchmark for the community. Most existing OCR benchmarks only go up to 20 or 30 pages, whereas real financial and legal documents often span 100+ pages with dense cross-page tables. The strong results on cross-page metrics (especially T-Table and T-Text) and the sliding-window KV cache streaming in Appendix B show that the proposed method scales well in practice.

Critical weaknesses

While the paper is technically solid and well-motivated, there are a few moderate weaknesses and presentation trade-offs to note:

  1. Visual cross-page dependencies are largely outsourced to the text decoder: In Eq. (3), constraining visual candidate blocks strictly to the current page Cpage(q)C_{\text{page}}(q) means that when complex structures (such as multi-page tables or split figures) cross page boundaries, the model cannot visually consult preceding pages. It entirely relies on the local text window L(q)L(q) to bridge cross-page semantics. This design is an effective engineering trade-off for scalability, but it somewhat simplifies the claim of "end-to-end multi-page visual modeling."

  2. Potential confounding factors in the main performance jump: The substantial performance jump on MarathonDocBench (Table 1) involves training on 400K pseudo-labeled samples generated by strong external OCR engines (MinerU-Popo and PaddleOCR-VL-1.6) under Gundam mode. Although the 200K ablation in Table 3 confirms a solid +5 point gain attributable to LASA itself, it remains somewhat difficult to disentangle how much of the headline gain comes from high-capacity data distillation versus the architectural innovations.

  3. Low information density in Figure 2: Figure 2 devotes three separate subplots across substantial main-text width merely to contrast three pairs of scalar metrics between MPDocBench and MarathonDocBench (document length, cross-page text, and cross-page tables). This comparison conveys essentially only six numbers and could be presented much more compactly as a small table or a single multi-metric bar chart, freeing up valuable main-text space for deeper analysis.

Questions for authors
  1. In Eq. (3), the candidate set Cpage(q)C_{\text{page}}(q) only keeps visual blocks that overlap with the current page g(q)g(q) (plus maybe one block sitting across the page boundary). When a table or paragraph splits across page i−1i-1 and page ii, does the model rely completely on the local text window L(q)L(q) (reading the already transcribed text from page i−1i-1) to merge them, or does it ever need visual tokens from page i−1i-1? Did you try letting Cpage(q)C_{\text{page}}(q) include page i−1i-1 as well?

  2. In Appendix A.1 and A.2, the final model is trained on 400K pseudo-labeled multi-page samples (using PaddleOCR-VL-1.6 and MinerU-Popo) with Gundam mode enabled, while the original UnlimitedOCR baseline used base mode for multi-page inputs. Table 3 gives a clean apples-to-apples comparison on the 200K subset (+5 points over retrained UOCR-1.0), which is great, but how much of the big jump in Table 1 (61.89 to 83.13) comes from the 400K training data and Gundam mode versus LASA itself?

  3. Minor typo to fix in the camera-ready version: in Fig. 4 (top green box), "Token Bounding" is mislabeled as "Text Bounding".

1 comment

1. Sufficiency of visual context for cross-page understanding, and the effect of expanding visual access (W1 & Q1)

A1: Thank you for these insightful comments. We address the concerns from three aspects.

"entirely relies on the local text window" (W1)

  1. Cross-page visual information in the current design. We agree that the local text window plays an important role in preserving cross-page information. However, cross-page understanding is not solely delegated to the text decoder. Since visual block boundaries are independent of page boundaries, selected boundary blocks may contain visual tokens from both the current and preceding pages, allowing the model to directly access visual context from adjacent pages. Appendix D in the revision adds a visualized example of a cross-page table showing how these boundary blocks are used and how g(q) evolves during inference.
  2. Contribution of cross-image boundary visual tokens. To test whether this pathway is actually used, we masked, at inference time only and without retraining, the visual tokens that come from the preceding page inside these boundary blocks. The cross-page text edit distance on MPDocBench (T-Text Edit, lower is better) nearly doubled, from 0.125 to 0.246. Since the model is unchanged and only this input is withheld, the degradation indicates that the trained model does rely on these few tokens rather than on the local text window alone.

"Did you try letting C_page(q) include page i-1 as well?" (Q1)

  1. Effect of expanding visual access. To test whether broader visual access helps, we expanded the visual candidate set to include blocks from the preceding page. The overall metric on MPDocBench improved by 0.5 points while the cross-page table metric dropped by 0.3 points, so the effect is not consistent across metrics. Including one more page of candidates also complicates candidate construction and raises peak memory under Sliding-Window Streaming.

Across these analyses, cross-page understanding in LASA rests on two complementary pathways, in which the boundary visual blocks play a substantive rather than an incidental role, and indiscriminately widening the visual scope does not translate into consistent improvements. The decisive factor is therefore which visual context is preserved rather than how much of it is retained, a principle that also guides our ongoing exploration of configurations such as linear attention for long-range visual dependencies.

2. Potential confounding factors in the main performance jump (W2 & Q2)

A2: Thank you for pointing out the potential confounding effect of data distillation and architectural improvements.

"difficult to disentangle ... data distillation versus the architectural innovations" (W2)

  1. Architectural gain at the final data scale. The preliminary ablations in Table 3 use only 200K pseudo-labeled samples under a 32K context window, so we repeated the comparison under the final training setting (400K samples, 64K context, Gundam representation). Under the same training data, LASA still outperforms the corresponding R-SWA baseline by 3.77 points (83.13 vs. 79.36), now reported in Appendix C of the revision.

"how much of the big jump ... versus LASA itself?" (Q2)

  1. Decomposing the headline jump. The headline number combines the architecture with the training configuration: the final setting adds a longer context, Gundam-mode multi-page inputs and a data mix balanced across document features. Retraining the R-SWA baseline on exactly the same 400K data and final setting still leaves LASA clearly ahead (Appendix C), and the same advantage appears in Table 3, whose 200K / 32K-context setting is the original UnlimitedOCR training configuration, which we kept so that the architecture is compared on the baseline's own terms.

3. Low information density in Figure 2 and typos (W3 & Q3)

A3: Thank you for this helpful suggestion. We have made Figure 2 more compact to improve its information density and readability, and corrected the typographical errors in the revised version, including the mislabeled "Text Bounding" in the top green box of Figure 4, which should read "Token Bounding".

We sincerely thank you for your careful reading and constructive suggestions, which have helped us strengthen the paper. We hope the revisions address your remaining concerns, and are glad to provide further details on any point.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Reviewers and Revision Submitted

We thank all reviewers for their insightful comments and constructive suggestions. We have revised the paper to address the concerns raised during the review process. Below, we summarize the major revisions (we also made minor layout adjustments and corrected typographical errors to meet the page limit), followed by our point-by-point responses to each reviewer. The major revisions are summarized as follows:

  1. Clarification of end-to-end speed evaluation (Figure 5). The reduction in attended visual tokens and attention computation follows from the Top-K budget itself and is independent of the implementation, whereas the resulting wall-clock speedup also depends on the attention kernels. To keep Figure 5 consistent with our current implementation, all methods have been re-measured under the same updated SGLang serving stack, with the FlashInfer attention backend for the baselines so that none of them is handicapped by its implementation. We have additionally expanded the explanation of the end-to-end speed evaluation in the main paper and provided a stage-by-stage breakdown of the time consumption in Appendix A.5. These revisions address the concerns raised by Reviewers mBxZ and 9J3v.
  2. Additional explanation of sliding-window streaming. We have expanded the explanation of Algorithm 1 in Appendix B to clarify the bounded KV-cache streaming procedure. Table 5 in the revision (Table 4 in the submission) now reports parsing quality on both MPDocBench and MarathonDocBench, and Table 6 adds the corresponding end-to-end throughput and KV-cache occupancy. These revisions address the questions raised by Reviewers mBxZ and 9J3v.
  3. Typographical and presentation improvements. We thank Reviewer 9hZS for pointing out these presentation issues. In response, we have corrected several typographical errors and refined Figure 2 by making it more compact to improve the overall presentation and accommodate the page limit.
  4. Ablation under the final training setting. To separate the contribution of each design from that of the training configuration, we have added Appendix C, which compares R-SWA, LASA without Token Bounding, and LASA under an identical final setting: the same 400K pseudo-labeled samples, the same 64K context length and the same Gundam visual representation. R-SWA uses the local window alone; LASA without Token Bounding adds Page Canonicalization, so each query attends to all visual blocks of its own page; LASA further adds Token Bounding, so only the Top-K blocks within that page are kept. We report both accuracy and throughput for each. This addresses the concerns raised by Reviewers 9hZS, mBxZ and 9J3v.
  5. Visualization of cross-page table parsing. We have added a visual example in Appendix D based on a cross-page table to illustrate how g(q) evolves during inference and how visual information from different pages is incorporated when parsing cross-page tables. This addresses the questions raised by Reviewers mBxZ and 9J3v regarding the changes in g(q), and illustrates the visual pathway that Reviewer 9hZS asked about when handling cross-page tables.

We appreciate the reviewers for their valuable comments and suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.