acceptodds
Under review as a conference paper at ICLR 2027

DocSEAM: Bidirectional Cross-Page Encoding for Multi-Page Document Understanding

Abstract

Multi-page document vision-language models encode each page independently, creating a feature-level discontinuity at page boundaries: visual tokens are formed without access to content on adjacent pages, and the language model must reason over these page representations formed in isolation. We introduce DocSEAM (Document Selective Enrichment via Asymmetric Masking), a lightweight module inserted between a pretrained vision encoder and the language model that injects the inductive bias of local cross-page context into visual representations. For each page, DocSEAM forms a local triplet with its two neighbors, applies asymmetric masked self-attention with directional salience gates to fuse information from neighbor pages, and returns only the enriched center-page tokens, leaving the visual token count unchanged. After a supervised cross-page warm-up and multi-page fine-tuning with DocSEAM frozen, our 9B-parameter model achieves state-of-the-art results among open-weight models on five multi-page document VQA benchmarks. We further show that DocSEAM makes the model robust to arbitrary re-pagination and significantly improves cross-page span localization, confirming that the gains stem from improved cross-page representations

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-page Document Visual Question Answering

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_j3kcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Multi-page document VLMs typically encode each page independently, which can create discontinuities when relevant content spans page boundaries. This paper proposes DocSEAM, a lightweight adapter placed after the vision encoder that enriches each page representation using information from its neighboring pages before passing the visual tokens to the LLM. The method uses asymmetric cross-page attention and directional salience gates to selectively model relationships between the previous, current, and next pages, without modifying the vision encoder or increasing the visual-token budget.

Critical strengths
  • The paper proposes a simple and practical solution that does not require modifying the pretrained vision encoder and can be added on top of its output.
  • The proposed method achieves significant improvements across several benchmark dataset.
  • The paper is generally well written
  • The paper provides extensive ablation studies
  • The two-stage training strategy is interesting: DocSEAM is first trained specifically to learn cross-page relationships and is subsequently frozen while the rest of the model is fine-tuned on downstream multi-page document tasks.
Critical weaknesses
  • Limited novelty relative to SVE. The general motivation of avoiding independent visual encoding and introducing relationships among multiple visual inputs before the LLM has already been explored by Stateful Visual Encoders (SVE; Wang et al., 2026). SVE similarly introduces cross-image feature interaction.

  • Missing direct comparison with SVE. SVE is arguably the closest conceptual prior and is already discussed in the paper. Since its code is publicly available, a comparison under the document understanding setting would substantially strengthen the empirical evidence. I particualry would like to see how they work when there are long relations between multiple pages, as DocSEAM mainly focus on neighbor bages, while SVE may learn better the long ones.

  • The benefit of the proposed two-stage training strategy is not fully isolated. Stage A trains DocSEAM specifically on cross-page boundary/alignment examples before downstream training. Although the paper includes several training-strategy ablations, it remains unclear whether the benefit comes specifically from this cross-page curriculum or simply from giving DocSEAM a dedicated adaptation stage. A useful additional control would train DocSEAM in Stage A using the full downstream training mixture, rather than the specially constructed cross-page dataset, while keeping the remaining training procedure unchanged.

Questions for authors

Please check the weaknesses. My main concern is the missing SVE baseline. It is not clear to me whether the main novelty is simply bringing the idea of SVE to multi-page document understanding, or whether the specific design choices in DocSEAM provide significant performance gains over SVE.

2 comments

Thank you for highlighting SVE as the closest prior and suggesting a direct comparison. We have added SVE results, including a breakdown by evidence topology, and the requested Stage A control.

Weakness 1: Both methods enable visual interaction before the LLM, but address different settings. SVE is motivated by visual changes and state tracking in temporal or interactive image sequences, where the current observation can access only past images. For fully observed documents, DocSEAM instead reads both the previous and next pages to connect content across page boundaries.

The methods also differ in architecture and training. SVE inserts causal Cross+FFN blocks inside the vision encoder and trains them during single-stage SFT with stop-gradient on predecessor features. DocSEAM leaves the vision encoder unchanged, operates on post-merger tokens, and uses a center-anchored mask and directional gates. It first trains the module on supervised boundary-alignment tasks with the backbone frozen (Stage A), then freezes the module during downstream fine-tuning (Stage B).

Our ablations support these document-specific choices: restricting DocSEAM to the previous page reduces performance from 58.3 / 62.9 to 53.1 / 55.8 on MMLongBench-Doc / LongDocURL; replacing the four directional gates with a shared gate reduces it to 57.1 / 60.2. The previous-only variant is an ablation of DocSEAM, while the following table evaluates SVE directly.

Weakness 2: We implement the released SVE on Qwen3.5-9B. SVE and the SFT baseline are trained on the combined Stage A and Stage B data in one stage. The SVE configuration uses Cross+FFN in every vision block, cloned input projections, zero-initialized output projections, predecessor stop-gradient, and first-page self-fallback. We also include our post-encoder variant without a mask or gates.

MethodMMLongBench-DocLongDocURLAdjacent two-pageNon-adjacent only
Zero-shot baseline57.154.558.048.8
SFT baseline44.542.847.040.0
SVE45.243.447.638.7
Post-encoder, no mask or gates55.257.864.150.3
DocSEAM58.362.969.554.7

The last two columns are LongDocURL evidence subsets. Although SVE can propagate information from earlier pages across successive encoder layers, it scores 38.7 on the non-adjacent subset, compared with 40.0 for SFT and 54.7 for DocSEAM. SVE's aggregate gains over SFT are +0.7 / +0.6, while DocSEAM's are +13.8 / +20.1. DocSEAM enriches local representations; the LLM can then integrate evidence across distant pages.

This difference in task requirements may help explain SVE's smaller gains. In the example in Figure 11, SVE's averaged source attention is diffuse and does not clearly concentrate on boundary-continuation evidence; this visualization is qualitative.

Weakness 3: As suggested, we replace the boundary-focused Stage A data with the full downstream mixture, keeping the Stage A steps, batch size, CE and auxiliary gate losses, and subsequent Stage B procedure unchanged.

Stage A variantMMLongBench-DocLongDocURL
No Stage A52.756.1
Downstream mixture, CE + auxiliary losses53.957.6
Boundary data, CE only55.859.5
Boundary data, CE + auxiliary losses (ours)58.362.9

The downstream-mixture warm-up improves over no Stage A by 1.2 / 1.5 points. Under the matched budget and loss terms, boundary-focused Stage A adds a further 4.4 / 5.3 points, supporting the value of the curriculum beyond a dedicated adaptation stage. We report this control in Section 4.4.

We hope these comparisons address your concerns about the distinction from SVE, distant dependencies, and Stage A and reflect positively in your final evaluation.

Dear Reviewer,

We wanted to follow up on our rebuttal, in which we addressed each of your concerns:

  1. SVE comparison: We implemented SVE on the same Qwen3.5-9B backbone and provided a full comparison (Table in Appendix L), showing that DocSEAM outperforms SVE by +13.8 / +20.1 points on MMLongBench-Doc / LongDocURL, including on the non-adjacent evidence subset.
  2. Stage A curriculum control: We trained DocSEAM's Stage A on the downstream mixture under matched budget and losses, isolating the contribution of the boundary-focused curriculum (+4.4 / +5.3 points over the downstream-mixture variant).
  3. Novelty relative to SVE: We provided both architectural analysis and empirical evidence showing that DocSEAM's document-specific design choices (bidirectional enrichment, center-anchored mask, directional gates) yield substantially larger gains than SVE in the document understanding setting.

We note that Reviewer Co5D has reviewed our rebuttal and raised their score from 3 to 4 after examining the additional experiments.

We would greatly appreciate it if you could review our response and let us know whether the additional results address your concerns, or if there are remaining questions we can clarify. Thank you for your time.

Reviewer_CzKUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, authors propose a cross-attention block applied to the output of visual encoders that capture the information at the end and beginning of the page, to improve the representation of documents, and thus the performance of multipage VQA systems. They apply their technique to benchmark datasets within the field, obtaining slight improvements in relation to other techniques that already exist.

Critical strengths

They tackle the VQA task in complex scenarios where the information is spread over more than one page of the document

Critical weaknesses

Overall, Section 3, Method. Is hard to follow. There are some terminological errors that could be easily solved with a couple of iterations but others, more conceptual, that make the work more difficult to understand. Some examples, not necessarily exhaustive or ordered in order of severity, are:

  1. In eq (1) NpN_p, I understand that it does not depend on p. All pages for a matter of normalization in the input will have the same number of visual tokens
  2. The dimension of XpX_p, I understand that is $ \mathbb{R}^{(N_{p-1}+ N_{p}+ N_{p+1})\times d}$, right?
  3. The matrix M, eq (4), is not asymmetric. What is the dimension?
  4. Figure 2, which illustrates the DocSEAM module, is not well understood. Moreover, Sections 3.1, 3.2, 3.3, which I imagine that describe the DOcSEAM module, neither.
  5. The notations of the salience gates (section 3.3) are confusing. Taken alone, I could imagine their meaning but within Equations (7), (8) and (10) the notation, and the section, is harder to follow.

The whole section would have been understood much better if the Eq (9), which I understand to be the DocSEAM module, was introduced at the beginning of the section and given the intuition first of what you want to do

At the experiment level, my main concern is that they seem to focus on single-column documents. How does their schema apply to multi-column documents?

Questions for authors
  • My main suggestion for authors is that they should fully rewrite the method description, with more precise and accurate notation.

  • At training, they propose a 2-phase training schema. Have they consider to repeat it like in an Expectation-Maximization scheme? I can understand that an EM scheme does not apply exactly to their training but it would be worth evaluating the impact on training repeating this 2-phase training

  • How do you plan to extend your scheme to multi-column documents?

7 comments

Thank you for identifying the points in Section 3 that need clarification. We have moved the module-level equation to the beginning, clarified token dimensions and mask terminology, and added a concrete explanation of the directional gates. Below, we walk through Figure 2 and report additional results on multi-column documents.

Question 1 We moved the module-level equation defining EpE_p to the beginning of the method section, before introducing the rest modules. This gives the input-output view of DocSEAM before the individual components are described.

Weakness 1 Qwen3.5 uses a native-resolution encoder, so NpN_p depends on each page's resolution and aspect ratio after preprocessing and merging. Triplet of pages are then concatenated, so pages with different token counts need no padding. If all pages were resized to one grid, NpN_p would be a constant NN.

Weakness 2 Yes, Xp∈R(Np−1+Np+Np+1)×dX_p \in \mathbb{R}^{(N_{p-1} + N_p + N_{p+1}) \times d}. The equation defining XpX_p now states this explicitly.

Weakness 3 You are right, the mask (M) is symmetric. For page groups ordered as (PREV, CURR, NEXT), the mask has dimension NΣ×NΣN_\Sigma\times N_\Sigma with NΣ=Np−1+Np+Np+1N_\Sigma=N_{p-1}+N_p+N_{p+1}, and its block structure is

\mathbf{0} & \mathbf{0} & -\infty \\\\ \mathbf{0} & \mathbf{0} & \mathbf{0} \\\\ -\infty & \mathbf{0} & \mathbf{0} \end{pmatrix}.$$ where each ($a, b$) block has shape $N_a\times N_b$. The asymmetry in our design lies in the functional roles (the center page is a hub that reads both neighbors, while neighbors read only the center and themselves) and in the directional gates, not in M. The word "asymmetric" was misleading so we now call it a "center-anchored mask" to describe its connectivity: the center page reads both neighbors, while direct previous-next interaction is blocked. We updated the acronym expansion to "Document Selective Enrichment via Anchored Masking", keeping "DocSEAM" unchanged. **Weakness 4** Figure 2 can be read from left to right. On the left, the vision encoder produces one token sequence $Z_p$ per page. For each page $p$, DocSEAM concatenates the previous, current, and next sequences into $X_p=[Z_{p-1};Z_p;Z_{p+1}]$, processes the triplet, and keep only the "enriched" center-page output $E_p$. The gray output tokens in the figure denote the neighbor outputs that are discarded. Repeating this operation for all pages produces $[E_1;\ldots;E_n]$, preserving the original visual-token count seen by the LLM. The right panel expands the attention computation within a DocSEAM layer. 1. The attention score measures query-key compatibility; 2. The anchored mask $M$ helps us "mimic" the natural information flow of a document through block direct communication between the previous and next pages so they cannot bypass the center page. The center page, in this design, acts as a local hub that mediates all neighbor context. 3. Page edges are full of noise like blank margins, headers, and page numbers, the directional bias $B$ modulates how strongly information is exchanged between pages. Each token predicts four gates: two for sending information to the previous and next pages, and two for receiving information from them. For a cross-page link, the sending and receiving gates jointly scale its unnormalized attention weight. For example, tokens near the two ends of a continued sentence can receive higher weights, while tokens corresponding to blank margins may receive lower weights. **Weakness 5** For clarity, consider one cross-page interaction. Let query $q$ belong to the current page $p$ and key $k$ to the previous page $p-1$. Information flows from $k$ to $q$, so the previous-page token uses its send-to-next gate $\sigma(s_k^{\to})$, while the current-page token uses its receive-from-previous gate $\sigma(r_q^{\gets})$: $$B_{qk}=\log\sigma(r_q^{\gets})+\log\sigma(s_k^{\to}).$$ The two gates describe the sending and receiving ends of the same link. Adding their log-values to the attention score multiplies its unnormalized softmax weight by $\sigma(r_q^{\gets})\sigma(s_k^{\to})$. Reading from the next page uses the other pair, $\sigma(r_q^{\to})$ and $\sigma(s_k^{\gets})$, with the same interpretation. The send/receive symbols denote four separate MLP outputs; same-page attention cells receive no salience bias. We have added a two-sentence explanation of this query-key example immediately after the previous-page bias equation in Section 3.3.

Question 2

Thank you for suggesting repeated alternation between the two stages. We have not evaluated this variant. Our current schedule assigns distinct roles to the stages: Stage A learns boundary-focused enrichment while the backbone is fixed, and Stage B adapts the downstream model while keeping the learned DocSEAM parameters fixed. Returning to Stage A after downstream adaptation would allow the two components to adapt further to each other, making alternating optimization a reasonable extension.

Our existing controls evaluate the value of the initial warm-up: omitting Stage A yields 52.7 / 56.1 on MMLongBench-Doc / LongDocURL. Using the downstream mixture in Stage A yields 53.9 / 57.6, while boundary-focused Stage A yields 58.3 / 62.9; these two Stage A variants use matched adaptation budgets and loss terms. These results support the benefit of the current single-pass schedule, although they do not determine whether additional alternations would improve it. Repeated alternation remains an unexplored extension.

Question 3 DocSEAM can be applied to multi-column documents without architectural changes. We manually opened each document to determine whether the document is single-column or multi-column. The following table shows the result:

BenchmarkLayoutQuestionsSFT (%)DocSEAM (%)Gain (points)
MMLongBench-DocSingle-column73745.159.4+14.3
MMLongBench-DocMulti-column34543.355.8+12.5
LongDocURLSingle-column1,69043.262.8+19.6
LongDocURLMulti-column63541.863.0+21.2

DocSEAM improves both layout groups, with comparable gains on multi-column documents (+12.5 / +21.2 points) and single-column documents (+14.3 / +19.6 points) on MMLongBench-Doc / LongDocURL, respectively.

Figure 8 in Appendix K additionally visualizes cross-page attention on a two-column document.

We hope these clarifications and the additional multi-column results address your concerns about the method's formulation and applicability. We would be grateful if you would reconsider your assessment in light of these revisions and results.

I thank the authors for their responses. The revised method section is clearly easier to follow and resolves many of the notation issues. However, the question of multi-column documents remains open, which in my opinion is a key property for these kind of approaches. The design of the module assumes that reading continues from the bottom of one page to the top of the next: the mask connects consecutive pages, while the gates and the frontier encoding rely on vertical position relative to the page boundaries. In a multi-column layout this assumption breaks down, as the bottom of a page holds the endings of all columns and only one of them continues on the next page. As each gate depends on a single token, nothing in the module can identify that fragment, so any gain in locating the continuation would have to come from generic attention rather than from the proposed components. The authors themselves note in Appendix K that Figure 8 is “not sufficient to verify alignment with the preceding page”. The per-layout results do not settle this, as they omit the variant without mask and gates and classify documents rather than questions. The applicability of the method to multi-column documents, and more generally to non-linear reading orders, therefore remains unclear.

Reviewer_Co5DOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces an approach to incorporate cross-page information enrichment for the visual tokens of a VLM with minimal addition of a trainable LoRA-esque module placed after the visual encoder to solve the inherent challenge for current VLMs to correlate content broken across page boundaries and to establish continuity amongst the tokens pertaining to neighboring pages. The paper details the design of the module, the training approach in two stages and a broad set of evaluations to verify their design choices and the benefit of such a module. The approach achieves SOTA performance on various long document VQA benchmarks confirming the benefit of having a module specifically designed to incorporate the task specific inductive biases.

Critical strengths
  1. The paper is well written. It is easy to follow. I was able to quickly understand the overall approach from the introduction itself. And further sections gave enough details and visualization for grasping the specifics.
  2. The approach, while not ground-breaking, is an interesting idea of encoding the inductive bias of "how to enforce pages attend to neighboring pages for improving their information representation" through a minimal module specifically designed for this end goal.
  3. The paper shows strong performance gains over the baselines across the relevant multi-page document vqa benchmarks.
  4. Overall I get a good idea on the intuition of the paper and how to implement it (for reproducibility) and feel this is a good contribution to the domain.
Critical weaknesses
  1. The pagination experiment, while it does seem to indicate that the model is sensitive to the document structure with fragmented document content spread across pages, does not necessarily rule out the effect of pre-training. The models would have simply not been trained on such fragmented inputs and thus could simply be a case for the data being OOD as far as the model is concerned. We should have additional baselines with models trained or augmented with re-paginated documents.
  2. The additional module ∼235M parameters (∼2.6% of a 9B backbone) added could just be doing the job of selective page-aware LoRA and so the paper should discuss that.
  3. Wouldnt claim that with DocSeam the model being able to see cross-page continuity in earlier layers is a by-product of the approach - it in fact is exactly what the approach is designing to do.
  4. "At document boundaries we use self-fallback, where Z0 = Z1, and ZP+1 = ZP, keeping the computation shape uniform without introducing a learned null-page embedding." - ablations for other variations?
  5. There is no justification for the need for Content-frontier positional encoding. No ablations regarding that design choices nor evaluating against standard PEs.
  6. Auxiliary gate loss should have been explained as part of the main paper and not supplementary since it leaves a key new term added of the implementation as auxiliary definition.
  7. In table 1, they haven't compared with similar sized Qwen baselines (specifically 4B and 7B) against their DocSeam included Qwen.
  8. "This confirms that DocSEAM encodes cross-page continuity into the visual tokens before the language model begins deep reasoning." - As mentioned previously, this is unsurprising since that is precisely what the design is aimed to do. Force cross-page correlations to be mapped out early on.
  9. Table 2's Existence Check result for SFT beating DocSeam is not discussed.
  10. Table 5 and Table 3's baselines are with the non-trained Qwen model. But that is compared with a trained DocSEAM model. That is not apples-to-apples comparison.
  11. The claim that the attention visualization "confirms cross-page continuity is encoded in the visual tokens before deep reasoning" is overstated. Earlier attention does not necessarily mean cross page continuity is encoded in the visual tokens. It could be that the language model may still be doing cross-page reasoning at Layer 7. Attention concentration need not mean attended information caused the prediction. To strengthen the claim, the authors could probe the visual tokens before the LLM or decode localization from intermediate layers, etc.
Questions for authors
  1. Could you provide additional baselines with models trained or augmented with re-paginated documents and compare your method to that?
  2. Why was the Content-frontier positional encoding needed? Do we have ablations on that?
  3. Can you please show comparisons with similar sized Qwen baselines (4B / 7B) and what about larger sized models?
  4. Table 5 and 3 needs to have the proper trained equivalent baseline.
  5. Did you test inclusion of the "Question" encodings to adapt DocSEAM's attention it pays to neighboring pages? As in, would conditioning on the question enable richer cross-page correlation for answering the question?
3 comments

Thank you for the careful and constructive assessment, and especially for recognizing DocSEAM's clear presentation, practical design, strong results, and reproducibility. Below are our responses to your concerns and questions:

Weakness 1 / Question 1: We trained the SFT baseline with re-paginated documents added to the Stage B data (half structure-preserving, half structure-breaking cuts), keeping everything else identical. We evaluate it on the re-paginated test sets and, importantly, on the original unmodified benchmarks.

ModelMMLongBench-Doc: originalpreservedbrokenLongDocURL: originalpreservedbroken
Baseline57.153.852.554.550.748.4
SFT44.542.140.942.841.539.9
SFT + re-paginated data in Stage B44.441.940.743.542.140.3
DocSEAM58.357.656.862.961.861.1

Two observations follow:

  1. Augmentation has little effect on MMLongBench-Doc and gives small gains on LongDocURL (+0.7 on original, +0.6 on preserved, and +0.4 on broken inputs). The pagination penalty remains 1.2 on MMLongBench-Doc and changes from 1.6 to 1.8 on LongDocURL. Thus, this augmentation does not reduce boundary-placement sensitivity in the tested configuration.
  2. DocSEAM still leads the augmented baseline by 16.1 / 20.8 points on broken inputs and by 13.9 / 19.4 points on the original benchmarks. Exposure to re-paginated training data under this protocol therefore does not close the gap, although this control does not rule out all possible effects of distribution shift.

Weakness 2: DocSEAM is adapter-like in the broad sense that it is a small trainable residual module. LoRA reparameterizes existing weight matrices, whereas DocSEAM introduces additional cross-page attention operations after the vision encoder.

More importantly, our existing capacity controls directly test whether extra parameters or page-local adaptation explain the gain. Three approximately 235M-parameter variants without real neighboring-page information, (self-as-neighbor, duplicated vision-encoder layers, and a single-page transformer), reach at most 46.2 on MMLongBench-Doc and 44.5 on LongDocURL, compared with DocSEAM's 58.3 and 62.9. These controls show that adding comparable page-local capacity does not recover the observed improvement.

Weakness 3 and 8: We agree. Earlier localization is an intended consequence of enriching the visual representations before the LLM, not an incidental by-product or a surprising discovery. We have revised the paper to present earlier attention concentration as a behavioral check consistent with the intended effect of cross-page enrichment.

Weakness 4: We compared self-fallback with two alternatives. The blank-white-page variant replaces missing neighbors with blank white pages only at inference. The learned null-page embedding variant is trained through both Stage A and Stage B.

Boundary handlingMMLongBench-DocLongDocURL
Self-fallback, Z0=Z1Z_0=Z_1 and ZP+1=ZPZ_{P+1}=Z_P (ours)58.362.9
Blank white page (inference only)58.162.7
Learned null-page embedding (retrained)57.962.4

The tested alternatives differ from self-fallback by at most 0.4 points on MMLongBench-Doc and 0.5 points on LongDocURL. We retain self-fallback because it requires no additional trainable parameters.

Weakness 5 / Question 2:

Document pages may have varying white margins, and absolute row positions do not directly indicate where document content begins or ends. We compared three gate inputs while keeping the remaining architecture, training data, objectives, and training schedule unchanged:

Gate inputMMLongBench-DocLongDocURL
Visual token only, no positional feature56.960.6
Standard sinusoidal encoding of the absolute row index57.461.5
Content-frontier encoding (ours)58.362.9

Content-frontier encoding improves performance by 0.9 and 1.4 points over absolute-row sinusoidal encoding on MMLongBench-Doc and LongDocURL, respectively. We have expanded the motivation in Section 3.3 and added the results in the appendix section "Full Ablation Details".

Weakness 6: We have expanded Section 3.4 in Method Section to include the auxiliary gate loss.

Weakness 7 / Question 3: We added zero-shot and SFT baselines for the two smaller backbones on all five benchmarks. The complete table is in the appendix, referenced from Section 4.1.

BackboneVariantDUDEMP-DocVQASlideVQAMMLongBench-DocLongDocURL
Qwen3.5-4BZero-shot43.679.267.351.951.4
Qwen3.5-4BSFT51.183.681.237.938.8
Qwen3.5-4B+ DocSEAM57.586.379.638.253.3
Qwen2.5-VL-7BZero-shot35.270.159.825.437.8
Qwen2.5-VL-7BSFT56.082.967.428.842.7
Qwen2.5-VL-7B+ DocSEAM58.088.380.039.553.3

DocSEAM improves over matched SFT on all five benchmarks for Qwen2.5-VL-7B and on four of five for Qwen3.5-4B. The exception is SlideVQA on Qwen3.5-4B, where it scores 79.6 versus 81.2 for SFT. On MMLongBench-Doc, the 4B model improves slightly over SFT (38.2 versus 37.9), but remains below zero-shot (51.9); on LongDocURL, it exceeds both SFT and zero-shot (53.3 versus 38.8 and 51.4). These results show that the gains vary across backbones and tasks.

Weakness 9: The existence check asks only whether a span occurs on one specified page and does not require cross-page linking. DocSEAM's main improvement occurs on the intended relational probe: 85.8 versus 65.0 for cross-page lookup. Both models remain strong on existence. We have explicitly discussed this trade-off after Table 2 in Section 4.2.

Weakness 10 / Question 4: We agree that a matched SFT comparison is needed to assess the benefit of DocSEAM after task adaptation. We retain zero-shot as a reference for the effect of fine-tuning and add the matched SFT baseline. DocSEAM uses the two-stage procedure described in the paper, while the SFT baseline trains on the combined Stage A and Stage B data in one stage. The revised tables have been incorporated into the paper.

Table 3 Revised:

MetricModelMMLongBenchLongDocURL
OriginalZero-shot baseline57.154.5
OriginalSFT baseline44.542.8
OriginalDocSEAM58.362.9
PreservedZero-shot baseline53.850.7
PreservedSFT baseline42.141.5
PreservedDocSEAM57.661.8
BrokenZero-shot baseline52.548.4
BrokenSFT baseline40.939.9
BrokenDocSEAM56.861.1
DegradationZero-shot baseline4.66.1
DegradationSFT baseline3.62.9
DegradationDocSEAM1.51.8
PenaltyZero-shot baseline1.32.3
PenaltySFT baseline1.21.6
PenaltyDocSEAM0.80.7

Table 5 Revised:

Evidence topologyZero-shot baselineSFT baselineDocSEAMGain (over base)Gain (over Base SFT)
Single page57.547.563.0+5.5+15.5
Adjacent two pages58.047.069.5+11.5+22.5
Multiple ($ \geq $ 3, with an adjacent pair)48.838.460.1+11.3+21.7
Non-adjacent only48.840.054.7+5.9+14.7
Overall54.542.862.9+8.4+20.1

Weakness 11: We agree. Attention maps only show correlation and do not prove causal impact on predictions or what is encoded inside pre-LLM tokens. We have softened our claims in Section 4.2 to report earlier attention focus without claiming causal effects or pre-LLM representation content, while our 1,500-sample benchmark (Table 2) confirms systematic output gains. We leave direct probing of pre-LLM representations to future work.

Question 5:

We have not evaluated a question-conditioned variant. In our design, DocSEAM produces question-independent visual tokens that are computed once per document and reused across multiple queries, leaving query-specific selection to the downstream LLM. While conditioning DocSEAM on the question could enable more targeted cross-page fusion, it would require recomputing visual enrichment for every query. We agree that question-conditioned routing as a promising future extension.

Thank you again for the detailed feedback. We hope the additional controls and revisions address your concerns about the design choices, comparison fairness, and interpretation of the results.

We sincerely thank you for the thorough re-evaluation and for raising the score to 4. Your detailed questions throughout the review process led to significant improvements in the paper, including the re-pagination augmentation control, the matched SFT baselines for Tables 3 and 5, and the backbone generality experiments. We are grateful for the constructive engagement and are glad that the revisions addressed your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.