- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Multi-page document VLMs typically encode each page independently, which can create discontinuities when relevant content spans page boundaries. This paper proposes DocSEAM, a lightweight adapter placed after the vision encoder that enriches each page representation using information from its neighboring pages before passing the visual tokens to the LLM. The method uses asymmetric cross-page attention and directional salience gates to selectively model relationships between the previous, current, and next pages, without modifying the vision encoder or increasing the visual-token budget.
Critical strengths
- The paper proposes a simple and practical solution that does not require modifying the pretrained vision encoder and can be added on top of its output.
- The proposed method achieves significant improvements across several benchmark dataset.
- The paper is generally well written
- The paper provides extensive ablation studies
- The two-stage training strategy is interesting: DocSEAM is first trained specifically to learn cross-page relationships and is subsequently frozen while the rest of the model is fine-tuned on downstream multi-page document tasks.
Critical weaknesses
-
Limited novelty relative to SVE. The general motivation of avoiding independent visual encoding and introducing relationships among multiple visual inputs before the LLM has already been explored by Stateful Visual Encoders (SVE; Wang et al., 2026). SVE similarly introduces cross-image feature interaction.
-
Missing direct comparison with SVE. SVE is arguably the closest conceptual prior and is already discussed in the paper. Since its code is publicly available, a comparison under the document understanding setting would substantially strengthen the empirical evidence. I particualry would like to see how they work when there are long relations between multiple pages, as DocSEAM mainly focus on neighbor bages, while SVE may learn better the long ones.
-
The benefit of the proposed two-stage training strategy is not fully isolated. Stage A trains DocSEAM specifically on cross-page boundary/alignment examples before downstream training. Although the paper includes several training-strategy ablations, it remains unclear whether the benefit comes specifically from this cross-page curriculum or simply from giving DocSEAM a dedicated adaptation stage. A useful additional control would train DocSEAM in Stage A using the full downstream training mixture, rather than the specially constructed cross-page dataset, while keeping the remaining training procedure unchanged.
Questions for authors
Please check the weaknesses. My main concern is the missing SVE baseline. It is not clear to me whether the main novelty is simply bringing the idea of SVE to multi-page document understanding, or whether the specific design choices in DocSEAM provide significant performance gains over SVE.
2 comments
Dear Reviewer,
We wanted to follow up on our rebuttal, in which we addressed each of your concerns:
- SVE comparison: We implemented SVE on the same Qwen3.5-9B backbone and provided a full comparison (Table in Appendix L), showing that DocSEAM outperforms SVE by +13.8 / +20.1 points on MMLongBench-Doc / LongDocURL, including on the non-adjacent evidence subset.
- Stage A curriculum control: We trained DocSEAM's Stage A on the downstream mixture under matched budget and losses, isolating the contribution of the boundary-focused curriculum (+4.4 / +5.3 points over the downstream-mixture variant).
- Novelty relative to SVE: We provided both architectural analysis and empirical evidence showing that DocSEAM's document-specific design choices (bidirectional enrichment, center-anchored mask, directional gates) yield substantially larger gains than SVE in the document understanding setting.
We note that Reviewer Co5D has reviewed our rebuttal and raised their score from 3 to 4 after examining the additional experiments.
We would greatly appreciate it if you could review our response and let us know whether the additional results address your concerns, or if there are remaining questions we can clarify. Thank you for your time.
Thank you for highlighting SVE as the closest prior and suggesting a direct comparison. We have added SVE results, including a breakdown by evidence topology, and the requested Stage A control.
Weakness 1: Both methods enable visual interaction before the LLM, but address different settings. SVE is motivated by visual changes and state tracking in temporal or interactive image sequences, where the current observation can access only past images. For fully observed documents, DocSEAM instead reads both the previous and next pages to connect content across page boundaries.
The methods also differ in architecture and training. SVE inserts causal Cross+FFN blocks inside the vision encoder and trains them during single-stage SFT with stop-gradient on predecessor features. DocSEAM leaves the vision encoder unchanged, operates on post-merger tokens, and uses a center-anchored mask and directional gates. It first trains the module on supervised boundary-alignment tasks with the backbone frozen (Stage A), then freezes the module during downstream fine-tuning (Stage B).
Our ablations support these document-specific choices: restricting DocSEAM to the previous page reduces performance from 58.3 / 62.9 to 53.1 / 55.8 on MMLongBench-Doc / LongDocURL; replacing the four directional gates with a shared gate reduces it to 57.1 / 60.2. The previous-only variant is an ablation of DocSEAM, while the following table evaluates SVE directly.
Weakness 2: We implement the released SVE on Qwen3.5-9B. SVE and the SFT baseline are trained on the combined Stage A and Stage B data in one stage. The SVE configuration uses Cross+FFN in every vision block, cloned input projections, zero-initialized output projections, predecessor stop-gradient, and first-page self-fallback. We also include our post-encoder variant without a mask or gates.
The last two columns are LongDocURL evidence subsets. Although SVE can propagate information from earlier pages across successive encoder layers, it scores 38.7 on the non-adjacent subset, compared with 40.0 for SFT and 54.7 for DocSEAM. SVE's aggregate gains over SFT are +0.7 / +0.6, while DocSEAM's are +13.8 / +20.1. DocSEAM enriches local representations; the LLM can then integrate evidence across distant pages.
This difference in task requirements may help explain SVE's smaller gains. In the example in Figure 11, SVE's averaged source attention is diffuse and does not clearly concentrate on boundary-continuation evidence; this visualization is qualitative.
Weakness 3: As suggested, we replace the boundary-focused Stage A data with the full downstream mixture, keeping the Stage A steps, batch size, CE and auxiliary gate losses, and subsequent Stage B procedure unchanged.
The downstream-mixture warm-up improves over no Stage A by 1.2 / 1.5 points. Under the matched budget and loss terms, boundary-focused Stage A adds a further 4.4 / 5.3 points, supporting the value of the curriculum beyond a dedicated adaptation stage. We report this control in Section 4.4.
We hope these comparisons address your concerns about the distinction from SVE, distant dependencies, and Stage A and reflect positively in your final evaluation.