- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes a two-stage framework called SURGE, which aims to solve the modality gap between text, images, and charts. In the offline stage, they use Sinkhorn algorithm to build a graph. Then in the online stage, they use Personalized PageRank to rank the nodes.
Critical strengths
The biggest strength of this paper is that the author does not stick to improving the embedding itself. Instead, the author admits the modality gap is an objective limitation, and smartly chooses to use classic graph algorithms to bypass this problem using document structure.
By shifting the heavy Sinkhorn graph construction to the offline preprocessing stage, the online retrieval only needs to run a lightweight Personalized PageRank. 0.99ms online latency is really impressive.
The author Evaluating the retrieval system by the strict downstream VLM answer accuracy is a very solid and correct choice for long-context multimodal tasks
Critical weaknesses
When building the graph, the author use a very simple rule in Section 3.2: if text match patterns like 'Table X', it directly force the similarity to 1. This will bring some concerns. It totally overwrite the real embedding distance computed by model, and also this trick easily fail if document use different words. Because this strong rule mix with the Sinkhorn graph, the paper should add ablation study to remove this rule, to show how much performance gain really come from the Sinkhorn algorithm itself
The main experiments heavily rely on human-annotated chunks. But in real world application, we must use auto chunking pipeline. From Figure 3(a), we can see the performance drop a lot when using raw chunks. This means the proposed method is very sensitive to chunking errors. If the model need perfect human cropping to beat baselines, its practical value is limited.
In Table 3, DMAP actually get higher MRR and nDCG than SURGE. SURGE only win on the final VLM answer accuracy. This make people doubt: does SURGE really retrieve the correct ground truth better, or it just retrieve more images (as shown in Fig 3b) to let VLM guess the answer? The author explain this as 'alternative evidence paths', but it is not very convincing without more proof
The paper uses very long documents (up to 620K tokens), which is great. However, the evaluation datasets are limited to highly structured domains like finance (FinanceBench) and hardware manuals (the custom Chip Dataset). These domains naturally have very strict and standard cross-references. It is doubtful if the proposed method, especially the regex-based cross-reference boosting, can generalize well to more casual or unstructured general-domain multimodal documents (like web pages or news reports). The authors should test on more diverse open-domain datasets to prove its robustness.
Questions for authors
-
Could you please provide an ablation study on the "Cross-Reference Boosting" rule mentioned in Section 3.2? I really want to know what is the exact performance drop if we totally remove the regex pattern matching (like forcing similarity to 1 for "Table X"). Does the Sinkhorn OT graph still work well and beat baselines without this hard-coded trick?
-
Most of the main results (like Table 4) are based on perfect human-annotated chunks. Can you provide the detailed evaluation results if the system uses a standard open-source PDF parser
4 comments
Dear Reviewer,
While the rebuttal is ending soon, we looking forward to your response to check whether our rebuttal addressed your concerns. We would be happy to make clarifications if you had concerns unresolved.
Best, Submission 12648 Authors
Thanks for the new results. For Q1, without the regex boosting, SURGE drops to 54.49%, which is only a little bit better than Top-k (52.95%). For Q2, the auto-chunking result is 48.3 vs Top-k 47.2. Only 1.1% gain in real-world setting is too small for such a complex method. Also, you did not really address my concern about why DMAP gets higher MRR and nDCG than SURGE. so I will keep my score as 2.
We thank you for your additional feedback and apologize if we overlooked some of your concerns. We would like to make a few additional points.
-
We agree that Top-k is a simple baseline for comparison, but a complex model cannot necessarily compensate for the modality gap. As you observed, all the other graph-based baselines, such as ColPali and DMAP, performed significantly worse than Top-k. This is because the modality gap distorts inter-chunk relationships (line 46), causing these methods to fail. Our rationale for comparing SURGE with Top-k is not simply that “our method is more complex and hence it must be better than all simple methods.” We improved the performance of graph-based retrieval to the point where our method achieved the best results, which is itself a contribution.
-
The statement “Without the regex boosting SURGE is only a little bit better than Top-k” is not a meaningful assessment. If the question is how important boosting is to the overall framework, we have shown that it is less important than the Sinkhorn graph or the PPR procedure. If the question is how well SURGE performs, we evaluate the framework as a whole, including this simple and useful module that is applicable to all datasets. There are many examples of researchers adding small, useful modules to a larger framework for an incremental improvement, such as constrained decoding in generative models. Such modules are not typically treated as separate from the main framework.
-
We agree that the gain over Top-k is small on automatically parsed PDFs. However, the improvement over ColPali and DMAP is apparent. As we noted in the paper, the automated parsing pipeline still makes substantial errors on long documents, including “OCR errors, wrong boundaries, and poorly extracted tables or figures” (line 430). We hypothesize that an average accuracy of approximately 50% might therefore could be an upper bound for methods in this setting. Moreover, we explicitly identify this issue as a limitation and include the identification of “several open questions including the multimodal document segmentation bottleneck and call for dedicated research to address them” (line 68) among our core contributions.
-
We apologize for overlooking the concern on retrieval metrics. In addition to the explanation and examples of alternative paths in Appendix C.3, another reason for SURGE’s lower scores on retrieval metrics is the evaluation setup. ColPali and DMAP operate at the page level, while SURGE operates solely at the chunk level. If page contains chunks , and is the gold-standard chunk, SURGE must identify , while ColPali and DMAP need only retrieve . We recognized this issue but had no fair way to convert between the two evaluation setups, so we chose to report the original scores despite the disadvantage this poses for SURGE.
We hope these points address your additional concerns and that you will reconsider your evaluation of our manuscript.
We thank the reviewer for their recognition of our work. We are looking forward to your feedback on whether our rebuttal addresses your concerns.
1. Cross Reference Boosting.
We have conducted the relevant experiments and reported the results in the global response for clarity. Overall, our method performs better with boosting, but the variant with only the graph and PageRank still outperforms all baselines. This supports our insight that the Sinkhorn graph and PageRank help address the modality gap, with cross-reference boosting providing an additional performance improvement.
2. Human annotated chunks.
We present an extension of Figure 3a that includes the state-of-the-art baselines. The experiments use GPT-5.2 as the backbone.
All methods show a significant drop in performance when switching to an automated PDF parsing pipeline. Our method remains the best-performing method.
Figure 3a shows the performance gap between chunks generated by a PDF parsing pipeline and those revised by human annotators. We would like to highlight two observations: 1. Our method still outperforms the baseline under these circumstances; and 2. As chunk quality improves through human revision, our method exhibits a steeper increase in performance.
3. Domain of Long documents.
We understand the reviewer’s concern and would like to make the following points:
[1] LongDocURL: A Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating. Deng et al. ACL 2025.
[2] M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Framework. Shia et al. EMNLP 2025.