acceptodds
Under review as a conference paper at ICLR 2027

SURGE: Sinkhorn-Seeded Graph Expansion for Multimodal Long-Document Retrieval

Abstract

Retrieval-augmented generation over long documents is increasingly important in practice, yet most existing approaches are designed for purely textual corpora and fail to handle the heterogeneous mix of text, tables, and figures found in real-world technical documents. We present SURGE, a two-stage retrieval framework for multimodal long-context documents. In the first stage, Sinkhorn-regularized optimal transport is applied over all pairwise chunk similarities to produce a globally calibrated co-association graph that encodes inter-chunk relationships across modalities without privileging text. In the second stage, Personalized PageRank is seeded from a query-proportional personalization vector and propagates relevance through this graph, surfacing tables and figures that cosine-similarity retrieval systematically misses due to the embedding-space modality gap. We evaluate SURGE on long multimodal datasets spanning up to 620K tokens across vision-language models of different sizes and demonstrate advantages over baselines on document querying. Our analysis further identifies multimodal document segmentation as the critical open bottleneck: retrieval quality is fundamentally chunking-quality-dependent, and we call for research on robust multimodal chunking pipelines.

Primary area:
generative models
Keywords:
Retrieval-augmented generation, Multimodal retrieval, Graph expansion, Long-context LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zvEVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a two-stage framework called SURGE, which aims to solve the modality gap between text, images, and charts. In the offline stage, they use Sinkhorn algorithm to build a graph. Then in the online stage, they use Personalized PageRank to rank the nodes.

Critical strengths

The biggest strength of this paper is that the author does not stick to improving the embedding itself. Instead, the author admits the modality gap is an objective limitation, and smartly chooses to use classic graph algorithms to bypass this problem using document structure.

By shifting the heavy Sinkhorn graph construction to the offline preprocessing stage, the online retrieval only needs to run a lightweight Personalized PageRank. 0.99ms online latency is really impressive.

The author Evaluating the retrieval system by the strict downstream VLM answer accuracy is a very solid and correct choice for long-context multimodal tasks

Critical weaknesses

When building the graph, the author use a very simple rule in Section 3.2: if text match patterns like 'Table X', it directly force the similarity to 1. This will bring some concerns. It totally overwrite the real embedding distance computed by model, and also this trick easily fail if document use different words. Because this strong rule mix with the Sinkhorn graph, the paper should add ablation study to remove this rule, to show how much performance gain really come from the Sinkhorn algorithm itself

The main experiments heavily rely on human-annotated chunks. But in real world application, we must use auto chunking pipeline. From Figure 3(a), we can see the performance drop a lot when using raw chunks. This means the proposed method is very sensitive to chunking errors. If the model need perfect human cropping to beat baselines, its practical value is limited.

In Table 3, DMAP actually get higher MRR and nDCG than SURGE. SURGE only win on the final VLM answer accuracy. This make people doubt: does SURGE really retrieve the correct ground truth better, or it just retrieve more images (as shown in Fig 3b) to let VLM guess the answer? The author explain this as 'alternative evidence paths', but it is not very convincing without more proof

The paper uses very long documents (up to 620K tokens), which is great. However, the evaluation datasets are limited to highly structured domains like finance (FinanceBench) and hardware manuals (the custom Chip Dataset). These domains naturally have very strict and standard cross-references. It is doubtful if the proposed method, especially the regex-based cross-reference boosting, can generalize well to more casual or unstructured general-domain multimodal documents (like web pages or news reports). The authors should test on more diverse open-domain datasets to prove its robustness.

Questions for authors
  1. Could you please provide an ablation study on the "Cross-Reference Boosting" rule mentioned in Section 3.2? I really want to know what is the exact performance drop if we totally remove the regex pattern matching (like forcing similarity to 1 for "Table X"). Does the Sinkhorn OT graph still work well and beat baselines without this hard-coded trick?

  2. Most of the main results (like Table 4) are based on perfect human-annotated chunks. Can you provide the detailed evaluation results if the system uses a standard open-source PDF parser

4 comments

We thank the reviewer for their recognition of our work. We are looking forward to your feedback on whether our rebuttal addresses your concerns.

1. Cross Reference Boosting.

We have conducted the relevant experiments and reported the results in the global response for clarity. Overall, our method performs better with boosting, but the variant with only the graph and PageRank still outperforms all baselines. This supports our insight that the Sinkhorn graph and PageRank help address the modality gap, with cross-reference boosting providing an additional performance improvement.

2. Human annotated chunks.

We present an extension of Figure 3a that includes the state-of-the-art baselines. The experiments use GPT-5.2 as the backbone.

Methodtop-kSURGEColpaliDMAP
Performance on human annotated PDF53.057.134.141.6
Performance on auto-chunked PDF47.248.324.835.2

All methods show a significant drop in performance when switching to an automated PDF parsing pipeline. Our method remains the best-performing method.

Figure 3a shows the performance gap between chunks generated by a PDF parsing pipeline and those revised by human annotators. We would like to highlight two observations: 1. Our method still outperforms the baseline under these circumstances; and 2. As chunk quality improves through human revision, our method exhibits a steeper increase in performance.

3. Domain of Long documents.

We understand the reviewer’s concern and would like to make the following points:

  1. General-domain documents, such as news articles and books, usually contain more text than figures. Their figures typically supplement the text rather than provide critical complementary information. Questions about these documents can therefore usually be answered using only the text, which conflicts with our original goal of a true assessment of multimodality.
  2. In practice, previous work on multimodal long-document benchmarks typically combines technical reports, user manuals, financial documents, and academic papers [1][2]. We generally follow this approach when evaluating our framework.
  3. In addition, we evaluate all methods on MMLongBench, which contains magazine pages, leaflets, etc., and hence represents the more general-interest side of the document collection.

[1] LongDocURL: A Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating. Deng et al. ACL 2025.
[2] M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Framework. Shia et al. EMNLP 2025.

Dear Reviewer,

While the rebuttal is ending soon, we looking forward to your response to check whether our rebuttal addressed your concerns. We would be happy to make clarifications if you had concerns unresolved.

Best, Submission 12648 Authors

Thanks for the new results. For Q1, without the regex boosting, SURGE drops to 54.49%, which is only a little bit better than Top-k (52.95%). For Q2, the auto-chunking result is 48.3 vs Top-k 47.2. Only 1.1% gain in real-world setting is too small for such a complex method. Also, you did not really address my concern about why DMAP gets higher MRR and nDCG than SURGE. so I will keep my score as 2.

We thank you for your additional feedback and apologize if we overlooked some of your concerns. We would like to make a few additional points.

  1. We agree that Top-k is a simple baseline for comparison, but a complex model cannot necessarily compensate for the modality gap. As you observed, all the other graph-based baselines, such as ColPali and DMAP, performed significantly worse than Top-k. This is because the modality gap distorts inter-chunk relationships (line 46), causing these methods to fail. Our rationale for comparing SURGE with Top-k is not simply that “our method is more complex and hence it must be better than all simple methods.” We improved the performance of graph-based retrieval to the point where our method achieved the best results, which is itself a contribution.

  2. The statement “Without the regex boosting SURGE is only a little bit better than Top-k” is not a meaningful assessment. If the question is how important boosting is to the overall framework, we have shown that it is less important than the Sinkhorn graph or the PPR procedure. If the question is how well SURGE performs, we evaluate the framework as a whole, including this simple and useful module that is applicable to all datasets. There are many examples of researchers adding small, useful modules to a larger framework for an incremental improvement, such as constrained decoding in generative models. Such modules are not typically treated as separate from the main framework.

  3. We agree that the gain over Top-k is small on automatically parsed PDFs. However, the improvement over ColPali and DMAP is apparent. As we noted in the paper, the automated parsing pipeline still makes substantial errors on long documents, including “OCR errors, wrong boundaries, and poorly extracted tables or figures” (line 430). We hypothesize that an average accuracy of approximately 50% might therefore could be an upper bound for methods in this setting. Moreover, we explicitly identify this issue as a limitation and include the identification of “several open questions including the multimodal document segmentation bottleneck and call for dedicated research to address them” (line 68) among our core contributions.

  4. We apologize for overlooking the concern on retrieval metrics. In addition to the explanation and examples of alternative paths in Appendix C.3, another reason for SURGE’s lower scores on retrieval metrics is the evaluation setup. ColPali and DMAP operate at the page level, while SURGE operates solely at the chunk level. If page pp contains nn chunks c1,...,cnc_1, ..., c_n, and cic_i is the gold-standard chunk, SURGE must identify cic_i, while ColPali and DMAP need only retrieve pp. We recognized this issue but had no fair way to convert between the two evaluation setups, so we chose to report the original scores despite the disadvantage this poses for SURGE.

We hope these points address your additional concerns and that you will reconsider your evaluation of our manuscript.

Reviewer_QhyaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents a multimodal Retrieval-Augmented Generation (RAG) framework aimed at improving retrieval leveraging links among different modalities contained in a document (text, tables, and figures). The core idea is to build a similarity graph across chunks of different modalities at indexing time by propagating similarity scores. During retrieval, this association graph is leveraged to find related chunks across modalities that might otherwise be missed by standard retrieval. The authors evaluate their approach on question-answering tasks, reporting improvements over existing other state-of-the-art (SoA) retrieval frameworks.

Critical strengths
  • The idea of propagating similarity across related multimodal chunks to construct an association graph is an interesting and promising contribution to multimodal retrieval.
  • The experimental results demonstrate improved performance on QA benchmarks compared to several existing state-of-the-art retrieval frameworks.
Critical weaknesses
  • The paper claims to resolve the modality gap between text, tables, and figures. However, since all modalities are mapped using the same embedding model, the underlying modality gap persists during the initial similarity graph construction. My intuition is that the the cross-reference boosting mentioned at the end of Section 3.2, and not the sickhorn transport graph or the pagerank retrieval, might serve as the actual bridge between modalities. In that sense, the paper lacks an ablation study evaluating the method with and without this specific preprocessing step to validate this claim.
Questions for authors
  • The primary evaluation relies on human-annotated chunk boundaries. In a realistic setting (e.g., Figure 3a), the advantage of the proposed method diminishes significantly, showing minimal gain over simple Top-k retrieval. The paper lacks comparison against other SoA RAG baselines under this realistic setting.
  • The retrieval performance is evaluated only on FinanceBench. Incorporating the other benchmarks would be helpful to demonstrate the generalization capability of the proposed framework.
3 comments

We thank the reviewer for their valuable feedback. We hope the following rebuttal addresses the concerns.

1. Cross-reference Boosting.

We understand the reviewer’s concern regarding cross-reference boosting. We have conducted the corresponding ablation experiments and report the results in the global response for clarity. Cross-reference boosting improves SURGE’s performance, but the variant using only the Sinkhorn graph and PageRank still outperforms all competing baselines. These results support the contribution of our graph construction and PageRank-based retrieval to addressing the modality gap, with cross-reference boosting providing an additional performance improvement.

2. Chunk Quality.

We extend Figure 3a to include the state-of-the-art baselines, using GPT-5.2 as the answerer backbone.

Methodtop-kSURGEColpaliDMAP
Performance on human annotated PDF53.057.134.141.6
Performance on auto-chunked PDF47.248.324.835.2

All methods exhibit substantial performance declines when switching to an automated PDF parsing pipeline, while SURGE remains the best-performing method in both settings.

In addition, we would like to point out that, in our understanding, Figure 3a illustrates how chunk quality affects the performance of all models. In the long, multimodal setting, the existing automated chunking method struggles to produce useful chunks. Inspecting these chunks reveals highly disorganized content, including misaligned text caused by OCR merging footnotes with paragraphs, incorrectly extracted figures due to bounding-box detection failures, and cases in which even human annotators struggle to distinguish text from figures. All models perform worse under these conditions. However, we consider two points important: (1) our method still outperforms the baseline under these circumstances; and (2) as chunk quality improves (i.e., through human annotation to clean the chunks), our method exhibits a steeper increase in performance.

3. Retrieval Performance:

The following tables report retrieval performance on the CHIP and MMLongBench-MQ datasets.

MMLongBench-MQ:

MethodMRRR@5R@10nDCG@5nDCG@10
SURGE0.5160.6100.7130.4800.518
ColPali0.7300.7440.8340.6750.709
DMAP0.6880.7120.8410.6310.679

Chip:

MethodMRRR@5R@10nDCG@5nDCG@10
SURGE0.3690.4330.5770.3290.383
ColPali0.3300.3400.4420.2900.329
DMAP0.3910.3520.4470.3270.362

These results largely align with our observation on FinanceBench that SURGE does not always perform best on these metrics. This is because methods such as ColPali split documents at the page level, whereas SURGE retrieves content only at the chunk level. The metrics evaluate the former at the page level, while the latter must identify the correct chunk within the same page to receive full credit.

Thank you for the clarifications and the additional details. I am still not convinced about the actual contribution of the sickhorn transport graph or the pagerank retrieval to the resolve the modality gap as claimed by the authors. Results without the cross-reference boosting show that this simple trick is the main reason for the improvement of results over simple top-k RAG, and that the actual improvement of the sickhorn graph is minor (only slightly signifficant in the FinanceBench dataset). This factor, along with the fact that using real chunks has a larger impact on the proposed method than on top-k RAG raises signifficant concerns about the actual contribution of the sickhorn graph and its ability to resolve the modality gap.

Thank you for your frank response. It helps us understand the source of your concern, and we appreciate the opportunity to clarify our claim and the comparisons that support it.

1. We do not claim that SURGE resolves the modality gap. In the introduction, we identify “the modality gap and embedding granularity” as limitations of multimodal retrieval (p. 1, line 45). We describe SURGE as a method that “captures fine-grained inter-chunk relationships across modalities” (p. 2, line 54). The term we use for addressing modality gap is “compensate” (p. 3, line 141 and p. 5, line 244). We never claimed to "resolve" it. Reviewer zvEV also recognized this distinction, describing our approach as using “classic graph algorithms to bypass this problem using document structure.”

2. Reranking chunks cannot, by itself, resolve the modality gap. By modality gap, we mean that embeddings of different modalities occupy distinct regions of the embedding space, as illustrated in Appendix C.5 and Figure 5. Changing the retrieval order of chunks leaves their geometry unchanged. SURGE instead uses the Sinkhorn graph to create associations between chunks to retrieve related content across modalities. As Figure 3(b) shows, SURGE increases the retrieval of multimodal chunks from approximately 30% to over 50%.

3. Top-k RAG is a strong baseline, and SURGE is the only evaluated method that outperforms it in this setting. The modality gap can place chunks from different modalities far apart even when they are semantically similar. This makes it difficult to identify inter-chunk relationships during preprocessing, and is the very reason that none of GraphRAG, Colpali and DMAP are competitive enough to beat the simple Top-k RAG. The latter ranks each chunk directly by its similarity to the query, so it avoids this particular difficulty. Among the methods we evaluated, SURGE outperforms both ColPali and DMAP and is the only method that exceeds Top-k RAG in average accuracy.

4. The conclusion that boosting is SURGE’s main contribution is incorrect because it is based on the wrong comparison. Comparing Top-k RAG, SURGE without boosting, and full SURGE shows how much boosting adds to SURGE. It does not reflect that it "is the main reason for the improvement of results" To show that, we need to compare SURGE w/o boosting with ablations that had either the Sinkhorn graph or PageRank removed. The relevant comparisons are shown in Table 5. Removing either the Sinkhorn graph drops the accuracy by 26.3, and removing PageRank drops it by 8.9. Removing boosting, by comparison, reduces average accuracy by only 2.77 percentage points, from 57.26% to 54.49%. Even without boosting and takes the accuracy drop, SURGE w/o boosting still outperforms every competing baseline on all three datasets, including top-k.

As we also showed in our response to Reviewer 7yC8, adding explicit references to Top-k RAG lowers its average accuracy from 52.95% to 52.46%. Taken together, these experiments show that the Sinkhorn graph and PageRank are the main contributors to SURGE’s performance, with reference boosting providing an additional gain.

MethodMMLongBench-MQFinanceBenchCHIPAverage
Top-k RAG87/181 (48.1%)87/150 (58.0%)38/72 (52.8%)52.95%
Top-k RAG with reference expansion86/181 (47.5%)92/150 (61.3%)35/72 (48.6%)52.46%
SURGE92/181 (50.8%)96/150 (64.0%)41/72 (56.9%)57.26%
SURGE without boosting88/181 (48.6%)91/150 (60.7%)39/72 (54.2%)54.49%

In conclusion, we do not claim resolving modality gaps, demonstrated a performance advantage across all baselines in all condition even with boosting disabled, and show that boosting is not the primary contributor to the performance. However, we do realize that we could have stated the scope of our claim more clearly in the manuscript to reduce your confusion. We welcome your suggestions on how to clarify the motivation and will be very happy to incorporate them into the revision.

Reviewer_7yC8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes SURGE, a training-free retrieval method for question answering over a single long multimodal document (text, tables, figures).

Method:

  • Offline, SURGE solves an entropic OT problem with uniform marginals on the chunk-chunk cosine-similarity matrix (Sinkhorn, lambda=0.05).
  • An optional step sets similarity to 1 for regex-detected text-to-table/figure cross-references; this step is enabled in the main experiments (App. B.2).
  • The transport plan is row-normalized into a transition matrix.
  • At query time, Personalized PageRank (alpha=0.7) is seeded with L1-normalized query-chunk cosine similarities, and the top-5 chunks are given to a VLM.

Evaluation:

  • Datasets: MMLongBench-MQ (a filtered subset of MMLongBench-Doc), FinanceBench, and a new 6-document Chip dataset.
  • Baselines: Top-k, a GraphRAG-style scorer, ColPali (ColQwen2), and a re-implemented DMAP.
  • Six VLM backbones.

Reported results:

  • Best accuracy on all three datasets when each method takes its best of the six backbones (Table 4).
  • Retrieval metrics on FinanceBench close to the baselines.
  • Low query-time latency.
  • Ablations, plus analyses of chunking quality, retrieved modality composition, and backbone capability.

The appendix contains a PPR path-expansion analysis, a perturbation analysis, and a failure case.

Critical strengths
  1. Simple, training-free and cheap at query time. The graph is cached per document and the retrieval kernel takes 0.99 ms (Sec. 3.4, p.5; Table 2, p.6). Against the cleanest control, Top-k with the same Voyage encoder, SURGE wins 16 of the 18 dataset x backbone cells, with 1 tie and 1 loss (Table 8, p.20); its three-dataset average is higher for every backbone (+0.3 to +4.2 pp); and it is higher at k = 5/10/20 (Fig. 2, p.8). The gains are small, though, and the cells are not independent evidence, since all backbones read the same retrieved chunks.

  2. The paper reports its failures and limitations alongside the wins, which makes it useful to practitioners. It analyzes a severe failure case with the transport map (Chip Doc 3, 58.3 to 16.7; Table 10, p.23; App. C.6, p.24), states that DMAP leads on FinanceBench for 5 of 6 backbones (p.21), notes that the latency comparison is not hardware-normalized (App. C.1, p.19), and compares the modality gap of three encoders (Table 9, p.23).

Critical weaknesses
  1. Reported accuracies do not match the stated query counts. Scoring is binary per query (p.6) and Table 1 lists 60 MMLongBench-MQ queries, yet no method's MMLongBench-MQ column in Table 8 is consistent with 60 queries; most values (e.g., 48.1, 44.9, 50.3) are not multiples of 1/60. The Top-k and GraphRAG columns fit ~185 scoring units, ColPali ~182 and DMAP ~181, and SURGE's six values fit no plausible common denominator. On Chip, the Top-k, GraphRAG and SURGE values match 72 queries (and Table 10), while the ColPali and DMAP values (40.0, 45.6, 32.2, 26.7, ...) fit 90. In addition, Fig. 2 (MMLongBench, GPT-5.2, k = 5: ~43 vs ~37) disagrees with Tables 5/8 for the same setting (50.3 vs 48.1). It is therefore unclear whether all methods were evaluated on the same items, and how large the MMLongBench gain over Top-k really is.

  2. The statement that SURGE "achieves the best result on every dataset" (p.7) relies on representing each method by its best of six backbones (Table 4). Averaged over backbones (Table 8), SURGE beats Top-k by only ~2.6 pp overall (50.3 vs 47.8). On FinanceBench, DMAP is higher on average (58.6 vs 55.2), and SURGE's best-backbone win there (64.0 vs 63.3) amounts to one question. The conclusion's "across all three evaluation datasets and all backbone models tested" (p.9) does not hold per cell; for example, on Chip with Qwen3-VL-8B, SURGE scores 30.6 and ranks fourth. Table 8 also underlines SURGE as the best FinanceBench result for GPT-4o, Gemini 2.5 Pro and Gemini 2.5 Flash, although DMAP is higher for all three. No seeds, variances, confidence intervals or paired tests are reported. For Chip/GPT-5.2 (41 vs 38 correct), the per-document counts in Table 10 imply an exact McNemar p of about 0.63 even under the most favourable split.

  3. The proposed mechanism, a Sinkhorn-calibrated graph that surfaces non-text evidence missed by cosine retrieval, is not isolated.

Questions for authors
  1. How many scoring units and runs were used for each method x dataset x backbone? The MMLongBench-MQ accuracies are not consistent with 60 queries; they fit 185 units for Top-k/GraphRAG, 182 for ColPali and 181 for DMAP, and the ColPali/DMAP values on Chip fit 90 rather than 72. Were all methods evaluated on the same items with the same scoring? For MMLongBench/GPT-5.2/k = 5, which numbers are correct, Fig. 2 (43/37) or Tables 5/8 (50.3/48.1)?
  2. Was cross-reference boosting enabled in all reported SURGE results and in both ablation variants? What are the results without it, and what does Top-k with cross-reference/same-page expansion (no PPR) achieve?
  3. Is the 0.05 sparsification threshold applied to the transport plan or to W? Are self-pairs masked before Sinkhorn? How are negative cosines handled in Eq. 2? Please report the column-sum distribution of the final W.
  4. Which LLM is the judge, what fraction of its judgments was checked by humans, and what was the agreement?
  5. On what data were λ, α and the sparsification threshold chosen?
4 comments

We thank the reviewer for their constructive feedback. We hope the following clarifications address their concerns.

1. Query Counts.

We thank the reviewer for identifying the discrepancy. The correct count of MMLongBench queries is 185. However, parsing failures in ColPali and DMAP affected four queries. We therefore use a common set of 181 queries for the revised evaluation and will update the corresponding results in the manuscript, highlighting the corrections in red. These corrections do not substantially change the results, and our method retains its leading performance.

For MMLongBench-MQ with GPT-5.2 and k=5, the reported pair of 50.3/48.1 is correct. We will revise Figure 2 accordingly.

2. Cross-reference boosting.

We have conducted the requested experiments. Since all reviewers raised this question, we present the results in the global response for clarity. Overall, cross-reference boosting improves our method’s performance, while the variant using only the graph and PageRank still outperforms all baselines.

Cross-reference boosting operates on the Sinkhorn graph in our implementation. It therefore cannot be directly applied to the graph-independent Top-k baseline in its current form.

3. Parameters.

The sparsification threshold is applied to the transport plan before constructing (W). Self-pairs are not masked before Sinkhorn normalization, i.e. Sii=1S_{ii}=1.

In our implementation, we apply softmax normalization to the cosine similarities to ensure nonnegative values. Although high-dimensional vector similarities tend to concentrate near zero, we empirically observe that only approximately 0.7% of the cosine similarities are negative. We will clarify the softmax normalization step in the manuscript.

The column sums of the final (W) have a mean of 1.000 and a standard deviation of 0.481. Their distribution is strongly long-tailed, ranging from 0 to 20.500.

4. LLM judges

We use GPT-4o as the judge. All judgments were checked by human evaluators, with approximately 98–99% agreement. We observed that, for long questions containing multiple subquestions, the judge sometimes marked an answer as correct despite deficiencies in its responses to individual subquestions.

5. Method Performance.

Our claim is that SURGE achieves the best result on each dataset when comparing the best-performing backbone configuration for each method, and the best average performance across datasets for each backbone tested. We do not claim that SURGE outperforms every baseline in every individual dataset–backbone combination.

Specifically, our results show that: (1) SURGE achieves the highest performance on each dataset with an appropriate backbone choice; and (2) for each tested backbone, SURGE achieves the highest average performance across the evaluated long, multimodal document QA datasets. We apologize for the ambiguity in the original wording and will revise the relevant statements to make this distinction explicit.

We included Table 4 rather than Table 8 in the main paper because of space constraints. In the revised manuscript, we will replace Table 4 with Table 8 to present the complete results. Finally, Table 10 is part of our failure-case analysis and should be interpreted in that context.

Thank you for the clarifications and the new boosting ablation. Three points remain:

  1. With the stated uniform marginals, each transport entry satisfies πij≤1/nπ_{ij} ≤ 1/n. Applying a threshold of 0.05 to the raw plan would therefore remove every entry for n > 20. Is the plan rescaled before thresholding, and how does the minimum-two-edge fallback operate?
  2. Without boosting, the gains over Top-k are 1, 4, and 1 correct answers across the three datasets. Could you provide the requested Top-k plus explicit-reference expansion control under the same five-chunk budget? This would help distinguish the benefit of reference links from graph propagation.
  3. The new table reports 92/181 = 50.8%, while the individual response identifies 50.3% as correct. Could you clarify which results correspond to the original and revised evaluation sets?

We thank you for your additional feedback. We have modified the manuscript, and hope the following points address your concerns:

  1. Yes, the raw Sinkhorn plan is rescaled by its global maximum before applying the threshold: π~=πmax⁡ijπij\tilde{\pi}=\frac{\pi}{\max_{ij}\pi_{ij}}. The threshold of 0.05 is a relative threshold selected empirically. We retain at least two entries per row. We first apply the threshold; if fewer than two entries pass it, we additionally retain the strongest remaining edges.

  2. We did not provide the Top-k plus explicit-reference expansion experiment in the previous rebuttal because our boosting operation cannot be directly applied to Top-k. Specifically, explicit-reference boosting operates on the similarity graph and sets the weights of edges between explicitly connected chunks to 1. Top-k does not use a graph; it simply retrieves the chunks with the highest query similarity and generates an answer.

    However, we provide a similar experiment below. We retrieve the top-k chunks and insert any explicitly cross-referenced chunks immediately after their corresponding retrieved chunk, and retain the first k chunks in the resulting sequence. For example, suppose the original Top-k retrieval with k=5k=5 returns:

    Figure 1 — high query similarity
    Figure 2 — high query similarity
    Figure 3 — high query similarity
    Figure 4 — high query similarity
    Figure 5 — high query similarity
    

    Top-k plus explicit-reference expansion will produce:

    Figure 1 — high query similarity
    Figure 2 — high query similarity
    Text 1 — connected to Figure 2
    Table 1 — connected to Figure 2
    Figure 3 — high query similarity
    
    Figure 4 — excluded
    Figure 5 — excluded
    

    We report this method’s performance alongside the relevant results from the global response:

MethodMMLongBench-MQFinanceBenchCHIPAvg.
Top-k RAG87/181 (48.1%)87/150 (58.0%)38/72 (52.8%)52.95%
Top-k RAG w/ boosting86/181 (47.5%)92/150 (61.3%)35/72 (48.6%)52.46%
SURGE92/181 (50.8%)96/150 (64.0%)41/72 (56.9%)57.26%
SURGE w/o boosting88/181 (48.6%)91/150 (60.7%)39/72 (54.2%)54.49%

The boosting method does not provide a significant positive impact on Top-k RAG. It reduces performance on MMLongBench-MQ and CHIP and improves performance only on FinanceBench, where it still underperforms SURGE with boosting. In our view, this is because boosting has limited impact when used solely to link explicit references, while its effectiveness also benefits from the Sinkhorn graph construction.

  1. The previously reported 50.3% corresponds to 93/185. At that time, we had not identified the parsing issues affecting ColPali and DMAP. We retained this value in the first rebuttal to remain consistent with the originally reported results. In the next version of the manuscript, we will update it to 50.8% using the 181-query subset. We would like to thank you again for your careful examination of our paper and will incorporate your feedback into the revised manuscript.

Thank you for clarifying the normalization and evaluation counts, and for adding the reference-expansion baseline under the same retrieval budget. These address my main concerns, and I will raise my score to weak accept. Please incorporate these clarifications and the new results into the revised manuscript.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response from the Authors

We thank the reviewers for their thoughtful comments and feedback. In particular, we appreciate the recognition of our approach of using document structure and graph-based similarity propagation to address the modality gap, as well as its training-free design and low query-time cost. We also appreciate their positive assessment of our downstream QA evaluation, performance improvements over existing retrieval methods, and transparent analysis of failure cases and limitations.

All reviewers requested an ablation of cross-reference boosting. We therefore evaluate SURGE with boosting disabled across all three datasets. The results using the GPT-5.2 backbone are presented below:

MethodMMLongBench-MQFinanceBenchCHIPAvg.
Top-k RAG87/181 (48.1%)87/150 (58.0%)38/72 (52.8%)52.95%
GraphRAG63/181 (34.8%)85/150 (56.7%)36/72 (50.0%)47.16%
ColPali43/181 (23.8%)57/150 (38.0%)29/72 (40.3%)34.01%
DMAP52/181 (28.7%)84/150 (56.0%)30/72 (41.7%)42.13%
SURGE92/181 (50.8%)96/150 (64.0%)41/72 (56.9%)57.26%
SURGE w/o boosting88/181 (48.6%)91/150 (60.7%)39/72 (54.2%)54.49%

Cross-reference boosting improves SURGE’s performance across all three datasets by providing explicit information about relationships between chunks. However, even without boosting, our graph construction and PageRank-based retrieval outperform all competing baselines on every dataset, achieving the second-highest average accuracy after the full SURGE method.

Dear ACs and Reviewers,

We thank the reviewers for their feedback. While the rebuttal period approaches to its end, we use this global post to summarize the strengths and weaknesses listed by the reviewers, and our corresponding clarification.

Strengths

1. Addressing the modality gap by document structure. Reviewer zvEV appreciated our use of document structure to retrieve related evidence across modalities. Reviewer Qhya highlighted the idea of propagating similarity across multimodal chunks through an association graph.

2. Efficient, training-free retrieval. Reviewers zvEV and 7yC8 recognized the practical benefit of constructing the Sinkhorn graph offline and reusing it across queries. At query time, the PPR retrieval kernel takes ~0.99 ms.

3. Outstanding performance on downstream answer accuracy. Reviewer zvEV appreciated evaluating retrieval through the VLM’s final answer accuracy, and Reviewer Qhya noted the improvements over existing retrieval frameworks. Reviewer 7yC8 highlighted that SURGE outperformed Top-k with the same encoder in 16 of 18 dataset–backbone combinations and appreciated our analysis of failure cases.

Questions and Our Responses

1. Contributions of cross-reference boosting (all reviewers). All three reviewers requested an ablation of cross-reference boosting. With GPT-5.2, removing boosting reduces average QA accuracy by 2.77 percentage points (see the Global Response). SURGE without boosting still outperforms every competing baseline on all three datasets. The component ablations in Table 5 show considerably larger drops when we remove the Sinkhorn graph (26.3 points) or PageRank (8.9 points). We also tested explicit-reference expansion with Top-k RAG (see our second post in Rebuttals to Reviewer 7yC8). It improves accuracy only on FinanceBench but decreases the performance overall. We showed that cross-reference boosting is not the main contributor of our method's performance, and is not universally beneficial to all methods.

2. The modality-gap claim and retrieval performance. Reviewer Qhya states that SURGE does not resolve the modality gap. We clarify that we do not claim to do so, and all reranking methods are unable to do so by definition, because changing the chunk order does not modify the underlying embedding geometry (see our second post in Rebuttals to Reviewer Qhya). Our method uses relationships between chunks to compensate for the modality gap during retrieval, as stated in the manuscript. We also reported additional performance on the retrieval metrics like MRR, nDCG, and recall, as requested (see our first post in Rebuttals to Reviewer Qhya).

3. Dependence on chunk quality and choice of document domains. Reviewer zvEV and Qhya requested a detailed evaluation on frameworks using an auto PDF parser instead of our human-annotated chunks. We extended Figure 3(a) to include ColPali and DMAP. With automatically parsed chunks (see our Rebuttal to Reviewer zvEV). Human revision improves all four methods, with SURGE benefits more. We also clarified our choice of document domains: technical and financial documents contain questions that require complementary evidence from text, figures, and tables, while MMLongBench-MQ broadens the evaluation beyond these specialized collections (see our Rebuttal to Reviewer zvEV).

4. Query counts, evaluation protocol, and performance claims. Reviewer 7yC8 pointed out that we had discrepancies in query results. We corrected the MMLongBench-MQ query count. The original set contained 185 queries, and parsing failures affecting four queries led us to report results on a common subset of 181. We have updated Figure 2 and all corresponding tables accordingly (see the Updated Manuscript). We also clarified that our dataset-level claim compares each method’s best backbone configuration, while our backbone-level claim compares average accuracy across datasets. Finally, human evaluators checked all GPT-4o judgments and agreed with approximately 98–99% of them.

We still consider this paper valuable to the community. It provides a lightweight, training-free way to bypass the modality gap problem in long-context retrieval, with impressive downstream VLM performance. We also contribute to the community by releasing the datasets and corresponding human-annotated benchmarks. Adequate modifications have been and will be made to reflect reviewers' feedback from this rebuttal period. We thank all reviewers and ACs again for their valuable effort in reviewing and improving our manuscript. We still look forward to the reviewers' additional feedback on whether we resolved their concerns to their end.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.