acceptodds
Under review as a conference paper at ICLR 2027

VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrieval

Abstract

Visually rich documents encode relevance through language, layout, structured visual elements, and corpus context, yet retrieval is typically evaluated by one-shot query–page matching. Agentic-search benchmarks usually score downstream question answering or report generation, leaving document ranking under iterative evidence acquisition underexplored. We introduce VisDocAgentBench, a closed-corpus benchmark comparing static and agentic retrieval under a shared ranked-output contract. It contains 2,375 pages from 100 documents and 120 unique-target queries balanced across direct, one-bridge, and two-bridge evidence structures. Relation-preserving construction yields semantic, relational, and visual queries, followed by full-document review and hard-negative validation. A strong late-interaction visual retriever reaches 97.50% Recall@1 on direct items but 2.50% on two-bridge items, exposing the limits of query–target matching when relevance depends on corpus context. Agents recover much of this loss, but planner choice and retrieval representation remain decisive. Every planner performs better with visual retrieval, whose best R@1 reaches 67.50% versus 37.50% for OCR-text. Ablations identify iterative search and page inspection as consequential capabilities, and providing the complete support context improves ranking on both routes. Trace analysis localizes the remaining losses to target discovery, candidate examination, and evidence-role integration. These findings motivate retrieval agents that combine modality-preserving discovery with evidence-directed verification.

Primary area:
datasets and benchmarks
Keywords:
Visually Rich Document Retrieval, Agentic Retrieval, Multimodal Information Retrieval, Multi-hop Reasoning, Tool-Using Agents, Benchmarking and Evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PQ54OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper explores the research question of document ranking under iterative evidence acquisition. It proposes a data curation pipeline that compose directed evidence paths from a visual corpus. This results in a self-contained benchmark to measure both static and agentic retrieval. Comprehensive experiments are done to evaluate the commonly used retrieval methods and agentic models. Ablations and analysis reveal key insights: e.g., a strong planner can recover the failures of static ranking, and direct page inspection is important for agentic retrieval.

Critical strengths
  1. The benchmark construction pipeline is well-designed. And the resulting benchmark could be useful for the community.
  2. The experiments are comprehensive, involving commonly used retrieval methods, closed-source and open-source models. The analysis provides useful insights.
Critical weaknesses
  1. The visual-vs-OCR comparison does not isolate the effect of modality. The agentic visual route uses Qwen3-VL-Embedding-8B, while OCR-text search uses Qwen3-Embedding-8B. Both use dense retrieval, but that does not guarantee comparable retrieval quality. The OCR dense model performs worse than BM25, yet this weaker OCR dense retriever is the one deployed in the agentic experiments.
Questions for authors
  1. I noticed that the input of search tools is text. Would you consider image search (image to image/image to text search) as useful for solving your benchmark?
3 comments
AuthorsOpenReview ↗

Response to Reviewer PQ54

We thank the reviewer for the thoughtful feedback and address the concerns below. We apologize for the delayed response, as the dataset expansion and human review required substantial time.

W1. The visual-vs-OCR comparison does not isolate the effect of modality. The agentic visual route uses Qwen3-VL-Embedding-8B, while OCR-text search uses Qwen3-Embedding-8B. Both use dense retrieval, but that does not guarantee comparable retrieval quality. The OCR dense model performs worse than BM25, yet this weaker OCR dense retriever is the one deployed in the agentic experiments.

We ran three comparisons to distinguish OCR retriever quality from the contribution of visual representations. All rows report full GPT-5.6-sol agents on the same 120 queries and corpus with the same 12-action budget. Scores are percentages.

The OCR retrieval comparison replaces Qwen3-Embedding-8B with BM25 over the same OCR text, keeping the other OCR-text tools unchanged. The dense representation comparison instead fixes the encoder at Qwen3-VL-Embedding-8B and compares indices over page images and complete OCR text. The BM25 fusion comparison adds the same BM25 branch to each of those dense indices, with identical fusion rules and tools. The two original agent results with dense retrieval are from Table 1 of the original submission.

ComparisonSearch backendR@1MRR@10
OCR retrievalQwen3-Embedding-8B, OCR36.6738.40
OCR retrievalBM25, OCR64.1767.36
Dense representationQwen3-VL-Embedding-8B, page images61.6763.58
Dense representationQwen3-VL-Embedding-8B, OCR42.5045.84
BM25 fusionBM25 + Qwen3-VL-Embedding-8B, page images66.6770.89
BM25 fusionBM25 + Qwen3-VL-Embedding-8B, OCR62.5066.42

The OCR retrieval pair shows that a stronger text backend substantially improves the OCR agent. In the dense representation pair, the image index exceeds the OCR index by 19.17 R@1 points with the encoder held fixed. The BM25 fusion pair retains a visual advantage of 4.17 R@1 and 4.47 MRR@10 points with the encoder, lexical retrieval, fusion rule, and tools held fixed.

These comparisons support our main conclusion that preserving visual structure improves overall retrieval, even when the encoder and lexical retrieval are held fixed. We have incorporated these comparisons into Section 6.1 and Appendix E.5 (Table 18) of the revised manuscript, with the experimental settings in Appendix D.7.

Q1. I noticed that the input of search tools is text. Would you consider image search (image to image/image to text search) as useful for solving your benchmark?

Yes, we implemented and evaluated an additional tool for image queries. The planner can use an already observed page or crop, optionally with text, to search the same page image index. Search with text queries remains available, and each image search consumes one of the same 12 actions. We evaluated this condition on the same 120 queries, using the visual agent from the original submission (Table 1) as the baseline.

ConditionR@1MRR@10
Standard visual agent61.6763.58
With image search65.0067.14

We also compared mean inference resources per query over the 119 queries with valid rankings in both conditions:

ResourceStandard visual agentWith image search
Model calls8.348.77
Pages viewed25.2228.18
Input tokens177,056207,808
Output tokens2,2502,948

The planner uses image search in 69/120 episodes, newly surfacing the target in 12. Its overall R@1 improvement is modest at 3.33 percentage points, with higher model usage and page exposure. Image search also uses actions from the same budget as other searches and verification. Image queries provide complementary retrieval clues, while the standard visual agent uses fewer model calls and tokens. We have added the settings to Appendix D.7 and the retrieval and resource comparisons to Appendix E.6 (Table 20) in the revised manuscript.

I thank the reviewers for their comprehensive response. The experiments are comprehensive. I would maintain my score of 3.

Thank you for your continued recognition!

Reviewer_364yOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VisDocAgentBench, a closed-corpus benchmark for visually rich document retrieval. The bench contains 2,375 pages from 100 scientific papers across 10 topics and 120 queries. For each query, the goal is to retrieve a unique target page. The authors use a VLM to extract query anchors, semantic bridges, visual target descriptors, and directed intra-page relations, then construct evidence paths through verified cross-page semantic alignments. The final queries hide the bridge identities and cover three evidence structures: direct, one-bridge, and two-bridge.

The authors compare static retrievers and agent retrievers using visual and OCR-text retrieval routes. A strong late-interaction visual retriever performs well on direct queries but drops sharply on bridged queries. Agents can improve performance, but results vary across planners. Ablations and trace analysis with GPT-5.6-sol highlight the importance of iterative search and page inspection.

Critical strengths

S1. The construction process of bench is clear, with quality checks throughout.

Figure 2 clearly presents the bench construction pipeline. The authors validate the extracted descriptors and directed relations, verify cross-page semantic alignments, and check the resulting paths for semantic coherence, evidence necessity, and visual necessity. Full-document human review and independent AI scoring provide further checks, followed by document-wide and hard-negative validation of the final queries. These checks at successive stages are a strength of the bench construction.

S2. The experiments cover a broad set of baselines and provide useful analysis.

The authors evaluate several static retrievers and agent retrievers on both visual and OCR-text retrieval routes. The results show a clear performance gap between direct and bridged queries. Ablations and the support-provided intervention with GPT-5.6-sol help explain the roles of iterative search, page inspection, and support context. The trace analysis also shows where targets are lost between discovery, examination, and final ranking.

Critical weaknesses

W1. The actual needs of the bench and contribution over prior work need stronger justification. 

W1.1. Some queries feel unnatural.  I agree with the motivation for retrieval that requires cross-page evidence. However, the query in Figure 1 (as shown below) is difficult to understand. 

Example query in Figure 1: "Find the grouped example-strip page for the benchmark contrasted with one containing a brightness example labeled by an action that minimizes the display level" 

It is unclear whether such queries reflect real user needs, or how much of their difficulty comes from cross-page reasoning rather than awkward wording. 

W1.2. The added value over existing evaluations remains unclear.  Recall@k and MRR@10 depend only on the rank of the unique target page; support pages and their ordering are not scored. ViDoSeek[1] already includes multi-hop visual-document questions and directly evaluates retrieval. The paper should better explain what its explicit bridge structures and level-wise evaluation reveal beyond the existing settings. 

W1.3. The single-target setting limits the evaluation scope.  The benchmark uses multiple support pages to locate one target page, but does not evaluate tasks where several pages jointly satisfy the information need. For example, retrieval may follow two branches, q→E1→E2q\rightarrow E_1\rightarrow E_2 and q→E3→E4q\rightarrow E_3\rightarrow E_4, before combining E2E_2 and E4E_4 to answer the question. The current metrics do not measure evidence completeness or answer quality in such cases. 

W2. Some conclusions need stronger statistical and annotation evidence. 

W2.1. The small sample size limits fine-grained conclusions. 

The benchmark has only 120 queries, with 40 per level and six cross-document cases. The cropping ablation changes overall R@1 by just three successful queries in either direction. Findings about tool effects and cross-document difficulty therefore need further validation. 

W2.2. Annotation reliability and target uniqueness need stronger validation.  The three authors jointly assign one human score, leaving agreement between independent annotators unreported. Uniqueness checks cover the involved source documents and ten candidates from one retriever, which may miss valid alternatives elsewhere in the corpus.

[1] Wang Q, Ding R, Chen Z, et al. Vidorag: Visual document retrieval-augmented generation via dynamic iterative reasoning agents[C]//Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025: 9124-9145.

Questions for authors

Q1. Were the final queries evaluated by readers who were not involved in the construction progress? Could you rewrite them more naturally (e.g., using shorter sentences or a recollection such as “I remember…”) while preserving the evidence requirements, and check whether the main findings still hold?

Q2. The paper emphasizes ranked outputs, but Recall@k and MRR@10 depend only on the position of the gold page. What does this ranking protocol add beyond standard target-page retrieval evaluation, and how does it advance prior work such as ViDoSeek? 

Q3. Many practical tasks require combining evidence from different pages. Could the bench be extended to retrieve more than one page to finish the tasks? 

Q4. Could you expand the query set to a larger corpus (e.g. maybe about 500 queries) and assess whether the reported trends hold?

Q5. Could multiple human annotators independently review the final queries and target labels, with agreement reported using an appropriate measure, such as Fleiss’ κκ for categorical judgments? Since the current uniqueness check retrieves ten candidates from the full corpus using one retriever, could you broaden the candidate pool or use additional retrievers to check for valid alternatives that this search may have missed?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 364y (1/4)

We thank the reviewer for the thoughtful feedback and address the concerns below. We apologize for the delayed response, as the dataset expansion and human review required substantial time.

W1.1. Some queries feel unnatural. I agree with the motivation for retrieval that requires cross-page evidence. However, the query in Figure 1 (as shown below) is difficult to understand.

Example query in Figure 1: "Find the grouped example-strip page for the benchmark contrasted with one containing a brightness example labeled by an action that minimizes the display level"

It is unclear whether such queries reflect real user needs, or how much of their difficulty comes from cross-page reasoning rather than awkward wording.

Q1. Were the final queries evaluated by readers who were not involved in the construction progress? Could you rewrite them more naturally (e.g., using shorter sentences or a recollection such as “I remember…”) while preserving the evidence requirements, and check whether the main findings still hold?

The intended information need is contextual lookup: a user remembers an example or relationship in one document and wants the corresponding visual result elsewhere. In Figure 1 of the original submission, the brightness example identifies one benchmark, and the contrast relation identifies the benchmark whose example page is requested. The evidence dependency carries the task. Language refinement should make that dependency easier to read.

Three readers who were not involved in benchmark construction evaluated all 120 requests on a 1–5 naturalness/clarity scale. Fleiss' κ measures agreement among readers beyond chance, with 0 indicating chance agreement and 1 perfect agreement. The results are:

MeasureValue
Mean score across readers and requests4.00 / 5
All three scores identical55/120 (45.83%)
Maximum score difference at most one116/120 (96.67%)
Fleiss' κ over the five score categories0.440

We finalized 57 paraphrases that preserve meaning, checked their targets and relational requirements, and evaluated both full GPT-5.6-sol routes on the original and paraphrased versions. The wording condition with 120 queries adds the 63 unchanged requests. Retrieval and coverage scores below are percentages. We also evaluated the static retrievers under the same wording condition and report the strongest static retriever, Nemotron, by evidence level. The results are:

SystemWordingQueriesR@1MRR@10
Visual agentOriginal5749.1251.48
Visual agentParaphrased5761.4064.02
OCR-text agentOriginal5731.5833.63
OCR-text agentParaphrased5726.3229.24
Visual agentWording condition12067.5069.54
OCR-text agentWording condition12034.1736.31
Static Qwen3-VL-Embedding-8B, page imagesWording condition12020.8328.09
Static Qwen3-Embedding-8B, OCRWording condition1201.676.26
Static BM25Wording condition1205.8311.94
Static NemotronWording condition12040.0049.39
Static Nemotron (L1)Wording condition4097.5098.12
Static Nemotron (L2)Wording condition4020.0036.11
Static Nemotron (L3)Wording condition402.5013.93

After paraphrasing, Nemotron still achieves 97.50% R@1 on L1 but only 2.50% on L3, indicating that clearer wording alone does not resolve the difficulty of retrieval requiring cross-page evidence. On the same 57 requests, the visual agent remains ahead of OCR-text under both the original wording (49.12% versus 31.58%) and the paraphrased wording (61.40% versus 26.32%). Both the difficulty of contextual retrieval and the visual agent's advantage persist under more natural phrasing. We have added the rating and wording robustness analysis to Appendix C.6 (Tables 9–10) in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 364y (2/4)

W1.2. The added value over existing evaluations remains unclear. Recall@k and MRR@10 depend only on the rank of the unique target page; support pages and their ordering are not scored. ViDoSeek[1] already includes multi-hop visual-document questions and directly evaluates retrieval. The paper should better explain what its explicit bridge structures and level-wise evaluation reveal beyond the existing settings.

Q2. The paper emphasizes ranked outputs, but Recall@k and MRR@10 depend only on the position of the gold page. What does this ranking protocol add beyond standard target-page retrieval evaluation, and how does it advance prior work such as ViDoSeek?

The added value is the explicit evidence dependency and its diagnosis. ViDoSeek includes multi-hop visual document questions and retrieval evaluation. The original submission already encodes these dependencies through local relations and verified alignments across pages (Sections 4.2–4.3), distinguishing direct requests and those with one or two bridges under the same page ranking endpoint.

We audited the full GPT-5.6-sol traces to measure the fraction of annotated supports found and examined, and the fraction of requests with all supports examined. Results for the visual and OCR-text agents are shown below. Each level contains 40 queries, and the Complete column also gives the query count:

Search representationLevelFoundExaminedComplete
VisualL265.0052.5052.50 (21/40)
VisualL382.5062.5047.50 (19/40)
OCR-TextL260.0037.5037.50 (15/40)
OCR-TextL371.2545.0030.00 (12/40)

For visual L3, support coverage falls from 82.50% found to 62.50% examined, and only 47.50% of requests have all annotated supports examined. Finding or examining just one of the two L3 supports contributes to average coverage, while Complete requires both to be examined. Higher average coverage can therefore coexist with lower completeness than on L2. This exposes where annotated evidence is lost between search and examination, beyond what the final target's Recall/MRR reveals. Our explicit bridge annotations thus complement the retrieval evaluation already present in work such as ViDoSeek by making intermediate evidence coverage and completeness measurable alongside the final target ranking. We have clarified the comparison with ViDoSeek in Section 2.3 of the revised manuscript and added this analysis to Section 6.3 and Appendix F.1 (Table 24).

AuthorsOpenReview ↗

Response to Reviewer 364y (3/4)

W1.3. The single-target setting limits the evaluation scope. The benchmark uses multiple support pages to locate one target page, but does not evaluate tasks where several pages jointly satisfy the information need. For example, retrieval may follow two branches, q→E1→E2q \rightarrow E_1 \rightarrow E_2 and q→E3→E4q \rightarrow E_3 \rightarrow E_4, before combining E2E_2 and E4E_4 to answer the question. The current metrics do not measure evidence completeness or answer quality in such cases.

Q3. Many practical tasks require combining evidence from different pages. Could the bench be extended to retrieve more than one page to finish the tasks?

Yes. We extended the evaluation to ten requests whose two branches address one information need and lead to two required target pages over the same corpus of 2,375 pages. The set contains 20 distinct target pages, with eight requests spanning documents and two staying within one document. For example, one request traces a memory method's model and benchmark choices to retrieve an accuracy–compute plot and a benchmark ablation table. We checked the branch relations, targets, plausible alternatives, and natural wording before freezing the requests.

The comparison uses both full GPT-5.6-sol agents and the static Nemotron retriever. The agents retain their tools, model settings, 12-action budget, and ranking of ten pages, with the task instruction updated to prioritize both required targets. Target Recall@kk averages the fraction of each request's two targets returned within the first kk ranks. Complete@kk is the proportion of requests with both targets returned there. All scores below are percentages.

SystemTarget Recall@3Target Recall@5Target Recall@10Complete@3Complete@5Complete@10
Visual agent45.0050.0050.0020.0020.0020.00
OCR-text agent30.0030.0050.0010.0010.0040.00
Static Nemotron10.0015.0025.000.0010.0020.00

The visual agent leads in both Target Recall and Complete within the top three and five ranks, showing that its advantage in early ranking extends to multiple required pages. Both agents also double Target Recall@10 relative to Nemotron. The extension directly evaluates joint retrieval of the pages required by a branched information need, with visual search placing more required evidence near the head of the ranking. We have added the settings, metrics, and results to Appendix E.8 (Table 23) of the revised manuscript, with a summary in Section 6.1.

W2.1. The small sample size limits fine-grained conclusions.

The benchmark has only 120 queries, with 40 per level and six cross-document cases. The cropping ablation changes overall R@1 by just three successful queries in either direction. Findings about tool effects and cross-document difficulty therefore need further validation.

The concern about sample size is whether a few individual queries drive the tool conclusions. We address this by quantifying how the inspection and cropping differences already reported in the original submission (Table 2 and Appendix D.5) vary when the same 120 queries are resampled.

The table reports the full agent's R@1 minus that of the corresponding tool ablation, in percentage points. Positive values favor retaining the tool. The 95% confidence intervals (CI) estimate uncertainty in these differences using 10,000 paired bootstrap draws. Query CI resamples queries within each level, and Document CI resamples groups sharing source documents. Each draw uses the same sampled queries for both settings. The interval endpoints are the 2.5th and 97.5th percentiles of the resampled differences.

ContrastRouteDifferenceQuery 95% CIDocument 95% CI
Full − no page inspectionVisual+15.83[7.50, 24.17][7.63, 24.63]
Full − no page inspectionOCR-Text+21.67[14.17, 30.00][14.37, 29.49]
Full − no regional cropVisual−2.50[−10.00, 5.00][−9.48, 5.98]
Full − no regional cropOCR-Text+2.50[−4.17, 10.00][−5.37, 10.92]

Page inspection improves R@1 on both routes, with both CI types entirely above zero. Crop effects are small and differ between routes, with both positive and negative effects inside the intervals. This matches the limited, route-dependent effects discussed in Section 6.2 of the original submission. We have added a brief CI explanation in Section 5.1, the full definitions and results in Appendix F.3 (Table 27), and their interpretation alongside the tool results in Section 6.2 of the revised manuscript.

Our response to Q4 below complements this analysis with 36 independent additional queries, expanding the cross-document L3 group from six to 30. The results on these additional queries support our main conclusions (Section 6.1 and Appendix E.7, Tables 21–22, in the revised manuscript).

Reviewer_DwuCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper claims that visuallly rich documents consists of multiple artifacts and should be evaluated beyond a single query-page matching strategy. Traditional visual retrieval benchmarks fails capturing this feature and this paper construct VisDocAgentBench to bridge the gap. 120 problems in the benchmark are asking either static retrieval model or agentic retrieval models to find the correct page of the figure or table in a specific paper given the descriptions of the figure. There are 3 levels of the quetions, where quetions of level 1 provides direct descriptions, and later levels requires hidden informaiton (in my mind, similar to multi-hop retrieval) before get the final answer. Results show static retrieval fails and agentic retreival can perform much better on VisDocAgentBench.

Critical strengths
  1. The paper identifies an important limitation of query–page matching: relevant pages may only be identifiable through evidence elsewhere in the corpus. This task definition is good.
  2. The multi-hop questions has further revealed the limitation of the statis retrieval. For me I think these mutli-hop questions can be very useful.
Critical weaknesses
  1. The comparison with static retrieval is not fully convincing to me. The agent has an additional planner model, which can rewrite queries and read pages. So the improvement may not all come from multi-round search. The no-iterative-search ablation helps, but I still have some doubts about whether the benchmark measures multi-round search specifically.
  2. The writing need to be improved. Many subsections only have 1 paragraph and not speaking clearly.
Questions for authors
  1. Could you use the same planner to rewrite the query once, retrieve once, and then read and rank the results? I would like to see how much the full agent improves over this baseline.
  2. Is section 4 AI generated contents? Many small subsections and the reading is not fluent at all.
2 comments
AuthorsOpenReview ↗

Response to Reviewer DwuC

We thank the reviewer for the thoughtful feedback and address the concerns below. We apologize for the delayed response, as the dataset expansion and human review required substantial time.

W1. The comparison with static retrieval is not fully convincing to me. The agent has an additional planner model, which can rewrite queries and read pages. So the improvement may not all come from multi-round search. The no-iterative-search ablation helps, but I still have some doubts about whether the benchmark measures multi-round search specifically.

Q1. Could you use the same planner to rewrite the query once, retrieve once, and then read and rank the results? I would like to see how much the full agent improves over this baseline.

To test whether iterative search adds value beyond initial query rewriting and page verification, we evaluated a pipeline using the same GPT-5.6-sol planner. Before seeing the corpus, it rewrites the request once and retrieves 30 candidates in a single call. It reads their page images and OCR text and produces a ranking over this fixed candidate set. Both routes use the same 120 queries and corpus as the full agent. Retrieval scores below are percentages.

RouteConfigurationR@1R@3R@5R@10MRR@10
VisualFull agent61.6765.0065.8368.3363.58
VisualOne rewrite, fixed search50.0052.5053.3354.1751.46
OCR-TextFull agent36.6738.3340.8344.1738.40
OCR-TextOne rewrite, fixed search20.8326.6726.6727.5023.56

The full agent scores higher on every reported metric, with R@1 improvements of 11.67 points on Visual and 15.83 on OCR-text. The pipeline with one initial rewrite provides 30 page observations and 12 planner calls, exceeding the full agent mean page exposure and call count on both routes reported in the original submission (Appendix D.6, Table 11). The full agent therefore improves retrieval despite fewer page observations and calls on average. This supports the value of adapting subsequent searches to observed evidence beyond initial rewriting and page verification.

We have added this comparison to Section 6.2 (Table 2) of the revised manuscript, with results by level in Appendix E.3 (Table 16) and settings in Appendix D.7.

W2. The writing need to be improved. Many subsections only have 1 paragraph and not speaking clearly.

Q2. Is section 4 AI generated contents? Many small subsections and the reading is not fluent at all.

The initial manuscript, including Section 4, was written by the human authors. It was not generated by AI. AI tools provided revision suggestions, which the authors reviewed and decided whether to adopt. This role in suggesting revisions was already disclosed in the AI use statement of the original submission.

We organized Section 4 around the construction stages shown in Figure 2, distinguishing their outputs and quality checks so that readers can follow how source pages become validated retrieval queries. Sections 4.1–4.3 correspond to stages 1–3 in Figure 2, respectively. Within stage 4, Section 4.4 describes path review and selection in step (a), while Section 4.5 covers query authoring and target validation in steps (b) and (c). We separate path review from query validation because a coherent path must still be expressed as a natural request with a unique target.

To improve continuity in the revised manuscript, we added a brief roadmap linking the subsections to Figure 2 at the start of Section 4 and explicit transitions into Sections 4.3 and 4.5. These targeted changes address the concern about readability and clarify how the subsections follow the construction workflow in Figure 2. We thank the reviewer for the constructive suggestions.

Dear Reviewer DwuC,

As the author-reviewer discussion period is drawing to a close, we would greatly appreciate it if you could review our responses and share your current assessment of the submission. If our responses have addressed your concerns, we would also be grateful if you could update your score accordingly.

Thank you very much for your time and consideration.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision Summary

In the revised manuscript, additions are highlighted in blue and deletions are shown in red with strikethrough.

  1. Added a baseline using the same planner for one initial query rewrite followed by fixed retrieval and reranking (Section 6.2, Table 2, Appendices D.7 and E.3, Table 16). Related: DwuC W1/Q1.

  2. Improved continuity in the construction description by linking its subsections to Figure 2 and adding transitions into path construction and query authoring (the opening paragraph of Section 4, Sections 4.3 and 4.5). Related: DwuC W2/Q2.

  3. Added reader ratings of query naturalness and clarity, agreement statistics, and an evaluation of paraphrases that preserve the evidence requirements (Appendix C.6, Tables 9–10). Related: 364y W1.1/Q1.

  4. Added a review of gold targets and evidence paths and broadened screening for alternative targets using multiple retrievers (Section 4.5, Appendix C.6, Table 9). Related: 364y W2.2/Q5.

  5. Clarified the relationship to ViDoSeek and added an analysis of support coverage and completeness alongside final target ranking (Sections 2.3 and 6.3, Appendix F.1, Table 24). Related: 364y W1.2/Q2.

  6. Added an evaluation with two required target pages per request, including metrics for target recall and request completion (Section 6.1, Appendix E.8, Table 23). Related: 364y W1.3/Q3.

  7. Added paired bootstrap confidence intervals and sensitivity analysis using groups of queries sharing source documents for the inspection and cropping ablations, with definitions and interpretation (Sections 5.1 and 6.2, Appendix F.3, Table 27). Related: 364y W2.1.

  8. Added an independent evaluation on 36 additional queries, including an expanded comparison of retrieval within and across documents (Section 6.1, Appendix E.7, Tables 21–22). Related: 364y W2.1/Q4.

  9. Added agent comparisons of OCR retrieval backends, image and OCR representations encoded by the same model, and BM25 fusion (Sections 5.2 and 6.1, Appendices D.7 and E.5, Tables 18–19). Related: PQ54 W1.

  10. Added search with observed images and compared its retrieval performance and inference resources with the standard visual agent (Section 5.2, Appendices D.7 and E.6, Table 20). Related: PQ54 Q1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.