- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper explores the research question of document ranking under iterative evidence acquisition. It proposes a data curation pipeline that compose directed evidence paths from a visual corpus. This results in a self-contained benchmark to measure both static and agentic retrieval. Comprehensive experiments are done to evaluate the commonly used retrieval methods and agentic models. Ablations and analysis reveal key insights: e.g., a strong planner can recover the failures of static ranking, and direct page inspection is important for agentic retrieval.
Critical strengths
- The benchmark construction pipeline is well-designed. And the resulting benchmark could be useful for the community.
- The experiments are comprehensive, involving commonly used retrieval methods, closed-source and open-source models. The analysis provides useful insights.
Critical weaknesses
- The visual-vs-OCR comparison does not isolate the effect of modality. The agentic visual route uses Qwen3-VL-Embedding-8B, while OCR-text search uses Qwen3-Embedding-8B. Both use dense retrieval, but that does not guarantee comparable retrieval quality. The OCR dense model performs worse than BM25, yet this weaker OCR dense retriever is the one deployed in the agentic experiments.
Questions for authors
- I noticed that the input of search tools is text. Would you consider image search (image to image/image to text search) as useful for solving your benchmark?
3 comments
I thank the reviewers for their comprehensive response. The experiments are comprehensive. I would maintain my score of 3.
Thank you for your continued recognition!
Response to Reviewer PQ54
We thank the reviewer for the thoughtful feedback and address the concerns below. We apologize for the delayed response, as the dataset expansion and human review required substantial time.
We ran three comparisons to distinguish OCR retriever quality from the contribution of visual representations. All rows report full GPT-5.6-sol agents on the same 120 queries and corpus with the same 12-action budget. Scores are percentages.
The OCR retrieval comparison replaces Qwen3-Embedding-8B with BM25 over the same OCR text, keeping the other OCR-text tools unchanged. The dense representation comparison instead fixes the encoder at Qwen3-VL-Embedding-8B and compares indices over page images and complete OCR text. The BM25 fusion comparison adds the same BM25 branch to each of those dense indices, with identical fusion rules and tools. The two original agent results with dense retrieval are from Table 1 of the original submission.
The OCR retrieval pair shows that a stronger text backend substantially improves the OCR agent. In the dense representation pair, the image index exceeds the OCR index by 19.17 R@1 points with the encoder held fixed. The BM25 fusion pair retains a visual advantage of 4.17 R@1 and 4.47 MRR@10 points with the encoder, lexical retrieval, fusion rule, and tools held fixed.
These comparisons support our main conclusion that preserving visual structure improves overall retrieval, even when the encoder and lexical retrieval are held fixed. We have incorporated these comparisons into Section 6.1 and Appendix E.5 (Table 18) of the revised manuscript, with the experimental settings in Appendix D.7.
Yes, we implemented and evaluated an additional tool for image queries. The planner can use an already observed page or crop, optionally with text, to search the same page image index. Search with text queries remains available, and each image search consumes one of the same 12 actions. We evaluated this condition on the same 120 queries, using the visual agent from the original submission (Table 1) as the baseline.
We also compared mean inference resources per query over the 119 queries with valid rankings in both conditions:
The planner uses image search in 69/120 episodes, newly surfacing the target in 12. Its overall R@1 improvement is modest at 3.33 percentage points, with higher model usage and page exposure. Image search also uses actions from the same budget as other searches and verification. Image queries provide complementary retrieval clues, while the standard visual agent uses fewer model calls and tokens. We have added the settings to Appendix D.7 and the retrieval and resource comparisons to Appendix E.6 (Table 20) in the revised manuscript.