- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work examines how multi-hop visual search can be improved by moving some of the sequential decision-making responsibilities from language-model agents into the retrieval component. To investigate this problem, the authors construct VHOP, a benchmark with five progressively challenging levels that test visual correspondence, spatial reasoning, and the ability to plan retrieval sequences. Building on this benchmark, they develop VHOP-ROUTER, a retrieval model that generates navigation decisions autoregressively. Rather than requiring an LLM to issue a new textual query at every step, the proposed model operates directly over visual representations, choosing subsequent images, recovering from unsuccessful retrieval paths, and determining when enough evidence has been collected. Its training procedure consists of supervised fine-tuning, online imitation learning, and reinforcement learning.
The experimental study considers both the retriever as an independent component and its integration into an agent-based question-answering system. For the L4 difficulty level, VHOP-ROUTER obtains a retrieval success rate of 76.3%. When incorporated into the agent, it raises end-to-end task success from 32.7% with a standard retriever to 85.4%. The paper further examines token efficiency, compares the proposed approach against more capable agent configurations and retrieval alternatives, and investigates its behavior across different levels of task complexity and in settings intended to be closer to practical visual search.
Critical strengths
1. An interesting architectural approach that gives the retriever greater autonomy in multi-step visual search.
A notable contribution of this work is its reconsideration of how responsibilities should be divided between the language agent and the retrieval system. Instead of having the agent coordinate every intermediate search through natural-language instructions, VHOP-ROUTER learns to traverse visual representations through a sequence of retrieval decisions. Its ability to select candidates, reconsider earlier choices, and terminate a search allows several planning operations to be handled within the retriever. This design directly targets the limitations of expressing complex visual dependencies as textual queries and offers an alternative way to organize multi-hop retrieval systems.
2. Substantial performance gains accompanied by meaningful baseline comparisons.
The experimental findings provide strong evidence for the effectiveness of the proposed approach within the constructed evaluation environment. Most notably, the end-to-end success rate on L4 tasks increases from 32.7% to 85.4% when the baseline retrieval component is replaced by VHOP-ROUTER, alongside a considerable reduction in token usage. The authors also examine larger candidate pools, stronger agent configurations, and tasks with varying levels of difficulty. These experiments strengthen the argument that improving the retriever's ability to make sequential decisions can offer significant benefits, rather than relying exclusively on a more powerful language agent to manage the search process.
Critical weaknesses
1. The experimental results do not yet establish reliable performance in realistic search environments.
The most convincing improvements are reported on synthetic tasks where scene composition, object relationships, and sources of difficulty are explicitly controlled. The additional evaluation intended to approximate real-world search is relatively small, covering only 50 queries, and its images are re-rendered rather than directly sampled from real-world photography. The results from this setting also reveal a substantial limitation: although the correct evidence is retrieved in approximately 34–36% of cases, the complete agent answers only 20% of questions correctly. This discrepancy indicates that successfully locating relevant visual content does not necessarily translate into reliable task completion. Consequently, the paper provides insufficient evidence that its reported advantages will carry over to visually diverse, less structured, and more challenging practical applications.
2. The learned navigation strategy appears to struggle when the required retrieval sequence becomes longer.
The four-hop evaluation in Appendix F.2 reveals a particularly concerning limitation. None of the investigated training configurations exceeds 8.2% accuracy, and the standard precise mixed-hop configuration achieves just 1.0%. Admittedly, these experiments change more than retrieval depth, since the search corpus is also enlarged and additional distractors are introduced. Nevertheless, the magnitude of the performance decline suggests that the proposed policy may not transfer effectively to more demanding retrieval conditions. This is especially important because learning to make sequential retrieval decisions is central to the paper's contribution. Without stronger evidence that this capability remains effective over longer search chains, the claimed generality and scalability of the framework remain difficult to establish.
Questions for authors
1. [Clarification] Understanding the limited performance on more realistic tasks.
Appendix F.3 reports approximately 34–36% retrieval success but only 20% final-answer accuracy in the realistic evaluation. Could the authors further investigate where the system fails, particularly separating unsuccessful evidence retrieval, premature or inappropriate stopping, and mistakes made during answer generation? It would also be helpful to explain what these findings imply for the framework's applicability outside the synthetic benchmark.
2. [Clarification / Additional Evaluation] Identifying the causes of four-hop performance degradation.
According to Appendix F.2, accuracy drops to 8.2% or below across all configurations on four-hop tasks, while the default configuration achieves only 1.0%. However, these tests simultaneously involve a larger retrieval corpus and more distracting candidates. Can the authors determine whether the main difficulty comes from longer decision sequences or from the increased complexity of the search space? An experiment that varies hop count independently of corpus size and distractor difficulty would provide more direct evidence about the model's ability to handle longer retrieval chains.
3. [Additional Evaluation] Separating autonomous retrieval from continued agent intervention.
The proposed retriever handles multiple retrieval decisions internally, but the surrounding LLM agent can still reformulate its requests and initiate further searches. To better identify the source of the performance improvement, could the authors evaluate a variant in which the agent is allowed to use VHOP-ROUTER only once, without subsequent query revision or repeated retrieval calls? Keeping the final-answer generation procedure unchanged would make it easier to assess how much of the improvement comes from the retriever's learned planning behavior rather than additional reasoning and intervention by the agent.
4. [Clarification] Evaluating computational efficiency beyond token consumption.
The reported savings in generated tokens and image inputs are encouraging, but these measurements may not capture the full computational overhead of the proposed system. Unlike a conventional retrieval call, VHOP-ROUTER may execute several internal decision steps before returning its results. Could the authors compare the methods using wall-clock latency, the total number of embedding computations, and overall computational expenditure under equivalent search budgets? Such measurements would help determine whether the proposed approach reduces total inference cost or mainly reallocates computation from the language agent to the retrieval component.
5. [Additional Evaluation] Measuring the individual effects of the training stages.
VHOP-ROUTER is developed through three successive stages: supervised fine-tuning, online imitation learning, and RLVR. Although the trajectory examples offer some insight into the learned behavior, the quantitative contribution of each stage remains insufficiently clear. Could the authors evaluate checkpoints obtained after each stage using the same test conditions and report retrieval success, backtracking accuracy, stopping accuracy, and end-to-end task performance? This comparison would help establish which capabilities emerge at each stage and whether all three stages are necessary to obtain the reported results.
8 comments
Thanks for your review (2/n)
Weakness 2
"2. The learned navigation strategy appears to struggle when the required retrieval sequence becomes longer.
The four-hop evaluation in Appendix F.2 reveals a particularly concerning limitation. None of the investigated training configurations exceeds 8.2% accuracy, and the standard precise mixed-hop configuration achieves just 1.0%. Admittedly, these experiments change more than retrieval depth, since the search corpus is also enlarged and additional distractors are introduced. Nevertheless, the magnitude of the performance decline suggests that the proposed policy may not transfer effectively to more demanding retrieval conditions. This is especially important because learning to make sequential retrieval decisions is central to the paper's contribution. Without stronger evidence that this capability remains effective over longer search chains, the claimed generality and scalability of the framework remain difficult to establish."
Response:
TL;DR: Three findings clarify transfer to longer retrieval chains: 1) Within the trained horizon (1–3 hops), dynamic variable-hop search works reliably: holding corpus size and distractors strictly fixed (Question 2), one single VHop-Router checkpoint dynamically handles unknown 1-, 2-, and 3-hop queries (99.7% / 90.0% / 76.3%, Section 5.4) without a depth oracle (knowing the ground-truth hop count H at test time, which prefix-conditioned Visual MDR [3] requires because it has no stopping head)—including on vague prompts and real YFCC-VHop photo chains (Weakness 1). 2) On out-of-horizon 4-hop queries (beyond 3 hops), while length extrapolation is a broad open challenge across sequence models [2] and multi-hop retrievers [3, 4], our component diagnosis shows that the main bottleneck is the lightweight stopping MLP head—specifically, distribution mismatch on its 2048-dim sequence-state input at unseen 5-image chain lengths, whereas local cosine-similarity drop features remain sharp at hop 4 (Question 2). 3) VHop-Router outperforms Visual MDR on 4-hop under both non-oracle score-drop stopping (24.7% vs. 16.5%) and oracle-depth K = 4 stopping (36.1% vs. 25.8%) (Appendix F.2, Table 21).
Experiment Setup:
- Dynamic Variable-Hop Control (1–3 Hops, Section 5.4 and Appendix E.1): Evaluated one single VHop-Router (Mixed-Hop) checkpoint across 1-, 2-, and 3-hop queries on the fixed 450-image L4 corpus (n = 300/hop, identical distractors, no depth oracle), under both precise and vague prompts (Question 2).
- Out-of-Horizon 4-Hop Diagnostic vs. Visual MDR (Appendix F.2, Tables 20–21, n = 97, 600 images): Evaluated matched 3-Hop Visual MDR [3] and VHop-Router checkpoints zero-shot on 4-hop L4 queries under both non-oracle 1D cosine-score-drop stopping (calibrated strictly on 1–3 hop dev queries, zero knowledge of K = 4) and matched oracle-depth K = 4 stopping across Beam = 1 and Beam = 4.
Results:
- Dynamic Hop Feasibility (1–3 Hops): Holding the 450-image corpus constant, Flash + Qwen3 drops sharply across 1 / 2 / 3 hops (75.7% → 66.7% → 32.7%, Appendix Tables 12a, 13a, and Section 5.1 Table 4a), whereas one precise Mixed-Hop VHop-Router checkpoint sustains 99.7% / 90.0% / 76.3% standalone without a depth oracle (Section 5.4, Tables 12a, 13a, 4a; and our vague-trained Mixed-Hop checkpoint sustains 97.0% / 84.7% / 50.7% vs. 21.3% / 6.7% / 2.0% with Flash on vague prompts, Tables 12d, 13d, 4b).
- 4-Hop Extrapolation Diagnostic (n = 97, Appendix F.2, Table 21): Decoupling stopping from the sequence state via the 1–3-hop-calibrated cosine-score drop (no K = 4 oracle, Beam = 4) lifts VHop-Router from 1.0% (Table 20) to 24.7% vs. 16.5% for Visual MDR, and under oracle-depth K = 4 stopping VHop-Router doubles Visual MDR at Beam = 1 (10.3% vs. 5.2%) via learned BACKTRACK and leads by +10.3 pp at Beam = 4 (36.1% vs. 25.8%; Mixed-Hop encoder: 32.0% vs. 29.9%):
| 3-Hop Trained Method on 4-Hop L4 (n = 97, Table 21) | Beam = 1 (K = 4 Oracle) | Beam = 4 (K = 4 Oracle) | Beam = 4 (No Oracle, Score-Drop Stop) |
|---|---|---|---|
| Visual MDR | 5.2% | 25.8% | 16.5% |
| VHop-Router | 10.3% | 36.1% | 24.7% |
References: [2] Anil et al., Exploring Length Generalization in Large Language Models, NeurIPS 2022. [3] Xiong et al., Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval, ICLR 2021. [4] Trivedi et al., MuSiQue: Multihop Questions via Single-Hop Question Composition, TACL 2022.
Thanks for your review (3/n)
Question 1
"1. [Clarification] Understanding the limited performance on more realistic tasks.
Appendix F.3 reports approximately 34–36% retrieval success but only 20% final-answer accuracy in the realistic evaluation. Could the authors further investigate where the system fails, particularly separating unsuccessful evidence retrieval, premature or inappropriate stopping, and mistakes made during answer generation? It would also be helpful to explain what these findings imply for the framework's applicability outside the synthetic benchmark."
Response:
TL;DR: Examining all 50 trajectories on the realistic test set (Section 5.6, Appendix F.3, Table 23) reveals three main takeaways: 1) Premature stopping by VHop-Router is rare (8%, 4/50); 2) 100% of the 34.0% → 20.0% gap (7/50 queries) comes from Gemini Flash's final answer selection after VHop-Router already returned the correct target image (6/7 picking a different wide-angle multi-object room view); and 3) In-domain real-photo training (YFCC-VHop, Weakness 1) directly resolves the synthetic-to-photo entry shift, lifting real-photo multi-hop standalone Hit@1 from 11.6% → 53.6% (+42.0 pp).
Experiment Setup: Examined every trajectory across all 50 human-authored queries over the 386 images of the realistic test set (Section 5.6, Appendix F.3, Tables 22–23) for Flash + VHop-Router (Mixed-Hop) (34.0% set retrieval, 17/50; 36.0%, 18/50 for 3-Hop) vs. Flash + Qwen3 (26.0%, 13/50), partitioning outcomes into five mutually exclusive categories: (1) correct final answer, (2) target returned in cumulative chain outputs but wrong image selected by Flash, (3) premature STOP, (4) mid-chain derailment, and (5) Hop-1 entry miss.
Results:
- 50-Query Failure Decomposition (Flash + VHop-Router Mixed-Hop): 20.0% (10/50) answered correctly (vs. 8.0%, 4/50 for Flash + Qwen3, in 3.3× fewer calls: 4.2 vs. 14.0); 14.0% (7/50; 8/50 for 3-Hop) returned the true target to the agent—accounting for the 17/50 → 10/50 (34.0% → 20.0%) retrieval-to-answer difference—where Flash picked a different cluttered multi-object room image (6/7 cases); premature stopping is only 8.0% (4/50; 2/50 = 4.0% for 3-Hop); 30.0% (15/50) derailed mid-chain; and 28.0% (14/50) missed Hop 1 (reflecting synthetic-to-everyday perceptual shift on query views).
- Implications Outside Synthetic Benchmarks: VHop-Router lowers combined mid-chain + early-stop attrition by 32%–37% relative (56%–60% on single-step baselines vs. 38%, 19/50), and adapting on natural multi-object photographs (YFCC-VHop, Weakness 1) lifts real-photo multi-hop retrieval from 11.6% → 53.6% standalone (n = 69) and 44.9% → 55.1% end-to-end.
Thanks for your review (1/n)
We sincerely thank Reviewer Ye8b for the thorough, high-confidence review and for recognizing how moving sequential navigation into the retriever reshapes the agent–retriever division of labor with substantial empirical gains. To directly address your weaknesses and five questions, we added natural Flickr photo evaluations, a 50-query realistic trajectory decomposition, fixed-gallery and 4-hop diagnostics, a strict single-call agent experiment, wall-clock GPU/API profiling, and a stage-wise checkpoint evaluation. Below, we address each item point by point.
Weakness 1
Response:
TL;DR: To directly evaluate generalization on naturally occurring photographs, we added three unmodified Flickr photo benchmarks (YFCC-VHop, DISBench-MVH, and Official DISBench [1]; Section 5.6, Figure 7, Table 8; Appendix F.5, Table 25 and Figure 14): across cluttered personal photo albums (about 750–1,900 real photos per user), VHop-Router raises multi-hop standalone Hit@1 from 11.6% → 53.6% (+42.0 pp) and Gemini 3.5 Flash agent Top-1 accuracy from 44.9% → 55.1% (+10.2 pp) on YFCC-VHop, while transferring zero-shot to Official DISBench and DISBench-MVH. Meanwhile, examining all 50 trajectories on the realistic test set (Question 1, Appendix F.3, Table 23) shows that premature stopping is rare (8%, 4/50) and all 7/50 (14%) cases separating retrieval (34%) from final accuracy (20%) occur after the target image is already returned to Flash.
Experiment Setup:
Results:
Reference: [1] DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Histories, 2026.