- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies why Multimodal Search Agents (MSAs) fail to utilize visual cues effectively. The authors attribute the failures to two modes: Visual-Cue Omission (overlooking task-relevant cues) and Visual-Cue Misuse (misinterpreting cues, leading to irrelevant retrieval or unsupported inferences). To address them, they propose V-CUES, a training-free framework that induces reusable visual-cue utilization patterns and tool-use patterns from successful OpenSearch-VL trajectories into an initial skill library, refines it through task-specific experiences from repeated executions and outcome feedback, and at inference adopts a Mentor–Actor architecture where the Mentor selects relevant knowledge and generates visual-cue guidance for the Actor's tool calls and evidence integration. Evaluated on four knowledge-intensive VQA benchmarks (MMSearch-Plus, VDR-Bench, LiveVQA, FVQA) with three backbones (GPT-5.2, GPT-5.4, Gemini-3-Flash), V-CUES outperforms both the standard tool-augmented agent and XSKILL on overall Avg@2, with ablations, cue-utilization analyses, and case studies further supporting the design.
Critical strengths
1. A practically relevant problem with a clear diagnostic motivation.
The paper reframes the visual understanding difficulty of MSAs as a gap between visual-cue perception and visual-cue utilization, rather than yet another tool-calling optimization. The diagnostic experiments show that correct trajectories exhibit higher effective cue-utilization rates while incorrect ones exhibit higher misuse rates, providing solid empirical support for the motivation.
2. Method design well aligned with the motivation, with consistent gains.
The skill representations (applicability conditions, appropriate uses, common misuses) map naturally onto the two identified failure modes, and the Mentor adapts them to specific tasks. Experiments across four benchmarks and three backbones show V-CUES consistently outperforms XSKILL on overall Avg@2; ablations confirm the contribution of Visual-Cue Utilization Patterns and the Mentor's task-specific adaptation. In addition, process-level metrics (CU, EU, CM) go beyond final accuracy and help explain where the improvements come from — a valuable aspect of the evaluation.
Critical weaknesses
1. Insufficient baselines, plus fairness and data-overlap concerns from the extra expert data.
First, among the baselines, Direct Answer and w/ Tools are only control settings, leaving XSKILL as the sole method-level comparison. Given the recent growth of experience-/skill-based agents, a single baseline cannot adequately establish the competitiveness of V-CUES.
Second, V-CUES builds its initial Skill Library from 3,500 successful expert trajectories of OpenSearch-VL-SFT-36K, but the paper does not clarify whether XSKILL is initialized with the same expert-data budget. If not, the observed gains may partly reflect the additional expert knowledge rather than the method design itself.
Third, the SFT data sources of OpenSearch-VL include FVQA and LiveVQA, both of which are also evaluation benchmarks here. Although the adaptation set and test set are disjoint within each benchmark, the paper does not clarify whether cross-source deduplication was performed between the induction trajectories and the evaluation samples.
2. Limited methodological novelty, and the experiments do not separately validate the two claimed failure modes.
The overall pipeline — skill induction from trajectories, experience accumulation, periodic revision, and retrieval-and-adaptation for new tasks — largely mirrors existing frameworks such as XSKILL. The novelty lies mainly in organizing knowledge around visual cues (When/Consider/Avoid fields) and prompt-based guidance, rather than in a new learning or optimization mechanism.
More importantly, the paper claims to address both Visual-Cue Omission and Misuse, yet no experiment isolates the mechanism responsible for each. V-CUES does not outperform XSKILL on CU across all backbones, so the results support "more effective utilization and less misuse" rather than "consistently less omission." Since addressing both failure modes is a central claim, more targeted mechanism-level ablations are needed to connect the design to each failure mode and to the resulting gains.
Questions for authors
Q1. Baselines and expert-data budget
Could the authors add more representative multimodal search / experience-/skill-based baselines under comparable settings? Also, does XSKILL have access to the same 3,500 expert trajectories (or an equivalent knowledge budget) for initialization? If not, could controlled comparisons under the same expert-data budget be provided?
Q2. Overlap between induction data and evaluation data
Were the 3,500 trajectories used for Initial Skill Induction checked for overlap in images, questions, and answers against the test samples of the four evaluation benchmarks?
4 comments
W4(Novelty): The novelty lies mainly in organizing knowledge around visual cues (When/Consider/Avoid fields) and prompt-based guidance, rather than in a new learning or optimization mechanism.
We respectfully argue that the core driver of our method's novelty stems from the gap between visual-cue awareness and visual-cue utilization in multimodal search agents discovered in our diagnostic experiments (Figure 2). Consequently, V-CUES is specifically designed to enable visual cues to fulfill their proper functional roles throughout multimodal search. Our analysis of the novelty is as follows:
| XSKILL | ACE-SKILL | V-CUES (Ours) | |
|---|---|---|---|
| Skill Induction | Distills workflow and tool templates, complemented by action-level tactical experiences, from visually grounded trajectories. | Distills category-level workflow, tool templates and instance-level lessons within semantically coherent task clusters. | Knowledge induction centers on a judgment: what search role do observed visual cues play relative to unresolved information gaps? We cluster candidate regularities from Visual First-Pass, Knowledge Gap, and execution traces, then verify role consistency against the original images and questions. |
| Knowledge Bootstrapping | Uses cross-rollout critique to identify effective strategies, execution failures, and corrective actions for improving tool use. | Driven by proficiency-based rollout allocation, coupled with cluster-specific knowledge consolidation to improve data efficiency and reduce interference. | A key challenge in knowledge bootstrapping is making feedback specific enough to guide what the agent learns from each trajectory. Since answer correctness only indicates overall search success, our cue-level analysis provides a more specific learning signal. |
| Task Execution | Decomposes task needs, retrieves and rewrites experiences, and adapts skills to the task. | Routes each task to its nearest semantic cluster, then retrieves and adapts knowledge within that cluster. | Grounds the guidance in visual cues present in the current image, verifies the applicability of retrieved experiences, and clarifies how these cues should guide retrieval. |
The task role of visual cues thus determines which trajectories or experiences can be jointly induced into a skill, and when the induced knowledge can be transferred. This represents a cognitive-level shift in knowledge bootstrapping.
W5(Mechanisms for Visual-Cue Omission and Misuse): Since addressing both failure modes is a central claim, more targeted mechanism-level ablations are needed to connect the design to each failure mode and to the resulting gains.
We sincerely thank you for your profound insights. Regarding this issue, first, compared to the w/ tools experimental setting, our method holds an overall advantage across three metrics—CU, EU, and CM (Figure 4).
| Backbone | Method | CU ↑ | EU ↑ | CM ↓ |
|---|---|---|---|---|
| GPT-5.2 | w/ Tools | 0.45 | 0.38 | 0.20 |
| GPT-5.2 | V-CUES (Ours) | 0.48 (+0.03) | 0.44 (+0.06) | 0.10 (−0.10) |
| GPT-5.4 | w/ Tools | 0.56 | 0.45 | 0.23 |
| GPT-5.4 | V-CUES (Ours) | 0.68 (+0.12) | 0.59 (+0.14) | 0.13 (−0.10) |
| Gemini-3-Flash | w/ Tools | 0.65 | 0.55 | 0.14 |
| Gemini-3-Flash | V-CUES (Ours) | 0.67 (+0.02) | 0.58 (+0.03) | 0.14 (0.00) |
According to our definitions of the three metrics in Appendix A.1.4, we believe this result indeed reflects that our proposed method resolves the problems of visual-cue omission and visual-cue misuse under the w/ tools setting to a certain extent, and is linked to the improvement in performance.
Second, regarding XSKILL, combining the analysis of visual-cue utilization in Section 4.4 and the analysis of agent execution turns in Section 4.5, this method prevents the agent from giving an answer prematurely by prompting the agent to perform further verification, which drives an increase in interaction turns (from 3.31 with w/ Tools to 5.30 with XSKILL, averaged across the four benchmarks using GPT-5.4; Figure 5a) and also alleviates the omission of visual cues .
We acknowledge that the frequency of occurrence of the two proposed failure modes and the degree of their impact on the execution results of multimodal search agents may differ. Currently, we can only reflect the relationship between the occurrence frequency of the two failure modes and the actual execution results separately through the three metrics of CU, EU, and CM. Considering the complexity of actual execution trajectories (for example, the two failure modes occurring sequentially within the same trajectory), it is currently difficult for us to provide a completely decoupled experiment that more directly reflects the causal relationship. This is a limitation of our paper, and we will clearly state this in our subsequent revision.
Q1: Baselines and expert-data budget.
We add a new baseline ACE-SKILL under comparable settings, and we control comparisons under the same expert-data budget, more detail results please see the W1 and W2.
Q2: Overlap between induction data and evaluation data.
We took steps to prevent data contamination, more detail results please see the W3.
Dear Reviewer,
Thank you again for your time and thoughtful feedback.
We have responded to your comments and would like to ensure that we have addressed all of your concerns (or at least part of them) satisfactorily. Please let us know if there are any remaining questions or areas requiring further clarification.
If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.
Thank you very much.
Best regards,
The Authors
We sincerely thank you for your careful reading and your insightful comments. We provide detailed clarifications below and hope to address your concerns.
W1(Additional baseline comparison): Given the recent growth of experience-/skill-based agents, a single baseline cannot adequately establish the competitiveness of V-CUES.
To broaden our comparison, we added ACE-SKILL [1] —a recent multimodal skill-learning framework cited in our paper that learns skills from trajectories via prioritized sampling and semantic clustering.
Follow the setting of our manuscript, we evaluated ACE-SKILL using
GPT-5.2andGemini-3-Flashon two benchmarks, MMSearch-Plus and VDR-Bench:As shown above, compared to ACE-SKILL and XSKILL, V-CUES maintains a consistent performance gains.
[1] Ace-skill: Bootstrapping multimodal agents with prioritized and clustered evolution, arixv 2026.05
W2(Expert-data budget): The observed gains may partly reflect the additional expert knowledge rather than the method design itself.
XSKILL itself does not use expert data. In Section 3.3 and Appendix A.1.1 of our paper, we specifically explained that our design of using expert data for Initial Skill Induction is intended to alleviate the cold-start problem in visual-cue learning: when an agent omits crucial cues in early queries, the final answer feedback can indicate search failure, but it is difficult for it to reveal how that cue could have helped obtain the missing evidence.
To address the concern regarding expert-data budget fairness, we supplemented a comparative experiment on the MMSearch-Plus benchmark using GPT-5.4. Following the way XSKILL induces skills during the accumulation stage, we used the samples actually utilized for generating initial skills after clustering in V-CUES to perform induction, re-forming an initial skill that conforms to XSKILL's design philosophy, and subsequently executed the complete workflow of XSKILL, obtaining the following results:
We can see that expert data does not directly bring a significant performance improvement. Under the same expert-data budget, the Avg@2 of XSKILL is 32.9%, whereas our V-CUES reaches 37.2%. This indicates that the leap in performance stems primarily from our core method design of "visual-cue utilization," rather than simply relying on external expert data.
W3(Data contamination risk): Although the adaptation set and test set are disjoint within each benchmark, the paper does not clarify whether cross-source deduplication was performed between the induction trajectories and the evaluation samples.
After obtaining the
OpenSearch-VL/Search-VL-SFT-36Kdataset from OpenSearch-VL, we checked for overlaps between the raw dataset and the test samples through question ID tracing, question text matching, and then image comparison, and removed potentially overlapping samples from the raw dataset to ensure that they would not be sampled and participate in Initial Skill Induction. After completing the sampling, we conducted a re-verification to ensure that the risk of data contamination was ultimately ruled out. We will clearly state this in our subsequent revision.