acceptodds
Under review as a conference paper at ICLR 2027

V-CUES: Improving Multimodal Search Agents by Centering Visual-Cue Utilization via Experience and Skills

10 views

Abstract

Multimodal Search Agents (MSAs) have emerged as a promising paradigm for solving real-world tasks by combining visual understanding with external tools. While recent works have improved the mechanics of tool use and information retrieval, agents still struggle to effectively utilize visual cues during task execution. As a result, agents remain susceptible to Visual-Cue Omission and Visual-Cue Misuse, which can lead to irrelevant retrieval and task failure. To address these limitations, we propose V-CUES, a framework that bootstraps reusable knowledge centered on visual-cue utilization. V-CUES extracts recurring patterns from historical trajectories to capture how visual cues should be utilized under different input contexts, while dynamically refining them through case-level experiences accumulated from new interactions. During execution, V-CUES contextualizes accumulated knowledge for each newly encountered task, providing task-specific supervision on how to incorporate visual cues throughout the search process. Evaluation across four knowledge-intensive benchmarks and three basic models shows that V-CUES consistently improves performance over the previous state-of-the-art methods, demonstrating the effectiveness of explicitly centering visual cues for multimodal search agents.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Search Agent, Visual-Cue Utilization, Experience and Skills

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Y9BAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why Multimodal Search Agents (MSAs) fail to utilize visual cues effectively. The authors attribute the failures to two modes: Visual-Cue Omission (overlooking task-relevant cues) and Visual-Cue Misuse (misinterpreting cues, leading to irrelevant retrieval or unsupported inferences). To address them, they propose V-CUES, a training-free framework that induces reusable visual-cue utilization patterns and tool-use patterns from successful OpenSearch-VL trajectories into an initial skill library, refines it through task-specific experiences from repeated executions and outcome feedback, and at inference adopts a Mentor–Actor architecture where the Mentor selects relevant knowledge and generates visual-cue guidance for the Actor's tool calls and evidence integration. Evaluated on four knowledge-intensive VQA benchmarks (MMSearch-Plus, VDR-Bench, LiveVQA, FVQA) with three backbones (GPT-5.2, GPT-5.4, Gemini-3-Flash), V-CUES outperforms both the standard tool-augmented agent and XSKILL on overall Avg@2, with ablations, cue-utilization analyses, and case studies further supporting the design.

Critical strengths

1. A practically relevant problem with a clear diagnostic motivation.

The paper reframes the visual understanding difficulty of MSAs as a gap between visual-cue perception and visual-cue utilization, rather than yet another tool-calling optimization. The diagnostic experiments show that correct trajectories exhibit higher effective cue-utilization rates while incorrect ones exhibit higher misuse rates, providing solid empirical support for the motivation.

2. Method design well aligned with the motivation, with consistent gains.

The skill representations (applicability conditions, appropriate uses, common misuses) map naturally onto the two identified failure modes, and the Mentor adapts them to specific tasks. Experiments across four benchmarks and three backbones show V-CUES consistently outperforms XSKILL on overall Avg@2; ablations confirm the contribution of Visual-Cue Utilization Patterns and the Mentor's task-specific adaptation. In addition, process-level metrics (CU, EU, CM) go beyond final accuracy and help explain where the improvements come from — a valuable aspect of the evaluation.

Critical weaknesses

1. Insufficient baselines, plus fairness and data-overlap concerns from the extra expert data.

First, among the baselines, Direct Answer and w/ Tools are only control settings, leaving XSKILL as the sole method-level comparison. Given the recent growth of experience-/skill-based agents, a single baseline cannot adequately establish the competitiveness of V-CUES.

Second, V-CUES builds its initial Skill Library from 3,500 successful expert trajectories of OpenSearch-VL-SFT-36K, but the paper does not clarify whether XSKILL is initialized with the same expert-data budget. If not, the observed gains may partly reflect the additional expert knowledge rather than the method design itself.

Third, the SFT data sources of OpenSearch-VL include FVQA and LiveVQA, both of which are also evaluation benchmarks here. Although the adaptation set and test set are disjoint within each benchmark, the paper does not clarify whether cross-source deduplication was performed between the induction trajectories and the evaluation samples.

2. Limited methodological novelty, and the experiments do not separately validate the two claimed failure modes.

The overall pipeline — skill induction from trajectories, experience accumulation, periodic revision, and retrieval-and-adaptation for new tasks — largely mirrors existing frameworks such as XSKILL. The novelty lies mainly in organizing knowledge around visual cues (When/Consider/Avoid fields) and prompt-based guidance, rather than in a new learning or optimization mechanism.

More importantly, the paper claims to address both Visual-Cue Omission and Misuse, yet no experiment isolates the mechanism responsible for each. V-CUES does not outperform XSKILL on CU across all backbones, so the results support "more effective utilization and less misuse" rather than "consistently less omission." Since addressing both failure modes is a central claim, more targeted mechanism-level ablations are needed to connect the design to each failure mode and to the resulting gains.

Questions for authors

Q1. Baselines and expert-data budget

Could the authors add more representative multimodal search / experience-/skill-based baselines under comparable settings? Also, does XSKILL have access to the same 3,500 expert trajectories (or an equivalent knowledge budget) for initialization? If not, could controlled comparisons under the same expert-data budget be provided?

Q2. Overlap between induction data and evaluation data

Were the 3,500 trajectories used for Initial Skill Induction checked for overlap in images, questions, and answers against the test samples of the four evaluation benchmarks?

4 comments

We sincerely thank you for your careful reading and your insightful comments. We provide detailed clarifications below and hope to address your concerns.

W1(Additional baseline comparison): Given the recent growth of experience-/skill-based agents, a single baseline cannot adequately establish the competitiveness of V-CUES.

To broaden our comparison, we added ACE-SKILL [1] —a recent multimodal skill-learning framework cited in our paper that learns skills from trajectories via prioritized sampling and semantic clustering.

Follow the setting of our manuscript, we evaluated ACE-SKILL using GPT-5.2 and Gemini-3-Flash on two benchmarks, MMSearch-Plus and VDR-Bench:

ModelMethodMMSearch-Plus Avg@2 (%)MMSearch-Plus Pass@2 (%)VDR-Bench Avg@2 (%)VDR-Bench Pass@2 (%)
GPT-5.2XSKILL25.3632.7025.7533.00
GPT-5.2ACE-SKILL28.4435.0725.2534.00
GPT-5.2V-CUES (Ours)31.04 (+2.60)37.91 (+2.84)27.75 (+2.00)38.50 (+4.50)
Gemini-3-FlashXSKILL48.5856.8728.5034.00
Gemini-3-FlashACE-SKILL46.9253.0827.2533.00
Gemini-3-FlashV-CUES (Ours)51.18 (+2.60)59.72 (+2.85)29.75 (+1.25)36.00 (+2.00)

Values in parentheses indicate improvements over the best results among the baselines.

As shown above, compared to ACE-SKILL and XSKILL, V-CUES maintains a consistent performance gains.

[1] Ace-skill: Bootstrapping multimodal agents with prioritized and clustered evolution, arixv 2026.05

W2(Expert-data budget): The observed gains may partly reflect the additional expert knowledge rather than the method design itself.

XSKILL itself does not use expert data. In Section 3.3 and Appendix A.1.1 of our paper, we specifically explained that our design of using expert data for Initial Skill Induction is intended to alleviate the cold-start problem in visual-cue learning: when an agent omits crucial cues in early queries, the final answer feedback can indicate search failure, but it is difficult for it to reveal how that cue could have helped obtain the missing evidence.

To address the concern regarding expert-data budget fairness, we supplemented a comparative experiment on the MMSearch-Plus benchmark using GPT-5.4. Following the way XSKILL induces skills during the accumulation stage, we used the samples actually utilized for generating initial skills after clustering in V-CUES to perform induction, re-forming an initial skill that conforms to XSKILL's design philosophy, and subsequently executed the complete workflow of XSKILL, obtaining the following results:

MMSearch-PlusXSKILL (w/o expert-data)XSKILL (w/ expert-data)V-CUES (Ours)
Avg@230.632.9 (+2.3)37.2 (+6.6)
Pass@239.340.8 (+1.5)47.9 (+8.6)

We can see that expert data does not directly bring a significant performance improvement. Under the same expert-data budget, the Avg@2 of XSKILL is 32.9%, whereas our V-CUES reaches 37.2%. This indicates that the leap in performance stems primarily from our core method design of "visual-cue utilization," rather than simply relying on external expert data.

W3(Data contamination risk): Although the adaptation set and test set are disjoint within each benchmark, the paper does not clarify whether cross-source deduplication was performed between the induction trajectories and the evaluation samples.

After obtaining the OpenSearch-VL/Search-VL-SFT-36K dataset from OpenSearch-VL, we checked for overlaps between the raw dataset and the test samples through question ID tracing, question text matching, and then image comparison, and removed potentially overlapping samples from the raw dataset to ensure that they would not be sampled and participate in Initial Skill Induction. After completing the sampling, we conducted a re-verification to ensure that the risk of data contamination was ultimately ruled out. We will clearly state this in our subsequent revision.

W4(Novelty): The novelty lies mainly in organizing knowledge around visual cues (When/Consider/Avoid fields) and prompt-based guidance, rather than in a new learning or optimization mechanism.

We respectfully argue that the core driver of our method's novelty stems from the gap between visual-cue awareness and visual-cue utilization in multimodal search agents discovered in our diagnostic experiments (Figure 2). Consequently, V-CUES is specifically designed to enable visual cues to fulfill their proper functional roles throughout multimodal search. Our analysis of the novelty is as follows:

XSKILLACE-SKILLV-CUES (Ours)
Skill InductionDistills workflow and tool templates, complemented by action-level tactical experiences, from visually grounded trajectories.Distills category-level workflow, tool templates and instance-level lessons within semantically coherent task clusters.Knowledge induction centers on a judgment: what search role do observed visual cues play relative to unresolved information gaps? We cluster candidate regularities from Visual First-Pass, Knowledge Gap, and execution traces, then verify role consistency against the original images and questions.
Knowledge BootstrappingUses cross-rollout critique to identify effective strategies, execution failures, and corrective actions for improving tool use.Driven by proficiency-based rollout allocation, coupled with cluster-specific knowledge consolidation to improve data efficiency and reduce interference.A key challenge in knowledge bootstrapping is making feedback specific enough to guide what the agent learns from each trajectory. Since answer correctness only indicates overall search success, our cue-level analysis provides a more specific learning signal.
Task ExecutionDecomposes task needs, retrieves and rewrites experiences, and adapts skills to the task.Routes each task to its nearest semantic cluster, then retrieves and adapts knowledge within that cluster.Grounds the guidance in visual cues present in the current image, verifies the applicability of retrieved experiences, and clarifies how these cues should guide retrieval.

The task role of visual cues thus determines which trajectories or experiences can be jointly induced into a skill, and when the induced knowledge can be transferred. This represents a cognitive-level shift in knowledge bootstrapping.

W5(Mechanisms for Visual-Cue Omission and Misuse): Since addressing both failure modes is a central claim, more targeted mechanism-level ablations are needed to connect the design to each failure mode and to the resulting gains.

We sincerely thank you for your profound insights. Regarding this issue, first, compared to the w/ tools experimental setting, our method holds an overall advantage across three metrics—CU, EU, and CM (Figure 4).

BackboneMethodCU ↑EU ↑CM ↓
GPT-5.2w/ Tools0.450.380.20
GPT-5.2V-CUES (Ours)0.48 (+0.03)0.44 (+0.06)0.10 (−0.10)
GPT-5.4w/ Tools0.560.450.23
GPT-5.4V-CUES (Ours)0.68 (+0.12)0.59 (+0.14)0.13 (−0.10)
Gemini-3-Flashw/ Tools0.650.550.14
Gemini-3-FlashV-CUES (Ours)0.67 (+0.02)0.58 (+0.03)0.14 (0.00)

According to our definitions of the three metrics in Appendix A.1.4, we believe this result indeed reflects that our proposed method resolves the problems of visual-cue omission and visual-cue misuse under the w/ tools setting to a certain extent, and is linked to the improvement in performance.

Second, regarding XSKILL, combining the analysis of visual-cue utilization in Section 4.4 and the analysis of agent execution turns in Section 4.5, this method prevents the agent from giving an answer prematurely by prompting the agent to perform further verification, which drives an increase in interaction turns (from 3.31 with w/ Tools to 5.30 with XSKILL, averaged across the four benchmarks using GPT-5.4; Figure 5a) and also alleviates the omission of visual cues .

We acknowledge that the frequency of occurrence of the two proposed failure modes and the degree of their impact on the execution results of multimodal search agents may differ. Currently, we can only reflect the relationship between the occurrence frequency of the two failure modes and the actual execution results separately through the three metrics of CU, EU, and CM. Considering the complexity of actual execution trajectories (for example, the two failure modes occurring sequentially within the same trajectory), it is currently difficult for us to provide a completely decoupled experiment that more directly reflects the causal relationship. This is a limitation of our paper, and we will clearly state this in our subsequent revision.

Q1: Baselines and expert-data budget.

We add a new baseline ACE-SKILL under comparable settings, and we control comparisons under the same expert-data budget, more detail results please see the W1 and W2.

Q2: Overlap between induction data and evaluation data.

We took steps to prevent data contamination, more detail results please see the W3.

Dear Reviewer,

Thank you again for your time and thoughtful feedback.

We have responded to your comments and would like to ensure that we have addressed all of your concerns (or at least part of them) satisfactorily. Please let us know if there are any remaining questions or areas requiring further clarification.

If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

Thank you very much.

Best regards,

The Authors

Reviewer_p2jfOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies visual-cue omission and misuse in multimodal search agents. To solve this problem, the authors proposes V-CUES, which builds reusable skills from successful trajectories and collects experiences from target tasks. Specifically, a mentor selects relevant knowledge and gives guidance to an actor. The method improves average performance over tool-only agents and XSKILL across four benchmarks and three backbone models.

Critical strengths
  1. The motivation of this paper is clear, and the distinction between cue omission and cue misuse is insightful.

  2. The experiments in Table 1 show gains across three backbones. Moreover, the comparison in Table 10 also supports the value of task-specific guidance.

Critical weaknesses
  1. As shown in Table 2, removing the experience library reduces Avg@2 by only 0.4 points for GPT-5.2 and 0.2 points for GPT-5.4. Therefore, the value of case-level experience is limited in this setting.

  2. As shown in Table 11, V-CUES uses 31.0k tokens per task, compared with 22.2k for the tool-only baseline. A comparison with similar actual token use would help separate the benefit of cue guidance from extra computation.

Questions for authors
  1. Does the experience library give larger gains on the other benchmarks? In Table 2, it presents only a small gain on MMSearch-Plus.

  2. Moreover, in Table 2, the authors do not test what happens when the image-based cluster check is removed. It is unclear how much this step helps the final performance.

2 comments

We sincerely thank you for your careful reading and your insightful comments. We provide detailed clarifications below and hope to address your concerns.

W1(Value of experience): The value of case-level experience is limited in this setting.

We understand your concern about the value of the Experience Library, and we add an ablation experiment on VDR-Bench [1] using GPT-5.4. The MMSearch-Plus focuses on fine-grained visual cues for web browsing, and the VDR-Bench emphasizes broader visual deep research across more diverse domains.

Metric (%)Full−Exp−Skill−S&E−Mentor
Avg@227.527.0 (−0.5)25.3 (−2.2)23.3 (−4.2)25.0 (−2.5)
Pass@236.035.0 (−1.0)33.5 (−2.5)32.5 (−3.5)34.0 (−2.0)

Based on the results of the ablation study, we acknowledge that when the Skill Library is present, the Experience Library mainly plays an auxiliary role, and its direct gain is not very large.

This is because during the evolution process on the benchmark's task distribution, periodic revisions of the skills are derived from the collected experiences; **the final skills already contain part of the useful information learned from the experiences and provide appropriate guidance through the Mentor. **

Besides, the comparison of the experimental results among the -Skill, -S&E, and -Mentor settings in our Table 2 also proves that learning experiences is valuable in itself: when skills are ablated, it can still provide the agent with usable guidance under specific contexts, making the operation of our framework more stable.

[1] Vision-deepresearch: Incentivizing deepresearch capability in multimodal large language models, ICML 2026

W2(Token consumption): A comparison with similar actual token use would help separate the benefit of cue guidance from extra computation.

As we pointed out, the gap between visual-cue awareness and visual-cue utilization is a core issue for agents in multimodal search tasks. One manifestation of this under the w/ tools experimental setting is that the agent lacks in-depth exploration of the problem and relies on language priors or simple text matching to give an answer. Our analysis of actual execution trajectories in Section 4.5 of the paper also shows that under the w/ tools setting, the agent executes fewer turns and lacks exploration and acquisition of information.

MethodAvg. turns / taskAvg. Tokens / taskAvg@2 (%)
w/ Tools3.3122.2k23.70
XSKILL5.30 (+1.99)30.2k (+8.0k)25.36 (+1.66)
V-CUES5.16 (+1.85)31.0k (+8.8k)31.04 (+7.34)

Therefore, stably improving multimodal search agents requires deeper information exploration, which naturally increases interaction turns and token consumption. Besides, XSKILL serves as a direct comparison with similar token usage: on MMSearch-Plus (GPT-5.2), V-CUES uses only 2.6% more tokens than XSKILL while achieving a 5.68-point higher Avg@2. This confirms that under comparable token budgets, cue guidance brings significant gains.

Q1(Value of experience): Does the experience library give larger gains on the other benchmarks? In Table 2, it presents only a small gain on MMSearch-Plus.

The additional VDR-Bench ablation study also shows a modest gain from the Experience Library, more detail results please see the W1.

Q2(Additional ablation study): In Table 2, the authors do not test what happens when the image-based cluster check is removed. It is unclear how much this step helps the final performance.

We agree that an ablation study should be conducted on the image-based cluster check. We supplemented an ablation experiment on MMSearch-Plus using GPT-5.4 by directly forming the initial skills from the backup results prior to the image-based cluster check, and then executing adaptation and testing. We obtained the following results:

MMSearch-Plusw/o image-based cluster checkw/ image-based cluster check
Avg@236.737.2 (+0.5)
Pass@246.947.9 (+1.0)

As can be seen from the ablation study, multimodal context validation provides some help, though it is not the primary part affecting the method's performance.

Regarding this issue, we acknowledge that during the actual induction process, only a few candidate clusters were split due to inconsistent task roles of visual cues; therefore, this step made limited changes to the initial cluster structure.

Besides, subsequent skill revisions use experiences from the target distribution to adjust the initial skills, and the Mentor also filters applicable knowledge based on the current image and question—these two steps further narrow the difference. We retain this quality-control step for initial skill induction as a validation step to ensure the soundness of the method, thereby reducing the risk of inducing different cue usages into the same skill.

Reviewer_bP2rOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argue that Multimodal Search agents often fail because they use visual cues poorly although recent works have improved tools mechanic. It issues 2 model failures: Visual-Cue Omission and Visual-Cue Misuse. To address these, it introduce V-CUES, a framework that bootstraps reusable knowledge centered on visual-cue utilization. Evaluation shows that V-CUES improves performance over previous methods.

Critical strengths

-A useful, well-motivated framework with supporting ablations: Shifting attention from tool use to the role of visual cues is sensible research direction. Experiments show that V-CUES outperform other baseline on GPT models -The method need no parameter updates, added test time cost(token consumption) which increase V-CUES practical values

Critical weaknesses

-The paper claim V-CUES “consistently outperforms” prior state of the art but Gemini 3 flash fall below to w /Tools baseline on LiveVQA benchmark -Only XSKILL is compared directly and also Gemini 3 flash backbone is old so it is unclear whether the gains hold on current frontier models -Novelty is incremental: The pipeline closely follows XSKILL. The contribution is mostly what content is stored and the Mentor/Actor split.

Questions for authors

-With Gemini-3-Flash, V-CUES is below w/ Tools on LiveVQA. Why does this happen? -Can authors evaluate on a more recent or stronger backbone for example: Gemini 3.8 flash,..?

3 comments

We sincerely thank you for your careful reading and your insightful comments. We provide detailed clarifications below and hope to address your concerns.

W1(Claim clarification). The paper claim V-CUES “consistently outperforms” prior state of the art but Gemini 3 flash fall below to w/ Tools baseline on LiveVQA benchmark.

Thank you for pointing this out. To express this more accurately and eliminate any misunderstanding, we will revise the phrasing to "consistently improves the overall performance across multiple benchmarks." referring specifically to the “Overall” results in Table 1.

W2(Additional baseline comparison). Only XSKILL is compared directly.

To broaden our comparison, we added ACE-SKILL [1] —a recent multimodal skill-learning framework cited in our paper that learns skills from trajectories via prioritized sampling and semantic clustering.

Follow the setting of our manuscript, we evaluated ACE-SKILL using GPT-5.2 and Gemini-3-Flash on two benchmarks, MMSearch-Plus and VDR-Bench:

ModelMethodMMSearch-Plus Avg@2 (%)MMSearch-Plus Pass@2 (%)VDR-Bench Avg@2 (%)VDR-Bench Pass@2 (%)
GPT-5.2XSKILL25.3632.7025.7533.00
GPT-5.2ACE-SKILL28.4435.0725.2534.00
GPT-5.2V-CUES (Ours)31.04 (+2.60)37.91 (+2.84)27.75 (+2.00)38.50 (+4.50)
Gemini-3-FlashXSKILL48.5856.8728.5034.00
Gemini-3-FlashACE-SKILL46.9253.0827.2533.00
Gemini-3-FlashV-CUES (Ours)51.18 (+2.60)59.72 (+2.85)29.75 (+1.25)36.00 (+2.00)

Values in parentheses indicate improvements over the best results among the baselines.

As shown above, compared to ACE-SKILL and XSKILL, V-CUES maintains a consistent performance gains.

[1] Ace-skill: Bootstrapping multimodal agents with prioritized and clustered evolution, arixv 2026.05

W3(Latest backbones comparison). Gemini 3 flash backbone is old so it is unclear whether the gains hold on current frontier models.

We have extended our experiments to the latest llm backbones, GPT-5.6-Luna (2026.07) and GPT-6-luna (2026.09).

We compare the V-CUES with w/ tools, XSKILL, and ACE-SKILL in on the MMSearch-Plus benchmark.

ModelMethodAvg@2 (%)Pass@2 (%)
GPT-5.6-lunaw/ tools35.5543.60
GPT-5.6-lunaXSKILL37.2045.02
GPT-5.6-lunaACE-SKILL36.4944.55
GPT-5.6-lunaV-CUES (Ours)39.57 (+2.37)48.34 (+3.32)
GPT-6-lunaw/ tools35.0744.07
GPT-6-lunaXSKILL36.0246.92
GPT-6-lunaACE-SKILL38.1547.39
GPT-6-lunaV-CUES (Ours)41.94 (+3.79)49.76 (+2.37)

Values in parentheses indicate improvements over the best results among the baselines.

Relative to the latest llm backbones, V-CUES maintains a consistent performance gains compared to ACE-SKILL and XSKILL.

W4(Novelty). Novelty is incremental: The pipeline closely follows XSKILL. The contribution is mostly what content is stored and the Mentor/Actor split.

We respectfully disagree that our innovation is merely "what content is stored and the Mentor/Actor split.". The core driver of our method's novelty stems from the gap between visual-cue awareness and visual-cue utilization in multimodal search agents discovered in our diagnostic experiments (Figure 2). Consequently, V-CUES is specifically designed to enable visual cues to fulfill their proper functional roles throughout multimodal search. Our analysis of the novelty is as follows:

XSKILLACE-SKILLV-CUES (Ours)
Skill InductionDistills workflow and tool templates, complemented by action-level tactical experiences, from visually grounded trajectories.Distills category-level workflow, tool templates and instance-level lessons within semantically coherent task clusters.Knowledge induction centers on a judgment: what search role do observed visual cues play relative to unresolved information gaps? We cluster candidate regularities from Visual First-Pass, Knowledge Gap, and execution traces, then verify role consistency against the original images and questions.
Knowledge BootstrappingUses cross-rollout critique to identify effective strategies, execution failures, and corrective actions for improving tool use.Driven by proficiency-based rollout allocation, coupled with cluster-specific knowledge consolidation to improve data efficiency and reduce interference.A key challenge in knowledge bootstrapping is making feedback specific enough to guide what the agent learns from each trajectory. Since answer correctness only indicates overall search success, our cue-level analysis provides a more specific learning signal.
Task ExecutionDecomposes task needs, retrieves and rewrites experiences, and adapts skills to the task.Routes each task to its nearest semantic cluster, then retrieves and adapts knowledge within that cluster.Grounds the guidance in visual cues present in the current image, verifies the applicability of retrieved experiences, and clarifies how these cues should guide retrieval.

The task role of visual cues thus determines which trajectories or experiences can be jointly induced into a skill, and when the induced knowledge can be transferred. This represents a cognitive-level shift in knowledge bootstrapping.

Q1(Performance Analysis). With Gemini-3-Flash, V-CUES is below w/ Tools on LiveVQA. Why does this happen?

Newly proposed benchmarks like MMSearch-Plus and VDR-Bench emphasize using fine-grained visual cues under retrieval noise. The images in LiveVQA tasks are relatively simple, and the visual cues are relatively few and explicit. Gemini-3-Flash can already formulate effective queries directly, leaving less room for additional cue guidance. In these cases, skill-based guidance may introduce unnecessary search constraints or verification steps, accounting for the small decline observed.

Q2(Latest backbones comparison). Can authors evaluate on a more recent or stronger backbone?

We introduce the latest llm backbones gpt-5.6-luna and gpt-6-luna, more detail results please see the W3.

Dear Reviewer,

Thank you again for your time and thoughtful feedback.

We have responded to your comments and would like to ensure that we have addressed all of your concerns (or at least part of them) satisfactorily. Please let us know if there are any remaining questions or areas requiring further clarification.

If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

Thank you very much.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: New Experiments and Revised Paper

We thank the reviewers for their constructive feedback. To address the questions raised, we conducted additional experiments and analyses.

We have updated the manuscript, with changes highlighted in blue. In particular, we added evaluations on two latest backbones, GPT-5.6 Luna and GPT-6 Luna (Appendix A.2.2), where V-CUES outperforms the compared baselines. We also strengthened XSKILL using the same expert data as V-CUES. V-CUES maintains its advantage in this comparison (Expert-Data Budget Analysis, Section 4.7).

Further additions include comparisons with ACE-SKILL as an additional baseline (Table 1 and Sections 4.1–4.2), component ablations assessing the value of the Experience Library on VDR-Bench, and an ablation of multimodal context validation (Appendix A.2.2). We also clarified the scope of our performance claims (Abstract and Conclusion), detailed the cross-source overlap checks (Appendix A.1.1), and discussed the limitations of separately attributing accuracy gains to reduced cue omission and misuse (Limitations).

We thank the reviewers again for their insightful comments and look forward to further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.