acceptodds
Under review as a conference paper at ICLR 2027

Match What You Know: Capability-Aligned Search Agents for Knowledge-Based Visual Question Answering

Abstract

Knowledge-Based Visual Question Answering (KBVQA) requires models to identify task-relevant entities in images and retrieve complementary information from external resources when their internal knowledge is insufficient. Existing retrieval-augmented generation methods rely on predefined retrieval pipelines, which may lead to either insufficient evidence acquisition or redundant retrieval. Although search agents can dynamically select tools, reinforcement learning often drives them towards a conservative full-pipeline retrieval workflow of "image search followed by text search". Consequently, even when the model already possesses the necessary entity-recognition or question-answering capabilities, it still tends to invoke external tools first, leading to underutilization of its existing capabilities and unnecessary tool calls. To address this issue, we propose , a multimodal search agent for KBVQA. Before reinforcement learning, we conduct tool-free behavioral probing to estimate the model’s initial capabilities in three aspects: identifying task-relevant entities, answering questions when the correct entity is provided, and directly answering questions from the original image-question pair. During training, we use the probing results to construct capability-boundary supervision, which encourages the model to prioritize its existing capabilities and invoke external tools only when the corresponding capabilities are insufficient. BoundarySA therefore learns to expand its problem-solving ability through tool use while continuing to leverage its original capabilities. Experiments on MMHops, InfoSeek, and E-VQA demonstrate that BoundarySA maintains competitive answer performance while making more effective use of the model’s existing capabilities and reducing reliance on the fixed workflow.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KBVQA, Search Agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_civdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies tool overuse in RL-trained KBVQA agents, which often collapse to a fixed image-search-then-text-search workflow even when retrieval is unnecessary. It first probes the model’s entity recognition, entity-conditioned answering, and direct answering capabilities for each training instance. It then converts these capability estimates into preferred tool-use paths and uses them as an additional RL reward, encouraging the agent to call external tools only when its own capabilities are insufficient.

Critical strengths
  • The paper identifies a practically relevant failure mode in RL-based KBVQA agents and motivates the use of a capability-aware tool well.
  • The capability probing and path-based reward provide an intuitive and interpretable mechanism for controlling tool use.
  • The paper provides useful behavioral analyses that help illustrate how capability-aware supervision changes the agent’s tool-use patterns.
Critical weaknesses
  • Offline capability probing may become stale as the model’s capabilities evolve during RL, since path preferences are fixed before training.
  • The method may generalize poorly to new tools or task structures because it relies on a task-specific capability decomposition and four predefined path categories.
  • The approach may be difficult to scale to less annotated datasets because probing requires ground-truth answers and relevant entity labels.
  • The method’s robustness in realistic agent settings remains unclear because evaluation is limited to closely related KBVQA benchmarks with relatively simple tool spaces.
Questions for authors
  • The benefit of the proposed RL reward is not fully disentangled from direct path supervision. A comparison with simpler alternatives, such as directly supervising the preferred path or using a routing classifier, would help isolate the contribution of the RL formulation.
  • The authors should explain how the framework would extend to richer tool spaces or tasks where the reasoning structure cannot be reduced to direct, image, text, and image-plus-text paths
  • The capability boundary is estimated before RL and remains fixed even though the model’s capabilities may change during training. The authors should provide stronger empirical evidence on capability drift, or compare against dynamic or periodically updated probing.
  • The current evaluation remains within relatively similar KBVQA settings and provides limited evidence for generalization to more realistic VQA scenarios. I strongly recommend evaluating on SUPERGLASSES, which is built from real-world egocentric images captured by smart glasses and covers diverse domains and query types with annotated search trajectories and reasoning processes.
3 comments
AuthorsOpenReview ↗

Response to Reviewer civd

We thank the reviewer for the thoughtful questions.

Q1. Are all three capability probes necessary?

We added two simpler alternatives:

  1. Correct-gated efficiency: fewer tool calls are rewarded only when the final answer is correct.

  2. Direct-only probing: only pdp_d is used to divide examples into Direct and Non-direct groups, without distinguishing Image-only, Text-only, and Image+Text requirements.

MethodAccuracy %Avg. calls
Ans + format43.803.90
Correct-gated efficiency44.603.14
Direct-only probing45.403.65
Full BoundarySA47.913.10

Direct-only probing improves over the Ans+format baseline because it can identify some examples that do not require retrieval. However, it cannot distinguish whether a Non-direct example requires ImageSearch, TextSearch, or both tools. Consequently, its Non-direct predictions remain concentrated on the conservative Image+Text workflow. Full BoundarySA improves Direct-only probing while reducing average tool use , demonstrating the complementary value of pep_e and pap_a.

Correct-gated efficiency achieves lower accuracy than BoundarySA because it applies the same “fewer calls are better” preference to every correctly answered instance, without modeling why different instances require different numbers or types of tool calls. For example, a correct Direct instance should ideally use no tools, whereas a correct Image+Text instance may require both ImageSearch and TextSearch, and therefore inherently needs more calls. Correct-gated efficiency observes only answer correctness and trajectory length; it cannot distinguish the necessary multi-tool calls of the latter from genuinely redundant retrieval. This can disrupt the appropriate multi-tool behavior of instances that genuinely require both tools.

In contrast, BoundarySA provides instance-specific supervision: three probes therefore distinguish zero-tool, single-tool, and multi-tool requirements, allowing the model to reduce redundant calls without uniformly penalizing necessary retrieval. This complementary capability decomposition explains why BoundarySA achieves a better accuracy–efficiency trade-off than either Correct-gated efficiency or Direct-only probing.


Q3. Does the pre-RL capability target remain applicable after SFT and RL?

We reported results further as follows:

Model stagepep_epap_apdp_dAgreement with Base target
Base0.310.180.10100.00%
After SFT0.340.210.1292.00%
After RL0.380.250.1486.14%

All three capability scores increase after SFT and RL, confirming that the model’s capability improves during training. This is consistent with our original formulation. As stated in the Capability-Boundary Shifts during Training paragraph, BoundarySA does not assume that the target remains fixed.

Our appendix shows that, even when the boundary shifts, the original answer and format rewards are preserved:

RBoundarySA=rans+βrformat+λransrbound.R_{\mathrm{BoundarySA}} = r_{\mathrm{ans}} + \beta r_{\mathrm{format}} + \lambda r_{\mathrm{ans}} r_{\mathrm{bound}}.

The additional boundary term is non-negative and gated by answer correctness. It therefore cannot reduce the baseline reward, and an incorrect trajectory cannot receive a bonus merely by matching the initial target.

The results provide direct empirical support: 92.00% of the SFT targets and 86.14% of the final RL targets remain identical to their Base targets. Moreover, BoundarySA improves accuracy from 43.80% to 47.91% while reducing average tool calls from 3.90 to 3.10. Thus, the initial target remains sufficiently stable to provide useful guidance while preserving the model’s ability to improve during training.


Q2,Q4: Questions about more tools and datasets.

In future work, we plan to extend our investigation of capability boundaries to a broader range of datasets and more diverse tool settings.

Thank you for the response. However, the restricted action space (Q2) is a fundamental limitation of the proposed method. In addition, the limited evaluation (Q4) does not sufficiently demonstrate the method's efficiency and generalizability. Therefore, I will maintain my original score.

AuthorsOpenReview ↗

Response to Reviewer civd [Supplementary Response]

For Restricted Paths

Thank you for your follow-up reply! We conducted a evaluation on the same MMHops test examples.

Probed target groupForced DirectForced ImageForced TextForced Image+Text
Direct62.24 / 0.0062.10 / 1.6762.05 / 2.2461.90 / 3.60
Image-only28.42 / 0.0055.52 / 2.1333.67 / 2.1554.28 / 3.69
Text-only24.46 / 0.0028.13 / 2.4052.57 / 2.5652.68 / 3.74
Image+Text11.39 / 0.0018.27 / 2.4913.43 / 2.6844.43 / 3.72

The results support an accuracy–cost interpretation of the probed targets. BoundarySA therefore does not indiscriminately suppress retrieval: it retains the full path when both tools are needed.

Allowing both ImageSearch and TextSearch gives the model access to more complete external evidence and can therefore remain beneficial for some examples. However, full retrieval is not uniformly advantageous across capability groups. Compared with the capability-compatible paths, forcing Image+Text decreases accuracy from 62.24% to 61.90% in the Direct group and from 55.52% to 54.28% in the Image-only group, while also requiring substantially more tool calls. This observation is consistent with our discussion in the original Tool Calls and Top-KK ablation: excessive retrieval may introduce irrelevant information and retrieval noise, increasing the burden on evidence summarization and subsequent reasoning and potentially interfering with questions that the model could already solve directly or with a single tool. In contrast, capability-aware routing selects tools according to the capability gap of each example, reducing the average number of calls from 3.71 to 3.10 while maintaining overall answer accuracy. Its benefit therefore does not come from simply minimizing search, but from avoiding redundant retrieval and its potential noise while preserving necessary evidence acquisition.

For More Evaluation

Thank you for this valuable suggestion. We agree that SUPERGLASSES is an important benchmark for evaluating VLM agents in realistic egocentric scenarios.

Although SUPERGLASSES is related to our task through its use of external knowledge retrieval, its focus on heterogeneous, real-world egocentric smart-glasses scenarios differs in scope from the long-tail, knowledge-intensive KBVQA setting studied in this work. We consider this an important direction and plan to extend BoundarySA to broader and more complex scenarios in future work.

Reviewer_SfWsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BoundarySA, a capability-aware search agent for Knowledge-Based Visual Question Answering (KB-VQA), to address the tendency of RL-based KBVQA search agents to over-rely on fixed retrieval pipelines. The method first probes the model's entity identification, entity-conditioned answering, and direct answering capabilities. These estimates are then used to construct compatibility scores for four tool-use paths, which guide RL training through a boundary-consistency reward. Experiments on MMHops, InfoSeek, and E-VQA demonstrate improved answer accuracy and reduced tool calls compared with existing baselines.

Critical strengths

S1. The paper identifies an important mode of RL-trained KB-VQA search agents: despite having the option to answer directly or use individual tools, they may converge to a conservative image-search-then-text-search workflow. Figures 3 and 4 provide useful behavioral evidence of this tendency. Focusing on when external evidence is actually needed, rather than only improving retrieval quality, addresses a practical source of unnecessary computation and potential retrieval noise. This problem framing is consequential because it targets the interaction between a model's internal capabilities and external tools.

S2. To address the overreliance on fixed retrieval pipelines identified above, the authors decompose the model's existing capabilities into three aspects: entity identification, entity-conditioned answering, and direct answering. Based on these capabilities, they construct four path-compatibility scores and incorporate a boundary-consistency reward to guide RL training, encouraging the agent to leverage its internal knowledge before invoking external tools. The proposed framework achieves better performance than the evaluated baselines on MMHops, InfoSeek, and E-VQA while reducing average tool calls, demonstrating its practical effectiveness.

Critical weaknesses

W1. The paper shows that BoundarySA improves overall accuracy (Tables 1–2) and encourages more diverse, capability-aligned retrieval paths (Figures 3–4). However, it remains unclear whether the accuracy gains actually result from avoiding unnecessary image/text searches, rather than other effects of boundary-aware RL training. In particular, the paper does not directly compare the answer accuracy of capability-compatible paths against the conventional image+text workflow on the same instances. Such a comparison is important to establish why capability-aware routing improves task success, rather than merely reducing tool calls.

W2. Table 5 demonstrates improved answer accuracy on InfoSeek and E-VQA, and Appendix Table 11 reports fewer tool calls, but neither establishes that the learned capability-aware routing behavior transfers to these datasets. The path-alignment analysis in Figure 4 is confined to the MMHops training setting. Without corresponding alignment results, preferably accompanied by answer accuracy broken down by target path, it remains unclear whether improvements on unseen benchmarks arise from capability-compatible tool selection or from more general gains in the trained agent. This weakens the paper's specific mechanism-generalization claim.

Questions for authors

Q1. Could the authors provide a controlled comparison of answer accuracy between capability-compatible retrieval paths and the conventional image+text workflow on the same instances, to clarify whether the improved routing behavior directly contributes to the accuracy gains?

Q2. Could the authors extend the path-alignment analysis in Figure 4 to InfoSeek and E-VQA, ideally with accuracy breakdowns by target path, to verify whether capability-aware tool selection generalizes across datasets?

Q3. As acknowledged in Appendix E.1, the four-category boundary reward cannot distinguish trajectories with different numbers of repeated search calls. Given that the image+text path remains dominant at later RL steps in Figure 3, while direct answering becomes rare, could the authors further analyze the distribution of tool-call counts and repeated searches within this category, particularly for multi-hop questions? This would help clarify whether BoundarySA effectively reduces redundant retrieval beyond encouraging more diverse tool-use paths.

1 comment
AuthorsOpenReview ↗

Response to Reviewer SfWs

We thank the reviewer for the thoughtful questions.

Q1 and Q3: Controlled path evaluation and tool use within the Image+Text category

For Q1, we reported results as follows.

Probed target groupForced DirectForced ImageForced TextForced Image+Text
Direct62.24 / 0.0062.10 / 1.6762.05 / 2.2461.90 / 3.60
Image-only28.42 / 0.0055.52 / 2.1333.67 / 2.1554.28 / 3.69
Text-only24.46 / 0.0028.13 / 2.4052.57 / 2.5652.68 / 3.74
Image+Text11.39 / 0.0018.27 / 2.4913.43 / 2.6844.43 / 3.72

The results support an accuracy–cost interpretation of the probed targets. BoundarySA therefore does not indiscriminately suppress retrieval: it retains the full path when both tools are needed.

Allowing both ImageSearch and TextSearch gives the model access to more complete external evidence and can therefore remain beneficial for some examples. However, full retrieval is not uniformly advantageous across capability groups. Compared with the capability-compatible paths, forcing Image+Text decreases accuracy from 62.24% to 61.90% in the Direct group and from 55.52% to 54.28% in the Image-only group, while also requiring substantially more tool calls. This observation is consistent with our discussion in the original Tool Calls and Top-KK ablation: excessive retrieval may introduce irrelevant information and retrieval noise, increasing the burden on evidence summarization and subsequent reasoning and potentially interfering with questions that the model could already solve directly or with a single tool. In contrast, capability-aware routing selects tools according to the capability gap of each example, reducing the average number of calls from 3.71 to 3.10 while maintaining overall answer accuracy. Its benefit therefore does not come from simply minimizing search, but from avoiding redundant retrieval and its potential noise while preserving necessary evidence acquisition.

To further address Q3, we analyze the examples whose probed target is Image+Text. In this group, BoundarySA achieves 44.43% accuracy with 3.72 average tool calls, whereas the Original baseline achieves 42.01% accuracy with 4.14 calls. BoundarySA therefore improves accuracy by 2.42 percentage points while reducing average tool use by 0.42 calls approximately101˙approximately 10\.1%. This shows that BoundarySA preserves the necessary Image+Text behavior while reducing tool calls within this path category.


Q2: Cross-dataset generalization of capability-aware routing

We extended the target-path alignment analysis to E-VQA and InfoSeek and results are as follows.

Target PathInfoSeek Origin Align.%InfoSeek BoundarySA Align.%InfoSeek Avg. CallsE-VQA Origin Align.%E-VQA BoundarySA Align.%E-VQA Avg. Calls
Direct6650.208680.30
Image-only42731.1045751.20
Text-only68791.3070801.40
Image+Text84861.7584861.71

The Original baseline is not trained with capability-aware path supervision. Its substantially higher alignment on the Image+Text target group than on the other groups is consistent with the behavioral collapse observed in Figure 4: it frequently adopts the conservative image-search-then-text-search workflow even when a simpler path is sufficient. In contrast, BoundarySA achieves higher alignment across all four target groups.

Together with the answer-accuracy and tool-call results on E-VQA and InfoSeek, these results show that BoundarySA’s capability-aware routing behavior generalizes beyond MMHops.

Reviewer_tL1POpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes BoundarySA to probe the model's capabilities for entity recognition, entity-conditioned answering, and direct answering. It constructs instance-specific path rewards for a search agent. The training process combines SFT with GSPO. Experiments on multiple datasets report improved answer accuracy and reduced tool usage.

Critical strengths
  1. The paper attempts to address the problem of redundant retrieval for KBVQA.
  2. It provides a basis for adapting tool use to individual instances.
  3. The results are reported in accuracy and tool usage across benchmarks. Cross-dataset evaluation provides evidence of practical benefits.
Critical weaknesses
  1. The path target obtained from detection lacks sufficient verification. It is not clear whether these scores are reliable in responding to the actual search needs. The summary returned by the image search may provide both the facts related to the identity of the entity and the answer. Therefore, there may not be a correspondence between the ability gap and the required tools assumed by the author.
  2. The necessity of using all three detections is unclear. The existing experiments support the overall effectiveness, but do not clearly establish the additional benefit of using all detections over some simpler alternatives.
  3. The claim of retaining the ability of the model requires more direct evidence. The path goal remains unchanged in the process of SFT and RL, but the model capability may change.
  4. Some reproduction details are unclear, including the prompts, judgment methods and entity matching rules. These affect the reliability of the path target and the interpretation of the experimental comparison.
Questions for authors
  1. What evidence shows that the target obtained from detection corresponds to an appropriate retrieval path? In particular, when the model is restricted to different retrieval paths, do the success rate and tool cost of each path support the mapping between ability and retrieval needs?
  2. How much additional benefit does using all three detections provide over simpler alternatives, such as using only direct answer ability detection or applying a simple efficiency reward only when the answer is correct?
  3. How will the abilities of the model change after SFT and RL? What evidence shows that the initial path target is still applicable?
2 comments
AuthorsOpenReview ↗

Response to Reviewer tL1P

We thank the reviewer for the thoughtful questions . We reported more details as follows:

Q1. For "Do the probed targets correspond to appropriate retrieval paths?"

We conducted a evaluation on the same MMHops test examples.

Probed target groupForced DirectForced ImageForced TextForced Image+Text
Direct62.24 / 0.0062.10 / 1.6762.05 / 2.2461.90 / 3.60
Image-only28.42 / 0.0055.52 / 2.1333.67 / 2.1554.28 / 3.69
Text-only24.46 / 0.0028.13 / 2.4052.57 / 2.5652.68 / 3.74
Image+Text11.39 / 0.0018.27 / 2.4913.43 / 2.6844.43 / 3.72

The results support an accuracy–cost interpretation of the probed targets. BoundarySA therefore does not indiscriminately suppress retrieval: it retains the full path when both tools are needed.

Allowing both ImageSearch and TextSearch gives the model access to more complete external evidence and can therefore remain beneficial for some examples. However, full retrieval is not uniformly advantageous across capability groups. Compared with the capability-compatible paths, forcing Image+Text decreases accuracy from 62.24% to 61.90% in the Direct group and from 55.52% to 54.28% in the Image-only group, while also requiring substantially more tool calls. This observation is consistent with our discussion in the original Tool Calls and Top-KK ablation: excessive retrieval may introduce irrelevant information and retrieval noise, increasing the burden on evidence summarization and subsequent reasoning and potentially interfering with questions that the model could already solve directly or with a single tool. In contrast, capability-aware routing selects tools according to the capability gap of each example, reducing the average number of calls from 3.71 to 3.10 while maintaining overall answer accuracy. Its benefit therefore does not come from simply minimizing search, but from avoiding redundant retrieval and its potential noise while preserving necessary evidence acquisition.

Q2. Are all three capability probes necessary?

We added two simpler alternatives:

  1. Correct-gated efficiency: fewer tool calls are rewarded only when the final answer is correct.

  2. Direct-only probing: only pdp_d is used to divide examples into Direct and Non-direct groups, without distinguishing Image-only, Text-only, and Image+Text requirements.

MethodAccuracy %Avg. calls
Ans + format43.803.90
Correct-gated efficiency44.603.14
Direct-only probing45.403.65
Full BoundarySA47.913.10

Direct-only probing improves over the Ans+format baseline because it can identify some examples that do not require retrieval. However, it cannot distinguish whether a Non-direct example requires ImageSearch, TextSearch, or both tools. Consequently, its Non-direct predictions remain concentrated on the conservative Image+Text workflow. Full BoundarySA improves Direct-only probing while reducing average tool use , demonstrating the complementary value of pep_e and pap_a.

Correct-gated efficiency achieves lower accuracy than BoundarySA because it applies the same “fewer calls are better” preference to every correctly answered instance, without modeling why different instances require different numbers or types of tool calls. For example, a correct Direct instance should ideally use no tools, whereas a correct Image+Text instance may require both ImageSearch and TextSearch, and therefore inherently needs more calls. Correct-gated efficiency observes only answer correctness and trajectory length; it cannot distinguish the necessary multi-tool calls of the latter from genuinely redundant retrieval. This can disrupt the appropriate multi-tool behavior of instances that genuinely require both tools.

In contrast, BoundarySA provides instance-specific supervision: three probes therefore distinguish zero-tool, single-tool, and multi-tool requirements, allowing the model to reduce redundant calls without uniformly penalizing necessary retrieval. This complementary capability decomposition explains why BoundarySA achieves a better accuracy–efficiency trade-off than either Correct-gated efficiency or Direct-only probing.

Q3. Does the pre-RL capability target remain applicable after SFT and RL?

We reported results further as follows:

Model stagepep_epap_apdp_dAgreement with Base target
Base0.310.180.10100.00%
After SFT0.340.210.1292.00%
After RL0.380.250.1486.14%

All three capability scores increase after SFT and RL, confirming that the model’s capability improves during training. This is consistent with our original formulation. As stated in the Capability-Boundary Shifts during Training paragraph, BoundarySA does not assume that the target remains fixed.

Our appendix shows that, even when the boundary shifts, the original answer and format rewards are preserved:

RBoundarySA=rans+βrformat+λransrbound.R_{\mathrm{BoundarySA}} = r_{\mathrm{ans}} + \beta r_{\mathrm{format}} + \lambda r_{\mathrm{ans}} r_{\mathrm{bound}}.

The additional boundary term is non-negative and gated by answer correctness. It therefore cannot reduce the baseline reward, and an incorrect trajectory cannot receive a bonus merely by matching the initial target.

The results provide direct empirical support: 92.00% of the SFT targets and 86.14% of the final RL targets remain identical to their Base targets. Moreover, BoundarySA improves accuracy from 43.80% to 47.91% while reducing average tool calls from 3.90 to 3.10. Thus, the initial target remains sufficiently stable to provide useful guidance while preserving the model’s ability to improve during training.

Q4. Reproducibility details

Thank you for pointing this out. We will provide the remaining implementation and reproducibility details in the camera-ready version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.