acceptodds
Under review as a conference paper at ICLR 2027

RACE: Learning Multimodal Search through Referential Abstraction and Compositional Evidence

Abstract

Training multimodal search agents requires tasks that make visually grounded search necessary for determining the answer. Yet synthetic questions can admit correct answers while bypassing the image or ignoring evidence from intended targets, weakening the resulting supervision. We propose RACE, an image-conditioned task synthesis framework that constructs dependencies from images to targets and from target-specific evidence to answers. Referential Abstraction (RA) replaces identity-revealing text with specific visual references or ambiguous relational references resolved by searching among depicted candidates. Compositional Evidence (CE) links retrieved facts about multiple entities in one image or entities from different images into joint questions whose answers require every designated contribution. Both operations also apply to images found during search, allowing new visual observations to determine subsequent search targets or supply answer evidence. Fine-tuning Qwen3.5-27B on only 12K teacher-generated trajectories from RACE raises average accuracy across ten multimodal benchmarks from 53.5 to 56.0, a relative improvement of approximately 4.7% over the matched base agent. Our code, data, and model will be publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal deep search agents, synthetic data generation, agent post-training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7ytrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces RACE, a data synthesis framework designed to train multimodal search agents on questions that genuinely require visual understanding and the integration of multiple pieces of evidence. RACE achieves this through two main strategies. The first, Referential Abstraction (RA), removes directly identifiable entity names and instead describes targets through visual references. Some of these references are deliberately ambiguous, requiring the agent to consult additional information to identify the correct entity. The second, Compositional Evidence (CE), combines information associated with multiple entities, either from the same image or from separate images, to create questions that cannot be answered without the specified evidence. Both strategies can be incorporated into intermediate stages of a search process to generate more involved reasoning trajectories. For data generation, the authors employ Qwen3.7-Plus to produce successful search trajectories and subsequently fine-tune Qwen3.5-27B using 12K synthesized examples. Evaluation on ten benchmarks shows that average accuracy rises from 53.5% to 56.0%. The behavioral analysis further suggests that the fine-tuned agent makes greater use of visual tools across successive search rounds.

Critical strengths

1. A thoughtful approach to improving the quality of multimodal search supervision.

One appealing aspect of this work is its attention to whether the generated questions actually demand the capabilities they are intended to teach. The authors highlight two recurring problems with existing training examples: the possibility of answering questions without inspecting visual content, and the inclusion of evidence that contributes little to the final answer. Rather than relying solely on more complicated question generation, RACE addresses these problems at the task-design level. RA makes entity identification dependent on visual information, while CE aims to ensure that multiple evidence sources contribute meaningfully to the solution. I consider this a valuable direction for developing more effective supervision for multimodal search agents.

2. Consistent performance improvements despite a modest amount of training data.

The experimental findings provide encouraging support for the proposed approach. RACE improves performance on nine of the ten evaluated benchmarks, suggesting that its benefits are not restricted to a single evaluation setting. Moreover, the model trained on only 12K RACE trajectories performs better than the counterpart trained on a 28K-example open-source mixture on each of the four main benchmarks. Another interesting observation is the change in the agent's search patterns, particularly its more frequent use of visual tools in later search rounds. Together, these results suggest that RACE can improve both downstream task performance and the way agents interact with visual evidence.

Critical weaknesses

1. The experimental evidence does not sufficiently distinguish the effects of the two proposed components.

RA and CE constitute the main methodological contributions of RACE, yet the current evaluation leaves their individual effectiveness unclear. In particular, the absence of controlled experiments that separately enable RA and CE makes it difficult to attribute the reported gains to either component. It also remains uncertain whether using both together offers meaningful advantages over adopting just one of them. Since the paper builds its main contribution around these two mechanisms, a clearer experimental breakdown would be important for understanding what drives the improvements and whether the full framework is justified.

2. The evaluation does not convincingly demonstrate that agents depend on the intended visual evidence.

The main motivation behind RACE is to generate tasks for which visual grounding and the combination of designated evidence sources are indispensable. However, the existing validation mainly examines properties of the generated questions and uses limited probes to identify potential shortcuts. It does not directly establish whether the trained agent uses the intended evidence when producing its answers. Higher answer accuracy or more frequent visual-tool calls cannot, by themselves, rule out alternative solution paths that bypass some of the required information. Consequently, the paper would benefit from stronger evidence that its training procedure leads agents to genuinely depend on the visual and compositional relationships encoded in the synthesized tasks.

Questions for authors

1. How much does each component contribute to the final performance?

Would it be possible to evaluate RA and CE separately, alongside the complete RACE framework, while keeping the training configuration and data scale consistent? This comparison would make it easier to understand whether the two strategies offer distinct benefits and how much is gained by combining them.

2. Do agents actually use every intended piece of evidence?

The proposed data construction process is designed to make particular images and evidence sources necessary for answering a question. However, how is this property verified in the behavior of the trained model? For example, could the authors selectively remove or substitute individual evidence sources and examine how the agent's predictions change? Such experiments could help establish whether the intended dependencies are learned rather than merely encoded during data generation.

3. How effective are the checks for unintended shortcuts?

The authors use question-only probing to filter examples that may reveal entity identities or answers without requiring the intended evidence. I would like to know how well these checks work in practice. What fraction of candidate questions fails the checks or requires modification? More importantly, is there any estimate of how frequently unintended shortcuts remain in the final training set?

4. Are the performance gains maintained under equivalent search budgets?

RACE-trained agents appear to invoke visual tools more frequently and make more tool calls overall. This raises the question of how much of the accuracy improvement comes from better search decisions, as opposed to additional search opportunities. Could the authors compare the baseline and RACE-trained models under the same limits on tool calls or computational expenditure? This would provide a clearer picture of the efficiency and effectiveness of the learned search behavior.

5. Could differences in the training-data distribution explain part of the improvement?

Compared with the other training datasets, RACE contains more examples involving multiple images and richer sequences of visual-tool interactions. These characteristics may independently affect downstream performance. Have the authors considered controlling for such differences, for instance by matching the proportion of multi-image examples or the frequency of visual-tool interactions? This would help separate the value of the proposed evidence-dependency mechanisms from the broader effects of training-data composition.

3 comments

Thank you for the detailed responses and additional experiments. The new RA-only ablation and data analysis have addressed some of my concerns and improved the paper. However, important issues remain regarding whether the trained agent genuinely depends on all intended evidence, whether the performance gains hold under matched search budgets, and whether the effects of training-data distribution are sufficiently controlled. Overall, while I appreciate the authors' efforts, these concerns have not been fully resolved. Therefore, I maintain my original rating.

AuthorsOpenReview ↗

Follow-up Response to Reviewer 7ytr (1/2)

Thank you for the follow-up. To address your three questions, we add two evaluation-time analyses of the trained agent (masking the returns of visual search tools and judging the benchmarks' annotated required steps), accuracy under matched tool-call budgets, and an open-source control with RACE's size and composition.

1. Does the trained agent depend on all intended evidence?

At evaluation, masking the returns of image search and text-to-image search lowers RACE's accuracy from 51.5 to 36.1 on VisB and from 40.2 to 35.2 on MS+; on MM-BC, RACE's 2.2-point improvement in overall accuracy is matched by a 2.2-point improvement in Strict Accuracy, which also requires every annotated step to be completed. Following your suggestion, the first analysis removes a source of evidence and measures how the trained agent's accuracy changes; the second checks whether the agent completes every required step. All other settings are as in Table 1.

(a) Masking the returns of visual search tools. The tool set and interaction limits are unchanged; we mask only the returns of image search, text-to-image search, or both, replacing them with a text placeholder, while image cropping, web search, and page fetching return normally and the input images are unchanged. RACE's accuracy (%) on VisB and MS+:

Masked returnsVisBMS+
None51.540.2
Image search43.835.7
Text-to-image search36.736.1
Both36.135.2

Masking both returns lowers RACE's accuracy by 15.4 points on VisB and by 5.0 points on MS+, and masking either one alone also lowers it on both benchmarks. These results show that the trained agent relies on the information returned by its visual search tools.

(b) Judging the benchmarks' annotated required steps. For every question, MM-BrowseComp provides a verified irreducible reasoning checklist of the steps required to reach the answer, and its official Strict Accuracy counts a question as correct only when the answer is correct and every checklist step is completed. BrowseComp-V3 provides expert-validated subgoals for every question. On the evaluation trajectories behind Table 1, we judge every step with each benchmark's official protocol and the same judge for both agents:

Metric (%)BaseRACEChange
MM-BC accuracy31.333.5+2.2
of which: every checklist step completed (Strict Accuracy)21.924.1+2.2
of which: correct with some checklist steps missing9.49.40.0
MM-BC checklist score53.055.3+2.3
BC-V3 accuracy31.736.7+5.0
BC-V3 subgoal completion56.062.5+6.5

RACE improves Strict Accuracy by 2.2 points (21.9% → 24.1%), matching its overall accuracy gain, while the percentage of questions answered correctly with incomplete checklists remains at 9.4%. RACE also completes more checklist steps overall (53.0 → 55.3) and more of BC-V3's expert-validated subgoals (56.0 → 62.5).

Revision (new Appendix G; Section 4.5): Appendix G reports the tables and protocols of both analyses, and Section 4.5 summarizes them.

AuthorsOpenReview ↗

Follow-up Response to Reviewer 7ytr (2/2)

2. Do the gains hold under matched search budgets?

On MM-BC and BC-V3, RACE achieves higher accuracy than the base agent at every search budget, from the tightest to the loosest. We apply a per-question tool-call budget to the evaluation trajectories behind Table 1; a trajectory that has not produced an answer within the budget is counted as incorrect. Accuracy (%) at each budget:

MM-BC≤30≤40≤50≤60≤100No budget
Base29.031.331.331.331.331.3
RACE29.932.132.132.633.533.5
RACE − Base+0.9+0.8+0.8+1.3+2.2+2.2
BC-V3≤10≤13≤15≤20≤30No budget
Base18.722.725.731.331.731.7
RACE23.029.730.734.036.336.7
RACE − Base+4.3+7.0+5.0+2.7+4.6+5.0

Under the same search budget, RACE is 0.8–2.2 points higher on MM-BC and 2.7–7.0 points higher on BC-V3, so the improvement reflects better search decisions rather than additional search opportunities.

Revision (Appendix E, new Table 13; Section 4.5): Appendix E reports the tables above, and Section 4.5 adds: "At every tool-call budget, RACE achieves higher accuracy than the base agent on MM-BC and BC-V3 (Appendix E)."

3. Is the effect of the training-data distribution controlled?

With RACE's size and multi-image share, open-source data still train a weaker agent than RACE on all four primary benchmarks (37.8 vs. 40.5 on average). We draw 12,309 trajectories from the 28K open-source mixture with RACE's composition (5,071 multi-image and 7,238 single-image, a 41.2% multi-image share), train the same backbone on them with the same SFT configuration as RACE, and evaluate with the settings of Table 2:

Qwen3.5-27B training dataTrajectoriesMulti-image shareMM-BCMS+BC-V3VisBAvg.
Open-source 28K (full)28,19818.1%32.137.936.348.538.7
Open-source, composition-matched12,30941.2%31.737.933.747.937.8
RACE12,30941.2%33.540.236.751.540.5

With the same number of trajectories and the same multi-image share, RACE is 1.8–3.6 points higher on every benchmark and 2.7 points higher on average. On MS+, which contains multi-image questions, the composition-matched control scores the same as the full mixture (37.9). The multi-image share therefore does not explain the gap between RACE and the open-source data.

Revision (Appendix B.3, new Table 7; Section 4.3): Appendix B.3 reports the table above and the sampling procedure, and Section 4.3 adds: "An open-source control drawn from this mixture with RACE's size and 41.2% multi-image share remains below RACE on all four primary benchmarks, by 2.7 points on average (Appendix B.3)."

These results and changes are marked in red in the revised PDF. We welcome further discussion.


Reviewer_zoneOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to synthesize training tasks that require multimodal search rather than merely permit it. It identifies two shortcuts: textual clues can reveal an image target's identity, and a multi-entity question can be answered without using every intended entity. RACE addresses these through Referential Abstraction (RA), which grounds target references in images, and Compositional Evidence (CE), which makes selected facts jointly necessary for the answer. The construction also extends to images retrieved during search. Fine-tuning Qwen3.5-27B on 12K teacher trajectories raises its mean accuracy across ten benchmarks from 53.5% to 56.0%.

Critical strengths
  1. The paper frames a concrete supervision problem as dependencies from images to targets and from target evidence to answers. RA's unique-reference checks and CE's necessity criterion make the intended task structure clearer than a generic call for more visual search.
  2. The matched base-versus-SFT comparison shows gains on all four primary search benchmarks. The paper also examines tool use during training and evaluation, and acknowledges that its bounded shortcut probes cannot rule out every alternative solution.
Critical weaknesses
  1. The motivation uses constructed examples but does not measure how often identity leakage or unused-entity evidence occurs in existing training data. This leaves the practical importance of the proposed gap, relative to prior visual-dependency and multi-entity work, unverified.
  2. The methodological novelty over related synthesis work is not well established. HyperEyes already combines visual references with questions requiring evidence from every selected entity, while DeepVoyager-VL enforces necessary visual evidence and minimal answer constraints. RACE adds search-based disambiguation of depicted candidates and applies these dependencies to natural and cross-image scenes, but the experiments do not isolate the value of these additions. There are no RA-only, CE-only, or reference-type ablations; Table 2 also changes training-data source, size, and composition.
  3. Most specialized-agent rows in Table 1 have missing scores, while published scores use different settings. This limits claims of superiority over prior agents, although the matched base-agent comparison still supports gains from RACE SFT.
  4. The behavioral analysis counts visual calls but does not show that visual evidence changes searches or answers; no solver-level image or evidence removal test is reported.
Questions for authors

All questions are presented in the weakness section.

Reviewer_FccKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the construction of training tasks for multimodal search agents. RACE combines Referential Abstraction, which replaces explicit entity identities with image-grounded references, and Compositional Evidence, which constructs questions requiring facts from multiple targets. These operations can also be applied to images retrieved during search. The resulting tasks are filtered and solved by a teacher model, and 12K successful trajectories are used to fine-tune Qwen3.5-27B. The trained model improves average accuracy across ten benchmarks from 53.5% to 56.0%, with gains on nine benchmarks. The paper also reports increased visual-tool use during later interaction rounds.

Critical strengths
  1. The paper makes a useful distinction between obtaining a correct answer and teaching the intended search behavior. The examples clearly illustrate how textual identity clues or unnecessary entity evidence can undermine multimodal supervision.

  2. RA and CE address complementary parts of task construction: identifying the intended target and making its evidence contribute to the answer. Applying these operations to both initial and retrieved images gives the framework scope beyond input-image recognition.

  3. The empirical gains are reasonably consistent. RACE improves all four primary benchmarks by 2.2–5.0 percentage points under matched evaluation settings, and its 12K trajectories outperform the compared 28K open-source SFT mixture on all four.

Critical weaknesses
  1. I'm not fully convinced that the designated images and evidence are truly necessary. Appendix C acknowledges that generator explanations are not solver tests under image or evidence removal. The question-only probe is useful, but the paper reports neither an independent audit nor tests withholding individual required images. It remains unclear how reliably the retained tasks satisfy the central construction criteria.

  2. There are no ablations separating the contributions of RA, CE, and shortcut filtering. Table 2 compares complete datasets with different sources and generation procedures, so it cannot identify which design choices account for the gains. A simpler synthesis baseline using the same teacher would also help establish the value of the proposed construction.

  3. The behavioral results show more visual-tool use, but do not establish more effective use of visual evidence. On MM-BrowseComp, average tool calls increase from 30.0 to 50.4 while accuracy improves by 2.2 points. Without accuracy comparisons at comparable actual cost or uncertainty estimates for the accuracy differences, the practical benefit is difficult to assess.

Questions for authors
  1. Have you tested retained tasks with individual required images withheld, or audited whether successful teacher trajectories use all designated evidence?
  2. What happens when RA or CE is removed while keeping the teacher and training budget comparable?
  3. Does RACE retain its advantage at comparable inference cost, and how uncertain are the reported accuracy differences?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.