- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work introduces RACE, a data synthesis framework designed to train multimodal search agents on questions that genuinely require visual understanding and the integration of multiple pieces of evidence. RACE achieves this through two main strategies. The first, Referential Abstraction (RA), removes directly identifiable entity names and instead describes targets through visual references. Some of these references are deliberately ambiguous, requiring the agent to consult additional information to identify the correct entity. The second, Compositional Evidence (CE), combines information associated with multiple entities, either from the same image or from separate images, to create questions that cannot be answered without the specified evidence. Both strategies can be incorporated into intermediate stages of a search process to generate more involved reasoning trajectories. For data generation, the authors employ Qwen3.7-Plus to produce successful search trajectories and subsequently fine-tune Qwen3.5-27B using 12K synthesized examples. Evaluation on ten benchmarks shows that average accuracy rises from 53.5% to 56.0%. The behavioral analysis further suggests that the fine-tuned agent makes greater use of visual tools across successive search rounds.
Critical strengths
1. A thoughtful approach to improving the quality of multimodal search supervision.
One appealing aspect of this work is its attention to whether the generated questions actually demand the capabilities they are intended to teach. The authors highlight two recurring problems with existing training examples: the possibility of answering questions without inspecting visual content, and the inclusion of evidence that contributes little to the final answer. Rather than relying solely on more complicated question generation, RACE addresses these problems at the task-design level. RA makes entity identification dependent on visual information, while CE aims to ensure that multiple evidence sources contribute meaningfully to the solution. I consider this a valuable direction for developing more effective supervision for multimodal search agents.
2. Consistent performance improvements despite a modest amount of training data.
The experimental findings provide encouraging support for the proposed approach. RACE improves performance on nine of the ten evaluated benchmarks, suggesting that its benefits are not restricted to a single evaluation setting. Moreover, the model trained on only 12K RACE trajectories performs better than the counterpart trained on a 28K-example open-source mixture on each of the four main benchmarks. Another interesting observation is the change in the agent's search patterns, particularly its more frequent use of visual tools in later search rounds. Together, these results suggest that RACE can improve both downstream task performance and the way agents interact with visual evidence.
Critical weaknesses
1. The experimental evidence does not sufficiently distinguish the effects of the two proposed components.
RA and CE constitute the main methodological contributions of RACE, yet the current evaluation leaves their individual effectiveness unclear. In particular, the absence of controlled experiments that separately enable RA and CE makes it difficult to attribute the reported gains to either component. It also remains uncertain whether using both together offers meaningful advantages over adopting just one of them. Since the paper builds its main contribution around these two mechanisms, a clearer experimental breakdown would be important for understanding what drives the improvements and whether the full framework is justified.
2. The evaluation does not convincingly demonstrate that agents depend on the intended visual evidence.
The main motivation behind RACE is to generate tasks for which visual grounding and the combination of designated evidence sources are indispensable. However, the existing validation mainly examines properties of the generated questions and uses limited probes to identify potential shortcuts. It does not directly establish whether the trained agent uses the intended evidence when producing its answers. Higher answer accuracy or more frequent visual-tool calls cannot, by themselves, rule out alternative solution paths that bypass some of the required information. Consequently, the paper would benefit from stronger evidence that its training procedure leads agents to genuinely depend on the visual and compositional relationships encoded in the synthesized tasks.
Questions for authors
1. How much does each component contribute to the final performance?
Would it be possible to evaluate RA and CE separately, alongside the complete RACE framework, while keeping the training configuration and data scale consistent? This comparison would make it easier to understand whether the two strategies offer distinct benefits and how much is gained by combining them.
2. Do agents actually use every intended piece of evidence?
The proposed data construction process is designed to make particular images and evidence sources necessary for answering a question. However, how is this property verified in the behavior of the trained model? For example, could the authors selectively remove or substitute individual evidence sources and examine how the agent's predictions change? Such experiments could help establish whether the intended dependencies are learned rather than merely encoded during data generation.
3. How effective are the checks for unintended shortcuts?
The authors use question-only probing to filter examples that may reveal entity identities or answers without requiring the intended evidence. I would like to know how well these checks work in practice. What fraction of candidate questions fails the checks or requires modification? More importantly, is there any estimate of how frequently unintended shortcuts remain in the final training set?
4. Are the performance gains maintained under equivalent search budgets?
RACE-trained agents appear to invoke visual tools more frequently and make more tool calls overall. This raises the question of how much of the accuracy improvement comes from better search decisions, as opposed to additional search opportunities. Could the authors compare the baseline and RACE-trained models under the same limits on tool calls or computational expenditure? This would provide a clearer picture of the efficiency and effectiveness of the learned search behavior.
5. Could differences in the training-data distribution explain part of the improvement?
Compared with the other training datasets, RACE contains more examples involving multiple images and richer sequences of visual-tool interactions. These characteristics may independently affect downstream performance. Have the authors considered controlling for such differences, for instance by matching the proportion of multi-image examples or the frequency of visual-tool interactions? This would help separate the value of the proposed evidence-dependency mechanisms from the broader effects of training-data composition.
3 comments
Follow-up Response to Reviewer 7ytr (1/2)
Thank you for the follow-up. To address your three questions, we add two evaluation-time analyses of the trained agent (masking the returns of visual search tools and judging the benchmarks' annotated required steps), accuracy under matched tool-call budgets, and an open-source control with RACE's size and composition.
1. Does the trained agent depend on all intended evidence?
At evaluation, masking the returns of image search and text-to-image search lowers RACE's accuracy from 51.5 to 36.1 on VisB and from 40.2 to 35.2 on MS+; on MM-BC, RACE's 2.2-point improvement in overall accuracy is matched by a 2.2-point improvement in Strict Accuracy, which also requires every annotated step to be completed. Following your suggestion, the first analysis removes a source of evidence and measures how the trained agent's accuracy changes; the second checks whether the agent completes every required step. All other settings are as in Table 1.
(a) Masking the returns of visual search tools. The tool set and interaction limits are unchanged; we mask only the returns of image search, text-to-image search, or both, replacing them with a text placeholder, while image cropping, web search, and page fetching return normally and the input images are unchanged. RACE's accuracy (%) on VisB and MS+:
| Masked returns | VisB | MS+ |
|---|---|---|
| None | 51.5 | 40.2 |
| Image search | 43.8 | 35.7 |
| Text-to-image search | 36.7 | 36.1 |
| Both | 36.1 | 35.2 |
Masking both returns lowers RACE's accuracy by 15.4 points on VisB and by 5.0 points on MS+, and masking either one alone also lowers it on both benchmarks. These results show that the trained agent relies on the information returned by its visual search tools.
(b) Judging the benchmarks' annotated required steps. For every question, MM-BrowseComp provides a verified irreducible reasoning checklist of the steps required to reach the answer, and its official Strict Accuracy counts a question as correct only when the answer is correct and every checklist step is completed. BrowseComp-V3 provides expert-validated subgoals for every question. On the evaluation trajectories behind Table 1, we judge every step with each benchmark's official protocol and the same judge for both agents:
| Metric (%) | Base | RACE | Change |
|---|---|---|---|
| MM-BC accuracy | 31.3 | 33.5 | +2.2 |
| of which: every checklist step completed (Strict Accuracy) | 21.9 | 24.1 | +2.2 |
| of which: correct with some checklist steps missing | 9.4 | 9.4 | 0.0 |
| MM-BC checklist score | 53.0 | 55.3 | +2.3 |
| BC-V3 accuracy | 31.7 | 36.7 | +5.0 |
| BC-V3 subgoal completion | 56.0 | 62.5 | +6.5 |
RACE improves Strict Accuracy by 2.2 points (21.9% → 24.1%), matching its overall accuracy gain, while the percentage of questions answered correctly with incomplete checklists remains at 9.4%. RACE also completes more checklist steps overall (53.0 → 55.3) and more of BC-V3's expert-validated subgoals (56.0 → 62.5).
Revision (new Appendix G; Section 4.5): Appendix G reports the tables and protocols of both analyses, and Section 4.5 summarizes them.
Follow-up Response to Reviewer 7ytr (2/2)
2. Do the gains hold under matched search budgets?
On MM-BC and BC-V3, RACE achieves higher accuracy than the base agent at every search budget, from the tightest to the loosest. We apply a per-question tool-call budget to the evaluation trajectories behind Table 1; a trajectory that has not produced an answer within the budget is counted as incorrect. Accuracy (%) at each budget:
| MM-BC | ≤30 | ≤40 | ≤50 | ≤60 | ≤100 | No budget |
|---|---|---|---|---|---|---|
| Base | 29.0 | 31.3 | 31.3 | 31.3 | 31.3 | 31.3 |
| RACE | 29.9 | 32.1 | 32.1 | 32.6 | 33.5 | 33.5 |
| RACE − Base | +0.9 | +0.8 | +0.8 | +1.3 | +2.2 | +2.2 |
| BC-V3 | ≤10 | ≤13 | ≤15 | ≤20 | ≤30 | No budget |
|---|---|---|---|---|---|---|
| Base | 18.7 | 22.7 | 25.7 | 31.3 | 31.7 | 31.7 |
| RACE | 23.0 | 29.7 | 30.7 | 34.0 | 36.3 | 36.7 |
| RACE − Base | +4.3 | +7.0 | +5.0 | +2.7 | +4.6 | +5.0 |
Under the same search budget, RACE is 0.8–2.2 points higher on MM-BC and 2.7–7.0 points higher on BC-V3, so the improvement reflects better search decisions rather than additional search opportunities.
Revision (Appendix E, new Table 13; Section 4.5): Appendix E reports the tables above, and Section 4.5 adds: "At every tool-call budget, RACE achieves higher accuracy than the base agent on MM-BC and BC-V3 (Appendix E)."
3. Is the effect of the training-data distribution controlled?
With RACE's size and multi-image share, open-source data still train a weaker agent than RACE on all four primary benchmarks (37.8 vs. 40.5 on average). We draw 12,309 trajectories from the 28K open-source mixture with RACE's composition (5,071 multi-image and 7,238 single-image, a 41.2% multi-image share), train the same backbone on them with the same SFT configuration as RACE, and evaluate with the settings of Table 2:
| Qwen3.5-27B training data | Trajectories | Multi-image share | MM-BC | MS+ | BC-V3 | VisB | Avg. |
|---|---|---|---|---|---|---|---|
| Open-source 28K (full) | 28,198 | 18.1% | 32.1 | 37.9 | 36.3 | 48.5 | 38.7 |
| Open-source, composition-matched | 12,309 | 41.2% | 31.7 | 37.9 | 33.7 | 47.9 | 37.8 |
| RACE | 12,309 | 41.2% | 33.5 | 40.2 | 36.7 | 51.5 | 40.5 |
With the same number of trajectories and the same multi-image share, RACE is 1.8–3.6 points higher on every benchmark and 2.7 points higher on average. On MS+, which contains multi-image questions, the composition-matched control scores the same as the full mixture (37.9). The multi-image share therefore does not explain the gap between RACE and the open-source data.
Revision (Appendix B.3, new Table 7; Section 4.3): Appendix B.3 reports the table above and the sampling procedure, and Section 4.3 adds: "An open-source control drawn from this mixture with RACE's size and 41.2% multi-image share remains below RACE on all four primary benchmarks, by 2.7 points on average (Appendix B.3)."
These results and changes are marked in red in the revised PDF. We welcome further discussion.
Thank you for the detailed responses and additional experiments. The new RA-only ablation and data analysis have addressed some of my concerns and improved the paper. However, important issues remain regarding whether the trained agent genuinely depends on all intended evidence, whether the performance gains hold under matched search budgets, and whether the effects of training-data distribution are sufficiently controlled. Overall, while I appreciate the authors' efforts, these concerns have not been fully resolved. Therefore, I maintain my original rating.