- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper points out a pretty important issue in multimodal search task synthesis: just because some evidence is used to generate a question doesn't mean the model actually needs that evidence to answer it. A lot of existing synthetic tasks have shortcuts, like answers being searchable through text, visual evidence being skippable, or intermediate answers basically being leaked in the question.
To fix this, the authors propose EvidenceWeave, which tries to make sure each reasoning step actually depends on the intended evidence. It first checks whether atomic QA pairs are supported by the source and whether the required evidence is really necessary, then connects these pairs into multi-hop reasoning chains while hiding intermediate answers. The goal is to generate harder and more meaningful multimodal search tasks that force agents to actually use different sources and tools.
For validation, the authors compare their synthesized queries against existing baselines and show that EvidenceWeave questions are harder to solve through shortcuts. They also use the generated trajectories to post-train Qwen3.5-27B, getting 5.2–9.3 percentage point improvements across five benchmarks. The trained model also shows more diverse and connected visual tool usage, suggesting the synthetic data helps it learn better multimodal search behaviors, rather than just improving answer accuracy.
Critical strengths
-
Identifying and addressing a fundamental issue in synthetic task generation. The paper makes a strong observation that evidence used to construct a question isn't necessarily needed to solve it. The authors clearly identify several common shortcuts and propose a systematic framework to address them through evidence qualification, dependency composition, and implicit intermediate answers. I think this is a meaningful contribution since the problem is quite general and the proposed approach could be useful for future multimodal agent data synthesis.
-
Strong empirical results with meaningful behavioral improvements. The paper shows consistent gains of 5.2–9.3 percentage points across five benchmarks after post-training on EvidenceWeave trajectories. More importantly, the analysis shows that the model uses visual tools more frequently and in more diverse ways, suggesting it actually learns better multimodal search behavior rather than just improving benchmark scores. This makes the practical value of the framework more convincing and is a major reason to support acceptance.
Critical weaknesses
-
Unfair comparison with existing methods. In Table 1, the authors compare EvidenceWeave-27B with existing search agents such as REDSearcher-MM, OpenSearch-VL, and MTA-DeepSearch. However, these baselines use older base models, while EvidenceWeave uses the newer and stronger Qwen3.5-27B. This makes the comparison less convincing, since it's unclear whether the improvements come from better training data or simply from a stronger base model.
-
Lack of controlled comparisons and unclear novelty. The paper does not compare different data synthesis methods using the same base model. For example, the authors could fine-tune Qwen3.5-27B on data generated by MTA or OpenSearch-VL and compare the results with EvidenceWeave. Also, similar verification steps have already been explored in MTA and other prior works, so the novelty of the proposed pipeline is not very clear. Without a fair comparison, it's hard to tell whether EvidenceWeave actually produces better training data or offers meaningful improvements over existing methods. This is a major concern since both data quality and the novelty of the synthesis pipeline are central to the paper's contributions.
Questions for authors
-
Ablation of key components: Can the authors provide an ablation study that separates the contributions of premise qualification, dependency composition, and intermediate-result masking? These are the main design choices of EvidenceWeave, and understanding their individual impact is important for assessing whether the proposed framework is actually necessary or if simpler alternatives could achieve similar gains.
-
Training Qwen-3-VL with your data or train Qwen3.5 with MTA or other baseline data to make the comparison fair.
-
Did I miss the cost analysis?
3 comments
I think my concern about baseline and novelty is not solved. I will keep my score. Thanks so much for your response.
Thank you for following up. In our response, we reported additional comparisons in which we trained the same Qwen3.5-27B backbone on queries from several baseline methods and from EvidenceWeave, using the same teacher and training settings (Response to the Common Questions (2/3): Baseline Comparisons). We also explained how EvidenceWeave differs from each method (Response to the Common Questions (3/3): Methodological Novelty). In light of these results, could you please clarify which baseline comparison you still find insufficient, or which aspect of the methodological distinction remains unclear? This would help us address your concern more directly.
Response to Weaknesses and Questions
Thank you for recognizing the framework’s value and the behavioral improvements. We address your central concern—whether the gains reflect data quality or backbone strength—through the same-backbone comparison you suggested, followed by component evidence and costs.
W1, W2 & Q2. Same-Backbone Training Comparisons
We trained Qwen3.5-27B on both EvidenceWeave and public baseline queries under the same settings. The comparison includes OpenSearch-VL, REDSearcher-MM, and combinations with Vision-DeepResearch, using the same quality control, Qwen3.7-Plus teacher, SFT configuration, and evaluation procedure. EvidenceWeave uses 335M training-data tokens, fewer than the alternatives’ 388M–994M.
EvidenceWeave leads on all four benchmarks, with a mean gain of 7.18 points, compared with changes of −0.48 to +2.93 points for the public-query alternatives. OpenSearch-VL data reduce performance by 1.4 points on MMBC and 3.5 points on VisB, whereas EvidenceWeave improves throughout. The table appears in “Response to the Common Questions (2/3): Baseline Comparisons” above and is now included in Figure 6 and Table 5 (Appendix E.5) of the revised manuscript.
In the original Table 1, the comparison is between the base model and EvidenceWeave-27B. External published scores marked with † provide context. The new comparison directly addresses the effect of query source with the backbone held fixed.
W2. Methodological Novelty Relative to Prior Work
Our methodological innovation is a task-construction model that derives qualification, composition, and query-formulation criteria from assessed information requirements. Verification addresses a common need in data synthesis, but its presence in MTA and other prior work does not establish equivalent construction methods. The specific distinctions are detailed in “Response to the Common Questions (3/3): Methodological Novelty” above. The same-backbone results further demonstrate the training value of the resulting queries under the same settings.
Q1. Component Ablations
The component evidence combines a small-scale development-stage training experiment with query-quality comparisons. Full ablations require repeating construction through training and evaluation. Data preparation through teacher-trajectory synthesis cost approximately US$34,000 for slightly over 10,000 queries and approximately 19,000 trajectories, averaging US$3.2 per query or US$1.8 per trajectory from the same total expenditure. “Response to the Common Questions (1/3): Component Ablations” above provides the detailed rationale and results:
Q3. Cost Analysis
Data preparation through teacher-trajectory synthesis cost approximately US$34,000, producing slightly over 10,000 queries and approximately 19,000 trajectories. Averaging the same total expenditure over queries or trajectories gives US$3.2 per query or US$1.8 per trajectory, respectively. The approximate breakdown is:
Inference tool use is reported under shared budget limits. Figure 12 shows mean calls increasing from 27.48 to 45.98 on MMBC, 8.42 to 13.80 on MMS+, 13.65 to 17.80 on BC-V³, and 9.52 to 13.71 on VisB. Figure 7 distinguishes increased multimodal retrieval and inspection from relatively stable web-search use. The revised manuscript reports the construction-cost summary in §5.1 and the breakdown in Appendix E.3 (Table 3).
We hope these comparisons and cost details address your concerns and welcome further questions.