acceptodds
Under review as a conference paper at ICLR 2027

More Evidence to Build Than to Answer: The Sufficiency-Necessity Gap in Task Synthesis for Multimodal Search Agent Post-Training

Abstract

Multimodal deep search agents are increasingly post-trained on complex synthetic tasks designed to demand iterative retrieval and multi-step reasoning across visual and textual web evidence. Existing synthesis methods typically organize interconnected evidence into multi-step solution structures, deriving questions via structural constraints and solver-based filtering to promote task difficulty. However, evidence sufficient to construct an answerable query is not necessarily required to solve it: visually grounded facts may be recovered through text alone, intended premises can be bypassed, and intermediate answers may leak into the prompt. A task can therefore remain valid yet fail to supervise its intended capabilities—a mismatch we term the sufficiency-necessity gap. To bridge this gap, we introduce EvidenceWeave, a task synthesis framework that jointly addresses evidence access, premise dependence, and information exposure. By composing qualified atomic relations into multi-step dependencies while keeping intermediate answers implicit, EvidenceWeave ties evidence acquisition and intermediate reasoning to the information requested by the query. Extensive experiments show that our queries yield lower direct-answer recovery rates than the public queries evaluated and elicit visual evidence acquisition and use in correct teacher solutions. Fine-tuning Qwen3.5-27B on the resulting trajectories improves performance across five core benchmarks and leads to more varied and connected evidence use. These results support EvidenceWeave as an approach to narrowing the sufficiency-necessity gap by turning construction evidence into targeted supervision for the acquisition, interpretation, and use of multimodal evidence. Our code, data, and model will be publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal deep search, agent post-training, task synthesis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jsofOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper points out a pretty important issue in multimodal search task synthesis: just because some evidence is used to generate a question doesn't mean the model actually needs that evidence to answer it. A lot of existing synthetic tasks have shortcuts, like answers being searchable through text, visual evidence being skippable, or intermediate answers basically being leaked in the question.

To fix this, the authors propose EvidenceWeave, which tries to make sure each reasoning step actually depends on the intended evidence. It first checks whether atomic QA pairs are supported by the source and whether the required evidence is really necessary, then connects these pairs into multi-hop reasoning chains while hiding intermediate answers. The goal is to generate harder and more meaningful multimodal search tasks that force agents to actually use different sources and tools.

For validation, the authors compare their synthesized queries against existing baselines and show that EvidenceWeave questions are harder to solve through shortcuts. They also use the generated trajectories to post-train Qwen3.5-27B, getting 5.2–9.3 percentage point improvements across five benchmarks. The trained model also shows more diverse and connected visual tool usage, suggesting the synthetic data helps it learn better multimodal search behaviors, rather than just improving answer accuracy.

Critical strengths
  1. Identifying and addressing a fundamental issue in synthetic task generation. The paper makes a strong observation that evidence used to construct a question isn't necessarily needed to solve it. The authors clearly identify several common shortcuts and propose a systematic framework to address them through evidence qualification, dependency composition, and implicit intermediate answers. I think this is a meaningful contribution since the problem is quite general and the proposed approach could be useful for future multimodal agent data synthesis.

  2. Strong empirical results with meaningful behavioral improvements. The paper shows consistent gains of 5.2–9.3 percentage points across five benchmarks after post-training on EvidenceWeave trajectories. More importantly, the analysis shows that the model uses visual tools more frequently and in more diverse ways, suggesting it actually learns better multimodal search behavior rather than just improving benchmark scores. This makes the practical value of the framework more convincing and is a major reason to support acceptance.

Critical weaknesses
  1. Unfair comparison with existing methods. In Table 1, the authors compare EvidenceWeave-27B with existing search agents such as REDSearcher-MM, OpenSearch-VL, and MTA-DeepSearch. However, these baselines use older base models, while EvidenceWeave uses the newer and stronger Qwen3.5-27B. This makes the comparison less convincing, since it's unclear whether the improvements come from better training data or simply from a stronger base model.

  2. Lack of controlled comparisons and unclear novelty. The paper does not compare different data synthesis methods using the same base model. For example, the authors could fine-tune Qwen3.5-27B on data generated by MTA or OpenSearch-VL and compare the results with EvidenceWeave. Also, similar verification steps have already been explored in MTA and other prior works, so the novelty of the proposed pipeline is not very clear. Without a fair comparison, it's hard to tell whether EvidenceWeave actually produces better training data or offers meaningful improvements over existing methods. This is a major concern since both data quality and the novelty of the synthesis pipeline are central to the paper's contributions.

Questions for authors
  1. Ablation of key components: Can the authors provide an ablation study that separates the contributions of premise qualification, dependency composition, and intermediate-result masking? These are the main design choices of EvidenceWeave, and understanding their individual impact is important for assessing whether the proposed framework is actually necessary or if simpler alternatives could achieve similar gains.

  2. Training Qwen-3-VL with your data or train Qwen3.5 with MTA or other baseline data to make the comparison fair.

  3. Did I miss the cost analysis?

3 comments
AuthorsOpenReview ↗

Response to Weaknesses and Questions

Thank you for recognizing the framework’s value and the behavioral improvements. We address your central concern—whether the gains reflect data quality or backbone strength—through the same-backbone comparison you suggested, followed by component evidence and costs.

W1, W2 & Q2. Same-Backbone Training Comparisons

We trained Qwen3.5-27B on both EvidenceWeave and public baseline queries under the same settings. The comparison includes OpenSearch-VL, REDSearcher-MM, and combinations with Vision-DeepResearch, using the same quality control, Qwen3.7-Plus teacher, SFT configuration, and evaluation procedure. EvidenceWeave uses 335M training-data tokens, fewer than the alternatives’ 388M–994M.

EvidenceWeave leads on all four benchmarks, with a mean gain of 7.18 points, compared with changes of −0.48 to +2.93 points for the public-query alternatives. OpenSearch-VL data reduce performance by 1.4 points on MMBC and 3.5 points on VisB, whereas EvidenceWeave improves throughout. The table appears in “Response to the Common Questions (2/3): Baseline Comparisons” above and is now included in Figure 6 and Table 5 (Appendix E.5) of the revised manuscript.

In the original Table 1, the comparison is between the base model and EvidenceWeave-27B. External published scores marked with †\dagger provide context. The new comparison directly addresses the effect of query source with the backbone held fixed.

W2. Methodological Novelty Relative to Prior Work

Our methodological innovation is a task-construction model that derives qualification, composition, and query-formulation criteria from assessed information requirements. Verification addresses a common need in data synthesis, but its presence in MTA and other prior work does not establish equivalent construction methods. The specific distinctions are detailed in “Response to the Common Questions (3/3): Methodological Novelty” above. The same-backbone results further demonstrate the training value of the resulting queries under the same settings.

Q1. Component Ablations

The component evidence combines a small-scale development-stage training experiment with query-quality comparisons. Full ablations require repeating construction through training and evaluation. Data preparation through teacher-trajectory synthesis cost approximately US$34,000 for slightly over 10,000 queries and approximately 19,000 trajectories, averaging US$3.2 per query or US$1.8 per trajectory from the same total expenditure. “Response to the Common Questions (1/3): Component Ablations” above provides the detailed rationale and results:

  • Premise qualification: the variant without qualification has 17.9% no-tool correctness on an approximately 1,000-query subset, compared with the paper’s 4.73% full-dataset rate. The development-stage model also trails full EvidenceWeave by 1.3–6.3 points.
  • Dependency composition: using predecessor answers as subsequent premises is established in MuSiQue [2], MTA-DeepSearch [1], and OpenSearch-VL [3]. Our contribution concerns the access and premise conditions governing composition, motivating the focus on qualification and masking.
  • Intermediate-result masking: the exposed-entity variant has 26.3% no-tool correctness on an approximately 1,000-query subset, compared with the paper’s 4.73% full-dataset rate. The higher rejection rate supports masking’s role in retaining more queries for teacher synthesis.

Q3. Cost Analysis

Data preparation through teacher-trajectory synthesis cost approximately US$34,000, producing slightly over 10,000 queries and approximately 19,000 trajectories. Averaging the same total expenditure over queries or trajectories gives US$3.2 per query or US$1.8 per trajectory, respectively. The approximate breakdown is:

Cost CategoryShare of Total Cost
Media screening27%
Textual-retrieval access probes19%
Atomic QA extraction15%
Atomic QA verification13%
Other construction operations26%

Inference tool use is reported under shared budget limits. Figure 12 shows mean calls increasing from 27.48 to 45.98 on MMBC, 8.42 to 13.80 on MMS+, 13.65 to 17.80 on BC-V³, and 9.52 to 13.71 on VisB. Figure 7 distinguishes increased multimodal retrieval and inspection from relatively stable web-search use. The revised manuscript reports the construction-cost summary in §5.1 and the breakdown in Appendix E.3 (Table 3).

We hope these comparisons and cost details address your concerns and welcome further questions.

[1] Peng et al. MTA-Agent: An Open Recipe for Multimodal Deep Search Agents. COLM, 2026.

[2] Trivedi et al. MuSiQue: Multihop Questions via Single-hop Question Composition. TACL, 2022.

[3] Chen et al. OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents. Preprint, 2026.

I think my concern about baseline and novelty is not solved. I will keep my score. Thanks so much for your response.

Thank you for following up. In our response, we reported additional comparisons in which we trained the same Qwen3.5-27B backbone on queries from several baseline methods and from EvidenceWeave, using the same teacher and training settings (Response to the Common Questions (2/3): Baseline Comparisons). We also explained how EvidenceWeave differs from each method (Response to the Common Questions (3/3): Methodological Novelty). In light of these results, could you please clarify which baseline comparison you still find insufficient, or which aspect of the methodological distinction remains unclear? This would help us address your concern more directly.

Reviewer_yAAZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EvidenceWeave for synthesizing multimodal search tasks. The authors argue that while constructing a problem may involve multiple images and multi-step reasoning, this does not mean the model genuinely needs such information when solving it. To address this, the paper generates training data by verifying evidence, filtering questions, and concealing intermediate answers, achieving performance improvements across five benchmarks.

Critical strengths

1.The problem addressed by this paper is of practical significance; existing multimodal search tasks can indeed allow models to bypass images or intermediate reasoning steps 2.The paper is well written, the overall methodology is easy to follow, and the experiments demonstrate certain performance gains.

Critical weaknesses

W1. Limited novelty.The main idea of this paper is quite close to existing work. MuSiQue [1] already proposed checking the dependencies between questions before composing single-hop questions into multi-hop ones. FORT-Searcher [2] studied how to avoid shortcuts in search tasks, and OpenSearch-VL [3] also used methods for concealing intermediate entities and filtering questions. Although this paper applies these ideas to multimodal search, it does not sufficiently clarify what new methods it actually proposes compared to these works. In addition, the paper does not cite MuSiQue [1].

W2. Lack of critical experiments.The paper combines several existing ideas, but does not separately verify whether these designs are truly useful. The current experiments only show that model performance improves after training on the generated data, but cannot demonstrate which of the proposed designs contribute to these gains. Comparisons with simpler data generation methods are recommended.

References [1] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multihop Questions via Single-hop Question Composition. Transactions of the Association for Computational Linguistics (TACL), 10:539–554. [2] Jia Deng, Yimeng Chen, Xiaoqing Xiang, Ziyang Zeng, Shuo Tang, Wayne Xin Zhao, Feng Chang, Chuan Hao, Yuan Wei, Ran Tao, Bryan Dai, and Ji-Rong Wen. 2026. FORT-Searcher: Synthesizing Shortcut-Resistant Search Tasks for Training Deep Search Agents. arXiv preprint arXiv:2606.12087.
[3] Shuang Chen, Kaituo Feng, Hangting Chen, Wenxuan Huang, Dasen Dai, Quanxin Shou, Yunlong Lin, Xiangyu Yue, Shenghua Gao, and Tianyu Pang. 2026. OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents. arXiv preprint arXiv:2605.05185.

Questions for authors

Q1. Compared to MuSiQue [1], FORT-Searcher [2], and OpenSearch-VL [3], what is the primary new method proposed in this paper?

Q2. If some of the checking and filtering steps are removed, can similar results still be achieved?

Q3. Can it be further verified whether the generated questions truly require visual information to answer? Under the same tool-call budget, does the performance still improve?

3 comments
AuthorsOpenReview ↗

Response to Weaknesses and Questions

Thank you for recognizing the practical importance of the problem and identifying relevant prior work. We clarify the methodological distinctions before addressing the component experiments and evaluation conditions.

W1 & Q1. Methodological Contribution and Relationship to Prior Work

EvidenceWeave introduces a task-construction model that derives qualification, composition, and query-formulation criteria from assessed information requirements. Dependency checks, shortcut controls, and intermediate-answer concealment address common concerns in complex QA synthesis. Sharing these concerns does not make the methods for addressing them equivalent. The distinctions from MuSiQue, FORT-Searcher, and OpenSearch-VL are detailed in “Response to the Common Questions (3/3): Methodological Novelty” above. The revised manuscript cites MuSiQue in §4.2 and clarifies our construction model in §§1 and 4.

W2 & Q2. Component Ablations and Comparisons with Existing Methods

The tested variants yield smaller training gains or higher no-tool correct-answer rates. The small-scale development experiment without premise qualification trails full EvidenceWeave by 1.3–6.3 points. On approximately 1,000-query subsets, the variants without premise qualification and with intermediate entities exposed have no-tool correct-answer rates of 17.9% and 26.3%, respectively, compared with the paper’s 4.73% full-dataset rate. These higher rates imply more query rejection under the same screening protocol.

The component results and cost rationale appear in “Response to the Common Questions (1/3): Component Ablations” above. “Response to the Common Questions (2/3): Baseline Comparisons” above further shows that EvidenceWeave achieves larger and more consistent gains with fewer training-data tokens.

Q3. Visual Information Requirements and Shared Tool-Call Budgets

Visual tools substantially increase the proportion of questions answered correctly. On the same full set retained after no-tool screening under otherwise identical settings, textual retrieval answers 6.18% correctly at least once, versus 45.29% with full tools. During construction, 39.3% of QA groups sharing visual context and premises contain both answers matched and not matched by textual probes (Fig. 3a), supporting access assessment per conditional QA.

Table 1 already uses the same maximum inference-step budget for the base and trained models. EvidenceWeave receives no larger allowance. Actual mean steps depend on tool-use and stopping decisions, so equal limits do not imply identical consumption. Tool selection, sequencing, and frequency are behaviors affected by post-training (Figs. 7 and 12). Forcing identical realized cost would constrain those behaviors and answer a different question from evaluation under an equal budget.

We hope these clarifications address your concerns and welcome your feedback or further questions.

Reviewer_yAAZOpenReview ↗

Final Assessment After Rebuttal

Thank you for your detailed responses and the additional experiments. I appreciate the time and effort you have invested in addressing the reviewers' concerns.

I agree that the same-backbone comparisons provide meaningful evidence for the effectiveness of the generated training data. The text-only versus full-tool comparison also addresses much of my concern regarding the importance of visual access. These results have strengthened my assessment of the paper's empirical contribution.

Regarding novelty, I acknowledge that EvidenceWeave differs from prior work by jointly considering premise dependence and evidence access when constructing tasks. My concern is not that an existing method implements exactly the same pipeline, but rather that the individual techniques are largely established, and the significance of their combination is not yet fully demonstrated.

The additional ablation results are helpful, and I understand the computational cost of conducting full-scale experiments. However, the current component comparisons remain somewhat limited in establishing how much the proposed design contributes beyond simpler alternatives.

Overall, I appreciate the authors' efforts and recognize the practical value of this work. While the rebuttal has addressed several of my concerns, I still have some reservations about the methodological novelty. Therefore, I will maintain my original rating of Weak Reject.

Thank you again for the constructive discussion.

Thank you for clarifying your assessment and for recognizing the empirical contribution. We respectfully disagree with your characterization of EvidenceWeave as a combination of established techniques. We also disagree with your assessment of what the component comparisons show about the value of our construction checks over simpler alternatives.

First, we distinguish the problems that arise at each stage of task construction from the techniques used to solve them. EvidenceWeave and prior work address common problems, including how to verify premise dependence and how to conceal intermediate answers. Our acknowledgment of that overlap concerns the problems themselves. Establishing that EvidenceWeave combines existing techniques would require showing that prior methods already use the specific checks and construction rules we describe. In “Response to the Common Questions (3/3): Methodological Novelty,” we compared those techniques directly and explained where they differ. Shared construction stages alone do not support the characterization of EvidenceWeave as a combination of established techniques.

Second, the simpler alternatives at issue were tested directly. In “Response to the Common Questions (1/3): Component Ablations,” both omitting premise qualification and exposing intermediate answers yielded substantially higher no-tool answer rates than the full-method reference under the same screening protocol. Since every query answered at this stage is discarded, the comparison measures a practical consequence of the design: how often a generated query can be solved through the very shortcut the construction check is intended to prevent. A request to assign each check a share of the final benchmark gain asks a different question. It does not justify discounting the observed differences between the proposed design and the simpler alternatives.

Our rebuttal explains how EvidenceWeave checks which premises and evidence a question requires. It also shows why the checks matter. In both simplified variants, the solver answered a larger share of questions without tools.

We respect your decision to retain your original rating, but we strongly disagree with how your assessment characterizes both the methodological distinctions and the experimental evidence, and therefore with the judgment of the work’s value that follows from it.

Reviewer_SPXPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1(−1)
Soundness
2
Read review
Paper summary

This paper asks whether evidence used to construct a multimodal search task is actually needed to solve it. EvidenceWeave checks atomic question-answer relations, combines their dependencies, and generates queries that hide intermediate answers. The authors then fine-tune Qwen3.5-27B on about 19K teacher trajectories. Table 1 reports gains of 5.2 - 9.3 percentage points across five search benchmarks. The formal results describe the constructed dependency structure; solver probes assess bypasses under specified access conditions.

Critical strengths
  1. I found the distinction between construction evidence and necessary evidence useful. A long dependency graph does not automatically make a task difficult for a solver with other retrieval routes. Sections 3-4 and Appendix D explain this distinction clearly and acknowledge what finite probes can establish.

  2. The improvements are consistent across the five main benchmarks, at 7.1, 5.2, 9.3, 7.1, and 6.5 percentage points (Table 1). These comparisons use the same backbone, making them useful evidence that the complete training pipeline helps.

Critical weaknesses
  1. My main hesitation is that Table 1 compares the original backbone with SFT on the new corpus. The roughly 19K teacher trajectories could help through ordinary supervision as well as the proposed construction checks. Appendix B explicitly acknowledges that the complete-framework comparison does not isolate those mechanisms. A same-teacher, same-training-budget construction control would help establish which design choices account for the improvement.

  2. The solving diagnostics support visual-tool use more directly than reduced bypass. The 4.73% no-tool success rate concerns direct answering from public input, including supplied media, without external tools. Visual-tool use in 96.1% of correct teacher trajectories establishes usage, not necessity. Completed-query textual probes are already described in Appendix C.4, but their aggregate recovery relative to a construction control remains unclear. Figure 11a also reports MMBC calls increasing from 27.48 to 45.98, about 67.3%. This describes an accuracy-cost trade-off under shared limits; it does not establish that additional calls caused the gains.

Questions for authors
  1. Is there an existing construction control using the same teacher, backbone, and training-token budget? If not, could you clarify which gains support the full supervision pipeline and which support the construction checks?

  2. For the completed-query probes already described in Appendix C.4, could you report recovery rates under textual retrieval and full tools with supplied media fixed? A bounded comparison against a minimally changed construction would help assess bypass reduction; a new benchmark is unnecessary.

I will further consider my score in light of the other reviewers’ comments and whether the authors’ response adequately addresses my concerns.

3 comments
AuthorsOpenReview ↗

Response to Weaknesses and Questions

Thank you for recognizing the distinction between construction evidence and necessary evidence. Your questions motivate a closer examination of what the training comparisons, construction controls, and tool-use observations each establish.

W1 & Q1. Construction Checks and Teacher Supervision

The public-query comparison holds the teacher and backbone fixed while changing the query source. Using the same Qwen3.7-Plus teacher, Qwen3.5-27B backbone, and downstream settings, EvidenceWeave achieves a mean gain of 7.18 points, versus changes of −0.48 to +2.93 points for public-query alternatives. It achieves these gains with 335M training-data tokens, while the alternatives use 388M–994M. A stronger teacher or a larger supervision corpus therefore cannot explain EvidenceWeave’s advantage. The full comparison appears in “Response to the Common Questions (2/3): Baseline Comparisons” above.

For individual mechanisms, the small-scale development experiment without premise qualification remains 1.3–6.3 points below the complete method. Query-level controls separately examine how qualification and masking affect direct answerability, as discussed below. The full tables and the cost rationale for these experiments appear in “Response to the Common Questions (1/3): Component Ablations” above.

W2 & Q2. Textual Bypass and Construction Controls

Removing online visual access substantially reduces the proportion of questions answered correctly. Across the same full set retained after no-tool screening under otherwise identical settings, textual retrieval answered 6.18% correctly at least once, compared with 45.29% under full tools. This 39.11-point gap directly supports dependence on online visual access under the evaluated conditions, beyond observing visual-tool calls in successful trajectories.

Given this proportion and the cost of retrieval attempts, we did not add textual-tool screening as a third mandatory filter in final query assessment. Construction-time textual probes remain part of evidence qualification and formulation (§4.1, Appendix C.4). Subsequent teacher-data selection excludes trajectories without visual-tool use, so retained demonstrations exercise multimodal tools (§5.1, Appendix E.2).

The minimally changed construction variants provide evidence of reduced direct-answer bypass. We assessed the variants without premise qualification and with intermediate entities exposed on approximately 1,000-query subsets. Below, we report these results alongside the paper’s full-dataset EvidenceWeave result, using the same four-attempt no-tool protocol with supplied media retained:

Construction VariantEvaluation ScopeAnswered Correctly at Least Once
Full EvidenceWeaveFull dataset (paper)4.73%
Without premise qualificationApproximately 1,000-query subset17.9%
Intermediate entities exposedApproximately 1,000-query subset26.3%

Both variants have higher no-tool correct-answer rates than the full-dataset reference, supporting the role of qualification and masking in limiting direct-answer bypass. This evidence concerns construction, while the textual-versus-full-tool comparison assesses dependence on online visual access. The component details appear in §§1 and 3 of “Response to the Common Questions (1/3): Component Ablations” above.

W2. Interpretation of Tool-Use Changes

The behavior analysis characterizes tool selection and evidence reuse, rather than attributing accuracy gains to call frequency. Mean MMBC calls increase from 27.48 to 45.98 under shared limits (Fig. 12a). Appendix B already states that “this comparison does not establish that increased visual-tool use caused those gains.”

Section 5.4 and Figure 7 distinguish three behavioral changes:

  • Tool selection: image cropping increases 6.5×–18.1×, reverse image search 2.6×–4.9×, text-to-image search 1.6×–2.1×, and webpage fetching 1.8×–2.8×, while web search changes only 0.9×–1.2×.
  • Evidence reuse: text-to-image search followed by cropping its output rises from 4.7% to 19.8% of tasks, and reverse search using a crop rises from 0.8% to 3.3%.
  • Tool diversity: tasks using at least two visual tool types increase from 17.3% to 48.8%, a change that repeated calls to one tool cannot explain.

Together, these observations show that higher accuracy accompanies broader and more connected use of multimodal evidence. We hope these results address your concerns and welcome further questions.

Thanks for the detailed rebuttal. The comparison with public queries using the same teacher and backbone makes a stronger case for the full EvidenceWeave pipeline. The text-only versus full-tool comparison also addresses part of my concern by showing that access to visual tools makes a substantial difference on the resulting tasks.

My main concern is still open. The public-query comparison changes the source of the training queries, and the premise-qualification ablation is small-scale. Taken together, these results still leave me unsure how much the proposed construction checks contribute to the gains at the main experimental scale. I share the other reviewers’ concern about the limited methodological advance over existing task-synthesis approaches. Given these remaining concerns, I’m keeping my original score

Thank you for clarifying your remaining concerns. We address each point below.

1. “The public-query comparison changes the source of the training queries.” Since the baseline methods differ in how they construct queries, we are unclear why varying the query source would weaken this comparison. Query source is the variable the experiment was designed to test. We held the backbone, teacher, quality control, training configuration, and evaluation fixed. Under these controls, EvidenceWeave queries yielded the strongest results on all four benchmarks, as reported in “Response to the Common Questions (2/3): Baseline Comparisons.” This experiment evaluates the training value of each pipeline’s output. We address individual construction checks separately.

2. “The premise-qualification ablation is small-scale,” leaving open how much the checks “contribute to the gains at the main experimental scale.” In “Response to the Common Questions (1/3): Component Ablations,” we tested the checks where their effects can be observed directly. Under the same four-attempt no-tool protocol, direct-answer recovery was 17.9% without premise qualification and 26.3% with intermediate entities exposed, each on approximately 1,000 queries, compared with 4.73% on the full EvidenceWeave dataset. Every query solved at this screen is discarded before teacher synthesis, so these differences directly affect query retention. We also reported a development-stage training comparison for premise qualification.

This evaluation strategy has clear precedents in search-agent post-training. MTA-Agent [1, Appendix C.1] evaluated construction changes on 100 generated questions per iteration and reported that its visual check reduced image redundancy from 14% to 6%. WebDancer [2, Fig. 3] compared data sources and trajectory filtering, while WebWatcher [3, §4.3] varied the tool-call criterion for selecting training trajectories. None of the three reports a separate main-scale training run for every query-construction step.

We disagree that such runs are necessary to support our component claims. They would address a distinct question about how many downstream benchmark points to attribute to each check. Our paper makes no per-check attribution of that kind. At the reported cost of approximately US$34,000 for data preparation through teacher synthesis, three comparable repetitions would cost about US$100,000 before training and evaluation. Given the direct measurements of shortcut frequency and the additional training comparison, we consider that requirement disproportionate to the claim being evaluated.

3. “I share the other reviewers’ concern about the limited methodological advance over existing task-synthesis approaches.” In “Response to the Common Questions (3/3): Methodological Novelty,” we compared EvidenceWeave with MuSiQue [4], FORT-Searcher [5], OpenSearch-VL [6], and MTA-Agent [1]. The distinction is specific. A question can require a preceding answer while still allowing a solver to recover the intended visual fact through text alone. EvidenceWeave checks premise dependence and evidence access separately, then uses both assessments to decide which relations to compose and which visual records are eligible for online acquisition. Could you please identify a prior method that already uses both assessments to make these decisions, or specify which part of our comparison you consider inaccurate?

The query-level experiments measure whether the checks reduce the shortcuts they target. The same-backbone experiments measure the training value of the resulting queries. We stand by both conclusions within those stated scopes.

[1] Peng et al. MTA-Agent: An Open Recipe for Multimodal Deep Search Agents. COLM, 2026.

[2] Wu et al. WebDancer: Towards Autonomous Information Seeking Agency. NeurIPS, 2025.

[3] Geng et al. WebWatcher: Breaking New Frontiers of Vision-Language Deep Research Agent. ICLR, 2026.

[4] Trivedi et al. MuSiQue: Multihop Questions via Single-hop Question Composition. TACL, 2022.

[5] Deng et al. FORT-Searcher: Synthesizing Shortcut-Resistant Search Tasks for Training Deep Search Agents. Preprint, 2026.

[6] Chen et al. OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents. Preprint, 2026.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Response to the Common Questions (1/3): Component Ablations

Thank you for the questions on component effectiveness. The central issue is how each construction component contributes to query quality and downstream training gains.

Overview of Component Analysis and Cost Constraints

Full training ablations require repeating query construction → quality control → teacher-trajectory synthesis → SFT → evaluation. Data preparation through teacher-trajectory synthesis cost approximately US$34,000, producing slightly over 10,000 queries and approximately 19,000 trajectories. This corresponds to US$3.2 per query or US$1.8 per trajectory, calculated from the same total expenditure. Repeating comparable data preparation for every component is prohibitive in time and cost. We therefore examine component effects through a small-scale development-stage training experiment and query-quality comparisons.

1. Premise Qualification

The variant without premise qualification yields smaller training gains and a higher no-tool correct-answer rate. During development, we trained Qwen3.5-27B on trajectories from queries constructed without premise qualification (§4.1, Eq. 1):

Qwen3.5-27B ConditionMMBCMMS+BC-V³VisB
Base model31.337.631.748.5
Without premise qualification32.140.839.752.1
Full EvidenceWeave38.442.841.055.6

The model improves over the base but trails full EvidenceWeave by 1.3–6.3 percentage points. The variant without premise qualification has a no-tool correct-answer rate of 17.9% on an approximately 1,000-query subset, compared with the paper’s 4.73% full-dataset rate for EvidenceWeave. Both follow the §5.1 protocol: at least one correct answer in four attempts by Qwen3.5-27B, retaining supplied media.

2. Dependency Composition

Using one question’s answer as another’s premise is an established composition principle, used by MuSiQue [1] (§§4–5), MTA-DeepSearch [2] (§3.2), and OpenSearch-VL [3] (§3.1). MuSiQue (§8.2) also demonstrates the value of dependency-aware filtering. Our contribution concerns the evidence-access and premise conditions governing composition (§§4.1–4.2, Appendix D.2): qualification determines admissible relations and whether visual evidence can be acquired through preceding results or remains a candidate for supplied media. Accordingly, we prioritized qualification and masking in the component experiments.

3. Intermediate-Result Masking and Query Retention

The masking comparison supports reduced direct-answer bypass and better query retention. Query formulation substitutes intermediate values with their defining relations (§4.3, Eq. 2). The same four-attempt no-tool assessment gives:

Query ConstructionEvaluation ScopeAnswered Correctly at Least Once
Standard maskingFull dataset (paper)4.73%
Intermediate entities exposedApproximately 1,000-query subset26.3%

The exposed-entity subset has a substantially higher no-tool correct-answer rate than the full-dataset reference. Since correctly answered queries are discarded before teacher synthesis, this finding supports masking’s role in reducing avoidable rejection and lowering the construction cost per retained query.

The high cost and long turnaround of post-training data construction make full training ablations of every design component prohibitively expensive and disproportionate in scope. Nevertheless, the query-retention results and development-stage training experiment together support the effectiveness of qualification and masking.

[1] Trivedi et al. MuSiQue: Multihop Questions via Single-hop Question Composition. TACL, 2022.

[2] Peng et al. MTA-Agent: An Open Recipe for Multimodal Deep Search Agents. COLM, 2026.

[3] Chen et al. OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents. Preprint, 2026.

AuthorsOpenReview ↗

Response to the Common Questions (2/3): Baseline Comparisons

Thank you for raising the question of fair baseline comparisons. The key question is whether EvidenceWeave’s gains reflect better queries rather than differences in teacher supervision, backbone, or training scale.

EvidenceWeave outperforms public-query alternatives under the same backbone and settings. We process queries from OpenSearch-VL [1], REDSearcher-MM [2], and combinations including Vision-DeepResearch (VDR) [3] through the same quality control, Qwen3.7-Plus teacher synthesis, Qwen3.5-27B SFT, and evaluation procedure (§5.1, Appendix E). Reusing public queries avoids repeating their construction, making this comparison more affordable than full component ablations.

Query Source Used to Train Qwen3.5-27BTraining-Data Tokens (M)MMBCMMS+BC-V³VisB
Base model—31.337.631.748.5
OpenSearch-VL38829.939.632.745.0
REDSearcher-MM51733.537.636.749.7
REDSearcher-MM + VDR60634.838.936.047.9
OpenSearch-VL + REDSearcher-MM + VDR99434.440.235.350.9
EvidenceWeave33538.442.841.055.6

EvidenceWeave uses 335M training-data tokens, fewer than every public-query alternative (388M–994M).

EvidenceWeave’s mean gain is 7.18 percentage points, compared with mean changes of −0.48 to +2.93 points for the alternatives. It improves on every benchmark, whereas OpenSearch-VL data cause declines of 1.4 points on MMBC and 3.5 points on VisB. These comparisons establish the practical training value of our constructed queries with backbone and teacher held fixed.

[1] Chen et al. OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents. Preprint, 2026.

[2] Chu et al. REDSearcher: A Scalable and Cost-Efficient Framework for Long-Horizon Search Agents. Preprint, 2026.

[3] Huang et al. Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal Large Language Models. ICML, 2026.

AuthorsOpenReview ↗

Response to the Common Questions (3/3): Methodological Novelty

Thank you for the questions on novelty. The question is what distinguishes EvidenceWeave’s construction method from prior approaches that address similar problems.

EvidenceWeave’s novelty lies in a task-construction model that derives qualification, composition, and query-formulation criteria from assessed information requirements. Evidence checking, dependency composition, and intermediate-answer concealment are common concerns in complex QA synthesis. Addressing these same concerns does not make the underlying construction methods equivalent. Sections 3–4 and Appendix D develop our model through an analysis of how successful solutions can bypass intended evidence and reasoning requirements.

This formulation gives each component a specific role in closing the sufficiency-necessity gap. Qualification assesses source support, premise dependence, and evidence-access requirements separately. Composition uses these assessments to determine admissible dependencies and whether evidence can be acquired through preceding results or remains a candidate for supplied media. Formulation conceals intermediate answers by expanding their qualified defining relations, preserving the dependencies that motivate concealment. The innovation therefore lies in modeling these construction decisions together and deriving their criteria from how successful solutions can bypass intended capabilities.

The distinctions from the cited methods concern how they address these shared requirements:

  • MuSiQue [1]: EvidenceWeave makes evidence availability part of the task being constructed. MuSiQue’s reported reading-comprehension construction checks predecessor-answer dependence with supporting passages supplied (§§4.2–4.3, Eq. 2). In EvidenceWeave, preceding results determine which evidence can be located and interpreted. These dependencies determine eligibility for online acquisition and which visual records remain candidates for supplied media (§4.1, Appendix D.2). Both methods use predecessor answers as subsequent premises, but their construction conditions differ. The revised manuscript cites MuSiQue in §4.2.

  • FORT-Searcher [2]: our construction criteria target the information requirements underlying post-training supervision. FORT analyzes cheap identifying routes and controls search-cost reductions to sustain long-horizon evidence acquisition (§§2–3). We examine how correct solutions bypass intended evidence-access and reasoning requirements. Weaker-access probes and premise checks determine which relations can participate in construction, linking task requirements to the capabilities successful solutions are intended to exercise.

  • OpenSearch-VL [3]: concealing a name does not establish that its predecessor remains necessary. Its rewriting requires answer invariance, unique identification, and absence of entity names or aliases (§3.1, Eq. 6). Table 6 in its Appendix D.2 marks as unique a candidate description identifying Steve Irwin through his fatal stingray incident, independently of the preceding zoo entity. EvidenceWeave checks for independently identifying clues and expands qualified defining relations (§4.1, Eq. 1 and §4.3, Eq. 2), making dependency preservation the purpose of concealment.

  • MTA-DeepSearch [4]: predecessor dependence does not establish the need to acquire the intended visual evidence. MTA checks bridge-entity dependence and whether search directly retrieves the merged question’s answer (§3.2.2). EvidenceWeave assesses evidence-access requirements separately from premise dependence, so an entity-dependent relation is not automatically treated as requiring online visual acquisition.

This makes the assessed requirements of solving a task the basis of its construction, linking the synthesis method to the capabilities its training examples are intended to exercise. The same-backbone comparisons in “Response to the Common Questions (2/3): Baseline Comparisons” further establish the training value of the resulting queries.

[1] Trivedi et al. MuSiQue: Multihop Questions via Single-hop Question Composition. TACL, 2022.

[2] Deng et al. FORT-Searcher: Synthesizing Shortcut-Resistant Search Tasks for Training Deep Search Agents. Preprint, 2026.

[3] Chen et al. OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents. Preprint, 2026.

[4] Peng et al. MTA-Agent: An Open Recipe for Multimodal Deep Search Agents. COLM, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.