acceptodds
Under review as a conference paper at ICLR 2027

FORT-Searcher: Synthesizing Shortcut-Resistant Search Tasks for Training Deep Search Agents

Abstract

Training deep-search agents requires verifiable questions that demand substantial evidence acquisition, yet structurally complex tasks may still admit cheap identifying routes when clues are overly selective, co-covered by the same evidence, directly searchable, or resolved from solver priors. We formalize this gap with a shortcut-aware difficulty framework that distinguishes intended construction complexity from identifying routes available through retrieval. Guided by this framework, we introduce FORT, a Framework of Shortcut-Resistant Training-Data Synthesis, which targets four shortcut risks across entity selection, evidence graph construction, question formulation, and adversarial refinement. We evaluate realized search behavior with solver-conditioned trajectory diagnostics rather than graph complexity or search length alone. Compared with existing open-source deep-search datasets, FORT shows later answer exposure, higher retrieval effort, lower single-clue selectivity, greater evidence dispersion, and higher realized pre-answer dependency costs. Using the resulting trajectories, we train FORT-Searcher with supervised fine-tuning alone. Across model scales, FORT-Searcher consistently improves over its backbones, achieving the strongest overall performance among similarly sized agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
deep-search agents, question synthesis, shortcut mitigation, agentic training data, information seeking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aNgCOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a data synthesize pipeline for deep research agents. It formulates 4 possible short-cuts of the generated question, and designs specific solutions in the pipeline to prevent these short-cuts. Performance of different backbone models can be improved after training on the proposed dataset. Experiments also show that later answer exposure is important for training effect.

Critical strengths
  1. The paper demonstrates an important phenomenon that answer exposure time is a significant metric besides total tool call number: within the same tool call number, later answer exposure in the data yields better training performance.
  2. The performance of the trained model is promising: 4B model achieves 74.4 in browsecomp, comparable with large models. Scores of other benchmarks are also competitive among baselines.
Critical weaknesses
  1. The paper formulates 4 different short-cut types during data synthesis, but only one of them (prior-knowledge binding) is clearly discussed and ablated in the experiments. The other short-cuts are not clearly identified and distinguished in the synthesized datasets and the baseline datasets.
  2. Most of the operations in the pipeline have been proposed in prior work (graph construction, long-tail entity selection, multi-source, entity fuzzing). Besides, the highlighted "adversarial refinement" relies on a strong model to refine the data and control the difficulty, hence the data quality is mainly determined by the intelligence of the used model, instead of the method itself. Prior work like WebShaper, InfoAgent and REDSearcher also use strong teacher model to refine and filter easy data. These baseline datasets can be easier mainly because the available model at that time is weaker than the current latest models.
  3. The comparison between baseline datasets mainly focus on analysis of trajectories from a solver model, e.g. accuracy, tool call number. Comprehensive comparison of training performance between these datasets is absent.
Questions for authors
  1. Can the authors clarify the exact model and prompts used in the data synthesis pipeline, especially the solver model in adversarial refinement.
  2. Can the authors provide detailed inference setting of the search agent, including context length, search environment and tool configurations? The reported performance of the base Qwen3.5-4B without context management in the paper is surprising (47.0 on BC in fig.2(a)). Other work [1] that trains Qwen3.5-4B via SFT and RL only obtains 37.3 on BC.
  3. The only comparison of training performance between open-source datasets that serve as baselines in the main content is reported in Table.11 (REDSearcher vs FORT-Searcher base on Qwen3-30B-A3B). Is the score of REDSearcher (57.4) obtained under the same inference setting of FORT-Searcher? If not, can the author evaluate them under the same setting (context management and search tool)?

[1]Lu Y, Ye R, Wang J, et al. ABSeeker: Training Long-Horizon Search Agents via Answer-Backtracked Credit Assignment[J]. arXiv preprint arXiv:2608.05102, 2026.

11 comments

We sincerely thank Reviewer aNgC for the time, the recognition of our work, and the constructive comments. We address each concern below and have revised the paper accordingly, with all changes marked in blue.

Response to Weakness 1

Weakness 1: The paper formulates 4 different short-cut types during data synthesis, but only one of them (prior-knowledge binding) is clearly discussed and ablated in the experiments. The other short-cuts are not clearly identified and distinguished in the synthesized datasets and the baseline datasets.

Response:

We thank the reviewer for asking how each shortcut type is identified and ablated.

1. Definitions, theory, and cases in the paper. The paper already defines, models, and illustrates each of the four shortcut types separately.

  • Definitions and controls: Table 1 maps each of the four risks to its mechanism and to the FORT control that targets it.
  • Theory: Section 2.3 and Appendix A link single-clue selectivity, evidence co-coverage, and exposed constants to the selectivity of identifying subsets, the evidence-acquisition cost MevM_{\mathrm{ev}}, and the dependency depth, respectively.
  • Cases: Appendix B.1 to B.4 give one diagnostic case for each risk, including the query and the round at which the shortcut is triggered.

This shows that the paper treats all four shortcut types, not only prior-knowledge binding.

2. Measurement on synthesized and baseline data. Table 15 already measures a separate proxy for each shortcut on 200 successful trajectories from FORT and 200 from open-source data:

ShortcutProxyOpen-sourceFORT
Single-clue selectivityRlowR_{\mathrm{low}} (%) ↓58.243.0
Evidence co-coverageRevR_{\mathrm{ev}} (%) ↑76.388.2
Exposed constantsC^dep\hat C_{\mathrm{dep}} ↑2.75.5
Prior-knowledge bindingp^prior\hat p_{\mathrm{prior}} (%) ↓17.013.0

FORT moves all four proxies in the intended direction relative to the baseline data. This shows that the four shortcuts are identified separately in both synthesized and baseline data, and that FORT reduces each of them.

3. Isolated ablation of each shortcut type. Table 19 removes the control for each shortcut type one at a time and keeps the rest of the pipeline fixed. For downstream results, we fine-tune Qwen3.5-4B on each configuration with the training budget fixed at 0.11B SFT tokens and evaluate on BrowseComp three times.

ConfigurationTargeted shortcutAcc. (%)Ω^\hat\OmegaThitT_{\mathrm{hit}}p^prior\hat p_{\mathrm{prior}} (%)BrowseComp
FullAll64.292.316.112.761.9 ± 0.5
− Long-tailPrior-knowledge binding72.076.413.015.360.0 ± 0.6
− DerivedEvidence co-coverage67.378.614.213.759.7 ± 0.5
− FuzzingExposed constants78.872.79.314.858.5 ± 0.7
− GenericSingle-clue selectivity70.080.214.813.259.6 ± 0.6

Every removal raises accuracy and lowers Ω^\hat\Omega, so each control on its own makes the search harder. The changes also match the targeted risk: removing long-tail selection gives the highest p^prior\hat p_{\mathrm{prior}}, and removing fuzzing gives the earliest ThitT_{\mathrm{hit}}. Every removal also lowers BrowseComp. This shows that each of the four controls makes the search harder and improves downstream training on its own.

In summary, all four shortcut types are defined, measured on both synthesized and baseline data, and individually ablated in both search diagnostics and downstream training, not only prior-knowledge binding.

Response to Weakness 2 (1)

Weakness 2: Most of the operations in the pipeline have been proposed in prior work (graph construction, long-tail entity selection, multi-source, entity fuzzing). Besides, the highlighted "adversarial refinement" relies on a strong model to refine the data and control the difficulty, hence the data quality is mainly determined by the intelligence of the used model, instead of the method itself. Prior work like WebShaper, InfoAgent and REDSearcher also use strong teacher model to refine and filter easy data. These baseline datasets can be easier mainly because the available model at that time is weaker than the current latest models.

Response:

We thank the reviewer for the detailed comparison with prior work. We agree that individual operations such as graph construction, long-tail entity selection, multi-source retrieval, and entity fuzzification have appeared in prior studies [1–4], and that stronger models can improve data quality. FORT's key contribution is a theory-driven framework that identifies the factors governing actual answer-identification difficulty, systematically addresses four complementary shortcut risks, and validates their effects through search trajectories and downstream training.

Importantly, with the teacher model held fixed and adversarial refinement disabled, FORT's construction controls increase solving cost by approximately 3× and improve downstream BrowseComp accuracy by 7.0 percentage points.

We first present this evidence, then clarify the methodological differences from prior work.

1. Construction Methodology Contributes Beyond Teacher Strength

(1) Construction-only ablation under a fixed teacher. Figure 3b already tests whether search difficulty is determined by model capability. Each configuration builds 3,000 questions with the same DeepSeek-V4-Pro-0813 model and no adversarial refinement, and FORT's construction controls are removed one after another.

ConfigurationΩ^\hat{\Omega}ThitT_{\mathrm{hit}}p^prior\hat p_{\mathrm{prior}}BrowseComp
Full FORT construction92.316.112.7%61.9
All six controls removed30.74.526.1%54.9

With the same teacher and no refinement, removing the construction controls cuts solving cost about threefold, and Table 18 already shows that downstream BrowseComp drops by 7.0 points. This shows that the same strong model produces much easier and less useful training data without FORT's construction controls, so search difficulty is not determined by model capability alone.

The new isolated ablations in Table 19 further show that removing any one of the four tested controls lowers BrowseComp from 61.9 to between 58.5 and 60.0, with the other controls retained and the training budget fixed at 0.11B SFT tokens. This shows that each tested control contributes on its own, and the benefit does not come from a single component.

(2) Construction is effective without adversarial refinement. FORT refines only shortcut-prone and initially-unsolved drafts, as described in Section 3.4 and Table 4. The composition of the final training pool is:

SourceShare of final successful poolΩ^\hat{\Omega}ThitT_{\mathrm{hit}}p^prior\hat p_{\mathrm{prior}}
Never-refined57.7%121.921.311.4%
Shortcut-prone23.5%55.612.112.4%
Initially-unsolved18.7%87.221.21.7%

57.7% of the final questions are never refined, and this group has the highest solving cost (121.9) and the latest answer exposure (21.3). The blind human audit in Table 17 already confirms their quality, with 96% answer correctness, 96% source support, 94% uniqueness, and 92% unambiguous formulation. This shows that most of the training data comes from construction alone, and this part is also the hardest of the three groups. The construction-only ablation above further disables refinement for every configuration and still shows large differences in search difficulty and downstream performance. This shows that the effects of FORT's construction do not depend on adversarial refinement, which serves as an additional mechanism for repairing problematic questions.

(3) Same-teacher comparisons with existing datasets. Table 3 already distills FORT and open-source questions with the same teacher. In the updated Table 3, with retrieval cost, topic distribution, and SFT tokens matched at 0.16B, FORT reaches 64.0 ± 0.5 versus 61.3 ± 0.6 on BrowseComp. In the new Table 12, we further re-distill REDSearcher, OpenSeeker, and DeepResearch-9K questions with the same teacher and train under the same recipe and evaluation setting, and FORT leads by 6.5 to 18.2 points.

Response to Weakness2 (2)

Rebuilding these baselines with a current model is not feasible, because their stage-specific agents and prompts are not fully released [1–4]. The construction-only ablation addresses the question within one pipeline: with the current DeepSeek-V4-Pro-0813, removing FORT's controls already lowers solving cost to 30.7. This shows that a current strong model alone does not produce hard questions, and that the construction controls contribute much of the difficulty.

2. Methodological Differences from Prior Work

FORT connects a theoretical characterization of answer-identification difficulty to targeted shortcut control, trajectory-guided refinement, and empirical validation.

(1) Identifying-route-based characterization of difficulty. Prior approaches model structural complexity, such as knowledge structures, graph topology, evidence dispersion, and hierarchical constraints [1–4]. Structural complexity does not guarantee a high identification cost: a question with many facts may still be solved through one highly selective clue or a cheaply retrievable combination. Section 2.2 and Appendix A already model difficulty through the evidence a solver needs to identify the answer. They define identifying subsets, minimum identifying-route cost, evidence-acquisition cost, and dependency depth, and establish lower bounds that characterize low-cost identifying routes. This explains why structurally complex questions can remain easy and identifies the shortcut risks that construction must address.

(2) Unified identification and targeted control of four shortcut risks. Prior methods address some of these issues individually, but the compared approaches [1–4] do not jointly characterize and control all four risks within one identifying-route framework.

Shortcut riskFORT controlPrior workFORT
Single-clue selectivityWeakly-identifying fact selection, Table 7Selects facts by knowledge structure, subtree, or constraint combination, and does not consider how strongly a single fact identifies the answer [1–4]. One highly distinctive clue can then make the other constraints unnecessary.Evaluates the identifying power of each fact and favors clues that identify the target only jointly.
Evidence co-coverageFour derived-fact constructors, Table 5Builds questions directly from extracted atomic facts and does not construct new facts [1–4], so a complete clue can often be retrieved verbatim.Constructs new facts across atomic facts and sources through coincidence bridging, count aggregation, numerical relations, and meta-fact extraction, so a complete clue rarely appears in one retrieved result.
Exposed constantsFive type-aware fuzzing strategies, Table 8Applies generic entity, numerical, or constraint fuzzification without considering the different types and roles of entities and values [2–4], which can leave clues directly searchable or recoverable through reasoning.Chooses the fuzzing operation by the type and role of each exposed value and validates answer uniqueness.
Prior-knowledge bindingLong-tail entity selectionDoes not consider or measure whether the solver answers from prior knowledge before retrieving evidence [1–4].Measures solver-side prior-knowledge shortcuts with the answer-before-evidence rate as an observable proxy.

This shows that the four controls target distinct ways of bypassing evidence acquisition, all derived from one analysis of identifying routes and solver behavior.

(3) Trajectory-guided diagnosis and targeted refinement. Prior methods use models for task expansion, difficulty assessment, and quality filtering [1–4]. FORT uses actual search trajectories to diagnose the specific shortcut or solvability problem and to guide the revision:

  • Shortcut-prone questions: FORT locates where the candidate space collapses, traces the keywords responsible for the low-cost identifying route, and revises those clues.
  • Initially-unsolved questions: FORT locates over-fuzzed or ambiguous clues and narrows them to restore solvability.

The diagnosis uses three observable trajectory diagnostics introduced in the paper: solving cost, answer hit time, and answer-before-evidence rate. Unlike the tool-call counts or structural complexity used in prior work [1–5], they jointly capture retrieval cost, answer exposure, and prior-based shortcuts. This shows that refinement is guided by diagnosed search behavior, beyond a generic difficulty score or a success label.

Reviewer_oLzVOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies why apparently complex search questions can admit inexpensive shortcuts. It formalizes an evidence-only reference search setting and identifies risks including co-covered evidence, selective clues, exposed constants, and prior-knowledge binding. FORT-Searcher combines long-tail entities, multi-source and derived evidence, clue transformations, and adversarial refinement to synthesize harder search tasks. Successful teacher trajectories provide 40,000 training examples. Supervised fine-tuning improves several search benchmarks across three student sizes, with additional matched-size, approximate search-cost, synthesis-component, cross-solver, and task-quality analyses.

Critical strengths
  1. The paper turns a useful analysis of search shortcuts into a practical data-generation pipeline. Cross-solver diagnostics and human checks strengthen the analysis.

  2. Gains extend across model sizes and benchmarks: the 35B model improves from 67.8 to 82.0 overall. Matched-data and component controls across multiple search settings make the evidence stronger than an uncontrolled dataset replacement.

Critical weaknesses
  1. Matching dataset size and mean search calls does not control training tokens, topics, or difficulty. The 3.2-point controlled BrowseComp gain supports practical usefulness, but its attribution to shortcut resistance needs token accounting and training-seed uncertainty, rather than just evaluation sampling.

  2. The audit does not establish the ambiguity rate of the full corpus, since refinement checks focus on successfully refined examples. Exact search and restart budgets also need clarification, as they affect measured difficulty and success.

Questions for authors
  1. In the matched-data experiment, are training-token counts comparable, and how much do results vary across seeds? These results would clarify whether the improvement is robust to training variability rather than partly explained by more training tokens.

  2. What are the exact search limits and context-restart rules? Please identify any differences across compared models, since budget and restart choices can affect both success rates and search-cost measurements.

  3. What is the ambiguity rate in a random sample of the final 40,000 examples, and how are failed refinements handled? A representative audit would help assess whether the reported quality checks extend to the dataset actually used for training.

6 comments

We sincerely thank Reviewer oLzV for the time, the positive assessment of our work, and the constructive comments. We address each concern below and have revised the paper accordingly, with all changes marked in blue.

Response to Weakness 1 & Question 1

Weakness 1: Matching dataset size and mean search calls does not control training tokens, topics, or difficulty. The 3.2-point controlled BrowseComp gain supports practical usefulness, but its attribution to shortcut resistance needs token accounting and training-seed uncertainty, rather than just evaluation sampling.

Question 1: In the matched-data experiment, are training-token counts comparable, and how much do results vary across seeds? These results would clarify whether the improvement is robust to training variability rather than partly explained by more training tokens.

Response:

We thank the reviewer for asking for token accounting and training-seed variance.

1. Matched training tokens and topic distribution. To rule out training tokens and topics as explanations, we re-run the matched-data experiment in Table 3 with both controlled. All four sets share the same distribution over the ten BrowseComp topic categories, assigned by DeepSeek-V4-Pro-0813. The open-source and FORT matched-cost sets contain the same 0.16B SFT tokens and the same average retrieval cost (Ω^=100\hat{\Omega}=100). Under these controls, FORT improves BrowseComp from 61.3 ± 0.6 to 64.0 ± 0.5 (Table 3). The gain therefore cannot be explained by more training tokens or a different topic mix.

2. Variance across training seeds. To measure training variability, we retrain both matched-cost sets in Table 3, open-source and FORT, with two additional random seeds under the same setting, and evaluate each model on BrowseComp three times.

Training dataSeed 42Seed 36Seed 23
Open-source (Ω^=100\hat{\Omega}=100)61.3 ± 0.661.1 ± 0.561.4 ± 0.6
FORT64.0 ± 0.563.7 ± 0.564.2 ± 0.5

Across training seeds, open-source data varies by 0.3 points and FORT by 0.5 points, while FORT exceeds open-source data by 2.6 to 2.8 points under every seed. The seed-level variation is much smaller than the gap. This shows that the gain is robust to training variability.

3. Attribution to shortcut resistance. At matched tokens and cost, the two sets differ mainly in shortcut-related properties: later answer exposure (ThitT_{\mathrm{hit}} 18.8 vs. 10.5) and fewer answer-before-evidence trajectories (p^prior\hat p_{\mathrm{prior}} 9.1% vs. 16.0%). The new downstream results in Table 19 link the gain to individual shortcut risks under a fixed 0.11B-token budget: removing any of the four individually ablated controls lowers BrowseComp from 61.9 to between 58.5 and 60.0. This supports attributing a large part of the gain to shortcut resistance.

In summary, the gain holds with training tokens and topics matched and is stable across training seeds, so it is not explained by more training tokens or training variability, but by the higher training value of FORT data.

Response to Weakness 2 & Question 3

Weakness 2: The audit does not establish the ambiguity rate of the full corpus, since refinement checks focus on successfully refined examples. Exact search and restart budgets also need clarification, as they affect measured difficulty and success.

Question 3: What is the ambiguity rate in a random sample of the final 40,000 examples, and how are failed refinements handled? A representative audit would help assess whether the reported quality checks extend to the dataset actually used for training.

Response:

We thank the reviewer for suggesting a representative audit of the final training set.

1. Existing quality audit in the paper. Our audit is not limited to refined examples. Table 17 (Appendix D.6) already reports a blind human audit of all three groups in the final data: questions accepted without refinement, and successfully refined shortcut-prone and initially-unsolved questions, with both pre- and post-refinement versions audited for the two refined groups. All final groups achieve at least 94% answer correctness and 88–92% unambiguous formulation. Every source of the final data is therefore covered by the audit.

2. Ambiguity rate on a random sample of the final 40K. We uniformly sampled 300 questions from the final 40K training set. The same three annotators as in Table 17 reviewed them under the same blind protocol, without knowing each question's source or refinement status. The unambiguous rate is 91.7% (275/300), so the ambiguity rate of the final training set is only 8.3%, in line with the 88–92% unambiguous rates of the subgroups in Table 17.

3. Handling of failed refinements. Drafts that remain unsolved after refinement are discarded; the final training set contains only successful trajectories.

4. Search and restart budgets.

  • Data synthesis (diagnostic search): up to 200 interaction turns in a single rollout, with no context restart.
  • Evaluation: a 256K-token context and a budget of 200 interaction turns. Each attempt is reset when it reaches the 80th-percentile turn of correct trajectories without context management, with at most five resets per question, and the final attempt runs to the full 200-turn budget.

Synthesis diagnostics use a single fixed rollout, so measured difficulty is not affected by restarts.

In summary, the ambiguity rate of the final training set is low at 8.3%, failed refinements are excluded from training, and the search and restart budgets are fixed and documented.

Response to Question 2

Question 2: What are the exact search limits and context-restart rules? Please identify any differences across compared models, since budget and restart choices can affect both success rates and search-cost measurements.

Response:

We thank the reviewer for asking for the exact search limits and restart rules and for how they differ across compared models.

1. Search limits and context-restart rules. All self-evaluated models use the same search limit and the same restart rule. Appendix D.1 specifies the search limit and the restart framework.

  • Search limit: the interaction budget is 200 turns and the maximum context length is 256K tokens.
  • Restart trigger: the restart point is the 80th percentile of interaction turns among all correct trajectories obtained without context management. In each attempt, if the model has not given a final answer by this turn, the context is reset and the model restarts from the original question. At most five resets are allowed per question, and the final attempt runs to the full 200-turn budget. Following MiroThinker, tool observations are retained within an attempt.
  • Where restarts apply: in Table 2, context management is enabled for BrowseComp and BrowseComp-ZH and disabled for xbench-DeepSearch and WideSearch, as stated in Section 4.2.
  • Search-cost measurement for data diagnostics: the diagnostic search that measures Ω^\hat{\Omega} runs a single rollout of up to 200 turns with no restart, and Figure 3a and Table 13 already measure all datasets with the same solver and budget.

The restart point is set by a fixed rule, so restarts follow the same criterion for every model and every benchmark where they are enabled.

2. Differences across compared models. Table 2 contains two kinds of scores, and only the officially reported scores differ in protocol.

  • Self-evaluated scores use exactly the same harness, search limit, and restart rule as FORT-Searcher. They cover all 9B- and 4B-scale models and Qwen3.6-35B-A3B on every benchmark, the large-scale agents on every benchmark except Kimi-K3 on BrowseComp, BigBang-v1 on BC-ZH, xbench, and WideSearch, and Apodex-1.0-mini on xbench and WideSearch.
  • Officially reported scores, marked with "-" in Table 2, follow each system's own protocol: REDSearcher on BrowseComp and BC-ZH, Apodex-1.0-mini on BrowseComp and BC-ZH, BigBang-v1 on BrowseComp, and Kimi-K3 on BrowseComp.

Budget and restart differences can therefore arise only for the officially reported scores.

3. Matched-budget comparison with the officially reported baselines. For a fair comparison with these baselines, the paper already re-evaluates the similar-size models under exactly the FORT-Searcher setting. Although Table 2 reports official BrowseComp scores for BigBang-v1 and Apodex-1.0-mini, Figure 2b already evaluates both models, together with Qwen3.6-35B-A3B and FORT-Searcher, on BrowseComp with the same harness, search limit, and restart rule. Figure 2b plots accuracy against mean cumulative Ω^\hat{\Omega} accumulated over all restarted attempts, so the searches spent after each restart are counted in every model's cost. At matched search cost, FORT-Searcher achieves the highest accuracy at medium-to-high budgets. This shows that FORT-Searcher's advantage over similar-size baselines holds when the evaluation settings are fully aligned and search cost is matched.

4. Gains without context restarts. To separate the effect of training from the effect of restarts, Figure 2a already evaluates both the original backbones and FORT-Searcher without context management. Without any restart, FORT training still improves BrowseComp by 5.3–18.9 points and BrowseComp-ZH by 16.4–25.8 points across the three model scales, and context management adds further gains at all three scales. This shows that the improvement from FORT training does not depend on the restart choice.

In summary, all self-evaluated models share the FORT-Searcher search limit and restart rule, the similar-size baselines with official scores are already re-evaluated under this setting in Figure 2b, where FORT-Searcher still achieves the highest accuracy at medium-to-high search budgets, and FORT training improves performance even without restarts, so the gains cannot be explained by budget or restart choices.


We hope these responses and the new experiments address the reviewer's concerns, and we would be grateful if the reviewer would consider further strengthening the support for our paper. We are happy to answer any further questions.

Reviewer_KVubOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FORT, a data synthesis method to generate websearch questions aiming to mitigate questions which can be answered using easy shortcuts. Starting from entities, it builds evidence graphs, selects indirect clues and revises questions using feedback from a search agent. Qwen3.6-35b-a3b trained on the successful trajectories from deepseek-v4-pro improved its overall score from 67.8 to 82.0.

Overall, the paper offers a useful data pipeline using existing components and results are interesting. However, the evidence for its claims remains incomplete with limited ablations and its novelty beyond an automated pipeline is unclear given prior synthesis methods like WebSailor also start from entities and build graphs.

Critical strengths
  1. The data helps improve across different model sizes ranging from 4B to 35B
  2. The paper includes training-data comparisons at matched average retrieval cost and evaluation at different inference budgets
  3. Studying when the answer first appears provides useful information beyond total trajectory length
Critical weaknesses
  1. The source for the gain is unclear whether is it because of deepseek-v4-pro or its data since the only same-teacher comparison (Table 3) is at 3K trajectories and shows a small gain 60.7 to 63.9 compared to the main 40K result
  2. Table 3 matches average retrieval count, but other differences like topic distribution and training-token counts remain uncontrolled. It is unclear which data properties explain the gains and what guidance researchers should take from them.
  3. The standard deviations are not reported and BrowseComp is evaluated only once (Appendix D.1). The improvement of the 60.7 to 63.9 improvement in Table 3 is therefore not significant
  4. Late answer discovery does not necessarily imply its a good trajectory to train on. It can reflect exploration, but could also make the model learn to perform ineffective early queries. No comparisons or analysis is made to show delayed discovery is a major factor in improving learning or showing that models preserve efficient search on easier questions
  5. Implementation details like prompts, criteria for removing redundant facts etc are missing
Questions for authors
  1. Can you report standard deviation for the Table 3 differences and main results table?
  2. Can you report the Table 3 same-teacher comparison at the full 40K scale on the 35B model to isolate the effect of data generation method over deepseek-v4-pro's ability?
  3. Can ablation on which individual shortcut controls improve downstream performance under matched training-token budgets be shown?
  4. Can you distinguish required evidence discovery with overall searches made before the answer appears, that would be a better factor than when the answer is discovered?
  5. Would combining efficient short trajectories with long multi-step trajectories improve better?
11 comments

We sincerely thank Reviewer KVub for the time, the recognition of our work, and the constructive comments. We address each concern below and have revised the paper accordingly, with all changes marked in blue.

Response to Weakness 1 & Question 2

Weakness 1: The source for the gain is unclear whether is it because of deepseek-v4-pro or its data since the only same-teacher comparison (Table 3) is at 3K trajectories and shows a small gain 60.7 to 63.9 compared to the main 40K result.

Question 2: Can you report the Table 3 same-teacher comparison at the full 40K scale on the 35B model to isolate the effect of data generation method over deepseek-v4-pro's ability?

Response:

We thank the reviewer for suggesting a full-scale same-teacher comparison to separate the contribution of the data from that of the teacher.

1. Existing same-teacher evidence in the paper. The gain does not come from the teacher alone. Table 3 already distills all four training sets with the same teacher under the same pipeline, backbone, and SFT recipe. At a matched retrieval cost of Ω^=100\hat{\Omega}=100, FORT improves BrowseComp from 61.3 to 64.0 in the updated Table 3. Since the teacher is shared, this gap reflects the data. Figure 3b and Table 18 already show that, with the same teacher and training recipe, progressively removing FORT's shortcut-resistant controls lowers BrowseComp from 61.9 to 54.9. This shows that, beyond what the teacher provides, FORT's data construction contributes a substantial part of the gain.

2. Same-teacher comparison at the full 40K scale on the 35B model. Reproducing the full Table 3 design at 40K scale is not feasible. Distilling all open-source datasets in the paper with DeepSeek-V4-Pro-0813 yields only about 50K correct trajectories, too few to draw three disjoint 40K sets with different solving costs. We therefore run the comparison that isolates the data generation method: we take 40K of these same-teacher open-source trajectories and fine-tune Qwen3.6-35B-A3B under the same recipe and inference setting as FORT-Searcher-35B-A3B.

Training data (40K, same teacher)BrowseCompBrowseComp-ZH
Open-source78.1 ± 0.476.5 ± 1.8
FORT83.4 ± 0.483.6 ± 1.0

With the teacher and data scale held fixed, FORT improves BrowseComp by 5.3 points and BrowseComp-ZH by 7.1 points. The contribution of FORT data persists, and grows, at the full 40K scale.

In summary, under the same teacher, FORT data improves performance both at 3K scale and at the full 40K scale on the 35B model, and removing FORT's construction components lowers performance from 61.9 to 54.9, so the gain cannot be attributed to DeepSeek-V4-Pro alone.

Response to Weakness 2

Weakness 2: Table 3 matches average retrieval count, but other differences like topic distribution and training-token counts remain uncontrolled. It is unclear which data properties explain the gains and what guidance researchers should take from them.

Response:

We thank the reviewer for pointing out the remaining confounders in Table 3.

1. Controlled comparison with matched topics and training tokens. To rule out topic distribution and training-token count as explanations, we re-run Table 3 with both controlled. We use DeepSeek-V4-Pro-0813 to assign every trajectory to one of the ten BrowseComp topic categories and sample all four sets with the same topic distribution. The two matched-cost sets also contain the same number of SFT tokens (0.16B). The teacher, distillation pipeline, backbone, SFT recipe, and evaluation setting are unchanged, and each model is evaluated on BrowseComp three times.

Ω^\hat{\Omega}ThitT_{\mathrm{hit}}p^prior\hat p_{\mathrm{prior}} (%)SFT tokensBrowseComp
30.04.68.50.06B56.1 ± 0.7
65.07.713.20.09B59.7 ± 0.5
100.010.516.00.16B61.3 ± 0.6
100.0 (FORT)18.89.10.16B64.0 ± 0.5

With topics, training tokens, and retrieval cost all matched, FORT still improves BrowseComp from 61.3 to 64.0. The updated results are reported in Table 3.

2. Which data properties explain the gain. At matched cost and tokens, the two sets differ mainly in answer exposure and prior binding: FORT exposes the answer later (ThitT_{\mathrm{hit}} 18.8 vs. 10.5) and has fewer answer-before-evidence trajectories (p^prior\hat p_{\mathrm{prior}} 9.1% vs. 16.0%). The paper already measures these properties, and the new isolated ablations link them to downstream performance:

  • Table 15 already shows that FORT trajectories have fewer highly selective clues (RlowR_{\mathrm{low}} 43.0% vs. 58.2%), more dispersed evidence (RevR_{\mathrm{ev}} 88.2% vs. 76.3%), and longer pre-answer dependency chains (C^dep\hat C_{\mathrm{dep}} 5.5 vs. 2.7).
  • The new downstream results in Table 19 individually ablate four controls, one targeting each shortcut risk, under a fixed 0.11B-token budget. Removing any of these four lowers BrowseComp, from 61.9 to between 58.5 and 60.0, and removing exact-value fuzzing, which most strongly advances answer exposure, causes the largest drop.

These results indicate that much of the gain is associated with resistance to cheap identifying routes, beyond search volume.

3. Guidance for researchers.

  • Retrieval cost alone is an insufficient target. Raising the average solving cost of open-source data from 30 to 100 improves BrowseComp by 5.2 points, but FORT adds a further 2.7 points at the same cost and token budget.
  • Data should be screened for cheap identifying routes, not only for length. Late answer exposure, low answer-before-evidence rate, and dispersed evidence are observable at synthesis time and are associated with higher training value.

In summary, with topic distribution and training tokens controlled, the gain persists and is associated with FORT's shortcut-resistant properties, which offers concrete guidance for building search training data.

Response to Weakness 3 & Question 1

Weakness 3: The standard deviations are not reported and BrowseComp is evaluated only once (Appendix D.1). The improvement of the 60.7 to 63.9 improvement in Table 3 is therefore not significant.

Question 1: Can you report standard deviation for the Table 3 differences and main results table?

Response:

We thank the reviewer for asking for run-level variance.

1. Repeated evaluation and standard deviations. The paper already averaged xbench-DeepSearch over 10 runs and BrowseComp-ZH and WideSearch over 5 runs (Appendix D.1). We now also evaluate BrowseComp three times and report standard deviations for all self-evaluated results in Table 2 and Table 3; officially reported scores are marked with "-". Appendix D.1 is updated accordingly. All reported gains are therefore accompanied by run-level variance.

2. Significance of the Table 3 difference. The FORT gain in Table 3 is significant. At matched retrieval cost, FORT reaches 64.0 ± 0.5 versus 61.3 ± 0.6 for open-source data. The 2.7-point gap is about five times the per-run standard deviation, and a two-sided Welch's t-test over the three runs gives p < 0.01.

3. Main results. In Table 2, the standard deviations are small relative to the gains: FORT-Searcher improves BrowseComp over its backbone by 20.7 points at 35B (83.4 ± 0.4 vs. 62.7 ± 0.6), 17.9 points at 9B, and 20.7 points at 4B, with standard deviations below 1 point. The main-result gains are therefore well beyond evaluation noise.

In summary, all self-evaluated results now include standard deviations, and the Table 3 gain is statistically significant.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.