- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a data synthesize pipeline for deep research agents. It formulates 4 possible short-cuts of the generated question, and designs specific solutions in the pipeline to prevent these short-cuts. Performance of different backbone models can be improved after training on the proposed dataset. Experiments also show that later answer exposure is important for training effect.
Critical strengths
- The paper demonstrates an important phenomenon that answer exposure time is a significant metric besides total tool call number: within the same tool call number, later answer exposure in the data yields better training performance.
- The performance of the trained model is promising: 4B model achieves 74.4 in browsecomp, comparable with large models. Scores of other benchmarks are also competitive among baselines.
Critical weaknesses
- The paper formulates 4 different short-cut types during data synthesis, but only one of them (prior-knowledge binding) is clearly discussed and ablated in the experiments. The other short-cuts are not clearly identified and distinguished in the synthesized datasets and the baseline datasets.
- Most of the operations in the pipeline have been proposed in prior work (graph construction, long-tail entity selection, multi-source, entity fuzzing). Besides, the highlighted "adversarial refinement" relies on a strong model to refine the data and control the difficulty, hence the data quality is mainly determined by the intelligence of the used model, instead of the method itself. Prior work like WebShaper, InfoAgent and REDSearcher also use strong teacher model to refine and filter easy data. These baseline datasets can be easier mainly because the available model at that time is weaker than the current latest models.
- The comparison between baseline datasets mainly focus on analysis of trajectories from a solver model, e.g. accuracy, tool call number. Comprehensive comparison of training performance between these datasets is absent.
Questions for authors
- Can the authors clarify the exact model and prompts used in the data synthesis pipeline, especially the solver model in adversarial refinement.
- Can the authors provide detailed inference setting of the search agent, including context length, search environment and tool configurations? The reported performance of the base Qwen3.5-4B without context management in the paper is surprising (47.0 on BC in fig.2(a)). Other work [1] that trains Qwen3.5-4B via SFT and RL only obtains 37.3 on BC.
- The only comparison of training performance between open-source datasets that serve as baselines in the main content is reported in Table.11 (REDSearcher vs FORT-Searcher base on Qwen3-30B-A3B). Is the score of REDSearcher (57.4) obtained under the same inference setting of FORT-Searcher? If not, can the author evaluate them under the same setting (context management and search tool)?
[1]Lu Y, Ye R, Wang J, et al. ABSeeker: Training Long-Horizon Search Agents via Answer-Backtracked Credit Assignment[J]. arXiv preprint arXiv:2608.05102, 2026.
11 comments
Response to Weakness 2 (1)
Weakness 2: Most of the operations in the pipeline have been proposed in prior work (graph construction, long-tail entity selection, multi-source, entity fuzzing). Besides, the highlighted "adversarial refinement" relies on a strong model to refine the data and control the difficulty, hence the data quality is mainly determined by the intelligence of the used model, instead of the method itself. Prior work like WebShaper, InfoAgent and REDSearcher also use strong teacher model to refine and filter easy data. These baseline datasets can be easier mainly because the available model at that time is weaker than the current latest models.
Response:
We thank the reviewer for the detailed comparison with prior work. We agree that individual operations such as graph construction, long-tail entity selection, multi-source retrieval, and entity fuzzification have appeared in prior studies [1–4], and that stronger models can improve data quality. FORT's key contribution is a theory-driven framework that identifies the factors governing actual answer-identification difficulty, systematically addresses four complementary shortcut risks, and validates their effects through search trajectories and downstream training.
Importantly, with the teacher model held fixed and adversarial refinement disabled, FORT's construction controls increase solving cost by approximately 3× and improve downstream BrowseComp accuracy by 7.0 percentage points.
We first present this evidence, then clarify the methodological differences from prior work.
1. Construction Methodology Contributes Beyond Teacher Strength
(1) Construction-only ablation under a fixed teacher. Figure 3b already tests whether search difficulty is determined by model capability. Each configuration builds 3,000 questions with the same DeepSeek-V4-Pro-0813 model and no adversarial refinement, and FORT's construction controls are removed one after another.
| Configuration | BrowseComp | |||
|---|---|---|---|---|
| Full FORT construction | 92.3 | 16.1 | 12.7% | 61.9 |
| All six controls removed | 30.7 | 4.5 | 26.1% | 54.9 |
With the same teacher and no refinement, removing the construction controls cuts solving cost about threefold, and Table 18 already shows that downstream BrowseComp drops by 7.0 points. This shows that the same strong model produces much easier and less useful training data without FORT's construction controls, so search difficulty is not determined by model capability alone.
The new isolated ablations in Table 19 further show that removing any one of the four tested controls lowers BrowseComp from 61.9 to between 58.5 and 60.0, with the other controls retained and the training budget fixed at 0.11B SFT tokens. This shows that each tested control contributes on its own, and the benefit does not come from a single component.
(2) Construction is effective without adversarial refinement. FORT refines only shortcut-prone and initially-unsolved drafts, as described in Section 3.4 and Table 4. The composition of the final training pool is:
| Source | Share of final successful pool | |||
|---|---|---|---|---|
| Never-refined | 57.7% | 121.9 | 21.3 | 11.4% |
| Shortcut-prone | 23.5% | 55.6 | 12.1 | 12.4% |
| Initially-unsolved | 18.7% | 87.2 | 21.2 | 1.7% |
57.7% of the final questions are never refined, and this group has the highest solving cost (121.9) and the latest answer exposure (21.3). The blind human audit in Table 17 already confirms their quality, with 96% answer correctness, 96% source support, 94% uniqueness, and 92% unambiguous formulation. This shows that most of the training data comes from construction alone, and this part is also the hardest of the three groups. The construction-only ablation above further disables refinement for every configuration and still shows large differences in search difficulty and downstream performance. This shows that the effects of FORT's construction do not depend on adversarial refinement, which serves as an additional mechanism for repairing problematic questions.
(3) Same-teacher comparisons with existing datasets. Table 3 already distills FORT and open-source questions with the same teacher. In the updated Table 3, with retrieval cost, topic distribution, and SFT tokens matched at 0.16B, FORT reaches 64.0 ± 0.5 versus 61.3 ± 0.6 on BrowseComp. In the new Table 12, we further re-distill REDSearcher, OpenSeeker, and DeepResearch-9K questions with the same teacher and train under the same recipe and evaluation setting, and FORT leads by 6.5 to 18.2 points.
Response to Weakness2 (2)
Rebuilding these baselines with a current model is not feasible, because their stage-specific agents and prompts are not fully released [1–4]. The construction-only ablation addresses the question within one pipeline: with the current DeepSeek-V4-Pro-0813, removing FORT's controls already lowers solving cost to 30.7. This shows that a current strong model alone does not produce hard questions, and that the construction controls contribute much of the difficulty.
2. Methodological Differences from Prior Work
FORT connects a theoretical characterization of answer-identification difficulty to targeted shortcut control, trajectory-guided refinement, and empirical validation.
(1) Identifying-route-based characterization of difficulty. Prior approaches model structural complexity, such as knowledge structures, graph topology, evidence dispersion, and hierarchical constraints [1–4]. Structural complexity does not guarantee a high identification cost: a question with many facts may still be solved through one highly selective clue or a cheaply retrievable combination. Section 2.2 and Appendix A already model difficulty through the evidence a solver needs to identify the answer. They define identifying subsets, minimum identifying-route cost, evidence-acquisition cost, and dependency depth, and establish lower bounds that characterize low-cost identifying routes. This explains why structurally complex questions can remain easy and identifies the shortcut risks that construction must address.
(2) Unified identification and targeted control of four shortcut risks. Prior methods address some of these issues individually, but the compared approaches [1–4] do not jointly characterize and control all four risks within one identifying-route framework.
| Shortcut risk | FORT control | Prior work | FORT |
|---|---|---|---|
| Single-clue selectivity | Weakly-identifying fact selection, Table 7 | Selects facts by knowledge structure, subtree, or constraint combination, and does not consider how strongly a single fact identifies the answer [1–4]. One highly distinctive clue can then make the other constraints unnecessary. | Evaluates the identifying power of each fact and favors clues that identify the target only jointly. |
| Evidence co-coverage | Four derived-fact constructors, Table 5 | Builds questions directly from extracted atomic facts and does not construct new facts [1–4], so a complete clue can often be retrieved verbatim. | Constructs new facts across atomic facts and sources through coincidence bridging, count aggregation, numerical relations, and meta-fact extraction, so a complete clue rarely appears in one retrieved result. |
| Exposed constants | Five type-aware fuzzing strategies, Table 8 | Applies generic entity, numerical, or constraint fuzzification without considering the different types and roles of entities and values [2–4], which can leave clues directly searchable or recoverable through reasoning. | Chooses the fuzzing operation by the type and role of each exposed value and validates answer uniqueness. |
| Prior-knowledge binding | Long-tail entity selection | Does not consider or measure whether the solver answers from prior knowledge before retrieving evidence [1–4]. | Measures solver-side prior-knowledge shortcuts with the answer-before-evidence rate as an observable proxy. |
This shows that the four controls target distinct ways of bypassing evidence acquisition, all derived from one analysis of identifying routes and solver behavior.
(3) Trajectory-guided diagnosis and targeted refinement. Prior methods use models for task expansion, difficulty assessment, and quality filtering [1–4]. FORT uses actual search trajectories to diagnose the specific shortcut or solvability problem and to guide the revision:
- Shortcut-prone questions: FORT locates where the candidate space collapses, traces the keywords responsible for the low-cost identifying route, and revises those clues.
- Initially-unsolved questions: FORT locates over-fuzzed or ambiguous clues and narrows them to restore solvability.
The diagnosis uses three observable trajectory diagnostics introduced in the paper: solving cost, answer hit time, and answer-before-evidence rate. Unlike the tool-call counts or structural complexity used in prior work [1–5], they jointly capture retrieval cost, answer exposure, and prior-based shortcuts. This shows that refinement is guided by diagnosed search behavior, beyond a generic difficulty score or a success label.
We sincerely thank Reviewer aNgC for the time, the recognition of our work, and the constructive comments. We address each concern below and have revised the paper accordingly, with all changes marked in blue.
Response to Weakness 1
Response:
We thank the reviewer for asking how each shortcut type is identified and ablated.
1. Definitions, theory, and cases in the paper. The paper already defines, models, and illustrates each of the four shortcut types separately.
This shows that the paper treats all four shortcut types, not only prior-knowledge binding.
2. Measurement on synthesized and baseline data. Table 15 already measures a separate proxy for each shortcut on 200 successful trajectories from FORT and 200 from open-source data:
FORT moves all four proxies in the intended direction relative to the baseline data. This shows that the four shortcuts are identified separately in both synthesized and baseline data, and that FORT reduces each of them.
3. Isolated ablation of each shortcut type. Table 19 removes the control for each shortcut type one at a time and keeps the rest of the pipeline fixed. For downstream results, we fine-tune Qwen3.5-4B on each configuration with the training budget fixed at 0.11B SFT tokens and evaluate on BrowseComp three times.
Every removal raises accuracy and lowers Ω^, so each control on its own makes the search harder. The changes also match the targeted risk: removing long-tail selection gives the highest p^prior, and removing fuzzing gives the earliest Thit. Every removal also lowers BrowseComp. This shows that each of the four controls makes the search harder and improves downstream training on its own.
In summary, all four shortcut types are defined, measured on both synthesized and baseline data, and individually ablated in both search diagnostics and downstream training, not only prior-knowledge binding.