- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies teacher-relative difficulty in deep-search distillation, arguing that structurally complex questions may require little search when the teacher already knows the underlying entities. The authors introduce Evidence-Verified Entity Familiarity (EVF) to investigate this relationship and propose Self-Reported Entity Familiarity (SEF) as a lightweight filtering mechanism during graph-based synthesis.
Critical strengths
S1. Meaningful problem formulation. The paper identifies a relevant limitation of existing deep-search data synthesis and introduces teacher-relative difficulty as a useful perspective.
S2. Simple and efficient approach. SEF provides a low-cost entity filtering mechanism that can be easily integrated into existing graph-based synthesis pipelines.
S3. Encouraging empirical results. The method consistently improves student performance across three teacher settings, and the cross-teacher reuse experiments demonstrate its practical potential.
Critical weaknesses
W1. The claimed improvement in search strategies is not sufficiently substantiated, and the diversity of the synthesized training data remains unclear.
The paper's central motivation is to elicit more informative search trajectories from teachers by selecting unfamiliar entities, thereby improving students' deep-search capabilities. However, the main downstream evidence is improved answer accuracy. While the authors report increased teacher-side search activity, they do not directly evaluate whether the student has learned more effective search strategies, such as better evidence acquisition, multi-hop reasoning, or search efficiency. It remains unclear whether the gains reflect improved search behavior or, at least partly, increased exposure to long-tail factual knowledge.
This concern is compounded by the limited characterization of the synthesized dataset. The paper does not provide sufficient analysis of question types, reasoning structures, or the diversity of evidence dependencies produced by the graph-based generator. Even with evaluation on external benchmarks, the absence of such analyses makes it difficult to determine whether the method teaches generalizable search strategies or primarily reinforces a narrow set of search patterns. This weakens the evidence supporting the paper's central mechanism.
W2. The predictive validity of SEF is not adequately established.
The paper motivates entity familiarity as a proxy for teacher-relative difficulty by showing that EVF predicts closed-book correctness, with AUC values of 0.60–0.706. However, the proposed synthesis method uses SEF rather than EVF. Although Table 2 reports differences in closed-book accuracy between SEF-YES and SEF-NO groups, the paper does not provide a direct, comparable evaluation of SEF's predictive performance against the same closed-book outcomes, such as SEF AUC with uncertainty estimates.
Instead, much of the justification for replacing EVF with SEF relies on their agreement and the retention of low-EVF entities. Agreement with another imperfect proxy does not establish equivalent predictive validity. Since SEF is the actual gating criterion and the central methodological contribution, this missing comparison leaves an important gap in the paper's justification of its proposed difficulty-control mechanism.
Questions for authors
Q1. Evidence of improved student search behavior.
Beyond improvements in final-answer accuracy, do the authors have evidence that SEF-gated training improves the student's search strategies? For example, can the authors compare student-side search trajectories before and after training in terms of effective evidence acquisition, search efficiency, or successful multi-hop reasoning? Such evidence would help distinguish improvements in search behavior from gains attributable primarily to exposure to additional long-tail factual knowledge.
Q2. Diversity and structural complexity of synthesized questions.
Could the authors provide a more detailed characterization of the synthesized training dataset, including the distribution of question types, graph depths, reasoning structures, and evidence dependencies? In particular, how often does a multi-hop exploration path translate into a question that genuinely requires multiple dependent retrieval steps, rather than one that can be resolved through a single direct search? This analysis would help assess whether the training data supports diverse and generalizable search strategies.
Q3. Direct predictive validity of SEF .
Table 1 reports the predictive performance of EVF against closed-book correctness, while Table 2 primarily evaluates SEF through its agreement with EVF and differences in group-level closed-book accuracy. Could the authors report SEF's predictive performance on the same closed-book outcomes used to evaluate EVF, including AUC and confidence intervals for each teacher? How much predictive performance is lost when replacing EVF with the cheaper binary SEF signal? This comparison is important for establishing whether SEF preserves the difficulty-prediction capability that motivates the proposed method.
2 comments
Response to Reviewer tpUU (2/2)
W1 (second part) / Q2. Characterization of the synthesized training data. As suggested, we characterized the three SEF-gated source pools from which the training trajectories were sampled (Table R11). Its rows cover the suggested aspects: question types, graph depths, reasoning structures and evidence dependencies.
| GLM-5.1 | DSV4-Flash | Kimi-K3 | |
|---|---|---|---|
| Question types | |||
| Answer type: person / org. / work / place / other | .27 / .27 / .14 / .11 / .21 | .27 / .24 / .17 / .08 / .24 | .36 / .22 / .16 / .08 / .18 |
| Largest of 15 domains | business .18 | science .19 | technology .17 |
| Graph depths | |||
| Exploration depth 1 / 2 / 3 / 4 | .24 / .40 / .30 / .06 | .29 / .38 / .28 / .05 | .21 / .39 / .33 / .07 |
| Reasoning structures | |||
| Constraints, median [IQR] | 7 [6, 9] | 6 [5, 8] | 11 [9, 14] |
| Evidence dependencies | |||
| Dependency depth 1 / 2 / 3 / ≥4 | .06 / .32 / .39 / .23 | .12 / .61 / .20 / .07 | .06 / .53 / .30 / .11 |
| ≥2 dependent sub-questions | 94.0% | 88.3% | 94.0% |
| Gold in top-10 | 1.3% | 1.7% | 1.0% |
| Teacher tool calls, median | 39 | 30 | 54 |
Table R11. Characterization of the SEF-gated source pools (fractions of questions unless marked %). Gold in top-10: the gold answer appears in the top-10 search results for the verbatim question.
Multiple dependent retrieval steps. An annotator model decomposed each sampled question into a minimal plan of sub-questions with dependencies. As shown in Table R11, 88–94% of questions involve at least two dependent sub-questions. Searching the question verbatim surfaces the gold answer in the top-10 results for only 1.0–1.7% of questions. Even questions with a single sub-question are not resolved by one search, taking the teachers a median of 17–39 tool calls.
Diversity. The training data vary along several dimensions: dependency depth spans one to four or more sub-questions, the number of constraints varies widely (median 6–11), and no answer type (at most 36%) or domain (at most 19%) dominates. Consistent with this, the gated student also outperforms the ungated one on WideSearch (Wong et al., 2025), whose multi-entity table-filling tasks differ in form from the training questions: item F1 rises from 61.50 to 65.94 and success rate from 8.50 to 12.00. This suggests that the learned search strategies are not tied to the generator's question form.
W2 / Q3. Direct predictive validity of SEF. As suggested, we evaluated SEF directly on the closed-book outcomes in Table 1, for each teacher. Table R12 reports both AUCs and their difference, with 95% confidence intervals from 2,000 paired bootstrap resamples over entities.
| Teacher | EVF AUC [95% CI] | SEF AUC [95% CI] | EVF − SEF [95% CI] |
|---|---|---|---|
| GLM-5.1 | .600 [.535, .661] | .541 [.484, .600] | .059 [.003, .117] |
| DSV4-Flash | .620 [.556, .686] | .594 [.536, .651] | .026 [−.022, .077] |
| Kimi-K3 | .706 [.641, .771] | .667 [.609, .726] | .039 [−.027, .105] |
Table R12. EVF and SEF AUC against the closed-book outcomes of Table 1.
As shown in Table R12, replacing EVF with SEF costs only 0.03–0.06 AUC, and the difference is not significant for DSV4-Flash and Kimi-K3. In return, SEF needs one tool-free call per entity and is 9–20× faster than EVF (Table 3).
For GLM-5.1, we further evaluated SEF on a larger set of 1,951 entities, where it reaches .594 [.572, .615], confirming that its signal exceeds chance. Overall, SEF retains most of EVF's predictive power, at a small fraction of EVF's cost.
Response to Reviewer tpUU (1/2)
We are grateful for the time and effort you devoted to our submission! We address each point below.
W1 (first part) / Q1. Evidence for improved student search behaviour. As suggested, we compared the student-side search trajectories of the Qwen3.8-27B base model and its ungated-trained and SEF-gated-trained students in the Kimi-K3 setting, along the three suggested aspects: evidence acquisition, multi-hop reasoning, and search efficiency. The analyses use the 130-question BrowseComp subset, with three evaluation runs per model.
Evidence acquisition. We measured whether retrieved content contains the gold answer and whether a retrieved page directly supports it, using an NLI-style check inspired by ALCE's entailment-based evaluation (Gao et al., 2023).
Table R9. Evidence acquisition of the Qwen3.8-27B students.
As shown in Table R9, the gated student acquires answer evidence most often on both measures: its retrieved results contain the gold answer in 69.0% of trajectories and its retrieved pages support the answer in 63.2%, 8.7 pp and 12.6 pp above the ungated student and 15.4 pp and 22.6 pp above the base model.
Multi-hop reasoning. Inspired by WebDetective’s evidence-chain evaluation (Song et al., 2026), we randomly sampled 20 questions with web-verified multi-hop solution routes and measured the fraction of intermediate entities on these routes that appear in retrieved content. This rate is 94.9% for the gated student, versus 79.6% before training and 82.9% for the ungated student.
Search efficiency. We report tool calls per correct answer, a tool-call analogue of cost-of-pass (Erol et al., 2026). The gated student's cost is 19% lower than the base model's. Table R10 further breaks down accuracy by observed tool-call count.
Table R10. Accuracy grouped by observed tool-call count.
As shown in Table R10, the gated student is more accurate than the base model in every tool-call bin, and is the most accurate of the three models in every bin below 200 calls.
Direct answer recall. With tools removed, the ungated and gated students answer only 1/390 and 5/390 attempts correctly, and training answers have no exact overlap with BrowseComp gold answers. The accuracy gain therefore mainly stems from improved search behaviour rather than from memorized long-tail knowledge.
References