acceptodds
Under review as a conference paper at ICLR 2027

Avoiding What Your Teacher Already Knows: Entity Familiarity for Deep-Search Distillation

Abstract

Deep-search agents are commonly trained by distilling tool-use trajectories from stronger teacher models. However, structurally complex questions may remain relatively easy for teachers familiar with key entities in the underlying entity graph, resulting in lower search demand and weaker supervision. We refer to this teacher-specific variation in question difficulty as teacher-relative difficulty and investigate entity familiarity as an early predictor. We first introduce Evidence-Verified Entity Familiarity (EVF), which estimates familiarity by verifying a teacher-generated factual brief against online evidence. Across multiple teachers, EVF is significantly correlated with teacher-relative difficulty with AUCs of 0.60–0.70. However, its high cost makes it impractical for constructing entity graphs at scale. To address this limitation, we propose Self-Reported Entity Familiarity (SEF), a lightweight gating signal that directly asks a teacher whether it is familiar with an entity before admitting that entity into the graph. Across multiple teachers, trajectories from SEF-gated synthesis exhibit greater observed search demand and yield higher downstream student accuracy than those from ungated synthesis. Because familiarity is teacher specific, we further study cross-teacher reuse. We screen graphs collected for other teachers using Kimi-K3's SEF judgments and use the retained graphs to produce 5,411 additional trajectories. Combined with 2,200 trajectories from newly constructed, SEF-gated graphs, these data improve Qwen3.8-27B's accuracy from 44.10% to 66.41% on a BrowseComp subset. Overall, SEF provides a low-cost way to synthesize more informative supervision for deep-search distillation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Deep-Search Agents, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_tpUUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies teacher-relative difficulty in deep-search distillation, arguing that structurally complex questions may require little search when the teacher already knows the underlying entities. The authors introduce Evidence-Verified Entity Familiarity (EVF) to investigate this relationship and propose Self-Reported Entity Familiarity (SEF) as a lightweight filtering mechanism during graph-based synthesis.

Critical strengths

S1. Meaningful problem formulation. The paper identifies a relevant limitation of existing deep-search data synthesis and introduces teacher-relative difficulty as a useful perspective.

S2. Simple and efficient approach. SEF provides a low-cost entity filtering mechanism that can be easily integrated into existing graph-based synthesis pipelines.

S3. Encouraging empirical results. The method consistently improves student performance across three teacher settings, and the cross-teacher reuse experiments demonstrate its practical potential.

Critical weaknesses

W1. The claimed improvement in search strategies is not sufficiently substantiated, and the diversity of the synthesized training data remains unclear.

The paper's central motivation is to elicit more informative search trajectories from teachers by selecting unfamiliar entities, thereby improving students' deep-search capabilities. However, the main downstream evidence is improved answer accuracy. While the authors report increased teacher-side search activity, they do not directly evaluate whether the student has learned more effective search strategies, such as better evidence acquisition, multi-hop reasoning, or search efficiency. It remains unclear whether the gains reflect improved search behavior or, at least partly, increased exposure to long-tail factual knowledge.

This concern is compounded by the limited characterization of the synthesized dataset. The paper does not provide sufficient analysis of question types, reasoning structures, or the diversity of evidence dependencies produced by the graph-based generator. Even with evaluation on external benchmarks, the absence of such analyses makes it difficult to determine whether the method teaches generalizable search strategies or primarily reinforces a narrow set of search patterns. This weakens the evidence supporting the paper's central mechanism.

W2. The predictive validity of SEF is not adequately established.

The paper motivates entity familiarity as a proxy for teacher-relative difficulty by showing that EVF predicts closed-book correctness, with AUC values of 0.60–0.706. However, the proposed synthesis method uses SEF rather than EVF. Although Table 2 reports differences in closed-book accuracy between SEF-YES and SEF-NO groups, the paper does not provide a direct, comparable evaluation of SEF's predictive performance against the same closed-book outcomes, such as SEF AUC with uncertainty estimates.

Instead, much of the justification for replacing EVF with SEF relies on their agreement and the retention of low-EVF entities. Agreement with another imperfect proxy does not establish equivalent predictive validity. Since SEF is the actual gating criterion and the central methodological contribution, this missing comparison leaves an important gap in the paper's justification of its proposed difficulty-control mechanism.

Questions for authors

Q1. Evidence of improved student search behavior.

Beyond improvements in final-answer accuracy, do the authors have evidence that SEF-gated training improves the student's search strategies? For example, can the authors compare student-side search trajectories before and after training in terms of effective evidence acquisition, search efficiency, or successful multi-hop reasoning? Such evidence would help distinguish improvements in search behavior from gains attributable primarily to exposure to additional long-tail factual knowledge.

Q2. Diversity and structural complexity of synthesized questions.

Could the authors provide a more detailed characterization of the synthesized training dataset, including the distribution of question types, graph depths, reasoning structures, and evidence dependencies? In particular, how often does a multi-hop exploration path translate into a question that genuinely requires multiple dependent retrieval steps, rather than one that can be resolved through a single direct search? This analysis would help assess whether the training data supports diverse and generalizable search strategies.

Q3. Direct predictive validity of SEF .

Table 1 reports the predictive performance of EVF against closed-book correctness, while Table 2 primarily evaluates SEF through its agreement with EVF and differences in group-level closed-book accuracy. Could the authors report SEF's predictive performance on the same closed-book outcomes used to evaluate EVF, including AUC and confidence intervals for each teacher? How much predictive performance is lost when replacing EVF with the cheaper binary SEF signal? This comparison is important for establishing whether SEF preserves the difficulty-prediction capability that motivates the proposed method.

2 comments
AuthorsOpenReview ↗

Response to Reviewer tpUU (1/2)

We are grateful for the time and effort you devoted to our submission! We address each point below.

 

W1 (first part) / Q1. Evidence for improved student search behaviour. As suggested, we compared the student-side search trajectories of the Qwen3.8-27B base model and its ungated-trained and SEF-gated-trained students in the Kimi-K3 setting, along the three suggested aspects: evidence acquisition, multi-hop reasoning, and search efficiency. The analyses use the 130-question BrowseComp subset, with three evaluation runs per model.

Evidence acquisition. We measured whether retrieved content contains the gold answer and whether a retrieved page directly supports it, using an NLI-style check inspired by ALCE's entailment-based evaluation (Gao et al., 2023).

Qwen3.8-27BRetrieved results contain the gold answerRetrieved page supports the answerAccuracy
Before training.536.406.441
Ungated-trained.603.506.551
SEF-gated-trained.690.632.636

Table R9. Evidence acquisition of the Qwen3.8-27B students.

As shown in Table R9, the gated student acquires answer evidence most often on both measures: its retrieved results contain the gold answer in 69.0% of trajectories and its retrieved pages support the answer in 63.2%, 8.7 pp and 12.6 pp above the ungated student and 15.4 pp and 22.6 pp above the base model.

Multi-hop reasoning. Inspired by WebDetective’s evidence-chain evaluation (Song et al., 2026), we randomly sampled 20 questions with web-verified multi-hop solution routes and measured the fraction of intermediate entities on these routes that appear in retrieved content. This rate is 94.9% for the gated student, versus 79.6% before training and 82.9% for the ungated student.

Search efficiency. We report tool calls per correct answer, a tool-call analogue of cost-of-pass (Erol et al., 2026). The gated student's cost is 19% lower than the base model's. Table R10 further breaks down accuracy by observed tool-call count.

Tool calls in a trajectory1–4950–99100–149150–199≥200
Before training.70.60.42.26.12
Ungated-trained.64.71.80.60.29
SEF-gated-trained.84.94.88.64.28

Table R10. Accuracy grouped by observed tool-call count.

As shown in Table R10, the gated student is more accurate than the base model in every tool-call bin, and is the most accurate of the three models in every bin below 200 calls.

Direct answer recall. With tools removed, the ungated and gated students answer only 1/390 and 5/390 attempts correctly, and training answers have no exact overlap with BrowseComp gold answers. The accuracy gain therefore mainly stems from improved search behaviour rather than from memorized long-tail knowledge.

References

  • Gao, T., Yen, H., Yu, J., and Chen, D. (2023). Enabling Large Language Models to Generate Text with Citations. EMNLP.
  • Song, M., et al. (2026). Demystifying Deep Search: A Holistic Evaluation with Hint-free Multi-hop Questions and Factorised Metrics. ICLR.
  • Erol, M. H., et al. (2026). Cost-of-Pass: An Economic Framework for Evaluating Language Models. ICLR.
AuthorsOpenReview ↗

Response to Reviewer tpUU (2/2)

W1 (second part) / Q2. Characterization of the synthesized training data. As suggested, we characterized the three SEF-gated source pools from which the training trajectories were sampled (Table R11). Its rows cover the suggested aspects: question types, graph depths, reasoning structures and evidence dependencies.

GLM-5.1DSV4-FlashKimi-K3
Question types
Answer type: person / org. / work / place / other.27 / .27 / .14 / .11 / .21.27 / .24 / .17 / .08 / .24.36 / .22 / .16 / .08 / .18
Largest of 15 domainsbusiness .18science .19technology .17
Graph depths
Exploration depth 1 / 2 / 3 / 4.24 / .40 / .30 / .06.29 / .38 / .28 / .05.21 / .39 / .33 / .07
Reasoning structures
Constraints, median [IQR]7 [6, 9]6 [5, 8]11 [9, 14]
Evidence dependencies
Dependency depth 1 / 2 / 3 / ≥4.06 / .32 / .39 / .23.12 / .61 / .20 / .07.06 / .53 / .30 / .11
≥2 dependent sub-questions94.0%88.3%94.0%
Gold in top-101.3%1.7%1.0%
Teacher tool calls, median393054

Table R11. Characterization of the SEF-gated source pools (fractions of questions unless marked %). Gold in top-10: the gold answer appears in the top-10 search results for the verbatim question.

Multiple dependent retrieval steps. An annotator model decomposed each sampled question into a minimal plan of sub-questions with dependencies. As shown in Table R11, 88–94% of questions involve at least two dependent sub-questions. Searching the question verbatim surfaces the gold answer in the top-10 results for only 1.0–1.7% of questions. Even questions with a single sub-question are not resolved by one search, taking the teachers a median of 17–39 tool calls.

Diversity. The training data vary along several dimensions: dependency depth spans one to four or more sub-questions, the number of constraints varies widely (median 6–11), and no answer type (at most 36%) or domain (at most 19%) dominates. Consistent with this, the gated student also outperforms the ungated one on WideSearch (Wong et al., 2025), whose multi-entity table-filling tasks differ in form from the training questions: item F1 rises from 61.50 to 65.94 and success rate from 8.50 to 12.00. This suggests that the learned search strategies are not tied to the generator's question form.

 

W2 / Q3. Direct predictive validity of SEF. As suggested, we evaluated SEF directly on the closed-book outcomes in Table 1, for each teacher. Table R12 reports both AUCs and their difference, with 95% confidence intervals from 2,000 paired bootstrap resamples over entities.

TeacherEVF AUC [95% CI]SEF AUC [95% CI]EVF − SEF [95% CI]
GLM-5.1.600 [.535, .661].541 [.484, .600].059 [.003, .117]
DSV4-Flash.620 [.556, .686].594 [.536, .651].026 [−.022, .077]
Kimi-K3.706 [.641, .771].667 [.609, .726].039 [−.027, .105]

Table R12. EVF and SEF AUC against the closed-book outcomes of Table 1.

As shown in Table R12, replacing EVF with SEF costs only 0.03–0.06 AUC, and the difference is not significant for DSV4-Flash and Kimi-K3. In return, SEF needs one tool-free call per entity and is 9–20× faster than EVF (Table 3).

For GLM-5.1, we further evaluated SEF on a larger set of 1,951 entities, where it reaches .594 [.572, .615], confirming that its signal exceeds chance. Overall, SEF retains most of EVF's predictive power, at a small fraction of EVF's cost.

Reviewer_LgVxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a critical limitation in deep search agent distillation: even structurally complex multi hop questions can be trivially solved by a strong teacher model using its internal parametric knowledge. In such cases, the teacher produces trajectories with little actual searching, yielding weak supervision for student agents. The authors formalize teacher relative difficulty, which describes how much external information a specific teacher must retrieve to solve a question, independent of structural graph complexity. First, they propose EVF (Evidence Verified Entity Familiarity). EVF estimates a teacher’s familiarity with an entity by verifying the model’s self generated factual summary against web evidence. Across three teachers, EVF correlates meaningfully with closed book question performance (AUC 0.60 0.70). However, EVF is computationally expensive for large scale graph construction. To mitigate cost, they introduce SEF (Self Reported Entity Familiarity), a lightweight binary gate: directly query the teacher whether it is familiar with a given entity. SEF filtering is applied at two points in graph based synthesis: seed entity admission and next hop expansion. Only entities marked unfamiliar by the teacher are kept for building evidence graphs. Experiments over GLM 5.1, DeepSeek V4 Flash and Kimi K3 demonstrate that SEF gated synthesis yields trajectories with higher search demand (more reasoning tokens, tool calls and turns). Students trained on SEF filtered trajectories obtain 6.15 8.46 percentage point accuracy gains compared to ungated baselines on the BrowseComp subset. Since entity familiarity is teacher specific, the paper further investigates cross teacher graph reuse. Though pairwise SEF judgments show 84.4 89.9% agreement between teachers, labels cannot be directly transferred. Instead, pre collected evidence graphs are reused after re screening entities with the target teacher’s SEF. Combining 2200 native Kimi K3 SEF gated trajectories plus 5411 trajectories generated from reused graphs improves Qwen3.8 27B accuracy from 44.10% to 66.41%. The three main contributions are: (1) formalizing teacher relative difficulty and validating entity familiarity as a predictive signal; (2) proposing cheap SEF gating for graph based synthesis and showing consistent student performance improvements; (3) presenting cross teacher evidence graph reuse to scale deep search training data. Presentation 3: Clear and easy to follow. The paper is well structured with clear research questions, diagrams, complete appendix containing prompts, hyper parameters and evaluation protocols. One minor caveat: some comparisons are not paired problem controlled; the authors note this limitation in text, yet readers need careful attention when interpreting experimental results. Informativeness 3: I gained a substantially deeper understanding of the problem or question. Prior work on synthetic deep search data mostly focuses on structural complexity (graph hops, constraints). This paper highlights an under appreciated dimension: a question can be structurally complex yet easy for a particular teacher due to its built in knowledge. It deepens our understanding of what constitutes high quality supervision for search agent distillation. Soundness 2: I have minor or fixable concerns about the paper’s soundness. Strengths: Evaluated on three teacher models and two student backbones; multiple runs with mean std reported; major benchmark BrowseComp adopted; table 8 empirically shows increasing graph depth does not reliably reduce closed book accuracy, supporting the core argument. Full reproducibility materials are provided in appendices. Concerns: 1.SEF is a model self reported binary signal, validated only via EVF (another model driven proxy), without human ground truth annotations for entity familiarity. Model hallucination or prompt bias may distort SEF outputs. 2.SEF gated and ungated synthesis generate disjoint sets of graphs and questions, rather than paired identical inputs. Therefore we observe correlation but cannot fully rule out confounding factors from entity distribution shift. 3.Cross teacher graph reuse mixes two variables: reuse of old graphs and increased total training sample size. Performance gain cannot be fully attributed to graph reuse alone. 4.Evaluations are limited to fact oriented graph based QA; generalization to open ended planning style search tasks remains unproven.

Critical strengths

1.Novel conceptual insight: The paper distinguishes structural complexity from teacher relative difficulty. It points out that multi hop structure alone does not guarantee useful supervision signal for distillation. This is an important conceptual contribution for synthetic search agent data research, with high community impact. 2.Practical, low cost intervention. SEF requires only one zero tool binary prompt call and can be plugged into existing graph synthesis pipelines with minimal modification. Cross teacher graph reuse further reduces the cost of building large scale deep search datasets, showing strong practical engineering value.

Critical weaknesses

1.Novel conceptual insight: The paper distinguishes structural complexity from teacher relative difficulty. It points out that multi hop structure alone does not guarantee useful supervision signal for distillation. This is an important conceptual contribution for synthetic search agent data research, with high community impact. 2.Practical, low cost intervention. SEF requires only one zero tool binary prompt call and can be plugged into existing graph synthesis pipelines with minimal modification. Cross teacher graph reuse further reduces the cost of building large scale deep search datasets, showing strong practical engineering value.

Questions for authors

1.SEF relies on model self report. Have you conducted small scale human validation to measure how often SEF YES/NO judgments deviate from actual model knowledge? 2.Could you design paired input experiments, fixing the same evidence graphs and toggling only the SEF gate, to mitigate entity distribution confounding and strengthen causal claims? 3.In the cross teacher reuse setting, could you add a control baseline with equal size newly sampled native SEF trajectories, to disentangle gains from graph reuse versus simply adding more training examples? 4.Would a continuous score variant of SEF bring further benefits compared to the current binary YES/NO formulation?

2 comments
AuthorsOpenReview ↗

Response to Reviewer LgVx (1/2)

Thank you very much for your thorough and insightful review! We address each point below.

 

C1 / Q1. External validation of SEF, and prompt sensitivity. As suggested, we validated SEF against the teacher’s actual knowledge using factual probes, independently of EVF. Because a model’s knowledge boundary is difficult to annotate directly, we measured it through the teacher’s accuracy on evidence-grounded questions. For each of the 294 seed entities, six annotators wrote probe questions grounded in retrieved web evidence and graded Kimi-K3’s closed-book answers. An entity’s probe accuracy, the fraction of its probes answered correctly, serves as a measure of how well the teacher knows that entity.

We manually audited this pipeline: all 60 randomly sampled probes were factually sound, and 96% of 160 gradings, sampled to cover both unanimous and split panel decisions, were correct.

As a result, SEF achieves an AUC of 0.794 against probe accuracy. To examine entity-level judgments, we label an entity as known when its probe accuracy exceeds a specified threshold; Table R3 compares SEF with these labels at three thresholds.

Self-report“Known” thresholdActually knownActually not known
YES (excluded by the gate), n=164≥ 0.685.4%14.6%
≥ 0.774.4%25.6%
≥ 0.866.5%33.5%
NO (retained), n=130≥ 0.638.5%61.5%
≥ 0.729.2%70.8%
≥ 0.822.3%77.7%

Table R3. SEF self-report versus the external knowledge label (Kimi-K3, 294 entities; rows sum to 100% within each threshold).

As shown in Table R3, SEF is largely consistent with the knowledge label. This holds in particular for the retained NO pool, which determines the entities admitted into the graph: entities the teacher does not know make up 61.5% of this pool at the 0.6 threshold, rising to 77.7% as the standard for knowing an entity becomes stricter.

The second part of your concern, prompt bias, we examined by re-running SEF with four prompts: the production prompt, a paraphrase, a paraphrase with the order of the YES/NO definitions swapped, and the production prompt without the sentence “If in doubt, answer NO.”

TeacherAUC vs. closed-book correctnessPairwise YES/NO agreement
Kimi-K30.663 ± 0.0060.893 ± 0.016
DSV4-Flash0.580 ± 0.0090.857 ± 0.019
GLM-5.10.562 ± 0.0210.892 ± 0.010

Table R4. SEF under four prompt variants (AUC ± s.d. across prompts; agreement over the six prompt pairs).

As shown in Table R4, judgments agree on 86–89% of entities across prompts, and predictive power varies little, so SEF is robust to these prompt variations. Together, these analyses show that SEF tracks the teacher's knowledge as measured independently of EVF, and that SEF is relatively robust to prompt variations.

 

C2 / Q2. Paired comparison on fixed evidence graphs. Following your suggestion, we fixed a common set of evidence graphs and toggled only the SEF gate. Because the gate acts while graphs are built, we implemented this comparison by applying SEF as a filter on a fixed pool: of the 10,495 graphs in the Kimi-K3 ungated pool, 2,414 contain only entities self-reported as unfamiliar and thus pass the gate, while the remaining 8,081 contain at least one familiar entity. We sampled 2,200 trajectories from each group and trained one student on each, using the recipe of Table 5. Since both training sets come from the same synthesis run, they differ only in whether their paths pass the gate.

Training data (Kimi-K3, same evidence graphs)# trajectoriesAccuracy
Contains YES (fails the gate)2,20056.92 ± 2.31
All NO (passes the gate)2,20061.03 ± 1.18

Table R5. Qwen3.8-27B accuracy on the 130-question BrowseComp subset with the evidence graphs fixed.

As shown in Table R5, with the evidence graphs fixed, the gated student is 4.11 pp more accurate. This confirms that the gain stems from entity screening itself rather than from differences between independently synthesized pools.

AuthorsOpenReview ↗

Response to Reviewer LgVx (2/2)

C3 / Q3. Size-matched control for graph reuse. Following your suggestion, we added a size-matched control that fixes the training set at 2,200 trajectories and varies only the data source. We sampled 2,200 Kimi-K3 SEF-gated trajectories from the reused-graph pool, matching the correct/incorrect ratio of the native gated set so that the two sets differ only in source, and trained a student with the same recipe as the native SEF-gated student.

Training data# trajectoriesAccuracy
Native, ungated2,20055.13 ± 4.37
Reused graphs, Kimi-K3 SEF-gated (new)2,20061.79 ± 1.94
Native, SEF-gated2,20063.59 ± 1.18

Table R6. Qwen3.8-27B accuracy on the 130-question BrowseComp subset (mean ± s.d. over three evaluation runs). The native rows are those of Table 5.

As shown in Table R6, at equal size the reused-graph student is close to the native gated one and above the native ungated one. Thus, with the number of training examples held fixed, supervision generated from reused graphs is comparable to native gated supervision.

The value of graph reuse lies in scaling this supervision at lower cost: whereas more native SEF-gated data requires constructing and exploring new graphs, reuse obtains supervision of comparable quality from graphs that already exist.

 

Q4. A continuous variant of SEF. Thank you for this constructive suggestion! We tested a continuous variant on Kimi-K3, in which the teacher returns a familiarity score in [0, 1] in a single call. To separate the effect of the format from that of prompt wording, we wrote four continuous prompts paralleling the four binary prompts of Table R4, and scored the same entities with the same endpoint.

Signal (Kimi-K3)AUC vs. closed-book correctnessRetained fraction
Binary SEF0.663 ± 0.00641–45%
Continuous score0.741 ± 0.007—
Continuous score, thresholded at 0.60.678 ± 0.01138–45%

Table R7. Binary versus continuous SEF (AUC ± s.d. over four prompts each; retained fraction gives the range over prompts).

As shown in Table R7, the continuous score predicts closed-book correctness better than the binary signal (0.741 vs. 0.663 AUC), so the teacher's self-report carries graded information beyond YES/NO. The suggested extension thus also supports the premise behind SEF: self-reported familiarity is informative about what the teacher can answer without tools.

To compare the continuous score with the current YES/NO gate, we also converted it into a binary signal at an illustrative threshold of 0.6: scores at or above 0.6 are treated as familiar and excluded, while scores below 0.6 are treated as unfamiliar and retained. This retains 38–45% of entities, close to the binary gate's 41–45%. The resulting binary signal has an AUC of 0.678, compared with 0.663 for direct YES/NO self-reports, indicating similar predictive discrimination at similar retention rates. At this threshold, the continuous format therefore gives a gate of similar quality to YES/NO; its additional information would matter only if used beyond a single cut.

Such uses include adjusting retention rates or using familiarity scores as weights rather than hard filters; testing whether they improve downstream student performance is future work. We will include this analysis in the appendix and discuss these extensions.

 

C4. Beyond single-answer QA. As suggested, we evaluated the two Kimi-K3 students, each trained on 2,200 trajectories, on two benchmarks whose output is a table rather than a single answer, to test generalization beyond single-answer QA. WideSearch (Wong et al., 2025) is a challenging benchmark for information gathering, synthesis, and reporting. DeepWideSearch (Lan et al., 2025) adds depth to this setting: the agent must first identify hidden core entities through multi-hop search and then collect their attributes into a table; core entity accuracy and column F1 measure depth, and row F1 and item F1 measure width.

WideSearch
Item F1Item recallItem precisionRow F1Success rate
Ungated61.5060.5267.8742.158.50
SEF-gated65.9465.4570.8647.8012.00
Δ+4.44+4.93+2.99+5.65+3.50
DeepWideSearch
Core entity accuracyColumn F1Row F1Item F1
Ungated83.6450.2728.0043.35
SEF-gated84.0954.6831.5348.02
Δ+0.45+4.41+3.53+4.67

Table R8. WideSearch and DeepWideSearch results of the Kimi-K3 2,200-trajectory students.

As shown in Table R8, the gated student is ahead on all nine metrics. These results indicate that the gains extend beyond single-answer QA to multi-entity information gathering that requires planning broad search coverage.

Reviewer_tMZfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how the target teacher’s entity familiarity can improve training data for deep-search distillation. The authors use Evidence-Verified Entity Familiarity (EVF) to examine the association between familiarity and closed-book correctness, then introduce the low-cost Self-Reported Entity Familiarity (SEF) signal to screen entities during seed selection and next-hop expansion. Experiments with three teachers and two student backbones show consistent gains under equal-sized trajectory training sets. The paper also demonstrates the feasibility of cross-teacher evidence graph reuse and reports competitive final-model performance across multiple benchmarks.

Critical strengths
  1. The problem is clearly defined and well motivated. The paper distinguishes structural complexity from teacher-relative difficulty. Quantitative analyses of entity familiarity, closed-book correctness, and search behavior provide a clear basis for the method.

  2. The method is simple and practically useful. SEF requires only one tool-free model call and can intervene in seed selection and graph expansion before question generation. Its substantially lower assessment cost than EVF makes it easy to incorporate into existing data synthesis pipelines.

  3. The downstream gains are consistent. SEF-gated data improve student accuracy over ungated data by 8.20, 6.15, and 8.46 percentage points across the three teacher settings. Consistency across teachers and student backbones supports the strategy’s effectiveness, while the multi-benchmark results further demonstrate the final system’s competitiveness.

Critical weaknesses

Controlled validation of the downstream gains remains limited in scope. Table 5 shows improvements across three teacher settings, but all comparisons use the same 130-question BrowseComp subset. Table 7 reports final-model results on the full benchmarks without corresponding ungated models, leaving unclear whether the SEF gains persist on a larger evaluation set or other benchmarks. Extending one representative gated/ungated comparison to the full BrowseComp benchmark or another reported benchmark would further strengthen the evidence.

Questions for authors
  1. Could the authors provide an additional gated/ungated comparison in a representative setting—for example, evaluating the two models trained on 2,200 trajectories each in the Kimi-K3 setting on full BrowseComp or another reported benchmark?
  2. The reasoning-token statistics in Table 4 are measured on the raw trajectory pools and exclude tool observations and final answers. Could the authors report sequence lengths, total supervised tokens, and training update counts for the actual training sets in Table 5 to clarify differences in supervision volume under the equal-trajectory-count comparison?
3 comments
AuthorsOpenReview ↗

Response to Reviewer tMZf

We sincerely thank you for your time and careful review! We address each point below.

 

W / Q1. Gated vs. ungated beyond the 130-question subset. As suggested, we compare the two Kimi-K3 students, each trained on 2,200 trajectories, on the full BrowseComp benchmark (1,266 questions) and on three additional benchmarks: BrowseComp-ZH, WideSearch and DeepWideSearch. As shown in Table R1, the gated student is ahead on all eleven metrics. These results further support the downstream gains of SEF gating beyond the 130-question subset.

BrowseComp (full)BrowseComp-ZHWideSearch
AccuracyAccuracyItem F1Item recallItem precisionRow F1Success rate
Ungated58.9860.78 ± 2.2261.5060.5267.8742.158.50
SEF-gated62.0164.01 ± 0.9265.9465.4570.8647.8012.00
DeepWideSearch
Core entity accuracyColumn F1Row F1Item F1
Ungated83.6450.2728.0043.35
SEF-gated84.0954.6831.5348.02

Table R1. Additional gated/ungated comparison in the Kimi-K3 setting.

 

Q2. Supervision volume in the Table 5 training sets. As suggested, we report the supervision volume of each Table 5 training set in Table R2.

TeacherArmMean turns / traj.Mean sequence length (tokens)Mean supervised tokens / traj.Optimizer steps
GLM-5.1ungated20.742.7k17.32k42
gated29.559.2k23.16k (+33.7%)60
DSV4-Flashungated34.044.0k17.05k42
gated36.653.3k19.75k (+15.8%)54
Kimi-K3ungated24.063.7k6.97k102
gated27.565.9k7.31k (+4.9%)102

Table R2. Supervision volume of the Table 5 training sets.

As shown in Table R2, for GLM-5.1 and DSV4-Flash the gated sets carry more supervision. The additional supervision reflects the additional search and reasoning effort that gated questions demand (Table 4), which is part of what gating produces.

The Kimi-K3 pair controls for this difference. Although its gated trajectories take more turns, the gated set has only 4.9% more supervised tokens per trajectory and the same 102 optimizer steps, yet its student is 8.46 pp more accurate on the browsecomp subset. The gain therefore holds when the supervision volume is nearly matched.

Thank the authors for the additional results, which substantially address my concerns.

And I also note that the gains on the full brosecomp (3.03) is smaller than on the subset. It is also true that at equal trajectory count the training volume is larger. Overall, the strengths outweigh the weaknesses, and I will maintain my original rating.

Thank you for the follow-up and for acknowledging the additional results. We briefly respond to the two remaining points.

Gain on the full BrowseComp. While the full-benchmark gain is smaller than on the subset, it holds consistently across all four benchmarks in Table R1, suggesting that SEF gating generalizes beyond BrowseComp.

Training volume. The Kimi-K3 pair is nearly volume-matched (+4.9% supervised tokens per trajectory, same 102 optimizer steps), and these two students are the ones evaluated in Table R1, where the gated student leads on all eleven metrics.

We sincerely appreciate your positive assessment and thoughtful consideration. In particular, we thank you for recognizing the teacher-relative view of difficulty and the practicality of SEF, and for your efforts in helping us improve the paper. We will incorporate these additional experiments and results into the revised version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.