acceptodds
Desk rejected at ICLR 2027

DR.FREE: YOU DON’T NEED DIFFICULTY REWARDS FOR SELF-EVOLVING SEARCH AGENTS

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

A central limitation of current data-free self-evolution methods for training search agents is their reliance on difficulty-based proposer rewards. These methods reward a proposer for generating questions that challenge a co-evolving solver, using solver difficulty as a proxy for question quality. Yet difficulty alone is insufficient to distinguish questions that require cross-passage evidence from those that are answerable via simpler shortcuts. In addition, measuring difficulty demands repeated solver rollouts for every candidate question, leading to substantial computational costs. In this paper, we introduce \methodname, the first self-evolving search framework that eliminates difficulty-based proposer rewards and directly optimizes for evidence necessity relative to shortcut contexts. DR.FREE samples relational chains from a knowledge graph and pairs them with aligned passages, giving question generation an explicit multi-hop structure. A generated question receives a positive Information Gain reward only when the likelihood of the target answer under the complete evidence passages exceeds the maximum likelihood under all evaluated shortcut contexts. Because this signal is computed from teacher-forced likelihoods, it removes the need for pass-rate estimation and reduces proposer training time by over 7x. Experiments on seven open-domain QA benchmarks show that DR.FREE outperforms prior data-free search agents and a supervised baseline, with large improvements on multi-hop QA benchmarks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
search agent, self evolving

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_CNtXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Dr. Free challenges the existing difficulty based approaches in self-evolving search agents. The main argument is that information gain based rewards better measure the dependence of generated questions compared to difficulty. Dr. Free also demonstrates impressive proposer training efficiency compared to prior art. The approach involves using verified knowledge graphs coupled with information gain rewards over end-to-end difficulty rewards. This prevents reward hacking and passage leakage in the proposed questions.

Critical strengths
  1. At a high level, understanding reward modeling for self-evolving search agents is an important problem and I'm glad that this paper explores this area.

  2. The paper demonstrates strong efficiency gains in proposer updates and improvement over several strong baselines on a large number of benchmarks.

  3. I found that the paper was easy to understand and read.

Critical weaknesses
  1. Table 2 provides evidence that IG reward does better than difficultly rewards. I have two core concerns:

    1. It is well established that using simple rewards and letting the model discover on its own often scales better. Does Dr. Free scale as well as difficulty rewards?
    2. A difficulty reward plus an equally strong evidence-dependence filter could achieve comparable quality under matched compute. An additional ablation would help establish the approach better,
  2. The reward measures improvement in answer likelihood but not whether the intended passage is necessary. A positive margin against the tested shortcut doesn't establish that every passage is necessary because the reward does not test all shortcuts. Specifically, the max\text{max} operator in Eq. 6 only tests one shortcut, which does not guarantee that no shortcut would be used. I would recommend the authors to do an empirical analysis validating that the rewarded questions actually reuire the evidence.

Questions for authors
  1. The paper missed some related works that could be used to make the evaluations and discussions about novelty more convincing. How is Dr. Free different from these approaches? How well does it do empirically?

    1. KELP (Liu et al ACL 2022) establishes the importance of KG-based task construction
    2. Some other works like IG-Search (Liang et al., 2026) also establish that information gain rewards help train stronger search agents.
  2. The paper uses standard benchmarks in multi-hop QA like HotpotQA, NQ, TriviaQA, MuSiQue, etc. Although widely used, these benchmarks are very old, and modern LLMs might be able to answer from their parametric memory.

    1. In addition to EM, a better measure for their capabilities would be Recall and Precision of retrieved documents.
    2. To isolate the effectiveness of Dr. Free, I would recommend testing the base models on all the benchmarks without any search. That would provide a better baseline for their parametric knowledge.
    3. Newer benchmarks like BrowseComp-Plus should be used for evaluation because they specifically provide a local index for search. Additionally they present a much harder challenge for language models because of their better multi-hop nature as opposed to hotpotqa and musique which can be solved in at most 2-4 hops.
    4. I would request the authors to conduct an analysis of the number of hops their approach actually does compared to a base model.
  3. Constructing a knowledge graph is non-trivial in practical settings. Although it provides verifiability, it is not scalable beyond Wikipedia. Can the authors show that it works in practical settings beyond wikipedia?

  4. One of the strengths of this paper is approx 7 times lower proposer update time. However, it removes solver rollouts from reward estimation, it requires likelihood evaluation across multiple contexts, KG construction and preprocessing, and alternating solver-proposer training. Kindly comment on the fairness of efficiency evaluation.

  5. In Appendix A.4, the authors share an interesting analysis on the performance of their approach across self evolution iterations. Although Qwen2.5-3B improves monotonically, the same is not true for the 7B variant. This raises questions about the efficacy of the proposed method. Is self evolution really helping? Would it help at scale in the newer generation models (say Qwen3.8)?

8 comments
AuthorsOpenReview ↗

Response to Reviewer CNtX(Part1)

We greatly appreciate Reviewer CNtX's detailed and thoughtful feedback, and we are glad that you find reward modeling for self-evolving search agents an important problem and the paper easy to follow.

Following your suggestions, we have added several new experiments, including the difficulty + filter ablation, BrowseComp-Plus, retrieval recall and precision, closed-book baselines, and newer Qwen3.5 backbones. We hope these address your concerns, and we would be grateful if you would consider raising your score in light of the new results. Our detailed responses are below.

W1.1: Does Dr. Free scale as well as difficulty rewards?

We agree that simple rewards often scale well, and we note that IG is itself a simple reward: a teacher-forced likelihood comparison that requires no solver rollouts, whereas the difficulty reward requires multi-turn rollouts for every question. The key difference is what the reward lets the model discover. Under the difficulty reward, the proposer discovers shortcuts: in our Qwen2.5-3B reproduction, 99.995% of Dr. Zero's nominally multi-hop questions are generated without any search (Table 4 in the original submission, now Table 5). IG closes this shortcut while keeping the reward simple and cheap.

Scaling across iterations. Average EM over seven benchmarks (Dr. Free from Table 9 in the original submission, now Table 10; Dr. Zero from Table 3 of its paper):

ModelMethodIter 1Iter 2Iter 3
Qwen2.5-3BDr. Zero0.3040.3190.326
Qwen2.5-3BDr. Free0.3540.3620.375
Qwen2.5-7BDr. Zero0.3660.3720.360
Qwen2.5-7BDr. Free0.3890.3970.401

Dr. Free improves monotonically on both backbones and leads at every iteration, while Dr. Zero's 7B model declines in the third iteration. These gains also come at lower cost: on the same 8×H800 setup, one 3B iteration takes 8.6 hours end-to-end for Dr. Free versus 22.0 hours for Dr. Zero (Appendix A.2), so Dr. Free can afford substantially more self-evolution under a fixed compute budget.

Scaling across model sizes and generations. Dr. Free outperforms Dr. Zero on all five tested backbones from 3B to 8B, including the Qwen3 family (Figure 1; Table 8 in the original submission, now Table 9). Our new results on the newer Qwen3.5-4B and 9B also show consistent gains over the base models (see our response to Question 5). Within the tested range, Dr. Free therefore scales at least as well as difficulty rewards, at a much lower cost.

W1.2: Difficulty reward plus an evidence-dependence filter

We have implemented this ablation. To make the filter as strong as possible, we use Dr. Free's own IG criterion: we train the proposer with the KG + difficulty reward, retain only generated questions with a strictly positive IG margin, and keep 12,800 of them with a balanced 1-/2-/3-hop distribution. The Qwen2.5-3B solver is then trained for 50 updates (batch size 256), matching the solver budget of one Dr. Free iteration (EM):

MethodNQTriviaQAPopQAHotpotQA2WikiMuSiQueBamboogleAvg.
Difficulty + IG filter0.3650.5550.3830.3210.3460.1270.2640.337
Dr. Free (Iter 1)0.3580.5630.3990.3360.3640.1380.3200.354

Even with this filter, the difficulty-trained proposer falls short of Dr. Free on six of seven benchmarks, with the largest gaps on Bamboogle and 2Wiki. The reason is that filtering can only remove questions after they are generated; it cannot change what the proposer learns to generate. A proposer trained with the difficulty reward rarely produces evidence-dependent questions: of 137,984 generated candidates, only 9.47% survive validity checks, deduplication, and the positive-IG filter. Training the proposer with IG instead steers its whole output distribution toward such questions.

This alternative is also more expensive than Dr. Free: it requires solver rollouts to estimate difficulty during proposer training, plus 10.78× over-generation and IG scoring of candidates for filtering, whereas Dr. Free only computes IG during proposer training.

AuthorsOpenReview ↗

Response to Reviewer CNtX(Part2)

W2: Does a positive IG reward establish that the evidence is necessary?

We first clarify that the max in Eq. 6 ranges over all shortcut contexts in B (closed-book, source-only, one-shot retrieval, and each individual passage), so a positive margin means that the full evidence outperforms every tested shortcut, not just one. We agree, however, that this establishes robustness only to the shortcuts in B, not a formal guarantee that every passage is necessary.

Following the recommendation, we provide three empirical analyses of whether rewarded questions depend on the full evidence:

  • Gold-blind audit (Table 5 in the original submission, now Table 6). With GPT-4o as the judge, Dr. Free's questions are answerable under any individual shortcut in only 2–5% of cases, compared with 17–88% for Dr. Zero and 20–51% for the difficulty-reward variant.
  • Per-question check. None of the 9 audited Dr. Free questions that trigger a shortcut receives a positive IG reward, so the rewarded questions do not admit any audited shortcut.
  • Beyond B. We evaluate all two-passage subsets for three-hop questions: for 41 of 49 (84%), the full evidence yields a higher answer likelihood than every two-passage subset.

In addition, removing any shortcut type from B lowers downstream accuracy (Table 3 in the original submission, now Table 4), indicating that each tested shortcut matters. Further details, including control conditions, are in our response to Reviewer eVk6 (W1, Q1, and W2).

Q1: Relation to KELP and IG-Search

We have added both works to Related Work (Section 2.2). The differences are as follows:

  • KELP selects KG paths as prompt context at inference time, whereas Dr. Free uses KG paths at training time to construct questions for self-evolving search agents. Our Related Work also discusses KG-based construction methods such as KG Paths (L139), which Dr. Free outperforms (0.401 vs. 0.358 average EM on Qwen2.5-7B, Table 1).
  • IG-Search applies IG on the solver side, rewarding each search step by its gain in answer likelihood so that the solver learns to search better. Dr. Free applies IG on the proposer side, contrasting the full evidence against the strongest shortcut context to judge question quality, without requiring solver rollouts for reward computation. IG-Search is trained on human-annotated QA pairs and is thus not directly comparable to our data-free setting, but the two are complementary: its solver-side reward could be plugged into our solver training.
AuthorsOpenReview ↗

Response to Reviewer CNtX(Part3)

Q2.1: Recall and precision of retrieved documents

We measure the recall (R) and precision (P) of gold supporting documents among all documents retrieved over each search trajectory. The best result in each column within each backbone is in bold.

MethodsNQ RNQ PHotpotQA RHotpotQA P2Wiki R2Wiki PMuSiQue RMuSiQue PAvg. RAvg. P
Qwen2.5-3B-Instruct
Base0.5170.2960.3740.2760.4200.2750.1820.1350.3730.245
Search-R10.6550.4080.4890.4420.4100.3310.2520.2040.4510.346
Dr. Zero0.6740.4180.5580.4130.5040.3410.3010.2140.5090.347
Dr. Free (Ours)0.6500.3980.5770.4320.5500.3480.3390.2240.5290.351
Qwen2.5-7B-Instruct
Base0.6370.3700.5570.4020.5060.3170.3060.2030.5010.323
Search-R10.6570.4120.5210.4450.4660.3390.3200.2280.4910.356
Dr. Zero0.6810.4280.5330.4490.4680.3680.2530.2060.4840.363
Dr. Free (Ours)0.6810.4230.5770.4340.5270.3530.3520.2470.5340.364

Dr. Free achieves the highest average recall at both scales, with comparable or slightly higher precision, and the recall gains concentrate on multi-hop benchmarks. Its EM improvements are therefore accompanied by retrieving more of the required evidence, rather than relying on parametric memory alone. These results have been added as Table 2 in the revision.

Q2.2: Base models without search

We evaluate the base models without search (closed-book) and with search (same agent prompt, no training), and compare them with Dr. Free (EM):

MethodsNQTriviaQAPopQAHotpotQA2WikiMuSiQueBamboogleAvg.
Qwen2.5-3B-Instruct
Base, no search0.1230.3660.1280.1470.1570.0320.2080.166
Base, with search0.1750.2930.2750.1710.2140.0670.1680.195
Dr. Free (Ours)0.3800.5720.4160.3530.3900.1720.3440.375
Qwen2.5-7B-Instruct
Base, no search0.1980.4740.1560.2070.2430.0520.3120.235
Base, with search0.3020.5440.3270.3320.3490.1350.3440.333
Dr. Free (Ours)0.3970.6100.4380.3870.3890.1640.4240.401

Parametric knowledge alone is limited on these benchmarks, especially for multi-hop questions, and Dr. Free substantially improves over the closed-book baseline at both scales (0.166 → 0.375 on 3B and 0.235 → 0.401 on 7B). Moreover, Dr. Free explicitly guards against parametric shortcuts during training: the closed-book context is one of the shortcut contexts in the IG reward, so questions answerable from parametric memory alone receive zero reward. These results have been added to Appendix B.3.

Reviewer_eVk6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Dr. Free trains self-evolving search agents using KG-based question construction and a proposer reward that contrasts answer likelihood under full evidence against shortcut contexts. It replaces rollout-based difficulty estimation with teacher-forced scoring. Experiments across 7 QA benchmarks and 5 backbones show improved average performance, particularly on multi-hop QA, and substantially lower proposer training cost.

Critical strengths
  1. The paper identifies a concrete weakness of difficulty-based question generation: difficult questions need not require the intended evidence.
  2. The reward ablations show downstream benefits, and replacing rollout-based proposer rewards with teacher-forced likelihood scoring substantially reduces proposer training cost.
Critical weaknesses
  1. Table 5 reports answerability under shortcut contexts without reporting answerability with complete evidence. Consequently, low shortcut success alone cannot distinguish evidence-dependent questions from ambiguous or unsupported ones. This leaves the interpretation of the audit incomplete.
  2. The reward compares full evidence against selected shortcuts, but does not systematically test multi-passage subsets. For three-hop examples, a two-passage solution may therefore remain undetected. The results support resistance to the evaluated shortcuts, rather than establishing that every intended hop is necessary.
  3. The relationship to prior likelihood-based rewards needs clarification. IG-Search and IGPO are not discussed, despite their related use of answer-likelihood gains.
Questions for authors
  1. Add control conditions for the IG reward e.g., a length-matched irrelevant context, shuffled passage order, and the final-hop passage alone and report whether SIGS_IG predicts the per-question shortcut labels behind Table 5?
  2. What are the exact reward scale, normalization, and HRPO settings for Difficulty Only and IG + Difficulty, and why does adding difficulty lower performance below IG alone (0.340 vs. 0.375)?
  3. Tables 1 and 8 show NQ drops on three of five backbones. Do you have evidence that this reflects over-searching by the solver? Please report average search turns/tokens per benchmark and whether the NQ gap persists under a matched inference search budget.
  4. Figure 2 calls the Abruka case “Actual: 2-hop,” but does not show the passages or IG scores. I suspect one hop may suffice—the lighthouse passage could already say “Abruka, Estonia.” Could you show the per-hop passages and likelihoods?
  5. Figure 2’s question is missing “is” (“where the Abruka rear lighthouse located at?”); small typo.
4 comments
AuthorsOpenReview ↗

Response to Reviewer eVk6(Part1)

We sincerely thank Reviewer eVk6 for the careful reading and the constructive suggestions. Our responses are as follows.

W1: Answerability with complete evidence

We thank the reviewer for this valuable suggestion. We have added the missing condition (Table 5 in the original submission, now Table 6), using the same judge and decision rule. A question is evidence-dependent if it is answerable with full evidence but not under any shortcut context. Entries are percentages over 100 questions per method.

MethodFull ↑Closed-book ↓Source-only ↓One-shot ↓Single-comp. ↓Evidence-dep. ↑
Dr. Zero87178165881
KG + Difficulty57482045512
Dr. Free22532517

The baselines reach high full-evidence answerability because their questions are effectively single-hop: 81% of Dr. Zero's questions are answerable from the source passage alone, and about half of KG + Difficulty's are answerable without evidence (48%) or from a single component (51%). Dr. Free targets genuine multi-hop questions, which are harder to construct in a well-posed form, so its full-evidence answerability is lower (22%). Yet it yields by far the most evidence-dependent questions: 17 per 100, versus 1 and 2 for the baselines.

Importantly, the remaining questions do little harm. Most of the 78 unanswerable questions lack sufficient evidence (41) rather than being ambiguous (4), and the solver fails on them too: 74 of the 78 are never answered correctly in sampled rollouts. Under GRPO such questions produce no gradient, whereas shortcut questions actively mislead the solver into answering from a single shortcut passage.

Q1: IG reward controls and per-question shortcut prediction

We ran all three suggested controls and the per-question analysis on the 100 Dr. Free questions audited in Table 6.

  • Irrelevant context. Full evidence yields a much higher answer log-likelihood than a length-matched irrelevant context (+4.60 nats on average). The likelihood signal is therefore driven by the relevance of the evidence, not by context length.
  • Shuffled passage order. The IG reward is largely preserved after shuffling (Spearman ρ = 0.86, 85% sign agreement), so it depends mainly on evidence content rather than passage order.
  • Final-hop passage alone. This control is already part of the shortcut set B (Eq. 5), which includes each individual passage in isolation, including the final-hop passage.

Per-question prediction. Only 9 of the 100 questions trigger at least one of the four shortcuts (closed-book, source-only, one-shot, or single-component). None of these 9 receives a positive IG reward, whereas 28 of the remaining 91 (30.8%) do. The reward is therefore conservative: it withholds credit from some valid questions, while none of the 9 audited shortcut questions were rewarded.

W2: Multi-passage subsets and the scope of the shortcut comparison

We agree with this scoping: our reward evaluates resistance to the tested shortcuts (L23–24, L293–294), not the necessity of every hop. Following the suggestion, we evaluated all two-passage subsets on the 49 three-hop questions in the audit.

  • For 41 of them (84%), the full evidence yields a higher answer likelihood than every two-passage subset.
  • For the remaining 8, a subset that drops the intermediate passage scores higher. None of these 8 can be answered from any single passage, and their final-hop passage is never retrieved by the question-only top-3 retriever (0/8).

We also note that the shortcut set B already includes one multi-passage context: the one-shot context, i.e., the top-3 passages retrieved with the question alone.

W3: Relation to IG-Search and IGPO

We have added both works to Related Work (Section 2.2). All three methods use answer-likelihood gains, but for different purposes:

  • IGPO and IG-Search apply IG on the solver side as dense credit assignment for learning to search. IGPO rewards the gain over the previous turn, and IG-Search rewards the gain over random documents. Both measure whether retrieved evidence is helpful, and both require human-annotated QA pairs for gold answers.
  • Dr. Free applies IG on the proposer side to judge question quality in data-free self-evolution. It contrasts the full evidence against the strongest shortcut context, which is relevant but insufficient, so it measures whether the full evidence adds support beyond every tested shortcut, and it needs no annotated answers.

The two directions are complementary, and solver-side IG rewards could be plugged into our solver training.

AuthorsOpenReview ↗

Response to Reviewer eVk6(Part2)

Q2: Reward settings of the ablations and why adding difficulty hurts

All ablations share the same KG pipeline, format reward (maximum 0.2), and HRPO settings, and differ only in the proposer reward. Difficulty Only replaces the IG reward (range [0, 3), Eqs. 6–7) with the pass-rate difficulty reward (range [0, 1]), while IG + Difficulty adds the two with equal weights. The components are summed without separate normalization, and HRPO then standardizes the total reward within each hop group (Eq. 9).

Adding difficulty hurts because it rewards any question that is solvable, even when it is solved through a shortcut, and thus re-rewards questions that IG rejects. For example, on one three-hop training question that can be answered in a single hop, one of five solver rollouts answered correctly via this shortcut, earning the maximum difficulty reward. IG instead assigns zero, since the strongest shortcut yields a higher answer log-likelihood than the full evidence (−4.98 vs. −10.37):

RewardFormatIGDifficultyTotalHRPO advantage
IG only0.20—0.2−0.59
IG + Difficulty0.2011.2+0.25

Adding difficulty thus turns a rejected shortcut question into a positively reinforced one. Consistently, IG + Difficulty performs on par with Difficulty Only (0.340 vs. 0.343) rather than with IG alone (0.375).

Q3: Whether the NQ gap is caused by over-searching

In short, our results do not support over-searching; the gap comes instead from the training data distribution.

Search statistics. Average search calls / generated tokens / retrieved tokens per question, under identical inference budgets for both methods (at most 5 searches, 512 generated tokens per turn, 1,024 tokens per retrieval, top-3 retrieval, greedy decoding):

Benchmark3B Dr. Zero3B Dr. Free7B Dr. Zero7B Dr. Free
NQ1.29 / 171 / 6541.23 / 168 / 5791.19 / 116 / 5931.40 / 192 / 742
TriviaQA1.28 / 182 / 6731.26 / 164 / 6141.19 / 122 / 5971.35 / 180 / 742
PopQA1.42 / 163 / 7501.42 / 149 / 6611.24 / 116 / 6421.46 / 190 / 822
HotpotQA1.62 / 230 / 10091.81 / 225 / 9911.39 / 166 / 7381.67 / 228 / 1055
2Wiki1.78 / 231 / 13412.08 / 254 / 12931.59 / 200 / 9941.83 / 246 / 1309
MuSiQue2.20 / 328 / 13182.09 / 289 / 11401.66 / 233 / 8892.02 / 283 / 1208
Bamboogle1.75 / 198 / 10051.82 / 188 / 9211.49 / 152 / 7541.82 / 194 / 977

Search volume does not track NQ accuracy: on 3B, Dr. Free uses fewer searches and tokens on NQ than Dr. Zero, whereas on 7B it uses more, yet it trails slightly in both cases.

Training data distribution. Our experiments show that the gap instead comes from the training data distribution: Dr. Zero's questions are almost all single-passage ones that resemble NQ, whereas Dr. Free spends two-thirds of its budget on multi-hop chains. When both methods are trained with the same number of single-hop questions, they perform on par on NQ (Δ = −0.07 ± 0.29 on 3B and +0.14 ± 0.27 on 7B, in EM percentage points). Please see our response to Reviewer X17h (W2 & Q3) for the full details.

Q4 & Q5: The Abruka example in Figure 2 and the typo

We thank the reviewer for this careful check. The reviewer is right: the lighthouse passage already mentions Estonia, so this example admits a source-only shortcut. Importantly, our IG reward detects this: the source-only context yields a higher answer likelihood than the full evidence (g < 0), so the example receives zero reward. It therefore illustrates how the reward filters shortcut questions rather than serving as a positive example.

We have replaced it with a positive-IG example: "At the mouth of which river is the city where Frances Itani was born?" (answer: Moira River), which also removes the typo. The table reports the answer log-likelihoods for every context in Eqs. (5)–(6) under the frozen Qwen2.5-3B-Instruct anchor:

ExampleClosed-bookSource-onlyOne-shotPassage 1Passage 2Full evidenceStrongest shortcutGain gIG reward
Abruka (original)−13.35−3.30−4.43−7.71−10.08−7.11−3.30−3.810
Frances Itani (new)−7.42−5.94−7.00−5.94−3.99−3.65−3.99+0.340.33

For Abruka, the source-only context is the strongest, so the reward is zero. For Frances Itani, the full evidence outperforms every evaluated shortcut context, so the reward is positive. The updated example appears in Figures 2 and 3, and the passages and likelihoods for both examples are in Appendix B.6 (Table 26).

I appreciate the author’s careful clarification and experiments, which has resolved some of my previous questions. I keep my original positive rating.

Thank you for your careful consideration of our responses and for maintaining your positive rating. We are glad that our clarifications and additional experiments have helped resolve some of your questions, and we sincerely appreciate your valuable feedback, which has helped us improve the paper.

Reviewer_X17hOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Dr. Free, a self-evolving framework for training search agents without human-annotated QA pairs. The authors first show that prior difficulty-based self-play methods (such as Dr. Zero) suffer from severe shortcut degeneration, where most nominally multi-hop questions are generated without search and can be answered directly from the seed passage. To address this, Dr. Free constructs questions from pre-verified Wikidata5M relational chains aligned with Wikipedia passages, and replaces rollout-based solver difficulty rewards with a teacher-forced, shortcut-aware information-gain (IG) reward that measures how much the full evidence chain increases target-answer likelihood over shortcut contexts (closed-book, source-only, one-shot, and single-passage). Experiments across five backbones and seven QA benchmarks show substantial gains on multi-hop QA and over 7x faster proposer training compared to Dr. Zero.

Critical strengths
  • The motivation is very clear and well-demonstrated. The audit of Dr. Zero (Table 4) convincingly exposes a major failure mode in difficulty-driven self-play, showing that over 85% of generated "multi-hop" questions involve zero search and leak the answer in the seed passage.
  • Replacing solver rollouts with teacher-forced likelihood contrasts (full evidence vs. shortcut contexts) to estimate proposer rewards makes a lot of sense. It directly targets multi-hop shortcuts while substantially improving proposer training efficiency.
  • The experiments and ablations are thorough and solid. The authors reports multi-seed training runs (Table 15), which is rare and appreciated in post-training work.
Critical weaknesses
  • The method relies heavily on a high-quality external knowledge graph (Wikidata5M), along with a curated whitelist of ~40 functional relations and detailed prompt rules. Moreover, all experiments are conducted exclusively on Wikipedia, without any out-of-domain validation. How to apply this framework in others domains without a clean knowledge graph remains an open question.
  • Relatedly, the performance gains come almost entirely from multi-hop benchmarks (such as 2WikiMultihopQA and MuSiQue) that were themselves constructed from Wikipedia and Wikidata relational structures, whereas performance on natural single-hop benchmarks (e.g., NQ and PopQA on 3B) is slightly lower than Dr. Zero.
Questions for authors
  • Have you evaluated whether a solver trained with Dr. Free generalizes to benchmarks built on out-of-domain corpora beyond Wikipedia?
  • Constructing the pool of 100,000 verified chains requires entity normalization, subgraph filtering, and two-phase retrieval verification across all intermediate hops on Wiki-18. Could you provide a more detailed breakdown of the computational and wall-clock cost of this KG construction and verification pipeline, and how the overall efficiency looks when this preprocessing cost is included?
  • On single-hop benchmarks such as NQ, Dr. Free slightly trails Dr. Zero on several backbones despite strong multi-hop gains. Is this because constraining question generation to KG relation chains narrows the diversity of single-hop questions compared to free-form passage generation?
4 comments
AuthorsOpenReview ↗

Response to Reviewer X17h (Part1)

We sincerely thank Reviewer X17h for the positive assessment and the constructive comments. And we address each point below.

W1 & Q1: Dependence on a clean KG and generalization beyond Wikipedia

We thank the reviewer for this important question. We address it from two angles: whether the trained solver generalizes beyond Wikipedia, and whether Dr. Free can be applied without a clean KG.

1. Generalization to out-of-domain corpora

(a) ConcurrentQA. We evaluate on ConcurrentQA [1], whose multi-hop questions require reasoning over both Wikipedia and a private Enron email corpus. The email part is entirely outside the Wikipedia domain used for training. Details have been added to Appendix B.1.

MethodsEM (Overall)F1 (Overall)F1 (Email-involved)F1 (Wiki-only)
Qwen2.5-3B-Instruct
Base0.1700.1990.1870.235
Search-R1 (Jin et al., 2025)0.3640.4280.4360.405
Dr. Zero (Yue et al., 2026)0.3210.3780.3730.394
Dr. Free (Ours)0.3630.4250.4170.449
Qwen2.5-7B-Instruct
Base0.3450.4290.4240.446
Search-R1 (Jin et al., 2025)0.4010.4600.4600.459
Dr. Zero (Yue et al., 2026)0.3930.4560.4630.436
Dr. Free (Ours)0.4090.4790.4850.464

Dr. Free consistently outperforms Dr. Zero on all metrics at both scales, including email-involved questions. It is on par with the fully supervised Search-R1 at 3B and outperforms it at 7B, without using any human-annotated QA data.

(b) BrowseComp-Plus. We further evaluate on BrowseComp-Plus [2], whose corpus consists of curated web documents rather than Wikipedia. Dr. Free outperforms Search-R1 in all eight settings and improves over the base model in seven of eight. Full results, including Dr. Zero and a retriever-level analysis, are provided in our response to Reviewer CNtX (Question 2, BrowseComp-Plus) and have been added to Appendix B.2.

Together, these results show that a solver trained on Wikipedia-derived chains transfers to private emails and open web corpora.

2. Applying Dr. Free without a clean KG

The KG is a replaceable upstream component rather than the core of our method. The IG reward only requires a question, a target answer, the intended evidence passages, and shortcut contexts; it never accesses graph structure. Likewise, our chain verification uses the solver's own retriever and corpus, so it is not tied to Wikipedia. When no curated KG is available, short evidence chains can be built from raw text with off-the-shelf LLM-based extractors such as KGGen [3]. Importantly, Dr. Free does not require a KG over the entire corpus: each iteration consumes only about 12,800 short chains, so extraction can be restricted to a small subset of documents, keeping the construction cost low.

[1] Reasoning over Public and Private Data in Retrieval-Based Systems. TACL, 2023.

[2] BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent

[3] KGGen: Extracting Knowledge Graphs from Plain Text with Language Models. NeurIPS 2025.

AuthorsOpenReview ↗

Response to Reviewer X17h (Part2)

W2 & Q3: Source of the multi-hop gains and the single-hop gap

We thank the reviewer for these insightful questions. In short, the multi-hop gains do not fully depend on Wikidata structure: the IG reward alone, without any KG chains, recovers about half of the improvement, and the gains also hold on a non-Wikipedia corpus. The small single-hop gap reflects how the training budget is allocated, not reduced question diversity. We explain both points below.

1. The multi-hop gains do not fully depend on Wikidata structure

Our ablation (Table 2 in the original submission, now Table 3; Qwen2.5-3B, EM) separates the contributions of the KG chains and the IG reward. "Multi-hop Avg." is the mean over the four multi-hop benchmarks, and "Avg." is over all seven benchmarks.

VariantKG chainsProposer rewardHotpotQA2WikiMuSiQueBamboogleMulti-hop Avg.Avg.
Dr. Zero×Difficulty0.2980.2910.0910.2000.2200.326
w/o KG×IG0.3190.3560.1290.2720.2690.335
Difficulty Only✓Difficulty0.3240.3570.1210.2640.2670.343
Dr. Free✓IG0.3530.3900.1720.3440.3150.375

Without any Wikidata chains (w/o KG), replacing the difficulty reward with IG already raises the multi-hop average from 0.220 to 0.269, including on the Wikidata-derived 2Wiki (0.291 → 0.356), so the gains cannot come from sharing relational structure with the benchmarks. KG chains contribute a comparable gain on their own (0.267), and the two are complementary (0.315 combined). The gains also hold beyond Wikipedia: on ConcurrentQA, Dr. Free outperforms Dr. Zero at both scales (see our response to W1 & Q1).

2. The single-hop gap reflects data allocation, not reduced diversity

As our audit shows (Table 4 in the original submission, now Table 5), most of Dr. Zero's nominally multi-hop questions can be answered from the seed passage, so Dr. Zero effectively trains on single-hop data, which matches NQ-style benchmarks. Dr. Free instead allocates two-thirds of its budget to two- and three-hop chains.

To test the diversity hypothesis directly, we train both methods on the same number of single-hop questions (3,816 for 3B and 3,584 for 7B) with identical training steps. The table reports Δ = Dr. Free − Dr. Zero in EM percentage points (mean ± std over three seeds):

BenchmarkQwen2.5-3BQwen2.5-7B
NQ−0.07 ± 0.29+0.14 ± 0.27
TriviaQA+0.08 ± 0.93−0.17 ± 0.43
PopQA+0.33 ± 0.23−0.31 ± 0.42
Avg. (7 benchmarks)+0.05 ± 0.68+0.48 ± 0.94

With equal single-hop data, the two methods perform on par: all differences are below 0.5 points. This indicates that constraining generation to KG chains does not reduce the usefulness of single-hop questions, so the original gap comes from the amount of single-hop data rather than its diversity.

Q2: Cost of KG construction and overall efficiency including preprocessing

We thank the reviewer for this question. The original submission stated in Appendix A.2 that the pipeline takes less than one hour, which was a conservative bound. A dedicated re-timed run on H800 gives the following upper bounds:

StageWall-clock
Candidate generation (entity normalization, subgraph filtering, random walks)< 3 min
Phase 1 verification (source-passage scan over Wiki-18)< 2 min
Phase 2 verification (per-hop retrieval probes) and I/O< 5 min
Total< 10 min

The pipeline is fast because no stage involves LLM inference: candidate generation consists of string matching and graph operations over existing Wikidata5M triples, and verification consists of substring checks and batched lookups in the existing index.

Including this one-time cost barely changes the overall efficiency. Even if it is charged entirely to a single iteration, Dr. Free remains about 7× faster than Dr. Zero in proposer training plus data generation (about 2.0 vs. 14.2 hours) and about 2.5× faster end-to-end (about 8.8 vs. 22.0 hours). Since the pool is reused across all iterations, later iterations incur no additional cost. We have updated Appendix A.2 accordingly.

Thank you for the clarificaiton and additional experiements, which has resolved some of my previous concerns. I maintain by positive rating.

Thank you for your recognition of our work from the initial review and for maintaining your positive rating. We are glad that our clarifications and additional experiments helped resolve some of your concerns, and we sincerely appreciate your constructive feedback throughout the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.