- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Dr. Free challenges the existing difficulty based approaches in self-evolving search agents. The main argument is that information gain based rewards better measure the dependence of generated questions compared to difficulty. Dr. Free also demonstrates impressive proposer training efficiency compared to prior art. The approach involves using verified knowledge graphs coupled with information gain rewards over end-to-end difficulty rewards. This prevents reward hacking and passage leakage in the proposed questions.
Critical strengths
-
At a high level, understanding reward modeling for self-evolving search agents is an important problem and I'm glad that this paper explores this area.
-
The paper demonstrates strong efficiency gains in proposer updates and improvement over several strong baselines on a large number of benchmarks.
-
I found that the paper was easy to understand and read.
Critical weaknesses
-
Table 2 provides evidence that IG reward does better than difficultly rewards. I have two core concerns:
- It is well established that using simple rewards and letting the model discover on its own often scales better. Does Dr. Free scale as well as difficulty rewards?
- A difficulty reward plus an equally strong evidence-dependence filter could achieve comparable quality under matched compute. An additional ablation would help establish the approach better,
-
The reward measures improvement in answer likelihood but not whether the intended passage is necessary. A positive margin against the tested shortcut doesn't establish that every passage is necessary because the reward does not test all shortcuts. Specifically, the operator in Eq. 6 only tests one shortcut, which does not guarantee that no shortcut would be used. I would recommend the authors to do an empirical analysis validating that the rewarded questions actually reuire the evidence.
Questions for authors
-
The paper missed some related works that could be used to make the evaluations and discussions about novelty more convincing. How is Dr. Free different from these approaches? How well does it do empirically?
- KELP (Liu et al ACL 2022) establishes the importance of KG-based task construction
- Some other works like IG-Search (Liang et al., 2026) also establish that information gain rewards help train stronger search agents.
-
The paper uses standard benchmarks in multi-hop QA like HotpotQA, NQ, TriviaQA, MuSiQue, etc. Although widely used, these benchmarks are very old, and modern LLMs might be able to answer from their parametric memory.
- In addition to EM, a better measure for their capabilities would be Recall and Precision of retrieved documents.
- To isolate the effectiveness of Dr. Free, I would recommend testing the base models on all the benchmarks without any search. That would provide a better baseline for their parametric knowledge.
- Newer benchmarks like BrowseComp-Plus should be used for evaluation because they specifically provide a local index for search. Additionally they present a much harder challenge for language models because of their better multi-hop nature as opposed to hotpotqa and musique which can be solved in at most 2-4 hops.
- I would request the authors to conduct an analysis of the number of hops their approach actually does compared to a base model.
-
Constructing a knowledge graph is non-trivial in practical settings. Although it provides verifiability, it is not scalable beyond Wikipedia. Can the authors show that it works in practical settings beyond wikipedia?
-
One of the strengths of this paper is approx 7 times lower proposer update time. However, it removes solver rollouts from reward estimation, it requires likelihood evaluation across multiple contexts, KG construction and preprocessing, and alternating solver-proposer training. Kindly comment on the fairness of efficiency evaluation.
-
In Appendix A.4, the authors share an interesting analysis on the performance of their approach across self evolution iterations. Although Qwen2.5-3B improves monotonically, the same is not true for the 7B variant. This raises questions about the efficacy of the proposed method. Is self evolution really helping? Would it help at scale in the newer generation models (say Qwen3.8)?
8 comments
Response to Reviewer CNtX(Part2)
W2: Does a positive IG reward establish that the evidence is necessary?
We first clarify that the max in Eq. 6 ranges over all shortcut contexts in B (closed-book, source-only, one-shot retrieval, and each individual passage), so a positive margin means that the full evidence outperforms every tested shortcut, not just one. We agree, however, that this establishes robustness only to the shortcuts in B, not a formal guarantee that every passage is necessary.
Following the recommendation, we provide three empirical analyses of whether rewarded questions depend on the full evidence:
- Gold-blind audit (Table 5 in the original submission, now Table 6). With GPT-4o as the judge, Dr. Free's questions are answerable under any individual shortcut in only 2–5% of cases, compared with 17–88% for Dr. Zero and 20–51% for the difficulty-reward variant.
- Per-question check. None of the 9 audited Dr. Free questions that trigger a shortcut receives a positive IG reward, so the rewarded questions do not admit any audited shortcut.
- Beyond B. We evaluate all two-passage subsets for three-hop questions: for 41 of 49 (84%), the full evidence yields a higher answer likelihood than every two-passage subset.
In addition, removing any shortcut type from B lowers downstream accuracy (Table 3 in the original submission, now Table 4), indicating that each tested shortcut matters. Further details, including control conditions, are in our response to Reviewer eVk6 (W1, Q1, and W2).
Q1: Relation to KELP and IG-Search
We have added both works to Related Work (Section 2.2). The differences are as follows:
- KELP selects KG paths as prompt context at inference time, whereas Dr. Free uses KG paths at training time to construct questions for self-evolving search agents. Our Related Work also discusses KG-based construction methods such as KG Paths (L139), which Dr. Free outperforms (0.401 vs. 0.358 average EM on Qwen2.5-7B, Table 1).
- IG-Search applies IG on the solver side, rewarding each search step by its gain in answer likelihood so that the solver learns to search better. Dr. Free applies IG on the proposer side, contrasting the full evidence against the strongest shortcut context to judge question quality, without requiring solver rollouts for reward computation. IG-Search is trained on human-annotated QA pairs and is thus not directly comparable to our data-free setting, but the two are complementary: its solver-side reward could be plugged into our solver training.
Response to Reviewer CNtX(Part3)
Q2.1: Recall and precision of retrieved documents
We measure the recall (R) and precision (P) of gold supporting documents among all documents retrieved over each search trajectory. The best result in each column within each backbone is in bold.
| Methods | NQ R | NQ P | HotpotQA R | HotpotQA P | 2Wiki R | 2Wiki P | MuSiQue R | MuSiQue P | Avg. R | Avg. P |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-3B-Instruct | ||||||||||
| Base | 0.517 | 0.296 | 0.374 | 0.276 | 0.420 | 0.275 | 0.182 | 0.135 | 0.373 | 0.245 |
| Search-R1 | 0.655 | 0.408 | 0.489 | 0.442 | 0.410 | 0.331 | 0.252 | 0.204 | 0.451 | 0.346 |
| Dr. Zero | 0.674 | 0.418 | 0.558 | 0.413 | 0.504 | 0.341 | 0.301 | 0.214 | 0.509 | 0.347 |
| Dr. Free (Ours) | 0.650 | 0.398 | 0.577 | 0.432 | 0.550 | 0.348 | 0.339 | 0.224 | 0.529 | 0.351 |
| Qwen2.5-7B-Instruct | ||||||||||
| Base | 0.637 | 0.370 | 0.557 | 0.402 | 0.506 | 0.317 | 0.306 | 0.203 | 0.501 | 0.323 |
| Search-R1 | 0.657 | 0.412 | 0.521 | 0.445 | 0.466 | 0.339 | 0.320 | 0.228 | 0.491 | 0.356 |
| Dr. Zero | 0.681 | 0.428 | 0.533 | 0.449 | 0.468 | 0.368 | 0.253 | 0.206 | 0.484 | 0.363 |
| Dr. Free (Ours) | 0.681 | 0.423 | 0.577 | 0.434 | 0.527 | 0.353 | 0.352 | 0.247 | 0.534 | 0.364 |
Dr. Free achieves the highest average recall at both scales, with comparable or slightly higher precision, and the recall gains concentrate on multi-hop benchmarks. Its EM improvements are therefore accompanied by retrieving more of the required evidence, rather than relying on parametric memory alone. These results have been added as Table 2 in the revision.
Q2.2: Base models without search
We evaluate the base models without search (closed-book) and with search (same agent prompt, no training), and compare them with Dr. Free (EM):
| Methods | NQ | TriviaQA | PopQA | HotpotQA | 2Wiki | MuSiQue | Bamboogle | Avg. |
|---|---|---|---|---|---|---|---|---|
| Qwen2.5-3B-Instruct | ||||||||
| Base, no search | 0.123 | 0.366 | 0.128 | 0.147 | 0.157 | 0.032 | 0.208 | 0.166 |
| Base, with search | 0.175 | 0.293 | 0.275 | 0.171 | 0.214 | 0.067 | 0.168 | 0.195 |
| Dr. Free (Ours) | 0.380 | 0.572 | 0.416 | 0.353 | 0.390 | 0.172 | 0.344 | 0.375 |
| Qwen2.5-7B-Instruct | ||||||||
| Base, no search | 0.198 | 0.474 | 0.156 | 0.207 | 0.243 | 0.052 | 0.312 | 0.235 |
| Base, with search | 0.302 | 0.544 | 0.327 | 0.332 | 0.349 | 0.135 | 0.344 | 0.333 |
| Dr. Free (Ours) | 0.397 | 0.610 | 0.438 | 0.387 | 0.389 | 0.164 | 0.424 | 0.401 |
Parametric knowledge alone is limited on these benchmarks, especially for multi-hop questions, and Dr. Free substantially improves over the closed-book baseline at both scales (0.166 → 0.375 on 3B and 0.235 → 0.401 on 7B). Moreover, Dr. Free explicitly guards against parametric shortcuts during training: the closed-book context is one of the shortcut contexts in the IG reward, so questions answerable from parametric memory alone receive zero reward. These results have been added to Appendix B.3.
Response to Reviewer CNtX(Part1)
We greatly appreciate Reviewer CNtX's detailed and thoughtful feedback, and we are glad that you find reward modeling for self-evolving search agents an important problem and the paper easy to follow.
Following your suggestions, we have added several new experiments, including the difficulty + filter ablation, BrowseComp-Plus, retrieval recall and precision, closed-book baselines, and newer Qwen3.5 backbones. We hope these address your concerns, and we would be grateful if you would consider raising your score in light of the new results. Our detailed responses are below.
W1.1: Does Dr. Free scale as well as difficulty rewards?
We agree that simple rewards often scale well, and we note that IG is itself a simple reward: a teacher-forced likelihood comparison that requires no solver rollouts, whereas the difficulty reward requires multi-turn rollouts for every question. The key difference is what the reward lets the model discover. Under the difficulty reward, the proposer discovers shortcuts: in our Qwen2.5-3B reproduction, 99.995% of Dr. Zero's nominally multi-hop questions are generated without any search (Table 4 in the original submission, now Table 5). IG closes this shortcut while keeping the reward simple and cheap.
Scaling across iterations. Average EM over seven benchmarks (Dr. Free from Table 9 in the original submission, now Table 10; Dr. Zero from Table 3 of its paper):
Dr. Free improves monotonically on both backbones and leads at every iteration, while Dr. Zero's 7B model declines in the third iteration. These gains also come at lower cost: on the same 8×H800 setup, one 3B iteration takes 8.6 hours end-to-end for Dr. Free versus 22.0 hours for Dr. Zero (Appendix A.2), so Dr. Free can afford substantially more self-evolution under a fixed compute budget.
Scaling across model sizes and generations. Dr. Free outperforms Dr. Zero on all five tested backbones from 3B to 8B, including the Qwen3 family (Figure 1; Table 8 in the original submission, now Table 9). Our new results on the newer Qwen3.5-4B and 9B also show consistent gains over the base models (see our response to Question 5). Within the tested range, Dr. Free therefore scales at least as well as difficulty rewards, at a much lower cost.
W1.2: Difficulty reward plus an evidence-dependence filter
We have implemented this ablation. To make the filter as strong as possible, we use Dr. Free's own IG criterion: we train the proposer with the KG + difficulty reward, retain only generated questions with a strictly positive IG margin, and keep 12,800 of them with a balanced 1-/2-/3-hop distribution. The Qwen2.5-3B solver is then trained for 50 updates (batch size 256), matching the solver budget of one Dr. Free iteration (EM):
Even with this filter, the difficulty-trained proposer falls short of Dr. Free on six of seven benchmarks, with the largest gaps on Bamboogle and 2Wiki. The reason is that filtering can only remove questions after they are generated; it cannot change what the proposer learns to generate. A proposer trained with the difficulty reward rarely produces evidence-dependent questions: of 137,984 generated candidates, only 9.47% survive validity checks, deduplication, and the positive-IG filter. Training the proposer with IG instead steers its whole output distribution toward such questions.
This alternative is also more expensive than Dr. Free: it requires solver rollouts to estimate difficulty during proposer training, plus 10.78× over-generation and IG scoring of candidates for filtering, whereas Dr. Free only computes IG during proposer training.