- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates how search agents use retrieved evidence to decide whether to stop or continue searching. Through controlled manipulations of entity mentions, answer-like candidates, and supporting facts, the authors find that fine-tuning shifts stopping behavior from candidate-based to mention-based cues, potentially reflecting correlations in the training data.
Critical strengths
1.Understanding what drives search agents to stop or continue searching is an important problem for reliable agentic search. Investigating how fine-tuning changes the evidence cues underlying these decisions is a promising direction. With a clearer research focus and stronger evidence of generalizability, this work has the potential to make a more substantial contribution.
2.The paper systematically manipulates entity mentions, answer-like candidates, and supporting facts to examine their effects on stopping decisions
Critical weaknesses
1.The main finding that fine-tuning shifts stopping decisions from answer-like candidates to entity mentions is primarily established using a specially constructed SFT dataset. However, the training data contain a strong shortcut: the presence of a page about the question's entity almost perfectly predicts the stopping label (AUROC = 0.98). More importantly, none of the 473 training examples labeled as "continue searching" contains the first-hop supporting page, meaning that the training data entirely omit a common multi-hop search situation where the first hop has been retrieved but further evidence is still needed. Under such conditions, it is not surprising that the fine-tuned model learns to rely on entity mentions when deciding whether to stop. The paper itself shows that this correlation is much weaker in public search-agent training datasets, and the externally evaluated models do not exhibit the same mention-dominant behavior. Therefore, it remains unclear whether the observed shift represents a general problem in search-agent fine-tuning or is mainly an artifact of the authors' training data construction. Evaluation with more representative SFT data would help establish the broader relevance of the findings.
2.The paper cites relevant studies on search stopping decisions, evidence sufficiency, shortcut learning, and counterfactual training, but does not sufficiently explain what new understanding it provides beyond these studies. Prior work has already identified premature stopping, unreliable evidence use, and training shortcuts in search agents, and has proposed methods to address these issues. While the paper provides a more fine-grained analysis of mention, candidate, and fact cues, it remains unclear which findings represent genuinely new insights rather than more specific demonstrations of known problems. The authors should more explicitly distinguish their contributions from the closest prior work and clarify what new understanding of search-agent behavior is established.
3.If the main objective is to understand how supervised fine-tuning changes search agents' stop-or-continue decisions, it is unclear how some of the additional experiments, particularly those comparing larger models with and without reasoning, directly contribute to this objective, since they do not examine fine-tuning effects. If fine-tuning is the central variable of interest, the authors should explain why reasoning is also investigated and what additional evidence these experiments provide for understanding the effects of fine-tuning. Alternatively, if the objective is to understand what determines search agents' stopping decisions more generally, the paper does not explain why fine-tuning is singled out for extensive analysis or provide a systematic framework for examining other influencing factors. As a result, although the individual findings are interesting, the connections between the central research question, experimental analyses, and overall conclusions could be made clearer.
Questions for authors
Please see the weaknesses.
2 comments
Response to Reviewer xQ6j (2/2): details for W1 and W3
Agents by training data (Qwen2.5-7B; effects on stopping in points, development pages).
| Agent | Mention | Candidate | Fact | Candidate answers on near misses | Answers on 157 natural first results |
|---|---|---|---|---|---|
| Before fine-tuning | 4.9 | 25.2 | 3.0 | 79.4% | 44.6% |
| Our data (three seeds) | 47.7 | 14.4 | 41.0 | 56.0% | 51.6% |
| Public trajectories, same recipe (three seeds) [P] | 2.5 | 34.3 | 60.6 | 22.8% | 9.1% |
| R3-RAG cold start / after RL (released) | 6.0 / 13.2 | 56.0 / 52.1 | 22.7 / 30.6 | 62.0% / 59.3% | 3.2% / 1.9% |
| Search-R1 (released) | 1.9 | 2.8 | 27.8 | 6.2% | not run |
"Fact" is the stopping rate on a page that states the fact minus that on its near miss (small before fine-tuning, when the model stops on nearly all of both, answering or declining). The natural first results contain the entity's page but not all the evidence.
Is it the statistic (W1)? We re-sampled the public data in two steps (same recipe, three seeds each; the design is post hoc). The first replaces all 211 search examples that contain the first-hop page without complete support, so that the entity's page remains in 5% of the search examples instead of 55%; the AUROC rises from 0.56 to 0.81. The second also replaces stop examples and raises it to 0.97. Mention effect: +2.5 [−1.1, 6.0], +2.3 [−1.4, 6.0] and +3.2 [−1.4, 8.2]; candidate effect: +34.3, +26.1 and +20.8. Relative to the candidate the mention gains 14.2 points [6.5, 23.1], so a higher coupling shifts the balance between the two cues, but the mention effect itself changes by only +0.8 points [−3.1, 4.9]. The re-sampled examples remain states of real trajectories and differ from ours in page format, query targets and the share of answer examples; the result does not show which property of our data produces the cue.
The situation you named, on real states. 40 states from public trajectories with insufficient evidence, verified by two blind reviewers; in 39 the first hop is present and the second missing. On these 39 the agents trained on public trajectories answer on 4, 4 and 6 (three seeds) and ours on 29, 31 and 27; on 40 states with sufficient evidence they answer correctly on 32, 32 and 34, and ours on 34, 35 and 37. The comparison is descriptive and was specified before the outputs. The states come from the same public source as the first group's training sample (other questions), so they favour that group.
A second public source. RQ-RAG trajectories, same recipe, three seeds, test fixed before any output, on pages from 80 new questions: mention effect −0.3 points [−2.6, 1.3], against +46.8 [32.4, 61.0] for our data on the same pages [P: direction; met]. These agents stop on only 19.5% of the pages that state the fact, so this shows the direction and little more.
The released R3-RAG agents. On development pages the name raises candidate answers by 35.2 points [25.5, 45.1] (after RL) and 28.5 [16.4, 40.5] (cold start) when it stands in the passage that states a relation of another entity, and by 6.9 [0.7, 12.7] and 3.2 [−4.2, 11.1] when the two stand in separate passages: the effect is several times larger when the two share a passage. For the checkpoint after RL we fixed a test in advance on held-out pages (166 questions; candidate shown, fact absent): it answers with the candidate on 108 when the passage names the asked entity and on 43 when it names another (+38.5 points [28.5, 48.4]; 103 against 39 by string match) [P; met]. Its stops rise from 55 to 112, and without a candidate go from 11 to 8 (difference 35.6 points [25.6, 45.5]). The error is shown on constructed pages only; on the 157 natural first results the two checkpoints answer on 3.2% and 1.9%.
One instrument for all agents (W3). Measured the same way as the 7B agents of the table, seven API models with reasoning on show mention and candidate effects of at most 7.5 points and fact effects of 20 to 92 points (one-hop pairs, short and long pages; given here only); two of them without reasoning show fact effects of 26 and 4 points.
Disclosures.
- The released agents entered through a supplementary eligibility check after our original gate was not passed; one further agent, ReSearch, did not pass it. The two R3-RAG checkpoints come from the same group.
- R3-RAG and Search-R1 have no before-fine-tuning row. In their prompt formats the untuned base model gave invalid decisions too often (15.3% and 7.5%; limit 5%).
- The public-trajectory arm entered through a follow-up eligibility check designed after the first attempt stopped at its gate. The prediction and both criteria had been fixed before any training: "weaker than with our data" was met, "still present" was not.
- One of the three RQ-RAG seeds missed our answer-correctness check (19 of 40 correct; threshold 20).
- The held-out pages were used twice during the discussion (Appendix H).
Response to Reviewer xQ6j (1/2): W1–W3
You wrote that "with a clearer research focus and stronger evidence of generalizability" the work could make a more substantial contribution. We ran the evaluation you suggested, and the revision reports what it found. [P] marks a prediction or criterion fixed before the run; "post hoc", an analysis designed after results were seen.
W1. A general problem, or an artifact of our training data? The paper's claim has two parts: "Features other than support move the decision substantially, and which of them matters most changes with fine-tuning" (Section 1). On the second part you are right: which feature it is depends on the training data. The first part holds beyond our data.
W2. What new understanding beyond prior work? Four statements; prior work is described as in Section 5.
W3. Focus and framework. The paper asks one question: which features of the retrieved page drive the decision to answer or to keep searching. One instrument, the edited pages, serves three purposes: identifying the cues, comparing the same model before and after fine-tuning, and testing how far the result extends. Fine-tuning is the factor we vary (Section 3 before; 4.1–4.3 after; 4.4 why; 4.5 what removes the cue). The larger models serve the third purpose. We run them with and without reasoning because our fine-tuned agent acts without a reasoning step. Without it most of them answer on near misses, but none responds more to the mention than to the candidate. We did not vary their training, so we do not attribute their behavior to the cause we identify for our agent.
Revised: the abstract, the end of Section 1 (a five-step map), Sections 4.4 and 5, the limitations and Appendix H.