Stop or Continue: How Fine-Tuning Changes Evidence Use in Search Agents

After every search, an agent must decide whether it has enough to answer or should keep searching. We ask which features of the retrieved page drive this decision, using matched edits that switch a mention of the question's entity, an answer-like string and the supporting fact on or off. Instruction-tuned models from three families are more likely to answer with an answer-like string when one appears, even if nothing supports it. After supervised fine-tuning on single answer-or-search decisions, Qwen2.5-7B-Instruct responds mainly to the mention instead: the effect of a mention on its rate of stopping (answering or declining) rises from 6 to 47 percentage points and that of an answer-like string falls from 29 to 5, a reversal no shift in its overall willingness to stop can produce. It held on preregistered held-out pages; Mistral-7B-Instruct-v0.3, fine-tuned the same way, showed it on development pages. Asked about pages that mention the entity but lack the fact, the fine-tuned agent says the fact is missing for 96% of them but answers on 54%. In real search episodes, fine-tuned agents often answer once the page about the question's entity arrives, even on questions this page cannot answer, where every answer was wrong. Instructions to check for the fact barely changed what the fine-tuned agent stops on. Unlike public search-agent trajectories, its training data almost never contain a page about the entity without the fact, a coupling measurable before training; in a preregistered study on new questions, counter-examples of this kind removed 82% of the mention's effect on stopping, while generic negatives had no measurable effect. Four larger open models rarely stopped on either cue alone, but with reasoning off, five of six models from five families often answered with the answer-like string on pages linking the right entities by the wrong relation.