- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how AI agents can be proactively helpful by seeking more necessary information on their own, rather than just reacting to explicit user prompts. The authors introduce two types of proactivity: horizontal (finding obvious missing information from the start) and vertical (following a trail where newly discovered evidence reveals the need to search for another). They use a need graph that maps out all the required evidence for a specific task. Then, the authors also propose Q&D, a new training method that teaches a smaller model to prioritize asking questions that uncover the most useful evidence. The results show that this approach helps the AI find deeper information, even allowing it to outperform a model 15x its size. Furthermore, when applied to a simulated customer service setting, the agent successfully solved more user problems.
Critical strengths
- The paper introduces horizontal and vertical proactivity and operationalizes them through need graphs and measurable metrics. This provides a structured way to evaluate several things: whether agents proactively seek information, what information it finds, and how it follows dependencies. The framework is a meaningful contribution because it enables more fine-grained, judge-free evaluation of behavior seeking information.
- The Q&D method consistently improves required-evidence coverage over the same prompted model across three multi-hop QA benchmarks. The experiments also include comparisons with larger models and ablations that help isolate the effect of training. Moreover, the improvements in retail task success on -bench, without additional training, provide encouraging evidence that the learned behavior can transfer beyond the original QA setting.
Critical weaknesses
- The evaluation relies on small, fixed retrieval pools where most prerequisite edges do not meaningfully constrain retrieval. Although the authors acknowledge and investigate this issue, the results mainly demonstrate better coverage of deeper decomposition steps rather than dependency-driven retrieval in more realistic settings.
- While the coverage of evidence is gained consistently, the improvements in QA (Table 2) remain modest and statistically inconclusive. Stopping behavior is also imperfect, particularly on StrategyQA, where the questioner often stops before collecting all required evidence. This raises questions about how effectively retrieval gains translate into end-to-end task performance.
- The comparisons with recent learning-based adaptive retrieval and stopping methods are quite limited. Furthermore, the authors report only two final-stage training seeds and acknowledge deviations from the original checkpoint selection rule. These factors limit confidence in the robustness and generality of the findings.
Questions for authors
Please address the weaknesses above. There are several questions starting from the weaknesses. Authors can treat them as suggested questions to answer during the rebuttal phase if they find them useful:
- Appendix I.4 provides a useful analysis of prerequisite dependencies. Could the authors elaborate on how they expect Q&D's vertical-proactivity gains to generalize to larger retrieval corpora, where following dependencies may be more important? Are there any observations from the current experiments that support this expectation?
- What do the authors see as the main bottleneck of translate evidence coverage into better answer quality? Can it be pẻmature stopping, the use of retrieved evidence by the answerer, or other factors?
- Could the authors clarify how Q&D differs from recent learning-based adaptive retrieval and stopping approaches, particularly in terms of training signals and decision-making? Additionally, given the shared training components and checkpoint selection limitations noted in Section 7, how should readers interpret the robustness of the reported improvements across training runs?
6 comments
Q2 Comment
Q2. Why doesn't better evidence coverage always improve answer quality?
We identify two bottlenecks between finding evidence and producing correct answers: limitations in final-answer generation and premature stopping: Q&D trains the questioner to seek missing information proactively, while a separate, fixed LLM uses the collected evidence to generate the final answer. On MuSiQue, Q&D retrieves evidence containing the answer more often than prompting (94.4% vs. 84.8%), but final-answer F1 remains similar when that evidence is available (47.8% vs. 48.0%). To examine it, we replaced the fixed answer-generating model with Claude Opus 5 (on the exact previous answerer input) and it increased Q&D's absolute F1 by 31.3%, suggesting substantial room for improving answer generation independently of information seeking. On StrategyQA, Q&D retrieves more required evidence than prompting at equal retrieval spend, but premature stopping limits this advantage to be translated to a better F1. Among runs ending without the evidence needed for the final answer, Q&D chooses to stop rather than reaching the retrieval limit in 88.5% of cases, compared with 39.0% for prompting.
Our new experiments show that improved evidence collection can lead to better final answers: In the larger document collections described in Q1, Q&D improves MuSiQue F1 over prompting by +5.8 [3.0, 9.1] points. To further examine this connection, we compared Q&D before and after its final training stage, which teaches the questioner to keep searching when evidence is missing. This training improves evidence coverage before stopping on all three benchmarks and increases answer F1 on MuSiQue (+3.3 [0.9, 6.4]) and 2WikiMultiHopQA (+3.0 [0.5, 5.9]), although it requires additional retrieval calls.
We have added these analyses in Appendices G.1 and D.1 and clarified the findings in Section 5.
Q3 Comment
Q3. How does Q&D differ from learned retrieval methods, and how robust are its gains?
Q&D learns what information to seek next and when to stop, using the evidence recovered by its questions as the training signal: This differs from Adaptive-RAG, which learns to select a retrieval strategy based on question complexity, and Search-R1, which learns search-and-answer decisions from final-answer rewards. Q&D instead compares candidate questions by the required evidence their subsequent retrievals recover, and trains the questioner to continue when evidence is still missing. Need graphs identify the required evidence during training but are not provided at inference time.
Compared with Search-R1, Q&D recovers more evidence and produces better answers when each method decides when to stop, although results at equal retrieval cost are mixed: Our paper already compares Q&D with PAR²-RAG, a structured retrieval approach that does not require training (Section 5). To address your concern about learned methods, we additionally evaluated the released Search-R1 7B on the same held-out tasks, keeping the retriever, document collections, answer-generation components, and retrieval limits fixed (Appendix E.2). When each method chooses when to stop, Q&D recovers more required evidence on all three benchmarks, improving answer F1 by +13.1, +4.2, and +11.5 percentage points on MuSiQue, StrategyQA, and 2WikiMultiHopQA, respectively (all statistically significant). Search-R1 generally stops earlier and uses fewer retrieval calls, which helps explain the difference. At equal retrieval cost, Q&D leads on MuSiQue, performs similarly on StrategyQA, and trails on 2WikiMultiHopQA. Since the released models also differ in their base models and training data, this comparison does not isolate the effect of their training objectives.
The coverage improvements are consistent across additional training runs, but these do not establish robustness of the full training pipeline: We extended the original two final-stage training runs to four. All four improve coverage over prompting on all three benchmarks, with all 12 confidence intervals excluding zero and only 0.4–0.7 percentage points of variation across runs. However, these runs share the same earlier checkpoint and training data, so they establish reproducibility of the final training stage rather than independent replication of the complete pipeline. No checkpoint satisfied all the original selection criteria, including the stopping requirement. We now explain the alternative selection procedure and its limitations in Section 7 and Appendix L.
Q1 comment
Thank you for finding the paper "clear and easy to follow" and for recognizing our need-graph framework as a "meaningful contribution" enabling "fine-grained, judge-free evaluation." We also appreciate your recognition of Q&D's consistent retrieval gains and its transfer beyond QA without additional training.
Your three points motivated additional informative experiments:
Q1. Do Q&D's gains hold in larger retrieval corpora where dependencies matter?
Q&D continues to recover more required evidence from larger document collections, including evidence that depends on earlier answers. Its subsequent questions also benefit from information retrieved in previous steps.
To test whether the original small document pools made retrieval too easy, we combined the pools of all 200 held-out tasks into one shared collection per benchmark, increasing their size to 1,524–3,207 paragraphs (126–160× their original size). Keeping the retriever, top-k settings, and retrieval budgets unchanged, Q&D still improves equal-spend coverage over prompting by +12.2 [8.7, 16.0], +5.4 [1.1, 10.0], and +3.1 [0.7, 5.6] points on MuSiQue, StrategyQA, and 2WikiMultiHopQA, respectively.
The gains extend to evidence whose retrieval depends on earlier answers: In the larger collections, we identified cases where providing an earlier answer was necessary to retrieve downstream evidence under our test. On these cases in MuSiQue, Q&D recovers that evidence 25.9 [6.9, 44.0] percentage points more often than prompting.
We also tested whether Q&D retrieves prerequisites before their downstream evidence, but found no clear improvement in this order. This does not necessarily contradict our objective of recovering required evidence efficiently, since one retrieval can return multiple pieces of evidence. More importantly, our existing evidence-hiding ablations (Section 5) show that preventing the questioner from seeing previously retrieved evidence reduces MuSiQue coverage by 8.4 and 8.7 points across the two seeds. This suggests that Q&D uses earlier discoveries to guide subsequent questions, even though the results do not establish that it systematically follows prerequisite chains.
We have added the larger-pool experiments and dependency analyses in Appendices E.1 and I.5.