acceptodds
Under review as a conference paper at ICLR 2027

ProSearch: Teaching LLMs When (Not) to Search over Streaming Inputs via Reinforcement Learning

Abstract

We introduce ProSearch, a reinforcement-learning framework for training proactive search agents for streaming inputs. Given a continuous input stream, the agent must decide whether to act, what external knowledge to retrieve, and how to respond, under streaming latency constraints. Existing systems either retrieve on every input or delegate the trigger to an external classifier, leaving the “whether to act” decision outside the learned policy. ProSearch instead internalizes it, training a single policy end-to-end with GRPO so that the trigger, query, and grounded answer are emitted by the same model and optimized against the same reward. The central challenge is reward sparsity: actionable segments are rare and correct behavior requires a three-step pipeline (trigger, retrieve, respond), so an outcome-only reward yields degenerate gradients. We address this with a structured reward decomposition whose three terms each score a different sub-event of the rollout, densifying the learning signal for GRPO relative to an outcome-only reward and yielding stable training. Across three proactive tasks spanning claim verification and knowledge-seeking turn detection, ProSearch consistently beats reactive and externally-orchestrated baselines, with up to a 2 gain in end-to-end task success.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Proactive agent, Retrieval augmented generation (RAG), GRPO

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FbBnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ProSearch trains a single language model to decide whether an input segment warrants external retrieval and, when appropriate, generate a query and a grounded response. The approach combines a synthetic supervised-finetuning initialization with GRPO using an additive reward for trigger correctness, retrieval quality, and answer quality. Retrieved passages are excluded from policy-gradient and KL calculations.

The paper evaluates the approach on a newly constructed, silver-labeled podcast claim-verification dataset and two knowledge-grounded dialogue benchmarks. Experiments with 3B and 8B Llama models, supplemented by Ministral-8B results, report substantial end-to-end improvements over reactive generators and externally gated cascades. Ablations investigate initialization, reward components, retrieval settings, and transfer. Despite the streaming motivation, the evaluated setting consists of independently scored, pre-segmented text inputs with supplied context and frozen retrieval corpora.

Critical strengths
  1. A useful problem formulation with a practical integrated solution. Explicitly learning when to abstain is valuable for systems that monitor inputs without requiring a response to every segment. ProSearch combines this decision with query generation and answering in one policy. The paper clearly explains that its reward decomposition supplies denser rollout-level rankings rather than independent token-level credit, and the implementation addresses practical issues such as retrieved-token masking and cold-start collapse.

  2. Substantial reported improvements across several settings. The full recipe improves end-to-end F1 over both SFT-only models and the evaluated gated Search-R1 cascades across three tasks and two primary model sizes. Additional Ministral experiments and independent-judge regrading provide useful evidence beyond a single backbone or evaluator. The ClaimVerif dataset also extends evaluation beyond preselected claims by retaining non-actionable utterances, with episode-disjoint splits and two label-quality audits.

Critical weaknesses
  1. The comparisons do not adequately isolate joint trigger learning or structured reward decomposition. The outcome-only control assigns no terminal credit to correct waiting, making its failure to learn abstention insufficient evidence that intermediate reward shaping is necessary. A stronger comparison would use the same complete SFT initialization and reward both correct waiting and successful end-to-end execution. Furthermore, the “matched-SFT” Search-R1 control truncates trajectories at </search>, so it does not receive the same answer-generation supervision as ProSearch. Appendix A also states that ProSearch receives validation-based learning-rate/checkpoint selection that the cascades do not. Finally, the paper does not compare against a separately gated generator trained with ProSearch’s retrieval and answer rewards. This matters because the external gates already achieve near-perfect trigger F1 on the dialogue tasks; much of the observed advantage could arise from improved generator training rather than joint optimization of the trigger.

  2. The evaluation contains an apparent numerical inconsistency and does not fully establish grounded correctness. Under Appendix E’s definition, when all true-positive rows have gold evidence, E2E-F1 cannot exceed trigger F1 multiplied by retrieval hit rate, because response credit is bounded by one. However, Table 18 reports trigger F1 = 0.84, retrieval hit = 0.57, and E2E-F1 = 0.65 for DSTC11→DSTC9; the corresponding upper bound is approximately 0.48. This requires reconciliation, including any missing-gold exceptions. Separately, Case 3 is presented as successful grounding of a claim about Jack Smith’s January 6 investigation, but the displayed evidence concerns Robert Mueller’s 2017 Russia investigation. Gold-document retrieval plus verdict agreement does not itself establish evidential support. These concerns are consequential given the silver ClaimVerif test labels, its 82.1% SUPPORTS prior, and the absence of reported multi-seed uncertainty.

Questions for authors
  1. Metric consistency: Please reconcile the DSTC11→DSTC9 row in Table 18 with Appendix E’s E2E definition. How many true-positive rows lack gold evidence, and are the trigger, retrieval, response, and E2E columns computed from identical predictions and checkpoints? Providing the underlying counts and summed response credits would resolve this issue.

  2. Central training claim: Does ProSearch retain its advantage over same-initialization GRPO using a terminal reward that credits both correct waiting and complete trigger–retrieve–answer success? Does it outperform an external gate combined with a generator trained using the same retrieval/answer supervision and comparable tuning budget? These comparisons would distinguish reward-design benefits from benefits of jointly learning the trigger.

  3. Grounding: Does Case 3 omit another retrieved passage that actually supports the claim, or is the example mislabeled as successful grounding? Please clarify what “fallback synthesis” means in evidence construction and whether any generated, rather than source-extracted, snippets enter the evidence corpus. Can you report model performance on human-adjudicated claim–evidence pairs, including per-stance results and a SUPPORTS-only baseline under the same retrieval predictions?

6 comments
AuthorsOpenReview ↗

Response (1/n)

Thank you for the detailed review. We address the training comparisons, metric consistency, and grounding questions below.

Q1: Does retrieval shaping help when the control already rewards waiting and end-to-end success?

ProSearch retains its advantage over a control with the same SFT initialization and credit for both correct waiting and successful execution. The control retains the trigger reward and evidence-gated answer reward, while removing the retrieval reward.

RewardClaimVerif E2E-F1DSTC9 E2E-F1
Trigger + answer0.390.83
ProSearch: trigger + retrieval + answer0.440.84

Retrieval shaping yields a larger gain on ClaimVerif (+0.05) than on DSTC9 (+0.01), where the control already performs strongly.

Q2: Does matched-SFT Search-R1 receive the same answer supervision?

Splitting at </search> adapts Search-R1's official implementation for SFT warm-up without losing answer supervision. After evidence insertion, the reference answer remains the supervised target for the continuation.

Warm-up improves gated Search-R1, but ProSearch retains an E2E-F1 advantage of 0.10/0.09 on ClaimVerif/DSTC9 (Tab. 4).

MethodClaimVerif E2E-F1DSTC9 E2E-F1
Search-R1 w/ gate0.310.66
Search-R1 + matched SFT w/ gate0.340.75
ProSearch0.440.84

Q3: Are learning-rate and checkpoint-selection procedures comparable?

Both ProSearch and Search-R1 use the same validation-based checkpoint-selection procedure. ProSearch's learning rate is validation-selected, while Search-R1 uses its official published setting. We have clarified this distinction in Appendix A.

Q4: Does joint ProSearch outperform a separately gated ProSearch-trained generator?

The gated ProSearch generator recovers most of the gain, with a smaller further improvement from joint training. It receives SFT warm-up and retrieval/answer rewards, omits the trigger reward, and uses the same SFT-Llama gate as Search-R1.

Method (Llama-3.1-8B)ClaimVerif E2E-F1DSTC9 E2E-F1
Search-R1 w/ gate0.310.66
ProSearch generator w/ gate0.4400.81
ProSearch0.4440.84

Joint ProSearch adds 0.004/0.03 E2E-F1 while integrating intervention and execution in one policy. The full component breakdown appears in the revised manuscript (Tab. 19).

Q5: How is the DSTC11→DSTC9 result reconciled with Appendix E's metric bound?

All four metric columns come from the same DSTC11→DSTC9 predictions and checkpoint. The table contained transcription errors. Across all 4,181 DSTC9 test examples, these predictions contain 1,422 true positives, 5 false positives, 559 false negatives, and 2,195 true negatives. The model therefore triggers on 1,427 examples, and there are 1,981 gold-trigger examples. The corrected values are:

MetricOriginally reportedRecomputed from saved predictions
Trigger rate0.421,427 / 4,181 ≈ 0.34
Trigger F10.842 × 1,422 / (1,427 + 1,981) ≈ 0.83
Retrieval hit0.571,294 / 1,422 ≈ 0.91

These trigger and retrieval statistics use the full test set. No true-positive example lacks gold evidence (0 / 1,422).

Response estimates use 326 judged TPs. Weighting the hit/miss group means by their full group sizes gives estimated response-credit totals of 1,178.8717 overall and 1,112.84 after retrieval gating. Thus, Resp = 1,178.8717/1422 = 0.8290 and E2E-F1 = 2 x 1,112.84/(1,427+1,981)=0.6531, retaining the reported 0.83 and 0.65 and satisfying the corrected upper bound of 0.7594. Appendix E provides the detailed calculation.

Q6: Did the model retrieve evidence supporting the claim in Case 3?

This was a figure-preparation error. The model retrieved supporting evidence about Jack Smith’s January 6 investigation, but the original figure displayed the wrong passage. We have corrected Case 3 to show the supporting retrieved evidence.

AuthorsOpenReview ↗

Response (2/n)

Q7: Does “fallback synthesis” introduce generated evidence into the released corpus?

None of the 20,767 released passages are simulated. “Fallback synthesis” inaccurately described a separate simulation mode that was not used for this corpus. ClaimVerif uses fetched web pages, with an LLM prompted to extract verbatim or near-verbatim passages stored with their source URLs. Failed fetches and empty extractions are discarded. Appendix D now clarifies this procedure.

Q8: How does ProSearch perform on human-adjudicated claims beyond the SUPPORTS majority?

The human-adjudicated results show that ProSearch's verdict performance is not explained by always predicting SUPPORTS. We evaluated the 39 human-labeled claims in the 200-utterance test audit (App. D, Tab. 15): 24 SUPPORTS and 15 REFUTES. Humans adjudicated the reference claim–evidence pairs; the model comparison measures verdict agreement with those labels using each model's own retrieved passages. Always-SUPPORTS uses ProSearch's retrieved passages and deterministically outputs SUPPORTS for every evaluated claim.

SUPPORTS recallSUPPORTS precisionREFUTES recallREFUTES precision
Always SUPPORTS1.00 (24/24)0.62 (24/39)0.00 (0/15)—
Search-R1 with the SFT-Llama gate0.88 (21/24)0.62 (21/34)0.13 (2/15)0.67 (2/3)
Published ProSearch0.92 (22/24)0.88 (22/25)0.80 (12/15)0.86 (12/14)

Search-R1's two WAIT decisions count as misses in recall. ProSearch correctly identifies 12/15 REFUTES cases, compared with 2/15 for gated Search-R1 and 0/15 for always-SUPPORTS.

Dear Reviewer FbBn,

Thank you again for your time and thoughtful feedback on our submission. As the discussion period is nearing its close, we want to kindly follow up on our rebuttal posted earlier. Please let us know if our responses have adequately addressed your questions or if there are any remaining concerns we can clarify before the end of the discussion period. We truly appreciate your engagement and consideration.

Best regards,

Authors

Reviewer_dddjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies when an LLM should search in a streaming setting. The main idea is quite heuristic: add a WAIT/SEARCH decision, use SFT to avoid collapse, balance the training samples, and manually design separate rewards for trigger, retrieval, and answer quality, then train the whole system with GRPO. The experiments show that this recipe works better than several reactive or gated baselines, but the method itself is mainly based on reward and training design rather than a new RL mechanism.

Critical strengths
  1. The problem setting is useful. Instead of assuming every input needs retrieval, the paper explicitly considers when the model should do nothing and treats WAIT as part of the policy.
  2. The experiments are relatively complete and the overall recipe works consistently.
Critical weaknesses
  1. The method feels mostly heuristic. It combines a WAIT/SEARCH action, SFT warm start, balanced sampling, and manually designed rewards for trigger, retrieval, and answer quality. These choices are reasonable, but I do not see a clear new RL mechanism or learning principle beyond reward engineering.
  2. It is not fully clear where the improvement comes from. The full recipe performs well, but the paper does not completely separate the benefit of joint trigger learning from the benefit of stronger supervision and structured rewards.
Questions for authors

How much of the gain comes from learning the WAIT/SEARCH decision jointly, rather than from the stronger SFT initialization and structured reward design? The method relies on several hand-designed reward terms and weights. How sensitive are the results to these choices, and is there evidence that the method still works well without careful tuning of the reward coefficients?

3 comments

Thank you for the questions. We address the contribution, attribution, and reward sensitivity in turn.

Q1: What is the contribution beyond combining existing training components?

ProSearch brings the decision to intervene into the learned search policy, jointly optimizing silence, query formulation, and grounded answering. Search-R1 assumes an answer is required, while gated cascades train intervention separately.

This creates a training tension: correct silence earns reliable reward on the non-actionable majority, while successful intervention requires several initially unreliable steps. Rewarding silence alone does not resolve this tension. SFT initializes executable search trajectories, balanced sampling maintains exposure to actionable inputs, and structured rewards distinguish correct silence, retrieval progress, and answer quality.

This formulation and training framework is supported by ablations and stronger controls (Tables 2–3 and Q2–Q3 below), with evaluation across three tasks and two model families. ClaimVerif enables intervention evaluation by retaining non-actionable utterances alongside verifiable claims.

Q2: How much does joint trigger learning add beyond SFT and generator rewards?

The gated-generator comparison (Tab. 19) tests the added value of joint trigger learning; the ablations test SFT (Tab. 2) and retrieval shaping (Tab. 3). The gated generator receives SFT and retrieval/answer rewards but no trigger reward, and uses Search-R1's gate. The no-retrieval-reward control retains trigger and answer rewards.

MethodClaimVerif E2E-F1DSTC9 E2E-F1
ProSearch generator w/ gate0.4400.81
Without SFT initialization0.260.71
Without retrieval reward0.390.83
ProSearch0.4440.84

Removing SFT causes the largest drop among these controls; retrieval shaping contributes more on ClaimVerif. These ablations measure changes to the full recipe, not independent contributions that can be added together.

Most of the improvement over gated Search-R1 is associated with generator training. Joint ProSearch adds 0.004 on ClaimVerif and 0.03 on DSTC9 while integrating intervention and execution in one policy.

Q3: How sensitive is performance to the reward design and coefficient tuning?

Doubling the judge weight retains 0.42 E2E-F1, compared with 0.44 at the default. We examine sensitivity on ClaimVerif with Llama-3.1-8B, using defaults selected through early validation pilots (Appendix A).

SettingRetrieval reward (hit / miss)Judge weight w_jClaimVerif E2E-F1
Default1.0 / 0.50.50.44
No retrieval reward0.0 / 0.00.50.39
Equal hit/miss rewards0.5 / 0.50.50.40
No judge reward1.0 / 0.50.00.35
Higher judge weight1.0 / 0.51.00.42

This result supports tolerance to the tested coefficient change; removing reward terms or the hit/miss distinction causes larger drops.

Dear Reviewer dddj,

Thank you again for your time and thoughtful feedback on our submission. As the discussion period is nearing its close, we want to kindly follow up on our rebuttal posted earlier. Please let us know if our responses have adequately addressed your questions or if there are any remaining concerns we can clarify before the end of the discussion period. We truly appreciate your engagement and consideration.

Best regards,

Authors

AuthorsOpenReview ↗

Last-day check

Dear Reviewer dddj,

Thank you again for your time and constructive feedback on our submission. As the discussion period is coming to a close soon, we wanted to check if our responses have adequately addressed your concerns or if you have any remaining questions we can clarify before the deadline.

Best regards,

The Authors

Reviewer_33C2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ProSearch, a reinforcement-learning framework for training language models to decide when to search, what to retrieve, and how to generate grounded responses over streaming inputs. The framework treats WAIT and SEARCH as explicit actions and jointly optimizes trigger detection, query formulation, and response generation using GRPO. To address sparse rewards and the tendency of the policy to collapse into always waiting, the authors introduce a structured reward consisting of trigger, retrieval, and answer components, together with a synthetic SFT cold-start and class-balanced sampling. Experiments on ClaimVerif, DSTC9-T1, and DSTC11-T5 report improvements in end-to-end F1 and action latency over reactive and externally gated baselines across multiple model backbones.

Critical strengths

The paper addresses a practically relevant problem of proactive retrieval, where an agent must decide whether an input warrants any intervention rather than assuming that every input requires a response.

Critical weaknesses
  1. In Section 3, the authors restrict the action space to WAIT and SEARCH, where WAIT produces no response, while SEARCH necessarily involves retrieval followed by answer generation. Does this formulation overlook the possibility of generating a response without performing external retrieval? Authors should clearly distinguish between two different decisions in their problem formulation: whether to retrieve external knowledge and whether to generate a response.

  2. As shown in Table 2, ProSearch does not outperform the baselines with external trigger classifiers in terms of trigger F1 in most settings. Does this suggest that the improvements in end-to-end performance primarily come from better retrieval and response generation, rather than more accurate modeling of the core decision of "when to search"?

  3. In Equation (5), the answer reward is activated only when the retrieved passages contain the annotated gold evidence. Does this mean that even if the model retrieves alternative but equally valid evidence and generates a correct answer, it may still receive no end-to-end credit simply because the retrieved evidence does not match the dataset's annotated evidence?

Questions for authors

Refer to weaknesses

3 comments

Thank you for the clear questions.

Q1: Why couple responding with retrieval?

External evidence makes a proactive response checkable against a source, even when the model already knows the answer. Whether to respond and whether retrieval is necessary are distinct decisions. The latter depends on the model's internal knowledge, which varies across models and changes during training. Shared references can evaluate answer correctness, but fixed annotations cannot establish when each model needs retrieval.

Coupling intervention with retrieval therefore provides a common evidence requirement for evaluating when to intervene. Distinguishing appropriate ANSWER-only choices from necessary SEARCH choices would require a model-dependent evaluation of retrieval necessity. Section 3 now explains this rationale.

Q2: Do the gains come from better retrieval and generation rather than better triggering?

ProSearch improves complete interventions through complementary contributions from retrieval, answering, and joint decision learning. The following Llama-3.1-8B results combine the component ablations (Tab. 3) and gated-generator comparison (Tab. 19).

Method / settingClaimVerif (Trig / Hit / Resp / E2E)DSTC9 (Trig / Hit / Resp / E2E)
Search-R1 w/ gate0.70 / 0.53 / 0.83 / 0.310.97 / 0.81 / 0.83 / 0.66
ProSearch generator w/ gate0.70 / 0.74 / 0.81 / 0.4400.97 / 0.93 / 0.87 / 0.81
ProSearch w/o retrieval reward0.67 / 0.70 / 0.84 / 0.390.96 / 0.92 / 0.94 / 0.83
ProSearch w/o answer reward0.65 / 0.66 / 0.83 / 0.350.97 / 0.91 / 0.78 / 0.69
Joint ProSearch0.66 / 0.79 / 0.87 / 0.4440.96 / 0.92 / 0.94 / 0.84

Removing retrieval or answer reward lowers E2E-F1 on both datasets, supporting the role of each. To assess joint decision learning, the gated ProSearch generator retains SFT and retrieval/answer rewards, omits the trigger reward, and uses Search-R1's external gate. Joint ProSearch improves over this control by 0.004/0.03 E2E-F1 on ClaimVerif/DSTC9, even though the external gate achieves higher trigger F1. Thus, higher standalone trigger accuracy does not guarantee better complete interventions: the joint policy achieves the strongest end-to-end performance in these comparisons while integrating decision and execution.

Q3: Annotated-evidence gating and valid alternatives

A correct answer does not by itself establish that the retrieved passage supports it. An unannotated passage may be equally valid, but that support must be verified; topical or semantic similarity alone is insufficient. Relaxing the gate without such verification risks rewarding ungrounded responses. Accepting alternatives would require an additional support assessment, with its own judgment errors and computational cost.

We therefore retain annotated evidence as a fixed, reproducible scoring criterion. When gold evidence is defined, this can exclude valid alternatives from answer reward and E2E credit, although a gold-trigger search still receives the retrieval-miss reward. The reward definition and Appendix E now explain this rationale and scoring consequence.

Dear Reviewer 33C2,

Thank you again for your time and thoughtful feedback on our submission. As the discussion period is nearing its close, we want to kindly follow up on our rebuttal posted earlier. Please let us know if our responses have adequately addressed your questions or if there are any remaining concerns we can clarify before the end of the discussion period. We truly appreciate your engagement and consideration.

Best regards,

Authors

AuthorsOpenReview ↗

Last-day check

Dear Reviewer 33C2,

Thank you again for your time and constructive feedback on our submission. As the discussion period is coming to a close soon, we wanted to check if our responses have adequately addressed your concerns or if you have any remaining questions we can clarify before the deadline.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.