- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ProSearch trains a single language model to decide whether an input segment warrants external retrieval and, when appropriate, generate a query and a grounded response. The approach combines a synthetic supervised-finetuning initialization with GRPO using an additive reward for trigger correctness, retrieval quality, and answer quality. Retrieved passages are excluded from policy-gradient and KL calculations.
The paper evaluates the approach on a newly constructed, silver-labeled podcast claim-verification dataset and two knowledge-grounded dialogue benchmarks. Experiments with 3B and 8B Llama models, supplemented by Ministral-8B results, report substantial end-to-end improvements over reactive generators and externally gated cascades. Ablations investigate initialization, reward components, retrieval settings, and transfer. Despite the streaming motivation, the evaluated setting consists of independently scored, pre-segmented text inputs with supplied context and frozen retrieval corpora.
Critical strengths
-
A useful problem formulation with a practical integrated solution. Explicitly learning when to abstain is valuable for systems that monitor inputs without requiring a response to every segment. ProSearch combines this decision with query generation and answering in one policy. The paper clearly explains that its reward decomposition supplies denser rollout-level rankings rather than independent token-level credit, and the implementation addresses practical issues such as retrieved-token masking and cold-start collapse.
-
Substantial reported improvements across several settings. The full recipe improves end-to-end F1 over both SFT-only models and the evaluated gated Search-R1 cascades across three tasks and two primary model sizes. Additional Ministral experiments and independent-judge regrading provide useful evidence beyond a single backbone or evaluator. The ClaimVerif dataset also extends evaluation beyond preselected claims by retaining non-actionable utterances, with episode-disjoint splits and two label-quality audits.
Critical weaknesses
-
The comparisons do not adequately isolate joint trigger learning or structured reward decomposition. The outcome-only control assigns no terminal credit to correct waiting, making its failure to learn abstention insufficient evidence that intermediate reward shaping is necessary. A stronger comparison would use the same complete SFT initialization and reward both correct waiting and successful end-to-end execution. Furthermore, the “matched-SFT” Search-R1 control truncates trajectories at </search>, so it does not receive the same answer-generation supervision as ProSearch. Appendix A also states that ProSearch receives validation-based learning-rate/checkpoint selection that the cascades do not. Finally, the paper does not compare against a separately gated generator trained with ProSearch’s retrieval and answer rewards. This matters because the external gates already achieve near-perfect trigger F1 on the dialogue tasks; much of the observed advantage could arise from improved generator training rather than joint optimization of the trigger.
-
The evaluation contains an apparent numerical inconsistency and does not fully establish grounded correctness. Under Appendix E’s definition, when all true-positive rows have gold evidence, E2E-F1 cannot exceed trigger F1 multiplied by retrieval hit rate, because response credit is bounded by one. However, Table 18 reports trigger F1 = 0.84, retrieval hit = 0.57, and E2E-F1 = 0.65 for DSTC11→DSTC9; the corresponding upper bound is approximately 0.48. This requires reconciliation, including any missing-gold exceptions. Separately, Case 3 is presented as successful grounding of a claim about Jack Smith’s January 6 investigation, but the displayed evidence concerns Robert Mueller’s 2017 Russia investigation. Gold-document retrieval plus verdict agreement does not itself establish evidential support. These concerns are consequential given the silver ClaimVerif test labels, its 82.1% SUPPORTS prior, and the absence of reported multi-seed uncertainty.
Questions for authors
-
Metric consistency: Please reconcile the DSTC11→DSTC9 row in Table 18 with Appendix E’s E2E definition. How many true-positive rows lack gold evidence, and are the trigger, retrieval, response, and E2E columns computed from identical predictions and checkpoints? Providing the underlying counts and summed response credits would resolve this issue.
-
Central training claim: Does ProSearch retain its advantage over same-initialization GRPO using a terminal reward that credits both correct waiting and complete trigger–retrieve–answer success? Does it outperform an external gate combined with a generator trained using the same retrieval/answer supervision and comparable tuning budget? These comparisons would distinguish reward-design benefits from benefits of jointly learning the trigger.
-
Grounding: Does Case 3 omit another retrieved passage that actually supports the claim, or is the example mislabeled as successful grounding? Please clarify what “fallback synthesis” means in evidence construction and whether any generated, rather than source-extracted, snippets enter the evidence corpus. Can you report model performance on human-adjudicated claim–evidence pairs, including per-stance results and a SUPPORTS-only baseline under the same retrieval predictions?
6 comments
Response (2/n)
Q7: Does “fallback synthesis” introduce generated evidence into the released corpus?
None of the 20,767 released passages are simulated. “Fallback synthesis” inaccurately described a separate simulation mode that was not used for this corpus. ClaimVerif uses fetched web pages, with an LLM prompted to extract verbatim or near-verbatim passages stored with their source URLs. Failed fetches and empty extractions are discarded. Appendix D now clarifies this procedure.
Q8: How does ProSearch perform on human-adjudicated claims beyond the SUPPORTS majority?
The human-adjudicated results show that ProSearch's verdict performance is not explained by always predicting SUPPORTS. We evaluated the 39 human-labeled claims in the 200-utterance test audit (App. D, Tab. 15): 24 SUPPORTS and 15 REFUTES. Humans adjudicated the reference claim–evidence pairs; the model comparison measures verdict agreement with those labels using each model's own retrieved passages. Always-SUPPORTS uses ProSearch's retrieved passages and deterministically outputs SUPPORTS for every evaluated claim.
| SUPPORTS recall | SUPPORTS precision | REFUTES recall | REFUTES precision | |
|---|---|---|---|---|
| Always SUPPORTS | 1.00 (24/24) | 0.62 (24/39) | 0.00 (0/15) | — |
| Search-R1 with the SFT-Llama gate | 0.88 (21/24) | 0.62 (21/34) | 0.13 (2/15) | 0.67 (2/3) |
| Published ProSearch | 0.92 (22/24) | 0.88 (22/25) | 0.80 (12/15) | 0.86 (12/14) |
Search-R1's two WAIT decisions count as misses in recall. ProSearch correctly identifies 12/15 REFUTES cases, compared with 2/15 for gated Search-R1 and 0/15 for always-SUPPORTS.
Follow-up
Dear Reviewer FbBn,
Thank you again for your time and thoughtful feedback on our submission. As the discussion period is nearing its close, we want to kindly follow up on our rebuttal posted earlier. Please let us know if our responses have adequately addressed your questions or if there are any remaining concerns we can clarify before the end of the discussion period. We truly appreciate your engagement and consideration.
Best regards,
Authors
Response (1/n)
Thank you for the detailed review. We address the training comparisons, metric consistency, and grounding questions below.
Q1: Does retrieval shaping help when the control already rewards waiting and end-to-end success?
ProSearch retains its advantage over a control with the same SFT initialization and credit for both correct waiting and successful execution. The control retains the trigger reward and evidence-gated answer reward, while removing the retrieval reward.
Retrieval shaping yields a larger gain on ClaimVerif (+0.05) than on DSTC9 (+0.01), where the control already performs strongly.
Q2: Does matched-SFT Search-R1 receive the same answer supervision?
Splitting at
</search>adapts Search-R1's official implementation for SFT warm-up without losing answer supervision. After evidence insertion, the reference answer remains the supervised target for the continuation.Warm-up improves gated Search-R1, but ProSearch retains an E2E-F1 advantage of 0.10/0.09 on ClaimVerif/DSTC9 (Tab. 4).
Q3: Are learning-rate and checkpoint-selection procedures comparable?
Both ProSearch and Search-R1 use the same validation-based checkpoint-selection procedure. ProSearch's learning rate is validation-selected, while Search-R1 uses its official published setting. We have clarified this distinction in Appendix A.
Q4: Does joint ProSearch outperform a separately gated ProSearch-trained generator?
The gated ProSearch generator recovers most of the gain, with a smaller further improvement from joint training. It receives SFT warm-up and retrieval/answer rewards, omits the trigger reward, and uses the same SFT-Llama gate as Search-R1.
Joint ProSearch adds 0.004/0.03 E2E-F1 while integrating intervention and execution in one policy. The full component breakdown appears in the revised manuscript (Tab. 19).
Q5: How is the DSTC11→DSTC9 result reconciled with Appendix E's metric bound?
All four metric columns come from the same DSTC11→DSTC9 predictions and checkpoint. The table contained transcription errors. Across all 4,181 DSTC9 test examples, these predictions contain 1,422 true positives, 5 false positives, 559 false negatives, and 2,195 true negatives. The model therefore triggers on 1,427 examples, and there are 1,981 gold-trigger examples. The corrected values are:
These trigger and retrieval statistics use the full test set. No true-positive example lacks gold evidence (0 / 1,422).
Response estimates use 326 judged TPs. Weighting the hit/miss group means by their full group sizes gives estimated response-credit totals of 1,178.8717 overall and 1,112.84 after retrieval gating. Thus, Resp = 1,178.8717/1422 = 0.8290 and E2E-F1 = 2 x 1,112.84/(1,427+1,981)=0.6531, retaining the reported 0.83 and 0.65 and satisfying the corrected upper bound of 0.7594. Appendix E provides the detailed calculation.
Q6: Did the model retrieve evidence supporting the claim in Case 3?
This was a figure-preparation error. The model retrieved supporting evidence about Jack Smith’s January 6 investigation, but the original figure displayed the wrong passage. We have corrected Case 3 to show the supporting retrieved evidence.