acceptodds
Under review as a conference paper at ICLR 2027

MedResearch: A Screening-Guided Agent for Medical Deep Research

Abstract

Medical deep research collects evidence that meets predefined eligibility criteria and synthesizes it into reports to support medical decision-making. A well-supported report, however, does not ensure comprehensive coverage of eligible evidence. Task-trained methods require additional supervision, while training-free methods reuse existing models but often use screening mainly to select sources rather than guide subsequent actions. Acting on screening judgments poses two challenges: exclusions may be incorrect, and the amount of undiscovered eligible evidence is unknown. We introduce MedResearch, a training-free agent built around Screen-to-Steer: screening judgments guide both evidence retention and subsequent agent actions. Its Protocol Reader reassesses first-pass exclusions with additional text from the same source; Eligibility Memory preserves the resulting judgments and search progress; and the Search Controller uses this state to direct further retrieval, trigger one-shot recovery, or stop. The retained evidence is then synthesized into a structured research report, with all model weights kept frozen throughout execution. Experiments on MetaSyn and CLEF show higher evidence-selection recall and F1 than the evaluated RAG and deep research baselines, together with strong report-level conclusion alignment. Ablations and evidence-flow analysis further highlight the complementary roles of reassessment, memory, and action control in collecting and retaining evidence for synthesis. Code is available in an anonymous repository.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Medical Deep Research, Evidence Discovery, LLM Agents, Systematic Review, Adaptive Retrieval, Evidence Screening

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_VYPCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a common problem in medical deep research: relevant papers can be found by the retriever but then incorrectly excluded during screening. MedResearch tries to address this with three components. The Protocol Reader revisits excluded papers using more abstract text, Eligibility Memory tracks screening decisions, and the Search Controller uses this information to guide further retrieval and stopping. The method does not require additional training. Results on MetaSyn and CLEF show better evidence-selection recall and F1 than the tested baselines. The paper also includes ablations and an analysis of how evidence is retained across the pipeline.

Critical strengths
  1. I think the paper identifies an important problem that is often overlooked in retrieval-based systems. Finding relevant papers is only part of the task; the system also needs to avoid discarding them during screening. The evidence-flow results illustrate this well. On MetaSyn with Qwen, MedResearch and MA-RAG have similar candidate recall (93.17% vs. 91.71%), but their final recall differs substantially (75.77% vs. 50.01%). This is probably the most convincing result in the paper, since it directly supports the motivation for improving the screening stage.

  2. The experimental evaluation is fairly thorough. The improvements in F1 are consistent across MetaSyn and CLEF, and the component ablations help identify which parts of the system are useful. I also like that the authors report retrieval coverage and evidence retention separately. Looking only at the final F1 scores would miss an important part of the story.

Critical weaknesses
  1. My main concern is that the experiments do not clearly separate the proposed mechanism from the benefits of additional context and inference. The reassessment module appears to contribute the largest improvement, but it also gives the model more abstract text and another LLM call. What happens if the initial screening simply uses 2,000 characters instead of 500? Or if the model screens the same 500 characters twice? Without these controls, I am not sure how much of the gain comes from the reassessment process itself. I also find it hard to tell how much the memory-guided retrieval contributes beyond the improved screening module.

  2. I am less convinced by the report-level evaluation. In CLEF, a positive label means that a paper is relevant at the abstract-screening stage, not necessarily that it should be included after full-text review. Yet the selected papers are used directly for report generation. One example that stood out to me is CD010680 in Figure 5: the system reports 24 included studies, while the original Cochrane review ultimately included only one. I think this distinction matters when interpreting the generated reports. The improvements in report-level conclusions are also less consistent than those in evidence selection.

Questions for authors
  1. Could you add two simple controls for the reassessment module: (a) single-pass screening with up to 2,000 abstract characters, and (b) repeating the 500-character screening without providing additional text? It would be helpful to compare both F1 and inference cost. This seems important for understanding where the improvement actually comes from.

  2. Are the candidate exposure, abstract lengths, and total LLM calls comparable across methods? I would also be interested in a fixed-retrieval baseline that uses the same Protocol Reader and a similar candidate budget. How much improvement would remain without memory-guided retrieval and search control?

  3. Could you clarify what an "included" study means in the CLEF report-generation experiments? For CD010680, are the 24 studies meant to be candidates for full-text assessment rather than finally eligible studies? If so, how is this distinction accounted for when evaluating the generated conclusions?

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer VYPC (Part 1/2: Weakness 1 & Questions 1-2)

We appreciate the reviewer's valuable time and insightful suggestions, which are important to our paper. We have uploaded a revised manuscript, where all of our modifications are highlighted in blue. Since Weaknesses 1–2 directly align with Questions 1–3, we address them jointly below in two parts (Part 1/2: Weakness 1 & Questions 1–2; Part 2/2: Weakness 2 & Question 3).

Weakness 1 (Part 1) & Question 1 — Controls for exclusion reassessment: 2,000-character and repeated 500-character screening

Weakness & Question. Exclusion reassessment adds up to 2,000 abstract characters and a second LLM call; the reviewer asks how it compares in F1 and inference cost against (a) single-pass 2,000-character screening and (b) repeating 500-character screening without additional text.

Response. We appreciate this core attribution question. Selective reassessment outperforms both controls on both benchmarks, indicating that its gains stem from combining focused exclusion verification with targeted context expansion rather than longer context or extra calls alone. We evaluated all four variants on the identical frozen 200-document initial pool with subsequent search disabled.

Screening VariantMetaSyn RecallMetaSyn F1CLEF RecallCLEF F1Tokens / Calls (MetaSyn | CLEF)
1-pass 500 chars60.98%61.08%34.05%39.09%39.5k / 8.0 | 47.9k / 8.0
(a) 1-pass 2,000 chars67.05%63.51%37.79%42.75%93.0k / 8.0 | 85.8k / 8.0
(b) 2-pass 500→500 chars68.56%63.16%38.82%42.22%72.3k / 15.6 | 85.4k / 14.6
Selective 500→2,000 (Ours)74.97%65.07%42.30%43.68%123.0k / 15.6 | 113.7k / 14.6

Repeating the original 500 characters ((b) vs. 1-pass 500, zero added text) accounts for 54%–58% of the recall gain (+7.58 and +4.77 points) and significantly improves F1 (+2.09 and +3.13 points, p < 0.05) by overturning conservative batch exclusions, while characters 500–2,000 contribute the remaining 42%–46%. Capping our Protocol Reader per review at (a)'s token budget (85.7k and 80.9k tokens, 12.0 and 11.2 calls) already raises recall by +6.90 and +2.20 points over (a) (73.95% recall and 64.87% F1 on MetaSyn), while completing remaining batches secures CLEF tail positives and beats equal-budget 500-character voting (60.09% and 37.00% F1). In the revised manuscript, we have included this controlled attribution analysis in Sec. 4.3 and App. E.

Weakness 1 (Part 2) & Question 2 — Cross-method comparability, fixed-retrieval baseline, and memory-guided search

Weakness & Question. The reviewer asks whether candidate exposure, abstract lengths, and total LLM calls are comparable across methods, and how much gain remains for a fixed-retrieval Protocol Reader baseline without memory-guided retrieval and search control.

Response. Thank you for raising this fairness and component-isolation question. Our exposure and token budgets closely match fixed-RAG baselines, indicating that our gains do not stem from larger exposure budgets, while Eligibility Memory and Search Controller prevent the precision loss of unsteered search beyond static retrieval (MetaSyn / CLEF below).

WorkflowExposed DocsScreen / Report CapCalls / ReviewNative TokensInc.F1
Fixed RAG (BM25 / Dense / MA)200.0 / 198.7≤2,000 / 1,2008.0 / 8.0 + 19.3M–9.9M / 2.5M–3.0M43.44%–51.01% / 27.71%–31.25%
Agents (GPT-R / OpenDR / ProtoMA)32.2–62.3 / 36.3–83.2Native / 1,200Multi-stepup to 51.7M / 14.0M25.91%–41.95% / 11.09%–18.54%
MedSR-Copilot (adapted)200.0 / 198.7Title+Abs / 1,200Multi-stage45.0M / 43.3M38.50% / 21.99%
Fixed 200 + Protocol Reader200.0 / 198.7500→2,000 / 1,20015.6 / 14.6 + 110.6M / 3.4M65.07% / 43.68%
MedResearch (Full)213.6 / 221.7500→2,000 / 1,20015.6 / 14.6 + 1–2 rds. + 111.0M / 4.1M65.45% / 44.05%

MedResearch also leads within-pool Cand.F1 across all three settings (67.64%, 69.82%, and 58.21%, Table D.3) using 3.4×–11× fewer tokens than OpenDR and MedSR-Copilot. Relative to Fixed 200 + Protocol Reader (65.07% and 43.68% F1), expanding search without memory steering and stopping (A only, Table E.1: 59.51% and 38.23% F1) degrades precision across five unsteered q0 rounds; coupling Memory and Controller (Full) preserves precision while adding statistically significant tail recall (75.77% and 42.91%, p < 0.05), with Memory yielding ~3× larger F1 gains alongside reassessment (+3.05 and +2.55) than alone (+1.00 and +0.94). In the revised manuscript, we have added this comparability ledger and fixed-retrieval baseline to Apps. D and E.

AuthorsOpenReview ↗

Author Response to Reviewer VYPC (Part 2/2: Weakness 2 & Question 3)

(Continued — Part 2/2: Weakness 2 & Question 3. In the revised manuscript, all of our modifications are highlighted in blue.)

Weakness 2 & Question 3 — CLEF abstract-stage candidates (CD010680) and report-level evaluation consistency

Weakness & Question. CLEF positive labels reflect abstract-screening relevance rather than final full-text inclusion (e.g., CD010680 reports 24 studies vs. 1 in Cochrane), and report-level gains are less uniform than selection gains; the reviewer asks what "included" means in CLEF report generation and how this distinction is accounted for when evaluating conclusions.

Response. We appreciate this important point regarding systematic-review stages. In CLEF, "included" records denote retained abstract-stage candidates for full-text assessment (such as the 24 records in CD010680) rather than finally eligible full-text studies, and MedResearch still achieves the highest recall on official full-text labels, indicating that screening retention strengthens downstream coverage alongside stage-calibrated synthesis. When full texts are paywalled in open corpora (a constraint shared across all eight methods), synthesizing abstract-screened candidates supports scoping prior to full-text retrieval; for instance, Schroder 2006 (PMID 16950430) in CD010680 met abstract criteria via duplex ultrasound confirmation, whereas Cochrane excluded its full text because target-subgroup data could not be isolated.

We account for this distinction in three ways: (1) all eight methods synthesize over identical abstract-only pools under the App. G.4 template, which mandates study-selection and quality-assessment limitation statements and prohibits inventing PRISMA full-text counts or pooled statistics; (2) evaluating retained sets against official CLEF content-level (full-text) labels shows that MedResearch achieves the highest full-text recall among all eight methods (57.33% vs. 44.45% for Dense-RAG, 40.25% for BM25-RAG, and 37.61% for MA-RAG); and (3) two blinded medical researchers across 40 paired reports explicitly graded evidence calibration alongside conclusion direction (86.25% agreement with Qwen, κ = 0.718; overall quality 3.92 vs. 3.77, p = 0.0164), complemented by GPT-6.1 across all 232 reports (+12.79 points on MetaSyn Dir.A). On MetaSyn-Qwen, removing App. B.6's 1,200-character synthesis truncation closes the Insights gap with unclipped Dense-RAG (33.02% vs. 32.09%, with Dir.A at 63.95% vs. 58.14%), while insight-extraction compression (App. C.4) and prohibiting invented pooling (App. G.3) bound automated report scores. In the revised manuscript, we have updated Fig. 5 and Sec. 4.5 to state "retained abstract-stage candidates," calibrated our report claims, and included the full-text and audit evaluations in Apps. D and F.

The authors have addressed my concerns well. I will maintain my weak accept rating.

Reviewer_TbnNOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MedResearch, an agent that collects studies meeting medical review criteria and synthesizes them into reports without additional training. It revisits initially excluded studies and uses screening results to guide subsequent searches. On MetaSyn and CLEF, it achieves higher evidence-selection recall and F1 than the compared baselines. Ablations show that removing exclusion reassessment causes the largest performance drop.

Critical strengths
  1. The paper addresses an important source of evidence loss: eligible studies that are retrieved but incorrectly excluded during screening. Revisiting these decisions and using screening feedback to guide further search is a practical approach to improving evidence collection without additional training.
  2. The evidence-selection results are consistent across the evaluated settings, with higher recall and F1 on both MetaSyn and CLEF. The retrieval–retention analysis helps explain where the gains come from, while component ablations show that all three components contribute to the final performance.
Critical weaknesses

The gains from exclusion reassessment are difficult to interpret. Reassessment both expands the abstract context from 500 to 2,000 characters and adds a second model call, while the ablation removes both. Without comparisons to single-pass screening with longer context or a second pass on unchanged text, it is unclear whether a simpler screening procedure could achieve similar gains.

Questions for authors
  1. How does exclusion reassessment compare with single-pass screening using 2,000 abstract characters, and with a second pass using the original 500 characters? These comparisons would help clarify whether reassessment offers benefits beyond additional context.
  2. The controller ablation removes both early stopping and recovery. What are their separate effects on recall, precision, and search cost? When the stopping rule is triggered, how much additional eligible evidence can be found by continuing the search?
1 comment
AuthorsOpenReview ↗

Author Response to Reviewer TbnN

We appreciate the reviewer's valuable time and insightful suggestions, which are important to our paper. We have uploaded a revised manuscript, where all of our modifications are highlighted in blue. Since Weakness 1 and Question 1 address the same attribution question, we respond to them jointly below, followed by Question 2.

Weakness 1 & Question 1 — Disentangling exclusion reassessment from 2,000-character context and a second call

Weakness & Question. Exclusion reassessment expands abstract context from 500 to 2,000 characters and adds a second call while the ablation removes both; the reviewer asks how it compares with single-pass 2,000-character screening and a second pass on the original 500 characters.

Response. We thank the reviewer for highlighting this key attribution question. Selective reassessment outperforms both controls even at matched budget, indicating that focused exclusion reassessment provides clear benefits beyond simply reading longer abstract context. We evaluated all four variants on the frozen 200-document initial pool (86 MetaSyn and 30 CLEF reviews, search disabled).

Screening VariantMetaSyn RecallMetaSyn F1CLEF RecallCLEF F1Tokens / Calls (MetaSyn | CLEF)
1-pass 500 chars60.98%61.08%34.05%39.09%39.5k / 8.0 | 47.9k / 8.0
1-pass 2,000 chars67.05%63.51%37.79%42.75%93.0k / 8.0 | 85.8k / 8.0
2-pass 500→500 chars68.56%63.16%38.82%42.22%72.3k / 15.6 | 85.4k / 14.6
Selective 500→2,000 (Ours)74.97%65.07%42.30%43.68%123.0k / 15.6 | 113.7k / 14.6

Decomposing our gain over 1-pass 500 (+13.99 and +8.25 recall; +3.99 and +4.59 F1 points on MetaSyn and CLEF), revisiting the unchanged 500 characters (2-pass 500→500) contributes 54%–58% of the recall gain (+7.58 and +4.77 points, p < 0.01) and lifts F1 (+2.09 and +3.13, p < 0.05) by overturning conservative 25-study batch exclusions; even on 2,000-character inputs, a second pass adds +7.11 and +4.15 recall (+1.42 CLEF F1, p < 0.05), while characters 500–2,000 contribute the remaining 42%–46% (+6.42 and +3.48 recall, p < 0.01). Capping our Reader per review at 1-pass 2,000's token budget (85.7k and 80.9k tokens, 0/116 overruns) already lifts recall by +6.90 and +2.20 points (73.95% recall and 64.87% F1 on MetaSyn), while full reassessment secures CLEF tail positives, saves 29.5% and 24.2% tokens vs. 2,000→2,000, and beats equal-budget 500-character voting (60.09% and 37.00% F1). In the revised manuscript, we have included this controlled decomposition and table in Sec. 4.3 and App. E.

Question 2 — Separate effects of early stopping and recovery, and undiscovered evidence after stopping

Question. The controller ablation removes both early stopping and recovery; the reviewer asks for their separate effects on recall, precision, and search cost, as well as how much eligible evidence remains if search continues after stopping.

Response. We appreciate this request for a finer controller breakdown. Early stopping cuts cost while one-shot recovery adds tail recall, indicating that the two controller actions play complementary efficiency and coverage roles with minimal post-stop evidence loss. In Table E.1, removing the Controller drops F1 by 1.91 and 1.22 points (w/o C: 63.54% and 42.83%); we isolate both actions along the identical five-round search trajectory below.

Search PolicyMetaSyn PrecisionMetaSyn RecallMetaSyn TokensCLEF PrecisionCLEF RecallCLEF Tokens
Unstopped 5-round search61.16%76.38%143.5k57.72%44.04%155.3k
Early stopping only62.45%75.44%128.0k58.27%42.85%135.0k
Early stop + Recovery (Full)62.53%75.77%128.3k58.26%42.91%135.3k

Early stopping reduces ordinary search from 5.0 rounds to 1.2–2.0 rounds, saving 10.8% and 13.1% Reader tokens and improving precision by +1.29 and +0.55 points while retaining 98.8% and 97.3% of 5-round recall. One-shot recovery adds fewer than 300 tokens per review (+296 and +258) and lifts MetaSyn F1 from 65.25% to 65.45% (+0.20 points, p < 0.05; 44.05% on CLEF). Continuing search through round 5 after zero-yield stopping retrieves zero additional reference studies in 79.1% (68/86) of MetaSyn reviews (0.38 retrieved gold and 0.21 retained TP per review overall); residual positives on broader CLEF diagnostic topics reflect isolated zero-yield rank gaps or lexically divergent deeper-rank studies where unstopped continuation lowers precision. In the revised manuscript, we have added this trajectory breakdown to App. E.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer XPUx (Part 1/2: Weakness 1 & Question 1)

We appreciate the reviewer's valuable time and constructive feedback, which are important to our paper. We have uploaded a revised manuscript, where all of our modifications are highlighted in blue. Since Weaknesses 1–2 directly align with Questions 1–2, we address them jointly below, followed by Question 3 (Part 1/2: Weakness 1 & Question 1; Part 2/2: Weakness 2 & Questions 2–3).

Weakness 1 & Question 1 — Matched screening controls, gain split, per-review costs, and Screen-to-Steer coupling

Weakness & Question. Reassessment adds 2,000-character context and a second call (even when no extra text exists) while query updates follow relevance feedback; the reviewer asks for matched 2,000-character and equal-budget controls, a split between extra-text and repeated-decision effects, per-review costs, and what Screen-to-Steer coupling adds.

Response. We thank the reviewer for pinpointing these attribution and coupling questions. Selective reassessment beats all matched controls below 2,000-character cost, indicating that second-pass verification and coupling reassessed labels with memory control each add distinct value beyond context length or standalone Rocchio updates. We evaluated all screening variants on identical frozen 200-document pools (86 MetaSyn and 30 CLEF reviews, search disabled).

Screening VariantMetaSyn RecallMetaSyn F1CLEF RecallCLEF F1Tokens / Calls (MetaSyn | CLEF)
1-pass 500 chars60.98%61.08%34.05%39.09%39.5k / 8.0 | 47.9k / 8.0
1-pass 2,000 chars67.05%63.51%37.79%42.75%93.0k / 8.0 | 85.8k / 8.0
2-pass 500→500 chars68.56%63.16%38.82%42.22%72.3k / 15.6 | 85.4k / 14.6
Selective 500→2,000 (Ours)74.97%65.07%42.30%43.68%123.0k / 15.6 | 113.7k / 14.6

Capping our Protocol Reader per review at 1-pass 2,000's token budget (85.7k and 80.9k tokens, 0/116 overruns) already raises recall by +6.90 and +2.20 points (p < 0.01 on MetaSyn: 73.95% recall and 64.87% F1), while full reassessment secures CLEF tail positives (42.30% recall) and outperforms equal-budget 500-character majority voting (60.09% and 37.00% F1, p < 0.01). Revisiting the unchanged 500 characters (500→500) contributes 54%–58% of the recall gain (+7.58 and +4.77 recall, p < 0.01; +2.09 and +3.13 F1, p < 0.05), explaining why App. B.4 still reassesses when Additional evidence is empty on short abstracts, whereas characters 500–2,000 contribute the remaining 42%–46% (+6.42 and +3.48 recall, p < 0.01).

Rather than claiming the Rocchio rule (Sec. 3.3) as new, Screen-to-Steer couples reassessed judgments with persistent memory and yield-gated stopping: in Table E.1, Memory B without reassessment (B only vs. Base) adds only +1.00 and +0.94 F1 due to noisy exclusions, whereas coupled with reassessment (Full vs. A+C) it adds +3.05 and +2.55 F1 (~3× larger), and Memory plus Controller add +5.94 and +5.82 F1 over unsteered search (Full 65.45% and 44.05% vs. A only 59.51% and 38.23%). Per review, Full uses 128.3k and 135.3k native tokens (138.7k and 161.0k with shared report), comparable to Dense-RAG (114.6k and 111.7k) and 3.4×–11× below OpenDR and MedSR-Copilot. In the revised manuscript, we have included these matched controls, coupling results, and per-review cost ledgers in Sec. 4.3 and Apps. D–E.

AuthorsOpenReview ↗

Author Response to Reviewer XPUx (Part 2/2: Weakness 2 & Questions 2-3)

(Continued — Part 2/2: Weakness 2 & Questions 2–3. In the revised manuscript, all of our modifications are highlighted in blue.)

Weakness 2 & Question 2 — Report-level claims, paired CIs, blinded evaluator audits, and CLEF stages

Weakness & Question. Report-level gains are less uniform than selection gains (MetaSyn-Qwen Insights 30.70% vs. 34.65%; GPT Dir.A 47.67%), rely on Qwen without paired CIs or expert audits, and compare CLEF abstract selections against completed reviews; the reviewer requests paired CIs and a blinded expert audit assessing Qwen agreement and clinically relevant conclusion changes.

Response. We appreciate this rigorous critique and agree that our strongest, most uniform gains lie in evidence collection. All 21/21 evidence-F1 CIs exceed zero and audits favor MedResearch, indicating that improved retention translates into calibrated, clinically relevant report gains once synthesis and evaluation ceilings are accounted for. Across Tables 1–2 (50k paired bootstrap, 95% CIs), all 21/21 baseline ΔInc.F1 intervals exceed zero (Holm p ≤ 0.00092), including +14.43 [9.46, 19.55] vs. Dense-RAG (MetaSyn-Qwen), +9.55 [4.32, 14.69] vs. GPT-R (MetaSyn-GPT), and +12.80 [5.84, 20.32] vs. Dense-RAG (CLEF). Paired Dir.A gains vs. Dense-RAG are +11.63 on MetaSyn-Qwen (+12.79 [3.49, 22.09] under GPT-6.1 across all 86 reviews) and +10.00 on CLEF (+11.76 under GPT-6.1), with +2.32 on MetaSyn-GPT (Table 1b, where MedResearch still leads all eight methods in both Dir.A at 47.67% and Insights at 28.24%, as forbidding invented pooled statistics causes GPT-5.6 to hedge qualitative summaries into Mixed). Removing App. B.6's 1,200-character synthesis clip on MetaSyn-Qwen closes the Insights gap with unclipped Dense-RAG (33.02% vs. 32.09%; Dir.A 63.95% vs. 58.14%), with remaining ceilings set by insight-extraction compression (App. C.4) and prohibiting invented pooling (App. G.3).

To audit report evaluation, two blinded medical researchers graded 40 paired reports (20 reviews vs. Dense-RAG), reaching 87.50% inter-expert direction agreement (κ = 0.756) and 86.25% agreement with Qwen (κ = 0.718; 91.43% on unanimous cases, alongside GPT-6.1's 93.75%–100% agreement on 32 audited reports). Experts significantly preferred MedResearch overall (3.92 vs. 3.77, Δ +0.15 [0.05, 0.25], p = 0.0164; 11 wins, 5 ties, and 4 losses), noting that retained trials resolved missed counter-evidence, shifting overstated Positive conclusions to calibrated Mixed and improving clinical relevance (+0.38, p = 0.0198) and reference alignment (+0.20). On CLEF, abstract synthesis under paywalls (CD010680: 24 retained candidates) and official content-level evaluation (#1 full-text recall across eight methods: 57.33% vs. 44.45% Dense-RAG) provide complementary views. In the revised manuscript, we have calibrated our report claims in the Abstract and Sec. 6, clarified Fig. 5, and added all paired CIs and blinded audits to Sec. 4.2, Tables 1–2, and Apps. D and F.

Question 3 — Zero-yield stopping frequency, undiscovered evidence, and recall–cost trade-off

Question. The reviewer asks how often the controller stops after its first zero-yield batch, what reference evidence remains undiscovered, and how stopping compares in recall versus cost against continuing the search.

Response. Thank you for requesting this trajectory analysis. Early stopping retains >97% of five-round recall at lower cost, indicating that the stopping policy preserves our coverage advantage without unproductive deeper-rank expansion. Specifically, stopping retains 98.8% and 97.3% recall while saving 10.8% and 13.1% Reader tokens across all 86 MetaSyn and 30 CLEF reviews below.

Search PolicyMetaSyn PrecisionMetaSyn RecallMetaSyn TokensCLEF PrecisionCLEF RecallCLEF Tokens
Unstopped 5-round search61.16%76.38%143.5k57.72%44.04%155.3k
Early stopping only62.45%75.44%128.0k58.27%42.85%135.0k
Early stop + Recovery (Full)62.53%75.77%128.3k58.26%42.91%135.3k

First-batch zero yield (Y1 = 0) stops 74/86 (86.0%) MetaSyn and 18/30 (60.0%) CLEF runs once top-200 screening captures surface-matched positives. Continuing through round 5 retrieves zero additional gold studies in 79.1% (68/86) of MetaSyn reviews (0.38 retrieved gold and 0.21 retained TP per review overall); residual CLEF positives stem from isolated zero-yield rank gaps or deeper-rank studies where unstopped continuation lowers precision (+1.29 and +0.55 precision advantage for stopping), avoiding Table E.1's w/o C drop (−1.91 and −1.22 F1). In the revised manuscript, we have included this recall–cost analysis in App. E.

AuthorsOpenReview ↗

Gentle Reminder: Follow-up on Our Rebuttal

Dear Reviewer,

We hope this message finds you well. Thank you once again for your time and for the insightful comments on our submission. In our rebuttal, we have carefully addressed each of your concerns with additional experiments and detailed clarifications, and we hope our response helps resolve the issues you raised.

As the author–reviewer discussion period is drawing to a close, we would be truly grateful if you could take a moment to look over our response and let us know whether any concerns remain. We would be more than happy to provide further clarification if needed.

Thank you very much for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.