- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies a common problem in medical deep research: relevant papers can be found by the retriever but then incorrectly excluded during screening. MedResearch tries to address this with three components. The Protocol Reader revisits excluded papers using more abstract text, Eligibility Memory tracks screening decisions, and the Search Controller uses this information to guide further retrieval and stopping. The method does not require additional training. Results on MetaSyn and CLEF show better evidence-selection recall and F1 than the tested baselines. The paper also includes ablations and an analysis of how evidence is retained across the pipeline.
Critical strengths
-
I think the paper identifies an important problem that is often overlooked in retrieval-based systems. Finding relevant papers is only part of the task; the system also needs to avoid discarding them during screening. The evidence-flow results illustrate this well. On MetaSyn with Qwen, MedResearch and MA-RAG have similar candidate recall (93.17% vs. 91.71%), but their final recall differs substantially (75.77% vs. 50.01%). This is probably the most convincing result in the paper, since it directly supports the motivation for improving the screening stage.
-
The experimental evaluation is fairly thorough. The improvements in F1 are consistent across MetaSyn and CLEF, and the component ablations help identify which parts of the system are useful. I also like that the authors report retrieval coverage and evidence retention separately. Looking only at the final F1 scores would miss an important part of the story.
Critical weaknesses
-
My main concern is that the experiments do not clearly separate the proposed mechanism from the benefits of additional context and inference. The reassessment module appears to contribute the largest improvement, but it also gives the model more abstract text and another LLM call. What happens if the initial screening simply uses 2,000 characters instead of 500? Or if the model screens the same 500 characters twice? Without these controls, I am not sure how much of the gain comes from the reassessment process itself. I also find it hard to tell how much the memory-guided retrieval contributes beyond the improved screening module.
-
I am less convinced by the report-level evaluation. In CLEF, a positive label means that a paper is relevant at the abstract-screening stage, not necessarily that it should be included after full-text review. Yet the selected papers are used directly for report generation. One example that stood out to me is CD010680 in Figure 5: the system reports 24 included studies, while the original Cochrane review ultimately included only one. I think this distinction matters when interpreting the generated reports. The improvements in report-level conclusions are also less consistent than those in evidence selection.
Questions for authors
-
Could you add two simple controls for the reassessment module: (a) single-pass screening with up to 2,000 abstract characters, and (b) repeating the 500-character screening without providing additional text? It would be helpful to compare both F1 and inference cost. This seems important for understanding where the improvement actually comes from.
-
Are the candidate exposure, abstract lengths, and total LLM calls comparable across methods? I would also be interested in a fixed-retrieval baseline that uses the same Protocol Reader and a similar candidate budget. How much improvement would remain without memory-guided retrieval and search control?
-
Could you clarify what an "included" study means in the CLEF report-generation experiments? For CD010680, are the 24 studies meant to be candidates for full-text assessment rather than finally eligible studies? If so, how is this distinction accounted for when evaluating the generated conclusions?
3 comments
Author Response to Reviewer VYPC (Part 2/2: Weakness 2 & Question 3)
(Continued — Part 2/2: Weakness 2 & Question 3. In the revised manuscript, all of our modifications are highlighted in blue.)
Weakness 2 & Question 3 — CLEF abstract-stage candidates (CD010680) and report-level evaluation consistency
Weakness & Question. CLEF positive labels reflect abstract-screening relevance rather than final full-text inclusion (e.g., CD010680 reports 24 studies vs. 1 in Cochrane), and report-level gains are less uniform than selection gains; the reviewer asks what "included" means in CLEF report generation and how this distinction is accounted for when evaluating conclusions.
Response. We appreciate this important point regarding systematic-review stages. In CLEF, "included" records denote retained abstract-stage candidates for full-text assessment (such as the 24 records in CD010680) rather than finally eligible full-text studies, and MedResearch still achieves the highest recall on official full-text labels, indicating that screening retention strengthens downstream coverage alongside stage-calibrated synthesis. When full texts are paywalled in open corpora (a constraint shared across all eight methods), synthesizing abstract-screened candidates supports scoping prior to full-text retrieval; for instance, Schroder 2006 (PMID 16950430) in CD010680 met abstract criteria via duplex ultrasound confirmation, whereas Cochrane excluded its full text because target-subgroup data could not be isolated.
We account for this distinction in three ways: (1) all eight methods synthesize over identical abstract-only pools under the App. G.4 template, which mandates study-selection and quality-assessment limitation statements and prohibits inventing PRISMA full-text counts or pooled statistics; (2) evaluating retained sets against official CLEF content-level (full-text) labels shows that MedResearch achieves the highest full-text recall among all eight methods (57.33% vs. 44.45% for Dense-RAG, 40.25% for BM25-RAG, and 37.61% for MA-RAG); and (3) two blinded medical researchers across 40 paired reports explicitly graded evidence calibration alongside conclusion direction (86.25% agreement with Qwen, κ = 0.718; overall quality 3.92 vs. 3.77, p = 0.0164), complemented by GPT-6.1 across all 232 reports (+12.79 points on MetaSyn Dir.A). On MetaSyn-Qwen, removing App. B.6's 1,200-character synthesis truncation closes the Insights gap with unclipped Dense-RAG (33.02% vs. 32.09%, with Dir.A at 63.95% vs. 58.14%), while insight-extraction compression (App. C.4) and prohibiting invented pooling (App. G.3) bound automated report scores. In the revised manuscript, we have updated Fig. 5 and Sec. 4.5 to state "retained abstract-stage candidates," calibrated our report claims, and included the full-text and audit evaluations in Apps. D and F.
The authors have addressed my concerns well. I will maintain my weak accept rating.
Author Response to Reviewer VYPC (Part 1/2: Weakness 1 & Questions 1-2)
We appreciate the reviewer's valuable time and insightful suggestions, which are important to our paper. We have uploaded a revised manuscript, where all of our modifications are highlighted in blue. Since Weaknesses 1–2 directly align with Questions 1–3, we address them jointly below in two parts (Part 1/2: Weakness 1 & Questions 1–2; Part 2/2: Weakness 2 & Question 3).
Response. We appreciate this core attribution question. Selective reassessment outperforms both controls on both benchmarks, indicating that its gains stem from combining focused exclusion verification with targeted context expansion rather than longer context or extra calls alone. We evaluated all four variants on the identical frozen 200-document initial pool with subsequent search disabled.
Repeating the original 500 characters ((b) vs. 1-pass 500, zero added text) accounts for 54%–58% of the recall gain (+7.58 and +4.77 points) and significantly improves F1 (+2.09 and +3.13 points, p < 0.05) by overturning conservative batch exclusions, while characters 500–2,000 contribute the remaining 42%–46%. Capping our Protocol Reader per review at (a)'s token budget (85.7k and 80.9k tokens, 12.0 and 11.2 calls) already raises recall by +6.90 and +2.20 points over (a) (73.95% recall and 64.87% F1 on MetaSyn), while completing remaining batches secures CLEF tail positives and beats equal-budget 500-character voting (60.09% and 37.00% F1). In the revised manuscript, we have included this controlled attribution analysis in Sec. 4.3 and App. E.
Response. Thank you for raising this fairness and component-isolation question. Our exposure and token budgets closely match fixed-RAG baselines, indicating that our gains do not stem from larger exposure budgets, while Eligibility Memory and Search Controller prevent the precision loss of unsteered search beyond static retrieval (MetaSyn / CLEF below).
MedResearch also leads within-pool Cand.F1 across all three settings (67.64%, 69.82%, and 58.21%, Table D.3) using 3.4×–11× fewer tokens than OpenDR and MedSR-Copilot. Relative to Fixed 200 + Protocol Reader (65.07% and 43.68% F1), expanding search without memory steering and stopping (A only, Table E.1: 59.51% and 38.23% F1) degrades precision across five unsteered q0 rounds; coupling Memory and Controller (Full) preserves precision while adding statistically significant tail recall (75.77% and 42.91%, p < 0.05), with Memory yielding ~3× larger F1 gains alongside reassessment (+3.05 and +2.55) than alone (+1.00 and +0.94). In the revised manuscript, we have added this comparability ledger and fixed-retrieval baseline to Apps. D and E.