- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper addresses a growing concern in medical AI research: as more clinically oriented papers are submitted to general machine learning venues, their clinical aspects, especially safety, are often not explicitly examined in peer review. The authors first motivate this with a diagnostic analysis of existing reviews, showing that clinical dimensions are discussed far less often than technical ones and are not simply captured by overall paper quality. To fill this gap, they propose EGCR, an evidence-grounded review system built on a simple intuition. Instead of critiquing a paper as a whole, it breaks the paper into individual clinical claims and checks each one against established clinical standards and guidelines that were available at submission time. The system also explicitly separates weak evidence from missing information and keeps a traceable record of how each finding was reached. Evaluated on papers from ICLR and Nature-series journals, EGCR is reported to preserve its evidence trail and to surface clinical dimensions not raised in the held-out human reviews.
Critical strengths
-
The paper targets a timely problem. As more clinical MedAI work enters general ML venues, clinical aspects such as safety are rarely made explicit in written reviews. Reviewing at the level of individual clinical claims is a sensible way to address this.
-
The design separates retrieval from applicability, and missing evidence from weak evidence. It also keeps each finding traceable to its source, which makes the system's judgments easier to inspect than free-form LLM reviews.
-
The evaluation is carefully scoped. Human reviews are held out during generation, and the authors do not overclaim the comparison as evidence of reviewer omissions.
Critical weaknesses
-
The technical contribution is limited. EGCR is mainly a prompt-based pipeline of claim extraction, retrieval, and LLM judgment on a single off-the-shelf model, so the novelty lies more in workflow design than in method.
-
The main complementarity result is largely self-defined. EGCR decides for itself which dimensions are assessable, and union coverage cannot fall below human coverage by construction, so the reported gain says little about whether the extra judgments are correct or useful.
-
Baselines and ablations are missing. The comparison with direct LLM review covers only two hand-picked cases, and it is unclear whether a simpler LLM reviewer with the same rubric would reach similar coverage, or what retrieval and applicability assessment actually contribute.
-
There is no validation by clinicians or experienced reviewers. The traceability audit mostly confirms properties guaranteed by the design, such as recoverable quotes and stored provenance, rather than the quality of the findings.
Questions for authors
-
Who performed the scoring in the 100-paper diagnostic audit? If it was an LLM, how should the PCA result be interpreted as a property of the papers rather than of the scoring model?
-
Could you report a quantitative comparison with direct LLM review on the full corpus? In particular, would a direct reviewer using the same rubric reach a similar union coverage?
-
Has any clinician or experienced reviewer checked a sample of the assessable-but-unasked findings? It would help to know what fraction are correct and genuinely useful to a reviewer.
-
What do retrieval and applicability assessment contribute on their own? An ablation removing each component would clarify which parts of the pipeline drive the reported results.
-
The 7% safety coverage in the nine-dimensional analysis appears to be computed per review, while the 73.38% human coverage in the seven-dimensional space is computed per paper and counts only concerns and requests. Could you clarify how these two analyses relate, and which dimensions account for most of the human coverage?