acceptodds
Withdrawn from ICLR 2027

ASSESSABLE BUT UNASKED: EVIDENCE-GROUNDED CLINICAL REVIEW FOR MEDICAL AI PAPERS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The clinical dimensions that a medical AI (MedAI) paper makes assessable do not fully overlap with those raised as substantive concerns or requests in its peer review. A nine-dimensional diagnostic analysis shows that clinical factors, espe- cially clinical safety, are less frequently made explicit in written review and are not fully represented by a single overall-quality axis. We introduce Evidence-Grounded Clinical Review (EGCR), a claim-level review system that converts clinical claims into evidence questions, retrieves submission- time external evidence, assesses source applicability, and aligns external require- ments with manuscript evidence to produce structured, inspectable findings. We evaluate EGCR on 168 MedAI papers from ICLR and Nature-series journals. For 107 papers, EGCR identifies at least one clinical dimension that it marks as assessable but that is not raised as a substantive concern or request in the held-out human review. Human-review dimension coverage is 73.38%, while the union coverage of human review and EGCR is 89.80%. A traceability audit shows 97.1% manuscript-evidence recoverability, complete preservation of external provenance chains, and 97.5% submission-time source availability. Paired case analysis against direct review further illustrates how evi- dence grounding can produce more specific validation questions and better align evidence requests with the paper’s stated claim scope. EGCR is not an accept/reject predictor; it provides an inspectable clinical evidence layer that makes the basis of review findings easier to trace and examine.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Medical AI, LLM-assisted peer review, Retrieval-augmented generation, Agent

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_M56HOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses a growing concern in medical AI research: as more clinically oriented papers are submitted to general machine learning venues, their clinical aspects, especially safety, are often not explicitly examined in peer review. The authors first motivate this with a diagnostic analysis of existing reviews, showing that clinical dimensions are discussed far less often than technical ones and are not simply captured by overall paper quality. To fill this gap, they propose EGCR, an evidence-grounded review system built on a simple intuition. Instead of critiquing a paper as a whole, it breaks the paper into individual clinical claims and checks each one against established clinical standards and guidelines that were available at submission time. The system also explicitly separates weak evidence from missing information and keeps a traceable record of how each finding was reached. Evaluated on papers from ICLR and Nature-series journals, EGCR is reported to preserve its evidence trail and to surface clinical dimensions not raised in the held-out human reviews.

Critical strengths
  1. The paper targets a timely problem. As more clinical MedAI work enters general ML venues, clinical aspects such as safety are rarely made explicit in written reviews. Reviewing at the level of individual clinical claims is a sensible way to address this.

  2. The design separates retrieval from applicability, and missing evidence from weak evidence. It also keeps each finding traceable to its source, which makes the system's judgments easier to inspect than free-form LLM reviews.

  3. The evaluation is carefully scoped. Human reviews are held out during generation, and the authors do not overclaim the comparison as evidence of reviewer omissions.

Critical weaknesses
  1. The technical contribution is limited. EGCR is mainly a prompt-based pipeline of claim extraction, retrieval, and LLM judgment on a single off-the-shelf model, so the novelty lies more in workflow design than in method.

  2. The main complementarity result is largely self-defined. EGCR decides for itself which dimensions are assessable, and union coverage cannot fall below human coverage by construction, so the reported gain says little about whether the extra judgments are correct or useful.

  3. Baselines and ablations are missing. The comparison with direct LLM review covers only two hand-picked cases, and it is unclear whether a simpler LLM reviewer with the same rubric would reach similar coverage, or what retrieval and applicability assessment actually contribute.

  4. There is no validation by clinicians or experienced reviewers. The traceability audit mostly confirms properties guaranteed by the design, such as recoverable quotes and stored provenance, rather than the quality of the findings.

Questions for authors
  1. Who performed the scoring in the 100-paper diagnostic audit? If it was an LLM, how should the PCA result be interpreted as a property of the papers rather than of the scoring model?

  2. Could you report a quantitative comparison with direct LLM review on the full corpus? In particular, would a direct reviewer using the same rubric reach a similar union coverage?

  3. Has any clinician or experienced reviewer checked a sample of the assessable-but-unasked findings? It would help to know what fraction are correct and genuinely useful to a reviewer.

  4. What do retrieval and applicability assessment contribute on their own? An ablation removing each component would clarify which parts of the pipeline drive the reported results.

  5. The 7% safety coverage in the nine-dimensional analysis appears to be computed per review, while the 73.38% human coverage in the seven-dimensional space is computed per paper and counts only concerns and requests. Could you clarify how these two analyses relate, and which dimensions account for most of the human coverage?

Reviewer_vDfHOpenReview ↗
Rating
2(−2)
Confidence
2
Informativeness
2(−1)
Presentation
2
Soundness
2(−1)
Read review
Paper summary

This paper asks whether peer review at general machine learning venues examines the clinical side of medical AI submissions. In 400 OpenReview reviews, clinical safety gets substantive discussion in 7% and clinical significance in 30%, while technical dimensions such as methodology and presentation sit between 67% and 88%. Accepted clinical papers at ICLR and ICML rose from 61 in 2024 to 278 in 2026, so the gap is widening as the volume grows. The authors respond with EGCR, a staged pipeline built on Claude Haiku 4.5. It extracts clinical claims from a manuscript, retrieves reporting standards and guidelines dated before submission, and records evidence gaps where a score would otherwise be forced. On 168 papers it raises a dimension absent from the human reviews in 107 (63.7%), mostly on problem definition, reference standards and safety.

Critical strengths

The diagnostic study stands whether or not EGCR works. The authors coded 400 reviews against nine dimensions and found clinical safety raised in 7% of them, almost always as a complaint. Set next to the growth in accepted clinical papers (61 to 278 in three years), that gives programme chairs a specific item to add to reviewer guidelines and a baseline to measure against next year. In my experience the people who raise safety are clinicians, and they are scarce in these reviewer pools, so I also read the 7% as a count of who is doing the reviewing.

EGCR keeps the retrieval of a guideline separate from the judgment that it applies to the paper at hand. That separation matters because guidelines are written for particular populations and care settings. A standard developed around a well-resourced tertiary hospital may say little about a model intended for a district clinic, and a system that retrieves it without asking the applicability question would penalize the wrong papers.

Recording not-assessable and evidence-gap as their own states is the right call. In clinical data, the absence of a measurement tells you something about how the patient was cared for, and filling it with a low value hides that. The same logic holds for a review: a dimension the manuscript gives no basis to judge is different from one it handles badly.

Most LLM reviewing systems are scored on similarity to human reviews. The first half of this paper shows why that target is flawed, since a system that matched human reviewers would reproduce the omissions the authors document. EGCR instead produces claim-level output with a provenance chain, and the traceability audit shows the chain was kept (97.1% of manuscript quotes recoverable). A reader can check each flag against its source, which is the minimum I would want before any LLM output informs an editorial decision.

Critical weaknesses

Every figure that depends on EGCR’s judgments is a count of what the pipeline emitted. That includes the 107 of 168 papers and the coverage gain from 73.4% to 89.8%. No clinician examined a sample of the findings to check that the dimension could be judged from the manuscript, or that the retrieved standard applied to the study’s population and care setting. The authors state this limitation plainly, which I appreciate, but the same numbers then appear in the abstract and conclusion without the qualifier.

The risk here is a familiar one in clinical AI, where a model can score well by responding to a surface cue. A pipeline that flags a dimension whenever a disease is named would produce similar “assessable but unasked” rates. A clinician-adjudicated sample would settle the question. The clinicians should come from more than one country and practice setting, because whether a safety question is worth asking depends on where the model would be used.

The claim that evidence grounding yields more specific validation questions rests on two cases the authors selected. The control condition (same model and budget, manuscript text only) was presumably run on more than two papers. I would ask the authors to report the full paired set, scored by clinicians blind to condition. Two chosen examples can illustrate a mechanism, and the abstract should describe them that way.

Reproducibility needs attention. Each paper was run through the pipeline once, the model version is not pinned, and the decoding parameters are not reported. Hosted commercial models change without notice, so another group may be unable to rerun these results in six months. A tool proposed for use in peer review should meet the reproducibility standard it would apply to submissions. Repeated runs per paper, with agreement between runs reported, would show whether the flags are stable.

Questions for authors

Questions

  1. Who or what performed the blinded 100-paper audit in Section 2 and Appendix A.3? The text says “the reviewer” assigned scores with no web access or tools. If that reviewer was Claude Haiku 4.5, then the PCA describes structure in one model’s scores, and the same model runs EGCR and the human-review mapping. Every result in the paper would then trace back to a single LLM’s judgments, with no human reading at any step.
  2. Did a clinician, or any of the authors, read a sample of the findings that EGCR raised and the human reviews did not? A check on even 20 of the 107 papers would change my assessment more than any other addition. I would also want to know where those clinicians practice, since a safety concern that is obvious in one health system can be irrelevant in another.
  3. On how many papers was the direct-review control run? If it was run on all 168, please report the per-dimension coverage for the full set alongside the two cases.
  4. How were the human reviews mapped to the seven dimensions, and by whom? If an LLM did the mapping, what was its agreement with a human annotator on a sample? The 73.4% coverage figure and the 30% and 7% diagnostic figures all depend on this step.
  5. What does “assessable” mean operationally? Is it a model judgment from a prompt or a rule applied to the evidence states? What fraction of dimensions ended up as not-assessable or evidence-gap across the corpus?
  6. Was the model version pinned for any runs, and was any paper run more than once? Five papers run three times each would show whether the extracted claims and the dimension assessments hold steady.
  7. Where do the retrieved standards and guidelines come from? If most were written by groups in a few high-income countries, EGCR may judge a paper from elsewhere against requirements that were never meant for its setting. A breakdown of sources by issuing body and country would help.

Suggestions

  1. Run a clinician validation study. Sample the EGCR-only findings by dimension and have two clinicians rate assessability and correctness blind to source. Report their agreement with each other as well as with the pipeline. Recruiting raters from different countries and including a nurse or pharmacist would test whether the seven dimensions reflect more than one profession’s view of clinical relevance.
  2. Test whether EGCR changes what reviewers write. Give reviewers the EGCR panel for half of a set of submissions and compare how often clinical concerns appear in their reviews. The same experiment should measure the opposite risk: reviewers who see the panel may copy its flags and stop reading the paper for clinical problems themselves.
  3. Compare against a checklist prompt that lists TRIPOD+AI and DECIDE-AI items with no retrieval, run on the same papers. If coverage is similar, the retrieval and applicability steps are adding less than the paper claims.
Reviewer_zSRVOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper looks at the issue of reviewing in medical AI papers across both conference and journal venues. The motivation is the lack of assessment in reviews along identifiable or "assessable" dimensions. The authors use an LLM pipeline to identify those assessment dimensions and then compare them against the actual reviews and what concerns or questions were raised by the reviewers. They then quantify and study the gap between the two.

Critical strengths

At a high level, the motivation is sound and useful. Reviewing faces major challenges and quantifying the relevance and comprehensiveness of reviews is useful. Particularly the temporal analysis that could give insights into how both science and reviewing are changing with AI tools.

The selected papers are decent covering both conference and journal venues.

Critical weaknesses

The main weakness is that it appears EGCR compares two different values. On one side it maps the entire assessable space using the Haiku pipeline, and then it checks which of those the reviewers raised concerns on or made requests for, specifically excluding things like praise. That means that an assessable dimension that a reviewer praised, would get marked as a gap.

On a slightly more fundamental but debatable level, I find that this setup implies that reviews should be all-encompassing and complete, and that any undiscussed but assessable dimension is marked as a gap. Without further context, it is very hard to say if these gaps are relevant or not to the paper at hand.

Analysis is still useful and quantification can be helpful, but a second issue here is that the experiments are not well described (for example, it’s not clear how the papers are mapped to dimensions) and it is a single run with no variance.

Questions for authors
  1. In figure 1, coverage is 76% in 2025 and 77% in 2026 despite more than a doubling in papers. Is that not coverage rising with paper growth? Or what is meant by coverage not rising with paper growth?
  2. How were the 400 reviews mapped to the dimensions? LLM or humans? What was the protocol.
  3. Two paragraphs starting on L158 and L162 seem to say very similar things.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.