- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper claims LLM judges reliably catch commissions (added/altered content) but are near-chance on omissions in AI-generated clinical notes. It releases a 500-pair benchmark (298 omissions graded by severity + surviving "trace," 202 commission controls) built transcript-first because reference notes are themselves discrepant.
Critical strengths
- The RAGAS control shares no prompt text and extracts with the judge model (not the constructor), yet still hits 0.817; concurrent independent work (DeLucia et al.) reports the same monolithic null. The recovery is a property of the task, not of their pipeline.
- Confounds are controlled, not just disclosed. GEPA compute confound → re-run with a control; single family → second judge family; commission arm → external MEDEC anchor; circularity → three separate controls.
- Genuinely novel, citable methodology. Transcript-first ground truth (motivated by the finding that 100% of reference notes are materially discrepant), severity + surviving-trace grading, and the per-fact-rule-vs-score distinction (2.7× detection at matched FAR).
- Honest about deployment limits. No threshold survives real scribe output, ~1/3 detection ceiling, restatement-trace failure — all reported by the authors rather than buried.
Critical weaknesses
- The entire negative result rests on model-generated fact-sheet labels, never independently clinician-labelled end to end; the only matching external benchmark is their own stratum. Self-disclosed (Limitation 1).
- All commissions are carried-over; most omissions are newly built. The within-cohort commission control doesn't exist in the release. The gap survives the uniform-omission slice and a MEDEC anchor, so the confound inflates the ~0.3 paired gap rather than inventing it — but the cleanest comparison still can't be made.
- One physician author, who designed the instruments and knew the claims; most stages are endorsements not blinded; the 10/10 headline is n=10 hand-picked disagreements; only the severity rubric got one external clinician (12 items). Weak for a clinical-safety framing, and the critical-class generosity (1/10 rule-fired criticals confirmed blind) undercuts the decision rule.
- Clinical validation is thin: one author-clinician, mostly unblinded endorsements, a 10/10 headline on n=10 cherry-picked cases, and only 12 items checked by an external rater. The same validation shows the "critical" class is generous (1/10 rule-fired criticals confirmed blind), which undercuts the rule that fires on it.
Questions for authors
- Every omission label is transcript-derived and model-generated. Is there any way to show the monolithic-vs-pipeline gap replicates on omissions labelled independently of your instrument — even a small clinician-labelled set not drawn from your own strata? Without it, how should a reader distinguish "judges can't see absence" from "judges don't reproduce your fact-sheet constructor"?
- All 202 commissions are carried-over; most omissions are newly built, and A.6 states the uniform cohort has no commission arm. Can you build even a small uniformly-constructed commission set (span-level insertion off the same site map) so the headline 0.94-vs-0.50 gap can be measured within one cohort? How large would a cohort effect have to be to account for the gap you see?
- Only 1/10 rule-fired "critical" facts were confirmed; 6/10 after reconciliation. Since the decision rule fires on "critical," what does detection/FAR look like if you re-score the stored verdicts against a clinician-aligned critical threshold rather than the audit stage's generous one?