acceptodds
Under review as a conference paper at ICLR 2027

LLM Judges Verify Presence, Not Absence: Omission Blindness in Generated Clinical Notes, Its Mechanism, and What Recovers It

Abstract

LLM judges score generated text at scale, and the benchmarks that measure them ask whether what a text says is correct, not whether what it should say is there. We measure that second capability where it is consequential and, unusually, constructible: ambient AI scribes writing clinical notes, where published human audits find omission the dominant error class. Public corpora cannot supply the answer key - every clinician reference note we audited is materially discrepant with its own transcript - so we release 500 single-error note pairs built from transcript-derived, audited fact sheets: 298 in which a named fact is certainly absent, graded by severity and by surviving trace, against 202 added-or-altered controls. Across an eight-design ablation, paired discrimination (0.5 is a coin flip) reads 0.79-0.94 on added or altered content against 0.50-0.63 on omissions, and on single notes every design flags notes with omissions about as often as perfect ones. Criterion scope, output format, voting, wording and automated prompt optimisation all move the operating point without creating detection, each with a bound on what we could have missed, and a larger reasoning budget lifts discrimination without closing the asymmetry. What recovers detection is restructuring the task - list the facts the source establishes, then verify each against the text as a closed presence check. Two methods reach that restructuring independently, and a control separates its parts, the fact list carrying about a third of the recovery and closed per-fact verdicts the rest. Held out, reading those verdicts as a rule rather than averaging them into a score is worth 2.7 times the detection at an identical false-alarm rate, and transfers across judge families near-unchanged, while no threshold set on constructed pairs survives real scribe output. Omissions whose restatement survives elsewhere defeat both. A physician author validated 70 items in a sitting covering six stages, three of them structurally blinded, and sided with the pipeline on all 10 notes where it and the best monolithic judge reach opposite conclusions; the severity rubric has since been graded blind by an independent clinician - not an author - agreeing to within a grade across the companion census's two clinician sittings. We release the benchmark, the prompts and every judgement.

Primary area:
datasets and benchmarks
Keywords:
LLM-as-a-judge, automatic evaluation, omission detection, benchmark, faithfulness and hallucination detection, evaluation methodology, clinical text generation, prompt optimisation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_qqPuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper claims LLM judges reliably catch commissions (added/altered content) but are near-chance on omissions in AI-generated clinical notes. It releases a 500-pair benchmark (298 omissions graded by severity + surviving "trace," 202 commission controls) built transcript-first because reference notes are themselves discrepant.

Critical strengths
  1. The RAGAS control shares no prompt text and extracts with the judge model (not the constructor), yet still hits 0.817; concurrent independent work (DeLucia et al.) reports the same monolithic null. The recovery is a property of the task, not of their pipeline.
  2. Confounds are controlled, not just disclosed. GEPA compute confound → re-run with a control; single family → second judge family; commission arm → external MEDEC anchor; circularity → three separate controls.
  3. Genuinely novel, citable methodology. Transcript-first ground truth (motivated by the finding that 100% of reference notes are materially discrepant), severity + surviving-trace grading, and the per-fact-rule-vs-score distinction (2.7× detection at matched FAR).
  4. Honest about deployment limits. No threshold survives real scribe output, ~1/3 detection ceiling, restatement-trace failure — all reported by the authors rather than buried.
Critical weaknesses
  1. The entire negative result rests on model-generated fact-sheet labels, never independently clinician-labelled end to end; the only matching external benchmark is their own stratum. Self-disclosed (Limitation 1).
  2. All commissions are carried-over; most omissions are newly built. The within-cohort commission control doesn't exist in the release. The gap survives the uniform-omission slice and a MEDEC anchor, so the confound inflates the ~0.3 paired gap rather than inventing it — but the cleanest comparison still can't be made.
  3. One physician author, who designed the instruments and knew the claims; most stages are endorsements not blinded; the 10/10 headline is n=10 hand-picked disagreements; only the severity rubric got one external clinician (12 items). Weak for a clinical-safety framing, and the critical-class generosity (1/10 rule-fired criticals confirmed blind) undercuts the decision rule.
  4. Clinical validation is thin: one author-clinician, mostly unblinded endorsements, a 10/10 headline on n=10 cherry-picked cases, and only 12 items checked by an external rater. The same validation shows the "critical" class is generous (1/10 rule-fired criticals confirmed blind), which undercuts the rule that fires on it.
Questions for authors
  1. Every omission label is transcript-derived and model-generated. Is there any way to show the monolithic-vs-pipeline gap replicates on omissions labelled independently of your instrument — even a small clinician-labelled set not drawn from your own strata? Without it, how should a reader distinguish "judges can't see absence" from "judges don't reproduce your fact-sheet constructor"?
  2. All 202 commissions are carried-over; most omissions are newly built, and A.6 states the uniform cohort has no commission arm. Can you build even a small uniformly-constructed commission set (span-level insertion off the same site map) so the headline 0.94-vs-0.50 gap can be measured within one cohort? How large would a cohort effect have to be to account for the gap you see?
  3. Only 1/10 rule-fired "critical" facts were confirmed; 6/10 after reconciliation. Since the decision rule fires on "critical," what does detection/FAR look like if you re-score the stored verdicts against a clinician-aligned critical threshold rather than the audit stage's generous one?
Reviewer_Y5d7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how, in clinical settings, LLM judges can catch incorrect information in notes but not missing information. The authors then try established fixes, such as checking each fact individually against the source. They take consultation transcripts and build a test set, which serves as the dataset.

Critical strengths

The core observation is not fully novel; it was discovered previously, but applying this existing phenomenon to clinical settings is interesting and addresses a real practical problem that can have impact. As reported in the paper, the authors were careful with their measurements.

Critical weaknesses

Poor presentation

Figure 1 is small and unreadable.

The text is hard to read. It is not that the text itself is not understandable, but it is hard to follow. I strongly suggest the authors put time into making sure the flow, the connections, and the writing are good. The abstract was verbose and vague, in my opinion, and the text contains many long and verbose sentences.

The opening of the introduction is very vague: “An LLM judge is a second model asked to read a source document and a generated text and report what is wrong with the text.” What do you mean by “second model”? It presumes there is a first LLM generating the text, but that may not be the case.

The introduction is very short. It is one paragraph explaining the setting, then suddenly the gap and three contributions; there is not much connection between the parts of the introduction.

The test set targets “was this specific sentence removed?” and does not consider whether a doctor would consider that a meaningful omission or not. I am not a physician and do not know how realistic and helpful this setting is for the medical domain. The authors themselves position their work relative to AbsenceBench, and I personally believe this paper’s novelty does not come from the phenomenon or the results, which were established before by AbsenceBench, but from applying a similar framework and idea to the medical domain.

This paper does not have a significant weakness, but it also does not have a significant or surprising novel strength, in my opinion; it is a sound study.

What does this mean? “If reference notes cannot be truth, the transcript has to be.” There are many sentences formatted like this. I strongly suggest the authors rewrite the manuscript and improve the presentation.

The paper does not have a smooth flow; it has long sentences, awkward framings, and also cases like “A written rubric fixes most of it, so the benchmark ships the grades of two cross-family graders working from that rubric, at κ 0.662 [0.59, 0.73] over the corpus’s 683 graded facts (the 613 in Figure A.2’s caption are the evaluation split alone), taking the lower grade where they disagree.” This refers to an appendix figure the reader has no context for, so what would the 613 number make clear?

There is not much grounding for “Removing a fact should then be mechanical, and it is not, because real notes repeat themselves.” How did the authors arrive at this? It seems like a jump.

Page 4 has a paragraph that is 24 lines long. I tried reading it multiple times, but it was very, very confusing to me.

The paper does not tell a story; it is more of a documentation report full of numbers and the rigorous statistics behind them. Since the paper is essentially the medical application of AbsenceBench, I believe the authors should take multiple passes at how that paper was presented and rewrite the full paper. My understanding of the paper came from my familiarity with the topic and related work; it must be hard to understand for readers without that background.

Questions for authors

Consider having a clinician rate a sample of the omissions by severity, so detection rates can be reported by how much each omission matters.

Reviewer_UfBWOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether LLM judges can detect omissions in generated clinical notes as well as they detect commissions (added or altered content). To evaluate this, the authors use LLMs to construct 500 note pairs from consultation transcripts, where each pair contains a verified clean note (the “clean twin”) and a copy with one planted error. They show that single-prompt (“monolithic”) judges on gpt-5.4 are near chance on omissions. This includes a plain prompt asking for a faithfulness and completeness rating. It holds for both paired discrimination (whether the flawed note scores lower than its clean twin, each scored separately) and single-note detection (whether the judge flags a flawed note on its own). Rewording, voting and GEPA prompt optimization do not help.

To address this, they propose a pipeline that (1) lists the transcript’s facts, (2) grades how important each fact is, and (3) checks whether each fact appears in the note. A note is flagged if any critical fact is missing (the “per-fact rule”). The paper claims this restructuring is what recovers detection. It also tests an optimized prompt that lists facts itself (the “evolved prompt”) and a control that gives a monolithic judge the pipeline’s fact list, mostly on a 151-pair held-out subset. Finally, the flagging cutoffs set on the benchmark do not transfer to 261 real AI-scribe notes.

Critical strengths
  1. The paper targets a practical problem. LLM judges are hard to evaluate, and this matters more in clinical AI, where judges are expected to catch missing clinical facts that pose a safety risk
  2. The benchmark design is good. Each omission is removed from a verified clean note, so its absence is certain, and each is graded by severity and by whether the fact survives elsewhere in the note.
Critical weaknesses
  1. Paper writing. The paper is hard to follow. Sentences often pack several numbers, caveats, and cross-references into one, and some are ambiguous. The main results and settings appear only in the appendix (e.g., the main text names no model). Several terms are never defined or are explained only in the appendix. (e.g., rung, acceptance split, engineered judge, etc.)

  2. Circularity between the benchmark and the pipeline. While building the benchmark, the authors used claude-opus-5 for the fact lists and gpt-5.5 as one of the two models grading each fact’s importance. The pipeline uses the same two models for the same two steps. The pipeline is therefore partly tested against answers produced by its own models, which may inflate its results. The paper does not test these two steps with different models, and Section 7.1 does not mention this overlap.

  3. Uncontrolled comparison. The baselines make one call to gpt-5.4 with no reasoning and a 1,024-token cap. The paper says this setting reflects “the cost and latency budgets an evaluation actually runs to in the field” (Section 4), but gives no evidence for it. The per-fact rule makes several calls to three models, all with reasoning (claude-opus-5 at medium effort, gpt-5.5 at high effort, and gpt-5.4 at medium effort). A reasoning budget alone raises the plain judge’s paired discrimination from 0.570 to 0.670 (Table B.2), but the paper reports no single-note results for baselines with reasoning. Only one monolithic judge has both reasoning and single-note results (gemini-3.1-pro-preview at “minimal” reasoning). It flags 19.9% of omission notes at 2.1% false alarms, with a cutoff fixed in advance and three runs. The per-fact rule flags 24.6% at 2.7% on the same 495-pair evaluation set, with a rule set after seeing held-out results and one run. The effect of restructuring is therefore not separated from the effect of reasoning and the extra models.

  4. Unsupported attribution of the pipeline’s gain. Contribution (iv) states that “the fact list accounts for about a third of the recovery, closed per-fact verdicts for the rest.” To test this, the authors give the fact list to the plain single-call judge and run it once (0.634 paired discrimination). Without the list, the same judge ranges from 0.508 to 0.603 over three runs (mean 0.570). Scores vary by up to 0.095 between runs, so a single run does not give a reliable estimate of the list’s effect. The authors also never give the list to stronger monolithic judges, such as the best monolithic judge (0.531 to 0.649 over three runs) or the engineered omission judge (0.635 over two runs). The claim is therefore not tested against stronger baselines.

Questions for authors
  1. Could you rerun the pipeline with a model from a different family, such as gemini-3.1-pro-preview, for listing facts and grading their importance? Please report the pipeline’s paired discrimination and the per-fact rule’s detection and false-alarm rates.

  2. Please clarify whether the two-stage pipeline makes one call per fact (Table B.2) or one call per note (Section 7, l. 403).

  3. Could you run the plain prompt, the best monolithic judge and the prompt changes from Section 6 with reasoning and a token budget matched to the pipeline’s total per note? Please report their single-note detection and false-alarm rates.

  4. Could you report the per-fact rule’s results when every pipeline step runs with no reasoning?

  5. Could you report the performance of the best monolithic judge when it is given the fact list?

  6. Could independent clinicians review a sample of the omission labels and confirm that each omission makes the note clinically deficient? The labels come from claude-opus-5’s fact extraction and an LLM importance grade. The census also shows that the share of notes with something reportable moves from 27.8% to 96.5% depending on the review instruction (Appendix J).

Reviewer_L8EuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether LLM judges can detect missing information in AI-generated clinical notes. Using 495 pairs of notes with controlled edits, the authors compare omissions with additions and alterations and test whether checking facts individually improves detection.

Critical strengths

Missing information is an important and easily overlooked problem in clinical documentation. The paired design holds the consultation fixed and distinguishes complete omissions from cases where some or all of the information remains in the note.

The paper reports detection rates alongside false alarms. Comparing two alert rules on the same set of fact-level judgments helps isolate the effect of how those judgments are combined.

Critical weaknesses

The construction process establishes which text was removed. Whether the removal warrants a clinical alert is determined mainly by models. This is particularly important for the 57 evaluation pairs in which the fact is still stated elsewhere in the note. The physician author reviewed 10 selected cases where the judges disagreed, and the independent clinician assessed severity on a separate sample. It remains unclear how often independent clinicians would consider the benchmark omissions worth flagging.

All 202 addition or alteration pairs come from an earlier build, while most omission pairs were created using a newer procedure. Restricting the analysis to the newer omission pairs leaves this difference in place. Some of the reported detection gap may therefore reflect how the two sets of examples were constructed.

The main detection rates are calculated per note. An alert can count as a hit without identifying the fact that was removed. This matters especially for commercial scribe notes, where several errors can coexist. Appendix J checks whether alerts name the verified omission for one method using lexical matching. The note-level results leave open how reliably each method identifies the specific information that is missing.

Questions for authors

Could independent clinicians review a sample of omission labels, including cases where the fact still appears elsewhere in the note?

Does the detection gap persist when the same fact is deleted in one version of a note and altered in another, using the same verification procedure?

After setting the alert threshold on one set of commercial scribe notes, what detection and false-alarm rates does it achieve on a separate set of consultations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.