acceptodds
Under review as a conference paper at ICLR 2027

Narrative Fidelity: Beyond Factuality in Evaluating Clinical Dialogue Summarization

Abstract

Clinical notes are usually evaluated as summaries, with ROUGE, BERTScore, or other overlap with a reference note. These metrics reward factual completeness but say little about whether a note preserves the patient's own framing, concerns, uncertainty, and perspective. We call this property narrative fidelity and introduce a six-dimension annotation taxonomy for it. We apply the taxonomy to notes from six systems (two current Claude models and four published baselines) on the ACI-Bench clinical dialogue corpus, and report two findings. First, narrative fidelity agrees with ROUGE on system ranking (Spearman ) but not on individual cases. Across cases and systems, its correlation with ROUGE-1 is moderate (– across core dimensions), well below the correlation among the ROUGE variants themselves (–), so the taxonomy captures variance that ROUGE does not. ROUGE tracks Clinical/Factual Content most closely () and Uncertainty/Epistemic Stance least (). BERTScore correlates less still (– per core dimension) and barely separates the systems: mean BERTScore varies by across systems whose narrative fidelity ranges from to . Second, we test sensitivity to a naturally occurring degradation, comparing ASR and human transcripts of the same 28 encounters. Aggregate narrative fidelity and ROUGE-1 change little, but Patient Concerns & Priorities declines (, paired -test; , Wilcoxon), and the decline is spread across cases rather than driven by outliers. This result does not survive correction for the six dimensions tested, so we treat it as a hypothesis for replication. It suggests that transcript quality can erode specific narrative content when overall note quality looks unchanged. Inter-annotator agreement is strong on three dimensions and lower for Perspective/Attribution, and we discuss what these results imply for evaluating AI scribes beyond factuality checks.

Primary area:
datasets and benchmarks
Keywords:
clinical dialogue summarization, narrative fidelity, LLM evaluation, benchmark, human annotation, inter-annotator agreement, ASR robustness, healthcare NLP

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_pW9FOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This study claims to contribute to evaluation methodology: it creates a six-dimension, five-status annotation taxonomy for narrative fidelity in clinical note generation and an empirical demonstration, across six generation systems and dialogues from ACI-Bench. The manuscript is poorly written and the contributions remain unclear.

Critical strengths

None

Critical weaknesses

"Clinical notes are usually evaluated as summaries, with ROUGE, BERTScore, or other overlap with a reference note. These metrics reward factual completeness..." ---> what do you mean? Is it true that ROUGE and BERTScore reward "factual completeness"?

Lacking detailed foundations and definitions of six-dimension, five-status annotation taxonomy

The data size is small, limiting its contribution: "a public corpus of 207 clinical dialogue–note pairs across three sub-collections (aci, virtassist, virtscribe)"

Questions for authors

Why six-dimension, five-status annotation taxonomy?

The study does not cite MED-OMIT, DocLens, TofuEval. It also did not cite Abacha et al, 2023 (Findings of ACL) and Asgari et al, npj Digital Medicine 2025.

Reviewer_qQCQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces a narrative fidelity evaluation method to assess whether AI-generated clinical notes preserve patients' concerns, uncertainty, and perspectives. The paper compares narrative fidelity with ROUGE and BERTScore on ACI-Bench, using a limited set of cases across several LLMs. The paper also conducts a small human evaluation, reporting a decline in preserving patient concerns.

Critical strengths
  1. The proposed evaluation taxonomy may be good for evaluating narrative fidelity. The human evaluation includes inter-annotator agreement and an independent clinician check.
  2. The human and ASR transcript experiment investigates whether transcription quality affects specific aspects of clinical note generation.
Critical weaknesses
  1. Abstract: ROUGE and BERTScore do not evaluate factuality. The abstract needs substantial improvement as it has many vague descriptions and is hard to understand. For example, what does “We call this property…” mean? What is ‘this’?
  2. What is the main contribution of this paper? Is it a new automatic metric, an evaluation protocol, or an evaluation pipeline? It is unclear throughout the manuscript.
  3. While the authors conduct independent clinical spot-checks, only a small part of the judgments are reviewed. Only a very small number of judgments are related to patient concerns, perspective, and uncertainty. This is insufficient to show the clinical validity. Therefore, the main contribution of this paper is not well-supported in my view. Further evaluation and validation are needed.
  4. The compared metrics are too limited to support the paper’s main claim. Only ROUGE and BERTScore are compared, however, these metrics assess text similarity and overlap instead of factual correctness. Factuality-related metrics are missing. For example, AlignScore, SummaC, QAFactEval, and PlainQAFact. Additionally, lower correlations with ROUGE do not mean the proposed evaluation method captures clinically meaningful errors.
  5. I’m concerned about the annotation quality. Perspective/Attribution has relatively low agreement, while the ASR shows low correlations across several dimensions. Further validation of the annotation quality is needed.
Questions for authors
  1. Why is BERTScore-F1 in Figure 1 roughly the same for six models? Any analysis for that?
  2. What is ChatGPT? Which specific model does it represent? The evaluated models are mostly outdated and rarely used.
  3. Table 1 reports a core narrative fidelity score of 0.965 for Claude Sonnet 5, while the four core dimensions in Appendix Figure 5 have an unweighted mean of around 0.82. How is the overall score calculated? Are the dimensions weighted differently? Also, the introduced content is excluded from the headline score.
  4. Why are the prompts used for the six LLMs different?
Reviewer_JJaxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper addressed qualitative evaluation of clinical dialogue summarization. Instead of automatic metrics the authors suggest a schema that ought to capture narrative fidelty build around six dimensions. They test the reliability of this schema with three annotators on 22 cases and show that it does not correlate well with existing automatics metrics. They extend this anaylsis to summaries based on dialogues transcribed by ASR.

Critical strengths
  1. Highly relevant problem to get better evaluation metrics for clinical diaglogue summarization. These tool will likely be more disseminated and existing automatic metrics seem to be insufficient.
Critical weaknesses
  1. There is no evidence for the validity of the introduced evaluation schema and, hence, it has unclear clinical relevance. The authors introduce the new concept of narrative fidelity and define six dimensions to measure it. However, there is no evidence given from the medical literature that these capture anything meaningful or relevant. The provided IRR measured the reliability of their introduced evaluation schema, but no evidence at all is given for the validity (e.g. content validity, construct validity, ...).
  2. The entire downstream analysis is based on the above evaluation schema and due to the limited validity it is not meaningful to interpret. I think overall it is interesting to consider a new evaluation schema, validated them with experts, and automize them with LLMs and show their added value to existing automatic metrics. However, the central part is to show that this newly introduced schema is valid and reliable o.w. the downstream analysis has limited value.
Questions for authors

Clarifications:

  1. I checked the supplement but could not find all three annotator results on the 22 cases. can you please provide it? I wanted to check if the IRR is simply high due to implicit bias of the annotators.
  2. How were the units chosen on which the IRR was validated? Did one person chose the units and then IRR was measured on the agreed on units? This would strongly bias the IRR numbers as the original schema also includes choosing the correct units.
  3. Please provide the full annotation schema, the exact input the annotators saw and their labels to validate the annotation procedure.
  4. Overall the related work seems incomplete from a summarization of medical text perspective (e.g. https://aclanthology.org/2021.naacl-main.382/; also evidence from the medical literature for the validity of the proposed concept is missing.
  5. I do not understand the justification of only using OpenAI: "without needing API access beyond a single provider"? There are many standardized interfaces across different LLM providers.

Suggestions (future work):

  1. This sentence seems misformatted "is reported alongside it but doesn’t."
  2. There seems to be a stray newline in the heading of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.