Narrative Fidelity: Beyond Factuality in Evaluating Clinical Dialogue Summarization

Clinical notes are usually evaluated as summaries, with ROUGE, BERTScore, or other overlap with a reference note. These metrics reward factual completeness but say little about whether a note preserves the patient's own framing, concerns, uncertainty, and perspective. We call this property narrative fidelity and introduce a six-dimension annotation taxonomy for it. We apply the taxonomy to notes from six systems (two current Claude models and four published baselines) on the ACI-Bench clinical dialogue corpus, and report two findings. First, narrative fidelity agrees with ROUGE on system ranking (Spearman $\rho =0.94$) but not on individual cases. Across cases and systems, its correlation with ROUGE-1 is moderate ($r\approx 0.43$–$0.68$ across core dimensions), well below the correlation among the ROUGE variants themselves ($r\approx 0.76$–$0.88$), so the taxonomy captures variance that ROUGE does not. ROUGE tracks Clinical/Factual Content most closely ($r\approx 0.68$) and Uncertainty/Epistemic Stance least ($r\approx 0.43$). BERTScore correlates less still ($r\approx 0.19$–$0.27$ per core dimension) and barely separates the systems: mean BERTScore varies by $0.014$ across systems whose narrative fidelity ranges from $0.243$ to $0.965$. Second, we test sensitivity to a naturally occurring degradation, comparing ASR and human transcripts of the same 28 encounters. Aggregate narrative fidelity and ROUGE-1 change little, but Patient Concerns & Priorities declines ($p=0.014$, paired $t$-test; $p=0.018$, Wilcoxon), and the decline is spread across cases rather than driven by outliers. This result does not survive correction for the six dimensions tested, so we treat it as a hypothesis for replication. It suggests that transcript quality can erode specific narrative content when overall note quality looks unchanged. Inter-annotator agreement is strong on three dimensions and lower for Perspective/Attribution, and we discuss what these results imply for evaluating AI scribes beyond factuality checks.