- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper categorizes clinician edits and filters out unsupported or personal changes before fine-tuning clinical summarization models. Experiments on semi-synthetic edit data examine how filtering affects reference alignment and faithfulness to input records.
Critical strengths
-
The paper addresses a practical problem: clinician edits may contain information unavailable to the model, making them unreliable training targets.
-
The taxonomy is informed by real deployment data, and the semi-synthetic dataset supports controlled comparisons of different edit filters.
Critical weaknesses
-
The contribution to AI and healthcare remains unclear. Better reference alignment does not establish clinical usefulness. “Without undesirable degradations” is too strong when Grounded + Global is less grounded in 4 of 10 cases. Treating personal edits as unsuitable supervision is also questionable, given 52% agreement with experts on this classification and evidence that some remain useful.
-
The 90 human judgments cover only 10 admissions and exclude clinical usefulness, completeness, and error severity. A larger evaluation should measure these outcomes and report uncertainty. Training benefits also need validation using real clinician edits.
-
[1] already uses synthetic clinical edits for training, and [2] corrects unsupported references before training clinical summarizers. The paper needs to explain what its taxonomy and filtering add.
-
Two controls are needed: directly cleaning unsupported content from final edited drafts using the same LLM and training budget, and randomly reverting edits while matching edit count and target length. The current filtering-rate experiment only reverts edits already tagged ungrounded, so it does not isolate the benefit of targeted selection.
-
The fine-tuned models are limited to Gemma-3-4B, Qwen-2.5-7B, and Llama-3.1-8B. I suggest adding a more recent, stronger open-weight model to test whether the findings generalize.
[1] Mishra, P., Yao, Z., Vashisht, P., Ouyang, F., Wang, B., Mody, V. D., & Yu, H. (2024, November). Synfac-edit: Synthetic imitation edit feedback for factual alignment in clinical summarization. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. 20061-20083).
[2] Adams, G., Shing, H. C., Sun, Q., Winestock, C., McKeown, K., & Elhadad, N. (2022, December). Learning to revise references for faithful summarization. In Findings of the Association for Computational Linguistics: EMNLP 2022 (pp. 4009-4027).
Questions for authors
Please refer to the weaknesses above.
5 comments
Response by Authors [2/4]
“The 90 human judgments cover only 10 admissions and exclude clinical usefulness, completeness, and error severity. A larger evaluation should measure these outcomes and report uncertainty. Training benefits also need validation using real clinician edits.”
First, we clarify that our experiments in Section 7 address a targeted question: how does training on raw vs. filtered edits impact the groundedness of the model-generated BHCs? The expert annotation study in Section 7.3 was designed to assess the impact on groundedness, rather than provide a comprehensive comparison across all dimensions of summarization quality. We agree that other aspects, such as completeness and error severity, are also relevant to assessing a model-generated BHC’s overall clinical utility. We explicitly acknowledge the need for such broader expert evaluation in our Limitations section (Appendix H, lines 2169–2172) and leave it as future work.
Second, assessing groundedness requires human experts to check clinical details in the model-generated BHCs against large collections of prior EHR notes. In our study, obtaining the 90 pairwise judgments required 30 hours of expert annotation, with at least 10 clinical notes per admission to reference (Appendix B.1). We agree that expert validation on a larger scale would provide more statistical power, but we leave it to future work given the substantial costs of expert annotation, as noted in the Limitations section (Appendix H, lines 2162–2163). Meanwhile, we report inter-annotator agreement for the annotation results in Appendix F.1 (63% unanimous agreement, 27% majority agreement).
Third, we agree that additional validation on real clinician-edit data would help establish how well our findings generalize to real-world editing workflows. As noted in our Limitations section (Appendix H, lines 2155–2163), while our semi-synthetic MIMIC-III-Ext-BHC-Edit dataset allows for controlled investigation of how edits can be used for model supervision, we do not claim that it can be used as a substitute for evaluation on real post-deployment edit data.
“[1] already uses synthetic clinical edits for training, and [2] corrects unsupported references before training clinical summarizers. The paper needs to explain what its taxonomy and filtering add.”
Thank you for highlighting these relevant works. We will include them in the Related Works section in our revision. However, we clarify that both Mishra et al. [1] and Adams et al. [2] address different problem settings.
Mishra et al. [1] study training models on synthetic edits made by frontier LLMs, in settings where clinician feedback is difficult to obtain at scale. These synthetic edits either make targeted improvements to drafts generated by a weaker LLM, or targeted degradations to clinician-written reference summaries to artificially construct edit pairs for preference learning. In both cases, the preferred version is known by construction. Adams et al. [2] propose training a model to revise unsupported sentences in clinician-written summaries, also with an emphasis on limited-data settings where simply removing training examples with noisy summaries is challenging.
In contrast, our work studies how to characterize, filter, and train models on edits made by clinicians, which can be collected at scale during their use of production clinical documentation systems. A key contribution of our work is showing that such edits should not be assumed to be improvements for the model to learn to reproduce. Our proposed taxonomy (Section 3.2) enables us to distinguish between different types of clinician edits (e.g., grounded vs. ungrounded, global vs. personal) and selectively use them as supervision signal. Applying our edit tagging + filtering method (Section 5) to these edits does not require any simulation of edits.
Given the limited availability of public clinician-edit datasets, we demonstrate the effectiveness of our framework on a semi-synthetic dataset based on MIMIC-III (Section 4). While the construction of this dataset involves simulating edits on top of LLM-drafted BHCs, every added or corrected detail is based on a real clinician-written BHC, and the types of simulated edits are informed by real-world clinician-editing behaviors observed in a deployed clinical documentation system.
Response by Authors [3/4]
“Two controls are needed: directly cleaning unsupported content from final edited drafts using the same LLM and training budget, and randomly reverting edits while matching edit count and target length. The current filtering-rate experiment only reverts edits already tagged ungrounded, so it does not isolate the benefit of targeted selection.”
To address your concerns, we provide additional results for the two suggested “direct cleaning” and “random reversion” baselines, using the same fine-tuning setup as in Section 6.
To implement the “direct cleaning” baseline, we consider two approaches for detecting ungrounded content in each post-edit BHC. First, we apply the off-the-shelf VeriFact pipeline by Chung et al. [3], which extracts the “atomic claims” from each BHC and labels each claim as grounded vs. ungrounded. Alternatively, we apply an adaptation of our GPT-5.5-based edit-tagger pipeline, which extracts the “atomic claims” and labels each claim as grounded vs. ungrounded. In either case, we then remove the ungrounded claims from the post-edit BHC, using our pipeline for constructing the grounded test set (Section 6, lines 358–363).
To implement the “random reversion” baseline, we randomly select and revert the same number of edits as the “Grounded” filter does for each edit pair. To account for randomness, we repeat this process three times with different random seeds. As reverting an edit can either shorten or lengthen the target (e.g., reverting an addition removes text, while reverting a deletion adds text), simultaneously matching the target lengths would require selectively reverting certain types of edits, rather than sampling uniformly from all edits. On average, the “random reversion” targets contain 234.3 words, while the “Grounded” targets contain 199.0 words, which are both much shorter than the targets before any filtering (305.3 words).
Below, we report the results for Qwen-2.5-7B-Instruct. We report both the reference-alignment and groundedness metrics as in Sections 6–7. For the “random reversion” baseline, we report the average across three seeds.
Reference Alignment
| Setting | DocLens-P | DocLens-R | MEDCON |
|---|---|---|---|
| Base | 0.2713 | 0.1736 | 0.2527 |
| Raw | 0.3029 | 0.3312 | 0.3796 |
| Direct Cleaning (GPT-5.5) | 0.3313 | 0.3231 | 0.3667 |
| Direct Cleaning (VeriFact) | 0.3173 | 0.3155 | 0.3642 |
| Random Reversion | 0.2942 | 0.3183 | 0.3482 |
| Grounded (Ours) | 0.3198 | 0.3249 | 0.3710 |
Groundedness
| Setting | MedScore | DocLens-Attr. | PDSQI-9 |
|---|---|---|---|
| Base | 0.9478 | 0.9004 | 0.5992 |
| Raw | 0.8742 | 0.7527 | 0.2357 |
| Direct Cleaning (GPT-5.5) | 0.9552 | 0.8674 | 0.6193 |
| Direct Cleaning (VeriFact) | 0.9290 | 0.8294 | 0.4323 |
| Random Reversion | 0.8932 | 0.7917 | 0.3034 |
| Grounded (Ours) | 0.9529 | 0.8804 | 0.5802 |
Our “Grounded” filter outperforms the VeriFact-based “direct cleaning” and “random reversion” baselines across all reference-alignment and groundedness metrics. Notably, randomly reverting edits still leads to a noticeable drop in groundedness (MedScore: 0.9478 0.8932) despite substantially shortening the target BHCs, while our “Grounded” filter retains similar levels of groundedness (MedScore: 0.9478 0.9529). These results demonstrate that targeted removal of ungrounded edits mitigates groundedness degradation from fine-tuning more effectively than randomly reverting the same number of edits.
GPT-5.5-based “direct cleaning” performs comparably to our “Grounded” filter. This is expected in our setting: the base model drafts are largely grounded, and nearly all ungrounded clinical content in the post-edit BHCs is introduced by the edits. Importantly, our goal is not to propose a better method for detecting ungrounded content, but to demonstrate that clinician edits need to be filtered appropriately for effective supervision. Filtering ungrounded edits is one concrete case study of our framework, which more generally enables filtering edits along other axes of our taxonomy.
Response by Authors [1/4]
Dear Reviewer b85S,
Thank you for your feedback and suggestions! We address your concerns below:
We agree that greater reference alignment does not guarantee higher clinical utility, as we note in our Limitations section (Appendix H, lines 2169–2172). We have revised the relevant text to clarify this point in our updated PDF: “improving the clinical utility” → “improving the alignment… to the clinician-written reference summary” (Section 6, lines 346–347).
We use the reference alignment metrics to assess how well the model-generated BHCs capture the clinical details documented in clinician-written references, recognizing that this is only one aspect of summarization quality. Indeed, one of our main findings is that naively training on raw, unfiltered edits can improve reference alignment but degrade groundedness in the input EHR notes (Section 7.1). Our analysis explicitly distinguishes these properties rather than treating reference alignment as a complete measure of task performance.
Our contribution is a framework for characterizing clinician edits to LLM drafts and translating them into effective supervision. Such edits accumulate at scale in deployed clinical documentation systems, but alongside useful corrections, a clinician’s revision can include additions based on information unavailable to the model (i.e., ungrounded edit) and changes reflecting local documentation preferences (i.e., personal edit). Indeed, we estimate that over 45% of edited notes in our proprietary corpus from an ambient medical scribing product contain at least one ungrounded edit (Appendix A.2). An edit that is preferable for clinical use is therefore not necessarily one the model should learn to reproduce. Our taxonomy makes these distinctions explicit, and our edit filtering method enables selective training on edits that provide useful supervision. On our MIMIC-III-based dataset, filtering for grounded edits improves reference alignment while mitigating degradation in groundedness.
Thank you for raising this point. We agree that the phrase “without undesirable degradations” in Figure 1 is stronger than intended. We have revised it to “while mitigating undesirable degradations” in our updated PDF.
We clarify that we do not regard personal edits as inherently unsuitable supervision. In Section 7.2 (lines 467–468), we note that some edits tagged as personal may still provide useful supervision, as suggested by the stronger reference-alignment results with the “Grounded” filter than with “Grounded + Global”. We consider removing personal edits since they reflect local preferences, and our goal is to improve the general BHC generation capabilities of the model. In other settings where the goal is to effectively adapt the model to local settings, personal edits may provide useful supervision, as we note in our Limitations section (Appendix H, lines 2174–2176).
Regarding the 52% tagger–expert agreement on “Global vs. Personal” classification, our annotation study also shows disagreement among experts: experts unanimously agree on only 64% of edits for “Global vs. Personal”, in contrast to 84% for “Clinical vs. Non-Clinical” on the same edits (Section 5, lines 319–325). These results suggest that determining “Global vs. Personal” is more subjective and may partly explain the difficulty in classifying edits along this axis. We explicitly acknowledge the limited reliability of our edit tagger for “Global vs. Personal” in line 325: “We thus treat Global vs. Personal tags as noisier signal than those on the other two axes.”