acceptodds
Under review as a conference paper at ICLR 2027

Actionable Feedback: Improving Model Capabilities From User Edit Data on Clinical Documentation

Abstract

Large language models (LLMs) are increasingly being deployed in clinical documentation workflows, where clinicians review and edit their outputs before use. In principle, these clinician edits represent a scalable source of supervision for model training. In practice, not all edits represent actionable feedback: for instance, on data drawn from real-world deployment of an AI clinical documentation product, our automated review suggests that over 45% of edited outputs contain user-added information that is not directly supported by the inputs to the model. Naively fine-tuning on such edits risks teaching the model to hallucinate. In this paper, we develop a method to address this challenge. Given that real clinician edits on deployed documentation tools are not publicly available, we first develop a taxonomy of clinician-editing behaviors, informed by data from a deployed AI clinical documentation product. Then, we develop a method for generating semi-synthetic clinician-edit data, informed by our taxonomy, on a discharge-summary generation task based on MIMIC-III, a public dataset. Finally, we introduce simple filtering methods for identifying actionable edits, with accuracy evaluated by expert review. We show empirically, on our semi-synthetic dataset, that naive fine-tuning on raw edited output increases ungrounded content, while our approach of progressively filtering toward grounded edits enables improvement on clinician-alignment metrics while retaining faithfulness to the input context.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI for healthcare, LLM-generated clinical documentation, clinician edit data

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_b85SOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper categorizes clinician edits and filters out unsupported or personal changes before fine-tuning clinical summarization models. Experiments on semi-synthetic edit data examine how filtering affects reference alignment and faithfulness to input records.

Critical strengths
  1. The paper addresses a practical problem: clinician edits may contain information unavailable to the model, making them unreliable training targets.

  2. The taxonomy is informed by real deployment data, and the semi-synthetic dataset supports controlled comparisons of different edit filters.

Critical weaknesses
  1. The contribution to AI and healthcare remains unclear. Better reference alignment does not establish clinical usefulness. “Without undesirable degradations” is too strong when Grounded + Global is less grounded in 4 of 10 cases. Treating personal edits as unsuitable supervision is also questionable, given 52% agreement with experts on this classification and evidence that some remain useful.

  2. The 90 human judgments cover only 10 admissions and exclude clinical usefulness, completeness, and error severity. A larger evaluation should measure these outcomes and report uncertainty. Training benefits also need validation using real clinician edits.

  3. [1] already uses synthetic clinical edits for training, and [2] corrects unsupported references before training clinical summarizers. The paper needs to explain what its taxonomy and filtering add.

  4. Two controls are needed: directly cleaning unsupported content from final edited drafts using the same LLM and training budget, and randomly reverting edits while matching edit count and target length. The current filtering-rate experiment only reverts edits already tagged ungrounded, so it does not isolate the benefit of targeted selection.

  5. The fine-tuned models are limited to Gemma-3-4B, Qwen-2.5-7B, and Llama-3.1-8B. I suggest adding a more recent, stronger open-weight model to test whether the findings generalize.

[1] Mishra, P., Yao, Z., Vashisht, P., Ouyang, F., Wang, B., Mody, V. D., & Yu, H. (2024, November). Synfac-edit: Synthetic imitation edit feedback for factual alignment in clinical summarization. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. 20061-20083).

[2] Adams, G., Shing, H. C., Sun, Q., Winestock, C., McKeown, K., & Elhadad, N. (2022, December). Learning to revise references for faithful summarization. In Findings of the Association for Computational Linguistics: EMNLP 2022 (pp. 4009-4027).

Questions for authors

Please refer to the weaknesses above.

5 comments
AuthorsOpenReview ↗

Response by Authors [1/4]

Dear Reviewer b85S,

Thank you for your feedback and suggestions! We address your concerns below:

“The contribution to AI and healthcare remains unclear. Better reference alignment does not establish clinical usefulness.”

We agree that greater reference alignment does not guarantee higher clinical utility, as we note in our Limitations section (Appendix H, lines 2169–2172). We have revised the relevant text to clarify this point in our updated PDF: “improving the clinical utility” →\rightarrow “improving the alignment… to the clinician-written reference summary” (Section 6, lines 346–347).

We use the reference alignment metrics to assess how well the model-generated BHCs capture the clinical details documented in clinician-written references, recognizing that this is only one aspect of summarization quality. Indeed, one of our main findings is that naively training on raw, unfiltered edits can improve reference alignment but degrade groundedness in the input EHR notes (Section 7.1). Our analysis explicitly distinguishes these properties rather than treating reference alignment as a complete measure of task performance.

Our contribution is a framework for characterizing clinician edits to LLM drafts and translating them into effective supervision. Such edits accumulate at scale in deployed clinical documentation systems, but alongside useful corrections, a clinician’s revision can include additions based on information unavailable to the model (i.e., ungrounded edit) and changes reflecting local documentation preferences (i.e., personal edit). Indeed, we estimate that over 45% of edited notes in our proprietary corpus from an ambient medical scribing product contain at least one ungrounded edit (Appendix A.2). An edit that is preferable for clinical use is therefore not necessarily one the model should learn to reproduce. Our taxonomy makes these distinctions explicit, and our edit filtering method enables selective training on edits that provide useful supervision. On our MIMIC-III-based dataset, filtering for grounded edits improves reference alignment while mitigating degradation in groundedness.

“‘Without undesirable degradations’ is too strong when Grounded + Global is less grounded in 4 of 10 cases. Treating personal edits as unsuitable supervision is also questionable, given 52% agreement with experts on this classification and evidence that some remain useful.”

Thank you for raising this point. We agree that the phrase “without undesirable degradations” in Figure 1 is stronger than intended. We have revised it to “while mitigating undesirable degradations” in our updated PDF.

We clarify that we do not regard personal edits as inherently unsuitable supervision. In Section 7.2 (lines 467–468), we note that some edits tagged as personal may still provide useful supervision, as suggested by the stronger reference-alignment results with the “Grounded” filter than with “Grounded + Global”. We consider removing personal edits since they reflect local preferences, and our goal is to improve the general BHC generation capabilities of the model. In other settings where the goal is to effectively adapt the model to local settings, personal edits may provide useful supervision, as we note in our Limitations section (Appendix H, lines 2174–2176).

Regarding the 52% tagger–expert agreement on “Global vs. Personal” classification, our annotation study also shows disagreement among experts: experts unanimously agree on only 64% of edits for “Global vs. Personal”, in contrast to 84% for “Clinical vs. Non-Clinical” on the same edits (Section 5, lines 319–325). These results suggest that determining “Global vs. Personal” is more subjective and may partly explain the difficulty in classifying edits along this axis. We explicitly acknowledge the limited reliability of our edit tagger for “Global vs. Personal” in line 325: “We thus treat Global vs. Personal tags as noisier signal than those on the other two axes.”

AuthorsOpenReview ↗

Response by Authors [2/4]

“The 90 human judgments cover only 10 admissions and exclude clinical usefulness, completeness, and error severity. A larger evaluation should measure these outcomes and report uncertainty. Training benefits also need validation using real clinician edits.”

First, we clarify that our experiments in Section 7 address a targeted question: how does training on raw vs. filtered edits impact the groundedness of the model-generated BHCs? The expert annotation study in Section 7.3 was designed to assess the impact on groundedness, rather than provide a comprehensive comparison across all dimensions of summarization quality. We agree that other aspects, such as completeness and error severity, are also relevant to assessing a model-generated BHC’s overall clinical utility. We explicitly acknowledge the need for such broader expert evaluation in our Limitations section (Appendix H, lines 2169–2172) and leave it as future work.

Second, assessing groundedness requires human experts to check clinical details in the model-generated BHCs against large collections of prior EHR notes. In our study, obtaining the 90 pairwise judgments required 30 hours of expert annotation, with at least 10 clinical notes per admission to reference (Appendix B.1). We agree that expert validation on a larger scale would provide more statistical power, but we leave it to future work given the substantial costs of expert annotation, as noted in the Limitations section (Appendix H, lines 2162–2163). Meanwhile, we report inter-annotator agreement for the annotation results in Appendix F.1 (63% unanimous agreement, 27% majority agreement).

Third, we agree that additional validation on real clinician-edit data would help establish how well our findings generalize to real-world editing workflows. As noted in our Limitations section (Appendix H, lines 2155–2163), while our semi-synthetic MIMIC-III-Ext-BHC-Edit dataset allows for controlled investigation of how edits can be used for model supervision, we do not claim that it can be used as a substitute for evaluation on real post-deployment edit data.

“[1] already uses synthetic clinical edits for training, and [2] corrects unsupported references before training clinical summarizers. The paper needs to explain what its taxonomy and filtering add.”

Thank you for highlighting these relevant works. We will include them in the Related Works section in our revision. However, we clarify that both Mishra et al. [1] and Adams et al. [2] address different problem settings.

Mishra et al. [1] study training models on synthetic edits made by frontier LLMs, in settings where clinician feedback is difficult to obtain at scale. These synthetic edits either make targeted improvements to drafts generated by a weaker LLM, or targeted degradations to clinician-written reference summaries to artificially construct edit pairs for preference learning. In both cases, the preferred version is known by construction. Adams et al. [2] propose training a model to revise unsupported sentences in clinician-written summaries, also with an emphasis on limited-data settings where simply removing training examples with noisy summaries is challenging.

In contrast, our work studies how to characterize, filter, and train models on edits made by clinicians, which can be collected at scale during their use of production clinical documentation systems. A key contribution of our work is showing that such edits should not be assumed to be improvements for the model to learn to reproduce. Our proposed taxonomy (Section 3.2) enables us to distinguish between different types of clinician edits (e.g., grounded vs. ungrounded, global vs. personal) and selectively use them as supervision signal. Applying our edit tagging + filtering method (Section 5) to these edits does not require any simulation of edits.

Given the limited availability of public clinician-edit datasets, we demonstrate the effectiveness of our framework on a semi-synthetic dataset based on MIMIC-III (Section 4). While the construction of this dataset involves simulating edits on top of LLM-drafted BHCs, every added or corrected detail is based on a real clinician-written BHC, and the types of simulated edits are informed by real-world clinician-editing behaviors observed in a deployed clinical documentation system.

AuthorsOpenReview ↗

Response by Authors [3/4]

“Two controls are needed: directly cleaning unsupported content from final edited drafts using the same LLM and training budget, and randomly reverting edits while matching edit count and target length. The current filtering-rate experiment only reverts edits already tagged ungrounded, so it does not isolate the benefit of targeted selection.”

To address your concerns, we provide additional results for the two suggested “direct cleaning” and “random reversion” baselines, using the same fine-tuning setup as in Section 6.

To implement the “direct cleaning” baseline, we consider two approaches for detecting ungrounded content in each post-edit BHC. First, we apply the off-the-shelf VeriFact pipeline by Chung et al. [3], which extracts the “atomic claims” from each BHC and labels each claim as grounded vs. ungrounded. Alternatively, we apply an adaptation of our GPT-5.5-based edit-tagger pipeline, which extracts the “atomic claims” and labels each claim as grounded vs. ungrounded. In either case, we then remove the ungrounded claims from the post-edit BHC, using our pipeline for constructing the grounded test set (Section 6, lines 358–363).

To implement the “random reversion” baseline, we randomly select and revert the same number of edits as the “Grounded” filter does for each edit pair. To account for randomness, we repeat this process three times with different random seeds. As reverting an edit can either shorten or lengthen the target (e.g., reverting an addition removes text, while reverting a deletion adds text), simultaneously matching the target lengths would require selectively reverting certain types of edits, rather than sampling uniformly from all edits. On average, the “random reversion” targets contain 234.3 words, while the “Grounded” targets contain 199.0 words, which are both much shorter than the targets before any filtering (305.3 words).

Below, we report the results for Qwen-2.5-7B-Instruct. We report both the reference-alignment and groundedness metrics as in Sections 6–7. For the “random reversion” baseline, we report the average across three seeds.

Reference Alignment

SettingDocLens-PDocLens-RMEDCON
Base0.27130.17360.2527
Raw0.30290.33120.3796
Direct Cleaning (GPT-5.5)0.33130.32310.3667
Direct Cleaning (VeriFact)0.31730.31550.3642
Random Reversion0.29420.31830.3482
Grounded (Ours)0.31980.32490.3710

Groundedness

SettingMedScoreDocLens-Attr.PDSQI-9
Base0.94780.90040.5992
Raw0.87420.75270.2357
Direct Cleaning (GPT-5.5)0.95520.86740.6193
Direct Cleaning (VeriFact)0.92900.82940.4323
Random Reversion0.89320.79170.3034
Grounded (Ours)0.95290.88040.5802

Our “Grounded” filter outperforms the VeriFact-based “direct cleaning” and “random reversion” baselines across all reference-alignment and groundedness metrics. Notably, randomly reverting edits still leads to a noticeable drop in groundedness (MedScore: 0.9478 →\rightarrow 0.8932) despite substantially shortening the target BHCs, while our “Grounded” filter retains similar levels of groundedness (MedScore: 0.9478 →\rightarrow 0.9529). These results demonstrate that targeted removal of ungrounded edits mitigates groundedness degradation from fine-tuning more effectively than randomly reverting the same number of edits.

GPT-5.5-based “direct cleaning” performs comparably to our “Grounded” filter. This is expected in our setting: the base model drafts are largely grounded, and nearly all ungrounded clinical content in the post-edit BHCs is introduced by the edits. Importantly, our goal is not to propose a better method for detecting ungrounded content, but to demonstrate that clinician edits need to be filtered appropriately for effective supervision. Filtering ungrounded edits is one concrete case study of our framework, which more generally enables filtering edits along other axes of our taxonomy.

Reviewer_YfRWOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Clinical summarization is an important task to potentially reduce the documentation burden of healthcare workers. LLMs are often used for summarization, but often clinician edits are not catered for appropriately. This paper tries to make use of step-by-step clinician edits for LLM fine-tuning. First, they use a proprietary dataset to create a taxonomy of clinican edits. Next, they use this to create a new synthetic dataset with such edits and show that fine-tuning on edit free summaries reduces unsupported facts generated from the LLM.

Critical strengths
  1. Very important task to improve evaluation of generated text in the clinical domain and make efective use of human edits. Many papers only treat the original and uneditited version as relevant artifact, but this work actually makes the effort to look into step-by-step edits.
  2. New dataset is created and potentially released publicly. The synthetic dataset created is probably made public and can be used to reproduce the results and conduct further research.
Critical weaknesses
  1. Taxonomy of clinical edits driven by private data set, ML models, and LLMs without any real-world signal of clinician intent or validation with healthcare professionals. As far as I understand it there is not true signal of clinician-intent. Instead, a proprietary dataset is used for which the pre- and post-edit versions exist. Then an LLM (GPT-4o) tries to reconstruct the original atomic changes and the original intent of the clinician while doing this change. Then topic modeling and a SAE is used to cluster these and another LLM (GPT-5-mini) is used to label these clusters. I think these are many unvalidated steps to reconstruct the original atomic edits and clinician intent. There is no validity that these reconstruction is correct and, hence, the downstream analysis based on it cannot be meaningfully interpreted.
  2. Contribution 4 was already shown in prior work for ungrounded facts that lead to hallucinations, which together with W1 limited the relevance of this work. In https://proceedings.mlr.press/v248/hegselmann24a.html a very similar analysis was performed for clinical summarization based on ungrounded facts in existing summaries. The paper showed that fine-tuning on the cleaned summaries reduces hallucinations, which is very similar to the fine-tuning without actionable edits performed in this work. The authors also released a dataset: https://physionet.org/content/ann-pt-summ/1.0.1/
Questions for authors

Clarifications:

  1. Why did you use MIMIC-III when MIMIC-IV-Notes is available which should be larger and richer?
  2. Please carefully describe the proprietary dataset used to develop the taxonomy? What kind of data is this, how was it collected, which patients are represented? All of this drives the downstream conclusions.
  3. How did you derive the taxonomy in 3.2 from 3.1? What were the full results in 3.1 and which process was used to come up with the taxonomy? This would be important to understand the validity of this introduced taxonomy.
  4. How does the human validation of the NPs validate that the LLM edits resemble real edits by clinicians? The NPs ought to check whether the edit fits in (Task 1) and whether the content is correct (Task 2), but there is no measurable way that this would be a reasonable edit by a human clincian.
  5. Can you release the full validation and annotation performed by the NPs?
  6. Will the dataset MIMIC-III-EXT-BHC-EDIT be released on PhysioNet?

Suggestions (future work):

  1. Overall, this is a very important research direction. However, to effectively use step-by-step edits any real-world signal supporting these step-wise edits and the intent would be necessary. Alternatively, the step-by-step changes could be discarded and the full unedited and editied version could be used.
  2. I think it is very useful to create new artifacts for the research community, but for the proposed dataset I see very little evidence that it resembles real-world edits performed by clinicians.
8 comments
AuthorsOpenReview ↗

Response by Authors [1/4]

Dear Reviewer YfRW,

Thank you for your feedback and suggestions! Before addressing your specific concerns, we would like to clarify a possible misunderstanding based on your summary:

“First, they use a proprietary dataset to create a taxonomy of clinician edits. Next, they use this to create a new synthetic dataset with such edits and show that fine-tuning on edit free summaries reduces unsupported facts generated from the LLM.”

We clarify that our method does not fine-tune on “edit free summaries”. Rather, we fine-tune on post-edit BHCs that retain only actionable edits (e.g., grounded edits), after reverting the non-actionable edits (e.g., ungrounded edits) using our edit filtering method (Section 5). Our experiments demonstrate that this approach helps improve reference alignment while mitigating degradation in groundedness.

Below, we address your specific concerns.

“Taxonomy of clinical edits driven by private data set, ML models, and LLMs without any real-world signal of clinician intent or validation with healthcare professionals. As far as I understand it there is not true signal of clinician-intent… I think these are many unvalidated steps to reconstruct the original atomic edits and clinician intent. There is no validity that these reconstruction is correct and, hence, the downstream analysis based on it cannot be meaningfully interpreted.”

“How did you derive the taxonomy in 3.2 from 3.1? What were the full results in 3.1 and which process was used to come up with the taxonomy? This would be important to understand the validity of this introduced taxonomy.”

We apologize for the potential confusion. First, in our caution to protect anonymity, we may have understated the role of the clinical team supporting this research. During the course of this work, we consulted clinical collaborators experienced with ambient medical scribing technology, and our edit taxonomy reflects their judgments. We may have inadvertently given an inflated sense of the role that the SAE and topic modeling (Section 3.1) played in constructing our taxonomy. These tools were leveraged for exploratory analysis, to confirm clinician intuition against edit patterns that surfaced organically from our empirical analysis of a large corpus of real clinician edits. We manually reviewed the resulting patterns against the underlying edit pairs and used selected patterns to construct our taxonomy (Section 3.2). However, due to the proprietary nature of the data, we are not able to share the fine-grained findings from this analysis. We have updated the PDF to explicitly clarify these points regarding the construction of our taxonomy (see Section 3.1, lines 178–181).

Second, we clarify that our downstream analysis and results do not depend on exact recovery of the clinicians’ intentions underlying the edits. Our atomic edit extraction step simply decomposes the textual changes in each edit pair into units that can be analyzed individually. The notion of “clinician intent” in Definition 1 and our prompts only serves as a heuristic for grouping semantically related textual changes into a single unit. We use these units to characterize and selectively filter the changes, without assuming that they match how the clinician originally grouped or conceived them. For example, assessing whether an added detail is supported by the input EHR context (i.e., grounded edit) does not require knowing precisely how and why the clinician made that change. For the downstream analysis, we separately assessed the accuracy of the edit tagger’s classifications by comparing its judgments against annotations from healthcare professionals (Section 5).

AuthorsOpenReview ↗

Response by Authors [2/4]

“Contribution 4 was already shown in prior work for ungrounded facts that lead to hallucinations, which together with W1 limited the relevance of this work… a very similar analysis was performed for clinical summarization based on ungrounded facts in existing summaries. The paper showed that fine-tuning on the cleaned summaries reduces hallucinations, which is very similar to the fine-tuning without actionable edits performed in this work. The authors also released a dataset”

Thank you for pointing us to this reference. We agree that Hegselmann et al. [1] demonstrate that removing unsupported content from training references can reduce hallucinations. However, we clarify that our work addresses a different problem.

Our contribution is in proposing a new framework for characterizing clinician edits to LLM drafts and translating them into effective supervision, rather than a new method for reducing hallucinations. Prior works on learning from user edits often treat user-edited outputs as improved targets [2,3]. In clinical documentation, alongside useful corrections, a clinician’s revision can include additions based on information unavailable to the model (i.e., ungrounded edit) and changes reflecting local documentation preferences (i.e., personal edits). Consequently, an edit that is preferable for clinical use is not necessarily an edit the model should learn to reproduce. Our taxonomy makes these distinctions explicit, and our MIMIC-III-based semi-synthetic testbed enables controlled investigation of their training consequences.

Our experiments in Sections 6–7 explore groundedness as a concrete case study for applying our framework. We examine the consequences of naively training on clinician edits in their entirety vs. selectively training on a subset of edits, including whether useful supervision can be preserved while excluding edits unsupported by the model’s inputs. Our results provide empirical support for selectively learning from clinician edits, and the benefit of excluding ungrounded edits aligns with findings from prior works such as Hegselmann et al. [1].

Our dataset also serves a different purpose. Our MIMIC-III-Ext-BHC-Edit dataset provides edits to LLM-generated BHC drafts for studying the consequences of selectively training on those edits. Hegselmann et al. [1] provide unsupported-fact annotations and cleaned reference summaries. The two datasets thus support related but distinct investigations of how clinical documentation should be used for model training.

“Why did you use MIMIC-III when MIMIC-IV-Notes is available which should be larger and richer?”

We used MIMIC-III because its greater diversity of longitudinal clinical notes provides richer source context for BHC generation and more closely reflects the documentation available in real-world production EHR systems. MIMIC-IV-Notes is a larger dataset but only includes discharge summaries and radiology reports. In contrast, MIMIC-III covers a greater diversity of clinical notes, also including e.g., nursing notes, cardiology reports, physician notes (e.g., admission and progress notes), and respiratory care notes. These additional notes also provide evidence for assessing the groundedness of edits.

Recent work such as VeriFact [4] (NEJM AI, 2026), which evaluates the groundedness of LLM-generated and clinician-written BHCs, also explicitly cites the availability of longitudinal notes as its reason for choosing MIMIC-III over MIMIC-IV (see “Dataset Creation” in the “Methods” section of [4]).

We also note that Hegselmann et al. [1], who use MIMIC-IV-Notes, consider a different task: summarizing an existing BHC section to generate the patient-facing Discharge Instructions (DI) section within the same discharge summary. Our task instead generates the BHC section itself from prior clinical notes recorded throughout a patient’s hospitalization, making MIMIC-III’s broader coverage of longitudinal notes particularly relevant.

AuthorsOpenReview ↗

Response by Authors [3/4]

“Please carefully describe the proprietary dataset used to develop the taxonomy? What kind of data is this, how was it collected, which patients are represented? All of this drives the downstream conclusions.”

As described in Section 3.1, the proprietary dataset consists of 30,000 de-identified and anonymized clinician–patient encounters between 2023 and 2026, spanning over 50 specialties across multiple US institutions, with edits recorded through routine clinical use of a deployed ambient medical scribing product. For each encounter, the input is a transcript of the clinician–patient conversation, and the output is the Assessment & Plan (A&P) section of the corresponding SOAP note. We have both the initial LLM-generated draft and the clinician-edited final version. Due to proprietary restrictions, we are unable to disclose further cohort details.

While the specifics of the cohort may affect what edit patterns we observe, the taxonomy axes we define capture broad distinctions that are relevant across specialties and patient populations, such as whether an edit changes clinical content (i.e., clinical edit) and whether the information it adds is supported by the model’s inputs (i.e., grounded edit). Our edit tagger, filtering method, and experiments are based on these definitions, and not directly dependent on the specific composition of the proprietary cohort. We also validate the edit tagger (Section 5) and the groundedness of model-generated BHCs (Section 7.3) against annotations from healthcare professionals, and conduct our main experiments using MIMIC-III. Thus, we believe that our downstream conclusions are not highly sensitive to the specifics of the proprietary dataset.

“Will the dataset MIMIC-III-EXT-BHC-EDIT be released on PhysioNet?” “Can you release the full validation and annotation performed by the NPs?”

Yes, we plan to share our MIMIC-III-Ext-BHC-Edit dataset and the nurse practitioner (NP) annotations on PhysioNet for the camera-ready version.

“Overall, this is a very important research direction. However, to effectively use step-by-step edits any real-world signal supporting these step-wise edits and the intent would be necessary. Alternatively, the step-by-step changes could be discarded and the full unedited and editied version could be used.”

As clarified in our response to Weakness 1 above, our analysis does not require exact recovery of the clinicians’ original sequence of editing actions or underlying intentions. Our approach already uses both the full pre-edit and post-edit BHCs, decomposing their textual differences into units that can be analyzed individually (i.e., atomic edits), without assuming that these units match how the clinician originally grouped or conceived them. This decomposition enables us to selectively retain or revert changes within the same document—e.g., retaining a correction supported by the model’s inputs (i.e., grounded) while reverting an unsupported addition (i.e., ungrounded).

Reviewer_EDGcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper studies how to learn from clinician edits to LLM-generated clinical documentation when some edits introduce information unavailable in the model’s inputs.

  • It proposes a taxonomy of edits, constructs a semi-synthetic discharge-summary editing dataset informed by real deployment data, and uses an LLM-based tagger to selectively revert unsupported or personal edits before fine-tuning.

Critical strengths
  • Experiments across three model families show that training on raw edited summaries improves reference alignment while degrading groundedness. Filtering unsupported edits substantially mitigates this degradation while retaining much of the alignment improvement.

  • The paper identifies an important distinction in learning from expert feedback

Critical weaknesses
  • The comparisons establish that filtering improves over training on raw targets, but do not show whether extracting atomic edits and inferring their intent offers an advantage over directly identifying and removing unsupported claims from the final summaries. A comparison using the same verification model and source notes would clarify

  • The Global/Personal tagger agrees with expert labels on only 13/25 edits (Section 5). And limited support for presenting progressively stricter filtering as reliably beneficial

Questions for authors
  • How does the proposed approach compare with directly filtering unsupported claims from post-edit summaries, using the same verifier and input evidence but without atomic-edit extraction or intent classification?

  • What evidence supports removing personal edits beyond groundedness filtering?

  • Appendix E.2 gives the base model an explicit system instruction to remain grounded, does the observed degradation relative to the base model persist under a matched inference prompt?

3 comments
AuthorsOpenReview ↗

Response by Authors [1/2]

Dear Reviewer EDGc,

Thank you for your feedback and suggestions! We address your concerns below:

“The comparisons establish that filtering improves over training on raw targets, but do not show whether extracting atomic edits and inferring their intent offers an advantage over directly identifying and removing unsupported claims from the final summaries. A comparison using the same verification model and source notes would clarify.” “How does the proposed approach compare with directly filtering unsupported claims from post-edit summaries, using the same verifier and input evidence but without atomic-edit extraction or intent classification?”

To address your concerns, we provide additional comparisons against “direct filtering” baselines, which use the same input EHR notes to directly identify and filter out ungrounded claims. We follow the same fine-tuning setup as in Section 6.

To implement this baseline, we adapt our GPT-5.5-based edit-tagger pipeline to extract “atomic claims” (as defined in VeriFact [1]) directly from each post-edit BHC and label each claim as grounded vs. ungrounded against the input EHR notes. We then remove the ungrounded claims from the post-edit BHCs, using the pipeline previously used to construct the grounded test set (Section 6, lines 358–363). We also consider an alternative implementation, which uses the off-the-shelf VeriFact pipeline to detect ungrounded claims, and removes them via the same process.

Below, we report the results for Qwen-2.5-7B-Instruct. We report both the reference-alignment and groundedness metrics as in Sections 6–7.

Reference Alignment

SettingDocLens-PDocLens-RMEDCON
Base0.27130.17360.2527
Raw0.30290.33120.3796
Direct Filtering (GPT-5.5)0.33130.32310.3667
Direct Filtering (VeriFact)0.31730.31550.3642
Grounded (Ours)0.31980.32490.3710

Groundedness

SettingMedScoreDocLens-Attr.PDSQI-9
Base0.94780.90040.5992
Raw0.87420.75270.2357
Direct Filtering (GPT-5.5)0.95520.86740.6193
Direct Filtering (VeriFact)0.92900.82940.4323
Grounded (Ours)0.95290.88040.5802

Our “Grounded” filter scores higher than the VeriFact-based “direct filtering” baseline across all six metrics and performs comparably to the GPT-5.5-based “direct filtering” baseline. The latter is expected, as both approaches use the same LLM to detect ungrounded content, and the base model drafts are largely grounded.

We highlight that our goal is not to propose a better method for detecting ungrounded content. Rather, our main contribution is a framework for characterizing clinician edits to LLM drafts and studying how to effectively translate them into useful supervision. Even when the edited document is preferable for clinical use, our work illustrates that not every change may reflect an improvement that the model should learn to reproduce. Our taxonomy makes these distinctions explicit, and our groundedness experiments (Sections 6–7) serve as one concrete case study of our framework.

“The Global/Personal tagger agrees with expert labels on only 13/25 edits (Section 5). And limited support for presenting progressively stricter filtering as reliably beneficial.”

“What evidence supports removing personal edits beyond groundedness filtering?”

We agree that progressively stricter filtering is not universally beneficial, but rather presents a trade-off: applying the “Grounded + Global” filter generally results in comparable or higher groundedness scores (Qwen-2.5-7B-Instruct, Llama-3.1-8B-Instruct), while applying the “Grounded” filter leads to better reference alignment (Figure 3). These results suggest that some edits tagged as personal may still provide useful supervision, as we acknowledge in Section 7.2 (lines 467–468).

We considered removing personal edits since our goal is to improve general BHC generation capabilities rather than adapt to local documentation preferences (Section 5, lines 339–340). In other settings where the goal is to effectively adapt the model to local settings, personal edits may provide useful supervision, as we note in our Limitations section (Appendix H, lines 2174–2176).

Regarding the 52% (13/25) tagger–expert agreement on “Global vs. Personal” classification, our annotation study also shows disagreement among experts: experts unanimously agree on only 64% of edits for “Global vs. Personal”, in contrast to 84% for “Clinical vs. Non-Clinical” on the same edits (Section 5, lines 319–325). These results suggest that determining “Global vs. Personal” is more subjective and may partly explain the difficulty of classifying these edits. We explicitly acknowledge the limited reliability of our edit tagger for “Global vs. Personal” in line 325: “We thus treat Global vs. Personal tags as noisier signal than those on the other two axes.”

AuthorsOpenReview ↗

Response by Authors [2/2]

“Appendix E.2 gives the base model an explicit system prompt instruction to remain grounded, does the observed degradation relative to the base model persist under a matched inference prompt?”

Yes, even under the matched inference prompt setup, we observe degradations in groundedness when we naively train on the raw post-edit BHCs.

Below, we provide additional results for Gemma-3-4B-IT and Qwen-2.5-7B-Instruct, where we include the system prompt instruction to stay grounded at inference time for both the base model and the fine-tuned models. We report both the reference-alignment and groundedness metrics as in Sections 6–7.

Gemma-3-4B-IT

Reference Alignment

SettingDocLens-PDocLens-RMEDCON
Base0.20000.17250.2300
Raw0.28220.33570.3709
Grounded (Ours)0.27660.32560.3549
Grounded + Global (Ours)0.22400.25730.3033

Groundedness

SettingMedScoreDocLens-Attr.PDSQI-9
Base0.92860.86400.3297
Raw0.89990.80040.2860
Grounded (Ours)0.93220.86770.5249
Grounded + Global (Ours)0.92400.86600.4307

Qwen-2.5-7B-Instruct

Reference Alignment

SettingDocLens-PDocLens-RMEDCON
Base0.27130.17360.2527
Raw0.28210.33700.3792
Grounded (Ours)0.33800.31000.3666
Grounded + Global (Ours)0.36250.26120.3172

Groundedness

SettingMedScoreDocLens-Attr.PDSQI-9
Base0.94780.90040.5992
Raw0.88220.76980.2824
Grounded (Ours)0.95830.87870.6508
Grounded + Global (Ours)0.96150.91890.6975

For both models, we observe that fine-tuning on raw post-edit BHCs consistently degrades groundedness relative to the base model, even under the matched inference prompts. In contrast, fine-tuning on our filtered post-edit BHCs matches or exceeds the base model on most groundedness metrics.

References:

[1] “Verifying Facts in Patient Care Documents Generated by Large Language Models Using Electronic Health Records” (Chung et al., 2026)

AuthorsOpenReview ↗

Gentle Reminder of the Discussion Period Deadline

Dear Reviewer EDGc,

Thank you once again for your time and effort in providing a thoughtful review of our work. We would like to send a kind reminder about the discussion period deadline, which is in about 1 day.

We understand that you have a busy schedule, and we sincerely appreciate your time. Should you have any additional questions or concerns, we are more than happy to address them.

Thank you, and we look forward to hearing from you.

Best, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.