acceptodds
Under review as a conference paper at ICLR 2027

ECG-RePAIR: Agentic Test-Time Repair with Evidence Grounding for ECG Interpretation

Abstract

Electrocardiography (ECG) is an essential diagnostic tool, where reliable interpretation requires clinical findings to be grounded in the underlying waveform. Recent multimodal large language models (MLLMs) have advanced automated ECG interpretation by integrating signal and image information, improving report completeness and diagnostic reasoning. However, these models still largely rely on one-pass generation, allowing uncertain or weakly grounded findings to persist in the final interpretation. In routine clinical practice, such findings are often revisited on the ECG waveform and reassessed against diagnostic criteria before an interpretation is finalized. Inspired by this practice, we propose ECG-RePAIR, an agentic test-time repair paradigm that organizes post-generation interpretation as a structured evidence-to-action process. ECG-RePAIR uses criterion-guided retrieval-augmented generation to determine what evidence should be examined, ECG tools to verify claims against the current waveform, and a repair advantage memory to determine whether the verified evidence warrants modifying the report. Extensive experiments show that ECG-RePAIR consistently improves diagnostic accuracy and report quality, while ablation studies and clinical expert evaluation further support its effectiveness.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
ECG, Agentic AI, Clinical AI, ECG interpretation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oXszOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ECG-RePAIR revisits an already generated ECG report in three stages:

  • Extract addressable claims and retrieve the relevant clinical criteria.
  • Measure the current waveform with two tool backends and classify the resulting evidence as supporting, contradicting, or uncertain.
  • Use an evidence-constrained action mechanism and a repair-advantage estimator to retain, add, or revise a finding.

The evaluation on ECG-Grounding combines component ablations, alternative base/evaluator models, and clinician assessment of sampled repaired reports.

Critical strengths
  • The separation between deciding what evidence is needed, obtaining that evidence from the waveform, and deciding whether to modify the report is well motivated. The explicit uncertain outcome and local repair mechanism are useful design choices for avoiding unsupported, broad report changes.

  • The paper provides multiple complementary checks of its approach, including tool and memory ablations, a different base model, a different evaluator, and independent clinician assessments. The reported improvement over ECG-R1 and the clinician findings make this a useful contribution to evidence-grounded ECG interpretation.

Taken together, these results and design choices lead me to a positive assessment and weak acceptance.

Critical weaknesses

The scope of the cross-fitted repair-memory results and the reliance on rubric-based automated report scoring are the main limitations to keep in mind. Human assessment and the other supporting analyses are relevant here. My remaining requests concern explaining the protocol and the scope of its conclusions, rather than identifying grounds for rejection.

Questions for authors

Q1. Are the five folds used for memory construction and action estimation patient-disjoint, including repeated ECGs? Please explain the isolation of fold-specific fitting and threshold selection, and distinguish this cross-fitted setting from deployment with a fixed memory on an independent cohort.

Q2. Please state the exact report-utility function J used to construct repair advantages and its relationship to the final evaluation metrics and judge. This is important for interpreting the incremental contribution of the memory-based decision rule.

Q3. Could you explain how shared errors between the two measurement backends are handled, especially near diagnostic thresholds? Please also clarify that the clinician sample evaluates the quality of executed repairs, rather than the completeness of detecting all errors in unchanged reports.

4 comments

Thank you for your positive assessment and constructive suggestions. We appreciate your recognition of ECG-RePAIR's evidence-grounded design, clinical relevance, and complementary evaluations. We have conducted additional experiments and analyses to address your concerns.

Response to W1 & Q1: Cross-fitted evaluation and independent repair memory

Patient-level isolation. The original five folds were partitioned by ECG record ID rather than patient ID. Therefore, repeated ECGs from the same patient could appear in different folds, although each ECG record was strictly excluded from its own memory construction and parameter selection.

Fold-specific fitting and threshold selection. For each held-out fold, memory construction, estimator fitting, and selection of the ridge coefficient and decision margin exclusively used the remaining four folds. The held-out fold was used only for test-time repair and evaluation, with the memory remaining read-only. Thus, the cross-fitted setting uses experience from other records within the benchmark, whereas independent-cohort deployment requires a fixed memory developed outside the test cohort.

Deployment with independent memory. Following the reviewer's suggestion, we constructed an independent Repair Advantage Memory exclusively using 27,185 ECGs from the ECG-R1 training set, with no patient overlap with the evaluation cohort. We obtained 6,128 tool-verified candidate repair actions, using 4,872 for estimator fitting and 1,256 for internal validation. The memory, estimator, and all parameters were fixed before evaluating the 2,380 ECG-Grounding test records.

MethodDiag. Acc.CreditCompl.Relev.LeadGround.Reason.Fidelity
ECG-R176.2468.654.625.785.8271.4171.3772.16
ECG-RePAIR (w/o Memory)79.7673.885.166.186.3174.6274.4874.71
ECG-RePAIR (Cross-fitted Memory)80.4974.945.246.336.4275.0975.0475.32
ECG-RePAIR (Independent Memory)81.1775.335.346.526.5976.0875.9375.91

Independent Memory improves Diagnosis Accuracy by 4.93 points over ECG-R1 and 1.41 points over the memory-free variant.

ECG-RePAIR achieves its best performance with a fixed repair memory and all parameters developed entirely outside the evaluation cohort, directly addressing the distinction between cross-fitted evaluation and independent-cohort deployment.

Response to W2 & Q2: Report-utility function and rubric-based evaluation

Exact report-utility function. J(y)J(y) is the per-report Diagnosis Accuracy score (0–100) assigned by DeepSeek-v4-flash-0731 under the original ECG-R1 rubric, corresponding to diagnosis_accuracy_binary in our implementation. Each diagnosis receives 2 points if correct and supported, 1 if largely correct but insufficiently supported, and 0 if incorrect or unsupported:

J(y) = 100 × (number of diagnoses with score > 0) / (total number of scored diagnoses).

The observed repair advantage is:

Δi=J(yiai)−J(yi),\Delta_i=J(y_i^{a_i})-J(y_i),

where yiy_i and yiaiy_i^{a_i} denote the reports before and after executing action aia_i, respectively. These differences serve as regression targets without additional metric weighting or normalization.

Relationship to the final evaluation metrics and judge. Diagnosis Accuracy is the offline optimization target and one of the eight final evaluation metrics, evaluated using the same DeepSeek judge and rubric. The remaining seven metrics are not directly optimized. No LLM judge is invoked during test-time repair. The estimator uses only current state–action features, and candidate repairs must satisfy the evidence-based action filter.

Incremental contribution of memory. Cross-fitted Memory improves Diagnosis Accuracy by 0.73 points over the memory-free variant, with a patient-level paired 95% confidence interval of [0.28, 1.18]. Independent Memory further increases this improvement to 1.41 points. Both memory configurations improve all eight evaluation metrics.

Judge-independent evaluation. To address the reliance on rubric-based automated report scoring, we additionally evaluated diagnostic Precision, Recall, and F1 using a fixed diagnostic vocabulary and identical concept-matching rules against reference diagnostic labels, without an LLM judge. Unmatched predicted diagnoses were counted as false positives.

MethodPrecisionRecallF1
ECG-R165.7570.7968.18
ECG-RePAIR (Cross-fitted Memory)71.8673.4272.65
ECG-RePAIR (Independent Memory)72.1473.7672.98

The simultaneous improvements in Precision (+6.39), Recall (+2.97), and F1 (+4.80) provide independent evidence of diagnostic improvement beyond rubric-based automated report scoring.

Response to Q3: Shared measurement errors, diagnostic thresholds, and clinical evaluation

Handling shared measurement errors. ECG-RePAIR uses NeuroKit2- and WFDB/XQRS-based measurement paths with predefined agreement tolerances:

MeasurementAgreement tolerance
Heart rate8 bpm
RR interval40 ms
PR interval30 ms
QRS duration20 ms
QT interval35 ms
QTc interval40 ms
ST amplitude0.08 mV
T-wave amplitude0.12 mV

Backend agreement alone cannot eliminate shared systematic errors. Therefore, ECG-RePAIR additionally checks signal quality, clinical prerequisites, and applicable exclusions before authorizing a repair. For example, first-degree AV block requires both reliable PR prolongation and a valid P–QRS relationship.

Handling measurements near diagnostic thresholds. Ambiguous evidence near diagnostic thresholds can trigger an Uncertain verdict, preventing automatic repair. The distance from the diagnostic threshold also informs the repair advantage estimator, enabling threshold-sensitive repair decisions.

We further evaluated 100 ECGs under baseline wander, electromyographic noise, power-line interference, and missing-lead conditions. Repair execution rates remain similar under noise perturbations (26–29% versus 28% on clean ECGs) and drop to 0% when Lead II is missing. These results support stable repair activity and uncertainty-aware abstention when essential waveform evidence is unavailable.

Scope of the original clinical evaluation. The original 50-case clinical evaluation specifically assessed the quality of executed repairs. It did not evaluate the completeness of detecting errors in unchanged reports.

To investigate both aspects, we expanded the clinical evaluation to ​**200 cases (100 repaired and 100 unchanged)**​. Two clinicians independently assessed anonymized and randomized reports against the original ECG waveforms, with method identities and before/after ordering concealed for repaired cases.

GroupMetricClinician AClinician B
Repaired (N = 100)ECG-supported modifications88%90%
Clinical improvement79%82%
No new errors87%89%
Unchanged (N = 100)Overall correctness79%76%
Missed clinical findings16%19%
Unsupported diagnoses8%10%
Cannot judge5%5%

Cohen's κ was 0.68 for clinical improvement and 0.73 for new-error assessment, indicating substantial inter-rater agreement. The unchanged-report assessment categories are not mutually exclusive.

​The expanded clinical evaluation directly examines both the quality of executed repairs and potential missed corrections in unchanged reports​, providing broader clinical evidence while remaining a sample-based assessment.

Given the time-intensive nature of expert assessment, we expanded the evaluation to 200 cases during the rebuttal period, twice ECG-R1's clinical evaluation sample size, and plan further expansion.

Thank you again for your positive assessment and valuable suggestions. We have incorporated these additional experiments, the updated Clinician Review Form, and methodological clarifications into the revised manuscript.

Thank you for taking the time to address my questions. The new evaluation with an independently constructed, patient-disjoint repair memory resolves my main concern about the evaluation setup. The judge-independent diagnostic results and the expanded clinical assessment are also helpful, particularly the inclusion of unchanged reports.

I am satisfied with these responses. They provide stronger support for the method and clarify how it would be used with a fixed memory at deployment. Please include the additional results and protocol details in the revised paper. I maintain my positive recommendation.

AuthorsOpenReview ↗

Reply Rebuttal Comment by Authors

Thank you very much for your valuable feedback! We sincerely appreciate the time and effort you devoted to reviewing our manuscript and providing constructive comments and suggestions. We will incorporate these additional results and protocol details into the revised manuscript.

— Authors

Reviewer_4jugOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ECG-RePAIR, an agentic test-time framework for improving ECG interpretation through evidence-grounded report correction. The method consists of three stages: ECG-aware planning with criterion-guided RAG, waveform verification using ECG measurement tools, and memory-conditioned repair based on historical action advantages.

Experiments on 2,380 ECG records demonstrate improvements in diagnostic accuracy and report quality over ECG-R1. The paper also includes ablation studies, evaluations with an alternative base model and judge, and clinical assessments by two experts.

Critical strengths

Clinically motivated problem formulation. The paper addresses the important problem of correcting unreliable ECG interpretations through waveform-grounded verification.

Well-structured framework. The integration of criterion-guided planning, ECG measurement tools, and memory-conditioned repair provides a systematic and interpretable workflow.

Promising empirical improvements. The method improves diagnostic accuracy and report quality, supported by ablation studies and preliminary clinical expert evaluations.

Critical weaknesses

Limited methodological novelty. The framework primarily integrates existing RAG techniques, ECG signal-processing tools, and memory-based decision mechanisms, with limited algorithmic innovation.

Concerns about the evaluation protocol. The repair advantage memory and decision parameters are learned from other folds of the designated test set. Although out-of-fold evaluation prevents direct record-level reuse, independent test-set validation is still needed.

Insufficient comparison with simpler baselines. The paper lacks strong comparisons against lightweight rule-based ECG correction and alternative tool-assisted post-processing approaches.

Heavy reliance on LLM-based evaluation. Most reported improvements depend on rubric-based LLM judges. Although an alternative judge is included, independent clinical validation remains limited.

Limited clinical expert evaluation. Only 50 repaired cases are assessed by two clinicians, with 74% and 78% considered improved. This provides insufficient evidence of reliable clinical benefits across diverse cases.

Limited coverage of actual report correction. Appendix B shows that the vast majority of repair actions are additions, with only two revisions and no recorded ST–T/ischemia repair actions. This raises questions about the framework's ability to correct diverse diagnostic errors.

Insufficient generalizability. Experiments are conducted on a single ECG dataset, with limited evaluation across hospitals, patient populations, and challenging signal-quality conditions.

Questions for authors

Methodological novelty: What distinguishes ECG-RePAIR from a simpler pipeline combining clinical rules, ECG measurement tools, and deterministic report correction?

Evaluation protocol: Could the authors evaluate the method on an independent test dataset, with the repair advantage memory constructed exclusively from separate training data?

Component contributions: How does ECG-RePAIR compare with a rule-based repair baseline using the same waveform measurements but without RAG or memory-based action selection?

Clinical reliability: Why does the framework perform almost exclusively Add actions rather than Revise actions? Can it effectively correct incorrect diagnoses, particularly ST–T abnormalities?

Generalizability: How robust is ECG-RePAIR to noisy ECG signals, different patient populations, and datasets collected from other clinical institutions?

3 comments

Thank you for your constructive comments. We conducted additional experiments to address your concerns.

Response to W1 & Q1: Methodological novelty

To the best of our knowledge, ECG-RePAIR is the first ECG-specific agentic framework that unifies criterion-guided retrieval, waveform verification, and experience-driven report repair. We address an important yet underexplored problem: how to correct unreliable clinical conclusions after initial report generation using verifiable waveform evidence.

We introduce three task-specific methodological designs:

  • Criterion-Guided RAG: Transforms clinical knowledge into executable evidence plans specifying relevant leads, measurements, thresholds, and clinical prerequisites.
  • Evidence-Grounded Verification: Introduces multi-tool agreement and three-way verification to construct reliable waveform evidence and handle clinical uncertainty.
  • Repair Advantage Memory: Learns historical state–action advantages to estimate the benefit of candidate repairs and enable selective correction.

These designs adapt and extend existing techniques for ECG report repair, as supported by ablations and controlled comparisons.

With Independent Memory, ECG-RePAIR achieves 81.17 Diagnosis Accuracy, outperforming ECG-R1, rule-based repair, and tool-assisted base-model revision by 4.93, 2.71, and 3.81 points, respectively, while achieving the best performance across all eight metrics.

Our novelty lies in the ECG-specific evidence-to-action architecture connecting these three mechanisms.

Response to W2 & Q2: Evaluation protocol

Our evaluation follows ECG-R1's ECG-Grounding protocol. The original five-fold protocol constructs repair memory with record-level out-of-fold isolation.

We constructed an independent Repair Advantage Memory using only 27,185 ECGs from the ECG-R1 training set, with no patient overlap with the evaluation cohort. Of 6,128 tool-verified candidate repair actions, 4,872 were used for estimator fitting and 1,256 for internal validation. All parameters were fixed before evaluating the 2,380 ECG-Grounding test records.

MethodDiag. Acc.CreditCompl.Relev.LeadGround.Reason.Fidelity
ECG-R176.2468.654.625.785.8271.4171.3772.16
ECG-RePAIR (w/o Memory)79.7673.885.166.186.3174.6274.4874.71
ECG-RePAIR (Cross-fitted Memory)80.4974.945.246.336.4275.0975.0475.32
ECG-RePAIR (Independent Memory)81.1775.335.346.526.5976.0875.9375.91

Independent Memory achieves 81.17 Diagnosis Accuracy (+4.93 over ECG-R1). With independent training data, the memory gain rises from +0.73 to +1.41 points over the memory-free variant, with further improvements across all eight metrics.

The best results are achieved with memory construction and parameter selection entirely outside the evaluation cohort, addressing independent test-set validation.

Response to W3 & Q3: Comparison with simpler baselines and component contributions

We evaluated two additional tool-assisted baselines using the same waveform measurements:

  • Measurement + Rules (rule-based repair): Uses fixed clinical rules and the same local executor, without RAG or memory-based action selection.
  • Measurement + ECG-R1 (tool-assisted post-processing): Provides the same measurements to ECG-R1 for direct report revision.
MethodDiag. Acc.CreditCompl.Relev.LeadGround.Reason.Fidelity
ECG-R176.2468.654.625.785.8271.4171.3772.16
Measurement + Rules78.4673.275.086.046.1573.1273.9874.48
Measurement + ECG-R177.3668.274.796.205.7173.7270.7573.58
w/o RAG79.3173.425.026.086.1674.2174.0874.35
w/o ECG Tools78.2271.834.815.845.5772.1472.3172.68
w/o Memory79.7673.885.166.186.3174.6274.4874.71
ECG-RePAIR (Cross-fitted Memory)80.4974.945.246.336.4275.0975.0475.32
ECG-RePAIR (Independent Memory)81.1775.335.346.526.5976.0875.9375.91

ECG-RePAIR with Cross-fitted Memory outperforms rule-based repair and tool-assisted base-model revision by 2.03 and 3.13 points in Diagnosis Accuracy, respectively, with improvements across all eight metrics.

Removing RAG, ECG tools, and memory reduces Diagnosis Accuracy by 1.18, 2.27, and 0.73 points, with patient-level paired 95% CIs of [0.58, 1.78], [1.63, 2.91], and [0.28, 1.18], respectively, all excluding zero.

These comparisons support our agentic architecture beyond simpler pipelines and its component contributions.

Response to W4: Heavy reliance on LLM-based evaluation

We conducted a judge-independent diagnostic evaluation of Precision, Recall, and F1 against reference diagnostic labels, using a fixed diagnostic vocabulary and identical concept-matching rules without an LLM judge. Unmatched predicted diagnoses were counted as false positives.

MethodPrecisionRecallF1
ECG-R165.7570.7968.18
ECG-RePAIR (Cross-fitted Memory)71.8673.4272.65
ECG-RePAIR (Independent Memory)72.1473.7672.98

With Independent Memory, ECG-RePAIR improves Precision, Recall, and F1 by 6.39, 2.97, and 4.80 points, respectively, demonstrating improved diagnostic completeness and precision independently of the LLM-based evaluation rubric.

Response to W5: Limited clinical expert evaluation

We expanded clinical evaluation from 50 to 200 cases (100 repaired and 100 unchanged). Two clinicians independently assessed anonymized, randomized reports against the original ECG waveforms, with method identities and before/after ordering concealed for repaired cases.

GroupMetricClinician AClinician B
Repaired (N=100)ECG-supported modifications88%90%
Clinical improvement79%82%
No new errors87%89%
Unchanged (N=100)Overall correctness79%76%
Missed clinical findings16%19%
Unsupported diagnoses8%10%
Cannot judge5%5%

Most repairs were ECG-supported and clinically beneficial, with few new errors. Cohen's κ was 0.68 for clinical improvement and 0.73 for new-error assessment, indicating substantial inter-rater agreement.

Most unchanged reports were judged clinically correct, supporting selective repair. Missed or unsupported findings indicate opportunities to improve repair coverage. These assessment categories are not mutually exclusive.

Given the time-intensive nature of expert assessment, we expanded the evaluation to 200 cases during rebuttal, twice ECG-R1's clinical evaluation sample size, and plan further expansion. These results and the updated Clinician Review Form are included in the revised manuscript.

Response to W6 & Q4: Actual report correction and ST–T abnormalities

Clarification of Add and Revise. Both actions are defined at the diagnostic concept level. Add introduces a missing diagnostic concept but may also replace or delete incorrect statements in the report body, whereas Revise modifies an existing diagnostic concept.

Both cases in Appendix C demonstrate actual correction: Case 1 corrects the normal-PR interpretation and identifies first-degree AV block; Case 2 corrects the normal-axis interpretation and identifies left-axis deviation.

ActionActionsReportsExisting content replaced/deleted
Add591548203 (37.0%)
Revise220
Total593550203 (36.9%)

203 of the 548 reports receiving Add actions (37.0%) also involved replacing or deleting existing content. Thus, the predominance of Add does not imply simply appending diagnoses.

Correction of ST–T abnormalities. In a targeted analysis of 2,380 ECG-Grounding test records, we identified four candidates with waveform-supported ST depression missing from the initial diagnostic answer. Two original reports explicitly described the ST segment as normal. ECG-RePAIR successfully corrected both erroneous statements and updated the diagnostic answers.

These results demonstrate that our waveform verification and local execution mechanisms can correct existing ST-segment interpretation errors, complementing the original automated evaluation with no executed ST–T repairs.

Response to W7 & Q5: Generalizability and robustness

Generalization across datasets. We evaluated ECG-RePAIR on 2,000 previously unseen PTB-XL test ECGs, with all parameters fixed beforehand. Precision, Recall, and F1 were evaluated against reference codes across 13 mappable diagnostic categories without an LLM judge.

PTB-XLPrecisionRecallF1
ECG-R132.1763.8442.78
ECG-RePAIR36.0867.7146.93

The gains of 3.91, 3.87, and 4.15 points in Precision, Recall, and F1 support cross-dataset generalization.

Robustness to challenging signal-quality conditions. We evaluated 100 ECGs under baseline wander, electromyographic noise, power-line interference, and missing-lead conditions. Repair execution rates remain similar under noise (26–29% versus 28% on clean ECGs) and drop to 0% when Lead II is missing, demonstrating stable repair activity and uncertainty-aware abstention when essential evidence is unavailable.

We have uploaded the revised manuscript incorporating the additional experiments, expanded clinical evaluation, targeted repair analysis, and methodological clarifications.

Dear Reviewer,

Thank you very much for your time and for overseeing our submission.

We have carefully prepared a detailed, point-by-point response to your comments and concerns in the rebuttal.

As the discussion period is drawing to a close, we wanted to ask whether these answers resolve your questions. If you find the explanations adequate, we respectfully ask you to consider re-evaluating your rating. If any point would benefit from further analysis, or additional ablation, we would be glad to provide it while the discussion is open.

Your comments sharpened the evaluation considerably, and we are grateful for them. Thanks again, we wish all the best in your work and life.

Best regards,

The authors

Reviewer_9XE4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ECG-RePAIR is a post-processing framework that verifies and locally repairs reports generated by a frozen ECG multimodal LLM, using ECG-R1-8B-RL as the default base model. It comprises three stages:

  • ECG-Aware Planning: A claim scanner identifies findings concerning rhythm, intervals, axis, voltage, and ST–T changes. The planner selects up to six claims and retrieves clinical criteria to specify the relevant leads, measurements, thresholds, and validity conditions.
  • Evidence-Grounded Verification: NeuroKit2- and WFDB/XQRS-based processing paths obtain waveform measurements, which are accepted when they satisfy predefined agreement tolerances. The verifier then assigns Supported, Contradicted, or Uncertain status.
  • Memory-Conditioned Repair: An evidence-based filter determines admissible Keep, Add, or Revise actions. A ridge-regression estimator predicts each action’s improvement in judge-defined report utility, and a local edit is executed only when its predicted advantage exceeds a selected margin.

The paper evaluates 2,380 ECG-Grounding test records from MIMIC-IV-ECG using eight ECG-R1 rubric metrics and a DeepSeek judge. The repair memory, regression estimator, and decision margin are constructed or selected through five-fold cross-validation within this evaluation cohort; each record is repaired using only the other four folds.

Critical strengths

1. The paper clearly motivates auditable, uncertainty-aware report correction.

  • Clear motivation and illustrative examples. The opening figure and case studies effectively illustrate the gap between waveform evidence and generated clinical assertions, motivating targeted, evidence-grounded repair.
  • Traceable corrections. Evidence Cards record measurements, relevant leads, backend agreement, and provenance, linking each edit to explicit clinical criteria. Examples such as correcting a reported PR interval of approximately 180 ms using a measured value of 225.5 ms make this connection concrete.
  • Explicit uncertainty handling. Detector disagreement, poor signal quality, missing prerequisites, and ambiguous measurements can trigger an Uncertain verdict rather than a forced correction. This is a valuable safeguard when modifying clinical assertions.

2. The repair stage is lightweight and shows preliminary transferability.

  • Low inference overhead. The framework keeps the base model frozen and requires no additional language-model generation or GPU computation during repair, with modest reported CPU overhead.
  • Evidence across models and evaluators. Improvements with a second base model and a second judge suggest that the benefit is not restricted to one model–evaluator pairing, supporting its potential as an add-on repair component.
Critical weaknesses

1. The evaluation does not establish whether the gains generalize beyond the benchmark and its scoring rubric.

  • Benchmark-specific supervision limits the generalization claim. The repair estimator and hyperparameters are fitted or selected using four-fifths of the benchmark test set and evaluated on the remaining fifth. Although cross-validation avoids using each record’s own outcome, it measures supervised adaptation within the benchmark rather than performance on an independently held-out cohort. Comparisons with unadapted base models therefore do not isolate the framework’s benefit from additional supervision.
  • The scoring protocol does not clearly account for unsupported additions. Unsupported diagnoses receive zero credit, but their effect on the aggregate Diagnosis Accuracy depends on an unspecified normalization. This matters because additions dominate the repairs (591 Add versus 2 Revise actions). A second judge confirms the improvement trend but uses the same rubric, leaving this concern unresolved. Explicit diagnostic precision and a transparent scoring formula are needed.
  • Independent clinical validation remains limited. Two clinicians assess only 50 repaired cases, with before/after order visible, and do not evaluate missed corrections in the 1,830 unchanged reports. Only 82–86% of sampled cases receive an explicit “No new errors” judgment. These results support preliminary edit quality but do not establish improved clinical reliability across the cohort.

2. The benefit over simpler measurement-based repair is unclear, and local edits do not ensure coherent reports.

  • Most of the gain is achieved without memory. Removing memory yields 79.76 Diagnosis Accuracy versus 80.49 for the full system and 76.24 for the base model. The additional 0.73-point gain lacks evidence of statistical or practical significance. Comparisons with simpler measurement-and-rule repair or base-model revision using the same waveform measurements would better isolate the proposed architecture’s contribution.
  • The technical distinction from rule-based post-processing is insufficiently established. The system combines curated criteria, conventional waveform processing, constrained actions, and ridge regression. The paper should clarify what criterion retrieval and the “agentic” formulation contribute beyond fixed claim-specific rules.
  • The illustrative outputs retain contradictions and editing artifacts. For example, adding first-degree AV block leaves assertions of normal conduction and intervals intact, while local deletions produce fragments such as “reveals PR interval but…”. These examples show that improved scores on targeted diagnoses do not necessarily translate into coherent, clinically reliable final reports.
Questions for authors

The paper has a clear motivation, but I would appreciate clarification on the following points:

  1. How well does the repair policy generalize without benchmark-specific supervision? Can you evaluate it on an untouched test cohort, with the estimator fitted and hyperparameters selected using a separate development cohort?

  2. How does Diagnosis Accuracy penalize unsupported additions? Please clarify the score aggregation formula and report diagnostic precision, particularly given the predominance of Add actions.

5 comments

Thank you for your time and constructive comments. We hope our responses address your concerns.

Response to W1.1 & Q1:

Evaluation protocol. ECG-R1 provides 27,185 training ECGs and 2,380 test ECGs. Our original five-fold split simulates experience-based test-time adaptation, constructing memory from four test folds and evaluating on the held-out fold.

We additionally constructed memory exclusively from the ECG-R1 training set, with no patient overlap, retaining the same 2,380 test ECGs and all other settings. We denote the two configurations as Cross-fitted Memory and Independent Memory, respectively.

MethodDiag. Acc.CreditCompl.Relev.LeadGround.Reason.Fidelity
ECG-R176.2468.654.625.785.8271.4171.3772.16
Cross-fitted Memory80.4974.945.246.336.4275.0975.0475.32
Independent Memory81.1775.335.346.526.5976.0875.9375.91

Independent Memory achieves the best results across all eight metrics. The further improvement may benefit from the larger training set, showing that our gains do not rely on test-set-derived memory.

External dataset evaluation. Using ECG-R1 training-set memory, we evaluated 2,000 PTB-XL test ECGs excluded from parameter selection, measuring diagnostic agreement against reference codes across 13 mappable categories.

PTB-XLPrecisionRecallF1
ECG-R132.1763.8442.78
ECG-RePAIR36.0867.7146.93

The 4.15-point F1 improvement supports generalization beyond the original benchmark and evaluation rubric.

Response to W1.2 & Q2:

Clarification of Add and Revise. Add introduces a missing diagnostic concept but may also replace or delete incorrect statements, whereas Revise modifies an existing diagnostic concept. Among 548 reports receiving Add actions, 203 (37.0%) involved replacing or deleting existing content. Thus, the predominance of Add (591 vs. 2 Revise) does not imply simply appending diagnoses. We have clarified these definitions in the revised manuscript.

Evidence-supported diagnostic improvement. Additional waveform evidence itself does not increase Diagnosis Accuracy; diagnostic gains require more accurate conclusions. Lead-Evidence Validity, Analysis Relevance, and Evidence-Based Reasoning assess the correctness, relevance, and diagnostic support of the added evidence, respectively. Their consistent improvements support evidence-grounded repair.

Diagnosis Accuracy calculation. Following ECG-R1's rubric, each diagnosis receives 2 points if correct and supported, 1 if largely correct but insufficiently supported, and 0 if incorrect or unsupported:

DAi=100×Ns>0Ntotal.DA_i=100\times\frac{N_{s>0}}{N_{\mathrm{total}}}.

Judge-independent diagnostic evaluation. We evaluated Precision, Recall, and F1 against reference diagnostic labels using fixed concept-matching rules without an LLM judge. Unmatched predicted diagnoses were counted as false positives.

MethodPrecisionRecallF1
ECG-R165.7570.7968.18
Cross-fitted Memory71.8673.4272.65
Independent Memory72.1473.7672.98

With Independent Memory, Precision improves by 6.39 points and F1 by 4.80 points, alongside increased Recall, demonstrating diagnostic gains without sacrificing precision.

Response to W1.3:

We expanded clinical evaluation from 50 to 200 cases (100 repaired and 100 unchanged). Two clinicians independently assessed anonymized, randomized reports against the original ECGs, with method identities and before/after ordering concealed for repaired cases.

GroupMetricClinician AClinician B
Repaired (N=100)ECG-supported modifications88%90%
Clinical improvement79%82%
No new errors87%89%
Unchanged (N=100)Overall correctness79%76%
Missed clinical findings16%19%
Unsupported diagnoses8%10%
Cannot judge5%5%

Clinical reliability of executed repairs. Most repairs were ECG-supported, clinically beneficial, and introduced no new errors.

Potential missed corrections. Most unchanged reports were clinically correct, supporting selective repair. Remaining missed or unsupported findings identify opportunities to improve coverage. These categories are not mutually exclusive.

Clinical reliability beyond repaired cases. Assessing both groups provides broader clinical evidence for our selective correction strategy.

Given the time-intensive nature of expert assessment, we expanded the evaluation to 200 cases during rebuttal, twice ECG-R1's clinical sample size, and plan further expansion. These results and the updated Clinician Review Form are included in the revised manuscript.

Response to W2.1:

Comparison with simpler repair alternatives. We evaluated two additional baselines using the same waveform measurements but different correction strategies:

  • Measurement + Rules (measurement-and-rule repair): Uses fixed clinical rules to determine corrections and the same local executor to apply them.
  • Measurement + ECG-R1 (base-model revision): Provides the same waveform measurements to ECG-R1 for direct revision of its initial report.
MethodDiag. Acc.CreditCompl.Relev.LeadGround.Reason.Fidelity
ECG-R176.2468.654.625.785.8271.4171.3772.16
Measurement + Rules78.4673.275.086.046.1573.1273.9874.48
Measurement + ECG-R177.3668.274.796.205.7173.7270.7573.58
ECG-RePAIR (w/o Memory)79.7673.885.166.186.3174.6274.4874.71
ECG-RePAIR (Cross-fitted Memory)80.4974.945.246.336.4275.0975.0475.32
ECG-RePAIR (Independent Memory)81.1775.335.346.526.5976.0875.9375.91

ECG-RePAIR with Cross-fitted Memory outperforms measurement-and-rule repair and base-model revision by 2.03 and 3.13 points in Diagnosis Accuracy, respectively, with improvements across all eight metrics. These comparisons validate our evidence-to-action architecture under identical measurement conditions.

Contribution and statistical significance of memory. Cross-fitted Memory improves Diagnosis Accuracy by 0.73 points over the memory-free variant (patient-level paired 95% CI [0.28, 1.18]). All eight paired confidence intervals exclude zero, supporting the statistical significance of memory-conditioned repair.

As described in W1.1, expanding repair memory using 27,185 training ECGs increases its contribution to Diagnosis Accuracy from +0.73 to +1.41 points, accompanied by further improvements across all eight metrics.

These results suggest that richer historical repair experiences can further improve memory-conditioned repair.

Response to W2.2:

ECG-RePAIR goes beyond fixed measurement and correction rules. Its agentic nature lies in dynamically updating its clinical belief state based on verification feedback and adapting subsequent planning and repair decisions. This is achieved through three components:

  • ECG-Aware Planning: Selects claims for verification and retrieves relevant clinical criteria to guide evidence acquisition.
  • Evidence-Grounded Verification: Checks patient-specific waveform evidence and updates the clinical belief state with supported, contradicted, or uncertain findings.
  • Memory-Conditioned Repair: Uses historical state–action advantages to decide whether an evidence-supported repair is beneficial, rather than simply applying fixed correction rules.

These components form a feedback-driven decision loop: verification results guide claim prioritization, subsequent evidence acquisition, and whether to execute repair. The LLM judge is used only for offline memory construction.

MethodDiag. Acc.Change
w/o Criterion Retrieval79.31−1.18
w/o ECG Tools78.22−2.27
w/o Memory79.76−0.73
ECG-RePAIR (Cross-fitted Memory)80.49—

These ablations and the simpler baseline comparisons in W2.1 support the value of adaptive evidence-driven planning and repair beyond fixed rule-based post-processing.

Response to W2.3:

Both cases in Appendix C successfully achieve their intended diagnostic repair objectives. Case 1 corrects the erroneous normal-PR interpretation using the measured PR interval of 225.5 ms and identifies first-degree AV block. Case 2 corrects the erroneous normal-axis interpretation using the measured axis of −35.8° and identifies left-axis deviation. Both corrections are supported by waveform evidence.

We attribute the residual contradictions primarily to hallucinations in the original MLLM-generated reports, where erroneous clinical assertions are repeated across multiple reasoning steps. In Case 1, although ECG-RePAIR correctly updates the PR measurement and diagnostic answer, hallucinated normal-conduction statements remain elsewhere in the report. In Case 2, local removal of the erroneous axis description leaves a grammatical fragment. Despite these residual hallucinations and textual imperfections, both cases achieve clinically correct diagnostic conclusions for the targeted findings, supported by objective waveform evidence.

We explicitly discussed these issues in Appendix C.1 and C.3 of the original manuscript (Lines 1324–1326 and 1660–1663), intentionally retaining the complete model outputs without manual post-editing to ensure transparency.

Further mitigating MLLM hallucinations and improving global report consistency will be an important direction for our future work.

Dear Reviewer,

Thank you very much for your time and for overseeing our submission.

We have carefully prepared a detailed, point-by-point response to your comments and concerns in the rebuttal.

As the discussion period is drawing to a close, we wanted to ask whether these answers resolve your questions. If you find the explanations adequate, we respectfully ask you to consider re-evaluating your rating. If any point would benefit from further analysis, or additional ablation, we would be glad to provide it while the discussion is open.

Your comments have helped us make our work more rigorous, and we sincerely appreciate your valuable feedback. Thanks again, we wish all the best in your work and life.

Best regards,

The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Revision

We have uploaded a revised version of our manuscript, incorporating additional experiments, expanded clinical evaluation, and methodological clarifications in response to the reviewers' comments. Our central claim remains unchanged: ECG-RePAIR formulates post-generation ECG interpretation as a structured evidence-to-action process, integrating criterion-guided planning, waveform-grounded verification, and memory-conditioned repair to improve the diagnostic accuracy and report quality of frozen ECG reasoning models.

All changes introduced in this second revision are highlighted in blue. The main changes are summarized below.

  1. Independent Repair Memory and Evaluation Protocol (Section 4.1; Appendices A.4, A.5, and E.1; Table 7). We have clarified the original record-level cross-fitted evaluation and introduced an Independent Memory setting using 27,185 ECGs from the ECG-R1 training set, with no patient overlap with the evaluation cohort. The memory, estimator, and repair parameters are fixed before evaluation on the 2,380 ECG-Grounding test records. Independent Memory achieves the best results across all eight metrics, increasing Diagnosis Accuracy from 76.24 to 81.17 (+4.93 points) over ECG-R1.

  2. Additional Repair Baselines and Statistical Analysis (Appendices E.1 and E.2; Tables 7 and 8). We have added Measurement + Rules and Measurement + ECG-R1 as controlled baselines using the same waveform measurements. ECG-RePAIR with Independent Memory outperforms them by 2.71 and 3.81 points in Diagnosis Accuracy, respectively. We also report patient-level paired 95% confidence intervals for all three component ablations. All reported intervals exclude zero, including the incremental memory contribution of 0.73 points [0.28, 1.18] under Cross-fitted Memory.

  3. Judge-Independent Diagnostic Evaluation (Section 4.1; Appendix E.3; Table 9). We have added diagnostic Precision, Recall, and F1 against reference diagnostic labels using fixed concept-matching rules without an LLM judge. Independent Memory improves Precision by 6.39 points and F1 by 4.80 points, alongside a 2.97-point increase in Recall, providing complementary evidence of diagnostic improvement beyond rubric-based evaluation.

  4. Expanded Blinded Clinical Expert Evaluation (Appendix E.4; Table 10). We have expanded clinical evaluation from 50 to 200 cases (100 repaired and 100 unchanged), with two clinicians independently assessing anonymized and randomized reports against the original ECGs. Clinical improvement is observed in 79% and 82% of repaired cases, with ECG-supported modifications in 88% and 90%. Cohen's κ is 0.68 for clinical improvement and 0.73 for new-error assessment. The expanded evaluation also examines potential missed corrections in unchanged reports and includes an updated Clinician Review Form.

  5. Clarified Repair Actions and Targeted ST–T Analysis (Section 3.4; Appendix B.2; Figure 10; Appendix E.5; Table 11). We have refined the Add and Revise definitions to distinguish missing-finding additions from corrections of existing clinical assertions. Under the revised categorization, 205 of the 550 repaired reports (37.3%) involve correction of existing assertions. A separate targeted analysis also demonstrates successful correction of two erroneous normal-ST interpretations supported by waveform measurements.

  6. External Dataset Evaluation (Section 4.1; Appendix E.6; Table 12). We have evaluated ECG-RePAIR on 2,000 previously unseen PTB-XL test ECGs using the fixed independent repair memory and parameters. Across 13 mappable diagnostic categories, ECG-RePAIR improves diagnostic F1 from 42.78 to 46.93 (+4.15 points) without an LLM judge, supporting generalization beyond ECG-Grounding.

  7. Evaluation under Waveform Perturbations (Appendix E.7). We have added experiments on 100 ECGs under baseline wander, electromyographic noise, power-line interference, and missing-lead conditions. Repair execution rates remain comparable under the evaluated noise perturbations (26–29% versus 28% on clean ECGs), and decrease to 0% when Lead II is missing. These results characterize the stability of repair activity and uncertainty-aware abstention when required waveform evidence is unavailable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.