- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ECG-RePAIR revisits an already generated ECG report in three stages:
- Extract addressable claims and retrieve the relevant clinical criteria.
- Measure the current waveform with two tool backends and classify the resulting evidence as supporting, contradicting, or uncertain.
- Use an evidence-constrained action mechanism and a repair-advantage estimator to retain, add, or revise a finding.
The evaluation on ECG-Grounding combines component ablations, alternative base/evaluator models, and clinician assessment of sampled repaired reports.
Critical strengths
-
The separation between deciding what evidence is needed, obtaining that evidence from the waveform, and deciding whether to modify the report is well motivated. The explicit uncertain outcome and local repair mechanism are useful design choices for avoiding unsupported, broad report changes.
-
The paper provides multiple complementary checks of its approach, including tool and memory ablations, a different base model, a different evaluator, and independent clinician assessments. The reported improvement over ECG-R1 and the clinician findings make this a useful contribution to evidence-grounded ECG interpretation.
Taken together, these results and design choices lead me to a positive assessment and weak acceptance.
Critical weaknesses
The scope of the cross-fitted repair-memory results and the reliance on rubric-based automated report scoring are the main limitations to keep in mind. Human assessment and the other supporting analyses are relevant here. My remaining requests concern explaining the protocol and the scope of its conclusions, rather than identifying grounds for rejection.
Questions for authors
Q1. Are the five folds used for memory construction and action estimation patient-disjoint, including repeated ECGs? Please explain the isolation of fold-specific fitting and threshold selection, and distinguish this cross-fitted setting from deployment with a fixed memory on an independent cohort.
Q2. Please state the exact report-utility function J used to construct repair advantages and its relationship to the final evaluation metrics and judge. This is important for interpreting the incremental contribution of the memory-based decision rule.
Q3. Could you explain how shared errors between the two measurement backends are handled, especially near diagnostic thresholds? Please also clarify that the clinician sample evaluates the quality of executed repairs, rather than the completeness of detecting all errors in unchanged reports.
4 comments
Response to Q3: Shared measurement errors, diagnostic thresholds, and clinical evaluation
Handling shared measurement errors. ECG-RePAIR uses NeuroKit2- and WFDB/XQRS-based measurement paths with predefined agreement tolerances:
| Measurement | Agreement tolerance |
|---|---|
| Heart rate | 8 bpm |
| RR interval | 40 ms |
| PR interval | 30 ms |
| QRS duration | 20 ms |
| QT interval | 35 ms |
| QTc interval | 40 ms |
| ST amplitude | 0.08 mV |
| T-wave amplitude | 0.12 mV |
Backend agreement alone cannot eliminate shared systematic errors. Therefore, ECG-RePAIR additionally checks signal quality, clinical prerequisites, and applicable exclusions before authorizing a repair. For example, first-degree AV block requires both reliable PR prolongation and a valid P–QRS relationship.
Handling measurements near diagnostic thresholds. Ambiguous evidence near diagnostic thresholds can trigger an Uncertain verdict, preventing automatic repair. The distance from the diagnostic threshold also informs the repair advantage estimator, enabling threshold-sensitive repair decisions.
We further evaluated 100 ECGs under baseline wander, electromyographic noise, power-line interference, and missing-lead conditions. Repair execution rates remain similar under noise perturbations (26–29% versus 28% on clean ECGs) and drop to 0% when Lead II is missing. These results support stable repair activity and uncertainty-aware abstention when essential waveform evidence is unavailable.
Scope of the original clinical evaluation. The original 50-case clinical evaluation specifically assessed the quality of executed repairs. It did not evaluate the completeness of detecting errors in unchanged reports.
To investigate both aspects, we expanded the clinical evaluation to **200 cases (100 repaired and 100 unchanged)**. Two clinicians independently assessed anonymized and randomized reports against the original ECG waveforms, with method identities and before/after ordering concealed for repaired cases.
| Group | Metric | Clinician A | Clinician B |
|---|---|---|---|
| Repaired (N = 100) | ECG-supported modifications | 88% | 90% |
| Clinical improvement | 79% | 82% | |
| No new errors | 87% | 89% | |
| Unchanged (N = 100) | Overall correctness | 79% | 76% |
| Missed clinical findings | 16% | 19% | |
| Unsupported diagnoses | 8% | 10% | |
| Cannot judge | 5% | 5% |
Cohen's κ was 0.68 for clinical improvement and 0.73 for new-error assessment, indicating substantial inter-rater agreement. The unchanged-report assessment categories are not mutually exclusive.
The expanded clinical evaluation directly examines both the quality of executed repairs and potential missed corrections in unchanged reports, providing broader clinical evidence while remaining a sample-based assessment.
Given the time-intensive nature of expert assessment, we expanded the evaluation to 200 cases during the rebuttal period, twice ECG-R1's clinical evaluation sample size, and plan further expansion.
Thank you again for your positive assessment and valuable suggestions. We have incorporated these additional experiments, the updated Clinician Review Form, and methodological clarifications into the revised manuscript.
Thank you for taking the time to address my questions. The new evaluation with an independently constructed, patient-disjoint repair memory resolves my main concern about the evaluation setup. The judge-independent diagnostic results and the expanded clinical assessment are also helpful, particularly the inclusion of unchanged reports.
I am satisfied with these responses. They provide stronger support for the method and clarify how it would be used with a fixed memory at deployment. Please include the additional results and protocol details in the revised paper. I maintain my positive recommendation.
Reply Rebuttal Comment by Authors
Thank you very much for your valuable feedback! We sincerely appreciate the time and effort you devoted to reviewing our manuscript and providing constructive comments and suggestions. We will incorporate these additional results and protocol details into the revised manuscript.
— Authors
Thank you for your positive assessment and constructive suggestions. We appreciate your recognition of ECG-RePAIR's evidence-grounded design, clinical relevance, and complementary evaluations. We have conducted additional experiments and analyses to address your concerns.
Response to W1 & Q1: Cross-fitted evaluation and independent repair memory
Patient-level isolation. The original five folds were partitioned by ECG record ID rather than patient ID. Therefore, repeated ECGs from the same patient could appear in different folds, although each ECG record was strictly excluded from its own memory construction and parameter selection.
Fold-specific fitting and threshold selection. For each held-out fold, memory construction, estimator fitting, and selection of the ridge coefficient and decision margin exclusively used the remaining four folds. The held-out fold was used only for test-time repair and evaluation, with the memory remaining read-only. Thus, the cross-fitted setting uses experience from other records within the benchmark, whereas independent-cohort deployment requires a fixed memory developed outside the test cohort.
Deployment with independent memory. Following the reviewer's suggestion, we constructed an independent Repair Advantage Memory exclusively using 27,185 ECGs from the ECG-R1 training set, with no patient overlap with the evaluation cohort. We obtained 6,128 tool-verified candidate repair actions, using 4,872 for estimator fitting and 1,256 for internal validation. The memory, estimator, and all parameters were fixed before evaluating the 2,380 ECG-Grounding test records.
Independent Memory improves Diagnosis Accuracy by 4.93 points over ECG-R1 and 1.41 points over the memory-free variant.
ECG-RePAIR achieves its best performance with a fixed repair memory and all parameters developed entirely outside the evaluation cohort, directly addressing the distinction between cross-fitted evaluation and independent-cohort deployment.
Response to W2 & Q2: Report-utility function and rubric-based evaluation
Exact report-utility function. J(y) is the per-report Diagnosis Accuracy score (0–100) assigned by DeepSeek-v4-flash-0731 under the original ECG-R1 rubric, corresponding to
diagnosis_accuracy_binaryin our implementation. Each diagnosis receives 2 points if correct and supported, 1 if largely correct but insufficiently supported, and 0 if incorrect or unsupported:J(y) = 100 × (number of diagnoses with score > 0) / (total number of scored diagnoses).
The observed repair advantage is:
Δi=J(yiai)−J(yi),
where yi and yiai denote the reports before and after executing action ai, respectively. These differences serve as regression targets without additional metric weighting or normalization.
Relationship to the final evaluation metrics and judge. Diagnosis Accuracy is the offline optimization target and one of the eight final evaluation metrics, evaluated using the same DeepSeek judge and rubric. The remaining seven metrics are not directly optimized. No LLM judge is invoked during test-time repair. The estimator uses only current state–action features, and candidate repairs must satisfy the evidence-based action filter.
Incremental contribution of memory. Cross-fitted Memory improves Diagnosis Accuracy by 0.73 points over the memory-free variant, with a patient-level paired 95% confidence interval of [0.28, 1.18]. Independent Memory further increases this improvement to 1.41 points. Both memory configurations improve all eight evaluation metrics.
Judge-independent evaluation. To address the reliance on rubric-based automated report scoring, we additionally evaluated diagnostic Precision, Recall, and F1 using a fixed diagnostic vocabulary and identical concept-matching rules against reference diagnostic labels, without an LLM judge. Unmatched predicted diagnoses were counted as false positives.
The simultaneous improvements in Precision (+6.39), Recall (+2.97), and F1 (+4.80) provide independent evidence of diagnostic improvement beyond rubric-based automated report scoring.