- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes ECG-RSI, a data-centric iterative training framework for ECG multimodal models. The forward pipeline generates question-answer pairs from measured features of real ECG waveforms. The reverse pipeline edits an existing training waveform to match a target feature change, re-measures the perturbed signal, and constructs evidence-aligned questions and explanations based on the achieved edits. A second feedback loop uses the current learner's error profile to decide which categories, difficulty levels, and supervision styles to generate in the next round. In each iteration, multiple sibling branches are trained from the same parent checkpoint, and the best-performing branch is promoted. The paper reports that the aggregate S7 score rises from 30.15 to 46.61 across four rounds, alongside waveform edit error statistics, a common-checkpoint ablation study, and a blinded review of 240 generated examples by two physicians.
Critical strengths
- The idea of reverse waveform perturbation for targeted instruction data construction is conceptually promising. Editing the waveform first and then regenerating supervision from the resulting signal provides a more controlled way to create feature-specific training examples than paraphrasing existing reports or reusing real-world annotations.
- The data construction pipeline is comprehensively engineered, with target attainment checks, non-target preservation checks, versioned repair mechanisms, replay buffers, same-parent branch comparison, and clinical expert review. The single-round ablation in Table 5 also provides preliminary evidence that reverse synthesis and feedback-guided selection add value beyond ordinary training questions at that specific checkpoint.
Critical weaknesses
- The core claim of recursive self-improvement lacks rigorous, isolated validation. The reported gain across four rounds conflates multiple confounding factors: cumulative data scaling, repeated supervised fine-tuning, targeted synthesis, extra branch-search compute, and repeated model selection. The most directly relevant ablation in Table 5 is only a single-round comparison from one checkpoint. The paper does not provide a matched multi-round control with the same cumulative data volume, number of SFT updates, and branch-search budget but without learner-driven feedback. Without this control, the main results cannot support the title-level claim that feedback-driven recursive self-improvement drives the improvement.
In addition, the automatic construction loop appears to reuse the same delineation/measurement framework for defining measurable targets and remeasuring edited signals. Although a separate quality-review stage and physician audit are provided, these do not directly establish that the numerical waveform measurements used for construction are independently verified. The physician audit further underscores this concern: non-target preservation is 0.79 and evidence-supported explanations reach only 0.76, leaving a notable residual error rate for data advertised as verified.
- Several decision-critical quantities are described procedurally but are not instantiated clearly enough to reproduce the reported trajectory. Large portions of the main text and appendices are dedicated to workflow descriptions, prompt templates, and case walkthroughs, while key numerical specifications — including the regression tolerances for branch promotion, valid-output gates, stopping criteria, and some edit-specific tolerance thresholds — are scattered, stated only qualitatively, or not explicitly reported. Core innovations are buried under procedural documentation, and the logical chain connecting method design, experimental evidence, and conclusions is consistently unclear. The overall presentation reads more like engineering process documentation than a focused scientific contribution.
Questions for authors
- Can you provide a multi-round baseline control with the same cumulative number of new examples, same replay policy, same number of SFT updates, and comparable branch-search compute budget, but using a fixed or random data recipe instead of learner-driven feedback? This control is necessary to disentangle the effect of RSI from plain data scaling and repeated training.
- Please state explicitly how , , and the seven final benchmarks are separated. Were any final-benchmark scores or benchmark-level error summaries used for recipe design, model promotion, stopping criteria, or hyperparameter decisions?
- Were the 240 physician-reviewed examples sampled from the final accepted training pool after all automatic quality gates? Also, is post-edit verification performed by the same delineator/measurement pipeline used to define the edit parameters, or by an independent verification module?
- Much of the paper and appendix is occupied by workflow descriptions, prompt templates, and case walkthroughs. Could you clearly state what the core technical contributions of this work are relative to prior work, and which parts are standard engineering implementation?