- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents ORAH, an ELM trained via pretraining and SFT stages on 12M ECG diagnosis text pairs. It exceeds the strongest reported baselines in all metrics on ECG-QA-CoT and the ECG-R1 Benchmark. The paper also studies the cross-dataset transfer, effects of data-scaling, pretraining benefits and whether RL improves model performance.
Critical strengths
S1. Large-scale data collection and model development.
S2. Strong empirical performance across multiple benchmarks.
S3. Useful empirical investigations into data scaling and cross-dataset transfer.
Critical weaknesses
W1. Although the paper presents several empirical studies, much of the analysis focuses on reporting rather than explaining. In particular, the rationale behind the multi-stage training pipeline, the roles of individual stages, and the causes of performance variations across metrics remain insufficiently investigated. I raise several related questions below.
W2. The model largely builds upon established components and training paradigms, including a SigLIP2-based ECG encoder, modality alignment MLP projector, autoregressive pretraining, and SFT. While the scale of the training corpus and the empirical evaluation are valuable, the paper introduces limited methodological innovation beyond combining these existing techniques into a large-scale ELM training pipeline.
W3. Although the appendix provides stage-specific hyperparameters, the paper does not sufficiently analyze the optimization dynamics or systematically justify key training decisions. In particular, the necessity and ordering of the individual stages are not isolated through controlled experiments, and the paper provides limited analysis of training loss curves and stage-transition behavior. Moreover, the data-scaling experiments increase both the amount of training data and the associated computation, making it difficult to disentangle their respective contributions.
Questions for authors
Q1. Can the authors provide deeper insights into the design of the multi-stage training pipeline? In particular, what motivates splitting SFT into two separate stages rather than training on the combined datasets jointly or using a different curriculum? How were the datasets, sample proportions, and training objectives selected for each of the five ORAH training stages?
Q2. Can the authors explain why pretraining benefits some capabilities while potentially impairing others? Could alternative training strategies mitigate these trade-offs?
Q3. Also note that the model performance degrades after RL finetuning, which contradict to common ideas. Since RL is currently a critical method for invoking LLM’s reasoning ability, will this incompatiblilty between ORAH and RL limits the model’s adaptation to more complex and generalized ECG tasks?
2 comments
Rebuttal to Review Part 2
Weakness 2
We thank the reviewer for recognizing the value of the data scale and empirical evaluation. We agree that Orah uses established architectural components and training methods. We chose this simple design to keep the architecture fixed while studying data scale, the effects of pretraining, and transfer across ECG sources. The main contribution is the approximately 12-million example training corpus and the empirical findings from these experiments, including where additional training helps and where targeted supervision remains necessary. We will revise the manuscript to make this scope clearer.
Question 2
We thank the reviewer for raising this point. Pretraining improves nearly every evaluated metric, with decreases only in ECG-R1 Diagnosis Accuracy (−1.03) and ECG-Reasoning Depth (−0.12). Diagnostic accuracy improves on ECG-QA-CoT (+3.72) and ECG-Reasoning IDA (+0.78), showing that the diagnostic effect depends on the benchmark. Our ablation establishes these differences but does not determine their underlying cause.
One possible approach is to introduce data aimed at improving diagnostic accuracy and reasoning depth during earlier training stages. Our targeted-supervision experiment (§4.3) motivates this direction, showing that a small amount of task-specific data improves all four ECG-Reasoning metrics. We leave other training strategies for mitigating these trade-offs to future work.
Question 3
We thank the reviewer for raising this point. We respectfully clarify that our results do not establish an incompatibility between Orah and RL. On the ECG-R1 Benchmark, RL improves Diagnosis Accuracy and Clinical Diagnostic Fidelity, while other metrics decline.
Prior work suggests that RL can increase the probability of generating successful reasoning paths already present in the starting model, without necessarily introducing new reasoning capabilities [1]. Thus, additional RL does not guarantee improvements across all metrics. ECG-R1 itself also shows modest gains from RL on its own benchmark [2] and on ECG-Reasoning [3]. On the ECG-Reasoning benchmark, ECG-R1's depth metric even decreases by 0.01 on the PTB-XL split.
Our experiments evaluate one RL configuration and do not establish a limit on Orah’s adaptation to more complex or general ECG tasks. Whether alternative RL strategies improve performance on such tasks remains an open question that we hope to investigate in future work.
References
[1] Yue et al. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? NeurIPS 2025.
[2] Jin et al. ECG-R1: Protocol-Guided and Modality-Agnostic MLLM for Reliable ECG Interpretation. ICML 2026.
[3] Oh et al. ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation. NeurIPS 2026.
Rebuttal to Review Part 1
Weakness 1 and Weakness 3 and Question 1
We thank the reviewer for raising these important questions. We selected datasets according to the supervision they provide. HEEDB, EchoNext, and our internal datasets provide ECG-report pairs without ECG-paired QA or conversations, so we use them for pretraining. Datasets with ECG-paired QA and conversations are used for SFT. Following Qwen3-VL [1], we allocate a smaller corpus to Orah pretraining 1 for projector alignment and a larger corpus to pretraining 2 for joint training. Pretraining 1 updates only the projector to align SigLEP representations with the frozen LLM. Pretraining 2 updates both the projector and LLM for ECG-conditioned report generation.
Our rationale for splitting SFT is to first establish instruction-following and conversational behavior using simple QA and conversation examples. SFT 2 includes different examples from some SFT 1 datasets to help maintain these capabilities, alongside additional instruction data and explicit reasoning traces. This progression is consistent with prior work introducing reasoning-enriched supervision after initial SFT [4]. Prior studies explore different mixed and staged SFT strategies [3] and report differing results for projector-only alignment [2]. These studies inform our design but do not establish which strategy is best for Orah.
We selected the within-stage dataset proportions heuristically to maintain diversity across ECG sources and supervision types. Table 1 reports the stage-specific datasets and sample counts, with additional dataset details in Appendix A. Our scaling experiments preserve the relative dataset proportions within each stage and evaluate the selected recipe without establishing that these proportions or the stage ordering are optimal. Additionally, our scaling experiments increase both data volume and training computation, so they do not isolate the effect of data volume at fixed computation.
We additionally evaluated two training choices. First, following Qwen3-VL's joint pretraining stage [1], we tried updating all parameters, including SigLEP, during Orah pretraining 2. Second, to examine whether additional SFT could recover the gains from Orah pretraining, we added a baseline that retains SigLEP pretraining but skips both Orah pretraining stages and increases each SFT stage from three to five epochs. Based on the training-token counts in Table 4, approximately 4.4 epochs per SFT stage would match the combined Orah pretraining and SFT token budget, so five epochs gives this baseline a slightly larger token budget.
The following tables report results on ECG-QA-CoT and the ECG-R1 Benchmark, respectively.
Updating all parameters did not yield consistently better performance, supporting our decision to freeze SigLEP during Orah pretraining 2 to save computation. Longer SFT narrows the performance gap, but pretraining plus SFT still outperforms the five-epoch SFT-only baseline on every reported metric. This supports allocating training tokens to Orah pretraining rather than additional passes over the same SFT data under our evaluated recipe.
We additionally report epoch-level training losses below. Training loss decreases across the recorded epochs in Orah pretraining 1 and both SFT stages.
These experiments do not isolate the necessity or ordering of individual stages, and the epoch-level losses do not resolve behavior at stage transitions. We will revise the manuscript to include these results and clarify these limitations. We leave controlled comparisons of alternative training strategies and detailed optimization analysis to future work.
References
[1] Qwen Team. Qwen3-VL Technical Report. 2025.
[2] Gröger et al. Rethinking Projector Training in Multimodal LLMs. NeurIPS 2026.
[3] Dong et al. How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition. ACL 2024.
[4] Goncharov et al. Complexity-aware fine-tuning. Findings of EACL 2026.