acceptodds
Under review as a conference paper at ICLR 2027

Scaling ECG-Language Models to 12 Million Training Examples

Abstract

ECG-language models (ELMs) show promise for automated ECG interpretation, but the effects of data scale and training strategies remain underexplored. We introduce Orah, an ELM combining a SigLIP2-adapted ECG encoder (SigLEP) with a 4 billion parameter large language model (LLM). Orah is trained on approximately 12 million examples across 12 datasets, the largest ELM training corpus to our knowledge. We develop a multi-stage training recipe spanning SigLEP pretraining, Orah pretraining, and supervised fine-tuning (SFT). We investigate the contributions of data scale and Orah pretraining across three benchmarks. Orah exceeds the strongest reported baselines in all metrics on ECG-QA-CoT and the ECG-R1 Benchmark. On the ECG-Reasoning Benchmark, adding approximately 3,000 task-specific examples from the opposing ECG source improves Initial Diagnosis Accuracy (IDA) by 27.26 points on PTB-XL and 30.45 points on MIMIC-IV-ECG, highlighting the value of small amounts of targeted supervision. Furthermore, our ablations show that increasing training data and computation under our recipe consistently improves performance across all three benchmarks. These results establish Orah as a strong ELM and provide empirical guidance for scaling data and training ELMs. Upon acceptance, we will release all publicly available datasets, model weights, and code for preprocessing, training, and evaluation used throughout this study.

Primary area:
datasets and benchmarks
Keywords:
ECG, Multimodal, Large Language Models, Vision-Language Models, Healthcare

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wEj2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ORAH, an ELM trained via pretraining and SFT stages on 12M ECG diagnosis text pairs. It exceeds the strongest reported baselines in all metrics on ECG-QA-CoT and the ECG-R1 Benchmark. The paper also studies the cross-dataset transfer, effects of data-scaling, pretraining benefits and whether RL improves model performance.

Critical strengths

S1. Large-scale data collection and model development.

S2. Strong empirical performance across multiple benchmarks.

S3. Useful empirical investigations into data scaling and cross-dataset transfer.

Critical weaknesses

W1. Although the paper presents several empirical studies, much of the analysis focuses on reporting rather than explaining. In particular, the rationale behind the multi-stage training pipeline, the roles of individual stages, and the causes of performance variations across metrics remain insufficiently investigated. I raise several related questions below.

W2. The model largely builds upon established components and training paradigms, including a SigLIP2-based ECG encoder, modality alignment MLP projector, autoregressive pretraining, and SFT. While the scale of the training corpus and the empirical evaluation are valuable, the paper introduces limited methodological innovation beyond combining these existing techniques into a large-scale ELM training pipeline.

W3. Although the appendix provides stage-specific hyperparameters, the paper does not sufficiently analyze the optimization dynamics or systematically justify key training decisions. In particular, the necessity and ordering of the individual stages are not isolated through controlled experiments, and the paper provides limited analysis of training loss curves and stage-transition behavior. Moreover, the data-scaling experiments increase both the amount of training data and the associated computation, making it difficult to disentangle their respective contributions.

Questions for authors

Q1. Can the authors provide deeper insights into the design of the multi-stage training pipeline? In particular, what motivates splitting SFT into two separate stages rather than training on the combined datasets jointly or using a different curriculum? How were the datasets, sample proportions, and training objectives selected for each of the five ORAH training stages?

Q2. Can the authors explain why pretraining benefits some capabilities while potentially impairing others? Could alternative training strategies mitigate these trade-offs?

Q3. Also note that the model performance degrades after RL finetuning, which contradict to common ideas. Since RL is currently a critical method for invoking LLM’s reasoning ability, will this incompatiblilty between ORAH and RL limits the model’s adaptation to more complex and generalized ECG tasks?

2 comments
AuthorsOpenReview ↗

Rebuttal to Review Part 1

Weakness 1 and Weakness 3 and Question 1

We thank the reviewer for raising these important questions. We selected datasets according to the supervision they provide. HEEDB, EchoNext, and our internal datasets provide ECG-report pairs without ECG-paired QA or conversations, so we use them for pretraining. Datasets with ECG-paired QA and conversations are used for SFT. Following Qwen3-VL [1], we allocate a smaller corpus to Orah pretraining 1 for projector alignment and a larger corpus to pretraining 2 for joint training. Pretraining 1 updates only the projector to align SigLEP representations with the frozen LLM. Pretraining 2 updates both the projector and LLM for ECG-conditioned report generation.

Our rationale for splitting SFT is to first establish instruction-following and conversational behavior using simple QA and conversation examples. SFT 2 includes different examples from some SFT 1 datasets to help maintain these capabilities, alongside additional instruction data and explicit reasoning traces. This progression is consistent with prior work introducing reasoning-enriched supervision after initial SFT [4]. Prior studies explore different mixed and staged SFT strategies [3] and report differing results for projector-only alignment [2]. These studies inform our design but do not establish which strategy is best for Orah.

We selected the within-stage dataset proportions heuristically to maintain diversity across ECG sources and supervision types. Table 1 reports the stage-specific datasets and sample counts, with additional dataset details in Appendix A. Our scaling experiments preserve the relative dataset proportions within each stage and evaluate the selected recipe without establishing that these proportions or the stage ordering are optimal. Additionally, our scaling experiments increase both data volume and training computation, so they do not isolate the effect of data volume at fixed computation.

We additionally evaluated two training choices. First, following Qwen3-VL's joint pretraining stage [1], we tried updating all parameters, including SigLEP, during Orah pretraining 2. Second, to examine whether additional SFT could recover the gains from Orah pretraining, we added a baseline that retains SigLEP pretraining but skips both Orah pretraining stages and increases each SFT stage from three to five epochs. Based on the training-token counts in Table 4, approximately 4.4 epochs per SFT stage would match the combined Orah pretraining and SFT token budget, so five epochs gives this baseline a slightly larger token budget.

The following tables report results on ECG-QA-CoT and the ECG-R1 Benchmark, respectively.

MethodF1Accuracy
Orah (SFT)63.7860.11
Orah (All parameters)63.4359.57
Orah (No Pretraining, 3 epochs)60.6456.39
Orah (No Pretraining, 5 epochs)62.5858.23
MethodDiagnosis AccuracyAnalysis CompletenessAnalysis RelevanceLead Evidence ValidityECG Feature GroundingEvidence-Based ReasoningClinical Diagnostic Fidelity
Orah (SFT)80.316.725.737.3485.0779.9885.97
Orah (All parameters)79.826.615.727.3084.4580.0284.78
Orah (No Orah pretraining, 3 epochs)81.343.974.044.2167.0964.8365.62
Orah (No Orah pretraining, 5 epochs)79.515.485.635.2678.8977.4380.68

Updating all parameters did not yield consistently better performance, supporting our decision to freeze SigLEP during Orah pretraining 2 to save computation. Longer SFT narrows the performance gap, but pretraining plus SFT still outperforms the five-epoch SFT-only baseline on every reported metric. This supports allocating training tokens to Orah pretraining rather than additional passes over the same SFT data under our evaluated recipe.

We additionally report epoch-level training losses below. Training loss decreases across the recorded epochs in Orah pretraining 1 and both SFT stages.

StageEpoch 1Epoch 2Epoch 3
Orah pretraining 10.670.300.23
Orah pretraining 20.19––
Orah SFT 10.430.300.14
Orah SFT 20.390.170.08

These experiments do not isolate the necessity or ordering of individual stages, and the epoch-level losses do not resolve behavior at stage transitions. We will revise the manuscript to include these results and clarify these limitations. We leave controlled comparisons of alternative training strategies and detailed optimization analysis to future work.

References

[1] Qwen Team. Qwen3-VL Technical Report. 2025.

[2] Gröger et al. Rethinking Projector Training in Multimodal LLMs. NeurIPS 2026.

[3] Dong et al. How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition. ACL 2024.

[4] Goncharov et al. Complexity-aware fine-tuning. Findings of EACL 2026.

AuthorsOpenReview ↗

Rebuttal to Review Part 2

Weakness 2

We thank the reviewer for recognizing the value of the data scale and empirical evaluation. We agree that Orah uses established architectural components and training methods. We chose this simple design to keep the architecture fixed while studying data scale, the effects of pretraining, and transfer across ECG sources. The main contribution is the approximately 12-million example training corpus and the empirical findings from these experiments, including where additional training helps and where targeted supervision remains necessary. We will revise the manuscript to make this scope clearer.

Question 2

We thank the reviewer for raising this point. Pretraining improves nearly every evaluated metric, with decreases only in ECG-R1 Diagnosis Accuracy (−1.03) and ECG-Reasoning Depth (−0.12). Diagnostic accuracy improves on ECG-QA-CoT (+3.72) and ECG-Reasoning IDA (+0.78), showing that the diagnostic effect depends on the benchmark. Our ablation establishes these differences but does not determine their underlying cause.

One possible approach is to introduce data aimed at improving diagnostic accuracy and reasoning depth during earlier training stages. Our targeted-supervision experiment (§4.3) motivates this direction, showing that a small amount of task-specific data improves all four ECG-Reasoning metrics. We leave other training strategies for mitigating these trade-offs to future work.

Question 3

We thank the reviewer for raising this point. We respectfully clarify that our results do not establish an incompatibility between Orah and RL. On the ECG-R1 Benchmark, RL improves Diagnosis Accuracy and Clinical Diagnostic Fidelity, while other metrics decline.

Prior work suggests that RL can increase the probability of generating successful reasoning paths already present in the starting model, without necessarily introducing new reasoning capabilities [1]. Thus, additional RL does not guarantee improvements across all metrics. ECG-R1 itself also shows modest gains from RL on its own benchmark [2] and on ECG-Reasoning [3]. On the ECG-Reasoning benchmark, ECG-R1's depth metric even decreases by 0.01 on the PTB-XL split.

Our experiments evaluate one RL configuration and do not establish a limit on Orah’s adaptation to more complex or general ECG tasks. Whether alternative RL strategies improve performance on such tasks remains an open question that we hope to investigate in future work.

References

[1] Yue et al. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? NeurIPS 2025.

[2] Jin et al. ECG-R1: Protocol-Guided and Modality-Agnostic MLLM for Reliable ECG Interpretation. ICML 2026.

[3] Oh et al. ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation. NeurIPS 2026.

Reviewer_CKaUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This is an interesting study describing the development of Orah, an ECG language model trained on a large collection of ECG datasets. Although the work offers limited methodological innovation, the scale of the data used for training and evaluation makes the study notable. Having said this, Orah’s initial diagnostic performance is underwhelming relative to that of other benchmark methods across two large datasets, and the manuscript does not adequately explain this result. Additional supervised training on smaller datasets improves performance; however, the need for benchmark-specific fine-tuning merely to match competing methods raises concerns that warrant further discussion. As a starting point, the related-work section should be expanded to better contextualize the baseline comparator models and clarify how Orah differs from them.

Critical strengths

The size and scope of the data used for Orah development is unparalleled.

Overall, the paper is clearly written and the experiments are extensive.

Critical weaknesses

The work is devoid of any real novelty as the approaches used for training are all standard.

The use of min-max normalization for ECGs seems non-standard, especially since the absolute value of the ECG itself is useful for some diagnostic tasks (e.g., LVH). If memory serves me correctly, this is not consistent with how ECGs were processed in the other benchmark methods used presented. Can the authors clarify?

The authors should better clarify the need for Orah SFT1 and Orah SFT2 stages and whether this additional supervised fine-tuning steps were used in the other benchmark methods. The extensive training procedure makes the relatively poor performance of Orah (without cross-data fine tuning) perplexing.

Questions for authors

Do the other benchmarks models use a similarly extensive training procedure? Contrasting the training method used by the authors with the training procedure of other baseline methods would help to put the results into context.

The authors highlight the amount of data used for training, but they do not discuss how their training procedure differs from that of baseline methods. Can they provide additional information along these lines? This is very important.

This reviewer does not find the arguments around "Evidence based reasoning" for ECG analyses very useful or compelling. At the end of the day it is the ability of these models to answer general diagnostic questions that require ECG signals. It would be good to understand precisely how well Orah performs in this regard, without targeted supervision across ECG sources. At present the data do not seem very compelling for this task.

I am certainly willing to reconsider my score if the authors can provide compelling answers to these questions.

2 comments
AuthorsOpenReview ↗

Rebuttal to Review Part 1

Weakness 1

We thank the reviewer for the critique. We agree that Orah builds on established training methods, especially from the vision-language modeling domain. Our contribution is an empirical study of data scale, pretraining, and transfer in ECG-language models. We collected approximately 12 million ECG-language training examples, the largest ELM training corpus to our knowledge, and evaluated a fixed architecture across three established benchmarks. Beyond benchmark performance, our experiments identify which capabilities benefit from pretraining and show that targeted supervision transfers across different ECG sources. We will revise the manuscript to make these empirical contributions clearer.

Weakness 2

We thank the reviewer for this question. We respectfully clarify that min-max normalization has been used in prior ECG work [1]. Our normalization is also consistent with ELF [2], the strongest ECG-QA-CoT baseline in Table 5, although it differs from that used by OpenTSLM (i.e., per-lead z-score normalization) and ECG-R1 (i.e., none).

We agree that our min-max normalization removes absolute voltage calibration, which may affect voltage-dependent tasks such as LVH assessment. Our results demonstrate strong overall benchmark performance with this preprocessing, but do not establish that it is optimal for all diagnostic tasks. We acknowledge this tradeoff and leave a systematic comparison of normalization strategies to future work.

Weakness 3, Question 1, and Question 2

Our rationale for the two SFT stages is to first introduce instruction following through simple QA and conversations using the chat template. In SFT 2, we include a different subset of examples from some SFT 1 datasets to help reduce forgetting, alongside new data with more complex conversations and reasoning traces. This rationale is consistent with prior work that first trains on simpler examples and then gradually introduces harder examples [3]. We will update the manuscript to include these rationales more explicitly.

The training datasets and strategies differ across baselines. For example, OpenTSLM uses a five-step curriculum covering time-series QA, captioning, human activity recognition, sleep staging, and ECG QA with reasoning [4]. ECG-R1 uses a single SFT stage combining ECG-Instruct and ECG Protocol-Guided Grounding CoT, followed by RL on the ECG-R1 dataset [5] (see Appendix A.1). Thus, other baselines also use multiple training stages, although their training sequences and datasets differ from Orah’s. We will add an explicit comparison of the baseline training stages and datasets to the manuscript.

Regarding Orah’s limited performance on ECG-Reasoning without benchmark-specific supervision, we agree that large-scale training alone does not adequately prepare Orah for this benchmark. Adding approximately 3,000 examples from the opposing ECG source improves IDA by 27.26 points on PTB-XL and 30.45 points on MIMIC-IV-ECG, highlighting the value of targeted supervision even after large-scale training.

References

[1] Uwaechia and Ramli. A Comprehensive Survey on ECG Signals as New Biometric Modality for Human Authentication: Recent Advances and Future Challenges. IEEE Access 2021.

[2] Han et al. ELF: A Family of Encoder-Free ECG-Language Models. MLHC 2026.

[3] Goncharov et al. Complexity-aware fine-tuning. EACL Findings 2026.

[4] Langer et al. OpenTSLM: Time-Series Language Models for Reasoning over Multivariate Medical Text- and Time-Series Data. ICML 2026.

[5] Jin et al. ECG-R1: Protocol-Guided and Modality-Agnostic MLLM for Reliable ECG Interpretation. ICML 2026.

AuthorsOpenReview ↗

Rebuttal to Review Part 2

Question 3

We thank the reviewer for raising this concern. We respectfully clarify that “Evidence-Based Reasoning” is the name of one of the seven evaluation metrics adopted from the ECG-R1 Benchmark [1]. We use this terminology to report results under that benchmark’s evaluation protocol.

We agree that answering diagnostic questions requiring ECG signals is a core capability to evaluate. We respectfully point out that all three benchmarks assess diagnostic performance: ECG-QA-CoT includes diagnostic questions, ECG-R1 reports Diagnosis Accuracy, and ECG-Reasoning reports Initial Diagnosis Accuracy (IDA). To assess diagnostic performance without the additional ECG-Reasoning examples, we report the following results on ECG-QA-CoT and the ECG-R1 benchmark respectively.

MethodF1Accuracy
Orah (SFT)63.7860.11
Orah (No Augmented Data)61.4558.03
MethodDiagnosis AccuracyAnalysis CompletenessAnalysis RelevanceLead Evidence ValidityECG Feature GroundingEvidence-Based ReasoningClinical Diagnostic Fidelity
Orah (SFT)80.316.725.737.3485.0779.9885.97
Orah (No Augmented Data)80.296.755.546.7285.1480.0384.76

Without these examples, Orah matches ECG-R1 (RL) in Diagnosis Accuracy (80.29) and scores higher on its other six metrics. ECG-QA-CoT F1 and accuracy decrease by 2.33 and 2.08 points, respectively. On ECG-Reasoning, Orah without the augmented examples achieves IDA of 65.54% on PTB-XL and 60.26% on MIMIC-IV-ECG (Table 7). We acknowledge that Orah’s diagnostic performance on this benchmark remains limited without targeted supervision.

References

[1] Jin et al. ECG-R1: Protocol-Guided and Modality-Agnostic MLLM for Reliable ECG Interpretation. ICML 2026.

Reviewer_a4MSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ECG-Language Model ORAH, which connects a modified SigLIP2 encoder with Qwen3.5-4B and is trained over multiple stages using approximately 12.47 million samples. Experiments investigate the effects of training scale, pre-training, and minimal task-specific supervision: while overall performance is strong, improvements in diagnostic accuracy do not translate into reliable step-by-step reasoning capabilities.

Critical strengths
  1. Valuable empirical evidence on large-scale ECG–language training. The paper assembles approximately 12.47 million training examples and compares 25%, 50%, and 100% data scales under a fixed model architecture and training recipe. Full-data training achieves broadly better performance across three benchmarks, providing useful evidence for scaling ECG–language models and a substantive reason to support acceptance.

  2. An informative finding on the complementarity of scale and targeted supervision. Adding approximately 3,000 benchmark-specific examples from the opposing ECG source substantially improves initial diagnosis accuracy (Figure 5, Table 7). This cross-source result demonstrates that large-scale training does not eliminate the value of targeted supervision, offering practical guidance for dataset construction and training-resource allocation.

Critical weaknesses
  1. The robustness of the scaling results is insufficiently established. Four ECG-R1 metrics decline when training data increases from 25% to 50%, before improving at 100% (Figure 6, Table 6). The paper reports neither repeated experiments nor uncertainty estimates and provides limited explanation for this non-monotonic behavior. Consequently, it remains unclear whether these differences exceed experimental variability, limiting confidence in the robustness of the scaling trends.

  2. The benefits of the training recipe are not fully disentangled. The pretraining ablation removes both ORAH pretraining stages and their associated computational investment (Figure 7), leaving unclear whether pretraining outperforms additional SFT under a matched compute budget. Direct comparisons supporting the two-stage pretraining and two-stage SFT schedules are also absent. These limitations do not invalidate the overall recipe but constrain the paper’s ability to provide general guidance on training strategies and budget allocation.

Questions for authors
  1. Robustness of scaling results. When training data was increased from 25% to 50%, the four metrics on the ECG-R1 Benchmark decreased, and then increased at 100% data (§4.4, Figure 6, Appendix Table 6). Were multiple random seeds, repeated data sampling, or repeated evaluations performed for each scale setting? Can the authors provide uncertainty estimates, or further discuss the possible reasons for these non-monotonic changes, to clarify the robustness of the scaling conclusions?

  2. Benefits and computational cost of ORAH pre-training. The existing ablation simultaneously removes both ORAH pre-training stages and their corresponding computational investment (§4.5, Figure 7). Under the same compute budget, does pre-training plus SFT still outperform directly increasing the SFT training amount? If no such controlled comparison exists, please clarify the boundary of conclusions that can be supported by the existing experiments.

  3. Design rationale for staged training. Can the authors further explain the design rationale for the two-stage ORAH pre-training and two-stage SFT (§3.3)? Is there preliminary evidence showing that training the projector first followed by joint training is superior to direct joint training from the start, and that staged SFT is superior to single-stage SFT with mixed data? If such comparisons were not conducted, please distinguish between gains supported by existing experiments and design choices that remain unverified.

Clarifications on the above issues, especially the robustness of scaling results and the interpretation of pre-training benefits, would help me re-evaluate this paper.

2 comments
AuthorsOpenReview ↗

Rebuttal to Review Part 1

Weakness 1 and Question 1

We thank the reviewer for pointing this out. We exactly follow the evaluation protocol of ECG-R1 [1]. Our inspection of its released evaluation scripts found no explicit procedure for averaging results across multiple random seeds, repeatedly subsampling the test set, or aggregating repeated runs under the same evaluation configuration. Thus, we provide new evaluations across three random seeds and report the mean ± standard deviation below. We did not repeat model training or resample the training subsets.

MethodDiagnosis AccuracyAnalysis CompletenessAnalysis RelevanceLead Evidence ValidityECG Feature GroundingEvidence-Based ReasoningClinical Diagnostic Fidelity
Orah (SFT)80.37 ± 0.176.69 ± 0.045.74 ± 0.087.41 ± 0.1885.01 ± 0.0980.03 ± 0.0685.91 ± 0.07
Orah (SFT, 25%)78.27 ± 0.094.66 ± 0.034.04 ± 0.055.08 ± 0.1874.13 ± 0.0470.84 ± 0.3574.20 ± 0.72
Orah (SFT, 50%)78.93 ± 0.223.81 ± 0.073.55 ± 0.214.17 ± 0.0671.02 ± 0.0977.25 ± 0.1479.15 ± 0.21

The decreases in these four metrics remain after repeated evaluation, suggesting that evaluation variability alone does not explain them. Differences in the training subsets and training randomness may contribute, but we have not tested these explanations. The 100% setting still achieves the highest mean across all seven metrics, supporting the benefit of using the full training set. We note that this quantifies evaluation variability, rather than uncertainty across independent training runs. We will update the manuscript with these reflections.

Weakness 2 and Question 2

We thank the reviewer for raising this important question. We added an SFT-only baseline with a comparable estimated compute budget by increasing each SFT stage from three to five epochs. Approximately 4.4 epochs per stage would match the estimated budget of pretraining plus SFT (see Table 4). For simplicity, we rounded up to five epochs, giving the SFT-only baseline a slightly larger budget. Results on ECG-QA-CoT and ECG-R1 are shown below, respectively:

MethodF1Accuracy
Orah (SFT)63.7860.11
Orah (SFT, No Pretraining)60.6456.39
Orah (SFT, No Pretraining and Similar Compute Budget)62.5858.23
MethodDiagnosis AccuracyAnalysis CompletenessAnalysis RelevanceLead Evidence ValidityECG Feature GroundingEvidence-Based ReasoningClinical Diagnostic Fidelity
Orah (SFT)80.316.725.737.3485.0779.9885.97
Orah (SFT, No Pretraining)81.343.974.044.2167.0964.8365.62
Orah (SFT, No Pretraining and Similar Compute)79.515.485.635.2678.8977.4380.68

Longer SFT narrows the performance gap, but pretraining plus SFT still outperforms the five-epoch SFT-only baseline on every reported metric. These results support the benefit of allocating compute to Orah pretraining rather than additional passes over the same SFT data under our evaluated training recipe. However, this comparison does not establish whether pretraining outperforms using additional unique SFT examples or identify the optimal allocation of compute. We leave this exploration to future works.

References

[1] Jin et al. ECG-R1: Protocol-Guided and Modality-Agnostic MLLM for Reliable ECG Interpretation. ICML 2026.

AuthorsOpenReview ↗

Rebuttal to Review Part 2

Question 3

We thank the reviewer for raising this important question. Our training recipe is inspired by Qwen3-VL [1]. Qwen3-VL's first pretraining stage updates only the projector to bridge the modality gap between the vision encoder and LLM. Similarly, Orah’s first pretraining stage aims to align the representation spaces of SigLEP and the LLM, followed by joint training of the projector and LLM in the second stage. Prior studies report conflicting results on the benefit of projector-only alignment [2, 3, 4], suggesting that its usefulness depends on the model, data, and task. We have not conducted a controlled comparison against direct joint training from the start.

Qwen3-VL's second pretraining stage updates all parameters, including the vision encoder [1]. We initially tried this for Orah pretraining stage 2 and report the following results on ECG-QA-CoT and ECG-R1 Benchmark, respectively:

MethodF1Accuracy
Orah (Only LLM and MLP)63.7860.11
Orah (All Params.)63.4359.57
MethodDiagnosis AccuracyAnalysis CompletenessAnalysis RelevanceLead Evidence ValidityECG Feature GroundingEvidence-Based ReasoningClinical Diagnostic Fidelity
Orah (Only LLM and MLP)80.316.725.737.3485.0779.9885.97
Orah (All Params.)79.826.615.727.3084.4580.0284.78

We see that updating all parameters did not yield consistently better performance. Therefore, to save compute, we decided to freeze SigLEP for the second pretraining stage, as well as all subsequent stages. This comparison supports our decision to freeze SigLEP, but does not establish the benefit of projector-only alignment over direct joint training.

For SFT, whether staged training or a single stage with mixed datasets is better has also been a subject of debate [5, 6]. We believe this choice also depends on the model, data, and task. We have not conducted a controlled comparison against single-stage SFT using the combined training data. Our rationale for the two SFT stages is to first establish instruction-following and conversational behavior using QA and conversation examples. In SFT 2, we include different subsets from some SFT 1 datasets to help reduce forgetting, alongside additional instruction data and explicit reasoning traces. This progression is consistent with prior work that introduces reasoning-enriched supervision after initial SFT [7]. We will update the manuscript to include these rationales.

The benefits of projector-first training over direct joint training and staged SFT over single-stage mixed-data SFT therefore remain unverified in our work. We acknowledge this limitation and leave these controlled comparisons to future work.

References

[1] Qwen Team. Qwen3-VL Technical Report. arxiv 2025.

[2] Gröger et al. Rethinking Projector Training in Multimodal LLMs. NeurIPS 2026.

[3] Karamcheti et al. Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models. ICML 2024.

[4] Tong et al. Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs. NeurIPS 2024.

[5] Dong et al. How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition. ACL Main 2024.

[6] Zhu et al. SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs. arxiv 2026.

[7] Goncharov et al. Complexity-aware fine-tuning. EACL Findings 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewers,

We thank the reviewers for their time and thoughtful feedback. We have revised the manuscript to incorporate the additional experiments and clarifications presented in our rebuttal. All changes are highlighted in blue in the manuscript. The main updates are:

  1. Repeated evaluations of scaling results. We added ECG-R1 evaluations across three random seeds and report means and standard deviations. The full-data setting achieves the highest mean on all seven metrics. We clarify that these results quantify evaluation variability, while variability across independent training runs remains unmeasured.

  2. Additional training ablations. We added a baseline without Orah pretraining that increases each SFT stage from three to five epochs, giving it a slightly larger training-token budget. Longer SFT narrows the gap, but pretraining plus SFT performs better on every reported metric. We also report the comparison supporting our decision to freeze SigLEP during Orah pretraining 2.

  3. Clearer training rationale and baseline comparisons. We expanded the explanation of each training stage, dataset allocation, and sampling proportions, added comparisons with baseline training procedures, and reported epoch-level training losses. We explicitly distinguish these design rationales from experimentally verified benefits.

  4. Diagnostic performance without additional ECG-Reasoning supervision. We added evaluations on the ECG-QA-CoT and ECG-R1 benchmarks using a model trained without the additional ECG-Reasoning examples. We also clarified Orah’s diagnostic performance and remaining limitations without this supervision.

  5. Clearer contributions and limitations. We clarified the framing around our empirical study of data scale, pretraining, and transfer, and expanded discussion of normalization trade-offs, mixed RL results, and training choices that remain unverified.

We appreciate the opportunity to improve the paper through this discussion. We welcome further questions and suggestions and would be happy to run additional experiments to address remaining concerns before the November 18 discussion deadline. If our responses and revisions have sufficiently addressed your concerns, we would appreciate your reconsideration of your assessment.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.