- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
ECGQuest is a literature-grounded True/False benchmark for evaluating and fine-tuning ECG-specific language models. A GPT-4o pipeline generates questions from 23 ECG references and Computing in Cardiology proceedings (2003–2025), yielding 10,904 unique True/False questions paired with their negated forms from the same source. The authors evaluate three commercial and twenty open-source models zero-shot, plus encoder baselines, and fine-tune five 7–14B open-source models with LoRA. They report zero-shot accuracy ranges, directional True/False bias, fine-tuning gains, and voting-ensemble results, with generalization checked on ECG-related subsets of MedMCQA and MedQA.
Critical strengths
- The topical coverage addresses a real gap. Spanning clinical, electrophysiological, signal-processing, and instrumentation knowledge, it goes meaningfully beyond existing evaluations that test either broad medical knowledge or the reading of individual ECG traces. This is the right problem to attack and is why the benchmark would be valuable if validated.
- The finding that general-purpose models outperform medically specialized ones is genuinely interesting and worth the community's attention, regardless of how one interprets its cause.
Critical weaknesses
- Dataset validity — the central contribution of a benchmark paper — is not established. Questions are generated by GPT-4o from parsed literature with no reported clinician validation, no inter-annotator agreement, and no error analysis of the generated items. Compounding this, every question appears in both affirmative and negated form from the same source, so models can plausibly exploit superficial negation cues rather than ECG knowledge; the paper reports directional bias as a finding but does not show that the reported accuracies are free of this artifact. Since essentially every conclusion in the paper rests on these numbers, this is decisive for my assessment and cannot be resolved without re-running the evaluation with human-verified labels and a de-paired control.
- The paper is not adequately situated against the ECG language-modeling literature. Recent ECG-MLLMs and ECG representation-learning work (e.g., GEM, D-BETA, FG-CLEP, anyECG-chat, MERL) is not discussed, which leaves unclear what capability gap this benchmark targets beyond what existing models already demonstrate. This also matters substantively: it is known that a substantial share of ECG reports omit detailed waveform descriptions, and since questions here are derived from text, whether the benchmark can actually probe waveform-level understanding deserves discussion rather than assumption.
Questions for authors
- What fraction of generated questions was verified by human experts, and what is the estimated label error rate? Were any items discarded during validation?
- Could you report accuracy separately on affirmative versus negated items, and on a de-paired subset, to demonstrate the numbers are not driven by negation cues?
- How should one interpret the finding that general-purpose models beat medically specialized ones — does the benchmark measure ECG knowledge, or general reading and reasoning ability?
- Could you relate ECGQuest to recent ECG-MLLMs and representation-learning methods, and clarify what capability gap it targets beyond what those models already show?
3 comments
To address this gap, we constructed ECGQuest using specialized ECG references and Computing in Cardiology proceedings from 2003 to 2025. The resulting dataset covers ECG diagnoses, waveform characteristics, acquisition technologies, and basic cardiovascular physiology. Unlike datasets designed to associate ECG signals with diagnoses or clinical reports, ECGQuest provides a literature-grounded resource for evaluating and adapting the domain-specific knowledge of language models independently of ECG input modality. Although ECGQuest can be used to evaluate ECG-specific factual knowledge through its weakly labeled questions, its primary goal is to support fine-tuning language models and transferring ECG-specific knowledge to them, enabling the acquired knowledge to be applied beyond the ECGQuest benchmark. Our experiments demonstrate improvements on ECGQuest after LoRA fine-tuning, while external evaluation on ECG-related questions from MedQA and MedMCQA provides evidence that these improvements can extend to independently sourced medical questions. These findings support ECGQuest's construction goal as a domain-adaptation resource rather than merely a benchmark for its own question format. Therefore, we view ECGQuest as complementary dataset for ECG related interpretations. Whereas multimodal approaches aim to learn how ECG signals and images correspond to clinically meaningful observations, ECGQuest supports fine-tuning models on the ECG-specific knowledge required to interpret those observations. In future work, ECGQuest-adapted models could be integrated with ECG encoders or signal-processing pipelines to interpret structured physiological measurements. Thus, while ECGQuest itself is the primary contribution of this paper, our external evaluation provides initial evidence that the dataset can serve its intended purpose of transferring ECG-specific knowledge to language models.
Regarding Question 3 (general-purpose versus medically specialized models): We agree that performance on ECGQuest may reflect both ECG-specific factual knowledge and general language comprehension and reasoning. However, medical specialization does not necessarily guarantee comprehensive knowledge of electrocardiography, which spans cardiovascular physiology, electrophysiology, signal processing, and recording technologies. Differences in model architecture, training objectives, and pretraining data may also contribute to the observed performance differences. Importantly, our primary objective is not to establish that general-purpose models are superior to medically specialized models, but to investigate whether ECG-specific knowledge can be transferred through fine-tuning. By comparing base and ECGQuest-fine-tuned versions of the same models on external ECG-related questions from MedQA and MedMCQA, we provide additional evidence that the learned information can improve performance beyond the original benchmark.
Regarding Question 1 (dataset validation and label reliability):
We agree that dataset validity is an important consideration. ECGQuest was developed using a literature-grounded generation and automated validation procedure, rather than relying solely on the language model's internal knowledge. During this process, 638 generated responses were rejected, and 10,904 True/False question pairs were accepted. We have further clarified the generation and validation procedures and added a representative accepted question pair, including its source text and corresponding labels, to the Supplementary Materials. We acknowledge that ECGQuest is a weakly labeled dataset rather than a fully clinician-validated benchmark. Consequently, we do not claim that automated validation alone establishes perfect label accuracy. Nevertheless, because the primary objective of ECGQuest is knowledge transfer through fine-tuning, the improvements observed on external ECG-related medical questions provide complementary evidence of its usefulness, although they do not replace direct expert validation of the generated labels. Furthermore, to provide additional evidence of the effectiveness of ECGQuest-based fine-tuning, we evaluated the models on the 1,050 original, held-out source passages from which the test questions were generated. These passages were extracted directly from published ECG references and were not used during fine-tuning. As shown in Table 3, the fine-tuned models demonstrated good performance in recognizing these original ECG facts. These provide additional evidence that ECGQuest-based fine-tuning can help models recognize ECG-related factual information beyond the generated question format, supporting the intended objective of ECG-specific knowledge transfer.
Regarding Question 2 (affirmative versus negated items and de-paired evaluation):
We thank the reviewer for this suggestion. We would like to clarify that the results reported in Table 1 already correspond to a de-paired evaluation. Specifically, the ECGQuest test set contains 1,050 True/False question pairs (2,100 statements), from which one statement per pair was selected for evaluation. Therefore, no question pair contributes both its true and false versions to the reported results, reducing the possibility that models used relationships between paired statements. Furthermore, to investigate potential label bias, we conducted an additional analysis reporting accuracy separately for the true and false statements, as presented in the newly added source-grounded consistency analysis (Table 3). The results show substantial True/False prediction imbalance in some zero-shot models, which is considerably reduced following ECGQuest fine-tuning. For example, the True/False accuracy gap decreased from 67.5 to 0.8 percentage points for Qwen2.5-14B and from 73.5 to 5.2 percentage points for BioMistral-7B. These findings suggest that the improvements after fine-tuning are not solely attributable to a preference for predicting one label.
We have revised the manuscript to clarify these distinctions, acknowledge the limitations of weakly labeled questions, and emphasize that ECGQuest targets ECG-specific factual knowledge rather than directly establishing waveform-level understanding or clinical diagnostic performance.
We thank the reviewer for these valuable questions and for highlighting recent advances in ECG multimodal language models and representation learning. We agree that these developments are highly relevant to our work, and we have expanded the Related Work and Introduction sections to clarify the distinction between ECGQuest and existing ECG interpretation frameworks.
Regarding Question 4 (relationship to recent ECG-MLLMs and the capability gap): Recent approaches, including MERL, D-BETA, FG-CLEP, GEM, and anyECG-chat, have made important contributions to ECG representation learning, waveform interpretation, and multimodal reasoning. These efforts demonstrate the potential of integrating physiological signals with language models for automated ECG analysis. However, we believe ECG-specific factual knowledge is another important capability to study separately, particularly given the diversity of ECG acquisition and presentation formats. ECG recordings are not always represented in a uniform format. For example, datasets such as PTB-XL provide simultaneous 12-lead recordings, whereas many clinical ECG devices display recordings in a 4 × 2.5-second layout accompanied by a continuous 10-second rhythm strip. In this latter representation, different groups of leads are displayed over different temporal windows rather than as twelve simultaneously displayed 10-second signals. This different format has resulted in large and valuable archives of ECG records, which motivated international challenges such as the George B. Moody PhysioNet Challenge 2024 and the Kaggle ECG Image Digitization Competition , which focus on recovering ECG signals from images of this format to have challenge for recovering these signals. Other acquisition and presentation formats, including reduced-lead and variable-duration recordings, introduce further diversity. Although recent multimodal approaches have made progress toward ECG interpretation, developing systems that robustly interpret these different representations remains challenging. An additional difficulty is the limited availability of comprehensive datasets covering these variations with corresponding clinical interpretations. One complementary direction is to use structured physiological measurements extracted from ECG recordings as inputs to language models. Established signal-processing methods and automated ECG analysis systems can extract clinically meaningful features, such as RR intervals, QRS durations, QT intervals, heart rate, electrical axes, and waveform amplitudes. These measurements can serve as an interpretable intermediate representation for a language model trained on ECG-specific knowledge to generate clinical interpretations. This approach is consistent with an important aspect of clinical ECG interpretation, in which cardiologists identify meaningful waveform characteristics and measurements and interpret them using their medical knowledge and clinical experience. From this perspective, developing ECG-aware language models involves two complementary capabilities: (1) understanding ECG-specific knowledge, including physiology, electrophysiology, waveform characteristics, diagnostic principles, and recording technologies, and (2) applying this knowledge to interpret patient-specific ECG measurements. While recent ECG-MLLMs primarily focus on ECG signal understanding and clinical interpretation, ECGQuest specifically targets the first capability. This distinction motivated the development of ECGQuest. Although existing general medical question-answering datasets, including MedQA and MedMCQA, contain ECG-related questions, they were designed to assess broader medical knowledge rather than systematically cover ECG-specific concepts. Even after filtering these datasets for ECG-related content, as described in our paper, the resulting subsets remain relatively limited and are insufficient for comprehensive ECG-specific fine-tuning.