acceptodds
Under review as a conference paper at ICLR 2027

ECGQuest: Benchmarking and Fine-Tuning Language Models for Electrocardiography

Abstract

Electrocardiogram (ECG) interpretation requires knowledge of cardiology, electrophysiology, clinical diagnosis, ECG waveforms, signal acquisition, and ECG instrumentation. While large language models have proven effective for such applications, existing language-models primarily assess broad medical knowledge or the interpretation of individual ECG signals and images, rather than comprehensive context knowledge required for ECG interpretation. We built ECGQuest, a literature-grounded resource for evaluating and fine-tuning ECG-specific language models. An automated GPT-4o-based pipeline generated questions from 23 ECG references and proceedings of the Computing in Cardiology conference between 2003–2025. The final dataset contains 10,904 unique True/False questions paired with their negated form derived from the same source (21,808 Q&A in total). We evaluated three commercial and 20 open-source language models on the held-out ECGQuest test set in a zero-shot setting. We also fine-tuned five open-source models with 7–14 B parameters using Low-Rank Adaptation and included BERT and BiomedBERT as supervised encoder baselines. Generalization was evaluated using ECG-related subsets of MedMCQA and MedQA datasets converted to binary True/False questions using their official examination answer keys. Zero-shot accuracy on ECGQuest ranged from 49.5% to 74.4%, with GPT-5 achieving the highest accuracy. General-purpose models outperformed all medically specialized models, and several models showed strong directional True/False biases. Encoder baselines performed near chance. Fine-tuning improved all open-source models on ECGQuest by 6.5–14.1%. The fine-tuned DeepSeek-R1-Distill-Qwen-14 B model achieved 76.3% accuracy, and a five-model voting ensemble achieved the highest overall accuracy of 78.5%. On MedMCQA and MedQA, fine-tuning primarily benefited weak or class-biased models and did not consistently improve already strong base models. ECGQuest is a reproducible benchmark for evaluating contextual ECG knowledge in language models and demonstrates that targeted, parameter-efficient fine-tuning can make small language models competitive with substantially larger commercial models.

Primary area:
datasets and benchmarks
Keywords:
Electrocardiography (ECG), Large Language Models (LLMs), Small Language Models (SLMs), Domain-specific benchmarking, ECG knowledge benchmarking, medical question answering, language model fine-tuning, parameter-efficient fine-tuning, Low-Rank Adaptation (LoRA)

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jABbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ECGQuest is a literature-grounded True/False benchmark for evaluating and fine-tuning ECG-specific language models. A GPT-4o pipeline generates questions from 23 ECG references and Computing in Cardiology proceedings (2003–2025), yielding 10,904 unique True/False questions paired with their negated forms from the same source. The authors evaluate three commercial and twenty open-source models zero-shot, plus encoder baselines, and fine-tune five 7–14B open-source models with LoRA. They report zero-shot accuracy ranges, directional True/False bias, fine-tuning gains, and voting-ensemble results, with generalization checked on ECG-related subsets of MedMCQA and MedQA.

Critical strengths
  • The topical coverage addresses a real gap. Spanning clinical, electrophysiological, signal-processing, and instrumentation knowledge, it goes meaningfully beyond existing evaluations that test either broad medical knowledge or the reading of individual ECG traces. This is the right problem to attack and is why the benchmark would be valuable if validated.
  • The finding that general-purpose models outperform medically specialized ones is genuinely interesting and worth the community's attention, regardless of how one interprets its cause.
Critical weaknesses
  • Dataset validity — the central contribution of a benchmark paper — is not established. Questions are generated by GPT-4o from parsed literature with no reported clinician validation, no inter-annotator agreement, and no error analysis of the generated items. Compounding this, every question appears in both affirmative and negated form from the same source, so models can plausibly exploit superficial negation cues rather than ECG knowledge; the paper reports directional bias as a finding but does not show that the reported accuracies are free of this artifact. Since essentially every conclusion in the paper rests on these numbers, this is decisive for my assessment and cannot be resolved without re-running the evaluation with human-verified labels and a de-paired control.
  • The paper is not adequately situated against the ECG language-modeling literature. Recent ECG-MLLMs and ECG representation-learning work (e.g., GEM, D-BETA, FG-CLEP, anyECG-chat, MERL) is not discussed, which leaves unclear what capability gap this benchmark targets beyond what existing models already demonstrate. This also matters substantively: it is known that a substantial share of ECG reports omit detailed waveform descriptions, and since questions here are derived from text, whether the benchmark can actually probe waveform-level understanding deserves discussion rather than assumption.
Questions for authors
  1. What fraction of generated questions was verified by human experts, and what is the estimated label error rate? Were any items discarded during validation?
  2. Could you report accuracy separately on affirmative versus negated items, and on a de-paired subset, to demonstrate the numbers are not driven by negation cues?
  3. How should one interpret the finding that general-purpose models beat medically specialized ones — does the benchmark measure ECG knowledge, or general reading and reasoning ability?
  4. Could you relate ECGQuest to recent ECG-MLLMs and representation-learning methods, and clarify what capability gap it targets beyond what those models already show?
3 comments

We thank the reviewer for these valuable questions and for highlighting recent advances in ECG multimodal language models and representation learning. We agree that these developments are highly relevant to our work, and we have expanded the Related Work and Introduction sections to clarify the distinction between ECGQuest and existing ECG interpretation frameworks.

Regarding Question 4 (relationship to recent ECG-MLLMs and the capability gap): Recent approaches, including MERL, D-BETA, FG-CLEP, GEM, and anyECG-chat, have made important contributions to ECG representation learning, waveform interpretation, and multimodal reasoning. These efforts demonstrate the potential of integrating physiological signals with language models for automated ECG analysis. However, we believe ECG-specific factual knowledge is another important capability to study separately, particularly given the diversity of ECG acquisition and presentation formats. ECG recordings are not always represented in a uniform format. For example, datasets such as PTB-XL provide simultaneous 12-lead recordings, whereas many clinical ECG devices display recordings in a 4 × 2.5-second layout accompanied by a continuous 10-second rhythm strip. In this latter representation, different groups of leads are displayed over different temporal windows rather than as twelve simultaneously displayed 10-second signals. This different format has resulted in large and valuable archives of ECG records, which motivated international challenges such as the George B. Moody PhysioNet Challenge 2024 and the Kaggle ECG Image Digitization Competition , which focus on recovering ECG signals from images of this format to have challenge for recovering these signals. Other acquisition and presentation formats, including reduced-lead and variable-duration recordings, introduce further diversity. Although recent multimodal approaches have made progress toward ECG interpretation, developing systems that robustly interpret these different representations remains challenging. An additional difficulty is the limited availability of comprehensive datasets covering these variations with corresponding clinical interpretations. One complementary direction is to use structured physiological measurements extracted from ECG recordings as inputs to language models. Established signal-processing methods and automated ECG analysis systems can extract clinically meaningful features, such as RR intervals, QRS durations, QT intervals, heart rate, electrical axes, and waveform amplitudes. These measurements can serve as an interpretable intermediate representation for a language model trained on ECG-specific knowledge to generate clinical interpretations. This approach is consistent with an important aspect of clinical ECG interpretation, in which cardiologists identify meaningful waveform characteristics and measurements and interpret them using their medical knowledge and clinical experience. From this perspective, developing ECG-aware language models involves two complementary capabilities: (1) understanding ECG-specific knowledge, including physiology, electrophysiology, waveform characteristics, diagnostic principles, and recording technologies, and (2) applying this knowledge to interpret patient-specific ECG measurements. While recent ECG-MLLMs primarily focus on ECG signal understanding and clinical interpretation, ECGQuest specifically targets the first capability. This distinction motivated the development of ECGQuest. Although existing general medical question-answering datasets, including MedQA and MedMCQA, contain ECG-related questions, they were designed to assess broader medical knowledge rather than systematically cover ECG-specific concepts. Even after filtering these datasets for ECG-related content, as described in our paper, the resulting subsets remain relatively limited and are insufficient for comprehensive ECG-specific fine-tuning.

To address this gap, we constructed ECGQuest using specialized ECG references and Computing in Cardiology proceedings from 2003 to 2025. The resulting dataset covers ECG diagnoses, waveform characteristics, acquisition technologies, and basic cardiovascular physiology. Unlike datasets designed to associate ECG signals with diagnoses or clinical reports, ECGQuest provides a literature-grounded resource for evaluating and adapting the domain-specific knowledge of language models independently of ECG input modality. Although ECGQuest can be used to evaluate ECG-specific factual knowledge through its weakly labeled questions, its primary goal is to support fine-tuning language models and transferring ECG-specific knowledge to them, enabling the acquired knowledge to be applied beyond the ECGQuest benchmark. Our experiments demonstrate improvements on ECGQuest after LoRA fine-tuning, while external evaluation on ECG-related questions from MedQA and MedMCQA provides evidence that these improvements can extend to independently sourced medical questions. These findings support ECGQuest's construction goal as a domain-adaptation resource rather than merely a benchmark for its own question format. Therefore, we view ECGQuest as complementary dataset for ECG related interpretations. Whereas multimodal approaches aim to learn how ECG signals and images correspond to clinically meaningful observations, ECGQuest supports fine-tuning models on the ECG-specific knowledge required to interpret those observations. In future work, ECGQuest-adapted models could be integrated with ECG encoders or signal-processing pipelines to interpret structured physiological measurements. Thus, while ECGQuest itself is the primary contribution of this paper, our external evaluation provides initial evidence that the dataset can serve its intended purpose of transferring ECG-specific knowledge to language models.

Regarding Question 3 (general-purpose versus medically specialized models): We agree that performance on ECGQuest may reflect both ECG-specific factual knowledge and general language comprehension and reasoning. However, medical specialization does not necessarily guarantee comprehensive knowledge of electrocardiography, which spans cardiovascular physiology, electrophysiology, signal processing, and recording technologies. Differences in model architecture, training objectives, and pretraining data may also contribute to the observed performance differences. Importantly, our primary objective is not to establish that general-purpose models are superior to medically specialized models, but to investigate whether ECG-specific knowledge can be transferred through fine-tuning. By comparing base and ECGQuest-fine-tuned versions of the same models on external ECG-related questions from MedQA and MedMCQA, we provide additional evidence that the learned information can improve performance beyond the original benchmark.

Regarding Question 1 (dataset validation and label reliability):

We agree that dataset validity is an important consideration. ECGQuest was developed using a literature-grounded generation and automated validation procedure, rather than relying solely on the language model's internal knowledge. During this process, 638 generated responses were rejected, and 10,904 True/False question pairs were accepted. We have further clarified the generation and validation procedures and added a representative accepted question pair, including its source text and corresponding labels, to the Supplementary Materials. We acknowledge that ECGQuest is a weakly labeled dataset rather than a fully clinician-validated benchmark. Consequently, we do not claim that automated validation alone establishes perfect label accuracy. Nevertheless, because the primary objective of ECGQuest is knowledge transfer through fine-tuning, the improvements observed on external ECG-related medical questions provide complementary evidence of its usefulness, although they do not replace direct expert validation of the generated labels. Furthermore, to provide additional evidence of the effectiveness of ECGQuest-based fine-tuning, we evaluated the models on the 1,050 original, held-out source passages from which the test questions were generated. These passages were extracted directly from published ECG references and were not used during fine-tuning. As shown in Table 3, the fine-tuned models demonstrated good performance in recognizing these original ECG facts. These provide additional evidence that ECGQuest-based fine-tuning can help models recognize ECG-related factual information beyond the generated question format, supporting the intended objective of ECG-specific knowledge transfer.

Regarding Question 2 (affirmative versus negated items and de-paired evaluation):

We thank the reviewer for this suggestion. We would like to clarify that the results reported in Table 1 already correspond to a de-paired evaluation. Specifically, the ECGQuest test set contains 1,050 True/False question pairs (2,100 statements), from which one statement per pair was selected for evaluation. Therefore, no question pair contributes both its true and false versions to the reported results, reducing the possibility that models used relationships between paired statements. Furthermore, to investigate potential label bias, we conducted an additional analysis reporting accuracy separately for the true and false statements, as presented in the newly added source-grounded consistency analysis (Table 3). The results show substantial True/False prediction imbalance in some zero-shot models, which is considerably reduced following ECGQuest fine-tuning. For example, the True/False accuracy gap decreased from 67.5 to 0.8 percentage points for Qwen2.5-14B and from 73.5 to 5.2 percentage points for BioMistral-7B. These findings suggest that the improvements after fine-tuning are not solely attributable to a preference for predicting one label.

We have revised the manuscript to clarify these distinctions, acknowledge the limitations of weakly labeled questions, and emphasize that ECGQuest targets ECG-specific factual knowledge rather than directly establishing waveform-level understanding or clinical diagnostic performance.

Reviewer_M2SiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ECGQuest, a literature-grounded resource for evaluating and fine-tuning ECG-specific language models. The dataset is consists of 10,904 True/False question pairs. The questions are parsed and extracted by GPT-4o from 23 scientific ECG resources and specialized ECG conference proceedings, spanning various ECG-related topics. The authors evaluate a diverse set of proprietary, general-purpose, biomedical, and open-access language models on the held-out benchmark in zero-shot settings. They also fine-tune several open-source models on ECGQuest and investigate the resulting performance improvements and generalization to external medical question-answering benchmarks.

Critical strengths

S1. Literature-grounded data construction with source traceability.

S2. Broad empirical evaluation and investigation of fine-tuning.

Critical weaknesses

W1. In 2026, the contribution of a purely text-based ECG benchmark appears limited, particularly given the rapid development of LLMs and the extensive use of large-scale textual corpora in their pretraining and SFT. The literature used to construct ECGQuest may have already been covered, directly or indirectly, by existing training datasets, which raises questions about the dataset’s novelty and its ability to evaluate genuinely unseen ECG knowledge. Furthermore, representing the data as simple True/False question pairs provides limited contextual richness and does not require models to generate explanations or demonstrate their reasoning. This formulation may be more suitable for conventional encoder-based classification models than for training and evaluating modern generative LLMs. More importantly, ECG-related knowledge cannot be fully assessed independently of actual ECG waveforms. Consequently, the benchmark may primarily measure text-based factual recall rather than meaningful ECG understanding, limiting both its scientific contribution and its relevance to practical ECG applications.

W2. The absence of comprehensive expert validation is therefore another significant limitation. The dataset is constructed with the assistance of GPT-4o. This raises concerns about factual correctness and the clinical relevance of the questions and their corresponding answers. Although grounding questions in source materials improves traceability, it does not guarantee that the generated statements faithfully represent the sources or that their labels are clinically correct.

Questions for authors

Q1. How do the authors verify that ECGQuest is not included in other pretraining dataset? The questions are derived from published scientific literature and conference proceedings, which may be publicly accessible and potentially included in the pretraining corpora of evaluated LLMs. Can the authors provide evidence that the dataset is not already present in other medical pretraining dataset?

Q2. How do the authors ensure the correctness of the constructed True/False statements? Revising an incorrect statement into a correct one, may require specialized ECG knowledge. What procedures are used to verify that the resulting True statements are unambiguous and factually correct?

Q3. How do the authors address potential GPT-4o-related bias or leakage? GPT-4o is used to construct the benchmark, while it is also one of the evaluated models. Could the relatively strong performance of GPT-4o be partly attributable to its familiarity with the source literature or to biases introduced by using the same model family for both data construction and evaluation?

4 comments

We thank the reviewer for the valuable comments and questions and for recognizing the literature-grounded construction, source traceability, and extensive experimental evaluation of ECGQuest. We would like to clarify the primary objective of ECGQuest and address the concerns regarding pretraining overlap, label correctness, and potential GPT-4o-related bias. Regarding the broader concern about the contribution of a text-based True/False benchmark: We agree that text-based factual knowledge alone does not establish waveform-level understanding or clinical diagnostic capability. However, we believe that ECG-specific factual knowledge is a complementary capability that deserves dedicated investigation. Recent approaches, including MERL, D-BETA, FG-CLEP, GEM, and anyECG-chat, have made important contributions to ECG representation learning, waveform interpretation, and multimodal reasoning. Nevertheless, ECG recordings exist in diverse acquisition and presentation formats, including simultaneous 12-lead recordings, clinical 4 × 2.5-second layouts with continuous rhythm strips (this different format has resulted in large and valuable archives of ECG records, which motivated international challenges such as the George B. Moody PhysioNet Challenge 2024 and the Kaggle ECG Image Digitization Competition, which focus on recovering ECG signals from images of this format to have challenges for recovering these signals), reduced-lead recordings, and variable-duration signals. Developing models capable of interpreting these representations remains challenging (as each of them have a specific format of representation), particularly given the limited availability of comprehensive datasets pairing different ECG formats with clinical interpretations. One complementary direction is to extract clinically meaningful measurements, including RR intervals, QRS durations, QT intervals, electrical axes, and waveform amplitudes, and provide them to ECG-aware language models for interpretation. Developing such a model can be conceptualized as a two-stage process: (1) equipping the model with ECG-specific factual knowledge concerning cardiac physiology, electrophysiology, waveform characteristics, diagnostic principles, and recording technologies; and (2) further training the model to apply this acquired knowledge to interpret patient-specific ECG measurements, analogous to how cardiologists build upon their foundational knowledge to perform clinical ECG interpretation. ECGQuest specifically focuses on the first problem. In addition to its True/False formulation each pair has a source that can be used as an explanation for the question labels. As a result, it provides a structured and scalable resource for ECG-specific knowledge adaptation. Its intended contribution is therefore not limited to benchmarking factual recall, but includes supporting the transfer of ECG-specific knowledge to language models through fine-tuning. Our experiments demonstrate improvements on ECGQuest after LoRA fine-tuning, while the external evaluations provide initial evidence that these gains can extend beyond the generated question format. In future work, ECGQuest-adapted language models could be integrated with ECG encoders or signal-processing pipelines to support interpretation of structured physiological measurements.

Regarding Q1 (potential overlap with pretraining datasets): We agree that the ECG references and conference proceedings used to construct ECGQuest may have been included, directly or indirectly, in the pretraining corpora of existing language models. Since the complete pretraining datasets of many evaluated models are not publicly available, we cannot definitively exclude such overlap. However, we would like to emphasize that the primary objective of ECGQuest is not to evaluate models exclusively on previously unseen ECG facts, but to provide a dedicated resource for fine-tuning language models and transferring ECG-specific factual knowledge to them. Although general medical question-answering datasets such as MedQA and MedMCQA contain ECG-related questions, they were designed to assess broader medical knowledge rather than systematically cover ECG physiology, waveform characteristics, diagnostic principles, and acquisition technologies. Even after filtering these datasets for ECG-related content, the resulting subsets remain relatively limited. This motivated us to construct ECGQuest from specialized ECG references and conference proceedings. More importantly, to investigate whether ECGQuest-based fine-tuning transfers knowledge beyond the original benchmark, we conducted additional external evaluations on ECG-related questions from MedQA and MedMCQA, whose labels were derived from official examination answer keys independently of ECGQuest. As shown in the newly added external evaluation table, fine-tuning improved accuracy for several models, including DeepSeek-R1-14B (62.3% to 70.7%), Qwen2.5-14B (67.1% to 72.5%), and BioMistral-7B (51.0% to 59.7%), while these datasets have existed online and may have been used to train the models. These results provide evidence that ECGQuest-based fine-tuning can generally improve performance on independently sourced medical questions, rather than merely improving accuracy on its own generated benchmark.

Regarding Q2 (correctness of the generated True/False statements): We agree that ensuring the factual correctness of generated statements is essential. As described in the Methods section, ECGQuest employs a source-grounded generation and automated validation procedure in which GPT-4o generates statements exclusively from provided ECG source passages. Generated items were checked for source consistency, ambiguous labels, malformed content, missing fields, and references to unavailable figures, resulting in 638 rejected responses. Each accepted statement was then paired with an opposite-label counterpart derived from the same source passage, with constraints on factual modifications and programmatic verification of source consistency, label reversal, and statement differences. To further improve transparency, we have clarified these procedures and added a representative accepted True/False pair, including its source passage and corresponding labels, to the Supplementary Materials. Additionally, as the purpose of constructing this dataset is just to transfer the ECG knowledge as much as possible to fine-tuned model on this dataset to validate the dataset, we have done two other analysis on the finetuned models: 1- to investigate whether ECGQuest-based fine-tuning enables models to recognize the original factual information beyond the generated statements, we conducted a source-grounded evaluation using the 1,050 held-out source passages. These original passages were presented independently of their corresponding True/False questions and were not included in the fine-tuning data. As shown in Table 4, the fine-tuned models demonstrated source-passage acceptance rate improvement. These findings provide additional evidence that ECGQuest-based fine-tuning can support transfer of ECG-related information. 2- Also, the fine-tuned models perform well on the external dataset, for which the labels were human-validated. This ensures that fine-tuning on this data can enable ECG knowledge to be transferred to the fine-tuned model. Nevertheless, we acknowledge that independent clinical expert validation could further improve the factual reliability of ECGQuest and potentially enhance knowledge transfer during fine-tuning. While ECGQuest remains a weakly labeled, literature-grounded resource rather than a fully clinician-validated dataset, the observed improvements on external human-validated datasets provide evidence that ECG-related knowledge has been successfully transferred to the fine-tuned models, despite the limitations of automated validation.

Regarding Q3 (potential GPT-4o-related bias or leakage): We would like to clarify that GPT-4o was primarily used to assist in constructing ECGQuest by generating questions from published ECG references. Its inclusion in the zero-shot evaluation was intended to provide a more comprehensive comparison across different language models, rather than to establish or validate the main conclusions of our study. In addition to GPT-4o, we evaluated a broad range of general-purpose and medically specialized language models in the zero-shot setting. Importantly, our main conclusions do not rely on GPT-4o's performance on ECGQuest, but rather on the effectiveness of ECGQuest as a resource for fine-tuning language models and transferring ECG-specific factual knowledge. These conclusions are supported by the improvements observed after LoRA fine-tuning across open-source models and by the additional evaluation on independently sourced ECG-related questions from MedQA and MedMCQA. Therefore, although we acknowledge that GPT-4o's involvement in dataset construction may introduce potential generation-related biases, its zero-shot evaluation was included for completeness and was not used as evidence to establish the effectiveness of ECGQuest-based knowledge transfer.

Reviewer_TBLwOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

ECGQuest is a benchmark of boolean questions on ECGs. GPT-4o generated questions, their "false counterparts", the verification and the categories from textbooks, manuals and tutorials. The authors evaluate 23 language models (3 commercial, 20 open weight). They also fine-tuned five open models, form an ensemble of the five (55B parameters in total), and test on external MedMCQA and MedQA. They conclude that general-purpose models beat medically specialized ones, that fine-tuning improves all open-source models by 6.5–14.1%. PEFT can make small models competitive with substantially larger models.

Critical strengths

General medical QA benchmarks do not measure conceptual ECG knowledge. The paired true/false, pair-level splitting, release of the generation and validation prompts, and the reporting of sensitivity/specificity, as well as further metrics follow good practice.

23 models with different sizes, and an external check that shows where fine-tuning does not transfer.

Critical weaknesses

Benchmark validity is not established. Every item is produced by GPT-4o, and the Limitations section admits "weak labels". Still there is no human audit of even a small random sample, no details from generated to accepted items (638 are mentioned as rejected), and no examples of accepted items. False items are made by altering one number or term (e.g. 120–200 ms → 120–400 ms) and may be findable by plausibility, thus the benchmark probably measures general knowledge rather than "contextual ECG knowledge". Risk of self-preference (GPT-4o generates data, GPT-family models are evaluated). The test mix differs strongly from train (ECG diagnosis 69.7% test vs 60.2% train; basic physiology 4.1% vs 10.9%).

The central conclusions are over-stated and sometimes counter-intuitive. "Competitive with substantially larger commercial models": on the external sets, fine-tuning lowers accuracy for the strongest model, and the ensemble performance is below GPT-4o. The text states that all five fine-tuned models are balanced, but the sensitivity/specificity analysis show several fine-tuned models biased toward True. "Medical specialization did not help" is not quite true: MedGemma-4B beats its own base Gemma-3-4B on ECGQuest and other datasets. results non-reproducible limited scientific contribution.

Questions for authors

Can you report a human audit on a random sample of test items (e.g. 200–300, two clinicians or ECG experts)? How was the train/test split performed; at random, or grouped by source? On which data were fine-tuning hyperparameters and the class weights selected? If on the test set, please re-run with a held-out validation set. Why does the abstract say three commercial when four closed models are in Table 1? Where is the significance test behind "significantly outperformed GPT-3.5"? Please state exactly what will be released and when.

6 comments

We thank the reviewer for these observations. Regarding potential GPT-4o-related bias, we clarify that GPT-4o was used primarily for dataset construction, while its inclusion in the zero-shot evaluation (Table 1) was intended to provide a comprehensive comparison across language models. Our main conclusions do not rely on GPT-4o's performance but on the effectiveness of ECGQuest-based fine-tuning and external evaluation. Importantly, our updated external evaluation on ECG-related questions from MedQA and MedMCQA (Table 5) provides further evidence of ECG-specific knowledge transfer beyond the original benchmark. These results demonstrate that ECGQuest-based fine-tuning can improve performance on independently sourced medical questions rather than merely adapting models to the generated True/False statements. Finally, we agree that medical specialization can improve performance, as demonstrated in Table 1, where MedGemma-4B outperformed Gemma-3-4B (59.5% versus 56.4%). Our intended conclusion is therefore not that medical specialization is ineffective, but rather that it does not consistently guarantee superior performance on ECG-specific knowledge tasks.

Q1. Can you report a human audit on a random sample of test items (e.g., 200–300, two clinicians or ECG experts)? We thank the reviewer for this suggestion. ECGQuest was constructed using a literature-grounded generation and automated validation procedure, where each question was linked to its original source passage. During validation, 638 generated responses were rejected, resulting in 10,904 accepted True/False question pairs. We acknowledge that the labels have not undergone independent clinical expert validation and should therefore be considered weak labels. However, we would like to clarify the primary motivation behind ECGQuest. ECG interpretation can be approached through multimodal language models that directly process ECG signals or images, or through a measurement-based framework that more closely resembles an important aspect of clinical ECG interpretation. Given the diversity of ECG acquisition and presentation formats, developing a single multimodal model capable of reliably interpreting all ECG representations is extremely challenging. Furthermore, comprehensive datasets pairing every ECG presentation format with corresponding clinical interpretations are not readily available. An alternative is to first extract clinically meaningful measurements, such as QRS durations, QT intervals, electrical axes, and waveform amplitudes, and subsequently interpret them, as cardiologists commonly do when assessing ECG findings. This framework can be divided into two stages: (1) transferring ECG-specific factual knowledge to a language model and (2) adapting the resulting model to interpret patient-specific ECG measurements. ECGQuest was developed to address the first stage. Existing medical question-answering datasets, including MedQA and MedMCQA, primarily cover general medical knowledge. Even after filtering them for ECG-related questions, as demonstrated in our manuscript, the resulting subsets remain relatively small and insufficient for comprehensive ECG-specific knowledge transfer through fine-tuning. Therefore, our primary objective was to construct a dedicated dataset capable of transferring ECG-specific factual knowledge to language models. Although ECGQuest has not yet undergone independent expert label validation, our additional experiments provide encouraging evidence of its effectiveness. As shown in Table 4, the fine-tuned models demonstrated good performance on the 1,050 original, held-out ECG source passages. More importantly, our external evaluations on ECG-related questions extracted from MedQA and MedMCQA demonstrated performance improvements for several fine-tuned models, providing evidence that the acquired ECG-specific knowledge can generalize beyond ECGQuest's generated question format. These findings support the intended purpose of ECGQuest as a resource for ECG-specific knowledge transfer, despite its current weak-label limitations. We believe that future independent clinical expert validation and correction of the generated labels could further improve the quality of the transferred knowledge and the effectiveness of ECGQuest-based fine-tuning.

Q2. How was the train/test split performed; at random, or grouped by source? ECGQuest was randomly partitioned at the question-pair level, rather than grouped by source document, with both members of each True/False pair assigned to the same partition. Before splitting, near-duplicates were removed using a Jaccard similarity threshold of 0.90 to reduce overlap across partitions. The final dataset contains 8,803 training pairs, 1,051 validation pairs, and 1,050 test pairs. This strategy was designed to evaluate ECG-specific knowledge across the range of reference materials rather than generalization to entirely unseen source documents. We have clarified the splitting procedure in the revised Methods section. Q3. On which data were fine-tuning hyperparameters and class weights selected? All model-specific hyperparameters, including LoRA rank, dropout, and class weights, were selected based on performance on the held-out validation set. The test set was reserved exclusively for final evaluation and was not used for hyperparameter selection. We have clarified this procedure in the Model Fine-Tuning subsection of the revised manuscript. Q4. Why does the abstract say three commercial models when four closed models are in Table 1? This was a typographical error in the abstract, and we have corrected the number of commercial models from three to four to ensure consistency with Table 1. Q5. Where is the significance test behind "significantly outperformed GPT-3.5"? Since no statistical significance test was conducted for this comparison, we have removed the term "significantly" to avoid implying statistical significance. We can add a significance test based on the reviewer's request, but as it cannot help the paper's objective, it has not been performed. The revised statement reads: "The recently released Jev 1.13 System One model outperformed GPT-3.5 on ECGQuest, achieving 70.0% accuracy compared with 63.1%." Q6. Please state exactly what will be released and when. We intend to publicly release the ECGQuest dataset, including the 10,904 True/False question pairs (21,808 individual statements). We also plan to provide the question-generation, validation, and label-balancing prompts (provided in supplementary material), together with the fine-tuned model adapters and the documented fine-tuning configurations. As stated in the manuscript, these resources are planned for release upon manuscript acceptance, with repository links to be included in the final version.

Thank you for the response. Several points are resolved: hyperparameters and class weights are selected on a validation set and the split is described. Commercial-model count added. MedGemma finding is acknowledged.

However, my main concerns remain. (1) Label validity: there is still no human audit. (2) Split: a random pair-level split within the same sources.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.