acceptodds
Under review as a conference paper at ICLR 2027

One Representation for Detection and Interpretation: Language-anchored Unified ECG Representation Learning via Signal, Image, and Clinical Text

11 views

Abstract

Electrocardiogram (ECG) representation is a core for computer-aided diagnosis of cardiovascular diseases (CVDs). The existing ECG-Text alignment-based methods are a promising paradigm for ECG representation learning. However, the learned representations struggle to concurrently execute both disease detection and interpretation in the diagnosis of CVDs, as proficiently as cardiologists do. Thus, we propose a LaUER, a Language-anchored Unified ECG Representation Learning framework for above tasks, like cardiologists. To approach the view of cardiologists, the ECG image is introduced to complement the ECG signal. Using clinical text, we design a multi-granularity language-anchored multimodal pre-training strategy to guide ECG representation learning in the language space, including a semantic-granularity learning, a concept-granularity learning, and a token-granularity learning. Through progressive guidance, a unified ECG representation is ultimately obtained that exhibits both detectability and interpretability. To evaluate the performance of LaUER, we conduct linear probing, zero-shot classification, and report generation evaluation on three ECG benchmark datasets, comparing it with the state-of-the-arts (SOTAs). Experimental results show that LaUER outperforms the SOTAs in both linear probing and zero-shot classification, while achieving performance close to SOTAs in report generation at a significantly lower cost compared to SOTAs.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Electrocardiogram (ECG), Representation learning, Multimodal Alignment, Zero-shot classification, Clinical report generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PSVrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LaUER is a language-anchored ECG representation learning framework intended to support both disease detection and interpretation from a single representation. The premise is that classification-oriented representations align with coarse disease concepts while generation-oriented ones emphasize fine-grained linguistic expression, even though cardiologists draw on the same underlying knowledge and differ only in expression granularity. To this end, ECG images are introduced alongside signals, since images are the format clinicians actually read. A multi-granularity pretraining strategy is built over clinical text: semantic-granularity learning anchors signal and image representations to report-level text via asymmetric cross-modal alignment; concept-granularity learning uses a standardized cardiovascular disease query bank as anchors with a query-guided multi-label readout over fused signal-image tokens; token-granularity learning trains a token-level decoder on the same fused tokens for report generation. Validation covers linear probing, zero-shot classification, and report generation.

Critical strengths
  • The premise is well-articulated and directly testable. Framing detection and interpretation as two readouts over shared knowledge at different linguistic granularities is a clean hypothesis, and it is the kind of claim whose success or failure is informative regardless of the outcome. This framing is what makes the paper worth engaging with rather than dismissing, and it is the main reason I view it as above clear rejection.
  • Introducing the ECG image modality on clinical grounds is a sensible, underexploited choice. Most ECG-text work uses signals only, yet images are what cardiologists actually read; the reported linear-probing and zero-shot gains suggest the fused representation is at least competitively discriminative.
Critical weaknesses
  • The unified-representation claim is conceded by the paper itself and then left unexamined. The abstract states that report generation is only "close to" state of the art while detection exceeds it, which means the single representation serves the discriminative readout better than the generative one — precisely the asymmetry the thesis set out to eliminate. The paper does not analyze why the generative side lags, nor whether the concept-level (discriminative) and token-level (generative) objectives interfere, for example through gradient conflict or through the shared fused-token bottleneck being pulled toward label-relevant rather than description-relevant content. Nor do the ablations isolate each granularity's marginal contribution, so it is not established that all three stages are necessary or even mutually reinforcing. Since "one representation for both" is the entire contribution, a result that supports it only halfway, without diagnosis, does not sustain the claim.
  • The framework is not positioned against the prior work it most directly extends. Text-anchored ECG pretraining using machine-generated or LLM-enhanced cardiological descriptions (e.g., ESI, Yu et al., TMLR 2024) established that clinical text can supervise signal encoders, and subsequent work aligned localized waveform segments with individual report tags (e.g., FG-CLEP, Li et al., 2025). LaUER's concept and token granularities are essentially alignment stages of this kind, so the marginal contribution of the multi-granularity curriculum over that line is unclear. This matters substantively rather than bibliographically: it has been reported that a large share of ECG reports omit detailed waveform descriptions, which bounds how much interpretive signal clinical text can supply as an anchor — a limitation directly relevant to a language-anchored design, and one that the same line of work addresses through text enhancement. MERL and MELP are cited, so the gap is conspicuous.
Questions for authors
  1. Why does report generation only approach state of the art while detection exceeds it? Do you observe interference between the concept-level discriminative objective and the token-level generative objective, e.g., degraded generation when the concept stage is trained longer?
  2. How do your concept- and token-granularity anchoring stages differ from prior fine-grained alignment between localized waveform segments and individual report tags? What does the semantic stage add beyond that?
  3. Given that many ECG reports omit detailed waveform descriptions, how does this bound the clinical text used as an anchor? Did you filter, augment, or otherwise enhance the report corpus?
  4. Can you provide a per-stage ablation showing the marginal contribution of each granularity, and confirm that the progressive ordering of the three stages is necessary rather than incidental?
  5. The manuscript is short for a three-stage pretraining framework. Are the fusion strategy, the query bank construction, and the image encoder adequately ablated in the appendix?
11 comments
AuthorsOpenReview ↗

Reply to Weakness 1 and Question 1 (Part A)

We thank the reviewer for accurately summarizing our central hypothesis and for the constructive feedback. The reviewer's description of two readouts of the same knowledge at different language granularities captures our intention. To facilitate verification, we have anonymously released the code, model weights, etc. The project code is available at https://github.com/anonymous-LaUER/LaUER_main. Model weights and configuration files are available at https://huggingface.co/anonymous-LaUER/LaUER. The ECG image data required for downstream evaluation are available at https://huggingface.co/datasets/anonymous-LaUER/LaUER-downstream-images, and the supervision annotations for the pretraining corpus are available at https://huggingface.co/datasets/anonymous-LaUER/LaUER-supervision-v4.

W1 & Q1. Examining the unified-representation claim, the generation gap, and objective interference

W1: “The unified-representation claim is conceded by the paper itself and then left unexamined. The abstract states that report generation is only ‘close to’ the state of the art while detection exceeds it … a result that supports it only halfway, without diagnosis, does not sustain the claim.”

Q1: “Why does report generation only approach state of the art while detection exceeds it? Do you observe interference between the concept-level discriminative objective and the token-level generative objective, e.g., degraded generation when the concept stage is trained longer?”

We thank the reviewer for this valuable question, which will help us clarify the main message in the revision. The reviewer accurately describes our claim. Detection and interpretation are two readouts of the same knowledge at different language granularities. Our question is whether a single representation can support both readouts. Following the progression of language understanding from coarse to fine, we use semantic-granularity learning (SGL), concept-granularity learning (CGL), and token-granularity learning (TGL) to learn a unified representation. In CGL, disease concepts serve as queries that read concept-related ECG information from the fused token sequence through cross-attention. A shared affine classifier maps the updated concept-query states to logits, each of which passes independently through a sigmoid to give the probability of the corresponding disease. In TGL, the decoder reads the same fused token sequence to generate reports and interpretations autoregressively. Classification requires concept-related information that distinguishes diseases, whereas generation requires fine-grained sequence information for clinical text. The two tasks therefore place different demands on the shared representation. We acknowledge this difference and investigate the cost at which one representation can support two readouts. This positioning appears in the abstract and introduction, but was not stated explicitly enough. W1 could consequently be read as rejecting the unified-representation claim. We will make the distinction explicit in the revision. Below, we examine one representation with two readouts through representation visualizations and the marginal changes of SGL, CGL, and TGL.

  1. Representation visualizations of the modules. As shown in the revised manuscript's Appendix D and the main text, the concept and token readouts act on the same fused representation. The concept-tracing layer and the sixth decoder layer exhibit different attention distributions for the same cases, but both read the same signal–image fusion tokens. The two readouts address the same cardiac pathological state, expressing it through class discrimination or natural language, respectively. This explains how the same representation can support both readouts. It differs fundamentally from using two independent models and underlies our description of the representation as unified.

  2. Marginal changes of the modules. We first consider the hypothesis that a severe conflict between the concept-level discriminative and token-level generative objectives would damage both classification and generation. The actual results in this response's Table R1 do not support this hypothesis. After TGL, concept zero-shot performance changes by only −0.22%, while fused-representation linear probing and cross-dataset zero-shot performance improve by 0.43% and 0.32%, respectively (this response's Table R1 for linear probing; the revised manuscript's Tables 33 and 34 in the revised manuscript's Appendix E.3 for cross-dataset zero-shot evaluation). We observe no evidence that the discriminative objective crowds out generative information or that the generative objective destroys discriminative information. We interpret this as the two forms of supervision sharing the same fused representation while acting along different readout directions. They are compatible under our training recipe.

(Please continue to read Part B.)

AuthorsOpenReview ↗

Reply to Weakness 1 and Question 1 (Part B)

Continue to reply W1/Q1.

Regarding the performance difference between report generation and disease detection, the revised manuscript adds a paired evaluation against frontier multimodal large language models on the same cases. We randomly sample 200 records from the MEETI (Zhang et al., 2025) test set. LaUER, DeepSeek-V4.1-Flash, and GPT-6.1-Sol each generate a report and an interpretation for every record in this same set of 200 records. The reports and interpretations are then independently scored by the same six cardiologists using GEM's seven five-point clinical criteria. As shown in this response's Table R2, LaUER's mean score is at least 4.0 on every criterion. Findings novelty reaches 4.7, the highest among the three models, while analytical accuracy and reasoning quality both reach 4.5, within 0.2 points of both frontier models. These results show that LaUER performs well in report generation compared with frontier multimodal large language models and offers clinical value.

These results also demonstrate the practical value of the proposed framework. Clinical ECG data are distributed across independent hospital systems. A single institution faces challenges in aggregating large-scale data to train a dedicated report-generation model from scratch and must also account for the computational costs of training and ongoing inference. LaUER has 267.6M parameters and completes three-stage pre-training on the MEETI corpus of 784,680 records using two RTX PRO 6000 GPUs, enabling the same fused representation to support both disease detection and report interpretation. Together with the stage-wise ablations in this response's Table R1, these results show that the model can acquire competitive report-generation capability while largely preserving classification performance. This provides a viable path for local deployment at individual institutions and addresses our central question of whether one representation can support two readouts.

Regarding the lack of isolated marginal contributions for each granularity, this response's Table R1 and the revised manuscript's Table 6 already report stage-wise marginal changes. LaUER is inspired by how humans understand natural language. At the language level (SGL), we first establish an overall association between things and language. At the concept level (CGL), we associate individual concepts with language and learn to distinguish them. At the token level (TGL), we learn to describe things based on our understanding of concepts. CGL therefore relies on trained semantic-granularity representations, and TGL requires trained concept-granularity representations. We compare the contributions of successive stages along LaUER's cumulative SGL, SGL+CGL, and SGL+CGL+TGL training sequence. We explain this further in Q4.

Regarding whether longer concept-stage training degrades generation, we do not train CGL to convergence in isolation and then freeze it. The CGL loss remains active during TGL, and the two objectives are jointly optimized throughout. Overfitting does require attention during training. The training-dynamics figure, provided as the revised manuscript's Figure 6 in the revised manuscript's Appendix D.2, is compiled from training logs and shows how the two modules remain balanced. It reports changes in concept zero-shot performance and generation quality across CGL and CGL+TGL training. The zero-shot values for CGL-1 through CGL-5 on the left are evaluated separately on downstream datasets using the CGL checkpoint retained after each epoch. Those for CGL/TGL-1 through CGL/TGL-5 on the right use the CGL+TGL checkpoint after each epoch under the same downstream evaluation protocol. AR_PPL on the right is measured on the validation set after each training epoch. The right axis reports AR_PPL, for which lower is better, so an upward curve indicates improved generation. During CGL, concept zero-shot AUROC rises rapidly from 24.0% to 82.26% and largely saturates. After TGL begins, it remains at 82.2%–82.3%, while validation AR_PPL falls from 2.60 to 2.35. Together, the curves show that CGL establishes the discriminative readout and TGL continues to improve generation without disrupting it.

(Response Table R1 and R2 are presented in Part C.)

AuthorsOpenReview ↗

Reply to Weakness 1 and Question 1 (Part C)

Continue to reply W1/Q1.

Response Table R1. Stage-wise ablation results for classification readouts and generative capability. AUROC (%) is averaged equally over the six downstream tasks.

StageSemantic-based ZSConcept-based ZSLinear 1% SignalLinear 1% ImageLinear 1% Fusion
SGL66.17N/A74.6877.38N/A
SGL+CGL66.1382.9474.3877.2680.49
SGL+CGL+TGL66.1182.7274.5277.2680.92

SGL/CGL/TGL results are averaged over three pre-training seeds (42, 66, and 20260915). The between-seed standard deviations of concept zero-shot AUROC are 0.90 and 1.41 for SGL+CGL and SGL+CGL+TGL, respectively. SGL has no trained fusion module or concept head, so the corresponding entries are N/A.

Response Table R2. Cardiologist ratings on a five-point scale (mean ± standard deviation) for LaUER and two frontier multimodal large language models (the revised manuscript's Table 5)

ModelParamsAnalytical RelevanceAnalytical AccuracyAnalytical CompletenessReasoning QualityFindings NoveltyClinical ValueOverall Satisfaction
DeepSeek-V4.1-FlashUndisclosed4.7±0.464.6±0.574.8±0.404.7±0.494.5±0.664.6±0.604.7±0.51
GPT-6.1-SolUndisclosed4.7±0.454.6±0.604.7±0.514.7±0.524.4±0.704.6±0.574.6±0.60
LaUER (ours)267.6M4.3±0.524.5±0.484.2±0.604.5±0.674.7±1.034.0±0.854.3±0.64

Note. The evaluation uses 200 MEETI test records (selection seed 42), independently scored by six cardiologists. All three models use the same cases, raters, and scoring criteria, namely GEM's seven five-point criteria (Lan et al., 2025). The prompt templates are provided in the revised manuscript's Appendix C.5. The parameter counts of the frontier models are undisclosed; LaUER has 267.6M parameters. Bold indicates the highest value among the three models.

Please see the revised manuscript's Figure 6 in the revised manuscript's Appendix D.2 for the changes in concept zero-shot AUROC and validation AR_PPL during CGL→TGL training. The left axis shows concept zero-shot AUROC (%), the right axis shows validation AR_PPL (lower is better), and the gray dashed line marks the boundary between CGL and TGL.

Reviewer_Qsw3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LaUER, a framework that learns a unified ECG representation from raw signals, ECG images, and clinical text. It introduces a three-stage pretraining strategy involving semantic alignment, concept-guided disease prediction, and token-level report generation. The learned representation is evaluated through linear probing and zero-shot classification on PTB-XL, CPSC2018, and CSN, along with cardiologist evaluation of generated reports on MEETI. The results show promising classification performance and competitive report quality.

Critical strengths
  1. Learning a shared representation for both disease detection and report generation is a meaningful objective, potentially reducing the need for separate task-specific ECG models.

  2. The evaluation covers multiple datasets and label budgets, with encouraging zero-shot classification results. The inclusion of cardiologist assessments also strengthens the practical relevance of the report generation task.

Critical weaknesses
  1. The contribution of the proposed three-granularity learning strategy is not sufficiently established. The concept-level supervision already provides strong disease-specific information, while adding token-level learning slightly reduces zero-shot AUROC (82.94 to 82.72). It remains unclear whether the full training strategy offers meaningful advantages over simpler combinations of existing objectives.

  2. The experimental comparisons are not fully convincing. Most baseline results are taken from previous work, and the cross-dataset evaluation uses different retained samples after label mapping. More importantly, the authors acknowledge that historical test results influenced prompt selection, raising concerns about the reliability of the reported zero-shot improvements.

Questions for authors
  1. LaUER essentially combines three established ECG-language learning paradigms. What evidence demonstrates that their integration provides genuine benefits beyond simply combining existing objectives, especially when adding TGL slightly reduces zero-shot performance?

  2. How much of the reported improvement comes from the proposed learning framework rather than the curated concept bank and prompt engineering? Table 7 changes multiple factors simultaneously, making their contributions difficult to distinguish.

  3. How can the claimed zero-shot improvements be considered reliable when the authors acknowledge that test-set results influenced prompt selection? Does this compromise the validity of the reported comparisons?

7 comments
AuthorsOpenReview ↗

Reply to Weakness 1 and Question 1 (Part A)

We thank the reviewer for recognizing the motivation for a unified representation, the evaluation across multiple datasets, and the cardiologist assessment. To facilitate verification, we have anonymously released the code, model weights and configurations, the pre-training corpus and its supervision annotations, and the ECG images used for downstream evaluation.

The project code and complete scripts for three-stage pretraining, linear probing, and zero-shot evaluation are available at https://github.com/anonymous-LaUER/LaUER_main. Model weights and configuration files are available at https://huggingface.co/anonymous-LaUER/LaUER. The ECG image data required for downstream evaluation are available at https://huggingface.co/datasets/anonymous-LaUER/LaUER-downstream-images, and the supervision annotations for the pretraining corpus are available at https://huggingface.co/datasets/anonymous-LaUER/LaUER-supervision-v4.

W1 & Q1. Does integrating the three granularities provide benefits beyond combining existing objectives?

W1: “The contribution of the proposed three-granularity learning strategy is not sufficiently established. The concept-level supervision already provides strong disease-specific information, while adding token-level learning slightly reduces zero-shot AUROC (82.94 to 82.72). It remains unclear whether the full training strategy offers meaningful advantages over simpler combinations of existing objectives.”

Q1: “What evidence demonstrates that their integration provides genuine benefits beyond simply combining existing objectives, especially when adding TGL slightly reduces zero-shot performance?”

W1 and Q1 raise the same question of whether integrating the three granularities provides benefits beyond simply combining existing objectives. We thank the reviewer for raising this question and apologize for not explaining this point in the previous manuscript. We clarify and discuss it below. First, we agree that concept-level supervision already provides strong disease-discriminative information, and that introducing token-granularity learning (TGL) does not improve zero-shot classification. Our claim, however, is one representation with two readouts. Classification relies on class-discriminative directions, whereas generation requires fine-grained sequential information. A unified representation is unlikely to achieve the optimum of both single-task models simultaneously. Learning a unified representation that supports both classification and report generation is therefore our central concern. TGL is intended to add the previously unavailable report-generation capability rather than improve classification. It achieves report-generation performance competitive with frontier multimodal large language models at a very small zero-shot performance cost (−0.22%). Moreover, its zero-shot performance of 82.72% remains above the existing state of the art, while linear evaluation and cross-domain zero-shot performance even exceed those before TGL training (this response's Table R1 for linear evaluation; the revised manuscript's Tables 33 and 34 in the revised manuscript's Appendix E.3 for cross-domain zero-shot evaluation). TGL thus adds generative capability without substantially degrading the classification readout.

(For the integration analysis of the LaUER framework, see Part B.)

AuthorsOpenReview ↗

Reply to Weakness 1 and Question 1 (Part B)

Continue to reply W1/Q1.

We next discuss whether the integration provides benefits beyond simply combining existing objectives. Semantic-granularity learning (SGL), concept-granularity learning (CGL), and TGL each have an indispensable role, both in their design and in the experimental results.

  1. Each granularity serves an indispensable readout function. The semantic, concept, and token granularities are inspired by how humans understand natural language. At the language level, we first establish an overall association between things and language. At the concept level, we associate individual concepts with language and learn to distinguish them. At the token level, we learn to describe things based on our understanding of concepts. Experimentally, this response's Table R1 shows that SGL provides semantic alignment and achieves 66.17% semantic zero-shot AUROC. After CGL introduces the standardized concept bank and query-guided readout, concept zero-shot AUROC reaches 82.94%, a gain of 16.77% over the semantic readout. This 16.77% gain comes from concept supervision together with the fused representation, rather than simply replacing the classification head. The concept readout requires disease evidence in the fused representation to be located individually by a fixed set of disease queries, whereas the semantic readout only requires the overall embedding to resemble the report text. The two readouts therefore organize the representation differently. As discussed above, TGL enables report-generation performance competitive with frontier multimodal large language models without degrading zero-shot classification capability. Each granularity thus serves an indispensable role.

  2. No simpler combination achieves the same outcome. SGL+CGL provides strong classification but has no generation path. SGL+TGL lacks a concept-level readout and cannot perform concept zero-shot classification. Only SGL+CGL+TGL supports both readouts. The value of integration lies in establishing a previously unavailable joint operating point, rather than adding the metrics of individual stages. The same fused representation can be read by both concept queries and the decoder. Furthermore, LaUER does not merely add three losses in parallel. CGL concept supervision and TGL token supervision act on the same fused representation. The former organizes the representation so that disease evidence can be located by queries, whereas the latter requires it to retain the fine-grained information needed to generate reports and explanations. Because the representation is shared, training in the later stage changes the input distribution of both readouts. Experimentally, the third row of this response's Table R1 shows that the fused representation learned with SGL+CGL+TGL achieves the highest linear evaluation performance, 80.92%. This result indicates that CGL and TGL do not exhibit a trade-off in the shared representation and provides evidence for this joint operating point.

Response Table R1. Stage-wise ablation results for classification readouts and generative capability. AUROC (%) is averaged equally over the six downstream tasks.

StageSemantic-based ZSConcept-based ZSLinear 1% SignalLinear 1% ImageLinear 1% Fusion
SGL66.17N/A74.6877.38N/A
SGL+CGL66.1382.9474.3877.2680.49
SGL+CGL+TGL66.1182.7274.5277.2680.92

SGL/CGL/TGL results are averaged over three pre-training seeds (42, 66, and 20260915). The between-seed standard deviations of concept zero-shot AUROC are 0.90 and 1.41 for SGL+CGL and SGL+CGL+TGL, respectively. SGL has no trained fusion module or query-guided tracing head, so the corresponding entries are N/A.

AuthorsOpenReview ↗

Reply to Weakness 2 and Question 2 (Part A)

W2 & Q2. Do the gains come from the framework, the concept bank, or prompt engineering?

Reviewer comments on the original submission

W2: “The experimental comparisons are not fully convincing. Most baseline results are taken from previous work, and the cross-dataset evaluation uses different retained samples after label mapping. …”

Q2: “How much of the reported improvement comes from the proposed learning framework rather than the curated concept bank and prompt engineering? Table 7 changes multiple factors simultaneously, making their contributions difficult to distinguish.”

We thank the reviewer for asking about the sources of the performance gains. The previous manuscript did not clearly describe the relevant experimental settings. We have corrected and clarified them.

First, although baseline results are taken from existing papers, their training and testing protocols are consistent with ours and are comparable. MERL (Liu et al., 2024) contributed benchmark results in this research direction. MELP (Wang et al., 2025) subsequently cited MERL's experimental results as reported values and evaluated on MELP's public splits and prompt bank. For pre-training, our MEETI (Zhang et al., 2025) corpus is derived from MIMIC-ECG, adding corresponding images to the existing MIMIC-ECG signals without changing the sample count. It therefore uses the same pre-training ECG samples as the baselines. For testing, we follow MELP's split files, including the official PTB-XL folds 1–8/9/10, the fixed CPSC2018/CSN splits, and the zero-shot protocol. Our evaluation procedure is therefore consistent with the baselines. The revised manuscript's Appendix C.1 identifies baseline values as results taken from the cited publications, and the revised manuscript's Table 16 summarizes the evaluation settings.

For cross-domain evaluation, we follow the label matching of MERL and MELP. The revised manuscript's Appendix C.3 states this protocol alignment, and the revised manuscript's Table 18 provides the category crosswalk.

Returning to whether the gains come from the framework or the concept bank and prompt engineering, the concept and prompt banks are part of our third contribution, the clinically curated multimodal corpus. Together with the proposed representation-learning framework, they constitute LaUER. We conducted the following experiments to examine the sources of the gains.

  1. Framework evaluation. As shown in this response's Table R1, adding CGL to SGL improves concept zero-shot classification by 16.77% over the semantic readout. With the same CKEPE prompt bank used by MELP, LaUER achieves native and cross-domain zero-shot classification AUROCs of 79.34% and 80.33%, exceeding MELP's 79.0% and 79.58%. These experiments show that the framework substantially improves zero-shot classification. Even without the proposed corpus, LaUER still outperforms the state of the art.

  2. Concept-bank evaluation. As shown in this response's Table R2, using our curated disease concept bank improves concept zero-shot AUROC from 76.45% to 83.20% (+6.75%) and fused-representation linear probing from 77.61% to 80.64% (+3.03%). Signal-only and image-only linear probing also improve by 7.34% and 4.20%, respectively. These results indicate that using our curated concept bank as CGL supervision is more effective for disease classification.

  3. Prompt-engineering evaluation. In this response's Table R3, only the prompt bank is changed, with the checkpoint, evaluation samples, and readout protocol held fixed. With CKEPE, native-label and cross-domain zero-shot AUROCs are 79.34% and 80.33%. Our standardized prompt bank improves these to 81.87% and 81.59%, yielding gains of 2.53% and 1.26%, respectively. This single-variable comparison shows that prompt-bank standardization itself improves zero-shot classification.

Overall, the framework determines how the representation is read out, the concept bank improves supervised training, and the prompt bank better matches LaUER's pre-training objective. Both the framework and the concept bank/prompt engineering improve zero-shot classification.

(The relevant experimental results and analysis are presented in Part B.)

Reviewer_qYocOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LaUER, a unified ECG representation learning framework for disease detection and report generation. It combines ECG signals, ECG images, and clinical text through semantic-, concept-, and token-level supervision. Experiments on three ECG datasets show improvements in classification and competitive report generation performance.

Critical strengths
  • The paper addresses a practically relevant problem by combining ECG disease detection and clinical interpretation within a unified framework.

  • The evaluation covers multiple ECG datasets and downstream tasks, including linear probing, zero-shot classification, and report generation.

  • The experimental results are generally competitive, particularly for zero-shot disease classification.

Critical weaknesses
  • The proposed framework largely combines established techniques, including contrastive alignment, cross-modal fusion, query-guided classification, and autoregressive decoding. The three-stage training strategy appears incremental, with limited evidence of substantial methodological innovation.

  • The evaluation focuses primarily on specialized ECG models and older architectures. Given recent advances in multimodal foundation models, it would be important to evaluate whether strong frontier models, such as GPT-6 Astra, can perform ECG image interpretation and report generation without task-specific training. Without such comparisons, the practical advantage of the proposed framework is difficult to assess.

  • Adding token-level training does not improve zero-shot classification, while the report generation comparison uses different evaluation cases. The claimed benefits of jointly learning detection and interpretation are therefore not convincingly established.

  • Several baseline results are taken from prior publications rather than matched reruns. More importantly, the paper acknowledges that historical test results informed prompt selection, raising concerns about the reliability of the reported zero-shot improvements.

Questions for authors

None.

7 comments
AuthorsOpenReview ↗

Reply to Weakness 1 (Part A)

We thank the reviewer for recognizing the practical value, experimental coverage, and zero-shot classification results of our work. The project code is available at https://github.com/anonymous-LaUER/LaUER_main. Model weights and configuration files are available at https://huggingface.co/anonymous-LaUER/LaUER. The ECG image data required for downstream evaluation are available at https://huggingface.co/datasets/anonymous-LaUER/LaUER-downstream-images, and the supervision annotations for the pretraining corpus are available at https://huggingface.co/datasets/anonymous-LaUER/LaUER-supervision-v4.

W1. The method mainly combines existing techniques, and the three-stage training strategy appears incremental

“The proposed framework largely combines established techniques, including contrastive alignment, cross-modal fusion, query-guided classification, and autoregressive decoding. The three-stage training strategy appears incremental, with limited evidence of substantial methodological innovation.”

Thank you for this question. We recognize that the previous manuscript did not clearly highlight its contributions, and we clarify them below. Contrastive alignment, cross-modal fusion, query-guided multi-label classification, and autoregressive decoding all have precedents in multimodal learning. These include contrastive alignment in CLIP (Radford et al., 2021), “align before fuse” in ALBEF (Li et al., 2021), joint understanding and generation in BLIP (Li et al., 2022), and the use of a shared visual encoder for contrastive learning and caption generation in CoCa (Yu et al., 2022). Contrastive alignment and cross-modal fusion have become standard paradigms for multimodal representation learning. The central questions of our method are at what granularity ECG and text should be aligned, and how the aligned ECG representation can support both disease detection and interpretation of cardiac state. This question has not previously been examined systematically. Following this approach, we use language anchors at semantic, concept, and token granularities and introduce query-guided classification and autoregressive decoding on a shared fused representation, progressively training the same representation to support both classification and generation readouts.

The contributions of this work therefore concern both the data corpus and the representation learning framework.

  1. For the text input, we introduce “report + long interpretation” for the first time in place of the report-level text used in existing ECG–text pretraining. The medical text in existing ECG–text studies mostly comes from clinical reports or short-text augmentation of those reports, and omits detailed waveform descriptions. To our knowledge, waveform descriptions contain rich semantic information. We use the medical text in MEETI (Zhang et al., 2025) derived from cardiologist-written reports and beat-level quantitative parameters, which contains extensive descriptions of waveform morphology, heart rate, intervals and amplitudes, and reasoning based on medical knowledge. To evaluate “report + long interpretation,” we add the comparison in this response's Table R1. When the semantic stage uses only report-level text, semantic-based zero-shot classification reaches 68.23%, but concept-based zero-shot classification reaches only 69.75%. This indicates that concept-granularity learning provides almost no additional benefit, and the model effectively reverts to semantic-granularity learning. With “report + long interpretation” at the semantic stage, semantic-based zero-shot classification reaches 66.17%, while concept-based zero-shot classification reaches 82.94%. The +13.19% gain in concept-based zero-shot classification indicates that the waveform and reasoning information in long interpretations allows the model to learn richer medical semantics during semantic-granularity learning. This is a substantive difference from existing work that uses only short reports or label phrases.

  2. For the ECG input, we use paired ECG “signal + image” views in place of the common single-view ECG paradigm, in which existing ECG–text alignment studies mostly use signals. ECG signals directly measure cardiac electrophysiological activity, while ECG images are the format that cardiologists actually read. The two provide complementary information and are jointly anchored by clinical text. This response's Table R2 below compares the three ECG readouts. The fused readout outperforms either unimodal readout in 17 of the 18 linear evaluation settings. With a 1% label budget, fusion reaches 64.58% AUROC on Form, compared with 59.69% for signal and 58.34% for image. This shows that the two views contain complementary disease information and that fusion is needed, rather than being introduced simply to increase the number of modalities.

(The contributions of the representation learning framework are detailed in Part B.)

AuthorsOpenReview ↗

Reply to Weakness 1 (Part B)

Continue to reply W1.

The contributions of representation learning are as follows.

  1. In semantic-granularity learning (SGL), we propose a feasible three-modal alignment strategy. Medical text consisting of “report + long interpretation” serves as the semantic anchor, and asymmetric cross-modal contrastive alignment maps ECG signals, ECG images, and clinical text into a shared semantic space to learn richer medical semantics than report-level text provides. Because the modality pairs differ in alignment difficulty, the manuscript assigns corresponding weights to the three alignment terms. Signals and images are different observations of the same ECG and have high cross-modal similarity, making their contrastive loss easier to fit. Language differs inherently from the other modalities and is harder to align. We therefore increase the weights of language-related alignment terms to emphasize the leading role of language in multimodal contrastive learning. The response's Table R1 shows that reverting to report-level text reduces concept-based zero-shot classification AUROC from 82.94% to 69.75%.

  2. In concept-granularity learning (CGL), the signal and image are first fused. Disease concepts calibrated against SNOMED-CT and reviewed on a sampled basis by clinical specialists form a standardized concept query bank. Query-guided cross-attention then performs multi-label readout from the fused signal–image representation. The comparison with SuPreME (Cai et al., 2025) concerns more than the standardized concept query bank. Our CGL query readout operates on the fused representation, whereas SuPreME performs query readout on a unimodal representation. Most importantly, this stage provides a way to fuse representations and use the resulting fused representation for other tasks. In the ablation, adding CGL raises zero-shot classification from 66.17% (semantic-based) to 82.94% (concept-based). This shows that guidance from the concept query bank gives the fused representation stronger discriminative ability.

  3. In token-granularity learning (TGL), a lightweight autoregressive decoder constrains the fused signal–image representation and gives it generation ability, rather than introducing a separate model dedicated to generation. This ordering also has a natural interpretation in human language learning. We first establish overall associations between objects and language, then associate concepts with language individually, and finally generate descriptions of those objects. The three-granularity design implements this coarse-to-fine process computationally. After TGL, LaUER shows good report-generation performance compared with frontier multimodal large language models in the paired evaluation on the same 200 cases (the response's Table R3), while largely preserving classification performance (the response's Table R4).

Response Table R1. Effect of the language view on zero-shot readouts (equally weighted mean AUROC over 6 downstream tasks, %)

SGL medical-text structureSemantic-based ZSConcept-based ZS
Reports containing only disease-description phrases68.2369.75
Phrase reports + long interpretations (ours)66.1782.94

Note. The second data row uses the text structure adopted during pretraining in our framework, report [SEP] interpretation. Both rows use the same concept query bank and readout protocol.

Response Table R2. Linear evaluation AUROC (%) for the three modalities. Values are taken from the Ours rows of the revised manuscript's Table 2.

MethodRhythm (1%)Rhythm (10%)Rhythm (100%)Sub (1%)Sub (10%)Sub (100%)Form (1%)Form (10%)Form (100%)Super (1%)Super (10%)Super (100%)CPSC2018 (1%)CPSC2018 (10%)CPSC2018 (100%)CSN (1%)CSN (10%)CSN (100%)
LaUER-Signal(Ours)77.0388.7792.0275.1387.0891.8759.6975.1083.9284.0988.1990.0877.9691.3994.4473.1988.9094.72
LaUER-Image(Ours)79.6091.3393.4180.4689.6792.9358.3474.7686.1186.5790.7491.8183.0793.8295.8975.5288.7994.67
LaUER-Fusion(Ours)86.3892.7293.7383.3990.3093.2364.5880.6386.5387.3090.5692.0685.5493.9295.9778.3690.8595.34

Overall, each component of LaUER is closely tied to the research objective. Both the extensions to ECG and text in the data corpus and the semantic-, concept-, and token-granularity learning in the framework serve the same goal of “one representation, two readouts.” Thus, the contributions are not an assembly of existing techniques, but a unified representation learning approach designed around the joint objective of disease detection and interpretation of cardiac state.

AuthorsOpenReview ↗

Reply to Weakness 2

W2. Missing comparisons with frontier multimodal foundation models

“The evaluation focuses primarily on specialized ECG models and older architectures. Given recent advances in multimodal foundation models, it would be important to evaluate whether strong frontier models, such as GPT-6 Astra, can perform ECG image interpretation and report generation without task-specific training. Without such comparisons, the practical advantage of the proposed framework is difficult to assess.”

We thank the reviewer for raising this issue and agree that comparisons with frontier models are important for assessing practical value. The revised manuscript adds a paired comparison on the same cases.

The evaluation reported in the revised manuscript's Table 5 (this response's Table R3) uses 200 records randomly sampled from the MEETI test set (selection seed 42). LaUER and two frontier multimodal large language models, DeepSeek-V4.1-Flash and GPT-6.1-Sol, each generate a short report and a detailed interpretation for every record. Six cardiologists independently score all three models on the same cases, using GEM's seven five-point criteria. Because the frontier models cannot directly read waveforms, they receive the same 12-lead ECG image as LaUER and additionally receive measurements derived from MEETI's beat-level quantitative parameters. These include per-lead heart rate, PR interval, QRS duration and amplitude, P-wave amplitude and duration, T-wave amplitude, QT/QTc, beat-level feature sequences, RR statistics, frontal-plane electrical axis, ST levels, and T-wave polarity. The two frontier models use exactly the same prompt and generate once per case. LaUER uses the complete three-stage pretrained model (training seed 42), without fine-tuning for the rating task, and generates directly from the fused ECG representation without a text prompt. The protocol and full prompts are provided in the revised manuscript's Appendix C.5.

This response's Table R3 shows that LaUER receives high ratings for both reliability and usefulness, with every criterion at or above 4.0. Findings novelty reaches 4.7, the highest of the three models, while analytical accuracy and reasoning quality both reach 4.5, within 0.2 points of the two frontier models. This result matters in real clinical settings. ECG data are distributed across independent hospital systems, whereas frontier large language models rely on Internet-scale corpora. A single institution can neither collect data on the same scale to train a dedicated report generator from scratch nor afford the cost of training and repeatedly running a large model. LaUER has only 267.6M parameters, yet provides both disease detection and report interpretation from the same fused representation. More importantly, this generation ability is obtained without sacrificing classification performance, as shown by the stage-wise ablation in this response's Table R4. LaUER therefore provides a low-cost approach that an institution can deploy locally.

Response Table R3. Cardiologist ratings on a five-point scale (mean ± standard deviation) for LaUER and two frontier multimodal large language models (the revised manuscript's Table 5)

ModelParamsAnalytical RelevanceAnalytical AccuracyAnalytical CompletenessReasoning QualityFindings NoveltyClinical ValueOverall Satisfaction
DeepSeek-V4.1-FlashUndisclosed4.7±0.464.6±0.574.8±0.404.7±0.494.5±0.664.6±0.604.7±0.51
GPT-6.1-SolUndisclosed4.7±0.454.6±0.604.7±0.514.7±0.524.4±0.704.6±0.574.6±0.60
LaUER (ours)267.6M4.3±0.524.5±0.484.2±0.604.5±0.674.7±1.034.0±0.854.3±0.64

Note. The evaluation uses 200 MEETI test records (selection seed 42), independently scored by six cardiologists. All three models use the same cases, raters, and scoring criteria, namely GEM's seven five-point criteria (Lan et al., 2025). The prompt templates are provided in the revised manuscript's Appendix C.5. The parameter counts of the frontier models are undisclosed; LaUER has 267.6M parameters. Bold indicates the highest value among the three models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.