acceptodds
Under review as a conference paper at ICLR 2027

Candidate Label Reconstruction with Cardinality Preservation for Multi-label ECG

Abstract

ECG recordings often contain multiple co-occurring diagnoses, while clinical annotations may also be ambiguous or incomplete. Partial Label Learning (PLL) provides a natural framework for this setting by learning from candidate label sets that contain the underlying true labels. However, existing PLL disambiguation strategies are not well-suited to multi-label ECG diagnosis for two reasons: uniform disambiguation can unnecessarily alter reliable candidate sets, while confidence-based approaches may suppress valid but less confident co-occurring diagnoses. We propose a novel disambiguation method, ReCal, which learns sample-specific reliability to determine how strongly each candidate set should be disambiguated and estimates label cardinality to preserve multiple valid diagnoses. Using several controlled ambiguity settings on PTB-XL, Chapman, and AMIGOS datasets, ReCal consistently improves both label disambiguation and downstream classification over existing methods.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Partial Label Learning, Partial Multi-Label Learning, Multi-Label Classification, Electrocardiography, Ambiguous Labels, Label Disambiguation, Label Cardinality

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_sYi3OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies partial multi-label learning for ECG diagnosis. Its main observation is that confidence-based disambiguation can hurt samples whose candidate labels are already correct and can also suppress valid co-occurring diagnoses. ReCal addresses these two issues with a learned sample weight that controls how much a candidate set is refined and an auxiliary cardinality predictor that estimates how many labels should be kept. The method is tested on PTB-XL, Chapman-Shaoxing, and AMIGOS under several controlled ambiguity settings. The paper reports classification results, recovered-cardinality analyses, reliable/ambiguous-sample deviation, and ablations of the two proposed components.

Critical strengths
  1. The paper identifies a meaningful failure mode of confidence-based disambiguation rather than only reporting a performance gap. The analysis explains why sharpening can move a correct multi-label target away from the ground truth and why high-cardinality samples are especially vulnerable. The two proposed components follow naturally from this analysis.
  2. The evaluation is reasonably broad within the stated PLL setting. Several representative baselines are compared using a shared ECG backbone, and the analysis goes beyond final Macro-F1. The recovered-cardinality and reliable-sample results directly test the failure modes targeted by the method.
Critical weaknesses
  1. The central clinical premise is broader than the actual problem the method solves. ReCal assumes Yi∗⊆SiY_i^* \subseteq S_i, so every true diagnosis is already present in the candidate set. The method can remove extra candidates, but it cannot recover a diagnosis that is missing from the annotation. Yet incomplete clinical annotation is used as a major motivation. The experiments also create ambiguity by adding candidates to datasets with known labels. This is not a minor edge case. It means the paper does not address one of the main forms of incompleteness suggested by its clinical framing.
  2. The learned sample weight is not an identifiable measure of annotation reliability. It is driven by how hard the current supervision is to fit. A rare, noisy, atypical, or simply difficult but correctly labeled ECG can therefore receive the same signal as a genuinely ambiguous label set. The cardinality module then learns targets from the model's own pseudo-labels on samples selected as reliable by this proxy. This makes the supervision circular. The paper shows that the components improve F1, but it does not show that the weight really tracks label reliability or that the cardinality loop corrects errors rather than reinforcing them.
Questions for authors
  1. Can the authors evaluate a setting in which one or more true labels are missing from the observed candidate set? If not, the clinical claim should be narrowed to the superset-candidate setting assumed by Yi∗⊆SiY_i^* \subseteq S_i.
  2. Can the authors show whether the learned weights distinguish label ambiguity from ordinary difficulty? For example, keep the labels correct and vary rarity, signal quality, or classification difficulty.
  3. Please report how accurately the learned reliable subset matches the known reliable/ambiguous split during training. An oracle-reliable subset or oracle-cardinality comparison would also help determine whether the cardinality loop is correcting or propagating early pseudo-label errors.
  4. The paper states that classification performance is evaluated using both Macro-F1 and AUC-ROC, but Table 1 only reports full Macro-F1 numbers while AUC-ROC results are mostly presented as class-wise heatmaps in Figure 5. Could the authors provide aggregate AUC-ROC values for all methods across all datasets and ambiguity settings in the main results table?
5 comments
AuthorsOpenReview ↗

part one of response

We thank the reviewer for the careful evaluation. We are glad that the analysis of why sharpening pulls a correct multi-label target away from the ground truth, and why high-cardinality samples suffer most, was found meaningful, and that the two components were seen as following from it. R3.W1 and R3.W2 are addressed in R3.Q1 and R3.Q2–R3.Q4, respectively.

R3.Q1: why not noisy partial label learning

We have not evaluated a setting in which true labels are missing from the candidate set, and we agree that the clinical claim should be narrowed accordingly. ReCal addresses partial label learning under the superset assumption. Every true diagnosis is in the candidate set, and the set may also contain false positives. This is the setting of our analysis, our controlled ambiguity generation and our evaluation on CODE-test. We chose it because it matches a common annotation protocol, in which several readers independently propose diagnoses and a consensus review or a senior adjudication then fixes the reference (Ribeiro et al., 2020; Rigueira et al., 2024). The candidate set can be defined as the union of the annotator's diagnosis. CODE-test follows this protocol, with six readers per recording and a cardiology expert as the gold standard, and the recent larger CODE-II-test (Abreu et al., 2026) is reviewed by multiple cardiologists, which will allow studies at a larger scale in future. Under this protocol the assumption fails only when all reader miss the diagnosis that the reference includes. The case where true labels may be missing is noisy partial label learning, which is a different problem. A disambiguation operator restricted to the candidate set cannot recover a missing diagnosis, so we do not claim that ReCal addresses it. We also have not tested how missing labels would affect the reliability weights or the cardinality estimator. We have made two changes to the paper. The abstract and introduction no longer describe the annotations as ambiguous candidate-label supervision under the superset assumption. We now state the assumption explicitly in the problem setup and have added a limitations paragraph that names missing diagnoses as an open direction. We consider noisy partial label learning an important extension for clinical annotation and leave it to future work.

R3.Q2: Difficulty vs reliability in weight

We agree that wiw_i is closely tied to the training loss. Minimizing the per-sample term of Eq. (6) over wiw_i gives wi⋆=log⁡(ℓi/λ)w_i^\star=\log(\ell_i/\lambda), so the weight reflects how well the current supervision is fit. The superset setting makes the loss informative for a specific reason. Every spurious candidate is a false positive that can mainly be fit by memorizing the sample, whereas the labels of a correct sample, even a difficult one, agree with patterns shared across its class and can be fit through generalization (Wu et al., 2023). Confusing a difficult but correctly labeled sample with an ambiguous one by receiving a high weight, has two consequences in our framework. First, the sigmoid-based gate partially disambiguates the sample. Since the update is smooth, the supervision moves toward the model confidence only in proportion to the gate value, and the resulting deviation is limited to the suppression term A(ρ)A(\rho) of the deviation decomposition (Eq. 5). Empirically, the deviation of reliable samples (RD) is 0.004 to 0.062 for ReCal, compared with 0.264 to 0.320 for Base-PLL (Table R3.2). Second, the sample may be excluded from the reliable set and be underrepresented in the training of the cardinality estimator. The recall of reliable samples for this set is at least 0.998 (Table R3.1). Both measurements are taken on data where correctly labeled samples are naturally difficult, since PTB-XL and Chapman are long-tailed and multi-label, include classes with roughly 100 samples, and show substantial differences in class-wise performance (Figs. 3 and 5). We further quantify the cost of this imperfect separation with oracle variants. Oracle-R replaces the learned weights with the ground-truth partition into reliable and ambiguous samples, for both the gate and the cardinality estimator. Table R3.2 shows that ReCal follows this oracle closely. The Macro-F1 of ReCal is within 0.004 to 0.028 of Oracle-R. The gap to the oracle is smaller than the gain of ReCal over Base-PLL (0.040 to 0.102) in every setting, so errors in reliability identification leave some headroom but do not remove the benefit of the method. These measurements are aggregated over samples and do not vary rarity, signal quality, or classification difficulty independently.

AuthorsOpenReview ↗

part two of response

R3.Q3: Quantize reliable sample recovery

The weight wiw_i is optimized on the loss of the current supervision (Eq. 6) and reflects how well the current supervision is fit, and it does not record whether the original candidate set was ambiguous. The weight is used in two places. First, it enters Eq. 8 as a continuous gate w~i\tilde{w}_i, scaling the update applied to the current supervision. Second, samples whose normalized weight is below 0.5 form the reliable set used to train the cardinality estimator. In both roles, the relevant property is the quality of the supervision a sample carries. Table R3.1 reports, on the training set at ambiguity proportion p=0.5p=0.5, the quality of the recovered reliable sample with the thresholded weight. Nearly all reliable samples are placed on the reliable side (recall ≥\ge 0.998), while part of the ambiguous samples are identified as unreliable. An ambiguous sample with supervision closer to the model’s prediction can become easy to fit, receives a low weight, and is grouped with the reliable samples. Whether this is harmful depends on whether the supervision it carries is correct, which accuracy cannot reveal.

We report the cardinality error and the deviation of the final supervision for the reliable and unreliable groups, and additionally for the ambiguous samples within the reliable group. Because the reliable group contains nearly all clean samples, the informative comparison is between ambiguous samples on both sides of the threshold. Ambiguous samples in the reliable group have a 1.7 to 2.8 times lower cardinality error and a 1.3 to 1.9 times lower deviation than those in the unreliable group, for example 0.462 and 0.593 against 1.301 and 1.147 on PTB-XL random. To measure the cost of the imperfect identification, we replace learned components with oracle ones (Table R3.2). Oracle-MLP trains the estimator on truly clean samples only, and Oracle-R additionally uses the ground-truth reliability for the gate. Replacing the learned gate with the ground-truth one (Oracle-R relative to Oracle-MLP) improves Macro-F1 by 0.003 to 0.020 on PTB-XL and Chapman, which is smaller than the 0.040 to 0.102 gain of ReCal over Base-PLL. This shows that the identification is imperfect and where the remaining headroom is. We will include Tables R3.1 and R3.2 in the appendix.

R3.Q4: Cardinality Estimation Accuracy

Table R3.2 contains these comparisons on PTB-XL and Chapman. We define each variant by the component it replaces or perturbs. 1) Oracle-C (oracle cardinality): The estimated cardinality in the disambiguation is replaced by the ground-truth cardinality. This removes all errors of the estimator, including those inherited from early pseudo-labels. 2) Oracle-MLP (oracle reliable subset for the estimator): The estimator is trained only on truly clean samples, so its training targets are free of pseudo-label error. The disambiguation remains similar to ReCal. 3) Oracle-R: Ground-truth reliability is used for the gate and for the estimator training. Oracle-R minus Oracle-MLP therefore isolates the effect of the gate. 4) Noisy-MLP (noisy estimator targets): The estimator is trained as in ReCal, but the cardinality target is increased by two for a random 50% of the training samples. This mimics the inflated cardinality that ambiguous candidate sets would contribute and tests the opposite direction of the oracles. If the loop propagated early errors, replacing with correct cardinalities or correct training targets should recover a substantial part of the gain over Base-PLL. While incorrect cardinality does cost some label quality, Oracle-C changes Macro-F1 by up to +0.003 and Oracle-MLP by less than +0.011 relative to ReCal. Noisy-MLP lowers Macro-F1 by 0.008 to 0.019 and increases AD by 0.131 to 0.239, and RD is mostly unchanged. While this noise is strong, the estimator is not insensitive to its targets, and contaminated targets mainly affect the recovery of label sets. However, Noisy-MLP improves performance over Base-PLL in all four settings. We therefore conclude that the cardinality loop does not propagate early errors to a degree that limits performance. Replacing the learned gate with the ground-truth one (Oracle-R relative to Oracle-MLP) improves Macro-F1 by up to 0.020, which is the main remaining headroom.

R3.Q5: Add AUCROC results to table

We agree that aggregate AUC-ROC should be available next to Macro-F1. Table 7 in the appendix now reports the AUC-ROC for all methods, datasets and ambiguity settings, in the same layout as Table 1. Figure 5 (class specific performance) remains in the paper to show where in the label space the methods differ, but it is not meant to replace the aggregate numbers. We kept the main table limited to Macro-F1 because a second metric for every method would roughly double its width within the page limit. We have added a sentence in Sec. 4 that points to the appendix table and summarizes the comparison.

AuthorsOpenReview ↗

part three of response

Table R3.1: Separation of cardinality error and deviation of the final supervision, grouped by the thresholded weight. Cardinality MAE compares the number of labels retained by the final supervision with the true cardinality. Mean ± std over three seeds.

DatasetScenarioCardinality MAE ↓Deviation ↓
reliablereliable$_{Amb}$unreliablereliablereliable_{Amb}\unreliable
PTB-XLRandom0.196±0.0380.462±0.0601.301±0.0570.251±0.0410.593±0.0581.147±0.052
Deep0.507±0.0181.365±0.0482.725±0.1700.358±0.0160.756±0.0231.399±0.037
ChapmanRandom0.186±0.0330.436±0.0701.023±0.0490.219±0.0250.528±0.0460.933±0.023
Deep0.414±0.0091.032±0.1421.802±0.0630.328±0.0150.880±0.0681.129±0.027

Table R3.2: Comparison of ReCal, oracle variants, and a noisy-cardinality variant in terms of predictive performance and disambiguation quality. Noisy-MLP trains the cardinality estimator on samples with 50% noise on their cardinality. Results are reported as mean ±\pm standard deviation. Bold and underlined values indicate the best and second-best results, respectively, within each setup.

DatasetScenarioMethodPredictive PerformanceDisambiguation Quality
Macro-F1↑AUROC↑AD↓RD↓
PTB-XLDeepBase-PLL0.332±0.0060.646±0.0051.327±0.5080.264±0.468
ReCal0.389±0.0070.697±0.0151.451±0.2550.013±0.097
Oracle-C0.390±0.0150.685±0.0151.250±0.3390.015±0.110
Oracle-MLP0.390±0.0150.690±0.0131.298±0.3160.012±0.095
Oracle-R0.393±0.0070.682±0.0051.204±0.3920.000±0.000
Noisy-MLP0.371±0.0060.681±0.0031.632±0.1000.013±0.098
PTB-XLRandomBase-PLL0.303±0.0090.630±0.0021.313±0.5590.281±0.486
ReCal0.343±0.0090.658±0.0081.131±0.3090.004±0.046
Oracle-C0.344±0.0080.654±0.0041.034±0.3300.003±0.037
Oracle-MLP0.348±0.0100.658±0.0051.196±0.3920.005±0.048
Oracle-R0.368±0.0020.666±0.0021.138±0.4510.000±0.000
Noisy-MLP0.324±0.0130.657±0.0101.370±0.1590.007±0.069
ChapmanDeepBase-PLL0.304±0.0040.642±0.0031.408±0.6780.305±0.485
ReCal0.372±0.0010.686±0.0051.355±0.3050.062±0.223
Oracle-C0.372±0.0090.684±0.0011.184±0.3950.060±0.228
Oracle-MLP0.383±0.0010.696±0.0011.254±0.3850.060±0.218
Oracle-R0.400±0.0060.695±0.0041.169±0.4120.000±0.000
Noisy-MLP0.364±0.0090.694±0.0071.486±0.0730.061±0.219
ChapmanRandomBase-PLL0.286±0.0020.634±0.0031.406±0.7060.320±0.499
ReCal0.388±0.0020.688±0.0021.192±0.4370.026±0.152
Oracle-C0.391±0.0090.681±0.0021.044±0.4560.030±0.174
Oracle-MLP0.391±0.0130.683±0.0061.168±0.4270.027±0.154
Oracle-R0.396±0.0120.681±0.0081.093±0.4580.000±0.000
Noisy-MLP0.375±0.0140.685±0.0061.342±0.163

Table R3.3: Performance of the auxiliary MLP in estimating ground-truth label cardinality. Micro denotes the metric over all test instances; Macro denotes the unweighted average across ground-truth cardinality groups.

MAE↓RMSE↓Bias(|.|)↓
DatasetScenarioMicroMacroMicroMacroMicroMacro
PTB-XLRandom0.484±0.0290.863±0.1000.859±0.0391.096±0.097+0.294±0.044-0.440±0.066
Deep1.009±0.2461.140±0.1991.574±0.3181.500±0.243+0.866±0.294+0.385±0.291
ChapmanRandom0.564±0.0751.529±0.1660.898±0.0851.695±0.155+0.368±0.119-1.141±0.233
Deep0.645±0.1781.901±0.1041.012±0.1572.073±0.105+0.393±0.256-1.519±0.180
Reviewer_T1C8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes ReCal, a partial multi-label learning framework for ECG classification that adaptively refines candidate labels based on sample reliability and estimated label cardinality. Experiments on three datasets demonstrate improvements in label disambiguation and classification under simulated annotation ambiguity.

Critical strengths

The paper identifies two meaningful limitations of existing disambiguation methods: modifying reliable labels and suppressing valid co-occurring diagnoses. The theoretical analysis provides a reasonable motivation for the proposed method.

The experiments cover multiple datasets and ambiguity settings, showing relatively consistent improvements over existing PLL baselines.

Critical weaknesses

The practical motivation is insufficiently justified. The framework assumes that candidate sets contain all true diagnoses, whereas real ECG annotations may also contain missing labels and diagnostic disagreements. Since all ambiguity is synthetically generated, it remains unclear whether the proposed problem accurately reflects real clinical annotation challenges.

The proposed mechanisms are not fully convincing. The reliability weights are closely related to training loss and may capture sample difficulty rather than annotation reliability, while the cardinality predictor learns from the model's own estimated labels. This raises concerns about whether the proposed components genuinely recover reliable supervision or merely regularize training.

Questions for authors

What real-world ECG annotation scenarios satisfy the assumption that all true diagnoses are included in the candidate set? Is there evidence that this type of ambiguity is common in clinical practice?

How does the reliability module distinguish incorrectly annotated samples from difficult but correctly labeled ECGs?

Given that cardinality supervision comes from the model's own predictions, how does the method avoid reinforcing incorrect cardinality estimates?

Would the method remain effective on real-world noisy or incomplete ECG annotations rather than synthetically generated candidate sets?

4 comments
AuthorsOpenReview ↗

part one of response

We thank Reviewer 2 for the careful assessment and for recognizing the two limitations of existing disambiguation we identify (modifying reliable labels and suppressing valid co-occurring diagnoses). We take the soundness concerns seriously. The reviewer raises two main issues: (1) the practical motivation, since our candidate sets assume all true diagnoses are present and ambiguity is synthetic; and (2) whether the reliability weights and the cardinality predictor recover reliable supervision. In the following, we provide detailed responses.

R2.Q1: Real-world utility of ambiguity in annotation

ReCal assumes that the candidate set contains all true diagnoses of a recording, which is the standard superset assumption in PLL. It holds when the candidate set is built from the diagnoses proposed by several readers and the true diagnosis is chosen among them. CODE Test (Ribeiro et al., 2020) follows this protocol. Each of its 827 ECGs was annotated independently by six clinicians of different expertise, and one cardiology expert serves as the gold standard. The union of the annotations forms the candidate set, and because the gold-standard annotator is one of the six readers, the superset property holds by construction. The readers disagree on 12.6% of the recordings (mean Fleiss’ kappa of 0.81), so these candidate sets do contain additional plausible labels.

Clinical datasets usually resolve such disagreement through consensus review (Rigueira et al., 2024) or adjudication by a senior expert (Ribeiro et al., 2020). This takes additional expert time, and ambiguity can persist in hard cases, for example when atrial fibrillation and atrial flutter overlap and observers disagree (Ribeiro et al., 2023). PLL keeps the readers’ proposals as candidates and lets the model resolve them. In this protocol the assumption means that the adjudicated diagnoses are among those proposed by at least one reader. We do not claim that the superset setting covers every form of clinical annotation, and we cannot say how common it is in practice. Multi-annotator ECG datasets are rare, and CODE Test is the only public one we know of that records individual annotations. The recently published CODE-II-test set (Abreu et al., 2026) contains 8,475 exams reviewed by multiple cardiologists, which will allow studies at a larger scale in future. We have clarified in the revised paper that our setting corresponds to the superset assumption.

Table R2.1: Separation of cardinality error and deviation of the final supervision, grouped by the thresholded weight. Cardinality MAE compares the number of labels retained by the final supervision with the true cardinality. Mean ± std over three seeds.

DatasetScenarioCardinality MAE ↓Deviation ↓
reliablereliable$_{Amb}$unreliablereliablereliable$_{Amb}$unreliable
PTB-XLRandom0.196±0.0380.462±0.0601.301±0.0570.251±0.0410.593±0.0581.147±0.052
Deep0.507±0.0181.365±0.0482.725±0.1700.358±0.0160.756±0.0231.399±0.037
ChapmanRandom0.186±0.0330.436±0.0701.023±0.0490.219±0.0250.528±0.0460.933±0.023
Deep0.414±0.0091.032±0.1421.802±0.0630.328±0.0150.880±0.0681.129±0.027
AuthorsOpenReview ↗

part two of response

R2.Q2: Difficulty vs reliability in weight

We first clarify how the weight is used, since this determines how it should be evaluated. The weight wiw_i is optimized on the loss of the current supervision (Eq. 6) and reflects how well the current supervision is fit, and it does not record whether the original candidate set was ambiguous. The weight is used in two places. First, it enters Eq. 8 as a continuous gate w~i\tilde{w}_i, scaling the update applied to the current supervision. Second, samples whose normalized weight is below 0.5 form the reliable set used to train the cardinality estimator. The relevant property is the quality of the supervision a sample carries.

Table R2.1 reports the quality of the recovered reliable samples through the thresholded weight on the training set at ambiguity proportion p=0.5p=0.5. Nearly all reliable samples are placed on the reliable side (recall$\ge$0.998), while part of the ambiguous samples are flagged as unreliable. An ambiguous sample with closer supervision to the model’s prediction can become easy to fit, receives a low weight, and is grouped with the reliable samples. Whether this is harmful depends on whether its supervision is correct. We report the cardinality error and the deviation of the final supervision for the reliable and unreliable groups, and additionally for the ambiguous samples within the reliable group. Reliable samples' supervision has over 4 times lower cardinality error and at least 3 times lower deviation than unreliable ones. Restricted to ambiguous samples on PTB-XL random, the reliable group has a cardinality error of 0.462 and a deviation of 0.593, compared with 1.301 and 1.147 for the unreliable group. To measure the effect of supervision quality, we replace learned components with oracle ones (Table R2.2). Oracle-C uses the ground-truth cardinality, Oracle-MLP trains the estimator on truly clean samples only, and Oracle-R uses ground-truth reliability for both the gate and the estimator training. If the estimator were reinforcing the errors, correct cardinalities or correct training samples should help substantially. While cardinality errors cost some label quality, Oracle-C changes Macro-F1 by 0.000 to 0.003 and Oracle-MLP by 0.001 to 0.011 relative to ReCal. The estimator is therefore not limited by the quality of its targets. The remaining gap lies in reliability identification. Replacing the learned gate with the ground-truth one improves Macro-F1 by 0.003 to 0.020, which is smaller than the 0.040 to 0.102 gain of ReCal over Base-PLL. This shows that the identification is imperfect and where the remaining headroom is. We will include Tables R2.1 and R2.2 in the appendix.

R2.Q3: Cardinality training from pseudo-label

The weight wiw_i reflects how well the current supervision is fit rather than whether the original candidate set was ambiguous. Samples whose normalized weight is below 0.5 form the reliable set used to train the cardinality estimator, and their regression targets are the numbers of labels recovered by their current supervision (Eq. 9). Because the estimator’s output in turn rescales the disambiguation (Sec. 3.3), a feedback loop exists in principle. Two design choices limit it. The estimator is trained only on the reliable set, and its output is clipped to [1,∣Si∣][1,|\mathcal{S}_i|], so it cannot introduce labels outside the candidate set. We also test it at three levels, which are the quality of the training targets, the performance of the estimator, and the effect of both on final performance.

Table R2.1 reports the cardinality error and the deviation of the final supervision of the train set for the reliable and unreliable groups, and for the ambiguous samples within the reliable group. The cardinality error of the reliable group, which is the error of the cardinality supervision against the true cardinality, ranges up to 0.51 across settings. Restricted to its ambiguous samples, the error is 0.44 to 1.37, compared with 1.02 to 2.73 for the unreliable group. To measure how much this contamination affects the final model, Oracle-MLP trains the estimator on truly clean samples only (Table R2.2). It changes Macro-F1 by +0.001 to +0.011 relative to ReCal, so the effect is small but not zero. Table R2.3 reports the estimator error on test samples against the ground-truth cardinality. Micro metrics average over all samples and are dominated by the frequent low-cardinality ones, whereas macro metrics weigh each cardinality group equally. The estimator has a macro MAE from 0.86 to 1.90 and micro of 0.48 to 1. To test whether this remaining error limits performance, Oracle-C supplies the ground-truth cardinality (Table R2.2). It changes Macro-F1 by less than +0.003, so the estimation error is not the limiting factor. As a complementary check, Table R2.4 reports the cardinality of the final disambiguated label sets. ReCal has the lowest macro MAE in all four settings. We will include Tables R2.1 to R2.4 in the appendix.

AuthorsOpenReview ↗

part three of response

R2.Q4: Real-world utility of ambiguity in annotation

CODE-test, described in our previous response, addresses it directly, since its candidate sets come from real disagreement between six annotators and not from our ambiguity generator. We split its 827 recordings into training and test sets with the same ratio as the other datasets, over three seeds and evaluate against the annotation of the cardiology expert, who serves as the gold standard. The results are in the main table (Table 1) and the dataset description is in the appendix. While Only 12.6% of the recordings carry spurious candidates, ReCal outperforms baselines. We should also be clear about what this does not cover. The method assumes that the candidate set is complete, so annotations that miss a true diagnosis fall outside our setting, and we have not evaluated them.

Table R2.2: Comparison of ReCal, oracle variants, and a noisy-cardinality variant in terms of predictive performance and disambiguation quality. Noisy-MLP trains the cardinality estimator on samples with 50% noise on their cardinality. Results are reported as mean ±\pm standard deviation. Bold and underlined values indicate the best and second-best results, respectively, within each setup.

DatasetScenarioMethodPredictive PerformanceDisambiguation Quality
Macro-F1↑AUROC↑AD↓RD↓
PTB-XLDeepBase-PLL0.332±0.0060.646±0.0051.327±0.5080.264±0.468
ReCal0.389±0.0070.697±0.0151.451±0.2550.013±0.097
Oracle-C0.390±0.0150.685±0.0151.250±0.3390.015±0.110
Oracle-MLP0.390±0.0150.690±0.0131.298±0.3160.012±0.095
Oracle-R0.393±0.0070.682±0.0051.204±0.3920.000±0.000
Noisy-MLP0.371±0.0060.681±0.0031.632±0.1000.013±0.098
PTB-XLRandomBase-PLL0.303±0.0090.630±0.0021.313±0.5590.281±0.486
ReCal0.343±0.0090.658±0.0081.131±0.3090.004±0.046
Oracle-C0.344±0.0080.654±0.0041.034±0.3300.003±0.037
Oracle-MLP0.348±0.0100.658±0.0051.196±0.3920.005±0.048
Oracle-R0.368±0.0020.666±0.0021.138±0.4510.000±0.000
Noisy-MLP0.324±0.0130.657±0.0101.370±0.1590.007±0.069
ChapmanDeepBase-PLL0.304±0.0040.642±0.0031.408±0.6780.305±0.485
ReCal0.372±0.0010.686±0.0051.355±0.3050.062±0.223
Oracle-C0.372±0.0090.684±0.0011.184±0.3950.060±0.228
Oracle-MLP0.383±0.0010.696±0.0011.254±0.3850.060±0.218
Oracle-R0.400±0.0060.695±0.0041.169±0.4120.000±0.000
Noisy-MLP0.364±0.0090.694±0.0071.486±0.0730.061±0.219
ChapmanRandomBase-PLL0.286±0.0020.634±0.0031.406±0.7060.320±0.499
ReCal0.388±0.0020.688±0.0021.192±0.4370.026±0.152
Oracle-C0.391±0.0090.681±0.0021.044±0.4560.030±0.174
Oracle-MLP0.391±0.0130.683±0.0061.168±0.4270.027±0.154
Oracle-R0.396±0.0120.681±0.0081.093±0.4580.000±0.000
Noisy-MLP0.375±0.0140.685±0.0061.342±0.1630.024±0.146
AuthorsOpenReview ↗

part four of response

Table R2.3: Performance of the auxiliary MLP in estimating ground-truth label cardinality. Micro denotes the metric over all test instances; Macro denotes the unweighted average across ground-truth cardinality groups.

MAE↓RMSE↓Bias(|.|)↓
DatasetScenarioMicroMacroMicroMacroMicroMacro
PTB-XLRandom0.484±0.0290.863±0.1000.859±0.0391.096±0.097+0.294±0.044-0.440±0.066
Deep1.009±0.2461.140±0.1991.574±0.3181.500±0.243+0.866±0.294+0.385±0.291
ChapmanRandom0.564±0.0751.529±0.1660.898±0.0851.695±0.155+0.368±0.119-1.141±0.233
Deep0.645±0.1781.901±0.1041.012±0.1572.073±0.105+0.393±0.256-1.519±0.180

Table R2.4: Cardinality recovery after candidate-label disambiguation. Micro denotes the metric over all test instances; Macro denotes the unweighted average across ground-truth cardinality groups. Bias values are signed, while ranking is based on absolute bias.

MAE↓RMSE↓Bias
DatasetScenarioMethodMicroMacroMicroMacroMicroMacro
PTB-XLRandomReCal0.350±0.0230.516±0.1230.782±0.0420.881±0.1520.300±0.028-0.129±0.136
Base-PLL0.373±0.0032.131±0.0700.781±0.0062.219±0.064-0.297±0.003-2.111±0.070
PRODEN0.425±0.0012.549±0.0270.881±0.0022.568±0.028-0.425±0.001-2.549±0.027
CAVL0.413±0.0002.500±0.0000.860±0.0002.500±0.000-0.413±0.000-2.500±0.000
LW0.617±0.0102.074±0.0290.897±0.0022.182±0.0320.107±0.011-1.905±0.033
PTB-XLDeepReCal0.971±0.0461.078±0.1061.798±0.0721.754±0.1550.945±0.0490.636±0.137
Base-PLL0.377±0.0021.960±0.0320.767±0.0032.085±0.033-0.208±0.003-1.912±0.034
PRODEN0.436±0.0012.569±0.0330.898±0.0032.596±0.036-0.436±0.001-2.569±0.033
CAVL0.413±0.0002.500±0.0000.860±0.0002.500±0.000-0.413±0.000-2.500±0.000
LW0.673±0.0092.077±0.0300.927±0.0082.182±0.0320.174±0.008-1.879±0.033
ChapmanRandomReCal0.341±0.0140.999±0.0760.697±0.0141.320±0.0850.241±0.021-0.753±0.094
Base-PLL0.381±0.0002.396±0.0270.751±0.0012.436±0.026-0.356±0.003-2.389±0.024
PRODEN0.393±0.0012.506±0.0030.773±0.0002.514±0.003-0.393±0.001-2.506±0.003
CAVL0.390±0.0002.500±0.0000.768±0.0002.500±0.000-0.390±0.000-2.500±0.000
LW0.618±0.0052.020±0.0760.840±0.0072.116±0.0800.247±0.006-1.812±0.076
ChapmanDeepReCal0.758±0.0681.310±0.0131.346±0.0891.718±0.0300.644±0.068-0.567±0.063
Base-PLL0.377±0.0002.320±0.0180.742±0.0012.394±0.013-0.306±0.007-2.298±0.020
PRODEN0.398±0.0012.526±0.0050.785±0.0022.541±0.005-0.398±0.001-2.526±0.005
CAVL0.390±0.0002.500±0.0000.768±0.0002.500±0.000-0.390±0.000-2.500±0.000
LW0.635±0.0122.013±0.0170.851±0.0092.108±0.0170.269±0.021-1.795±0.019

Ribeiro et al. "Automatic diagnosis of the 12-lead ECG using a deep neural network." Nature communications 2020.

Ribeiro et al. "Can deep learning models differentiate atrial fibrillation from atrial flutter?." medRxiv 2023.

Rigueira et al. "Optimizing ECG Audits: Clustering-Based Identification of Ambiguous Exams." ENIAC 2024.

Abreu et al. "CODE-II: A large-scale dataset for artificial intelligence in ECG analysis." npj Digital Medicine 2026.

Comments on the paper

5 comments

We thank the reviewer for the careful assessment. We appreciate that the reviewer finds the problem meaningful, namely the corruption of reliable labels and the suppression of valid co-occurring diagnoses during disambiguation, and that the gains of ReCal are consistent across ambiguity settings and supported by the ablations.

R1.W1: Methodological novelty

We study why confidence-based disambiguation fails on multi-label data, and show that the standard disambiguation is invariant to both the reliability of a sample and its ground-truth cardinality (Lemma 1), that this corrupts reliable multi-label samples (Corollary 1), and that sharpening drives the recovered cardinality toward one (the cardinality-collapse proposition in the appendix). Sample reweighting and cardinality-aware prediction do exist, but they play a different role in prior work. Reweighting typically scales the loss of hard or noisy samples, whereas in ReCal the weight controls how much the supervision of each sample is refined. Cardinality-aware methods decide how many labels to output at prediction time (Cortes et al., 2024), whereas in ReCal, cardinality is estimated during training from the reliable subset and used to reshape the disambiguated label set. Label cardinality is a defining attribute of multi-label samples, and to our knowledge, it has not been treated as an explicit part of the disambiguation design in PML. The two components are also coupled, since the reliability module supplies the samples from which cardinality is learned, and the ablation shows complementary effects, with removing reliability raising RD and removing cardinality raising AD.

R1.Q1: Difficulty vs reliability in weight

We agree that wiw_i is closely tied to the training loss. Minimizing the per-sample term of Eq. (6) over wiw_i gives wi⋆=log⁡(ℓi/λ)w_i^\star=\log(\ell_i/\lambda), so the weight reflects how well the current supervision is fit. The superset setting makes the loss informative for a specific reason. Every spurious candidate is a false positive that can mainly be fit by memorizing the sample, whereas the labels of a correct sample, even a difficult one, agree with patterns shared across its class and can be fit through generalization (Wu et al., 2023). The weights are constrained in three ways. First, the term λ∑iwi\lambda\sum_i w_i in Eq. (6) prevents the trivial solution wi→∞w_i\rightarrow\infty and fixes the scale of the weights (λ=10−6\lambda=10^{-6}). Second, the weights are min-max normalized to [0,1][0,1] over all training samples before they are used as a reliability measure. Third, the normalized weights pass through the sigmoid gate of Eq. (7), so the weight scales the update of the current supervision continuously (Eq. (8)).

Cost of confusing difficulty with ambiguity: The same error, a difficult but correctly labeled sample receiving a high weight, has two consequences in our framework. First, the gate opens and the sample is partially disambiguated. Since the update is smooth, the supervision moves toward the model confidence only in proportion to the gate value, and the resulting deviation is limited to the suppression term A(ρ)A(\rho) of the deviation decomposition (Eq. 5). Empirically, the deviation of reliable samples (RD) is 0.004 to 0.062 for ReCal, compared with 0.264 to 0.320 for Base-PLL (Table R1.2). Second, the sample may be excluded from the reliable set and be underrepresented in the training of the cardinality estimator. The recall of reliable samples for this set is at least 0.998 (Table R1.1). Both measurements are taken on data where correctly labeled samples are naturally difficult, since PTB-XL and Chapman are long-tailed and multi-label, include classes with roughly 100 samples, and show substantial differences in class-wise performance (Figs. 3 and 5). We further quantify the cost of this imperfect separation with oracle variants. Oracle-R replaces the learned weights with the ground-truth partition into reliable and ambiguous samples, for both the gate and the cardinality estimator. Table R1.2 shows that ReCal follows this oracle closely. The Macro-F1 of ReCal is within 0.004 to 0.028 of Oracle-R. The gap to the oracle is smaller than the gain of ReCal over Base-PLL (0.040 to 0.102) in every setting, so errors in reliability identification leave some headroom but do not remove the benefit of the method. These measurements are aggregated over samples and do not vary rarity, signal quality, or classification difficulty independently.

AuthorsOpenReview ↗

part two of response

Table R1.1: Separation of cardinality error and deviation of the final supervision, grouped by the thresholded weight. Cardinality MAE compares the number of labels retained by the final supervision with the true cardinality. Mean ± std over three seeds.

DatasetScenarioCardinality MAE ↓Deviation ↓
reliablereliable$_{Amb}$unreliablereliablereliable$_{Amb}$unreliable
PTB-XLRandom0.196±0.0380.462±0.0601.301±0.0570.251±0.0410.593±0.0581.147±0.052
Deep0.507±0.0181.365±0.0482.725±0.1700.358±0.0160.756±0.0231.399±0.037
ChapmanRandom0.186±0.0330.436±0.0701.023±0.0490.219±0.0250.528±0.0460.933±0.023
Deep0.414±0.0091.032±0.1421.802±0.0630.328±0.0150.880±0.0681.129±0.027

R1.Q2: Quantize reliable sample recovery

As discussed in our previous response, the weight wiw_i reflects how well the current supervision is fit, not whether the original candidate set was ambiguous. It acts as a continuous gate w~i\tilde{w}_i on the update in Eq. 8, and samples with a normalized weight below 0.5 form the reliable set used to train the cardinality estimator. Accordingly, the relevant property is the quality of the supervision a sample carries. Table R1.1 reports the quality of the recovered reliable samples with the thresholded weight on the training set at ambiguity proportion p=0.5p=0.5. Nearly all reliable samples are placed on the reliable side (recall$\ge$0.998), while part of the ambiguous samples are flagged as unreliable. An ambiguous sample with closer supervision to the model’s prediction can become easy to fit, receives a low weight, and is grouped with the reliable samples. Whether this is harmful depends on whether the supervision it carries is correct, which accuracy cannot reveal. We report the cardinality error and the deviation of the final supervision for the reliable and unreliable groups, and additionally for the ambiguous samples within the reliable group. Reliable samples' supervision has over 4 times lower cardinality error and at least 3 times lower deviation than unreliable ones. The cardinality error of the reliable group quantifies how much error can be passed to the cardinality estimator through its own training signal. Restricted to ambiguous samples on PTB-XL random, the reliable group has a cardinality error of 0.462 and a deviation of 0.593, compared with 1.301 and 1.147 for the unreliable group. To measure the effect of supervision quality, we replace learned components with oracle ones (Table R1.2). Oracle-C uses the ground-truth cardinality, Oracle-MLP trains the estimator on truly clean samples only, and Oracle-R uses ground-truth reliability for both the gate and the estimator training. If the estimator were reinforcing the errors, correct cardinalities or correct training samples should help substantially. While cardinality errors cost some label quality, Oracle-C changes Macro-F1 by 0.000 to 0.003 and Oracle-MLP by 0.001 to 0.011 relative to ReCal. The estimator is therefore not limited by the quality of its targets. The remaining gap lies in reliability identification. Replacing the learned gate with the ground-truth one improves Macro-F1 by 0.003 to 0.020, which is smaller than the 0.040 to 0.102 gain of ReCal over Base-PLL. This shows that the identification is imperfect and where the remaining headroom is. We will include Tables R1.1 and R1.2 in the appendix.

AuthorsOpenReview ↗

part three of response

Table R1.2: Comparison of ReCal, oracle variants, and a noisy-cardinality variant in terms of predictive performance and disambiguation quality. Noisy-MLP trains the cardinality estimator on samples with 50% noise on their cardinality. Results are reported as mean ±\pm standard deviation. Bold and underlined values indicate the best and second-best results, respectively, within each setup.

DatasetScenarioMethodPredictive PerformanceDisambiguation Quality
Macro-F1↑AUROC↑AD↓RD↓
PTB-XLDeepBase-PLL0.332±0.0060.646±0.0051.327±0.5080.264±0.468
ReCal0.389±0.0070.697±0.0151.451±0.2550.013±0.097
Oracle-C0.390±0.0150.685±0.0151.250±0.3390.015±0.110
Oracle-MLP0.390±0.0150.690±0.0131.298±0.3160.012±0.095
Oracle-R0.393±0.0070.682±0.0051.204±0.3920.000±0.000
Noisy-MLP0.371±0.0060.681±0.0031.632±0.1000.013±0.098
PTB-XLRandomBase-PLL0.303±0.0090.630±0.0021.313±0.5590.281±0.486
ReCal0.343±0.0090.658±0.0081.131±0.3090.004±0.046
Oracle-C0.344±0.0080.654±0.0041.034±0.3300.003±0.037
Oracle-MLP0.348±0.0100.658±0.0051.196±0.3920.005±0.048
Oracle-R0.368±0.0020.666±0.0021.138±0.4510.000±0.000
Noisy-MLP0.324±0.0130.657±0.0101.370±0.1590.007±0.069
ChapmanDeepBase-PLL0.304±0.0040.642±0.0031.408±0.6780.305±0.485
ReCal0.372±0.0010.686±0.0051.355±0.3050.062±0.223
Oracle-C0.372±0.0090.684±0.0011.184±0.3950.060±0.228
Oracle-MLP0.383±0.0010.696±0.0011.254±0.3850.060±0.218
Oracle-R0.400±0.0060.695±0.0041.169±0.4120.000±0.000
Noisy-MLP0.364±0.0090.694±0.0071.486±0.0730.061±0.219
ChapmanRandomBase-PLL0.286±0.0020.634±0.0031.406±0.7060.320±0.499

Table R1.3: Performance of the auxiliary MLP in estimating ground-truth label cardinality. Micro denotes the metric over all test instances; Macro denotes the unweighted average across ground-truth cardinality groups.

MAE↓RMSE↓Bias(∣⋅∣\vert\cdot\vert)↓
DatasetScenarioMicroMacroMicroMacroMicroMacro
PTB-XLRandom0.484±0.0290.863±0.1000.859±0.0391.096±0.097+0.294±0.044-0.440±0.066
Deep1.009±0.2461.140±0.1991.574±0.3181.500±0.243+0.866±0.294+0.385±0.291
ChapmanRandom0.564±0.0751.529±0.1660.898±0.0851.695±0.155+0.368±0.119-1.141±0.233
Deep0.645±0.1781.901±0.1041.012±0.1572.073±0.105+0.393±0.256-1.519±0.180

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.