- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies partial multi-label learning for ECG diagnosis. Its main observation is that confidence-based disambiguation can hurt samples whose candidate labels are already correct and can also suppress valid co-occurring diagnoses. ReCal addresses these two issues with a learned sample weight that controls how much a candidate set is refined and an auxiliary cardinality predictor that estimates how many labels should be kept. The method is tested on PTB-XL, Chapman-Shaoxing, and AMIGOS under several controlled ambiguity settings. The paper reports classification results, recovered-cardinality analyses, reliable/ambiguous-sample deviation, and ablations of the two proposed components.
Critical strengths
- The paper identifies a meaningful failure mode of confidence-based disambiguation rather than only reporting a performance gap. The analysis explains why sharpening can move a correct multi-label target away from the ground truth and why high-cardinality samples are especially vulnerable. The two proposed components follow naturally from this analysis.
- The evaluation is reasonably broad within the stated PLL setting. Several representative baselines are compared using a shared ECG backbone, and the analysis goes beyond final Macro-F1. The recovered-cardinality and reliable-sample results directly test the failure modes targeted by the method.
Critical weaknesses
- The central clinical premise is broader than the actual problem the method solves. ReCal assumes , so every true diagnosis is already present in the candidate set. The method can remove extra candidates, but it cannot recover a diagnosis that is missing from the annotation. Yet incomplete clinical annotation is used as a major motivation. The experiments also create ambiguity by adding candidates to datasets with known labels. This is not a minor edge case. It means the paper does not address one of the main forms of incompleteness suggested by its clinical framing.
- The learned sample weight is not an identifiable measure of annotation reliability. It is driven by how hard the current supervision is to fit. A rare, noisy, atypical, or simply difficult but correctly labeled ECG can therefore receive the same signal as a genuinely ambiguous label set. The cardinality module then learns targets from the model's own pseudo-labels on samples selected as reliable by this proxy. This makes the supervision circular. The paper shows that the components improve F1, but it does not show that the weight really tracks label reliability or that the cardinality loop corrects errors rather than reinforcing them.
Questions for authors
- Can the authors evaluate a setting in which one or more true labels are missing from the observed candidate set? If not, the clinical claim should be narrowed to the superset-candidate setting assumed by .
- Can the authors show whether the learned weights distinguish label ambiguity from ordinary difficulty? For example, keep the labels correct and vary rarity, signal quality, or classification difficulty.
- Please report how accurately the learned reliable subset matches the known reliable/ambiguous split during training. An oracle-reliable subset or oracle-cardinality comparison would also help determine whether the cardinality loop is correcting or propagating early pseudo-label errors.
- The paper states that classification performance is evaluated using both Macro-F1 and AUC-ROC, but Table 1 only reports full Macro-F1 numbers while AUC-ROC results are mostly presented as class-wise heatmaps in Figure 5. Could the authors provide aggregate AUC-ROC values for all methods across all datasets and ambiguity settings in the main results table?
5 comments
part two of response
R3.Q3: Quantize reliable sample recovery
The weight is optimized on the loss of the current supervision (Eq. 6) and reflects how well the current supervision is fit, and it does not record whether the original candidate set was ambiguous. The weight is used in two places. First, it enters Eq. 8 as a continuous gate , scaling the update applied to the current supervision. Second, samples whose normalized weight is below 0.5 form the reliable set used to train the cardinality estimator. In both roles, the relevant property is the quality of the supervision a sample carries. Table R3.1 reports, on the training set at ambiguity proportion , the quality of the recovered reliable sample with the thresholded weight. Nearly all reliable samples are placed on the reliable side (recall 0.998), while part of the ambiguous samples are identified as unreliable. An ambiguous sample with supervision closer to the model’s prediction can become easy to fit, receives a low weight, and is grouped with the reliable samples. Whether this is harmful depends on whether the supervision it carries is correct, which accuracy cannot reveal.
We report the cardinality error and the deviation of the final supervision for the reliable and unreliable groups, and additionally for the ambiguous samples within the reliable group. Because the reliable group contains nearly all clean samples, the informative comparison is between ambiguous samples on both sides of the threshold. Ambiguous samples in the reliable group have a 1.7 to 2.8 times lower cardinality error and a 1.3 to 1.9 times lower deviation than those in the unreliable group, for example 0.462 and 0.593 against 1.301 and 1.147 on PTB-XL random. To measure the cost of the imperfect identification, we replace learned components with oracle ones (Table R3.2). Oracle-MLP trains the estimator on truly clean samples only, and Oracle-R additionally uses the ground-truth reliability for the gate. Replacing the learned gate with the ground-truth one (Oracle-R relative to Oracle-MLP) improves Macro-F1 by 0.003 to 0.020 on PTB-XL and Chapman, which is smaller than the 0.040 to 0.102 gain of ReCal over Base-PLL. This shows that the identification is imperfect and where the remaining headroom is. We will include Tables R3.1 and R3.2 in the appendix.
R3.Q4: Cardinality Estimation Accuracy
Table R3.2 contains these comparisons on PTB-XL and Chapman. We define each variant by the component it replaces or perturbs. 1) Oracle-C (oracle cardinality): The estimated cardinality in the disambiguation is replaced by the ground-truth cardinality. This removes all errors of the estimator, including those inherited from early pseudo-labels. 2) Oracle-MLP (oracle reliable subset for the estimator): The estimator is trained only on truly clean samples, so its training targets are free of pseudo-label error. The disambiguation remains similar to ReCal. 3) Oracle-R: Ground-truth reliability is used for the gate and for the estimator training. Oracle-R minus Oracle-MLP therefore isolates the effect of the gate. 4) Noisy-MLP (noisy estimator targets): The estimator is trained as in ReCal, but the cardinality target is increased by two for a random 50% of the training samples. This mimics the inflated cardinality that ambiguous candidate sets would contribute and tests the opposite direction of the oracles. If the loop propagated early errors, replacing with correct cardinalities or correct training targets should recover a substantial part of the gain over Base-PLL. While incorrect cardinality does cost some label quality, Oracle-C changes Macro-F1 by up to +0.003 and Oracle-MLP by less than +0.011 relative to ReCal. Noisy-MLP lowers Macro-F1 by 0.008 to 0.019 and increases AD by 0.131 to 0.239, and RD is mostly unchanged. While this noise is strong, the estimator is not insensitive to its targets, and contaminated targets mainly affect the recovery of label sets. However, Noisy-MLP improves performance over Base-PLL in all four settings. We therefore conclude that the cardinality loop does not propagate early errors to a degree that limits performance. Replacing the learned gate with the ground-truth one (Oracle-R relative to Oracle-MLP) improves Macro-F1 by up to 0.020, which is the main remaining headroom.
R3.Q5: Add AUCROC results to table
We agree that aggregate AUC-ROC should be available next to Macro-F1. Table 7 in the appendix now reports the AUC-ROC for all methods, datasets and ambiguity settings, in the same layout as Table 1. Figure 5 (class specific performance) remains in the paper to show where in the label space the methods differ, but it is not meant to replace the aggregate numbers. We kept the main table limited to Macro-F1 because a second metric for every method would roughly double its width within the page limit. We have added a sentence in Sec. 4 that points to the appendix table and summarizes the comparison.
part three of response
Table R3.1: Separation of cardinality error and deviation of the final supervision, grouped by the thresholded weight. Cardinality MAE compares the number of labels retained by the final supervision with the true cardinality. Mean ± std over three seeds.
| Dataset | Scenario | Cardinality MAE ↓ | Deviation ↓ | ||||
|---|---|---|---|---|---|---|---|
| reliable | reliable$_{Amb}$ | unreliable | reliable | reliable_{Amb}\ | unreliable | ||
| PTB-XL | Random | 0.196±0.038 | 0.462±0.060 | 1.301±0.057 | 0.251±0.041 | 0.593±0.058 | 1.147±0.052 |
| Deep | 0.507±0.018 | 1.365±0.048 | 2.725±0.170 | 0.358±0.016 | 0.756±0.023 | 1.399±0.037 | |
| Chapman | Random | 0.186±0.033 | 0.436±0.070 | 1.023±0.049 | 0.219±0.025 | 0.528±0.046 | 0.933±0.023 |
| Deep | 0.414±0.009 | 1.032±0.142 | 1.802±0.063 | 0.328±0.015 | 0.880±0.068 | 1.129±0.027 |
Table R3.2: Comparison of ReCal, oracle variants, and a noisy-cardinality variant in terms of predictive performance and disambiguation quality. Noisy-MLP trains the cardinality estimator on samples with 50% noise on their cardinality. Results are reported as mean standard deviation. Bold and underlined values indicate the best and second-best results, respectively, within each setup.
| Dataset | Scenario | Method | Predictive Performance | Disambiguation Quality | ||
|---|---|---|---|---|---|---|
| Macro-F1↑ | AUROC↑ | AD↓ | RD↓ | |||
| PTB-XL | Deep | Base-PLL | 0.332±0.006 | 0.646±0.005 | 1.327±0.508 | 0.264±0.468 |
| ReCal | 0.389±0.007 | 0.697±0.015 | 1.451±0.255 | 0.013±0.097 | ||
| Oracle-C | 0.390±0.015 | 0.685±0.015 | 1.250±0.339 | 0.015±0.110 | ||
| Oracle-MLP | 0.390±0.015 | 0.690±0.013 | 1.298±0.316 | 0.012±0.095 | ||
| Oracle-R | 0.393±0.007 | 0.682±0.005 | 1.204±0.392 | 0.000±0.000 | ||
| Noisy-MLP | 0.371±0.006 | 0.681±0.003 | 1.632±0.100 | 0.013±0.098 | ||
| PTB-XL | Random | Base-PLL | 0.303±0.009 | 0.630±0.002 | 1.313±0.559 | 0.281±0.486 |
| ReCal | 0.343±0.009 | 0.658±0.008 | 1.131±0.309 | 0.004±0.046 | ||
| Oracle-C | 0.344±0.008 | 0.654±0.004 | 1.034±0.330 | 0.003±0.037 | ||
| Oracle-MLP | 0.348±0.010 | 0.658±0.005 | 1.196±0.392 | 0.005±0.048 | ||
| Oracle-R | 0.368±0.002 | 0.666±0.002 | 1.138±0.451 | 0.000±0.000 | ||
| Noisy-MLP | 0.324±0.013 | 0.657±0.010 | 1.370±0.159 | 0.007±0.069 | ||
| Chapman | Deep | Base-PLL | 0.304±0.004 | 0.642±0.003 | 1.408±0.678 | 0.305±0.485 |
| ReCal | 0.372±0.001 | 0.686±0.005 | 1.355±0.305 | 0.062±0.223 | ||
| Oracle-C | 0.372±0.009 | 0.684±0.001 | 1.184±0.395 | 0.060±0.228 | ||
| Oracle-MLP | 0.383±0.001 | 0.696±0.001 | 1.254±0.385 | 0.060±0.218 | ||
| Oracle-R | 0.400±0.006 | 0.695±0.004 | 1.169±0.412 | 0.000±0.000 | ||
| Noisy-MLP | 0.364±0.009 | 0.694±0.007 | 1.486±0.073 | 0.061±0.219 | ||
| Chapman | Random | Base-PLL | 0.286±0.002 | 0.634±0.003 | 1.406±0.706 | 0.320±0.499 |
| ReCal | 0.388±0.002 | 0.688±0.002 | 1.192±0.437 | 0.026±0.152 | ||
| Oracle-C | 0.391±0.009 | 0.681±0.002 | 1.044±0.456 | 0.030±0.174 | ||
| Oracle-MLP | 0.391±0.013 | 0.683±0.006 | 1.168±0.427 | 0.027±0.154 | ||
| Oracle-R | 0.396±0.012 | 0.681±0.008 | 1.093±0.458 | 0.000±0.000 | ||
| Noisy-MLP | 0.375±0.014 | 0.685±0.006 | 1.342±0.163 |
Table R3.3: Performance of the auxiliary MLP in estimating ground-truth label cardinality. Micro denotes the metric over all test instances; Macro denotes the unweighted average across ground-truth cardinality groups.
| MAE↓ | RMSE↓ | Bias(|.|)↓ | |||||
|---|---|---|---|---|---|---|---|
| Dataset | Scenario | Micro | Macro | Micro | Macro | Micro | Macro |
| PTB-XL | Random | 0.484±0.029 | 0.863±0.100 | 0.859±0.039 | 1.096±0.097 | +0.294±0.044 | -0.440±0.066 |
| Deep | 1.009±0.246 | 1.140±0.199 | 1.574±0.318 | 1.500±0.243 | +0.866±0.294 | +0.385±0.291 | |
| Chapman | Random | 0.564±0.075 | 1.529±0.166 | 0.898±0.085 | 1.695±0.155 | +0.368±0.119 | -1.141±0.233 |
| Deep | 0.645±0.178 | 1.901±0.104 | 1.012±0.157 | 2.073±0.105 | +0.393±0.256 | -1.519±0.180 |
part one of response
We thank the reviewer for the careful evaluation. We are glad that the analysis of why sharpening pulls a correct multi-label target away from the ground truth, and why high-cardinality samples suffer most, was found meaningful, and that the two components were seen as following from it. R3.W1 and R3.W2 are addressed in R3.Q1 and R3.Q2–R3.Q4, respectively.
R3.Q1: why not noisy partial label learning
We have not evaluated a setting in which true labels are missing from the candidate set, and we agree that the clinical claim should be narrowed accordingly. ReCal addresses partial label learning under the superset assumption. Every true diagnosis is in the candidate set, and the set may also contain false positives. This is the setting of our analysis, our controlled ambiguity generation and our evaluation on CODE-test. We chose it because it matches a common annotation protocol, in which several readers independently propose diagnoses and a consensus review or a senior adjudication then fixes the reference (Ribeiro et al., 2020; Rigueira et al., 2024). The candidate set can be defined as the union of the annotator's diagnosis. CODE-test follows this protocol, with six readers per recording and a cardiology expert as the gold standard, and the recent larger CODE-II-test (Abreu et al., 2026) is reviewed by multiple cardiologists, which will allow studies at a larger scale in future. Under this protocol the assumption fails only when all reader miss the diagnosis that the reference includes. The case where true labels may be missing is noisy partial label learning, which is a different problem. A disambiguation operator restricted to the candidate set cannot recover a missing diagnosis, so we do not claim that ReCal addresses it. We also have not tested how missing labels would affect the reliability weights or the cardinality estimator. We have made two changes to the paper. The abstract and introduction no longer describe the annotations as ambiguous candidate-label supervision under the superset assumption. We now state the assumption explicitly in the problem setup and have added a limitations paragraph that names missing diagnoses as an open direction. We consider noisy partial label learning an important extension for clinical annotation and leave it to future work.
R3.Q2: Difficulty vs reliability in weight
We agree that wi is closely tied to the training loss. Minimizing the per-sample term of Eq. (6) over wi gives wi⋆=log(ℓi/λ), so the weight reflects how well the current supervision is fit. The superset setting makes the loss informative for a specific reason. Every spurious candidate is a false positive that can mainly be fit by memorizing the sample, whereas the labels of a correct sample, even a difficult one, agree with patterns shared across its class and can be fit through generalization (Wu et al., 2023). Confusing a difficult but correctly labeled sample with an ambiguous one by receiving a high weight, has two consequences in our framework. First, the sigmoid-based gate partially disambiguates the sample. Since the update is smooth, the supervision moves toward the model confidence only in proportion to the gate value, and the resulting deviation is limited to the suppression term A(ρ) of the deviation decomposition (Eq. 5). Empirically, the deviation of reliable samples (RD) is 0.004 to 0.062 for ReCal, compared with 0.264 to 0.320 for Base-PLL (Table R3.2). Second, the sample may be excluded from the reliable set and be underrepresented in the training of the cardinality estimator. The recall of reliable samples for this set is at least 0.998 (Table R3.1). Both measurements are taken on data where correctly labeled samples are naturally difficult, since PTB-XL and Chapman are long-tailed and multi-label, include classes with roughly 100 samples, and show substantial differences in class-wise performance (Figs. 3 and 5). We further quantify the cost of this imperfect separation with oracle variants. Oracle-R replaces the learned weights with the ground-truth partition into reliable and ambiguous samples, for both the gate and the cardinality estimator. Table R3.2 shows that ReCal follows this oracle closely. The Macro-F1 of ReCal is within 0.004 to 0.028 of Oracle-R. The gap to the oracle is smaller than the gain of ReCal over Base-PLL (0.040 to 0.102) in every setting, so errors in reliability identification leave some headroom but do not remove the benefit of the method. These measurements are aggregated over samples and do not vary rarity, signal quality, or classification difficulty independently.