acceptodds
Under review as a conference paper at ICLR 2027

Learning from Ugly Ducklings: Hierarchical Modeling of Intra-Patient Variance for Melanoma Prediction

Abstract

Medical AI shows great potential to improve the efficiency and accuracy of diagnosis from medical images. However, disease manifestations vary across individuals, such that a similar observation may be typical for one patient but concerning for another. Most medical image classifiers assume independent and identically distributed (IID) observations and therefore do not account for intrapatient variation. In melanoma detection, dermatologists account for this variation by assessing both a lesion’s appearance and whether it stands out from the patient’s other lesions, a pattern known as the ugly-duckling principle. We model this setting with a hierarchical model that estimates a patient reference from unlabeled same-patient lesions and derives how deviations from this reference modify lesion-level evidence. We introduce EIDER (Evidence In Deviations from an Estimated Reference), which estimates this reference and learns how lesion-to-reference deviations adjust lesion-level predictions. We evaluate EIDER across three dermatology settings. On total-body photography, EIDER achieves an AUROC of 0.959, compared with 0.921 for our strongest image-only baseline. Controlled analyses show that replacing a patient's reference with that of another patient decreases performance, larger reference sets are associated with greater gains over lesion-only prediction, and combining signed and squared deviations performs better than either restricted deviation form; we further evaluate EIDER in dermoscopic and longitudinal settings, and assess its applicability to EEG seizure detection.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
melanoma detection, medical imaging, patient context, ugly-duckling, hierarachical model, dermatology

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ce61OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on melanoma prediction, particularly interested in ugly-duckling principle, which is a pattern that associate skin lesions within a patient. The authors use a hierachical model (patient as reference) to derives the variations of unlabeled lesions, and named their contribution as Evidence In Deviations from an Estimated Reference (EIDER). to learns lesion-to-reference deviation and adjust the lesion-level predictions.

The method is evaluated on total-body photograph (TBP), dermoscopy and EEG seizure detection. For TBP, the authors compared it with image-only baseline and demonstrated improved AUROC of 0.038. Further analysis involved changing patient references and the size of reference sets.

Critical strengths

Formulate hierarchical model for patient as reference and a method named Evidence In Deviations from an Estimated Reference (EIDER). Results evaluated on different datasets, imaging and EEG, and demonstrated generalisability of the proposed method.

Critical weaknesses

The split of datasets and issues with the imbalanced datasets are not clearly stated in this paper. It is difficult to follow as many of the details are in the appendices. Key information should be in the main manuscript. Difficulties in understanding the novelty of the method. There are no clear descriptions on technical novelty of the work. The evaluation requires more information on what are the test datasets and what is the sensitivity and specificity. More metrics will help to understand the merit of the paper.

Questions for authors

The task is for melanoma prediction. However, the datasets with only 393 melanomas (ISIC 2024), 584 melanomas (SIIM-ISIC 2020), do every patients with a melanoma to be used as reference and how many of the patients with melanoma? What is the technical novelty of this paper? As the datasets are very imbalanced, how reliable is the results on sensitivity and specificity? For the longitudinal cohort of 458 patients, how did you split them in the experiment? From your analysis, would your method produce more false positives when your references are patients with melanoma?

2 comments

We thank the reviewer for their feedback and appreciation of our theoretical framework and evaluations. We first clarify the technical novelty of our work (W2), then address dataset splitting and class imbalance (W1), the reliability of sensitivity and specificity estimates (W3), and false-positive rates in patients with malignant lesions (Q5).

W2 / Q2 - Technical novelty

We refer to our response to ZzWh W2 for the overall novelty of our work. Here, we clarify the technical rationale behind EIDER's design.

EIDER uses a modular architecture that augments a frozen lesion-only classifier, supports different lesion representations, and allows patient-comparison contributions to be analyzed independently (Fig. 1). This design is motivated by three theoretical results:

  1. Lemma 3.3 motivates separating lesion-only predictions from patient-comparison corrections, allowing the latter to be learned independently (Eq. 2).
  2. Proposition B.4 establishes the predictive sufficiency of a partially pooled reference and reference-set size under our hierarchical model, motivating a compact representation of patient context (Eq. 3).
  3. Proposition B.6 motivates using signed and squared deviations to compare each lesion with its patient reference (Eq. 4).

We will clarify these theoretical motivations in Section 3.

W1 / Q1, Q4 - Dataset splitting and class imbalance

We use five patient-disjoint folds, with separate training, validation, and calibration patients. For the longitudinal cohort, training uses only each patient's final visit. Your question helped us identify an inconsistency: the submitted Table 1 evaluated all visits. We have corrected this to use only final visits (8,884 observations, 458 patients, 111 malignancies from 57 patients), as detailed in Additional Corrections.

ISIC 2024 contains 393 malignancies (157 melanomas) from 259/1,042 patients, and SIIM 2020 contains 584 melanomas from 428/2,056 patients. All other lesions from the same patient serve as unlabeled references, restricted to the same visit for the longitudinal cohort.

To handle class imbalance, training retains all positives and samples up to 50 negatives per positive. Evaluation uses all eligible held-out lesions.

We will move some of the training procedure to the experimental section.

W3 / Q3: Experimental clarifications and sensitivity

As clarified in ZzWh Q3, we use five patient-disjoint folds without test-set subsampling.

Sensitivity and specificity are already reported in Table 7 (E.1), alongside AUROC, AP, and F1. For EIDER, they are:

DatasetSensitivity (%)Specificity (%)
ISIC 202486.6088.74
SIIM 202079.5179.26
Longitudinal92.1985.31

These metrics depend on the classification threshold, selected using Youden's J on calibration patients. With severe imbalance (only 0.1% malignant lesions on ISIC 2024), even moderate false-positive rates produce many false positives, explaining the low F1 (1.49%).

We therefore prioritize pAUC80 for ranking at high sensitivity, alongside PR15 for patient-level retrieval and AP for precision across recall levels. Unlike sensitivity and specificity, these metrics are threshold-independent.

We will clarify this rationale and highlight the additional metrics in the experimental section.

Q5 - Impact of malignant lesions in the reference on false positives

Following our analysis for s9u5 W5 on the impact of malignant lesions, we also compared false-positive rates (FPR) between patients with and without malignant lesions.

DatasetLesion branch FPR (%) without / with malignancyEIDER FPR (%) without / with malignancy
ISIC 202414.19 / 16.5011.51 / 10.86
SIIM 202022.62 / 39.3517.00 / 36.16
Longitudinal16.26 / 29.4412.87 / 27.39

On ISIC 2024, EIDER's FPR is slightly lower for patients with malignancy (10.86% vs. 11.51%). On the other datasets, FPR is higher for these patients under both models. Nevertheless, EIDER reduces FPR relative to the lesion branch in both groups, although the reduction is uncertain for patients with malignancy in the longitudinal cohort.

Thus, the higher FPR observed in some patients with malignancy cannot be attributed directly to malignant references. We will add this analysis to the Appendix.

Reviewer_ce61OpenReview ↗

Technical novelty and paper organization

Thanks for your responses. It's great to see that reviewers' comments helped you to correct your paper. Due to this reason, it is crucial to double check all your results throughout the paper. The technical novelty is still unclear and as stated, about the issues of key information is not in the main manuscript, I'll need to check your revised manuscript.

Reviewer_s9u5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper formalizes melanoma prediction using a target lesion together with an unlabeled reference set of the same patient's other lesions, building on the "ugly duckling" principle. It derives, via a Gaussian hierarchical model, a predictively sufficient patient reference representation, a partially pooled patient reference estimate plus reference-set size, and shows theoretically that both signed and squared deviations from this reference provide useful evidence for identifying malignancies. Based on this, the authors propose EIDER, which combines a frozen lesion-only branch with a learned reference-adjustment branch. Experiments on three dermatology datasets (ISIC 2024 TBP, SIIM-ISIC 2020 dermoscopic and a longitudinal cohort) and one non-dermatology dataset (EEG seizure detection) show consistent, though uneven, gains over lesion-only and intrapatient baselines, with controlled reference-substitution interventions supporting the claim that EIDER exploits patient-specific reference information.

Critical strengths
  1. Interesting combination of a theoretical derivation (Bayes decomposition + hierarchical/random-effects model) with a concrete, testable architecture (signed + squared deviations, partial pooling via κ\kappa).
  2. Strong empirical validation of the proposed solution. Specifically, the shrinkage-scale estimation matches the fixed κ\kappa and the predicted vs. observed reference-uncertainty curves align well. Insightful ablations are also provided, including single vs. multi-component reference and signed/squared deviation.
  3. Controlled interventions (reference substitution, shuffling, zeroing) indicate that the model depends on the specific patient reference and not on reference-set size or acquisition site.
  4. Generalization to EEG seizure detection is a nice test of external validity of the formulation beyond dermatology.
  5. Honest reporting of a negative/mixed result (longitudinal cohort, where EIDER barely beats its own lesion branch) rather than cherry-picking.
Critical weaknesses
  • The main gains appear almost exclusively on the ISIC 2024 total body photography dataset. For SIIM-ISIC 2020 the gains are marginal (from 13.0 to 13.4 pAUC80) and the longitudinal cohort does not benefit from EIDER in practice. The paper would benefit from a clearer discussion of when the method is expected to help rather than presenting it as a general-purpose gain.

  • Following on the previous comment, in the case of ISIC 2024 the patient branch uses the 33 tabular WB360 morphology features, unlike the dermoscopic and longitudinal datasets, where the patient branch uses visual embeddings (MONET). Hence, it is unclear whether EIDER's improvement on TBP stems from the intrapatient deviation mechanism or simply from multimodal fusion between image representations and rich morphological measurements. An ablation using image-only embeddings in the patient branch on ISIC 2024 would be needed to isolate the contribution of the reference mechanism.

  • Evaluation is limited to datasets drawn from essentially the same imaging setup as used for method development, and all three primary dermatology datasets provide fairly rich per-lesion and per-patient metadata or large reference sets. It would be important to test the method on additional public skin-lesion datasets with more varied metadata availability and reference-set sizes such as PAD-UFES-20, HIBA, and HAM10000, which have substantially smaller and more heterogeneous per-patient lesion counts and metadata completeness compared to ISIC 2024/SIIM-ISIC 2020. This would help clarify whether the reported gains are specific to total-body-photography-scale reference sets or also apply to smaller, noisier clinical datasets.

  • The comparison set for "intrapatient" methods may also consider additional multi-modal approaches such as TFormer [R1], SLIMP [R2] and AID [R3] that also exploit patient/lesion hierarchy as well as stronger general-purpose dermatology representations such as WhyLesionCLIP [R4] as frozen-feature baselines on the same datasets, to better contextualize how much of EIDER's gain is attributable to the reference-deviation mechanism specifically versus general representation quality.

  • The model assumes that reference lesions are benign. In practice, the reference being unlabeled may contain malignancies. It would be important to perform a sensitivity analysis to evaluate how malignant contamination of the reference set may bias downstream predictions.

  • On a related note, the target-lesion/reference-set split assumes a Gaussian, diagonal-covariance hierarchical model. This is a strong simplification, and while Appendix C tests some implications, there's limited discussion of how violations (e.g., multimodal per-patient lesion distributions) could affect real deployment.

  • Patient-level interpretability (Fig. 3) is shown for one representative patient (plus a few more in the appendix), which also corresponds to a critical case due to the increased number of melanomas. A more systematic, quantitative interpretability analysis across patients would strengthen the claims in Section 4.5.

References

[R1] Zhang, et al. "TFormer: A throughout fusion transformer for multi-modal skin lesion diagnosis". Computers in Biology and Medicine (2023).

[R2] Christopoulos, et al. "Skin lesion phenotyping via nested multi-modal contrastive learning." arXiv preprint arXiv:2505.23709 (2025).

[R3] Liu, et al. "Unlocking the Power of Medical Tabular Data via Semantic-Aware Multimodal Pre-training." MICCAI (2026).

[R4] Yang, et al. "A textbook remedy for domain shifts: Knowledge priors for medical image analysis". NeurIPS (2024).

Questions for authors
  1. What are the main factors leading to a substantial gain? Can the difference between the gains on ISIC 2024 compaer to the other datasets be attributed to the characteristics of the pratient-branch features?
  2. How does reference set contamination affect downstream performance?
  3. How well do the assumptions on the splits (Gaussian, diagonal-covariance) hold in the real data?
5 comments
AuthorsOpenReview ↗

Official Comment by Authors [1/2]

We thank the reviewer for recognizing our theoretical analysis, controlled interventions, and evaluation beyond dermatology. Their suggestions helped us strengthen our evaluation with additional datasets and baselines (W3, W4), as well as robustness and interpretability analyses (W5, W6, W7).

We first characterize when patient context helps (W1, W2). By isolating the contribution of patient comparison from lesion features (W2) and examining how gains depend on reference-set size (W1), which helps explain the strong improvements on ISIC 2024, smaller gains on SIIM 2020 and the longitudinal cohort, and uncertain benefits on HIBA and PAD-UFES-20 (W3).

W1: When is the method expected to help?

EIDER benefits most from large, informative reference sets. Figure 6a shows that gains increase with reference-set size, helping explain the larger improvements on ISIC 2024 (median 241 lesions/patient) than SIIM 2020 (median 12).

To isolate this effect, we added a sensitivity analysis on ISIC 2024. Keeping target lesions and the trained model fixed, we randomly subsampled reference sets and recomputed predictions.

Reference size0135102050100Full
pAUC8015.0515.2415.2415.4015.5815.7115.9315.9816.29

Performance increases with reference-set size, confirming the value of additional lesions. However, Figures 6b–c show that heterogeneous or distinctive references can reduce this benefit. Thus, both size and informativeness matter.

We will clarify these conditions in Section 4.3 and the Discussion, and examine smaller reference sets in W3.

W2 / Q1: What drives the gains on ISIC 2024?

Both lesion features and patient comparison contribute. As detailed in our response to ZzWh W1, adding WB360 improves pAUC80 by 0.73 points, while patient comparison adds another 1.28 points. Thus, multimodal fusion alone does not explain the gain.

Reference-set size also matters. Restricting ISIC 2024 to 10–20 references reduces the gain to 0.53–0.66 points, closer to SIIM 2020 (median 12 references). On HIBA and PAD-UFES-20 (medians 2 and 1), the benefit of patient comparison remains uncertain (W3).

We will include explanations in Section 4.3 and the Discussion.

W3: Including additional datasets

We initially focused on total-body photography because more classical datasets such as HAM10000 lack patient identifiers, preventing intra-patient comparison.

Nevertheless, we added HIBA and PAD-UFES-20, providing a challenging setting with median reference-set sizes of only 2 and 1:

  • HIBA: One image per patient-lesion pair, prioritizing dermoscopic, then close-up and overview images. We obtained 1,257 lesions from 628 patients (503 malignant), represented using frozen MONET embeddings.
  • PAD-UFES-20: One image per patient-lesion pair, yielding 1,891 lesions from 1,373 patients. We classified BCC, SCC, and melanoma as positive (833 cases) and used available metadata.

We report AP instead of PR15 because most patients have fewer than 15 lesions.

ModelHIBAPAD-UFES-20
pAUC80APpAUC80AP
Lesion-only
ResNet-189.3972.325.8864.18
LightGBM8.1069.8117.6093.74
PanDerm MLP15.8189.8611.3183.35
MONET MLP14.1883.5412.6786.09
WhyLesionCLIP14.9187.0211.9181.80
AID11.0974.268.5673.58
TFormer13.0281.8616.5989.32
Intra-patient
Image–LightGBM13.9682.9117.8794.65
Deep Sets14.0483.4613.0285.33
Set Transformer14.1282.6513.0984.60
DMT-Quadruplet3.6858.724.0857.92
EIDER ablations
EIDER (lesion branch)14.1685.9313.6887.32
Deep Sets reference14.7784.4216.9392.47
Set Transformer reference14.6982.7916.5991.24
No patient comparison14.3085.8417.4293.82
EIDER14.9184.3017.2193.15

On HIBA, EIDER slightly improves over its lesion branch and no-patient control, but the gains are not significant. PanDerm MLP performs best, suggesting that richer lesion embeddings may help. This could motivate using PanDerm embeddings instead of MONET in EIDER lesions representation, which we leave for future work. On PAD-UFES-20, EIDER improves over its lesion branch but not the no-patient control. Image–LightGBM performs best, suggesting lesion features matter more than patient comparison here.

Overall, these results show that the benefit of patient comparison with very small reference sets remains uncertain. We will include results in Appendix and discuss them in the main text.

AuthorsOpenReview ↗

Official Comment by Authors [2/2]

W4: Additional dermatology baselines

We added TFormer, AID, and WhyLesionCLIP, however, these are not intrapatient methods as suggested by the reviewer, as they classify individual lesions without comparing them to other lesions from the same patient. We adapted them as follows:

  • TFormer: We supplied the same image to both streams due to datasets limitations. We included age, sex, and anatomical site, froze Swin-Tiny, and trained the fusion and diagnosis layers.
  • WhyLesionCLIP: We followed the original linear-probing protocol, using frozen image features and logistic regression.
  • AID: We followed the original protocol and used frozen features with a linear classifier, including image, metadata, and cross-modal features on ISIC 2024, and image features alone elsewhere.
  • SLIMP: Its released checkpoints were unavailable, so we just included it in Related Work.

Results on our three original datasets follow (HIBA and PAD results are in W3).

MethodISIC 2024SIIM 2020Longitudinal
pAUC80PR15pAUC80PR15pAUC80PR15
TFormer11.3150.1910.6994.6313.6598.25
AID13.0168.737.6196.507.0794.74
WhyLesionCLIP9.0246.3311.3496.968.6989.47
EIDER16.2970.2712.9796.4215.92100.00

EIDER achieves the highest pAUC80 on all three datasets and the highest PR15 on two. AID and WhyLesionCLIP slightly outperform it on SIIM 2020 PR15. We note that WhyLesionCLIP was pretrained on SIIM 2020 and HIBA images, and AID on ISIC 2024, which may affect comparisons.

We will add these baselines to Table 1 and details to Appendix D.

W5: Do malignant lesions in the reference set corrupt prediction?

This is a good observation, however, we would like to note that dermatologists also rely on a similar assumption when applying the ugly-duckling principle, as patients generally have many more benign lesions than malignant ones.

Still, malignant lesions in the reference set could influence predictions. To test this, we removed malignant reference lesions and compared performance with removing the same number of randomly selected benign lesions per patient to control for set size. We kept the model fixed and repeated benign sampling three times. We found no clear change in pAUC80 across the five datasets (all 95% CIs include zero).

We will add this analysis to Appendix E.

W6: Gaussian and diagonal covariance assumptions

Indeed, Gaussian assumption simplifies patient lesion distributions, which is known to be multimodal with multiple recurring benign patterns (e.g. "signature nevi"). Our diagnostics also reveal multimodality, especially with larger reference sets.

However, Appendix C.3 compares single and multiple partially pooled references and shows that more components consistently reduce pAUC80 (Fig. 5b). One explanation is that distribution modes may not correspond to malignancy-relevant differences, while estimating more references from fewer lesions increases uncertainty. Thus, a single reference can remain a better predictive summary.

We assumed diagonal covariance to simplify notation and interpretation, but it is not required for our theoretical results. It is used only in the final step of Proposition B.3 to simplify U^n\hat U_n. All results hold for general positive-definite covariance matrices using:

(T^{-1}\mu+n\Sigma_c^{-1}\bar Z).$$ We also observed empirically that only **13.3–13.5\% of feature pairs** on ISIC 2024 have absolute correlation above 0.5, suggesting strong correlations are relatively uncommon. ## W7: Quantitative interpretability analysis We agree that a systematic evaluation across patients would strengthen our interpretability claims. We therefore conducted two additional analyses on ISIC 2024, complementing Fig. 3: 1. Larger deviations from the patient reference **correlate with EIDER adjustments** (Spearman $\rho = 0.384$). This extends Fig. 3a across patients and confirms that deviation magnitude alone does not determine the adjustment. 2. Relative to the zero-reference ablation, patient context decreases scores for 65.55\% of **lesion-only false positives**. This suggests that patient context can **help reduce false alarms**, although these decreases do not necessarily correct the classifications. We will include these analyses in the appendix. We will also discuss future directions involving **expert evaluation** to assess clinical relevance: comparing EIDER's adjustments with dermatologists' judgments of unusual lesions and evaluating whether corrections help experts understand changes in predictions.

I thank the authors for their detailed reply and for their transparency. My main concerns have been addressed, I have no additional questions. The method is well-motivated and theoretically well-grounded. At the same time, the updated results indicate that the gains over strong baselines are marginal.

Reviewer_ZzWhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims to model the “ugly-duckling principle” in melanoma classification (using patient level information to aid image level classification). The authors propose a hierarchical method of utilizing unlabeled same-patient lesions to estimate what a typical lesion might look like and how a target lesion is deviating from it. This is utilized to modify lesion-level classification. The proposed method has strong performance on ISIC 2024, marginal increase on ISIC 2020, but no increase on Longitudinal results. The authors also evaluate generalization to EEG seizure detection.

Critical strengths

The proposed method performs strongly on ISIC 2024 dataset, with PR15 going 64.5 to 75.7 and pAUC80 going from 14.3 to 16.5 over the strongest baseline. Given that ISIC 2024 is the dataset with highest median lesions per patient, the performance gain here makes it worth considering their proposed method.

Critical weaknesses

In ISIC 2024, the authors incorporate additional tabular features (WB360), which can potentially improve performance without the proposed novel modeling. So it is important to isolate the gains from the tabular data over the gains that came from the proposed patient-level processing. This is particularly relevant for establishing the importance of this method since the gains are less prominent in ISIC 2020 where there are no tabular features (they used image embeddings for patient level features instead). It is important to establish that patient-level processing improves even beyond tabular data processing in the same-architecture.

Existing methods (Soenksen 2021, Yu 2021) already utilize ugly-duckling modeling in melanoma classification, so it is important to isolate what the proposed architectural changes contribute. It is unclear whether Deep Sets and Set Transformer use the same ConvNextV2 image branch, so it is important to show the performance of these methods with that image branch for fair comparison.

Questions for authors

Experiments:

  1. Can the authors demonstrate how the same image branch + same MLP perform on ISIC 2024 with only the target lesion’s tabular features (without any patient level comparison)?
  2. Can the authors confirm that the Deep Sets and Set Transformer use the ConvNextV2 classifier too? If they do not use it, can the authors show how they perform when attached to the same frozen image branch?

Clarifications:

  1. Can the authors clarify whether in the data split, the validation patients are separate from test patients in each fold?
  2. Is the image branch score (the lesion-level logits used to train the patient branch) taken from the patients that image branch was not trained on?
1 comment

We thank the reviewer for their careful assessment and for recognizing the improvements on ISIC 2024. Their suggestions helped us strengthen our evaluation (W1, W3). We first address the novelty concern (W2) to clarify the paper's broader contributions.

W2: Distinction from existing ugly-duckling methods

Previous work has modeled the ugly-duckling principle. Our contributions extend beyond its application to melanoma:

  1. We formulate a general learning problem using unlabeled observations from the same individual (Tab. 2), and derive a prediction decomposition (Eq. 2) that provides a common framework for comparing existing methods (Tab. 3).
  2. Assuming a hierarchical structure in the data, we derive EIDER, which goes beyond prior work by preserving deviation direction rather than using scalar atypicality scores as in Soenksen et al., and by separating patient-context contributions from lesion-only predictions, unlike Yu et al.
  3. We demonstrate competitive performance across dermatology datasets (Tab. 1) and applicability beyond melanoma on EEG seizure detection (Fig. 4).
  4. We validate our modeling choices through ablations (Tab. 1; W1), reference interventions (Fig. 2), theoretical validation (Fig. 5), and sensitivity analyses across reference-set characteristics (Fig. 6). These show the benefit of patient comparison, especially for high reference set sizes (s9u5 W1), and the effectiveness of our reference representation compared with matched set models (W3).

We will clarify these contributions in the Introduction and Related Work.

W1 / Q1: Isolating patient-level comparison

Indeed, our original ablation did not separate gains from WB360 features and patient comparison. We therefore trained two additional ablations on ISIC 2024, keeping the image branch frozen and retraining the patient MLP:

  1. No patient comparison: The MLP receives only target WB360 features, without the patient reference U^n\hat U_n or deviations rr.
  2. No WB360: As requested by s9u5, we replace WB360 with frozen MONET embeddings for target and reference lesions to align with our other datasets.
ModelpAUC80PR15AUROCSens.Spec.F1
Lesion branch (image only)14.2867.1892.4084.9984.911.09
No patient comparison (WB360 only)15.0168.3493.3184.8287.401.31
No WB360 (MONET comparison)15.1567.8293.5684.9086.661.23
EIDER (WB360 comparison)16.2970.2795.1286.6088.741.49

Adding WB360 improves pAUC80 by 0.73, while patient comparison provides a further 1.28 (CI [0.70, 1.93]). This confirms that multimodal fusion alone does not explain the gain.

We will add these ablations to Table 1 and implementation details to Appendix D.

W3 / Q2: Augmenting set baselines with the EIDER lesion branch

Our original comparison excluded the image branch to assess prediction from lesion relationships alone. However, combining these models with the image branch raises another interesting question: Does EIDER's patient reference better summarize patient information than learned set representations?

We therefore trained Deep Sets and Set Transformer to produce patient references instead of label predictions. We retained EIDER's frozen image predictor and patient MLP, replacing the reference and deviation features (U^n\hat U_n, rr, r2r^2) with learned set representations. Both encoders receive the patient's lesions and are trained jointly with the MLP.

Patient referenceISIC 2024SIIM 2020Longitudinal
pAUC80PR15pAUC80PR15pAUC80PR15
Deep Sets15.5672.7212.4795.9514.0299.42
Set Transformer14.4969.2412.3295.9514.2898.83
Eq. 3 (ours)16.2970.2712.9796.4215.92100.00

EIDER's reference achieves higher pAUC80 than both alternatives on all three datasets, although CIs overlap. Differences in PR15 are smaller and mixed.

We will add these new baselines to the main text and detail their implementation in the Appendix.

Q3: Clarifications on data splits

We confirm that training, validation, and test patients are disjoint within each fold. The image branch is trained only on training patients and frozen before training the patient branch.

Q4: Clarifications on in-sample logits

Indeed, the patient branch originally used logits from an image model fitted on the same patients. As we were rerunning EIDER's evaluation, we took the opportunity to address this issue. We now split training patients into two groups, train an image model on each, and generate logits for each group using the model trained on the other. The patient branch is then trained on these out-of-group logits.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Additional Corrections

While preparing our rebuttal, we identified and corrected some minor issues in the submission. We report them here for transparency, and will submit a revised version of the paper.

Longitudinal results

The submitted Table 1 evaluated multiple visits per patient, although our protocol focuses on single-examination comparisons. We corrected the evaluation to use only each patient's final visit and same-visit references, yielding 8,884 observations (111 malignant) from 458 patients.

The corrected results are:

MethodpAUC80PR15AUROCSens.Spec.F1
Lesion-only
ResNet-1815.39100.0092.5886.4985.6013.06
PanDerm MLP15.01100.0091.9883.7882.2310.62
MONET MLP15.23100.0091.9483.1884.0911.57
EIDER (lesion branch)14.74100.0091.2190.9982.0811.32
Intrapatient baselines
Image–LightGBM10.9796.4982.5562.1683.028.26
Deep Sets reference14.0299.4290.7687.9981.5110.68
Set Transformer reference14.2898.8391.0492.1979.9610.43
DMT-Quadruplet3.8291.2369.3645.9589.569.46
EIDER15.92100.0092.7992.1985.3113.63

EIDER now achieves the highest pAUC80 (15.92) and AUROC (92.79), outperforming ResNet-18 (15.39 and 92.58), with higher sensitivity (92.19 vs. 86.49) at similar specificity.

We will update Table 1 and its interpretation accordingly.

EIDER implementation

While preparing the rebuttal, we found that the submitted EIDER results were generated using an earlier implementation rather than the version described in the manuscript. We have rerun the evaluations using the described implementation, without changing the proposed method.

The corrected EIDER results are marginally different from previous results:

DatasetpAUC80PR15AUROCSens.Spec.F1
ISIC 202416.2970.2795.1286.6088.741.49
SIIM 202012.9796.4288.2379.5179.2611.94

EIDER still achieves the highest pAUC80 on ISIC 2024, while slightly trailing ConvNeXt V2 on SIIM 2020 (12.97 vs. 13.10). We will update the results, figures, and discussion throughout the manuscript.

Bibliographic corrections

We corrected the author lists of Lawhern et al. 2018 and Haenssle et al. 2018, and the article number of Ghahari et al. 2025. These bibliographic corrections do not affect the findings.

AuthorsOpenReview ↗

Revised manuscript and summary of changes

Dear Reviewers,

Thank you again for your careful reviews and constructive feedback. We have revised the manuscript to incorporate the additional experiments and clarifications discussed in our rebuttal.

The main changes are:

  1. Expanded evaluation: We added TFormer, AID, and WhyLesionCLIP baselines, along with two additional dermatology datasets (HIBA and PAD-UFES-20) to evaluate settings with few reference lesions.
  2. Additional ablations and analyses: We isolated the contribution of patient comparison from target features, compared our patient reference with learned Deep Sets and Set Transformer representations, and added controlled experiments on reference-set size, reference identity, and malignant reference lesions (Table 1, Fig. 2, App. E).
  3. Methodological corrections and clarifications: We updated the EIDER implementation and longitudinal evaluation, clarified the training and evaluation protocol, and expanded the discussion of the theoretical assumptions and model interpretability (Sec. 3 and 4, App. C, F).

We hope the revised manuscript addresses the main concerns raised during the discussion and makes our contributions and experimental findings clearer.

Although little time remains in the discussion period, we would be happy to answer any further questions or concerns before it closes.

Thank you again for your time and feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.