- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on melanoma prediction, particularly interested in ugly-duckling principle, which is a pattern that associate skin lesions within a patient. The authors use a hierachical model (patient as reference) to derives the variations of unlabeled lesions, and named their contribution as Evidence In Deviations from an Estimated Reference (EIDER). to learns lesion-to-reference deviation and adjust the lesion-level predictions.
The method is evaluated on total-body photograph (TBP), dermoscopy and EEG seizure detection. For TBP, the authors compared it with image-only baseline and demonstrated improved AUROC of 0.038. Further analysis involved changing patient references and the size of reference sets.
Critical strengths
Formulate hierarchical model for patient as reference and a method named Evidence In Deviations from an Estimated Reference (EIDER). Results evaluated on different datasets, imaging and EEG, and demonstrated generalisability of the proposed method.
Critical weaknesses
The split of datasets and issues with the imbalanced datasets are not clearly stated in this paper. It is difficult to follow as many of the details are in the appendices. Key information should be in the main manuscript. Difficulties in understanding the novelty of the method. There are no clear descriptions on technical novelty of the work. The evaluation requires more information on what are the test datasets and what is the sensitivity and specificity. More metrics will help to understand the merit of the paper.
Questions for authors
The task is for melanoma prediction. However, the datasets with only 393 melanomas (ISIC 2024), 584 melanomas (SIIM-ISIC 2020), do every patients with a melanoma to be used as reference and how many of the patients with melanoma? What is the technical novelty of this paper? As the datasets are very imbalanced, how reliable is the results on sensitivity and specificity? For the longitudinal cohort of 458 patients, how did you split them in the experiment? From your analysis, would your method produce more false positives when your references are patients with melanoma?
2 comments
Technical novelty and paper organization
Thanks for your responses. It's great to see that reviewers' comments helped you to correct your paper. Due to this reason, it is crucial to double check all your results throughout the paper. The technical novelty is still unclear and as stated, about the issues of key information is not in the main manuscript, I'll need to check your revised manuscript.
We thank the reviewer for their feedback and appreciation of our theoretical framework and evaluations. We first clarify the technical novelty of our work (W2), then address dataset splitting and class imbalance (W1), the reliability of sensitivity and specificity estimates (W3), and false-positive rates in patients with malignant lesions (Q5).
W2 / Q2 - Technical novelty
We refer to our response to ZzWh W2 for the overall novelty of our work. Here, we clarify the technical rationale behind EIDER's design.
EIDER uses a modular architecture that augments a frozen lesion-only classifier, supports different lesion representations, and allows patient-comparison contributions to be analyzed independently (Fig. 1). This design is motivated by three theoretical results:
We will clarify these theoretical motivations in Section 3.
W1 / Q1, Q4 - Dataset splitting and class imbalance
We use five patient-disjoint folds, with separate training, validation, and calibration patients. For the longitudinal cohort, training uses only each patient's final visit. Your question helped us identify an inconsistency: the submitted Table 1 evaluated all visits. We have corrected this to use only final visits (8,884 observations, 458 patients, 111 malignancies from 57 patients), as detailed in Additional Corrections.
ISIC 2024 contains 393 malignancies (157 melanomas) from 259/1,042 patients, and SIIM 2020 contains 584 melanomas from 428/2,056 patients. All other lesions from the same patient serve as unlabeled references, restricted to the same visit for the longitudinal cohort.
To handle class imbalance, training retains all positives and samples up to 50 negatives per positive. Evaluation uses all eligible held-out lesions.
We will move some of the training procedure to the experimental section.
W3 / Q3: Experimental clarifications and sensitivity
As clarified in ZzWh Q3, we use five patient-disjoint folds without test-set subsampling.
Sensitivity and specificity are already reported in Table 7 (E.1), alongside AUROC, AP, and F1. For EIDER, they are:
These metrics depend on the classification threshold, selected using Youden's J on calibration patients. With severe imbalance (only 0.1% malignant lesions on ISIC 2024), even moderate false-positive rates produce many false positives, explaining the low F1 (1.49%).
We therefore prioritize pAUC80 for ranking at high sensitivity, alongside PR15 for patient-level retrieval and AP for precision across recall levels. Unlike sensitivity and specificity, these metrics are threshold-independent.
We will clarify this rationale and highlight the additional metrics in the experimental section.
Q5 - Impact of malignant lesions in the reference on false positives
Following our analysis for s9u5 W5 on the impact of malignant lesions, we also compared false-positive rates (FPR) between patients with and without malignant lesions.
On ISIC 2024, EIDER's FPR is slightly lower for patients with malignancy (10.86% vs. 11.51%). On the other datasets, FPR is higher for these patients under both models. Nevertheless, EIDER reduces FPR relative to the lesion branch in both groups, although the reduction is uncertain for patients with malignancy in the longitudinal cohort.
Thus, the higher FPR observed in some patients with malignancy cannot be attributed directly to malignant references. We will add this analysis to the Appendix.