- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a new method that classifies pulmonary nodules malignancy on a binary basis (malignant or benign). For training the authors use input that is labeled both on malignancy and that contains several attribution labels. These attribution labels are generally considered helpful for classification. In the training setting a teacher network is trained using these helpful attributes and the CT scans and a student network is trained to classify only based on the CT scans. Information from the teacher network is distilled to the student network based using several methods such as a distillation loss. The contribution is a new method for CT pulmonary nodule classification using multimodal-to-unimodal hierarchical knowledge distillation and graph neural networks. Validation is done using two different datasets. Both datasets are evaluated using five-fold cross-validation and external validation is being done by training the network on one dataset and evaluating it on the other one.
Critical strengths
Valuable addition to the landscape: The paper is novel in its application rather than its methodology: it brings knowledge distillation to a domain that was previously dominated by CLIP-style models. Thus, it fills a valuable gap in this domain.
Critical weaknesses
-
No clear methodological contribution: For a venue like ICLR I would expect a clear methodological contribution. This contribution in itself is quite sound, but the contribution is rather by using existing methods on a new domain than by providing a new method of tackling a problem. All methods of this contribution are sound, but none seems to be new. Sometimes this is stated in the paper and sometimes it is not. I would also expect a contribution that goes beyond one specific domain and application case. I would suggest to rather go for a medical venue like e.g. MICCAI. For a methodological contribution I would rather expect a comparison to knowledge-distillation methods in the same domain, to see whether this methods outperforms other knowledge-distillation methods.
-
Some results are non-comparable: a) Some of the results were copied from other papers and not replicated. This, by itself is not an issue, but the other papers partly used different evaluation and exclusion criteria e.g. results from Nodule-Clip were copied. However, Nodule-Clip had a more difficult learning and evaluation setting, since they did not exclude any nodules (they classified all scores below and equal to 2.5 as benign and above 2.5 as malignant). This paper has a different setting and excludes the indeterminate nodules (mean score exactly 3), which are likely among the hardest to classify. This makes the comparison between these methods incomparable. I just checked this for Nodule-Clip, but this might as well be the case for other methods (notably it is not for CLIP-Lung, which seems to be replicated). I understand, that it is not always possible to replicate all the findings and it is even stated, that some values were copied (not stated which ones though). But this comparison makes it seem as if MUGAT-KD outperforms Nodule-CLIP on every metric, while the honest comparison would be, that they are non-comparable. An ablation using different thresholds and including all datapoints might solve this issue and provide some fair comparison.
b) This is a weaker issue than above, but also comparing MUGAT-KD with 2D and 3D CNN based method is without further comment not sound. This comparison is interesting, however these methods were trained without any attributes and thus with a different training setting. I see the usefulness of them as a type of "lower bound", but the setting is fundamentally different. I agree with putting them there, but that should also be explicitly stated. In the same breath, the teacher (Tab. 3) uses the attributes at inference and thus acts as an "upper bound", but it is not compared to any other method that uses the attributes at inference (e.g., the ones discussed in Sec. 2). It would be interesting to see how the teacher performs against them.
To conclude: I think for now the method is truly only comparable with CLIP-Lung among the listed comparisons. This is however nowhere explicitly stated.
Questions for authors
There are no comparable methods for the cross-dataset evaluation method. Since CLIP-Lung is already implemented, could you do a comparison at least with this method to see how it performance compares. I would also be interested in how inputs are encoded. This is at least for the attributes never listed. How were the teacher checkpoint and the epoch thresholds of Eq. 16 chosen, given that the five folds are reported as test folds? And how many nodules (benign/malignant) remain in LIDC-IDRI after excluding score-3 nodules?
Here just some minor issues (no effect on score), which you could improve: The paper has several spelling errors and some formatting errors(these are just the ones I found maybe use a LLM to parse for all of them) in Fig. 2(b), "Maskd CT Slice" → "Masked CT Slice" and "Masked Central Node Neature" → "Feature"; in Fig. 4, "Probabilitles" → "Probabilities"; in Sec. 4.1, "benign andmalignant,respectively" is missing spaces, and "Exclusion with a score equals 3" is ungrammatical (e.g., "Nodules with an average score of exactly 3 are excluded"); in the Conclusion (l. 480–481), the line break after "feature-" leaves the comma at the start of the next line (", slice-, and region-level").
TO Critical Weakness 1 : Our contribution is a generalizable multimodal-to-unimodal distillation framework for classification tasks involving 3D images and structured data, rather than a method limited to medical imaging. We represent a 3D image as a sequence of slices and construct a multimodal teacher with a slice-attribute graph that models relationships between image slices and structured attributes. An image-only student then learns through three complementary objectives: global representation alignment, slice-level attention alignment, and saliency-guided masked distillation. Unlike CLIP-Lung, which mainly aligns global image and attribute representations, our framework transfers knowledge at the representation, slice-attention, and region-conditioned levels. It assumes that structured attributes have observable manifestations in the images and is therefore potentially applicable to other 3D imaging tasks with complementary structured or multimodal information.To establish a new paradigm for applications in computer vision, audio, language, and other modalities. TO Critical Weakness 2.a: We have now marked results reported in the literature with asterisks (*). Table 13 in the supplementary material summarizes the dataset sizes, class distributions, and label definitions of all methods marked with an asterisk. The remaining methods in Table 1 and all methods in Table 2 were reproduced using the same patient-level data split. We have also clarified that LIDC-IDRI labels are based on radiologists’ malignancy ratings, whereas LIDP uses pathology-confirmed diagnostic labels. TO Critical Weakness 2.b: The training information available to MUGAT-KD differs from that available to conventional 2D/3D CNN methods. These CNN methods use only CT images during both training and inference, whereas MUGAT-KD leverages structured attributes as privileged information during training while requiring only CT images at inference. We retain these methods to provide reference performance under image-only training conditions and to assess the benefits of incorporating attribute knowledge during training. As the reviewer noted, these methods serve as a performance lower bound. We will explicitly clarify the scope and interpretation of these comparisons in the revised manuscript. Furthermore, in Table 12, we provide a comparison between our teacher model and other multimodal methods, serving as a performance upper-bound reference. TO Question:
We thank the reviewer for pointing out these spelling and formatting issues. We have carefully checked and corrected each of the identified errors and conducted a further review of the language and formatting throughout the manuscript to improve its accuracy and readability.