- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper identifies fixed regions, patches, or slices as the bottleneck and proposes ASNet, which iteratively identifies subject-specific supervoxels that can localize the abnormalities better on PET scans using the ASP module. Then, it proposes an AEC module that can identify anomalous supervoxels using a reference to perform classification (AD vs CN and MCI vs CN) tasks on the ADNI and AIBL datasets. ASNet shows a good performance across the datasets and tasks it was evaluated for, although the dataset sizes were small.
Critical strengths
The paper proposes a supervoxel-based iterative partitioning technique on PET scans with task-specific adaptation. ASNet shows a good performance across 2 datasets and tasks, although the dataset sizes were small. However, I am currently unable to consider this a substantive strength for the reasons mentioned in the weaknesses and questions.
Critical weaknesses
1. Regarding the normative reference considered:
- A normative reference in clinical neuroimaging is generally conditioned on age, which is not considered in this work. According to ASNet, the normative reference is obtained from all the CN subjects across all ages, which, to my knowledge, should not be called normative.
- This also means that AEC characterizes the supervoxels using a common reference, which could be confounded by age, acquisition-related variations, etc. What if the CN cohort differs, or say new CN subjects from different cohorts get added for further training, or new test data from different scanners or age ranges are tested? I suspect the evaluations would change. There is no analysis of the framework's stability.
- Also, with differences in brain volumes/structures across ages, and since the preprocessing performs only affine registration, there is no true correspondence across scans at the voxel level.
2. Potential overfiting: There is no mention of val split in the protocol, only 5-fold cross-val (meaning only train and test). How was the best ckpt chosen? This raises doubts of potential overfitting. Also, ADNI and AIBL provide longitudinal scans per subject, there is no mention of the train-test splitting strategy.
3. Missing details and experiments in the paper:
- Missing training loss details. Sec 3.3 states that ASNet was trained in 3 stages, and mentions the use of distillation loss in stage 2, but the details of this loss is missing. What is getting distilled?
- Missing architectural details, like the paper just mentions 3D convolution stem, MHSA, etc,. Details of the design are missing. There is no code to verify as well.
- Missing generalizability test for cross-dataset.
- Missing baselines that used supervoxels. SVGFormer, AdaSLIC, etc. are works that have used supervoxel for 3d medical volumes. The authors could consider either using their supervoxel partitioning approach or training these encoders for the classification tasks, or comparing the ASP with other established supervoxel-generation methods.
- Missing ablations like using ASP’s K-means clustering fixed without the iterative update, random voxel selection or spatial groups without k-means, without abnormality gating, without reference, etc. These are necessary to understand the importance of the proposed framework.
- No confounder analysis was performed, which is essential for understanding its effect on partitioning and, in turn, performance.
4. Regarding the claims:
- Superiority over use of fixed atlas: AAGN uses HOA atlas in its framework and performs 95.19% AUC and 89.03% ACC. The proposed ASNet ablation of the same HOA atlas replacing ASP shows 85.86%AUC and 79.14%ACC. This shows quite a contrast of two things: a) its either the framework of ASNet does not support/handle atlas based inputs or the later part of the framework, AEC is overly fragile, b) The claim of superiority over atlas based grouping by 10% is misleading as the difference in AAGN vs ASNet is only ~1% in both AUC and ACC, thus, table 2 ablation cannot be considered for evaluating the atlas ablations.
- Interpretable design by construction: the scores are not truly region-specific scores but rather coupled by the global features and hence have an influence from the global context. Also, the paper does not establish that the identified regions are truly anomalous, and hence, it's difficult to assess.
Questions for authors
-
Regarding the motivation, if the abnormality lies on two parcels, we can consider both parcels to show anomalous behavior. If a well-trained model can identify both parcels as abnormal, why would that be considered unreliable? For example, say a scan has abnormality in hippocampus, at a later stage it spreads to the nearby parahippocampal region, then the model can flag that the hippocampus and parahippocampus shows abnormality, so what’s wrong here?
-
How is the whole-brain context vector derived from the voxel features? What is the motivation to use FiLM? Because it is derived from the voxel features, it’ll just globally shift all the supervoxel representations in the same direction channel-wise. Another question here, if a scan is classified as say abnormal, then its not just certain supervoxels which are responsible, but also the global context, right? Then, how do I know what exactly was responsible? In other words, since each regional score depends on whole-brain context, its value need not originate exclusively from evidence inside the displayed region. How do the authors distinguish exact additive accounting from faithful spatial localization?
-
In fig 4, I understand that the interpretations are different, but how do I read what is expected? Which methods failed? How do you claim that ASNet performs better? For example, even if say MAD-Former produces block-wise responses, are they pointing to the right region of abnormality? Also, the conclusion in Sec 4.4 is not entirely right. Similar to ASNet, AAGN also assigns a contribution to a parcel, which can be used to directly interpret its contribution for a classification. So, what is the additional interpretability property introduced by AEC.
-
This also leads to my next question, do you have any ground truth to validate the regions identified to be abnormal? Or, in other words, how should I validate your qualitative images? So, I am not convinced by the evaluation of the partitions. The performance on tasks alone cannot support the partitioning technique.
-
While the ASP uses a 3D conv stem, which I am assuming to be some k layers of CNN, ideally the output would not truly represent the exact voxel’s feature due to the design of kernel overlaps. Then is this assumption still valid? This matters because your assigned boundaries are wrt these voxel features, so we cannot assume that the boundaries are truly aligned to the corresponding voxels in the image space.
-
When atlas was used, was it directly overlayed and ablated? Because mostly the atlases are designed for MNI152 1mm space. So, if it's different, then the overlaid regions will be misaligned, which could be the reason for poor performance rather than the strict atlas partitioning.
5 comments
Response to Reviewer kJB5 (Part 2/2)
[Part 2/2]
Experiments and other details
Weakness 2. There is a validation split: within each of the five folds, the data are split 3:1:1 into training, validation and test sets, and the best checkpoint is chosen by validation AUC; the test set is never used for training or checkpoint selection. For both ADNI and AIBL, only the baseline scan of each subject is used, and all splits are made at the subject level.
Weakness 3. What is distilled is the class posterior of the auxiliary classifier G-Cls: G-Cls is trained with ASP in stage 1 and then frozen as the teacher, and AEC is trained with a standard KL-divergence distillation loss to match its temperature-softened predictions, in addition to the cross-entropy loss.
The architecture is as follows:
- Input and stem. Input 121×145×121. The stem has four 3×3×3 convolutions (stride 1, no downsampling) with InstanceNorm and GELU, and produces 16-channel voxel features.
- ASP. 32-d supervoxel tokens; and ; candidate set = the 10 spatially nearest centres; 4-head self-attention over the supervoxel tokens.
- AEC. 16-d whole-brain context vector; abnormal and preserved experts are residual MLPs with LayerNorm, and the abnormal expert ends with softplus. A supervoxel is routed to the abnormal expert when its mean deviation exceeds CN standard deviation.
Cross-dataset generalization is not a claim of the paper. Because PET intensities differ across tracers and acquisition protocols, ASNet is trained and evaluated within each cohort and tracer, where it achieves consistent gains on ADNI (FDG, AV45, AV1451) and AIBL.
The SVGFormer and AdaSLIC partitions and the K-Means clustering of ASP without the iterative update perform as follows in the same pipeline (, FDG, AD vs. CN):
| Partition () | AUC | ACC |
|---|---|---|
| Fixed K-Means (ASP initialization, no iterative update) | 93.93±2.82 | 86.45±3.94 |
| SVGFormer partition | 94.55±2.56 | 86.24±3.93 |
| AdaSLIC partition | 90.75±5.70 | 82.58±7.21 |
| ASNet (learned) | 96.20±2.53 | 90.54±2.09 |
The learned partition of ASP outperforms all of them. SVGFormer and AdaSLIC group voxels by image intensity, and Fixed K-Means by coordinates, all before training; ASP's assignment is learned with the classifier and updated with region context, so the regions are shaped by diagnostically relevant features.
Without the gate and the reference, AEC reduces to an unconstrained classifier on the same supervoxel representation, i.e. the G-Cls head, which Table 2 reports on the same ASP backbone: 94.14 AUC / 87.53 ACC, against 96.07 / 89.68 for AEC. Removing the monotonic prior tied to the reference, by letting the abnormal expert produce evidence for both classes, also lowers performance (FDG, AD vs. CN):
| Abnormal expert | AUC | ACC |
|---|---|---|
| Evidence for CN and AD (no monotonic prior) | 95.65±2.78 | 88.82±3.44 |
| Evidence for AD only (ASNet) | 96.20±2.53 | 90.54±2.09 |
Q3 and Q4. ADNI and AIBL contain no voxel-level ground truth for PET abnormality, and none of the compared interpretable methods is evaluated against one. Explanations are therefore judged by their agreement with the image and with known AD patterns: a good explanation follows the uptake pattern of the subject's own image, covers the abnormal tissue without extending over preserved tissue, and emphasizes the most affected regions. In Fig. 4:
- 3DSC-TF and MAD-Former respond in blocks that follow the patch grid; they point to the abnormality only roughly and do not provide a comprehensive explanation across the brain.
- LA-GMF and LEAR highlight large areas, so the most affected regions do not stand out.
- AAGN and PRGNN. Their explanation values change abruptly at parcel boundaries, which is inconsistent with the smooth disease patterns in brain images.
- ASNet assigns its strongest AD evidence to the area of visibly reduced uptake in the upper part of the sagittal slice, with boundaries that follow the reduced-uptake pattern, consistent with the cortical hypometabolism typical of AD on FDG PET (Minoshima et al., 1995; Mosconi, 2005).
References.
-
Minoshima S. et al., A diagnostic approach in Alzheimer's disease using three-dimensional stereotactic surface projections of fluorine-18-FDG PET, J Nucl Med 36:1238–1248, 1995.
-
Mosconi L., Brain glucose metabolism in the early and specific diagnosis of Alzheimer's disease, Eur J Nucl Med Mol Imaging 32:486–510, 2005.
Thanks for the response. I have further questions:
Weakness 1:
- The effect of confounding by age is not addressed. Even if the method does not consider age as input, the PET scans are confounded by age. So this is essential to understand its effect as you are considering CN reference irrespective of age.
- Regarding correspondence, even if voxel correspondence is not required, since brains across ages or demographics can vary in sizes, there could be regional variations if the supervoxel identified is small. The authors also say that the boundaries need to fit the regions of abnormality in more detail. Is there any analysis conducted on the regions obtained and its correspondence to the reference?
Weakness 4:
- If the 10% gap is with respect to ASNet itself and not wrt to atlas-based methods in general, then I feel you should clarify this in the paper, as it gives an impression that ASNet is better than atlas-based methods…
Q2: Could you provide ablations with and without this global feature?
Q5: Even if the grid size is the same, the receptive field varies with kernel size, overlapping kernels, and depth of the conv stem. So, feature location in the output does not truly correspond to the voxel in the input due to its local receptive field. The 4-layer stem creates a receptive field, meaning each "voxel" feature already blends a wide neighborhood before supervoxel assignment. So, there is no reliable test to check how good the boundaries identified by the supervoxels are, as you claim that boundaries matter.
Weakness 2: Is it 3:1:1 split within each of the 5 folds? Please clarify the exact protocol in the paper.
Weakness 3:
- It is suggested to present your final loss function used rather than as verbose as in the paper. Is there any ablation of this distillation loss?
- The confounder analysis is still not addressed, which includes effects of manufacturer, demographics, etc. and its effects on partitioning.
Response to Reviewer kJB5 (Part 1/2)
[Part 1/2]
Main concerns
Weakness 1 (also the confounder analysis in Weakness 3). The CN reference in ASNet is our own operational definition and differs from a normative reference in clinical neuroimaging: it provides a weak prior on the PET image based on the CN subjects of the training split, which only decides whether a supervoxel is evaluated by the abnormal or the preserved expert, and does not by itself determine the diagnosis.
ASNet deliberately does not condition this reference on age, because there is no established age-specific uptake level that defines AD, MCI or CN. Like all compared methods, ASNet is trained on cohorts of mixed age without explicit age modelling. Regarding acquisition, PET values have different meanings for different tracers, so for an accurate interpretation each experiment is restricted to PET acquired with the same class of tracer.
ASNet does not require exact voxel correspondence across scans. Its comparison with the CN reference is made at the level of regions rather than individual voxels: each supervoxel is judged by its average deviation over all of its voxels, about 4,400 on average, so what matters is the overall uptake of a region, not exact voxel-to-voxel differences.
Weakness 4 (also Q6). The 85.86% AUC / 79.14% ACC is ASNet itself with ASP replaced by the fixed HOA partition; the 95.19% / 89.03% is AAGN, a different method designed end to end around atlas ROIs. The two numbers come from different models, so the contrast shows neither that ASNet cannot handle atlases nor that AEC is fragile: with only the partition fixed (Fixed K-Means), ASNet still reaches 93.93 AUC. The 10-point gap refers only to the atlas ablation within ASNet, not to atlas-based methods. We learn the partition instead of using fixed partitions such as atlases because such units are fixed in advance, so AD-related PET abnormalities, whose location and extent vary across individuals, may be fragmented by them or misaligned with them. ASP instead learns the partition dynamically for each subject under diagnostic supervision, so it can adapt to the different abnormality patterns of different subjects. To our knowledge, no existing PET-based AD diagnosis method learns such a subject-specific partition under diagnostic supervision.
The atlases are not misaligned: all PET images are registered to the SPM12 template (MNI) space and resampled to a 121×145×121 grid with 1.5 mm voxels, and AAL, BNA and HOA, defined in MNI space, were resampled to the same grid.
Q1. PET measures physiological uptake, and its abnormal patterns are not suitably explained with the concept of structural parcels. Abnormal areas are usually finer-grained than structural parcels and more patient-specific, so describing a structure as "abnormal" is not precise enough; their boundaries need to be fitted in more detail.
Q5. The assumption holds, and the boundaries stay tied to the image voxels. Every convolution in the stem has stride 1 and padding 1, with no pooling or downsampling, so the output feature map has exactly the same grid as the input image. Overlapping receptive fields make the feature of each voxel describe the voxel together with its immediate neighbourhood, as in any voxel-wise segmentation network, but do not move it to another voxel.
Q2 (also "interpretable by construction" in Weakness 4). The whole-brain context vector is a 16-d vector obtained as a learned weighted sum of the features of all brain voxels. FiLM is used because the uptake level of a PET region is partly related to the uptake of the whole brain and of its surroundings, AD-related changes are distributed, and clinical reading interprets a regional finding against the whole-brain pattern. A region should therefore be assessed with evidence from both the region itself and the whole brain, and requiring purely region-specific scores would not suit PET; this is by design.
Exact additive accounting and spatial localization are different properties, and the paper claims them separately. Exact additive accounting means that the image-level logit is exactly the sum of the supervoxel scores; this is what the paper means by contributions "faithful to the prediction", and it only concerns how the logit is decomposed over supervoxels. Spatial localization refers to the region boundaries, which ASP fits to the subject's PET at the voxel level, more finely than atlas parcels or patches. We do not claim that a region's score comes only from evidence inside its boundary: for the reason above, neither the pathology nor the model design is purely local.