- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper investigates the privacy risks of releasing feature embeddings extracted from frozen pretrained encoders applied to longitudinal T1-weighted brain MRI scans. The authors define three threat model, patient liknability, re-identification, and attribute inferenc, and evaluate five pretrained encoders alongside a handcrafted baseline across two longitudinal Alzheimer's disease cohorts (OASIS-2 and MIRIAD). Using Canberra distance for similarity scoring, they measure system-level linkability and matching accuracy, finding that most encoders retain substantial subject-specific information. They also train a random forest classifier on the frozen embeddings to predict binary recorded sex under subject-disjoint cross-validation, showing strong balanced accuracies and observe that linkage risk and sex-inference risk do not rank encoders in the same order.
Critical strengths
- Motivated by recent debates over whether medical foundation model retrieval experiments actually constitute re-identification (Engelmann et al., 2026), the paper carefully distinguishes record linkability, identity-linked re-identification, and reference-free attribute inference and pairs retrieval accuracy with a calibrated system-level linkability audit metric
- Showing that privacy risk is multidimensional: Comparing Table 3 (linkage) and Table 4 / Table 6 (sex inference) reveals an inverse relationship across the extreme models with AnatCL having the highest linkage risk but the lowest sex-inference accuracy whereas SAM-Med3D has the lowest linkage risk but high within-dataset and across-dataset sex-inference acc. This provides vidence that low risk on one privacy dimension does not guarantee safety on another and is an important result for the community
Critical weaknesses
- On page 4, you state that you used all 708 scans in MIRIAD and claim that "these are longitudinal acquisitions rather than near-duplicate acquisitions of the same scan: MIRIAD follow-ups are separated by approximately two weeks to two years." However, there is a problem here: by the design of the MIRIAD study (Malone et al., 2013), see https://pmc.ncbi.nlm.nih.gov/articles/PMC3809512/pdf/main.pdf page 34 and Table 2, two back-to-back repeatability scans were acquired during the exact same visit on the same day at weeks 0, 6, and 38, accounting for around 370 to 380 out of the scans. If your evaluation protocol places every other scan in the dataset into the reference gallery and only excludes the query scan itself, more than half of your queries in MIRIAD have a same-visit, back-to-back scan in the gallery. This inflates your MIRIAD metrics and would explain the otherwise confusing finding that unaligned models like SAM-Med3D jump from 13.9% ReID@1 on OASIS-2 to 89.6% on MIRIAD. -Each evaluated encoder uses a different preprocessing and input pipeline. You correctly cite Engelmann who show that untrained random CNNs and raw downsampled images can match or exceed foundation models in medical image linkage but then you include no randomly initialised (untrained) encoder baselines on the same preprocessed inputs. The paper also contains an internal contradiction regarding preprocessing: on page 4 and page 15 you state that OASIS-2 is already distributed in a motion-corrected, registered, and skull-stripped form and argue that this removes identifying information relative to raw MIRIAD but on page 8 you argue that on OASIS-2, 3D-Neuro-SimCLR has higher risk than SAM-Med3D because 3D-Neuro-SimCLR uses skull-stripped and linearly registered volumes while SAM-Med3D applies no brain-specific preprocessing.
- Sensitive-attribute inference is evaluated only on binary recorded sex, omitting clinically sensitive attributes available in both OASIS-2 and MIRIAD such as Alzheimer's disease diagnosis, cognitive scores and age. Moreover, in App A2, cross-dataset sex inference collapses to near-chance balanced accuracy for four of the five encoders with only SAM-Med3D generalising across cohorts which suggets that the within-dataset RF classifiers for the other encoders may be exploiting dataset-specific acquisition or preprocessing artefacts rather than generalisable representations.
- Primitive methodology: For an ICLR paper, the methodology of this work is primitive. In my opinion, it would be better suited to a medical journal, not an AI conference.
Questions for authors
- Please recompute Table 3 and Table 7 on MIRIAD after strictly excluding same-visit back-to-back scans from the reference gallery or retaining at most one scan per visit per subject and evaluate a stricter two-split linkage setting where the gallery contains only a single baseline scan per subject and queries come from subsequent visits
- What are the linkability and ReID1 scores when using randomly initialised weights for each encoder on the exact same preprocessed inputs? Also, can you report the exact quantitative numbers for the experiment mentioned only in passing on page 8 (lines 392-394) using only AnatCL's "three anatomical descriptors"?
- Since OASIS-2 scans are already distributed skull-stripped and atlas-registered, what additional preprocessing difference between 3D-Neuro-SimCLR and SAM-Med3D on OASIS-2 accounts for the discussion on page 8 and how do you reconcile page 8 (where preprocessing is said to increase re-identification risk) with page 15 (where preprocessing on OASIS-2 is said to remove identifying information)?
- Can you extend the attribute-inference evaluation in Sec 4.2 to include age and alzheimer's disease status (on healthy vs. AD subjects, or inferring clinical status), and explain why four out of five encoders fail to generalise sex prediction across OASIS-2 and MIRIAD in Table 6?
2 comments
Thank you for your response and for re-running the MIRIAD evaluation without the same-visit back-to-back repeatability scans. While fixing the MIRIAD protocol addresses one experimental flaw, several substantive concerns remain that prevent me from raising my score.
- Removing same-visit repeatability scans caused SAM-Med3D's ReID@1 on MIRIAD to drop by 12%, confirming that same-session scans substantially inflated retrieval performance for unaligned models. I cannot find a stricter gallery split with a single reference scan per subject.
- A randomly initialised encoder baseline is not "merely an alternative attacker scenario that can be deferred to future work". Because each evaluated encoder relies on a different preprocessing and spatial registration pipeline, running the same architectures with random weights on the exact same preprocessed inputs is a necessary control to isolate how much linkability comes from foundation model pretraining versus the preprocessing pipeline itself. Given that inference on ca. 700 preprocessed scans requires minimal compute, omitting this control leaves a central question of the paper unanswered.
- You state that classifier performance for age prediction was "close to chance across the evaluated encoders" and conclude that the embeddings do not meaningfully encode age. However, brain age is one of the strongest signals in T1-weighted MRI so reporting near-chance age prediction across all models suggests a methodological issue in how age inference was formulated.
- In a general-purpose embedding sharing scenario, clinical diagnosis (eg. Alzheimer status) is a critical sensitive health attribute whenever the downstream task is not AD classification itself. Moreover, the fact that SAM-Med3D generalises across OASIS-2 and MIRIAD for sex prediction while the other four encoders collapse toward chance on the exact same sample sizes indicates that within-dataset classifiers for those four models are sensitive to cohort- or preprocessing-specific artifacts rather than capturing a dataset-invariant demographic signal.
We thank the reviewer for carefully examining the experimental methodology and identifying the same-visit repeatability acquisitions in MIRIAD. This observation has helped us strengthen the evaluation protocol.
For the back-to-back scans of MIRIAD : We are grateful to the reviewer for catching this. We reran the evaluation keeping only one scan per session and excluding the back-to-back repeatability scans. Across most models the results are stable: AnatCL, DINO, Brain-MRI-SSL 3DNeuro-SimCLRr show negligible change in both Linkability and ReID@1. SAM-Med3D’s ReID@1drops from 89.58% to 77.52%. It may be explained by the fact that SAM-Med3D’s local, segmentation-oriented features are similarly sensitive to positioning and intensity. Models with more anatomy-driven or acquisition-invariant representations (Brain-MRI-SSL, AnatCL, DINO) are unaffected. Its Linkability score, however, remains essentially stable (0.5139 to 0.5111), and ReID@5 drops by a smaller margin (96.62% to 91.62%), showing that even where the top-1 ranking is disturbed, the correct match is still recovered within the top five candidates, so the underlying identity signal persists.
The overall conclusion is unchanged: all five models still leak substantial identity-related information. We updated the tables 3 and 7 in the paper with the correct results.
As for the experiment with an untrained encoder: Our setting assumes a realistic attacker who uses the same publicly released encoder as the target site, a reasonable assumption since foundation-model weights are typically open-sourced alongside the embeddings. A weaker, non-learned baseline is not the scenario we evaluate, but it would be a valuable addition to isolate how much of the apparent leakage is attributable to the learned representation itself versus general properties of high-dimensional embedding spaces. Due to time constraints for this revision, we are not able to include it now, but we see it as a future work.
For the experiment using only ANATCL’s three anatomical descriptors : here are the results on oasis2: re-identification using only the three anatomical descriptors that guide its weakly contrastive training yields a Linkability of 0.8221 and ReID@1 of 79.31%, close to the 0.9155 Linkability and 90.91% ReID@1 obtained with the full AnatCL embeddings, suggesting that these anatomical features drive much of its risk. We reported these results in line 378.
Regarding the preprocessing statements: These two statements operate at two different levels of analysis and are not contradictory. At the dataset level (pages 4 and 15), we observe that OASIS-2 (distributed already skull-stripped and atlas-registered) yields lower re-identification rates than raw MIRIAD, consistent with the idea that this preprocessing removes some identifying information (e.g. facial structure).
At the model level (page 8), the comparison between 3D-Neuro-SimCLR and SAM-Med3D on OASIS-2 concerns a different analysis: here, 3D-Neuro-SimCLR (trained on skull-stripped volumes) produces embeddings better aligned with the residual structural identity signal present in OASIS-2, whereas SAM-Med3D, trained for organ segmentation on heterogeneous medical images, extracts features that are less correlated with that signal.
On the omission of diagnosis and cognitive scores as sensitive attributes: we do not consider the pathological class (e.g. Alzheimer’s diagnosis) a sensitive attribute in the sense intended here. It is the clinical outcome variable, not something we want to suppress or hide from the embeddings. However on age: we did run this experiment. We do not find that the embeddings encode age to a meaningful degree alongside identity and sex; classifier performance for age prediction was close to chance across the evaluated encoders
About the cross-dataset collapse and generalization: We agree this is a good and important observation. However, this reflects a common machine learning issue rather than something specific to our method: if we had trained on both datasets jointly and tested across different distributions, we would likely have obtained better generalization. Our datasets are not large enough to support this kind of training, and we added this statement this explicitly as a limitation.
We sincerely thank the reviewer for their thorough assessment and for identifying methodological aspects that warranted further investigation. Your feedback has been particularly valuable in motivating our additional experiments excluding same-visit acquisitions and in helping us refine the interpretation of our findings. We hope that these new results, together with our clarifications and acknowledgment of the remaining limitations, help address your main concerns regarding experimental soundness. We would respectfully appreciate your reconsideration of the paper's assessment and score in light of this additional evidence.