acceptodds
Under review as a conference paper at ICLR 2027

Quantifying privacy risks in foundation model representations of brain MRI

Abstract

Pretrained representations are increasingly used to share and reuse information from medical images without exposing the original scans, but it is unclear how much patient-specific information these representations retain. We investigate this question for brain magnetic resonance imaging (MRI) by evaluating five frozen pretrained encoders spanning different architectures, pretraining objectives, and training domains on two longitudinal T1-weighted datasets, OASIS-2 and MIRIAD. Under an attacker model with access to released embeddings, the corresponding public encoder, and auxiliary MRI scans, we evaluate patient linkage and sensitive-attribute inference. Across the evaluated representations, we observe substantial subject-specific information: rank-1 patient linkage reaches up to 99% on MIRIAD and exceeds 90% on OASIS-2 for the highest-risk representation, despite the absence of an identification objective during encoder training. We further show that the released embeddings retain information about a demographic attribute, with recorded-sex prediction reaching balanced accuracy of 90.9% on OASIS-2 and 95.5% on MIRIAD. Overall, our results demonstrate that replacing raw brain MRI with learned embeddings does not, by itself, eliminate patient-linkage or attribute-inference risk, and is therefore insufficient on its own to establish anonymity.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
brain MRI, pre-trained encoders, Foundation models, privacy, re-identification, membership inference, embeddings, self-supervised learning, medical imaging privacy, attribute inference.

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_UzNhOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates the privacy risks of releasing feature embeddings extracted from frozen pretrained encoders applied to longitudinal T1-weighted brain MRI scans. The authors define three threat model, patient liknability, re-identification, and attribute inferenc, and evaluate five pretrained encoders alongside a handcrafted baseline across two longitudinal Alzheimer's disease cohorts (OASIS-2 and MIRIAD). Using Canberra distance for similarity scoring, they measure system-level linkability and matching accuracy, finding that most encoders retain substantial subject-specific information. They also train a random forest classifier on the frozen embeddings to predict binary recorded sex under subject-disjoint cross-validation, showing strong balanced accuracies and observe that linkage risk and sex-inference risk do not rank encoders in the same order.

Critical strengths
  • Motivated by recent debates over whether medical foundation model retrieval experiments actually constitute re-identification (Engelmann et al., 2026), the paper carefully distinguishes record linkability, identity-linked re-identification, and reference-free attribute inference and pairs retrieval accuracy with a calibrated system-level linkability audit metric
  • Showing that privacy risk is multidimensional: Comparing Table 3 (linkage) and Table 4 / Table 6 (sex inference) reveals an inverse relationship across the extreme models with AnatCL having the highest linkage risk but the lowest sex-inference accuracy whereas SAM-Med3D has the lowest linkage risk but high within-dataset and across-dataset sex-inference acc. This provides vidence that low risk on one privacy dimension does not guarantee safety on another and is an important result for the community
Critical weaknesses
  • On page 4, you state that you used all 708 scans in MIRIAD and claim that "these are longitudinal acquisitions rather than near-duplicate acquisitions of the same scan: MIRIAD follow-ups are separated by approximately two weeks to two years." However, there is a problem here: by the design of the MIRIAD study (Malone et al., 2013), see https://pmc.ncbi.nlm.nih.gov/articles/PMC3809512/pdf/main.pdf page 34 and Table 2, two back-to-back repeatability scans were acquired during the exact same visit on the same day at weeks 0, 6, and 38, accounting for around 370 to 380 out of the scans. If your evaluation protocol places every other scan in the dataset into the reference gallery and only excludes the query scan itself, more than half of your queries in MIRIAD have a same-visit, back-to-back scan in the gallery. This inflates your MIRIAD metrics and would explain the otherwise confusing finding that unaligned models like SAM-Med3D jump from 13.9% ReID@1 on OASIS-2 to 89.6% on MIRIAD. -Each evaluated encoder uses a different preprocessing and input pipeline. You correctly cite Engelmann who show that untrained random CNNs and raw downsampled images can match or exceed foundation models in medical image linkage but then you include no randomly initialised (untrained) encoder baselines on the same preprocessed inputs. The paper also contains an internal contradiction regarding preprocessing: on page 4 and page 15 you state that OASIS-2 is already distributed in a motion-corrected, registered, and skull-stripped form and argue that this removes identifying information relative to raw MIRIAD but on page 8 you argue that on OASIS-2, 3D-Neuro-SimCLR has higher risk than SAM-Med3D because 3D-Neuro-SimCLR uses skull-stripped and linearly registered volumes while SAM-Med3D applies no brain-specific preprocessing.
  • Sensitive-attribute inference is evaluated only on binary recorded sex, omitting clinically sensitive attributes available in both OASIS-2 and MIRIAD such as Alzheimer's disease diagnosis, cognitive scores and age. Moreover, in App A2, cross-dataset sex inference collapses to near-chance balanced accuracy for four of the five encoders with only SAM-Med3D generalising across cohorts which suggets that the within-dataset RF classifiers for the other encoders may be exploiting dataset-specific acquisition or preprocessing artefacts rather than generalisable representations.
  • Primitive methodology: For an ICLR paper, the methodology of this work is primitive. In my opinion, it would be better suited to a medical journal, not an AI conference.
Questions for authors
  • Please recompute Table 3 and Table 7 on MIRIAD after strictly excluding same-visit back-to-back scans from the reference gallery or retaining at most one scan per visit per subject and evaluate a stricter two-split linkage setting where the gallery contains only a single baseline scan per subject and queries come from subsequent visits
  • What are the linkability and ReID1 scores when using randomly initialised weights for each encoder on the exact same preprocessed inputs? Also, can you report the exact quantitative numbers for the experiment mentioned only in passing on page 8 (lines 392-394) using only AnatCL's "three anatomical descriptors"?
  • Since OASIS-2 scans are already distributed skull-stripped and atlas-registered, what additional preprocessing difference between 3D-Neuro-SimCLR and SAM-Med3D on OASIS-2 accounts for the discussion on page 8 and how do you reconcile page 8 (where preprocessing is said to increase re-identification risk) with page 15 (where preprocessing on OASIS-2 is said to remove identifying information)?
  • Can you extend the attribute-inference evaluation in Sec 4.2 to include age and alzheimer's disease status (on healthy vs. AD subjects, or inferring clinical status), and explain why four out of five encoders fail to generalise sex prediction across OASIS-2 and MIRIAD in Table 6?
2 comments

We thank the reviewer for carefully examining the experimental methodology and identifying the same-visit repeatability acquisitions in MIRIAD. This observation has helped us strengthen the evaluation protocol.

For the back-to-back scans of MIRIAD : We are grateful to the reviewer for catching this. We reran the evaluation keeping only one scan per session and excluding the back-to-back repeatability scans. Across most models the results are stable: AnatCL, DINO, Brain-MRI-SSL 3DNeuro-SimCLRr show negligible change in both Linkability and ReID@1. SAM-Med3D’s ReID@1drops from 89.58% to 77.52%. It may be explained by the fact that SAM-Med3D’s local, segmentation-oriented features are similarly sensitive to positioning and intensity. Models with more anatomy-driven or acquisition-invariant representations (Brain-MRI-SSL, AnatCL, DINO) are unaffected. Its Linkability score, however, remains essentially stable (0.5139 to 0.5111), and ReID@5 drops by a smaller margin (96.62% to 91.62%), showing that even where the top-1 ranking is disturbed, the correct match is still recovered within the top five candidates, so the underlying identity signal persists.

The overall conclusion is unchanged: all five models still leak substantial identity-related information. We updated the tables 3 and 7 in the paper with the correct results.

As for the experiment with an untrained encoder: Our setting assumes a realistic attacker who uses the same publicly released encoder as the target site, a reasonable assumption since foundation-model weights are typically open-sourced alongside the embeddings. A weaker, non-learned baseline is not the scenario we evaluate, but it would be a valuable addition to isolate how much of the apparent leakage is attributable to the learned representation itself versus general properties of high-dimensional embedding spaces. Due to time constraints for this revision, we are not able to include it now, but we see it as a future work.

For the experiment using only ANATCL’s three anatomical descriptors : here are the results on oasis2: re-identification using only the three anatomical descriptors that guide its weakly contrastive training yields a Linkability of 0.8221 and ReID@1 of 79.31%, close to the 0.9155 Linkability and 90.91% ReID@1 obtained with the full AnatCL embeddings, suggesting that these anatomical features drive much of its risk. We reported these results in line 378.

Regarding the preprocessing statements: These two statements operate at two different levels of analysis and are not contradictory. At the dataset level (pages 4 and 15), we observe that OASIS-2 (distributed already skull-stripped and atlas-registered) yields lower re-identification rates than raw MIRIAD, consistent with the idea that this preprocessing removes some identifying information (e.g. facial structure).

At the model level (page 8), the comparison between 3D-Neuro-SimCLR and SAM-Med3D on OASIS-2 concerns a different analysis: here, 3D-Neuro-SimCLR (trained on skull-stripped volumes) produces embeddings better aligned with the residual structural identity signal present in OASIS-2, whereas SAM-Med3D, trained for organ segmentation on heterogeneous medical images, extracts features that are less correlated with that signal.

On the omission of diagnosis and cognitive scores as sensitive attributes: we do not consider the pathological class (e.g. Alzheimer’s diagnosis) a sensitive attribute in the sense intended here. It is the clinical outcome variable, not something we want to suppress or hide from the embeddings. However on age: we did run this experiment. We do not find that the embeddings encode age to a meaningful degree alongside identity and sex; classifier performance for age prediction was close to chance across the evaluated encoders

About the cross-dataset collapse and generalization: We agree this is a good and important observation. However, this reflects a common machine learning issue rather than something specific to our method: if we had trained on both datasets jointly and tested across different distributions, we would likely have obtained better generalization. Our datasets are not large enough to support this kind of training, and we added this statement this explicitly as a limitation.

We sincerely thank the reviewer for their thorough assessment and for identifying methodological aspects that warranted further investigation. Your feedback has been particularly valuable in motivating our additional experiments excluding same-visit acquisitions and in helping us refine the interpretation of our findings. We hope that these new results, together with our clarifications and acknowledgment of the remaining limitations, help address your main concerns regarding experimental soundness. We would respectfully appreciate your reconsideration of the paper's assessment and score in light of this additional evidence.

Thank you for your response and for re-running the MIRIAD evaluation without the same-visit back-to-back repeatability scans. While fixing the MIRIAD protocol addresses one experimental flaw, several substantive concerns remain that prevent me from raising my score.

  • Removing same-visit repeatability scans caused SAM-Med3D's ReID@1 on MIRIAD to drop by 12%, confirming that same-session scans substantially inflated retrieval performance for unaligned models. I cannot find a stricter gallery split with a single reference scan per subject.
  • A randomly initialised encoder baseline is not "merely an alternative attacker scenario that can be deferred to future work". Because each evaluated encoder relies on a different preprocessing and spatial registration pipeline, running the same architectures with random weights on the exact same preprocessed inputs is a necessary control to isolate how much linkability comes from foundation model pretraining versus the preprocessing pipeline itself. Given that inference on ca. 700 preprocessed scans requires minimal compute, omitting this control leaves a central question of the paper unanswered.
  • You state that classifier performance for age prediction was "close to chance across the evaluated encoders" and conclude that the embeddings do not meaningfully encode age. However, brain age is one of the strongest signals in T1-weighted MRI so reporting near-chance age prediction across all models suggests a methodological issue in how age inference was formulated.
  • In a general-purpose embedding sharing scenario, clinical diagnosis (eg. Alzheimer status) is a critical sensitive health attribute whenever the downstream task is not AD classification itself. Moreover, the fact that SAM-Med3D generalises across OASIS-2 and MIRIAD for sex prediction while the other four encoders collapse toward chance on the exact same sample sizes indicates that within-dataset classifiers for those four models are sensitive to cohort- or preprocessing-specific artifacts rather than capturing a dataset-invariant demographic signal.
Reviewer_eS81OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper sets out to explore the privacy risks that can exist in datasets made up of embeddings of Brain MRI Images. Even though precautions are taken to hide the identitiy of a patient in the dataset, multiple occurances in the dataset could be linked or sensitive attributes like sex can be inferred from the released embeddings. Existing studies show that even when defacing is applied, brain shape can be used to identify individuals. The authors conduct a study of five foundation models, four of which are train using unsupervised learning to embedding image data (brain mri or other medical/natural images). Using two brain MRI datasets, they conduct attacks to determine what information is recoverable from the released embeddings. The attackers are assumed to have access to the embedding model, the released embeddings and a collection of labelled auxillary data. A random forest attack model is trained for each encoder/dataset and it's linkability and re-identification and sex classification is reported. The vulnerabilities of the different model show by these results is then discussed.

Critical strengths

-The paper is very clearly written, with a strong motivation: the real-world issue of how to 'anonymously' release brain-MRI data, as is required by some governments. The literature review shows the gap in previous works : privacy is only implemented by removing names/birthdates or defacing, existing privacy approaches did not specify their attack assumptions.

The authors conduct a thorough audit of existing foundation models that vary greatly in their training data, preprocessing, and training objective. They clearly specify multiple types of data leakage that could occur, and design an attack model to exploit this. This successfully demonstrates that virtually all the embedding of the two brain MRI datasets present some privacy risk. They conclude that existing models are never truly anonymizing their data.

Critical weaknesses

-While this paper thoroughly demonstrates that existing medical foundation models are not private, they stop short of answering important questions such as WHY are they not private (what contributes to the risk), and how can the models be protected. Since the 5 evaluated models all have different training objectives and preprocessing, it is impossible to isolate the effects of certain choices. Therfore, discussion is essentially limited to guessing at why AnatCL (with the 'most detailed preprocessing') is the most vulnerable model.

-The paper would be stronger if authors also conducted a full analysis of their own foundation models, each trained using slightly different objectives or preprocessing, to isolate the differences in leakage. Furthermore, it would be very useful to implement any existing defences to compare their effect against unprotected models.

-The current attack model assumes full access to the embedding model f. This is generally a strong assumption for adversaries, since many only can obtain input output pairs from their auxillary dataset. If possible, it would be good to look at vulnerabilities of the models under varying strengths of attack assumptions.

-The paper has very little acknoweledgment of existing defense techniques in the related works section,the discussion and conclusion. While I am not familiar with defenses specialized to foundation models, it would be worth discussing existing methods for other models like neural networks (e.g. differential privacy, federated learning, PATE, LoRA). Defacing is breifly mentioned in related work, can this be applied to your experiments? Or is it already incorporated during preprocessing?

Questions for authors

-Would it be possible to isolate the effects of the differences in the 5 foundation models, by changing small aspects and evaluate privacy risks? If not, why (possibly too expensive)? Until this sort of analyses is conducted, the claims of what makes a foundation model more or less private (e.g. supervised training seems to be less private) are unsupported.

-It is good that you clearly specify your attack arhcitechture and assumptions, but again it would be best if you could shows results for multiple different attack models (e.g. try using an MLP instead of RF....). Is this possible to add? Why choose random forest and assume embedding model access (access to f)? What type of attack can be done if target model architechture is not known?

-In the discussion/conclusion, what defense techniques might help defend against the attacks? What are the tradeoffs of these techniques (generally there is a privacy-utlility tradeoff).

-Since some of the 5 embedding models seem to be more private, it seems natural to choose the one that has the lowest attack scores. Is there any accuracy trade-off to this? Are the embeddings from the most private model more noisy/hard to derive clinical meaning from? (This is a common issue in privacy for medical data).

2 comments

We thank the reviewer for the detailed feedback on controlled comparisons, attacker assumptions, and possible defenses. We first clarify the distinction between our two attack procedures. Random Forest is used exclusively for recorded-sex inference. Patient linkage and re-identification are performed directly from the frozen embeddings using Canberra-based similarity, without training an additional classifier.

On the choice of Random Forest: prior work on sex inference from brain MRI embeddings has used Random Forest, which motivated our choice. To test this more rigorously, we additionally evaluated five other classifiers (Logistic Regression, SVM, KNN, Naive Bayes, Decision Tree) across both datasets. Random Forest was competitive with the best classifier in every setting, and the ranking of the five foundation models by sex-inference accuracy (Brain-MRI-SSL > SAMMed3D > DINO > SimCLR > AnatCL) was identical across OASIS-2 and MIRIAD regardless of classifier.

On isolating the effects of differences between models: we agree this is important for interpretability. However, isolating each design axis (training objective, supervised vs. self-supervised, backbone, pretraining domain) would require a controlled ablation, retraining matched variants differing in one factor at a time, which is costly at foundation-model scale and outside this paper’s scope. Our goal here was narrower: to show that, across five existing models spanning different architectures and objectives, none is privacy-preserving as a replacement for sharing raw images. We report this as a limitation and plan to investigate these factors in future work alongside our planned protective method.

On the white-box assumption: this is meant to be realistic, not worst-case. Training a foundation-model-scale encoder from scratch is expensive, so hospitals and research groups tend to reuse publicly released encoders rather than build their own (e.g., UNI2 released model weights and pre-extracted features for over 25,000 pathology slides [1]). An attacker need only know which public model a site adopted, then use it directly. If the architecture is genuinely unknown, the black-box scenario is still possible: the attacker can train an encoder from scratch on public data using a comparable self-supervised objective.

For privacy techniques: defacing only removes facial voxels; identity in brain MRI is encoded in brain structure itself, so we do not consider it a solution, and our tested scans are not defaced. A relevant distinction is between protecting training data and protecting new patients’ embeddings shared at inference time; our focus is the latter. LoRA can support an identity-suppressing fine-tuning objective but gives no privacy guarantee alone. Differentially private training and PATE protect training data; federated learning decentralizes data without itself guaranteeing privacy. None automatically prevents identity leakage from new patients’ embeddings; post-processing (identity-suppressing transformations or differential privacy applied to embeddings before release) is a more direct fit. We see three directions for our own protection method: private fine-tuning (privacy constraints during fine-tuning), post-processing (transforming embeddings after inference), and private retraining (retraining from scratch with a privacy objective).

On privacy-utility tradeoff: we use balanced accuracy on Alzheimer’s classification (demented vs. control; OASIS-2 “Converted” subjects excluded) as the utility metric against the re-identification risk in Table 3. Utility drops for OASIS-2 vs. MIRIAD; this needs further investigation, but may relate to OASIS-2’s existing preprocessing. The most private encoder, SAM-Med3D, still retains usable utility, though the lowest among higher-utility encoders. The least private, AnatCL, offers no compensating advantage. Among higher-utility encoders (Brain-MRI-SSL, 3D-Neuro-SimCLR, DINO-ViT), re-identification risk is consistently high (ReID@1 ≥ 0.82 on MIRIAD). No encoder is simultaneously most private and most useful; we recommend none of the evaluated models. We reported the results are in Appendix A4.

We thank the reviewer again for their thoughtful comments and valuable suggestions, which have helped us strengthen the experimental analysis and better articulate our contributions. We hope that our additional results and clarifications, particularly regarding the robustness of attribute inference across different classifiers and the multidimensional nature of privacy leakage, provide a clearer picture of the significance of our findings. We would be grateful if the reviewer could reconsider their overall assessment and score in light of these improvements.

[1] UNI2, Mahmood Lab, github.com/mahmoodlab/UNI

Reviewer_eS81OpenReview ↗

Response to official comment

Thank you for clarifying the attack procedures and classifier choice. Can you please highlight where or whether this clarification has been main in the revised pdf?

I understand now that the controlled ablation may be too costly to add to this paper, however I still feel it is a big limitation of this work. It should certainly be discussed in the paper as a limitation and direction for future work, with explicit arguments as to why it is too expensive. Please highlight this and let me know when/if it is added.

The white box and privacy techniques explanation is satisfactory now, however again please highlight how this is improved in the text.

Privacy Utility tradeoff: I appreciate the experiments exploring this, however this statement concerns me: " No encoder is simultaneously most private and most useful; we recommend none of the evaluated models." While it is common knowledge that we often cannot optimize these two objectives, this statement makes the work feel like a 'negative results' paper. In that case, since your analyses is only on a subset of models, it is not enough to suggest that 'no foundations models are private'. Without this claim, I feel the paper really needs to point to some cases where the privacy could me maintained. Otherwise, the contribution feels too weak.

Overall, I read this paper as demonstrating that foundation models have privacy risks, but that to mean seems like common knowledge. It does not conduct an exhaustive review of all models, or prove an impossibility of privacy, nor does it propose an improved approach for privacy in these models.

If you have the time, could you please provide a more thorough explanations of exactly what you have updated in the main text based on my comments. Please make the updates in a highlighted colour (e.g. blue), so that I can easily identify them and assess whether they have been sufficiently addressed.

Reviewer_5gN3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce an attack model designed to address the linkability and release of withheld patient attributes in the embedding space of foundation models. Using the dataset of OASIS-2 and MIRIAD, they highlight the risk of treating embeddings as sufficiently privacy preserving.

Critical strengths
  1. I appreciate the authors’ characterization of the linkage and withheld patient attributes in the context of the guidelines of the EDPB. I think this is a timely topic to discuss with an increasing amount of foundation models being released and perhaps to trigger a discussion of the privacy characteristics of embeddings - instead of treating them as anonymized latent representations via deep learning by default.

  2. The authors are incredibly transparent about the attack models in question and the evaluations are clear, easy to understand and to the point.

  3. The work clearly points to the issues with treating embeddings as sufficiently safe from the perspective of anonymization and points towards a need of perhaps targeted mechanisms to make them privacy safe.

Critical weaknesses
  1. I think the following characterizes one of the experimental advantages, but also broadly highlights the weakness of both MIRIAD and OASIS-2.

This acquisition consistency is advantageous for our privacy analysis because scanner/site membership and acquisition setting do not vary systematically across subjects, reducing the possibility that subject matching is driven by between-subject acquisition-condition differences

The scans are incredibly homogenous in scanner type and site, while correctly being heterogenous in time. This does weaken the generalized takeaways of the work.

..when a patient undergoes MRI examinations at multiple clinics and an attacker with access to scans from one clinic attempts to link them to embeddings released by another

This example attack scenario is basically not covered by the authors’ datasets.

  1. The weakened experimentation above casts doubt on such a strong statement.

Overall, these results show that pretrained brain MRI encoders are not privacy-safe

There are already models that show near 50% linkage - perhaps this will be further lower in multi-site, multi-scanner settings but the phrasing could be made cautionary instead of absolute until experiments exist to support that.

Questions for authors
  1. Can the authors comment on the lack of mutli-site data in their dataset and possible issues for the generalizability of their results? I ask this while understanding the challenge of acquiring such data, but also commenting on their own example in the paper as stated in the Weaknesses.

  2. Does the fact that every single scan has a matching scan in their auxiliary embeddings makes a difference to their results?

  3. Are the authors aware of any documented examples of the conditions of such an attack made possible in any recent foundation model release? I ask because without a compromised infrastructure problem, it seems unlikely that a site/hospital/clinic would knowingly release their embeddings. This is not necessarily a problem - the attack model is still an interesting thought experiment.

2 comments

We thank the reviewer for the constructive comments and for recognizing the relevance of auditing privacy risks in released medical representations.

On dataset homogeneity: OASIS-2 and MIRIAD are each single-site, but they differ substantially from each other in acquisition protocol, scanner characteristics, preprocessing pipeline, and cohort composition. We evaluate the same five frozen encoders on both, and the consistently high re-identification risk across these two distinct settings is some evidence that the risk is not an artefact of one particular acquisition context. That said, both datasets remain single-site, so this does not establish generalization to genuinely multi-site or multi-scanner deployment, and it has been added explicitly as a limitation.

We should also flag a real constraint: a publicly available, longitudinal dataset with multiple scans per subject that is also confirmed absent from the pretraining data of all five evaluated foundation models is a demanding combination of requirements, and OASIS-2 and MIRIAD are, to our knowledge, among the very few datasets that satisfy all of them at once. We considered testing on a multi-site dataset such as ON-Harmony[1], but it has only 20 subjects, which would not make the evaluation robust. Given this, we see a genuinely multi-site evaluation as future work, once we identify a dataset that meets these requirements, rather than something we can commit to for this revision. In the paper, we will change the statement to use two different moments in time instead of different hospitals, in line 211.

On the matching-scan question: this requirement is necessary for Re-ID rate, which is only defined when a true match exists in the gallery, so it is needed to measure that attack. For linkability, it is not mandatory that every subject has a match in the gallery: the metric is computed from the distribution of mated and non-mated scores, so as long as there are enough positive pairs (scans of the same subject) in the set, we can still measure it even when some subjects have no matching scan in the gallery. On the strong statement: We agree the original phrasing overstated what our results support. We have already revised the statement at line 381 to: “These results suggest that pretrained brain MRI encoders may carry privacy risks,” which better reflects the scope of our evaluation.

On the realism of the attack scenario:, we note that releasing precomputed image embeddings instead of raw scans is an established practice. NeuroCBIR [2], for instance, publicly shares more than 26,000 precomputed T1-weighted brain MRI embeddings specifically to support reproducible research and reduce the compute and storage cost of downstream tasks. Also, PhysioNet [3] distributes precomputed embeddings from the MIMIC-CXR chest radiography database for the same resource-saving reason. These examples show that embedding-sharing, including for brain MRI, is a real and growing practice. To the best of our knowledge, no attack on these embeddings has been documented yet. Our framing is intentionally anticipatory: foundation models and their embeddings are being adopted rapidly, and we want to flag this risk before it becomes standard deployed practice.

References:

[1] https://openneuro.org/datasets/ds004712/versions/2.0.1

[2] NeuroCBIR: github.com/minnelab/NeuroCBIR

[3] Image Embeddings for MIMIC-CXR, PhysioNet, v1.0.0

We sincerely thank the reviewer for their constructive feedback and for recognizing the relevance of our work. Your comments have helped us better clarify the scope of our findings, the assumptions underlying our threat model, and the limitations regarding cross-site generalizability. We hope these clarifications adequately address your concerns and would greatly appreciate your reconsideration of the paper's overall assessment and score.

Reviewer_5gN3OpenReview ↗

Thank you for your response

I thank the authors for their detailed responses. I believe that the authors do have an interesting direction for their work. However, while the section of 4.1 PATIENT LINKAGE AND RE-IDENTIFICATION is interesting (I note the correction of the MIRIAD issue pointed out by reviewer UzNh), I still believe that the closed set nature of the attack scenario affects the linkability and reidentification scores based on the prevalence of matches between the two sets. It's not merely the computability of the metrics that my question was based on but the effect on the performance of the metrics if an increasing number of the patients in the auxilliary set are not necessarily represented in the released embeddings.

I do not object to the authors' defense of their selection of OASIS and MIRIAD, or the assertion of the realism of the attack setting. Alternatives like ON-Harmony, as they have stated, would have been questionable.

However, I believe that my score in the initial review still reflects my position with respect to the above mentioned issue in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.