- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies genotype reconstruction from genomic language model trained on related individuals. GenoFA combines model predictions with Mendelian transmission probabilities to infer a missing genotype. They measure the separation between reconstruction scores from models trained with and without a target's entire family, aiming to distinguish training-related leakage from imputation. They evaluate a fine-tuned Nucleotide Transformer and a MLP including real trios and larger families with simulated offspring. They then apply family-level client-DP and audit family membership with group-LiRA.
Critical strengths
-
This paper addresses a privacy issue: relatives can contribute information about the same individual and genomic training can divide one person's data across several records. Their family-aware reconstruction attack connects this problem to a concrete inference procedure. Evaluating both a genomic model and a MLP helps establish that the concern extends beyond one architecture.
-
The comparison with nonmember reconstruction is useful. An attacker can recover genotypes through imputation even when the family did not contribute to training. This gives a more informative assessment than reporting reconstruction accuracy on training examples alone.
Critical weaknesses
-
The privacy-utility comparison does not match the full family-level privacy guarantee. Under the bound used in this paper, Client-DP achieves better reported utility despite the tighter delta, which is encouraging, but the comparison does not establish the claimed gain at the same privacy guarantee. The authors also describe dependence as breaking an independence assumption of standard DP. Standard DP bounds changes under its specified adjacency without requiring independent records. The distinction between record-level protection, family-level protection, and inferential privacy needs to be stated correctly.
-
The evidence for leakage increasing with family size relies on a substantial change to the biological setting. The author explains that this creates diversity within short windows, but do not establish how this choice affects the attack advantage. The real-trio experiments support a narrower empirical claim. Without a sensitivity analysis, the stronger results on simulated families provide limited evidence about the effect of family size in human cohorts.
Questions for authors
-
Can you compare client-DP and group-DP at matched family-level epsilon and delta, and specify the adjacency and accountant used for each? Please also distinguish the limitations of record-level protection from the validity of the standard DP guarantee.
-
Does GenoFA retain its advantage, and does leakage increase with family size, when offspring generation uses a human-scale recombination rate? A sensitivity check within the existing simulation setup would help determine how much of the reported effect depends on this choice.
4 comments
Weakness 2 and Question 2: Does larger family size widen the gap between GenoFA and naive leakage, and what happens at human-scale recombination rates?
Both points are fair, and we agree the rate is a substantial deviation from biology. We ran two follow-up experiments: one isolating family size at a fixed recombination rate, one varying the recombination rate. Our simulations use 10,000 cM/Mb against a human-scale rate of roughly 1 cM/Mb because the short windows permitted by the gLM’s context length do not, at human rates, yield enough haplotype diversity to separate in-distribution from out-of-distribution statistically. The family-aware leakage is nonetheless not a simulation artifact: the real-trio experiments (Table 3, 1000 Genomes) use actual average human recombination, and the target’s relatives being in training still raises reconstruction accuracy there. The two experiments are updated in Appendix A.8 and Appendix A.21.
Experiment 1 holds the recombination rate at 10,000 and varies the number of correlated family members k, with all children’s genomes simulated at that rate; the high rate ensures enough unique genomic sequences for members and non-members. The gap between GenoFA and naive leakage widens on average as family size grows. Results are averaged across all 6 attack sites used in the manuscript.
| Number of kids | Average GenoFA leakage measure | Average naive leakage measure | Gap |
|---|---|---|---|
| 1 | 0.591 | 0.579 | +0.0114 |
| 2 | 0.583 | 0.568 | +0.0149 |
| 4 | 0.574 | 0.557 | +0.0174 |
| 8 | 0.557 | 0.538 | +0.0194 |
| 16 | 0.551 | 0.532 | +0.0192 |
Experiment 2 fixes one child per family and varies the recombination rate from 1 to 10,000. The effect is minor, so the comparison in the paper is not invalidated. We acknowledge that recombination rates vary across genomic regions, and that estimating them from HapMap rather than using an average rate of 1 may produce more pronounced differences for longer sequences, as noted among the simplifications in Appendix A.2. This is worth exploring in future work.
| Recombination rate (cM/Mb) | GenoFA accuracy %, members | Naive accuracy %, members | GenoFA accuracy %, non-members | Naive accuracy %, non-members |
|---|---|---|---|---|
| 1 | 96.9 | 90.6 | 72.6 | 63.7 |
| 10 | 96.9 | 90.6 | 72.6 | 63.7 |
| 100 | 97.0 | 90.9 | 72.9 | 63.7 |
| 1000 | 97.1 | 91.0 | 72.4 | 63.0 |
| 10000 | 97.1 | 91.3 | 70.3 | 61.2 |
Thank you again for your detailed review. As the discussion closes today, we wanted to check whether our responses have addressed your questions. We compared client-level and group-level DP at matched ε and δ, and clarified the scope of the DP guarantee. We also added a recombination-rate sensitivity analysis, which shows that GenoFA retains its advantage at human-scale rates. If any concerns remain, we would be glad to address them. Otherwise, we would be grateful if you would consider updating your score.
Thank you for the rebuttal and the additional experiments. I appreciate the authors' detailed responses to my questions, particularly the new tables provided.
After considering the rebuttal and the other reviewers' comments, I think my current score is still appropriate, so I will keep it unchanged.
My primary concerns regarding the theoretical alignment of the privacy-utility comparison and the biological realism of the recombination rates in the simulations remain partially unaddressed. The new experiments do not fully change my assessment of the paper's core claims.
Weakness 1: Record-DP limitations against the DP definition, and DP against inferential privacy:
[1] Shiva P. Kasiviswanathan and Adam Smith. On the ‘semantics’ of differential privacy: A Bayesian formulation. Journal of Privacy and Confidentiality, 6(1), Jun. 2014. doi: 10.29012/jpc.v6i1.634.
Weakness 1 and Question 1: Matching epsilon and delta for group-DP and client-DP: The table below gives matching family-level epsilon and delta for client-DP and group-DP (epsilon and delta scaled from record-DP) with maximum family size capped at 3 (k=3); the manuscript reported the case where client-DP uses the stricter delta in Figure 3a. Both record-DP training with DP-SGD (Algorithm 1) and client-DP training (Algorithm 2) use a Rényi DP accountant. Under record-DP, adjacency is D, D′ differing in one record (Definition 1). Under client-DP (Definition 5), D and D′ differ in all data points of the entire family, following prior implementations (McMahan et al., 2018; Geyer et al., 2018) that treat all of a family’s data points as one client.
Group-DP and client-DP comparison (maximum family size 3):