acceptodds
Under review as a conference paper at ICLR 2027

End-to-End Differential Privacy for Multi-stage Foundation Model Training

20 views

Abstract

Differential privacy is increasingly used in multi-stage training processes of foundation models, from initial pretraining over different fine-tuning and alignment stages, down to task-specific in-context learning. However, with current privacy accounting, we cannot report an end-to-end privacy guarantee of the final model. The two main obstacles are that (1) the different training stages may protect different privacy units, i.e., they are defined with respect to different neighborhood relations, such as fixed-length token sequences vs. task examples, or documents vs. users, and that (2) the training data overlap between the various stages might be unknown. Since differential privacy guarantees are defined with respect to their own neighborhood relation, individual per stage guarantees cannot be composed directly, and standard composition results require either perfect knowledge of data overlap or pessimistically assuming complete overlap of data between all stages. We close this gap with the first formal framework for end-to-end privacy guarantees under heterogeneous privacy units and unknown but constrained data overlap. Concretely, we formalize each stage as a private mechanism with a stage-specific neighborhood relation and derive tools that give a guarantee for the end-to-end mechanism under generic constraints on data overlap. We show that our bounds are black-box optimal in several cases, and that, surprisingly, converting to a smaller privacy unit, even for DP-SGD, can weaken the guarantee. Turning to practice, we study end-to-end accounting for Gaussian DP mechanisms, showing that our bound is easy to compute even with a large number of stages, and that the most expensive stage(s) dominate the final privacy cost. Our advances enable rigorous end-to-end privacy guarantees for modern multi-stage foundation model training pipelines.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Differential privacy, Multi-stage Training, Foundation Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZGCVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies providing an end-to-end privacy guarantee with a target relationship that can be represented by the actual neighborhood relationships. It approaches to the problem by having privacy guarantees for individual stages how one can achieve a guarantee for the whole system as a single stage.

Using the f-DP definitions and the tools for the composition for this definition, they have derived a way to end up with a privacy guarantee for all. Also, they have analyzed their approaches' optimality stating that their approach is optimal (convert-then-compose) when the conversion and the composition is optimal already. Finally, they claimed the privacy unit conversion theorem they have can not be improved further and state that unit conversion is basically group-privacy.

They have numerical experiments supporting their corollaries at the end.

Critical strengths
  • [S1] I think the idea stated in the paper is really important and interesting.
  • [S2] Given examples in the paper clarifies what the paper tries to achieve.
Critical weaknesses
  • [W1] Even though the problem definition is really interesting the methodology is I believe modest. The proofs mainly use already existing tools and combine them. The optimality results sounds as if the convert-then-compose methodology is optimal when the conversion and composition is optimal however this does not say anything on whether convert-then-compose is the theoretical pathway one should consider to have an end-to-end guarantee.
  • [W2] The idea is important however it resembles already existing works on different privacy definitions and perspectives such as Pufferfish privacy where the conversion resembles the definition of the secrets in that case.
  • [W3] The narrative on the paper mention about the privacy on LLMs and provide use cases from that perspective however there is no actual relationship between LLM privacy and the papers contribution. Instead I would expect to see more real life use cases on this. Following this there might be synthetic and/or real world experiments representing those use cases.
Questions for authors
  • [Q1] Can you explain what exactly Theorem 5.2 contributes to the paper if that can be seen as group privacy?
  • [Q2] I could not really catch the part where the target unit is actually defined as a more fine-grained feature, in that case what would be the values of "a(u)"s?
  • [Q3] Can you have an interesting real-world scenario and actually prove that a final end-to-end privacy guarantee matters?
  • [Q4] Is there a way to enforce the setup to follow the assumptions and what would be the effects of them?
4 comments

Thank you for the review. We are happy you recognize that the idea of the paper is “important and interesting”. We have responded to your questions and comments below. We have also added a numerical experiment of end-to-end accounting of pretraining + SFT + DPO and updated the paper pdf. We have highlighted new and changed text with blue.

[W1] Methodology is modest

It is true that our proofs mainly combine existing tools, but we are the first to combine them to solve this problem, despite the fact that our problem is important (agreed by Reviewer HRdW).

Optimality result doesn’t say whether convert-then-compose is the best theoretical pathway

The optimality result shows that it is not possible to obtain tighter end-to-end privacy guarantees than those convert-then-compose gives. This means that one should not look for other pathways in order to obtain tighter bounds. Of course, there may be alternative pathways that give the same guarantees, but improve on some other axis such as computational cost. In summary, while the result does not prove that convert-then-compose is the best path in all possible ways, it shows that we cannot hope for improvements on the certified privacy bound, which is arguably the most important axis.

[W2] Connection to Pufferfish privacy

We do not claim that we are the first to consider that different privacy units can be used with differential privacy, and we agree that the secrets in Pufferfish privacy generalise the notion of privacy unit. Our core contribution is end-to-end accounting for multi-stage pipelines, which is orthogonal to Pufferfish: end-to-end accounting could be studied in Pufferfish as we study it in DP, though we are not aware of any work doing so. We chose to focus on DP, since it is the most common privacy framework in machine learning at the moment.

[W3] Use cases with connection to LLMs

We have added a privacy accounting experiment on a pipeline of VaultGemma pretraining + SFT + DPO on OASST2 data to Section 6.2. If overlap between OASST2 and pretraining data can be ruled out, we obtain an up to 15% decrease in the end-to-end μ\mu compared to full overlap, depending on what the pretraining contribution cap is. This shows that significant privacy budget savings are possible when overlap can be ruled out, so DP pretrained LLM providers should provide documentation to support determining whether there is overlap.

[Q1] Can you explain what exactly Theorem 5.2 contributes to the paper if that can be seen as group privacy?

Theorem 5.2 includes the K=1K = 1 case, which is not covered by group privacy. It also summarises how group privacy (and the K=1K = 1 case) apply in our problem.

[Q2] I could not really catch the part where the target unit is actually defined as a more fine-grained feature, in that case what would be the values of "a(u)"s?

This is considered in Appendix B, where we look at conversions between different sequence-length privacy units. The values of a(u)a(u) can be read from the appropriate table in Theorem B.1. Note that the tables give values for the worst-case user, which is why they use KK to denote the value, but the same logic can be applied to an individual user.

[Q3] Can you have an interesting real-world scenario and actually prove that a final end-to-end privacy guarantee matters?

Marek (2025, Section 4.5) studied how pretraining influences the empirical privacy leakage from differential private adaptations, like fine-tuning. They found that at the same formal privacy guarantee, differential private adaptations leak significantly differently, depending on the distributional closeness of pretraining and adaptation data.This motivates that an end-to-end guarantee is needed to properly characterize and bound privacy leakage in multi-stage foundation model training pipelines. We have added a citation to this paper as additional motivation. Their findings would have a significant impact if pretraining and finetuning on for example medical data.

We are not aware of any actual cases where data had leaked due to improper handling of end-to-end privacy. However, for privacy, it is necessary to anticipate and prepare beforehand, because leaked data can never be “unleaked”.

Reference:

  • Marek, Bartłomiej, Lorenzo Rossi, Vincent Hanke, et al. “Benchmarking Empirical Privacy Protection for Adaptations of Large Language Models.” ICLR 2025.

[Q4] Is there a way to enforce the setup to follow the assumptions and what would be the effects of them?

In the LLM pipeline experiment we added, we use OASST2 data containing user ids, which allow us to bound the number of contributions users can make by removing enough data that all users are under a set cap. In our setup, we still need to make assumptions on pretraining, since the we do not know the details of the pretraining data. In practice, this can be avoided by providing an interface or proper documentation about the pretraining data.

As the rebuttal period is coming to an end, we would like to thank the Reviewer once again for their constructive feedback. We have made substantial updates to address their comments and questions:

  • A numerical experiment of end-to-end accounting of an LLM training pipeline. We applied our privacy accounting theory to a real-world pipeline of VaultGemma pretraining, followed up by SFT+DPO on OASST2 data. OASST2 contains user ids, so it is possible to bound user contributions on that dataset. We find that if overlap between pretraining data and OASST2 can be ruled out, significant savings on the end-to-end privacy bound are possible (up to 15%).
  • A privacy audit of the LLM training pipeline. We run a GDP privacy audit on a similar pipeline, with continued DP pretraining of Pythia-160M replacing VaultGemma. The audit shows that end-to-end privacy leakage exceeds the per-stage bound (at 80% confidence), so reporting only stagewise bounds underestimates the privacy cost.
  • Scenario where end-to-end guarantee matters. Our audit shows that the end-to-end privacy leakage can exceed the stagewise privacy bounds. We also added a citation to a paper that found the distributional closeness of pretraining and adaptation data significantly affects privacy leakage as an additional motivation.
  • Clarifications. We clarified the interpretation of the convert-and-compose optimality result, the relationship of our theory to Pufferfish privacy, the significance of Theorem 5.2, and the relationship between a(u)a(u) and the target privacy unit.

We believe these additions significantly strengthen the paper, and would be happy if the Reviewer increased their score as a result.

Thank you for the clarifications and your responses.

I understand that the convert-then-compose approach is optimal when each component is considered/assumed to be optimal. I also understand from the black-box perspective this can be considered the most optimal answer. However, I think an analysis on how converting is optimal in different scenarios would be great for the paper. As far as I understand with the current setup only the relatively simple case is reported as an experiment where the conversions can be seen as groups belonging to a user or a component by simply counting the subcomponents.

I acknowledge that though as an initial idea and a mathematical framework it can be considered as a good contribution therefore I will increase my recommendation to 3.

Thanks again for the discussion.

We would like to sincerely thank the reviewer for their helpful feedback and the score increase. We will look into the optimality question for more complicated scenarios for the camera-ready version. If there are any other concerns that are only partially resolved, we would be happy to address them in the camera-ready version.

Reviewer_VUUhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is mainly about deriving theoretical mathematical bounds for the case of applying differential privacy principles to multi-stage foundation model training. By multi-stage, each stage can be pretraining, instruction tuning, and so forth. And each stage may also have the respective datasets it learns from. They may also have different privacy units (sequences vs. examples vs. documents). The theoretical mathematical bounds for a single stage are known, but when we apply multiple training stages, each having its own dataset, how are these bounds affected? This is the main problem the paper attempts to address.

They achieve this by using the f-DP framework (from Dong et. al. 2022) and applying some assumptions, mainly that the contributions related to a particular privacy unit and each training stage are bounded by some finite amount, each privacy unit can only appear in a finite number of stages, and the total contribution across all stages is bounded by some finite amount. Under these constraints, the sequential composition of the worst-case privacy unit works as the privacy guarantee. This is the main contribution of the paper. There are additional contributions related to the tightness of the bounds and converting between different privacy units.

There is also a numerical evaluation of the bounds, suggesting that training stages with large privacy cost will dominate the overall privacy cost. So when improving privacy for a multi-stage training pipeline, effort should be focused on these training stages.

Critical strengths
  1. The core mathematical argument is solid. The inclusion of the small example 1 note in the paper was useful for understanding some of the mathematical terms. There are also helpful notes in the Appendix section of the paper.
Critical weaknesses
  1. The experiment section is very confusing. It really is a numerical evaluation of the derived bounds. However, the experiment wording suggests there were some practical experiments conducted to evaluate the privacy costs of multi-stage foundation model training.
  2. In the experiment section itself, it suggests stage-wise contribution caps. But where did these stage-wise contribution caps even come from? The paper itself mentions plausible, but it is not clear how are this numbers plausible.
  3. Since the experimental section is weak here, it is hard to evaluate the real-world applicability of the mathematical results.
Questions for authors
  1. Could the authors complement Sections 5–6 with a membership inference audit on a small multi-stage pipeline, comparing an empirical lower bound on privacy loss with the theoretical upper bound under the same release setting?
  2. Does the end-to-end guarantee assume that the contribution bounds in Eq. (3) and the tokenizer in Appendix A.2 are fixed independently of the private data? If these are learned from private data, or if cross-stage deduplication changes the stage datasets, what conditions are needed for the guarantee to remain valid?
5 comments

Thank you for the review. We are happy you recognize the mathematical argument is solid, and the examples clarify the mathematical terms. We have responded to your comments and questions below. We have also added a numerical experiment of end-to-end accounting of pretraining + SFT + DPO and updated the paper pdf. We have highlighted new and changed text with blue.

W1 The experiment section is very confusing. It really is a numerical evaluation of the derived bounds. However, the experiment wording suggests there were some practical experiments conducted to evaluate the privacy costs of multi-stage foundation model training.

We have added a numerical experiment of end-to-end accounting of an LLM training pipeline to Section 6.2. We also have changed the name of the section to “Numerical Evaluation”.

W2 In the experiment section itself, it suggests stage-wise contribution caps. But where did these stage-wise contribution caps even come from? The paper itself mentions plausible, but it is not clear how are this numbers plausible.

In the grid experiment (now in Section 6.1), we set the caps to a decreasing sequence to reflect the fact that the early-stage datasets are often large and there may be little information about them.

In the LLM pipeline experiment (Section 6.2), we set the caps of the SFT and DPO stages so that 10%, 1% or 0.1% of users are above the cap. We use the OASST2 dataset containing user ids, which makes this possible. After setting the cap, we remove data such that all users respect the cap. We still need to assume a pretraining cap, since we have very little information on the pretraining data, so we sweep over a large range of values, 1 to 10 000.

W3 Since the experimental section is weak here, it is hard to evaluate the real-world applicability of the mathematical results.

Our LLM pipeline experiment strengthens the experiments. We observe that if overlap between OASST2 and pretraining data can be ruled out, we obtain an up to 15% decrease in the end-to-end μ\mu compared to full overlap, depending on what the pretraining contribution cap is. This shows that significant privacy budget savings are possible when overlap can be ruled out, so DP pretrained LLM providers should provide documentation to support determining whether there is overlap.

Q1 Could the authors complement Sections 5–6 with a membership inference audit on a small multi-stage pipeline, comparing an empirical lower bound on privacy loss with the theoretical upper bound under the same release setting?

We are planning to implement a one-run privacy audit on a similar pipeline as the one we have added (Section 6.2).

Q2 Does the end-to-end guarantee assume that the contribution bounds in Eq. (3) and the tokenizer in Appendix A.2 are fixed independently of the private data? If these are learned from private data, or if cross-stage deduplication changes the stage datasets, what conditions are needed for the guarantee to remain valid?

Yes, the contribution bounds and tokenizer need to be fixed independently of the private data. Learning them from private data would require significant theoretical generalisation of DP, since no DP definition can handle a privacy unit that depends on private data. This would be the case even if they were learned under DP, since it is not clear whether a composition where one stage can affect the privacy unit of the other is valid. For the tokenizer, this only applies if the privacy unit depends on the tokenizer, for example sequence-level units.

Cross-stage deduplication can be handled with our theory, since it only changes the contribution bounds for the post-deduplication stages.

AuthorsOpenReview ↗

Auditing experiment completed

We have completed the auditing experiment we planned earlier. We ran a white-box audit of a pipeline consisting of continuing the pretraining of Pythia-160M under DP on the FineWeb-10BT dataset and DP SFT + DPO on OASST2 dataset, using the same contribution bounding our earlier experiment in Section 6.2 used.

The audit constructs canary users separately for continued pretraining and SFT + DPO, with 1000 users in each group. This simulates a setting where pretraining data and OASST2 are disjoint. We construct white-box gradient canaries, where each user is assigned a parameter of the model, and their examples inject a gradient of clipping norm size to that coordinate.

We use the f-DP auditing process of Mahloujifar et al. (2025). We present the results in the table below, see the updated paper for details (Appendix E.3, highlighted in orange). The audit shows that the SFT+DPO users have a total μSFT+DPO≥1.084\mu_{\mathrm{SFT+DPO}} \geq 1.084 (at 80% confidence level) that is larger than the per-stage μSFT=μDPO=0.936\mu_{\mathrm{SFT}} = \mu_{\mathrm{DPO}} = 0.936, meaning that reporting just the per-stage bound underestimates the privacy cost. For pretraining, the audit got very little signal. This is not surprising, since we had to limit the sample rate and number of iterations for pretraining due to its computational cost so much that one run only samples 4% of the data and the included canaries.

Canaries inAudited afterFraction correctμ ≥ (95%)μ ≥ (80%)Accounting μ ≤
pretraining onlypretraining0.5370.0000.0030.936
SFT and DPOSFT0.9210.6890.7990.936
SFT and DPODPO0.9710.9391.0841.324

As we have now completed the experiments the reviewers asked for, we kindly ask the reviewers to consider raising their scores.

Reference:

Mahloujifar, Saeed, Luca Melis, and Kamalika Chaudhuri. “Auditing f-Differential Privacy in One Run.” ICML 2025

As the rebuttal period is coming to an end, we would like to thank the Reviewer once again for their constructive feedback. We have added the following to address their comments and questions:

  • A numerical experiment of end-to-end accounting of an LLM training pipeline. We apply our privacy accounting theory to a practical pipeline of VaultGemma pretraining, followed up by SFT+DPO on OASST2 data. OASST2 contains user ids, so it is possible to bound user contributions on that dataset. We find that if overlap between pretraining data and OASST2 can be ruled out, significant savings on the end-to-end privacy bound are possible (up to 15%).
  • A privacy audit of the LLM training pipeline. We run a GDP privacy audit on a similar pipeline, with continued DP pretraining of Pythia-160M replacing VaultGemma. The audit shows that end-to-end privacy leakage exceeds the per-stage bound (at 80% confidence), so reporting only stagewise bounds underestimates the privacy cost.
  • Clarified that contribution bounds and tokenizer are fixed.

We believe these changes substantially strengthen the paper, especially the experimental portion. We would be happy if the Reviewer would increase their score as a result.

Reviewer_HRdWOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates end-to-end privacy accounting for multi-stage training pipelines. In such pipelines, different privacy units are employed at each stage, and while the extent of data overlap between stages is unknown, it remains subject to certain constraints. By leveraging the concept of "group privacy," we map the neighborhood relations specific to each stage onto a unified target relation, subsequently aggregating the overall privacy guarantees based on worst-case contribution distributions. Our results are formulated within the ff-differential privacy (ff-DP) framework. Specifically, by employing Gaussian Differential Privacy (Gaussian DP), we demonstrate that it is possible to maximize a weighted quadratic objective function while simultaneously satisfying constraints on individual stages, total contributions, and inter-stage data overlap. We derive a structural property for the Gaussian DP optimization problem, examine conditional black-box optimality and the tightness of the transformation, and construct a "zeroing-out" counterexample to show that privacy guarantees may not be preserved when mapped to smaller units. Numerical experiments illustrate the sensitivity of privacy guarantees to contribution constraints and confirm the computational feasibility of this approach for training pipelines comprising up to 15 stages.

Critical strengths
  1. When the units protected vary across different stages, privacy metrics for each stage cannot be compared directly. Privacy accounting must also account for whether a target unit participates in multiple stages. Clarifying these dependencies addresses a practical gap in how privacy guarantees for foundation models are currently reported.

  2. Section 4.1 introduces a framework based on "feasible contribution vectors" that integrates privacy transformation and composition mechanisms, without requiring prior assumptions about whether units are entirely overlapping or completely independent. The constraints imposed on per-stage contributions, total contributions, and the number of affected stages are not only clearly interpretable but also encompass various known edge cases.

  3. Corollary 4.3 formulates a specific quadratic optimization problem; Proposition 4.4 further establishes that the optimal solution to this problem contains at most one non-zero coordinate that has not reached its upper bound. This property effectively narrows the search space, enabling the use of exact algorithms for scenarios with a moderate number of stages.

Critical weaknesses
  1. The aforementioned guarantees require the determination of reliable bounds—AtA_t, rr, and BB—for each target unit. While organizational policies or data provenance information might provide such bounds, actual pre-training data often contain duplicates, repackaged records, user data with ambiguous attribution, and contributions introduced during preprocessing. Although sensitivity analysis is useful, upper bounds established based on reasonable assumptions are not equivalent to rigorously certified upper bounds. Providing a concrete example of a pipeline that demonstrates how these bounds are implemented—or derived in a conservative manner—would significantly strengthen the practical contribution of this research. If valid constraints cannot be obtained, the framework defaults to "pessimistic composition"; this limitation should be explicitly acknowledged.

  2. The μ\mu values ​​selected in Section 6 are intended to approximate previously published or commonly used ϵ/δ\epsilon/\delta values. In general, a single pair of ϵ/δ\epsilon/\delta values ​​is insufficient to fully construct a GDP (Gaussian Differential Privacy) trade-off curve. Applying GDP-based reasoning to a practical pipeline requires valid GDP bounds for each stage, or a sound method for transforming or deriving these bounds using more granular privacy accounting information. The examples provided in the text should be clearly framed as "scenario analyses"; an end-to-end example based on "certified stage-wise curves" would provide a more compelling demonstration of the method's direct applicability in real-world settings.

  3. Proposition D.2 assumes optimal transformations and employs tight sequential composition for the transformed mechanisms. This is a conditional result rather than an unconditional theorem regarding "tightness" for all "constrained-overlap" scenarios covered in Theorem 4.1. While "smaller-to-larger" Gaussian constructions establish certain meaningful special cases, they should not be interpreted as proving that every entry in the "neighborhood-conversion table" or every "feasible-overlap envelope" achieves tightness. The abstract and the statement of contributions should explicitly clarify which specific scenarios are indeed optimal.

  4. Section 3 defines the mechanism as releasing all intermediate models, even though, in practice, one is typically only concerned with the final model. Since the "transcript guarantee" can also protect the final model via post-processing, this is a reasonable and conservative choice. However, for certain mechanisms, "final-only privacy" might be more robust. Furthermore, the tightness demonstrated by a "black-box" privacy-consumption transcript does not necessarily imply that a specific training process—focused solely on the final result—achieves the same level of tightness. This distinction should be clearly noted when discussing optimality and practical deployment.

  5. In the worst case, the computational complexity of exact enumeration remains exponential, depending on the number of stages and the overlap bound. For many standard training workflows, 15 stages represent a reasonable range; however, scenarios involving frequent adaptive adjustments, continuous updates, or inputs from multiple data sources could significantly increase the scale of the problem. Additionally, general ff-DP (differential privacy based on ff-divergence) results currently lack equally mature numerical computational support; how to implement these—perhaps via tradeoff functions or composition accounting based on privacy loss distributions (PLD)—remains a topic for future research. The scope of the framework’s applicability should be clearly defined when drawing conclusions.

  6. The current experiments rely on manually set contribution caps and privacy parameters for each stage, rather than conducting an audit tailored to specific multi-stage training workflows. Conducting small-scale, reproducible case studies is sufficient; there is no need to retrain large foundation models. The practical argument for the framework could be further strengthened by demonstrating its improvements over "full-overlap accounting" within a realistic training workflow, validating the effectiveness of the optimizer through small-scale examples (including comparisons against exhaustive search results), and providing detailed documentation on numerical precision. These steps are intended to refine the validation process, rather than implying that the absence of large-scale model training experiments negates the work's theoretical contributions.

Questions for authors
  1. Following data tokenization, packing, deduplication, and reuse, how should practitioners derive or determine the values ​​for AtA_t, rr, and BB? Are there concrete examples that provide evidence-based (rather than purely hypothetical) bounds?

  2. Are the values ​​of μ\mu in Section 6 intended merely as illustrative scenarios? To translate published ϵ/δ\epsilon/\delta guarantees into valid inputs for Corollary 4.3, what specific information regarding the "stages" is required?

  3. Beyond the case of "full-overlap composition" combined with "optimal conversions," what other optimality results hold? Could the authors clarify the distinctions between the following concepts: the optimality of a "convert-then-compose" strategy under certain conditions, the tightness of Gaussian conversion, and the tightness results established for the general "constrained-overlap" setting?

  4. How would the results change if only the final model were released? Please clarify whether the optimality results discussed in the paper pertain to the full training transcript, the final output (which might encode the training process), or a specific training mechanism designed for a "final-model-only" release.

  5. Regarding Algorithm 1, are there any correctness verifications or implementation details available for small-scale instances? At what stage counts and overlap bounds does this method become infeasible? Furthermore, is it possible to handle the "stage curves" encountered in practical applications without assuming Gaussian Differential Privacy (GDP)?

3 comments

Thank you for the review and recognizing that our work “addresses a practical gap in how privacy guarantees for foundation models are currently reported.”. We have responded to the comments and questions below. We have also added a numerical experiment of end-to-end accounting of pretraining + SFT + DPO and updated the paper pdf. We have highlighted new and changed text with blue.

W1, Q1 Providing a concrete example of a pipeline that demonstrates how these bounds are implemented

We have added privacy accounting for a pipeline of VaultGemma pretraining, followed by SFT and DPO, both on the OASST2 dataset, see Section 6.2 of the updated paper. If overlap between OASST2 and pretraining data can be ruled out, we obtain an up to 15% decrease in the end-to-end μ\mu compared to full overlap, depending on what the pretraining contribution cap is. This shows that significant privacy budget savings are possible when overlap can be ruled out, so DP pretrained LLM providers should provide documentation to support determining whether there is overlap.

W2, Q2 Selection of μ\mu values based on (ϵ,δ(\epsilon, \delta)-bounds.

Yes, they are intended as illustrative scenarios. They are not intended as claims that (ϵ,δ)(\epsilon, \delta)-guarantees could be translated to GDP. We picked the μ\mu values based on (ϵ,δ)(\epsilon, \delta)-pairs since (ϵ,δ)(\epsilon, \delta) is how privacy guarantees are often reported in the literature. The specific ϵ\epsilon values ϵ∈{1,3,5,8}\epsilon \in \{1, 3, 5, 8\} we chose are common choices (Duan et al. 2023), and the δ\delta is the same as VaultGemma (Sinha et al. 2025).

W3, Optimality result clarification

The requirement of optimal transformation in Proposition D.2 is necessary, since any convert-then-compose instantiation that does not use optimal transformations can clearly be improved by using the optimal transformations.

The abstract and contribution statements do not claim that we have optimality results for every case. Explicitly listing all the specific scenarios as suggested by the reviewer would make both of them too detailed.

W4, Q4 Releasing only the final model

The optimality results assume all of the models are released. We have added a note that our tightness results assume all models are released to the beginning of Section 5, which is reasonable in our setting where model checkpoints are often released publicly. While it may be theoretically possible to get improvements if only the final model is released, this is unlikely in practice. All of the results where only releasing the final model improves privacy bounds for training a single stage require unrealistic (for modern models) assumptions on the loss and model architecture (Chourasia et al. 2021, Ye and Shokri 2022, Annamalai 2024).

References:

  • Chourasia, Rishav, Jiayuan Ye, and Reza Shokri. “Differential Privacy Dynamics of Langevin Diffusion and Noisy Gradient Descent.” NeurIPS 2021.
  • Ye, Jiayuan, and Reza Shokri. “Differentially Private Learning Needs Hidden State (Or Much Faster Convergence).” NeurIPS 2022.
  • Annamalai, Meenatchi Sundaram Muthu Selva. “It’s Our Loss: No Privacy Amplification for Hidden State DP-SGD With Non-Convex Loss.” arXiv:2407.06496. 2024.

W5 Adaptive adjustment and continual updates (see Q5 for general f-DP support)

We have added a note to the beginning of Section 3 that we assume the training setup is fixed, and leave adaptive updates and continual learning to future work.

W6 Documentation on numerical precision (see W1 concrete example and Q5 validating optimizer)

We run all calculations with 64-bit floating point numbers. We have added this to Appendix E.

Q3 What other optimality results hold, clarify distinctions between optimality results?

The optimality results for convert-then-compose shows that if optimal conversion and composition are used, then combining them gives the optimal end-to-end bound. Assuming optimality for both is trivially required, since if one is not optimal, replacing it with the optimal version is an improvement. The conversion results show that the conversion bounds cannot be improved without mechanism-specific analyses. This implies that they are the optimal conversion for the mechanism in their counterexamples, the Gaussian mechanism.

Q5, How scalable is Algorithm 1, is it possible to handle general f-DP numerically?

Figure 2(d) shows that it is possible to run Algorithm 1 over the whole grid of potential rr and BB values with 15 stages. Running it just once would be possible for even more stages.

Handling general trade-off curves in Theorem 4.1 would require a numerical implementation of the f-DP group privacy operation Γk\Gamma_k. We are not aware of any published implementation of that.

Q5, Correctness verification of Algorithm 1

The following Python code contains a small example of running Algorithm 1 that can be easily checked manually. The code can be placed in the constrained_overlap.ipynb notebook in the supplementary code and run there. It is easy to verify that in the first case with r=1r = 1, it is optimal to fill the first stage, resulting in μ=4\mu = 4. In the second case with r=2r = 2, it is optimal to put two contributions in the first stage and one in the second stage, so μ=17\mu = \sqrt{17}.

import numpy as np
from constrained_overlap_optimization import find_worst_case_mu

stagewise_caps = np.array([2, 3])
stagewise_mus = np.array([2.0, 1.0])
max_stages = 1
max_contributions = 3
max_objective, _, _ = find_worst_case_mu(stagewise_caps, stagewise_mus, max_stages, max_contributions)
print(f"Max objective: {max_objective}")

max_stages = 2
max_objective, _, _ = find_worst_case_mu(stagewise_caps, stagewise_mus, max_stages, max_contributions)
print(f"Max objective: {max_objective}")
print(f"sqrt(4^2 + 1^2): {np.sqrt(17)}")

As the rebuttal period is coming to an end, we would like to thank the Reviewer once again for their constructive feedback. We have added the following based on their suggestions:

  • A privacy accounting experiment for a pipeline of VaultGemma pretraining, followed by SFT and DPO, both on the OASST2 dataset. We use the user ids in the OASST2 dataset to bound user contributions to that dataset, and our results show that if overlap between pretraining data and OASST2 can be ruled out, significant privacy budget savings (up to 15%) are possible.
  • Correctness verification of Algorithm 1. We have provided a short code snippet that tests Algorithm 1 in a simple setting that is easy to verify manually.
  • Additional clarifications. We have clarified the scope of our optimality results, numerical precision, selection of μ\mu values, the scalability of Algorithm 1, the possibility of handling f-DP numerically, potential for adaptive adjustments or continual updates, and discussed whether releasing only the final model could improve privacy bounds.

We believe these updates have substantially improved the paper and would be happy if the Reviewer would update their score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to reviews

We would like to thank the reviewers for their work. They agree the problem we study is significant (Reviewers HRdW and ZGCV), and the theory is solid (all reviewers), covering various known edge cases (Reviewer HRdW), and that the examples make the framework easy to follow (VUUh, ZGCV). Reviewer HRdW highlights that our work “addresses a practical gap in how privacy guarantees for foundation models are currently reported.”
During the rebuttal, we made significant progress by practically addressing the Reviewers’ comments. We indicate the changes made in the paper through colored font, and present the rebuttal highlights below:

Highlight 1: Practical Experiments with real-world LLM use-case: End-to-end accounting for a DP LLM pipeline. We account for a pipeline of DP pretraining (VaultGemma, sequence-level), DP-SFT and DP-DPO on OASST2. To our knowledge, this is the first end-to-end privacy guarantee reported for a DP-trained LLM across pretraining and alignment. We have added an experiment that applies Corollary 4.3 to end-to-end privacy accounting a training pipeline with pretraining, supervised finetuning (SFT) and direct preference optimization (DPO). We use VaultGemma (Sinha et al. 2025) as the model, which was pretrained under DP. We consider the OASST2 dataset for SFT and DPO, which contains user ids that allow us to bound the number of contributions each user can make in that dataset. We have added results and further details to the paper, see Section 6.2 and Appendices E.2 and G. Related additions and modifications are highlighted in blue.

Highlight 2: Contribution bounds derived from data and enforced: OASST2 contains user IDs, so the SFT and DPO contribution caps are derived from the data (set so that 10%, 1% or 0.1% of users exceed them) and enforced by removing contributions above the cap. This shows concretely how the constraints of Theorem 4.1 can be obtained and enforced in practice.

Highlight 3: Ruling out overlap yields substantial savings: We observe that if overlap between OASST2 and pretraining data can be ruled out, we obtain an up to 15% decrease in the end-to-end μ\mu compared to full overlap, depending on what the pretraining contribution cap is. This shows that significant privacy budget savings are possible when overlap can be ruled out, so DP pretrained LLM providers should provide documentation to support determining whether there is overlap.

Highlight 4: Practical pivacy audit confirms the need for end-to-end accounting: Following the suggestion of Reviewers VUUh and HRdW, we audited a multi-stage pipeline of DP continued pretraining, DP-SFT and DP-DPO with white-box canaries. For users who contribute to several stages, the measured privacy leakage exceeds what the per-stage guarantees allow. This shows that reporting per-stage guarantees understates the actual privacy cost. At the same time, the leakage stays below our end-to-end bound, which the audit confirms is valid and reasonably tight (Appendix E.3). Related additions and modifications are highlighted in orange.

Further improvements:

  • We state the scope of all results explicitly, namely the release of all checkpoints, and a pipeline, contribution bounds and tokenizer fixed independently of the private data.
  • We added a correctness check for Algorithm 1.
  • We clarified the relation to Pufferfish privacy and added Marek et al. (2025), which shows empirically that pretraining affects the privacy leakage of DP adaptations.

We believe these additions have significantly strengthened the paper and would be grateful if the reviewers would take them into account in their final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.