- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies providing an end-to-end privacy guarantee with a target relationship that can be represented by the actual neighborhood relationships. It approaches to the problem by having privacy guarantees for individual stages how one can achieve a guarantee for the whole system as a single stage.
Using the f-DP definitions and the tools for the composition for this definition, they have derived a way to end up with a privacy guarantee for all. Also, they have analyzed their approaches' optimality stating that their approach is optimal (convert-then-compose) when the conversion and the composition is optimal already. Finally, they claimed the privacy unit conversion theorem they have can not be improved further and state that unit conversion is basically group-privacy.
They have numerical experiments supporting their corollaries at the end.
Critical strengths
- [S1] I think the idea stated in the paper is really important and interesting.
- [S2] Given examples in the paper clarifies what the paper tries to achieve.
Critical weaknesses
- [W1] Even though the problem definition is really interesting the methodology is I believe modest. The proofs mainly use already existing tools and combine them. The optimality results sounds as if the convert-then-compose methodology is optimal when the conversion and composition is optimal however this does not say anything on whether convert-then-compose is the theoretical pathway one should consider to have an end-to-end guarantee.
- [W2] The idea is important however it resembles already existing works on different privacy definitions and perspectives such as Pufferfish privacy where the conversion resembles the definition of the secrets in that case.
- [W3] The narrative on the paper mention about the privacy on LLMs and provide use cases from that perspective however there is no actual relationship between LLM privacy and the papers contribution. Instead I would expect to see more real life use cases on this. Following this there might be synthetic and/or real world experiments representing those use cases.
Questions for authors
- [Q1] Can you explain what exactly Theorem 5.2 contributes to the paper if that can be seen as group privacy?
- [Q2] I could not really catch the part where the target unit is actually defined as a more fine-grained feature, in that case what would be the values of "a(u)"s?
- [Q3] Can you have an interesting real-world scenario and actually prove that a final end-to-end privacy guarantee matters?
- [Q4] Is there a way to enforce the setup to follow the assumptions and what would be the effects of them?
4 comments
As the rebuttal period is coming to an end, we would like to thank the Reviewer once again for their constructive feedback. We have made substantial updates to address their comments and questions:
- A numerical experiment of end-to-end accounting of an LLM training pipeline. We applied our privacy accounting theory to a real-world pipeline of VaultGemma pretraining, followed up by SFT+DPO on OASST2 data. OASST2 contains user ids, so it is possible to bound user contributions on that dataset. We find that if overlap between pretraining data and OASST2 can be ruled out, significant savings on the end-to-end privacy bound are possible (up to 15%).
- A privacy audit of the LLM training pipeline. We run a GDP privacy audit on a similar pipeline, with continued DP pretraining of Pythia-160M replacing VaultGemma. The audit shows that end-to-end privacy leakage exceeds the per-stage bound (at 80% confidence), so reporting only stagewise bounds underestimates the privacy cost.
- Scenario where end-to-end guarantee matters. Our audit shows that the end-to-end privacy leakage can exceed the stagewise privacy bounds. We also added a citation to a paper that found the distributional closeness of pretraining and adaptation data significantly affects privacy leakage as an additional motivation.
- Clarifications. We clarified the interpretation of the convert-and-compose optimality result, the relationship of our theory to Pufferfish privacy, the significance of Theorem 5.2, and the relationship between and the target privacy unit.
We believe these additions significantly strengthen the paper, and would be happy if the Reviewer increased their score as a result.
Thank you for the clarifications and your responses.
I understand that the convert-then-compose approach is optimal when each component is considered/assumed to be optimal. I also understand from the black-box perspective this can be considered the most optimal answer. However, I think an analysis on how converting is optimal in different scenarios would be great for the paper. As far as I understand with the current setup only the relatively simple case is reported as an experiment where the conversions can be seen as groups belonging to a user or a component by simply counting the subcomponents.
I acknowledge that though as an initial idea and a mathematical framework it can be considered as a good contribution therefore I will increase my recommendation to 3.
Thanks again for the discussion.
We would like to sincerely thank the reviewer for their helpful feedback and the score increase. We will look into the optimality question for more complicated scenarios for the camera-ready version. If there are any other concerns that are only partially resolved, we would be happy to address them in the camera-ready version.
Thank you for the review. We are happy you recognize that the idea of the paper is “important and interesting”. We have responded to your questions and comments below. We have also added a numerical experiment of end-to-end accounting of pretraining + SFT + DPO and updated the paper pdf. We have highlighted new and changed text with blue.
It is true that our proofs mainly combine existing tools, but we are the first to combine them to solve this problem, despite the fact that our problem is important (agreed by Reviewer HRdW).
The optimality result shows that it is not possible to obtain tighter end-to-end privacy guarantees than those convert-then-compose gives. This means that one should not look for other pathways in order to obtain tighter bounds. Of course, there may be alternative pathways that give the same guarantees, but improve on some other axis such as computational cost. In summary, while the result does not prove that convert-then-compose is the best path in all possible ways, it shows that we cannot hope for improvements on the certified privacy bound, which is arguably the most important axis.
We do not claim that we are the first to consider that different privacy units can be used with differential privacy, and we agree that the secrets in Pufferfish privacy generalise the notion of privacy unit. Our core contribution is end-to-end accounting for multi-stage pipelines, which is orthogonal to Pufferfish: end-to-end accounting could be studied in Pufferfish as we study it in DP, though we are not aware of any work doing so. We chose to focus on DP, since it is the most common privacy framework in machine learning at the moment.
We have added a privacy accounting experiment on a pipeline of VaultGemma pretraining + SFT + DPO on OASST2 data to Section 6.2. If overlap between OASST2 and pretraining data can be ruled out, we obtain an up to 15% decrease in the end-to-end μ compared to full overlap, depending on what the pretraining contribution cap is. This shows that significant privacy budget savings are possible when overlap can be ruled out, so DP pretrained LLM providers should provide documentation to support determining whether there is overlap.
Theorem 5.2 includes the K=1 case, which is not covered by group privacy. It also summarises how group privacy (and the K=1 case) apply in our problem.
This is considered in Appendix B, where we look at conversions between different sequence-length privacy units. The values of a(u) can be read from the appropriate table in Theorem B.1. Note that the tables give values for the worst-case user, which is why they use K to denote the value, but the same logic can be applied to an individual user.
Marek (2025, Section 4.5) studied how pretraining influences the empirical privacy leakage from differential private adaptations, like fine-tuning. They found that at the same formal privacy guarantee, differential private adaptations leak significantly differently, depending on the distributional closeness of pretraining and adaptation data.This motivates that an end-to-end guarantee is needed to properly characterize and bound privacy leakage in multi-stage foundation model training pipelines. We have added a citation to this paper as additional motivation. Their findings would have a significant impact if pretraining and finetuning on for example medical data.
We are not aware of any actual cases where data had leaked due to improper handling of end-to-end privacy. However, for privacy, it is necessary to anticipate and prepare beforehand, because leaked data can never be “unleaked”.
Reference:
In the LLM pipeline experiment we added, we use OASST2 data containing user ids, which allow us to bound the number of contributions users can make by removing enough data that all users are under a set cap. In our setup, we still need to make assumptions on pretraining, since the we do not know the details of the pretraining data. In practice, this can be avoided by providing an interface or proper documentation about the pretraining data.