- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposed a PAC-private diffusion model adaptation to directly achieve reconstruction privacy (RP) to overcome the limitation of DP-SGD that it is does not have a direct relationship to RP. A diffusion model private adaptation is done with PAC-privacy for sensitive image synthesis. It trains compact LoRA adapters or Textual Inversion embeddings and applies a single Gaussian perturbation after training, with anisotropic noise calibrated from the covariance. It also canonicalizes LoRA factors to reduce parameterization ambiguity. It tried to match the privacy budge between DP with PAC privacy which is later used for experimental comparisons. The reported results under their matching mechanism show improved subject fidelity and classification accuracy over DP-SGD.
Critical strengths
- The motivation is clear and promising. The observation that DP is not directly related to how much information is revealed before and after private training is a long-standing vital problem of the field of privacy. I am happy to see progress in this field.
- The experiments cover both personalization and downstream synthetic-data utility, which is really nice since it does not stays only on the "theoretical" part, but moves toward the "application" part.
Critical weaknesses
- My biggest concern is about the theoretical part. I did not reproduce the proof myself, but the claimed privacy-budget matching between PAC privacy and DP is insufficiently justified. The conversion matches reconstruction-success bounds only at the selected prior success probability . There are concerns to me: the tightness of these two upper bounds are not discussed, so they may or may not give the same constraint even when the value is equal; even when we assume that they exactly equal, one is matched to millions of different pairs of which forms a continuous "line". Consequently, I doubt that the utility comparisons is not conducted under equivalent reconstruction privacy.
- Another concern is about the theoretical proof. I did not reproduce the theorems and proofs myself, but I used GPT-6.1-Sol (extra-high) to give them a check. The LLM pointed out a major concern that I checked myself and believe to be real. On page 5, in paragraph Noise calibration., is drawn from private training dataset , and then noise is drawn from it in Theorem 1. This contradicts with the assumption also in Theorem 1 that the noise is independent of , the private dataset used for the final adaptation. If the same dataset is used in the experiment, I would completely question the soundness of the results. Correct me if I make any mistakes or misunderstandings here.
- Although PAC privacy is the key part of the paper, the formulation of it is never given throughout the paper, even with the formulation of DP given.
- I am bit concerned that the tasks used in the experiment part is too simple.
Questions for authors
See Weaknesses. Especially the theoretical part, which is hard be really vital given the topic and scope claimed by the paper.
3 comments
Thank you for the clarification. I agree that independent resamples from a fixed dataset can be independent. However, this does not resolve the central concern: the response changes the privacy model without establishing the guarantee claimed in the paper. The manuscript explicitly aims to protect the original sensitive training records (lines 151–165), formalizes privacy through (lines 180–182), and claims for in Theorem 1 (lines 262–269). Algorithm 1 instead calibrates noise using bootstrap samples from the private dataset (lines 220–228 and 233–235). Your explanation establishes independence conditional on . This can justify independence within a fixed-source resampling model, but it does not establish confidentiality of the source records themselves. If is the private object to be protected, drawing fresh Gaussian randomness does not remove the dependence of the noise distribution on that dataset. Please explicitly reconcile the protected variable, sampling distribution, and adversary’s prior across the privacy definition, theorem, and experimental procedure.
Furthermore, I agree with the first point raised in "Critical Weaknesses" by Reviewer 3PUr. Therefore, I believe the theoretical part of this paper needs deeper revisits. I recommend the authors use some cutting models for help in re-check and re-verification of the proposed theoretical part.
Response to the Follow-up
Thank you for the follow-up. We agree that the distinction between the protected secrets and their underlying probability distribution should be made explicit.
In our PAC Privacy formulation, the privacy guarantee concerns information leakage about the realized sensitive data, modeled as random variables under a specified input distribution, rather than the distribution itself. In particular, let denote the dataset used for the final adaptation, where specifies the underlying data distribution. The guarantee bounds the mutual information between and the released output, thereby covering the sensitive records contained in . This is a distributional privacy guarantee and is not tied to any particular realization of .
Our Monte Carlo calibration assumes access to a sampling procedure that generates calibration datasets independently from the same fixed input distribution . Crucially, the calibration procedure does not access the realized protected data in or any information derived from them. Since the calibration datasets and are sampled independently, the estimated noise covariance and the Gaussian perturbation generated from it are independent of . This establishes the independence condition underlying Theorem 1.
We will revise the algorithm description and privacy analysis to explicitly distinguish the realized protected secrets, their underlying probability distribution, and the independent calibration procedure. When an empirical distribution is constructed from a reference dataset, the independence argument requires that the reference information used for calibration be independent of the protected input. The stated guarantee does not automatically extend to records used to construct that empirical distribution.
Thank you also for considering Reviewer 3PUr's comments. We have addressed those concerns in our separate response.
Author Response
Thank you for recognizing the importance of studying reconstruction privacy and for appreciating our evaluation of both personalized generation and downstream synthetic-data utility. We also appreciate your close reading of the theoretical analysis and the questions it raised. Below, we address the budget conversion, noise calibration, PAC Privacy formulation, and experimental setting in turn.
Q1: Is the DP-to-PAC budget conversion justified when the reconstruction success bounds are matched at only one prior success rate?
Thank you for raising the question of non-uniqueness. The trade-off function fε,δ is an equivalent representation of the (ε,δ)-DP guarantee and thus provides a valid posterior reconstruction-success bound at each prior success rate. These are analytically derived upper bounds, not measurements of realized attack success. Privacy–utility comparisons ask which method achieves higher utility under the same stated privacy constraint. Here, we use the upper bound on posterior reconstruction success at α⋆ as that constraint: we choose the PAC mutual information budget to match the DP bound at this point, then compare utility of DP-SGD and PAC perturbation.
We agree that picking different points on the f-DP curve yields different values of KL divergence (β). Conversely, a fixed β is compatible with many prior–posterior success pairs and may be associated with different DP parameters as well.
Choosing a prior success rate arbitrarily gives a valid pointwise comparison, but may obscure differences in the strength of DP guarantees. For example, at α=0.1, the resulting KL budgets are 2.021, 2.252, and 2.302 for ε=3,5,10, respectively, despite the substantial differences in ε. We therefore do not choose the prior success rate arbitrarily. Instead, we choose the prior success rate where DP permits its maximum reconstruction advantage, and compute the corresponding KL budget. With δ fixed, β is strictly increasing in ε, giving a unique mapping under this selection rule. Thus, the conversion matches the reconstruction-success upper bounds at the chosen operating points and preserves the ordering of the DP budgets as ε increases.
Q2: Is ΣB independent of X⋆? Thank you for raising the independence question. Let D be the fixed dataset, and let Q(D) denote the distribution induced by our resampling rule. Calibration uses independent draws X(1),…,X(K)∼Q(D), while the final adaptation uses a fresh draw X⋆∼Q(D), independent of the calibration draws conditional on D. The estimated noise covariance depends only on the calibration runs, and the Gaussian noise is drawn afresh. Consequently, the added noise is independent of X⋆. Sampling from the same fixed distribution does not make the calibration and final draws dependent.
Q3: PAC Privacy definition.
Due to space constraints and readability, we presented the reconstruction advantage and mutual information formulation directly used in our analysis, while citing the original PAC Privacy paper for the full definition.
Q4: Experimental tasks.
Our evaluation builds on established private image-generation tasks: DPDM evaluates FashionMNIST and downstream classification using synthetic data, while DP-LoRA studies private latent-diffusion fine-tuning. We additionally evaluate EuroSAT, BloodMNIST, and few-shot object and face personalization to test the methods across different data domains and adaptation settings.
Thank you again for your careful review. We hope our responses clarify and address your concerns.