- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies differentially private text-to-image generation using a publicly pretrained autoregressive spectral image model. DP-Wavelet defines private supervision over a fixed prefix of coarse-to-fine wavelet tokens. The Coarse configuration privately learns the lowest-frequency prefix and uses the frozen public model to complete the remaining tokens, while Full privately learns and generates the complete sequence. Generation and public completion preserve the training guarantee through DP post-processing. Experiments on MS-COCO and MM-CelebA-HQ compare diffusion and raster-autoregressive baselines, report quality across resolutions and training costs, and provide a membership-inference audit. The main finding is a tradeoff between coarse-scale quality, final-resolution FID, and training cost.
Critical strengths
-
The reported results demonstrate promising utility for private spectral image modeling. Full achieves the lowest final-resolution FID in three of the four private dataset–budget settings. The COCO results are particularly encouraging, with FID@256 of 8.9 and 10.3 at the reported budgets of and , respectively. Improvements over the same model’s pretrained performance also indicate that private adaptation contributes useful target-distribution information.
-
The same-backbone comparison provides an informative empirical tradeoff. Comparing Coarse and Full controls the pretrained architecture and shows that restricting supervision can improve coarse-scale FID and reduce training cost, while full-sequence adaptation improves final-resolution FID. Appendix E quantifies the training-cost difference on identical hardware, and Appendix F extends the quality evaluation to additional resolutions. These comparisons are useful for understanding the consequences of choosing the scope of private supervision.
Critical weaknesses
-
The demonstrated contribution of scale-selective private adaptation remains modest. SIT already provides the spectral representation, coarse-to-fine ordering, partial decoding, and autoregressive completion capabilities. The contribution here is their use with DP optimization and a selectable supervision prefix. This is a plausible new combination, but the strongest final-resolution results come from full-sequence adaptation. Coarse has a legitimate advantage at its native evaluation scale and saves approximately 13% of training time, while yielding worse final-resolution FID in every reported finetuned comparison. I find the empirical result useful, but remain insufficiently convinced that the demonstrated benefits of selecting the private supervision scope constitute a substantial advance.
-
Privacy calibration is insufficiently specified. The submission does not report the actual , noise multipliers, or training dataset sizes needed to reproduce the privacy accounting. Without these values and confirmation that the sampling implementation matches the accountant’s assumptions, the reported numerical privacy guarantees cannot be independently verified.
-
Cross-model differences limit attribution of the utility gains. The cross-model comparison combines differences in pretrained backbones, pretraining conditions, and trainable parameter counts, without reporting total and trainable parameter counts. The pretrained results and same-backbone Coarse/Full comparison provide useful controls, but do not fully isolate the contribution of spectral private adaptation. Reporting the parameter counts and relevant pretraining conditions would make the source of the reported gains easier to assess.
-
The chosen training horizons lack reported convergence evidence. Appendix E reasonably selects training horizons separately based on preliminary convergence, yet provides no supporting convergence evidence. Providing this evidence would help assess whether the baselines were adequately optimized under the stated privacy budgets.
Questions for authors
-
Privacy calibration (Weakness 2): Please report the training dataset size, , noise multiplier , and sampling rate for each private configuration, and confirm that the actual sampling procedure matches the PLD accountant’s assumptions.
-
Model size and pretraining conditions (Weakness 3): Please report total and trainable parameter counts for the compared configurations, and clarify the pretraining datasets behind the selected public checkpoints.
-
Baseline convergence (Weakness 4): Please provide the existing convergence evidence supporting Appendix E’s selected private baseline training horizons. Alternatively, a limited horizon-sensitivity check in one representative private setting, with noise recalibrated to preserve the same , would help address this concern.
-
Coarse-scale quality after public completion (Weakness 1): Using the existing generated samples, could you report FID@32 for DP-Wavelet-Coarse’s final outputs after public completion, resized to using the same procedure and reference-image preprocessing as for the other methods? How does this compare with the decoded coarse intermediary’s FID@32, and does Coarse retain its advantage when all methods are evaluated using resized final outputs?
2 comments
Rebuttal by Authors Continued
- The chosen training horizons lack reported convergence evidence.
Response: We agree that our training horizon selection could be better clarified. As discussed in our response to Reviewer 6uhv (Weakness 2), we selected the training horizons based on preliminary convergence of the training loss and, for DP-LDM, also intermediate FID evaluations because the loss was relatively noisy. The following table summarizes the loss values at different stages of training on MS-COCO with progress normalized by each method's total number of updates:
| Training progress | DP-LDM loss | DP-LDM FID@256 | DP-LlamaGen loss | DP-Wavelet loss |
|---|---|---|---|---|
| 25% | ~0.15 | 13.01 | 6.49 | ~6.30 |
| 50% | ~0.15 | 12.54 | 6.19 | ~6.28 |
| 75% | ~0.15 | 12.64 | 5.97 | ~6.25 |
| 100% | ~0.15 | 12.61 | 5.97 | ~6.24 |
| Total updates | 9,690 | — | 1,125 | 100,000 |
To more rigorously address the concern about differences in training horizons, we additionally ran a matched sample-exposure experiment for where each method processes 1,152,000 samples during training with accordingly recalibrated DP noise multipliers to maintain the same privacy budget. The results are:
| Dataset | DP-Wavelet-Coarse | DP-Wavelet-Full | DP-LDM | DP-LlamaGen |
|---|---|---|---|---|
| MS-COCO | 14.23 | 9.28 | 12.41 | 13.80 |
| MM-CelebA-HQ | 26.98 | 23.09 | 24.17 | 63.70 |
Under this comparison, DP-Wavelet-Full achieves the lowest FID@256 on both datasets, so its advantage over the baselines cannot be attributed solely to processing more training examples. We have included this comparison in Appendix F of the updated paper.
Questions:
- Please report the training dataset size, , noise multiplier, and sampling rate for each private configuration, and confirm that the actual sampling procedure matches the PLD accountant’s assumptions.
Response: As described in our response to Weakness 2, Appendix E now reports the dataset sizes, , noise multipliers, and sampling rates for each private configuration. We have also clarified consistent sampling assumptions for the PLD and PRV accountants in Appendix E.
- Please report total and trainable parameter counts for the compared configurations and clarify the pretraining datasets behind the selected public checkpoints.
Response: As discussed in our response to Weakness 3, we have added a table in Appendix E reporting the total and trainable parameter counts with the data used for public pretraining. Specifically, DP-Wavelet uses an AR-SIT checkpoint pretrained on WebLI, DP-LDM uses Stable Diffusion v1.4 pretrained on LAION-derived datasets, and DP-LlamaGen uses LlamaGen-XL pretrained on LAION-COCO.
- Please provide the existing convergence evidence supporting Appendix E’s selected private baseline training horizons. Alternatively, a limited horizon-sensitivity check in one representative private setting, with noise recalibrated to preserve the same , would help address this concern.
Response: We provide the training loss progression and intermediate FID evaluations that we used to assess convergence and inform selecting the training horizons in the response to Weakness 4. These show similar stabilization trends across the compared methods. We additionally report a matched sample-exposure experiment in Appendix F with noise recalibrated to preserve the same privacy budget, which provides complementary evidence that the performance differences are not solely explained by training duration.
- Using the existing generated samples, could you report FID@32 for DP-Wavelet-Coarse’s final outputs after public completion resized to using the same procedure and reference-image preprocessing as for the other methods? How does this compare with the decoded coarse intermediary’s FID@32, and does Coarse retain its advantage when all methods are evaluated using resized final outputs?
Response: Upon your recommendation and that of Reviewer 6uhv, we have updated the FID@32 evaluations to use the symmetric protocol where Coarse's final outputs are resized to before evaluation using the same procedure as for the other methods. Under this protocol, the trends are largely the same as when using the decoded coarse intermediary's FID@32. Specifically, DP-Wavelet-Coarse retains lower FID@32 than Full on MM-CelebA-HQ and the two demonstrate comparable quality on MS-COCO. Coarse also continues to outperform both DP-LDM and DP-LlamaGen at the coarse scale across all reported privacy budgets.
Rebuttal by Authors
Thank you very much for your thoughtful review and constructive feedback. We are pleased that you find our private image generation results promising, particularly on MS-COCO, and the same-backbone comparison informative. We address your comments below and hope this will help improve your evaluation of the paper.
Weaknesses:
Response: Thank you for highlighting this concern. It is a correct characterization that our contribution is to leverage the properties already provided by AR-SIT for private generation by restricting the DP finetuning objective to selected wavelet scales and studying how this choice affects image quality and training cost. To better demonstrate the value of selecting the supervision scope, we have extended the experiments from the original Coarse and Full endpoints to include two intermediate prefix lengths, for a total of four configurations Kp∈{256,512,768,1024} corresponding to supervision through the 32×32, 64×64, 128×128, and 256×256 scales. The updated results in Table 1 and Figure 2 provide a more complete characterization of the tradeoff that we study. In particular, Figure 2 illustrates that, while Full generally achieves the best final-resolution FID, shorter prefixes can approach this performance with lower per-update training cost. For example, on MM-CelebA-HQ at ε=10, supervision through 128×128 achieves FID@256 of 18.2 compared with 18.4 for Full and Coarse achieves a better FID@32. We have also added a prefix supervision and completion ablation in Table 2, which shows that coarse supervision captures substantial target-domain information and that the frozen public model can limit final-resolution quality when generating the remaining details. Together, these experiments demonstrate that supervising the entire sequence is not always necessary to obtain strong image quality and that selecting the supervision prefix provides a useful way to balance quality across resolutions and private finetuning cost without changing the pretrained architecture or requiring a separate super-resolution model.
Response: Thank you for identifying the missing privacy-accounting details. We have expanded the hyperparameter tables in Appendix E to report the numerical values of δ, calibrated noise multipliers σ, training set sizes, and sampling rates. We have also clarified the privacy accountants used by each method: DP-Wavelet and DP-LDM use PLD accounting under a fixed-size subsampling model and DP-LlamaGen uses Opacus's PRV accountant with Poisson sampling. Together with the reported batch sizes and training horizons, these details allow the privacy accounting for each configuration to be independently reproduced.
Response: Thank you for raising this point. We agree that differences in model capacity and pretraining can contribute to the observed performances and, for the sake of transparency, we have extended Appendix E to include the total and trainable parameter counts of each method in addition to the pretraining datasets used for their public checkpoints. As discussed in our responses to Reviewer 6uhv (Question 2) and Reviewer Myyp (Weakness 1), our comparisons across DP-Wavelet prefix lengths use the same pretrained AR-SIT checkpoint, which allows us to study the effect of restricting private supervision while keeping the architecture and pretraining fixed. The comparisons with DP-LDM and DP-LlamaGen complement these experiments by evaluating diffusion and raster-autoregressive approaches on the same datasets, privacy budgets, and evaluation protocol, with the additional model information providing context for differences in performance across architectures.