acceptodds
Under review as a conference paper at ICLR 2027

Decomposing Private Image Generation via Coarse-to-Fine Wavelet Modeling

Abstract

Generative models trained on sensitive image datasets have been shown to risk memorizing and reproducing individual training examples, making strong privacy guarantees essential. While differential privacy (DP) provides a principled basis for such guarantees, applying standard DP finetuning (e.g., with DP-SGD) to generative models often results in severe degradation of image quality. To address this, we introduce a spectral DP framework that uses wavelet representations to control which image scales are included in private finetuning, motivated by the hypothesis that dataset-specific information needed for adaptation, such as face geometry and object shape, can be captured by low-frequency wavelet components while fine-scale details can be supplied by a public prior. Specifically, we privately finetune a publicly pretrained autoregressive spectral image model to generate intermediaries at a chosen level of detail, then use the frozen public model to complete any remaining wavelet scales without additional privacy loss. Experiments with coarse-prefix and full-sequence configurations on MS-COCO and MM-CelebA-HQ demonstrate strong distributional image quality relative to diffusion and raster-autoregressive baselines. Full-sequence adaptation generally yields better final-resolution quality while coarse-prefix adaptation achieves stronger results at the coarse evaluation scale with lower training cost.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
differential privacy, image generation, transformer, wavelet

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YNgmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies differentially private text-to-image generation using a publicly pretrained autoregressive spectral image model. DP-Wavelet defines private supervision over a fixed prefix of coarse-to-fine wavelet tokens. The Coarse configuration privately learns the lowest-frequency prefix and uses the frozen public model to complete the remaining tokens, while Full privately learns and generates the complete sequence. Generation and public completion preserve the training guarantee through DP post-processing. Experiments on MS-COCO and MM-CelebA-HQ compare diffusion and raster-autoregressive baselines, report quality across resolutions and training costs, and provide a membership-inference audit. The main finding is a tradeoff between coarse-scale quality, final-resolution FID, and training cost.

Critical strengths
  1. The reported results demonstrate promising utility for private spectral image modeling. Full achieves the lowest final-resolution FID in three of the four private dataset–budget settings. The COCO results are particularly encouraging, with FID@256 of 8.9 and 10.3 at the reported budgets of ε=10\varepsilon=10 and 11, respectively. Improvements over the same model’s pretrained performance also indicate that private adaptation contributes useful target-distribution information.

  2. The same-backbone comparison provides an informative empirical tradeoff. Comparing Coarse and Full controls the pretrained architecture and shows that restricting supervision can improve coarse-scale FID and reduce training cost, while full-sequence adaptation improves final-resolution FID. Appendix E quantifies the training-cost difference on identical hardware, and Appendix F extends the quality evaluation to additional resolutions. These comparisons are useful for understanding the consequences of choosing the scope of private supervision.

Critical weaknesses
  1. The demonstrated contribution of scale-selective private adaptation remains modest. SIT already provides the spectral representation, coarse-to-fine ordering, partial decoding, and autoregressive completion capabilities. The contribution here is their use with DP optimization and a selectable supervision prefix. This is a plausible new combination, but the strongest final-resolution results come from full-sequence adaptation. Coarse has a legitimate advantage at its native evaluation scale and saves approximately 13% of training time, while yielding worse final-resolution FID in every reported finetuned comparison. I find the empirical result useful, but remain insufficiently convinced that the demonstrated benefits of selecting the private supervision scope constitute a substantial advance.

  2. Privacy calibration is insufficiently specified. The submission does not report the actual δ\delta, noise multipliers, or training dataset sizes needed to reproduce the privacy accounting. Without these values and confirmation that the sampling implementation matches the accountant’s assumptions, the reported numerical privacy guarantees cannot be independently verified.

  3. Cross-model differences limit attribution of the utility gains. The cross-model comparison combines differences in pretrained backbones, pretraining conditions, and trainable parameter counts, without reporting total and trainable parameter counts. The pretrained results and same-backbone Coarse/Full comparison provide useful controls, but do not fully isolate the contribution of spectral private adaptation. Reporting the parameter counts and relevant pretraining conditions would make the source of the reported gains easier to assess.

  4. The chosen training horizons lack reported convergence evidence. Appendix E reasonably selects training horizons separately based on preliminary convergence, yet provides no supporting convergence evidence. Providing this evidence would help assess whether the baselines were adequately optimized under the stated privacy budgets.

Questions for authors
  1. Privacy calibration (Weakness 2): Please report the training dataset size, δ\delta, noise multiplier σ\sigma, and sampling rate for each private configuration, and confirm that the actual sampling procedure matches the PLD accountant’s assumptions.

  2. Model size and pretraining conditions (Weakness 3): Please report total and trainable parameter counts for the compared configurations, and clarify the pretraining datasets behind the selected public checkpoints.

  3. Baseline convergence (Weakness 4): Please provide the existing convergence evidence supporting Appendix E’s selected private baseline training horizons. Alternatively, a limited horizon-sensitivity check in one representative private setting, with noise recalibrated to preserve the same (ε,δ)(\varepsilon,\delta), would help address this concern.

  4. Coarse-scale quality after public completion (Weakness 1): Using the existing generated samples, could you report FID@32 for DP-Wavelet-Coarse’s final 256×256256\times256 outputs after public completion, resized to 32×3232\times32 using the same procedure and reference-image preprocessing as for the other methods? How does this compare with the decoded coarse intermediary’s FID@32, and does Coarse retain its advantage when all methods are evaluated using resized final outputs?

2 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you very much for your thoughtful review and constructive feedback. We are pleased that you find our private image generation results promising, particularly on MS-COCO, and the same-backbone comparison informative. We address your comments below and hope this will help improve your evaluation of the paper.

Weaknesses:

  1. The demonstrated contribution of scale-selective private adaptation remains modest.

Response: Thank you for highlighting this concern. It is a correct characterization that our contribution is to leverage the properties already provided by AR-SIT for private generation by restricting the DP finetuning objective to selected wavelet scales and studying how this choice affects image quality and training cost. To better demonstrate the value of selecting the supervision scope, we have extended the experiments from the original Coarse and Full endpoints to include two intermediate prefix lengths, for a total of four configurations Kp∈{256,512,768,1024}K_p\in\lbrace{256,512,768,1024\rbrace} corresponding to supervision through the 32×3232\times32, 64×6464\times64, 128×128128\times128, and 256×256256\times256 scales. The updated results in Table 1 and Figure 2 provide a more complete characterization of the tradeoff that we study. In particular, Figure 2 illustrates that, while Full generally achieves the best final-resolution FID, shorter prefixes can approach this performance with lower per-update training cost. For example, on MM-CelebA-HQ at ε=10\varepsilon=10, supervision through 128×128128 \times 128 achieves FID@256 of 18.2 compared with 18.4 for Full and Coarse achieves a better FID@32. We have also added a prefix supervision and completion ablation in Table 2, which shows that coarse supervision captures substantial target-domain information and that the frozen public model can limit final-resolution quality when generating the remaining details. Together, these experiments demonstrate that supervising the entire sequence is not always necessary to obtain strong image quality and that selecting the supervision prefix provides a useful way to balance quality across resolutions and private finetuning cost without changing the pretrained architecture or requiring a separate super-resolution model.

  1. Privacy calibration is insufficiently specified.

Response: Thank you for identifying the missing privacy-accounting details. We have expanded the hyperparameter tables in Appendix E to report the numerical values of δ\delta, calibrated noise multipliers σ\sigma, training set sizes, and sampling rates. We have also clarified the privacy accountants used by each method: DP-Wavelet and DP-LDM use PLD accounting under a fixed-size subsampling model and DP-LlamaGen uses Opacus's PRV accountant with Poisson sampling. Together with the reported batch sizes and training horizons, these details allow the privacy accounting for each configuration to be independently reproduced.

  1. Cross-model differences limit attribution of the utility gains.

Response: Thank you for raising this point. We agree that differences in model capacity and pretraining can contribute to the observed performances and, for the sake of transparency, we have extended Appendix E to include the total and trainable parameter counts of each method in addition to the pretraining datasets used for their public checkpoints. As discussed in our responses to Reviewer 6uhv (Question 2) and Reviewer Myyp (Weakness 1), our comparisons across DP-Wavelet prefix lengths use the same pretrained AR-SIT checkpoint, which allows us to study the effect of restricting private supervision while keeping the architecture and pretraining fixed. The comparisons with DP-LDM and DP-LlamaGen complement these experiments by evaluating diffusion and raster-autoregressive approaches on the same datasets, privacy budgets, and evaluation protocol, with the additional model information providing context for differences in performance across architectures.

AuthorsOpenReview ↗

Rebuttal by Authors Continued

  1. The chosen training horizons lack reported convergence evidence.

Response: We agree that our training horizon selection could be better clarified. As discussed in our response to Reviewer 6uhv (Weakness 2), we selected the training horizons based on preliminary convergence of the training loss and, for DP-LDM, also intermediate FID evaluations because the loss was relatively noisy. The following table summarizes the loss values at different stages of training on MS-COCO with progress normalized by each method's total number of updates:

Training progressDP-LDM lossDP-LDM FID@256DP-LlamaGen lossDP-Wavelet loss
25%~0.1513.016.49~6.30
50%~0.1512.546.19~6.28
75%~0.1512.645.97~6.25
100%~0.1512.615.97~6.24
Total updates9,690—1,125100,000

To more rigorously address the concern about differences in training horizons, we additionally ran a matched sample-exposure experiment for ε=10\varepsilon=10 where each method processes 1,152,000 samples during training with accordingly recalibrated DP noise multipliers to maintain the same privacy budget. The results are:

DatasetDP-Wavelet-CoarseDP-Wavelet-FullDP-LDMDP-LlamaGen
MS-COCO14.239.2812.4113.80
MM-CelebA-HQ26.9823.0924.1763.70

Under this comparison, DP-Wavelet-Full achieves the lowest FID@256 on both datasets, so its advantage over the baselines cannot be attributed solely to processing more training examples. We have included this comparison in Appendix F of the updated paper.

Questions:

  1. Please report the training dataset size, δ\delta, noise multiplier, and sampling rate for each private configuration, and confirm that the actual sampling procedure matches the PLD accountant’s assumptions.

Response: As described in our response to Weakness 2, Appendix E now reports the dataset sizes, δ\delta, noise multipliers, and sampling rates for each private configuration. We have also clarified consistent sampling assumptions for the PLD and PRV accountants in Appendix E.

  1. Please report total and trainable parameter counts for the compared configurations and clarify the pretraining datasets behind the selected public checkpoints.

Response: As discussed in our response to Weakness 3, we have added a table in Appendix E reporting the total and trainable parameter counts with the data used for public pretraining. Specifically, DP-Wavelet uses an AR-SIT checkpoint pretrained on WebLI, DP-LDM uses Stable Diffusion v1.4 pretrained on LAION-derived datasets, and DP-LlamaGen uses LlamaGen-XL pretrained on LAION-COCO.

  1. Please provide the existing convergence evidence supporting Appendix E’s selected private baseline training horizons. Alternatively, a limited horizon-sensitivity check in one representative private setting, with noise recalibrated to preserve the same (ϵ,δ)(\epsilon, \delta), would help address this concern.

Response: We provide the training loss progression and intermediate FID evaluations that we used to assess convergence and inform selecting the training horizons in the response to Weakness 4. These show similar stabilization trends across the compared methods. We additionally report a matched sample-exposure experiment in Appendix F with noise recalibrated to preserve the same privacy budget, which provides complementary evidence that the performance differences are not solely explained by training duration.

  1. Using the existing generated samples, could you report FID@32 for DP-Wavelet-Coarse’s final 256×256256\times256 outputs after public completion resized to 32×3232\times32 using the same procedure and reference-image preprocessing as for the other methods? How does this compare with the decoded coarse intermediary’s FID@32, and does Coarse retain its advantage when all methods are evaluated using resized final outputs?

Response: Upon your recommendation and that of Reviewer 6uhv, we have updated the FID@32 evaluations to use the symmetric protocol where Coarse's final 256×256256\times256 outputs are resized to 32×3232\times32 before evaluation using the same procedure as for the other methods. Under this protocol, the trends are largely the same as when using the decoded coarse intermediary's FID@32. Specifically, DP-Wavelet-Coarse retains lower FID@32 than Full on MM-CelebA-HQ and the two demonstrate comparable quality on MS-COCO. Coarse also continues to outperform both DP-LDM and DP-LlamaGen at the coarse scale across all reported privacy budgets.

Reviewer_MYypOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a method that privately finetunes a public AR-SIT checkpoint (spectral image tokenizer with coarse-to-fine wavelet token ordering) using DP-Adam with PLD accounting. There are two configurations. DP-Wavelet-Coarse computes the loss only on the 32×32 LL0 prefix and lets the frozen public model complete the remaining scales. DP-Wavelet-Full is ordinary DP finetuning over the full token sequence. Privacy follows from record-wise preprocessing plus post-processing (Theorem 1). Experiments on MS-COCO and MM-CelebA-HQ at ε ∈ {1, 10, ∞} report FID and LPIPS at 32x32 to 256x256 (16x16 and 64x64 resolutions are reported in the appendix) against DP-LDM and DP-LlamaGen. Moreover, a canary-based audit of the Coarse model on COCO is included.

Critical strengths
  1. The gains on MM-CelebA-HQ dataset under DP are substantial. At ε=10, FID@256 drops from 55.3 (pretrained) to 17.8 for DP-Wavelet-Full, against 24.9 for DP-LDM (Table 1). This is a strong result of the paper.

  2. The idea of using the coarse-to-fine ordering of wavelet tokens as a knob for how much image detail is learned from private data is interesting and timely.

Critical weaknesses
  1. The paper’s proposal is that selecting which wavelet scales are learned privately improves the privacy-utility tradeoff. Low frequencies are learned from the private data and fine scales are supplied by a public prior. However, only the two endpoints of this design space are evaluated: K_p = K0 (Coarse) and K_p = K (Full). The Full endpoint seems to be the standard DP finetuning of the pretrained AR-SIT model, so the knob that defines the method is never actually swept. Regarding the endpoints that are evaluated, Full outperforms Coarse in FID@256 in every finetuned setting, often by a large margin: 8.9 vs 13.9 and 10.3 vs 14.2 on MS-COCO at ε = 10 and ε = 1. This appears to contradict the hypothesis that fine-scale detail can be delegated to the public prior. Moreover, we see at Tables 3 and 4 that the two configurations use different clipping norms (0.1 vs 0.01). In addition, the coarse-scale advantage seemsto be due to the backbone . On MS-COCO, the pretrained , with no finetuning at all, already achieves a lower FID@16 (7.5) and FID@32 (10.3) than every finetuned baseline at any privacy level (Tables 1 and 6).

  2. The DP-LDM baseline that is included included in the experimental evalaution does not appear to adapt. On MS-COCO at ε = 1, it reproduces the Pretrained row exactly in both FID (12.0/15.0) and LPIPS (0.600/0.745). The non-private finetuning barely moves its FID@256 (15.0 to 14.9) and worsens its FID@32 (12.0 to 13.5). Its results are also non-monotone in ε: on MM-CelebA-HQ, FID@256 is 24.9 at ε = 10 vs 29.6 at ε = ∞ (Table 1). Furthermore, the training budgets seem to be unequal. Under DP on MS-COCO, DP-Wavelet processes roughly 12.8M examples, compared with about 0.83M for DP-LDM and 1.15M for DP-LlamaGen (DP-Wavelet (Coarse and Full): 100,000 updates × batch 128 = 12.8M DP-LDM: 3,230 × 256 = 826,880 ≈ 0.83M DP-LlamaGen: 1,125 × 1,024 = 1,152,000 ≈ 1.15M). In addition , the benefit of the method seems to vanish where privacy matters most. On MM-CelebA-HQ at ε = 1, DP-LDM achieves the best FID@256 (35.1, vs 36.6 and 36.7 for DP-Wavelet). DP-LlamaGen also achieves the best LPIPS in every private setting at both 32 and 256 px.

Questions for authors
  1. I was wondering why DP-RDM (Lebensold et al., 2024), which is evaluated on MS-COCO at ε = 10, is not cited and it is not included in the experimental evaluation? Moreover, apart from one work (FETA-Pro (Gong et al., 2026)) which was published in 2026, the rest of the cited SOTA works are older. Could you please confirm that there is no other recent SOTA work? Moreover, the experimental evaluation contains one work published in 2024 and one work what has been adjusted by the authors. I was wondering why more works were not included ? Maybe a table that categorizes SOTA would be helpful, because the current SOTA seems outdated and the works that they are experimentally evaluates are limited.

  2. Would it be possible to clarify the doubts in the weaknesses regarding the experiments ? Specifically:

  • Is the Full configuration (K_p = K) equivalent to standard DP finetuning of the pretrained AR-SIT model? If so, what distinguishes it methodologically from applying DP-SGD to AR-SIT directly?

  • Could you report results for intermediate prefixes (e.g., prefixes ending at 64 and 128 px) at ε = 1 and ε = 10, so that the effect of the number of privately learned wavelet scales can be assessed beyond the two endpoints?

  • Full outperforms Coarse in FID@256 in every finetuned setting. How do you reconcile this with the hypothesis that fine-scale detail can be supplied by the public prior? Do you attribute the gap to domain mismatch in the unadapted public completion, or to distribution shift between DP-generated prefixes and the prefixes seen during pretraining?

  • Why do Coarse and Full use different clipping norms (0.1 vs 0.01, Tables 3 and 4)? Could you report the comparison with matched clipping norms, so that the difference can be attributed to the scope of supervision rather than to tuning?

  • On MS-COCO, the pretrained AR-SIT already achieves lower FID@16 (7.5) and FID@32 (10.3) than every finetuned baseline. Could you please provide an ablation that separates the contribution of spectral ordering from the strength of the backbone? Examples would be a DP-finetuned next-scale autoregressive model such as VAR, or a low-resolution DP generator followed by public super-resolution.

  • On MS-COCO at ε = 1, DP-LDM reproduces the Pretrained row exactly in both FID and LPIPS, and non-private finetuning barely changes its FID@256. Could you explain why the baseline does not appear to adapt, and how its hyperparameters and training horizon were tuned?

  • On MM-CelebA-HQ, DP-LDM’s FID@256 is better at ε = 10 (24.9) than at ε = ∞ (29.6). What causes this non-monotone behavior?

  • Based on Table 3, DP-Wavelet processes roughly 12.8M examples under DP on MS-COCO, compared with about 0.83M for DP-LDM and 1.15M for DP-LlamaGen. Why were the training horizons chosen so differently? Could you report baseline results under a comparable training budget, together with their compute (chip-hours, as in Table 5)?

  • On MM-CelebA-HQ at ε = 1, DP-LDM achieves the best FID@256. What advantage, if any, does DP-Wavelet offer in the strong-privacy regime?

  • DP-LlamaGen achieves the best LPIPS in every private setting at both 32 and 256 px, and its lead over DP-Wavelet is already present before finetuning. Do all methods share the same evaluation pipeline, including resizing and LPIPS computation, given that DP-LlamaGen is implemented separately in PyTorch?

  1. The code is not inlcuded in the submission. It is a good practice to include the code because it helps he reproducability of the work. I am not sure if it is possible to include the code at this point of the submission , so please report any hyperparams that are missing so that it is possible to reproduce the experimental results.
5 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you very much for taking the time to review our paper and for the detailed feedback. We are glad to hear that you find the MM-CelebA-HQ results to be strong and the methodology interesting. We address your comments below and hope this will help improve your evaluation of the paper.

Weaknesses:

  1. Only two endpoints of the framework are evaluated: Kp=K0K_p = K0 (Coarse) and Kp=KK_p = K (Full) so the knob that defines the method is not swept. Full outperforms Coarse in FID@256 in every finetuned setting which contradicts the hypothesis that fine-scale detail can be delegated to the public prior. The configurations use different clipping norms and the coarse scale advantage seems to be due to the backbone.

Response: Thank you for highlighting these concerns. We agree that evaluating only the Coarse and Full endpoints did not sufficiently characterize how many wavelet scales should be supervised privately. We have therefore extended the experiments to include two intermediate prefix lengths, for a total of four configurations $K_p\in \lbrace{256,512,768,1024\rbrace} $ corresponding to supervision through the 32×3232\times32, 64×6464\times64, 128×128128\times128, and 256×256256\times256 scales. The updated results in Table 1 and Figure 2 show that extending supervision to finer scales generally improves FID@256 on MS-COCO, with substantial gains already at intermediate prefix lengths and modest additional per-update training cost. This is expected since supervising finer scales allows the model to adapt more image details to the private data distribution rather than relying on the frozen public model for completion. On MM-CelebA-HQ, the improvements are more dependent on the privacy budget and show diminishing returns from supervising additional scales. Overall, the optimal prefix length depends on the dataset and target resolution with no universal saturation point.

Regarding the stronger final-resolution performance of DP-Wavelet-Full, we would like to clarify that our hypothesis is that coarse wavelet scales can capture much of the target-domain structure while the remaining details are generated by a frozen public model. We do not mean to imply that using public completion will always match the quality of privately adapting the entire sequence. To better illustrate this difference, we have added a prefix supervision and completion ablation in Table 2 for models finetuned with ε=10\varepsilon=10. First, using the same frozen public continuations, we compare models finetuned specifically on shorter prefixes with the corresponding prefixes generated by Full. We find comparable or better coarse-scale quality with prefix-specific supervision, including improved final-resolution quality for Coarse on MM-CelebA-HQ. Second, when holding the generated prefixes fixed, we replace public completion with a non-privately finetuned target-domain model. This substantially improves FID@256 while leaving FID@32 largely unchanged. These results suggest that coarse supervision can capture important target-domain information and the frozen public model's ability to generate matching finer-scale details can limit final-resolution quality.

We have also addressed the configuration differences within DP-Wavelet models by studying their sensitivity to the clipping norm and selecting one value for all configurations. As described in Appendix F, we repeated the Coarse-vs-Full comparison with matched optimization parameters and varying clipping norms on MM-CelebA-HQ at ε=10\varepsilon=10. The results indicate that Coarse retains a roughly 1.25-point FID@32 advantage under matched settings and Full continues to achieve better FID@256. For C∈{0.01,0.1}C\in\lbrace{0.01,0.1\rbrace}, we found that nearly all per-sample gradients are clipped so changing CC scales both the clipped gradient and the added noise rather than producing a tenfold difference in their effective ratio. We have consequently updated the paper to use C=0.1C=0.1 and learning rate 2×10−52\times10^{-5} consistently across all DP-Wavelet configurations as reported in Tables 4 and 5.

AuthorsOpenReview ↗

Rebuttal by Authors Continued

Finally, we agree that differences in the pretrained backbones likely contribute to differences in measured performances, however we have done our best to maintain fair comparisons (described in more depth in the next response). Our Coarse-to-Full comparisons use the same pretrained AR-SIT checkpoint, which allows us to study the effect of restricting private supervision while keeping the architecture and pretraining fixed. Moreover, DP-Wavelet-Coarse improves FID@32 relative to its own pretrained checkpoint, showing that its coarse-scale performance reflects target-domain adaptation rather than only the strength of the pretrained model. Our comparisons with DP-LDM and DP-LlamaGen complement these experiments by evaluating diffusion and raster-autoregressive approaches on the same datasets, privacy budgets, and evaluation protocol. The models differ in their pretraining data, architecture, and tokenization, and fully controlling these differences would require pretraining all the models ourselves, which is prohibitively expensive computationally.

  1. The DP-LDM baseline does not appear to adapt as the same performance is given for eps=1 as pretrained on MS-COCO and there are small FID changes in other settings. The results are non-monotone for MM-CelebA-HQ FID@256 24.9 at eps=10 and 29.6 at eps=inf. The training budgets are not equal across methods and baselines achieve stronger performance in some settings.

Response: Thank you for highlighting these concerns about baseline adaptation and selecting training horizons. To assess whether additional training could improve performance for DP-LDM, we extended the DP-LDM runs on MS-COCO from 3,230 to 9,690 updates and updated the corresponding results in Table 1. With the new results, we find that DP-LDM obtains improved FID@256 values after finetuning and follows nearly monotonic trends. Specifically, DP-LDM obtains 15.0 before finetuning, 13.4 with non-private finetuning, 12.1 at ε=10\varepsilon=10, and 12.6 at ε=1\varepsilon=1. As discussed in our response to Reviewer 6uhv (Weakness 2), we selected the training horizons based on preliminary convergence of the losses, however since the losses were noisy for DP-LDM, we also monitored intermediate FID evaluations and observed their convergence by 9,690 updates.

Regarding the non-monotonic performance of DP-LDM on MM-CelebA-HQ, a plausible explanation is that the gradient noise introduced by DP acts as a regularizer during finetuning, potentially reducing overfitting and improving generalization to the target dataset. This could explain why the ε=10\varepsilon=10 run achieves a lower FID@256 than non-private finetuning (24.9 vs. 29.6) since FID measures the quality of generated images rather than training loss and this need not decrease monotonically as the privacy constraint strengthens. More importantly, we observe that the non-private and privately finetuned models substantially improve over the FID of the pretrained model on MM-CelebA-HQ, which demonstrates successful target-domain adaptation.

Thank you for suggesting a comparison between the methods with matched training exposure. Upon your recommendation and to complement the method-specific training horizons discussed above, we conducted an additional experiment at ε=10\varepsilon=10 where all four methods process the same 1,152,000 training samples with DP noise multipliers recalibrated to maintain the same privacy budget. Under this comparison, DP-Wavelet-Full achieves the lowest FID@256 on both MS-COCO and MM-CelebA-HQ. We report the complete results in Appendix F of the updated paper, demonstrating that DP-Wavelet-Full retains its advantage over the baselines when training exposure and privacy budgets are matched.

Finally, our updated results also change the comparison under the strongest privacy constraint. On MM-CelebA-HQ at ε=1\varepsilon=1, DP-Wavelet-Full now achieves FID@256 of 34.1 compared with 35.1 for DP-LDM while Coarse achieves 36.7. We therefore observe an advantage for Full even in this setting. We also acknowledge that DP-LlamaGen consistently achieves the lowest LPIPS in the private comparisons. However, this advantage is already present in the pretrained setting, which suggests that it is partly attributable to differences between the pretrained models rather than private finetuning alone. We treat LPIPS as a complementary measure of perceptual similarity to caption-paired references under DP and emphasize FID as our primary metric for distributional image quality, where DP-Wavelet generally performs better.

AuthorsOpenReview ↗

Rebuttal by Authors Continued

Questions:

  1. Why is DP-RDM (Lebensold et al. 2024), which is evaluated on MS-COCO at ε=10\varepsilon=10, not cited and or included in the experimental evaluation? Moreover, many of the cited SOTA works are old. Please confirm there is no other recent SOTA work. Why were more baselines not included? Maybe a table that categorizes SOTA would be helpful.

Response: Thank you for highlighting DP-RDM and the suggestion to include a more systematic discussion of recent work on private image generation. We agree that DP-RDM (Lebensold et al. 2024) is relevant and we have added it to the related works section. DP-RDM evaluates text-to-image generation on MS-COCO at ε=10\varepsilon=10, however they use a different approach for accessing private data than we consider. Specifically, DP-RDM keeps the diffusion generator fixed and uses a differentially private retrieval mechanism to condition generation on information from a private image database where the privacy budget accounts for a bounded number of retrieval queries (up to 10,000 in the reported experiments). In contrast, DP-Wavelet privately finetunes a reusable image generator to learn the target distribution, after which sampling with public captions incurs no additional privacy cost. Thus, DP-RDM addresses a complementary setting and its reported FID cannot be directly compared under our generation and privacy-accounting protocol.

Regarding covering more recent work, we have expanded the related-work discussion to build on our existing coverage of DP-LoRA (Tsai et al., 2025), private autoregressive generation (Shaikh et al., 2025), and SPTI (Wang et al., 2025) and reflect additional developments from 2025-2026. We additionally discuss DP-SAPF (Gong et al., 2026), which selects parameters for private finetuning, and RPGen (Wang et al., 2026), which improves robustness to DP noise through public pretraining. Following your suggestion, we have also added a categorization table in Appendix G that distinguishes approaches according to how they use private data and whether the generator itself is adapted.

Our experimental evaluation focuses on DP-LDM and DP-LlamaGen as baselines because both support caption-conditioned generation and can be evaluated on the same datasets, privacy budgets, and evaluation protocol as DP-Wavelet. These methods provide representative diffusion and autoregressive comparisons while our experiments across wavelet prefix lengths examine the effect of restricting private supervision within the same pretrained backbone.

  1. Would it be possible to clarify the doubts in the weaknesses regarding the experiments? Is DP-Wavelet-Full equivalent to standard DP finetuning of the pretrained AR-SIT model and if so what distinguishes it from applying DP-SGD to AR-SIT directly? Please report results for intermediate prefixes at ε=1\varepsilon=1 and ε=10\varepsilon=10. Full outperforms Coarse in FID@256 in every finetuned setting, how do you reconcile this with the hypothesis that fine-scale detail can be supplied by the public prior? Why do Coarse and Full use different clipping norms and could you report the comparison with matched clipping norms? On MS-COCO, the pretrained AR-SIT already achieves lower FID@16 and FID@32 than every finetuned baseline, please provide an ablation that separates the contribution of spectral ordering from the strength of the backbone? Why does the DP-LDM baseline not appear to adapt on MS-COCO and how were its hyperparameters and training horizon tuned? What causes the non-monotone behavior of DP-LDM's FID@256 values on MM-CelebA-HQ? Why were the training horizons chosen differently for different methods and can you report baseline results under a comparable training budget? What advantage does DP-Wavelet offer in the strong-privacy regime considering DP-LDM achieves the best FID@256? Do all methods share the same evaluation pipeline given that DP-LlamaGen is implemented separately in PyTorch?

Response: We appreciate the detailed questions and address each question below, referring to the additional experiments and earlier responses where appropriate.

Full configuration. Yes, DP-Wavelet-Full with Kp=KK_p = K corresponds to standard DP finetuning of the pretrained AR-SIT model using DP-Adam. Our methodological contribution is restricting private supervision to selected wavelet scales and using the frozen public model to complete the remaining sequence. As shown in Table 1 and Figure 2, Full generally achieves the best final-resolution quality but shorter prefixes can approach this performance at lower training cost, with diminishing returns from supervising additional scales in some settings. Shorter prefixes can also achieve better coarse-scale quality. Thus, selecting KpK_p allows us to balance image quality across resolutions and private finetuning cost depending on the target dataset and privacy budget.

Reviewer_6uhvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DP-Wavelet for differentially private text-to-image generation in a public-text/private-image setting. Building on the public Spectral Image Tokenizer and its autoregressive model AR-SIT, whose wavelet-domain tokens are ordered coarse-to-fine and support partial decoding, DP-Wavelet privately finetunes the transformer with DP-Adam and PLD accounting on a spectral token prefix of length Kp — the coarse LL0 approximation only (DP-Wavelet-Coarse) or the full sequence (DP-Wavelet-Full) — with any remaining scales completed by the frozen public model at inference. A theorem shows via the post-processing property of DP that generation, completion, and decoding preserve the (eps, delta) guarantee established during finetuning. Experiments on MS-COCO and MM-CelebA-HQ report FID and LPIPS at four resolutions against DP-LDM and DP-LlamaGen across pretrained, non-private, eps=10, and eps=1 regimes, plus a one-run membership-inference audit (Coarse, MS-COCO only) and a Gemini-refinement exploration. Coarse wins FID at the 32x32 evaluation scale at lower training cost; Full achieves the lowest FID@256 in five of six finetuned settings.

Critical strengths
  1. A correct, precisely stated privacy design with honest reporting: the end-to-end (eps, delta) guarantee follows cleanly from DP post-processing with explicit scope conditions (Appendix D), and the paper plainly reports its unfavorable results — DP-LlamaGen wins all eight private LPIPS cells, DP-LDM wins FID@256 on MM-CelebA-HQ at eps=1, and the Gemini-refinement variant is metric-worse despite smoother outputs.

  2. A concrete, reusable tradeoff with good hygiene: coarse-prefix finetuning wins FID at the coarse evaluation scale at lower training cost (3.15h vs 3.63h per 100K updates, identical hardware) while full-sequence finetuning wins FID@256 in all six finetuned comparisons; per-method hyperparameter tables, a same-hardware cost comparison, and a modern one-run membership-inference audit are provided.

Critical weaknesses
  1. The methodological contribution is thin, the paper's own design axis is evaluated at only two extreme points, and the headline coarse-scale comparison is protocol-asymmetric. The framework is DP-Adam finetuning of a public AR-SIT checkpoint with the loss truncated at a prefix, plus a direct post-processing theorem; the closest private-AR prior work (Shaikh et al., 2025) is cited but never compared. The central axis — which wavelet scales to privatize — is sampled only at Kp = K0 and Kp = K, with no intermediate-prefix sweep and the two motivating hypotheses never measured. At 32x32, Coarse is evaluated on its natively decoded intermediary while all other methods are evaluated on downscaled 256x256 outputs, so the "substantially lower coarse FID" claim partly measures the evaluation protocol.

  2. The comparisons are uncontrolled at every level. Baselines run 22-89x fewer updates (1,125-4,680 vs 100,000) under undocumented per-method horizons, and cited strong methods (PrivImage, dp-promise, FETA-Pro) are not compared — at eps=1 on MM-CelebA-HQ the one strong baseline already wins FID@256 (35.1 vs 36.6). Even the internal Coarse-vs-Full comparison is not configuration-matched (10x clipping norm, 2x learning rate, 8x non-private batch size), so the absolute DP noise scale differs 10x between the two arms. All metrics are single runs yet sub-1-FID margins are interpreted; delta and noise multipliers are never reported; and the privacy audit covers only Coarse on MS-COCO — not the identity-sensitive face dataset, not Full, no baseline.

Questions for authors
  1. What are the FID@32/@256 results for intermediate prefix lengths (one and two detail scales), and where does the quality/cost curve saturate? This is the decisive evidence for the paper's central claim.

  2. How does a non-spectral control — DP finetuning of a low-resolution raster AR model plus public super-resolution — perform under the same budget? This isolates the value of the spectral representation over any coarse parameterization.

  3. If all methods are evaluated at the coarse scale on downscaled final outputs (symmetric protocol), does the Coarse configuration still win FID@32?

  4. Can you re-run the Coarse-vs-Full comparison at matched clipping norm, learning rate, and batch size? How does the prefix-truncated loss (K0-token vs K-token average) interact with the 10x clipping-norm difference — how much of the Coarse advantage survives an effective-SNR-matched comparison?

  5. Were DP-LDM and DP-LlamaGen trained to convergence under each epsilon? Please provide learning curves or a stated symmetric tuning protocol, and reconcile your DP-LDM numbers with those reported by Liu et al. (2024). Why is DP-LlamaGen's MM-CelebA-HQ performance so unstable across regimes (FID@256: 32.5 at eps=inf, 63.7 at eps=10, 74.2 at eps=1)?

  6. Can the membership-inference audit be extended to MM-CelebA-HQ (the identity-sensitive dataset), DP-Wavelet-Full, and at least one baseline? And do per-sample LPIPS-to-reference scores correlate with audit scores — i.e., could DP-LlamaGen's LPIPS advantage (verified: it wins all eight private LPIPS cells) partly reflect proximity to individual references (a leakage proxy) rather than pure utility?

Additional concerns (for the record):

  1. All FID/LPIPS entries are single runs, yet margins below 1 FID are interpreted; please report run-to-run variability (seeds or bootstrap over caption subsets). The privacy accounting is under-specified: delta is never stated numerically and noise multipliers are not reported.

  2. The abstract's "lower training cost" holds only relative to the paper's own Full configuration (3.15h vs 3.63h); the baselines run orders of magnitude fewer updates, and the Coarse configuration adds unreported inference-time completion cost per sample.

  3. In Table 2, the confidence-adjusted Clopper-Pearson lower bound (7.04) exceeds the raw point estimate (5.20); please clarify the construction or verify the number. Relatedly, the audit-method reference (McKenna et al., 2026) does not appear to be publicly locatable — please provide a preprint link, since the estimator construction is needed to interpret Table 2. Also, the motivating medical-imaging setting (Appendix A) is never instantiated with a medical dataset.

6 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you very much for taking the time to review our paper and for the comprehensive feedback. We are pleased to hear that you find our framework to be correct and precisely stated and the quality-cost tradeoff to be clear. We address your comments below and hope this will help improve your evaluation of the paper.

Weaknesses:

  1. The framework is only evaluated at two extreme points and the headline coarse-scale comparison is protocol-asymmetric. The closest related method Shaikh et al. is cited but not compared.

Response: Thank you for raising these points. We agree that adding evaluating intermediate wavelet scales provides a more complete picture of the framework's performance. Following your suggestion, we have extended DP-Wavelet to two intermediate prefix lengths Kp=512K_p=512 and Kp=768K_p=768 corresponding to the 64×6464 \times 64 and 128×128128 \times 128 scales respectively so the updated paper evaluates all four prefix lengths. The results show that extending private supervision to finer scales generally improves FID@256, particularly on MS-COCO, although the gains diminish or become non-monotonic on MM-CelebA-HQ. This is likely partly because finetuning on additional scales allows the model to adapt finer image details to the private data distribution rather than relying on a frozen public model for the completion. Since increasing KpK_p changes both the training objective and the amount of public completion, we have added an ablation that holds generated prefixes fixed and compares completion using the public model with completion using a non-privately finetuned model for the target domain as a diagnostic. We find that adapting the completion model can substantially improve FID@256 while leaving FID@32 largely unchanged. This suggests that fine-scale adaptation can improve final-resolution quality independently of the quality of the coarse representation.

Thank you for also highlighting this difference in the method for FID@32 computation between DP-Wavelet-Coarse and the other models. We have updated the evaluation protocol to be symmetric and consistently report FID at lower resolutions using the final generated images downscaled to the corresponding resolution. Under this comparison, Coarse retains a lower FID@32 on MM-CelebA-HQ while Full performs slightly better on MS-COCO. We provide the detailed results in the responses to Questions 1 and 3 below and also in the paper.

Regarding Shaikh et al. (2025), we agree this work is relevant for private autoregressive image generation, however their experiments finetune ImageNet-pretrained, class-conditioned VAR models on different downstream datasets so their reported FIDs are not directly comparable with our caption-conditioned image generation setting. A fair comparison would require adapting VAR's conditioning and pretrained generation pipeline to our datasets rather than directly applying their existing experimental setup. We instead include DP-LlamaGen as a raster-autoregressive baseline since LlamaGen operates in our same text-conditioned setting and acknowledge that this does not fully isolate the effect of spectral ordering. We discuss this further in response to Question 2.

  1. Some parameters for comparisons are uncontrolled such as the number of updates and the clipping norm, learning rate, and batch size. Some information is under-documented such as per-method horizons, delta and noise multipliers are not reported, cited strong methods are not compared, and the privacy audit only covers coarse image generation.

Response: Thank you for highlighting these places where our baseline selection and experimental configurations could be better justified.

We chose the number of finetuning updates based on method-specific preliminary convergence of the loss and, for DP-LDM, also intermediate FID evaluations since the training loss was relatively noisy. To assess whether additional training could improve DP-LDM's performance, we extended the MS-COCO runs to use 9,690 updates and have updated the corresponding results in Table 1. The following table summarizes the loss values at different stages of training on MS-COCO with progress normalized by each method's total number of updates:

Training progressDP-LDM lossDP-LDM FID@256DP-LlamaGen lossDP-Wavelet loss
25%~0.1513.016.49~6.30
50%~0.1512.546.19~6.28
75%~0.1512.645.97~6.25
100%~0.1512.615.97~6.24
Total updates9,690—1,125100,000
AuthorsOpenReview ↗

Rebuttal by Authors Continued

To more rigorously address the concern about differences in training horizons, we additionally ran a matched sample-exposure experiment for ε=10\varepsilon=10 where each method processes 1,152,000 samples during training with accordingly recalibrated DP noise multipliers to maintain the same privacy budget. The results are:

DatasetDP-Wavelet-CoarseDP-Wavelet-FullDP-LDMDP-LlamaGen
MS-COCO14.239.2812.4113.80
MM-CelebA-HQ26.9823.0924.1763.70

Under this comparison, DP-Wavelet-Full achieves the lowest FID@256 on both datasets, so its advantage over the baselines cannot be attributed solely to processing more training examples. We have included this comparison in Appendix F of the updated paper.

We also repeated the Coarse vs Full performance comparison using matched clipping norms, learning rates, and batch sizes for models finetuned on MM-CelebA-HQ at ε=10\varepsilon=10 with C=0.1C=0.1, learning rate 2×10−52\times10^{-5}, batch size 128, and 100k updates. The results are given below:

Clipping norm CCCoarse FID@32 / FID@256Full FID@32 / FID@256
0.0113.82 / 19.3114.98 / 18.32
0.113.71 / 19.3414.96 / 18.39
5.013.91 / 19.3114.89 / 18.19
10.015.16 / 21.1516.10 / 19.11

In the matched setting, we find that Coarse keeps its 1.25 point FID@32 advantage (compared with around 2.5 points in the original comparison), while Full achieves better FID@256. For C∈{0.01,0.1}C\in\{0.01,0.1\}, we observe that essentially all the per-sample gradients are clipped, so changing CC scales both the clipped gradient and the added noise rather than producing a tenfold change in their effective ratio. Consistent with this, the resulting FIDs are very similar. To maintain consistency, we have updated the paper to use C=0.1C=0.1 and learning rate 2×10−52\times10^{-5} for all DP-Wavelet runs, including the new versions with intermediate prefix lengths.

To address the concern about evaluation uncertainty, we computed bootstrap standard errors using 50 resamplings of generated and reference Inception features with 30,720 samples per distribution. We found that the standard errors range from approximately 0.03 to 0.17 FID and have added this analysis to Appendix F to better qualify conclusions based on small FID differences. We have also added the data-specific privacy parameters δ\delta and calibrated noise multipliers σ\sigma to the training hyperparameters in Appendix E.

Following your suggestion, we have also extended the membership inference audit to DP-Wavelet-Full on MS-COCO. At ε=10\varepsilon=10, both private models achieve an attack AUROC of approximately 0.501, compared with 0.715 for non-private Coarse and 0.765 for non-private Full models. The attack therefore detects a membership signal in the non-private models and performs approximately at chance level for the private models we audited. For MM-CelebA-HQ, we use all 30,000 images for training, which leaves no held-out samples for the same auditing protocol. While an additional audit could provide insight into identity-specific empirical privacy risks, the formal (ϵ,δ)(\epsilon, \delta)-DP guarantee applies independently of the dataset and is preserved under our public completion procedure by post-processing. We also agree that auditing the baselines would provide a useful empirical comparison, however our audit relies on changes in autoregressive log-likelihood relative to the public checkpoint and extending this to DP-LDM would require a different attack score, while DP-LlamaGen would require a separately implemented and calibrated audit to support the PyTorch model. We therefore do not draw comparative privacy conclusions from these results and they are intended only to provide an empirical check that DP-Wavelet finetuning reduces the membership signal detected by this attack relative to non-private finetuning.

AuthorsOpenReview ↗

Rebuttal by Authors Continued

Regarding baseline selection, we agree that our criteria could be made more clear. The suggested methods (PrivImage, dp-promise, and FETA-Pro) address different aspects of private image generation: PrivImage selects semantically matched public data for pretraining, dp-promise exploits the forward-process noise of diffusion models, and FETA-Pro introduces a spatial-frequency training curriculum with additional private feature extraction. Our focus in this work is instead a representation-level design choice: which wavelet scales to supervise when privately finetuning an existing public autoregressive model (without modifying its pretraining procedure or introducing additional private-data processing stages). Moreover, the evaluations for these approaches primarily consider unconditional or label-conditioned generation on different datasets and generally lower resolutions, rather than our caption-conditioned generation with 256×256256 \times 256 resolution. Consequently, their published FID values are not directly comparable with ours without adapting the generation and evaluation pipelines. We instead select DP-LDM and DP-LlamaGen as baselines because their pretrained diffusion and autoregressive models support the same caption-conditioned generation as DP-Wavelet, allowing direct comparisons on the same datasets, privacy budgets, and evaluation protocol.

Questions:

  1. What are the FID@32/@256 results for intermediate prefix lengths and where does the quality/cost curve saturate?

Response: We now report the FID@32 and FID@256 results for intermediate prefix lengths Kp=512K_p=512 and Kp=768K_p=768 in Table 1 of the updated paper. We also added Figure 2 to characterize the quality/cost tradeoff. On MS-COCO, we find that extending private supervision to finer wavelet scales improves the final resolution quality at a modest additional training cost, with substantial gains already at intermediate prefix lengths. On MM-CelebA-HQ, the improvements are more dependent on the privacy budget, with coarse supervision and public completion already performing best in some settings. Overall, the quality-cost tradeoff is dataset-dependent with no universal saturation point.

  1. How does DP finetuning of a low resolution raster AR model with public upscaling perform comparatively?

Response: Thank you for this comparison suggestion. For reference, the original AR-SIT paper already compares spectral and raster autoregressive image generation in the non-private setting and finds comparable full-resolution quality while demonstrating that spectral ordering enables efficient generation of complete low-resolution images from shorter token prefixes. Our goal in this work is not to establish that spectral tokenization is necessarily the best approach to private coarse-to-fine generation, but rather to investigate how this existing multiscale property can be exploited under DP by restricting private supervision to selected spectral scales while using the frozen public model to complete the remaining details. AR-SIT provides a natural framework for this because we can vary the privately supervised prefix without changing the pretrained architecture or requiring a separate super-resolution model. Our Coarse-to-Full ablations examine how the extent of private supervision affects image quality and training cost and we use DP-LlamaGen as a baseline for a raster-order autoregressive comparison under DP.

Studying whether a raster-order low-resolution generator with public upscaling works equally well would be interesting, however this would require a new generation pipeline that is not easily available from existing checkpoints. Specifically, the publicly released text-conditioned LlamaGen models support 256×256256\times256 and 512×512512\times512 generation and truncating their raster token sequences does not yield complete low-resolution images, so we would need to train or adapt a caption-conditioned raster generator for lower resolutions and pair it with a compatible public super-resolution model. Simply downsampling full-resolution LlamaGen outputs and applying super-resolution would still require privately finetuning and generating the full raster sequence, so this would not isolate the benefit of coarse private supervision. In summary, we agree that evaluating such a pipeline would be nice but this requires additional model adaptation and evaluation that is unfortunately beyond the scope of experiments we can run for this rebuttal.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overview of Improvements

We would like to thank the reviewers again for their time and valuable comments. We have uploaded a revised version of the manuscript incorporating the following main changes highlighted in blue:

  • Extended the DP-Wavelet experiments to four supervision prefixes and added an ablation examining the effects of prefix supervision and public completion.
  • Added matched clipping-norm and training-exposure comparisons, together with additional convergence evidence and training cost measurements.
  • Clarified the evaluation protocol, expanded the privacy audit, and reported FID uncertainty estimates.
  • Expanded the implementation and privacy-accounting details, model comparisons, and coverage of recent related work.

We believe that these changes have substantially enhanced the quality of the paper and invite the reviewers to reconsider our submission, potentially adjusting their scores. If there are any remaining concerns, please let us know and we will address them!

Area_Chair_gP3VOpenReview ↗

Dear Reviewers

Since the author-reviewer discussion ends soon, I encourage the reviewer to review the authors' rebuttals and let them know whether their responses are satisfactory and whether any concerns remain.

Thank you for your time and effort in reviewing for ICLR 2027.

Best wishes,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.