- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces DP-λCGD, a new correlated-noise variant of DP-SGD that aims to improve the privacy-utility tradeoff without the memory and runtime cost of existing correlated-noise methods. DP-λCGD uses a simple one-parameter family of noise correlations, where each step cancels a fraction of the noise added at the previous step. To avoid extra memory, the previous noise is regenerated from a stored random number generator state instead of being kept in memory. The paper provides a theoretical analysis of the error of the proposed method and a practical heuristic for choosing its parameter. Experiments on image and language models show that DP-λCGD has runtime close to DP-SGD and improves accuracy compared to DP-SGD and other correlated-noise baselines in several settings.
Critical strengths
- The paper studies an important problem. Removing the memory overhead of correlated noise is practically relevant for large models.
- The idea of regenerating the noise from a stored PRNG state is an interesting and uncommon approach.
- The paper is generally well written, and the evaluation across different model sizes supports the efficiency claim.
Critical weaknesses
- The novelty is not very significant. The correlation matrix is acknowledged to already appear in prior work (Choquette-Choo et al. 2024a, Pillutla et al. 2025, Kalinin et al. 2026), so the novelty largely reduces to treating λ as a tunable hyperparameter and using regeneration.
- In the main comparison (Fig. 4a) on CIFAR-10, BLT and BISR (p=16) appear a lot stronger at several privacy levels. Even though the authors claim that BLT requires either storing a noise buffer or regenerating the full sequence, there is no empirical evidence that characterizes this tradeoff between BLT and DP-λCGD.
- It appears that DP-λCGD needs to be tuned over λ which is dependent on task, batch size, number of epochs, etc. This is a big disadvantage compared to other baselines.
Questions for authors
- How does the measured peak memory of DP-λCGD compare to BLT, BISR, and BandInvMF?
- How should λ be selected when no public validation set is available, and how robust is the "2-4x smaller than amplified RMSE" rule on other tasks?
4 comments
>> How does the measured peak memory of DP-CGD compare to BLT, BISR, and BandInvMF?
We are happy to provide an empirical comparison. In particular, BISR and BandInvMF have identical memory requirements for a fixed bandwidth (as they differ only in coefficients), so for simplicity, we report only BISR. BLT has approximately the same memory requirements as BISR with a stored noise history of bandwidth 5, while DP-CGD corresponds to BISR with bandwidth .
We report the peak GPU memory usage when fine-tuning BERT-Base on IMDb with batch size 512 and physical batch size 16, corresponding to the fine-tuning experiments in Figure 6(a). We compare DP-SGD, DP-CGD, BLT with buffer size 5, and BISR with bandwidth 16. The results are reported in the table below.
| Method | Regenerated (GB) | Stored on GPU (GB) |
|---|---|---|
| DP-SGD | 9.741 | – |
| DP-CGD | 9.752 | 10.575 |
| BLT, () | - | 11.857 |
| BISR, () | 9.752 | 16.281 |
With noise regeneration, DP-CGD has almost identical memory usage to DP-SGD (additional overhead is due to the size of the largest parameter group used for noise generation). Both DP-CGD and BISR (as well as BandInvMF) can benefit from efficient noise regeneration. In contrast, the noise correlation structure of BLT prevents the same practical regeneration, requiring the noise history to be stored in memory. In our experiment, this results in approximately 2.1 GB of additional memory usage compared with DP-CGD with noise regeneration.
For larger models, this overhead can become particularly important. First, models that fit on a single GPU with DP-CGD might not fit with BLT due to the additional memory requirements. Second, the memory saved by DP-CGD can potentially be allocated to larger physical batch sizes, allowing faster training.
We have included the table in the revised manuscript; see Table 3.
>> “In the main comparison (Fig. 4a) on CIFAR-10, BLT and BISR (p=16) appear a lot stronger at several privacy levels”
We agree that BLT and BISR () achieve higher accuracy at some privacy levels in Figure 4(a). However, this comes with additional memory or computational costs. Moreover, in the fine-tuning experiments (Figure 6), we do not observe substantial accuracy improvements from BLT or BISR over DP-CGD. Our goal is to provide a favorable trade-off between accuracy, memory, and computational efficiency, rather than optimizing accuracy alone. DP-CGD achieves substantial accuracy improvements over DP-SGD, while our runtime experiments show that its computational cost remains close to that of DP-SGD, particularly for larger models.
>> How should be selected when no public validation set is available, and how robust is the "2-4x smaller than amplified RMSE" rule on other tasks?
When no public validation set is available, we propose selecting based on amplified RMSE. This does not need access to any data and can be computed knowing only the number of iterations, number of epochs, batch size, and privacy level. As mentioned above, our experiments show that this choice already gives performance reasonably close to the accuracy-optimal .
The 2-4× rule of thumb is based on our CIFAR-10 and IMDb experiments (Figure 3), where the accuracy optimal choice is smaller than the amplified RMSE suggests for CIFAR-10 training, while the two coincide for fine-tuning on IMDb. During the rebuttal, we did additional experiments comparing fine-tuning of ViT-Base on CIFAR-100 and training WRN-16-4 from scratch on CIFAR-100. The results are consistent with our previous observations: the amplified RMSE-optimal values are almost optimal for fine-tuning, while for training from scratch, the optimal values are 2-4 times smaller (on the scale ). Please find the corresponding plots in Figure 7 of the updated manuscript.
We also realized that the wording of this rule was imprecise. We intended to say that, for training from scratch, the accuracy optimal choice is typically 2-4x smaller than the value suggested by the amplified RMSE (on the scale ), whereas for fine-tuning, we have not observed a significant difference. We have clarified the rule in the revised manuscript.
Dear Reviewer KHgG,
As the rebuttal period will end soon, we kindly ask you to clarify whether any of the concerns remain.
Best regards, Authors
I appreciate the response from the author. While most of my concerns are addressed, I still find the novelty to be limited. I will maintain my current score.
We thank the reviewer for their time and valuable feedback! Below we address the concerns and questions raised.
>> On the novelty
We agree that the correlation Toeplitz matrix of DP-λCGD has appeared in the literature before, but we would like to emphasize that, to the best of our knowledge, we are the first to systematically study how the correlation structure can be tuned for the actual training accuracy in a practical setting. This is different from simply selecting the best λ based on RMSE. In particular, a major contribution of our paper is the empirical study of the discrepancy between the RMSE optimal, amplified RMSE optimal, and accuracy optimal choices of λ. The proposed factorization provides a simple, single parameter setting to study these questions, which we believe are important for understanding the practical behavior of correlated noise methods.
Another major contribution is noise regeneration, as the reviewer also acknowledges. This can be seen as complementary to our choice of correlation structure: DP-λCGD is particularly well suited for regeneration, as it requires regenerating only a single noise vector per iteration, while still providing substantial utility improvements over DP-SGD.
>> It appears that DP-λCGD needs to be tuned over λ which is dependent on task, batch size, number of epochs, etc.
We would like to clarify that task-specific tuning of λ is not required to use DP-λCGD. Our main point is that the metrics commonly used to construct noise factorizations, such as Root Mean Squared Error (RMSE), do not perfectly predict the training accuracy.
Like other factorizations, our factorization can be selected by optimizing RMSE. In particular, we observed that selecting λ based on amplified RMSE already gives performance reasonably close to the accuracy optimal one. Importantly, amplified RMSE can be computed without access to any training or validation data.
The advantage of our factorization is that its simple structure allows us to study the relationship between the factorization and actual training accuracy. Other factorizations generally do not provide such a simple mechanism for exploring this dependence. Moreover, there is no reason to expect that their fixed factorizations are universally optimal across different tasks, batch sizes, privacy levels, and numbers of epochs. They are typically selected by optimizing RMSE, given the number of iterations and separation, which does not necessarily reflect downstream training performance.
Therefore, we see the possibility of tuning λ for downstream accuracy as an additional flexibility of our method, rather than a requirement. In fact, we consider the study of the relationship between RMSE and practical training performance to be one of the major contributions of our paper.