acceptodds
Under review as a conference paper at ICLR 2027

Beyond Square Roots: A Memory-Efficient Explicit Factorization for Multi-Epoch Private Learning

Abstract

Correlated-noise mechanisms are among the most promising approaches for improving the utility of differentially private model training, but rigorous guarantees require explicit, analyzable factorizations, and practical deployment requires memory efficiency. Recent works have developed banded inverse factorizations, which address both requirements by exploiting a banded structure in the correlation matrix. The bandwidth controls the size of the noise buffer used to correlate noise across iterations, and thus governs the tradeoff between utility and memory cost. At the two ends of this tradeoff, DP-CGD achieves high memory efficiency by using only a one-step noise buffer, but this limits its utility gains, while the banded inverse square root (BISR) factorization exploits larger correlation windows and is asymptotically optimal for large bandwidths but performs poorly at low bandwidths. To bridge this gap, we introduce -BIFR, a unified generalization of both factorizations. In the low-memory, low-bandwidth regime, -BIFR improves RMSE, amplified RMSE, and private training performance, while yielding tighter theoretical guarantees for multi-participation error in multi-epoch training.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
differential privacy, private model training, matrix factorization mechanism, correlated noise, memory efficiency

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fob7OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper considers the problem of using the matrix mechanism for answering prefix sum queries with differential privacy, with applications to DP ML. We are given an input stream (e.g. of gradients) G and add correlated noise C^{-1} Z, where Z is Gaussian noise and C is a "correlation matrix". The goal is to minimize the error of the prefix sums of the stream E C^{-1} Z, where E is the lower-triangular all-ones matrix while preserving differential privacy (which requires bounding some norm of C).

Typically memory is limited, so storing C^{-1} Z all at once is inefficient and we would like to be able to generate each row of C^{-1} Z in a streaming manner using limited memory. The authors are further interested in a setting where we can generate each row from scratch efficiently, rather than generating each row from the previous row. This is helpful since the 'noise state' can be saved as a few PRNG seeds rather than as full (high-dimensional) noise vectors, and also allows us to regenerate the noise efficiently if we need to restart in the middle of training due e.g. to device availability (past approaches might require running all previous iterations of noise generation to retrieve the noise state from the PRNG seeds). For this reason the authors focus on banded inverse matrices, i.e. C such that C^{-1} only has pp non-zero diagonals, which natively have these properties for noise generation.

They propose a family of banded-inverse matrices called γ\gamma-Banded Inverse Fractional Root (BIFR), parameterized by γ∈[0,1]\gamma \in [0, 1], where the entries of C−1C^{-1} decay as (γi)\binom{\gamma}{i}. BIFR recovers the previous work of BISR at γ=1/2\gamma = 1/2 making it optimal for larger bandwidths pp, and matches the asymptotic error scaling of the prior work λ\lambda-CGD, which was designed for p = 2, but extends it to larger pp.

The authors conduct empirical studies demonstrating (i) the error (RMSE) of BIFR is much closer to that of the numerically optimized solution than other approaches (ii) how the choice of parameter γ\gamma effects the RMSE as well as model accuracy at different bandwidths (iii) how much the resulting mechanisms benefit from privacy amplification (iv) experimental comparisons on model training between different matrix factorizations in memory-limited settings. Namely, compared to other methods BIFR does well at high-epsilon regimes closest to practice.

Critical strengths
  • The asymptotic error bound demonstrates that γ\gamma-BIFR unifies and subsumes results from past work. Providing unifying frameworks like γ\gamma-BIFR gives a cleaner overall understanding of the problem space, rather than having many seemingly disjoint results with separate intuition. Furthermore, I would characterize this primarily as a theory paper, so having a theoretical result subsuming and expanding beyond past work constitutes a meaningful contribution.
  • The empirical studies are thorough, looking at a wide variety of factors. For example, comparisons on RMSE and model training between different methods, robustness of error/accuracy to the γ\gamma parameter, comparing the amplification in different settings, and more. These thorough studies both give better qualitative understanding of different methods, and also make it more useful for practitioners by identifying regimes where the gain is largest, and also the importance of certain parameters.
Critical weaknesses
  • The practical significance of the method is somewhat limited. BIFR is not always the best performing method in the experiments. The main qualitative practical advantage of banded-inverse methods is the setting when the noise state is required to be low memory (a few PRNG keys) and also fast to regenerate. However, other methods which the authors mention as being less practical in these regimes due to regenerating more slowly are still used in practice (as a recent production example, see the use of BLTs in https://arxiv.org/abs/2609.31494) despite not satisfying these criteria, so it's not clear how common these restrictions are.
  • In the empirical results, DP-SGD seems to be a highly competitive method even at larger ϵ\epsilon, which seems to contradict past work (e.g. https://arxiv.org/abs/2306.08153) which observes DP-SGD is outperformed even by e.g. BandMF at almost all epsilons. It would be good to explain this discrepancy with past work, otherwise it's a bit hard to trust these empirical results. Even in the low-memory regime where DP-SGD would be more competitive, I would expect 2 bands to offer most methods enough flexibility to beat DP-SGD at high epsilons.
Questions for authors
  • See above question on why DP-SGD performs so well in the experiments
4 comments

We thank the reviewer for their thoughtful feedback and positive assessment of our theoretical and empirical contributions! We address the remaining concerns below.

>> On the practical significance

As correctly pointed out by the reviewer, our method belongs to the class of banded-inverse methods, which offer several advantages over the Buffered Linear Toeplitz (BLT) method.

  1. Theoretical guarantees: Banded-inverse methods have explicit RMSE guarantees in the multi-participation (multi-epoch) setting. While BLT supports privacy accounting for multiple participations, its existing theoretical utility guarantees do not cover this setting. Moreover, even in the single-participation setting, the theoretical construction differs from the numerically optimized BLT implementation used in practice.
  2. Simplicity: Banded-inverse methods are relatively easy to implement, analyze, and understand. In particular, our explicit construction requires selecting only a single scalar parameter, rather than optimizing an entire factorization. We believe this simplicity can help make matrix mechanisms more accessible to the broader DP community.
  3. Noise regeneration: Banded-inverse methods allow efficient regeneration of past noise vectors from PRNG seeds, avoiding the need to store a noise buffer whose memory requirement scales with the model size.

Regarding the practical significance, we acknowledge the successful BLT deployment cited by the reviewer. However, we wish to emphasize that their reported models have only up to 10M parameters, making the storage of a few additional noise vectors relatively easy. In contrast, DP-SGD has already been used to train substantially larger models, such as the 1B parameter VaultGemma [1]. At this scale, storing multiple additional noise vectors becomes considerably more expensive, making efficient noise regeneration particularly appealing.

To show this advantage, we have done additional memory and runtime experiments, reported in Tables 8, 9 and 10 of Appendix A.2 of the revised manuscript (see also our response to reviewer yG27). In particular, when fine-tuning VaultGemma, γ\gamma-BIFR with p=64p=64 and noise regeneration requires only 28.4 GB of GPU memory, compared with 44.2 GB for BLT, while causing only 10% additional runtime. The savings in memory can easily be translated into time savings by allowing larger physical batches to be used. In particular, we show a 29% reduction in time compared to BLT when using noise regeneration with a larger physical batch size that would otherwise not fit on a single GPU with BLT.

We also note that optimizing BLT with larger numbers of buffers can be challenging. In particular, the JAX_privacy implementation explicitly states that "the default of 3 is a good choice in general; large number of buffers can cause numerical instability in the optimization." [2]. The cited BLT deployment also uses only three buffers, leaving substantial utility gains on the table, particularly in the low-privacy regime, as shown in Figures 4 and 5 of our paper.

We therefore believe that banded-inverse methods offer a promising direction for scaling correlated noise private training to substantially larger models, while retaining theoretical guarantees and implementation simplicity.

[1] Sinha et al. “VaultGemma: A Differentially Private Gemma Model” 2025 arXiv:2510.15001

[2] https://github.com/google-parfait/jax_privacy/blob/main/jax_privacy/matrix_factorization/buffered_toeplitz.py#L958

>> DP-SGD seems to be a highly competitive method

We note that DP-SGD was already competitive in the high-privacy regime (ε≤1\varepsilon \leq 1) in the cited BandMF paper (McKenna, 2025), see Figure 1. Even at larger privacy budgets, the improvements in CIFAR-10 accuracy are relatively modest, on the order of a few percentages. Moreover, the larger improvements in their Figure 6 are shown in the federated learning setting without amplification, which is more favorable to matrix factorization and less comparable to our centralized experiments.

There are also several important differences between the experimental settings that may explain the observed discrepancy.

  1. BandMF optimizes the bandwidth for each privacy budget, whereas we consider fixed, predefined bandwidths to study the utility-memory tradeoff. These bandwidths are not necessarily optimal for every method and privacy budget.
  2. The privacy amplification schemes differ. DP-SGD benefits from tight accounting under Poisson subsampling, whereas the available Poisson accounting for general matrix mechanisms remains very loose. We use the Balls-in-Bins amplification, which provides tighter accounting for banded-inverse mechanisms but offers weaker amplification than Poisson subsampling. This difference limits the achievable improvements over DP-SGD, particularly in the high-privacy regime.
  3. The optimization procedures differ. In Appendix H.1, BandMF reports using momentum 0.950.95 for matrix-factorization methods and no momentum for DP-SGD. In contrast, we do not use momentum in our CIFAR experiments to isolate the effect of noise correlation from the gradient correlation introduced by momentum.

We agree with the reviewer's intuition that even bandwidth p=2p=2 can outperform DP-SGD at larger privacy budgets, as also shown in Figure 4 of the DP-λ\lambdaCGD paper (Kalinin et al, 2026a), where λ\lambda was selected based on the validation accuracy. Our results similarly show higher accuracy than DP-SGD at ε=8\varepsilon=8 (see Tables 2-5).

We believe these differences in bandwidth selection, privacy amplification, and momentum help explain the discrepancy with the results reported in BandMF.

Thanks to the authors for their response. The authors provide several compelling reasons why their empirical setting is different and the comparison between DP-SGD and BandMF may not align with prior work. I also appreciate the inclusion of the new empirical comparisons in terms of memory usage. These results are very useful for comparing the different methods across multiple dimensions besides just utility. I have raised my Soundness score accordingly.

However, some of my concerns still remain.

  • First, the comparisons the authors make in terms of memory show BIFR with 4 bands is better in terms of efficiency than BLTs with 4 buffers. Again, I believe this is a useful result to show, to allow comparing the methods on multiple axes. However, this is in the setting of Figure 4a, where for smaller epsilons BLT has higher accuracy (sometimes by a non-trivial margin), and even for larger epsilons seems to be within the margin of error. In turn, it's plausible that despite BIFR achieving better efficiency in Tables 8-10, if we fix a time and memory constraint per iteration, BLTs still achieve the best utility, even if we convert the wins in efficiency for BIFR to wins in utility by e.g. using a larger batch size or more iterations of training (of course, the opposite is also plausible).
  • Second, if DP-SGD is competitive we can also take the argument above in the opposite direction: In settings where DP-SGD is competitive, we should just use DP-SGD and then there's no need to track noise state at all, and perhaps this enables larger batch sizes or more iterations of training. If DP-SGD achieves the best utility, then the problem of picking the best of different MF variants is moot.

In short, while I think the authors' new results are valuable for comparing the different methods, and even if BIFR does not uniformly improve on BLTs it's a meaningful contribution that BIFR allows one to trade off on multiple axes with other DP-MF methods like BLTs, I don't feel the concern is fully gone and don't feel comfortable raising my overall review score accordingly.

Dear Reviewer,

We appreciate your feedback and kindly ask you to consider the following arguments, which we hope will address your remaining concerns.

Efficiency. We also compare γ\gamma-BIFR with 64 noise vectors in large scale VaultGemma fine-tuning (see Table 8 of the revised manuscript). With 64 bands, γ\gamma-BIFR achieves substantially greater noise cancellation and outperforms BLT. Our comparison was done using the same physical batch size, under which γ\gamma-BIFR was only 10% slower than BLT due to noise regeneration. However, the substantial memory savings offered by γ\gamma-BIFR allow for larger physical batch sizes, which could further improve runtime.

Utility. We agree that BLT and DP-SGD can outperform γ\gamma-BIFR, particularly in the high-privacy regime (ϵ≤1\epsilon \leq 1). However, we believe that the boundary at which this occurs could be shifted through improved privacy accounting (in the comparison with DP-SGD) and a better tuning metric than amplified RMSE for selecting the bandwidth and γ\gamma (in the comparison with BLT). In other words, the exact boundary at which one method begins to outperform another is not necessarily fundamental and could potentially be improved through further research.

We also agree that DP-SGD should be preferred in high-privacy regimes where it outperforms γ\gamma-BIFR. However, we argue that these regimes may be less relevant to practical production deployments. For example, in the aforementioned work on practical deployment [Daly et al., 2026], the authors use a privacy budget of ρ=0.232\rho = 0.232 under concentrated DP. Converting this to (ϵ,δ)(\epsilon, \delta)-DP with δ=10−6\delta = 10^{-6} yields ϵ≈3.8\epsilon \approx 3.8. This corresponds to a low privacy regime, in which our γ\gamma-BIFR outperforms both BLT and DP-SGD by a meaningful margin; see Figures 4b and 5b.

Reviewer_xwyrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes γ\gamma-BIFR, an explicit banded-inverse matrix factorization for correlated-noise differentially private learning. A tunable fractional-power parameter generalizes BISR and DP-λ\lambda CGD, aiming to improve utility at small bandwidths while supporting noise regeneration to reduce storage requirements. The theoretical contributions include a norm-based characterization of multi-participation sensitivity and an error bound that improves the low-bandwidth dependence of BISR’s guarantee. Validation combines numerical comparisons of RMSE and subsampling-amplified RMSE with private training experiments on CIFAR-10, CIFAR-100, and IMDb. Results demonstrate improvements over BISR in low-bandwidth settings, while gains over other strong baselines are smaller and task-dependent.

Critical strengths
  1. Useful theoretical analysis of the low-bandwidth regime. Lemma 1 relates multi-participation sensitivity to standard matrix norms, while Theorem 2 establishes an improved low-bandwidth error guarantee. These results provide analytical insight beyond numerical parameter tuning and constitute the paper’s strongest contribution.

  2. A simple, interpretable unification of existing mechanisms. γ\gamma -BIFR recovers BISR and DP-λ\lambda CGD as special cases and introduces only one scalar parameter at a fixed bandwidth. The numerical results demonstrate that adjusting this parameter can substantially improve on BISR at small bandwidths. This offers a useful alternative to optimizing an entire matrix while retaining an explicit, analyzable construction.

Critical weaknesses
  1. The central efficiency–utility tradeoff is insufficiently validated. The paper motivates γ\gamma -BIFR through memory efficiency and negligible computational overhead, but provides no direct peak-memory or per-step runtime comparison for its regeneration implementation, particularly at p=64p=64 (Sections 3–4). Moreover, accuracy gains over strong alternatives are often modest: Table 5 reports 68.51±2.2168.51\pm2.21 versus 68.18±2.5068.18\pm2.50 for BSR at ϵ=8\epsilon=8. Together, these gaps leave the practical significance of the proposed extension insufficiently established.

  2. The privacy scope of the experimental procedure is unclear. Appendix A describes 25 learning-rate trials followed by retraining on the full training set, without specifying independent public tuning data or privacy accounting for selection and retraining. It is therefore unclear whether the reported (ϵ,δ)(\epsilon,\delta) budgets cover the complete procedure or only training with a fixed configuration. This distinction matters because the claimed utility improvements are evaluated at those stated privacy budgets.

Questions for authors
  1. Clarification — privacy scope: Do the reported (ϵ,δ)(\epsilon,\delta) budgets cover learning-rate selection and final retraining, or only a single fixed-configuration training run? What public-data assumptions or privacy accounting support the tuning procedure described in Appendix A?

  2. Clarification and supporting evidence — efficiency: Did the reported training experiments actually use noise regeneration? What measured peak-memory and per-step runtime results support the negligible-overhead claim, particularly at p=64p=64, relative to DP-SGD and buffered implementations?

  3. Clarification and supporting evidence — utility gains: Were baseline comparisons paired across random seeds? For the principal comparisons with BSR and DP- λ\lambda CGD, can you provide the paired accuracy differences and their uncertainty to establish whether the reported gains are reliable?

2 comments

We thank the reviewer for their feedback! We address the raised concerns and provide additional clarifications below.

>> Clarification of the privacy scope

The reported (ε,δ)(\varepsilon,\delta) budgets cover individual training runs with fixed hyperparameters, but do not include learning rate tuning. As in related work, we select learning rates using a held-out validation set and report the mean accuracy and confidence intervals over three runs. This is a common benchmarking protocol that makes it as easy as possible to compare DP learning methods with each other, because in practical deployments, hyperparameter tuning can be done in different ways, let it be by hypothesis testing on private data (see e.g. [Xiang et al., NDSS 2026]), or using public validation data without additional privacy costs.

We wish to emphasize that our method does not introduce additional data dependent hyperparameter tuning, because γ\gamma is selected by minimizing amplified RMSE (equation 6), which is just a function of the matrices BB and CC, not of any potentially private validation data. We have included the clarification in the Experiments and Appendix A sections of the revised manuscript.

>> Clarification and supporting evidence for efficiency

By design, noise regeneration reproduces the same noise vectors as storing the past noise vectors in memory when using the same random seeds, and therefore does not affect the reported accuracies. Since our original experiments fit within the available GPU memory, we used the noise buffer for simplicity. We agree that direct memory and runtime comparisons are useful, particularly at p=64p=64. We have now done these experiments and reported the results in our response to reviewer yG27, as well as in Tables 8, 9 and 10 of Appendix A.2 of the revised manuscript.

In particular, for fine-tuning the 1B parameter VaultGemma [1] model on IMDb dataset, γ\gamma-BIFR with p=64p=64 and noise regeneration requires only 28.4 GB of GPU memory, compared with 44.2 GB for BLT, at the cost of only 10% additional runtime. The savings in memory can easily be translated into time savings by allowing larger physical batches to be used. In particular, we show a 29% reduction in time compared to BLT when using noise regeneration with a larger physical batch size that would otherwise not fit on a single GPU with BLT.

These results also extend the evaluation of noise regeneration in DP-λ\lambdaCGD (Kalinin et al., 2026a) to larger bandwidths (p=64p=64), showing that substantially more noise cancellation is possible with relatively little additional runtime overhead in large scale training.

[1] Sinha et al. “VaultGemma: A Differentially Private Gemma Model” 2025 arXiv:2510.15001

>> Clarification and supporting evidence for utility gains

All methods were evaluated using the same three random seeds, ensuring paired comparisons. We already report the mean accuracies and their 95% confidence intervals over these runs in Tables 2-7 of Appendix A. We have now clarified this in the Experiments section of the revised manuscript.

Dear Reviewer xwyr,

We hope that our rebuttal has addressed the questions and concerns raised in the review. In case any concerns remain, we kindly ask you to let us know before the rebuttal deadline.

Best regards, Authors

Reviewer_yG27OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper considers matrix factorization mechanism in DP and proposes a generalization of DP-λCGD. The generalized algorithm preserves the memory-free property of the original algorithm.

The authors characterize the sensitivity of Toeplitz matrix under multi-participation and derive the RMSE bound of the proposed algorithm. The experiments are carried out to compare the algorithm with BISR, among others.

Critical strengths

The presentation is clear and the paper is well motivated and easy to follow. Theoretical results are well established, the lemma of which might be of independent interest (bound the sensitivity of Toeplitz matrix).

Critical weaknesses

The paper builds on DP-λCGD and thus the contribution might be incremental. Experiments did not seem to establish significant advantages of the algorithm.

Questions for authors

For experiments, do you have stats of memory usage and can you explain how these experiments show the advantages of your algorithm? It's hard for me to follow Figure 4, as the line plots with shades have a lot of overlap.

2 comments

We thank the reviewer for their time and thoughtful feedback!

>> On the novelty of our contribution:

Our proposed method indeed generalizes the DP-λ\lambdaCGD mechanism. However, there are, in principle, many ways one could generalize DP-λ\lambdaCGD to larger bandwidths, and we would argue that our construction is especially appealing because it achieves several important properties simultaneously:

  1. It still only has a single hyperparameter to tune.
  2. Is theoretically tractable for RMSE analysis.
  3. Achieves RMSE errors close to the numerically optimal ones, see Figure 1.
  4. Achieves theoretically optimal RMSE in the large-bandwidth limit.
  5. It unifies the prior works on DP-λ\lambdaCGD and BISR.

We would argue that the combination of these properties makes our proposed method more than an incremental improvement over DP-λ\lambdaCGD.

>>On the experimental improvements:

On the training tasks (CIFAR-10 in Figure 4 and CIFAR-100 in Figure 5), we show an improvement over previous methods, at least in the lower-privacy regime ε≥1\varepsilon \ge 1. Indeed the improvements themselves are not large. One possible explanation is that, after a series of works proposing increasingly optimized matrix mechanisms, CIFAR-10 as a DP benchmark is approaching saturation and making any further progress is increasingly hard. Quoting a remark made in “Near Exact Privacy Amplification for Matrix Mechanisms” by Choquette-Choo et al. (ICLR 2025):

"It is worth mentioning that accuracy of training on the CIFAR-10 benchmark with correlated noise has been highly optimized for over a long series of papers (e.g. (Choquette-Choo et al., 2022, 2024a)). So, any tangible and unconditional improvement is significant."

>> On the readability of Figure 4

We appreciate the concern. In the revised manuscript, we have made the shaded regions more transparent to improve readability. However, since the curves are quite close to each other, some overlap is difficult to avoid. The numerical accuracies are also available in Tables 4 and 5 for a more precise comparison.

>> On memory usage

We are happy to provide memory usage measurements for different methods, together with the corresponding mean time per epoch.

The advantage of our proposed method is most clear in large scale training. During the rebuttal period, we did additional experiments fine-tuning VaultGemma [1], a 1B-parameter model originally trained with DP-SGD. We fine-tuned it on the IMDb dataset, using a batch size of 512 and a physical batch size of 2. In the table below, we observe that BLT requires approximately 1.6 times as much memory as γ\gamma-BIFR with noise regeneration, while γ\gamma-BIFR with p=64p=64 causes only a 10% increase in runtime compared with BLT. In principle, the memory usage of noise regeneration could be further reduced to that of DP-SGD. This would require generating noise in smaller chunks (rather than all at once for each parameter group), potentially at the cost of additional runtime, bringing the memory ratio between BLT and γ\gamma-BIFR closer to 1.9. This has two practical implications: first, γ\gamma-BIFR could be used to train larger models that may not fit on a single GPU with BLT; and second, the additional available memory could allow us to use a larger physical batch size for γ\gamma-BIFR.

The savings in memory can easily be translated into time savings by allowing larger physical batches to be used, which we demonstrate in the following tables. In particular, we show a 29% reduction in time compared to BLT when using noise regeneration with a larger physical batch size that would otherwise not fit on a single GPU with BLT.

MethodMemory w/o regen. (MB)Memory w/ regen. (MB)Time w/o regen. (s)Time w/ regen. (s)
γ\gamma-BIFR p=64p=64--28420.0--1114.0 ± 1.5
γ\gamma-BIFR p=4p=453221.828421.01021.9 ± 4.71022.0 ± 4.4
DP-λ\lambdaCGD p=2p=244916.928421.51018.1 ± 1.31018.1 ± 0.7
BLT b=4b=444155.9--1012.7 ± 0.3--
DP-SGD23423.2--1025.5 ± 8.8--

To test whether increasing the physical batch size helps, we also did experiments with a physical batch size of 4 for noise regeneration with p=4p=4. Note that storing the noise for p=4p=4 in memory with a physical batch size of 4 would not fit on a single GPU, and increasing the physical batch size to, e.g. 8 would not fit on the GPU even with regeneration. The results show a 29% reduction in time compared to BLT with a physical batch size of 2, as BLT would not fit on the GPU with a larger physical batch size.

MethodPhysical batch sizePeak allocated (MB)Mean epoch time ± 95% CI (s)
BLT b=4b=4244155.901012.7 ± 0.3
γ\gamma-BIFR p=4 (w/o regen.)253221.751021.9 ± 4.8
γ\gamma-BIFR p=4 (w/ regen.)228420.991022.1 ± 4.4
γ\gamma-BIFR p=4 (w/ regen.)436721.56716.9 ± 0.5

Second, we report the results for the VGG model on CIFAR-10, a smaller scale experiment with a physical batch size of 32 and a batch size of 512. In this setting, storing noise history for p=4p=4 has little impact on memory usage, and even for p=64p=64, memory requirements remain modest (455 MB). Noise regeneration at p=4p=4 introduces only about 17% runtime overhead, corresponding to our "low-memory" experiments. However, for the "high-memory" experiments with p=64p=64, the overhead is 2.4x, making noise storage preferable for smaller models where memory is less constrained.

MethodMemory w/o regen. (MB)Memory w/ regen. (MB)Time w/o regen. (s)Time w/ regen. (s)
γ\gamma-BIFR p=64p=64455.4294.93.83 ± 0.029.13 ± 0.03
γ\gamma-BIFR p=4p=4303.9294.93.71 ± 0.014.33 ± 0.02
DP-λ\lambdaCGD p=2p=2297.6294.93.71 ± 0.014.02 ± 0.03
BLT b=4b=4304.4--3.80 ± 0.01--
DP-SGD294.9--3.65 ± 0.10--

Taken together, these results show that γ\gamma-BIFR makes higher-bandwidth (higher accuracy) matrix mechanisms practical across different memory constraints. When sufficient memory is available, storing the noise history is preferable, while in memory-constrained settings, noise regeneration substantially reduces memory usage, which in turn enables larger physical batch sizes and reduced training time. These results have been included in the revised manuscript; see Tables 8, 9, and 10 of Appendix A.2.

[1] Sinha et al. “VaultGemma: A Differentially Private Gemma Model” 2025 arXiv:2510.15001

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.