- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes TIDE. It adds fresh Gaussian noise to clipped per-example gradients and maintains a client-level exponential moving average correction. The aggregated perturbations telescope into boundary terms. Under smoothness, exact gradients, inactive clipping and a fixed learning rate, the paper derives an O(η²) stationarity upper bound and a tuned O(T^(-2/3)) bound, compared with O(η) and O(T^(-1/2)) bounds for independent noise. It also gives an exact-gradient-preimage distance bound and a conditional matrix-Gaussian sensitivity analysis. Experiments cover CIFAR-10/100, Tiny-ImageNet, ViT fine-tuning, partial participation and several inversion attacks, including a TIDE-aware multi-round objective. The main tables suggest useful accuracy retention at larger noise levels. However, they do not establish a general privacy advantage at a matched end-to-end privacy budget. I recommend weak rejection: the practical idea is promising, but the security interpretation, novelty positioning and inconsistent experimental reporting require substantial revision.
Critical strengths
- Simple mechanism with a useful restricted optimization analysis. The identity sum_t(ξ̄_t + e_(t-1)) = (e_(-1) - e_(T-1))/(1-β) is transparent, and the auxiliary-iterate argument in Appendix A.2 explains why temporally correlated perturbations can reduce optimization damage. The stationary initialization and explicit dependence on β make the construction understandable and implementable. This is a credible basis for further study, although it does not itself prove privacy.
- Broad empirical evidence of utility retention. At σ = 0.003, Table 1 reports CIFAR-10 accuracy of 91.16 ± 0.17% for TIDE versus 87.69 ± 0.39% for DP-SGD; CIFAR-100 gives 68.57 ± 0.50% versus 62.86 ± 0.41%. Appendix I extends utility evaluation to 100 non-IID clients, and Appendix F explicitly considers temporal correlation through a covariance-aware attack. Including these settings, clipping ablations and perceptual metrics is valuable. The gains over the stronger DP-Step-Decay baseline are smaller, and the privacy results remain specific to the evaluated attacks.
Critical weaknesses
- The claimed security and theoretical scope exceed what is established. Theorem 3.1 bounds the distance to an exact solution of G_w(x′) = g′. If the noisy gradient lies outside the gradient map's image, the defined distance is infinite; approximate inversion may still recover x accurately. For an attack with residual ρ, the same argument gives only max(0, (||g′ - G_w(x)|| - ρ)/l_g), so a perturbation-norm bound alone is insufficient. Appendix A.6 usefully distinguishes conditional sensitivity energy from subsampled RDP, but no full-transcript (ε, δ) comparison is reported. Table 2 even gives a 34.1-fold conditional divergence ratio at T = 200, full participation and 10-fold TIDE noise; its q < 1 rows are not privacy guarantees. Larger marginal noise therefore does not establish stronger overall privacy. Appendix F tests only T = 10, lacks a GLS positive control and omits crucial tuning details. Its raw-gradient forward model also needs reconciliation with active clipping. Separately, Appendix A.2 analyzes exact gradients without active clipping, momentum or learning-rate decay, whereas Appendix B uses all these practical features. Mini-batch sampling noise does not telescope with the injected noise. The abstract should describe bounds for this restricted model rather than a demonstrated general convergence-rate separation for practical DP-SGD.
- Novelty and empirical reliability are insufficiently resolved. At β = 0, TIDE's first-difference noise matches the Anti-PGD construction up to scaling/initialization; Orvieto et al. (ICML 2022) and Koloskova et al. (NeurIPS 2023) are directly relevant but absent from the references. The latter already studies correlated-noise optimization, including nonconvex objectives. Appendix A.3 mentions DP-FTRL/matrix mechanisms, yet there is no substantive comparison establishing the incremental contribution. Reporting conflicts further weaken the evidence: Section 5.4 gives ViT accuracy 90.52% versus 83.17%, but Table 8 gives 89.27% versus 86.65% at σ = 0.03. The 27.45% DP result is at σ = 0.1 in Table 8, not 0.3, and Table 9 gives TIDE SSIM 0.0470, not 0.034. All five SSIM means and standard deviations for TIDE at σ = 0.01 are identical across Tables 1a and 1b. These require source-log verification, not an assumption of misconduct. Algorithm 1 line 4 also uses σ where Equation (1) uses σ_p, changing variance by a factor of B. Without resolving the released object, noise normalization and experiment provenance, the comparisons cannot be reliably reproduced.
Questions for authors
Clarifications and evidence that could change the recommendation:
- What does the server actually receive: all per-example gradients, a batch mean, or local model updates? Please reconcile Algorithm 1 line 4 with Equations (1)-(2), and identify whether every table uses σ or σ_p, the clipping threshold, and the training/attack batch sizes. How does Appendix A.6 model the shared batch state for that exact transcript?
- Can the safety argument bound approximate reconstruction error under the stated attacker, and can privacy/utility be compared at a common full-training (ε, δ), adjacency and sampling protocol? Otherwise, please narrow the privacy claims explicitly to the tested attacks.
- Please provide corrected ViT results and source logs for the repeated CIFAR SSIM rows. Were DP-SGD and DP-Step-Decay attack metrics intentionally reused, and at which checkpoint/noise level were they measured?
- What are the GLS attack's λ, γ, optimizer, restarts, sample count and clipping-aware forward model? Does it succeed on an undefended or weak-noise positive control? A longer observation horizon would materially test the claimed temporal robustness.
- What is new relative to Anti-PGD and correlated-noise/DP-FTRL analyses? Please separate the EMA/defense contribution from prior noise-cancellation ideas and explain which convergence claims survive mini-batch noise and active clipping. Please also report main-benchmark seed/sample counts, error-bar definitions, training lengths, aggregation details and checkpoint selection.
4 comments
Response to Reviewer Y6oy
W2 & Q5. Contribution, convergence analysis, and experimental evidence
Prior work and contribution. We expanded Related Work to include Anti-PGD (Orvieto et al., 2022), Koloskova et al. (2023), and DP-FTRL/matrix-factorization methods. Anti-PGD's first-difference noise corresponds to TIDE's case up to scaling and initialization. Koloskova et al. provide a general linearly correlated-noise framework recovering the same asymptotic order in the corresponding fixed-step setting. TIDE develops an EMA-parameterized, client-local realization for gradient-inversion defense and studies reconstruction resistance and training utility. The reported multi-round ablation identifies as the weakest reconstruction-defense setting among the evaluated values.
Direct correlated-noise baseline. We added a binary-tree DP-FTRL comparator using the same CIFAR training protocol and nominal noise-scale parameter. Three-seed best-validation accuracy (%, mean sample SD) is:
| Dataset | DP-SGD | Tree DP-FTRL | TIDE | |
|---|---|---|---|---|
| CIFAR-10 | 0.003 | 87.69 0.39 | 91.37 0.20 | 91.16 0.17 |
| CIFAR-100 | 0.003 | 62.86 0.41 | 68.11 0.34 | 68.57 0.50 |
| CIFAR-10 | 0.03 | — | 87.60 0.20 | 90.60 0.45 |
| CIFAR-100 | 0.03 | — | 60.12 0.07 | 67.46 0.30 |
At , TIDE and tree DP-FTRL achieve comparable utility; at , TIDE has 3.00/7.34 percentage-point higher validation accuracy on CIFAR-10/CIFAR-100, respectively.
Convergence assumptions and stochastic gradients. The theorem uses fixed learning rate and inactive clipping to isolate the impact of injected temporally correlated noise. For conditionally unbiased mini-batch gradients , satisfying and , the Appendix extends the stationarity bound with additional and terms for TIDE and DP-SGD, respectively, under the proof's constants. The corresponding injected-noise terms scale as for TIDE and for DP-SGD in the simplified setting. The principal experiments additionally evaluate active clipping and scheduled learning rates.
Theory-aligned training controls. We added no-clipping/fixed-LR results (200 epochs, three clients, three seeds), reporting best-validation accuracy (%, mean sample SD):
| Dataset | DP-SGD | TIDE | |
|---|---|---|---|
| CIFAR-10 | 0.003 | 84.25 0.48 | 88.24 0.57 |
| CIFAR-100 | 0.003 | 58.29 0.71 | 66.09 0.23 |
| CIFAR-10 | 0.03 | 31.66 0.66 | 87.16 0.54 |
| CIFAR-100 | 0.03 | 4.66 0.38 | 60.67 0.37 |
These fixed-step comparisons complement the theorem's convergence-speed bound with same-epoch-budget training outcomes. The CIFAR training setup uses batch size 128, 200 epochs, three clients, and three training seeds; the reported training accuracy is best validation accuracy. Single-round reconstruction uses ten target batches of size four (40 images) at a shared initialized sigmoid-activated ResNet-18. GLS SSIM error bars are calculated across five attack evaluations. The manuscript and supplementary materials contain the relevant evaluation protocol and implementation.
Q1. Released gradient, noise normalization, and sensitivity protocol
Clients communicate one defended batch-averaged gradient per training step and retain their EMA correction states internally. Algorithm 1 now distinguishes the client-side computation of per-example gradients, used for clipping and batch aggregation, from the final communicated gradient:
e_t=\beta e_{t-1}-(1-\beta)\bar\xi_t.$$ The experimental $\sigma$ specifies the **batch-level Gaussian innovation scale**; the equivalent independent per-example construction uses $\sigma_p=\sqrt B\,\sigma$. The CIFAR training evaluation uses **batch size 128**, while inversion is conducted **at initialization** using a shared sigmoid-activated ResNet-18, **batch size four**, and **ten target batches (40 images)**. This attack batch size allows informative reconstruction at a manageable joint-inversion difficulty. The training protocol specifies the clipping threshold and scheduled-learning-rate configuration, while the fixed-LR, inactive-clipping experiment separately tests the assumptions used in the convergence proof. Appendix A.6 measures the **conditional sensitivity energy of the time-ordered sequence of released batch means** for a sample participating at designated training steps. The calculation includes the covariance generated by the client-local EMA correction and examines full and partial participation scenarios. This identifies the release transcript and participation patterns underlying the conditional divergence calculation.Thank you for the detailed response and additional experiments. Some of my concerns have been addressed.
However, I believe two major concerns remain insufficiently resolved. First, the paper still lacks a privacy analysis covering the full training process and a privacy–utility comparison under the same end-to-end privacy budget. The existing conditional sensitivity analysis and empirical attack results are insufficient to fully support the paper's broader privacy claims. Second, although the additional convergence analysis and experiments are valuable, the claimed convergence-rate advantage has not been sufficiently established under practical training conditions involving mini-batch sampling noise, gradient clipping, and learning-rate schedules. Furthermore, while the authors have further clarified the differences from existing correlated-noise methods, the incremental theoretical contribution remains limited.
I appreciate the authors' efforts to improve the paper. Nevertheless, these remaining concerns continue to affect my overall assessment. Therefore, I will maintain my original score.
Thank you for your follow-up and for acknowledging our additional analyses and experiments. We would like to further clarify the following three points.
1. End-to-end privacy accounting. While tree-based DP-FTRL constrains cumulative noise variance to , TIDE can retain growth with a substantially reduced coefficient and bounded relative sensitivity energy by generalizing its error-correction update to , with . Within this generalized framework, DP-SGD and the original TIDE correspond to and , respectively.
For , the inverse noise-generation matrix exhibits exponentially decaying entries, and the conditional sensitivity energy ratio in Appendix A.6 satisfies
(1-q)\left(1+\frac{\gamma^2}{1-(\beta+\gamma)^2}\right) +q\left(\frac{1-\beta}{1-\beta-\gamma}\right)^2.$$ This upper bound is independent of the training horizon. For example, with $\beta=0.5$, reducing $\gamma$ from $0.5$ to $0.45$ reduces the 34.1-fold conditional divergence ratio highlighted by the reviewer to 0.862 under the same setting ($T=200$, full participation, and 10-fold noise scale), while still eliminating 99% of the asymptotic cumulative noise variance relative to independent noise of the same scale. This provides a principled extension of TIDE toward balancing gradient reconstruction resistance, optimization utility, and end-to-end differential privacy. We further empirically validate this generalized formulation on CIFAR-10 with $\beta=0.5$ and $\gamma=0.45$: | Method | $\sigma=0.003$ | $\sigma=0.03$ | | --- | --- | --- | | DP-SGD | $87.69\pm0.39$ | - | | Original TIDE | $91.16\pm0.17$ | $90.60\pm0.45$ | | Generalized TIDE | $91.17\pm0.05$ | $90.56\pm0.32$ | Generalized TIDE achieves nearly identical test accuracy to the original TIDE while substantially reducing its conditional sensitivity energy. These theoretical and empirical results demonstrate the flexibility of TIDE in balancing noise cancellation and transcript sensitivity, providing a principled direction toward end-to-end privacy–utility optimization. **2. Convergence under practical training conditions.** Thank you for bringing up this remaining concern. We have provided two additional theoretical analyses in Proposition A.5 (*Convergence with unbiased mini-batch gradients*) and Proposition A.6 (*Synchronous fixed-weight client aggregation*). The former explicitly incorporates mini-batch gradient variance into the convergence bounds of both TIDE and DP-SGD, while preserving TIDE's control over accumulated injected noise. The latter establishes that the noise cancellation property extends to synchronous multi-client training. Our main experiments incorporate active gradient clipping and scheduled learning rates, and the additional experiments with inactive clipping and fixed learning rates further validate the theoretical predictions under the corresponding assumptions. Together, these results establish the optimization benefit of temporal noise cancellation in the analyzed settings and demonstrate its empirical effectiveness under practical training configurations. **3. Theoretical contribution.** We thank the reviewer for highlighting the connections to Orvieto et al. (ICML 2022) and Koloskova et al. (NeurIPS 2023). We would like to emphasize three distinctions that clarify the contributions of TIDE. First, TIDE generalizes the Anti-PGD mechanism of Orvieto et al. through an EMA-based error-correction framework. Anti-PGD corresponds to the special case of $\beta=0$, while TIDE allows $\beta\in[0,1)$, providing an additional degree of freedom for controlling temporal noise correlations. Second, while the general convergence framework of Koloskova et al. can encompass TIDE's noise structure, our theoretical analysis addresses a different objective: gradient inversion defense. Specifically, Theorems 3.1 and 3.2 jointly characterize the instantaneous perturbation governing gradient reconstruction and the accumulated perturbation governing optimization. This connects reconstruction resistance with convergence behavior within a unified, explicitly parameterized mechanism. Third, our empirical findings demonstrate that $\beta=0$ does not necessarily provide the best reconstruction resistance (Table 9). This observation is particularly important because it shows that the special case considered in prior studies is not always the most favorable choice in practice. We hope these clarifications further highlight the motivation and contributions of our work, and we appreciate your constructive feedback throughout the discussion.
Response to Reviewer Y6oy
We thank the reviewer for the detailed and technically insightful feedback. We have refined the theoretical statements, expanded the experimental comparisons and attack evaluations, and verified and corrected the reported numerical results. We address the theoretical, experimental, and reproducibility questions in detail below.
W1 & Q2. Approximate inversion theorem and privacy analysis
We refined the gradient-matching reconstruction objective to allow an explicit matching residual. For a clean gradient map Gw, defended observation g′, and tolerance r, define
dw(r)(x;g′)=infx′:∥Gw(x′)−g′∥≤r∥x′−x∥.
Under lg-Lipschitz continuity, the geometric argument gives
dw(r)(x;g′)≥lg[∥g′−Gw(x)∥−r]+.
When g′=Gw(x)+ξ, the numerator becomes [∥ξ∥−r]+. The Gaussian reconstruction-distance estimate is stated with the same residual tolerance. The revised formulation explicitly relates perturbation magnitude, achievable gradient-matching accuracy, and reconstruction displacement.
Appendix A.6 analyzes the conditional sensitivity energy of a sequence of released batch-averaged gradients, given a target sample's specified training-step participation pattern. It includes full- and partial-participation examples and accounts for the correlations induced by the client-local EMA correction. The reported conditional divergence energies characterize these participation patterns; the empirical privacy assessment reports reconstruction performance for the specified inversion attacks. A full-training privacy budget also depends on the adjacency relation, sampling protocol, and composition of the observed transcript. The new tree DP-FTRL training comparison uses the stated nominal noise scales and is interpreted as an optimization-utility comparison.
W1 & Q4. Covariance-aware multi-round reconstruction and attack protocol
We expanded multi-round reconstruction evaluation with covariance-aware GLS applied to T=10 and T=20 consecutive gradient releases from the same fixed batch of four images. The attack uses λ=1, Adam with learning rate 0.06, and five attack evaluations per condition. SSIM is reported as mean ± standard deviation across those evaluations:
The undefended controls yield SSIM above 0.63 at both horizons, establishing an effective reconstruction reference. Longer observation horizons increase reconstruction SSIM for both defended configurations. TIDE maintains low SSIM at the evaluated noise scale. We report TGIAs-RO and GLS in separate Appendix tables and describe the consecutive same-batch observation protocol and the attack's stated optimization settings. The experimental comparisons use the distinct reported noise scales for DP-SGD and TIDE.
W2 & Q3. Numerical corrections and experimental provenance
We verified the result tables and corrected the affected numerical descriptions. For ViT-B/16 on CIFAR-100 at σ=0.03, the accuracies are 89.27% for TIDE and 86.65% for DP-SGD, and TIDE's CI-Net SSIM is 0.0470. The DP-SGD accuracy of 27.45% corresponds to σ=0.1.
The duplicated TIDE σ=0.01 CIFAR reconstruction values came from inserting the CIFAR-100 measurements into the CIFAR-10 row. The corrected CIFAR-10 means for GI, IG, iDLG, CI-Net, and DGGI are 0.0267, 0.0180, 0.0186, 0.0281, and 0.0235, respectively; the CIFAR-100 measurements are retained.
The inversion attacks are evaluated at model initialization. At that point, DP-SGD and DP-Step-Decay use the same gradient perturbation mechanism and noise scale, so their reported attack metrics use the same initialization-stage attack measurements. Model-training accuracy is evaluated separately. The manuscript clarifies the shared measurements, the batch-level σ convention, and the corresponding experimental provenance.