acceptodds
Withdrawn from ICLR 2027

Understanding the Impact of Differentially Private Training on Memorization of Long-Tailed Data

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent research shows that modern deep learning models achieve high predictive accuracy partly by memorizing individual training samples. Such memorization raises serious privacy concerns, motivating the widespread adoption of differentially private training algorithms such as DP-SGD. However, a growing body of empirical work shows that DP-SGD often leads to suboptimal generalization performance, particularly on long-tailed data that contain a large number of atypical samples. Despite these observations, a theoretical understanding of this phenomenon remains largely unexplored, and existing differential privacy analysis are difficult to extend to the nonconvex and nonsmooth neural networks commonly used in practice. In this work, we develop the first theoretical framework for analyzing DP-SGD on long-tailed data from a feature learning perspective. We show that the test error of DP-SGD-trained models on the long-tailed subpopulation is significantly larger than the test error over the entire dataset. Our analysis further characterizes the training dynamics of DP-SGD, demonstrating how gradient clipping and noise injection jointly adversely affect the model’s ability to memorize informative but atypical samples. Finally, we validate our theoretical findings through extensive experiments on both synthetic and real-world datasets.

Primary area:
learning theory
Keywords:
differential privacy, data memorization, learning theory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_G14ROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the impact of memorization using DP-SGD on long-tailed data. Providing new insights on the training dynamics and generalization.

Critical strengths

This paper has more fine-grained analysis than the prior work.

Critical weaknesses
  1. It would be good to have a table of notations for the symbols.
  2. While proving some bounds is useful, the expressions are so complicated. It's difficult to understand the behavior or use it in practice.
Questions for authors
  1. The long-tailed data are identified by non-DP tranining. But the analysis is on DP training, how do you ensure these data are still long-tailed?
  2. Can you explain how do you define the signal-to-noise ratio and more importantly, the noise correlation ratio?
  3. What are the main difficulties to generalize to deep models?
  4. There're several conditions (4.1), does your experiments satisfy all of them?
  5. Does your analysis applied to other DP algorithms, such as the DP-FTRL?
Reviewer_Y3eeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper investigates an idea of memorization in DP-SGD, especially in long-tailed data. The authors suggest the theoretical results on 2-layer neural networks, based on the well-studied concepts in standard (non-DP) training.
Critical strengths
  • The paper investigates a specific concept of memorization and long-tailed data in DP-SGD scenarios.
  • The paper has lots of theoretical analysis on the specific scenario.
Critical weaknesses
  • Please refer to the Questions below.
Questions for authors
  • I believe that the most critical weakness of this paper is the contribution of this paper. I agree that the theoretical understanding of both memorization and long-tailed data is important. However, the paper investigates both concepts in a very specific scenario suggested by [1].

  • Furthermore, I believe that most of the theoretical explanation and experiments merely utilize the concepts in [1]. This is worth investigating, but I think the fitness of the setups in [1] and DP is not well explained. Can you explain the novelty in this perspective or the detailed rationale for choosing the framework in [1]?

  • The tightness of the bound or alignment with conventional theoretical papers in DP or DP-SGD is ambiguous. Comparing the proposed theoretical results with those of previous theoretical papers on the DP side should be investigated.

  • To experimentally back up the memorization ⟨wy,r,ξi⟩\langle w_{y,r},\xi_i\rangle in DP-SGD setups, can the authors at least do an ablation study of clipping-only or perturbation-only analysis and coherence with the theoretical parts? As the inner product itself contains both the norm and cosine similarities, can the authors explain this in detail?

  • The concept of long-tailed data is different from the conventional concepts. In Appendix A, the authors visualized the noise pattern memorization of long-tailed data. Can the authors clarify the connection between noise memorization and long-tailed distribution in detail?

  • The explanation of the given conditions in Condition 4.1 should be provided.

[1] Rethinking Benign Overfitting in Two-Layer Neural Networks, ICML 25

Reviewer_KphZOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how DP-SGD affects memorization and generalization on long-tailed data. Through theoretical analysis of two-layer ReLU CNNs and empirical experiments, the authors argue that gradient clipping and Gaussian noise suppress noise memorization, leading to worse generalization on long-tailed samples.

Critical strengths

The paper investigates the impact of DP-SGD on memorization and generalization of long-tailed data and provides insights into the underlying mechanisms.

Critical weaknesses
  1. Lemma E.1 claims that the activation set is monotonically non-decreasing. However, the proof does not properly account for Gaussian noise in DP-SGD, which may deactivate previously activated neurons.

  2. The proof appears to use full-batch gradient updates instead of mini-batch SGD. In particular, the proofs assume that each training sample contributes to every iteration, and the sampling scheme is not clearly specified.

  3. The theoretical assumptions are not satisfied by the synthetic experimental settings. Moreover, the real-world experiments use a covariance-based influence score to identify long-tailed samples, without establishing its connection to the theoretical definition in Definition 4.7.

Questions for authors
  1. How can the authors guarantee activation-set monotonicity despite Gaussian noise in DP-SGD?

  2. Does the proof consider mini-batch SGD? If so, how is sampling randomness handled?

  3. Why do the synthetic experiments not satisfy the theoretical assumptions? How is the real-world definition of long-tailed data related to Definition 4.7?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.