Can Variable Noise Be Data-Independent? Differentially Private Bimodal SGD

DP-SGD serves as the de facto baseline for private deep learning, yet its constant noise scale imposes an unfavorable dilemma: small noise rapidly depletes privacy budgets, while large noise degrades gradient signals. While adaptive noise mitigates this issue, its data dependence violates essential data independence, risking leakage. We propose Differentially Private Bimodal SGD (DP-BiSGD), a strictly data-independent baseline that stochastically alternates between high-probability small perturbations for local exploitation and low-probability large perturbations for global exploration. To translate this concept into a viable baseline, we address two challenges inherent in this bimodal formulation. Specifically, we develop a bypass-scaled update scheme that shields the momentum buffer from extreme-variance corruption, coupled with a dynamic GMM-PLD accountant to overcome loose standard accounting and ensure tight privacy bounds. Extensive experiments demonstrate that DP-BiSGD consistently outperforms DP-SGD in model utility under identical privacy guarantees, preserves strong resilience against inference attacks, and serves as a seamless drop-in replacement.