Structured Sampling for Stochastic Depth via Boolean Fourier Analysis

Stochastic depth is a key technique for training residual networks, which randomly drops residual blocks during training to mitigate overfitting. Standard estimation of the expected gradient may result in high variance and unstable convergence due to averaging over i.i.d. masks. To address this issue, we formulate gradient estimation under stochastic depth as a Fourier projection problem over the Boolean cube. Building on this formulation, we develop two structured sampling designs that eliminate selected spectral components while preserving the original expectation: a coordinate-balanced design that removes first-order components at low sample cost, and an equiprobable stratification framework that cancels a user-specified set of modes exactly and guarantees covariance dominance. We derive explicit lower bounds on the variance reduction for both designs, and analyze their implications for stochastic gradient descent, including improved parameter-error decay and lower averaged training loss, under mild assumptions. Numerical experiments on CIFAR-10 and CIFAR-100 show that our designs substantially reduce gradient-estimation variance and slightly improving the accuracy and stability of training.