Training-Data Leakage from Optimizer State in Resumable Checkpoints

Modern training systems routinely save resumable checkpoints that contain optimizer state in addition to model weights. These artifacts are widely stored and shared, yet their privacy implications remain largely unexplored. In this paper, we show that optimizer state can reveal information about the training data because it records a transformed history of training gradients. The information exposed depends on how many checkpoints are available. A single checkpoint can reveal whether specific features, such as tokens or classes, were present in the training data through the traces they leave in optimizer state. These traces can remain detectable long after their original contribution to the optimizer state has decayed, a persistence we call privacy memory. With two adjacent checkpoints, differencing signed optimizer states can recover the optimizer-input gradient. This allows existing gradient-inversion attacks to be applied directly to saved checkpoints. We validate these mechanisms across models, optimizer representations, and standard training frameworks. Our results show that resumable checkpoints should be treated as sensitive training artifacts rather than ordinary model snapshots.