acceptodds
Under review as a conference paper at ICLR 2027

Training-Data Leakage from Optimizer State in Resumable Checkpoints

Abstract

Modern training systems routinely save resumable checkpoints that contain optimizer state in addition to model weights. These artifacts are widely stored and shared, yet their privacy implications remain largely unexplored. In this paper, we show that optimizer state can reveal information about the training data because it records a transformed history of training gradients. The information exposed depends on how many checkpoints are available. A single checkpoint can reveal whether specific features, such as tokens or classes, were present in the training data through the traces they leave in optimizer state. These traces can remain detectable long after their original contribution to the optimizer state has decayed, a persistence we call privacy memory. With two adjacent checkpoints, differencing signed optimizer states can recover the optimizer-input gradient. This allows existing gradient-inversion attacks to be applied directly to saved checkpoints. We validate these mechanisms across models, optimizer representations, and standard training frameworks. Our results show that resumable checkpoints should be treated as sensitive training artifacts rather than ordinary model snapshots.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
training-data privacy, optimizer-state leakage, resumable checkpoints, gradient inversion, attribute inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZovJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates privacy leakage from optimizer states stored in resumable training checkpoints, demonstrating that a single checkpoint can reveal the provenance of training features. The paper validates these leakage mechanisms across different models, optimizers, and standard training frameworks.

Critical strengths
  1. The paper introduces and analyzes the concept of privacy memory, demonstrating that detectable traces of training features can persist beyond the decay of the optimizer's exponential moving average due to their separation from evolving background activity.

  2. The paper distinguishes between single-checkpoint feature provenance inference and multi-checkpoint gradient recovery, providing insights into different privacy leakage mechanisms associated with optimizer states.

Critical weaknesses
  1. The reconstruction attack relies on consecutive checkpoints, which may be uncommon in real-world training scenarios.

  2. The single-checkpoint attack mainly focuses on feature detection, while its effectiveness on complete training samples remains unclear.

  3. The privacy-memory experiments are limited to specific features and training settings, leaving its generality insufficiently explored.

Questions for authors
  1. How effective is the reconstruction attack under larger checkpoint intervals commonly used in practical training?

  2. Can the single-checkpoint attack effectively identify complete training samples or documents instead of individual features?

  3. How does the persistence of privacy memory vary across different datasets, feature frequencies, and optimizer settings?

4 comments
AuthorsOpenReview ↗

Response to Reviewer ZovJ (Part I)

We thank the reviewer for the constructive questions. We provide further evidence and clarification addressing the concerns below.

1. Practicality under larger checkpoint intervals

Adjacent signed first moments permit exact single-step gradient recovery. The original experiment shows rapid degradation with larger gaps: spacing 1 gives 72.5 dB MNIST PSNR versus 9.7 dB for a weight-difference proxy, with reconstruction near the noise floor around spacing 10. Non-adjacent first moments instead expose an exponentially weighted gradient window (Eq. 10), enabling interval-level provenance even when instance reconstruction is weak.

For distilGPT-2 (d=768), we use AdamW (learning rate 5×10⁻⁵, weight decay 0.01, 100 warm-up steps), 64 post-warm-up steps, and three seeds. A DAGER-style span check is applied to the layer-0 attention input projection. Entries are mean top-|T| recall over the vocabulary, where |T| is the number of distinct tokens in the Δ-step window; row parentheses give tokens per step and cell annotations give saturated rank.

Batch B, length L (tokens/step)Δ = 1248163264
1, 16 (16)0.9560.9440.9460.9440.9490.9040.242 (rank 764)
1, 32 (32)0.9650.9740.9790.9700.9660.335 (rank 768)0.264
4, 32 (128)0.9770.9780.9650.263 (rank 768)0.2150.2150.210

Recall drops when effective rank approaches the layer width.

A separate Δ=1 fp32 packed-WikiText-2 experiment (two seeds) varies tokens per step; parentheses give mean distinct-token counts and cells are top-|T| recall from the exact adjacent gradient using layer-0 q_proj.

Model (d)2,048 (874)4,096 (1,445)8,192 (2,458)16,384 (4,170)
Qwen3-0.6B (1024)0.9910.0050.0000.008
Qwen3-1.7B (2048)0.9900.9960.6800.320
Qwen3-4B (2560)0.9910.9960.9960.011
Qwen3-8B (4096)0.9900.9960.9980.990

Values are means over two seeds; recovery degrades when distinct-token count exceeds model width. The Qwen3-8B result at 16,384 tokens is one tested configuration, not a universal guarantee.

For interval inference, dv=v_b−β2^Δv_a (β2=0.999), v_b is the single-checkpoint baseline, and dθ the weight-difference baseline. Values range over a∈{2,000,5,000,10,000}; hard negatives precede but are absent from the interval. TPR@1%, hard/all calibrates on hard/all non-interval negatives.

Gapdv: AUC, harddv: TPR@1%, harddv: TPR@1%, allv_b alone: AUC / TPR@1%, harddθ: AUC / TPR@1%, hard
10.997–1.0000.94–1.001.0000.94–0.98 / 0.51–0.600.95–0.97 / 0.04–0.17
100.994–0.9960.85–0.860.997–1.0000.85–0.90 / 0.28–0.360.94–0.97 / 0.11–0.30
500.982–0.9930.70–0.820.985–0.9960.82–0.89 / 0.25–0.290.94–0.96 / 0.23–0.29
1000.979–0.9900.65–0.790.981–0.9880.80–0.89 / 0.24–0.280.94–0.96 / 0.26–0.29
5000.969–0.9830.51–0.720.967–0.9720.76–0.89 / 0.20–0.270.89–0.92 / 0.08–0.19
1,0000.963–0.9810.50–0.650.961–0.9670.78–0.90 / 0.21–0.290.80–0.87 / 0.01–0.05
2,0000.958–0.9790.53–0.640.961–0.9660.82–0.92 / 0.30–0.380.66–0.78 / 0.01–0.06
5,0000.945–0.9680.46–0.530.953–0.9560.89–0.95 / 0.43–0.510.45–0.60 / 0.01–0.04

At smaller β2 values:

Gapβ2 = 0.99β2 = 0.95
1000.980–0.992 / 0.65–0.820.975–0.991 / 0.62–0.78
1,0000.922–0.964 / 0.38–0.500.788–0.838 / 0.25–0.31
5,0000.811–0.837 / 0.31–0.360.729–0.750 / 0.18–0.21

We scanned two ranked samples of Hugging Face Hub repositories for optimizer-state files and multiple checkpoints from the same training run.

SampleReposWith optimizer state≥ 2 checkpoints of one runMin gap ≤ 100 steps≤ 10Adjacent (gap 1)
Most recently updated4,995107392862
Most downloaded5,0003471 (smallest gap is 100)00

These are counts from ranked samples rather than hub-wide prevalence estimates.

Thus, larger gaps reduce complete-example reconstruction but can still expose interval-level token sets or feature provenance.

AuthorsOpenReview ↗

Response to Reviewer ZovJ (Part II)

2. Identification of complete training samples or documents

The single-checkpoint attack targets feature provenance rather than complete-example recovery or order-preserving extraction. Its score is order-invariant and therefore does not generally recover token order; the document experiment tests whether optimizer-state information adds to model-only attacks.

This experiment uses distilGPT-2 and 600 four-sentence WikiText documents, with half of the documents included in training, over three seeds. The v_t score is the mean optimizer-state row norm over the document's distinctive tokens (tokens appearing in at most two documents). The model-only attacks use loss and Min-20% Prob; the shadow-fit rows use logistic regression trained on a disjoint shadow run. The results are means and standard deviations over three seeds:

EpochsAttackAUROCTPR@1%TPR@5%
1v_t0.951 ± 0.0130.16 ± 0.080.68
1loss0.905 ± 0.0200.16 ± 0.190.53
1Min-20% Prob0.916 ± 0.0050.27 ± 0.080.59
1model-only: loss + Min-20%, shadow-fit logistic regression0.921 ± 0.0110.20 ± 0.140.59
1v_t + loss + Min-20%, shadow-fit logistic regression0.979 ± 0.0080.45 ± 0.280.93
1v_t + loss, rank average (no fitting)0.979 ± 0.0060.53 ± 0.220.90
3model-only, shadow-fit logistic regression0.995 ± 0.0050.77 ± 0.321.00
3v_t + loss + Min-20%, shadow-fit logistic regression0.996 ± 0.0041.00 ± 0.001.00

TPR@1% rests on only 3 of 300 negative examples, so it varies widely between seeds; TPR@5% is the more stable metric.

The combined attack uses a disjoint shadow run only for auxiliary calibration; the core optimizer-state provenance attack does not require a shadow model. Optimizer state provides a substantial complementary signal after one epoch, but the additional benefit becomes small after three epochs.

We also tested multi-token secrets. This experiment uses distilGPT-2 trained for three epochs with the template Contact record: {}., 48 candidates per secret type, half of which are trained, and three seeds. The v_t score averages row norms over each candidate's rare tokens (reference-corpus frequency at or below the 25th percentile), the loss score is the mean token loss on the secret span, and the combination is a rank average. In the first v_t row, the candidate is scored using its raw-string tokenization; the second v_t row uses the tokens actually present in training. The results are mean AUROC by secret type:

ScoreEmailPhoneAPI keyUUIDSurnameSingle token
v_t, raw-string tokenization (string encoded on its own)0.680.790.640.580.601.00
v_t, the tokens actually trained0.700.800.630.580.731.00
loss on the secret span (model only)0.781.000.991.000.741.00
v_t + loss, rank average0.760.860.760.750.761.00

For high-entropy multi-token secrets, model loss can already be very effective, and optimizer state does not consistently provide an additional advantage. These results support a feature-level and complementary interpretation rather than a universal instance-extraction claim.

AuthorsOpenReview ↗

Response to Reviewer ZovJ (Part III)

3. Generality of privacy memory

We evaluated privacy memory across token frequencies, token categories, optimizer coefficients, and optimizer-state precision. The persistence experiments therefore span two training corpora: CodeAlpaca, used for the Qwen3-0.6B natural run, and WikiText, used for the controlled distilGPT-2 marker experiments. The final CodeAlpaca row in the horizon table provides an additional corpus replication.

The token-category analysis is offline on saved trajectories. The primary fp32 trajectory is the Qwen3-0.6B natural run with CodeAlpaca and β2 = 0.999; the bf16 values come from the paper's original run. The table compares trained and never-seen tokens within each category at the final checkpoint (step 19,999), using embedding-row v_t norms. Categories are whitespace, punctuation/symbol, stopword, identifier/numeric, alphabetic content, and other.

CategoryTrained nMedian countAUC, fp32TPR@1%, fp32AUC, bf16TPR@1%, bf16
whitespace12290.9920.890.9990.91
punct/symbol2,30340.9940.840.9950.91
stopword797100.9910.900.9930.96
identifier/numeric1,94830.9820.790.9890.84
alphabetic (content)14,83030.9840.810.9910.89
other3,27030.9960.790.9980.80
all23,270--0.9910.81----

For the following table, “initially detectable” means above the 99th percentile of never-seen-token scores at the first snapshot at or after the token's last occurrence. τ denotes the number of steps remaining after that last occurrence, and later entries are the fractions still above the corresponding threshold.

OccurrencesnInitially detectableτ = 1k2k5k10k15k19k
16,7780.9930.990.980.920.34 (2,942)0.25 (1,382)0.19 (207)
2–35,5380.9961.000.990.940.48 (1,447)0.27 (488)0.18 (74)
4–105,0960.9961.001.000.970.65 (383)0.27 (77)0.10 (10)
11–1004,5780.9991.001.000.960.54 (13)----

Single-occurrence tokens remain detectable with retention 0.92 after 5,000 steps and 0.34 after 10,000 steps in the fp32 β2 = 0.999 setting. Here τ is measured after the token's only occurrence.

A controlled replication inserts 384 otherwise-unused marker tokens exactly three times and never again. Across three seeds, the markers remain detectable after 10,000 steps with AUC 0.991 ± 0.000 and TPR@1% FPR 0.675 ± 0.021.

In this controlled experiment, 384 otherwise-unused marker token ids are injected k times during steps 0–199 and never used again. The experiments use distilGPT-2 on WikiText, learning rate 5 × 10⁻⁵, 10,000 training steps, and snapshots every 500 steps. Each marker is scored by its optimizer-state row norm against unused token ids. Adafactor is scored using exp_avg_sq_row and Lion using exp_avg. The horizon is the last snapshot at which TPR@1% is at least 0.5, after subtracting the 200-step injection window. Markers are synthetic and one model is used.

ConfigurationSeedsFirst post-window TPR@1%Final AUCFinal TPR@1%Horizon (steps)
AdamW β2 = 0.999, k = 130.980.9690.297,300
AdamW β2 = 0.999, k = 331.000.9910.66≥ 9,800 (end of run)
AdamW β2 = 0.999, k = 1031.000.9970.98≥ 9,800 (end of run)
AdamW β2 = 0.99, k = 320.9970.560.01800
AdamW β2 = 0.95, k = 320.070.5090.005< 300
SGD with momentum 0.9, k = 320.060.6300.031< 300
Adafactor, k = 320.9970.9510.0493,050
Lion, k = 320.6050.5380.014300–800
AdamW β2 = 0.999, k = 3, CodeAlpaca31.000.9960.997≥ 9,800 (end of run)

Under the same controlled AdamW setting (β2 = 0.999, k = 3), the WikiText run and the CodeAlpaca replication both retain detectable markers through the end of the 9,800-step post-injection evaluation window, although their final AUC/TPR@1% differ (0.991/0.66 versus 0.996/0.997).

State precision also matters. The table varies model and optimizer-state storage precision; entries are early-only AUC at approximately 1k, 5k, 10k, and 20k steps.

StorageQwen3-0.6BdistilGPT-2
bf16 weights + bf16 state (the paper's runs)0.999 / 0.997 / 0.998 / 0.99550.999 / 0.998 / 0.996 / 0.9935
fp32 weights + bf16 state0.999 / 0.996 / 0.998 / 0.9971.000 / 0.999 / 0.999 / 0.998
fp32 weights + fp32 state0.998 / 0.991 / 0.980 / 0.9171.000 / 0.994 / 0.974 / 0.858

At 20k, Qwen3-0.6B early-only AUC is 0.9955 with bf16 state versus 0.917 with fp32 state, showing representation- and optimizer-dependent persistence.

Reviewer_ZovJOpenReview ↗

Response

Thank you for your response. My concerns have been adequately addressed, and I will maintain my positive score.

Reviewer_unPMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors demonstrate that not only intermediate checkpoints, but also intermediate optimizer states can leak additional privacy about training data points. The authors empirically evaluated the proposed attack method on distilGPT-2 models.

Critical strengths

The paper analyzes how intermediate optimize states can potentially leak more information about training data points, where previous works mostly investigate the leakage from the final model weights, gradients, and intermediate model weights.

Critical weaknesses

This work is very similar to the line of work on privacy auditing when the adversary knows the intermediate checkpoints [1][2]. The authors did not compare or position the proposed attack against this line of work, so it is unclear what the contribution is on top of that. [1] Milad Nasr, Shuang Songi, Abhradeep Thakurta, Nicolas Papernot, and Nicholas Carlin. Adversary instantiation: Lower bounds for differentially private machine learning. In 2021 IEEE Symposium on security and privacy (SP), pages 866–882. IEEE, 2021. [2] Milad Nasr, Jamie Hayes, Thomas Steinke, Borja Balle, Florian Tramèr, Matthew Jagielski, Nicholas Carlini, and Andreas Terzis. Tight auditing of differentially private machine learning. In USENIX Security, 2023.

Questions for authors

Similar to the weaknesses above, how is this work different from the line of work on privacy auditing when the adversary knows the intermediate checkpoints?

4 comments
AuthorsOpenReview ↗

Response to Reviewer unPM (Part I)

We thank the reviewer for highlighting the connection to privacy auditing with access to intermediate training states. The two lines of work differ in their observation surfaces, attacker access, and objectives; we clarify this distinction below and will make it explicit in the related work discussion.

Difference in threat model and objective

The two lines of work overlap in recognizing that intermediate training information can increase privacy leakage, but they study different observation surfaces and objectives.

AspectPrivacy auditing with intermediate updatesOur work
Attacker accessMay observe intermediate model updates during training; stronger settings may allow active interactionReads serialized checkpoint files after training
Training interactionMay require update access or adversarially selected training dataNo live optimizer, training access, or original examples for the core attacks
Main objectiveEstimate privacy loss or lower bounds on the DP parameterInfer feature provenance and recover optimizer-input gradients
Main artifactModel outputs or model-update trajectorySerialized optimizer moments and checkpoint state
Main settingPrimarily DP-SGD auditingStandard non-DP training and released resumable checkpoints

Our single-checkpoint attack uses one released optimizer state to infer whether specific features influenced training. Our multi-checkpoint attack uses released signed first moments to recover the optimizer-input gradient after training has finished. We do not claim to introduce a new DP auditing method or a new gradient-inversion optimizer.

The core single- and multi-checkpoint attacks do not require a shadow model; shadow calibration is used only in the auxiliary document-combination experiment.

Matched one-run privacy audit

To quantify the additional information exposed by optimizer state, we conducted a matched one-run audit across different released artifacts. This follows a one-run canary protocol: distilGPT-2 is trained on 4,000 natural records and 1,000 canary records, where each canary is a natural sentence containing one unique, otherwise-unused token and is included independently with probability 1/2. Training uses Poisson sampling with q ≈ 0.014 (expected batch size 64), 600 steps, and Adam at learning rate 10⁻⁴. The DP runs use per-example clipping C = 1 and Gaussian noise σC; ε ∈ {1, 4, 8} is obtained with an RDP accountant at δ = 10⁻⁵, cross-checked against TF-Privacy reference values.

For each release, the pre-registered guessing rule labels the top 100 scores as IN and the bottom 100 as OUT, giving 200 guesses. A perfect 200/200 result therefore certifies at most ε = 4.19 under the 95%-confidence binomial-dominance lower-bound procedure used here. Score orientation is fixed using the other seed, and each setting uses two seeds. The non-DP results are AUC values from two seeds. The DP column summarizes the AUC range across ε ∈ {1, 4, 8} and the corresponding empirical ε lower bound.

Release seen by the auditorNon-DP AUC (2 seeds)Non-DP correct / 200 → ε lower boundDP at ε = 1 / 4 / 8: AUC, ε lower bound
final model: canary loss0.987 / 0.990200 → 4.19 (the ceiling)0.47--0.49, 0.00
final weights: current row norm0.515 / 0.501116, 105 → 0.08, 0.000.51--0.52, ≤ 0.16
final weights + base model: displacement (orientation flipped)0.913 / 0.897194, 185 → 2.78, 2.060.50, 0.00
resumable checkpoint: v_t row norm0.9998 / 0.9998200 → 4.19 (the ceiling)0.52--0.53, ≤ 0.04

The four rows correspond respectively to canary loss from the final model, the current embedding-row norm, the displacement from the base model, and the v_t row norm in a resumable checkpoint. The latter is the only release that contains optimizer state. This audit is intended to compare release surfaces, rather than to establish a tight empirical DP audit or a formal privacy guarantee. Model utility was not evaluated.

AuthorsOpenReview ↗

Response to Reviewer unPM (Part II)

Release-surface comparison

The same distinction is visible in the full single-checkpoint release-surface comparison. This reruns the single-checkpoint protocol underlying Table 6 on Qwen3-0.6B with fp32 weights and optimizer state for 20,000 steps, using a disjoint calibration run at the same horizon. The weight-based learned rows use the same gradient-boosted classifier class as Table 6, with features from the current weights and, where indicated, the base model, together with the indicated m_t/v_t row-norm features. The direct v_t and m_t rows use the corresponding optimizer-state row norms without a classifier. Entries are TPR at a nominal 1% FPR operating point.

ReleasedStep 3001,0003,00020,000
current weights only (no base model)0.2250.1610.1160.135
weights + base model0.5560.5890.6060.894
weights + base model + m_t0.8720.7940.7390.914
weights + base model + v_t0.9820.9670.9650.983
weights + base model + v_t + m_t0.9800.9700.9640.983
v_t alone (no classifier, no base model)0.9610.9370.9410.928
m_t alone0.6870.5750.5790.566

Thresholds for all rows were calibrated at a nominal 1% false-positive rate on the disjoint calibration run and then frozen. The realized test FPR can differ from the nominal calibration value; in the matched run it is about 1--2% for individual detectors.

These results show that optimizer state provides an additional observation channel beyond model weights, especially early in training. At later horizons, the model weights accumulate a stronger and more similar signal.

Position relative to privacy auditing

We view our work as complementary to privacy auditing. Privacy-auditing work asks how much privacy loss can be empirically certified for a specified training mechanism under a particular adversary model. Our work asks what information is contained in a resumable checkpoint that may be copied, backed up, or published after training.

The distinction is therefore not that intermediate updates are harmless or that optimizer state is unrelated to training trajectories. Rather, our contribution is to identify optimizer state as a persistent, data-dependent release artifact, characterize how its representation determines the information that survives, and show that adjacent serialized states can transfer existing inversion attacks to a passive post-training reader.

We will clarify this distinction in the related work discussion and position our work as complementary to privacy auditing.

I thank the authors for their detailed response and additional experiments. The technical execution and the breadth of evaluation look solid to me, and I appreciate that. However, my main concern is still about the conceptual positioning and the scope of this work. Here are some thoughts that may help the authors when refining their manuscript:

  1. Positioning: From a threat-model perspective, the core contribution still feels a little bit incremental relative to existing literature on privacy auditing with intermediate model checkpoints. Since both the model weights and optimizer states are fundamentally downstream products of the gradients, analyzing the optimizer states can be viewed as an extension of existing work on “reverting gradients to data points.”

  2. Context of recovery from adjacent checkpoints: I think similar ideas have not only been explored in privacy auditing with intermediate checkpoints, but also in Federated Learning literature where the server has access to adjacent updates from clients. This is commented in the related work section, but overall it feels very similar to prior work. The primary difference in this work is leveraging optimizer states in addition to model weights. The mathematical formulation for the cancellation itself is neat, but the overall effect is data-point recovery from adjacent model updates.

  3. Scope: The authors evaluate across different optimizers (e.g., Adam with momentum and variance buffers), which is extensive. However, there are many optimizers that can explicitly store or cache historical model checkpoints in their optimizer states (e.g., Lookahead [1]). If the main contribution of the work is to find out how much an optimizer state can leak about training data points, discussing or including optimizers that store multi-step training trajectories would provide a much more complete picture.

Despite these concerns regarding positioning and scope, demonstrating how standard optimizers like Adam expose training data through their internal buffers is a practically valuable contribution. I encourage the authors to re-visit how they frame their contributions. For example, positioning the paper explicitly as a systematic study of how optimizer states retain and leak training signals, which may better align with the actual strengths of the work.

[1] Michael R. Zhang, James Lucas, Geoffrey Hinton, & Jimmy Ba. (2019). Lookahead Optimizer: k steps forward, 1 step back.

AuthorsOpenReview ↗

Follow-up response to Reviewer unPM

Thank you for recognizing the technical quality and practical value of the work. We clarify the three remaining points below.

1. Positioning relative to privacy auditing

A shared origin in gradients does not imply that model weights and optimizer states are informationally equivalent. The cited privacy-auditing papers estimate privacy loss under specified adversary capabilities. Our contribution is to characterize feature provenance, privacy memory, and gradient exposure in serialized optimizer representations. The provenance attack uses one saved optimizer state without adjacent-update differencing; the privacy-memory experiments track how long this signal remains detectable during subsequent training. The matched release-surface comparison above demonstrates the complementary information provided by optimizer state.

2. Relationship to federated update inversion

We do not claim a new reconstruction algorithm or novelty in the cancellation identity alone. The adjacent-checkpoint result establishes how saved optimizer moments provide the observations required by existing inversion attacks. Representation comparisons, native-framework validation, and sanitization experiments characterize the conditions and limitations of this channel. Adjacent-checkpoint inversion is therefore one component of the study, alongside single-checkpoint provenance and privacy memory.

3. Scope and Lookahead

Lookahead's additional slow-weight state is a relevant extension beyond the representations evaluated here. Our findings show that feature provenance and privacy memory arise in gradient-history buffers even without explicitly cached historical model snapshots. The existence of additional weight states does not by itself establish what ordinary moment buffers reveal or how long their traces remain detectable.

Our claims concern the tested state representations. We will discuss Lookahead as an extension and clarify the study's scope without claiming exhaustive coverage of every optimizer.

Reviewer_HT9ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies training-data leakage from optimizer states stored in released checkpoints. It considers two threat settings. With a single checkpoint, the optimizer state can reveal feature-level provenance, such as whether particular tokens, classes, or expert routes were involved in training. With multiple checkpoints, especially adjacent ones, optimizer state differences can be used to recover the training gradient and enable existing gradient-inversion attacks. The paper evaluates these attacks across different models, tasks, and optimizers settings, and further studies the persistence of leakage over longer training as well as potential checkpoint sanitization defenses.

Critical strengths
  1. The paper identifies a new and usually overlooked privacy attack surface that recover information from released optimizer state checkpoints.
  2. The evaluation is extensive, covering single- and multiple-checkpoint settings across various models, tasks, and optimizers.
  3. The paper discusses practical defenses sanitization strategies to mitigate the issues.
Critical weaknesses
  1. The practicality of the multiple-checkpoint attack is less convincing. As shown in Figure 3 and Line 414, strong reconstruction requires checkpoints separated by a single optimizer step, while performance degrades close to the noise floor once the spacing exceeds 10 steps. This assumes a strong attacker capability, and it is unclear whether such an attack setting has practical significance.

  2. The technical contribution of the multiple-checkpoint attack is somewhat limited. Given two adjacent signed optimizer states, the training gradient can be directly recovered from the optimizer update rule, after which the reconstruction relies on well-established gradient-inversion attacks.

  3. The privacy significance of the single-checkpoint attack is unclear. For example, in the Qwen3-0.6B feature provenance experiment in Line 320, the recovered features are tokens aggregated across many training samples, rather than meaningful phrases or sentences with preserved order (also reflected in Appendix B.4). It is also unclear whether the strongest signals correspond to privacy-sensitive features rather than frequent common tokens such as punctuation and stopwords.

Questions for authors
  1. How general is the reported privacy memory phenomenon? In Line 370, 22 tokens occurring only in the first 100 sentences remain separable after 20,000 steps. Are these all early-only tokens, or only a subset that remains detectable? What fraction of initially detectable features retain this long-term separability?
  2. Which types of tokens remain most separable after training? Are frequent common tokens such as punctuation and stopwords more likely to remain detectable because of their high frequency, or can rare tokens also remain separable? This distinction is relevant to understanding why such token-level detectability should be considered privacy-sensitive in the sense implied by the term “privacy memory.”
  3. Table 6 shows that after longer training, the performance gap between optimizer-state and weight-only token recovery becomes quite small. How much overlap is there between the tokens recovered by the two methods? In particular, does optimizer-state access reveal many additional tokens that would not already be identified from the model weights alone?
7 comments
AuthorsOpenReview ↗

Response to Reviewer HT9R (Part I)

We thank the reviewer for the careful and constructive comments. We provide further evidence and clarification addressing the concerns below.

1. Practical significance of the multi-checkpoint attack

Exact single-step recovery requires two adjacent checkpoints containing a signed first moment; arbitrary checkpoint schedules are not assumed. Complete-example MNIST reconstruction reaches 72.5 dB PSNR at spacing 1 (vs. 9.7 dB for the weight-difference proxy) and nears the noise floor by spacing 10. Non-adjacent first moments expose an exponentially weighted gradient window (Eq. 10), so larger gaps can still reveal interval-level token provenance even when instance reconstruction is weak.

For the distilGPT-2 experiment, Δ = b − a, d = 768, and AdamW uses learning rate 5 × 10⁻⁵, weight decay 0.01, and 100 warm-up steps. We record 64 post-warm-up steps over three seeds and apply a DAGER-style span check to the layer-0 attention input projection. Each entry is mean top-|T| recall over the full vocabulary, where |T| is the number of distinct tokens in the Δ-step window. Parenthetical row values are total tokens per step; rank annotations denote numerical input rank at saturation.

Batch B, length L (tokens/step)Δ = 1248163264
1, 16 (16)0.9560.9440.9460.9440.9490.9040.242 (rank 764)
1, 32 (32)0.9650.9740.9790.9700.9660.335 (rank 768)0.264
4, 32 (128)0.9770.9780.9650.263 (rank 768)0.2150.2150.210

Recall drops when the effective input rank approaches or exceeds the layer width.

Separately, at Δ = 1, we vary the total tokens per optimizer step. Columns show total tokens, with mean distinct-token counts in parentheses. Each cell is top-|T| recall from the exact adjacent-checkpoint gradient using the layer-0 q_proj span check. We use fp32 packed WikiText-2 sequences and two seeds.

Model (d)2,048 (874)4,096 (1,445)8,192 (2,458)16,384 (4,170)
Qwen3-0.6B (1024)0.9910.0050.0000.008
Qwen3-1.7B (2048)0.9900.9960.6800.320
Qwen3-4B (2560)0.9910.9960.9960.011
Qwen3-8B (4096)0.9900.9960.9980.990

Values are means over two seeds. Recovery is strongest when the number of distinct tokens is below or near the model width and degrades when the effective rank exceeds it. The Qwen3-8B result at 16,384 tokens per step is one tested configuration rather than a universal guarantee.

For interval inference, dv = v_b − β2^Δ v_a with β2 = 0.999; v_b is the single-checkpoint baseline and dθ the weight-difference baseline. Values are ranges over anchors a ∈ {2,000, 5,000, 10,000}. Hard negatives were trained before but absent from the interval. TPR@1%, hard calibrates on hard negatives, while TPR@1%, all calibrates on all non-interval tokens.

Gapdv: AUC, harddv: TPR@1%, harddv: TPR@1%, allv_b alone: AUC / TPR@1%, harddθ: AUC / TPR@1%, hard
10.997–1.0000.94–1.001.0000.94–0.98 / 0.51–0.600.95–0.97 / 0.04–0.17
100.994–0.9960.85–0.860.997–1.0000.85–0.90 / 0.28–0.360.94–0.97 / 0.11–0.30
500.982–0.9930.70–0.820.985–0.9960.82–0.89 / 0.25–0.290.94–0.96 / 0.23–0.29
1000.979–0.9900.65–0.790.981–0.9880.80–0.89 / 0.24–0.280.94–0.96 / 0.26–0.29
5000.969–0.9830.51–0.720.967–0.9720.76–0.89 / 0.20–0.270.89–0.92 / 0.08–0.19
1,0000.963–0.9810.50–0.650.961–0.9670.78–0.90 / 0.21–0.290.80–0.87 / 0.01–0.05
2,0000.958–0.9790.53–0.640.961–0.9660.82–0.92 / 0.30–0.380.66–0.78 / 0.01–0.06
5,0000.945–0.9680.46–0.530.953–0.9560.89–0.95 / 0.43–0.510.45–0.60 / 0.01–0.04

At smaller β2 values:

Gapβ2 = 0.99β2 = 0.95
1000.980–0.992 / 0.65–0.820.975–0.991 / 0.62–0.78
1,0000.922–0.964 / 0.38–0.500.788–0.838 / 0.25–0.31
5,0000.811–0.837 / 0.31–0.360.729–0.750 / 0.18–0.21

We scanned two ranked Hugging Face Hub samples for optimizer-state files and multiple checkpoints from one training run; gaps were parsed from checkpoint directory names.

SampleReposWith optimizer state≥ 2 checkpoints of one runMin gap ≤ 100 steps≤ 10Adjacent (gap 1)
Most recently updated4,995107392862
Most downloaded5,0003471 (smallest gap is 100)00

These are counts from ranked samples rather than estimates of hub-wide prevalence.

Thus, larger spacing reduces complete-example reconstruction quality, but can still expose interval-level token sets or feature provenance. We will state this distinction explicitly and avoid presenting wide-gap token-set inference as complete-example reconstruction.

AuthorsOpenReview ↗

Response to Reviewer HT9R (Part II)

2. Technical contribution of the multi-checkpoint attack

The algebraic identity for adjacent signed first moments is simple. Our contribution is to identify serialized optimizer state as a post-training privacy interface that can recover the optimizer-input gradient required by existing attacks.

The paper characterizes which state representations preserve signed gradients, magnitudes, factored statistics, or quantized approximations. We also validate the reduction on HuggingFace Trainer, Accelerate, and raw PyTorch checkpoints, and show that deleting one state tensor may leave related leakage recoverable through other states or the weight trajectory.

Therefore, the contribution is that existing gradient-inversion attacks can be transferred to a passive reader of released checkpoint files, without live optimizer access, training interaction, or access to the original examples.

This observation also applies beyond one optimizer name. The relevant property is the representation stored in the checkpoint: adjacent signed linear states support gradient recovery in exact arithmetic under the known update rule, whereas magnitude, factored, and quantized states expose different, generally weaker observations in our experiments.

AuthorsOpenReview ↗

Response to Reviewer HT9R (Part III)

3. Privacy significance of the single-checkpoint attack

The single-checkpoint attack is intentionally a feature-provenance attack, rather than a general phrase- or sentence-extraction method. Its score is order-invariant and does not generally recover token order or complete training examples.

The additional token-category analysis gives the following results. The scores are embedding-row norms of v_t at the final checkpoint (step 19,999), comparing trained tokens with never-seen tokens within each category. The fp32 columns use the Qwen3-0.6B natural run with β2=0.999\beta_2 = 0.999; the bf16 columns use the paper's original run.

CategoryTrained nMedian countAUC, fp32TPR@1%, fp32AUC, bf16TPR@1%, bf16
whitespace12290.9920.890.9990.91
punct/symbol2,30340.9940.840.9950.91
stopword797100.9910.900.9930.96
identifier/numeric1,94830.9820.790.9890.84
alphabetic (content)14,83030.9840.810.9910.89
other3,27030.9960.790.9980.80
all23,270--0.9910.81----

Thus, the signal is not limited to punctuation or stopwords. The document experiment further tests whether optimizer-state information adds to model-only document attacks. It uses distilGPT-2 and 600 four-sentence WikiText documents, half of which are included in training, over three seeds. The v_t score is the mean optimizer-state row norm over each document's distinctive tokens (tokens appearing in at most two documents). The model-only attacks use loss and Min-20% Prob; the shadow-fit rows use logistic regression trained on a disjoint shadow run. The results are means and standard deviations over three seeds.

EpochsAttackAUROCTPR@1%TPR@5%
1v_t0.951 ± 0.0130.16 ± 0.080.68
1loss0.905 ± 0.0200.16 ± 0.190.53
1Min-20% Prob0.916 ± 0.0050.27 ± 0.080.59
1model-only: loss + Min-20%, shadow-fit logistic regression0.921 ± 0.0110.20 ± 0.140.59
1v_t + loss + Min-20%, shadow-fit logistic regression0.979 ± 0.0080.45 ± 0.280.93
1v_t + loss, rank average (no fitting)0.979 ± 0.0060.53 ± 0.220.90
3model-only, shadow-fit logistic regression0.995 ± 0.0050.77 ± 0.321.00
3v_t + loss + Min-20%, shadow-fit logistic regression0.996 ± 0.0041.00 ± 0.001.00

TPR@1% rests on only 3 of 300 negative examples, so it varies widely between seeds; TPR@5% is the more stable metric.

The combined attack uses a disjoint shadow run only for auxiliary calibration; the core optimizer-state provenance attack does not require a shadow model. These results show that optimizer state can provide complementary document-level evidence in some settings, but is not universally stronger than model-only attacks.

For multi-token secrets, optimizer state is also not universally stronger than model-only extraction. This experiment uses distilGPT-2 trained for three epochs with the template Contact record: {}., 48 candidates per secret type, half of which are trained, and three seeds. The v_t score averages row norms over each candidate's rare tokens (reference-corpus frequency at or below the 25th percentile), the loss score is the mean token loss on the secret span, and the combination is a rank average. In the first v_t row, each candidate is tokenized independently; the second v_t row uses the tokens actually present in training. Results are mean AUROC by secret type:

ScoreEmailPhoneAPI keyUUIDSurnameSingle token
v_t, independently tokenized candidate0.680.790.640.580.601.00
v_t, the tokens actually trained0.700.800.630.580.731.00
loss on the secret span (model only)0.781.000.991.000.741.00
v_t + loss, rank average0.760.860.760.750.761.00

For high-entropy multi-token secrets, model loss can already be very effective, and the optimizer-state score does not consistently provide an additional benefit. We will therefore avoid presenting feature provenance as a universal instance-extraction method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.