- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates privacy leakage from optimizer states stored in resumable training checkpoints, demonstrating that a single checkpoint can reveal the provenance of training features. The paper validates these leakage mechanisms across different models, optimizers, and standard training frameworks.
Critical strengths
-
The paper introduces and analyzes the concept of privacy memory, demonstrating that detectable traces of training features can persist beyond the decay of the optimizer's exponential moving average due to their separation from evolving background activity.
-
The paper distinguishes between single-checkpoint feature provenance inference and multi-checkpoint gradient recovery, providing insights into different privacy leakage mechanisms associated with optimizer states.
Critical weaknesses
-
The reconstruction attack relies on consecutive checkpoints, which may be uncommon in real-world training scenarios.
-
The single-checkpoint attack mainly focuses on feature detection, while its effectiveness on complete training samples remains unclear.
-
The privacy-memory experiments are limited to specific features and training settings, leaving its generality insufficiently explored.
Questions for authors
-
How effective is the reconstruction attack under larger checkpoint intervals commonly used in practical training?
-
Can the single-checkpoint attack effectively identify complete training samples or documents instead of individual features?
-
How does the persistence of privacy memory vary across different datasets, feature frequencies, and optimizer settings?
4 comments
Response to Reviewer ZovJ (Part II)
2. Identification of complete training samples or documents
The single-checkpoint attack targets feature provenance rather than complete-example recovery or order-preserving extraction. Its score is order-invariant and therefore does not generally recover token order; the document experiment tests whether optimizer-state information adds to model-only attacks.
This experiment uses distilGPT-2 and 600 four-sentence WikiText documents, with half of the documents included in training, over three seeds. The v_t score is the mean optimizer-state row norm over the document's distinctive tokens (tokens appearing in at most two documents). The model-only attacks use loss and Min-20% Prob; the shadow-fit rows use logistic regression trained on a disjoint shadow run. The results are means and standard deviations over three seeds:
| Epochs | Attack | AUROC | TPR@1% | TPR@5% |
|---|---|---|---|---|
| 1 | v_t | 0.951 ± 0.013 | 0.16 ± 0.08 | 0.68 |
| 1 | loss | 0.905 ± 0.020 | 0.16 ± 0.19 | 0.53 |
| 1 | Min-20% Prob | 0.916 ± 0.005 | 0.27 ± 0.08 | 0.59 |
| 1 | model-only: loss + Min-20%, shadow-fit logistic regression | 0.921 ± 0.011 | 0.20 ± 0.14 | 0.59 |
| 1 | v_t + loss + Min-20%, shadow-fit logistic regression | 0.979 ± 0.008 | 0.45 ± 0.28 | 0.93 |
| 1 | v_t + loss, rank average (no fitting) | 0.979 ± 0.006 | 0.53 ± 0.22 | 0.90 |
| 3 | model-only, shadow-fit logistic regression | 0.995 ± 0.005 | 0.77 ± 0.32 | 1.00 |
| 3 | v_t + loss + Min-20%, shadow-fit logistic regression | 0.996 ± 0.004 | 1.00 ± 0.00 | 1.00 |
TPR@1% rests on only 3 of 300 negative examples, so it varies widely between seeds; TPR@5% is the more stable metric.
The combined attack uses a disjoint shadow run only for auxiliary calibration; the core optimizer-state provenance attack does not require a shadow model. Optimizer state provides a substantial complementary signal after one epoch, but the additional benefit becomes small after three epochs.
We also tested multi-token secrets. This experiment uses distilGPT-2 trained for three epochs with the template Contact record: {}., 48 candidates per secret type, half of which are trained, and three seeds. The v_t score averages row norms over each candidate's rare tokens (reference-corpus frequency at or below the 25th percentile), the loss score is the mean token loss on the secret span, and the combination is a rank average. In the first v_t row, the candidate is scored using its raw-string tokenization; the second v_t row uses the tokens actually present in training. The results are mean AUROC by secret type:
| Score | Phone | API key | UUID | Surname | Single token | |
|---|---|---|---|---|---|---|
| v_t, raw-string tokenization (string encoded on its own) | 0.68 | 0.79 | 0.64 | 0.58 | 0.60 | 1.00 |
| v_t, the tokens actually trained | 0.70 | 0.80 | 0.63 | 0.58 | 0.73 | 1.00 |
| loss on the secret span (model only) | 0.78 | 1.00 | 0.99 | 1.00 | 0.74 | 1.00 |
| v_t + loss, rank average | 0.76 | 0.86 | 0.76 | 0.75 | 0.76 | 1.00 |
For high-entropy multi-token secrets, model loss can already be very effective, and optimizer state does not consistently provide an additional advantage. These results support a feature-level and complementary interpretation rather than a universal instance-extraction claim.
Response to Reviewer ZovJ (Part III)
3. Generality of privacy memory
We evaluated privacy memory across token frequencies, token categories, optimizer coefficients, and optimizer-state precision. The persistence experiments therefore span two training corpora: CodeAlpaca, used for the Qwen3-0.6B natural run, and WikiText, used for the controlled distilGPT-2 marker experiments. The final CodeAlpaca row in the horizon table provides an additional corpus replication.
The token-category analysis is offline on saved trajectories. The primary fp32 trajectory is the Qwen3-0.6B natural run with CodeAlpaca and β2 = 0.999; the bf16 values come from the paper's original run. The table compares trained and never-seen tokens within each category at the final checkpoint (step 19,999), using embedding-row v_t norms. Categories are whitespace, punctuation/symbol, stopword, identifier/numeric, alphabetic content, and other.
| Category | Trained n | Median count | AUC, fp32 | TPR@1%, fp32 | AUC, bf16 | TPR@1%, bf16 |
|---|---|---|---|---|---|---|
| whitespace | 122 | 9 | 0.992 | 0.89 | 0.999 | 0.91 |
| punct/symbol | 2,303 | 4 | 0.994 | 0.84 | 0.995 | 0.91 |
| stopword | 797 | 10 | 0.991 | 0.90 | 0.993 | 0.96 |
| identifier/numeric | 1,948 | 3 | 0.982 | 0.79 | 0.989 | 0.84 |
| alphabetic (content) | 14,830 | 3 | 0.984 | 0.81 | 0.991 | 0.89 |
| other | 3,270 | 3 | 0.996 | 0.79 | 0.998 | 0.80 |
| all | 23,270 | -- | 0.991 | 0.81 | -- | -- |
For the following table, “initially detectable” means above the 99th percentile of never-seen-token scores at the first snapshot at or after the token's last occurrence. τ denotes the number of steps remaining after that last occurrence, and later entries are the fractions still above the corresponding threshold.
| Occurrences | n | Initially detectable | τ = 1k | 2k | 5k | 10k | 15k | 19k |
|---|---|---|---|---|---|---|---|---|
| 1 | 6,778 | 0.993 | 0.99 | 0.98 | 0.92 | 0.34 (2,942) | 0.25 (1,382) | 0.19 (207) |
| 2–3 | 5,538 | 0.996 | 1.00 | 0.99 | 0.94 | 0.48 (1,447) | 0.27 (488) | 0.18 (74) |
| 4–10 | 5,096 | 0.996 | 1.00 | 1.00 | 0.97 | 0.65 (383) | 0.27 (77) | 0.10 (10) |
| 11–100 | 4,578 | 0.999 | 1.00 | 1.00 | 0.96 | 0.54 (13) | -- | -- |
Single-occurrence tokens remain detectable with retention 0.92 after 5,000 steps and 0.34 after 10,000 steps in the fp32 β2 = 0.999 setting. Here τ is measured after the token's only occurrence.
A controlled replication inserts 384 otherwise-unused marker tokens exactly three times and never again. Across three seeds, the markers remain detectable after 10,000 steps with AUC 0.991 ± 0.000 and TPR@1% FPR 0.675 ± 0.021.
In this controlled experiment, 384 otherwise-unused marker token ids are injected k times during steps 0–199 and never used again. The experiments use distilGPT-2 on WikiText, learning rate 5 × 10⁻⁵, 10,000 training steps, and snapshots every 500 steps. Each marker is scored by its optimizer-state row norm against unused token ids. Adafactor is scored using exp_avg_sq_row and Lion using exp_avg. The horizon is the last snapshot at which TPR@1% is at least 0.5, after subtracting the 200-step injection window. Markers are synthetic and one model is used.
| Configuration | Seeds | First post-window TPR@1% | Final AUC | Final TPR@1% | Horizon (steps) |
|---|---|---|---|---|---|
| AdamW β2 = 0.999, k = 1 | 3 | 0.98 | 0.969 | 0.29 | 7,300 |
| AdamW β2 = 0.999, k = 3 | 3 | 1.00 | 0.991 | 0.66 | ≥ 9,800 (end of run) |
| AdamW β2 = 0.999, k = 10 | 3 | 1.00 | 0.997 | 0.98 | ≥ 9,800 (end of run) |
| AdamW β2 = 0.99, k = 3 | 2 | 0.997 | 0.56 | 0.01 | 800 |
| AdamW β2 = 0.95, k = 3 | 2 | 0.07 | 0.509 | 0.005 | < 300 |
| SGD with momentum 0.9, k = 3 | 2 | 0.06 | 0.630 | 0.031 | < 300 |
| Adafactor, k = 3 | 2 | 0.997 | 0.951 | 0.049 | 3,050 |
| Lion, k = 3 | 2 | 0.605 | 0.538 | 0.014 | 300–800 |
| AdamW β2 = 0.999, k = 3, CodeAlpaca | 3 | 1.00 | 0.996 | 0.997 | ≥ 9,800 (end of run) |
Under the same controlled AdamW setting (β2 = 0.999, k = 3), the WikiText run and the CodeAlpaca replication both retain detectable markers through the end of the 9,800-step post-injection evaluation window, although their final AUC/TPR@1% differ (0.991/0.66 versus 0.996/0.997).
State precision also matters. The table varies model and optimizer-state storage precision; entries are early-only AUC at approximately 1k, 5k, 10k, and 20k steps.
| Storage | Qwen3-0.6B | distilGPT-2 |
|---|---|---|
| bf16 weights + bf16 state (the paper's runs) | 0.999 / 0.997 / 0.998 / 0.9955 | 0.999 / 0.998 / 0.996 / 0.9935 |
| fp32 weights + bf16 state | 0.999 / 0.996 / 0.998 / 0.997 | 1.000 / 0.999 / 0.999 / 0.998 |
| fp32 weights + fp32 state | 0.998 / 0.991 / 0.980 / 0.917 | 1.000 / 0.994 / 0.974 / 0.858 |
At 20k, Qwen3-0.6B early-only AUC is 0.9955 with bf16 state versus 0.917 with fp32 state, showing representation- and optimizer-dependent persistence.
Response
Thank you for your response. My concerns have been adequately addressed, and I will maintain my positive score.
Response to Reviewer ZovJ (Part I)
We thank the reviewer for the constructive questions. We provide further evidence and clarification addressing the concerns below.
1. Practicality under larger checkpoint intervals
Adjacent signed first moments permit exact single-step gradient recovery. The original experiment shows rapid degradation with larger gaps: spacing 1 gives 72.5 dB MNIST PSNR versus 9.7 dB for a weight-difference proxy, with reconstruction near the noise floor around spacing 10. Non-adjacent first moments instead expose an exponentially weighted gradient window (Eq. 10), enabling interval-level provenance even when instance reconstruction is weak.
For distilGPT-2 (d=768), we use AdamW (learning rate 5×10⁻⁵, weight decay 0.01, 100 warm-up steps), 64 post-warm-up steps, and three seeds. A DAGER-style span check is applied to the layer-0 attention input projection. Entries are mean top-|T| recall over the vocabulary, where |T| is the number of distinct tokens in the Δ-step window; row parentheses give tokens per step and cell annotations give saturated rank.
Recall drops when effective rank approaches the layer width.
A separate Δ=1 fp32 packed-WikiText-2 experiment (two seeds) varies tokens per step; parentheses give mean distinct-token counts and cells are top-|T| recall from the exact adjacent gradient using layer-0 q_proj.
Values are means over two seeds; recovery degrades when distinct-token count exceeds model width. The Qwen3-8B result at 16,384 tokens is one tested configuration, not a universal guarantee.
For interval inference,
dv=v_b−β2^Δv_a(β2=0.999),v_bis the single-checkpoint baseline, anddθthe weight-difference baseline. Values range over a∈{2,000,5,000,10,000}; hard negatives precede but are absent from the interval.TPR@1%, hard/allcalibrates on hard/all non-interval negatives.At smaller β2 values:
We scanned two ranked samples of Hugging Face Hub repositories for optimizer-state files and multiple checkpoints from the same training run.
These are counts from ranked samples rather than hub-wide prevalence estimates.
Thus, larger gaps reduce complete-example reconstruction but can still expose interval-level token sets or feature provenance.