acceptodds
Withdrawn from ICLR 2027

Anchored or Drifting: What Recursive Self-Generation Reveals About Training Data

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large generative models are known to memorize their training data, posing severe privacy risks. Yet, current methods to detect training membership typically rely on the weak signals of a single forward pass. In this work, we find that training samples and unseen (held-out) data follow visibly different trajectories under recursive self-generation - repeatedly feeding a model's output back as its next input. Held-out samples *drift*: they lose the specifics of the original within a few steps. Training samples stay *anchored*, degrading far more slowly. We show that the membership signal this produces holds across model modalities, architectures and scales, spanning language, diffusion, and autoregressive vision models. Furthermore, these recursive trajectories provide a signal that raises membership inference TPR at FPR for nearly every attack we evaluate, roughly doubling it on the weakest baselines and still improving the strongest, which shows that a model's behavior under recursion carries membership evidence that a single query does not.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Memorization, Data Privacy, Membership Inference Attacks, Dataset Inference, Generative Models

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_4LtAOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Recursive Self-Generation (RSG), a framework for improving membership inference attacks by repeatedly feeding a generative model’s output back as its next input. The authors observe that training samples tend to remain anchored near their original states, whereas non-members drift more rapidly toward the model’s learned distribution. They extract trajectory features and combine them with existing attack scores through an L1-regularized logistic regression classifier. Experiments across diffusion models, image autoregressive models, Transformers, and state-space language models show improved TPR at 1% FPR in 118 of 122 attack–model combinations, suggesting that recursive trajectories contain membership information missed by single-query attacks. However, the method requires labeled member and non-member data for calibration, incurs multiple generation queries, and sometimes reduces AUROC on LLMs, indicating that its gains are concentrated in the low-FPR region rather than reflecting uniformly better membership discrimination.

Critical strengths

The anchored-versus-drifting phenomenon identified in this paper is both novel and conceptually compelling. The experiments demonstrate that recursive self-generation trajectories provide substantial membership signals beyond those captured by existing attacks, particularly in the low-FPR regime. Most notably, the phenomenon is consistently observed across different modalities and model families. This cross-modal and cross-architectural generality significantly strengthens the paper’s contribution and suggests that the proposed phenomenon may reflect a broadly shared property of generative models. More broadly, this finding opens up a promising new perspective on memorization in generative models and is likely to inspire substantial follow-up work on privacy auditing, membership inference, and generative-model dynamics.

Critical weaknesses
  1. The method requires labeled calibration data. It trains a logistic regression classifier using known members and non-members. However, in a realistic MIA setting, the attacker may not know which samples were used to train the target model. The paper should clarify where these labels come from and test whether the method works without target-specific labeled data.

  2. AUROC decreases on some LLMs. The method improves TPR at 1% FPR, but Table 13 shows lower AUROC for several attacks. Low-FPR performance is important, but AUROC still measures the attack’s overall ability to separate members from non-members. The paper should explain this trade-off and show more complete results, such as ROC curves and partial AUC.

Questions for authors

Where do the labeled member and non-member samples used to train the logistic regression classifier come from? Can the method still work without target-specific labeled data?

Why does the method improve TPR at 1% FPR but reduce AUROC for several LLM attacks? Could you provide full ROC curves or partial AUC results to explain this trade-off?

3 comments

We sincerely thank the reviewer for their time and feedback.

W1: This is the standard protocol, not an additional assumption. Examples of established works that use the same procedure are LiRA [1], CDI [2], and MIA for IARs [3].

Q1: The labels come from the benchmark’s own ground-truth split, which is what makes evaluation possible at all: ImageNet and COCO train/test splits for vision models, and MIMIR’s member/non-member sets for LLMs. Concretely, we draw repeated stratified train/test splits, fit the L1L_1-regularized logistic regression on the training half of each, evaluate on the held-out half, and aggregate scores across splits. This is the standard protocol in the MIA literature [1], [2].

W2/Q2: The two metrics measure different regions of the ROC curve, and our features act on one of them. Trajectory features sharpen the high-confidence tail, where a positive call can be acted on, while contributing little or occasionally costing in the mid-FPR regime that dominates AUROC. Following [1], we judge attacks by TPR@1%FPR rather than AUC, because AUC averages over the high-FPR regime where no audit would operate, so a method that improves confident detections while losing marginally in aggregate ranking is nonetheless the stronger attack. Also, Table 13 shows the effect on AUC is small and two-sided.

We hope our responses adequately address the concerns and appreciate your consideration of these clarifications in your final assessment.

[1] Carlini et al. Membership Inference Attacks From First Principles, 2022

[2] Dubiński et al. CDI: Copyrighted Data Identification in Diffusion Models, 2025

[3] Kowalczuk et al. Privacy Attacks on Image AutoRegressive Models, 2025

I understand that member and non-member labels are needed to evaluate attack performance. My concern is that your method also uses known members of the target model to train the attack classifier. This is a separate requirement: an evaluator can use ground-truth labels to measure performance, while an attacker may not have access to confirmed members of the target model’s training set. The held-out evaluation does not address the practical availability of this calibration data. Could you clarify which realistic attack scenarios provide such known members, and test whether the gains persist when the classifier is calibrated on an auxiliary model without using target-specific membership labels?

Sorry for the misunderstanding. Even established MIAs use some calibration data. From the Min-K++ paper: "The authors of Min-K% were using validation data to determine the hyperparameter k, yet the validation data is not released. For this reason, we choose to sweep over 10, 20, ..., 100 and report the best result for both Min-K% and Min-K%++ for fair comparison". CAMIA also uses calibration data to fit LR.

But our approach's gains stay even under the more practical setup:

Assume an auditor has (i) the target data X whose membership is in question (unlabelled) and (ii) a set N of known non-members (the established assumption of MIAs). We replace the LR with an LDA estimated from X and N alone. The approach therefore needs nothing beyond the standard MIA calibration assumption. Results (TPR@1%FPR):

Model, attackBase+ours, LDA+ours, LR
VAR-30, Loss4.06.46.6
VAR-30, Min-K%8.412.512.7
RAR-XXL, Loss6.09.29.4
RAR-XXL, Min-K%10.913.114.5
DiT-XL, PIA2.825.321.2
DiT-XL, Denoising loss1.326.027.4

Without any member labels, the LDA variant recovers all of the LR gain or even beats it.

We hope this clarification adequately addresses the reviewer's concern.

Reviewer_vyDBOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work demonstrates that several types of recent generative models show common behavior of anchoring to a training sample when given part of that training sample. This tendency can be used to construct self-referential membership attacks that only requires the black box access to the generative model tested. In particular, this work suggests that recursive self-generation, which is applied in slightly different means for each modality tested, can be used as a generic strategy that uses not a single query but a sequence of recursive queries.

Critical strengths

The main idea is clearly stated and intuitively demonstrated at PoC level. Specifically, relative performance comparisions between trajectory-based attacks and sample-based attacks are executed in many aspects (including satisfactory ablation studies). This is the fundamental idea under this work, which is well demonstrated for language models, diffusion models, and autoregressive vision models (albeit the models tested are somewhat limited in size due to the authors' hardware specs).

Critical weaknesses

I was so expecting rigorous consideration on how to measure the distance of two trajectories, yet this seemed to be taken a very practical perspective of perusing a collection of pre-existing "distance"s for each modality (image and langauge). Yes. I even went to the appendices for deeper insights, but sadly I could not find justification that will help linking theoretical analysis with the practical phenomena.

Due to the authors' GPU hardware setting, the models tested in this work is somewhat limited in their sizes. I think this behavior deserves more extensive analysis (be it empirical) over a varying model size range. More on this at the questions section.

Questions for authors
  1. Any theoretical justifications for the composite distance function used in this work? I am curious how this δ\delta will fare with the theory and/or extract deeper insights from the empirical results.

  2. I am intrigued by how anchoring behavior frequencies change over different training set sizes. Naturally, I am also curious of how this trends over different model sizes. Would you mind giving comments or some results for further discussion?

1 comment

We thank the reviewer for their thoughtful comments and feedback.

Q1: Proposition J.4 is proved for the Euclidean norm on the sample space, so MSE instantiates it directly; LPIPS and SSIM do not inherit the bound as stated. What the theory does license for all three is the shape, since any metric that is locally Lipschitz-comparable to l2l_2 around x⋆x^\star preserves the geometric-decay-to-a-floor structure of Corollary J.5, changing only the constants λ\lambda and εσ/(1−λ)\varepsilon_\sigma/(1-\lambda). This is why we treat the composite distance as several estimators of the same pair (λ,δ∞)(\lambda, \delta_\infty) under different geometries rather than as one distance: MSE is sensitive to pixel-level return, LPIPS to semantic return, SSIM to structural return. There are of course many ways to measure distance along a trajectory, and more sophisticated choices are often richer empirically, but they tend to be considerably harder to analyze.

Q2: Training-set size is the one axis we control cleanly in Section 4.3 with the DDPM (see Fig. 3). Anchoring weakens monotonically as the set grows - member endpoint displacement stays flat while non-members travel much further, and over 90% of non-member trajectories terminate closer to the training pool than they began at every size. Model size is examined in Appendix H.1 (not directly in the same experiment, but still very relevant) across eight vision models. E.g., in VAR the gap grows with scale.

The limit on model sizes is not a compute constraint: evaluating MIA requires ground-truth membership labels, and no larger open model provides a verifiable train/test split, which is why benchmarks are restricted to exactly the families we use.

We hope our responses adequately address the concerns and would greatly appreciate your reconsideration of the score.

Reviewer_B7JUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Recursive Self-Generation (RSG) for detecting training-data membership in generative models. The main idea is that training samples remain close to their original representations under repeated generation, while non-members gradually drift away.

The method extracts trajectory features and combines them with existing membership inference attacks using logistic regression. Experiments cover LLMs, diffusion models, and image autoregressive models. The paper reports higher TPR@1%FPR in 118 of 122 attack model pairs, and says matched-compute augmentation does not recover the gain.

Critical strengths
  1. Interesting and useful idea. Recursive generation provides a different way to study memorization. The controlled DDPM experiments (Figure 3) offer good supporting evidence.

  2. Broad experimental coverage. The method is evaluated across different architectures and modalities. The matched-query comparison in Table 6 is particularly useful for testing whether improvements come from recursion rather than simply using more queries.

Critical weaknesses
  1. The theoretical connection between anchoring and membership is not fully established. Appendix J, Assumption J.2, defines memorization through local density sharpness rather than actual training-set membership.

  2. The fused-vs-baseline comparison is not like-for-like, so the headline gain cannot yet be attributed to recursion. Sec. 3.4 says scorer, splits and base attacks are identical in both arms. Sec. 4.7 says any gain is “information … the base attack does not already extract”. The released code differs.

  3. Potential evaluation leakage in the released code. In src/mia_utils/metrics.py, pick_best_column() selects attack configurations using labels from the complete dataset (lines 22–35), while orient_and_bootstrap() determines score direction using full-data AUC (lines 116–139).

Questions for authors

Along with weaknesses, please answer following questions.

  1. How many member and non-member samples per model and domain are used? How many LLM documents remain after the 328-token filter? Are the baseline columns computed on that same subset and on the same 200 tokens?

  2. Why does fusion lower AUC in Table 13 when the base score is a feature? And what exactly are the +aug perturbations?

  3. Why are attack parameters and score directions selected using the full labeled dataset in metrics.py rather than training data only?

  4. How does Assumption J.2 distinguish a memorized training example from an unseen example located near a sharp density mode?

1 comment

We thank the reviewer for their feedback.

W2: If an attack is a single score, adding LR on top of it does not change its results – fitting a logistic regression on one feature is a monotone transform, so the ranking, and therefore the AUC and TPR@FPR, are unchanged – while +ours needs the aggregation since it has multiple features. When the attack is multidimensional, we use LR for BOTH baseline and +ours, so the protocol is matched in every case. The difference visible in the code is this shorter path for scalar scores, which is equivalent.

Q1: Baselines are computed over the same tokens. For vision models, we use 5000 members from the ImageNet/COCO train split and 5000 non-members from the corresponding validation split. For LLMs, we take 1000 members and 1000 non-members per MIMIR domain. After filtering, about 30% of samples remain.

W3/Q3: We are extremely grateful for this question. It was an unintended data leakage bug in our implementation. We have fixed this problem in our code. We present the corrected results below along with 95% CIs. As the updated tables demonstrate, while the absolute numbers shifted slightly, the significant performance gains of our method hold.

VAR-d30 (TPR@1%FPR)

MIABase TPR+ours TPRΔ ± 95% CI
Loss4.06.6+2.7 ± 1.7
Zlib4.57.7+3.2 ± 2.0
Min-K%8.412.7+4.2 ± 2.5
SURP4.06.6+2.7 ± 1.7

DiT-XL (TPR@1%FPR)

MIABase TPR+ours TPRΔ ± 95% CI
SecMI2.115.3+13.2 ± 3.8
Denoising loss1.327.4+26.1 ± 6.1
PIA2.821.2+18.4 ± 4.7
CDI77.283.7+6.6 ± 3.2

VAR-d30 (AUC)

MIABase AUC+ours AUCΔ ± 95% CI
Loss74.380.4+6.0 ± 0.8
Zlib74.580.6+6.1 ± 0.8
Min-K%82.986.4+3.5 ± 0.5
SURP74.380.4+6.0 ± 0.8

DiT-XL (AUC)

MIABase AUC+ours AUCΔ ± 95% CI
SecMI65.990.4+24.5 ± 1.2
Denoising loss52.292.0+39.8 ± 1.2
PIA63.992.3+28.4 ± 1.4
CDI98.499.0+0.6 ± 0.1

Q2: The corrected results presented above have no such effect. The +aug perturbations are EDA-style [1] word-level edits (random word delete, swap, insert) for text, while for vision they are standard image augmentations consisting of RandomResizedCrop, color jitter, Gaussian blur, and random patch masking. We do not use heavy flips or crops.

W1/Q4: Assumption J.2 does not distinguish them, and is not intended to, since it is a property of the density at a point, so any sharp mode of pθ(σ)p_\theta^{(\sigma)} satisfies it whether or not that point was trained on. As stated in Appendix J.1, that training induces such modes at training points is the empirical premise of this work, not something the assumption establishes. The discrimination enters instead through what we measure. Proposition J.4 is anchored at the mode x⋆x^\star, but we measure δt=δ(Z0,Zt)\delta_t = \delta(Z_0, Z_t) – displacement from the query point. For a memorized sample Z0≈x⋆Z_0 \approx x^\star and the drift floor is small; for an unseen sample near a mode at x⋆≠Z0x^\star \neq Z_0 the trajectory contracts toward x⋆x^\star, leaving δ∞≈∥Z0−x⋆∥>0\delta_\infty \approx \|Z_0 - x^\star\| > 0. This is the transport reported in Section 4.3, where over 90% of non-member trajectories end closer to the training pool than they began. Membership is thus indicated by small λ\lambda and small δ∞\delta_\infty, which is why Section 3.3 uses both.

We hope our responses address the concerns, and we would greatly appreciate your reconsideration of the score.

[1] Wei et al., EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks, EMNLP 2019

Reviewer_WpWjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes recursive self-generation (RSG) for membership inference: a sample is repeatedly fed back through a frozen generative model. Training samples stay close to the original (“anchored”), while held-out samples drift away. Trajectory features are concatenated with existing MIA scores and fused by a logistic regression. This improves TPR@1%FPR in 118 of 122 attack–model pairs across image autoregressive, diffusion, and language models.

Critical strengths
  1. The idea is simple and broadly applicable. It needs only the ability to generate a sample of the same type, adds no shadow models, and is evaluated across three model families and two modalities. Reframing model-collapse style recursion as an inference-time measurement is a nice conceptual move.

  2. Evidence that the signal is instance-specific. In the occlusion experiment, members recover the exact original far more often than non-members when ranked against same-class distractors. The controlled DDPMs show non-members being pulled toward the training set.

  3. The evaluation focuses on TPR at low FPR rather than AUC alone, which is the right metric for auditing. It also includes a matched-compute control. That control addresses the most obvious objection, that the gain comes only from spending more queries.

Critical weaknesses
  1. The contribution of recursion is not isolated. Compared with the baseline, the fused attack adds both recursion and a set of generation-based features (e.g., per-step MSE/LPIPS/SSIM, or 12 text features for LLMs), combined by a trained logistic regression. Even a single regeneration step (T=1) gives a reconstruction-error signal that is known to be informative for MIA. Without fusing the base score with T=1 features under the same classifier, it is unclear how much of the gain comes from recursion rather than from adding reconstruction-type features. Appendix I studies this only on small CIFAR-10 DDPMs.

  2. For LLMs, the gain may not come from recursion, and distribution shift is not ruled out. Most LLM trajectory features (match ratio, MRR, log-prob, edit distance) compare the model’s output with the true continuation of the original text, so they are close to the likelihood signals that Loss or Min-K% already use, just computed at more positions. Consistent with this, all base attacks converge to nearly the same fused score within each domain (Table 5). It is also unexpected that fusion lowers AUC for several attacks (Table 13). Because the fusion is a supervised classifier over many features, it may also exploit differences between member and non-member text rather than memorization; the very large jump on DM Math (Zlib 11.6 → 74.2) should be checked with a blind baseline, e.g., the same features computed with a reference model not trained on the Pile.

  3. No uncertainty estimates. Test-set sizes, confidence intervals, and multiple splits are not reported for TPR@1%FPR, and many gains are 0.2–2 points. It is unclear how generation strength s was selected for the main tables.

  4. The theory is weakly connected to the method. It covers only diffusion denoisers and treats memorized points as local modes. The contraction rate it defines is never estimated in the experiments.

Questions for authors
  1. What is the TPR@1%FPR when the base score is fused with T=1 (single-step) features using the same logistic regression?

  2. Does the +aug control use the same LR fusion and number of features as +ours?

  3. For LLMs, how is the true continuation defined after the original prefix leaves the window? Which features dominate the fitted LR?

  4. How many test samples are used, and what are the confidence intervals over multiple splits?

2 comments

We sincerely thank the reviewer for dedicating their time to evaluating our manuscript and for providing constructive comments.

Q1: The requested experimental results are as follows:

RAR-XXL: T=1 vs T=5 (TPR@1%FPR)

MIABase TPRT=1T=5
Loss6.09.110.1
Hinge6.18.79.4
Min-K%12.614.715.1
SURP4.48.49.1
CAMIA3.97.17.0

Both are fused with the base score by the same L1 logistic regression. For instance, in diffusion models, the gain is even higher, since there are the strongest signals.

Q2: In the paper, we tested the +aug version where we average the MIA features from multiple augmentations and then score once. Now, we also computed the approach that uses MIA features from the original instance and concatenated trajectory features computed on augmented images (the same features as our approach uses, and the same LR fusion). The results:

DiT-XL: +aug vs +ours (TPR@1%FPR)

MIA+aug TPR+ours TPRΔ ± 95% CI
SecMI4.615.3+10.7 ± 4.0
Denoising loss4.527.4+22.9 ± 6.0
PIA5.621.2+15.6 ± 4.9
CDI80.183.7+3.6 ± 3.1

RAR-XXL: +aug vs +ours (TPR@1%FPR)

MIA+aug TPR+ours TPRΔ ± 95% CI
Loss7.69.4+1.8 ± 1.6
Zlib10.411.2+0.9 ± 1.9
Min-K%13.214.5+1.3 ± 1.9
SURP5.28.9+3.7 ± 1.7
CAMIA4.16.4+2.3 ± 1.5

Q3: The true continuation is never undefined in this specific approach. It is always the ground-truth document tokens at the window’s current offset, which is tracked independently of what the model sees. At iteration tt, the model generates mm tokens from its own current context, and these are compared against the real document tokens at positions [tm+k, tm+k+m)[tm + k,\, tm + k + m). It is also possible to do this procedure without relying on ground truth using distributional features, e.g., JSD, mean entropy, logit margin.

The table summarizing the 6 most important features for the fitted LR:

attack scoreentropy_meanedit_distjsdlogproblogit_margin
share38.1%10.4%6.8%6.7%5.0%4.1%

These numbers are the share of the absolute standardised coefficients in the LR on [attack score ‖ trajectory features], averaged over 20 bootstrap refits for each of 6 models × 8 attacks × 7 MIMIR domains.

Q4: For the vision domain, we use 10k samples from ImageNet and from COCO. While the MIMIR dataset has ~3.5k samples, we use it all. The results with 95% CIs over 50 splits:

VAR-d30 (TPR@1%FPR)

MIABase TPR+ours TPRΔ ± 95% CI
Loss4.06.6+2.7 ± 1.7
Zlib4.57.7+3.2 ± 2.0
Min-K%8.412.7+4.2 ± 2.5
SURP4.06.6+2.7 ± 1.7

DiT-XL (TPR@1%FPR)

MIABase TPR+ours TPRΔ ± 95% CI
SecMI2.115.3+13.2 ± 3.8
Denoising loss1.327.4+26.1 ± 6.1
PIA2.821.2+18.4 ± 4.7
CDI77.283.7+6.6 ± 3.2

We hope our responses address the concerns, and we would greatly appreciate your reconsideration of the score.

Reviewer_WpWjOpenReview ↗

Part of concerns solved

Thanks for the response and the new results. The +aug control and the CIs for VAR-d30 and DiT-XL help, and the DiT-XL gains look solid.

My main concern is still whether the gain comes from recursion. In the new RAR-XXL table, T=1 already gets most of the improvement (e.g., Loss 6.0 → 9.1 at T=1 vs. 10.1 at T=5), and for CAMIA T=1 is slightly better than T=5. The extra gain from T=5 is within about one point, which is inside the CIs you report for RAR-XXL. Since the paper's claim is about recursion, I would like to see the same comparison with CIs on DiT-XL and at least one LLM.

For LLMs, my W2 points (a blind or reference-model baseline, the DM Math jump, the lower AUC in Table 13) are not addressed, and no CIs are given. Also, the leakage bug mentioned in your reply to Reviewer B7JU seems to affect the main tables, but corrected numbers are only shown for two vision models. For these reasons, I will keep my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final statement

We sincerely thank the reviewers for their careful and constructive comments, which were highly valuable to us and useful to find where we should improve our evaluation. We kindly ask the AC to take the good quality of these reviews into account - as our feedback for this rebuttal.

While addressing the issues raised about our LLM evaluation, we discovered that the member/non-member split of the benchmark used in our evaluation is affected by a distribution shift between the two sets, so the benchmark partly measures distribution shift rather than membership. Re-evaluating our method for LLMs on a properly constructed, shift-free benchmark was not feasible within the short rebuttal period, and we therefore decide to withdraw the submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.