acceptodds
Under review as a conference paper at ICLR 2027

Learning to Detect Training Data via Active Reconstruction

Abstract

Detecting LLM training data is generally framed as a membership inference attack (MIA) problem. However, most MIAs operate passively on fixed model weights, using log-likelihoods or text generations. In this work, we introduce **Active Data Reconstruction Attack** (ADRA), a family of MIAs that actively induces a model to reconstruct a given text through training. We hypothesize that training data are *more reconstructible* than non-members, and the difference in their reconstructibility can be exploited for membership inference. Motivated by findings that reinforcement learning (RL) sharpens behaviors already encoded in weights, we leverage on-policy RL to actively elicit data reconstruction by finetuning a policy initialized from the target model. To effectively use RL for MIA, we design reconstruction metrics and contrastive rewards. The resulting algorithms, ADRA and its adaptive variant ADRA+, improve both reconstruction and detection given a pool of candidate data. Across pre-training, post-training, and distillation, our best method improves over the strongest available loss-based baseline by 11.2 AUROC points on average across 14 evaluation settings, and the strongest of all evaluated baselines by 6.8 points. As a highlight, ADRA+ improves over Min-K%++ by 18.8 points on BookMIA and 7.6 points on AIME.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Training Data Detection, Memorization, Membership Inference Attack, Data Extraction, Meta-Learning.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mMBPOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Active Data Reconstruction Attack (ADRA), a reinforcement-learning-based approach to membership inference attacks (MIAs) for LLMs. ADRA fine-tunes a policy initialized from the target model using on-policy reinforcement learning with reconstruction-oriented rewards. The authors introduce lexical reconstruction metrics and a contrastive matching reward to encourage the model to reconstruct candidate text. An adaptive variant, ADRA+, further incorporates membership priors derived from conventional loss-based MIA scores. The paper evaluates the proposed methods across pre-training, post-training, and distillation settings. The results show that ADRA and ADRA+ generally outperform the evaluated passive baselines in terms of AUROC, with particularly strong performance on BookMIA, Dolma3 arXiv, AIME, and paired distillation-trace discrimination. Additional experiments investigate reconstruction quality, optimization controls, computational costs, and the dependence of the attack on prior training exposure.

Critical strengths
  1. This paper leverages the ability of RL to reinforce behaviors already encoded in model parameters, progressively amplifying the differences in reconstructibility between data seen and unseen during pre-training, thereby facilitating more effective membership inference. The core idea is simple and straightforward, and the proposed method is relatively easy to implement.
  2. The authors evaluate their approach across three different training stages (pre-training, post-training, and distillation) and several model families. The proposed methods achieve substantial AUROC improvements over existing baselines in several settings, especially on BookMIA, Dolma3 arXiv, and controlled AIME contamination.
Critical weaknesses
  1. Compared with training-free MIA methods, the RL process introduces additional computational and time overhead and requires an additional training data construction process. Consequently, the overall cost of ADRA is higher than that of conventional loss-based MIA methods.
  2. ADRA+ incorporates scores obtained from conventional loss-based MIA methods as prior information, effectively constructing a form of pseudo-labeling signal. From this perspective, ADRA+ can be viewed as a mechanism for amplifying existing membership signals provided by conventional loss-based MIAs. Therefore, it may be more appropriate to regard ADRA+ as a supplementary enhancement rather than a standalone methodological contribution, and its contribution appears less significant than that of ADRA.
Questions for authors
  1. Why does contrastive matching improve membership separation? Could the authors report how frequently the true suffix achieves the maximum similarity for member and non-member examples throughout training? Additionally, how sensitive is the method to the number and difficulty of distractors?
  2. Given that FSD has already demonstrated the effectiveness of target-model fine-tuning for MIA, could the authors further clarify the specific methodological novelty of ADRA beyond replacing fine-tuning-based score deviation with reconstruction-oriented RL? In particular, what evidence establishes the necessity and unique benefit of on-policy RL for extracting membership signals?
7 comments
AuthorsOpenReview ↗

Response to Reviewer mMBP (1/3): Cost

Thank you for the detailed review and close-read of our paper! We are glad to see you find our method to be "simple and straightforward", our paper to conduct comprehensive experiments across trainings tages, and show "proposed methods achieve substantial AUROC improvements over existing baselines in several settings".

We respond to your weaknesses/questions inline below, please also feel free to see our general response about additional results, ablation, and analysis.

Compared with training-free MIA methods, the RL process introduces additional computational and time overhead and requires an additional training data construction process.

Agree that our method is more expensive than training-free MIAs, we discuss this more in our limitation. We think the method compensates the training cost by trading the RL compute with more MIA accuracy and fewer False Positives (best TPR@1%FPR in 12 of 14 settings; Tables 18, 20, and 21). In high-stakes scenario (like copyrighted data, private medical/presonal data audit), MIA accuracy and low FPR is more important than cost-effectiveness, as noted in [1].

For large-scale screening of data, a cheap passive MIA can serve as a first-pass filter, with ADRA used for the later selected candidates.

The cost is also quite small in absolute terms. Under strict RL budgets (Section 6 and Appendix D.2), a <$1 run already beats all passive baselines in AUROC, and a <$7 run recovers near-full performance on both BookMIA and Dolma3 arXiv:

AUROCBest baseline (N-Sampling)1 RL step ($0.68)10 steps ($6.82)30 steps, full ($20.45)
BookMIA, ADRA73.674.775.676.2
BookMIA, ADRA+73.675.877.678.4
Dolma3 arXiv, ADRA81.584.289.991.8
Dolma3 arXiv, ADRA+81.584.990.792.4

*Costs are for one full attack run over 512 candidates on 8×H200, prorated from measured 30-step runtimes.

References:

[1] Carlini, Nicholas, et al. "Membership inference attacks from first principles." 2022 IEEE symposium on security and privacy (SP). IEEE, 2022.

AuthorsOpenReview ↗

Response to Reviewer mMBP (2/3): Method Details

ADRA and ADRA+

ADRA+ amplifies the membership signal of existing loss-based MIAs, so it may be better viewed as a supplementary enhancement than a standalone contribution, and is less significant than ADRA.

Agree that ADRA+ is a supplementary improvement over ADRA, that's why we termed it "ADRA+"! We think it's natural to combine our method with loss-based MIAs as they are fairly orthogonal, and report the ADRA+ results seperately.

ADRA+ improves over ADRA in 10 of the 12 settings where it applies. ADRA alone, without any loss-based prior, already outperforms the strongest baseline in 13 of 14 settings (Tables 2, 4, and 6). We will make this clearer in the final version.

Contrastive Matching & Similarity

Why does contrastive matching improve membership separation? Could the authors report how frequently the true suffix achieves the maximum similarity for member and non-member examples throughout training?

Contrastive matching widens the gap between member and non-member reconstruction, and the bigger this gap, the better the MIA performance.

Since our attack does not know which candidates are members, without distractors (K = 0) every candidate, member or non-member, is pushed toward its own true suffix. With distractors, a candidate is no longer forced to match its true suffix, Member rollouts still can match their true suffix, since the model has seen it, while non-member rollouts mostly match distractors instead. This contrastive RL therefore improves member reconstruction much more than non-member reconstruction.

We show this empirically. Removing distractors drops AUROC on AIME from 85.8 to 81.3 (no training: 73.1; Section 6 and Figure 2). We also report how often the true suffix achieves the maximum similarity:

Table R2. How often the true suffix wins the matching reward (% of rollouts; AIME, K = 7).

Before RLStep 25After RL
Member58.1%68.0%77.5%
Non-member15.3%13.9%17.8%

The member match rate grows from 58.1% to 77.5% during training, while the non-member rate stays roughly flat, so the gap widens from 42.8 to 59.7 points. We hope this clarifies the concern!

Additionally, how sensitive is the method to the number and difficulty of distractors?

Great point!

Below, we show AIME AUROC as we vary the number of distractors K (each rollout is scored against K + 1 suffixes):

Table R5. AUROC on AIME vs. number of random distractors K.

K Distractors0 (no matching)137 (default)1531
ADRA81.381.684.585.886.185.5

We see that after the benefits of adding more distractors K saturates at around K=7.

We also conduct additional experiment where instead of randomly drawing from the pool, for each candidate we select distractors based on their token simialrity with the true suffix.

Table R6. AUROC on AIME with random vs. hard distractors (K = 7).

DistractorsRandom (default)Hard (Top K=7 similar suffixes in the pool)
ADRA85.886.7

We see that increasing the difficulty actually further improves MIA performance, albeit at the cost of ranking the pool by similarity for every candidate.

AuthorsOpenReview ↗

Response to Reviewer mMBP (3/3): RL vs. SFT

Given that FSD has already demonstrated the effectiveness of target-model fine-tuning for MIA, could the authors further clarify the specific methodological novelty of ADRA beyond replacing fine-tuning-based score deviation with reconstruction-oriented RL?

Thank you for raising FSD (we discuss our disction in Appendix K). Replacing FSD's fine-tuning with reconstruction-oriented RL changes two things, each requiring additional method components or assumptions:

  1. No labeled data. FSD fine-tunes on a labeled set of known non-members from the same domain. ADRA trains only on the unlabeled candidate pool it scores, so our method need no membership labels.
  2. A different membership signal. FSD measures how a likelihood score shifts after fine-tuning. ADRA measures how well the model reconstructs each candidate's true suffix from its own generations.

Compared to SFT which forces models to directly learn both member and non-member suffixes, On-policy RL has the unique advantage of reinforcing what the target already generates. And the contrastive reward credits the true suffix only when it beats the distractors. We show this empirically to be true in response (2/3).

In sum, FSD uses labeled non-members to calibrate likelihood scores, while ADRA elicits reconstruction of the candidates themselves without labels. To our knowledge, it is the first RL-based active MIA.

In particular, what evidence establishes the necessity and unique benefit of on-policy RL for extracting membership signals?

Our optimization controls on AIME test this directly (Section 6, Figure 2). All runs start from the same target model and the same unlabeled candidate pool (10 SFT epochs vs. 60 RL steps):

Attack-time trainingAUROC
None (N-Sampling)73.1
SFT on the candidates (3 learning rates, final)59.8–67.1
RL, random reward58.1
RL, our reward81.3
RL, our reward + matching (ADRA)85.8
RL, our reward + matching + prior (ADRA+)85.5

SFT performs below no training (N-Sampling) at all three learning rates. RL with random rewards also degrades. Only RL with reconstruction rewards improves detection, so the benefit requires both on-policy RL and reconstruction-aligned rewards.

We provide a justification in theoretical analysis (please see General Response 4/4): SFT trains on every candidate's true suffix, members and non-members alike, so it teaches the model the non-member suffixes too and erases the gap. On-policy RL only reinforces what the model already generates, so it amplifies the head start that members already have.

Reviewer_T14yOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Active Data Reconstruction Attack, a reinforcement learning-based approach for MIAs on large language models. Unlike conventional MIAs that rely on fixed model weights, ADRA uses on-policy RL to fine-tune the target model and improve the reconstruction of candidate training data. The authors introduce reconstruction-based rewards and two variants, ADRA and ADRA+, which use matching and adaptive matching strategies to guide RL training. The method is evaluated across pre-training, post-training, and distillation settings using multiple LLMs and datasets. Experimental results show that ADRA and ADRA+ generally outperform existing loss-based and reconstruction-based MIA baselines in terms of AUROC.

Critical strengths
  1. Extensive experimental evaluation. The paper evaluates the proposed method across pre-training, post-training, and distillation settings, with comparisons against several existing MIA baselines.
Critical weaknesses
  1. Poor presentation and unclear motivation. The paper does not clearly motivate the research problem, even in the abstract and introduction. The limitations of existing MIAs are not sufficiently discussed, making it unclear what specific gap the paper aims to address and why an RL-based approach is needed. The threat model and attacker capabilities are also not clearly introduced. In addition, the motivation for several key design choices in the proposed method is weak, making the overall approach difficult to follow.
  2. Limited methodological novelty. The proposed method mainly applies existing RL techniques (GRPO) with similarity-based rewards to improve text reconstruction. While the authors introduce matching and adaptive matching rewards, these designs appear relatively straightforward, and their advantages over simpler alternatives are not sufficiently justified.
Questions for authors
  1. Could the authors clarify the specific limitations of existing passive MIAs that motivate the proposed active approach? Why is RL necessary to address these limitations?
  2. Could the authors clarify the practical attack scenario in which an adversary can update the target model using the candidate data?
  3. How does this assumption affect the applicability of ADRA compared with conventional MIAs?
  4. What are the key methodological contributions beyond applying GRPO with reconstruction-based rewards?
  5. In Equation (5), the proposed contrastive reward takes the maximum similarity across the true suffix and distractors, without explicitly separating positive and negative samples. Why is this formulation preferred over a conventional contrastive objective, and what makes it particularly suitable for membership inference?
10 comments
AuthorsOpenReview ↗

Response to reviewer T14y (1/4): Motivations & Limitations

We thank the reviewer for their time and perspective. We are encouraged that the reviewer find our paper to conduct "Extensive experimental evaluation" and show effectiveness across "pre-training, post-training, and distillation settings, with comparisons against several existing MIA baselines".

However, we respectfully disagree that our paper does not contain novel elements, nor that our paper lacks sufficient motivation and discussions of assumptions. Below, we addressed explicitly some misconception and questions, and we will revise the manuscript to make them clearer.

Motivation and Limiations

The reviewer finds the motivation unclear: which limitation of existing MIAs ADRA addresses, why RL is needed, what the threat model is, and why the key design choices are made (Weakness 1, Questions 1–3). Each of these is stated explicitly in the paper, and we quote the relevant passages below.

The paper does not clearly motivate the research problem, even in the abstract and introduction.

We state the problem and its motivation are the opening sentences of both the abstract and the introduction:

  • Abstract (L11–13): "Detecting LLM training data is generally framed as a membership inference attack (MIA) problem. However, most MIAs operate passively on fixed model weights, using log-likelihoods or text generations."
  • Introduction (L56-58): "Modern language models are known to memorize and regurgitate training data, raising concerns about copyright, privacy, and data contamination."

The limitations of existing MIAs are not sufficiently discussed, making it unclear what specific gap the paper aims to address. Could the authors clarify the specific limitations of existing passive MIAs that motivate the proposed active approach?

The limiation is that existing MIAs are passive and did not sufficiently utilize membership signal in the weights. In Introduction paragraph 2 (L60–65), we wrote: "Most prior MIAs query the target model without modifying its weights, passively extracting signals exposed through the output layer [...] We hypothesize that model weights encode latent membership signal that passive methods fail to reliably expose." Section 3 also restates this (L131–132): "They assume that passive querying extracts all available membership signal."

Our results show this limitation concretely. On WikiMIA2024 Hard, loss-based MIA baselines score below chance, between. 44.5–50.6 AUROC (Table 2), and at 1% FPR "passive baselines largely collapse: methods such as Zlib and Min-K%++ frequently fall below 1% TPR" (Appendix, Table 18). In comparison, we show that our method significantly improve MIA accuracy and reduce false positives rate.

Why is RL necessary to address these limitations?

Thank you for raising this point.

We think RL has two unique advantages. First, on-policy RL only reinforces what the model already generates, so it amplifies the reconstruction advantage that members already have, rather than forcefully teaching the model new text as SFT does. Second, RL allow non-differentiable rewards and reward shaping; in our case it's the data reconstruction similarity and contrastive matching. This is our core contribution: ADRA is the first RL-based active MIA that also outperforms all other MIA baselines.

This is also well motivated by recent literature (Introduction, paragraph 3, L68–71): RL "sharpens behaviors already encoded in weights" [1, 2] "while transferring minimal new information" [3].

Empirically, we show that RL is superior to loss-based MIAs and sampling (Tables 2, 4, and 6) and to SFT (Section 6, Figure 2). Additionally, we also provide a theoretical justification in General Response (4/4).

References:

[1] Yue et al. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? NeurIPS 2025 Oral.

[2] Wang et al. Reinforcement Learning for Reasoning in Large Language Models with One Training Example. arXiv 2025.

[3] Schulman and Thinking Machines Lab. LoRA Without Regret. Thinking Machines Lab: Connectionism, 2025.

AuthorsOpenReview ↗

Response to reviewer T14y (2/4): Threat Models & Assumptions

The threat model and attacker capabilities are also not clearly introduced.

Could the authors clarify the practical attack scenario in which an adversary can update the target model using the candidate data?

Thanks for bringing this up!

We discuss this explicitly in Section 3, paragraph 'Model access' (L136–140). ADRA applies to any open-weight model: "ADRA targets open-weight models or managed APIs supporting on-policy RL with custom rewards." Such targets are widely available: "Recent open-weight releases—Kimi K3, Qwen3.8, and Inkling—and API training services such as Tinker support these two assumptions" (Section 3, footnote 2, L160). We discuss this more in detail in E.3 Model Access Assumptions.

How does this assumption affect the applicability of ADRA compared with conventional MIAs?

Importantly, our method share the same assumption as loss-based MIAs, which also require open weights. Again, we describe in Section 3 (L136–141): "On our evaluated open-weight targets, ADRA and likelihood-based MIAs share the same model-availability boundary. Note that closed-source model APIs provide neither the teacher-forced candidate likelihoods required by the likelihood-based MIAs nor ADRA's update capability."

AuthorsOpenReview ↗

Response to reviewer T14y (3/4): Novelty

Limited methodological novelty. The proposed method mainly applies existing RL techniques (GRPO) with similarity-based rewards to improve text reconstruction. What are the key methodological contributions beyond applying GRPO with reconstruction-based rewards?

Our proposed method is a new class of MIA, active MIA, which (a) updates a local copy of the target to elicit membership signal that passive methods miss, and (b) converts more compute into better MIA accuracy. To our knowledge, ADRA is the first RL-based active MIA.

In contrast, passive MIAs cannot reliably convert computeinto better performance. Loss-based MIAs cannot utilize more compute, increasing N-Sampling's budget 16x (N = 512) changes AUROC by less than one point (Appendix D), while ADRA improves with more RL training (Table 15).

Making RL work for MIA required three components that standard RL fine-tuning does not provide. We propose 3 new componenents:

  1. Reconstruction rewards (Section 3.2): dense lexical similarity instead of sparse exact match.
  2. Contrastive matching (ADRA, Section 3.3): rewarding reconstruction directly would teach the model every candidate's suffix, including non-members. Matching against distractors credits the true suffix only when a rollout resembles it more than the other candidates, so RL amplifies the advantage members already have.
  3. Adaptive Contarstive matching (ADRA+, Section 3.3): a label-free, loss-based prior sets how often the true suffix is included in matching.
Reviewer_2K9KOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes ADRA/ADRA+, which employs reward restructuring and GRPO (Group Relative Policy Optimization) reinforcement learning to actively enhance a model's ability to reconstruct candidate texts, subsequently using this capability to perform membership inference on training data. Experiments cover pre-training, post-training, and distillation scenarios, demonstrating performance superior to various baselines.

Critical strengths
  • Extensive experiments were conducted to verify the effectiveness of the method.
  • The method’s performance—in terms of both effectiveness and efficiency—was compared and analyzed.
Critical weaknesses
  • The intuition that "if the actual text appeared during training, the model-generated content is more likely to match the true suffix" lacks verification through direct experimentation or theoretical analysis.
  • ADRA is not suitable for large-scale, low-cost screening across all types of text.
  • There is a lack of experiments involving more cutting-edge models, such as those in the Qwen 3.5 series with comparable parameter sizes.
Questions for authors
  • Is it possible to demonstrate—through direct experimentation and theoretical analysis—that if a specific text sequence has appeared in the training data, the model is more likely to generate content that matches the actual subsequent text (the "true suffix")?
  • Would the same conclusion hold for more advanced models, such as those in the Qwen 3.5 series?
  • Regarding the reward function in GRPO: What is the individual contribution of each metric within the TRIO framework? It is recommended to conduct ablation studies.
8 comments
AuthorsOpenReview ↗

Response to Reviewer to 2K9K (1/4): Qwen 3.5 Experiments

Thank you for your thoughtful feedback and review! We are encouraged that you find our paper to be extensive in our experiments and show our method empirically effective across pre-training, post-training, and distillation.

We will address your questions and concerns point by point:

Would the same conclusion hold for more advanced models, such as those in the Qwen 3.5 series?

This is a reasonable concern. Below we perform the same experiment with Qwen3.5-9B on Tulu3 Mix - Wildchat dataset, following our post-training MIA setup.

Table R3. AUROC on Tulu3 Mix – WildChat (post-training detection). R-Loss uses Qwen3.5-9B-Base as the reference.

ModelADRA+ADRAN-SampLossR-LossZlibMin-K%Min-K%++
Qwen3.5-9B82.182.860.649.177.744.850.157.8

ADRA performs significnatly outperforms baselines that requires no additional calibartion, and even outperform R-Loss by 5.1 points. We hope this clarifies concerns about its effectiveness on newer models.

AuthorsOpenReview ↗

Response to Reviewer to 2K9K (2/4): Verification on Hypothesis

If the actual text appeared during training, the model-generated content is more likely to match the true suffix. Hypothesis lacks verification through direct experimentation or theoretical analysis.

Empirically, we show this is true in three ways: (a) reconstruction with just sampling, without RL, (b) how RL changes member and non-member reconstruction, and (c) a controlled experiment that removes training data exposure.

Last but not least, we also give a simple theoretical analysis in our general response (4/4). Thank you for the suggestion!

Experiment (a) N-Sampling already reconstructs members better. Our N-Sampling baseline tests the hypothesis with no attack-time training: it samples continuations from the original target model and scores their overlap with the true suffix. It is comparable with other loss-based MIAs in all 14 settings (AUROC 54.3–90.4) and is the strongest baseline in 8 of them (Tables 2, 4, and 6).

Experiment (b) After RL, the member–non-member gap widens. Below we show the reconstruction performance of member and non-member data for our distillation experiment (S1.1). "Before RL" is N-Sampling from the original model, and "after RL" is ADRA. Members are already reconstructed better before RL, and the gap grows after training with ADRA. This assyemetry in true suffix reconstruction is also why MIA performance improves (AUROC 90.4 → 98.4).

Table R1 (condensed; full table in General Response 2/4). Distillation (S1.1); before RL is N-Sampling, after RL is ADRA.

Jaccard (%)LCS (words)Coverage (%)
Member gain+19.0+61+25.4
Non-member gain+3.0+6+0.6
Gap, before → after RL13.4 → 29.451 → 10610.6 → 35.4

The true-suffix win rate shows the same pattern (Table R2, General Response 2/4).

Experiment (c) Without training exposure, there is no MIA gain. We also show the reverse is true. If the model never saw the data, our method doesn't work. On Olympia Math, where members and non-members are a random split, we rerun the full attack from the Tulu2-7B checkpoint before contamination:

Target (Olympia Math, AUROC)N-SamplingADRAADRA+
Contaminated (members seen in training)59.567.872.4
Uncontaminated (same texts, never seen)53.553.152.8

In the uncontaminated case, every method falls to near chance (around 50%), and ADRA+'s +12.9 AUROC advantage over N-Sampling drops to −0.7. This shows that the signal has to come from the model having seen the data during training; attack-time RL on the candidate suffixes does not create it on its own. (We discuss this more in Section 6, "Disentangling Membership from Attack-Time Learning", and Appendix J.)

Finally, in the paper, we show qualitative examples for the pre-training, post-training, and distillation setups, highlighting the parts of the suffix that the model reconstructs (please see Appendix M, Reconstruction Examples).

AuthorsOpenReview ↗

Response to Reviewer to 2K9K (3/4): A Theoretical View on our Method

We provide a theoretical analysis in General Response (4/4). In brief, maximum likelihood training lowers the loss on training text, so members are more likely to be generated near their true suffix. On-policy RL reinforces a completion only when it is sampled, so members get more high-reward samples and improve faster, while unseen text is either much harder to generate from scratch.

We hope this clarifies our hypothesis, and will include this into our paper.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response (1/4): Summary

We thank each reviewer for their time and feedback! Reviewers found our evaluation extensive across pre-training, post-training, and distillation (2K9K, T14y, mMBP), with our method achieving "substantial AUROC improvements over existing baselines" (mMBP). On average, we improve 11.2 AUROC over the strongest loss-based baseline across 14 different training settings.

We summarize the main concerns and new results below, with full results in Parts 2–4. We will add all new results to the revision.

  • Novelty (T14y, mMBP). ADRA is the first RL-based active MIA. It trains a local copy of the target to elicit membership signal that passive MIAs miss, and it turns compute into accuracy, which passive MIAs cannot.
  • Hypothesis Verification (2K9K, mMBP). RL improves members far more than non-members (new, Tables R1–R2). On a target that never saw the members, the attack falls to chance (Table 24), and SFT or random-reward RL degrade AUROC (Section 6). We also add a theoretical analysis (Part 4).
  • Threat model (T14y). ADRA needs the same access as loss-based MIAs such as Loss and Min-K%: all assume open weights access. Closed APIs support neither kind of attack (Section 3).
  • Cost (2K9K, mMBP). We show the absolute compute cost is cheap. Under cheap RL budget, one RL step (< 1 USD) already beats every passive baseline; ten steps (< 7 USD) recover near-full performance (Table 15).
  • Additional Analsysis & Ablation (T14y, 2K9K, mMBP). Each component beats simpler alternatives (Section 6), and new ablations show the gains are robust to the reward metric and to the number and difficulty of distractors (Tables R4–R6).
  • Additional baseline: FSD (T14y, mMBP, AC). We show that FSD, which requires an additional set of labeled non-members, performs worse than ADRA in both settings we tested (Table R7).

New results summary

New resultFindingRaised by
Member vs. non-member reconstruction (Table R1)Gap more than doubles after RL (13.4 → 29.4 Jaccard)2K9K
True-suffix win rate (Table R2)Members 58.1% → 77.5%; non-members flat at 14–18%mMBP
Qwen3.5-9B on WildChat (Table R3)ADRA 82.8 vs. best baseline 77.7 AUROC2K9K
Theoretical analysis (Part 4)On-policy RL amplifies members' head start2K9K, T14y, mMBP
Single-metric rewards (Table R4)Each alone gains 10–12 AUROC over no training; Trio is best2K9K
Distractor count and difficulty (Tables R5–R6)Saturates at K = 7; hard distractors reach 86.7mMBP
FSD comparison (Table R7, reply to AC)Without labels, ADRA beats FSD with 256 labeled non-members (67.8 vs. 63.5 on Olympia Math; 82.8 vs. 45.0 on WildChat)T14y, mMBP
AuthorsOpenReview ↗

General Response (2/4): Additional Results

Additional Results #1: RL amplifies existing membership signal

ADRA never sees membership labels, so RL can raise AUROC only by improving members more than non-members in reconstruction. We measure this directly.

Table R1. Suffix reconstruction before and after RL (distillation, S1.1). Before RL is N-Sampling from the original model; after RL is ADRA. Member rows are from Table 7; non-member rows are new.

Jaccard (%)LCS (words)Coverage (%)
Member, before RL25.58222.3
Member, after RL44.514347.7
Member gain+19.0+61+25.4
Non-member, before RL12.13111.7
Non-member, after RL15.13712.3
Non-member gain+3.0+6+0.6
Gap (member − non-member)13.4 → 29.451 → 10610.6 → 35.4

Table R2. How often the true suffix wins the matching reward (% of rollouts; AIME, K = 7).

Before RLStep 25After RL
Member58.168.077.5
Non-member15.313.917.8

After our attack-time RL training, the reconstruction gap more than doubles (AUROC 90.4 → 98.4), and the member win rate rises 19.4 points while the non-member rate stays flat. Without training exposure there is no gain: on the pre-contamination Olympia Math target, every method falls to near chance (ADRA+ 72.4 → 52.8; see Appendix J).

Additional Results #2: Newer model family (Qwen3.5-9B)

Table R3. AUROC on Tulu3 Mix – WildChat (post-training detection). R-Loss uses Qwen3.5-9B-Base as the reference.

ModelADRA+ADRAN-SampLossR-LossZlibMin-K%Min-K%++
Qwen3.5-9B82.182.860.649.177.744.850.157.8

ADRA beats every baseline, including R-Loss, which requires a reference model, by 5.1 points.

AuthorsOpenReview ↗

General Response (3/4): Additional Ablations

Additional Ablation #1: Reward metric

Our proposed reward Trio averages three lexical metrics over words: token set similarity (which words of the true suffix appear), LCS (whether they appear in order), and n-gram coverage (whether exact phrases are reproduced).

Table R4. AUROC on AIME with each reward metric alone.

RewardAUROC
Spurious reward (random)58.1
No training (N-Sampling)73.1
LCS only83.1
Token set only83.9
N-gram coverage only85.1
Trio (average of all three)85.8

Each metric alone gains 10–12 points over no training, so the gains do not hinge on one metric. We average them because the average is harder to reward-hack and gives a denser learning signal.

Additional Ablation #2: Number and difficulty of distractors

Table R5. AUROC on AIME vs. number of random distractors K.

K0 (no matching)137 (default)1531
ADRA81.381.684.585.886.185.5

Table R6. AUROC on AIME with random vs. hard distractors (K = 7). Hard distractors are the 7 pool suffixes with the highest token similarity to the true suffix.

Random (default)Hard
85.886.7

Gains saturate around K = 7 (K = 3–31 within 1.6 points). Hard distractors help further, at the cost of ranking the pool for every candidate.

AuthorsOpenReview ↗

General Response (4/4): A Theoretical View of Our Method

Below we provide a theoretical analysis of why our hypothesis and method work. We will include this to our revised manuscript.

For a prefix cc and true suffix y⋆y^{\star}, the probability of generating y⋆y^{\star} exactly is

pθ(y⋆∣c)=exp⁡(−L(y⋆∣c;θ)).p_{\theta}(y^{\star} \mid c) = \exp\big(-\mathcal{L}(y^{\star} \mid c; \theta)\big).

Maximum-likelihood training lowers L\mathcal{L} on training data, so trained-on text becomes more likely to be generated; this is the same loss gap that loss-based MIAs exploit. Prior work confirms this at scale. Exact extraction grows with model size, duplication, and prompt length [1], and its probability can be measured by sampling [2]. Our rewards relax exact match to partial overlap (token set, LCS, n-gram coverage), so partial recovery still receives credit.

For RL, the idealized optimum of our KL-regularized objective is [3, 4]

π⋆(y∣c)∝pθ0(y∣c)exp⁡(r(y,y⋆)/β),\pi^{\star}(y \mid c) \propto p_{\theta_{0}}(y \mid c) \exp\big(r(y, y^{\star}) / \beta\big),

where rr is our reward and β\beta is the KL coefficient. This reweights the target model's own distribution, so membership enters only through pθ0p_{\theta_{0}}. Within our finite on-policy budget, a completion is reinforced only when it is sampled, so its update scales with its probability [5].

Members start with more probability near y⋆y^{\star} (Table R1, before RL), so they get more high-reward samples and improve faster (Tables R1–R2), consistent with findings that RL sharpens existing behaviors [6]. Without training exposure, RL creates no reliable gap (please see our experiments in Appendix J "Disentangling Membership From Attack-time Learning"). SFT, in contrast, trains on every candidate's true suffix, so it also teaches the non-member suffixes and erases the gap (more ablations in Section 6).

References:

[1] Carlini et al. Quantifying Memorization Across Neural Language Models. ICLR 2023.

[2] Hayes et al. Measuring Memorization in Language Models via Probabilistic Extraction. NAACL 2025.

[3] Korbak et al. RL with KL Penalties is Better Viewed as Bayesian Inference. Findings of EMNLP 2022.

[4] Rafailov et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023.

[5] Mei et al. On the Global Convergence Rates of Softmax Policy Gradient Methods. ICML 2020.

[6] Yue et al. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? NeurIPS 2025 Oral.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.