- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Active Data Reconstruction Attack (ADRA), a reinforcement-learning-based approach to membership inference attacks (MIAs) for LLMs. ADRA fine-tunes a policy initialized from the target model using on-policy reinforcement learning with reconstruction-oriented rewards. The authors introduce lexical reconstruction metrics and a contrastive matching reward to encourage the model to reconstruct candidate text. An adaptive variant, ADRA+, further incorporates membership priors derived from conventional loss-based MIA scores. The paper evaluates the proposed methods across pre-training, post-training, and distillation settings. The results show that ADRA and ADRA+ generally outperform the evaluated passive baselines in terms of AUROC, with particularly strong performance on BookMIA, Dolma3 arXiv, AIME, and paired distillation-trace discrimination. Additional experiments investigate reconstruction quality, optimization controls, computational costs, and the dependence of the attack on prior training exposure.
Critical strengths
- This paper leverages the ability of RL to reinforce behaviors already encoded in model parameters, progressively amplifying the differences in reconstructibility between data seen and unseen during pre-training, thereby facilitating more effective membership inference. The core idea is simple and straightforward, and the proposed method is relatively easy to implement.
- The authors evaluate their approach across three different training stages (pre-training, post-training, and distillation) and several model families. The proposed methods achieve substantial AUROC improvements over existing baselines in several settings, especially on BookMIA, Dolma3 arXiv, and controlled AIME contamination.
Critical weaknesses
- Compared with training-free MIA methods, the RL process introduces additional computational and time overhead and requires an additional training data construction process. Consequently, the overall cost of ADRA is higher than that of conventional loss-based MIA methods.
- ADRA+ incorporates scores obtained from conventional loss-based MIA methods as prior information, effectively constructing a form of pseudo-labeling signal. From this perspective, ADRA+ can be viewed as a mechanism for amplifying existing membership signals provided by conventional loss-based MIAs. Therefore, it may be more appropriate to regard ADRA+ as a supplementary enhancement rather than a standalone methodological contribution, and its contribution appears less significant than that of ADRA.
Questions for authors
- Why does contrastive matching improve membership separation? Could the authors report how frequently the true suffix achieves the maximum similarity for member and non-member examples throughout training? Additionally, how sensitive is the method to the number and difficulty of distractors?
- Given that FSD has already demonstrated the effectiveness of target-model fine-tuning for MIA, could the authors further clarify the specific methodological novelty of ADRA beyond replacing fine-tuning-based score deviation with reconstruction-oriented RL? In particular, what evidence establishes the necessity and unique benefit of on-policy RL for extracting membership signals?
7 comments
Response to Reviewer mMBP (2/3): Method Details
ADRA and ADRA+
ADRA+ amplifies the membership signal of existing loss-based MIAs, so it may be better viewed as a supplementary enhancement than a standalone contribution, and is less significant than ADRA.
Agree that ADRA+ is a supplementary improvement over ADRA, that's why we termed it "ADRA+"! We think it's natural to combine our method with loss-based MIAs as they are fairly orthogonal, and report the ADRA+ results seperately.
ADRA+ improves over ADRA in 10 of the 12 settings where it applies. ADRA alone, without any loss-based prior, already outperforms the strongest baseline in 13 of 14 settings (Tables 2, 4, and 6). We will make this clearer in the final version.
Contrastive Matching & Similarity
Why does contrastive matching improve membership separation? Could the authors report how frequently the true suffix achieves the maximum similarity for member and non-member examples throughout training?
Contrastive matching widens the gap between member and non-member reconstruction, and the bigger this gap, the better the MIA performance.
Since our attack does not know which candidates are members, without distractors (K = 0) every candidate, member or non-member, is pushed toward its own true suffix. With distractors, a candidate is no longer forced to match its true suffix, Member rollouts still can match their true suffix, since the model has seen it, while non-member rollouts mostly match distractors instead. This contrastive RL therefore improves member reconstruction much more than non-member reconstruction.
We show this empirically. Removing distractors drops AUROC on AIME from 85.8 to 81.3 (no training: 73.1; Section 6 and Figure 2). We also report how often the true suffix achieves the maximum similarity:
Table R2. How often the true suffix wins the matching reward (% of rollouts; AIME, K = 7).
| Before RL | Step 25 | After RL | |
|---|---|---|---|
| Member | 58.1% | 68.0% | 77.5% |
| Non-member | 15.3% | 13.9% | 17.8% |
The member match rate grows from 58.1% to 77.5% during training, while the non-member rate stays roughly flat, so the gap widens from 42.8 to 59.7 points. We hope this clarifies the concern!
Additionally, how sensitive is the method to the number and difficulty of distractors?
Great point!
Below, we show AIME AUROC as we vary the number of distractors K (each rollout is scored against K + 1 suffixes):
Table R5. AUROC on AIME vs. number of random distractors K.
| K Distractors | 0 (no matching) | 1 | 3 | 7 (default) | 15 | 31 |
|---|---|---|---|---|---|---|
| ADRA | 81.3 | 81.6 | 84.5 | 85.8 | 86.1 | 85.5 |
We see that after the benefits of adding more distractors K saturates at around K=7.
We also conduct additional experiment where instead of randomly drawing from the pool, for each candidate we select distractors based on their token simialrity with the true suffix.
Table R6. AUROC on AIME with random vs. hard distractors (K = 7).
| Distractors | Random (default) | Hard (Top K=7 similar suffixes in the pool) |
|---|---|---|
| ADRA | 85.8 | 86.7 |
We see that increasing the difficulty actually further improves MIA performance, albeit at the cost of ranking the pool by similarity for every candidate.
Response to Reviewer mMBP (3/3): RL vs. SFT
Given that FSD has already demonstrated the effectiveness of target-model fine-tuning for MIA, could the authors further clarify the specific methodological novelty of ADRA beyond replacing fine-tuning-based score deviation with reconstruction-oriented RL?
Thank you for raising FSD (we discuss our disction in Appendix K). Replacing FSD's fine-tuning with reconstruction-oriented RL changes two things, each requiring additional method components or assumptions:
- No labeled data. FSD fine-tunes on a labeled set of known non-members from the same domain. ADRA trains only on the unlabeled candidate pool it scores, so our method need no membership labels.
- A different membership signal. FSD measures how a likelihood score shifts after fine-tuning. ADRA measures how well the model reconstructs each candidate's true suffix from its own generations.
Compared to SFT which forces models to directly learn both member and non-member suffixes, On-policy RL has the unique advantage of reinforcing what the target already generates. And the contrastive reward credits the true suffix only when it beats the distractors. We show this empirically to be true in response (2/3).
In sum, FSD uses labeled non-members to calibrate likelihood scores, while ADRA elicits reconstruction of the candidates themselves without labels. To our knowledge, it is the first RL-based active MIA.
In particular, what evidence establishes the necessity and unique benefit of on-policy RL for extracting membership signals?
Our optimization controls on AIME test this directly (Section 6, Figure 2). All runs start from the same target model and the same unlabeled candidate pool (10 SFT epochs vs. 60 RL steps):
| Attack-time training | AUROC |
|---|---|
| None (N-Sampling) | 73.1 |
| SFT on the candidates (3 learning rates, final) | 59.8–67.1 |
| RL, random reward | 58.1 |
| RL, our reward | 81.3 |
| RL, our reward + matching (ADRA) | 85.8 |
| RL, our reward + matching + prior (ADRA+) | 85.5 |
SFT performs below no training (N-Sampling) at all three learning rates. RL with random rewards also degrades. Only RL with reconstruction rewards improves detection, so the benefit requires both on-policy RL and reconstruction-aligned rewards.
We provide a justification in theoretical analysis (please see General Response 4/4): SFT trains on every candidate's true suffix, members and non-members alike, so it teaches the model the non-member suffixes too and erases the gap. On-policy RL only reinforces what the model already generates, so it amplifies the head start that members already have.
Response to Reviewer mMBP (1/3): Cost
Thank you for the detailed review and close-read of our paper! We are glad to see you find our method to be "simple and straightforward", our paper to conduct comprehensive experiments across trainings tages, and show "proposed methods achieve substantial AUROC improvements over existing baselines in several settings".
We respond to your weaknesses/questions inline below, please also feel free to see our general response about additional results, ablation, and analysis.
Agree that our method is more expensive than training-free MIAs, we discuss this more in our limitation. We think the method compensates the training cost by trading the RL compute with more MIA accuracy and fewer False Positives (best TPR@1%FPR in 12 of 14 settings; Tables 18, 20, and 21). In high-stakes scenario (like copyrighted data, private medical/presonal data audit), MIA accuracy and low FPR is more important than cost-effectiveness, as noted in [1].
For large-scale screening of data, a cheap passive MIA can serve as a first-pass filter, with ADRA used for the later selected candidates.
The cost is also quite small in absolute terms. Under strict RL budgets (Section 6 and Appendix D.2), a <$1 run already beats all passive baselines in AUROC, and a <$7 run recovers near-full performance on both BookMIA and Dolma3 arXiv:
*Costs are for one full attack run over 512 candidates on 8×H200, prorated from measured 30-step runtimes.
References:
[1] Carlini, Nicholas, et al. "Membership inference attacks from first principles." 2022 IEEE symposium on security and privacy (SP). IEEE, 2022.