acceptodds
Under review as a conference paper at ICLR 2027

Membership Inference Attacks on Language Models with Multiple Alignment Methods

Abstract

Large language models (LLMs) are fine-tuned by different methods on different data during their alignment for knowledge acquisition or preference optimization. The data used in this stage may leak sensitive information. In this work, we propose membership inference attacks for multiple and multi-stage LLM alignment methods. We introduce a extensible set of signals covering different aspects of the alignment pipeline to map the sample data to a multi-dimensional vector. Depending on the availability of shadow models, we propose shadow-free **Envelope (ENV)** and shadow-based **shadow-calibrated likelihood ratio (SLR)** to construct the final composite score for membership inference. We evaluate the framework on five model families (GPT-2, Llama-3.2-1B, Qwen2.5-1.5B-Instruct, Qwen2.5-3B-Instruct, and Llama-3.1-8B-Instruct) under two reward granularities(a continuous learned reward and a binary verifier) and multiple alignment pipelines. The extensive results indicate our proposed methods can achieve stronger average AUC than existing methods across all configurations.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Membership inference attack, large language model, fine tuning, alignment training, MIA for multiple alignment methods

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5zaDOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the membership inference problem for aligned language models. The goal is to determine whether a candidate sample participated in training when the specific alignment method is unknown and the model may have undergone multi-stage training. The authors integrate prompt statistics, preference response likelihoods, token probability changes, and reward distributions into a 19-dimensional detector vector, hoping to capture the complementary signals left by different training objectives. The method includes two types of scoring: ENV, which requires no aligned shadow model, sorts the detector vector and calibrates it against rank statistics in a reference population, taking the largest degree of abnormality; and SLR, which relies on shadow models, trains one shadow model for each of SFT, DPO, NPO, GRPO, and PPO, uses fingerprint distances on a reference prompt set to blend member and non-member distributions, and then computes likelihood ratios using QDA, naive Bayes, or an AUC-gated weighted version. The experiments cover five model families, use IMDB, HH-RLHF, and GSM8K, and examine continuous rewards, binary verifiable rewards, and four training chains. The authors report that on Llama-3.2-1B, SLR-NB-W achieves good average AUC results under both reward types, and that detectable signals are also preserved in multi-stage experiments.

Critical strengths
  1. The research problem has practical value. The paper not only examines the situation under a single training objective, but also incorporates unknown alignment methods and multi-stage training into the membership inference problem. By integrating behavioral signals and likelihood signals from different sources into a unified vector, it enables a clearer comparison of how different alignment methods each leak information.

  2. The method provides choices under different resource conditions. ENV requires no training of aligned shadow models, while SLR uses five types of shadow distributions for calibration; QDA, NB, and NB-W clearly reflect the trade-offs among covariance modeling, independence assumptions, and signal selection. The sorted rank calibration is also a design worth verifying.

Critical weaknesses
  1. The contribution of this paper is not clearly stated. The Gaussian likelihood ratio, QDA, and NB of SLR all have mature statistical foundations. The existing results are insufficient to show whether the improvement mainly comes from the new signals, the five shadows, the fingerprint, or the scorer. If the features and shadow budget could be fixed, and LR/RF, NB, and QDA compared, with the fingerprint, rank calibration, and dimension weighting removed separately, the experiments would be more convincing.

  2. The multi-stage experiments do not sufficiently demonstrate the preservation or overwriting of membership signals across stages. Table 3 does not clearly specify whether the same training data are used across stages. It also does not distinguish samples used only in SFT, only in later stages, in multiple stages, or in none of the stages. Without such distinctions, it is difficult to determine which stage contributes to the final membership inference results.

Questions for authors
  1. Could the authors clarify whether the improvement comes from the new signals, shadows, fingerprint, or scorer?

  2. Would it be possible to compare LR/RF, NB, and QDA under fixed features and a fixed shadow budget, and to provide corresponding ablation results?

  3. The multi-stage experiments do not state whether the data in each stage are the same, nor do they distinguish different stage member types, making it difficult for readers to determine which stage the detection results come from. Please clarify the data setup and the definition of member types at each stage.

4 comments

Answer to Weakness 1.

We thank the reviewer and clarify the contributions as follows. Our pipeline starts with 19-dim signal extraction and branches into ENV (shadow-free rank calibration) for the no-shadow budget setting and SLR (fingerprint-based multi-shadow calibration). After SLR, the scoring variants are DIBA (classifier trained on shadow data) and the shadow-statistics scorers QDA, NB, NB-W, top-K. These designs are not parallel, they are layered, as shown in the hierarchy added in Figure 3 in Section 5.3 of the revised version.

Specifically, rank calibration is used only for ENV, i.e., the shadow-free multi-signal budget. Multi-shadow calibration is the combined name for the five shadows, fingerprint, and dimension weighting. Only with multiple shadows do we obtain different shadow fingerprints and then use softmax weighting for multi-shadow calibration. After multi-shadow calibration, the scoring variants can be either our proposed LLR-based statistics family or DIBA's LR/RF scorer.

Our contributions can be grouped into three parts: new signals, multiple shadow calibration, and LLR-based scoring variants. To further demonstrate the effect of each of our technique components, we explicitly list the performance of different combinations of our proposed techniques as an ablation study in Table 4, which we have added in the revised version Section 5.3, so that the individual contribution of each technique is clear.

Another advantage of our scorers is that LR/RF trains a classifier on shadow data, whereas our LLR-based scorers require no training: the blended Gaussians are fit in closed form from the selected shadow's statistics, so the scorer adds no optimizer, no iteration, and no failure mode of its own.

Answer to Weakness 2.

Our multi-stage setup is stated in the Multi-Stage Alignment section: every stage of every curriculum trains on the same 1000 prompts. To additionally separate which stage contributes the detectable trace, we add a per-stage trace analysis on a more realistic multi-stage chain.

The new data split as follows. Configs: SFT = 1500 steps on all prompts; GRPO = 1500 steps on F∪R, the reference model is sft; NPO = 1500 steps on F∪R, the reference model is sft; NPO = 1500 steps on F, the reference model is grpo.

GroupSizeStages
S1000SFT only
R1200SFT + GRPO
F300SFT + GRPO + NPO

Table 5. AUCs of SLR-NB on different data splits (S, R, F).

ChainFRS
SFT→GRPO.977.976.940
SFT→DPO.851.870.741
SFT→GRPO→NPO.993.6300.601

Analysis:

  1. A second fine tuning stage strengthens detection: groups that saw stage 2 stay above SFT-only.
  2. Later fine-tuning erases part of the earlier trace: groups not seen in the later fine-tuning stage have lower AUC in the final model.
  3. NPO rewrites rather than erases the trace: the canonical NPO loss changes the rejected-response log-ratios, and our signal set contains dedicated rejected-response detectors (D2: LR-Asym and D3: |LR-Rej|), so the tail leaves a new, detectable trace on F.

Answer to Question 1.

Please refer to Answer to Weakness 1.

Answer to Question 2.

Please refer to Answer to Weakness 1.

Answer to Question 3.

Please refer to Answer to Weakness 2.

The authors have supplemented Table 4 and Table 5; however, they still have not fully addressed my concerns. For example, regarding specific quantitative attribution, the ablation experiments lack comparative results for LR/RF, NB, and QDA under fixed features and a fixed shadow budget, and the definition of member type at each stage in Question 3 remains insufficiently clear. Some issues have not been effectively addressed, so I maintain the original score.

Response to Comment 1

We thank the reviewer for this comment. We believe there may be a misunderstanding about what Table 4 already reports. The comparison the reviewer requests, namely LR/RF, NB and QDA under fixed features and a fixed shadow budget, is exactly the design of the component ablation table. This holds for both the one shadow block and the five shadow block. In both blocks the target data are the same batch of target data, so the features are identical across scorers. The shadows are trained on the same batch of shadow data, so the shadow budget is also fixed. The blocks report all five scorers side by side, including LR/RF, NB and QDA. The requested fixed feature and fixed shadow comparison is therefore already present in the revised paper.

We are not fully certain what additional comparison the reviewer has in mind. If the reviewer means a table restricted to a different fixed feature set, or a different shadow budget, we would be grateful for a more precise pointer. Please specify which features, which budget and which table or column, so that we can address it directly.

Response to Comment 2

We thank the reviewer for pressing us to define member type operationally. In plain language, a prompt is a member of a stage if and only if that stage’s training data contained that prompt. The three prompt groups are F (the forget set, 300 prompts), R (the RL set, 1200 prompts) and S (the SFT only set, 1000 prompts). These three groups are mutually exclusive and together cover all 2500 prompts. SFT is the canonical sft@1500 over all 2500 prompts, so every prompt is an SFT member. GRPO trains on F ∪ R, which is 1500 prompts, so F and R are GRPO members and S is not. NPO trains on F only, which is 300 prompts, so only F is an NPO member.

Therefore the member type of each prompt is as follows. F is a member of SFT, GRPO and NPO. R is a member of SFT and GRPO. S is a member of SFT only. The three groups are disjoint, so no prompt belongs to more than one of F, R and S. This is what we mean by stage specific membership.

To explain the purpose of each stage, SFT is the warm up that all 2500 prompts go through, because RL run directly on the base model is unstable and produces unreadable output, so the standard pipeline finetunes first and then runs RL from the SFT checkpoint, and in our chain SFT also plays the role of making a prompt a member at all since unlearning presupposes that something was learned.

GRPO is the RL alignment stage, where the model is updated against a reward, which is a reward model in the HH RLHF regime and a rule verifier in the RLVR regime, and its purpose is to align the SFT checkpoint with that reward.

NPO is the unlearning stage, where naive gradient ascent on the forget data collapses the model, so NPO instead unlearns the forget set while staying anchored to the reference model, and its purpose is to remove the forget set without destroying the model.

The chain matters because real models are trained in stages and a later deletion request hits a model that has already been aligned, so membership is stage specific, and the exposure nesting lets us attribute the trace stage by stage.

The member type definitions, the purpose of each stage and the per stage audits are now stated explicitly in the revised paper and quantified by the experiment above. We would be grateful if the reviewer could let us know whether any further concern remains. If so, please state it explicitly so that we can address it directly.

Thank you for the additional clarifications. I think most of my concerns have been addressed, so I’m willing to raise my score to 3.

Reviewer_P9BQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies membership inference over LLM alignment data across multiple different alignment strategies and multi-stage settings. Due to the target alignment method being possibly unknown, their proposed method combines signals that cover different aspects of alignment strategies and merge them into a 19-dimensional vector. To convert the vector into a score, they propose both a shadow-free version, Envelope, that calibrates with a held-out data population, and a shadow-based method that creates near-identical shadows using different alignment strategies for likelihood ratio comparisons. Their combinatorial experiments over five base models and five alignment strategies as well as multi-stage aligned models show that their SLR method generally achieves the highest AUC, while their ENV beats out shadow-free baselines.

Critical strengths
  • The problem of alignment-method-agnostic membership inference is well-motivated since many models may not disclose specific technical details related to which alignment methods are used. Providing both a shadow-free and shadow-based version with notable empirical gains also supports auditing under different attacker-access assumptions, making their method more generalizable.
  • The multi-stage analysis provides a useful first step for studying MI on models with chained training strategies that are closer in construction to real-world models.
  • The evaluation is extensive, with coverage over various alignment strategies, reward granularities, model families, and multi-stage pipelines. Additional analysis isolating the MI success of the individual signals helps us understand the contributions of different signals.
Critical weaknesses
  • From my read of the DIBA paper [1], it seems that the existing baseline is implemented differently. While the current work uses the following signals: a_max, a_mean, a_std, b_KL, and b_token, whereas the DIBA work uses the tuned and base model rewards, k3 KL divergence, and likelihood ratio (and a zlib normalized version). Importantly, a key signal is the advantage between the finetuned and base rewards, which seems to be the exact definition of D19 (which also seems to be critical to a lot of the success for detection in the reward-based alignment strategies). Unless the authors are referring to a different DIBA, I would be curious how the results change when this is corrected.
  • The method contribution in general seems quite limited. For example, it’s not clear why each of the 19 signals is necessary; in fact, the appendix single signal analysis seems to indicate that most success can be attributed to a few signals (e.g., D3, D5, D19). Secondly, the choice of merging (e.g., SLR), doesn’t appear to have a notable advantage. While Table 2 seems to indicate the SLR methods are substantially better than the S-DIBA baseline, Table 14 demonstrates that, when DIBA is provided with both the same 5 shadow models and 19 signals, the observed performance is about equivalent to SLR. The extent of the contribution seems to be recognizing the need for different MI signals for different alignment methods. More RL-MIA baseline methods such as the Self-Critique method from [2] would be valuable to include since previous work has also already established that token-level metrics may not translate to MI over reward-based alignment methods.

[1] Yule Liu, Heyi Zhang, Jinyi Zheng, Zhen Sun, Zifan Peng, Jiaheng Wei, Tianshuo Cong, Yilong Yang, and Xinlei He. Auditing data membership in reinforcement learning with verifiable rewards. arXiv preprint arXiv:2511.14045, 2025. [2] Yongding Tao, Tian Wang, Yihong Dong, Huanyu Liu, Kechi Zhang, Xiaolong Hu, and Ge Li. Detecting Data Contamination from Reinforcement Learning Post-Training for Large Language Models. arXiv preprint arXiv:2510.09259, 2025.

Questions for authors
  • Given that some signals seem dominant (e.g., D19 from the ablation results), is there a way to run some sort of leave-one-out analysis for signal families to determine which signals are actually important in aggregate or if the signal vectors can be greatly reduced in dimension? Extending the single-signal analysis from the appendix may yeild interesting insights on what signals work best depending on the type of alignment method the model one is auditing has undergone.
  • In addition to reporting AUC, can the authors also share TPR@lowFPR?
  • Can more details be provided for the multistage analysis to help understand what’s happening in the results? For example, what is the member set being audited? Does each stage use the same training prompt dataset?
  • How close do the shadows need to be to the audited model’s training pipeline? For example, how robust are the methods to mismatches in training data, scale, training hyperparameters, etc. (knowledge an auditor may not know when constructing their own shadows).
7 comments

Answer to Weakness 1.

We thank the reviewer for pointing this out. One factual correction: original DIBA has no advantage axis. Its five features are FT Score, Base Score, sDiv (k3), sLikelihood, and sZlib, none of which is an advantage signal. D19 is not an original DIBA feature; it is our composite, D19 = FT Score − Base Score, i.e., the target policy's mean reward minus the base policy's mean reward on the same prompt.

To settle the comparison definitively, we now replace our S-DIBA implementation with the faithful original DIBA implementation and compute it on every panel. We have updated all affected numerical results in the revised version. The updated results show that DIBA improves on the PPO and GRPO panels, but even with this improvement, it remains comparable to or lower than ours on both panels. Moreover, this does not affect our claim: DIBA performs equally poorly on the other panels, indicating that it is only suitable for reward-model-based alignment, and its average performance therefore remains far below ours.

Answer to Weakness 2.

We thank the reviewer and clarify our contributions in Section 5.3 of the revised version. Our pipeline starts with 19-dim signal extraction and branches into ENV (shadow-free rank calibration) for the no-shadow budget setting and SLR (fingerprint-based multi-shadow calibration). After SLR, the scoring variants are DIBA (a classifier trained on shadow data) and the shadow-statistics scorers QDA, NB, NB-W, and top-K. The technique hierarchy is shown in Figure 3 of the revised version in Section 5.3.

(1) The 19-dim signal vector is intended to comprehensively study different statistics, but it is not our only contribution. Most important is fingerprint-based multi-shadow calibration for the setting where the alignment method is unknown and multi-stage. Its advantage is reflected by the average performance across the five alignment trainings. To further demonstrate the individual role of each signal, we conduct a leave-one-out analysis over all 19 signals. The results are as follows:

Table 2. Leave-one-out analysis. removes each of the 19 signals individually, the table shows the three largest single-axis drops:

PanelNB (19 axes)largest single-axis drop2nd3rd
l1b RM SFT.970D5:Token10pct −.007D4:TokenMean −.006D12:TokenLR-Mean −.006
l1b RM DPO.869D2:LR-Asym −.048D22:r_lift −.005D10:Reward-Adv −.003
l1b RM NPO.846D2:LR-Asym −.021D3:LR-Rej −.021D12:TokenLR-Mean −.015
l1b RM GRPO.721D9:RewardStd −.015D16:RewardGap-Mean −.012D11:Reward-Range −.007
l1b RM PPO.671D22:r_lift −.051D11:Reward-Range −.016D9:RewardStd −.014
l1b RLVR SFT.923D14:TokenLR-Skew −.014D5:Token10pct −.012D12:TokenLR-Mean −.008
l1b RLVR DPO.924D2:LR-Asym −.076D3:LR-Rej −.013D16:RewardGap-Mean −.001
l1b RLVR NPO.972D2:LR-Asym −.006D5:Token10pct −.006D13:TokenLR-Std −.005
l1b RLVR GRPO.594D22:r_lift −.121D4:TokenMean −.008D13:TokenLR-Std −.005
l1b RLVR PPO.608D22:r_lift −.024D5:Token10pct −.010D14:TokenLR-Skew −.009

Analysis: Although a few signals are indeed the most critical, the three most relied-upon signals vary across different alignment methods.

(2) As for Table 14, the observed performance of DIBA is roughly equivalent to SLR. This is because S-DIBA in Table 14 differs substantially from original DIBA: it uses our 19-dim signals, fingerprint, and multi-shadow calibration, and only keeps DIBA's scorer.

To further demonstrate the effect of each of our technique components, we explicitly list the performance of different combinations of our proposed techniques as an ablation study in Table 4, which we have added in the revised version Section 5.3, so that the individual contribution of each technique is clear. Moreover, DIBA trains a classifier on shadow data, whereas our scorers require no training: the blended Gaussians are fit in closed form from the selected shadow's statistics, so the scorer adds no optimizer, no iteration, and no failure mode of its own.

(3) Self-Critique: We have added the Self-Critique method as a new baseline in the revised version. It performs well on some reward-model-based alignment settings, but like DIBA, even on these settings its performance is lower than ours, while on the other panels and on average it is far below our method.

Answer to Question 1.

Please refer to Answer to Weakness 2.

Answer to Question 2.

The TPR at low FPR results are now reported in Tables 15 and 16 of the appendix in the revised version. Because recomputing TPR at low FPR requires rerunning all experimental cells, we have first completed the results on Llama-3.2-1B, the remaining results will be updated in the revised version once completed.

Answer to Question 3.

The setup is stated in the Multi-Stage Alignment section: every stage of every curriculum trains on the same 1000 prompts. To additionally separate which stage contributes the detectable trace, we add a per-stage trace analysis on a realistic multi-stage chain.

The new data split as follows. Configs: SFT = 1500 steps on all prompts; GRPO = 1500 steps on F∪R, the reference model is sft; NPO = 1500 steps on F∪R, the reference model is sft; NPO = 1500 steps on F, the reference model is grpo:

GroupSizeStages
S1000SFT only
R1200SFT + GRPO
F300SFT + GRPO + NPO

Table 3. AUCs of SLR-NB on diferent data splits (S, R, F):

ChainFRS
SFT→GRPO.977.976.940
SFT→DPO.851.870.741
SFT→GRPO→NPO.993.630.601

Analysis:

  1. A second fine tuning stage strengthens detection: groups that saw stage 2 stay above SFT-only.
  2. Later fine-tuning erases part of the earlier trace: groups not seen in the later fine-tuning stage have lower AUC in the final model.
  3. NPO rewrites rather than erases the trace: the canonical NPO loss changes the rejected-response log-ratios, and our signal set contains dedicated rejected-response detectors (D2: LR-Asym and D3: |LR-Rej|), so the tail leaves a new, detectable trace on F.

Answer to Question 4.

To test how robust the methods are to mismatches, we train shadows under mismatched data sizes (100/500/2000 prompts vs the canonical 1000), mismatched learning rates (×0.25, ×0.1), mismatched step budgets (2500/3000 steps), and mismatched member identities, auditing each of the five single-alignment llama1b targets and reporting the mean AUC over the five targets with different alignment training:

Table 4. Shadow mismatch robustness:

Shadows used for calibrationNB AUC (mean over 5 targets)
canonical (1000 prompts, standard lr, standard steps).824
Mismatched training data
100-prompt shadows.849
500-prompt shadows.818
2000-prompt shadows.829
member sets from a different prompt subset.833
Mismatched learning rate
lr × 0.25.814
lr × 0.1.805
Mismatched step budget
2500-step shadows.828
3000-step shadows.824

Analysis: Data-size, step, and member-identity mismatches are largely absorbed, and 100-prompt shadows can even improve performance: the fingerprint reweights the ensemble toward the usable shadows, and the closest-shadow weight remains on the true method. Learning-rate mismatches cost relatively little when some canonical shadows remain for the fingerprint to lock onto. When every shadow is trained at a wrong learning rate (as shown in the Table above), however, no canonical shadow remains, and the loss concentrates on the DPO and GRPO panels (DPO .941 → .868; GRPO .685 → .633), whose preference and reward statistics depend most on how closely the shadow tracks the target's optimization dynamics.

Area_Chair_XXZXOpenReview ↗

Reviewer interaction required

Dear reviewer, The authors have replied to your comments and the discussion is close to the end. We encourage you to participate in the discussion to see whether your questions and concerns have been answered by the authors. Thank you! Best, Your AC

Reviewer_WJ2DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the problem of membership inference for language-model fine-tuning when the alignment objective is unknown or training involves multiple stages. The authors combine likelihood, preference, and rollout-reward features in two proposed attacks where one calibrates sorted feature percentiles against known nonmembers, while the other combines Gaussian member/nonmember models from alignment-specific shadows. In experiments, the shadow based method was shown to have competitive performance under the different alignment objectives.

Critical strengths
  1. The paper addresses a relevant problem: membership inference when the alignment objective is unknown or training involves multiple stages.

  2. The authors conducted evaluation covering several models and alignment objectives, with useful feature and classifier ablations.

Critical weaknesses
  1. The attack requires access to the reward function used during policy training, but this requirement is not stated in the threat-model description. Such access may be unavailable to an external attacker, particularly when the provider uses a private learned reward model. The paper does not evaluate a substitute scorer or a variant without reward features, so the practical effectiveness under unknown alignment remains unclear.

  2. The experiments show that the expanded feature set provides useful signals for membership inference across different alignment methods. However, the additional benefit of the proposed scoring rules is less clear. It is shown that S-DIBA achieves comparable performance when given the same features in table 14. The relative performance of the proposed scoring variants also varies across settings, with limited explanation of why these differences arise or how to choose among them.

Questions for authors
  1. The required access to the training reward function needs to be stated explicitly. A comparison using a substitute scorer or excluding reward features would help establish how much the reported effectiveness depends on this access.

  2. PREMIA's implementation needs clarification, the score described in equation 8 appears to be different from original implementation.

  3. The paper should explain which shadows supply the feature AUCs used by NB-W and how these AUCs are combined for each target model. This should clarify how the weights are obtained without knowing the target's training method or using target evaluation membership labels.

4 comments

Answer to Weakness 1.

We have made this assumption explicit in the revised Section 1 Threat Model: “The attacker can query the target policy and has access to the reward function used during policy training.”

More importantly, to show that this assumption is not strong, we now replace the training reward (Skywork-RM-8B) with two substitute rewards that an auditor can obtain without any knowledge of the training reward: (i) Skywork-Reward-Gemma-2-27B, a publicly available reward model which is different from the real one; and (ii) a small 1B reward model (Llama-3.2-1B with a scalar reward head) that the auditor fine-tunes for 2 epochs on the public HH-RLHF preference dataset, which is held out from the audited models’ training data. The new results on Llama-1B are as follows:

Table 1. Ablation study of the reward function on Llama-1B (HH-RLHF dataset):

MethodScorerNBQDANB-W
SFTtraining (Skywork-8B).970.973.974
SFTpublic (Gemma-2-27B).933.913.924
SFTself-trained (1B).859.873.890
DPOtraining (Skywork-8B).869.861.865
DPOpublic (Gemma-2-27B).861.874.865
DPOself-trained (1B).877.865.864
NPOtraining (Skywork-8B).846.821.848
NPOpublic (Gemma-2-27B).782.780.820
NPOself-trained (1B).847.799.804
GRPOtraining (Skywork-8B).721.515.750
GRPOpublic (Gemma-2-27B).729.556.731
GRPOself-trained (1B).735.616.735
PPOtraining (Skywork-8B).671.520.646
PPOpublic (Gemma-2-27B).606.534.572
PPOself-trained (1B).657.578.581

Analysis: Across the five alignment methods and three scorers, replacing the training reward with the public Gemma-2-27B or the self-trained 1B reduces MIA performance by only an acceptable margin (On average, Gemma-2-27B and the self-trained model are 2.47 and 1.80 percentage points lower than the real training reward function, respectively). The attacker therefore does not need access to the real training reward function.

Answer to Weakness 2.

We thank the reviewer and clarify the contributions, the S-DIBA (Table 14) comparison, and scorer choice as follows.

Our pipeline starts with 19-dim signal extraction and branches into ENV (shadow-free rank calibration) for the no-shadow budget setting and SLR (fingerprint-based multi-shadow calibration). After SLR, the scoring variants are DIBA (classifier trained on shadow data) and the shadow-statistics scorers QDA, NB, NB-W, top-K. The technique hierarchy is shown in Figure 3 of the revised version in Section 5.3.

S-DIBA in Table 14 varies the original DIBA a lot, it uses our 19-dim signals, fingerprint, and multi-shadow calibration, and only keeps DIBA's scorer. We explicitly list the performance of different combinations of our proposed techniques as the ablation study in Table 4, which we have added in the revised version, so that the individual contribution of each technique is clear.

How to choose among scoring variants: DIBA trains a classifier on shadow data, whereas our scorers require no training: the blended Gaussians are fit in closed form from the selected shadow's statistics, so our proposed scorers add no optimizer, no iteration, and no failure mode of its own. All our scorers are the same LLR test over those shadow-fitted Gaussians, differing only in how they trust the shadow calibration. we prefer NB as the default, QDA under clean calibration with real covariance structure, NB-W when calibration is trustworthy and per-dimension reliability differs, and top-K when only the shadow-AUC ranking is trustworthy.

Answer to Question 1.

Please refer to Answer to Weakness 1.

Answer to Question 2.

We thank the reviewer for pointing this out. We have corrected Eq. (8) to match the original PREMIA implementation and updated all affected numerical results in the revised version. As the updated results show, PREMIA's average performance on the alignment methods improves after the correction but still remains below our SLR method. Moreover, PREMIA's improvement is mainly on non-reward-model-based alignment, while on reward-model-based alignment (PPO, GRPO), it still remains close to random.

Answer to Question 3.

We thank the reviewer for the opportunity to clarify. NB-W weights each of the 19 dimensions by that dimension's member-vs-non-member AUC, which is estimated on the shadow model ensembled by the fingerprint mechanism. The detailed mechanism is shown in Algorithm 1 in Section 4, newly added in the revised version, which clearly presents the details of each method in pseudocode.

Thank you for conducting the additional experiments. I have a few follow up questions for clarification:

  1. Whether the proxy reward models are also used to train the GRPO/PPO shadow policies, or only to compute attack features.
  2. How many HH-RLHF pairs are used to train the 1B proxy reward model, whether they overlap with reference population, shadow calibration examples or membership-evaluation examples.
  3. The correlation of reward scores from the proxy and one used for training on the same generated response for examples in reference population, shadow calibration examples or membership-evaluation examples.

We thank the reviewer for this follow-up. It exposes a gap between our reward ablation experiment and the threat model: the earlier substitute-scorer study only used substitute rewards to compute attack features, and did not train the GRPO/PPO shadow policies under those rewards. To close this gap, we conducted the following supplementary experiments.

Data separation: the members in the membership-evaluation examples and the shadow examples are drawn from the training set and are disjoint from each other. The nonmembers in the membership-evaluation examples and the reference-population examples are drawn from the test set and are also disjoint. The proxy uses 10,000 HH-RLHF pairs from the training set, disjoint from all the above data.

Results: Table A. Detection under shadow policies retrained with substitute reward models (AUC; NB / QDA / NB-W).

Reward Modeltraining rewardself-trained 1Bgemma27
SFT.970/.973/.974.964/.955/.966.945/.929/.942
DPO.869/.861/.865.867/.865/.864.863/.867/.862
NPO.846/.821/.848.841/.821/.844.870/.855/.883
GRPO.721/.515/.750.737/.633/.739.728/.559/.728
PPO.671/.520/.646.651/.561/.567.605/.523/.635

Across all 15 AUC values, the mean for self-trained 1B and gemma27 are essentially identical to the training-reward baseline (-0.21% and −0.47%, respectively). These results of replacing shadow reward model are better than the non-replacing results reported in Table 1 of our previous rebuttal.

Analysis: Our pipeline first computes the 19-dimensional features of the membership evaluation examples on the policy. It then computes the 19-dimensional features of the reference population on both the policy and the shadow model; these latter features constitute the fingerprint.

In our original rebuttal experiments, the reward model was replaced only when computing the 19-dim features. This makes the reward-related components of the 19-dim features not fully accurate. The correlation between reward scores from the proxy and those from the real reward model is shown in Table B. Table B shows that all correlations are clearly above random and are close to Skywork's agreement with itself. Moreover, this does little harm: the substitute reward model only needs sufficient discriminative power to preserve the relative differences between members and nonmembers. The subsequent shadow calibration, however, is affected: the fingerprint comes from the proxy reward model while the shadow trained from the true reward model, creating a fingerprint-shadow mismatch and biasing calibration.

Table B. Reward agreement on the same generated responses (Pearson r / Spearman rho). The last column is the reference ceiling: Skywork's agreement with itself across different rollouts of the same prompt (same-response self-agreement is 1.0 by construction and is not informative).

Datasky↔trained-1Bsky↔gemma27ftsky↔sky (cross-rollout)
reference example.415/.407.407/.397.491/.484
shadow example.494/.507.432/.426.559/.556
evaluation example.444/.447.423/.409.456/.452

In the supplementary experiments, we also train the shadow model with the substitute reward model, so that the fingerprint and the shadow model are matched. This introduces a new mismatch between the shadow and the policy because they are trained with different reward models.

Table C reports calibration weights with and without replacing the shadow reward model. The fingerprint–shadow mismatch stems from the reward-model-dependent dimensions of the 19-dim features, whereas the shadow–policy mismatch arises for the GRPO and PPO shadows, the comparison therefore depends on which mismatch dominates. The higher average AUC and more accurate calibration weights of replacing the shadow reward model indicate that the fingerprint–shadow mismatch is the dominant factor. A plausible reason is that the aligned shadows and policy induced by different reward models are close, so the shadow–policy mismatch is small, whereas different reward models induce larger differences in the features.

Table C. Difference in fingerprint blend (0–1) between replacing and not replacing the shadow reward model. (trained-1B proxy; weight of the dominant shadow):

panelFeature-only blend weightshadow-retrained blend weight
SFTppo .77, sft .16sft .78
DPOdpo 1.00dpo 1.00
NPOppo .90, npo .05npo .80
GRPOgrpo 1.00grpo 1.00
PPOppo .74ppo .41

Thank you for the additional experiments and clarifications, which mostly addressed the concern regarding reward-access and proxy implementation, though robustness to proxies with substantially different reward preferences remains uncertain. The benefit of the proposed scoring rules is still unclear. My assessment of technical contribution maintains.

Comments on the paper

1 comment

We thank all reviewers for their careful reading and constructive feedback. We address the main concerns through new experiments and clarified presentation:

(i) A threat-model clarification and a reward-function substitution study demonstrating that the attack does not require access to the training reward model (WJ2D-W1/Q1);

(ii) Faithful re-implementations of the original DIBA baseline and a corrected PREMIA formulation, evaluated under identical calibration on every panel, together with a clarified NB-W weighting mechanism presented in Algorithm 1 in the revised version (WJ2D-Q2/Q3, P9BQ-W1);

(iii) A contribution decomposition and a technique-component ablation study in Section 5.3 in the revised version (WJ2D-W2, P9BQ-W2/Q1, 5zaD-W1/Q2);

(iv) Controlled multi-stage member-type experiments, including a realistic three-stage SFT→GRPO→NPO chain (P9BQ-Q3, 5zaD-W2);

(v) A systematic shadow-mismatch robustness study (P9BQ-Q4).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.