acceptodds
Under review as a conference paper at ICLR 2027

Leaky Students: Membership Inference against On-Policy Distillation

Abstract

On-policy distillation (OPD) trains a student to match a teacher's next-token distributions on student-generated trajectories. However, privileged information supplied to the teacher for OPD training may contain sensitive data. Whether the student leaks private information about the records supplied to the teacher during distillation remains poorly understood. To the best of our knowledge, we present the first systematic study of membership inference in this setting. We find that fresh student trajectories expose sparse membership signals that fixed reference-answer losses often miss. These signals are mixed with probability changes caused by training on other records. We introduce LEAKY, which samples fresh trajectories from the target model and compares its token log-probabilities with the maximum across matched reference models trained without the candidate records. It applies Leaky ReLU to the resulting gaps, preserving positive gaps and downweighting negative gaps as an approximate correction for incidental positive gaps in non-members. Across fifteen targets spanning mathematics, medical question answering, and code generation, LEAKY outperforms all evaluated baselines and achieves mean AUROC 0.875, compared with 0.614 for the strongest baseline on each target in the main evaluation. On the same sampled trajectories, the strongest baseline achieves mean AUROC 0.826. These results show that students trained through OPD can expose the membership of records used for teacher supervision, even when fixed reference-answer losses provide little evidence of membership.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
On-policy Distillation, Membership Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7Y34OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors studied membership inference attacks tailed for student models trained with OPD. Based on the threat model that the attacker knows: 1) the candidate prompt xx and the reference answer rr, 2) the OPD training recipe including the student model state pre-OPD, the teacher model, the OPD configuration and the OPD training data distribution, 3) student model’s token-level log probability during inference, but not model weights, they proposed Leaky that utilizes KK reference models trained via OPD following the same OPD setup as the target student model, but trained with auxiliary data from the same task distribution, with the candidates excluded, to compute a per-token gap between the target student model and the reference model with the highest log-probability, then the gap is aggregated via LeakyReLU. They compared the performance of Leaky with existing MIAs on five models across three tasks, and show higher attack success rates on all tasks. They also run ablation studies to investigate the effect of the number of reference models and black-box attacks.

Critical strengths

The authors studied a timely privacy issue for OPD setting, and this paper seems to be the first attempt to perform MIA on OPD setting. It also shows that existing MIAs are not very effective on OPD setting. The empirical performance of Leaky seems to be very good comparing to existing MIA attacks on OPD setting. The authors also provide link to the code for reproducibility purpose.

Critical weaknesses

I list the main concerns here and minor issues in the Questions for Authors box.

  1. Most of the Leaky components are existing techniques, for example, using reference models and sampling from the models, and per-token gaps. It is not very clear which part of Leaky is essential for OPD setting that make this attack effective for OPD. The measured benefit over the best baseline, on the same sampled trajectories, is 0.049 mean AUROC, very small.
  2. It is not clear if Leaky only works for OPD setting or other post-trained models, such as models post-trained with SFT or other distillation methods, therefore, it is not clearly if this type of attack shows an unique weakness of OPD or not.
  3. The selection of parameter α\alpha seems to be dataset dependent, it is unclear how sensitive Leaky is on the selection of α\alpha.
  4. Based on the threat model, the attacker knows most of the information regarding OPD, such as OPD configuration and data distribution. More experiments should be performed to examine with limited or even no access of OPD configuration, does Leaky perform the same or not?
  5. There is no discussion on how the size of the training dataset will affect the performance of Leaky.
Questions for authors
  1. Does leakage come from the teacher's conditioning on rr, or from training on prompts and rollouts? Does Leaky still work if rr is replaced by a wrong or shuffled answer at attack time, or if the attacker has xx only?
  2. Can you add (a) non-DP LoRA with DP-LoRA's learning rate and clipping, (b) DP full fine-tuning, and (c) a utility-matched early-stopped baseline, on more than one student? Is near-chance AUROC under DP-LoRA explained by the model barely learning?
  3. It will be good to know if one applies the Leaky ReLU or max-reference gap to InfoRMIA's token scores, and replace the Leaky aggregator with mean, Min-K or window aggregation on max-reference gaps, on identical trajectories, what will be the results?
  4. There is no cost analysis on Leaky. How many records, steps and GPU hours does each reference model need, and how does AUROC scale with auxiliary set size? What is the minimum attacker budget?
Reviewer_b8gjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper presents the study of membership inference attacks of on-policy distillation. The proposed Leaky attack has three steps: 1)for each candidate training record, sample 8 fresh conditional trajectories from the student; 2) compare the target's token log-probabilities on these trajectories against 8 shadow models trained with the same recipe; 3) aggregate the gap into a membership score. Experiments are conducted across 5 students and 3 domains reaching 0.875 AUROC.

Critical strengths
  1. OPD is becoming a mainstream post-training recipe, and the attack surface of its training records has not previously been studied systematically.
  2. Extensive experiments are conducted across 5 models and 3 domains against 10+ baselines.
  3. The writing is clear and internally consistent.
Critical weaknesses
  1. The threat model is too strong to be realistic. LEAKY needs 1)the base model, 2)the teacher, 3)the exact training recipe, 4)shadow data that largely overlap the victim's training pool. In most cases, these resources are not available to outside attackers.
  2. The attack itself has limited algorithmic novelty. It is essentially a combination of LiRA-style shadow reference + trajectory-conditioned InfoRMIA + Leaky-ReLU aggregation; the teacher plays no role in the attack proper (only in the OPD training pipeline and in replicating the shadows).
  3. Under the paper's matched comparison, where both attacks score the same 8 sampled trajectories, Leaky's advantage over InfoRMIA is only +0.049 mean AUROC; most of the headline gap therefore comes from the trajectory channel rather than the aggregator. 4.The math-domain targets train on only 512 records repeated ~8 epochs; this "small pool × many passes" corner limits the validity of the leakage conclusions for realistic OPD deployment.
Questions for authors
  1. Could the authors report the overlap rate between the shadow and victim training pools for each domain?
  2. Could the authors report the total compute cost of the full attack pipeline and how it compares with the shadow budget required by each baseline.
  3. Could Leaky still work if the OPD pipeline adopted mixed on-policy distillation?
Reviewer_Tm1fOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors propose Leaky, a reference-model-based membership inference attack against language models trained with on-policy distillation (OPD), where a teacher conditioned on privileged reference answers supervises a student that only sees the prompt. Specifically, the paper shows that conventional loss-based MIAs on the fixed reference answer are near chance. Leaky samples trajectories from the target conditioned on pairs, computes per-token gaps between the target log-probability and the maximum over K OPD-trained reference models, and aggregates them with a Leaky ReLU whose slope is motivated by an extreme-value argument for correcting chance leads caused by model disagreement. Additionally, across 15 targets, Leaky reaches a mean AUROC of 0.875 versus 0.614 for the strongest baseline in the standard setting and 0.826 when baselines score the same sampled trajectories. A text-only variant remains effective (AUROC 0.66–0.85). LoRA reduces but does not remove leakage, while DP-LoRA brings AUROC to chance at a substantial utility cost. Ablations cover the number of reference models and trajectories, the sampling model, OPSD, and synthetic or external auxiliary data.

Critical strengths

1.Timely and interesting problem for me: Privileged-information OPD/OPSD is rapidly being adopted, and the paper identifies a concrete risk: excluding r from the student input does not imply membership privacy. This corrects a plausible but incorrect intuition among practitioners.

2. Attractive Preliminary experiments: The analysis in Fig. 2 and Sec. 3.1 cleanly separates two obstacles, the objective mismatch (fixed-answer loss is uninformative) and shared, non-candidate-specific probability changes. These two obstacles directly motivate the trajectory-based scoring and reference calibration.

3. Broad and careful evaluation: The evaluation spans 15 targets across 3 model families and 3 domains, uses low-FPR metrics, and includes stratified member/non-member matching for code. It also adds a fairer “same-trajectory” baseline comparison, black-box and OPSD settings, auxiliary-data variants, and reference-model controls that stay near chance.

Critical weaknesses

1. Novelty of the Leaky method: Calibrating token-level log-probabilities against reference models is already central to existing attacks (e.g., RMIA and InfoRMIA). It is therefore unclear how much of Leaky’s improvement comes from new methodology rather than from applying these ideas to freshly sampled trajectories. In Tab.10, InfoRMIA on the same trajectories reaches a mean AUROC of 0.826 versus 0.875 for Leaky, and further clarification of the methodological contribution beyond the max-reference calibration and Leaky ReLU aggregation would be helpful.

2.Computational cost and efficiency: Leaky requires training K=8 full OPD reference models, each with a large teacher served online, and then scoring N=8 sampled trajectories per candidate with all nine models. The black-box variant additionally needs roughly 4k–8k queries per candidate. The paper reports neither wall-clock time nor GPU-hours, nor how this cost compares with the baselines, which makes it hard to assess how practical the attack is.

3.Limited baselines and defense evaluation: The paper does not compare against SFT or fixed-sequence KD on the same data, so it remains unclear whether OPD leaks more or less than directly training on r. Defenses are limited to LoRA and DP-LoRA at a single budget. Missing are a privacy–utility curve over ε, simpler mitigations (e.g., fewer epochs, early stopping, deduplication), and the cited DP-OPD method.

Questions for authors

I find the paper’s study focus and insights interesting, I hope the authors can address the concerns (e.g., Methodological contribution, Attack cost, Comparison with SFT) , and I will consider raising my score accordingly.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your time and effort in reviewing this paper. Please take a moment to read the authors’ rebuttal and the other reviewers’ comments, and participate in the discussion. It would be helpful to indicate which concerns have been addressed and which remain unresolved, and to follow up with the authors where further clarification is needed. Please also update your review and score if warranted, or briefly explain why your assessment remains unchanged. Your engagement will be valuable in reaching a well-informed decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.