- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, the authors studied membership inference attacks tailed for student models trained with OPD. Based on the threat model that the attacker knows: 1) the candidate prompt and the reference answer , 2) the OPD training recipe including the student model state pre-OPD, the teacher model, the OPD configuration and the OPD training data distribution, 3) student model’s token-level log probability during inference, but not model weights, they proposed Leaky that utilizes reference models trained via OPD following the same OPD setup as the target student model, but trained with auxiliary data from the same task distribution, with the candidates excluded, to compute a per-token gap between the target student model and the reference model with the highest log-probability, then the gap is aggregated via LeakyReLU. They compared the performance of Leaky with existing MIAs on five models across three tasks, and show higher attack success rates on all tasks. They also run ablation studies to investigate the effect of the number of reference models and black-box attacks.
Critical strengths
The authors studied a timely privacy issue for OPD setting, and this paper seems to be the first attempt to perform MIA on OPD setting. It also shows that existing MIAs are not very effective on OPD setting. The empirical performance of Leaky seems to be very good comparing to existing MIA attacks on OPD setting. The authors also provide link to the code for reproducibility purpose.
Critical weaknesses
I list the main concerns here and minor issues in the Questions for Authors box.
- Most of the Leaky components are existing techniques, for example, using reference models and sampling from the models, and per-token gaps. It is not very clear which part of Leaky is essential for OPD setting that make this attack effective for OPD. The measured benefit over the best baseline, on the same sampled trajectories, is 0.049 mean AUROC, very small.
- It is not clear if Leaky only works for OPD setting or other post-trained models, such as models post-trained with SFT or other distillation methods, therefore, it is not clearly if this type of attack shows an unique weakness of OPD or not.
- The selection of parameter seems to be dataset dependent, it is unclear how sensitive Leaky is on the selection of .
- Based on the threat model, the attacker knows most of the information regarding OPD, such as OPD configuration and data distribution. More experiments should be performed to examine with limited or even no access of OPD configuration, does Leaky perform the same or not?
- There is no discussion on how the size of the training dataset will affect the performance of Leaky.
Questions for authors
- Does leakage come from the teacher's conditioning on , or from training on prompts and rollouts? Does Leaky still work if is replaced by a wrong or shuffled answer at attack time, or if the attacker has only?
- Can you add (a) non-DP LoRA with DP-LoRA's learning rate and clipping, (b) DP full fine-tuning, and (c) a utility-matched early-stopped baseline, on more than one student? Is near-chance AUROC under DP-LoRA explained by the model barely learning?
- It will be good to know if one applies the Leaky ReLU or max-reference gap to InfoRMIA's token scores, and replace the Leaky aggregator with mean, Min-K or window aggregation on max-reference gaps, on identical trajectories, what will be the results?
- There is no cost analysis on Leaky. How many records, steps and GPU hours does each reference model need, and how does AUROC scale with auxiliary set size? What is the minimum attacker budget?
Dear Reviewers,
Thank you for your time and effort in reviewing this paper. Please take a moment to read the authors’ rebuttal and the other reviewers’ comments, and participate in the discussion. It would be helpful to indicate which concerns have been addressed and which remain unresolved, and to follow up with the authors where further clarification is needed. Please also update your review and score if warranted, or briefly explain why your assessment remains unchanged. Your engagement will be valuable in reaching a well-informed decision.