- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This work investigates the privacy risk of best-of-n (BoN) sampling and analyzes whether it is possible to distill a reward model and its ranking. The analysis explores 3 deployment scenarios, when only the winner of the BoN is available, when the ranking is available and when the scores are also available. The results show that it is possible to partially recover the reward model’s preference ranking even under the most restrictive setting. Finally, the work evaluates an initial mitigation strategy (perturbing the winner selection), which only slightly weakens the attack.
Critical strengths
- The work explores new and mostly unexplored settings on the privacy risks of the reward models used at inference time and whether it is possible to distill them. This is important because BoN is commonly used, and identifying it as a side channel is useful for practitioners.
- The evaluation and empirical experiments are reasonably extensive and cover all the main settings. The results are consistent across five reward models, five seeds, and out-of-distribution prompts.
Critical weaknesses
- It is unclear whether the ordering agreement of the attack comes from actual knowledge of the reward model or it comes from a distribution shift between the responses and . The responses don’t necessarily follow the same distribution (for instance the reward model might be biased toward longer responses), so a simple heuristic on the response features could achieve high agreement without any access to BoN. Blind baselines (e.g. response length and base policy probability) are missing from Table 1. On RewardBench-2 the base policy log probability alone outperforms all L1 shadow models. Similarly, the novelty of the approach seems limited. The setting is very similar to the standard distillation settings, however standard distillation baselines are not compared.
- The work focuses on privacy risks however it does not seem to measure privacy risk. There is no member against non-members comparison and the focus is on distilling the reward model, which I think overstate the privacy risk. Moreover, agreement on the math and code pairs suggest that the shadow models learn the reward model’s ordering rather than information specific to its training data.
Questions for authors
- How do blind baselines perform on Table 1 pairs?
- What is the report agreement on member versus non-member pairs to support the privacy claim?
4 comments
Response to Reviewer SKsd (2/2)
W3. Novelty, and comparison with standard distillation baselines.
We would like to clarify that this paper does not claim a new distillation method (see also point 4 of our general response). Our contribution is to identify that BoN might leak the preferences of its private RM, and to study this leak systematically, with three exposure levels, five RMs, and a first attempt at a defense. The setting also differs from standard distillation, where the student model can control the input and directly query the teacher model. Our L1 attacker never queries , and it observes only the BoN winner, without access to the scores or the rejected candidates (§2, l.156–161). As for the baselines, Table 3 (App. B) compares our objectives with the common alternatives, namely the Bradley–Terry loss with a single reference at L1 and score regression (MSE) at L3.
W4 and Q2. Whether the paper measures a privacy risk, and the member versus non-member comparison.
We think reviewer SKsd might misread which privacy risk we study (see point 1 of our general response). We identify two risks, namely (i) recovering the preferences behind 's training data, and (ii) distilling itself. The reviewer writes that "the focus is on distilling the reward model", but distillation is only risk (ii). Risk (i) concerns the preferences in 's training data, and we measure it separately on the curated pairs, which are pairs from 's own training data (§5.1, l.331–334).
Regarding Q2, a member versus non-member comparison evaluates membership inference, which is a different attack from the two we study. We do not claim it, and we have listed it as future work (§6, l.475–477). Membership inference is also not the only kind of privacy attack. The survey of Rigaki & Garcia [1] also covers attacks that recover training samples or their labels, and model extraction. Risk (i) recovers the preference labels of 's training pairs (not the pairs themselves), and risk (ii) is model extraction.
Regarding the math and code results, a reward model is trained precisely to learn the preference signal in its training data (Stiennon et al. [2]; Ouyang et al. [3]), and its ordering is that signal. This preference signal is what the shadow model recovers, and the leak of this signal is exactly the privacy risk that we study. Other information about the training data, such as membership, is not what we aim to study, and we leave it to future work as noted above.
[1] Rigaki and Garcia. A Survey of Privacy Attacks in Machine Learning. ACM Computing Surveys, 2023.
[2] Stiennon et al. Learning to Summarize with Human Feedback. NeurIPS, 2020.
[3] Ouyang et al. Training Language Models to Follow Instructions with Human Feedback. NeurIPS, 2022.
Thank you for the detailed response and the blind baselines, which address my concern for L2 and L3.
My main concern about privacy remains. L1 recovers preferences at a similar rate on pairs from the training data of the RM and on the math and code pairs, which are not in its training data. This suggests that the shadow model learns the general preferences of the RM rather than information specific to its training data, so risk (i) reduces to risk (ii).
Risk (ii) is also limited at L1, since the L1 shadow models are close to chance on RewardBench 2 and the attack fails with mismatched references. Model extraction is therefore shown only at L2 and L3, where the setting is close to standard distillation.
Finally, L1 outperforms the blind baselines only on the RMs trained on the same preference dataset, so this result may depend on that data.
For these reasons, I still find the privacy claims overstated and I maintain my score.
We thank the reviewer for the follow-up and we are glad the blind baselines resolved the reviewer's concern about L2 and L3. We would like to explain briefly why we believe that the remaining objections of Reviewer SKsd do not support the conclusion that our privacy claims are overstated.
1. Does risk (i) reduce to risk (ii)? The reviewer argues that because L1 recovers preferences at similar rates on training pairs and on math and code pairs, the shadow model learns only the general preferences of and risk (i) reduces to risk (ii).
First, the premise of that argument does not hold. Each row of Table 1 reports a separate attack. We repeat the attack on math and code prompts, training a separate shadow model for each domain and evaluating each model on pairs from that domain (§5.1, l.313–316). Each row has its own evaluation sets, so we specify in the Table 1 caption that values should be compared only within a row. Furthermore, the blind baselines the reviewer requested show how much these sets differ in difficulty. L1 scores look similar on the Skywork-RM and math curated pairs (0.754 and 0.746), yet the best blind baseline scores 0.596 on Skywork-RM and 0.793 on math. Similar L1 scores across these rows therefore do not imply that the shadow model learns only the general preferences of .
Second, even if the shadow model learns the general preferences of , risk (i) does not reduce to risk (ii). Because learned those "general preferences" from its own training data, a shadow model that learns the "general preferences" also recovers some of the preferences behind those data. Our experiments show that for the three Skywork-trained RMs, the L1 attacker sees only the BoN winner yet correctly predicts the preferred response on about three out of four pairs from 's training data (74% to 75%). The best blind baseline uses no BoN outputs and reaches only 58% to 60%.
2. Risk (ii) at L1. The reviewer writes that "Risk (ii) is also limited at L1". We would like to stress again that the L1 shadow model agrees with on up to 85% of the generated pairs, whose responses are not in 's training data. On the math and code prompts, which lie outside 's training distribution, this agreement is 88% and 80%, respectively (Table 1). The agreement on these new responses shows that the L1 shadow model has learned part of 's behavior (risk (ii)), though it struggles on RewardBench-2, which is a much harder benchmark. Most prompts in RewardBench-2 require the model to choose one response out of four, with even more choices in its Ties category. Even itself achieves only 0.63 to 0.81 accuracy (§5.3, l.365–369 and l.376). Finally, our attack is unoptimized (§1, l.108–109). We made a first attempt to exploit BoN, and future work may yield a stronger attacker for distilling (§6, l.472–474).
3. Are the privacy claims overstated? We disagree with the reviewer's assessment that we overstate the privacy risk of BoN. Every privacy claim in the paper rests on our experimental results. We reported all the L1 limitations the reviewer lists in the follow-up, either in the paper or in our earlier replies. For all these reasons, the reviewer's remaining objections do not support the conclusion that our privacy claims are overstated.
Response to Reviewer SKsd (1/2)
We thank the reviewer for the detailed comments. The suggestion to add blind baselines was especially helpful, and we have added them to the paper.
W1 and Q1. Blind baselines on the Table 1 pairs, and the distribution shift between yc and yr.
We agree that blind baselines are a good control for this concern. In the submission we reported the two likelihood baselines only on RewardBench-2 (App. E.5). Following the reviewer's request, we have now evaluated three blind baselines on every evaluation set of Table 1 (see point 2 of our general response). The table below reports the best of the three baselines next to our shadow models, as curated / generated agreement. An L1 number is in bold where the blind baseline loses to the L1 shadow model.
We tested the length heuristic that the reviewer gives as an example on both the curated and the generated pairs. Choosing the preferred response by length alone reaches only 54% to 64% agreement on the curated pairs and 62% to 68% on the generated pairs, whereas the L1 shadow model reaches 63% to 75% and 74% to 88%, respectively. The L1 shadow model is above this length baseline on every set, by 7 to 21 points on the curated pairs and by 10 to 24 points on the generated pairs. Response length alone is therefore not sufficient to capture the preference signal of r⋆. The full results are in the new App. E.7.
W2. On RewardBench-2, the base-policy log-probability outperforms the L1 shadow models.
We report this result ourselves in App. E.5, and we include this experiment since we want to show what our L1 attack can and cannot do. Our L1 shadow model indeed does not exceed the blind likelihood baselines on RewardBench-2, but we do not think that this result undermines our contribution. We identify two risks, namely (i) recovering the preferences behind r⋆'s training data and (ii) distilling r⋆ itself. RewardBench-2 covers only one part of (ii), as it tests whether the shadow model generalizes to other, harder tasks. We also ran the attack on math and code prompts, which lie outside r⋆'s training data, and there the L1 shadow model still reproduces r⋆'s behavior on 88% and 80% of the generated pairs (Table 1). The RewardBench-2 result therefore only shows that our L1 shadow model does not generalize well to these harder problems. The L1 attacker already achieves (i) by recovering up to 75% of the preferences behind r⋆'s training data (Table 1), which is a serious privacy risk on its own.