acceptodds
Under review as a conference paper at ICLR 2027

Preference Stealing from Best-of-: The Privacy Risk of Inference-Time Alignment

Abstract

Inference-time alignment methods are gaining popularity since they improve the helpfulness and safety of language models' responses without training. Most methods rely on a reward model (RM) to raise the probability of good responses. The RM encodes preferences from expensive, proprietary, and often sensitive human-annotated training data. Both the RM itself and these preferences are therefore valuable targets for attackers. However, prior work on preference privacy has focused on training time, leaving the risk at inference time unexplored. We first raise the concern that inference-time alignment methods might leak the RM's preferences through the responses they serve. We take Best-of- (BoN) sampling, one of the most widely used methods, as the representative case. BoN keeps its RM in the serving loop, so every served response implicitly reveals a preference signal from the RM. We study how attackers might acquire the RM's preferences under different deployment scenarios, which expose different amounts of information about BoN's alignment process. Our experiments show that even under the strictest scenario in which BoN serves only the final response, attackers can still recover the preferences behind the RM's training data with about accuracy, against for chance. Finally, we take a first step toward a defense through output perturbation, yet the attack survives almost intact. BoN keeps leaking the RM's preferences, and closing the leak remains an open problem.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
privacy, reward distillation, preference reconstruction, inference-time alignment, Best-of-N

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SKsdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work investigates the privacy risk of best-of-n (BoN) sampling and analyzes whether it is possible to distill a reward model and its ranking. The analysis explores 3 deployment scenarios, when only the winner of the BoN is available, when the ranking is available and when the scores are also available. The results show that it is possible to partially recover the reward model’s preference ranking even under the most restrictive setting. Finally, the work evaluates an initial mitigation strategy (perturbing the winner selection), which only slightly weakens the attack.

Critical strengths
  • The work explores new and mostly unexplored settings on the privacy risks of the reward models used at inference time and whether it is possible to distill them. This is important because BoN is commonly used, and identifying it as a side channel is useful for practitioners.
  • The evaluation and empirical experiments are reasonably extensive and cover all the main settings. The results are consistent across five reward models, five seeds, and out-of-distribution prompts.
Critical weaknesses
  • It is unclear whether the ordering agreement of the attack comes from actual knowledge of the reward model or it comes from a distribution shift between the responses ycy_c and yry_r. The responses don’t necessarily follow the same distribution (for instance the reward model might be biased toward longer responses), so a simple heuristic on the response features could achieve high agreement without any access to BoN. Blind baselines (e.g. response length and base policy probability) are missing from Table 1. On RewardBench-2 the base policy log probability alone outperforms all L1 shadow models. Similarly, the novelty of the approach seems limited. The setting is very similar to the standard distillation settings, however standard distillation baselines are not compared.
  • The work focuses on privacy risks however it does not seem to measure privacy risk. There is no member against non-members comparison and the focus is on distilling the reward model, which I think overstate the privacy risk. Moreover, agreement on the math and code pairs suggest that the shadow models learn the reward model’s ordering rather than information specific to its training data.
Questions for authors
  1. How do blind baselines perform on Table 1 pairs?
  2. What is the report agreement on member versus non-member pairs to support the privacy claim?
4 comments
AuthorsOpenReview ↗

Response to Reviewer SKsd (1/2)

We thank the reviewer for the detailed comments. The suggestion to add blind baselines was especially helpful, and we have added them to the paper.

W1 and Q1. Blind baselines on the Table 1 pairs, and the distribution shift between ycy_c and yry_r.

We agree that blind baselines are a good control for this concern. In the submission we reported the two likelihood baselines only on RewardBench-2 (App. E.5). Following the reviewer's request, we have now evaluated three blind baselines on every evaluation set of Table 1 (see point 2 of our general response). The table below reports the best of the three baselines next to our shadow models, as curated / generated agreement. An L1 number is in bold where the blind baseline loses to the L1 shadow model.

r⋆r^\starbest blind baselineL1L2L3
Skywork-RM0.596 / 0.7230.754 / 0.8290.942 / 0.9790.942 / 0.983
URM-8B0.602 / 0.7760.746 / 0.8390.945 / 0.9810.952 / 0.984
Gemma-27B-RM0.576 / 0.7450.738 / 0.8500.937 / 0.9850.939 / 0.985
Skywork-RM, math0.793 / 0.7200.746 / 0.8840.980 / 0.9890.984 / 0.993
Skywork-RM, code0.649 / 0.7510.664 / 0.7970.909 / 0.9660.903 / 0.960
Tulu-RM0.778 / 0.8420.704 / 0.7500.967 / 0.9630.974 / 0.970
RB2-RM0.683 / 0.8050.631 / 0.7410.953 / 0.9740.959 / 0.980
  • L2 and L3. The best blind baseline loses to the L2 and L3 shadow models on all 14 evaluation sets, by 0.12 to 0.36. This gap suggests that the agreement of the L2 and L3 shadow models does not come from response features alone, and that these shadow models have learned the preference signal of r⋆r^\star.
  • L1 on the three Skywork-trained RMs. The best blind baseline also loses to the L1 shadow model on both sets of the three RMs trained on the Skywork preference data (Skywork-RM, URM-8B and Gemma-27B-RM), and on the generated pairs of the math and code prompts.
  • Where L1 does not exceed the blind baseline. On Tulu-RM and RB2-RM, and on the curated pairs of the math prompts, the L1 shadow model stays below the mean log-probability baseline, and on the curated pairs of the code prompts the two are on par (0.664 against 0.649). In these cases we cannot tell whether the agreement of the L1 shadow model comes from having really learned r⋆r^\star's preferences, or simply because r⋆r^\star itself prefers the responses that πref\pi_{\text{ref}} finds more likely. The main claim of the paper nevertheless holds, because the L1 shadow model clearly exceeds the blind baselines on the three Skywork-trained RMs. The details are in point 2 of our general response.

We tested the length heuristic that the reviewer gives as an example on both the curated and the generated pairs. Choosing the preferred response by length alone reaches only 54% to 64% agreement on the curated pairs and 62% to 68% on the generated pairs, whereas the L1 shadow model reaches 63% to 75% and 74% to 88%, respectively. The L1 shadow model is above this length baseline on every set, by 7 to 21 points on the curated pairs and by 10 to 24 points on the generated pairs. Response length alone is therefore not sufficient to capture the preference signal of r⋆r^\star. The full results are in the new App. E.7.

W2. On RewardBench-2, the base-policy log-probability outperforms the L1 shadow models.

We report this result ourselves in App. E.5, and we include this experiment since we want to show what our L1 attack can and cannot do. Our L1 shadow model indeed does not exceed the blind likelihood baselines on RewardBench-2, but we do not think that this result undermines our contribution. We identify two risks, namely (i) recovering the preferences behind r⋆r^\star's training data and (ii) distilling r⋆r^\star itself. RewardBench-2 covers only one part of (ii), as it tests whether the shadow model generalizes to other, harder tasks. We also ran the attack on math and code prompts, which lie outside r⋆r^\star's training data, and there the L1 shadow model still reproduces r⋆r^\star's behavior on 88% and 80% of the generated pairs (Table 1). The RewardBench-2 result therefore only shows that our L1 shadow model does not generalize well to these harder problems. The L1 attacker already achieves (i) by recovering up to 75% of the preferences behind r⋆r^\star's training data (Table 1), which is a serious privacy risk on its own.

AuthorsOpenReview ↗

Response to Reviewer SKsd (2/2)

W3. Novelty, and comparison with standard distillation baselines.

We would like to clarify that this paper does not claim a new distillation method (see also point 4 of our general response). Our contribution is to identify that BoN might leak the preferences of its private RM, and to study this leak systematically, with three exposure levels, five RMs, and a first attempt at a defense. The setting also differs from standard distillation, where the student model can control the input and directly query the teacher model. Our L1 attacker never queries r⋆r^\star, and it observes only the BoN winner, without access to the scores or the rejected candidates (§2, l.156–161). As for the baselines, Table 3 (App. B) compares our objectives with the common alternatives, namely the Bradley–Terry loss with a single reference at L1 and score regression (MSE) at L3.

W4 and Q2. Whether the paper measures a privacy risk, and the member versus non-member comparison.

We think reviewer SKsd might misread which privacy risk we study (see point 1 of our general response). We identify two risks, namely (i) recovering the preferences behind r⋆r^\star's training data, and (ii) distilling r⋆r^\star itself. The reviewer writes that "the focus is on distilling the reward model", but distillation is only risk (ii). Risk (i) concerns the preferences in r⋆r^\star's training data, and we measure it separately on the curated pairs, which are pairs from r⋆r^\star's own training data (§5.1, l.331–334).

Regarding Q2, a member versus non-member comparison evaluates membership inference, which is a different attack from the two we study. We do not claim it, and we have listed it as future work (§6, l.475–477). Membership inference is also not the only kind of privacy attack. The survey of Rigaki & Garcia [1] also covers attacks that recover training samples or their labels, and model extraction. Risk (i) recovers the preference labels of r⋆r^\star's training pairs (not the pairs themselves), and risk (ii) is model extraction.

Regarding the math and code results, a reward model is trained precisely to learn the preference signal in its training data (Stiennon et al. [2]; Ouyang et al. [3]), and its ordering is that signal. This preference signal is what the shadow model recovers, and the leak of this signal is exactly the privacy risk that we study. Other information about the training data, such as membership, is not what we aim to study, and we leave it to future work as noted above.

[1] Rigaki and Garcia. A Survey of Privacy Attacks in Machine Learning. ACM Computing Surveys, 2023.

[2] Stiennon et al. Learning to Summarize with Human Feedback. NeurIPS, 2020.

[3] Ouyang et al. Training Language Models to Follow Instructions with Human Feedback. NeurIPS, 2022.

Thank you for the detailed response and the blind baselines, which address my concern for L2 and L3.

My main concern about privacy remains. L1 recovers preferences at a similar rate on pairs from the training data of the RM and on the math and code pairs, which are not in its training data. This suggests that the shadow model learns the general preferences of the RM rather than information specific to its training data, so risk (i) reduces to risk (ii).

Risk (ii) is also limited at L1, since the L1 shadow models are close to chance on RewardBench 2 and the attack fails with mismatched references. Model extraction is therefore shown only at L2 and L3, where the setting is close to standard distillation.

Finally, L1 outperforms the blind baselines only on the RMs trained on the same preference dataset, so this result may depend on that data.

For these reasons, I still find the privacy claims overstated and I maintain my score.

We thank the reviewer for the follow-up and we are glad the blind baselines resolved the reviewer's concern about L2 and L3. We would like to explain briefly why we believe that the remaining objections of Reviewer SKsd do not support the conclusion that our privacy claims are overstated.

1. Does risk (i) reduce to risk (ii)? The reviewer argues that because L1 recovers preferences at similar rates on training pairs and on math and code pairs, the shadow model learns only the general preferences of r⋆r^\star and risk (i) reduces to risk (ii).

First, the premise of that argument does not hold. Each row of Table 1 reports a separate attack. We repeat the attack on math and code prompts, training a separate shadow model for each domain and evaluating each model on pairs from that domain (§5.1, l.313–316). Each row has its own evaluation sets, so we specify in the Table 1 caption that values should be compared only within a row. Furthermore, the blind baselines the reviewer requested show how much these sets differ in difficulty. L1 scores look similar on the Skywork-RM and math curated pairs (0.754 and 0.746), yet the best blind baseline scores 0.596 on Skywork-RM and 0.793 on math. Similar L1 scores across these rows therefore do not imply that the shadow model learns only the general preferences of r⋆r^\star.

Second, even if the shadow model learns the general preferences of r⋆r^\star, risk (i) does not reduce to risk (ii). Because r⋆r^\star learned those "general preferences" from its own training data, a shadow model that learns the "general preferences" also recovers some of the preferences behind those data. Our experiments show that for the three Skywork-trained RMs, the L1 attacker sees only the BoN winner yet correctly predicts the preferred response on about three out of four pairs from r⋆r^\star's training data (74% to 75%). The best blind baseline uses no BoN outputs and reaches only 58% to 60%.

2. Risk (ii) at L1. The reviewer writes that "Risk (ii) is also limited at L1". We would like to stress again that the L1 shadow model agrees with r⋆r^\star on up to 85% of the generated pairs, whose responses are not in r⋆r^\star's training data. On the math and code prompts, which lie outside r⋆r^\star's training distribution, this agreement is 88% and 80%, respectively (Table 1). The agreement on these new responses shows that the L1 shadow model has learned part of r⋆r^\star's behavior (risk (ii)), though it struggles on RewardBench-2, which is a much harder benchmark. Most prompts in RewardBench-2 require the model to choose one response out of four, with even more choices in its Ties category. Even r⋆r^\star itself achieves only 0.63 to 0.81 accuracy (§5.3, l.365–369 and l.376). Finally, our attack is unoptimized (§1, l.108–109). We made a first attempt to exploit BoN, and future work may yield a stronger attacker for distilling r⋆r^\star (§6, l.472–474).

3. Are the privacy claims overstated? We disagree with the reviewer's assessment that we overstate the privacy risk of BoN. Every privacy claim in the paper rests on our experimental results. We reported all the L1 limitations the reviewer lists in the follow-up, either in the paper or in our earlier replies. For all these reasons, the reviewer's remaining objections do not support the conclusion that our privacy claims are overstated.

Reviewer_hzHYOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies whether a hidden reward model’s preferences can be recovered from a Best-of-NN service. The authors train shadow reward models under three access levels: selected answers only, candidate rankings, and reward scores. Across five reward models, they find that even winner-only access enables some preference recovery, while rankings allow much stronger model extraction. They also test noise-based defenses, which only modestly weaken the attacks at the evaluated settings while reducing BoN’s reward gains.

Critical strengths

The paper tries to answer a practically useful question: how much can someone learn about a hidden reward model just from the answers a BoN service returns?

The comparison across access levels is helpful. Testing what happens when attackers see only winners, full rankings, or scores makes it clear how much each extra piece of information helps them.

The defense experiments make the study more complete. The authors also test whether adding noise weakens the attack and how much answer quality is lost along the way

Critical weaknesses
  1. The most realistic attack that is L1 has limited generalization. this is the paper’s most interesting setting in my eyes. However, its shadow models achieve only about 50–59% agreement with the r* on RewardBench-2. The stronger extraction results require rankings or scores. This limits the evidence that ordinary output-only access enables a broadly useful copy of the hidden reward model.

  2. The method novelty needs clearer separation from existing reward learning. The paper itself cites Pukdee et al. (2026) as showing that Bradley–Terry training on a BoN winner versus fresh base-model samples recovers the reward model’s ranking. Its L2 attack uses a weighted ListMLE objective. The contribution may therefore lie primarily in the security evaluation and exposure-level comparison. so the authors should clarify what the proposed objectives add over these existing approaches.

Questions for authors
  1. How does your noise-based defense compare with PrivBoN and PrivITP from [1]? It should be cited in your paper as it looks quite similar/related to your contribution.

  2. Also wanted to know your thoughts on this question: how much does knowing the base model matter? Would the winner-only attack still work if the attacker used a different generator or did not know the service’s sampling settings?

[1] Privacy Without Regret: Differentially Private Inference-Time Alignment — Jain, Bhattad, and Ray Chowdhury

3 comments
AuthorsOpenReview ↗

Response to Reviewer hzHY (1/2)

We thank the reviewer for the positive assessment and the thoughtful questions.

W1. The L1 attack has limited generalization on RewardBench-2.

We agree with this observation, and the paper also reports that the L1 attack "does not reach that far" on RewardBench-2 (§5.3, l.405–406). However, our experiments show that the L1 shadow model recovers up to 75% of the preferences behind r⋆r^\star's training data (Table 1), which is clearly above the blind baselines on the three Skywork-trained RMs (see point 2 of our general response). Our attack is also unoptimized, so the true leak may be worse (l.108–109).

W2. Separation from Pukdee et al. (2026) and existing reward learning.

The reviewer asks what the proposed objectives add over existing reward learning, but a new objective is not a contribution that we claim (see point 4 of our general response). In the paper we attribute both objectives to existing work. The L1 loss resembles the Bradley–Terry loss, for which we cite Pukdee et al. [1] as support (l.236–238), and the L2 loss is a weighted form of the ListMLE loss of Xia et al. [2] (l.244–245). Pukdee et al. [1] study how to train a reward model from Best-of-NN preference data, whereas we show that BoN acts as an exploitable side channel for an adversary (l.80–81), and we measure how much BoN leaks at all three exposure levels against the same RM (§2, l.153–155). The contribution of this paper lies in the security evaluation and the exposure-level comparison.

[1] Pukdee, Balcan, and Ravikumar. Reward Learning from Best-of-N Preference Data: Targets, Tradeoffs, and Design Principles. arXiv:2605.30619, 2026.

[2] Xia, Liu, Wang, Zhang, and Li. Listwise Approach to Learning to Rank: Theory and Algorithm. ICML, 2008.

AuthorsOpenReview ↗

Reviewer hzHY (2/2)

Q1. Comparison with PrivBoN and PrivITP.

Thank you for pointing us to this excellent work. Although PrivBoN [1] is concurrent with our submission (arXiv, Aug. 26, 2026) and ICLR's policy on contemporaneous work does not require us to compare with it, we find it highly relevant to our work and believe that it deserves credit. We therefore cite it in the revised paper (§2) and discuss it below.

As explained in point 3 of our general response, PrivBoN is closely related to our soft-BoN defense (10). The two differ in how the noise level is chosen. PrivBoN derives it from a privacy budget and the sensitivity of the reward model, which gives its formal guarantee, whereas we choose it to reach a given retained gain (App. D) and do not bound the scores. If we nevertheless apply PrivBoN's accounting, with the sensitivity bounded by the range of r⋆r^\star's scores, our soft-BoN cells would correspond to ε\varepsilon between 16 and 117. This is a lower noise level than the budgets evaluated in [1], so Fig. 2 does not evaluate PrivBoN in its intended regime, and evaluating our attack there is a natural next step. We did not evaluate PrivITP, which privatizes a different selection rule (rejection sampling), and we leave it to future work.

Q2. How much does knowing the base model matter?

Thank you for this question. We ran this experiment for the rebuttal and added it to the paper (new App. E.8). In the second and third rows of the table below, only the attacker's K=7K = 7 references are replaced, and the shadow model's backbone is still Llama-3.1-8B-Instruct. In the last row, the shadow model's backbone is replaced as well, so that it is the same model that generates the references.

attacker's referencesshadow backbonecuratedgenerated
πref\pi_{\text{ref}} with the deployment's settings (temperature 1, top-pp 1), as in Table 1Llama-3.1-8B-Instruct0.754 ± 0.0510.829 ± 0.058
πref\pi_{\text{ref}} with temperature 0.7, top-pp 0.9Llama-3.1-8B-Instruct0.5400.368
Qwen3-8B (temperature 1, top-pp 1)Llama-3.1-8B-Instruct0.5430.527
Qwen3-8B (temperature 1, top-pp 1)Qwen3-8B0.5250.573

The first row is the five-run mean and standard deviation of Table 1, and the other three rows are one run each. Random guessing gives 0.5 on both the curated and the generated pairs.

With mismatched references (rows 2, 3 and 4), the L1 agreement falls to near or below the 0.5 of chance. We can therefore conclude that the L1 attack does rely on knowledge of the base model (the first assumption of our threat model, l.198–199) and of its sampling settings, which is a limitation of this attack that we now state in the paper (§6, l.480–482). The L1 setting nevertheless remains realistic when the BoN service is built on an open-weight model with default or documented sampling settings. The L2 and L3 attacks use no references, so they do not depend on this knowledge.

Our explanation is that, when the references are mismatched with the BoN winner y⋆y^\star, the style difference between the responses (in features such as length, vocabulary diversity and Markdown formatting) can be so pronounced that the shadow model can simply fit the L1 objective by memorizing this style difference, without really learning r⋆r^\star's preference. What the shadow model learns instead can even run against r⋆r^\star's preference, which makes an agreement below 0.5 possible, as in the 0.368 of the second row. In fact, prior work on reward modeling has noted that training on response pairs struggles to disentangle the preference from artifacts such as response length and format [2], and a proposed remedy is to train on pairs whose two responses differ along a specific attribute of interest [3].

[1] Jain, Bhattad, and Ray Chowdhury. Privacy Without Regret: Differentially Private Inference-Time Alignment. arXiv:2608.26324, 2026.

[2] Liu et al. RRM: Robust Reward Model Training Mitigates Reward Hacking. ICLR, 2025.

[3] Srivastava et al. Robust Reward Modeling via Causal Rubrics. ICLR, 2026.

Thanks for the helpful response and for the new experiment. Your clarifications cleared up my concerns, and I’ll keep my positive rating.

Reviewer_2HgPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Authors study distillation attacks on the reward model when best-of-N inference-time alignment is used. Authors study multiple threat models: L1: only winner is available, L2: ranking is available, L3: ranking as well as scores are available. Authors design attack methods for each of the threat model. Through empirical experiments majorly based on Skywork Reward models and preference data, authors show that even with conservative threat models and out-of-domain transfer are assumed, distilled models agree with the original reward model much more frequently than random. Two defense methods and their effectiveness are also empirically studied.

Critical strengths

Authors establish a novel connection between reward model privacy and best-of-N alignment. As both are relatively well studied area of research, the connection will lead to more cross-pollination between two areas, which would make this paper impactful. Authors also make good connections to the learning to rank literature, which is similarly another opportunity for impact.

Authors study multiple threat models, which are reasonably practical. The breadth of threat models enriches the understanding of the threat: readers get a very good sense of which piece of information is critical.

Critical weaknesses

Rather than differential privacy or membership inference privacy, authors focus on how well the reward model can be distilled. And I am not sure the distillation itself would be considered as privacy risk in practical scenarios. It is difficult to think of practical scenarios which the preference given pairs of responses is considered very private, although pairs of responses are accessible. In prior studies, differential privacy and membership inference privacy were studied because they connect to more realistic privacy risk, but I don't see a practical privacy risk in this setting.

The degree of technical novelty is not as strong as I expect for ICLR. The fact that an exposed reward model can be distilled is only mildly surprising for L1; for L2 and L3, it is quite straightforward. L2 and L3 are simply distillation of a classification model given labels and scores. L1 has a very nice connection to Bradley-Terry model, but then, the connection was already known by Pukdee et. al, so the authors' contribution is mostly making the connection to the distillation attack.

Questions for authors

Do L1 loss (8) and L2 loss (9) have a clear connection to BoN distribution as with L1, or are they simply heuristics? Using ListMLE and cross entropy loss makes practical sense, but it is unclear whether they have as strong foundation as L1's loss.

Isn't (11) known to be differentially private? How does differential privacy guarantees connect to the "preference stealing" setting authors study?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 2HgP (1/2)

We thank the reviewer for the careful reading and the clear questions.

W1. Is recovering the preferences a privacy risk, and when are the response pairs accessible while the preference is private?

We identify two risks, namely (i) recovering the preferences behind r⋆r^\star's training data and (ii) distilling r⋆r^\star itself (see point 1 of our general response). The reviewer's remark that we "focus on how well the reward model can be distilled" describes risk (ii) only. In fact, the scenario that the reviewer asks about (response pairs are public while the preference label is private) is the setting of our curated set, which we use to evaluate risk (i).

The curated set consists of triples (x,yc,yr)(x, y_c, y_r) taken from r⋆r^\star's training data, where xx is a prompt and ycy_c and yry_r are the chosen and the rejected response (§5.1, l.331–334). We use these triples to test whether the shadow model recovers the preference signal in the training data, which is risk (i). This setting is reasonable in practice. For example, Ray Chowdhury et al. [1] describe the same situation in differentially private reward learning. They adopt label differential privacy because "in most applications" the prompt and the two responses presented to the annotator are "public (or pre-collected)", while the annotator's preference "needs to be protected".

W2. Technical novelty.

The reviewer argues that the novelty of our work is limited for three reasons, and we respond to each of them below.

  1. "The fact that an exposed reward model can be distilled is only mildly surprising for L1". In the L1 setting the attacker cannot access r⋆r^\star directly. It can infer the reward signal only from the difference between the BoN winner and the reference responses that it sampled from πref\pi_{\text{ref}}. The attacker sees none of the scores that r⋆r^\star assigns, and it cannot control which candidate responses are passed to r⋆r^\star. These restrictions are much stricter than those of a standard distillation attack (§2, l.156–161). The fact that the L1 attacker still recovers up to 75% of the preferences on the curated set (Table 1) is therefore non-trivial.
  2. "L1 has a very nice connection to Bradley-Terry model, but then, the connection was already known by Pukdee et. al". Indeed, we do not claim this connection as our contribution, and we cite Pukdee et al. [2] (l.236–238) to make this clear. Our contribution is to identify a privacy risk that the community had not been aware of, namely that BoN leaks the preferences of its private reward model at inference time. Most prior work on preference privacy studies the training-time setting (l.83–86), and to the best of our knowledge, such a privacy risk in BoN had not been identified in the literature (l.87–89; see also point 4 of our general response).
  3. "for L2 and L3, it is quite straightforward. L2 and L3 are simply distillation of a classification model given labels and scores." We study L2 and L3 not to propose new attack methods for these two settings, but because different deployment scenarios of BoN give the attacker different levels of information about r⋆r^\star to train its shadow model (§4, l.201–203). We test several objectives for training the shadow model and report their evaluation results (App. B, App. E.2), which gives the community a quantitative measurement of the attack in the L2 and L3 settings.

[1] Ray Chowdhury, Zhou, and Natarajan. Differentially Private Reward Estimation with Preference Feedback. AISTATS, 2024.

[2] Pukdee, Balcan, and Ravikumar. Reward Learning from Best-of-N Preference Data: Targets, Tradeoffs, and Design Principles. arXiv:2605.30619, 2026.

AuthorsOpenReview ↗

Response to Reviewer 2HgP (2/2)

Q1. Do (8) and (9) have a clear connection to the BoN distribution, or are they heuristics?

We assume the question refers to the L2 loss (8) and the L3 loss (9). Both are principled in the same sense as the L1 loss. All three losses fit the same family of preference models, namely Bradley–Terry and its listwise extension Plackett–Luce, to what the attacker observes at each level.

For L2, the attacker observes how r⋆r^\star ranks the NN candidates. Loss (8) with uniform weights is the negative log-likelihood of this ranking under the Plackett–Luce model (§3), and the Bradley–Terry model behind the L1 loss is the N=2N = 2 case of the Plackett–Luce model. The connection to the BoN distribution lies in the first term of (8), which is the likelihood that the winner beats the other candidates. The BoN distribution (1) describes the same event, and the L1 loss models this event with fresh πref\pi_{\text{ref}} samples in place of the unobserved candidates (l.249–253). The remaining terms of (8) add the order among the other candidates (App. E.2).

For L3, the attacker observes r⋆r^\star's scores, so it can compute r⋆r^\star's own Bradley–Terry preference probability pij⋆p^\star_{ij} for every pair of candidates. Loss (9) is the Bradley–Terry loss (2) with pij⋆p^\star_{ij} as a soft label in place of the hard label. We also compare it with regressing directly on the scores (ℓMSE\ell_{\text{MSE}} in Table 3), which performs worse.

Q2. Is (11) differentially private, and how do DP guarantees connect to preference stealing?

We did not carry out a differential privacy analysis in this work, although the two defenses that we evaluate are closely related to standard DP mechanisms. As we note in §4.2 (l.290–293), (11) is related to the Report Noisy Max mechanism, which with Gaussian noise is differentially private when the scores are bounded (App. E of [1]), and soft-BoN (10) is closely related to PrivBoN [1] (see point 3 of our general response). Our defenses are only a first step toward mitigating the leak (§4.2, l.273–274). We compare them by the retained gain of BoN (§5.4) and do not calibrate the noise to a privacy budget.

A DP guarantee might not stop preference stealing, because the two concern different things. A DP guarantee protects each single comparison in r⋆r^\star's training data, in the sense that the served responses reveal little about each one of them. In our preference stealing attack, however, the attacker recovers the preferences that r⋆r^\star learned from all the comparisons together. Thus an inference-time alignment process with a DP guarantee does not necessarily prevent the attacker from recovering the preference signal which is depicted by all the comparisons in r⋆r^\star's training data.

Our experiments point the same way. Under the two defenses, the attack loses at most 6 points of agreement on the curated pairs (risk (i)) and at most 7 points on the generated pairs (risk (ii)), even when the deployment gives up 27% of BoN's gain (§5.4, App. E.6).

[1] Jain, Bhattad, and Ray Chowdhury. Privacy Without Regret: Differentially Private Inference-Time Alignment. arXiv:2608.26324, 2026.

The curated set consists of triples taken from's training data, where ... is a prompt and ... and ... are the chosen and the rejected response (§5.1, l.331–334). We use these triples to test whether the shadow model recovers the preference signal in the training data, which is risk (i). This setting is reasonable in practice. For example, Ray Chowdhury et al. [1] describe the same situation in differentially private reward learning. They adopt label differential privacy because "in most applications" the prompt and the two responses presented to the annotator are "public (or pre-collected)", while the annotator's preference "needs to be protected"

I was challenging exactly this setting- maybe my language wasn't very clear. I don't find it convincing that the response pair public yet the preference is so sensitive that we want it protected. In open source preference datasets, of course triplets are public, but the preference is rarely considered private. I cannot think of practical scenarios which this setting is important, hence it is difficult to establish the significance of the contribution. And I don't think citing Jain et al helps- in my opinion they make exactly the same mistake of not sufficiently explaining the consequence of the problem they are solving. If reward models are personal, it makes sense we want to protect people's value judgments, but authors only consider the setting which a universal reward model is trained.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers (1/2)

We thank all reviewers for their constructive feedback. We are encouraged that all three reviewers find the problem new, and that all three rate the soundness of the paper at 3. Below we summarize our answers to the concerns that several reviewers share. The details are in the individual replies.

1. Which privacy risk the paper studies (2HgP, SKsd). We identify two risks that BoN poses to its private reward model r⋆r^\star (§1, l.49–53 and l.88–90).

  • (i) An adversary can recover the preferences behind r⋆r^\star's training data, which harms the users and annotators who provided or labeled these data.

    We evaluate this attack on the curated pairs, which come from r⋆r^\star's own training data. Our experiments show that even the winner-only (L1) attacker, which is the strictest setting, recovers up to 75% of the preferences, far above the 50% of chance, and the L2 and L3 attackers recover 94% to 97% (Table 1).

  • (ii) An adversary can distill r⋆r^\star itself, which harms the RM's owner.

    We evaluate this attack on the generated pairs, whose responses are sampled from πref\pi_{\text{ref}} and are not in r⋆r^\star's training data. These pairs therefore test whether the shadow model behaves like r⋆r^\star on responses outside the training data. The L1 attacker agrees with r⋆r^\star on up to 85% of these pairs, and the L2 and L3 attackers agree on 96% to 99% (Table 1).

These results show that BoN serves as an exploitable side channel through which an attacker can steal the preferences of r⋆r^\star. Note that we do not claim membership inference. How an attacker could leverage its shadow model for membership inference is nevertheless an interesting problem, and we list it as future work (§6, l.475–477).

2. Blind baselines (SKsd). On all evaluation sets of Table 1, we evaluated three blind baselines that never use any output of BoN, namely the response length, and the mean log-probability and the summed log-probability of the response under πref\pi_{\text{ref}}.

The L2 and L3 shadow models exceed the best of these three blind baselines on every set, by 0.12 to 0.36. The L1 shadow model exceeds the best blind baseline on the three RMs trained on the Skywork preference data (Skywork-RM, URM-8B and Gemma-27B-RM), where L1 recovers 74% to 75% of the curated preferences against 58% to 60% for the best blind baseline. These leaks therefore cannot be explained by a blind heuristic.

On Tulu-RM and RB2-RM, and on the curated pairs of the math prompts, the L1 shadow model stays below the mean log-probability baseline. Tulu-RM and RB2-RM follow πref\pi_{\text{ref}}'s likelihood unusually closely, as this baseline alone agrees with these two RMs on 84% and 81% of the generated pairs and on 78% and 68% of the curated pairs. In these cases we therefore cannot tell whether the agreement of the L1 shadow model comes from having really learned r⋆r^\star's preferences, or simply because r⋆r^\star itself prefers the responses that πref\pi_{\text{ref}} finds more likely. The main claim of the paper nevertheless holds. On the three Skywork-trained RMs the L1 shadow model clearly exceeds the blind baselines, so even in the L1 setting, an attacker can still exploit BoN to recover the preferences behind r⋆r^\star's training data, with up to 75% accuracy.

AuthorsOpenReview ↗

General Response to All Reviewers (2/2)

3. Differential privacy and PrivBoN (2HgP, hzHY). We thank Reviewer hzHY for pointing us to PrivBoN [1], an excellent concurrent work. We now cite it in the revised paper (§2) and discuss how it relates to our defenses below, although PrivBoN appeared on arXiv on Aug. 26, 2026, and the ICLR reviewer guidelines do not require authors to compare with such contemporaneous work. PrivBoN is closely related to our soft-BoN defense (10). PrivBoN adds Gumbel noise to the reward scores and serves the response with the highest noisy score. By the Gumbel-max trick, this samples the served response from a softmax over the scores (the exponential mechanism [2]), which is also how soft-BoN selects its response. The difference is that PrivBoN calibrates the noise to a privacy budget and to the sensitivity of the reward model, which gives it a formal differential privacy guarantee. Our defenses, in contrast, are only a first step toward mitigating the leak (§4.2, l.273–274). We set the temperature of soft-BoN according to the retained gain and do not bound the scores, so soft-BoN carries no such guarantee.

The two works are complementary. PrivBoN gives a formal guarantee for every single training comparison, and our attack measures how much of the preferences that r⋆r^\star learned from the whole dataset can still be recovered from the served responses.

4. Novelty (all reviewers). The novelty concerns of all three reviewers are about the attack method. Reviewers 2HgP and hzHY note that our training objectives build on existing ones, namely the Bradley–Terry loss, ListMLE and cross-entropy, and Reviewer SKsd finds the setting similar to standard distillation. A new training objective, however, is not a contribution that we claim. As stated in §1, our contributions are to show that RM-based inference-time alignment creates an exploitable privacy side channel, to measure systematically how much BoN leaks at three exposure levels against five RMs, and to show that output perturbation does not close the leak unless the deployment gives up a substantial share of BoN's gain. The fact that a simpler attack works well actually strengthens our contributions, as it demonstrates that the privacy risk we identify is indeed practical and serious. The reviewers themselves list these contributions among the strengths of the paper.

Revisions (marked in blue). (a) New App. E.7 with the blind baselines. (b) §2: the citation of PrivBoN as concurrent work, and its relation to our defenses. (c) App. E.5: we corrected a wording error. The summed log-probability baseline favors shorter responses, not longer ones. (d) New App. E.8 with the L1 attack under mismatched references, and a sentence in §6 that states the resulting limitation of the L1 attack (Reviewer hzHY, Q2). (e) App. E.6: Table 12 now reports the five-seed results that Fig. 2 plots. (f) Minor wording fixes in the caption of Fig. 1 and in §6.

[1] Jain, Bhattad, and Ray Chowdhury. Privacy Without Regret: Differentially Private Inference-Time Alignment. arXiv:2608.26324, 2026.

[2] McSherry and Talwar. Mechanism Design via Differential Privacy. FOCS, 2007.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.