- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies differentially private inference-time alignment. The main motivation is to address two challenges. The first is reward hacking, where selecting responses with high predicted rewards can lead to poor actual performance. The second is privacy leakage, where selected responses may reveal sensitive information from the reward model's training data.
The paper has three main contributions: (1) proposing PrivBoN (Algorithm 1), which adds Gumbel noise to reward scores and provides DP guarantees while achieving optimal regret rates; (2) introducing PrivITP (Algorithm 2), which uses -regularized rejection sampling and provides ex-post DP guarantees; and (3) evaluating both algorithms across different language models, reward models, and datasets, demonstrating improvements in reward-hacking mitigation and privacy–utility tradeoffs.
Critical strengths
The paper provides formal central differential privacy guarantees for Best-of-N sampling, a widely used inference-time alignment method. This is a valuable contribution. Moreover, the connection the paper establishes between privacy-preserving randomization and mitigating reward hacking is particularly interesting.
Critical weaknesses
The privacy guarantees of both PrivBoN and PrivITP algorithms depend on a global sensitivity of the reward model (Theorems 3.1 and 4.1). Although the authors mentioned in Appendix G.3 that they bound , it is unclear how this assumption is enforced for the neural reward models used in the experiments. In particular, if these models produce unbounded reward scores, clipping or normalization would be needed to guarantee the assumed bound. Moreover, using the worst-case bound () may require adding noise with substantial variance. This can potentially affect the privacy–utility tradeoff. Since the sensitivity directly determines the privacy guarantees of both algorithms, I believe this point needs further clarification to assess their practical applicability.
The paper claims that PrivBoN and PrivITP are scaling-monotonic, as stated in the abstract and Sections 3 and 4. However, Equations (2) and (5) only provide regret upper bounds that decrease with the number of candidates . A decreasing upper bound does not necessarily imply that the actual regret is non-increasing. Since avoiding performance degradation as grows is one of the main motivations of the paper, I believe the authors should clarify whether the claimed scaling monotonicity is theoretically guaranteed or only supported by the experimental results.
Questions for authors
How is the global reward-model sensitivity estimated or bounded for the neural reward models used experimentally?
Could the authors clarify whether the scaling-monotonicity claim refers only to the regret upper bounds in Equations (2) and (5), or whether the actual regret of PrivBoN and PrivITP is theoretically guaranteed to be non-increasing as the number of candidates grows?
3 comments
Thank you for your responses and clarifications.
I encourage the authors to incorporate these clarifications into the final version of the paper, particularly regarding how is bounded in practice and how reward models without bounded output heads are handled, i.e., through clipping.
Thank you for your constructive feedback. We will add a Remark in the updated draft.
If any other point remains unclear, please let us know; we would be glad to clarify further during the discussion period. If you feel your concerns have been resolved, we would be grateful if you would consider updating your assessment.
We thank the reviewer for their feedback. We address the concerns below.
How Δr is bounded. The reviewer is right that Δr≤Rmax is valid only if scores are actually confined to a known range. Every reward model in our experiments has a bounded output head with range [Rmin,Rmax], and we set Δr=Rmax−Rmin. For example, 8.79 for Oasst, 10.22 for Gemma, 9.34 for Llama. For a reward model without a bounded head, clipping its output to a fixed range before noise is added enforces the same bound by construction. Clipping is post-processing, so the analysis is unchanged.
We agree this bound is conservative and forces larger noise. It is, however, the only bound that can be certified when the reward model is frozen, which is the setting we consider. When the deployer does control training, tighter bounds are available via per-example gradient clipping (as in DP-SGD).
Is scaling monotonicity a guarantee or an empirical observation? The reviewer is correct, and we will revise the wording. Equations (2) and (5) are upper bounds, and a decreasing upper bound does not always imply that the true regret is non-increasing in n. We do not claim pointwise monotonicity of the true regret theoretically; we show it only for the upper bounds. The pointwise monotone behavior reported in the paper is an empirical finding.