acceptodds
Under review as a conference paper at ICLR 2027

Privacy Without Regret: Differentially Private Inference-Time Alignment

Abstract

Best-of-N (BoN) sampling is the simplest and most widely deployed inference-time alignment strategy, but it suffers from two distinct problems: reward hacking, in which the selected response exploits errors in the proxy reward model, and the absence of any privacy protection for the sensitive human preference data used to train that reward model. We show that a single intervention – adding calibrated noise to reward scores before selection – resolves both. Our first result, Private Best-of-N (PrivBoN), establishes that Gumbel noise at an appropriate scale simultaneously provides -differential privacy and implements KL-regularized alignment. Whenever the privacy budget exceeds a critical threshold , the privacy-mandated noise is the regret-optimal regularization, and privacy imposes zero additional alignment cost – matching the information-theoretic skyline of huang2025bon. Because depends on an unknown coverage coefficient, we introduce Private Inference-Time Pessimism (PrivITP), which combines -regularized rejection sampling with a two-phase Gaussian mechanism. PrivITP achieves ex-post -DP with a privacy cost independent of the number of responses , cleanly decouples the regularization parameter from the privacy parameter, and attains the skyline up to a noise-inflation term. Experiments across several language models, datasets, and reward models confirm our results: PrivBoN and PrivITP are scaling-monotonic (unlike BoN, which degrades past a critical ), and PrivITP outperforms PrivBoN at equivalent privacy levels, with the largest gains in the strong-privacy regime.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Alignment algorithms, Differential privacy, Reward hacking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YYngOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies differentially private inference-time alignment. The main motivation is to address two challenges. The first is reward hacking, where selecting responses with high predicted rewards can lead to poor actual performance. The second is privacy leakage, where selected responses may reveal sensitive information from the reward model's training data.

The paper has three main contributions: (1) proposing PrivBoN (Algorithm 1), which adds Gumbel noise to reward scores and provides DP guarantees while achieving optimal regret rates; (2) introducing PrivITP (Algorithm 2), which uses χ2\chi^2-regularized rejection sampling and provides ex-post DP guarantees; and (3) evaluating both algorithms across different language models, reward models, and datasets, demonstrating improvements in reward-hacking mitigation and privacy–utility tradeoffs.

Critical strengths

The paper provides formal central differential privacy guarantees for Best-of-N sampling, a widely used inference-time alignment method. This is a valuable contribution. Moreover, the connection the paper establishes between privacy-preserving randomization and mitigating reward hacking is particularly interesting.

Critical weaknesses

The privacy guarantees of both PrivBoN and PrivITP algorithms depend on a global sensitivity Δr\Delta_r of the reward model (Theorems 3.1 and 4.1). Although the authors mentioned in Appendix G.3 that they bound Δr≤Rmax⁡\Delta_r\leq R_{\max}, it is unclear how this assumption is enforced for the neural reward models used in the experiments. In particular, if these models produce unbounded reward scores, clipping or normalization would be needed to guarantee the assumed bound. Moreover, using the worst-case bound (Rmax⁡R_{\max}) may require adding noise with substantial variance. This can potentially affect the privacy–utility tradeoff. Since the sensitivity directly determines the privacy guarantees of both algorithms, I believe this point needs further clarification to assess their practical applicability.

The paper claims that PrivBoN and PrivITP are scaling-monotonic, as stated in the abstract and Sections 3 and 4. However, Equations (2) and (5) only provide regret upper bounds that decrease with the number of candidates nn. A decreasing upper bound does not necessarily imply that the actual regret is non-increasing. Since avoiding performance degradation as nn grows is one of the main motivations of the paper, I believe the authors should clarify whether the claimed scaling monotonicity is theoretically guaranteed or only supported by the experimental results.

Questions for authors

How is the global reward-model sensitivity Δr\Delta_r estimated or bounded for the neural reward models used experimentally?

Could the authors clarify whether the scaling-monotonicity claim refers only to the regret upper bounds in Equations (2) and (5), or whether the actual regret of PrivBoN and PrivITP is theoretically guaranteed to be non-increasing as the number of candidates nn grows?

3 comments

We thank the reviewer for their feedback. We address the concerns below.

How Δr\Delta_r is bounded. The reviewer is right that Δr≤Rmax⁡\Delta_r \le R_{\max} is valid only if scores are actually confined to a known range. Every reward model in our experiments has a bounded output head with range [Rmin⁡,Rmax⁡][R_{\min}, R_{\max}], and we set Δr=Rmax⁡−Rmin⁡\Delta_r = R_{\max} - R_{\min}. For example, 8.79 for Oasst, 10.22 for Gemma, 9.34 for Llama. For a reward model without a bounded head, clipping its output to a fixed range before noise is added enforces the same bound by construction. Clipping is post-processing, so the analysis is unchanged.

We agree this bound is conservative and forces larger noise. It is, however, the only bound that can be certified when the reward model is frozen, which is the setting we consider. When the deployer does control training, tighter bounds are available via per-example gradient clipping (as in DP-SGD).

Is scaling monotonicity a guarantee or an empirical observation? The reviewer is correct, and we will revise the wording. Equations (2) and (5) are upper bounds, and a decreasing upper bound does not always imply that the true regret is non-increasing in nn. We do not claim pointwise monotonicity of the true regret theoretically; we show it only for the upper bounds. The pointwise monotone behavior reported in the paper is an empirical finding.

Thank you for your responses and clarifications.

I encourage the authors to incorporate these clarifications into the final version of the paper, particularly regarding how Δr\Delta_r is bounded in practice and how reward models without bounded output heads are handled, i.e., through clipping.

Thank you for your constructive feedback. We will add a Remark in the updated draft.

If any other point remains unclear, please let us know; we would be glad to clarify further during the discussion period. If you feel your concerns have been resolved, we would be grateful if you would consider updating your assessment.

Reviewer_CtWrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studied the problem of privacy in inference-time alignment, especially in the best-of-N method. The authors first proposed the PrivaBoN algorithm, which achieves pure differential privacy by adding Gumbel noise on reward before selection. Then they establish the theoretical regret upper bound and find the algorithm achieves DP and KL-regularized alignment simultaneously. In order to solve the dependence on the coverage coefficient in the algorithm, they proposed the PrivITP algorithm and provided theoretical regret analysis for it. Finally, they ran experiments on various language models, datasets, and reward models to verify their methods.

Critical strengths
  1. The writing is clear and organized.

  2. The problem of private inference time alignment is interesting and cutting-edge.

  3. The paper provides both theoretical guarantees and an experimental implementation. The theoretical side is solid.

Critical weaknesses
  1. The notation is confusing. The authors used  epsilon\ epsilon for privacy and then used it for error. It is confusing for readers. I suggest using other notation for error for distinguishing.

  2. The assumption is strong. The theoretical results under the reward error follow a Gaussian distribution. This might not be practical.

  3. There are no lower bounds to show the optimality of the regret bounds.

Questions for authors
  1. Why did you choose Gumbel noise to guarantee DP in Algorithm 1? Typically, people choose Laplacian noise to guarantee pure DP. How about changing Gumbel noise to Laplacian noise?

  2. Is it possible to provide a regret lower bound for your problem? If yes, give the proofs and results, or a proof sketch. If no, explain why.

  3. Why did you achieve an approximate DP in Algorithm 2 but achieve pure DP in Algorithm 1? What is the reason for the inconsistency?

5 comments
AuthorsOpenReview ↗

Response to Reviewer CtWr

We thank the reviewer for the feedback. We address the concerns below.

Gaussian error assumption. This assumption is milder than it appears, because:

  • The privacy guarantees (Theorems 3.1 and 4.1) make no distributional assumption on the reward error.
  • PrivITP's regret bound (Theorem 4.3) assumes only bounded mean-squared error, not Gaussianity.
  • PrivBoN's regret bound: the Gaussian assumption is used only to get the closed form of the threshold ε⋆(x)\varepsilon^\star(x). Under a general MSE assumption, the qualitative conclusions still hold (the bias–variance trade-off in σ\sigma, the existence of a privacy-is-free threshold, and the regret bound decreasing in nn), but the threshold loses its closed form (Appendix B.4). All results go through unchanged up to constants for Sub-Gaussian errors. We have already discussed this on page 5 in the main text.

Why Gumbel rather than Laplace noise. Laplace report-noisy-max mechanism is also ε\varepsilon-DP. We use Gumbel because of the selection distribution it induces. By the Gumbel-max trick, Gumbel-perturbed argmax is exactly softmax sampling, i.e., the exponential mechanism (Theorem 3.1). This yields the closed-form policy in Proposition 3.2, which converges to the KL-regularized optimum π∞∝π0exp⁡(r^/σ)\pi_\infty \propto \pi_0 \exp(\hat r/\sigma). That identity is what makes the privacy noise coincide with the regret-optimal regularization and gives the "privacy is free" threshold ε⋆\varepsilon^\star. Laplace argmax has no such closed form and does not converge to a regularized optimum, so the regret analysis would lose this structure.

That being said, we compare PrivBoN with Gumbel(0, σ\sigma) versus Laplace(0, bb) noise at equal noise scale (σ=bσ = b) for Armo-RM evaluated on the GSM8K dataset with Phi-3-Mini-Instruct base policy. We observe that both give almost similar Accuracy lift (%) over the base policy.

ScaleNoiseN=16N=64N=256N=1024
0.50Gumbel3.70 ± 0.823.83 ± 0.813.84 ± 0.813.85 ± 0.81
Laplace3.84 ± 0.813.85 ± 0.813.87 ± 0.813.86 ± 0.81
1.00Gumbel2.32 ± 0.842.42 ± 0.842.41 ± 0.842.42 ± 0.84
Laplace2.45 ± 0.842.42 ± 0.842.43 ± 0.842.42 ± 0.84

Lower bounds. The paper already states a matching lower bound: Proposition 2.1 (the regret skyline of Huang et al., 2025). For any inference-time alignment algorithm, private or not, there exist a true reward and a reward model with error εRM\varepsilon_{\mathrm{RM}} such that Reg≥14Cπ⋆εRM2\mathrm{Reg} \ge \frac14\sqrt{C^{\pi^\star}\varepsilon_{\mathrm{RM}}^2}. Our upper bounds match it up to constants in the large nn limit:

  • PrivBoN attains 3Cπ⋆εRM23\sqrt{C^{\pi^\star}\varepsilon_{\mathrm{RM}}^2} for ε≥ε⋆\varepsilon \ge \varepsilon^\star (Theoremm 3.4).
  • PrivITP attains 3Cπ⋆(εRM2+Rmax⁡σ)3\sqrt{C^{\pi^\star}(\varepsilon_{\mathrm{RM}}^2 + R_{\max}\sigma)} (Theorem 4.3).

So PrivBoN is optimal up to constants, provided the threshold is exceeded. For PrivITP, optimality depends on the additive noise-inflation term Rmax⁡σR_{\max}\sigma. We have discussed this after the respective Theorems.

Pure DP for PrivBoN versus approximate DP for PrivITP. The difference comes from Phase 1 of PrivITP. Phase 1 releases the normalizer λ\lambda through the Gaussian mechanism, which is (ε1,δ)(\varepsilon_1, \delta)-DP. Phase 2 is pure ex-post DP (Theorem 4.1). We chose Gaussian noise in Phase 1 to be consistent with the Gaussian above-threshold framework (Lebensold et al., 2024) used in Phase 2, and because the regret analysis uses Gaussian smoothing. If pure DP is preferred, replacing the Phase 1 Gaussian with Laplace noise of scale bb gives pure ε1\varepsilon_1-DP. We will add this as a remark.

Notation. We used εRM\varepsilon_{\mathrm{RM}} for reward model error and ε\varepsilon for privacy to keep them distinguished. In the final version, we will rename the reward-model error, for example, to ηRM\eta_{\mathrm{RM}}, and keep ε\varepsilon only for privacy.

We thank the reviewer for the constructive feedback. We have revised the manuscript with new results and clarifications, and we hope these, together with our responses, address your concerns. If any point remains unclear, please let us know; we would be glad to clarify further during the discussion period. If you feel your concerns have been resolved, we would be grateful if you would consider updating your assessment.

Thank you for the response. Proposition 2.1 has no dependence on ε\varepsilon. It therefore cannot certify whether any privacy-dependent term in your upper bounds is necessary. My original question concerned the cost of privacy, and that question remains open.

Reviewer_8ua2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies differentially private inference-time alignment for LLMs. The authors propose PrivBoN, which adds Gumbel noise to reward scores to provide DP and mitigate reward hacking, and PrivITP, which combines χ²-regularized rejection sampling with Gaussian noise to achieve ex-post DP guarantees. They establish regret bounds for both methods, and perform experiments to show the alignment performance.

Critical strengths
  • The paper studies differentially private inference-time alignment, which is an interesting and relatively underexplored problem.

  • The paper provides DP guarantees and regret bounds for both methods. The connection between privacy noise and alignment regularization is interesting.

  • They perform experiments on multiple language models and benchmarks, to show the effectiveness of the proposed methods.

Critical weaknesses
  • The experiments compare PrivBoN and PrivITP under the same noise scale instead of the same privacy budget. Since the two methods have different privacy guarantees, I recommend evaluating their performance under matched privacy budgets (ϵ,δ)(\epsilon, \delta) for a fair comparison.

  • I recommend including empirical comparisons with training-time DP approaches, such as privately training a reward model followed by standard BoN selection, to better clarify the advantages of inference-time DP.

  • The experiments are limited to relatively small language models. I recommend evaluating larger models to show the generalizability of the proposed methods.

  • I recommend including “Differentially Private Preference Data Synthesis for Large Language Model Alignment” (ICML 2026) in the literature review and discussing its relevance to privacy-preserving LLM alignment.

Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 8ua2

We thank the reviewer for the constructive review. We address each point below.

Matched privacy budgets. We have run the comparison at matched ε\varepsilon: we calibrate σ\sigma for PrivBoN and (σX,σZ)(\sigma_X, \sigma_Z) for PrivITP so that both meet that budget. We keep δ\delta fixed for PrivITP. We tabulate Accuracy lift (%) for PrivBoN and PrivITP over the base policy (Phi-3-Mini-Instruct) for Armo-RM evaluated on the GSM8K dataset.

ε\varepsilonMethodN=16N=16N=64N=64N=256N=256N=1024N=1024
18PrivBoN6.63 ± 0.786.80 ± 0.786.91 ± 0.796.88 ± 0.79
PrivITP5.25 ± 0.685.77 ± 0.785.82 ± 0.785.81 ± 0.78
8PrivBoN6.38 ± 0.776.58 ± 0.776.70 ± 0.786.72 ± 0.78
PrivITP4.30 ± 0.754.43 ± 0.804.43 ± 0.804.41 ± 0.80
5.5PrivBoN6.02 ± 0.786.20 ± 0.776.29 ± 0.776.29 ± 0.77
PrivITP3.44 ± 0.783.45 ± 0.823.43 ± 0.823.42 ± 0.82
4PrivBoN5.55 ± 0.785.69 ± 0.785.75 ± 0.785.76 ± 0.78
PrivITP2.82 ± 0.802.77 ± 0.832.76 ± 0.832.74 ± 0.83

We see that PrivBoN outperforms PrivITP across all budgets, and the gap widens as privacy tightens. Note that this doesn't dilute PrivITP's advantages over PrivBoN: decoupling privacy from reward-hacking mitigation and ex-post accounting, which lets FSRC answer ≈3×\approx 3\times more queries under a fixed total budget.

Comparison with training-time DP. A DP-trained reward model spends its budget once, and subsequent uses are post-processing. Our mechanisms spend per-query budget on an undegraded model. A single "matched ϵ\epsilon" therefore means different things for the two: a one-time training budget in one case, a per-query budget in the other. Any comparison has to fix a query volume and a DP-SGD configuration, and those choices would determine the outcome. Remark 3.6 shows the trade-off. DP training raises the reward-model error to ϵRMtrain(ϵ)>ϵRM\epsilon_{\text{RM}}^{\text{train}}(\epsilon) > \epsilon_{\text{RM}}, so every query pays the higher irreducible floor Cπ⋆ϵRMtrain(ε)\sqrt{C^{\pi^\star}}\epsilon_{\text{RM}}^{\mathrm{train}}(\varepsilon) (Proposition 2.1). Inference-time DP keeps the floor at Cπ⋆ϵRM\sqrt{C^{\pi^\star}}\epsilon_{\text{RM}}, at the cost of a finite query budget. Neither dominates. Training-time DP is preferable at high query volumes, where its one-time cost is amortized. Inference-time DP is preferable when the reward model is a frozen third-party checkpoint, where retraining is not an option. Since the two approaches are complementary rather than competing, each suited to a different deployment regime, we believe the comparison above characterizes their relationship more faithfully than empirical results.

Larger language models. Our methods work within the sample-and-evaluate framework, so they need no model internals, and the theory has no size dependence either. The current evaluation covers three base policies (Phi-3-Mini 3.8B, Gemma-2-2B, and Llama-3.2-3B) and reward models ranging from 1.4B to 8B.

We add Qwen2.5-7B-Instruct as a base policy on MATH-500 with Armo-RM and report the accuracy lift (%) of each method over the base policy. PrivBoN uses σ=0.75\sigma = 0.75. PrivITP uses the same total noise, split evenly across phases (σX=σZ=0.375\sigma_X = \sigma_Z = 0.375). ITP and PrivITP both use β=0.005\beta = 0.005. The results show the same qualitative pattern as the smaller models in the paper.

MethodN=8N=16N=32N=64
BoN2.30 ± 1.572.55 ± 1.623.05 ± 1.664.15 ± 1.78
ITP2.30 ± 1.572.55 ± 1.622.89 ± 1.653.22 ± 1.69
PrivBoN1.10 ± 1.541.24 ± 1.541.23 ± 1.541.24 ± 1.54
PrivITP2.33 ± 1.202.65 ± 1.352.52 ± 1.472.26 ± 1.53

Related work. Thank you for pointing us to the paper. It protects the preference data at the data level by releasing a DP synthetic dataset for training. That places it on the training-time side of the trade-off discussed in Remark 3.6, and it complements our inference-time mechanism. We will add it to the related work.

We thank the reviewer for the constructive feedback. We have revised the manuscript with new results and clarifications, and we hope these, together with our responses, address your concerns. If any point remains unclear, please let us know; we would be glad to clarify further during the discussion period. If you feel your concerns have been resolved, we would be grateful if you would consider updating your assessment.

Thank you for the response. The new results and clarifications have addressed most of my concerns. I will maintain my rating of 3 (Weak Accept), which reflects my positive assessment of the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Changes in Revised Draft

We thank all reviewers for their constructive feedback and have revised the paper accordingly, using the additional page allowed for the rebuttal. The main changes are:

  • PrivBoN and PrivITP are now compared at matched privacy budgets. Results are given in Section 5, Table 2.
  • New results with Qwen2.5-7B-Instruct as the base policy on MATH-500 are added in Appendix G.4, Tables 8,9.
  • The bound on Δr\Delta_r (bounded head range, or clipping) is mentioned in Remark 4.5. Values for different RMs are given in Section 5.
  • Remark 3.3 is added on the choice of Gumbel over Laplace noise, and comparison results are given in Appendix G.5, Table 10.
  • The limited role of the Gaussian-error assumption is expanded in Remark 3.5.
  • A reference to DP preference-data synthesis (Gao and Yang, 2026) is added in Section 1.
  • A conclusion section is added.
  • A regret lower bound showing the cost of privacy is added in Appendix H.

We welcome any further questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.