acceptodds
Under review as a conference paper at ICLR 2027

Fully Private AI Alignment to Human Values

Abstract

Reinforcement learning from human feedback is the preeminent method for aligning powerful AI systems with human preferences and values. Training data for RLHF typically consists of a prompt taken from a conversation with a human along with multiple AI responses, ranked according to the human’s preferences. Data of this form is collected both from professional raters, as well as from everyday users of publicly available AI chatbots, thus presenting a serious privacy risk if used for training. In this paper we study the problem of alignment from human preferences while preventing privacy leaks of human-written training data. We design algorithms that provably achieve -differential privacy in the standard Bradley-Terry and Plackett-Luce models that fully privatize the entire interaction between human and AI. Furthermore, our algorithms achieve asymptotic statistical efficiency that matches the minimax optimal non-private RLHF algorithms in this setting. These results demonstrate that, in a standard theoretical setup, there is no tension between maintaining privacy and efficient alignment training.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
fully private, AI alignment, human values

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_d7p2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies differentially private RLHF under linear reward models and Bradley-Terry/Plackett-Luce preference assumptions. The authors propose private estimation of reward parameters and feature covariance, followed by pessimistic policy optimization. The main claim is that privacy can be achieved while retaining the statistical efficiency of non-private RLHF. The analysis covers pairwise comparisons, K-wise rankings, and trajectory-level preferences in finite-horizon MDPs.

Critical strengths
  1. The paper addresses a relevant privacy problem in RLHF by considering protection of prompts, model responses, and preference rankings together.
  2. Combining private reward estimation with private covariance estimation is useful because the resulting quantities can support subsequent policy optimization through DP post-processing.
  3. The paper provides a theoretically motivated treatment of pairwise preferences and extends the framework to K-wise rankings and trajectory comparisons.
  4. The comparison with the non-private results provides a clear reference point for evaluating statistical efficiency.
Critical weaknesses
  1. K-wise comparison analysis: The gradient and Hessian expressions in Appendix B.1 appear inconsistent with the Plackett-Luce likelihood. In particular, the gradient in Claim B.1 does not recover the standard Bradley-Terry gradient when K=2. Since these expressions support the subsequent analysis, their correctness needs to be established.
  2. General MDP extension: The trajectory covariance in Section 5 is defined using a sum of stepwise outer products, whereas the preference likelihood depends on the sum of feature differences across a trajectory. These expressions are generally different because the former excludes cross terms. This raises a concern about the curvature argument and Theorem 5.2.
  3. Private covariance estimation: Algorithm 2 adds independent Gaussian noise to the covariance matrix without explicitly ensuring symmetry. Since the subsequent analysis uses the perturbed matrix to define norms and confidence sets, this requires clarification.
  4. Scope of the privacy guarantee: The analysis relies on known bounded features and linear reward models. Moreover, the guarantee is record-level rather than user-level, which limits its direct applicability when an individual contributes multiple preference records.
  5. Lack of empirical validation: The paper provides theoretical guarantees but no experimental results to evaluate the proposed approach. Even a small synthetic experiment comparing private and non-private RLHF across different privacy budgets, sample sizes, and feature dimensions would help demonstrate the finite-sample behavior and practical utility of the proposed method.
Questions for authors
  1. How is the trajectory covariance in Section 5 consistent with the aggregated feature differences used in the Bradley-Terry likelihood? Does the performance bound in Theorem 5.2 remain valid under the stated definition?
  2. Is the Gaussian noise matrix in Algorithm 2 intended to be symmetric? If so, how is the noise calibrated to maintain the stated privacy guarantee?
  3. Could the authors clarify the conditions under which the privacy-dependent error term is dominated by the non-private term, particularly when the dimension is fixed?
3 comments

We are glad to see that you found that our paper addresses a relevant privacy problem in RLHF that considers protection of prompts, model responses, and preference rankings together, provides a theoretically motivated treatment of pairwise preferences, extends the framework to K-wise rankings and trajectory comparisons, and the comparison with the non-private results provides a clear reference point for evaluating statistical efficiency.

  1. ”How is the trajectory covariance in Section 5 consistent with the aggregated feature differences used in the Bradley-Terry likelihood? Does the performance bound in Theorem 5.2 remain valid under the stated definition?”

This is a typo in Section 5. The correct covariance is 1n∑i(∑hΔϕih)(∑hΔϕih)⊤\frac1n\sum_i(\sum_h\Delta\phi_{ih})(\sum_h\Delta\phi_{ih})^{\top}, which is what is used by the proof. The performance bound of Theorem 5.2 is correct as stated with this corrected covariance.

  1. Is the Gaussian noise matrix in Algorithm 2 intended to be symmetric? If so, how is the noise calibrated to maintain the stated privacy guarantee?

The matrix in Algorithm 2 indeed must be symmetric, thank you for pointing this out. All that we require for the stated privacy guarantee is standard concentration of measure bounds which hold just as well for symmetric Gaussian noise as they do for asymmetric noise.

  1. Could the authors clarify the conditions under which the privacy-dependent error term is dominated by the non-private term, particularly when the dimension is fixed?

The non-private term dominates when d>log⁡(1/δ)ϵ2d > \frac{\log(1/\delta)}{\epsilon^2}. Typically ϵ\epsilon is some fixed constant, and δ\delta is inverse polynomial in nn, thus this just requires dd to be larger than O(log⁡n/ϵ2)O(\log n/\epsilon^2).

  1. “K-wise comparison analysis: The gradient and Hessian expressions in Appendix B.1 appear inconsistent with the Plackett-Luce likelihood.”

Thank you for pointing this out, there was an index error in calculating these terms. We give the corrected gradient and Hessian below. This calculation error does not change any of the results in the paper, because the bounds of Lemma B.3 and B.4 remain valid after the corrections.

The corrected gradient is given by:

∇lDK(θ)=−1n∑i=1n∑j=1K∑k=jK(exp⁡(⟨θ,ϕ(si,aσi(k)i)⟩)∑m=jKexp⁡(⟨θ,ϕ(si,aσi(m)i)⟩))⋅(ϕ(si,aσi(j)i)−ϕ(si,aσi(k)i))\nabla l_{\mathcal{D}_K}(\theta) = -\frac{1}{n} \sum_{i=1}^n \sum_{j=1}^K \sum_{k=j}^K \left( \frac{\exp(\langle \theta, \phi(s^i, a_{\sigma_i(k)}^i) \rangle)}{\sum_{m=j}^K \exp(\langle \theta, \phi(s^i, a_{\sigma_i(m)}^i) \rangle)} \right) \cdot \Big( \phi(s^i, a_{\sigma_i(j)}^i) - \phi(s^i, a_{\sigma_i(k)}^i) \Big)

The corrected Hessian is given by:

∇2lDK(θ)=12n∑i=1n∑j=1K∑k=jK∑l=jK(exp⁡(⟨θ,ϕ(si,aσi(k)i)⟩)exp⁡(⟨θ,ϕ(si,aσi(l)i)⟩)(∑m=jKexp⁡(⟨θ,ϕ(si,aσi(m)i)⟩))2)⋅xklixkli⊤\nabla^2 l_{\mathcal{D}_K}(\theta) = \frac{1}{2n} \sum_{i=1}^n \sum_{j=1}^K \sum_{k=j}^K \sum_{l=j}^K \left( \frac{\exp(\langle \theta, \phi(s^i, a_{\sigma_i(k)}^i) \rangle) \exp(\langle \theta, \phi(s^i, a_{\sigma_i(l)}^i) \rangle)}{\left( \sum_{m=j}^K \exp(\langle \theta, \phi(s^i, a_{\sigma_i(m)}^i) \rangle) \right)^2} \right) \cdot x_{kl}^i {x_{kl}^i}^\top

Thank you for the detailed response and clarifications. I appreciate the authors acknowledging the errors in the K-wise derivations, trajectory covariance definition, and covariance perturbation, and providing the corresponding corrections. These clarifications are helpful. However, I would still like the authors to add corrected derivations fully incorporated into the paper, particularly to verify that the main theoretical guarantees remain unchanged. The lack of empirical validation also leaves some questions about the finite-sample privacy–utility trade-off.

Overall, while the response addresses several of my concerns, I believe some issues remain open. Therefore, I will maintain my original rating of weak rejection.

Thank you for your response. We have now updated the paper with these corrections. As discussed before, the theoretical guarantees remain the same.

Reviewer_xHDgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies differentially private reinforcement learning from human feedback (RLHF), aiming to protect the entire preference record, including prompts, responses, and human rankings. The authors propose a framework that combines objective-perturbed maximum likelihood estimation, private covariance estimation, and pessimistic policy optimization. Under linear reward assumptions and Bradley–Terry/Plackett–Luce preference models, they establish differential privacy guarantees and policy suboptimality bounds that asymptotically match the minimax-optimal non-private rates. The theoretical results cover both contextual bandits and general MDPs.

Critical strengths
  1. The paper addresses a privacy problem in RLHF by considering the complete preference record rather than only the preference labels. The combination of private reward estimation and private covariance estimation allows subsequent policy optimization to operate on privatized quantities, providing a modular framework for differentially private RLHF.

  2. The main theoretical contribution is the derivation of policy suboptimality bounds that asymptotically match the minimax-optimal non-private rates under the stated assumptions. The analysis goes beyond establishing differential privacy and explicitly studies the impact of privacy on policy performance.

Critical weaknesses
  1. Standard DP-SGD can also provide record-level differential privacy over the entire preference tuple, including prompts, responses, and preference labels. Therefore, full-record privacy alone does not distinguish the proposed method from existing DP training approaches. While the paper establishes theoretical utility guarantees using objective perturbation, it does not sufficiently clarify whether comparable guarantees could be obtained using DP-SGD-based reward estimation combined with private covariance estimation. This makes it difficult to assess the necessity and relative advantages of the proposed approach.

  2. The analysis relies on a known, fixed feature representation, linearly parameterized rewards, and specific probabilistic preference models. These assumptions substantially simplify reward modeling compared with practical LLM alignment, where both representations and reward functions are often nonlinear. Moreover, the paper provides no empirical evaluation, even on synthetic contextual bandit problems. Consequently, the finite-sample privacy–utility trade-off and practical advantages of the proposed algorithms remain unclear.

Questions for authors
  1. Since DP-SGD can also protect complete preference records, could the authors clarify the advantages of objective perturbation over DP-SGD under the same linear reward assumptions? In particular, can a DP-SGD-based reward estimator achieve comparable policy suboptimality guarantees when combined with private covariance estimation and pessimistic policy optimization?

  2. The theoretical results establish asymptotically optimal rates under specific assumptions. However, the additional privacy-dependent error term may remain significant at practical sample sizes. Could the authors provide numerical or synthetic evaluations comparing private and non-private algorithms across different values of the privacy budget, sample size, and feature dimension?

  3. The analysis assumes a fixed feature representation and a linear reward function. How essential are these assumptions to the privacy–utility guarantees? In particular, which components of the analysis would fail or require substantial modification when extending the method to nonlinear reward models?

3 comments

We are glad to hear that you found that our paper addresses the privacy problem in RLHF by considering the complete preference record rather than only the preference labels, and that our main theoretical contribution goes beyond establishing differential privacy and shows that performance asymptotically match the minimax-optimal non-private rates. We also appreciate you valuing presentation, soundness and quality of our paper highly.

  1. ”Since DP-SGD can also protect complete preference records, could the authors clarify the advantages of objective perturbation over DP-SGD under the same linear reward assumptions? In particular, can a DP-SGD-based reward estimator achieve comparable policy suboptimality guarantees when combined with private covariance estimation and pessimistic policy optimization?”

Our analysis of objective perturbation applies to any optimization algorithm that returns an approximate solution to the regularized loss function. This provides maximum flexibility, allowing any reward-learning optimization algorithm to be plugged in, and immediately used.

In contrast DP-SGD is a specific optimization algorithm that requires its own tuning, and may not be appropriate for the particular setting. For example, the level of noise added in each step may lead to a significantly larger number of total steps needed for the algorithm to reach convergence. It is possible that one could use DP-SGD on the regularized objective function to obtain similar private reward estimation guarantees, though this would require careful analysis of how the various approximation errors interact in our method, as well as an evaluation of how this affects total number of steps and computational cost.

  1. ”The theoretical results establish asymptotically optimal rates under specific assumptions. However, the additional privacy-dependent error term may remain significant at practical sample sizes. Could the authors provide numerical or synthetic evaluations comparing private and non-private algorithms across different values of the privacy budget, sample size, and feature dimension?”

In the differential privacy field this is the standard theoretical benchmark and asymptotically optimal guarantees are the gold standard for showing how privacy can be achieved for important learning problems, and this is exactly what we prove for our algorithms. This approach has been tremendously successful over the past decade, and has led to the development of many practical privacy-preserving learning algorithms.

  1. ”The analysis assumes a fixed feature representation and a linear reward function. How essential are these assumptions to the privacy–utility guarantees? In particular, which components of the analysis would fail or require substantial modification when extending the method to nonlinear reward models?”

We follow the vast majority of prior theoretical work in using a linear feature representation so that interpretable provable guarantees on sample complexity are possible. We view these results as providing qualitative theoretical evidence that can be used to inform design choices for full RLHF with LLMs. In particular, our results use the feature covariance matrix of the dataset to construct the appropriate confidence sets for pessimistic policy optimization. The use of the data covariance matrix is specific to the linear feature map setting. However, our algorithm gives a clear qualitative picture of how to extend these results beyond this setting: the key is to construct a confidence set for pessimistic policy optimization in a differentially private way. Training heuristic estimators for the confidence set with DP-SGD, for example, could be used in standard RLHF pipelines for LLMs.

Thanks for the author's response. Since you mentioned that DPSGD may lead to significantly slower convergence, does the paper provide theoretical and empirical evaluations to support your claim? The concern in my second point is about empirical validation of the proposed method, rather than theoretical analysis. I am not sure how the response, "In the differential privacy field this is the standard theoretical benchmark, ..., This approach has been tremendously successful over the past decade...", helps address this concern. Overall, the raised issues remain unsolved.

  1. “Since you mentioned that DPSGD may lead to significantly slower convergence, does the paper provide theoretical and empirical evaluations to support your claim?”

Prior work [1] has already shown that DP with objective perturbation can be achieved in time O(nlog⁡n)O(n\log n), while DP-SGD requires O(n2)O(n^2) time in the same setting.

[1] Improving the privacy and practicality of objective perturbation for differentially private linear learners. NeurIPS 2023.

  1. ”The concern in my second point is about empirical validation of the proposed method, rather than theoretical analysis. “

As noted in the ICLR reviewer guide, submissions bring value when they demonstrate new, impactful theoretical knowledge, and claims should be evaluated based on the rigor of the mathematical proofs rather than the presence of empirical benchmarks.

Furthermore, the theoretical foundations of privacy is a major, well-established machine learning field where purely theoretical contributions are studied, published and highlighted at top venues [1,2,3,4,5,6].

[1] Improved sample complexity for private non-smooth non-convex optimization. ICML 2025.

[2] Purifying approximate differential privacy with randomized post-processing. NeurIPS 2025. [Spotlight Presentation]

[3] Differentially private reinforcement learning with self-play.NeurIPS 2024.

[4] Privacy amplification by random allocation. NeurIPS 2025. [Spotlight Presentation]

[5] Private mechanism design via quantile estimation. ICLR 2025.

[6] Instance-optimality for private KL distribution estimation. NeurIPS 2025. [Spotlight Presentation]

Reviewer_AdbPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies privacy-preserving RLHF. It assumes fixed features and linear rewards. It combines objective-perturbed reward estimation, private covariance estimation, and pessimistic policy optimization. It protects full preference records, including prompts, responses, and rankings. The authors provide privacy and policy-error guarantees for contextual bandits and finite-horizon MDPs. They claim learning rates comparable to non-private methods under suitable conditions. The paper provides theoretical proofs but no experiments.

Critical strengths
  1. The paper addresses an important privacy problem in RLHF. It aims to protect full preference records, including prompts and responses, rather than only preference labels.
  2. The approach is modular. It privatizes the reward estimate and data covariance. Later policy optimization can use these outputs without accessing the private records. This separates privacy protection from policy learning.
Critical weaknesses
  1. The optimization accuracy condition is insufficient as stated. Algorithm 1 allows any β<1/n\beta<1/n. With fixed d,ϵ,ηd,\epsilon,\eta and δ=n−2\delta=n^{-2}, choosing β=c/n\beta=c/n gives ν2=Θ(log⁡n)\nu^2=\Theta(\sqrt{\log n}). The output noise does not vanish. This conflicts with Lemma A.11 in nondegenerate feature directions. A stricter condition on β\beta is needed to support the error guarantee.
  2. The trajectory covariance is inconsistent with the curvature analysis. Section 5 uses 1n∑i∑hΔϕihΔϕih⊤\frac1n\sum_i\sum_h\Delta\phi_{ih}\Delta\phi_{ih}^{\top}. The BTL Hessian instead involves 1n∑i(∑hΔϕih)(∑hΔϕih)⊤\frac1n\sum_i(\sum_h\Delta\phi_{ih})(\sum_h\Delta\phi_{ih})^{\top}. Per-step differences can cancel, so bound (21) fails under the stated definition. This may be a definition error, but the algorithm and proof must be made consistent to establish the MDP guarantee.
Questions for authors

Q1: What condition on β\beta is needed for Lemma A.11? Does choosing β=O(n−2)\beta=O(n^{-2}) resolve the output-noise issue and preserve the main performance bounds?

Q2: Could you clarify the intended definition of the trajectory covariance? Section 5 uses 1n∑i∑hΔϕihΔϕih⊤\frac1n\sum_i\sum_h\Delta\phi_{ih}\Delta\phi_{ih}^{\top}, while Appendix C appears to use 1n∑i(∑hΔϕih)(∑hΔϕih)⊤\frac1n\sum_i(\sum_h\Delta\phi_{ih})(\sum_h\Delta\phi_{ih})^{\top}. Is this a notation error? Would using the latter definition leave Theorem 5.2 unchanged?

1 comment

We are glad to hear that you found our paper addressing an important privacy problem in RLHF that aims to protect full preference records, including prompts and responses, rather than only preference labels, and our approach is modular and separates privacy protection from policy learning.

  1. “What condition on β\beta is needed for Lemma A.11? Does choosing β=O(n−2)\beta = O(n^{-2}) resolve the output-noise issue and preserve the main performance bounds?”

This is a typo, thank you for pointing this out. It should be β<1/n2\beta < 1/n^2 in Algorithm 1. This is all that is required for the results to go through.

  1. ”Could you clarify the intended definition of the trajectory covariance? Section 5 uses 1n∑i∑hΔϕihΔϕih⊤\frac1n\sum_i\sum_h\Delta\phi_{ih}\Delta\phi_{ih}^{\top}, while Appendix C appears to use 1n∑i(∑hΔϕih)(∑hΔϕih)⊤\frac1n\sum_i(\sum_h\Delta\phi_{ih})(\sum_h\Delta\phi_{ih})^{\top}. Is this a notation error? Would using the latter definition leave Theorem 5.2 unchanged?”

This is a notation error. The version in Appendix C is the correct one and it leaves Theorem 5.2 unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.