- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies differentially private RLHF under linear reward models and Bradley-Terry/Plackett-Luce preference assumptions. The authors propose private estimation of reward parameters and feature covariance, followed by pessimistic policy optimization. The main claim is that privacy can be achieved while retaining the statistical efficiency of non-private RLHF. The analysis covers pairwise comparisons, K-wise rankings, and trajectory-level preferences in finite-horizon MDPs.
Critical strengths
- The paper addresses a relevant privacy problem in RLHF by considering protection of prompts, model responses, and preference rankings together.
- Combining private reward estimation with private covariance estimation is useful because the resulting quantities can support subsequent policy optimization through DP post-processing.
- The paper provides a theoretically motivated treatment of pairwise preferences and extends the framework to K-wise rankings and trajectory comparisons.
- The comparison with the non-private results provides a clear reference point for evaluating statistical efficiency.
Critical weaknesses
- K-wise comparison analysis: The gradient and Hessian expressions in Appendix B.1 appear inconsistent with the Plackett-Luce likelihood. In particular, the gradient in Claim B.1 does not recover the standard Bradley-Terry gradient when K=2. Since these expressions support the subsequent analysis, their correctness needs to be established.
- General MDP extension: The trajectory covariance in Section 5 is defined using a sum of stepwise outer products, whereas the preference likelihood depends on the sum of feature differences across a trajectory. These expressions are generally different because the former excludes cross terms. This raises a concern about the curvature argument and Theorem 5.2.
- Private covariance estimation: Algorithm 2 adds independent Gaussian noise to the covariance matrix without explicitly ensuring symmetry. Since the subsequent analysis uses the perturbed matrix to define norms and confidence sets, this requires clarification.
- Scope of the privacy guarantee: The analysis relies on known bounded features and linear reward models. Moreover, the guarantee is record-level rather than user-level, which limits its direct applicability when an individual contributes multiple preference records.
- Lack of empirical validation: The paper provides theoretical guarantees but no experimental results to evaluate the proposed approach. Even a small synthetic experiment comparing private and non-private RLHF across different privacy budgets, sample sizes, and feature dimensions would help demonstrate the finite-sample behavior and practical utility of the proposed method.
Questions for authors
- How is the trajectory covariance in Section 5 consistent with the aggregated feature differences used in the Bradley-Terry likelihood? Does the performance bound in Theorem 5.2 remain valid under the stated definition?
- Is the Gaussian noise matrix in Algorithm 2 intended to be symmetric? If so, how is the noise calibrated to maintain the stated privacy guarantee?
- Could the authors clarify the conditions under which the privacy-dependent error term is dominated by the non-private term, particularly when the dimension is fixed?
3 comments
Thank you for the detailed response and clarifications. I appreciate the authors acknowledging the errors in the K-wise derivations, trajectory covariance definition, and covariance perturbation, and providing the corresponding corrections. These clarifications are helpful. However, I would still like the authors to add corrected derivations fully incorporated into the paper, particularly to verify that the main theoretical guarantees remain unchanged. The lack of empirical validation also leaves some questions about the finite-sample privacy–utility trade-off.
Overall, while the response addresses several of my concerns, I believe some issues remain open. Therefore, I will maintain my original rating of weak rejection.
Thank you for your response. We have now updated the paper with these corrections. As discussed before, the theoretical guarantees remain the same.
We are glad to see that you found that our paper addresses a relevant privacy problem in RLHF that considers protection of prompts, model responses, and preference rankings together, provides a theoretically motivated treatment of pairwise preferences, extends the framework to K-wise rankings and trajectory comparisons, and the comparison with the non-private results provides a clear reference point for evaluating statistical efficiency.
This is a typo in Section 5. The correct covariance is n1∑i(∑hΔϕih)(∑hΔϕih)⊤, which is what is used by the proof. The performance bound of Theorem 5.2 is correct as stated with this corrected covariance.
The matrix in Algorithm 2 indeed must be symmetric, thank you for pointing this out. All that we require for the stated privacy guarantee is standard concentration of measure bounds which hold just as well for symmetric Gaussian noise as they do for asymmetric noise.
The non-private term dominates when d>ϵ2log(1/δ). Typically ϵ is some fixed constant, and δ is inverse polynomial in n, thus this just requires d to be larger than O(logn/ϵ2).
Thank you for pointing this out, there was an index error in calculating these terms. We give the corrected gradient and Hessian below. This calculation error does not change any of the results in the paper, because the bounds of Lemma B.3 and B.4 remain valid after the corrections.
The corrected gradient is given by:
∇lDK(θ)=−n1∑i=1n∑j=1K∑k=jK(∑m=jKexp(⟨θ,ϕ(si,aσi(m)i)⟩)exp(⟨θ,ϕ(si,aσi(k)i)⟩))⋅(ϕ(si,aσi(j)i)−ϕ(si,aσi(k)i))
The corrected Hessian is given by:
∇2lDK(θ)=2n1∑i=1n∑j=1K∑k=jK∑l=jK((∑m=jKexp(⟨θ,ϕ(si,aσi(m)i)⟩))2exp(⟨θ,ϕ(si,aσi(k)i)⟩)exp(⟨θ,ϕ(si,aσi(l)i)⟩))⋅xklixkli⊤