acceptodds
Under review as a conference paper at ICLR 2027

Population-Relative RLHF for LLM Output Diversity: A Bi-Level Optimization Approach

Abstract

We study the problem of aligning large language models (LLMs) when output diversity is critical alongside output quality. Standard alignment methods (implicitly) optimize a reward model that scores individual responses in isolation, which can lead to mode collapse and overly concentrated policies. We propose population-relative RLHF, a framework that conditions pairwise preference feedback on a reference population of responses, enabling the reward model to capture a holistic quality-diversity trade-off. We formulate population-relative RLHF as a bi-level optimization problem, where the upper level learns a population-relative reward model and the lower level solves for the best-response LLM policy induced by that reward. We show that this formulation is an instance of a broader class of bi-level problems whose lower level constitutes a mean-field-type stochastic game. For this general problem class, we develop a first-order Hessian-free algorithm with provable finite-time convergence guarantees. We apply this algorithm to population-relative RLHF, and also introduce a simplified, memory-efficient variant suitable for large-scale implementation. Empirical evaluations demonstrate that our methods outperform prior art on the quality–diversity trade-off.

Primary area:
reinforcement learning
Keywords:
RLHF, bi-level optimization, LLM post-training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jj6WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Population-Relative RLHF, a framework for improving the quality-diversity trade-off in LLM alignment by incorporating pairwise preference feedback conditioned on a reference population of responses. The problem is formulated as a bi-level optimization task, where the upper level learns a population-relative reward model and the lower level optimizes the LLM policy through a mean-field-type game. They develop a first-order, Hessian-free algorithm (FD-S-MFG) with finite-time convergence guarantees, alongside a simplified, memory-efficient variant for practical LLM training. Experiments shows that the proposed methods achieve a good quality-diversity Pareto frontier compared to the baselines.

Critical strengths
  1. The paper develops a novel first-order, Hessian-free algorithm with finite-time convergence guarantees for a general class of bi-level optimization problems involving mean-field-type games.
  2. The new method achieves a superior trade-off between quality and diversity over several competitive baselines, while its simplified variant demonstrates comparable performance with lower computational and memory overhead.
Critical weaknesses
  1. Limited experimental datasets. The experimental evaluation is limited to two benchmarks, Happy Tweet Generation and LitBench. The relatively narrow selection of datasets raises concerns about the generalizability and robustness of the proposed method. Evaluating the approach on additional datasets would help provide stronger empirical evidence for its effectiveness.
  2. Lack of computational efficiency analysis. The paper lacks quantitative comparisons of computational costs against existing baselines. Reporting training time, GPU memory usage, and computational requirements would help assess the practical efficiency and scalability of the proposed approach.
  3. Limited justification of theoretical assumptions. The convergence guarantee in Theorem 1 relies on three key assumptions. Although the authors briefly discuss these assumptions and cite related literature, they provide limited justification for their validity in practical LLM alignment settings. A more detailed discussion of their applicability would strengthen the theoretical contribution.
Questions for authors
  1. Could the authors empirically validate the convergence rate and sample complexity bound established in Theorem 1, perhaps through some controlled tabular experiments?
  2. Could the authors clarify the motivation for selecting Jaccard distance and Vendi score as diversity metrics? Additionally, evaluating the proposed method using alternative diversity metrics would help establish whether the improvements generalize beyond the metrics.
  3. I think there is a typo in the point annotations of Figure 1 and 2. It should be δ\delta instead of λ\lambda.
Reviewer_uSupOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Many tasks in real life require reflecting plurality and diversity, and thus, LLMs should be able to reflect diverse outputs alongside correct sensible outputs. While existing methods maximize the reward of individual responses, each in isolation, this paper proposes population-relative RLHF, where a response is judged against a reference set of outputs from the current model. The annotator must choose between two candidates alongside another NN outputs from the current reference policy. The motivation is to reward responses that are individually useful but also contribute something different to the current population.

The framework is as follows. Denote ω\omega the reward-model parameters, θ\theta the policy parameters, and πθ\pi_\theta the LLM policy. Then Jω(π,π′)J_\omega(\pi,\pi') is the regularized expected reward obtained by π\pi when evaluated against a reference policy π′\pi'. The lower-level problem optimizes an equilibrium policy π⋆(ω)\pi^\star(\omega), which optimal when the reference population follows that same policy. Specifically, Eq. (4) defines the lower-level objective: Jω(π,π′)=E[∑t=0∞γt(rω(st,at∣π′)−τlog⁡π(at∣st))].J_\omega(\pi,\pi')=\mathbb{E}[\sum_{t=0}^{\infty}\gamma^t(r_\omega(s_t,a_t\mid\pi')-\tau\log\pi(a_t\mid s_t))]. The upper-level objective is F(ω)=f(ω,π⋆(ω))F(\omega)=f(\omega,\pi^\star(\omega)), where f(ω,π)f(\omega,\pi) is the reward model’s preference-prediction loss on data generated by policy π\pi. Eq. (5) then formulates the bilevel problem as min⁡ωF(ω)\min_\omega F(\omega) subject to π⋆(ω)∈arg⁡max⁡πJω(π,π⋆(ω))\pi^\star(\omega)\in\arg\max_\pi J_\omega(\pi,\pi^\star(\omega)).

Lemma 1 gives the hypergradient, which is the derivative of the upper-level objective accounting for the equilibrium policy’s dependence on the reward model: ∇F(ω)=∇ωf−∇ω,12Jω(∇1,12Jω+∇2,12Jω)−1∇θf.\nabla F(\omega) = \nabla_\omega f - \nabla^2_{\omega,1}J_\omega \left( \nabla^2_{1,1}J_\omega+\nabla^2_{2,1}J_\omega \right)^{-1} \nabla_\theta f. Here, ∇ωf\nabla_\omega f and ∇θf\nabla_\theta f are the partial gradients of the preference loss w.r.t. reward and policy parameters, ∇1,12Jω\nabla^2_{1,1}J_\omega describes curvature w.r.t the acting policy, ∇2,12Jω\nabla^2_{2,1}J_\omega captures coupling to the reference population, and ∇ω,12Jω\nabla^2_{\omega,1}J_\omega couples reward-model and acting-policy parameters. Under Assumptions 1–3 and suitable step sizes, Theorem 1 claims min⁡0≤k≤KE∥∇F(ωk)∥2≤O(ϵMFG(K+1)1/5)+O~((K+1)−2/5),\min_{0\le k\le K}\mathbb E\|\nabla F(\omega_k)\|^2 \le O\left(\epsilon_{\mathrm{MFG}}(K+1)^{1/5}\right) + \widetilde O\left((K+1)^{-2/5}\right), where ωk\omega_k is the reward parameter at outer iteration kk (KK is the final iteration) and ϵMFG\epsilon_{\mathrm{MFG}} bounds the squared errors in the inner solver’s policy parameters and advantage estimates. To keep expected squared gradient norm at most ϵ>0\epsilon>0, O~(ϵ−5/2)\widetilde O(\epsilon^{-5/2}) outer iterations and O~(ϵ−4)\widetilde O(\epsilon^{-4}) total sample complexity are needed under tabular softmax parameterization (Section 2.2, lines 153–157) and equilibrium stability assumptions.

The empirical experiments study positive-tweet generation with a small four-layer transformer and creative writing by LoRA finetuning Llama-3.1-8B-Instruct. Both use synthetic preferences and measure the quality-diversity tradeoff using the following metrics: tweets measured with sentiment scores plus word-level Jaccard diversity, while creative writing is assesed using an LLM quality score with embedding-based marginal Vendi diversity. The reported trade offs are encouraging; they report improved frontiers over RLHF, DPO, DivPO, DDPO, CD-RLHF, and its own Set RLHF baseline. The simplified method matches or sometimes exceeds the full method, where the simplified method drops the indirect gradient term and alternates between fitting the reward model on sampled comparisons from the current policy and optimizing the policy using the learned population-conditioned reward, avoiding the inverse Hessian and finite-difference calculations.

Critical strengths

The feedback formulation addresses a real modeling limitation. Evaluating candidates relative to other existing outputs w.r.t the current policy seems like a sensible approach when users care about receiving a useful collection of answers. Though I am not entirely familiar with the other literature that deals with similar problems of diversity, the formulation here seems intuitive and justifiable. The theoretical formulation is relatively neat, though I am unsure how meaningful the guarantees are in practicality.

Critical weaknesses
  1. There are some possible errors in the mathematical presentation and proof, unless the paper incorporates stronger assumptions.

Section 3.2 claims that Algorithm 2 provides "unbiased hypergradient estimation and stronger convergence guarantees", but Theorem 1 and Propositions 1 and 2 analyze Algorithm 1. I might have missed it but it is unclear which guarantee is claimed for Algorithm 2. There's also a possible error in Algorithm 2. In particular, Eq. (16) appears to omit the KL regularizer’s curvature, so the displayed estimator does not directly approximate the operator required by the theoretical hypergradient. Although Algorithm 2 includes KL in the policy update, its curvature estimator H^1,k\widehat H_{1,k} of Algorithm 2 step 13 uses only the learned reward RωR_\omega. However, the lower-level objective includes KL regularization, whose second derivative also contributes to the hypergradient.

As a demonstrative example, consider two responses. Let πt(a)=σ(t)\pi_t(a)=\sigma(t), use a uniform anchor policy, and assign rewards Rw(a)=wR_w(a)=w and Rw(b)=0R_w(b)=0, with no population dependence. The regularized optimum is t⋆=w/τt^\star=w/\tau. Writing p⋆=σ(t⋆)p^\star=\sigma(t^\star), the correct curvature is −τp⋆(1−p⋆)-\tau p^\star(1-p^\star), whereas Eq. (16) converges to wp⋆(1−p⋆)(1−2p⋆)wp^\star(1-p^\star)(1-2p^\star) for a unit perturbation direction. The population term has zero expectation in this example. Thus, the estimator targets a different operator even at equilibrium and with an arbitrarily small finite-difference step.

Omitting the KL may be a valid practical approximation, but I am unclear then whether the claims about "unbiased hypergradient estimation and stronger convergence guarantees" for Algorithm 2 is valid. Does the implementation include the regularizer contribution despite its omission from Eq. (16)? If so, the displayed estimator needs correction. If the omission is intentional, the writing should reflect the justification. Also, the claims about unbiased hypergradient estimation and applicable guarantees should be clarified and possibly revised for Algorithm 2.

On the other hand ,the convergence proof uses stronger smoothness than the assumptions provide. Assumption 2 assumes bounded, Lipschitz first derivatives of the upper objective ff. In the proof of Lemma 9, Appendix H.9, Eq. (67), lines 2142–2144, the authors use a Lipschitz bound on the derivative of b=∇θfb=\nabla_\theta f.

More explicitly, writing ζ=(ω,θ)\zeta=(\omega,\theta) for the combined parameters and LfL_f for a smoothness constant, the proof requires

\le L_f\|\zeta_1-\zeta_2\|.$$ This is a Lipschitz condition on the relevant second derivatives of $f$. It does not follow from Lipschitz continuity of the first derivatives. The resulting regularity of the auxiliary solution is subsequently used in the tracking argument of Lemma 10. The authors could strengthen the assumptions, yet as written, the proof does not establish the claimed result under the stated assumptions. 2. The experiments may bring into question the theoretical contribution for the full hypergradient method. The full method does not seem to provide clear advantage over the simplified version in the tweet experiments. Furthermore, only the simplified version is evaluated for an off-the-shelf LLM experiment. The evaluation also relies on synthetic preferences, even though the work's original motivation is grounded in real concern about practical diversity, but provides limited evidence about actual human preferences. The experiments did not demonstrate sufficient robustness to be practical, while the theory shows a gap after various approximations. The theoretical merit lies on providing a framework to begin from.
Questions for authors

My questions are incorporated in the weaknesses. Please address them there. Thank you.

Reviewer_kZyAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the problem of aligning an AI LLM model. It poses this problem as a bi-level optimization problem where the upper level sets a reward function and the lower level finds the optimal policy given this reward function. The upper level is modeled as a single player trying to minimize a cross entropy cost with respect to a reward parameter. The lower level is modeled as a mean field game where RLHF is performed given the other feedback. This allows the authors to pose the system as a bi-level optimization problem where computations can be done using two time scale stochastic approximation with the faster timescale being the lower level while the slower timescale being the upper level. The paper then provides convergence results for the minimum value of the iterates.

Critical strengths

The paper's formulation is interesting . The use of a bilevel architecture while not novel is a very useful tool to analyze the llm.

RLHF as a mean field process when the other responses are provided with conditions on continuity circumvents the need to play a more complicated Stackelberg game and minimizes the need for coordination among the lower level agents.

Theorem 1 establishes the first finite-time and finite-sample complexity bound for a bi-level optimization algorithm targeting a lower-level regularized mean-field-type game

Critical weaknesses

First of all in section two the sequence of natural language prompts is a poorly defined space. Is it separable? Polish? What kind of space is it? This might seem pedantic but this determines what kind of distributions are used when constructing sampling process on x. It also determines the size of parameter for

Following this line, what is the order of d? Are we sure a finite d can be used to parameterize such a space?

Both these questions are very important when considering the markovian state space characterization that follows. How do we know that the state space induced for this process is finite? The policy parameterization is R^{|S||A|}, this parameterization will fail if |S| is not finite. Next, line 119: A sufficiently large regularization weight τ guarantees that π⋆(ω) is unique for any ω, ensuring that the objective in (3) is well-defined. This is largely meaningless since as \tau tends to infinity the value of \pi_* is just \pi_{KL}. Returning to the finiteness question before, the KL divergence in discrete cases can be well defined however, in more general spaces the question of absolute continuity of the policies will becomes very important.

A major missing result follows from line 211, why is it trivial that stochastic approximation algorithms will converge? The paper does not verify the lyapunov condition requirement or boundedness or martingale difference of the updates anywhere. This is a non trivial result.

Questions for authors

Is D (line 103) just the distribution with respect to P_w? If yes, then why create a new variable for it, if not how is it different?

Doesn't the order of human feedback then affect the value of P_w? In mean field process there is a degree of simultaneous actions among the clients that induce fixed points. However, if one is to query the agents sequentially one may see a polya urn like behavior where previous responses may induce clustering and trivial fixed points as a result. How do you address these problems?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.