acceptodds
Under review as a conference paper at ICLR 2027

Groupwise Distortion Guarantees for Preference-Based Alignment

Abstract

Preference-based alignment methods such as RLHF and NLHF aggregate ordinal pairwise comparisons to learn an LLM policy, but a natural goal is maximizing (average cardinal utility), which comparisons do not identify. Gölz, Haghtalab, and Yang (GHY) measure the gap by : the worst-case ratio between the welfare of the best fixed lottery (distribution over responses) and of the learned lottery. They show NLHF is optimal when every user receives the same lottery. Account-based LLMs, however, have information about their users and can serve different lotteries to different people. We give an efficient algorithm, GLHF, that learns a single group-conditioned policy from one comparison per user. This policy attains GHY's optimal distortion bound simultaneously on every group in a prespecified, possibly overlapping collection, with sample complexity growing logarithmically in the number of groups and inversely with the smallest group mass. When the members of a group share a feasible favorite response, that group's distortion approaches one. On human coffee ratings and synthetic LLM-generated ratings, GLHF lowers distortion in every group relative to NLHF and substantially reduces worst-group distortion relative to group-agnostic baselines.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Alignment, Social Choice, Distortion, NLHF, Multigroup Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_q37HOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

There exists a gap between the goal of social welfare and status quo alignment methods. These methods take pairwise comparisons and learn a policy, but cannot identify social welfare. Ordinal preference data is too limited to allow for this. Existing work measures the gap between the methods and the goal of social welfare using the distortion as their measure (GHY), but assumes a fixed lottery for all users. In practical settings, however, LLMs are tailored to their users, meaning different users use LLMs with different probability distributions and are being served separate lotteries. The paper proposes GLHF, which instead learns lotteries on a per-group basis. The paper shows that they are able to obtain GHY’s bounds on each of these groups. However, users can belong to multiple groups, and adding lotteries for each intersection of groups is infeasible. They instead return a learned weighted average for each user and blend the group policies. The authors use synthetic LLM-generated data and the real-world Great American Coffee Taste Test dataset to demonstrate their method. They find that their method achieves lower worst-group distortion than NLHF, pooled GLHF, and a uniform reference.

Critical strengths
  • The problem is solved with a simple method that addresses the real issues with overlapping groups when assigning lotteries to users who belong to multiple groups. The guarantees hold per group, and the strength of those guarantees increases with how often each group is observed.

  • The use of the real-world Great American Coffee Taste Test dataset, which has real demographic groups (e.g., age and gender), to complement the synthetic data is appreciated. The math seemed sound and I did not notice any errors upon inspection.

Critical weaknesses
  • The settings in the paper seem unrealistic for LLM alignment. Each study only uses one prompt with a few fixed responses and known group labels and group sizes. I would like to see some discussion of how this would extend to practical settings. In addition, the theory is vacuous at the scale of the experiments. For example, the Age 45+ group in the coffee study has about 435 training users, which gives a bound of about 1.71 under Theorem 3, and since the margins in M(x)M(x) lie in [−1,1][-1, 1], exploitability is at most 1.

  • Each individual element of the paper is presented adequately. As a whole, however, the number of equations, the dense spacing, and the near-total lack of figures and tables make the paper hard to follow. It is difficult to get a clear grasp of the method, its performance, and its tradeoffs. I would suggest at least the addition of a visual on the group policy blending.

Questions for authors

My score can be changed with presentation changes, and by addressing the practicality and level of gain of the method.

  • Can Theorem 3 be extended to cover the adaptive learning rates used in the experiments?
  • How does per-intersection NLHF compare to GLHF, especially on the smaller groups?
  • How would GLHF extend to an actual LLM policy or to multiple prompts?
3 comments

Thank you for the careful review, especially the worked Age 45+ example and the suggestions for improving the presentation. The theory was updated to directly resolve the gap between the learning rates in the theorem and those in the experiments you identified.

Response to criticisms:

The fixed-prompt, finite-response setting follows GHY's analysis of the utility obtained from preference-based learning. It captures the aggregation of feedback over candidate responses that occurs in LLM training; for example, InstructGPT collected rankings of 4–9 responses per prompt for reward-model training. We now explain this connection and the possible use of learned distributions as fine-tuning targets. Groups can be selected using a separate discovery sample, as in our experiments, and the adaptive learner does not need their sizes. Memberships must still be observable when learning and serving.

You are right that the worst-case bound is vacuous at the experimental scale. With about 435 expected Age 45+ training observations, the revised adaptive theorem gives 2.55 under active sampling and 3.92 under passive sampling at 95% confidence, both above one. The value of the theorem is that it rules out persistent groupwise exploitability as data grow and quantifies how the required feedback depends on group prevalence and the number of alternatives, a guarantee we would wish to have for any preference learning method. The experiments complement this by showing welfare improvements at finite budgets, although those improvements are not certified by the worst-case bound.

Thank you for suggesting a visual explanation of the blend. We added Figures 2 and 3 to show how matching group policies are combined, how one comparison updates them, and how a response is sampled at deployment. We also moved the exact updates beside the full pseudocode in Appendix A so the main text can focus on the procedure and performance as you suggested.

Response to questions:

The revised Theorem 3 now covers the adaptive rates used in both studies, under both active and passive sampling. The proof controls the outer and inner updates separately, then combines their bounds through the second moment of the shared comparison score. The added dependence is O(log⁡log⁡T)O(\log \log T) in the logarithmic factor. Neither learning rate requires a group mass (Appendices A–E).

For per-intersection NLHF, the answer depends on the available data. In the separate exploratory Coffee study (ten folds and three learner seeds per fold), worst-group distortion with six groups is nearly tied (1.101 versus 1.099 for multigroup GLHF). With fifteen groups, the gap widens to 1.230 versus 1.100. This is consistent with per-intersection NLHF becoming more sensitive to limited data as the number of groups grows (Appendix G.6).

As these groups are added, the median training intersection shrinks from about 25 respondents to one. With fifteen groups, 18% of test users fall in intersections absent from training and receive the uniform reference. GLHF can still use comparisons from everyone in each matching group, avoiding the need to split the data into separately learned membership cells.

For an actual LLM, the learned lottery specifies how to distribute responses among the candidates for a prompt. It can be used directly for selection, or the distributions learned for several prompts can become targets for fine-tuning a model conditioned on both the prompt and the user's memberships. This is the route discussed briefly in Section 7. As in GHY, studying the response distributions first lets us establish what utility guarantees the preference-learning procedure can achieve. Carrying those guarantees through neural fine-tuning would require bounding how accurately the model reproduces the distributions and generalizes to new prompts.

Reviewer_q37HOpenReview ↗

Response to authors

I thank the authors for their revisions and response to my questions.

I am satisfied by the Theorem 3 extension, and I appreciate the addition of figures.

However, my concerns about the setting remain, and the gains over the group NLHF baseline seem fairly modest. I appreciate the potential LLM extension, but it is not detailed enough to fully convince me at this stage.

As a result, I will maintain my score at this point.

Thank you for evaluating our revisions. We appreciate your consideration.

We would like to briefly clarify the theoretical motivation for our method. GLHF provides simultaneous worst-case comparison guarantees for every group. As illustrated by Figure 1, averaging independently learned NLHF policies fails in general to preserve a group’s comparison guarantee. Beyond the empirical improvements, our main contribution is to establish these simultaneous theoretical guarantees in the presence of overlapping groups.

Our abstraction also follows GHY’s Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?, which studies welfare guarantees for distributions over candidate responses while abstracting from practical LLM fine-tuning. We adopt the same framework to establish standalone theoretical guarantees for overlapping groups that we would hope a practical implementation would preserve.

Reviewer_j9ncOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper extends GHY’s (Golz, Haghtalab and Yang) NLHF idea to learn a single "fair policy" for multiple users with overlapping preference group membership. The proposed algorithm GLHF simultaneously learns multiple specialist policies serving each preference group and a mixture weight that combines them using a no-regret algorithm to get a single policy to minimize groupwise exploitability. Additionally, the paper presents a theoretical guarantee and experiments to support their claims.

Critical strengths
  1. NLHF for users with overlapping group membership is a useful and well motivated extension to the problem.
  2. The algorithm is pretty neat. It only collects a single pair-wise preference per user at every round. The theoretical bound is also pretty interesting and non-trivial even though NLHF and multi-group learning concepts are already known.
Critical weaknesses
  1. The empirical validation and possibly practical relevance is weak. Collected using preference online is very expensive and somewhat unrealistic and the experiment is small-scale with no-real fine-tuning of the LLM. Additionally, GLHF is informationally richer than other methods as it sees the group membership. However, the theory holds on its own and I recommend borderline accept.
Questions for authors

Are you able to add the simple baseline where you do GLHF independently for every group and mix/average it? It will help resolve the information issue for the baselines.

1 comment

Thank you for the encouraging assessment of the algorithm and theory, and for pointing out the information advantage over pooled baselines. As described in the general response, we have added a group NLHF average to the main comparison. This helps distinguish the benefit of observing memberships from the benefit of the learning method.

Response to criticisms:

We should have made clearer that GLHF can learn from feedback collected in advance. The revised main experiments use passive sampling, so all trained methods receive the same log of one uniformly sampled comparison per user. Active sampling is optional and could be implemented through an online preference-collection platform. It uses the same one-comparison-per-user budget; its additional benefit is now reported separately in Section 6.4.

The fixed-prompt setting is a deliberate abstraction, following GHY, for understanding what utility guarantees preference-based learning can provide. Small candidate sets are also used in practical preference collection: InstructGPT trained its reward model from rankings of 4–9 responses per prompt before fine-tuning the LLM. We now explain this connection in the paper. Our aim is to establish GHY's welfare guarantee simultaneously across overlapping groups for a policy that uses user context; implementing and analyzing a shared neural policy across prompts is a further step, discussed briefly in Section 7.

Response to questions:

Thank you for suggesting independent group fits followed by averaging. In a separate auxiliary check, we used one GLHF learner per named group and one for the full population, using each group's comparisons from the shared passive log. Equally averaging the fits applicable to each user gives mean worst-group distortion of 1.135 on Coffee and 1.207 on Completions, versus 1.091 and 1.099 for passive joint GLHF. These are auxiliary results; the revised manuscript reports the corresponding NLHF averaging baseline, which also receives the same memberships and comparisons.

This comparison gives both methods access to the same group memberships and feedback. Figure 1 shows why learning the group policies jointly can still matter: even if each independently fitted NLHF policy is optimal for its group, averaging them for users in overlapping groups can violate a group’s comparison guarantee. GLHF instead updates the group policies against the blend users actually receive, providing simultaneous guarantees for the deployed policy.

Reviewer_N5VuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Groupwise Learning from Human Feedback (GLHF), an algorithm for learning a single context-dependent policy that accommodates potentially overlapping user groups. GLHF maintains group-specific lotteries and adaptively combines them based on group membership, providing simultaneous guarantees across overlapping groups. Specifically, the authors establish asymptotically worst-case optimal groupwise welfare distortion guarantees under the Bradley-Terry model, with stronger guarantees for groups with aligned preferences. Experiments on human coffee ratings and synthetic LLM-generated ratings show that GLHF reduces worst-group distortion compared with baselines, including NLHF.

Critical strengths

The theoretical contribution of the paper is clear, as it shows that worst-case optimal groupwise welfare distortion guarantees can be achieved simultaneously even for overlapping groups and establishes rigorous theoretical bounds. The problem is timely and relevant to the growing literature on AI alignment, with potential implications for accommodating diverse user contexts in LLMs while providing welfare guarantees for different subgroups.

Critical weaknesses

The practical efficiency and scalability of the proposed algorithm seem somewhat limited, as it requires prior knowledge of the group structure, maintains a separate lottery for each group, and stores intermediate policies to construct the final averaged policy. The empirical evaluation could be strengthened, as the experiments use simple rating-based utility models and only a small number of fixed alternatives. The baselines are mostly group-agnostic, and comparisons with group-aware methods would provide a more convincing evaluation.

Questions for authors

Regarding Section 3.1, since the stronger welfare guarantee applies to any policy with low exploitability, could the authors clarify how Proposition 5 relates to GLHF? Also, shouldn't this guarantee be understood in terms of a trade-off, since finer-grained groups may reduce residual welfare loss but lead to weaker statistical guarantees due to smaller population masses? In the experiments, wouldn't it be more informative to compare GLHF with a mixture of groupwise NLHF policies? For example, one could train NLHF separately for each group and average the resulting lotteries for each context based on group membership. The distortion values in the coffee-rating experiment are generally quite low, even for the uniform reference. Could this be due to the relatively simple or favorable structure of the underlying utility distributions? Could the authors comment on how well these results would generalize to more complex and heterogeneous preference settings?

1 comment

Thank you for the thoughtful review and for recognizing the theoretical contribution. Your suggestions about group-aware baselines, storage, and the cost of finer groups were directly implemented in our revisions.

Response to criticisms:

The groups need to be fixed before policy learning, but they can be chosen using a separate discovery sample. This is what we did in both experiments: we reserved about 10% of respondents to identify groups with different preferences, then fixed the groups before training and evaluation (Appendix G.1). The learner needs observable memberships, but the adaptive rates do not require group sizes. The current learner state uses O(Km)O(Km) memory, in addition to storage for saved rounds.

To address the storage cost, we added Proposition 7. Retaining s rounds sampled uniformly with replacement before training reduces policy storage from O(TKm)O(TKm) to O(sKm)O(sKm), with a welfare loss relative to the full time average of at most log⁡(2Kδs)/2s\sqrt{\log(\frac{2K}{\delta_s})/{2s}}, simultaneously across groups with probability at least 1−δs1-\delta_s. Appendix A gives the computational and storage costs explicitly.

Ratings let us construct pairwise feedback and evaluate the resulting welfare on the same utility scale. In the Completions study, we explicitly asked the model to adopt the persona and “Rate how well the answer works for you.” This makes the intended utility interpretation direct, while remaining a synthetic evaluation that treats equal rating increments as equal utility increments. The number of alternatives is also comparable to preference collection in InstructGPT, which used 4–9 responses per prompt. Section 6 now explains this connection.

Thank you for suggesting a group-aware comparison. We added the groupwise NLHF mixture with exactly the same memberships and passive comparison log as GLHF. We also added an example showing why averaging independent group fits does not automatically preserve their guarantees, as discussed below.

Response to questions:

On Proposition 5, you are correct: it describes the welfare guaranteed by a policy with low exploitability, regardless of how that policy is learned. GLHF is our algorithm for obtaining the comparison certificates underlying GHY's welfare analysis simultaneously across overlapping groups. Proposition 5 gives a sharper interpretation of what those certificates imply when a group's preferences are more aligned. Section 3.1 now explicitly applies this bound to GLHF, using the certificates from Theorem 3.

You are also right about the tradeoff from finer groups. They can reduce residual welfare loss, but smaller population masses mean fewer comparisons and a larger statistical error. The leading error scales as 1/(Tpk)1/\sqrt(Tp_k), so refinement helps the bound only when the reduction in the residual term outweighs this increase. We now say this explicitly in Section 3.1. Adding groups of fixed mass has only logarithmic statistical overhead in their number; making a group rarer can have a much larger cost.

For the NLHF mixture, we added both a theoretical example and an empirical comparison. Figure 1 shows that even exact groupwise fits can lose a group's comparison guarantee when averaged on overlapping users.

In the experiments, we fit one NLHF lottery per named group and one for the full population, then equally average the fits applicable to each user. At the main setting, mean worst-group distortion is 1.13 for this baseline versus 1.09 for passive GLHF on Coffee, and 1.18 versus 1.10 on Completions. Group NLHF does better on Completions at β=0.5 and 1, so we do not claim that GLHF always wins empirically.

On Coffee's low distortion, yes: we think the alternatives and utility distributions matter. In the Completions study, two of the six responses were factually incorrect and rarely preferred, which helps explain why uniform has much higher worst-group distortion there (1.74 versus 1.12 on Coffee). Coffee's low value is consistent with a more favorable set of alternatives. The benchmark also matters: it is each group's best fixed lottery within a KL ball of radius 0.5, rather than giving every person their favorite alternative.

The comparison certificates themselves allow arbitrary pairwise preferences, including heterogeneous and cyclic preferences. Their welfare interpretation assumes individual Bradley–Terry comparisons with a common finite β, but does not require simple or similar utilities across users. At the main experimental setting, β=∞, the comparison guarantee still applies, while the worst-case welfare factor diverges.

Comments on the paper

1 comment

We thank the reviewers for their careful reading. Their questions led to three substantial improvements.

  1. The theory now covers the adaptive algorithm used in the experiments. The submitted Theorem 3 analyzed fixed learning rates, whereas the experiments used adaptive rates. We now prove simultaneous groupwise guarantees for those adaptive updates under both active and passive sampling. The adaptive learning rates do not require group sizes. Section 3.1 now makes explicit the tradeoffs of more refined groups: narrower groups may have more similar preferences but also supply fewer comparisons, increasing statistical error.
  2. The main experiments now control for both sampling protocol and access to group memberships. Previously our main setting used active sampling to learn the multigroup GLHF policies, but now the main setting uses the same passive comparison log across all trained methods. We also added a baseline that fits NLHF separately on each named group and the full population, then equally averages the fits applicable to each user. At the main setting (β=∞, with learner and benchmark KL radii both 0.5), passive GLHF achieves lower mean worst-group distortion on held-out users than the group NLHF baseline: 1.09 versus 1.13 on Coffee and 1.10 versus 1.18 on Completions.
  3. We clarify why and how GLHF learns jointly and reduce the storage needed for deployment. Figure 1 shows why averaging exact groupwise NLHF policies need not preserve their guarantees when groups overlap. Figures 2 and 3 trace how GLHF learns and serves its group-conditioned policy. A new Proposition 7 gives a simultaneous welfare guarantee when only a random sample of training rounds is stored, reducing the storage needed for deployment.

We also want to clarify how this setting relates to LLM training and fine-tuning. In OpenAI's InstructGPT training pipeline, annotators ranked 4–9 candidate responses per prompt; these rankings trained a reward model, which then guided LLM fine-tuning (Ouyang et al., 2022, Appendix A.3). Studying preferences over a small candidate set therefore reflects a concrete part of the training pipeline. Like GHY and the subsequent distortion analysis of Oko et al. (2026), we study distributions over responses directly, abstracting from neural parameterization. This lets us ask what utility guarantees can follow from comparison feedback before addressing how a shared model learns across prompts.

Our contribution is a first step toward utility guarantees for context-dependent LLM policies: under individual Bradley–Terry comparisons with a common finite β, we show that GHY's asymptotically optimal population-level distortion guarantee can hold simultaneously for overlapping groups, each relative to its own best feasible fixed lottery. The learned distributions can select among LLM-generated candidates or provide targets for subsequent fine-tuning, as discussed in Section 7.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.