- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
There exists a gap between the goal of social welfare and status quo alignment methods. These methods take pairwise comparisons and learn a policy, but cannot identify social welfare. Ordinal preference data is too limited to allow for this. Existing work measures the gap between the methods and the goal of social welfare using the distortion as their measure (GHY), but assumes a fixed lottery for all users. In practical settings, however, LLMs are tailored to their users, meaning different users use LLMs with different probability distributions and are being served separate lotteries. The paper proposes GLHF, which instead learns lotteries on a per-group basis. The paper shows that they are able to obtain GHY’s bounds on each of these groups. However, users can belong to multiple groups, and adding lotteries for each intersection of groups is infeasible. They instead return a learned weighted average for each user and blend the group policies. The authors use synthetic LLM-generated data and the real-world Great American Coffee Taste Test dataset to demonstrate their method. They find that their method achieves lower worst-group distortion than NLHF, pooled GLHF, and a uniform reference.
Critical strengths
-
The problem is solved with a simple method that addresses the real issues with overlapping groups when assigning lotteries to users who belong to multiple groups. The guarantees hold per group, and the strength of those guarantees increases with how often each group is observed.
-
The use of the real-world Great American Coffee Taste Test dataset, which has real demographic groups (e.g., age and gender), to complement the synthetic data is appreciated. The math seemed sound and I did not notice any errors upon inspection.
Critical weaknesses
-
The settings in the paper seem unrealistic for LLM alignment. Each study only uses one prompt with a few fixed responses and known group labels and group sizes. I would like to see some discussion of how this would extend to practical settings. In addition, the theory is vacuous at the scale of the experiments. For example, the Age 45+ group in the coffee study has about 435 training users, which gives a bound of about 1.71 under Theorem 3, and since the margins in lie in , exploitability is at most 1.
-
Each individual element of the paper is presented adequately. As a whole, however, the number of equations, the dense spacing, and the near-total lack of figures and tables make the paper hard to follow. It is difficult to get a clear grasp of the method, its performance, and its tradeoffs. I would suggest at least the addition of a visual on the group policy blending.
Questions for authors
My score can be changed with presentation changes, and by addressing the practicality and level of gain of the method.
- Can Theorem 3 be extended to cover the adaptive learning rates used in the experiments?
- How does per-intersection NLHF compare to GLHF, especially on the smaller groups?
- How would GLHF extend to an actual LLM policy or to multiple prompts?
3 comments
Response to authors
I thank the authors for their revisions and response to my questions.
I am satisfied by the Theorem 3 extension, and I appreciate the addition of figures.
However, my concerns about the setting remain, and the gains over the group NLHF baseline seem fairly modest. I appreciate the potential LLM extension, but it is not detailed enough to fully convince me at this stage.
As a result, I will maintain my score at this point.
Thank you for evaluating our revisions. We appreciate your consideration.
We would like to briefly clarify the theoretical motivation for our method. GLHF provides simultaneous worst-case comparison guarantees for every group. As illustrated by Figure 1, averaging independently learned NLHF policies fails in general to preserve a group’s comparison guarantee. Beyond the empirical improvements, our main contribution is to establish these simultaneous theoretical guarantees in the presence of overlapping groups.
Our abstraction also follows GHY’s Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?, which studies welfare guarantees for distributions over candidate responses while abstracting from practical LLM fine-tuning. We adopt the same framework to establish standalone theoretical guarantees for overlapping groups that we would hope a practical implementation would preserve.
Thank you for the careful review, especially the worked Age 45+ example and the suggestions for improving the presentation. The theory was updated to directly resolve the gap between the learning rates in the theorem and those in the experiments you identified.
Response to criticisms:
The fixed-prompt, finite-response setting follows GHY's analysis of the utility obtained from preference-based learning. It captures the aggregation of feedback over candidate responses that occurs in LLM training; for example, InstructGPT collected rankings of 4–9 responses per prompt for reward-model training. We now explain this connection and the possible use of learned distributions as fine-tuning targets. Groups can be selected using a separate discovery sample, as in our experiments, and the adaptive learner does not need their sizes. Memberships must still be observable when learning and serving.
You are right that the worst-case bound is vacuous at the experimental scale. With about 435 expected Age 45+ training observations, the revised adaptive theorem gives 2.55 under active sampling and 3.92 under passive sampling at 95% confidence, both above one. The value of the theorem is that it rules out persistent groupwise exploitability as data grow and quantifies how the required feedback depends on group prevalence and the number of alternatives, a guarantee we would wish to have for any preference learning method. The experiments complement this by showing welfare improvements at finite budgets, although those improvements are not certified by the worst-case bound.
Thank you for suggesting a visual explanation of the blend. We added Figures 2 and 3 to show how matching group policies are combined, how one comparison updates them, and how a response is sampled at deployment. We also moved the exact updates beside the full pseudocode in Appendix A so the main text can focus on the procedure and performance as you suggested.
Response to questions:
The revised Theorem 3 now covers the adaptive rates used in both studies, under both active and passive sampling. The proof controls the outer and inner updates separately, then combines their bounds through the second moment of the shared comparison score. The added dependence is O(loglogT) in the logarithmic factor. Neither learning rate requires a group mass (Appendices A–E).
For per-intersection NLHF, the answer depends on the available data. In the separate exploratory Coffee study (ten folds and three learner seeds per fold), worst-group distortion with six groups is nearly tied (1.101 versus 1.099 for multigroup GLHF). With fifteen groups, the gap widens to 1.230 versus 1.100. This is consistent with per-intersection NLHF becoming more sensitive to limited data as the number of groups grows (Appendix G.6).
As these groups are added, the median training intersection shrinks from about 25 respondents to one. With fifteen groups, 18% of test users fall in intersections absent from training and receive the uniform reference. GLHF can still use comparisons from everyone in each matching group, avoiding the need to split the data into separately learned membership cells.
For an actual LLM, the learned lottery specifies how to distribute responses among the candidates for a prompt. It can be used directly for selection, or the distributions learned for several prompts can become targets for fine-tuning a model conditioned on both the prompt and the user's memberships. This is the route discussed briefly in Section 7. As in GHY, studying the response distributions first lets us establish what utility guarantees the preference-learning procedure can achieve. Carrying those guarantees through neural fine-tuning would require bounding how accurately the model reproduces the distributions and generalizes to new prompts.