acceptodds
Under review as a conference paper at ICLR 2027

Who Should We Listen to More? Welfare-Aware Preference Acquisition for Pluralistic Alignment

Abstract

Pluralistic alignment needs to accommodate diverse preferences under limited human feedback. Using AI simulators to label preference comparisons reduces annotation costs, but variation in simulation accuracy across individuals can leave some people's preferences poorly represented in training. We study how to allocate human feedback across people to construct training data for pluralistic alignment, and how this allocation affects the trained policy. To make this allocation explicit and controllable, we define a social welfare objective over individual supervision utilities that measure label quality. Each query's marginal welfare gain depends on its expected supervision improvement, the recipient's current utility, and application-designated priority. Welfare-Aware Preference Acquisition (WAPA) uses auxiliary reference models to estimate query gains and individual utilities, then selects queries by estimated marginal welfare gain while updating recipient utilities. We prove exact optimality of the acquisition algorithm for the estimated welfare objective under a fixed query budget and model predictions. Experiments on six Roleplay collections and four OpinionQA regions demonstrate improved supervision and controllable allocation tradeoffs. Policy training on four Roleplay collections improves preference prediction for both seen and unseen individuals. When prioritizing a designated group and poorly represented individuals, acquiring human labels for 20% of comparisons recovers 75-86% of the average predictive utility gap between policies trained entirely on AI-generated labels and entirely on ground-truth labels.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Pluralistic Alignment, Preference Acquisition, Social Welfare

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5QH1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how to assign a limited human-labeling budget across preference comparisons to maximize social welfare. The proposed method, WAPA, estimates each comparison’s expected label error using three reference LLMs and greedily queries the comparison with the largest marginal gain in estimated welfare. The authors prove that this greedy rule exactly maximizes the estimated welfare objective under a fixed budget.

Critical strengths
  • The greedy assignment is simple and cheap. Moreover, its effectiveness is justified by theory.
  • The experimental ablations are extensive.
Critical weaknesses
  • The preference labels in the Roleplay experiments are all simulated. Real human evaluation is only involved at the last stage to grade the generated responses.
  • The personas are from FSPO, which uses GPT-4o to generate synthetic personas and preference labels. Therefore, the preferences are simulated by GPT-4o for different personas instead of being collected from real humans.
  • The paper involves multiple different concepts, e.g., persona, preference, and welfare, but the presentation is not clear enough.
Questions for authors

Could the authors clarify the assumptions required to guarantee global optimality of WAPA for the estimated welfare objective?

6 comments
AuthorsOpenReview ↗

Thank you for your comment on evidence from real human preferences

Weakness 1

Roleplay acquisition uses simulated preference labels, with human evaluation only at the final response-grading stage. This limits the evidence from real human feedback.

We respectfully clarify that our acquisition evaluation also includes OpinionQA, which contains preferences derived from real human survey responses. WAPA achieves higher Mean and Tail acquisition utility than the compared baselines across all four regional subsets and both welfare settings. Section 4 (lines 298–300) introduces both benchmarks, and Appendix E.2 (lines 1379–1388) describes the OpinionQA evaluation. Roleplay uses synthetic preference labels; OpinionQA provides a complementary test of allocation using preferences derived from respondents' survey answers.

OpinionQA originates from the Pew Research Center's American Trends Panel. We use the processed release from Group-Adaptive-Elicitation, covering questions from survey waves W50, W54 and W92 on family, economic inequality and political typology. Our four regional subsets contain 3,130 respondents and 30 questions per respondent, with 3–5 answer options per question (Appendix E.2, lines 1379–1388). This evaluates whether a limited query budget can improve the representation of heterogeneous survey preferences.

The existing results in Table S3 show higher Mean and Tail acquisition utility than all budgeted baselines in every region under both welfare settings. We reproduce the WAPA results below. Values are utilities in percent; parentheses give percentage-point gains over the strongest of Random, Entropy and Reference Disagreement for each region, setting and metric, all at a 20% query budget. These differences are calculated from the displayed Table S3 values (Appendix G.1, lines 1621–1634).

(γ,ρ)(\gamma,\rho)OpinionQA subsetMeanTailTarget
(1,1)(1,1)Midwest58.40 (+3.52)39.75 (+3.97)58.19 (+3.04)
(1,1)(1,1)Northeast59.89 (+3.57)40.78 (+3.14)60.61 (+4.09)
(1,1)(1,1)South57.78 (+3.49)38.91 (+3.54)57.53 (+1.60)
(1,1)(1,1)West59.89 (+3.56)38.87 (+3.09)59.60 (+2.64)
(2,−5)(2,-5)Midwest57.91 (+2.95)40.62 (+4.84)68.27 (+2.00)
(2,−5)(2,-5)Northeast59.42 (+3.21)42.67 (+5.03)69.10 (+1.86)
(2,−5)(2,-5)South57.38 (+3.03)39.77 (+4.40)65.53 (−0.97)
(2,−5)(2,-5)West59.53 (+3.25)42.12 (+6.23)68.41 (+0.70)

Thus, the acquisition gains extend beyond synthetic Roleplay labels: Mean and Tail improve in all eight region–setting comparisons, and Target improves in seven.

OpinionQA tests acquisition on survey choices; it does not establish downstream personalization of generated responses for real users. Our generated-response human evaluation provides a separate check of suitability for supplied profiles. Appendix B (lines 852–858) already states that this is not direct satisfaction feedback from the represented users and calls for broader evaluation with those users. Together, the benchmarks establish controlled allocation and downstream results on Roleplay and acquisition results on survey-derived preferences, while direct validation with represented users remains an important next step.

AuthorsOpenReview ↗

Thank you for your comment on the validity of synthetic preference labels

Weakness 2

The FSPO personas and preference labels are synthetic rather than collected from real people, raising concerns about the external validity of the preference experiments.

We respectfully clarify that Roleplay serves as a controlled testbed for feedback allocation, with complementary evidence from FSPO’s real-user transfer study and our OpinionQA experiments. To directly examine the validity of Roleplay’s synthetic labels, we additionally compared them with human judgments on the same preference comparisons.

Roleplay provides the coverage and comparability needed to test how a shared feedback budget should be distributed across heterogeneous preferences. As disclosed in Appendix E.1 (lines 1328–1332), its personas and preference labels are synthetic. Its explicit profiles and repeated comparisons over a common prompt set let us compare allocation methods at the same budget, train policies on the resulting labels, and evaluate on separate prompts and personas (Appendix E.1, lines 1333–1344). This coverage matters: an unqueried comparison must still have an evaluation label if we are to measure the effect of choosing another query allocation. Collecting such repeated judgments from each real participant is demanding, so Roleplay offers a practical controlled testbed for this question.

The underlying FSPO work also provides evidence that training on these synthetic preferences can transfer to real-user personalization. In FSPO, Section 7, a study with 50 real participants reports personalized-response win rates of 68.2% against the base model and 72.3% against supervised fine-tuning. Its construction procedure also filters labels that change when response order is swapped, and persona refinement reduces repeated-label entropy from 0.64 to 0.13 nats (Section 6). The consistency checks support label stability, while the human study supports synthetic-to-real transfer. These findings support using FSPO as a testbed. Our additional experiment below directly evaluates agreement between synthetic labels and human judgments on the same comparisons.

To address label validity directly, we conducted an additional human validation experiment. We randomly sampled 200 preference comparisons from Roleplay. Five human annotators independently evaluated each comparison using the supplied persona, with the original labels concealed until all judgments were finalized. Agreement with the original labels averaged 75.8% across annotators, with individual agreement rates ranging from 73.5% to 77.5%. This provides supplementary evidence that Roleplay’s synthetic labels agree with human judgments conditioned on the supplied personas.

Our response to Weakness 1 presents the complementary OpinionQA evidence: allocation gains also hold on survey-derived preferences across four regional subsets. Roleplay enables controlled acquisition and downstream evaluation, while OpinionQA tests acquisition beyond synthetic preferences. The remaining need for direct feedback from represented users is acknowledged in Appendix B (lines 852–858). Richer human preference benchmarks with repeated judgments per participant would make this evaluation substantially stronger and are an important direction for future dataset work.

AuthorsOpenReview ↗

Clarification of persona, preference, and welfare

Weakness 3

The distinctions among persona, preference, and welfare need clearer explanation.

The concepts describe successive levels of the acquisition problem: whose preferences are represented, what labels express those preferences, and how supervision quality is aggregated across people. These concepts are introduced in the Introduction and formalized in Section 2 through the preference-acquisition setting, supervision utility (Eq. (1)), and social welfare (Eq. (2)). Section 3.1 explains how they determine WAPA’s acquisition rule. We summarize their roles and connections below:

ConceptMeaning in this paperOriginal location
PersonaA user profile conditioning the preference comparisons and the policy. It identifies whose preferences the model should represent.Section 2, lines 105–107; Figure 1A, lines 108–127
PreferenceFor persona ii and prompt jj, the label Yij∈0,1Y_{ij}\in\\{0,1\\} indicates which of two fixed responses is preferred; pij∗=Pr⁡(Yij=1)p^*_{ij}=\Pr(Y_{ij}=1) allows this preference to be stochastic.Section 2, lines 128–133
Supervision utilityThe quality of the completed preference labels for one persona, measured as one minus average expected Brier completion loss.Section 2, Eq. (1), lines 134–143
Social welfareA weighted aggregation of these persona-level supervision utilities, determining how the shared query budget is allocated.Section 2, Eqs. (2)–(4), lines 145–161

Specifically, the individual quantity is (Section 2, Eq. (1), lines 134–143)

Ui(S)=1−1M∑j:(i,j)∉SE[(p^ij−Yij)2].U_i(S)=1-\frac1M\sum_{j:(i,j)\notin S}\mathbb E[(\hat p_{ij}-Y_{ij})^2].

A query replaces a pseudo-label with the observed preference label, removing its completion loss and increasing that persona's supervision utility. The population objective is (Section 2, Eq. (2), lines 145–153)

J(S)=∑iwihρ(Ui(S)+ε),hρ(u)={uρ/ρ,ρ≠0,log⁡u,ρ=0.J(S)=\sum_i w_i h_\rho(U_i(S)+\varepsilon), \qquad h_\rho(u)= \begin{cases} u^\rho/\rho,&\rho\ne0,\\\\ \log u,&\rho=0. \end{cases}

Here, fixed positive weights wiw_i specify persona priorities, while ρ≤1\rho\le1 controls sensitivity to low supervision utility. For ρ<1\rho<1, hρ′(u)=uρ−1h'_\rho(u)=u^{\rho-1} is larger at lower utility, so an equal utility increment receives more value for a lower-utility persona when weights are equal. Section 2, Eq. (3), lines 154–160, sets the weight ratio between a designated Target persona and a non-Target persona to γ\gamma. Thus, γ\gamma expresses subgroup priority and ρ\rho expresses sensitivity to supervision disadvantage.

For example, two personas can prefer different responses to the same retirement question, as in Figure 1A (lines 108–127). If the model supplies poorer preference labels for one persona, that persona has lower supervision utility. Welfare specifies how much priority to give improving that supervision when labels are scarce. In this paper, “welfare” therefore aggregates the quality of preference supervision; it does not measure the person's real-world wellbeing or assign a value to the content of their preferences. Section 3.1, Eq. (5), lines 190–203, estimates these utilities before querying, and Section 3.2, Eq. (7), lines 218–227, uses the completed labels for downstream alignment.

Reviewer_tsWEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how to allocate a limited human feedback budget among different individuals in multi-modal LLM alignment. The authors propose the Welfare-Aware Preference Acquisition (WAPA) method, which acquires the expected benefits of human preference labels from auxiliary reference models and prioritizes related queries through social welfare objectives. This method takes into account both low-supervision-quality individuals and the application of specified subgroup priority levels, and provides rigorous theoretical proofs. The method improves the quality of preference supervision on multiple datasets, and further demonstrates its effectiveness through several DPO experiments.

Critical strengths
  1. The research problem has practical value. This paper focuses on an important fundamental question in multi-preference alignment: how to reasonably allocate preference annotation resources among different users under a limited human feedback budget. The authors incorporate the individual's current supervision quality and the specified group priorities into a social welfare objective, making the feedback allocation not only dependent on the information value of the samples but also reflect the different supervision needs of users.
  2. The theoretical analysis is rigorous. The authors prove through rigorous mathematical reasoning that under fixed model prediction and annotation budgets, the greedy strategy of WAPA can precisely optimize the defined estimated social welfare objective. Compared to existing baselines, WAPA achieves more consistent performance improvements during the data acquisition phase.
Critical weaknesses
  1. There is a lack of sufficient experimental evidence for the method's effectiveness in aligning downstream multi-preference: The core goal of the paper is to improve multi-preference alignment through feedback allocation based on social welfare perception. However, although the method can significantly change resource allocation at the data acquisition stage, its impact on the downstream DPO model is very limited. In the two parameter sensitivity experiments conducted, the maximum difference in downstream utility brought about by different settings is only 0.1254 percentage points. Moreover, increasing the annotation priority of the target group sometimes reduces the final predictive utility of that group.
  2. Where does the core performance improvement of WAPA come from? In Appendix G.4, the authors compare the difference between the complete method and the method using only model prediction differences, but the difference in Mean Utility between the two methods is only 0.07, and it is even slightly worse for some datasets.
Questions for authors
  1. Does this method provide substantial improvements in downstream multi-consistency? Appendix G.12 shows that changing the welfare parameters only produces minor downstream utility differences (up to 0.1254 percentage points), and sometimes prioritizing the target group actually reduces its downstream utility. Is there sufficient evidence to indicate that optimizing the welfare of supervision can lead to meaningful improvements in downstream group-level consistency?
  2. Does the proposed supervised utility accurately reflect the generalizable learning gains? WAPA assigns zero to the supervised loss for obtaining labels, even though the underlying preferences are inherently uncertain. When the true preference probability and model prediction are both 0.5, obtaining labels still results in a positive estimated gain, despite the model being probabilistically correct. How do the authors distinguish between improvements in fixed pool supervision quality and genuine improvements in predicting preferences for unseen data?
  3. How much of WAPA's improvement comes from selective human feedback rather than stronger reference model supervision? In Appendix G.13, the reference model predictions typically provide better initial supervision than the base model. Since WAPA has already calculated these reference predictions, will the downstream DPO performance be more effective compared to directly using the reference model's pseudo-labels?
9 comments
AuthorsOpenReview ↗

Thank you for your comment on welfare-aware acquisition — Part 1: Objective and downstream gains

Weakness 1 — Part 1

The downstream evidence for welfare-aware allocation appears limited: changing welfare parameters produces utility differences of at most 0.1254 percentage points, and prioritizing the Target group can reduce its downstream utility.

We respectfully clarify that WAPA’s welfare objective encompasses both equal-weight supervision gains and additional distributional priorities. Under equal-weight mean welfare, (γ,ρ)=(1,1)(\gamma,\rho)=(1,1), WAPA already improves downstream Mean and Tail utility over the compared acquisition baselines (Section 4.3; Appendix G.9). The 0.1254 percentage-point span concerns changes between welfare settings. The small and sometimes reversed effects in Appendix G.12 motivate examining a broader parameter range, which we investigate through additional experiments with stronger welfare priorities.

  • Equal-weight mean welfare already guides effective label acquisition by comparing estimated supervision gains across personas and comparisons. Specifically, given different personas represented with different accuracy by the base model's pseudo-labels, WAPA estimates how much querying each comparison would improve its recipient's supervision quality, then allocates the shared budget according to those gains and the chosen welfare preferences. The score sijs_{ij} is the estimated supervision Brier loss, and sij/Ms_{ij}/M is the recipient's estimated utility gain from querying the label (Section 3.1, Eq. (5); Appendix D.1, Eq. (45)). The allocator values this improvement through the exact marginal welfare gain (Appendix C.1, Eq. (10), written in terms of U^i\widehat U_i):

    ΔJ^ij=wi[hρ!(U^i+sijM+ε)−hρ(U^i+ε)].\Delta\widehat J_{ij} =w_i\left[h_\rho\\!\left(\widehat U_i+\frac{s_{ij}}M+\varepsilon\right) -h_\rho(\widehat U_i+\varepsilon)\right]. Here, NN and MM are the numbers of personas and comparisons per persona; wiw_i encodes Target priority, and hρh_\rho specifies sensitivity to supervision disadvantage. At (γ,ρ)=(1,1)(\gamma,\rho)=(1,1), Section 2, Eqs. (2)–(3), give wi=1/Nw_i=1/N and h1(u)=uh_1(u)=u, hence

    J^1,1(S)=1N∑iU^i(S)+ε,ΔJ^ij=sijNM.\widehat J_{1,1}(S)=\frac1N\sum_i\widehat U_i(S)+\varepsilon, \qquad \Delta\widehat J_{ij}=\frac{s_{ij}}{NM}.

    Thus, at (γ,ρ)=(1,1)(\gamma,\rho)=(1,1), WAPA already directs queries toward comparisons with larger estimated supervision errors. Personas with more such comparisons can therefore receive more feedback: maximizing average supervision quality does not mean dividing the query budget equally. Lowering ρ\rho gives greater marginal value to supervision gains for recipients with lower current utility; γ\gamma sets the Target-to-non-Target persona weight ratio (Section 2, Eqs. (2)–(3)).

  • The method comparisons establish WAPA’s overall downstream gains. Section 4.3 reports higher Mean and Tail utility than the compared acquisition baselines under both welfare settings. At (γ,ρ)=(2,−5)(\gamma,\rho)=(2,-5) and a 20% query budget, WAPA recovers most of the downstream Mean utility improvement achieved by fully labeled training across four datasets and both Seen/Held-out splits (Figures 4–5). Our reference-gap metric measures how much of the improvement from all-pseudo-label DPO to all-ground-truth-label DPO is recovered:

    Recovery=100×UWAPA−UproxyUGT−Uproxy.\mathrm{Recovery}=100\times \frac{U_{\mathrm{WAPA}}-U_{\mathrm{proxy}}} {U_{\mathrm{GT}}-U_{\mathrm{proxy}}}.

    A recovery of 0% corresponds to the all-pseudo-label policy and 100% to the fully labeled reference (Appendix F.4). The main-text results are summarized below; ranges cover four datasets and both persona splits under (γ,ρ)=(2,−5)(\gamma,\rho)=(2,-5) (Section 4.3, lines 426–429).

    Acquisition methodQuery budgetMean reference-gap recovery (%)
    Random20%≈36.4–43.2
    APL20%≈18.9–26.6
    Entropy20%40.23–49.38
    BAL-PM20%49.72–59.05
    WAPA20%74.84–85.74

    Random and APL ranges are approximated from the rounded utilities in Appendix G.9, Tables S9–S10; the other ranges are reported directly in Section 4.3.

    At the same query budget, WAPA recovers substantially more of the reference improvement than the compared baselines.

AuthorsOpenReview ↗

Thank you for your comment on welfare-aware acquisition — Part 2: Additional group-level evidence

Weakness 1 — Part 2

Continuing our response to Weakness 1, we present the additional controlled experiments on downstream group outcomes below.

  • The additional experiments examine how stronger welfare priorities affect downstream group outcomes. We extend Appendix G.12 on North America, varying one welfare parameter at a time while holding query scores, the 20% query budget, group membership, and the training and evaluation protocol fixed.

    The tables report absolute utility values, 100×(1−Brierloss)100\times(1-\mathrm{Brier\\ loss}). Mean averages all personas, Tail averages the lowest-utility 10% separately for each policy, and Target averages the designated priority group.

    Fixed ρ=−50\rho=-50: vary γ∈1,5,50\gamma\in\\{1,5,50\\} to test downstream Target utility.

    γ\gammaMean utility (%)Tail utility (%)Target utility (%)
    184.482880.378085.2105
    584.483480.218685.3932
    5084.484380.175485.5912

    Target utility increases at each step as γ\gamma rises, gaining 0.3807 percentage points from γ=1\gamma=1 to 50 while Mean utility remains nearly unchanged. The accompanying Tail decrease makes the tradeoff from Target prioritization explicit.

    Beneficiary rate is the percentage of personas whose downstream utility increases relative to the starting setting of a parameter path. It measures how broadly the gains are shared, complementing the average utility reported above.

    For the fixed-γ\gamma path, we track the Initial supervision Tail: the same personas in the bottom 10% of estimated supervision utility before acquisition.

    ρ\rho (fixed γ=5\gamma=5; reference ρ=1\rho=1)Initial supervision Tail beneficiary rate
    −1057.1%
    −50100.0%

    At ρ=−50\rho=-50, every persona in this fixed initially disadvantaged group has higher downstream utility than under the ρ=1\rho=1 reference, with γ=5\gamma=5 fixed.

    For the fixed-ρ\rho path, we measure the beneficiary rate in the designated Target group.

    γ\gamma (fixed ρ=−50\rho=-50; reference γ=1\gamma=1)Target beneficiary rate
    528.6%
    5042.9%

    Increasing γ\gamma from 5 to 50 raises the Target beneficiary rate by 14.3 percentage points. Thus, more Target personas benefit relative to the same γ=1\gamma=1 reference.

Together, these results demonstrate that WAPA’s downstream benefits are already present under equal-weight mean welfare, while the welfare parameters provide additional control over how supervision gains are distributed. Our expanded experiments further show that increasing Target priority raises downstream Target utility in the tested settings. These findings directly address the parameter-sensitivity concern and illustrate the downstream allocation tradeoffs.

AuthorsOpenReview ↗

Thank you for your question about label uncertainty — Part 1: Score derivation and selection

Weakness 2 — Part 1

What drives WAPA’s performance gains? Appendix G.4 reports Mean utility differences of at most approximately 0.07 percentage points between the full score and discrepancy-only, with the full score sometimes performing worse. What additional benefit does the uncertainty component provide?

We thank the reviewer for raising this question. We would like to highlight that WAPA’s score is derived from expected supervision loss under the reference-estimated preference distribution. Discrepancy-only omits the label-uncertainty term in this expectation. The original Roleplay ablation shows small Average differences and distributional tradeoffs; our extended OpinionQA ablation shows consistent Average gains under (γ,ρ)=(1,1)(\gamma,\rho)=(1,1) and Tail gains under (2,−5)(2,-5).

Let pp denote the base model's predicted preference probability and Y∈0,1Y\in\\{0,1\\} the observed preference label. An unqueried comparison uses pp as its training target, with supervision loss (p−Y)2(p-Y)^2. Querying replaces this target with YY, removing that comparison's completion loss. Before observing YY, we estimate its probability using the reference-model mean qq (Section 3.1, Eq. (5), lines 190–203). The expected loss decomposes as follows (Appendix D.1, Eqs. (44)–(46), lines 1228–1250):

s=EY∼Bernoulli(q)[(p−Y)2]=q(1−p)2+(1−q)p2=(p−q)2⏟prediction discrepancy+q(1−q)⏟label uncertainty.s=\mathbb E_{Y\sim\mathrm{Bernoulli}(q)}[(p-Y)^2] =q(1-p)^2+(1-q)p^2 =\underbrace{(p-q)^2}_{\text{prediction discrepancy}} +\underbrace{q(1-q)}_{\text{label uncertainty}}.

The discrepancy term measures deviation from the reference mean; the uncertainty term captures the remaining expected loss of the realized label. Averaging supervision loss over MM comparisons per persona gives an estimated utility increment of s/Ms/M from this query (Appendix D.1, Eq. (45)). Both components therefore arise from the same loss calculation, with no separately chosen mixing coefficient. Appendix G.4's component ablation explores their empirical contributions to this mathematically derived score (Eq. (58), lines 1696–1700).

To see how uncertainty can change a selection, write dk=(pk−qk)2d_k=(p_k-q_k)^2 and uk=qk(1−qk)u_k=q_k(1-q_k), so sk=dk+uks_k=d_k+u_k. Under (γ,ρ)=(1,1)(\gamma,\rho)=(1,1), allocation reduces to selecting the highest scores. For two candidates with da>dbd_a>d_b, discrepancy-only prefers aa, but the full score prefers bb precisely when

ub−ua>da−db.u_b-u_a>d_a-d_b.

This follows directly from comparing db+ubd_b+u_b with da+uad_a+u_a in Eq. (58). Consider the following illustrative pair:

CandidateBase probability ppReference mean qqDiscrepancy ddUncertainty uuFull score ss
aa0.30.10.040.090.13
bb0.50.50.000.250.25

If one query is available, discrepancy-only selects aa. The full score selects bb, whose expected completion loss under the reference probabilities is larger despite exact agreement between pp and qq. Thus, uncertainty can identify valuable queries that discrepancy undervalues. In a larger pool, the selected set changes when such reversals cross the budget cutoff. Under nonlinear welfare, the score also affects the estimated persona-utility states used by the allocator, so the ablation measures both effects (Appendix G.4, lines 1715–1720). The example explains the mechanism; its empirical contribution is evaluated below.

Part 2 presents the extended OpinionQA experiments.

Reviewer_i2nnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to allocate a limited human preference annotation budget across individuals for pluralistic alignment. The authors propose Welfare-Aware Preference Acquisition (WAPA), which formulates preference acquisition as the maximization of a social welfare objective over individual supervision utilities. The objective incorporates both application-specific subgroup priorities and sensitivity to individuals with lower supervision quality.

WAPA estimates query values using predictions from multiple reference models and greedily selects comparisons according to their marginal welfare gains. The authors prove that this procedure exactly optimizes the estimated welfare objective under a fixed query budget and frozen predictions. They also provide theoretical results characterizing the effects of welfare weights and a conditional generalization bound connecting supervision quality to downstream prediction risk.

Experiments on Roleplay and OpinionQA demonstrate improvements in acquisition utility over active learning baselines. Downstream DPO experiments on Roleplay further show improved preference prediction for both seen and held-out personas. With a 20% human-label budget, WAPA recovers approximately 75–86% of the mean predictive-utility gap between fully pseudo-labeled and fully ground-truth-labeled training under the main Qwen3-4B setup.

Critical strengths
  1. A meaningful formulation of preference acquisition for pluralistic alignment.

The paper identifies an important problem that is not explicitly addressed by conventional active preference learning: allocating a shared annotation budget across individuals with different supervision needs.

The proposed social welfare objective provides a principled way to distinguish between expected query benefit, current supervision disadvantage, and application-designated subgroup priorities. I find this formulation valuable because it makes the distributional consequences of preference acquisition explicit, rather than optimizing only aggregate informativeness.

  1. Clear algorithmic design with theoretical analysis and extensive empirical evaluation.

WAPA provides a simple acquisition algorithm with an exact optimality guarantee for the estimated welfare objective. The paper also includes theoretical analysis of subgroup prioritization and sensitivity to low individual utility.

The empirical evaluation is relatively comprehensive, covering multiple datasets, welfare configurations, acquisition baselines, and downstream DPO experiments. In particular, the inclusion of both seen and held-out personas is valuable for assessing whether acquisition improvements transfer beyond the individuals whose labels were queried. The improvements over welfare-matched baselines provide evidence that the proposed acquisition strategy is effective.

Critical weaknesses
  1. The proposed supervision utility may not accurately reflect improvements in preference learning.

The proposed utility assigns zero supervision loss to queried comparisons, regardless of the uncertainty in the underlying preferences. For example, even when the base model perfectly predicts the true preference probability p^ij=pij∗=0.5\hat p_{ij}=p^*_{ij}=0.5, WAPA still assigns a positive query benefit. While querying resolves the realized label, it does not necessarily improve estimation of the underlying preference distribution. This raises concerns about whether maximizing supervision welfare is an appropriate surrogate for improving downstream preference alignment.

  1. Limited evidence that welfare-aware acquisition improves downstream pluralistic alignment.

Although varying the welfare parameters substantially affects acquisition-level Target and Tail utilities, these effects do not consistently translate into downstream alignment improvements. Appendix G.12 shows that downstream utilities vary by at most approximately 0.13 percentage points across welfare configurations, and increasing Target priority can even reduce downstream Target utility. These findings suggest that WAPA's improvements may primarily come from more effective label selection, rather than its welfare-aware allocation mechanism. Stronger evidence connecting welfare priorities to downstream alignment outcomes would be needed to substantiate the paper's central motivation.

Questions for authors

No.

4 comments
AuthorsOpenReview ↗

Thank you for your question about supervision utility and downstream learning

Weakness 1

Is supervision utility an appropriate surrogate for downstream preference learning? Queried comparisons receive zero supervision loss and positive acquisition benefit even when the base model already predicts the true preference probability. How does resolving realized labels support learning the underlying preference distribution?

We thank the reviewer for raising this question.

Supervision utility measures the quality of the acquired training targets; its relevance to downstream learning is supported by Theorem C.6’s population-risk bound and the downstream evaluations on unseen prompts and held-out personas in Section 4.3.

  • The zero loss refers to completion of the queried training label, not prediction of a fresh preference. When a comparison is queried, its training target is replaced by the observed label (Appendix C.4, Eq. 39). In your example, where the prediction and true preference probability both equal 0.5, this removes an expected completion loss of 0.25 for that pool entry, while uncertainty about a fresh label remains. Appendix D.1 discusses this case: the positive supervision gain reflects resolving that comparison’s realized label, not correcting an already accurate preference-probability estimate.

  • Theorem C.6 connects mean supervision utility to a population-risk upper bound. Under the assumptions in Appendix C.4, with probability at least 1−δ1-\delta,

    R(fθ^)⏟population Brier risk≲1−1N∑i=1NUi(S)⏟residual supervision error+Rn(F)⏟model-class complexity+(1N+1M)log⁡2δ⏟sampling term+εfit⏟training fit.\underbrace{R(f_{\hat\theta})}_{\text{population Brier risk}} \lesssim \underbrace{1-\frac{1}{N}\sum_{i=1}^{N}U_i(S)}_{\text{residual supervision error}} +\underbrace{\mathfrak{R}_n(\mathcal F)}_{\text{model-class complexity}} +\underbrace{\sqrt{\left(\frac{1}{N}+\frac{1}{M}\right)\log\frac{2}{\delta}}}_{\text{sampling term}} +\underbrace{\varepsilon_{\mathrm{fit}}}_{\text{training fit}}.

    Here, R(f)=Ea,x,Y[(f(a,x)−Y)2]R(f)=\mathbb E_{a,x,Y}[(f(a,x)-Y)^2] evaluates fresh personas and prompts; Ui(S)U_i(S) is persona ii's expected completed-supervision utility, NN and MM are the numbers of personas and prompts, and n=min⁡N,Mn=\min\\{N,M\\}. The bound requires independent persona/prompt sampling, a model class fixed before sampling, and empirical KL fitting error at most εfit\varepsilon_{\mathrm{fit}}. Conditional on the sampled inputs, labels are independent, and both the frozen predictions and the acquisition set SS are independent of all pool labels; ≲\lesssim hides a universal constant.

    Under these assumptions, higher mean supervision utility tightens the population-risk upper bound when the other terms are fixed. This provides a theoretical basis for the acquisition surrogate and is compatible with your example: the connection is a bound on downstream risk, not a guarantee that every queried label improves estimation of the underlying preference probability.

  • The downstream evaluations test generalization beyond the acquisition pool. The submitted paper reports trained-policy utility on unseen prompts for Seen personas and on Held-out personas whose labels were never queried. At the same 20% query budget and (γ,ρ)=(2,−5)(\gamma,\rho)=(2,-5), the results are summarized below. Ranges cover four datasets and both evaluation splits (Section 4.3, Figures 4–5, lines 426–429; Appendix G.9, Tables S9–S10).

    Acquisition methodQuery budgetMean reference-gap recovery (%)
    Random20%≈36.4–43.2
    APL20%≈18.9–26.6
    Entropy20%40.23–49.38
    BAL-PM20%49.72–59.05
    WAPA20%74.84–85.74

    Following Appendix F.4, reference-gap recovery is

    Recovery=100×Umethod−UproxyUGT−Uproxy,\mathrm{Recovery}=100\times\frac{U_{\mathrm{method}}-U_{\mathrm{proxy}}}{U_{\mathrm{GT}}-U_{\mathrm{proxy}}},

    where UproxyU_{\mathrm{proxy}} and UGTU_{\mathrm{GT}} are downstream Mean utilities after training on all base-model pseudo-labels and all ground-truth labels, respectively. Random and APL ranges are calculated from the rounded values in Tables S9–S10; the other ranges are reported in Section 4.3.

    With only 20% of labels queried, WAPA recovers approximately 75–86% of the fully labeled reference improvement, compared with approximately 50–59% for BAL-PM and less for the other baselines. Because this evaluation includes unseen prompts and Held-out personas, these are improvements in trained-policy prediction beyond the completed acquisition pool. They provide empirical transfer evidence alongside Theorem C.6's population-risk bound.

AuthorsOpenReview ↗

Thank you for your comment on welfare-aware acquisition — Part 1: Objective and downstream gains

Weakness 2 — Part 1

Do welfare priorities improve downstream group alignment beyond effective label selection? In Appendix G.12, welfare settings change downstream utility by at most approximately 0.13 percentage points, and higher Target priority can reduce Target utility despite substantial acquisition-level changes.

We respectfully clarify that WAPA’s welfare objective encompasses both equal-weight supervision gains and additional distributional priorities. Under equal-weight mean welfare, (γ,ρ)=(1,1)(\gamma,\rho)=(1,1), WAPA already improves downstream Mean and Tail utility over the compared acquisition baselines (Section 4.3; Appendix G.9). The approximately 0.13 percentage-point span concerns changes between welfare settings. The small and sometimes reversed effects in Appendix G.12 motivate examining a broader parameter range, which we investigate through additional experiments with stronger welfare priorities.

  • Equal-weight mean welfare already guides effective label acquisition by comparing estimated supervision gains across personas and comparisons. Specifically, given different personas represented with different accuracy by the base model's pseudo-labels, WAPA estimates how much querying each comparison would improve its recipient's supervision quality, then allocates the shared budget according to those gains and the chosen welfare preferences. The score sijs_{ij} is the estimated supervision Brier loss, and sij/Ms_{ij}/M is the recipient's estimated utility gain from querying the label (Section 3.1, Eq. (5); Appendix D.1, Eq. (45)). The allocator values this improvement through the exact marginal welfare gain (Appendix C.1, Eq. (10), written in terms of U^i\widehat U_i):

    ΔJ^ij=wi[hρ!(U^i+sijM+ε)−hρ(U^i+ε)].\Delta\widehat J_{ij} =w_i\left[h_\rho\\!\left(\widehat U_i+\frac{s_{ij}}M+\varepsilon\right) -h_\rho(\widehat U_i+\varepsilon)\right]. Here, NN and MM are the numbers of personas and comparisons per persona; wiw_i encodes Target priority, and hρh_\rho specifies sensitivity to supervision disadvantage. At (γ,ρ)=(1,1)(\gamma,\rho)=(1,1), Section 2, Eqs. (2)–(3), give wi=1/Nw_i=1/N and h1(u)=uh_1(u)=u, hence

    J^1,1(S)=1N∑iU^i(S)+ε,ΔJ^ij=sijNM.\widehat J_{1,1}(S)=\frac1N\sum_i\widehat U_i(S)+\varepsilon, \qquad \Delta\widehat J_{ij}=\frac{s_{ij}}{NM}.

    Thus, at (γ,ρ)=(1,1)(\gamma,\rho)=(1,1), WAPA already directs queries toward comparisons with larger estimated supervision errors. Personas with more such comparisons can therefore receive more feedback: maximizing average supervision quality does not mean dividing the query budget equally. Lowering ρ\rho gives greater marginal value to supervision gains for recipients with lower current utility; γ\gamma sets the Target-to-non-Target persona weight ratio (Section 2, Eqs. (2)–(3)).

  • The method comparisons establish WAPA’s overall downstream gains. Section 4.3 reports higher Mean and Tail utility than the compared acquisition baselines under both welfare settings. At (γ,ρ)=(2,−5)(\gamma,\rho)=(2,-5) and a 20% query budget, WAPA recovers most of the downstream Mean utility improvement achieved by fully labeled training across four datasets and both Seen/Held-out splits (Figures 4–5). Our reference-gap metric measures how much of the improvement from all-pseudo-label DPO to all-ground-truth-label DPO is recovered:

    Recovery=100×UWAPA−UproxyUGT−Uproxy.\mathrm{Recovery}=100\times \frac{U_{\mathrm{WAPA}}-U_{\mathrm{proxy}}} {U_{\mathrm{GT}}-U_{\mathrm{proxy}}}.

    A recovery of 0% corresponds to the all-pseudo-label policy and 100% to the fully labeled reference (Appendix F.4). The main-text results are summarized below; ranges cover four datasets and both persona splits under (γ,ρ)=(2,−5)(\gamma,\rho)=(2,-5) (Section 4.3, lines 426–429).

    Acquisition methodQuery budgetMean reference-gap recovery (%)
    Random20%≈36.4–43.2
    APL20%≈18.9–26.6
    Entropy20%40.23–49.38
    BAL-PM20%49.72–59.05
    WAPA20%74.84–85.74

    Random and APL ranges are approximated from the rounded utilities in Appendix G.9, Tables S9–S10; the other ranges are reported directly in Section 4.3.

    At the same query budget, WAPA recovers substantially more of the reference improvement than the compared baselines.

These comparisons establish WAPA’s overall downstream gains. In Part 2 of this response, we examine how changing welfare priorities affects downstream group outcomes through additional controlled experiments.

AuthorsOpenReview ↗

Thank you for your comment on welfare-aware acquisition — Part 2: Additional group-level evidence

Weakness 2 — Part 2

Continuing our response to Weakness 2, we present additional experiments examining how stronger welfare priorities affect downstream group outcomes. We extend Appendix G.12 on North America, varying one welfare parameter at a time while holding query scores, the 20% query budget, group membership, and the training and evaluation protocol fixed.

The tables report absolute utility values, 100×(1−Brierloss)100\times(1-\mathrm{Brier\\ loss}). Mean averages all personas, Tail averages the lowest-utility 10% separately for each policy, and Target averages the designated priority group.

Fixed ρ=−50\rho=-50: vary γ∈1,5,50\gamma\in\\{1,5,50\\} to test downstream Target utility.

γ\gammaMean utility (%)Tail utility (%)Target utility (%)
184.482880.378085.2105
584.483480.218685.3932
5084.484380.175485.5912

Target utility increases at each step as γ\gamma rises, gaining 0.3807 percentage points from γ=1\gamma=1 to 50 while Mean utility remains nearly unchanged. The accompanying Tail decrease makes the tradeoff from Target prioritization explicit.

Beneficiary rate is the percentage of personas whose downstream utility increases relative to the starting setting of a parameter path. It measures how broadly the gains are shared, complementing the average utility reported above.

For the fixed-γ\gamma path, we track the Initial supervision Tail: the same personas in the bottom 10% of estimated supervision utility before acquisition.

ρ\rho (fixed γ=5\gamma=5; reference ρ=1\rho=1)Initial supervision Tail beneficiary rate
−1057.1%
−50100.0%

At ρ=−50\rho=-50, every persona in this fixed initially disadvantaged group has higher downstream utility than under the ρ=1\rho=1 reference, with γ=5\gamma=5 fixed.

For the fixed-ρ\rho path, we measure the beneficiary rate in the designated Target group.

γ\gamma (fixed ρ=−50\rho=-50; reference γ=1\gamma=1)Target beneficiary rate
528.6%
5042.9%

Increasing γ\gamma from 5 to 50 raises the Target beneficiary rate by 14.3 percentage points. Thus, more Target personas benefit relative to the same γ=1\gamma=1 reference.

These additional experiments show that increasing Target priority raises downstream Target utility in the tested settings, while stronger sensitivity to supervision disadvantage extends benefits across the initially disadvantaged group. The accompanying Tail tradeoff shows how welfare priorities affect the distribution of downstream gains.

AuthorsOpenReview ↗

Following up on your concerns — we would welcome your feedback

Thank you again for your time and feedback. We wanted to briefly return to the two main points in our replies:

  • The surrogate has a conditional connection to learning, without requiring every query to improve probability estimation. In your 0.5 example, querying resolves the training label while uncertainty about fresh preferences remains (Appendix D.1). Under its stated assumptions and with other terms fixed, Theorem C.6 links higher mean supervision utility to a tighter population-risk upper bound. This explains why the example is compatible with the surrogate; Section 4.3 separately tests whether acquisition gains transfer to unseen prompts and held-out personas.
  • Effective label selection is part of the welfare objective itself. Equal-weight mean welfare allocates queries by their expected supervision gains across people; equal weights do not mean equal allocations. Section 4.3 already reports downstream Mean and Tail gains over acquisition baselines under both welfare settings. At 20% budget, the joint setting recovers 75–86% of the Mean improvement from all-pseudo-label to fully labeled training, versus 50–59% for BAL-PM, across four datasets and both persona splits. Appendix G.12 is a parameter sensitivity analysis of downstream group outcomes, rather than a comparison against other acquisition methods. To examine your specific concern about the incremental effects of priorities, our North America extension varies stronger priorities at a fixed budget and reports group utilities, beneficiary rates, and Target/Tail tradeoffs.

We would appreciate hearing whether these explanations address your concerns and, if so, your reconsideration of the assessment. With the author–reviewer discussion closing on Nov 18, we would especially appreciate hearing from you during this final day. Your feedback would help us further improve the paper.

Thank you again for your time.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to the Area Chair and all reviewers

Dear Area Chair and Reviewers,

Thank you for your time and feedback. As the discussion period draws to a close, we would appreciate your feedback on our detailed replies. For ease of reference, we highlight the main concerns below, distinguishing evidence already present in the submitted paper from additional experiments conducted during the rebuttal.

  • Supervision utility and downstream learning (i2nn, tsWE). The submitted paper already provides a population-risk bound in Theorem C.6 and downstream evaluations on unseen prompts and held-out personas in Section 4.3. With a 20% query budget, WAPA recovers approximately 75–86% of the fully labeled reference improvement across four datasets and both persona splits. Our replies explain the distinction between training-label completion and fresh-preference prediction, the bound's stated assumptions, and the calibrated 0.5 example discussed in Appendix D.1.
  • Welfare priorities and group-level outcomes (i2nn, tsWE). Section 4.3 evaluates overall downstream method gains, while Appendix G.12 separately examines welfare-parameter effects. Our replies make this distinction explicit and supplement the existing analysis with controlled stronger-priority experiments on North America at a fixed query budget. These report Target and Tail utility and beneficiary rates within the Target and initially disadvantaged groups, characterizing setting-dependent distributional effects and tradeoffs.
  • Acquisition-score components (tsWE). The score decomposition and component ablation in Appendix G.4 explain and evaluate the contributions of discrepancy and uncertainty. Building on this analysis, we extended the ablation to OpinionQA: retaining uncertainty improves Mean acquisition utility under the mean objective and Tail utility under the joint setting across all four regions, with a Target-utility tradeoff in the latter.
  • Direct reference-model supervision (tsWE). Appendix G.13 compares reference-based supervision quality at the acquisition stage. To test the proposed downstream alternative directly, we added the requested reference-only DPO experiment. WAPA achieves higher downstream Mean, Tail, and Target utility than direct training on reference-model pseudo-labels across all four datasets and both evaluation splits under both evaluated WAPA settings.
  • Human-preference evidence and conceptual clarity (5QH1). The submitted paper already includes acquisition experiments on OpinionQA's survey-derived human preferences (Appendix E.2; Table S3), alongside the controlled Roleplay experiments. We additionally validated 200 Roleplay comparisons with five independent human annotators using the supplied personas: agreement with the original labels averaged 75.8% (individual rates: 73.5%–77.5%). Our replies also point to Section 2’s definitions of persona, preference, supervision utility, and welfare, and explain the conditions and proof of exact acquisition optimality in Theorem 3.3 and Appendix C.3.

We welcome any remaining questions and would appreciate knowing whether the evidence and explanations in our replies address your concerns. Where they do, we kindly ask you to reconsider your assessment accordingly.

Thank you again for your time and consideration.

AuthorsOpenReview ↗

Revised PDF: additional evidence and clarifications

Dear Area Chair and Reviewers,

Thank you for the time, care, and thoughtful questions you have brought to this discussion. We are glad that our responses have helped address some of the concerns raised here. We have prepared a revised PDF incorporating the additional evidence and clarifications, including the human validation of Roleplay labels. Additions are marked in blue for ease of review.

  • Downstream effectiveness and welfare priorities. Section 4.3 already evaluates WAPA against acquisition baselines on unseen prompts and held-out personas. The revision clarifies the distinction between these method gains and the parameter sensitivity analysis in Appendix G.12, which we extend with stronger priorities, group utilities, and beneficiary rates.
  • The contribution of label uncertainty. Appendix G.4 adds the OpinionQA component ablation. Retaining uncertainty improves Mean acquisition utility under mean welfare and Tail utility under joint welfare across all four regions. Appendix D.1 explains how both score components follow from the expected supervision loss.
  • Direct reference-model supervision. Appendix G.13 adds the requested reference-only DPO comparison. WAPA achieves higher downstream Mean, Tail, and Target utility across all four datasets and both persona splits under both evaluated settings.

We hope the revised PDF makes the evidence easier to assess. We would be grateful to hear whether our replies and these additions address your concerns, or what we could clarify further. Your feedback has helped us strengthen the paper, and we truly appreciate your time and continued consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.