- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates how to assign a limited human-labeling budget across preference comparisons to maximize social welfare. The proposed method, WAPA, estimates each comparison’s expected label error using three reference LLMs and greedily queries the comparison with the largest marginal gain in estimated welfare. The authors prove that this greedy rule exactly maximizes the estimated welfare objective under a fixed budget.
Critical strengths
- The greedy assignment is simple and cheap. Moreover, its effectiveness is justified by theory.
- The experimental ablations are extensive.
Critical weaknesses
- The preference labels in the Roleplay experiments are all simulated. Real human evaluation is only involved at the last stage to grade the generated responses.
- The personas are from FSPO, which uses GPT-4o to generate synthetic personas and preference labels. Therefore, the preferences are simulated by GPT-4o for different personas instead of being collected from real humans.
- The paper involves multiple different concepts, e.g., persona, preference, and welfare, but the presentation is not clear enough.
Questions for authors
Could the authors clarify the assumptions required to guarantee global optimality of WAPA for the estimated welfare objective?
6 comments
Thank you for your comment on the validity of synthetic preference labels
Weakness 2
The FSPO personas and preference labels are synthetic rather than collected from real people, raising concerns about the external validity of the preference experiments.
We respectfully clarify that Roleplay serves as a controlled testbed for feedback allocation, with complementary evidence from FSPO’s real-user transfer study and our OpinionQA experiments. To directly examine the validity of Roleplay’s synthetic labels, we additionally compared them with human judgments on the same preference comparisons.
Roleplay provides the coverage and comparability needed to test how a shared feedback budget should be distributed across heterogeneous preferences. As disclosed in Appendix E.1 (lines 1328–1332), its personas and preference labels are synthetic. Its explicit profiles and repeated comparisons over a common prompt set let us compare allocation methods at the same budget, train policies on the resulting labels, and evaluate on separate prompts and personas (Appendix E.1, lines 1333–1344). This coverage matters: an unqueried comparison must still have an evaluation label if we are to measure the effect of choosing another query allocation. Collecting such repeated judgments from each real participant is demanding, so Roleplay offers a practical controlled testbed for this question.
The underlying FSPO work also provides evidence that training on these synthetic preferences can transfer to real-user personalization. In FSPO, Section 7, a study with 50 real participants reports personalized-response win rates of 68.2% against the base model and 72.3% against supervised fine-tuning. Its construction procedure also filters labels that change when response order is swapped, and persona refinement reduces repeated-label entropy from 0.64 to 0.13 nats (Section 6). The consistency checks support label stability, while the human study supports synthetic-to-real transfer. These findings support using FSPO as a testbed. Our additional experiment below directly evaluates agreement between synthetic labels and human judgments on the same comparisons.
To address label validity directly, we conducted an additional human validation experiment. We randomly sampled 200 preference comparisons from Roleplay. Five human annotators independently evaluated each comparison using the supplied persona, with the original labels concealed until all judgments were finalized. Agreement with the original labels averaged 75.8% across annotators, with individual agreement rates ranging from 73.5% to 77.5%. This provides supplementary evidence that Roleplay’s synthetic labels agree with human judgments conditioned on the supplied personas.
Our response to Weakness 1 presents the complementary OpinionQA evidence: allocation gains also hold on survey-derived preferences across four regional subsets. Roleplay enables controlled acquisition and downstream evaluation, while OpinionQA tests acquisition beyond synthetic preferences. The remaining need for direct feedback from represented users is acknowledged in Appendix B (lines 852–858). Richer human preference benchmarks with repeated judgments per participant would make this evaluation substantially stronger and are an important direction for future dataset work.
Clarification of persona, preference, and welfare
Weakness 3
The distinctions among persona, preference, and welfare need clearer explanation.
The concepts describe successive levels of the acquisition problem: whose preferences are represented, what labels express those preferences, and how supervision quality is aggregated across people. These concepts are introduced in the Introduction and formalized in Section 2 through the preference-acquisition setting, supervision utility (Eq. (1)), and social welfare (Eq. (2)). Section 3.1 explains how they determine WAPA’s acquisition rule. We summarize their roles and connections below:
| Concept | Meaning in this paper | Original location |
|---|---|---|
| Persona | A user profile conditioning the preference comparisons and the policy. It identifies whose preferences the model should represent. | Section 2, lines 105–107; Figure 1A, lines 108–127 |
| Preference | For persona and prompt , the label indicates which of two fixed responses is preferred; allows this preference to be stochastic. | Section 2, lines 128–133 |
| Supervision utility | The quality of the completed preference labels for one persona, measured as one minus average expected Brier completion loss. | Section 2, Eq. (1), lines 134–143 |
| Social welfare | A weighted aggregation of these persona-level supervision utilities, determining how the shared query budget is allocated. | Section 2, Eqs. (2)–(4), lines 145–161 |
Specifically, the individual quantity is (Section 2, Eq. (1), lines 134–143)
A query replaces a pseudo-label with the observed preference label, removing its completion loss and increasing that persona's supervision utility. The population objective is (Section 2, Eq. (2), lines 145–153)
Here, fixed positive weights specify persona priorities, while controls sensitivity to low supervision utility. For , is larger at lower utility, so an equal utility increment receives more value for a lower-utility persona when weights are equal. Section 2, Eq. (3), lines 154–160, sets the weight ratio between a designated Target persona and a non-Target persona to . Thus, expresses subgroup priority and expresses sensitivity to supervision disadvantage.
For example, two personas can prefer different responses to the same retirement question, as in Figure 1A (lines 108–127). If the model supplies poorer preference labels for one persona, that persona has lower supervision utility. Welfare specifies how much priority to give improving that supervision when labels are scarce. In this paper, “welfare” therefore aggregates the quality of preference supervision; it does not measure the person's real-world wellbeing or assign a value to the content of their preferences. Section 3.1, Eq. (5), lines 190–203, estimates these utilities before querying, and Section 3.2, Eq. (7), lines 218–227, uses the completed labels for downstream alignment.
Thank you for your comment on evidence from real human preferences
Weakness 1
We respectfully clarify that our acquisition evaluation also includes OpinionQA, which contains preferences derived from real human survey responses. WAPA achieves higher Mean and Tail acquisition utility than the compared baselines across all four regional subsets and both welfare settings. Section 4 (lines 298–300) introduces both benchmarks, and Appendix E.2 (lines 1379–1388) describes the OpinionQA evaluation. Roleplay uses synthetic preference labels; OpinionQA provides a complementary test of allocation using preferences derived from respondents' survey answers.
OpinionQA originates from the Pew Research Center's American Trends Panel. We use the processed release from Group-Adaptive-Elicitation, covering questions from survey waves W50, W54 and W92 on family, economic inequality and political typology. Our four regional subsets contain 3,130 respondents and 30 questions per respondent, with 3–5 answer options per question (Appendix E.2, lines 1379–1388). This evaluates whether a limited query budget can improve the representation of heterogeneous survey preferences.
The existing results in Table S3 show higher Mean and Tail acquisition utility than all budgeted baselines in every region under both welfare settings. We reproduce the WAPA results below. Values are utilities in percent; parentheses give percentage-point gains over the strongest of Random, Entropy and Reference Disagreement for each region, setting and metric, all at a 20% query budget. These differences are calculated from the displayed Table S3 values (Appendix G.1, lines 1621–1634).
Thus, the acquisition gains extend beyond synthetic Roleplay labels: Mean and Tail improve in all eight region–setting comparisons, and Target improves in seven.
OpinionQA tests acquisition on survey choices; it does not establish downstream personalization of generated responses for real users. Our generated-response human evaluation provides a separate check of suitability for supplied profiles. Appendix B (lines 852–858) already states that this is not direct satisfaction feedback from the represented users and calls for broader evaluation with those users. Together, the benchmarks establish controlled allocation and downstream results on Roleplay and acquisition results on survey-derived preferences, while direct validation with represented users remains an important next step.