- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses a fundamental issue with preference data, where biases among data contributors may lead to an imbalanced representation of preferences across different demographic groups. To address this issue, the paper proposes a sortition-based approach that constructs Soft and Hard panels to filter and reweight preference data according to demographic composition. The paper provides a theoretical analysis to demonstrate the effectiveness of the proposed approach and further validates it through experiments. The experimental results show that models aligned using this approach better reflect American values regarding AI.
Critical strengths
- The paper raises a meaningful and research-worthy issue regarding bias in preference data.
- The paper is written and organized clearly.
- The authors honestly acknowledge the limitations of their work.
Critical weaknesses
- The paper overclaims its applicability to both DPO and RLHF in the title and the second contribution. However, the authors only conduct experiments with DPO, which strictly speaking does not belong to RLHF. To support the claim about RLHF, the authors should apply the proposed data processing method to RLHF (e.g., training a reward model) and analyze the experimental results.
- The evaluation uses only GPT-5.2 as the LLM-as-a-judge for ranking, which may introduce GPT-5.2's own preference biases. Using at least three models from different model families to vote on the rankings would provide a fairer evaluation.
- The PRISM Alignment and Community Alignment experiments show inconsistent results, with Hard > US-Rep > Soft and Soft > Full-US > Hard, respectively. This inconsistency raises concerns about the actual effectiveness of the proposed Hard and Soft panels, as discussed in Question 2.
- Appendix A.4 shows that only 56 out of 145 raters reside in the U.S., meaning most are not U.S. residents. Although the panel matches the U.S. demographic composition, differences in countries, environments, and education may lead to different values. Despite the authors' claim that this is a practical compromise, such differences may weaken the credibility of the conclusions, even if the results show better alignment with overall U.S. values.
- The proposed method has significant limitations, as it requires detailed demographic information about preference annotators, as provided by PRISM. Since most preference datasets lack such information, I doubt the potential for further research and the scalability of this approach.
Questions for authors
- The authors fix the number of training steps for a fair comparison. However, since Hard Panel has fewer training samples, this results in more training epochs and may lead to overfitting. Why not downsample Full PRISM and Soft Panel to achieve comparable numbers of training steps, epochs, raters, and preference pairs for a fairer comparison?
- In the "Cross-dataset replication on Community Alignment" section, the authors state that the choice between Hard Panel and Soft Panel depends on data quality. However, given a preference dataset, how should I decide which method to use? The dataset's quality may be unknown and is not necessarily quantifiable. Could the authors provide concrete criteria for choosing between Hard Panel and Soft Panel?
4 comments
W4: Only 56/145 original panelists are U.S. residents.
This is a substantive limitation that demographic matching cannot remove. A mixed-locale panel satisfying U.S. demographic margins is not a representative sample of U.S. residents. We now state the 56/145 composition in the main limitations and add a U.S.-only pool of 348 raters, with feasible panel size k=141.
| U.S.-only condition | Held-out accuracy [95% CI] | U.S. survey similarity | GPT BT |
|---|---|---|---|
| Hard | .513 [.444, .571] | .720 | −.012 |
| Soft | .511 [.454, .565] | .715 | .230 |
| Full (U.S.) | .490 [.428, .552] | .732 | .025 |
| US-Rep | .508 [.440, .571] | .703 | .241 |
| Base | — | .746 | −.484 |
Soft is above Full on the constitutional proxy, the human-accuracy difference is uncertain, and survey similarity is worse. Hard does not exceed Full on the constitutional proxy. These results do not establish improved population representation in the all-U.S. setting. They appropriately restrict the conclusion from the mixed-locale experiment.
Restricting locale also changes the available data and feasible design, so we do not treat this as a one-variable ablation.
W5: Demographic metadata and scalability.
The requirement is real. The method applies where consented rater attributes, a justified target public, and sufficient coverage are available. It is not a repair for metadata-free datasets. Metadata-poor settings may use a smaller set of ethically collected attributes, but narrower attributes also support narrower representativeness claims. We do not propose inferring sensitive attributes from text.
The selection/weighting layer is compatible with existing per-example losses and does not require a larger model or an additional forward pass. It does incur metadata-collection, quota-specification, and sortition-solver costs. Multiple attributes or small cells may make quotas infeasible or weights concentrated; those outcomes must be reported rather than hidden. The contribution’s scope is to make these participation assumptions visible and auditable, not to solve all feedback-data coverage problems.
Q1: Match steps, epochs, raters, and preference-pair counts.
The original shared-step comparison did not isolate these dimensions. With a fixed effective batch size, optimizer steps determine total row draws, while passes depend on dataset size. One cannot simultaneously hold steps and passes fixed across datasets of different sizes. Distinct raters and within-rater pair counts add further constraints. We therefore separate the controls rather than call one comparison “fully matched.”
We use a 5,095-pair Hard training budget and compare:
- A same-size random-rater panel.
- Uniform Full subsampling.
- Soft-weight-proportional row subsampling, trained unweighted.
Pair matching alone does not ensure identical per-rater exposure or effective sample size. We run both a shared 3,538-step regime and two passes through each condition’s own data.
| Condition | Same steps: Judge BT | Same steps: Accuracy | Same steps: Survey | Same passes: Judge BT | Same passes: Accuracy | Same passes: Survey |
|---|---|---|---|---|---|---|
| Hard | .187 | .551 | .729 | .025 | .579 | .748 |
| Random panel | .373 | .544 | .706 | .099 | .597 | .751 |
| Full-sub | .136 | .539 | .671 | −.032 | .591 | .739 |
| Soft-sub | −.171 | .533 | .720 | .091 | .578 | .768 |
| Soft (all data) | .091 | .597 | .761 | .288 | .594 | .762 |
| Full (all data) | .052 | .595 | .734 | .113 | .600 | .733 |
Same steps: 3,538 optimizer steps. Same passes: two passes through each condition’s own data. Metrics are Gemini constitutional-judge BT, post-stratified held-out accuracy, and U.S. survey similarity. BT values are comparable within a regime’s candidate set.
Three conclusions survive these controls:
- Random exceeds Hard on the constitutional proxy under both regimes, so the original judge advantage does not identify quota composition.
- Soft-sub exceeds Full-sub on survey similarity at equal pair counts: +.049 at shared steps and +.029 at equal passes. This suggests that the weighting distribution matters beyond raw pair count.
- All-data Soft and Full are close on prediction accuracy, while Soft has higher survey similarity in both regimes.
These are complementary observations, not a claim that all confounds have been eliminated.
We also vary panel draw (seeds 42, 7, and 13) and training seed. Random exceeds Hard on constitutional BT in each of the four reported comparisons. Hard survey similarity varies from .697 to .729 across panel draws, which is why we now present panel sensitivity rather than one favorable realization. Length/refusal adjustment does not remove the Random–Hard gap in three of the four reported checks, but this association does not rule out every response-style effect.
Q2: How should a practitioner choose Hard or Soft without knowing “data quality”?
We have replaced that recommendation with criteria that do not require an unobserved data-quality label.
-
When all feedback is available, start with fixed-normalizer Soft. It retains comparisons and removes panel-draw randomness from the objective. Evaluate it against Full on the intended outcome. This is a practical default, not a universal empirical dominance claim. Raw linear weights preserve the exact expected-panel identity; floors and other transformations must be disclosed and checked separately.
-
Use Hard when the deliverable or collection design requires a realized panel. Examples include a named, auditable, quota-feasible electorate or a budget permitting collection from only k raters. Report multiple draws and a size-matched random-panel baseline.
-
Choose using a separate validation split and an appropriate target. Use population-matched held-out preferences or survey distributions when those are the intended outcomes. Use constitutional evaluation for consensus compliance. The test split used in this response must not also be treated as an untouched test after method selection.
Weight concentration, Kish effective sample size, Hard data fraction, and effective pair counts are useful diagnostics for unstable weighting and regime confounding. They are not validated predictors of which method will represent a public better. Where population-matched validation is unavailable, that limitation should remain explicit.
Thank you for the detailed response. I have a clear understanding of the additional experiments and the corresponding explanations.
However, I still have significant concerns regarding Question 2. If additional data analysis is required to determine which specific method should be applied, this would substantially undermine the practical applicability of the proposed work. This is because this work aims to propose a methodology rather than a data selection or preprocessing. A proposed methodology should be clear and easily reproducible, without requiring extensive or complicated preliminary steps to determine how it should be applied.
We thank the reviewer for a careful review. Below we address each of the concerns made in the review and also included it in the revised version of the paper. Note that in the revised version of the paper, all the changes are highlighted in blue.
W1: Claims about RLHF are broader than the DPO experiments.
We have changed the title to Pluralistic Preference Alignment via Sortition-Weighted Preference Learning (please see the revised pdf) and narrowed the contribution language. DPO is related to the KL-regularized preference-learning formulation, but our DPO experiments do not constitute an online RLHF pipeline. We do not use that relationship to claim empirical coverage of reward-model-plus-policy-optimization methods.
We additionally trained weighted Bradley–Terry reward models using the same data-construction principle:
LRM(φ)=A1∑jaj[−logsigmoid(rφ(xj,yj+)−rφ(xj,yj−))].
Each reward model selects from a shared pool of 16 base-model candidates per prompt. This tests reward-model training and selection while keeping the candidate pool fixed. It is not PPO or another online policy-optimization experiment.
Reward-model extension. Bradley–Terry (BT) scores are jointly fitted within this selection comparison.
Every trained selector has a better BT point estimate than the no-selection baselines, but the sortition-weighted reward model does not improve on Full. Full has the highest accuracy point estimate, and the reported uncertainty does not establish superiority among trained selectors. We report this result as an empirical scope check rather than evidence of a general RLHF benefit.
W2: Reliance on one judge family.
We added three judge families. The original 8B checkpoints have the same model-level ordering under all four judges:
Model-level Kendall τ=1 across all judge pairs. Per-question cross-family agreement is lower (τ=.72–.75; top-1 agreement .70–.74), close to GPT self-agreement (τ≈.775).
BT values are fitted separately within each judge and must not be compared across columns as a shared scale. This result addresses dependence on one judge family. It does not eliminate correlated model biases or make the constitutional rubric a population-distribution measure.
W3: Opposite Hard/Soft orderings on PRISM and Community Alignment.
We agree that the reversal prevents a general Hard-over-Soft claim. The revision retains this result and adds two datasets , we also used a different model family (Qwen3-8B-Base):
Constitutional BT scores. Each row is a separate comparison; magnitudes across rows are not comparable.
Soft has a positive Full-relative point difference on all four datasets, but the Community Alignment gap is small and its marginal intervals overlap. We do not claim statistically established gains everywhere.
Hard’s advantage depends on the regime. PRISM’s random-panel controls, together with a near-tie between Hard and Random on CoVal, show why its judge score cannot be interpreted as a composition effect. We have replaced the untested “data-quality tradeoff” explanation with a description of the observed regimes and an explicit statement that data quality was not independently manipulated.
References for new datasets:
Coval : Learning values-aware rubrics from the crowd (https://alignment.openai.com/coval/)
OvertonBench : Benchmarking Overton Pluralism in LLMs (https://arxiv.org/abs/2512.01351)