acceptodds
Under review as a conference paper at ICLR 2027

Pluralistic Preference Alignment via Sortition-Weighted RLHF

Abstract

Preference-based alignment methods such as reinforcement learning from human feedback (RLHF) and Direct Preference Optimization (DPO) implicitly determine whose values a model learns through the composition of their rater pools. Because these pools are often convenience samples, the resulting preference signal may systematically overrepresent some demographic groups while marginalizing others. We introduce sortition-weighted preference learning, a representativeness-aware framework that incorporates algorithmic sortition—the sampling mechanism underlying citizens' assemblies—into preference-based fine-tuning. We study two implementations. Hard Panel trains exclusively on preferences from a quota-satisfying mini-public selected through sortition, whereas Soft Panel retains the full dataset while weighting raters according to their inclusion probabilities under the same sampling design. Using PRISM rater demographics and preference data, we fine-tune Llama models with DPO and evaluate their behavior against a 75-clause constitution elicited from a representative panel of U.S. residents. Across multiple preference-aggregation rules, Hard Panel achieves the strongest constitutional alignment, while Soft Panel consistently improves upon training on the full, unadjusted PRISM sample. Additional analyses of weighting functions, panel sizes, training gradients, and the Community Alignment dataset reveal an important data-quality tradeoff: hard selection performs best when representative subsamples retain sufficiently informative preferences, whereas soft weighting becomes more competitive when filtering would discard substantial amounts of high-quality data. Our results demonstrate that the demographic composition of preference feedback is an empirically consequential design choice and provide an explicit, auditable mechanism for aligning models with a specified target population. We also provide theoretical analysis of our framework.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Pluralistic Alignment, RLHF, Sortition, Computational Social Choice

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JkYHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a fundamental issue with preference data, where biases among data contributors may lead to an imbalanced representation of preferences across different demographic groups. To address this issue, the paper proposes a sortition-based approach that constructs Soft and Hard panels to filter and reweight preference data according to demographic composition. The paper provides a theoretical analysis to demonstrate the effectiveness of the proposed approach and further validates it through experiments. The experimental results show that models aligned using this approach better reflect American values regarding AI.

Critical strengths
  • The paper raises a meaningful and research-worthy issue regarding bias in preference data.
  • The paper is written and organized clearly.
  • The authors honestly acknowledge the limitations of their work.
Critical weaknesses
  1. The paper overclaims its applicability to both DPO and RLHF in the title and the second contribution. However, the authors only conduct experiments with DPO, which strictly speaking does not belong to RLHF. To support the claim about RLHF, the authors should apply the proposed data processing method to RLHF (e.g., training a reward model) and analyze the experimental results.
  2. The evaluation uses only GPT-5.2 as the LLM-as-a-judge for ranking, which may introduce GPT-5.2's own preference biases. Using at least three models from different model families to vote on the rankings would provide a fairer evaluation.
  3. The PRISM Alignment and Community Alignment experiments show inconsistent results, with Hard > US-Rep > Soft and Soft > Full-US > Hard, respectively. This inconsistency raises concerns about the actual effectiveness of the proposed Hard and Soft panels, as discussed in Question 2.
  4. Appendix A.4 shows that only 56 out of 145 raters reside in the U.S., meaning most are not U.S. residents. Although the panel matches the U.S. demographic composition, differences in countries, environments, and education may lead to different values. Despite the authors' claim that this is a practical compromise, such differences may weaken the credibility of the conclusions, even if the results show better alignment with overall U.S. values.
  5. The proposed method has significant limitations, as it requires detailed demographic information about preference annotators, as provided by PRISM. Since most preference datasets lack such information, I doubt the potential for further research and the scalability of this approach.
Questions for authors
  1. The authors fix the number of training steps for a fair comparison. However, since Hard Panel has fewer training samples, this results in more training epochs and may lead to overfitting. Why not downsample Full PRISM and Soft Panel to achieve comparable numbers of training steps, epochs, raters, and preference pairs for a fairer comparison?
  2. In the "Cross-dataset replication on Community Alignment" section, the authors state that the choice between Hard Panel and Soft Panel depends on data quality. However, given a preference dataset, how should I decide which method to use? The dataset's quality may be unknown and is not necessarily quantifiable. Could the authors provide concrete criteria for choosing between Hard Panel and Soft Panel?
4 comments

We thank the reviewer for a careful review. Below we address each of the concerns made in the review and also included it in the revised version of the paper. Note that in the revised version of the paper, all the changes are highlighted in blue.

W1: Claims about RLHF are broader than the DPO experiments.

We have changed the title to Pluralistic Preference Alignment via Sortition-Weighted Preference Learning (please see the revised pdf) and narrowed the contribution language. DPO is related to the KL-regularized preference-learning formulation, but our DPO experiments do not constitute an online RLHF pipeline. We do not use that relationship to claim empirical coverage of reward-model-plus-policy-optimization methods.

We additionally trained weighted Bradley–Terry reward models using the same data-construction principle:

LRM(φ)=1A∑jaj[−log⁡sigmoid⁡(rφ(xj,yj+)−rφ(xj,yj−))].L_{RM}(φ) = \frac{1}{A}\sum_j a_j[-\log\operatorname{sigmoid}(r_φ(x_j,y_j^{+})-r_φ(x_j,y_j^{-}))].

Each reward model selects from a shared pool of 16 base-model candidates per prompt. This tests reward-model training and selection while keeping the candidate pool fixed. It is not PPO or another online policy-optimization experiment.

Reward modelHeld-out accuracy [95% CI]U.S.-rater accuracyWorst-group accuracyBest-of-16 BT [95% CI]
Full.610 [.575, .642].614.584.078 [.042, .115]
Soft.595 [.555, .633].612.554.007 [−.035, .046]
US-Rep.591 [.551, .627].589.476.074 [.037, .110]
Hard.588 [.556, .620].575.508.046 [.007, .088]
Random panel.571 [.536, .603].594.510.035 [−.004, .074]
Base greedy———−.109 [−.157, −.060]
Random candidate———−.131 [−.175, −.082]

Reward-model extension. Bradley–Terry (BT) scores are jointly fitted within this selection comparison.

Every trained selector has a better BT point estimate than the no-selection baselines, but the sortition-weighted reward model does not improve on Full. Full has the highest accuracy point estimate, and the reported uncertainty does not establish superiority among trained selectors. We report this result as an empirical scope check rather than evidence of a general RLHF benefit.


W2: Reliance on one judge family.

We added three judge families. The original 8B checkpoints have the same model-level ordering under all four judges:

ModelGPT-5.2Claude Sonnet 5.5Gemini 3.8 FlashDeepSeek v4.1 Flash
Hard.384.428.453.390
US-Rep.101.117.097.095
Soft (original).017.017.016−.023
Full−.069−.083−.054−.088
Base−.433−.479−.513−.373

Model-level Kendall τ=1τ=1 across all judge pairs. Per-question cross-family agreement is lower (τ=.72τ=.72–.75.75; top-1 agreement .70–.74), close to GPT self-agreement (τ≈.775τ ≈ .775).

BT values are fitted separately within each judge and must not be compared across columns as a shared scale. This result addresses dependence on one judge family. It does not eliminate correlated model biases or make the constitutional rubric a population-distribution measure.


W3: Opposite Hard/Soft orderings on PRISM and Community Alignment.

We agree that the reversal prevents a general Hard-over-Soft claim. The revision retains this result and adds two datasets , we also used a different model family (Qwen3-8B-Base):

Dataset / runHardSoftFull
PRISM original.384.017−.069
PRISM retrained, fixed-normalizer Soft.133.033−.026
CoVal (Qwen3-8B-Base, k=72).157.010−.059
OvertonBench (Qwen3-8B-Base, k=492).015−.008−.056
Community Alignment original.016.141.125

Constitutional BT scores. Each row is a separate comparison; magnitudes across rows are not comparable.

Soft has a positive Full-relative point difference on all four datasets, but the Community Alignment gap is small and its marginal intervals overlap. We do not claim statistically established gains everywhere.

Hard’s advantage depends on the regime. PRISM’s random-panel controls, together with a near-tie between Hard and Random on CoVal, show why its judge score cannot be interpreted as a composition effect. We have replaced the untested “data-quality tradeoff” explanation with a description of the observed regimes and an explicit statement that data quality was not independently manipulated.

References for new datasets:

Coval : Learning values-aware rubrics from the crowd (https://alignment.openai.com/coval/)

OvertonBench : Benchmarking Overton Pluralism in LLMs (https://arxiv.org/abs/2512.01351)

W4: Only 56/145 original panelists are U.S. residents.

This is a substantive limitation that demographic matching cannot remove. A mixed-locale panel satisfying U.S. demographic margins is not a representative sample of U.S. residents. We now state the 56/145 composition in the main limitations and add a U.S.-only pool of 348 raters, with feasible panel size k=141.

U.S.-only conditionHeld-out accuracy [95% CI]U.S. survey similarityGPT BT
Hard.513 [.444, .571].720−.012
Soft.511 [.454, .565].715.230
Full (U.S.).490 [.428, .552].732.025
US-Rep.508 [.440, .571].703.241
Base—.746−.484

Soft is above Full on the constitutional proxy, the human-accuracy difference is uncertain, and survey similarity is worse. Hard does not exceed Full on the constitutional proxy. These results do not establish improved population representation in the all-U.S. setting. They appropriately restrict the conclusion from the mixed-locale experiment.

Restricting locale also changes the available data and feasible design, so we do not treat this as a one-variable ablation.


W5: Demographic metadata and scalability.

The requirement is real. The method applies where consented rater attributes, a justified target public, and sufficient coverage are available. It is not a repair for metadata-free datasets. Metadata-poor settings may use a smaller set of ethically collected attributes, but narrower attributes also support narrower representativeness claims. We do not propose inferring sensitive attributes from text.

The selection/weighting layer is compatible with existing per-example losses and does not require a larger model or an additional forward pass. It does incur metadata-collection, quota-specification, and sortition-solver costs. Multiple attributes or small cells may make quotas infeasible or weights concentrated; those outcomes must be reported rather than hidden. The contribution’s scope is to make these participation assumptions visible and auditable, not to solve all feedback-data coverage problems.

Q1: Match steps, epochs, raters, and preference-pair counts.

The original shared-step comparison did not isolate these dimensions. With a fixed effective batch size, optimizer steps determine total row draws, while passes depend on dataset size. One cannot simultaneously hold steps and passes fixed across datasets of different sizes. Distinct raters and within-rater pair counts add further constraints. We therefore separate the controls rather than call one comparison “fully matched.”

We use a 5,095-pair Hard training budget and compare:

  1. A same-size random-rater panel.
  2. Uniform Full subsampling.
  3. Soft-weight-proportional row subsampling, trained unweighted.

Pair matching alone does not ensure identical per-rater exposure or effective sample size. We run both a shared 3,538-step regime and two passes through each condition’s own data.

ConditionSame steps: Judge BTSame steps: AccuracySame steps: SurveySame passes: Judge BTSame passes: AccuracySame passes: Survey
Hard.187.551.729.025.579.748
Random panel.373.544.706.099.597.751
Full-sub.136.539.671−.032.591.739
Soft-sub−.171.533.720.091.578.768
Soft (all data).091.597.761.288.594.762
Full (all data).052.595.734.113.600.733

Same steps: 3,538 optimizer steps. Same passes: two passes through each condition’s own data. Metrics are Gemini constitutional-judge BT, post-stratified held-out accuracy, and U.S. survey similarity. BT values are comparable within a regime’s candidate set.

Three conclusions survive these controls:

  1. Random exceeds Hard on the constitutional proxy under both regimes, so the original judge advantage does not identify quota composition.
  2. Soft-sub exceeds Full-sub on survey similarity at equal pair counts: +.049 at shared steps and +.029 at equal passes. This suggests that the weighting distribution matters beyond raw pair count.
  3. All-data Soft and Full are close on prediction accuracy, while Soft has higher survey similarity in both regimes.

These are complementary observations, not a claim that all confounds have been eliminated.

We also vary panel draw (seeds 42, 7, and 13) and training seed. Random exceeds Hard on constitutional BT in each of the four reported comparisons. Hard survey similarity varies from .697 to .729 across panel draws, which is why we now present panel sensitivity rather than one favorable realization. Length/refusal adjustment does not remove the Random–Hard gap in three of the four reported checks, but this association does not rule out every response-style effect.


Q2: How should a practitioner choose Hard or Soft without knowing “data quality”?

We have replaced that recommendation with criteria that do not require an unobserved data-quality label.

  • When all feedback is available, start with fixed-normalizer Soft. It retains comparisons and removes panel-draw randomness from the objective. Evaluate it against Full on the intended outcome. This is a practical default, not a universal empirical dominance claim. Raw linear weights preserve the exact expected-panel identity; floors and other transformations must be disclosed and checked separately.

  • Use Hard when the deliverable or collection design requires a realized panel. Examples include a named, auditable, quota-feasible electorate or a budget permitting collection from only k raters. Report multiple draws and a size-matched random-panel baseline.

  • Choose using a separate validation split and an appropriate target. Use population-matched held-out preferences or survey distributions when those are the intended outcomes. Use constitutional evaluation for consensus compliance. The test split used in this response must not also be treated as an untouched test after method selection.

Weight concentration, Kish effective sample size, Hard data fraction, and effective pair counts are useful diagnostics for unstable weighting and regime confounding. They are not validated predictors of which method will represent a public better. Where population-matched validation is unavailable, that limitation should remain explicit.

Thank you for the detailed response. I have a clear understanding of the additional experiments and the corresponding explanations.

However, I still have significant concerns regarding Question 2. If additional data analysis is required to determine which specific method should be applied, this would substantially undermine the practical applicability of the proposed work. This is because this work aims to propose a methodology rather than a data selection or preprocessing. A proposed methodology should be clear and easily reproducible, without requiring extensive or complicated preliminary steps to determine how it should be applied.

Reviewer_Tdv1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a sortition-weighted preference learning framework to address demographic representation in alignment. The authors introduce two methods: hard panel, which trains on a representative subset of raters, and soft panel, which reweights the full dataset using inclusion probabilities. The paper provides theoretical analyses of both methods and evaluates them on PRISM and Community Alignment datasets. Experimental results suggest that demographic representativeness can improve alignment, with different tradeoffs between hard selection and soft weighting.

Critical strengths
  • The paper addresses an important aspect of pluralistic alignment, and the proposed methods provide a simple and practical framework that can be readily integrated into existing RLHF/DPO pipelines
  • The experimental results are promising, as hard panel outperforms the full-data baseline despite using substantially fewer raters.
Critical weaknesses
  • The theoretical results are largely limited to characterizing the optimization properties of hard and soft panel methods and do not provide particularly novel insights. Most results follow from standard sampling and optimization arguments, without establishing generalization guarantees on alignment performance or introducing substantial methodological novelty.

  • The experimental evaluation relies primarily on an LLM judge and a 75-clause constitution, which may not accurately reflect the heterogeneous preferences of the target population.

Questions for authors
  • In line 211, what is the precise definition of the external risk R(θ)R(\theta)?

  • In Theorem 3, does the optimal lottery depend on the choice of WW and the parameter θ\theta? If so, what is the practical relevance of the proposed optimization problem?

  • Could the authors provide further discussion or evidence on how accurately the LLM judge and the 75-clause constitution used for evaluation reflect the preferences of the target population?

  • In Theorem 6, maybe the illustrative example could be placed immediately after the theorem statement, rather than after the Remark.

3 comments

We thank the reviewer for a careful review. Below we address each of the concerns made in the review and also included it in the revised version of the paper. Note that in the revised version of the paper, all the changes are highlighted in blue.

W1: Limited novelty of the theory and absence of alignment generalization guarantees.

We agree that the mathematical ingredients are classical. Our contribution is their adaptation to this setting, we do not claim that any theorem establishes generalization to a population’s values. The revised contribution statement makes this explicit. The proposed contribution is the mapping from a quota-constrained sortition lottery to auditable preference-learning objectives, together with its implementation analysis and controlled empirical evaluation.

Why sortition is the organizing construction. The claim is not that demographic weighting itself is new. A single feasibility specification yields both a realized quota-satisfying panel and compatible inclusion probabilities for an all-data objective. LEXIMIN adds an explicit individual-participation fairness criterion within that feasible design. This is useful when the same governance requirement must support either restricted panel participation or weighting of existing feedback. It does not recover unknown recruitment propensities or guarantee that demographic groups share homogeneous values.

This mapping has a concrete consequence beyond a formal analogy: a mathematically specified weighted DPO loss can optimize a different effective electorate when weights are normalized inside small microbatches. For independent uniform row draws JtJ_t, row weights aj=wrj/Nrja_j = w_{r_j}/N_{r_j}, and known A=∑jajA = \sum_j a_j, the estimator

g^=NbA∑t=1baJt∇ℓJt(θ)\widehat{g} = \frac{N}{bA}\sum_{t=1}^{b} a_{J_t}\nabla\ell_{J_t}(θ)

is unbiased for the intended weighted gradient. By contrast, the self-normalized estimator

∑t=1baJt∇ℓJt(θ)∑t=1baJt\frac{\sum_{t=1}^{b} a_{J_t}\nabla\ell_{J_t}(θ)}{\sum_{t=1}^{b} a_{J_t}}

generally is not, and becomes unweighted when b=1b=1. This distinction motivates the fixed-normalizer retraining and limits how the original Hard–Soft comparison can be interpreted. The correction is useful without claiming novelty for importance sampling itself or attributing the entire retraining gain to this one change.


Q1: Precise definition of external risk R(θ)

We now define external risk before Theorem 2. Let TT be the target public, QQ a prompt distribution, dθd_θ the deployed response rule, and sT(x,y)∈[0,1]s_T(x,y) ∈ [0,1] the probability that a randomly drawn member of TT is satisfied with response yy. Then

RT(θ)=1−Ex∼QEy∼dθ(⋅∣x)[sT(x,y)].R_T(θ) = 1 - E_{x∼Q} E_{y∼d_θ(·|x)}[s_T(x,y)].

For greedy decoding, dθd_θ is a point mass. In the binary example, sT(x,A)=pTs_T(x,A)=p_T and sT(x,B)=1−pTs_T(x,B)=1-p_T. This makes the external criterion explicit and separates it from the training losses.

The held-out pairwise prediction error is not an estimator of this generated-response satisfaction risk: it evaluates which existing candidate a model’s implicit reward ranks higher. Similarly, survey similarity and constitutional compliance are separate external criteria. No theoretical ordering is asserted for any of them.


Q2: Does the optimal lottery depend on W and θ, and what is its practical relevance?

Yes. Define the individual gradients and panel-average gradient as

gi(θ)=∇Li(θ),gS=k−1∑i∈Sgi(θ).g_i(θ) = \nabla L_i(θ), \qquad g_S = k^{-1}\sum_{i ∈ S} g_i(θ).

For a fixed feasible inclusion vector π∗π^{*}, the objective is

∑SλS(gS−gˉ)⊤W(gS−gˉ),gˉ=k−1∑iπi∗gi(θ).\sum_S λ_S(g_S-\bar{g})^{\top}W(g_S-\bar{g}), \qquad \bar{g} = k^{-1}\sum_i π_i^{*}g_i(θ).

It depends on the reference parameter through the gradients and on the chosen positive-semidefinite matrix WW. At fixed θθ, WW, and π∗π^{*}, it is linear in the lottery probabilities λSλ_S, subject to the panel-probability and marginal constraints.

The main practical implication is simpler than the optional optimization: fair first-order inclusion probabilities do not determine Hard’s variance. Different lotteries can have the same Soft objective but different Hard variability. Our panel-draw checks make this sensitivity concrete: Hard survey similarity is .729, .706, and .697 across the three reported draws; held-out accuracy is .551, .535, and .539. These observations motivate reporting multiple draws, rather than establishing that the theorem predicts the numerical spread.

We move the optimization program to the appendix and state explicitly that it was not used experimentally. One possible extension would fix θ=θrefθ=θ_{ref} and W=IW=I before training. Optimizing over only the panels returned by a solver gives a restricted-support optimum, not necessarily the full-family optimum. We also remove the draft’s suggestion that the cube method directly solves this particular variance-cost program. No uniform training-path or external-risk improvement is guaranteed.

W2 and Q3: Do the judge and constitution reflect heterogeneous population preferences?

Not on their own. Four judge families reproduce the original constitutional ordering, but agreement on the same consensus rubric cannot validate a distributional population claim. We therefore add the human-preference and survey-distribution evaluations reported in the population-evaluation table, explain the construct distinction in the main text, and retain the negative checks.

Those checks matter. In the all-U.S. pool, Soft and Full achieve survey similarities of .715 and .732, respectively, both below Base (.746). Their held-out accuracies are .511 and .490, with wide, overlapping intervals. Soft also does not improve worst-group accuracy over Full in the main mixed-locale comparison (.556 versus .557).

Thus, our evidence supports a survey-distribution benefit in the main mixed-locale setting, not a universal or subgroup-wise population-alignment guarantee. This is the principal change to our interpretation.


Q4: Place the illustrative example immediately after Theorem 6.

Done. The two-rater example now immediately follows Theorem 6, and the longer survey-sampling remark is retained in the appendix. We explicitly describe the example as an external-target counterexample, rather than evidence for the mechanism behind the dataset reversal.

I appreciate the authors' efforts to address my concerns by clarifying the scope of the theoretical results and adding new evaluations. However, I am still not fully convinced that the contribution is sufficiently clear and substantial to recommend acceptance. I will keep my current score for now, but I am open to update it based on further discussion with the other reviewers.

Reviewer_YWQ8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes using sortition to select or reweight the annotators whose preferences are used for fine-tuning via DPO. Hard Panel trains on a sampled, demographically representative panel, while Soft Panel keeps all annotators but weights their preferences by their probability of being selected.

The authors fine-tune 1B, 3B and 8B Llama models and evaluate them on 3,000 questions derived from a 75-clause constitution collected through public input, with GPT-5.2 judging the responses. On PRISM, Hard Panel ranks above both PRISM's existing U.S.-representative subset and Soft Panel, while Soft Panel improves over training on the full dataset without adjustment. The ordering changes on Community Alignment, where Soft Panel does better than Hard Panel. So the results show that selecting and weighting raters affects performance on this constitutional evaluation, but the advantage of hard selection depends on the dataset.

The paper also presents several theoretical results about the relationship between Hard and Soft Panel, which the authors connect to established results in survey sampling and estimation.

Critical strengths

I appreciate the ablations considered in the paper including different model sizes, weighting choices and a second dataset. I also appreciate that the authors show cases where Hard Panel does better and cases where Soft Panel does better. More generally, applying sortition as part of the fine-tuning pipeline seems like a very natural idea and I am glad that the authors pursued this.

Critical weaknesses

My main concern is what the evaluation actually shows. The constitution was collected through public input, but the votes were processed into principles with consensus across groups. This does not preserve the distribution of people's preferences. So I am not convinced that doing better against this constitution shows better representation of the target population. The evaluation is interesting, but does it support the message the authors take from it? I would like a clearer explanation of this connection, or an additional evaluation that directly looks at the different preferences in the population.

At first the writing felt fine, but I found the parts I wasn't already familiar with surprisingly hard to follow. There is some explanation around the theoretical results, but I often missed what they mean and why they matter here. Similarly, the experiments go straight into describing the dataset without first explaining what they are trying to establish. The paper could really do with more context and some substantial editing.

Questions for authors

Could the authors explain what each theoretical result means, why it matters for the paper, and why it is important enough to be included in the main text?

I would generally expect Soft Panel to be better at representing the target population, since it keeps all the data while weighting it to reflect the target population. If Hard Panel performs better on the PRISM evaluation, doesn't this raise questions about whether the evaluation actually measures population representation?

See also the first weakness for further questions.

2 comments

We thank the reviewer for a careful review. Below we address each of the concerns made in the review and also included it in the revised version of the paper. Note that in the revised version of the paper, all the changes are highlighted in blue.

W1 and Q2: What does the constitution measure, and why does Hard outperform Soft?

We agree with the construct-validity concern. The publicly elicited constitution retains principles with cross-group consensus, it does not preserve the distribution of individual votes or the full range of disagreement. Constitutional ranking therefore measures compliance with these consensus principles. It is not an estimator of the target population’s preference distribution. We have now stated this explicitly in the abstract and added two complementary evaluations. It is also mentioned explicitly in sections of experimental framing, evaluation protocol, and limitations.

1. Held-out human preference prediction. We exclude 10% of PRISM conversations from every retraining condition: 815 conversations, 5,910 comparisons, and 636 raters. Each model’s implicit DPO reward predicts the chosen versus rejected response. We average accuracy within rater and aggregate using the specified U.S. demographic targets, with rater-level bootstrap intervals. We also report U.S.-locale-only and worst-group summaries.

2. Opinion-distribution similarity. We compare model answer distributions with U.S. survey answer distributions on 1,076 GlobalOpinionQA questions from Pew Global Attitudes and the World Values Survey, using mean 1−JSD1-\mathrm{JSD}, where JSD denotes Jensen–Shannon divergence, and paired question resampling. This directly evaluates a distributional target, while remaining a survey-elicitation proxy rather than a complete measure of population alignment.

ConditionHeld-out accuracy [95% CI]Worst-group accuracyU.S. survey similarity [95% CI]
Fixed-normalizer Soft.597 [.557, .639].556.761 [.753, .768]
Full.595 [.555, .635].557.734 [.725, .743]
US-Rep.552 [.516, .593].456.703 [.694, .712]
Hard.551 [.516, .589].473.729 [.721, .738]
Random panel.544 [.512, .579].482.706 [.697, .715]
Base.500 (reward ties)—.746 [.737, .754]

What these results support. Soft’s paired survey gain over Base is .015 [95% CI: .011, .019], and its gain over Full is .027. Soft–Full survey gains remain positive with another training seed (.015) and under equal passes (.029). On held-out accuracy, Soft and Full are effectively tied: the .002 point difference and worst-group scores do not establish a Soft advantage. Thus, the new evidence supports a survey-distribution benefit in this setting, without establishing improvement on every notion of representation.

Why the constitutional ordering differs. A random-rater panel matched to Hard’s size scores above Hard on the constitution in all four reported panel/training-seed checks. In the shared-step Gemini comparison, Random and Hard have Bradley–Terry scores of .373 and .187, respectively, despite survey similarities of .706 and .729. Under equal passes, the Hard–Random survey ordering reverses (.748 versus .751). These controls show that the original Hard judge gain cannot, by itself, be attributed to quota composition. They are consistent with a substantial training-regime effect, but do not establish a complete causal explanation of the gain.

There is also no theoretical contradiction. For linear weights, at a fixed parameter value θ\theta,

Lsoft(θ)=ES[Lhard(θ∣S)].L_{soft}(θ) = E_{S}[L_{hard}(θ | S)].

This identity does not commute with optimization or imply an ordering under external evaluation. The binary DPO example is a counterexample to a universal ordering, not an explanation of the empirical mechanism. With pooled preference pˉ=∑iqipi\bar p=\sum_i q_i p_i and reference probability rr,

psoft∗(A)=sigmoid⁡(logit⁡(r)+β−1logit⁡(pˉ)).p^{*}_{soft}(A) = \operatorname{sigmoid}(\operatorname{logit}(r) + β^{-1}\operatorname{logit}(p̄)).

Each realized Hard panel instead fits its own preference mixture. Either policy can perform better on a shifted external target. Keeping all data removes panel-draw variance from the objective; it does not ensure that every external rubric ranks the trained model first.

A necessary implementation correction. Original Soft runs normalized weights separately within microbatches of four. Theorem 5 shows that this changes expected relative row influence; at microbatch size one, it cancels the weights entirely. New Soft runs use the known-normalizer estimator. The original and retrained survey scores (.672 and .761) do not constitute a pure normalization ablation because the training split also changes. Interpreting the reported weight-floor diagnostic also requires distinguishing raw from transformed weights. We therefore do not attribute the entire difference to the estimator correction.

W2 and Q1: Explain the theoretical results and improve the experimental narrative.

We have replaced the theory section’s extended survey-literature discussion with a short explanation before each result, retaining the details in the appendix. We also added a “What the experiments establish” paragraph before introducing the datasets.

ResultMeaningPractical consequence and scope
1. ExpectationLinear Soft is the fixed-parameter average Hard objective, and its demographic masses satisfy the quota bounds.Correctly implemented raw linear weights eliminate panel-draw randomness from that objective. This does not establish population preference recovery.
2. OptimizationThe optimum of the mean and the mean of panel-specific optima can differ; external risks are not universally ordered.Neither Hard nor Soft is universally superior after training.
3. Panel varianceJoint inclusion probabilities, not only marginal inclusion probabilities, determine design variance.Report multiple Hard draws. The optional reference-parameter variance program moves to the appendix.
4. StabilityUnder strong convexity, gradient disagreement bounds the distance between Hard and Soft optima.This is a conditional stability result, not a theorem about nonconvex neural-DPO generalization. Secondary bounds move to the appendix.
5. MinibatchesRandom-denominator self-normalization can change the effective weighted objective.Use a known normalizer or weight-proportional row sampling. Gradient accumulation alone does not fix separately normalized microbatches.
6. CounterexampleBinary DPO admits either external ordering under an exact LEXIMIN design.This rules out universal superiority. The numerical example now directly follows the theorem.

These results rely on established sampling and optimization arguments. Their role is to specify what the construction guarantees, identify implementation failure modes, and motivate the experimental controls. We no longer present them as a generalization theory of alignment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.