- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper propose a way to scale pluralistic alignment data by building a synthetic data generation pipeline. Starting from candidates of pluralistic alignment specifications (I will call them PAS from now on, since specifications alone sometimes misinterpreted), the pipeline filter it out using LLM judges validated by human annotator, then use a set of LLMs to generation consitutions, then QA pairs, further iteratively refined by large, frontier-class open-weights LLMs, which results in ~1.2M QA pairs covering 9 values categories from 243 PAS. The paper also propose an alignment method called Grouped Robustness Optimization (GRO) which selects interpolation that most robust across all specifications. To prove the usefulness of the synthetic alignment data and GRO, they also propose a rubric-evolving benchmark called PluralValueEval with 1,976 criteria across 36 PAS
Critical strengths
- The proposed GRO is the strongest contribution of the paper. This kind of optimization is surely needed in the literature of pluralistic alignment, regardless of the alignment data being used.
Critical weaknesses
- Lack of proof or evidence regarding the reliability and quality of the synthetically generated data (both the alignment and the benchmark data)
- Source of the initial specification candidates are unknown (for both alignment and benchmark data), clearly not grounded into actual data produced by human
- (Synthetic data generation) The pipeline itself is so lengthy and full of LLM-generated steps. No validation of whether the LLM are doing the right thing except in the early parts in filtering the specification candidates. Iterative refinement using "better, larger" LLMs may help, but the concern is that there is no proof that the generated data is aligned with the intended PAS or broader cases of pluralistic values. Given the pipeline, it is more likely the data is aligned to values held by the LLM that generated the answer.
- (Benchmark data) Similar critical concerns although the human annotation is slightly better here to ensure the validity of the rubric. However, the validation for the judge part is missing. It is the most important validation that is needed, without it it's unclear how reliable the judge is. Calibration with other model-based judges does not validate this. It only measures model-to-model agreement, and it does not explain reliability or truthfulness.
- Self-preference loop, where models that parts in generating either of the alignment data or the benchmark data, also being used in evaluation without any proof that there is no self-preference bias.
Questions for authors
- Have the authors try to evaluate using existing and more-grounded alignment benchmark such as PRISM, WorldValuesBench (from WVS), or any others? Instead of evaluate using a very similar synthetic benchmark data compared to the alignment data.
- Can the authors provide the positioning of the work more clearly, such as what makes the paper unique and advantageous towards similar works in synthetic data generation for pluralistic alignment (such as PLURAL)?
4 comments
W3. Benchmark data: We agree that agreement between automated judges does not establish agreement with human judgment. Our original human study validated rubric quality. We supplemented it with three separate studies covering generated training data, model-response comparisons, and automated-judge decisions, with 108 evaluator sessions per study and 324 overall.
For judge validation, we sampled five responses per specification across all 36 benchmark specifications. Each response was evaluated against three response-specific criteria, yielding 540 criterion decisions assessed by three independent evaluators each. Participants first determined whether a criterion applied and, if applicable, whether the response adhered, partially adhered, or ignored it. Jev’s decisions were hidden from participants. Criterion sampling included different automated applicability and adherence labels to examine agreement across decision types.
Jev agreed with human majority judgments on 84% of applicability decisions, with Cohen’s kappa of 0.66. On cases judged applicable by both, adherence agreement was 78%, with kappa of 0.61. Human–human Krippendorff’s alpha was 0.64 for applicability and 0.60 for the combined four-category judgment.
These comparisons directly evaluate the judge against humans rather than another model. We will report disagreement patterns and sampling limitations alongside agreement, without interpreting the results as evidence that the judge is universally reliable or unbiased.
W4. Self-preference: We designed the evaluation so that the primary judge takes no part in generation. Every result in the paper, including all fine-tuning results, is scored by Jev, a third-party evaluation model used in no stage of data or benchmark construction (Table 3). Kimi-K3, which wrote the training answers, serves only as a second judge to check Jev’s model rankings.
The rankings show no sign of self-preference:
- The data generator wrote no rubrics. Kimi-K3 had no role in building the benchmark. It ranks first under Jev too, so its lead does not depend on its own judging.
- Rubric authors are not favored. The two evaluated models that proposed rubrics, GPT-OSS-120B and Qwen3.5-397B, rank fourth and fifth of twelve. Qwen3.8-27B, which had no role, outscores Qwen3.5-397B by 6.55 points.
- Judges from three providers agree. Jev and Kimi-K3 agree on the model ranking (Spearman ρ = 0.965 across twelve models), and Kimi-K3 in turn preserves GPT-5.6 Sol’s ranking.
This agreement reflects rubrics that leave judges little room for their own preferences. Each specification has about 55 fine-grained criteria (1,976 in total), proposed by four models from three providers and grounded in retrieved sources. Rubric meta-evaluation averages judges from two providers to offset each judge’s preference for its own provider’s models. A majority of three human annotators rated 90%+ of the criteria authentic, actionable, and specific. Judges therefore make the same calls on the same answers: Jev and Kimi-K3 agree on 83.2% of applicability decisions and 76.2% of criterion ratings.
To test the judge against humans directly, two human studies are ongoing. One compares human preferences between model responses with the automated rankings. The other compares human decisions on individual rubric criteria with Jev’s. We will report the results once they are ready.
Q1. PRISM validation: Thank you for the suggestion. We evaluated the Gemma-4-31B-it models on PRISM (Kirk et al., 2024), a human-grounded preference benchmark, and it has been added to the main content (line 480) and Appendix R Table 22. Training on our corpus carries over to real participants’ preferences, beyond our synthetic benchmark.
For each participant’s opening message, we generate one reply per model under the specification matching the participant’s group (by country or religion). Given the participant’s own PRISM ratings of other replies, Jev predicts which of two new replies they would prefer. GRO is preferred over the untuned model in every group, by up to 78.8%. SFT loses in four of the five groups, mirroring the robustness gap on our benchmark:
| Participants | Specification | GRO vs untuned | SFT vs untuned |
|---|---|---|---|
| South Africa | Ubuntu philosophy | 78.8 ± 0.4 | 56.1 ± 1.2 |
| Chile, Mexico, Uruguay | Latin American society | 73.5 ± 0.6 | 41.2 ± 0.3 |
| Catholic | Catholic moral theology | 65.9 ± 0.3 | 35.5 ± 0.5 |
| Muslim | Islamic finance | 61.5 ± 1.1 | 38.9 ± 0.3 |
| Jewish | Orthodox halachic reasoning | 60.1 ± 1.6 | 21.9 ± 0.3 |
Q2. PLURAL comparison: We will add this comparison to the Related Work section. PLURAL (Agarwal et al., 2026) turns survey responses into preference data for 20 countries. Our work differs in three ways:
- Natural language value system, not only surveyed countries. Specifications written in natural language are expanded into about 200 prioritized clauses each. This covers 243 value systems, including ones no survey captures, such as Kantian deontology or Socratic questioning.
- Data and evaluation that resist surface adherence. Answers are refined by a critic that never sees the grading rubric, and questions are hardened adversarially. PluralValueEval then measures adherence to an explicit specification with human-validated criteria (94.0% approval), rather than matching a population’s survey answers.
- One steerable model with robust gains. We present a fine-tuning method GRO, that trains a single model to follow multiple specifications individually. On held-out questions, it raises our main metric by 7.9 points while cutting the share of degraded specifications from 47% to 8%.
PLURAL captures how representative populations answer; ours covers explicitly specified value systems that surveys do not reach. In addition to the above and aligning with third point, our focus is more on the steerability of language models towards varied value systems. We have revised the abstract to highlight this aspect.
Thank you for the response and the additional work to revise the paper, I appreciate the effort. Initially, I want to wait for the follow-up human evaluation. However, I think even with that results, some of my concerns below still holds, mainly regarding how the human evaluation itself being done.
Annotators
Prolific filters cover demographics like religion, country, or politics, which as far as I know, do not cover specific knowledge like value systems. Therefore, I am not sure if the annotator helds the value that required to properly judge a specific PAS.
No baseline or evidence of distinct values
To my view, fidelity score of 4.4/5 and 99% rubric approval shows that are some issues with how the human evals is done, probably one of them because of the annotator effect or could be a design effect or both. Without a baseline, these scores cannot show fidelity. The current design can only show consensus, but it cannot show pluralism. Even under streerable pluralism that being the focus of the paper, a model is only pluralistic if its outputs for different value systems are actually different and are judged as different by the people who hold them, in which the paper currently has no evidence of this.
For the reason above, I keep my score as it is for now.
Thank you for the interesting review.
W1. Initial specifications: Our nine categories correspond to the value domains used in established value research. Spranger’s six value types (theoretical, economic, aesthetic, social, political, religious; Spranger 1928), adopted in the Allport-Vernon-Lindzey Study of Values, map onto our philosophical, professional, creative, well-being, political, and religious categories. The thematic structure of the World Values Survey (Haerpfer et al. 2022) and European Values Study covers the same domains and adds culture/society, communication, and well-being. Weber’s value spheres (1915/1946) reinforce our categories. Our composite specifications are combinations of the above to enable interesting and nuanced alignment goals. Our categories were ideated over weeks of discussion to ensure plausible coverage and diversity of human values while being mindful of application potential regarding LLM alignment, and were later selected via human annotations. The table below provides the full background.
References:
W2. Synthetic data generation:
The table below summarizes how each stage is validated; we will add it to the paper. Every automated step is either compared with human judgments or scored against human-validated rubrics.
We are running a direct human evaluation of the whole pipeline in addition to human-evaluated LLM judge rubrics. Prolific annotators rate 180 training question–answer pairs across all 36 specifications, with three annotators per pair. They rate whether each answer reflects the values and priorities of its specification, contains a substantive error, or stereotypes a group. If the data reflected the generator’s own values instead of the specification, it would score low on the first question. Because this evaluates the final output, it checks the whole pipeline end to end, from goals and constitutions to questions and answers. We will report the results once it’s done.