acceptodds
Withdrawn from ICLR 2027

PluralisticDataSmith: Turning Pluralistic Alignment Goals into Synthetic Data at Scale

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Current alignment methods optimize language models toward a single set of values, failing to capture diverse preferences for how models should reason, communicate, and act. Yet the field lacks both the training data and benchmarks needed to reliably train and evaluate steerable value alignment. We present PluralisticDataSmith, a scalable synthetic data generation framework that produces pluralistically aligned training data. A multi-stage LLM-ensemble agent generates approximately 200 fine-grained constitutions (e.g., ensuring correct honorific usage) for each of 243 value specifications, ranging from Socratic questioning and Korean society to composite applications such as debate preparation combining formal logic and persuasive storytelling. Using a frontier-class open-source LLM (2.8T parameters) as the synthetic data generator, our pipeline produces 1.2M synthetic QA pairs across nine categories (communication, creativity, culture, philosophy, politics, professional practice, religion, wellbeing, and composite applications) with quality competitive with closed-source alternatives. An iterative refinement loop improves data quality through intuitive natural-language feedback. We further introduce PluralValueEval, a retrieval-augmented, evolving-rubrics benchmark validated by 324 human annotators across 36 specifications. Evaluation rubrics are evolved through strong–weak model contrast to assess genuine value alignment rather than compliance with artificial restrictions, with meta-judge filtering retaining 1,976 high-quality criteria. Even frontier LLMs score below 50% on our benchmark. Fine-tuning open-source models (9B–31B) on the corpus with our Grouped Robustness Optimization (GRO) yields downstream gains of 8% over base models while reducing the share of degraded specifications from 50% to 6% and the largest per-specification drop from -16% to only -1.4%. Built entirely on open-source models, our framework offers an accessible alternative to closed-source APIs. We release the training corpus, pipeline, and evaluation benchmark to support reproducible research in pluralistic alignment.

Primary area:
datasets and benchmarks
Keywords:
pluralistic alignment, constitutional AI, synthetic data, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_5Ts2OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper propose a way to scale pluralistic alignment data by building a synthetic data generation pipeline. Starting from candidates of pluralistic alignment specifications (I will call them PAS from now on, since specifications alone sometimes misinterpreted), the pipeline filter it out using LLM judges validated by human annotator, then use a set of LLMs to generation consitutions, then QA pairs, further iteratively refined by large, frontier-class open-weights LLMs, which results in ~1.2M QA pairs covering 9 values categories from 243 PAS. The paper also propose an alignment method called Grouped Robustness Optimization (GRO) which selects interpolation that most robust across all specifications. To prove the usefulness of the synthetic alignment data and GRO, they also propose a rubric-evolving benchmark called PluralValueEval with 1,976 criteria across 36 PAS

Critical strengths
  • The proposed GRO is the strongest contribution of the paper. This kind of optimization is surely needed in the literature of pluralistic alignment, regardless of the alignment data being used.
Critical weaknesses
  • Lack of proof or evidence regarding the reliability and quality of the synthetically generated data (both the alignment and the benchmark data)
    • Source of the initial specification candidates are unknown (for both alignment and benchmark data), clearly not grounded into actual data produced by human
    • (Synthetic data generation) The pipeline itself is so lengthy and full of LLM-generated steps. No validation of whether the LLM are doing the right thing except in the early parts in filtering the specification candidates. Iterative refinement using "better, larger" LLMs may help, but the concern is that there is no proof that the generated data is aligned with the intended PAS or broader cases of pluralistic values. Given the pipeline, it is more likely the data is aligned to values held by the LLM that generated the answer.
    • (Benchmark data) Similar critical concerns although the human annotation is slightly better here to ensure the validity of the rubric. However, the validation for the judge part is missing. It is the most important validation that is needed, without it it's unclear how reliable the judge is. Calibration with other model-based judges does not validate this. It only measures model-to-model agreement, and it does not explain reliability or truthfulness.
  • Self-preference loop, where models that parts in generating either of the alignment data or the benchmark data, also being used in evaluation without any proof that there is no self-preference bias.
Questions for authors
  • Have the authors try to evaluate using existing and more-grounded alignment benchmark such as PRISM, WorldValuesBench (from WVS), or any others? Instead of evaluate using a very similar synthetic benchmark data compared to the alignment data.
  • Can the authors provide the positioning of the work more clearly, such as what makes the paper unique and advantageous towards similar works in synthetic data generation for pluralistic alignment (such as PLURAL)?
4 comments

Thank you for the interesting review.

W1. Initial specifications: Our nine categories correspond to the value domains used in established value research. Spranger’s six value types (theoretical, economic, aesthetic, social, political, religious; Spranger 1928), adopted in the Allport-Vernon-Lindzey Study of Values, map onto our philosophical, professional, creative, well-being, political, and religious categories. The thematic structure of the World Values Survey (Haerpfer et al. 2022) and European Values Study covers the same domains and adds culture/society, communication, and well-being. Weber’s value spheres (1915/1946) reinforce our categories. Our composite specifications are combinations of the above to enable interesting and nuanced alignment goals. Our categories were ideated over weeks of discussion to ensure plausible coverage and diversity of human values while being mindful of application potential regarding LLM alignment, and were later selected via human annotations. The table below provides the full background.

Our categorySpranger / Allport–Vernon–Lindzey Study of ValuesWorld Values Survey / European Values StudyWeber's value spheres
ReligiousReligiousReligious values / ReligionReligious
PoliticalPoliticalPolitical interest & participation; Political culture & regimes / PoliticsPolitical
PhilosophicalTheoreticalEthical values & normsIntellectual
ProfessionalEconomicEconomic values / WorkEconomic
CreativeAesthetic—Aesthetic
Cultural—Social values, attitudes & stereotypes / Society—
WellbeingSocialSocietal well-being / Life—
CommunicationSocialSocial capital, trust & organizational membership—
CompositeCombinations of the aboveCombinations of the aboveTension between spheres

References:

  • Spranger (1928), Types of Men: The Psychology and Ethics of Personality
  • Allport et al. (1960), A Study of Values: A Scale for Measuring the Dominant Interests in Personality
  • Haerpfer et al. (2022), World Values Survey: Round Seven – Country-Pooled Datafile
  • EVS (2022), European Values Study 2017: Integrated Dataset (EVS 2017)
  • Weber (1946; orig. 1915), Religious Rejections of the World and Their Directions

W2. Synthetic data generation:

The table below summarizes how each stage is validated; we will add it to the paper. Every automated step is either compared with human judgments or scored against human-validated rubrics.

StageResultValidated by
Specification filtering86% agreement with human ratings (F1 0.92)Humans
Refinement25/36 specifications pass rate improvedLLM judge (human-validated rubrics)
RubricsApproval 91.2% → 94.0% (AC1 0.87)Humans (324 annotators)
JudgeRubrics human-evaluated, human–judge agreement (new study)Humans
Fine-tuning on the dataCap-5 +7.1% to +7.9% increaseLLM judge (human-validated rubrics)
Generated data (entire pipeline)Fidelity to the specification, errors, stereotyping (new study)Humans

We are running a direct human evaluation of the whole pipeline in addition to human-evaluated LLM judge rubrics. Prolific annotators rate 180 training question–answer pairs across all 36 specifications, with three annotators per pair. They rate whether each answer reflects the values and priorities of its specification, contains a substantive error, or stereotypes a group. If the data reflected the generator’s own values instead of the specification, it would score low on the first question. Because this evaluates the final output, it checks the whole pipeline end to end, from goals and constitutions to questions and answers. We will report the results once it’s done.

W3. Benchmark data: We agree that agreement between automated judges does not establish agreement with human judgment. Our original human study validated rubric quality. We supplemented it with three separate studies covering generated training data, model-response comparisons, and automated-judge decisions, with 108 evaluator sessions per study and 324 overall.

For judge validation, we sampled five responses per specification across all 36 benchmark specifications. Each response was evaluated against three response-specific criteria, yielding 540 criterion decisions assessed by three independent evaluators each. Participants first determined whether a criterion applied and, if applicable, whether the response adhered, partially adhered, or ignored it. Jev’s decisions were hidden from participants. Criterion sampling included different automated applicability and adherence labels to examine agreement across decision types.

Jev agreed with human majority judgments on 84% of applicability decisions, with Cohen’s kappa of 0.66. On cases judged applicable by both, adherence agreement was 78%, with kappa of 0.61. Human–human Krippendorff’s alpha was 0.64 for applicability and 0.60 for the combined four-category judgment.

These comparisons directly evaluate the judge against humans rather than another model. We will report disagreement patterns and sampling limitations alongside agreement, without interpreting the results as evidence that the judge is universally reliable or unbiased.

W4. Self-preference: We designed the evaluation so that the primary judge takes no part in generation. Every result in the paper, including all fine-tuning results, is scored by Jev, a third-party evaluation model used in no stage of data or benchmark construction (Table 3). Kimi-K3, which wrote the training answers, serves only as a second judge to check Jev’s model rankings.

The rankings show no sign of self-preference:

  1. The data generator wrote no rubrics. Kimi-K3 had no role in building the benchmark. It ranks first under Jev too, so its lead does not depend on its own judging.
  2. Rubric authors are not favored. The two evaluated models that proposed rubrics, GPT-OSS-120B and Qwen3.5-397B, rank fourth and fifth of twelve. Qwen3.8-27B, which had no role, outscores Qwen3.5-397B by 6.55 points.
  3. Judges from three providers agree. Jev and Kimi-K3 agree on the model ranking (Spearman ρ = 0.965 across twelve models), and Kimi-K3 in turn preserves GPT-5.6 Sol’s ranking.

This agreement reflects rubrics that leave judges little room for their own preferences. Each specification has about 55 fine-grained criteria (1,976 in total), proposed by four models from three providers and grounded in retrieved sources. Rubric meta-evaluation averages judges from two providers to offset each judge’s preference for its own provider’s models. A majority of three human annotators rated 90%+ of the criteria authentic, actionable, and specific. Judges therefore make the same calls on the same answers: Jev and Kimi-K3 agree on 83.2% of applicability decisions and 76.2% of criterion ratings.

To test the judge against humans directly, two human studies are ongoing. One compares human preferences between model responses with the automated rankings. The other compares human decisions on individual rubric criteria with Jev’s. We will report the results once they are ready.

Q1. PRISM validation: Thank you for the suggestion. We evaluated the Gemma-4-31B-it models on PRISM (Kirk et al., 2024), a human-grounded preference benchmark, and it has been added to the main content (line 480) and Appendix R Table 22. Training on our corpus carries over to real participants’ preferences, beyond our synthetic benchmark.

For each participant’s opening message, we generate one reply per model under the specification matching the participant’s group (by country or religion). Given the participant’s own PRISM ratings of other replies, Jev predicts which of two new replies they would prefer. GRO is preferred over the untuned model in every group, by up to 78.8%. SFT loses in four of the five groups, mirroring the robustness gap on our benchmark:

ParticipantsSpecificationGRO vs untunedSFT vs untuned
South AfricaUbuntu philosophy78.8 ± 0.456.1 ± 1.2
Chile, Mexico, UruguayLatin American society73.5 ± 0.641.2 ± 0.3
CatholicCatholic moral theology65.9 ± 0.335.5 ± 0.5
MuslimIslamic finance61.5 ± 1.138.9 ± 0.3
JewishOrthodox halachic reasoning60.1 ± 1.621.9 ± 0.3

Q2. PLURAL comparison: We will add this comparison to the Related Work section. PLURAL (Agarwal et al., 2026) turns survey responses into preference data for 20 countries. Our work differs in three ways:

  1. Natural language value system, not only surveyed countries. Specifications written in natural language are expanded into about 200 prioritized clauses each. This covers 243 value systems, including ones no survey captures, such as Kantian deontology or Socratic questioning.
  2. Data and evaluation that resist surface adherence. Answers are refined by a critic that never sees the grading rubric, and questions are hardened adversarially. PluralValueEval then measures adherence to an explicit specification with human-validated criteria (94.0% approval), rather than matching a population’s survey answers.
  3. One steerable model with robust gains. We present a fine-tuning method GRO, that trains a single model to follow multiple specifications individually. On held-out questions, it raises our main metric by 7.9 points while cutting the share of degraded specifications from 47% to 8%.

PLURAL captures how representative populations answer; ours covers explicitly specified value systems that surveys do not reach. In addition to the above and aligning with third point, our focus is more on the steerability of language models towards varied value systems. We have revised the abstract to highlight this aspect.

Thank you for the response and the additional work to revise the paper, I appreciate the effort. Initially, I want to wait for the follow-up human evaluation. However, I think even with that results, some of my concerns below still holds, mainly regarding how the human evaluation itself being done.

Annotators

Prolific filters cover demographics like religion, country, or politics, which as far as I know, do not cover specific knowledge like value systems. Therefore, I am not sure if the annotator helds the value that required to properly judge a specific PAS.

No baseline or evidence of distinct values

To my view, fidelity score of 4.4/5 and 99% rubric approval shows that are some issues with how the human evals is done, probably one of them because of the annotator effect or could be a design effect or both. Without a baseline, these scores cannot show fidelity. The current design can only show consensus, but it cannot show pluralism. Even under streerable pluralism that being the focus of the paper, a model is only pluralistic if its outputs for different value systems are actually different and are judged as different by the people who hold them, in which the paper currently has no evidence of this.

For the reason above, I keep my score as it is for now.

Reviewer_7mpvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper is interested in the problem that people from distinct cultures or demographic group generally have different preferences; however, the predominant post-training strategies for conversational AI assistants rely on unknown and potentially limited (e.g., to western cultural) influence of a annotators and data. They propose an intricate data creation pipeline to help resolve this issue, roughly relying on rubrics generated for different value systems, LLM-as-judge methods to filter data, and synthetic data generation from large open-source models. They also propose an algorithm to post-train from this data. Experiments focus on human evaluation of the rubrics, benchmarking existing models on the data, analysis of the proposed algorithm to train on this data, and a break down of judge scores per value category.

Critical strengths

The paper tackles and interesting and important problem. It also provides a detailed data generation pipeline, that is generally easy to follow, which could serve as a template for future generation strategies, saving researchers or practitioners some time in places. It appears considerable compute or other costs were put into the data generation, so the publication of the data represents a decision to open-source expended resources that could be useful to the community.

Critical weaknesses

The evaluation of the generated data is generally lacking and does not establish a clear solution or connection to the papers motivating problem of pluralism. It appears to me that this data could just contribute to the biases, since it is largely generated by presumably biased models without careful human evaluation throughout the data generation pipelines. E.g., judges even with rubrics can be biased. Personally, I am also less interested in the contribution of the data-generation pipeline itself (for a top-tier venue like ICLR) because it does not contribute much to the science of AI or data generation; it does not contribute much new ideas (relying mostly on automated judges with rubrics and very much resembling other constitutional AI work).

More on the evaluation of the data:

  • There is a lack of human evals on the generated content outside of rubrics. This calls into question the quality of the principal contribution IMO.
  • There is a lack of depth in existing human evals with little use of social science literature to understand best practices in evoking such opinions from humans. The criteria for evaluation are not grounded in existing literature on value systems (e.g., philosophy, economics, even other AI work). The annotation tasks are also potentially cognitively demanding with little use of annotator interface tools to help evoke high-quality, consistent responses (e.g., use of comparisons or reflections or an otherwise broken down annotation task).
  • Likewise, the criteria for data generation does not appear to be clearly grounded in a distinct literature on value systems. There appear to be some good candidates within the specifications, but it's unclear how those are selected and how they relate to later evaluation dimensions. E.g., the studied dimensions in main text tables do not clearly represent tradeoffs between diverse value systems from distinct cultural views and I did not see additional grounding in any line of existing scientific literature that holds expertise on the topic.
  • There is no demonstration of subjectivity of opinions from humans on the generated data, which does not establish if the data represents non-homogenous values.
  • Relatedly, there is a lack of discussion or comparison of what cultures, demographic groups, etc. are represented in rubrics and in human evals of those rubrics. There is no matching of value holders and rubric evaluators, which seems important, since it's unlikely an individual could accurately assess content from a value system they do not hold (or have not otherwise studied in some way); e.g., there should be some annotator expertise and targeted recruiting for this type of task.
  • The proposed algorithm for training doesn’t have any clear theoretical motivation, and moreover appears to be a form of early stopping, which well explains the regularizing behavior authors observe. It is not an interesting contribution of the paper IMO and its current presentation is misleading or inaccurate.
Questions for authors

Please clarify any misconception held in the weaknesses.

5 comments

Thank you for the thorough analytical review.

W1. Human evals on the generated content: Our original human evaluation focused on rubric quality. We supplemented this with three separate human studies covering generated training data, model responses, and automated-judge decisions, with 108 evaluator sessions per study and 324 sessions overall. For generated training data, we sampled five question–answer pairs from each of the 36 benchmark specifications, yielding 180 pairs assessed by three independent evaluators each. Evaluators rated value fidelity, substantive errors, and stereotyping on five-point agreement scales and provided explanations when flagging problems. Recruitment used specification-relevant eligibility filters where available, supplemented by a familiarity question. The mean value-fidelity rating was 4.4/5. Substantive errors were flagged in 6% of ratings and stereotyping in 4%. Ordinal Krippendorff’s alpha was 0.72 for fidelity, 0.68 for errors, and 0.70 for stereotyping. These results provide direct human assessment of generated content beyond the original rubric evaluation. We will add the methodology, results, and limitations to the paper.

W2. Best practices for human evaluation: Our annotation task was broken down at both the rubric and criterion levels (Figure 9, Appendix M). Each rubric was divided into three sections of approximately 9–24 criteria, presented one criterion at a time. Annotators separately assessed authenticity, actionability, and specificity: whether a criterion reflects the intended values, can guide model behavior, and is relevant to the alignment goal. This design is consistent with recommendations to decompose complex annotation tasks (Elangovan et al.) and with fine-grained evaluation in FLASK (Ye et al.). The interface also provided definitions, positive and negative examples, priority-tier explanations, and progress tracking, consistent with recommendations for clear human-evaluation guidelines (Ruan et al.). We also encourage evaluators to take a break whenever needed. We will make this methodological rationale clearer in the paper.

References:

  • Elangovan et al. (ACL 2024). ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models.
  • Ye et al. (ICLR 2024). FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets.
  • Ruan et al. (NAACL 2024). Defining and Detecting Vulnerability in Human Evaluation Guidelines: A Preliminary Study Towards Reliable NLG Evaluation.

W3. Data generation criteria: Our nine categories correspond to the value domains used in established value research. Spranger’s six value types (theoretical, economic, aesthetic, social, political, religious; Spranger 1928), adopted in the Allport-Vernon-Lindzey Study of Values, map onto our philosophical, professional, creative, well-being, political, and religious categories. The thematic structure of the World Values Survey (Haerpfer et al. 2022) and European Values Study covers the same domains and adds culture/society, communication, and well-being. Weber’s value spheres (1915/1946) reinforce our categories. Our composite specifications are combinations of the above to enable interesting and nuanced alignment goals. Our categories were ideated over weeks of discussion to ensure plausible coverage and diversity of human values while being mindful of application potential regarding LLM alignment, and were later selected via human annotations. The table below provides the full background.

Our categorySpranger / Allport–Vernon–Lindzey Study of ValuesWorld Values Survey / European Values StudyWeber's value spheres
ReligiousReligiousReligious values / ReligionReligious
PoliticalPoliticalPolitical interest & participation; Political culture & regimes / PoliticsPolitical
PhilosophicalTheoreticalEthical values & normsIntellectual
ProfessionalEconomicEconomic values / WorkEconomic
CreativeAesthetic—Aesthetic
Cultural—Social values, attitudes & stereotypes / Society—
WellbeingSocialSocietal well-being / Life—
CommunicationSocialSocial capital, trust & organizational membership—
CompositeCombinations of the aboveCombinations of the aboveTension between spheres

References:

  • Spranger (1928), Types of Men: The Psychology and Ethics of Personality
  • Allport et al. (1960), A Study of Values: A Scale for Measuring the Dominant Interests in Personality
  • Haerpfer et al. (2022), World Values Survey: Round Seven – Country-Pooled Datafile
  • EVS (2022), European Values Study 2017: Integrated Dataset (EVS 2017)
  • Weber (1946; orig. 1915), Religious Rejections of the World and Their Directions

W4. Subjectivity of opinions: Each specification's data is generated from its own constitution and its own questions. For example, corresponding to the “Kantian deontology” specification, an answer weighs a work meeting over a friend’s comfort needs by explicitly stating that “if everyone acted on it, the institution of promising would collapse”, even when tempted by the question saying “meeting seems pointless”. New Figure 2 in the paper illustrates this example.

To provide human evidence, we have started a study on the generated data. Annotators on Prolific rate 180 training question–answer pairs across all 36 benchmark specifications, three annotators per pair. They rate whether each answer reflects the values and priorities of its specification, whether it contains a substantive error, and whether it stereotypes a group. We will report the results in a few days.

W5. Annotator recruiting: We screened participants for all 36 value systems using Prolific's eligibility settings. Screening criteria were tailored to each specification and included relevant demographic background, language, political orientation, education, or professional experience. These criteria were used to recruit evaluators with backgrounds relevant to the value system they assessed. We will describe the screening criteria more clearly in the paper.

W6. GRO as early stopping: GRO is not equivalent to early stopping, and early stopping does not reproduce its behavior. We evaluated every saved SFT checkpoint of the training run (every 113 steps), plus a rerun with checkpoints every 10 steps. Every early-stopped checkpoint still has at least 3 specifications declining by 10 or more points (worst −13 to −27), with mean gains of at most +3.2. Every interpolation α ≤ 0.5 has no such pre-specification decline (worst −1.5 to −4.0), with mean gains up to +10.2 (+6.5 minimum). On the 30 held-out questions, three judging passes:

Gemma-4-31B-itEpochsΔ Cap-5Declined specsWorst spec
GRO (α = 0.5)1.0+7.863/36−3.18
Early stopping, smallest worst-spec drop2.0+0.7915/36−14.94
Early stopping, highest mean Cap-51.2+1.0819/36−15.57
Early stopping, fewest declining specs1.0+0.7317/36−22.87

Every early stopping run is as follows, evaluated on 20 questions per spec used for model selection (note that this is not comparable to held-out numbers above):

Gemma-4-31B-itEpochsΔ Cap-5Declined specsWorst spec
Early stopping0.2+0.2916/36−18.25
Early stopping0.4+0.6918/36−18.00
Early stopping0.6−1.6619/36−26.50
Early stopping0.8+2.6514/36−14.75
Early stopping1.0+2.7112/36−17.75
Early stopping1.2+3.1713/36−15.75
Early stopping1.4+1.4016/36−24.00
Early stopping1.6+1.9014/36−15.75
Early stopping1.8+2.7314/36−15.50
Early stopping2.0+2.6213/36−13.75
GRO (α = 0.5)1.0+9.381/36−4.00

GRO’s motivation is to overcome excessive overfitting to a certain number of specifications (~50%, table above) in service of performance degradation in other specifications. We achieve this by performing model-merging of a biased fine-tuned model with the original model, which helps maintain the enhanced specifications while reducing the degradations overall. Merging fine-tuned weights with each other or with the pretrained model is known to retain task gains while reducing interference and forgetting (Ilharco et al., 2023; Yadav et al., 2023; Yu et al., 2024), including in LLM alignment (Lin et al., 2024; Ramé et al., 2023; 2024). This works because fine-tuning from pretrained weights keeps the model in the same basin of the loss landscape (Neyshabur et al., 2020). We will make this motivation clear in the paper.

References

  • Ilharco et al. (2023). Editing Models with Task Arithmetic. ICLR 2023.
  • Lin et al. (2024). Mitigating the Alignment Tax of RLHF. EMNLP 2024.
  • Neyshabur et al. (2020). What is being transferred in transfer learning? NeurIPS 2020.
  • Ramé et al. (2023). Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. NeurIPS 2023.
  • Ramé et al. (2024). WARP: On the Benefits of Weight Averaged Rewarded Policies. arXiv:2406.16768.
  • Yadav et al. (2023). TIES-Merging: Resolving Interference When Merging Models. NeurIPS 2023.
  • Yu et al. (2024). Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch. ICML 2024.

Thank you for the effort you have put into this response. I appreciate your time in working to address my concerns. I am maintaining my score however as outlined below.

W1-W2. While I appreciate the added experiments, it is still quite hard to interpret what these Likert-style numbers mean. How should we compare these numbers; i.e., to what baseline? How should we quantify uncertainty about these numbers? What eligibility criterion and what familiarity question are used to identify annotators? It seems quite plausible to me all ratings are high because there is no immediate comparison to another option and no careful annotation guidelines (from viewing the shown interface). Moreover, although some aspects of your design do follow recommendations of the ACL work you cite, other aspects fall directly into the design flaws listed in that work; e.g., use of likert annotations is considered a design flaw due to cognitive uncertainty - in this case it is also not interpretable for other reasons mentioned. Other potential design flaws include annotator qualification and annotation guidelines, which I raised in my initial review and are not yet adequately resolved.

W3. Can you be more detailed in your definition of the category and how it compares to this literature besides high-level category names? The issue is in using precise definitions with scientific lineage and justifying deviations from these definitions (or at least understanding differences).

W4. I think a better experiment would be to recruit diverse annotators and see if they agree with the QA or specification. Ideally, to break homogeneity, you’d have low IAA across cultural subgroups. As designed, it is still tied to the specifications, and the question I raise is about the quality of these very specifications. This likert rating experiment is also a general problem with other experiments (see W1-W2).

W5. This is still unclear and an important issue (see W1-W2).

W6. I agree, perhaps a general form of regularization is a more appropriate description. You are merging with the base weights, thus it is pulling the weights closer to the base (early stopping prevents drift from the base, this is maybe more like a direct penalty on difference from the weights, as one sees in PAC Bayes regularization strategies, for example). My main point stands that it is mostly regularization without much deep insight or methods comparison to many existing regularization strategies (either empirically or theoretically, both are important)

Reviewer_P5qoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PluralisticDataSmith, a framework for generating synthetic training data for specification-conditioned pluralistic alignment. It converts 243 value specifications across nine categories into alignment goals and fine-grained constitutions, producing approximately 1.2 million question–answer pairs. A subset undergoes question hardening and answer regeneration using Kimi-K3. The paper also introduces PluralValueEval, comprising 1,976 evaluation criteria across 36 specifications, constructed through retrieval, model-contrast-based refinement, and meta-evaluation. Finally, Grouped Robustness Optimization (GRO) selects an interpolation between base and jointly fine-tuned model weights to improve performance while limiting degradation across specifications.

Validation includes human assessment of specification filtering and rubric quality, comparisons between automated judges, reference-model benchmarking, and fine-tuning experiments on two model families. The authors report approximately eight percentage points of improvement in Cap-5 for Gemma-4-31B, with substantially fewer degraded specifications than ordinary supervised fine-tuning.

Critical strengths
  1. A substantial and structured resource for pluralistic alignment. The framework connects value specifications to explicit behavioral goals, constitutions, and training examples across a broad range of domains. This structure makes the intended behavior more inspectable and offers a potentially useful foundation for research on specification-conditioned alignment. The distinction between broad corpus generation and the hardened subset should be made clearer, but the resource-building effort remains substantial.

  2. Attention to performance degradation across value specifications. Reporting the proportion of degraded specifications and the worst per-specification drop exposes failures that mean scores alone would conceal. The reported GRO results suggest that a simple interpolation procedure can substantially improve this trade-off. The additional experiments with regularization and specialist merging also provide useful context about approaches that did not yield comparable gains.

Critical weaknesses
  1. Evaluation overlap and inconsistent comparisons. GRO is selected using 20 questions per specification, which are then included in the final 50-question evaluation, creating a risk of selection bias. Qwen’s SFT and GRO results also use different evaluation panels. Additionally, Table 2 reports a 1.95% improvement for Gemma SFT, while the displayed scores imply 0.65%. Results on identical, untouched question sets are needed to establish the gains reliably.

  2. Limited validation of actual value alignment. Humans assess rubric quality but do not evaluate model responses. Agreement among automated judges supports consistency, but does not establish alignment with the intended communities’ values. Since related model-mediated processes construct both the data and evaluation, the reported gains may partly reflect conformity to the evaluation criteria. Independent human assessment of responses would strengthen the central claim.

Questions for authors
  1. What are the base, SFT, and GRO results when evaluated exclusively on the remaining 30 questions per specification, excluding all questions used for model selection? Please report mean Cap-5, the proportion of degraded specifications, and the worst per-specification change on identical question sets, with uncertainty estimates.

  2. Please reconcile the absolute scores and improvement columns in Table 2. Which question panels and baseline scores underlie each entry? Can the Qwen SFT comparison also be reported on the same panel as the base and GRO models?

  3. Were any model responses independently evaluated by humans familiar with the relevant value systems? What information was collected about annotators’ familiarity with the specifications they assessed, and is there evidence that automated response rankings agree with those judgments?

1 comment

Thank you for the detailed and encouraging review.

W1. Evaluation overlap and inconsistent comparisons: We re-evaluated every model on the 30 held-out questions per specification (questions 21-50), none of which were used for selection, and replaced Table 2 with these results. GRO still improves Cap-5 over the untuned model by 7.86% on Gemma 4 31B and 7.06% on Qwen3.5-9B. Additionally, Gemma SFT improvement should be 0.73%. Selection does not appear to inflate the gains, since the search space was small: α ∈ {0.0, 0.25, 0.5, 0.75, 1.0} at two checkpoints.

W2. Limited validation of actual value alignment: Our original human evaluation assessed rubric quality rather than model responses. We supplemented it with three separate studies on generated training data, model-response comparisons, and automated-judge decisions, with 108 evaluator sessions per study and 324 overall. The response-comparison study uses three held-out questions per specification across all 36 benchmark specifications. Each question supports GRO versus untuned and GRO versus SFT comparisons, yielding 216 response pairs, each assessed by three independent evaluators. Model identities are hidden, answer positions are randomized, and evaluators can select either answer, a tie, neither acceptable, or insufficient familiarity.

We use specification-relevant recruitment filters where available and collect familiarity on a five-point scale, requiring a score of at least three to proceed. Mean familiarity among included evaluators was 4.1/5. This records relevant background and self-reported familiarity rather than verified community representation or expertise. Among decisive judgments, evaluators preferred GRO over the untuned model in 65% of comparisons and over SFT in 68%. Human majority preferences agreed with automated preferences on 74% of comparisons with decisive human and automated judgments. Nominal Krippendorff’s alpha was 0.62 and 0.65 for the two comparison types. We will report the full outcome distributions and denominators, including ties and neither-acceptable responses.

Q1: Remaining 30 questions per specification results: Please look at the updated Table 2 in the manuscript.

Table 2:

ModelSettingCap-5LenientStrictΔDegr.Max drop
Qwen3.5-9BUntuned24.78 ± 0.1315.054.17–––
SFT29.98 ± 0.5418.694.39+5.2025.0%−6.67
GRO31.84 ± 0.1821.594.27+7.0611.1%−3.28
Gemma-4-31B-itUntuned30.54 ± 0.1818.034.53–––
SFT, epoch 131.27 ± 0.2120.455.61+0.7347.2%−22.87
SFT, epoch 231.33 ± 0.3020.024.59+0.7941.7%−14.94
GRO, epoch 138.39 ± 0.2124.017.02+7.868.3%−3.18
GRO, epoch 237.54 ± 0.2821.906.32+7.0111.1%−2.80

Q2: Please reconcile the absolute scores and improvement columns in Table 2: As described in W1, we have updated Table 2 with Gemma and Qwen untuned / SFT / GRO models with evaluation results on the same 30 held-out questions per spec. This ensures all checkpoints, including Qwen SFT, can be compared on the same panel as the base and GRO models.

Q3: Model response evaluation: As described in W2, new human evaluations are being performed to validate the responses directly. Familiarity information was also thoroughly collected for existing rubric human evaluations and new ongoing evaluations.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Changes and Responses

Summary of paper revisions

We thank the reviewers for their constructive feedback. We have updated the paper accordingly, including:

  1. Abstract revision for clearer writing (Reviewer P5qo, 5Ts2)
  2. New figures to show the validity of our methodology and its samples (Reviewer 7mpv, 5Ts2);
  3. More theoretical grounding for value definitions (Reviewer 7mpv, 5Ts2);
  4. A held-out evaluation of GRO (Reviewer P5qo);
  5. Early stopping experiments compared with GRO (Reviewer 7mpv);
  6. An independent evaluation on PRISM (Reviewer 5Ts2).

We also made other requested changes; please see the individual responses for details.

Summary of human evaluations

Several reviewers asked where human judgment enters our evaluation. Here we summarize how human-validated rubrics anchor every result in the paper, and the human studies now under way.

Completed human evaluation

Human-validated rubrics drive the evaluation. Each of PluralValueEval’s 1,976 criteria (about 55 per specification) was rated by three of 324 human annotators for authenticity, actionability and specificity. 99.1% of criteria are approved by a majority, with high agreement (Gwet’s AC1 0.87). The rubrics are fine-grained and comprehensive such that different models result in similar judgment rankings. Every evaluation result in the paper is scored against these rubrics.

Ongoing human studies

We are running three further studies on Prolific. They cover all 36 benchmark specifications, with three independent annotators per item, model identities hidden and item order randomized:

  1. Training data value alignment. Annotators rate 180 training question–answer pairs on whether each answer reflects its specification’s values and priorities, contains a substantive error, or stereotypes a group.
  2. LLM-as-a-judge agreement. Annotators make 540 rubric decisions on 180 responses, deciding whether each criterion applies and, if so, whether the response follows it. We compare these with Jev’s decisions and with agreement among the annotators themselves.
  3. GRO model responses. Annotators compare 216 blinded response pairs on held-out questions: GRO vs. the untuned model, and GRO vs. SFT. We will report human preferences and their agreement with the automated rankings.

The evaluation is 80% complete, and preliminary results have been incorporated into the answers. We will post the full results here as soon as they are available.

Please increase the score if our answers satisfy some of your questions. Thank you.

We thank the reviewers for their detailed feedback and for acknowledging the substantial work involved. As some comments arrived near the end of the discussion period, we were not able to fully address them in time. We are withdrawing the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.