- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how the demographic composition of post-training data, post-training algorithms, and training-stage order jointly shape the value-related behavior of LLMs. Using OpinionQA, the authors construct controlled corpora in which the proportion of majority answers associated with two demographic groups is varied from 0% to 100%. Experiments cover party affiliation, education, and religion, using Llama-3.1-8B, Qwen3-8B-Base, and OLMo-2-7B under five post-training paths: SFT, SFT→DPO, SFT→GRPO, SFT→DPO→GRPO, and SFT→GRPO→DPO. The paper measures group alignment using an agreement score and introduces an amplification score to quantify whether the model's final alignment exceeds the opinion imbalance in the training corpus. It further studies preference removal, intervention stability, and task-only distillation. The main findings are that different post-training paths amplify corpus opinion mixtures differently, DPO/GRPO ordering matters, previously learned preferences can be differentially removed depending on training history, and preference-related behavioral differences can propagate to student models through distillation.
Critical strengths
- The controlled mixture × post-training-path design is systematic and useful. The paper varies demographic opinion mixtures while keeping the underlying prompts and corpus size largely fixed, and compares several post-training paths under the same setup. In particular, comparing DPO→GRPO with GRPO→DPO provides relatively direct evidence that stage ordering itself can matter.
- The paper goes beyond static preference measurement. In addition to studying amplification, it examines whether learned preferences can be removed, whether models with similar final preferences differ in intervention stability because of their training histories, and whether preference-related behavior transfers through task-only distillation. This gives the paper a broader view of how post-training history affects subsequent model behavior.
Critical weaknesses
1. The interpretation of “value preference” is broader than what the main measurement directly supports.
OpinionQA provides full demographic-group response distributions, but the main analysis reduces each group to its majority answer and keeps questions where the two groups' modal answers differ. This is a reasonable construction for controlled conflicting-opinion experiments, but it discards within-group heterogeneity and can treat two very similar distributions as opposing preferences when their modal answers barely differ. The primary measurement therefore more directly captures alignment with demographic groups' modal survey responses than a general latent “value preference.” The paper includes distribution-based robustness analysis, but the distinction between the operational measurement and the broader value interpretation should be made more explicit.
2. The experiments identify how post-training reshapes measured alignment, but not whether post-training is the primary source of the underlying preference.
Because the pretrained base models do not provide a meaningful pre-training preference baseline under the paper's evaluation format, the current design cannot distinguish preferences newly created during post-training from latent preferences already acquired during pre-training and later exposed, amplified, or redirected. The strongest supported claim is therefore that post-training systematically reshapes observed preference conditional on a fixed pretrained model, rather than that the preferences themselves originate during post-training.
3. Cross-path comparisons do not always isolate the effect of the post-training algorithm.
Several post-training paths differ simultaneously in optimization method, number of stages, accumulated training exposure, and inherited intermediate checkpoints. The DPO→GRPO versus GRPO→DPO comparison cleanly supports an order effect because the component methods and stage counts are matched, but broader comparisons such as SFT versus multi-stage paths are better interpreted as path-level effects rather than isolated algorithm effects. Stage-matched or update-matched controls would make this attribution cleaner.
4. The novelty is mainly in the controlled joint characterization, rather than in the broad phenomenon itself.
Prior work already shows that different post-training methods can lead to different value-related behaviors and that later stages can modify behavior learned earlier. The clearest contribution here is the systematic characterization of opinion-mixture proportion × post-training path × stage order under a common controlled setup. I think the paper would benefit from positioning this narrower contribution more explicitly, rather than making the broader phenomenon of “post-training shapes values” appear novel.
5. External-validity and robustness coverage remain somewhat limited.
The main experiments use three models in a relatively narrow 7B–8B scale range, so it is unclear whether amplification magnitude, stage-order effects, and preference removability persist for substantially larger models. In addition, the multi-seed robustness analysis is strongest for the party-based amplification experiment and is not equally extensive for education, religion, preference removal, intervention stability, or distillation transfer.
Questions for authors
- How broadly do the authors intend the term “value preference” to be interpreted? Would “alignment with demographic-group survey responses” be a more precise description of the primary measurement?
- Can the authors more systematically compare the majority-answer metric with the full-distribution metric, especially for questions where the two groups have different modal answers but very similar response distributions?
- Is there a way to measure the pretrained models' initial preferences using an elicitation protocol that does not depend on the post-SFT option-number format? If not, would the authors consider narrowing the causal claim to post-training reshaping observed alignment?
- Can stage-matched or update-matched controls be added to separate optimization-method effects from generic continued-training or additional-exposure effects?
- Relative to Bhatia et al. (2026), Value Drifts: Tracing Value Alignment During LLM Post-Training (TACL), which already studies how post-training algorithms and datasets affect value alignment and reports that different preference-optimization algorithms can produce different value outcomes even with fixed preference data, which findings in this paper are genuinely new? In particular, can the authors distinguish contributions arising from the controlled mixture sweep and stage-order design from broader phenomena already established in prior work?
- In the task-only distillation experiments, what exactly is being transferred: a semantic value representation, generic response tendency, source-task prediction behavior, or another correlated property? Can differences in teacher output distributions on the source tasks be controlled more directly?
- Do the authors expect the amplification, ordering, and preference-removal results to hold for substantially larger models? If larger-scale experiments are infeasible, the corresponding generalization claims could be scoped more explicitly.
- Can at least a subset of the preference-removal, intervention-stability, and distillation-transfer experiments be repeated across independent seeds to test whether those findings are as stable as the main amplification result?
7 comments
Response (2/3)
W4 (same as Q5). The novelty is mainly in the controlled joint characterization, rather than in the broad phenomenon itself.
We agree with the reviewer. The contribution of this paper is a systematic comparison, under one controlled setting, of how post-training methods, the order of post-training stages, and the opinion mixture in the corpus jointly shape value preferences. The new findings from this design are the following. Post-training amplifies the opinion imbalance in the corpus, and the degree of amplification differs across paths. The order of the stages determines whether an existing preference is easy to remove. Models with similar preferences differ in their stability under interventions. The revised PDF states this contribution explicitly at the end of the introduction (Section 1, page 2).
W5-a (same as Q7). External-validity and robustness coverage remain somewhat limited. The main experiments use three models in a relatively narrow 7B–8B scale range, so it is unclear whether amplification magnitude, stage-order effects, and preference removability persist for substantially larger models.
We repeated the experiments on models from 0.6B to 32B parameters (Qwen3 at 0.6B, 1.7B, 4B, and 14B, and OLMo-2 at 1B, 13B, and 32B; Appendix G.7, page 61). Amplification occurs at all sizes, and the amplification scores of the 14B and 32B models are close to those of the 8B and 7B models in the main experiments (Figure 30, page 62). SDG produces weak amplification at all sizes, and the order effect that SDG amplifies less than SGD holds except for the smaller Qwen3 models. The conclusions on preference removal are unchanged. SFT removes most of an existing preference, whereas the paths that include DPO or GRPO retain most of it (Table 35, page 63). We have also limited the scope of our conclusions in Section 7 (page 9) of the revised PDF. Our experiments cover base models of up to 32B parameters, and whether the findings hold for larger models remains to be tested.
W5-b (same as Q8). In addition, the multi-seed robustness analysis is strongest for the party-based amplification experiment and is not equally extensive for education, religion, preference removal, intervention stability, or distillation transfer.
We repeated the preference-removal, intervention-stability, and distillation-transfer experiments with two additional random seeds (Appendix G.1, page 49). For preference removal, we retrained the whole path for each seed, and the overall trends are unchanged (Figure 20, page 51). For stability under interventions, the two findings of Section 5 (page 7) hold for every seed (Table 30, page 52). For distillation transfer, we retrained all student models with the new seeds, and the average preference gap is positive for every seed (Tables 31 and 32, page 53).
Q1. How broadly do the authors intend the term “value preference” to be interpreted? Would “alignment with demographic-group survey responses” be a more precise description of the primary measurement?
Please refer to our responses to W1-a.
Q2. Can the authors more systematically compare the majority-answer metric with the full-distribution metric, especially for questions where the two groups have different modal answers but very similar response distributions?
Please refer to our responses to W1-b.
Q3. Is there a way to measure the pretrained models' initial preferences using an elicitation protocol that does not depend on the post-SFT option-number format? If not, would the authors consider narrowing the causal claim to post-training reshaping observed alignment?
Please refer to our responses to W2.
Q4. Can stage-matched or update-matched controls be added to separate optimization-method effects from generic continued-training or additional-exposure effects?
Please refer to our responses to W3.
Q5. Relative to Bhatia et al. (2026), Value Drifts: Tracing Value Alignment During LLM Post-Training (TACL), which already studies how post-training algorithms and datasets affect value alignment and reports that different preference-optimization algorithms can produce different value outcomes even with fixed preference data, which findings in this paper are genuinely new? In particular, can the authors distinguish contributions arising from the controlled mixture sweep and stage-order design from broader phenomena already established in prior work?
Please refer to our responses to W4.
Response (3/3)
Q6-a. In the task-only distillation experiments, what exactly is being transferred: a semantic value representation, generic response tendency, source-task prediction behavior, or another correlated property?
We analyze the source of the transfer in Appendix F.3 (page 45). On the task with numeric labels (WinoGrande), the label distribution of the teacher's responses leans in the same direction as the OpinionQA majority answers of the group that the teacher is aligned with (Table 22, page 46), which shows that the option labels are one route of the transfer. On the task with alphabetic labels (MMLU), the numeric labels favoured by the distilled students match the group that their teacher is aligned with (Table 23, page 47). On the task with free-form responses (GSM8K), the semantics of the teacher's responses is closer to the group that the teacher is aligned with (Table 24, page 47). Therefore, the transfer includes both a response tendency and a semantic component.
Q6-b. Can differences in teacher output distributions on the source tasks be controlled more directly?
To control the differences between the teachers' outputs more directly, we distill only on the GSM8K questions for which the two teachers give the same final answer, and we use a random subset of the same size as a control (Appendix F.3, page 45; Table 25, page 48). On these questions, the average preference gap of the students is still positive. Therefore, value transfer cannot be avoided by controlling the distillation corpus.
Q7. Do the authors expect the amplification, ordering, and preference-removal results to hold for substantially larger models? If larger-scale experiments are infeasible, the corresponding generalization claims could be scoped more explicitly.
Please refer to our responses to W5-a.
Q8. Can at least a subset of the preference-removal, intervention-stability, and distillation-transfer experiments be repeated across independent seeds to test whether those findings are as stable as the main amplification result?
Please refer to our responses to W5-b.
Response (1/3)
We thank the reviewer for the detailed and thorough comments. Please refer to the revised PDF (marked in blue and green backgrounds) for more details. Below, we respond point by point.
We agree with the reviewer. In this paper, value preference refers to the alignment of the model's answers with demographic-group survey responses, and we do not claim that it is a more general latent value preference. The revised PDF states this in two places, namely the footnote where the term first appears in the introduction (Section 1, page 1) and the place where we define the two scores (Section 2.6, page 4). Both places use the wording suggested by the reviewer.
Beyond the highest-probability answer, we also evaluate the models with an agreement score based on their full answer distributions, and amplification remains (Appendix C.5, page 34). In addition, we divide the questions into three sets by the distance between the two groups' answer distributions (Similar, Middle, and Different), and in each set we compute the amplification scores from the highest-probability answer, by top-p sampling, and from the full answer distribution (Appendix C.6, page 36; Figure 15, page 36). In the Similar set, where the two groups' distributions are closest, all three amplification scores remain positive but are lower than in the Different set (10.2 versus 13.1 with the highest-probability answer, and 6.3 versus 9.3 with the full answer distribution). Therefore, amplification remains on questions where the two groups' distributions are close, and it is stronger on questions where they are further apart.
We measure the initial preferences of the base models by continuation, which does not rely on the option-number format learned in SFT (Appendix A.2, page 18; Table 5, page 20). All three base models show measurable initial preferences, and the direction is the same across them. They lean toward g1 on the party and education attributes and toward g2 on the religion attribute. In addition, we state in Section 2.3 (page 3) of the revised PDF that this paper examines how post-training changes the preferences of a given pretrained model, rather than where these preferences originate.
We have added two kinds of controls, one matched in the number of stages and one matched in the number of training updates (Appendix C.2, page 30). In the stage-matched controls, we compare each path with the SFT-only path that has the same number of stages (S2S and SSS). An additional SFT stage hardly changes the amplification score, whereas S2G is lower than S2S for every LLM and demographic attribute (Table 10, page 30). In the update-matched controls, the final stage of every path is trained on the same corpus for the same number of steps, and the paths ending in GRPO still move less toward the majority of the corpus than the SFT-only paths (Figure 11, page 32). Therefore, the weaker amplification of the paths ending in GRPO comes from the post-training method rather than from the number of stages or the amount of training. Section 3 (page 5) of the revised PDF also points to these results.