acceptodds
Under review as a conference paper at ICLR 2027

How Does Post-Training Shape LLM Value Preferences?

Abstract

During post-training, large language models (LLMs) learn value preferences from corpora that may contain imbalanced proportions of opinions associated with different demographic groups. Amplifying these imbalances may further underrepresent some groups' opinions. However, how opinion mixtures shape value preferences across post-training paths remains unclear. To answer this question, we conduct controlled experiments using post-training corpora with different opinion mixtures and five post-training paths based on SFT, DPO, and GRPO. We find that post-training amplifies opinion imbalances differently across post-training paths. Once value preferences are amplified, whether further post-training can remove them depends on both the post-training method and the opinion mixture in the corpus. More concerningly, a model's stability under interventions depends on whether further post-training changes its value preference. Moreover, these value preferences are transferred to student models even when distillation uses corpora with no explicit opinion questions. These findings suggest that both the opinion mixture in post-training corpora and the choice of post-training method should be considered when developing, evaluating, and reusing LLMs.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLMs, Value Alignment, Post-Training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_JUAvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how the demographic composition of post-training data, post-training algorithms, and training-stage order jointly shape the value-related behavior of LLMs. Using OpinionQA, the authors construct controlled corpora in which the proportion of majority answers associated with two demographic groups is varied from 0% to 100%. Experiments cover party affiliation, education, and religion, using Llama-3.1-8B, Qwen3-8B-Base, and OLMo-2-7B under five post-training paths: SFT, SFT→DPO, SFT→GRPO, SFT→DPO→GRPO, and SFT→GRPO→DPO. The paper measures group alignment using an agreement score and introduces an amplification score to quantify whether the model's final alignment exceeds the opinion imbalance in the training corpus. It further studies preference removal, intervention stability, and task-only distillation. The main findings are that different post-training paths amplify corpus opinion mixtures differently, DPO/GRPO ordering matters, previously learned preferences can be differentially removed depending on training history, and preference-related behavioral differences can propagate to student models through distillation.

Critical strengths
  1. The controlled mixture × post-training-path design is systematic and useful. The paper varies demographic opinion mixtures while keeping the underlying prompts and corpus size largely fixed, and compares several post-training paths under the same setup. In particular, comparing DPO→GRPO with GRPO→DPO provides relatively direct evidence that stage ordering itself can matter.
  2. The paper goes beyond static preference measurement. In addition to studying amplification, it examines whether learned preferences can be removed, whether models with similar final preferences differ in intervention stability because of their training histories, and whether preference-related behavior transfers through task-only distillation. This gives the paper a broader view of how post-training history affects subsequent model behavior.
Critical weaknesses

1. The interpretation of “value preference” is broader than what the main measurement directly supports.

OpinionQA provides full demographic-group response distributions, but the main analysis reduces each group to its majority answer and keeps questions where the two groups' modal answers differ. This is a reasonable construction for controlled conflicting-opinion experiments, but it discards within-group heterogeneity and can treat two very similar distributions as opposing preferences when their modal answers barely differ. The primary measurement therefore more directly captures alignment with demographic groups' modal survey responses than a general latent “value preference.” The paper includes distribution-based robustness analysis, but the distinction between the operational measurement and the broader value interpretation should be made more explicit.

2. The experiments identify how post-training reshapes measured alignment, but not whether post-training is the primary source of the underlying preference.

Because the pretrained base models do not provide a meaningful pre-training preference baseline under the paper's evaluation format, the current design cannot distinguish preferences newly created during post-training from latent preferences already acquired during pre-training and later exposed, amplified, or redirected. The strongest supported claim is therefore that post-training systematically reshapes observed preference conditional on a fixed pretrained model, rather than that the preferences themselves originate during post-training.

3. Cross-path comparisons do not always isolate the effect of the post-training algorithm.

Several post-training paths differ simultaneously in optimization method, number of stages, accumulated training exposure, and inherited intermediate checkpoints. The DPO→GRPO versus GRPO→DPO comparison cleanly supports an order effect because the component methods and stage counts are matched, but broader comparisons such as SFT versus multi-stage paths are better interpreted as path-level effects rather than isolated algorithm effects. Stage-matched or update-matched controls would make this attribution cleaner.

4. The novelty is mainly in the controlled joint characterization, rather than in the broad phenomenon itself.

Prior work already shows that different post-training methods can lead to different value-related behaviors and that later stages can modify behavior learned earlier. The clearest contribution here is the systematic characterization of opinion-mixture proportion × post-training path × stage order under a common controlled setup. I think the paper would benefit from positioning this narrower contribution more explicitly, rather than making the broader phenomenon of “post-training shapes values” appear novel.

5. External-validity and robustness coverage remain somewhat limited.

The main experiments use three models in a relatively narrow 7B–8B scale range, so it is unclear whether amplification magnitude, stage-order effects, and preference removability persist for substantially larger models. In addition, the multi-seed robustness analysis is strongest for the party-based amplification experiment and is not equally extensive for education, religion, preference removal, intervention stability, or distillation transfer.

Questions for authors
  1. How broadly do the authors intend the term “value preference” to be interpreted? Would “alignment with demographic-group survey responses” be a more precise description of the primary measurement?
  2. Can the authors more systematically compare the majority-answer metric with the full-distribution metric, especially for questions where the two groups have different modal answers but very similar response distributions?
  3. Is there a way to measure the pretrained models' initial preferences using an elicitation protocol that does not depend on the post-SFT option-number format? If not, would the authors consider narrowing the causal claim to post-training reshaping observed alignment?
  4. Can stage-matched or update-matched controls be added to separate optimization-method effects from generic continued-training or additional-exposure effects?
  5. Relative to Bhatia et al. (2026), Value Drifts: Tracing Value Alignment During LLM Post-Training (TACL), which already studies how post-training algorithms and datasets affect value alignment and reports that different preference-optimization algorithms can produce different value outcomes even with fixed preference data, which findings in this paper are genuinely new? In particular, can the authors distinguish contributions arising from the controlled mixture sweep and stage-order design from broader phenomena already established in prior work?
  6. In the task-only distillation experiments, what exactly is being transferred: a semantic value representation, generic response tendency, source-task prediction behavior, or another correlated property? Can differences in teacher output distributions on the source tasks be controlled more directly?
  7. Do the authors expect the amplification, ordering, and preference-removal results to hold for substantially larger models? If larger-scale experiments are infeasible, the corresponding generalization claims could be scoped more explicitly.
  8. Can at least a subset of the preference-removal, intervention-stability, and distillation-transfer experiments be repeated across independent seeds to test whether those findings are as stable as the main amplification result?
7 comments
AuthorsOpenReview ↗

Response (1/3)

We thank the reviewer for the detailed and thorough comments. Please refer to the revised PDF (marked in blue and green backgrounds) for more details. Below, we respond point by point.

W1-a (same as Q1). The primary measurement more directly captures alignment with demographic groups' modal survey responses than a general latent “value preference.” The paper includes distribution-based robustness analysis, but the distinction between the operational measurement and the broader value interpretation should be made more explicit.

We agree with the reviewer. In this paper, value preference refers to the alignment of the model's answers with demographic-group survey responses, and we do not claim that it is a more general latent value preference. The revised PDF states this in two places, namely the footnote where the term first appears in the introduction (Section 1, page 1) and the place where we define the two scores (Section 2.6, page 4). Both places use the wording suggested by the reviewer.

W1-b (same as Q2). OpinionQA provides full demographic-group response distributions, but the main analysis reduces each group to its majority answer and keeps questions where the two groups' modal answers differ. This is a reasonable construction for controlled conflicting-opinion experiments, but it discards within-group heterogeneity and can treat two very similar distributions as opposing preferences when their modal answers barely differ.

Beyond the highest-probability answer, we also evaluate the models with an agreement score based on their full answer distributions, and amplification remains (Appendix C.5, page 34). In addition, we divide the questions into three sets by the distance between the two groups' answer distributions (Similar, Middle, and Different), and in each set we compute the amplification scores from the highest-probability answer, by top-pp sampling, and from the full answer distribution (Appendix C.6, page 36; Figure 15, page 36). In the Similar set, where the two groups' distributions are closest, all three amplification scores remain positive but are lower than in the Different set (10.2 versus 13.1 with the highest-probability answer, and 6.3 versus 9.3 with the full answer distribution). Therefore, amplification remains on questions where the two groups' distributions are close, and it is stronger on questions where they are further apart.

W2 (same as Q3). The experiments identify how post-training reshapes measured alignment, but not whether post-training is the primary source of the underlying preference.

We measure the initial preferences of the base models by continuation, which does not rely on the option-number format learned in SFT (Appendix A.2, page 18; Table 5, page 20). All three base models show measurable initial preferences, and the direction is the same across them. They lean toward g1g_1 on the party and education attributes and toward g2g_2 on the religion attribute. In addition, we state in Section 2.3 (page 3) of the revised PDF that this paper examines how post-training changes the preferences of a given pretrained model, rather than where these preferences originate.

W3 (same as Q4). Cross-path comparisons do not always isolate the effect of the post-training algorithm.

We have added two kinds of controls, one matched in the number of stages and one matched in the number of training updates (Appendix C.2, page 30). In the stage-matched controls, we compare each path with the SFT-only path that has the same number of stages (S2S and SSS). An additional SFT stage hardly changes the amplification score, whereas S2G is lower than S2S for every LLM and demographic attribute (Table 10, page 30). In the update-matched controls, the final stage of every path is trained on the same corpus for the same number of steps, and the paths ending in GRPO still move less toward the majority of the corpus than the SFT-only paths (Figure 11, page 32). Therefore, the weaker amplification of the paths ending in GRPO comes from the post-training method rather than from the number of stages or the amount of training. Section 3 (page 5) of the revised PDF also points to these results.

AuthorsOpenReview ↗

Response (2/3)

W4 (same as Q5). The novelty is mainly in the controlled joint characterization, rather than in the broad phenomenon itself.

We agree with the reviewer. The contribution of this paper is a systematic comparison, under one controlled setting, of how post-training methods, the order of post-training stages, and the opinion mixture in the corpus jointly shape value preferences. The new findings from this design are the following. Post-training amplifies the opinion imbalance in the corpus, and the degree of amplification differs across paths. The order of the stages determines whether an existing preference is easy to remove. Models with similar preferences differ in their stability under interventions. The revised PDF states this contribution explicitly at the end of the introduction (Section 1, page 2).

W5-a (same as Q7). External-validity and robustness coverage remain somewhat limited. The main experiments use three models in a relatively narrow 7B–8B scale range, so it is unclear whether amplification magnitude, stage-order effects, and preference removability persist for substantially larger models.

We repeated the experiments on models from 0.6B to 32B parameters (Qwen3 at 0.6B, 1.7B, 4B, and 14B, and OLMo-2 at 1B, 13B, and 32B; Appendix G.7, page 61). Amplification occurs at all sizes, and the amplification scores of the 14B and 32B models are close to those of the 8B and 7B models in the main experiments (Figure 30, page 62). SDG produces weak amplification at all sizes, and the order effect that SDG amplifies less than SGD holds except for the smaller Qwen3 models. The conclusions on preference removal are unchanged. SFT removes most of an existing preference, whereas the paths that include DPO or GRPO retain most of it (Table 35, page 63). We have also limited the scope of our conclusions in Section 7 (page 9) of the revised PDF. Our experiments cover base models of up to 32B parameters, and whether the findings hold for larger models remains to be tested.

W5-b (same as Q8). In addition, the multi-seed robustness analysis is strongest for the party-based amplification experiment and is not equally extensive for education, religion, preference removal, intervention stability, or distillation transfer.

We repeated the preference-removal, intervention-stability, and distillation-transfer experiments with two additional random seeds (Appendix G.1, page 49). For preference removal, we retrained the whole path for each seed, and the overall trends are unchanged (Figure 20, page 51). For stability under interventions, the two findings of Section 5 (page 7) hold for every seed (Table 30, page 52). For distillation transfer, we retrained all student models with the new seeds, and the average preference gap is positive for every seed (Tables 31 and 32, page 53).

Q1. How broadly do the authors intend the term “value preference” to be interpreted? Would “alignment with demographic-group survey responses” be a more precise description of the primary measurement?

Please refer to our responses to W1-a.

Q2. Can the authors more systematically compare the majority-answer metric with the full-distribution metric, especially for questions where the two groups have different modal answers but very similar response distributions?

Please refer to our responses to W1-b.

Q3. Is there a way to measure the pretrained models' initial preferences using an elicitation protocol that does not depend on the post-SFT option-number format? If not, would the authors consider narrowing the causal claim to post-training reshaping observed alignment?

Please refer to our responses to W2.

Q4. Can stage-matched or update-matched controls be added to separate optimization-method effects from generic continued-training or additional-exposure effects?

Please refer to our responses to W3.

Q5. Relative to Bhatia et al. (2026), Value Drifts: Tracing Value Alignment During LLM Post-Training (TACL), which already studies how post-training algorithms and datasets affect value alignment and reports that different preference-optimization algorithms can produce different value outcomes even with fixed preference data, which findings in this paper are genuinely new? In particular, can the authors distinguish contributions arising from the controlled mixture sweep and stage-order design from broader phenomena already established in prior work?

Please refer to our responses to W4.

AuthorsOpenReview ↗

Response (3/3)

Q6-a. In the task-only distillation experiments, what exactly is being transferred: a semantic value representation, generic response tendency, source-task prediction behavior, or another correlated property?

We analyze the source of the transfer in Appendix F.3 (page 45). On the task with numeric labels (WinoGrande), the label distribution of the teacher's responses leans in the same direction as the OpinionQA majority answers of the group that the teacher is aligned with (Table 22, page 46), which shows that the option labels are one route of the transfer. On the task with alphabetic labels (MMLU), the numeric labels favoured by the distilled students match the group that their teacher is aligned with (Table 23, page 47). On the task with free-form responses (GSM8K), the semantics of the teacher's responses is closer to the group that the teacher is aligned with (Table 24, page 47). Therefore, the transfer includes both a response tendency and a semantic component.

Q6-b. Can differences in teacher output distributions on the source tasks be controlled more directly?

To control the differences between the teachers' outputs more directly, we distill only on the GSM8K questions for which the two teachers give the same final answer, and we use a random subset of the same size as a control (Appendix F.3, page 45; Table 25, page 48). On these questions, the average preference gap of the students is still positive. Therefore, value transfer cannot be avoided by controlling the distillation corpus.

Q7. Do the authors expect the amplification, ordering, and preference-removal results to hold for substantially larger models? If larger-scale experiments are infeasible, the corresponding generalization claims could be scoped more explicitly.

Please refer to our responses to W5-a.

Q8. Can at least a subset of the preference-removal, intervention-stability, and distillation-transfer experiments be repeated across independent seeds to test whether those findings are as stable as the main amplification result?

Please refer to our responses to W5-b.

Reviewer_gdtVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper studies how opinion mixtures and post-training methods shape the preferences expressed by LLMs. Using OpinionQA, the authors vary the proportions of two demographic groups’ answers and compare five training paths involving SFT, DPO, and GRPO. They also examine whether learned preferences can be changed, how they respond to interventions, and whether they transfer to students through ordinary task data.

Critical strengths
  1. The controlled mixture experiments offer a useful framework for comparing how data composition and training paths affect model behavior.

  2. The findings highlight why training history and preference transfer through distillation may matter when evaluating and reusing models.

Critical weaknesses
  1. Reversing option contents often reverses the observed preference trend. This makes it unclear how much the measurements capture semantic preferences versus label or position bias.

  2. The distribution based amplification score is compared directly with the training proportion pp, although training mixes modal answers and evaluation uses full survey distributions. This comparison needs a justified baseline to support the amplification claims.

  3. The stability analysis uses broad preference categories. Without matching final agreement scores and answer confidence, it is difficult to separate the effect of training history from differences in the models’ current behavior.

Questions for authors
  1. After randomizing option orders and evaluating agreement by answer content, do the main amplification patterns and differences between training paths remain?

  2. What score does an ideal model that exactly matches the training label distribution receive under Equation 14? How would using this baseline affect the reported amplification and path rankings?

  3. Could you report preference gaps for individual distillation configurations and repeat key experiments across student training seeds? Confidence intervals would help assess the smaller effects.

4 comments
AuthorsOpenReview ↗

Response (1/2)

We thank the reviewer for carefully reading our paper and for the valuable comments. Please refer to the revised PDF (marked in yellow and green backgrounds) for more details. Below, we respond point by point.

W1 (same as Q1). Reversing option contents often reverses the observed preference trend. This makes it unclear how much the measurements capture semantic preferences versus label or position bias.

Most OpinionQA questions are Likert questions, so the option labels indicate degree. We compute the loss on both the option label and the option content, so the models learn both. trevt_{\text{rev}} keeps the order of the labels and reverses the content, which puts the labels in conflict with the content. When we change the labels from numbers to letters at evaluation (tletter-revt_{\text{letter-rev}}, Figure 22, page 55), the agreement scores increase with the mixture proportion. This shows that the models follow the content once the labels no longer apply. When we shuffle the order of the options at evaluation (tshufflet_{\text{shuffle}}, Figure 22), the curves are close to those under treft_{\text{ref}}, which keeps the original order, and the agreement scores still increase with the mixture proportion. This shows that the models learn only a small position bias. When we determine the selected option from the content of the answer, the agreement scores also still increase with the mixture proportion (Figure 23, page 56). Therefore, the models learn to produce value-related content, and not only option labels.

In addition, we post-train and evaluate the LLMs again on corpora with randomized option order (Appendix G.3, page 57; Figure 24, page 58). Amplification remains under the original order (treft_{\text{ref}}), under randomized order (tshufflet_{\text{shuffle}}), and when the answer is determined from its content (tgent_{\text{gen}} and tnolabelt_{\text{nolabel}}). The differences between post-training paths also remain. Under each of the four templates, SFT and S2D produce the strongest amplification, and the two paths ending in GRPO (S2G and SDG) produce the weakest, with SDG producing the weakest overall, as in the main experiments.

W2 (same as Q2). The distribution based amplification score is compared directly with the training proportion, although training mixes modal answers and evaluation uses full survey distributions. This comparison needs a justified baseline to support the amplification claims.

We thank the reviewer for pointing out the error in Equation 14 (page 35). We have revised Appendix C.5 (page 34). We introduce one-hot distributions in Equation 14, so that the agreement score of the ideal answer distribution lies on the diagonal. Figure 14c (page 35) compares the amplification scores of the five paths when agreement is computed from the highest-probability answer, by top-pp sampling, and from the full answer distributions. The amplification scores of all five paths are positive under all three. The ranking of the paths does not change except for SFT. The amplification score of SFT is much lower because its answer confidence is lower. We analyze this in detail in Appendix C.5.

W3. The stability analysis uses broad preference categories. Without matching final agreement scores and answer confidence, it is difficult to separate the effect of training history from differences in the models’ current behavior.

We thank the reviewer for the suggestion. We compare checkpoints with similar final agreement scores and answer confidence, and we also run a regression that controls for these two quantities (Appendix E.4, page 39; Table 17, page 42). The table below shows the difference in flip rate between two sets of checkpoints on Llama-3.1-8B (checkpoints whose first-stage preference is neutral or differs from the final-stage preference, minus checkpoints whose first-stage and final-stage preferences are the same).

InterventionUnadjustedRegression [95% CI]Matched
Endorsement+0.050+0.044 [+0.033, +0.056]+0.044
Override+0.163+0.157 [+0.111, +0.205]+0.142
CAA+0.169+0.164 [+0.138, +0.193]+0.164
TaskVec+0.185+0.184 [+0.137, +0.235]+0.162
CFT+0.193+0.196 [+0.159, +0.239]+0.221

After the two quantities are controlled, the effect of the first-stage preference on the flip rate is almost unchanged, and none of the 95% confidence intervals of the five interventions contains zero. Therefore, the differences in flip rate cannot be explained by differences in the current behavior of the models.

AuthorsOpenReview ↗

Response (2/2)

Q1. After randomizing option orders and evaluating agreement by answer content, do the main amplification patterns and differences between training paths remain?

Please refer to our responses to W1.

Q2. What score does an ideal model that exactly matches the training label distribution receive under Equation 14? How would using this baseline affect the reported amplification and path rankings?

Please refer to our responses to W2.

Q3. Could you report preference gaps for individual distillation configurations and repeat key experiments across student training seeds? Confidence intervals would help assess the smaller effects.

We thank the reviewer for the suggestion. We retrained all student models with two additional random seeds, and the results are in Appendix G.1 (page 49). Table 31 (page 53) reports the preference gap of each of the 135 distillation configurations (5 teacher post-training paths × 3 source task datasets × 3 demographic attributes × 3 student models), as the mean and standard deviation over the three seeds. Table 32 (page 53) reports the average preference gaps by teacher post-training path, demographic attribute, student model, and source task dataset, with their 95% bootstrap confidence intervals. Averaged over the three seeds and the 135 configurations, the preference gap is +3.49 on the training-covered set ([+2.46, +4.53]) and +3.52 on the training-uncovered set ([+2.38, +4.66]). By group, the confidence intervals exclude zero except for S2D teachers and Qwen3-1.7B-Base students, and the average preference gaps of these two groups are still positive. We have also added error bars across seeds to Figure 4 (page 9) in the main text.

Thanks for the rebuttal. I’m raising my score from 2 to 3 because the new experiments address part of my main concerns. The amplification effect remains when models are trained with shuffled options and evaluated by answer content, and it also holds under the corrected baseline. The additional runs support a small average effect in the distillation experiments, but I’m still unsure how much of that effect reflects preferences about answer content and how much comes from answer label habits or other behavior learned from the teacher.

Thank you for reading our rebuttal and for recognizing the new experiments, and especially for raising the score.

Thank you also for raising the new concern. To address it, we have added Appendix F.4 (page 47), which examines how much of the preference gap comes from the option contents, how much from the option labels, and how much from other behaviors learned from the teacher.

Option labels and contents. We evaluate the students with two types of prompts. One provides only the option labels in the options, and the other provides only the option contents. We compute the preference gap under each and report the content share (Equation 17; Table 26, page 48). The results show that both the option labels and the option contents contribute, and on average the option labels contribute slightly more. For GSM8K, whose distillation corpus consists of free-form responses, the option contents contribute slightly more.

Other behaviors learned from the teacher. We examine six behaviors on the source task and six on OpinionQA (Table 27, page 49). The students mainly learn how their teachers answer the source task, and to a lesser extent how they answer the OpinionQA questions. We then adjust for the differences in these behaviors with a regression (Table 28, page 49). The preference gap changes from +3.48 to +3.47 and from +2.79 to +2.07 on the training-uncovered set. Thus, although the students learn these behaviors, the behaviors do not account for the preference gap.

If you have any further concerns, we would be glad to address them.

Reviewer_9jRrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper varies the share of opinions from two groups in a post-training corpus. It trains three base models along five SFT/DPO/GRPO paths and measures how often the model’s answer matches each group. It then studies preference removal, stability under interventions, and transfer through distillation. The authors conclude that post-training amplifies opinion imbalances and that the effect depends on the method and the order of stages.

Critical strengths

The experiments are systematic and cover a lot of ground. The path differences, especially DPO then GRPO versus GRPO then DPO, are large compared with the seed noise reported. The appendix is honest about results that cut against the main story.

Critical weaknesses

The central claim is that models learn value preferences. I’m not convinced the evidence shows that. Training and evaluation use the same fixed option order and numeric labels. In Appendix G.2 the authors reverse the order of the options, and in most cases the direction of the mixture effect reverses. They write that the results suggest the models learned preferences tied to option labels. Section 7 cites G.2 only to say amplification remains evident across templates, and the main text never mentions the reversal. If the models mainly learned which option number goes with which question, then the results in Sections 3–6 describe label behavior, not values.

Second, "amplification" is judged against the diagonal using the model’s top answer. If each question’s variants carry both groups’ answers, a model that merely matches the mixture would already give the majority answer on most questions. The paper doesn’t derive a baseline for this.

Questions for authors
  1. Does the mixture effect persist when option order is randomized in both training and evaluation, and does it follow the content of the answer?

  2. How were the two groups' answers assigned across the variants of each question, and what does a per-question majority readout give on the same metric?

4 comments

We thank the reviewer for the careful and constructive comments. Please refer to the revised PDF (marked in red and green backgrounds) for more details. Below, we respond point by point.

W1-a (same as Q1). With a fixed option order and numeric labels, the models may have learned option labels rather than values.

Most OpinionQA questions are Likert questions, so the option labels indicate degree. We compute the loss on both the option label and the option content, so the models learn both. trevt_{\text{rev}} keeps the order of the labels and reverses the content, which puts the labels in conflict with the content. When we change the labels from numbers to letters at evaluation (tletter-revt_{\text{letter-rev}}, Figure 22, page 55), the agreement scores increase with the mixture proportion. This shows that the models follow the content once the labels no longer apply. When we shuffle the order of the options at evaluation (tshufflet_{\text{shuffle}}, Figure 22), the curves are close to those under treft_{\text{ref}}, which keeps the original order, and the agreement scores still increase with the mixture proportion. This shows that the models learn only a small position bias. When we determine the selected option from the content of the answer, the agreement scores also still increase with the mixture proportion (Figure 23, page 56). Therefore, the models learn to produce value-related content, and not only option labels.

In addition (for Q1), we post-train and evaluate the LLMs again on corpora with randomized option order (Appendix G.3, page 57; Figure 24, page 58). Amplification remains under the original order (treft_{\text{ref}}), under randomized order (tshufflet_{\text{shuffle}}), and when the answer is determined from its content (tgent_{\text{gen}} and tnolabelt_{\text{nolabel}}).

W1-b. The main text does not mention the reversal reported in Appendix G.2.

In Section 7 (page 9) of the revised PDF, where we cite Appendix G.2 (page 52), we have added a brief discussion of the reversal under trevt_{\text{rev}}.

W2-a. With the top answer as the readout, a model that merely matches the mixture would already give the majority answer on most questions.

We agree with the mechanism that the reviewer describes. When the highest-probability answer is taken, a model that merely reproduces the corpus proportion would also give the majority answer on most questions. Following prior work on bias amplification (Zhao et al., 2017; Wang & Russakovsky, 2021), we take the imbalance in the corpus (the diagonal) as the reference, and an output that is more imbalanced than the corpus counts as amplification. The revised PDF states this in Section 2.6 (page 4), where we define the amplification score. Section 3 (page 4) also states that, under SFT, amplification arises mainly from selecting the highest-probability answer. Amplification also does not come only from selecting the highest-probability answer. When we use top-pp sampling or compute agreement from the full answer distributions, the amplification scores of all five paths remain positive (Appendix C.5, page 34).

W2-b (same as Q2-b). The paper doesn’t derive a baseline for this. What does a per-question majority readout give on the same metric?

We have added this baseline to Figure 14c (page 35) (dashed line, corpus majority, with an amplification score of 20.7). It is the amplification score computed directly from the corpus by taking the answer that is more frequent among the variants of each question. This is an extreme case in which the model learns and outputs only the majority answer of every question and learns none of the minority answers. Therefore, none of the paths reaches it.

Q1. Does the mixture effect persist when option order is randomized in both training and evaluation, and does it follow the content of the answer?

Please refer to our responses to W1-a.

Q2-a. How were the two groups' answers assigned across the variants of each question?

We assign the answers to individual examples rather than to whole questions. All examples are shuffled once with a fixed seed, and the two groups' answers are assigned according to the mixture proportion pp. Therefore, the proportion of g1g_1's answers in the whole corpus is exactly pp, while the share of g1g_1's answers among the 25 variants of each question varies around pp. Details are given in Table 8 (page 26) in Appendix B.3 (page 24) of the revised PDF.

Q2-b. What does a per-question majority readout give on the same metric?

Please refer to our responses to W2-b.

Dear Reviewer 9jRr,

We appreciate the time you spent on our submission. We have responded to each of your comments and revised the paper accordingly. Should anything remain unclear, we would be happy to clarify it within the discussion period.

Thanks to the authors for the detailed response. The revision addresses several of my concerns, and I appreciate the effort.

The randomized-option experiments provide good evidence that the models learn more than option labels. The authors have also acknowledged the reversal in the main text and clarified how answers are assigned across question variants. These are meaningful improvements.

Two points still leave me uncertain. First, the effect becomes weaker when labels are removed, and the shuffled-option test does not fully separate label bias from content preference. Second, while the new sampling results are helpful, the majority baseline does not resolve my concern about amplification. A well-calibrated model can produce the same majority-only answers under argmax without amplifying the learned distribution.

The paper is stronger now and merits a 3 rating. I am more persuaded by the main findings, though I would still be cautious about interpreting all of the observed effects as learned value preferences.

Thank you for your response and for raising the rating. We respond to the two remaining points below.

Point 1. We added an experiment (Figure 25, page 58). When the prompt provides only the option labels, the agreement score still increases with the mixture proportion, but there is no amplification. Amplification occurs when the option contents are also provided, even though the LLM still answers with an option label. This indicates that the LLMs bind each option label to the value preference expressed by its option content, rather than treating the label itself as a value preference.

Point 2. Even when the amplification score is computed from the full answer distribution, all paths still show amplification (Figure 14c, page 35), although it is weaker for SFT because of its lower answer confidence. This shows that the answer distributions learned by the LLMs are themselves more imbalanced than the corpus.

We thank you again for your time and constructive suggestions. If you have any further concerns, we would be glad to address them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal

We thank all reviewers for their constructive comments and for the discussion, and we are grateful that the reviewers acknowledged our rebuttal. Their comments have made the paper clearer and more robust. Below we summarize the main changes in the revised PDF:

  1. We added experiments and analyses on how the option labels and the option contents relate to the value preferences. (9jRr and gdtV)
  2. We corrected the distribution-based agreement score, and analyzed amplification under top-p sampling and under the distribution-based agreement score. (JUAv, 9jRr, and gdtV)
  3. We added analyses that control for the number of post-training stages and training updates. (JUAv)
  4. We added an analysis of the stability under interventions that controls for the final agreement score and the answer confidence. (gdtV)
  5. We added a comparison across model sizes from 0.6B to 32B parameters. (JUAv)
  6. We repeated the experiments behind all four main findings (Sections 3 to 6) with three random seeds. (JUAv and gdtV)
  7. We further analyzed what is transferred in distillation, and how the option labels, the option contents, and other learned behaviors contribute to it. (JUAv and gdtV)
  8. We clarified the definition of the term "value preference" and highlighted our contributions more explicitly. (JUAv)

All main conclusions of the paper remain unchanged, and the revisions provide additional support for them.

We thank the reviewers again for their time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.