acceptodds
Under review as a conference paper at ICLR 2027

What Is Lost in Post-Training? Default Collapse and the Loss of In-Context Steerability Across Diverse Perspectives

19 views

Abstract

AI models serving a heterogeneous population need to be able to act according to the goals or principles appropriate to different users and contexts. Post-training has been shown to narrow the range of perspectives reflected in large language model outputs, but existing evidence primarily concerns behavior under neutral prompting. We show that post-training methods can additionally degrade models' ability to be steered in-context toward perspectives they were not trained to favor. In controlled experiments, we fine-tune policies to favor one side of cultural-value disagreements and evaluate checkpoints throughout optimization. The selected target becomes increasingly dominant in ordinary use, while the ability to recognize and faithfully enact the opposing viewpoint declines. These findings reveal a tension between prioritizing a single set of values and preserving the technical capacity needed to serve diverse stakeholders. We then develop and theoretically characterize a principled alternative that optimizes reward subject to a prescribed distribution over expressed perspectives and present stance-distribution matching as one practical implementation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
alignment, in-context steerability, language model post-training, pluralistic alignment, preference collapse

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rzZvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes to study default collapse, steerability, and recognition when post-training language models towards one pole of a preference spectrum. Models lean more towards the pole across the three settings, as expected. Then the authors propose some theory and empirical methods to alleviate default collapse.

Critical strengths
  • preserving pluralism in post-training is an important research question
  • studying how things change when post-training towards one side is interesting
Critical weaknesses
  • the evaluation is limited
  • the second part could be elevated
Questions for authors
  • In line 78, ``we call this ability in-context steerability'': how is this different from steerable pluralism in [1]? I feel like authors are inclined to reiterate concepts already introduced in existing works, also cited by the authors, and brand it under a new name.

  • I actually love the idea of testing what happens under one-sided post-training. My question is: why would people do that? The only case where all these findings have empirical significance is when malicious actors intentionally steer models towards one side. Then the focus should be on how we develop defenses/resistance mechanisms so that models won't be easily swayed through this lopsided training, if the authors agree with that statement? I guess what I'm saying is that: training models towards one side, ``look the models are now one-sided!'' feels kind of expected, while the actual interesting question of defense and malicious intent goes unaddressed and is not the focus here.

  • The second part of tweaking training to alleviate default collapse has great potential, but currently the technical method, empirical results, and findings, are all thrown into the appendix, instead some theore is kept in the main paper. This perhaps should be done in the other way around: the proposed method, along with serious evals and baselines, is actually more interesting than the theory with a lot of asterisks and limits anyways.

  • What's the difference between steerability and recognition, when they are being evaluated?

  • The vast majority of results concentrate on one dataset, CDEval, a workshop paper (correct?). There is great diversity in how ``diversity'' and pluralism is measured and evaluated in the LLM space, and the current version is greatly falling short of capturing that. Any pluralism dataset is imperfect in there way, and only through a consortium of evals would the conclusions be solid. The authors could 1) discuss the limitations of CDEval, 2) look through the many good works cited here and adopt their evals and resources.

  • The introduction is weirdly long...

[1] Sorensen et al. "A roadmap to pluralistic alignment." ICML.

3 comments

Thank you for the review. Following your review, among others, the revision broadens the evaluation.

In line 78, ``we call this ability in-context steerability'': how is this different from steerable pluralism in [1]? I feel like authors are inclined to reiterate concepts already introduced in existing works, also cited by the authors, and brand it under a new name.

We do not mean to introduce a new concept here. Following Sorensen et al. (2026), we use in-context steerability to describe a model’s ability to follow a perspective supplied in its prompt. This specifies the form of steering we measure within the broader notion of steerable pluralism in [1]. The revised version no longer says “we call”.

My question is: why would people do that? The only case where all these findings have empirical significance is when malicious actors intentionally steer models towards one side.

We disagree that these findings matter only under malicious use, because training pressure is present whenever a training signal rewards one perspective more than another. Real-world signals are rarely as one-sided as in our experiments, but even diverse feedback supports some views more than others. When a reward model consistently scores one perspective higher—whether intentional or not—optimization amplifies that advantage into a stronger behavioral tendency. For example, we observe this when only 75% or 55% of the preferences used to train the reward model rank one side above the other (Figures 3 and C.9).

The pressure also does not have to be intended. In a new experiment, we train a model on an unrelated formatting task whose target answer is correlated with one perspective, and its default shifts toward that perspective while it becomes harder to steer toward the other (Section 3.3).

[...] training models towards one side, “look the models are now one-sided!” feels kind of expected

We agree that a model trained toward one perspective is expected to adopt it by default. Our main finding, which was unclear a priori, is that the more strongly a model is “aligned” with one vision of what is good, the harder it becomes to steer it toward another (even when a user explicitly asks for it).

[...] the actual interesting question of defense and malicious intent goes unaddressed and is not the focus here.

We also agree that how to prevent this loss is an important question, and the paper takes two steps toward it with recognition tuning (Section 3.4) and a modified training objective (Section 4).

The second part of tweaking training to alleviate default collapse has great potential, but currently the technical method, empirical results, and findings, are all thrown into the appendix [...]

We agree that the practical method and empirical results should receive greater prominence. We have added further SDM evaluations in Appendix E, with the main findings summarized in Section 4.2. We are currently extending this evaluation and plan to give the method and expanded results more space in the next revision’s main text. However, these experiments are unlikely to finish before the rebuttal deadline.

What's the difference between steerability and recognition, when they are being evaluated?

Both multiple-choice tasks show the same two options. Steerability asks the model to choose according to a given view. Recognition asks which option expresses it. Comparing them helps distinguish whether the model can’t or won’t comply, and our results point mainly to the former (Section 3.4).

The vast majority of results concentrate on one dataset, CDEval, a workshop paper (correct?). There is great diversity in how ``diversity'' and pluralism is measured and evaluated in the LLM space, and the current version is greatly falling short of capturing that. Any pluralism dataset is imperfect in there way, and only through a consortium of evals would the conclusions be solid. [...]

We agree that the conclusions should not rest on one dataset, and the revision now reports results on three datasets. Besides CDEval, the original submission included American Trends Panel questions where two demographic groups hold opposing majority positions (Appendix C.5). Following your suggestion, we added test sets based on DailyDilemmas (Chiu et al., ICLR 2025) to the main text (Figure 2 and Figures C.16 and C.17)

DailyDilemmas reproduces our main findings: training toward one perspective makes it dominant by default and the other harder to recognize and follow, while recognition tuning restores steerability without undoing the default. Default collapse and steerability loss now appear across three datasets that define perspectives differently.

Our experiments require distinct, reasonable answers and perspectives that recur across questions, so we can train on some questions and test on others (Appendix B.4). Most resources lack one of these properties. Even DailyDilemmas required aggressive filtering for recurring value contrasts (Appendix C.4).

I would like to thank the authors for their response. Thank you for adding DailyDilemma. The authors' new claim on main finding, the more strongly a model is “aligned” with one vision of what is good, the harder it becomes to steer it toward another, is sort of expected too? So the second part of tuning to combat this becomes really essential in my opinion. As the revised paper currently stands, the second part is still mainly represented by theorem proving (with less value? since this is a very empirical research question), and not yet furnished with the interesting methods and results. As a result, the results and analysis in the main paper still felt kind of hollow given 10 pages.

I understand the authors might need more time beyond rebuttal for that: the area chair could potentially weigh in on whether they believe the authors could complete them within the scope of camera ready.

Thank you for the follow-up!

The authors' new claim on main finding, the more strongly a model is “aligned” with one vision of what is good, the harder it becomes to steer it toward another, is sort of expected too?

Let us explain the scientific value we see in the paper in its current form. Our finding — that aligning a model toward one set of perspectives reduces its ability to follow others on request — had not been isolated or explained in previous work. Previous work found that released models are harder to steer than their pretrained versions (e.g., Sorensen et al., 2026), but the cause remained a priori unclear, because released models differ from their base models in many ways at once. By changing only how strongly a model is aligned toward one set of values, we show that alignment can directly induce strong priors that are hard to shake off even with in-context instructions.

Further, our analysis goes beyond demonstrating the general effect, and some of the findings surprised us. We analyze when and how quickly it starts (within 48 training examples), how one-sided the training signal needs to be (a 75/25 annotator population suffices for full collapse, and even 55/45 produces a clear shift), how much it generalizes (it transfers to domains never trained on and to free-form responses), how it is influenced by the model’s initial behavior (the loss follows the direction of training, even against the model’s prior lean), how it connects with the model’s ability to understand a perspective (what is lost is recognition of the requested perspective, not willingness to comply), and whether steerability and default collapse are always tied to each other (they are not).

Identifying the connection between post-training and steerability matters for pluralistic alignment. A common view is that a single model can accommodate users with different values through inference-time instructions. Recent methods that guide a fixed model toward a specified perspective build on this idea. Under this view, the choice of whose values to align to affects only the model’s default, which users can override through prompting. Our finding suggests that this choice decides much more, since aligning a model toward the values of some users can reduce its ability to serve others, even when they ask. We believe that demonstrating and analyzing it rigorously is necessary to raise awareness and enable benchmarking and counteracting in practice.

So the second part of tuning to combat this becomes really essential in my opinion. As the revised paper currently stands, the second part is still mainly represented by theorem proving (with less value? since this is a very empirical research question), and not yet furnished with the interesting methods and results. As a result, the results and analysis in the main paper still felt kind of hollow given 10 pages.

We agree that showing how to counter these effects is important. For the final version, we will continue our expansion of the empirical evaluation of SDM and move it to the main body; we refrained from putting work-in-progress results in the main body of the revised version and instead added them to the appendix. We will test the method in more settings and compare it with standard reward-only training, reporting the resulting perspective distribution and the steerability of the trained model. These experiments build on the comparison already in the paper, and we are confident that we can complete them before the camera-ready deadline.

Reviewer_rtjVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the issues of steerability loss and default collapse arising from different post-training methods (SFT, DPO, and RLHF) under a controlled one-sided post-training setting. The authors conduct experiments across three models to demonstrate the presence of these issues under the proposed training setup. To address steerability loss, they introduce Recognition Tuning to recover the model's ability to follow alternative perspectives. Additionally, they propose a modification to the standard KL-regularized RLHF objective by incorporating a stance-distribution constraint, aiming to mitigate default collapse by controlling the distribution of expressed perspectives.

Critical strengths
  • The issues of steerability loss and default collapse raised in this paper are worth discussing.
Critical weaknesses
  1. The discussion of contributions in the Introduction is too long and does not clearly present the main points, making it difficult to understand what the paper proposes and which problems it addresses. This significantly affects readability.
  2. The paper only uses CDEval and ATP as dataset. Although Appendix B.1 explains why other datasets are unsuitable for the experiments, the empirical evidence remains limited. The authors could consider adapting other datasets for additional generalization evaluations.
  3. The authors perform one-sided post-training on three models that have already undergone post-training and claim that post-training leads to steerability loss and default collapse. Although the authors state that "This setup is not meant to characterize deployed LLMs," I still think this experimental design unconvincing. Repeatedly strengthening a single perspective would intuitively lead to steerability loss and default collapse. This is similar to catastrophic forgetting, where the model loses its original capabilities.
  4. The main paper lacks ablation studies, as the authors put all of them in the appendix. Ablation studies are important, and their key findings should be presented in the main paper, with the appendix serving as a supplement.
Questions for authors
  1. Why is SFT + DPO only applied to GPT-4.1 mini, SFT + RLHF only to Llama-3.1-8B-Instruct, and SFT the only method applied to Gemma-4-12B-it? This introduces two independent variables into the comparison, namely differences between models and differences between post-training methods.
  2. As mentioned in Weakness 3, steerability loss and default collapse do not seem to be common issues in standard post-training. For example, at the 0% training point in Figures 1 and 2, the original post-trained models do not exhibit these issues. Could the authors provide examples or experimental evidence showing that these issues can also occur under standard post-training settings, to demonstrate that the problems identified in this paper can arise in practice?
3 comments

We thank the reviewer for their comments and questions!

The discussion of contributions in the Introduction is too long and does not clearly present the main points

We agree that the Introduction could present our contributions more clearly. Based on your suggestion, we have shortened and reorganized it.

The paper only uses CDEval and ATP as dataset.

We agree that testing our findings on additional datasets would strengthen the empirical evidence. Following the reviewer’s suggestion, we added experiments based on a third dataset, DailyDilemmas (Chiu et al., 2025), which now appear in the main text (Figure 2, Appendix C.4).

The results reproduce our main findings. Training toward one perspective makes it increasingly dominant by default and weakens the model’s ability to recognize and follow the other, and recognition tuning restores steerability while the trained default remains.

The authors perform one-sided post-training on three models that have already undergone post-training and claim that post-training leads to steerability loss and default collapse. Although the authors state that "This setup is not meant to characterize deployed LLMs," I still think this experimental design unconvincing. Repeatedly strengthening a single perspective would intuitively lead to steerability loss and default collapse [...]

Let us clarify what our experimental design is intended to establish and why its implications extend beyond one-sided training.

Our goal is to understand how aligning a model toward one perspective affects its ability to follow others. We start from post-trained models that can initially follow both perspectives, then introduce a training signal that consistently rewards one of them and vary the strength of optimization. This controlled experiment allows us to identify the causal effect of increasing optimization pressure on steerability. As expected, the model's default shifts toward the rewarded perspective. But our main finding goes beyond this expected shift: the more strongly a model is “aligned” with one vision of what is good, the harder it becomes to steer it toward another (even when a user explicitly asks for it).

Real-world training signals are rarely this one-sided, but they do not need to be for default collapse and steerability loss to appear. Even when feedback contains diverse views, some perspectives receive more positive feedback than others. When a reward model consistently assigns higher scores to one perspective—whether intentional or not—optimization amplifies that advantage into a stronger behavioral tendency. For example, we observe this even when we introduce disagreement into the feedback used to train reward models, so that only 75% or 55% of preferences rank one side above the other (Figures 3 and C.9).

The training pressure also does not have to be intended. In a new experiment (Appendix C.1), we train a model on an unrelated formatting task whose target answer is correlated with one perspective, and its default shifts toward that perspective while it becomes harder to steer toward the other. In practice, other signals may reinforce or counteract this pressure, so how much steerability is lost depends on the rest of training.

The main paper lacks ablation studies, as the authors put all of them in the appendix.

Following your suggestion, we moved key ablations into the main text (Sections 3.2 & 3.3).

Why is SFT + DPO only applied to GPT-4.1 mini, SFT + RLHF only to Llama-3.1-8B-Instruct, and SFT the only method applied to Gemma-4-12B-it?

The mix of models and methods does not affect our conclusions, because we do not compare one model or method against another. Each configuration is measured relative to its starting checkpoint as training pressure increases, and the same qualitative effects recur in all of them. Instead of an exhaustive grid, we chose combinations that cover different algorithms (SFT, RLHF/DPO) and different notions of pressure (weaker regularization, more one-sided feedback).

[...] steerability loss and default collapse do not seem to be common issues in standard post-training. For example, at the 0% training point in Figures 1 and 2, the original post-trained models do not exhibit these issues. [...]

These effects are not confined to our experimental setting. Both have been measured in released models, although in a milder form. Compared with their pretrained versions, released models express fewer views by default (Sorensen et al., 2024b; Murthy et al., 2025; Ryan et al., 2024) and are less steerable (Sorensen et al., 2026). These studies document the effects in practice, while our experiments help explain and understand them.

The 0% point in Figure 1 does not show that the starting models have no default. Each curve averages runs trained toward opposite poles, so it starts at 50% by construction. Shown per pole, the starting model already leans clearly toward one side in five of the six dimensions (Figure C.2).

I apologize for my delayed response.

Thank you to the authors for their detailed response. Given that the authors may have limited time to respond to this follow-up, I would like to focus directly on my primary concern.

My main concern remains whether the current post-training experimental design is well justified.

When a reward model consistently assigns higher scores to one perspective—whether intentional or not—optimization amplifies that advantage into a stronger behavioral tendency.

While I acknowledge that unintended biases in reward models can also lead to perspective shifts, this observation alone does not sufficiently justify the use of one-sided post-training in the experimental setup.

The authors also mention that “The 0% point in Figure 1 does not show that the starting models have no default.” This suggests that the original models may already exhibit some degree of steerability issues.

I would prefer to see an analysis of these issues in the original models and methods that directly address them. However, the current experimental design introduces an additional post-training stage that makes the steerability loss issue more severe, and then proposes a method to fix them.

This makes it difficult to determine whether the proposed method is truly effective, or whether the results simply show that the method can fix problems introduced by the one-sided post-training itself. Therefore, I am still not convinced that the current experimental design provides sufficient evidence to support the effectiveness of the proposed method.

Thank you for the follow-up! We would like to clarify that we view the paper’s main goal as understanding the loss of steerability caused by post-training. Below, we describe the scope of our analysis of recognition tuning and the role of our controlled experimental design.

[...] This makes it difficult to determine whether the proposed method is truly effective, or whether the results simply show that the method can fix problems introduced by the one-sided post-training itself.

Experimental Setup & Recognition Tuning

We agree that our experiments do not establish the effectiveness of applying recognition tuning directly to the original released checkpoints. We view recognition tuning as a secondary contribution within our analysis, intended primarily as a diagnostic experiment: we find that it restores much of the induced steerability loss while largely preserving the trained default. This shows that default collapse and steerability loss do not have to coincide, that recovering the ability to follow alternative perspectives need not undo the learned preference, and it provides evidence linking recognition and steerability. Still, the survey experiments show that even our full training sequence can improve steerability beyond the original released checkpoint: on politics survey questions, Llama-3.1-8B-Instruct starts at 70%, falls to 37% after one-sided SFT, and reaches 90% after recognition tuning (Appendix C.5).

If “the proposed method” refers to SDM, its training starts from the released checkpoints, without first applying the collapse-inducing training from Section 3.

While I acknowledge that unintended biases in reward models can also lead to perspective shifts, this observation alone does not sufficiently justify the use of one-sided post-training in the experimental setup.

Experimental Setup & Analysis of Steerability Loss and Default Collapse

We use unanimous preferences as a controlled setting to isolate the effect of directional training pressure. Within each configuration, we vary how strongly the model is trained toward one perspective while holding the other settings fixed. The additional training stage is therefore the experiment itself: it lets us trace how defaults and steerability change as optimization proceeds. We note that isolating this factor offers a possible explanation for why released models already appear less steerable than their pre-trained versions (Sorensen et al., 2026), which cannot be determined by observing those models alone.

At the same time, we would like to emphasize that the scope of our observation regarding default collapse and steerability loss goes beyond unanimous one-sided post-training. For instance, in Figure 3, we present a simulated setup with mixed preferences, as appears in a standard annotator dataset. For intuition, consider two answers expressing individualist and collectivist perspectives: in our setting, 75% of annotators prefer the individualist answer, while the remaining 25% prefer the collectivist answer. Default collapse and steerability loss also occur under these mixed signals. We also show that they occur when the perspective enters only as an incidental correlation of an unrelated formatting objective (Appendix C.1). In sum, we use the simplified setup to isolate the effect and to understand it better, and then establish that our findings hold beyond the case of a unanimous one-sided training signal.

[...] “The 0% point in Figure 1 does not show that the starting models have no default.” This suggests that the original models may already exhibit some degree of steerability issues.

Our remark concerned default preferences, which we measure separately from steerability. A model can favor one perspective by default while still following either perspective when requested: On CDEval and DailyDilemmas, the initial models do so on most questions (Figures 1&2).

The default curve shows how often a model chooses the value it is trained toward. It averages over copies trained toward opposite poles, making the initial score 50% by construction, regardless of the starting preference. For example, suppose the starting model prefers work-life balance over career success on 70% of the questions. The copy trained toward work-life balance starts at 70%, the copy trained toward career success starts at 30%, and the average is 50%.

Training can also reduce steerability toward the perspective the model initially preferred. Our design controls for the starting default in the sense that we test both training directions from the same starting model. After training, each copy chooses its trained value more often by default and becomes harder to steer toward the other. This includes the copy trained toward career success, which becomes harder to steer toward work-life balance, the value the starting model preferred. In other words, the loss of steerability follows the direction of training, not the model’s initial lean.

Reviewer_NbSzOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how post-training toward a single perspective affects LLM behavior. The authors distinguish between default collapse, where models increasingly favor one perspective under neutral prompts, and steerability loss, where models become less able to follow alternative perspectives when explicitly instructed.

Through controlled experiments with SFT, RLHF, and DPO, the authors show that stronger training pressure leads to default collapse and reduced recognition and steerability. They also find that recognition tuning can restore much of the lost steerability while largely preserving the learned default preference. Finally, the paper introduces a constrained reward optimization framework and a practical method, Stance-Distribution Matching (SDM), to control the distribution of expressed perspectives.

Critical strengths
  1. The distinction between default collapse and steerability loss provides a useful perspective on the effects of post-training. The authors conduct controlled experiments across different models and training methods, examining how these behaviors change with training pressure. I find the experimental setup well designed and the observations valuable.

  2. The finding that recognition and steerability often decline together is interesting. More importantly, recognition tuning restores much of the lost steerability while largely preserving the trained default preference. Section 4 also provides a clear theoretical characterization of reward optimization with stance distribution constraints, adding to the paper's technical contribution.

Critical weaknesses

1.The difference between multiple-choice and free-form steerability deserves more careful interpretation.

The paper reports a substantial loss of steerability in multiple-choice tasks, dropping to 0.9% after strong SFT. However, Appendix G shows a milder decline in free-form generation, where average steerability decreases from 6.28 to 5.85. The authors discuss this difference and provide additional checks on A/B option probabilities, which I appreciate. Still, it remains unclear how well the multiple-choice results reflect steerability loss in more natural interactions. Further free-form evaluation would make the conclusions more convincing.

2.The empirical evaluation of SDM is still limited.

The theoretical framework in Section 4.1 considers per-prompt distribution constraints, while the practical SDM implementation matches distributions pooled across prompts. Although this difference is acknowledged, the experiments cover only two settings and lack a direct reward-only baseline under the same training budget. It is also unclear whether SDM preserves in-context steerability. Additionally, Appendix H reports a decrease in valid response rates, particularly from 94% to 79% on IDV, while stance distributions are computed only over valid responses. These issues make it difficult to fully assess the practical effectiveness of SDM.

Questions for authors
  1. Given the substantial difference between multiple-choice and free-form steerability results, could the authors further clarify how this affects the main conclusions? Would additional free-form evaluations with multiple samples help validate the observed steerability loss?

  2. Could the authors compare SDM with a reward-only PPO baseline under the same training budget, reporting reward performance, stance distributions, and steerability? Also, how does the decrease in valid response rates affect the distribution matching results?

2 comments

Thank you for the review. We address your questions below.

The difference between multiple-choice and free-form steerability deserves more careful interpretation. The paper reports a substantial loss of steerability in multiple-choice tasks, dropping to 0.9% after strong SFT. However, Appendix G shows a milder decline in free-form generation, where average steerability decreases from 6.28 to 5.85. The authors discuss this difference and provide additional checks on A/B option probabilities, which I appreciate. Still, it remains unclear how well the multiple-choice results reflect steerability loss in more natural interactions. Further free-form evaluation would make the conclusions more convincing. Q1. Given the substantial difference between multiple-choice and free-form steerability results, could the authors further clarify how this affects the main conclusions?

Both formats support our main conclusion that the more strongly a model is “aligned” with one vision of what is good, the harder it becomes to steer it toward another. The difference you note affects how much steerability is lost, which is indeed less in free-form generation.

In free-form answers, continued training still makes models follow the opposed perspective less closely (Appendix C.2.1, Figure C.3 & C.8). The average you cite also included the trained perspective and, as a result, understated this drop; we revised the figures to report steerability to the opposed perspective for a cleaner evaluation. As in multiple choice, recognition tuning reverses it (Figure C.14).

We agree that the near-complete failure is specific to multiple choice, possibly because it is the only format we train on. Models that almost never select the opposing perspective on multiple-choice questions can still express that view in their own words, which Section 3.2 now states explicitly.

Would additional free-form evaluations with multiple samples help validate the observed steerability loss?

We agree that an evaluation with multiple samples would strengthen this result. In a preliminary check on a subset of questions, repeated responses received the same judge scores.

The theoretical framework in Section 4.1 considers per-prompt distribution constraints, while the practical SDM implementation matches distributions pooled across prompts.

Our theorems cover the per-prompt constraint and SDM trains on a pooled relaxation of it (Appendices D.6 and E.1), so the evidence for SDM as implemented is empirical, and we have extended it as described below. For a shared target, any model that satisfies the per-prompt constraints also satisfies the pooled constraint, but the converse need not hold. We average across prompts so that the model can maintain a consistent stance across repeated answers to the same question, and because estimating a separate distribution for every prompt would require many more samples.

Q2. Could the authors compare SDM with a reward-only PPO baseline under the same training budget, reporting reward performance, stance distributions, and steerability?

Following your suggestion, we added this comparison (Section 4.2, Appendix E.2.1), and SDM remains much more steerable than reward-only PPO on both values of interest while both learn the rewarded perspective to a similar degree. The two runs use the same initial policy, reward model, prompts, and training budget. This is a single setting, and we are extending the comparison to further targets and value dimensions for the final version.

Also, how does the decrease in valid response rates affect the distribution matching results?

We agree that the drop in validity limits what the distribution-matching results show, because the stance distribution is computed over valid responses only. In the experiment where validity dropped most, the improvement comes mainly from fewer neutral answers, and part of it may reflect answers that avoid a position being judged invalid instead of neutral. The other original experiment depends less on this, since there the answers move toward both ends of the scale, as the target prescribes.

SDM penalizes invalid responses during training, but this did not fully prevent the decline. Nevertheless, in the new comparison, validity is substantially higher under SDM than reward-only PPO (76.2% versus 41.1%).

Thank you for the detailed response and additional experiments. The clarifications and new results have addressed some of my initial concerns. I appreciate the authors' efforts and will maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

With the author-reviewer discussion phase ending, we want to thank the reviewers again for their feedback. Below, we summarize how we have addressed the three recurring concerns and revised the paper.

  1. The motivation and significance of one-sided training need to be clearer.

    Reviewers rtjV and rzZv questioned the practical relevance of our one-sided training setup and what our findings establish beyond expected effects of training toward one perspective.

    Our main finding is that as a model is more strongly "aligned" with one vision of what is good, it becomes steadily less able to adopt another on request. To isolate this effect, we use a training signal that rewards one perspective and vary the strength of optimization, which lets us identify its causal effect.

    We do not believe these effects are confined to our setting. Real-world signals are rarely as one-sided as in our experiments, but even feedback from a diverse annotator population will support some views more than others, and optimization amplifies any such advantage, whether intentional or not, into a stronger behavioral tendency. For example, we observe the same effects, more gradually, when only 75% or 55% of the preferences used to train the reward model rank one side above the other (see Figure 3, which we moved to the main body, and Figure C.9).

  2. The empirical evaluation is too narrow.

    Reviewers rtjV and rzZv noted a heavy reliance on CDEval and asked for additional datasets.

    The revised version reports our experiments on three datasets. All three give qualitatively the same result: CDEval (Section 3), American Trends Panel survey questions where two demographic groups hold opposing majority positions (Appendix C.3), and DailyDilemmas (Chiu et al., 2025).

    We added DailyDilemmas during the rebuttal phase and report our findings in Figure 2 in Section 3 and Appendix C.4. Across all three datasets, we find that training toward one perspective makes it dominant by default and makes the model harder to steer toward the other, and recognition tuning restores steerability without undoing the default.

    As we discuss in Appendix B.4, our experiments need distinct, reasonable perspectives that recur across questions, so that we can train on some questions and test on others (see Appendix B.4). Existing resources usually meet these requirements only after substantial processing, such as the filtering we applied to the survey questions and to DailyDilemmas (Appendices C.3 and C.4).

  3. The introduction is too long.

    Reviewers rtjV and rzZv found that the Introduction did not clearly present our contributions.

    We agree, and we shortened and reorganized it following the reviewers' suggestion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.