- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes to study default collapse, steerability, and recognition when post-training language models towards one pole of a preference spectrum. Models lean more towards the pole across the three settings, as expected. Then the authors propose some theory and empirical methods to alleviate default collapse.
Critical strengths
- preserving pluralism in post-training is an important research question
- studying how things change when post-training towards one side is interesting
Critical weaknesses
- the evaluation is limited
- the second part could be elevated
Questions for authors
-
In line 78, ``we call this ability in-context steerability'': how is this different from steerable pluralism in [1]? I feel like authors are inclined to reiterate concepts already introduced in existing works, also cited by the authors, and brand it under a new name.
-
I actually love the idea of testing what happens under one-sided post-training. My question is: why would people do that? The only case where all these findings have empirical significance is when malicious actors intentionally steer models towards one side. Then the focus should be on how we develop defenses/resistance mechanisms so that models won't be easily swayed through this lopsided training, if the authors agree with that statement? I guess what I'm saying is that: training models towards one side, ``look the models are now one-sided!'' feels kind of expected, while the actual interesting question of defense and malicious intent goes unaddressed and is not the focus here.
-
The second part of tweaking training to alleviate default collapse has great potential, but currently the technical method, empirical results, and findings, are all thrown into the appendix, instead some theore is kept in the main paper. This perhaps should be done in the other way around: the proposed method, along with serious evals and baselines, is actually more interesting than the theory with a lot of asterisks and limits anyways.
-
What's the difference between steerability and recognition, when they are being evaluated?
-
The vast majority of results concentrate on one dataset, CDEval, a workshop paper (correct?). There is great diversity in how ``diversity'' and pluralism is measured and evaluated in the LLM space, and the current version is greatly falling short of capturing that. Any pluralism dataset is imperfect in there way, and only through a consortium of evals would the conclusions be solid. The authors could 1) discuss the limitations of CDEval, 2) look through the many good works cited here and adopt their evals and resources.
-
The introduction is weirdly long...
[1] Sorensen et al. "A roadmap to pluralistic alignment." ICML.
3 comments
I would like to thank the authors for their response. Thank you for adding DailyDilemma. The authors' new claim on main finding, the more strongly a model is “aligned” with one vision of what is good, the harder it becomes to steer it toward another, is sort of expected too? So the second part of tuning to combat this becomes really essential in my opinion. As the revised paper currently stands, the second part is still mainly represented by theorem proving (with less value? since this is a very empirical research question), and not yet furnished with the interesting methods and results. As a result, the results and analysis in the main paper still felt kind of hollow given 10 pages.
I understand the authors might need more time beyond rebuttal for that: the area chair could potentially weigh in on whether they believe the authors could complete them within the scope of camera ready.
Thank you for the follow-up!
The authors' new claim on main finding,
the more strongly a model is “aligned” with one vision of what is good, the harder it becomes to steer it toward another, is sort of expected too?
Let us explain the scientific value we see in the paper in its current form. Our finding — that aligning a model toward one set of perspectives reduces its ability to follow others on request — had not been isolated or explained in previous work. Previous work found that released models are harder to steer than their pretrained versions (e.g., Sorensen et al., 2026), but the cause remained a priori unclear, because released models differ from their base models in many ways at once. By changing only how strongly a model is aligned toward one set of values, we show that alignment can directly induce strong priors that are hard to shake off even with in-context instructions.
Further, our analysis goes beyond demonstrating the general effect, and some of the findings surprised us. We analyze when and how quickly it starts (within 48 training examples), how one-sided the training signal needs to be (a 75/25 annotator population suffices for full collapse, and even 55/45 produces a clear shift), how much it generalizes (it transfers to domains never trained on and to free-form responses), how it is influenced by the model’s initial behavior (the loss follows the direction of training, even against the model’s prior lean), how it connects with the model’s ability to understand a perspective (what is lost is recognition of the requested perspective, not willingness to comply), and whether steerability and default collapse are always tied to each other (they are not).
Identifying the connection between post-training and steerability matters for pluralistic alignment. A common view is that a single model can accommodate users with different values through inference-time instructions. Recent methods that guide a fixed model toward a specified perspective build on this idea. Under this view, the choice of whose values to align to affects only the model’s default, which users can override through prompting. Our finding suggests that this choice decides much more, since aligning a model toward the values of some users can reduce its ability to serve others, even when they ask. We believe that demonstrating and analyzing it rigorously is necessary to raise awareness and enable benchmarking and counteracting in practice.
So the second part of tuning to combat this becomes really essential in my opinion. As the revised paper currently stands, the second part is still mainly represented by theorem proving (with less value? since this is a very empirical research question), and not yet furnished with the interesting methods and results. As a result, the results and analysis in the main paper still felt kind of hollow given 10 pages.
We agree that showing how to counter these effects is important. For the final version, we will continue our expansion of the empirical evaluation of SDM and move it to the main body; we refrained from putting work-in-progress results in the main body of the revised version and instead added them to the appendix. We will test the method in more settings and compare it with standard reward-only training, reporting the resulting perspective distribution and the steerability of the trained model. These experiments build on the comparison already in the paper, and we are confident that we can complete them before the camera-ready deadline.
Thank you for the review. Following your review, among others, the revision broadens the evaluation.
We do not mean to introduce a new concept here. Following Sorensen et al. (2026), we use in-context steerability to describe a model’s ability to follow a perspective supplied in its prompt. This specifies the form of steering we measure within the broader notion of steerable pluralism in [1]. The revised version no longer says “we call”.
We disagree that these findings matter only under malicious use, because training pressure is present whenever a training signal rewards one perspective more than another. Real-world signals are rarely as one-sided as in our experiments, but even diverse feedback supports some views more than others. When a reward model consistently scores one perspective higher—whether intentional or not—optimization amplifies that advantage into a stronger behavioral tendency. For example, we observe this when only 75% or 55% of the preferences used to train the reward model rank one side above the other (Figures 3 and C.9).
The pressure also does not have to be intended. In a new experiment, we train a model on an unrelated formatting task whose target answer is correlated with one perspective, and its default shifts toward that perspective while it becomes harder to steer toward the other (Section 3.3).
We agree that a model trained toward one perspective is expected to adopt it by default. Our main finding, which was unclear a priori, is that the more strongly a model is “aligned” with one vision of what is good, the harder it becomes to steer it toward another (even when a user explicitly asks for it).
We also agree that how to prevent this loss is an important question, and the paper takes two steps toward it with recognition tuning (Section 3.4) and a modified training objective (Section 4).
We agree that the practical method and empirical results should receive greater prominence. We have added further SDM evaluations in Appendix E, with the main findings summarized in Section 4.2. We are currently extending this evaluation and plan to give the method and expanded results more space in the next revision’s main text. However, these experiments are unlikely to finish before the rebuttal deadline.
Both multiple-choice tasks show the same two options. Steerability asks the model to choose according to a given view. Recognition asks which option expresses it. Comparing them helps distinguish whether the model can’t or won’t comply, and our results point mainly to the former (Section 3.4).
We agree that the conclusions should not rest on one dataset, and the revision now reports results on three datasets. Besides CDEval, the original submission included American Trends Panel questions where two demographic groups hold opposing majority positions (Appendix C.5). Following your suggestion, we added test sets based on DailyDilemmas (Chiu et al., ICLR 2025) to the main text (Figure 2 and Figures C.16 and C.17)
DailyDilemmas reproduces our main findings: training toward one perspective makes it dominant by default and the other harder to recognize and follow, while recognition tuning restores steerability without undoing the default. Default collapse and steerability loss now appear across three datasets that define perspectives differently.
Our experiments require distinct, reasonable answers and perspectives that recur across questions, so we can train on some questions and test on others (Appendix B.4). Most resources lack one of these properties. Even DailyDilemmas required aggressive filtering for recurring value contrasts (Appendix C.4).