acceptodds
Under review as a conference paper at ICLR 2027

Post-Training Reshapes Truth–Sycophancy Geometry

Abstract

Language models can favor agreement with users over truthful answers, a behavior known as sycophancy. We investigate how post-training changes the internal relationship between truth and sycophancy directions, and whether this relationship affects model responses. We introduce the Geometric Fragility Signature (GFS), which measures how the truth and sycophancy directions are related in activation space. Across 22 matched base and instruction-tuned model pairs from nine families, we find that post-training makes the truth and sycophancy directions more negatively related. We find a similar relationship across two different factual datasets. We then change the angle between the truth and sycophancy directions to test whether this relationship can influence model responses. Under an instruction to prioritize agreement even when it requires flexibility with facts, this changes how often models abandon previously correct answers in seven of eight models. However, other internal directions can sometimes produce similar changes. Finally, we use the same directions to try to keep models truthful under pressure to agree. This helps some models but not others, and can even make models abandon more correct answers. Overall, our results show that post-training reshapes the internal relationship between truth and sycophancy directions, while interventions on these directions can change model responses but do not consistently reduce sycophancy across models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
sycophancy, mechanistic interpretability, activation steering, representation geometry, post-training, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8Y7tOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether post-training systematically changes the internal relationship between representations associated with truthfulness and sycophancy. It constructs a truth direction from the activation difference between false and true statements, and a sycophancy direction from the activation difference between agreeing with a user’s false claim and rejecting it. The paper then measures the cosine similarity between these directions and averages them into the Geometric Fragility Signature (GFS). A cosine near 0 means the directions are roughly orthogonal, a negative cosine means they are more opposed, and a positive cosine means they are more aligned.

The authors compare matched base models and instruction-tuned models and find that post-training generally shifts the geometry in the negative direction: the truth and sycophancy directions become more opposed. This is the paper's main empirical finding and is relatively consistent across model families.

They then ask whether this geometry can be manipulated and whether doing so changes model behavior. They construct aggregate truth and sycophancy directions and rotate the sycophancy direction toward different target angles relative to the truth direction. This changes the rate at which models abandon answers they previously answered correctly when given an instruction to prioritize agreement with the user.

Finally, they investigate whether this manipulation can be used for sycophancy mitigation. They directly steer models away from the sycophancy direction or toward the truth direction. They observe substantial effects in some models, but the effects are inconsistent across models and can come with capability costs.

Importantly, the authors themselves acknowledge two major limitations: the behavioral intervention is not uniquely attributable to the proposed sycophancy direction, and the behavioral setup does not cleanly isolate sycophancy from general instruction following/compliance.

Critical strengths
  • Broad and systematic empirical evaluation. The paper evaluates the proposed geometric signature across 22 matched base/instruction-tuned models spanning nine model families and different model scales, and further examines the finding across multiple factual datasets. This breadth makes the central observation (post-training is associated with a shift toward more negative truth–sycophancy geometric alignment) more convincing than a result demonstrated on only one or two models. I also appreciate that the authors are appropriately cautious in interpreting the cross-dataset result and explicitly acknowledge its limitations rather than overclaiming generality.

  • The target-angle intervention is a creative and informative probing method. Constructing a modified sycophancy vector with a prescribed angle relative to the truth vector while preserving its norm provides a systematic way to probe whether the geometric relationship itself can influence model behavior.

Critical weaknesses
  • The proposed sycophancy direction is not cleanly disentangled from other factors. The direction is constructed from the difference between agreeing and disagreeing with false user claims. Consequently, it may encode generic agreement, response polarity, social compliance, semantic differences, and/or the truth value of the underlying claim, rather than sycophancy specifically. This is consequential because the paper's central geometric interpretation depends on treating this contrast as a sycophancy representation.
Questions for authors
  • How large is the observed GFS shift in representation space, and what does a change from approximately −0.01 to −0.03 imply geometrically? The reported average change is only about 0.02 cosine units. Is this shift large relative to the variance across examples, model families, or unrelated contrast directions? Can the authors quantify the corresponding angular change and demonstrate that it is sufficiently large to explain the observed behavioral effects?
  • Why are different subsets of models used for the different steering and control experiments? Geometry-derived steering is evaluated on eight models, the random control on five, and all three control directions on only three models. How were these models selected, and were the selection criteria determined before observing the results? Could computational constraints or model-specific properties have influenced which models received the more complete set of controls?
  • I have a concern about the sycophancy direction definition. How do the authors disentangle sycophancy from generic agreement, response polarity, and the truth value of the underlying statement? Since the sycophancy direction is constructed only from false statements, could the observed geometry instead reflect an interaction between agreement and truth value?
  • How do the authors establish that the behavioral effects of target-angle steering are specific to the proposed truth–sycophancy geometry? In several experiments, structured control directions can produce comparable behavioral changes. If multiple unrelated directions can alter collapse rates, what evidence supports interpreting the effect as manipulation of a sycophancy-specific representation rather than a generic effect of activation perturbation?
Reviewer_nsf4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper argues that post-training reshapes the truth-sycophancy geometry of language models. The authors define the geometric fragility signature (GFS), which is the average cosine similarity between truth and sycophancy directions. They show that across 22 base-chat model pairs, GFS at the final layer decreases after post-training. They steer the activations in the middle layer and change whether the model agrees with the users' answers, while random controls often produce similar results. Finally, they use steering as mitigation method, but the results are mixed for different models.

Critical strengths
  • The paper is clear about several limitations, including negative and ambiguous results. The authors also include several robustness check, including bootstrap tests, cross-item tests, etc.
  • The breadth of empirical experiments is strong, covering 22 pairs of models.
Critical weaknesses
  • Significant novelty issue: the headline result is the same as Ying et al. (2026), which came out more than 7 months before the deadline and is not cited! They also show that post-training reduces the alignment between truth directions and sycophancy directions on several base-chat model pairs. So this paper should not claim "post-training reshapes truth-sycophancy geometry" as a novel result and use it as the title. The authors need to overhaul the title/abstract/intro/related work so the distinctions from Ying et al. (2026), and the contributions are clear.
  • The sycophancy dataset construction is concerning. "Agree" is always associated with sycophantic labels, while "disagree" is always non-sycophantic. This creates very simple lexical confounds that the probe direction likely picks up on.
  • Why is the geometry evaluated at the final layer? Truth and sycophancy geometry is clearest in the middle layers, while the final layer normally shows output geometry. The layer where the geometry is measured is also different from the layer that is steered on. This is a very confusing choice, and the authors should elaborate.
  • Why use GPS? It is more straightforward and standard to compute cosine similarity of the diff-mean directions for truth vs. sycophancy contrasts. Also, the proposed aggregate direction steering is diff-in-means steering, and the authors should at least cite Marks & Tegmark 2023.
  • A sample size of only 71 statements is too small. The authors can consider including other large-scale paired true/false statement datasets from Ying et al. 2026, Marks & Tegmark 2024, Azaria & Mitchell, 2023.
  • Mitigation results are weak. Steering reduces sycophantic agreements in two of the three models, and prompting outperforms it in all settings.
  • Figure 2 wrapped figure breaks the format.

Reference:

  • Ying, Z. J., Ravfogel, S., Kriegeskorte, N., & Hase, P. (2026). The truthfulness spectrum hypothesis. arXiv preprint arXiv:2602.20273.
  • Azaria, A., & Mitchell, T. (2023). The internal state of an LLM knows when it’s lying. In Findings of the Association for Computational Linguistics: EMNLP 2023 (pp. 967-976).
  • Marks, S., & Tegmark, M. (2023). The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824.
Questions for authors
  • How do your results relate to Ying et al. (2026)?
  • Does the geometric shift hold at intermediate layers?
  • Why did six model pairs fail the reproduction check?
Reviewer_rYc4OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how post-training changes the relationship between truth-related and sycophancy-related representations. It introduces GFS, an average cosine similarity between matched activation differences, and compares 22 public base–instruction model pairs. The main finding is that GFS generally becomes more negative after post-training. The authors also test activation interventions constructed from these directions. These interventions affect responses in several models, but the effects are not consistently specific to the sycophancy direction, and mitigation does not work reliably across models.

Critical strengths
  1. The comparison across 22 model pairs is useful. The family-aware analysis, resampling checks, and non-sycophancy controls make the descriptive finding more convincing.

  2. I appreciate the inclusion of structured intervention controls and capability evaluations. These help show where the proposed interpretation holds and where it becomes less clear.

  3. The paper is generally clear and candid about its limitations. The inconsistent mitigation results, including cases where a simple prompt works better, are worth reporting.

Critical weaknesses
  1. My main hesitation is what we learn from GFS beyond the measured shift itself. The paper provides evidence that this statistic changes after post-training, but its broader significance is less clear. GFS does not reliably rank models by collapse, and the intervention results do not clearly establish its explanatory role. A descriptive result can certainly be valuable without predicting behavior. Here, though, I would like a clearer account of what the observation tells us about truth and sycophancy that existing representation and steering studies do not already tell us.

  2. The sycophancy direction compares agreement and disagreement with a false statement. Without the corresponding contrast on true statements, it is hard to separate agreement from factual correctness. Similarly, the main behavioral experiment explicitly asks the model to prioritize agreement and permits flexibility with facts, without supplying a false user belief. The resulting collapse may therefore reflect instruction following as well as sycophancy. The authors acknowledge this, but it limits how much the behavioral results support the interpretation of the geometry.

  3. The connection between the geometry measurement and the intervention remains ambiguous. GFS is measured using item-level contrasts at the final layer, while steering uses aggregate directions at an intermediate layer. Appendix B.5 helps compare itemwise and aggregate summaries, but does not fully bridge these settings. Also, the angle-based intervention adds an activation displacement; it does not directly change the measured GFS. Since other structured directions sometimes produce comparable effects, I read these results as evidence that the perturbations can affect responses, rather than strong evidence that the measured truth–sycophancy relationship explains those changes.

Questions for authors
  1. What should GFS be taken to measure? Is the intended contribution primarily a descriptive observation about these contrasts, or should GFS tell us something about behavioral fragility? Making this distinction more explicit would help.

  2. What evidence most directly connects the final-layer GFS shift to the intermediate-layer steering results? What do we learn from the target-angle construction beyond what could be learned from additive steering using combinations of the same two directions?

  3. How do these findings relate to 【1】? Section 6 and Figure 4 report reduced alignment between sycophantic lying and other truth representations after post-training. The direction definitions and similarity metrics differ, so the results are not equivalent, but discussing their relationship would help clarify the contribution. This is a request for context rather than a requirement to compare against an arXiv preprint.

【1】Ying et al., The Truthfulness Spectrum Hypothesis (arXiv:2602.20273)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.