- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates whether post-training systematically changes the internal relationship between representations associated with truthfulness and sycophancy. It constructs a truth direction from the activation difference between false and true statements, and a sycophancy direction from the activation difference between agreeing with a user’s false claim and rejecting it. The paper then measures the cosine similarity between these directions and averages them into the Geometric Fragility Signature (GFS). A cosine near 0 means the directions are roughly orthogonal, a negative cosine means they are more opposed, and a positive cosine means they are more aligned.
The authors compare matched base models and instruction-tuned models and find that post-training generally shifts the geometry in the negative direction: the truth and sycophancy directions become more opposed. This is the paper's main empirical finding and is relatively consistent across model families.
They then ask whether this geometry can be manipulated and whether doing so changes model behavior. They construct aggregate truth and sycophancy directions and rotate the sycophancy direction toward different target angles relative to the truth direction. This changes the rate at which models abandon answers they previously answered correctly when given an instruction to prioritize agreement with the user.
Finally, they investigate whether this manipulation can be used for sycophancy mitigation. They directly steer models away from the sycophancy direction or toward the truth direction. They observe substantial effects in some models, but the effects are inconsistent across models and can come with capability costs.
Importantly, the authors themselves acknowledge two major limitations: the behavioral intervention is not uniquely attributable to the proposed sycophancy direction, and the behavioral setup does not cleanly isolate sycophancy from general instruction following/compliance.
Critical strengths
-
Broad and systematic empirical evaluation. The paper evaluates the proposed geometric signature across 22 matched base/instruction-tuned models spanning nine model families and different model scales, and further examines the finding across multiple factual datasets. This breadth makes the central observation (post-training is associated with a shift toward more negative truth–sycophancy geometric alignment) more convincing than a result demonstrated on only one or two models. I also appreciate that the authors are appropriately cautious in interpreting the cross-dataset result and explicitly acknowledge its limitations rather than overclaiming generality.
-
The target-angle intervention is a creative and informative probing method. Constructing a modified sycophancy vector with a prescribed angle relative to the truth vector while preserving its norm provides a systematic way to probe whether the geometric relationship itself can influence model behavior.
Critical weaknesses
- The proposed sycophancy direction is not cleanly disentangled from other factors. The direction is constructed from the difference between agreeing and disagreeing with false user claims. Consequently, it may encode generic agreement, response polarity, social compliance, semantic differences, and/or the truth value of the underlying claim, rather than sycophancy specifically. This is consequential because the paper's central geometric interpretation depends on treating this contrast as a sycophancy representation.
Questions for authors
- How large is the observed GFS shift in representation space, and what does a change from approximately −0.01 to −0.03 imply geometrically? The reported average change is only about 0.02 cosine units. Is this shift large relative to the variance across examples, model families, or unrelated contrast directions? Can the authors quantify the corresponding angular change and demonstrate that it is sufficiently large to explain the observed behavioral effects?
- Why are different subsets of models used for the different steering and control experiments? Geometry-derived steering is evaluated on eight models, the random control on five, and all three control directions on only three models. How were these models selected, and were the selection criteria determined before observing the results? Could computational constraints or model-specific properties have influenced which models received the more complete set of controls?
- I have a concern about the sycophancy direction definition. How do the authors disentangle sycophancy from generic agreement, response polarity, and the truth value of the underlying statement? Since the sycophancy direction is constructed only from false statements, could the observed geometry instead reflect an interaction between agreement and truth value?
- How do the authors establish that the behavioral effects of target-angle steering are specific to the proposed truth–sycophancy geometry? In several experiments, structured control directions can produce comparable behavioral changes. If multiple unrelated directions can alter collapse rates, what evidence supports interpreting the effect as manipulation of a sycophancy-specific representation rather than a generic effect of activation perturbation?