acceptodds
Under review as a conference paper at ICLR 2027

: Revealing Truth-Direction Sensitivity Loss under Sycophantic Pressure

Abstract

Language models can retain decodable truth-related information while their answers change under user pressure. Assessing the robustness of these representations requires measuring how strongly answers respond to the same information. We introduce Probing–Sensitivity Inspection (PSI), a protocol that pairs a frozen linear probe with a two-sided intervention along the same fitted truth direction. On single-token true/false tasks with Qwen2.5 (7B, 14B, 32B) and Gemma-2-9B, PSI reveals three findings. First, decodability–sensitivity dissociation: probe accuracy remains at 1.000 across the tested prompt conditions and cue levels, while intervention responses vary and reverse under instructed inversion; Qwen2.5-7B then answers truthfully on 0.133 of items. Second, sensitivity attenuation under preserved accuracy: under honest system prompts, sensitivity declines with sycophantic pressure in both model families while truthful-answer accuracy remains high. At the first cue level on Qwen2.5-7B, the intervention effect falls by more than one fifth while accuracy is 0.933. Sycophantic cues produce larger reductions than length-matched neutral prompts. Third, projection-based truth recovery: under preference-conditioned prompts, amplifying each item's existing projection raises truthful-answer accuracy from 0.700 to 0.883 on Qwen2.5-7B and from 0.500 to 0.850 on Gemma-2-9B, with little change under norm-matched random directions. PSI reveals pressure-dependent changes in truth-direction sensitivity that remain hidden behind stable probe accuracy and strong answering performance.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, sycophancy, truthfulness, linear probing, activation interventions, representation analysis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_G5GXOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Probing–Sensitivity Inspection (PSI), a diagnostic protocol to investigate the relationship between truth-related representations and model outputs under sycophantic pressure. PSI combines a frozen linear probe for measuring truth decodability with bidirectional activation interventions along the same fitted truth direction to quantify output sensitivity. The authors further introduce normalized sensitivity and a truth–preference competition ratio to characterize how model responses change across different prompting conditions. Experiments on Qwen2.5 models (7B, 14B, and 32B) and Gemma-2-9B reveal three observations. First, truth-related information remains linearly decodable even when models produce incorrect or intentionally inverted answers. Second, sensitivity to the fitted truth direction decreases under increasing sycophantic pressure, despite relatively stable probe accuracy and truthful-answer accuracy. Third, amplifying the existing projection onto the truth direction can improve truthful-answer accuracy under preference-conditioned prompts.

Critical strengths
  1. An interesting question.

The paper investigates whether truth-related information can remain accessible in internal representations even when model outputs are influenced by user preferences or misleading instructions. Distinguishing representation decodability from output sensitivity is meaningful for understanding LLM truthfulness and sycophantic behavior.

  1. A relatively straightforward and interpretable framework.

PSI combines linear probing and activation interventions within a unified evaluation protocol. The mathematical formulation is relatively simple, and the use of a frozen truth direction provides a consistent reference for comparing responses under different prompting conditions.

Critical weaknesses
  1. Insufficient construct validity of the proposed truth-direction sensitivity metric.

My primary concern is whether the proposed sensitivity metric actually measures the intended property of truth utilization. In Eq. (2), truth-direction sensitivity is defined as the difference in answer logits resulting from positive and negative perturbations along the fitted truth direction. Mathematically, this quantity measures the directional responsiveness of the output to an externally imposed activation perturbation. However, it does not necessarily reflect how strongly the model utilizes truth-related information during an unperturbed forward pass.

For instance, a model may exhibit substantial intervention sensitivity even when its natural prediction is dominated by unrelated features or user preferences. Conversely, a relatively small intervention effect does not necessarily imply that the model relies less on truth-related representations. The observed sensitivity attenuation under sycophantic pressure may reflect changes in local response geometry, nonlinear transformations, or output margins rather than a genuine reduction in truth utilization.

  1. The intervention scale and direction construction introduce additional confounding factors.

The sensitivity measurements rely on a fixed intervention scale of α = 1.0, but the paper does not justify why this scale is appropriate or comparable across different models, layers, and prompting conditions. Table 1 shows that intervention strength substantially affects answer accuracy, suggesting that the selected perturbation may alter model behavior. Stronger evidence is needed to demonstrate that the central conclusions remain consistent under smaller perturbations and appropriately normalized intervention magnitudes. The truth direction is constructed using the difference between mean activations of true and false statements. High probe accuracy alone cannot show that this direction captures a generalizable truth representation.

Questions for authors

Q1. Novelty of PSI. What fundamentally distinguishes PSI from existing frameworks combining linear probing with activation steering or causal interventions? Can the authors demonstrate a diagnostic capability that cannot be achieved using conventional probing and intervention techniques?

Q2. Construct validity of truth-direction sensitivity. How do the authors justify interpreting the two-sided intervention effect as a measure of truth utilization? Could the observed attenuation arise from changes in local response geometry or output margins without any reduction in the natural contribution of truth-related representations?

Reviewer_j2DaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Probing–Sensitivity Inspection (PSI) to study how language models use truth-related representations under sycophantic pressure. PSI applies a linear probe and an activation intervention along the same truth direction, measuring both how well truth can be decoded and how strongly changes along that direction affect model outputs. The authors evaluate Qwen2.5 (7B/14B/32B) and Gemma-2-9B under different system prompts and increasing levels of sycophantic pressure. Beyond these experimental concerns, I also find the broader significance of the work somewhat unclear. Although the observed dissociation is interesting, the paper does not sufficiently explain what researchers can learn from it or how it could guide future work. A clearer discussion of the practical value and implications of PSI would help strengthen the paper's contribution.

Critical strengths
  1. The distinction between probe accuracy, intervention sensitivity, and answer accuracy is interesting. While previous work has shown that decodable information is not necessarily used by the model, this paper further shows that sensitivity can decrease before answer accuracy changes.
  2. The paper includes several relevant controls, including random and shuffled-label directions, length-matched neutral prompts, and out-of-sample prediction. These experiments help rule out simple explanations for the observed effects.
Critical weaknesses
  1. The paper measures sensitivity using a fixed truth direction learned under the honest condition. However, this direction may change as sycophantic pressure increases. For example, the model could still encode truth information, but the relevant direction might rotate in the representation space. In this case, interventions along the original direction would become less effective, even if the model's ability to use truth information has not weakened. I suggest fitting a new truth direction at each cue level and measuring:
  • The cosine similarity between the original and newly fitted directions.
  • Whether sensitivity still decreases when measured along the newly fitted direction.

This experiment is important for distinguishing changes in representation geometry from reduced sensitivity to truth information. Without it, the main interpretation remains uncertain.

  1. The paper reports a probe accuracy of 1.0 across all conditions. However, this result is based on a single difference-of-means direction and a binary classification threshold.Perfect classification accuracy does not mean that the representation remains unchanged. For example, the separation margin could decrease, the variance could increase, or the direction could rotate while classification accuracy remains at 1.0.The authors could strengthen this finding by reporting continuous measurements, such as projection distributions, classification margins, and variance. Comparing with other probes, such as logistic regression or CCS, would also help verify that the result is not specific to the current probe.

While the paper presents interesting observations, I find it difficult to understand how these findings could guide future research. The authors show that truth-direction sensitivity can decrease even when answer accuracy remains high, but the implications of this observation are not sufficiently discussed.

Questions for authors
  • Does the truth direction change under sycophantic pressure? If you refit the truth direction at each cue level, how similar is it to the original direction? Does sensitivity still decrease when using the new direction? This is the most important question for my assessment.
  • Could you report projection distributions, margins, or other continuous measures as pressure increases? These results would help determine whether the representation changes despite perfect probe accuracy.
  • Specifically, does removing the truth-direction component have a smaller effect as pressure increases? This would help distinguish changes in normal model behavior from responses to large interventions.
1 comment

Thank you for the thorough and well-organized responses. The additional experiments directly address my three main concerns. I have therefore raised my rating.

Reviewer_HmCGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Probing–Sensitivity Inspection (PSI), a framework that combines linear probing with activation interventions to study truth-related representations under sycophantic pressure. It reports that truth information can remain decodable while output sensitivity changes. It also combines truth and preference sensitivities to predict model outputs and shows that amplifying existing truth-direction projections can improve truthful answering under preference-conditioned prompts.

Critical strengths
  • Strength 1: Simple and clear methodology. The paper proposes a straightforward framework that pairs probing with interventions along the same fitted direction. The approach makes the distinction between decodable information and output sensitivity easy to understand, without introducing unnecessary methodological complexity.

  • Strength 2: Interesting intersection. The paper explores an interesting intersection between truth-related representations and sycophancy, using intervention sensitivity to examine how factual information and user preferences relate to model outputs.

  • Strength 3: Clear presentation and supporting experiments. The paper is well organized, with clearly defined measurements and accessible explanations of its main findings.

Critical weaknesses
  • Weakness 1: Incomplete comparisons across models. The abstract and introduction describe an evaluation of four models, but complete results across those models are missing from key experiments. In particular:

    • Dissociation results: Section 4 establishes the dissociation between decodability and sensitivity, one of the paper’s central findings, without providing a complete model-by-model comparison of both measurements.
    • Prediction comparisons: The competition-prediction analysis compares the proposed rule with the truth-only baseline only for Qwen2.5-7B, without reporting corresponding comparisons for the other three models.

    These results should be provided across all four models to assess whether the central finding and predictive benefits hold consistently.

  • Weakness 2: The added value of the proposed sensitivity measure needs clearer validation. The improvement from adding preference information does not by itself establish the benefit of using sensitivity to quantify the relative influence of truth and preference. A suitable comparison would give a baseline with both truth and preference information, but use simple fixed or learned weights instead of measured sensitivities. This comparison should be reported across all four models, using the same held-out examples for competing predictors within each model. It would clarify whether the proposed measurement provides additional predictive value beyond including both information sources.

  • Weakness 3: Limited depth of mechanistic insight. The proposed framework is clean and useful, but the experiments primarily establish and quantify the observed phenomenon. The paper provides limited investigation into why user pressure weakens the output’s response to the truth direction, or how the model’s internal computation mediates the competition between factual truth and user preference. The measurement is validated more extensively than the underlying computation is explained, which limits the depth and significance of the mechanistic contribution.

Questions for authors
  • In Table 2, Gemma-2-9B has honest sensitivity +10.72 and preference-conditioned sensitivity +6.79, but normalized sensitivity −0.15. If these are the values used in Equation 3, their ratio would be approximately +0.63, not negative. Could the authors clarify whether this is a reporting error or whether different conditions or calculations were used?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.