- Rating
- 3(+1)
- Confidence
- 1(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents Probing–Sensitivity Inspection (PSI), a diagnostic protocol to investigate the relationship between truth-related representations and model outputs under sycophantic pressure. PSI combines a frozen linear probe for measuring truth decodability with bidirectional activation interventions along the same fitted truth direction to quantify output sensitivity. The authors further introduce normalized sensitivity and a truth–preference competition ratio to characterize how model responses change across different prompting conditions. Experiments on Qwen2.5 models (7B, 14B, and 32B) and Gemma-2-9B reveal three observations. First, truth-related information remains linearly decodable even when models produce incorrect or intentionally inverted answers. Second, sensitivity to the fitted truth direction decreases under increasing sycophantic pressure, despite relatively stable probe accuracy and truthful-answer accuracy. Third, amplifying the existing projection onto the truth direction can improve truthful-answer accuracy under preference-conditioned prompts.
Critical strengths
- An interesting question.
The paper investigates whether truth-related information can remain accessible in internal representations even when model outputs are influenced by user preferences or misleading instructions. Distinguishing representation decodability from output sensitivity is meaningful for understanding LLM truthfulness and sycophantic behavior.
- A relatively straightforward and interpretable framework.
PSI combines linear probing and activation interventions within a unified evaluation protocol. The mathematical formulation is relatively simple, and the use of a frozen truth direction provides a consistent reference for comparing responses under different prompting conditions.
Critical weaknesses
- Insufficient construct validity of the proposed truth-direction sensitivity metric.
My primary concern is whether the proposed sensitivity metric actually measures the intended property of truth utilization. In Eq. (2), truth-direction sensitivity is defined as the difference in answer logits resulting from positive and negative perturbations along the fitted truth direction. Mathematically, this quantity measures the directional responsiveness of the output to an externally imposed activation perturbation. However, it does not necessarily reflect how strongly the model utilizes truth-related information during an unperturbed forward pass.
For instance, a model may exhibit substantial intervention sensitivity even when its natural prediction is dominated by unrelated features or user preferences. Conversely, a relatively small intervention effect does not necessarily imply that the model relies less on truth-related representations. The observed sensitivity attenuation under sycophantic pressure may reflect changes in local response geometry, nonlinear transformations, or output margins rather than a genuine reduction in truth utilization.
- The intervention scale and direction construction introduce additional confounding factors.
The sensitivity measurements rely on a fixed intervention scale of α = 1.0, but the paper does not justify why this scale is appropriate or comparable across different models, layers, and prompting conditions. Table 1 shows that intervention strength substantially affects answer accuracy, suggesting that the selected perturbation may alter model behavior. Stronger evidence is needed to demonstrate that the central conclusions remain consistent under smaller perturbations and appropriately normalized intervention magnitudes. The truth direction is constructed using the difference between mean activations of true and false statements. High probe accuracy alone cannot show that this direction captures a generalizable truth representation.
Questions for authors
Q1. Novelty of PSI. What fundamentally distinguishes PSI from existing frameworks combining linear probing with activation steering or causal interventions? Can the authors demonstrate a diagnostic capability that cannot be achieved using conventional probing and intervention techniques?
Q2. Construct validity of truth-direction sensitivity. How do the authors justify interpreting the two-sided intervention effect as a measure of truth utilization? Could the observed attenuation arise from changes in local response geometry or output margins without any reduction in the natural contribution of truth-related representations?
Thank you for the thorough and well-organized responses. The additional experiments directly address my three main concerns. I have therefore raised my rating.