acceptodds
Under review as a conference paper at ICLR 2027

Evaluating stability and generalization of truth and safety probes in LLMs

Abstract

Linear probing is widely used to investigate how language models represent concepts such as truthfulness and safety, and has been proposed for monitoring and steering model behavior. Prior studies report contradictory results: some find shared linear directions that transfer across datasets, while others report poor cross-dataset robustness or stability. However, these studies differ in language models, datasets, prompt formats, and evaluation procedures, making their conclusions difficult to reconcile. Guided by the Predictability–Computability–Stability (PCS) framework for veridical data science, we treat these experimental choices as reasonable perturbations within a unified study and ask which conclusions about linear probing remain stable or change across them. Our study spans twelve datasets across medical, factual, statistical, safety, and social-reasoning domains, as well as different language model layers, pretrained architectures, prompt formats, input feature types, probe classification algorithms, and training-domain coverage. We find that truth- and safety-related activation representations across domains are better described by hierarchical linear probes that jointly model shared structure alongside dataset-specific variation over flat linear or nonlinear probes. In assessing the role of distribution shift, we find that probes trained on data including target distribution samples consistently outperform those trained out-of-domain, while aligning prompts to a standard True/False format improves cross-domain transfer. Nonlinear probe architectures and alternative feature representations either match or underperform linear probes trained on raw activations. Finally, predictive performance and steering effectiveness are distinct properties: the directions that most strongly steer the evaluated models align with maximal activation variance rather than discriminative power.

Primary area:
interpretability and explainable AI
Keywords:
activation probing, interpretability, large language models, AI safety, truthfulness, activation steering, generalization, stability, distribution shift

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_wKcNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper evaluates 12 linear and nonlinear probe architectures across 12 truth and safety datasets, two models (Gemma-3-4B-IT, Qwen2.5-7B-Instruct), four layer depths, and two prompt framings under the Predictability–Computability–Stability (PCS) framework. The authors introduce hierarchical random-slopes linear probes to model shared versus dataset-specific directions and compare discriminative performance against activation steering efficacy.

Critical strengths
  1. The experimental grid covers 12 datasets (including two new human-validated datasets, Anatomy and VDS) with grouped cross-validation that prevents data leakage across paired prompts.
  2. Applying multilevel random-slopes logistic regression (b0+ag+zig⊤(w0+ug)b_0 + a_g + z_{ig}^\top(w_0 + u_g)) provides a clear mathematical formulation to separate shared concept directions (w0w_0) from dataset-specific adjustments (ugu_g).
  3. Applying Ledoit-Wolf shrinkage to the target covariance matrix Σ^t\hat{\Sigma}_t in the Mahalanobis cosine similarity corrects the circularity between unregularized geometric alignment and prediction correlation in prior work.
Critical weaknesses
  1. Sec. 3.6 concludes that steering aligns with maximal activation variance (PC1\text{PC}_1) rather than discriminative power. However, within each target dataset, the cosine similarity between PC1\text{PC}_1 and the difference-in-means direction (μ1−μ0\mu_1 - \mu_0) is ≥0.97\ge 0.97. Because the two directions are collinear in these curated datasets, the ablation cannot distinguish whether steering relies on variance (PC1\text{PC}_1) or class separation (μ1−μ0\mu_1 - \mu_0).
  2. Random-slopes probes require known dataset identity gg to apply the adjustment ugu_g. In out-of-distribution settings where gg is unknown (VDS in Figure 5), weighting adjustments by unsupervised activation similarity fails to improve over the shared direction w0w_0 because label-free geometry does not predict alignment across domains.
  3. Evaluations are limited to two small instruction-tuned models (4B and 7B), and steering is measured only by next-token log-odds shifts (log⁡[P(True)/P(False)]\log[P(\text{True})/P(\text{False})]) rather than multi-token generation quality or refusal rates.
Questions for authors
  1. In Appendix B.1, how is the covariance Σ\Sigma of the random effects (ag,ug)(a_g, u_g) estimated in practice—via penalized maximum likelihood with fixed penalty ratios (10×10\times or 100×100\times) or empirical Bayes / REML?
  2. Why do Mixture-of-Experts (MoE) probes underperform random-slopes probes when dataset identity is 99%99\% linearly decodable from the activation space ?
  3. Does the cross-concept transfer between truth and safety (Figure A4) persist in base (non-instruction-tuned) models, or is it induced by post-training alignment?
1 comment

We sincerely thank the reviewer for the detailed methodological comments and constructive suggestions. Our contributions and novelty relative to prior work are detailed in the first comment of our response to Reviewer QAF7.

W1. Diff-in-means versus PC1. We agree that near-collinearity between PC1 and the difference-in-means direction prevents the existing steering experiments from distinguishing maximal variance from class separation as the relevant causal factor, and thank the reviewer for pointing this out. We have since done an analysis using target-absent experiments to help distinguish between the two; the experimental results and details are in the second comment of our response to Reviewer QAF7, but we summarize them below.

When the held-out data corpus determines the direction vectors, the collinearity between PC1 and diff-in-means greatly decreases (cosine similarity between 0.37-0.94). Diff-in-means has significantly higher steering effect than PC1, and PC1 steers as or less effectively than random vectors on Gemma once diff-in-means is projected out, and far less effectively than the diff-in-means on Qwen. This provides more clear evidence other directions such as PC1 appear to steer proportionally to their alignment with the diff-in-means; diff-in-means is causally implicated.

W2. Known dataset identity. Known dataset identity is an explicit assumption of the random-slopes model’s descriptive application, since the model jointly represents shared predictive structure and source-specific variation through partial pooling. Its descriptive contribution can therefore be evaluated independently of the ability to select adjustments for an entirely new domain without labels. For an unknown dataset, the implementation uses the shared component unless an adjustment is supplied separately.

W3. Scale and generation. We agree that the architecture rankings are supported for the two evaluated models and single-turn settings; their persistence at larger scales or multi-turn prompts remains unestablished. Our study uses model sizes and single-turn responses consistent with the majority of prior work we built upon.

We have also re-run our steering experiments to additionally capture the first 50 tokens of the model’s output, and used LLM-as-a-judge (Claude Opus) to judge whether the response coherently reflects the label being steered towards. Full results and experimental details are in Appendix secion C.6, but to summarize: we find that judged free-responses for Qwen show significant response to steering, whereas Gemma shows weaker effects. We find congruent relationships between the effect of L2 regularized probes and diff-in-means to the next-token probability experiment; when L2 regularized directions are projected onto the top PC (almost perfectly reflecting the diff-in-means), the steering effect is far stronger than when projecting onto the top 50 components. We have modified the paper's conclusions about steering to reflect this new result.

Q1. Estimation. We fit the random-slopes probe using Bayesian inference rather than restricted maximum likelihood due to the binary objective and limited number of groups. We have updated the corresponding section in the Appendix with the implementation details.

Q2. MoE probes. Linear decodability of dataset identity does not imply that a MoE router trained to predict truth or safety labels will recover the dataset partition. Our MoE router is trained jointly with the experts using the prediction objective, without supervision on dataset identity or a constraint assigning one expert to each dataset. In assessing how prompts are routed, we found that individual experts handle prompts from multiple sources, showing that the unconstrained routing does not impose the grouping used by the random-slopes model; it does not indicate that source information was unavailable in the activations. In comparing the random-slopes to MoE, we are able to distinguish the benefit of explicitly encoding dataset identity in hierarchical probe performance across known sources.

Q3. Base models. We have run a new experiment to evaluate how cross-concept transfer between truth and safety changes on the base models corresponding to the instruction-tuned Qwen and Gemma models. We show results below for diff-in-means directions trained on the Geometry of Truth dataset (GoT, factual) and XSTest (safety) and applied to one another, using the true/false prompts as input at the 88% depth (base / instruction-tuned AUC using 5-fold CV). We observe that the degree of cross-concept transfer is greater in the instruction-tuned model for Gemma but still quite significant in base models, particularly for Qwen, where the two are comparable.

Trained on → tested onGemma-3-4BQwen2.5-7B
GoT → GoT0.688 / 0.9910.993 / 0.995
GoT → XSTest0.707 / 0.8970.936 / 0.881
XSTest → GoT0.621 / 0.9450.992 / 0.985
XSTest → XSTest0.874 / 0.9470.953 / 0.966
Reviewer_5xc6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper systematically investigates the stability and cross-domain generalization of truth and safety probes in large language models. The authors evaluate 12 datasets spanning five domains using Gemma-3-4B and Qwen2.5-7B, comparing linear, nonlinear, and hierarchical probes across different layers, activation features, prompt formats, and training distributions. The results show that linear probes generally match nonlinear alternatives, while hierarchical random-slope probes improve prediction by combining shared and dataset-specific components. Standardized True/False prompting also improves cross-domain transfer. Finally, the steering experiments reveal that predictive accuracy does not necessarily translate into greater influence on model outputs, with difference-in-means directions producing the strongest steering effects.

Critical strengths

The paper provides a broad comparison of probing methods across 12 datasets, two language models, different prompt formats, model layers, and training distributions. The use of grouped cross-validation and multiple transfer settings helps ensure consistent comparisons. This unified evaluation provides a useful empirical reference for understanding how experimental choices influence probe performance.

Critical weaknesses
  1. Limited novelty and insufficiently distinctive scientific contributions: The paper provides a broad empirical evaluation, but its main findings largely confirm observations from existing literature rather than establish substantially new insights. Specifically, the competitiveness of linear probes, the importance of training-domain coverage for transfer, and the mismatch between probe accuracy and steering effectiveness have all been investigated in prior work. The hierarchical random-slopes approach is also based on established multilevel regression techniques rather than a new methodological contribution.

  2. Insufficient experimental evidence to support the central generalization and steering claims: The largest gains in Figure 4 rely on including the target dataset during training. Figure 5 evaluates held-out transfer, but detailed results cover only two datasets, limiting evidence of generalization. Moreover, Appendix C.5 shows that geometric similarity provides little guidance for selecting adjustments on unseen domains; Steering is evaluated mainly through next-token True/False log-odds rather than factual correctness or safety in generated responses.

Questions for authors

As mentioned in Weaknesses

Reviewer_QAF7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The contribution of the present paper on truth and safety probes is that it takes a relatively broad view across 12 datasets of statements and questions, including general factual knowledge, medicine, statistics, safety, and social reasoning. They study two language models, Gemma-3-4B and Qwen2.5-7B. They train truth probes on the final prompt token’s hidden representation in different layers of these models. The probes include linear and non-linear classifiers. Results show that out-of-distribution generalization of truth probes is difficult. Truth probes perform better when the training data includes samples from the distribution of the test data. Linear probes appear preferable, performing as well or better than non-linear alternatives. To perform well on a given dataset, a probe benefits from adjustments to adapt it to that dataset. The authors explore a hierarchical logistic regression approach, using shared weights and regularized adjustments. An interesting finding is that linear probe axes that perform best for readout are not necessarily best for steering. Simple difference-of-means probes perform better for steering than probes optimized for readout accuracy.

Critical strengths

Applying a hierarchical linear decoding approach to truth probing.

Exploration of 12 different data sets, 2 models, different layers, various probe formats, various probe architectures and training data compositions.

Separate evaluation of readout and steering performance.

Critical weaknesses

Most of the concepts the paper is about have been investigated in previous studies, so the advance is somewhat incremental.

Steering does not make the models more of less truthful in general, it just steers binary answers toward the literal labels “true” or “false”, not, for example, toward the true answer (which may be the response “false”). Whether the results will generalize to larger model and more natural contexts of generation in long chats is questionable.

Questions for authors

Given the alignment of the difference-of-mean vector and the first PC, does steering work equally well of both directions, or does the small difference have a big impact on effectiveness? It would be good to compare steering signals of matched magnitude along the difference of means, PC1, and the difference-of-means component that is orthogonal to PC1. In addition, it would be good to report the original-space angles, and label-based variance decomposition.

2 comments
AuthorsOpenReview ↗

Response to comment about novelty

We thank the reviewer for recognizing the value of the hierarchical analysis and the separate evaluation of readout and steering. We respond to the concerns below: this first part addresses novelty, and the second addresses the steering questions.

1. Novelty. The reviewer is correct that many of the topics we study have precedent. However, previous studies each evaluate a far narrower set of probe architectures, training/evaluation settings, hypotheses about probe performance and transfer, and data sources. Our work was motivated by the fact that it was unclear how previous findings should be integrated or generalized, particularly where experimental setups and conclusions disagreed. We address these gaps by more comprehensively evaluating which results regarding linear probing hold across domains and experimental perturbations, alongside a much more extensive comparison with non-linear and hierarchical probe architectures than we have seen in prior work. This systematic assessment of the generalizability of findings in truth/safety probing is itself an important contribution. We additionally highlight four results that, to our knowledge, have not been established elsewhere.

First, prior work establishes that linear probes are effective in particular settings (Marks and Tegmark, 2024) and that diverse training data is beneficial (Liu et al., 2024). It does not establish how source coverage affects in-domain and out-of-domain performance, or whether nonlinear probe architectures become more beneficial as the number and diversity of training sources increase. For the latter, we show that this is not the case: the best linear and best nonlinear probes differ by at most 0.011 median AUC in every training setting across our truth and safety datasets, a result that has not previously been established. Within the models and settings evaluated, it supports the use of linear probes even as training sources become more diverse, providing useful evidence for the broader research community.

Second, introducing the random-slopes architecture as a descriptive model of concept homogeneity and heterogeneity across domains is novel. Multilevel regression is an established statistical method; existing truth probes based on logistic regression, LDA, and difference in means also use established estimators. The methodological contribution here is the application of partial pooling to provide a unified, interpretable account of representational heterogeneity in truth and safety representations. While previous work (e.g., Ying et al., 2026) has established that variation in truth representations exists across datasets, neither the relationship between shared signal and domain-specific heterogeneity nor the relationship between domain similarity and cross-domain transfer has previously been characterized in this way within one comprehensive framework. The descriptive value of the random-slopes model should therefore be assessed independently of adaptation to unseen, unlabeled domains, which we included but did not intend as the approach’s primary value. We will clarify that Figure 4 evaluates this descriptive contribution and does not establish adaptation to an unseen dataset.

Third, our finding that truth and safety concept representations partially transfer to one another is novel; previously examined transfer across factual datasets does not establish cross-concept transfer. This is an important addition to the literature on truth probing and truth concept representations, particularly given the value for alignment research of understanding how models represent these two concepts. During the rebuttal period, we have additionally performed a comparison of the cross-concept transfer in instruction-tuned versus base models, finding that the degree of transfer is greater in the instruction-tuned model for Gemma but still present in base models, particularly for Qwen. Details and results will be added to the appendix.

Fourth, we show that coefficient vectors steer the model's next token label probabilities only through their overlap with difference-in-means directions. This accounts for the earlier observation that diff-in-means directions steer more effectively than classifier weights (Marks and Tegmark, 2024; Im and Li, 2025); we provide more causal evidence through experiments that control the degree of coefficient overlap with the diff-in-means direction to pinpoint its influence. During the rebuttal period, we conducted additional experiments showing that the diff-in-means direction, rather than simply PC1, is responsible for steering; we provide details in point 4 of the second part of this response. We have included additional results for free generated model responses in Appendix C.6.

We will revise the contribution statement to clarify the significance of these contributions and distinguish them from previous work on truth probing.

AuthorsOpenReview ↗

Responding to the remaining points

2. Steering towards labels, not the true answer. We agree that our steering experiments test how effectively we can move the model output probabilities toward each label; this is precisely their aim, rather than improving factual accuracy. Specifically, steering toward ‘True’ along a direction fitted to separate ‘True’ from ‘False’ statements should push the answer toward ‘True’ for any statement, regardless of its label, which is what we observe in practice. For an actually false statement, this would reduce the model accuracy in determining the correct label, while still constituting successful steering towards the label of interest. Steering toward the correct answer would instead require conditioning the push on each prompt’s ground-truth label. Our experiments therefore aim to establish the causal influence of the tested directions on the model’s judgment toward the desired class, which provides evidence for how those concepts are encoded within activation space.

We have also re-run our steering experiments to additionally capture the first 50 tokens of the model’s output, and used LLM-as-a-judge (Claude Opus) to judge whether the response coherently reflects the label being steered towards. Full results and experimental details are in Appendix secion C.6, but to summarize: we find that judged free-responses for Qwen show significant response to steering, whereas Gemma shows weaker effects. We find congruent relationships between the effect of L2 regularized probes and diff-in-means to the next-token probability experiment; when L2 regularized directions are projected onto the top PC (almost perfectly reflecting the diff-in-means), the steering effect is far stronger than when projecting onto the top 50 components.

3. Larger models and long chats. We agree that the architecture rankings are supported for the two evaluated models and single-turn settings; their persistence at larger scales or multi-turn prompts remains unestablished and is a direction for future work. Our study uses model sizes and single-turn responses consistent with the majority of prior work we built upon. We will explicitly note this in the Limitations section.

4. Diff-in-means versus PC1. We agree that near-collinearity between PC1 and the difference-in-means direction prevents the existing steering experiments from distinguishing maximal variance from class separation as the relevant causal factor. We have since done the following analysis using target-absent experiments to help distinguish between the two. While within each target, PC1 and diff-in-means are nearly perfectly collinear (all with cosine similarity at least 0.97), when we evaluate steering performance on each target using the held-out data corpus to determine the direction vectors, the collinearity between PC1 and diff-in-means greatly decreases (with cosine similarity between 0.37-0.94). Below, we show the log-odds shift per steering step for the two targets per model for which the training corpus produced the least collinear PC1 and diff-in-means; we will put the results for all datasets in the appendix. Diff-in-means has significantly higher steering effect than PC1 in these experiments, and once the diff-in-means is projected out, PC1 steers as or less effectively than random vectors on Gemma, and far less effectively than the diff-in-means on Qwen. This provides more clear evidence that diff-in-means is the direction that steers the model; other directions such as PC1 appear to steer proportionally to their alignment with the diff-in-means.

Model, targetcos(diff, PC1)DiffPC1PC1, diff projected outRandom (max abs.)
Gemma, CommonClaim0.5418.112.12.73.9
Gemma, PubMedQA-Labeled0.5513.86.51.33.1
Qwen, CommonClaim0.3712.26.21.91.1
Qwen, PubMedQA-Labeled0.4210.22.22.30.8

5. Plain cosine angles. The paper reports regularized Mahalanobis cosine similarity between vectors, to distinguish variability involved in predictive differences versus unrelated variability (App. B.2). Here we report plain cosine similarities between diff and PC1 for comparison: on Gemma, per dataset, they are between 0.83-1.00, and on Qwen between 0.95-1.00, except for the Geometry of Truth dataset on Gemma, where PC1 is unaligned with the diff-in-means (cosine sim 0.05) but PC2 is nearly collinear. Pooled corpus L2 coefficients and diff-in-means have Mahalanobis cosine 0.81 (Gemma) and 0.85 (Qwen) but plain cosine 0.01-0.05 and 0.03-0.24, since most of the logistic regression-based weights lie in low-variance coordinates.

6. Variance decomposition. PC1 explains 26-77% of a dataset’s variance on Gemma and 35-67% on Qwen. Between-class variance is 31-55% of the total variance along PC1 on the six Gemma steering targets and 42-65% on Qwen, so PC1 is not purely a class label axis. We will add these details to the appendix.

Reviewer_4Wk2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work is an extensive study that compares 6 linear probes and 6 non-linear probes for evaluating truth and safety in LLMs. The authors conduct experiments with 2 models (Gemma-3-4B-IT and Qwen2.5-7B-Instruct) and 12 datasets. They conclude that training-distribution coverage is the most important for determining probe performance, and linear probe >= non linear probe, and predictive accuracy does not determine the model's capability in activation steering.

Critical strengths
  • 2 new datasets that can be valuable to the AI safety community.
  • Broad coverage of techniques and datasets.
Critical weaknesses

Despite the soundness and wide coverage, I don't think this work is sufficiently novel. The conclusions are not new and mostly similar to existing works, so this work mostly provides further validation to previous results. The dataset contribution is novel but is only a small part of the work.

Questions for authors

No questions.

1 comment
AuthorsOpenReview ↗

Response to comment about novelty

We thank the reviewer for recognizing the value of our new dataset and the comprehensive experiments. We respond to this concern below.

The reviewer is correct that many of the topics we study have precedent. However, previous studies each evaluate a far narrower set of probe architectures, training/evaluation settings, hypotheses about probe performance and transfer, and data sources. Our work was motivated by the fact that it was unclear how previous findings should be integrated or generalized, particularly where experimental setups and conclusions disagreed. We address these gaps by more comprehensively evaluating which results regarding linear probing hold across domains and experimental perturbations, alongside a much more extensive comparison with non-linear and hierarchical probe architectures than we have seen in prior work. This systematic assessment of the generalizability of findings in truth/safety probing is itself an important contribution. We additionally highlight four results that, to our knowledge, have not been established elsewhere.

First, prior work establishes that linear probes are effective in particular settings (Marks and Tegmark, 2024) and that diverse training data is beneficial (Liu et al., 2024). It does not establish how source coverage affects in-domain and out-of-domain performance, or whether nonlinear probe architectures become more beneficial as the number and diversity of training sources increase. For the latter, we show that this is not the case: the best linear and best nonlinear probes differ by at most 0.011 median AUC in every training setting across our truth and safety datasets, a result that has not previously been established. Within the models and settings evaluated, it supports the use of linear probes even as training sources become more diverse, providing useful evidence for the broader research community.

Second, introducing the random-slopes architecture as a descriptive model of concept homogeneity and heterogeneity across domains is novel. Multilevel regression is an established statistical method; existing truth probes based on logistic regression, LDA, and difference in means also use established estimators. The methodological contribution here is the application of partial pooling to provide a unified, interpretable account of representational heterogeneity in truth and safety representations. While previous work (e.g., Ying et al., 2026) has established that variation in truth representations exists across datasets, neither the relationship between shared signal and domain-specific heterogeneity nor the relationship between domain similarity and cross-domain transfer has previously been characterized in this way within one comprehensive framework. The descriptive value of the random-slopes model should therefore be assessed independently of adaptation to unseen, unlabeled domains, which we included but did not intend as the approach’s primary value. We will clarify that Figure 4 evaluates this descriptive contribution and does not establish adaptation to an unseen dataset.

Third, our finding that truth and safety concept representations partially transfer to one another is novel; previously examined transfer across factual datasets does not establish cross-concept transfer. This is an important addition to the literature on truth probing and truth concept representations, particularly given the value for alignment research of understanding how models represent these two concepts. During the rebuttal period, we have additionally performed a comparison of the cross-concept transfer in instruction-tuned versus base models, finding that the degree of transfer is greater in the instruction-tuned model for Gemma but still present in base models, particularly for Qwen. Details and results will be added to the appendix.

Fourth, we show that coefficient vectors steer the model's next token label probabilities only through their overlap with difference-in-means directions. This accounts for the earlier observation that diff-in-means directions steer more effectively than classifier weights (Marks and Tegmark, 2024; Im and Li, 2025); we provide more causal evidence through experiments that control the degree of coefficient overlap with the diff-in-means direction to pinpoint its influence. During the rebuttal period, we conducted additional experiments showing that the diff-in-means direction, rather than simply PC1, is responsible for steering; we provide details in point 4 of the second part of this response. We have included additional results for free generated model responses in Appendix C.6.

We will revise the contribution statement to clarify the significance of these contributions and distinguish them from previous work on truth probing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.