acceptodds
Under review as a conference paper at ICLR 2027

Plausibility Confounds Linear Truth Probes in Large Language Models

Abstract

Recent work has shown that large language models have linear directions in their internal representations that can distinguish true from false factual statements, raising the possibility of verifying factual claims directly from hidden states without relying on the LLM’s generated output. Despite this promise, we find that existing truth-probing datasets, methods, and evaluation protocols overlook an important confound: plausibility, namely false statements that are factually or semantically close to the truth. Building on the datasets and probing framework of Marks and Tegmark (2024), we mine near-miss falsehoods from the model’s own conditional distribution and construct a plausibility direction using only plausible-false versus random-false statements. With no true statements used for probe training, this false-only direction is strongly aligned with the original truth direction (cosine similarity 0.78–0.84) and, when evaluated on the true-versus-false task, it reaches 88–100% of the original truth probe’s AUC. These results suggest that truth and plausibility are strongly aligned in representation space. This observation also exposes an important weakness in standard truth-probe evaluation. Existing work primarily reports ranking-based metrics such as AUC on test sets containing true statements and relatively easy random falsehoods. While such metrics are threshold-free, they do not directly characterize the reliability of a probe used to classify a single input statement. We therefore complement standard AUC with fixed-threshold single-statement classification and explicitly stress-test probes on plausible falsehoods. Replacing random falsehoods with near-miss negatives causes a modest degradation in AUC, and a much larger failure in deployment-style classification: the false-positive rate on plausible falsehoods is 3–8 that on standard random negatives. Importantly, near-miss falsehoods mined by one reference LLM remain challenging for other model families, indicating that the effect reflects transferable structure rather than model-specific adversarial examples. Geometric analyses further show that the truth and plausibility signals are coupled in ways that cannot be sufficiently disentangled by simple linear or nonlinear post-probe processing without sacrificing useful truth information. Therefore, we propose a strategy of incorporating plausible falsehoods directly into the negative training set, which aims at discouraging reliance on plausibility during truth probing, and it improves truth probing effects across multiple factuality datasets.

Primary area:
interpretability and explainable AI
Keywords:
Truth Probing, Mechanistic Interpretability, Large Language Models, LLM Safety

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fcMyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors show that the concept "plausibility" might be spuriously correlated and entangled with the concept "truth" in the latent space of LLMs. They present a procedure for constructing plausible but false statements. They construct a direction dPd_P as the mass mean between the plausible false and false statements and show that this direction can separate true and false statements reasonably well. Moreover, they show that standard truth probes, trained on true and false statements, achieve a lower AUC on the plausible false/false statements compared to the true/false statements. They propose the method "local contrastive readout" which achieves a higher AUC between plausible false and false statements.

Critical strengths
  • The authors demonstrate the existence of a direction dPd_P constructed only from false statements, that can nevertheless separate true and false statements with high AUC. That is a significant contribution.
  • Local contrastive readout seems to be a promising technique for more accurate classification of true and plausible false statements. The method has a number of limitations but these are communicated very transparently by the authors.
Critical weaknesses

Novelty and Significance

  • While the finding of the authors is interesting, it is quite narrow and rests heavily on prior work by Marks and Tegmark.
  • The authors narrowly investigate one potential confound to truth but there may be many others.
  • True/P in Table 2 is still quite high. I do not find it that surprising that more plausible falsehoods are harder to distinguish from true statements than blatant falsehoods. For sp_en_trans and cities the AUC of True/P is even 1, showing that dTd_T can separate true from plausible false.

Conceptual and Methodological concerns

  • Marks and Tegmark specifically study truth representations for statements that are "simple, uncontroversial, and unambiguous", i.e. for statements that the model "knows" to be true or false. The claim that truth directions separate true and false statements can only possibly be made for statements for which the model "knows" to some extent whether the given statement is true or false (possibly with some uncertainty). For the "plausible but false" statements you construct, it remains unclear whether the model "knows" that these are false or whether it takes them to be true. Could you please evaluate this, e.g. by asking the model to classify a given statement as true or false? Is the model able to accurately classify the plausible false statements as false? This would significantly clarify the results in my opinion. If the model can accurately predict that the plausible false statements are false but the truth probe classifies them as true, this points to some surprising confounding factors in truth probe training. If the model is uncertain about these statements (e.g. make a few prompt variations + sample), this is evidence that truth directions capture some notion of model uncertainty, which would be a cool result! However, if the model consistently classifies the plausible false statements as true, then the model is simply mistaken about these statements and we probably also should not expect a truth probe to classify them accurately. For this check, please focus specifically on the false positives of the standard truth probe.

  • Line 348: I do not understand why you use the plausible false statements generated with Llama for all other models as well. Wouldn't it be cleaner to generate the plausible false statements for each model individually?

Presentation

The presentation of the paper could be improved. For instance, a nice Figure 1 could help with the understanding, the notation is sometimes a bit confusing (e.g. "render" in Line 408 is never explained), and some explanations are not very clear (e.g. "We also remove multi-valued relations" in Line 194 is not clear).

Questions for authors
  • In equation (2) you describe the construction of the plausible but false statements. Did you run a check on each of these statements that it is actually false (e.g. using a strong LLM judge)? I saw that you did an alias screening but maybe there could be other ways in which the LLM generates a statement that is not false, i.e. either true or without any truth-value? If you did not check that each plausible statement is clearly false, please include such a check in the rebuttal.
  • What is the saliency of the plausibility direction compared to the truth direction? I.e. what is ∥μT−μR∥2\lVert \mu_T - \mu_R \rVert_2 compared to ∥μP−μR∥2\lVert \mu_P - \mu_R \rVert_2?
Reviewer_y1bHOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors argue that linear truth probes trained on simple true/false statement datasets are confounded by factual plausibility. The authors construct plausible falsehoods using language models' own probability distribution. They show that a difference-of-means probe direction fitted on plausible and random falsehoods aligns well with the truth probe direction and recovers most of its performance. They also show that using plausible falsehoods instead of random ones lowers truth probe performance. The paper proposes a local contrastive readout, which compares the probe score on a statement with its most plausible alternative, and shows improved performance on test set.

Critical strengths
  • The problem of addressing probe picking up on confounds is important.
  • The plausible falsehood construction is a reasonable test, though prior work has proposed similar perplexity-based negative sampling to construct plausible falsehoods (Servedio et al. 2025).
  • The authors did careful screening and filtering of the datasets. The results include six model families.
Critical weaknesses
  • The plausibility confound is not new. Servedio et al. (2025) show that plausible falsehoods lower probe test performance and Cheang et al. (2025) show that plausible hallucinations overlap with true facts in hidden-state space. Neither is cited, and it seems that the new thing here is that the false-only direction aligns well with the truth direction.
  • The probe direction projection analysis is unnatural. The most natural experiment is to train a probe with both random and plausible falsehoods as negatives and evaluate on T/R and T/P. This reviewer expects an LR probe trained on it to still perform well on both test sets.
  • The authors exclude ranking-ambiguous samples from the test set. These are exactly the hallucination-like cases the paper says matter most. The authors should report all main results with these samples included, or show both filtered and unfiltered splits.
  • A 0.5 threshold on a logistic probe is not how probes are deployed. The standard protocol fixes FPR (e.g., 1% or 5%) on a control set (e.g. alpaca dataset) and reports accuracy at that threshold. Table 2 should report this for both the logistic and the mass-mean probe, along with TPR at fixed FPR on both R and P negatives. Also, the use of difference-of-means vs. logistic regression is not consistent throughout the paper. This reviewer would suggest reporting results on both probes under the same metrics.
  • Evidence in realistic and on-policy settings is limited. The local contrastive probe requires an answer slot and a closed candidate pool and is not tested on on-policy data. Since on-policy data is already highly plausible according to the model, the proposed method is very likely to fail.

Reference:

  • Servedio, G., De Bellis, A., Di Palma, D., Anelli, V. W., & Di Noia, T. (2025, July). Are the hidden states hiding something? testing the limits of factuality-encoding capabilities in llms. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 6089-6104).
  • Cheang, C. S., Chan, H. P., Zhang, W., & Deng, Y. (2026, July). Do LLMs Really Know What They Don’t Know? Internal States Mainly Reflect Knowledge Recall Rather Than Truthfulness. In Findings of the Association for Computational Linguistics: ACL 2026 (pp. 713-730).
Questions for authors
  • What is the performance (AUROC) of a simple likelihood/perplexity classifier on the tested datasets?
  • How does a probe trained with both R and P as negative samples perform on T/R and T/P?
  • What do the main results look like when ranking-ambiguous samples are included?
  • Can you show examples of constructed plausible falsehoods for each dataset?
Reviewer_jAsGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies linear truth probes, which read whether a statement is true from an LLM's hidden states. The authors focus on the false statements used to fit and evaluate these probes, which in standard benchmarks are usually random and unrelated to the subject, and are therefore much easier to reject than the errors models actually make. They construct more plausible false statements by replacing the answer with the wrong candidate that the model itself finds most likely, while keeping the rest of the statement fixed. Using these statements, they show that a direction fitted only from plausible versus random false statements closely aligns with the standard truth direction, and that truth probes lose accuracy and accept many more false statements as true when evaluated on plausible negatives. They also propose a local contrastive readout, which judges each statement relative to its most likely alternative answer and reduces these false positives on datasets with an identifiable answer slot. Experiments on six models and six datasets, including long-form generated text, support that plausible negatives expose errors of truth probes that evaluation with random negatives misses.

Critical strengths

The paper identifies a real weakness of current truth-probe benchmarks and offers a practical way to test it.

The false statements in standard benchmarks are mostly random, while the errors that matter in practice are plausible ones. By keeping the subject, relation, and template fixed and replacing only the answer with the model's most likely wrong candidate, the paper builds harder negatives that isolate this difference. With these negatives, standard truth probes that look accurate and well calibrated on random negatives accept plausible false statements as true far more often, consistently across six models, and a similar pattern appears on real generated spans. This gives a concrete reason to include plausible negatives and fixed-threshold metrics when evaluating truth probes.

Critical weaknesses

While the paper raises an important problem, I have three concerns about its evaluation.

  1. The evaluation leaves out the facts the model does not know, which is where hallucinations occur.

The paper flags an item as ranking-ambiguous when the model assigns a higher likelihood to a wrong candidate than to the true answer, and excludes these items from the primary held-out evaluation. This removes 23% of CounterFact and 41% of Companies. The remaining items are those for which the model already ranks the true answer first, that is, facts the model knows. Hallucinations, however, arise mainly on facts the model does not know, so the setting most relevant to the paper's motivation is left out of the evaluation.

  1. The local contrastive readout is not compared with the baselines its design calls for.

The readout is compared only with global thresholds applied to the same probe, and two natural baselines are missing.

(1) A likelihood-only rule. For a false input, the readout compares it with the model's most likely alternative answer. When the model knows the fact, this alternative is the true statement, and a rule that uses only the model's likelihood ranking, labeling a statement true when its answer is the model's top candidate, would already classify every statement correctly without any probe. This is the case for all items in Cities and sp en trans, where no item is ranking-ambiguous and where the readout reaches nearly perfect results. When the model does not know the fact, the alternative is itself false, and the readout compares two false statements, so it is unclear whether it still helps. Results with this baseline, and on the ranking-ambiguous items, are needed to show that the probe contributes to the improvement.

(2) A probe trained with plausible negatives. The paper's own finding suggests a direct fix, which is to include plausible false statements when fitting the probe. The readout is motivated as working without retraining, but without this comparison it is unclear how much it gains over the simpler fix.

  1. The alignment between the two directions may partly follow from their shared reference.

Both directions are fitted relative to the same random false statements, since the truth direction is the mean of true minus random false and the plausibility direction is the mean of plausible false minus random false. True and plausible false statements both fit their context, while random false statements do not. If this difference drives both directions, their cosine similarity and the false-only direction's AUC will be high by construction, and the result would show that random negatives are easy to separate rather than that the truth direction encodes plausibility. The permutation null partitions the same false statements at random and does not control for this shared reference. A comparison that does not share the reference, for example the alignment between a direction fitted from true versus plausible false statements and the plausibility direction, would help separate these two explanations.

Questions for authors
  1. (Ranking-ambiguous items) How do the standard truth probes and the local contrastive readout perform on the ranking-ambiguous items? Is the ranking-ambiguity filter also applied to the evaluations in Sections 4.2 and 4.3?
  2. (Baselines) How do a rule that uses only the model's likelihood ranking over the candidates and a probe trained with plausible negatives perform in the Table 3 settings? Which model provides the likelihood used to select the alternative statement in the readout?
  3. (Direction construction) Could you report the alignment and AUC results using a direction that does not share the random false reference, for example a truth direction fitted from true versus plausible false statements?

In addition, although not a question, I would like to note one point about the presentation. The local contrastive readout is listed as a main contribution, but the paper has no dedicated method section. The readout is introduced for the first time inside the results (Section 4.3) and evaluated with a single table, while the definitions of the truth and plausibility directions appear in the experimental setup. Presenting the readout in its own method section, with its motivation, its assumptions, and the information it uses at test time, would make the contribution easier to identify and evaluate. Section 3.2 also interleaves the construction of plausible false statements, the data quality controls (ranking ambiguity, multi-valued relations, and alias screening), and the definitions of the probes. Separating the method from the experimental setup, and moving the data quality controls after the core definitions, would make the paper considerably easier to follow.

2 comments

Update: CounterFact plausible falsehoods verified by two judges

Following Reviewer fcMy's suggestion, we re-audited the mined plausible falsehoods of CounterFact with two strong judges, Claude Opus 5.5 and GPT-6.1, and kept only the statements that both label FALSE (see our response to Reviewer fcMy for the audit). We re-ran the comparison of decision rules, which is the most important set of results in our responses, on this verified subset. The same filter is applied to the training negatives of the plausible- and mixed-negative probes, while the original true and random false statements are unchanged. The other three datasets are not changed.

As shown in the updated Tables 4a/4b below, the conclusions do not change. On CounterFact, the local contrastive readout remains comparable to the likelihood-only rule on T/P (F1 0.811 vs. 0.813) and better than all three global-threshold baselines (0.750-0.768). Averaged over the four datasets, the readout still attains the highest F1 on T/P among all the rules compared (0.910 vs. 0.862-0.883). Due to time constraints, the other results that involve the CounterFact plausible-false subset will be updated in the revised paper, but we believe the updates above are sufficient to support the core claims of the paper.

Table 4a (updated). Average performance of the decision rules over the four datasets (CounterFact, Cities, sp_en_trans, and Companies). Each entry is shown as excluding → including ranking-ambiguous items. All rules except likelihood-only are evaluated at the same TPR, matched to the local contrastive readout. CounterFact plausible statements are kept only if both judges label them FALSE.

Decision ruleTPRFPR_RFPR_PF1 T/RF1 T/PF1 pooled
Global threshold, matched TPR0.941 → 0.8900.088 → 0.0920.200 → 0.2080.929 → 0.8970.892 → 0.8620.861 → 0.831
Plausible-negative probe, matched TPR0.941 → 0.8900.202 → 0.1770.197 → 0.1610.886 → 0.8660.894 → 0.8780.830 → 0.819
Mixed-negative probe, matched TPR0.941 → 0.8900.112 → 0.1020.176 → 0.1690.919 → 0.8930.900 → 0.8750.861 → 0.838
Likelihood-only (no probe)1.000 → 0.8730.000 → 0.0000.000 → 0.1161.000 → 0.9271.000 → 0.8831.000 → 0.883
Local contrastive0.941 → 0.8900.020 → 0.0380.046 → 0.0760.959 → 0.9210.951 → 0.9100.942 → 0.894

Table 4b (partially updated). Only CounterFact rows. The rows for Cities, sp_en_trans and Companies are unchanged.

DatasetDecision ruleTPRFPR_RFPR_PF1 T/RF1 T/PF1 pooled
CounterFactGlobal threshold, matched TPR0.860 → 0.7650.167 → 0.1740.385 → 0.4030.848 → 0.7890.804 → 0.7500.746 → 0.691
CounterFactPlausible-negative probe, matched TPR0.860 → 0.7650.395 → 0.3610.393 → 0.3360.763 → 0.7200.802 → 0.7670.678 → 0.650
CounterFactMixed-negative probe, matched TPR0.860 → 0.7650.184 → 0.1840.327 → 0.3310.841 → 0.7850.820 → 0.7680.754 → 0.704
CounterFactLikelihood-only (no probe)1.000 → 0.7710.000 → 0.0000.000 → 0.1841.000 → 0.8711.000 → 0.8131.000 → 0.813
CounterFactLocal contrastive0.860 → 0.7650.034 → 0.0680.116 → 0.1770.908 → 0.8350.884 → 0.8110.869 → 0.783

I thank the authors for the extensive additiional experiments and corresponding detailed responses.

In my review, I raised concerns about the exclusion of ranking-ambiguous items, the missing baselines for the local contrastive readout, and the shared random false reference in the alignment analysis. The new results and analyses on the comp arison with probes trained with plausible negatives and the new alignment results address my concerns on these two points.

However, new results show that both the standard truth probes and the proposed method perform close to random guessing on the ranking-ambiguous subset, which confirms my original concern. This subset consists of facts the model does not know, which is where halucinations mainly occur, so the proposed method appears not to work in the setting most relevant to the paper's motivation.

Overall, I will keep my score of 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.