acceptodds
Under review as a conference paper at ICLR 2027

Reading Between the Logits: Uncovering Vocabulary-Grounded Belief in LLMs

Abstract

Understanding the internal representations guiding an LLM's judgement, particularly when these diverge from its verbalised responses, is important for interpretability, reliability and safety. Existing approaches typically extract such signals using supervised probes, which require labelled data and training for each new setting. Lens methods, such as the logit lens and J-lens, instead project hidden states into vocabulary space, providing readouts of intermediate model representations. We introduce an unsupervised method that clusters vocabulary-level patterns in lens readouts to recover concepts: interpretable groups of tokens that express the LLM's judgement of a factual claim, such as whether it treats the claim as true, false or uncertain. These concepts generalise across prompt styles and model families, and exchanging their directions flips the model's verdict across claims. From these concepts, we compute a training-free signal that approaches the performance of supervised linear probes and remains comparatively stable when a user's conflicting belief alters the model's verbalised judgement. Finally, a complementary topic modelling analysis reveals readout patterns related to truth, falsity, misconception and contested claims, and how these develop across layers. These patterns are already present in base models, but become aligned with verbalised judgements only after instruction tuning. Together, concepts and readout patterns give an unsupervised, interpretable and inexpensive account of the representations behind a model's judgements, and of when these diverge from what it verbalises.

Primary area:
interpretability and explainable AI
Keywords:
large language models, mechanistic interpretability, linear probes, logit lens, jacobian lens, unsupervised learning, topic modelling, representation analysis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QvmPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

This paper studies how a language model's internal judgment of a factual claim relates to its final answer. It groups words with similar patterns in intermediate readouts and uses acceptance and rejection groups to score claims. The authors test these groups through interventions, misleading user prompts, and comparisons with supervised probes. A separate analysis tracks judgment-related patterns across layers and compares base and instruction-tuned models.

Critical strengths
  1. The authors test the approach across several model families and a varied set of datasets and tasks. The ablations, intervention studies, and transfer experiments provide a substantial empirical basis for the work.
  2. The authors provide strong interventional evidence that the recovered concepts are causally involved in the model’s verdict, rather than merely correlated with it. In the tested prompts, layers, and model families, swapping opposite-polarity concept directions reliably flips the model’s judgment, whereas same-polarity and random swaps have negligible effect. The effect also survives removal of answer tokens from the clusters, which strengthens the case that the recovered directions capture more than surface answer tokens.
  3. The authors offer a practical, training-free, and interpretable readout of model belief. Their concept clusters are recovered directly in vocabulary space and require no separately trained supervised classifier, and the resulting acceptance margin approaches supervised probes within a task. Under user-pressure/sycophancy settings, this margin can preserve a correctness signal even when the model’s verbalized answer is swayed, making it valuable for detecting belief–output divergence.
Critical weaknesses
  1. The paper provides a useful way to organize and inspect internal representations, but I am less convinced by its methodological novelty. It builds on J-lens and established algorithms such as PCA and HDBSCAN, while hidden-state readouts and interventions that flip answers have already been demonstrated in prior lens work. For the clustering approach, the key issue is what automatically discovered token groups add beyond individual tokens. Although the yes/no interventions show some benefit, the paper does not sufficiently isolate the practical value gained from automatic clustering.
  2. I am not convinced that the comparison supports the claim that internal judgments feed into free-form answers 'only' in the instruction-tuned model (contribution 4 in Introduction). The claim should be limited to the tested settings or supported by a more direct comparison. The instruction-tuned OLMo model receives a bare claim in chat, whereas the base model is prompted to choose true or false. Although the authors explain this choice, it remains unclear whether the difference would persist under comparable prompts that elicit free-form responses.
  3. The experiments cover varied factual judgment and question-answering settings, but evidence for cluster interventions still mainly concerns claim verdicts. Their value for multi-step mathematical reasoning, code generation, planning, or long-form generation is less clear, and interventions during generated chain-of-thought reasoning are not directly tested. I would welcome discussion of these limits to clarify the findings' scope.
Questions for authors
  1. Which contributions are new relative to J-lens and related work, and which extend existing methods? Please clarify what automatic token clustering adds beyond individual tokens, and discuss its potential use in practical settings such as safety monitoring and intervention. It would help to distinguish demonstrated benefits from possible future applications.
  2. Could the authors clarify the scope of 'only' in Contribution 4? How do the results distinguish differences associated with instruction tuning from those associated with the different response formats? Please support the claim under more comparable prompts that permit free-form answers from both models, or restrict it to the tested settings.
3 comments

We thank the reviewer for acknowledging the breadth of our evaluation, the interventional evidence that the recovered concepts are causally involved in the verdict, and the value of the acceptance margin for detecting belief-output divergence. The reviewer’s questions on what clusters add and on Contribution 4 led to new experiments (G2 and G3 of the General Response), which clarified both. We address the concerns below.

W1, Q1. Our contribution is a way to extract more from existing lens readouts without supervision. J-lens readouts, PCA, HDBSCAN and NMF are existing methods. What’s new:

  • the unsupervised pipeline that recovers groups of tokens expressing the model’s judgement, without labels or hand-picked tokens;
  • the acceptance margin, a training-free score that approaches supervised probes (AUROC 0.94 vs 0.99, App. K) and stays comparatively stable under sycophantic pressure (Sec. 6.2);
  • the NMF analysis of readout patterns across layers and models (Sec. 6.3).

We also extended the J-lens swap, from single tokens to clusters, with centring and norm matching (Sec. 4.3, App. H.3).

Prior lens work reads a judgement from a single token or phrase chosen by hand for each judgement, prompt and model, while supervised probes need labels and training for each setting. Our concepts need neither.

On what clusters add beyond individual tokens (G2), single tokens often work as well, but clusters remove the need to hand-pick them, which matters when the obvious choice is not the best. On the yes/no prompt, the clusters flip 0.810 of verdicts against 0.560 for the prompt’s own pair _yes↔_no (Tab. 1), and, in a new experiment, still 0.777 after we remove every form of yes, no, true and false, so the gain comes from vocabulary that the clustering found and that is not in the prompt.

W2, Q2. We agree, and will restrict the claim. We extended the analysis of Sec. 6.3 (G3) to a second Base/Instruct pair, Llama-3.1-8B, alongside OLMo-3-7B and Qwen3-8B (Instruct only). As the reviewer suggested, we compared Base and Instruct models on 6 free-form templates. In both, mid-layer truth and falsity registers correlate with late-layer opening or verdict topics and predict the late-layer readouts: for OLMo-3-7B, r = 0.90-0.95 (Base) and 0.91-0.98 (Instruct) on 5 of the 6 templates. The exception is the bare-claim template that Sec. 6.3 used for the Instruct models: the Base model forms no such registers on it, which matches App. C.3, where the Base model needed a forced choice to give a verdict. For Llama-3.1-8B, the Instruct model is stronger on 3 of 6 templates and significantly weaker on 1. The “only” in Contribution 4 therefore does not hold. We will add these results to a new appendix and revise the abstract, Contribution 4 and Sec. 6.3. Due to lack of space, the exact changes are in reply W3 to Reviewer 9G9v.

W3. We agree, and will add to the Limitations (Sec. 7): “Our experiments are limited to claim verification. Whether the readouts form coherent token groups in more complex settings, such as open-ended generation and multi-step reasoning, remains open.” We chose claim verification because there an internal judgement can be compared with the truth, at a known position, and with existing probe and sycophancy work (App. J, K). Extending it is not a straightforward application: with chain-of-thought, for instance, it is unclear which intermediate tokens carry a judgement. As a first step beyond single claims, in a new experiment where a fixed claim is preceded by evidence that supports or contradicts it, the acceptance margin is higher with supporting evidence in 199 of 200 pairs (see reply W1 to Reviewer w1sW).

Q1 (practical use). We see at least 4 applications, based on different results from the paper.

  • Monitoring. The acceptance margin detects when a model’s verbalised answer diverges from its internal judgement under sycophantic pressure (Sec. 6.2), so it could serve as a cheap, label-free monitor during deployment.
  • Cheaper probing. The margin approaches supervised probes in predicting whether the model will accept its own answer (App. K), so a probe is needed only where higher accuracy matters.
  • Half-truths. The partial cluster separates partially true from false claims (App. I), a possible half-truth monitor.
  • Intervention. Swaps flip the verdict, and steering along the uncertain cluster makes the model less certain rather than changing its answer (Sec. 6.1). This suggests interventions that make a model express uncertainty instead of a confident wrong answer.

We thank the reviewer again for their suggestions. We hope that these clarifications and new results address their concerns about novelty, the instruction-tuning claim and the scope of our findings. If so, we kindly ask the reviewer to consider updating their score to reflect this, and we are happy to address any further questions.

We wanted to let the reviewer know that we have uploaded a new version of our work, with all changes highlighted in green. The changes fall into two groups: new experiments prompted by the reviewer's questions on what clusters add and on Contribution 4, and updated wording that restricts the instruction-tuning claim and clarifies the scope of our findings.

W1, Q1 (what clusters add beyond individual tokens).

  • Sec. 6.1, Exchanging opposite polarities flips the verdict (l. 358-360). We added: "Even after every form of yes, no, true and false is removed from the clusters, the swap still flips 0.777 of verdicts, so the gain comes from vocabulary that the clustering found and that is not in the prompt."
  • Tab. 1 (l. 379, 387). The yes/no block has a new row, confirm∘^\circ ↔\leftrightarrow reject∘^\circ (0.777, S/R 0.993/0.560), and the caption now defines ∘^\circ as "all forms of true, false, yes and no" removed.
  • Sec. H.12 (Clusters Without Answer Forms), new appendix. It reports this ablation, isolating the practical value of automatic clustering that the reviewer asked about.

W2, Q2 (scope of "only" in Contribution 4).

  • Contribution 4 (l. 106-107): "Both base and instruction-tuned models track truthfulness, and under most free-form prompts this signal predicts the late-layer readouts in both".
  • Abstract (l. 025-026): "The prominence of the patterns in the middle layers predicts the model's late-layer readouts."
  • Fig. 1 caption (l. 072): "falsity patterns predict the late-layer readout".
  • Sec. 6.3. The title (l. 449) and paragraph titles (l. 455, 462) no longer attribute the topic structure to instruction tuning, and the Tab. 2 (l. 463-464) and Fig. 6 (l. 444) captions are restricted to the templates of Tab. 6. The section points to the extended analysis in Sec. L.9 (l. 459-460), and closes with (l. 477-479): "In OLMo, instruction tuning therefore does not create the verdict, but computes it earlier. Under free-form prompts, the base model also routes it to the reply."
  • Sec. C.3 (l. 1153-1154). The base model gives no verdict without a forced choice "under the bare-claim template; under the other free-form templates it does."
  • Sec. 7 and conclusion. The instruction-tuning results now "use two model pairs" (l. 503), and the conclusion now says that "base models can already form this verdict in the middle layers, where it predicts the late-layer readouts. Instruction tuning does not create it, but makes it form more reliably across prompts" (l. 509-511).
  • Sec. L.9 (Readout Patterns Across Templates and Models), new appendix. It provides the more direct comparison the reviewer requested: base and instruction-tuned models from two families, under comparable free-form and closed-form prompts. It also explains our choice of each model's native format.

W3 (scope beyond claim verdicts).

  • Sec. 7 (l. 496-500). We added the promised limitation: "Second, our experiments are limited to claim verification. …\ldots Whether they also appear during unprompted claim processing, or form coherent token groups in more complex settings such as open-ended generation and multi-step reasoning, remains open."
  • Sec. 6.1 (l. 374-375) and Sec. H.11. A new experiment with evidence given in context provides a first step beyond single claims, as discussed in our response.

The new appendices currently report the results of our response. The full descriptions and result tables will be ready for the camera-ready version.

We thank the reviewer again for their suggestions, which led us to improve both the case for clustering and the instruction-tuning claims. Do the new ablation in Sec. H.12, which isolates what automatic clustering adds beyond the prompt's own answer words, and the revised scope of Contribution 4, supported by the comparison under free-form prompts in Sec. L.9, address the reviewer's concerns? If so, we would be grateful if the reviewer would consider updating their score. Otherwise, we would welcome suggestions for what would make the paper stronger.

I appreciate the detailed explanations provided by the authors.

The authors added experiments and specifically revised the paper to fix the minor issues mentioned. I believe the concerns raised in the original review have been well addressed. I therefore updated my score accordingly.

Reviewer_9G9vOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper clusters concepts such as acceptance, rejection, and uncertainty from the vocabulary reading of the LLM's intermediate layer, and constructs an acceptance margin measurement model to evaluate the internal judgment of the statements. The author discovers the concepts on FEVER and intervenes, evaluates the internal signals in the conformity tests based on TriviaQA and TruthfulQA, and compares with the supervised probes on TriviaQA and NQOpen. The experiments show that exchanging the directions of acceptance and rejection can significantly reverse the model's judgment, and some effects can be transferred across prompts and models; in the conformity scenarios, internal reading often retains the correct information that is not reflected in the output, providing ideas for model monitoring and error detection.

Critical strengths
  1. The method is intuitive and easy to understand: directly identify the concepts at the lexical level, without the need for training task classifiers; the centralized and clustering design also has ablation support.

  2. The intervention experiment is solid: it includes random directions, polarity exchanges, and removal of answer words as controls, and verified the cross-prompt and cross-model effects.

  3. The experiments are comprehensive: combined with accommodating behavior, supervised probes, and cross-dataset transfer, it demonstrates the application and boundaries of internal judgment signals.

Critical weaknesses
  1. Intervention can reverse the judgment, but it cannot yet distinguish whether it has changed the factual judgment or merely shifted the response tendency; the interpretation of "belief" still requires stronger verification.

  2. The advantage of the relatively simple baseline is limited: word reading and swapping have achieved similar effects in some experiments, and the unsupervised scheme still has a significant gap compared to the supervised probe.

  3. The instruction fine-tuning analysis used different base/instruct prompt formats, and only involved a pair of models, making it difficult to clearly attribute the observed differences to the fine-tuning.

Questions for authors

see weakness

3 comments

We thank the reviewer for finding our method intuitive, and our experiments comprehensive. The reviewer’s remarks on response tendency and on clusters vs single tokens led to new experiments (G1 and G2 of the General Response), which refined our understanding of what the clusters carry and when they help. We address the concerns below.

W1. Our results indicate that the swaps change the factual judgement, not only the response tendency. In the paper, swaps flip verdicts in both directions, same-polarity swaps have almost no effect although the clusters are geometrically similar (Tab. 1, App. H.9), the effect holds across prompts with different answer words (Tab. 1), and steering along uncertain raises the model’s uncertainty rather than only the probability of the word (Sec. 6.1).

We agree that the belief interpretation needed stronger evidence, and this remark led to new tests (G1). For 1,086 FEVER claims, we compared each claim with a flipped version that differs in one fact and has the opposite truth value. Making the claim and its flip identical along the true and false cluster directions, leaving the rest of the hidden state unchanged, shrinks the difference between the model’s verdicts on them by 25.8% [25.4%, 26.2%], against less than 1% for random directions. A shift in response tendency would move both verdicts alike and leave this difference unchanged. This indicates that the model uses these directions for part of its factual judgement. We will add the experiments of G1 to a new appendix, and the following limitation to Sec. 7: “Fixing the concept cluster directions removes about a quarter of the gap between the model’s verdicts on true and false claims. The clusters therefore carry only part of the model’s true/false judgement.”

W2. We agree that single tokens often work as well as clusters (Tab. 1, App. G.1, H.6). Clusters help when the obvious token is not the best one, and they remove the need to choose it by hand: on the yes/no prompt, the prompt’s own pair _yes↔_no flips 0.560 of verdicts, against 0.810 for the clusters (Tab. 1). To further understand the usefulness of clusters, we developed a new experiment in which we removed every form of yes, no, true and false from the yes/no clusters. With the remaining forms, the resulting cluster still flips 0.777, so the gain comes from vocabulary that the clustering found and that is not in the prompt (G2).

On the probe gap, we do not claim that the margin outperforms supervised probes, only that it recovers much of their signal (AUROC 0.94 vs 0.99, App. K) with no training or per-claim labels. It is a cheap first reading, after which a probe can be trained if accuracy matters most.

W3. We agree. We extended the analysis of Sec. 6.3 (G3) to a second Base/Instruct pair, Llama-3.1-8B, alongside OLMo-3-7B and Qwen3-8B (Instruct only), and to 12 templates (6 free-form, 6 closed-form). Topic features match or beat SVD features in 59 of 60 model-template combinations, Base models included, and the Instruct advantage is small or absent, so these properties are not unique to instruction-tuned models. We will add these results to a new appendix, and revise the paper as follows:

  • Abstract (l. 025-027): “These patterns … only after instruction tuning.” → “The prominence of the patterns in the middle layers predicts the model’s late-layer readouts.”
  • Fig. 1 caption (l. 072): “instruction tuning routes” → “models route”
  • Contribution 4 (l. 106-107): “…but only in the instruction-tuned model does it feed into a free-form verbalised answer” → “…and under most free-form prompts this signal predicts the late-layer readouts in both”
  • Sec. 6.3 title (l. 412) → “Truth-Related Structure Is Organised into Vocabulary Topics”
  • l. 419: “Instruction tuning organises existing information into topics.” → “Topics carry the category information in base and instruction-tuned models.”
  • Tab. 2 caption (l. 420): “…with instruction tuning.” → “…with instruction tuning under the templates of Tab. 6.”
  • l. 426-427: “…but only the instruction-tuned model expresses it as topics.” → “…but the instruction-tuned model expresses it more clearly as topics under these templates.”
  • l. 449: “Instruction tuning connects truth and falsity topics to the answer.” → “Truth and falsity topics feed into the answer.”
  • l. 456-457: “Instruction tuning therefore does not create the verdict: …” → “In OLMo, instruction tuning therefore does not create the verdict, but computes it earlier. Under free-form prompts, the base model also routes it to the reply.”

We thank the reviewer again for their suggestions. We hope that these clarifications and new results address their concerns about response tendency, the value of clusters over single tokens, and the instruction-tuning analysis. If so, we kindly ask the reviewer to consider updating their score to reflect this, and we are happy to address any further questions.

We wanted to let the reviewer know that we have uploaded a new version of our work, with all changes highlighted in green. The changes fall into two groups: new experiments prompted by the reviewer's comments on response tendency, clusters versus single tokens and the instruction-tuning comparison, and updated wording that restricts our claims to what the results support. In the paper, we call the one-fact variant of a claim a counterfactual rather than a flip, so that it is not confused with the flip rates of our swap experiments.

W1 (factual judgement vs response tendency).

  • Sec. 6.1 (l. 362-375), new paragraph The directions follow the fact. It addresses the reviewer's concern directly with a test that a shift in response tendency cannot pass, and states why such a shift would leave the measured gap unchanged.
  • Sec. H.11 ( Does the Representation Follow the Fact? ), new appendix. It presents the setup and full results behind this paragraph.
  • Sec. 7 (l. 494-496). We added the promised limitation: "the clusters carry only part of the model's true/false judgement, since fixing their directions removes about a quarter of the gap between the model's verdicts on true and false claims."

W2 (clusters vs single tokens).

  • Sec. 6.1, Exchanging opposite polarities flips the verdict (l. 358-360). We added: "Even after every form of yes, no, true and false is removed from the clusters, the swap still flips 0.777 of verdicts, so the gain comes from vocabulary that the clustering found and that is not in the prompt."
  • Tab. 1 (l. 379, 387). The yes/no block has a new row, confirm∘^\circ ↔\leftrightarrow reject∘^\circ (0.777, S/R 0.993/0.560), and the caption now defines ∘^\circ as "all forms of true, false, yes and no" removed.
  • Sec. H.12 (Clusters Without Answer Forms), new appendix. It reports this ablation, isolating what the clustering adds beyond the prompt's own answer words.

W3 (prompt formats and a single model pair).

  • Abstract (l. 025-026): "The prominence of the patterns in the middle layers predicts the model's late-layer readouts."
  • Fig. 1 caption (l. 072): "falsity patterns predict the late-layer readout".
  • Contribution 4 (l. 106-107): "Both base and instruction-tuned models track truthfulness, and under most free-form prompts this signal predicts the late-layer readouts in both".
  • Sec. 5, Readout patterns. Readout patterns now "show how it develops across layers in base and instruction-tuned models" (l. 334-335), and a new sentence introduces the repeated analysis across 12 templates and a second model pair (l. 343-346).
  • Sec. 6.3. The title is now "Truth-Related Structure Is Organised into Vocabulary Topics" (l. 449), and the paragraph titles are now "Topics carry the category information in base and instruction-tuned models" (l. 455) and "Truth and falsity topics feed into the answer" (l. 462). The Tab. 2 (l. 463-464) and Fig. 6 (l. 444) captions are restricted to the templates of Tab. 6. The text now says that the instruction-tuned model "expresses it more clearly as topics under these templates", points to the extended analysis in Sec. L.9 (l. 459-460), and closes with (l. 477-479): "In OLMo, instruction tuning therefore does not create the verdict, but computes it earlier. Under free-form prompts, the base model also routes it to the reply."
  • Sec. 7 and conclusion. The instruction-tuning results now "use two model pairs" (l. 503), and the conclusion now says that "base models can already form this verdict in the middle layers, where it predicts the late-layer readouts. Instruction tuning does not create it, but makes it form more reliably across prompts" (l. 509-511).
  • Sec. L.9 (Readout Patterns Across Templates and Models), new appendix. It answers the reviewer's concern about formats and model pairs by repeating the analysis of Sec. 6.3 with Llama-3.1-8B Base/Instruct and 12 templates. It also explains why each model reads the templates in its native format, and what that choice implies for attributing differences to instruction tuning.
  • Other appendices. Sec. C.3, L.1, L.5, L.7 and Tab. 3 are updated for consistency with these changes.

The new appendices currently report the results of our response. The full descriptions and result tables will be ready for the camera-ready version.

We thank the reviewer again for their suggestions, which substantially improved both the belief interpretation and the instruction-tuning analysis. Do the new counterfactual test (Sec. 6.1, Sec. H.11) and the revised instruction-tuning claims (Sec. 6.3, Sec. L.9) resolve the reviewer's concerns on response tendency and on the base/instruct comparison? If so, we would be grateful if the reviewer would consider updating their score. If any concern remains, we would welcome suggestions on how to further improve the paper.

Thanks for your rebuttal! I will keep my positive score.

Reviewer_w1sWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Addresses question of how to examine a model’s internal judgment in unsupervised way (without probes). The paper decodes intermediate hidden states into scores over vocabulary words and groups words whose scores vary together into concepts, in particular "acceptance," "rejection," and "uncertainty", and uses these groups to measure the model’s stance. Authors find that the internal score can favor correct statements even when the final response agrees with a user’s false suggestion. Swapping the directions in hidden states changes the model’s verdict; viewing the swap as a causal intervention then suggests these representations causally affect its answers rather than merely correlate with them. Authors show recurring patterns across layers. Also, authors find internal signals distinguishing true from false claims in both pretrained and instruction-tuned models, but where in the instruction-tuned model these signals appear earlier and are more closely associated with whether the final reply affirms or corrects the claim -- suggests that instruction tuning doesn't inject the model with information on how to create an answer, but rather changes how the model turns information it already has into an answer.

Critical strengths
  1. Swapping approach (with random controls) tests causal influence, not just correlation.
  2. Provides insight into sycophancy - under conditions of sycophancy conditions, internal scores favor correct statements even when the final answer follows the user’s false suggestion -- useful for figuring out when the model starts to diverge into sycophancy.
Critical weaknesses

Fails to address reasoning. An internal signal saying “false” does not explain which evidence made the model judge the claim false. A model could reach the correct verdict for an irrelevant reason. The interventions demonstrate causal control, but do not establish whether the representation carries the influence of particular evidence input as it computes the answer. Stronger evidence would trace how changing relevant input facts changes the internal representation and, through it, the answer.

Questions for authors

Can you run some tests that evaluate whether the discovered representations carry information the model uses to reach its judgment, as opposed to simply provide a way to change which verdict it spits out?

3 comments

We thank the reviewer for recognising that our swaps test causal influence beyond mere correlation, and that our readouts can help detect when a model starts to diverge into sycophancy. We are grateful for the reviewer’s suggestion to trace how a changed fact reaches the answer, which directly drove the new experiments in G1 of the General Response. These new results have improved our understanding of what the concept clusters carry. We address the concerns below.

W1. The claims fed to the model come directly from FEVER and have no accompanying evidence, so the only input the model receives is the fact stated in the claim itself. We therefore tested whether the representation follows that fact (G1). For 1,086 FEVER claims, we compared each claim with a paraphrase (same fact, new wording) and a flip (one entity, date, number or attribute replaced, which reverses the truth value). Reading the hidden state along the true and false cluster directions, the side on which a claim falls (true or false) changes for 90.9% of flips, but only for 5.7% of paraphrases. Thus, the representation follows the relevant fact, not the wording.

To also test evidence given in context, we built 200 synthetic pairs in which a fixed claim is preceded by evidence that supports or contradicts it, differing in one fact (e.g. “Mira lives in the city where Orion is headquartered. Orion is headquartered in Rome / Berlin.” followed by the claim “Mira lives in Rome.”), with the Y/N verification protocol of App. K. On Qwen3-8B (logit lens, layer 18), the acceptance margin before the verification instruction is higher with supporting evidence in 199 of 200 pairs, and the final Y/N logits are correct in all 400 prompts, so the readout also follows the relevant evidence when it is given in context. We will add both experiments to a new appendix.

Q1. We ran new tests (see G1), and they show that the model uses the cluster directions for part of its verdict, so they are not only a way to change it. Following the reviewer’s suggestion, we compared each claim with its flip (W1): the two differ in one fact only, so the difference between the model’s verdicts on them measures the effect of that fact on the answer. We then made the claim and its flip identical along the true and false cluster directions, by setting the hidden state’s component along these directions to the same value in both, and left the rest of the hidden state unchanged. This shrinks the difference between the two verdicts by 25.8% [25.4%, 26.2%], against less than 1% when we do the same along random directions. If the model did not use the cluster directions, the difference would not change. The cluster directions are not the only carrier of the judgement: a direction computed with labels carries most of the signal, and the cluster directions read it through the small component they share with it. Interestingly, the acceptance margin tracks this direction closely (Pearson r = 0.979, G1). We will add the experiments of G1 to a new appendix, and the following limitation to Sec. 7: “Fixing the concept cluster directions removes about a quarter of the gap between the model’s verdicts on true and false claims. The clusters therefore carry only part of the model’s true/false judgement.”

We thank the reviewer again for their suggestions. We hope that the new experiments address their concern about whether the concept clusters carry information the model uses to reach its verdict. If so, we kindly ask the reviewer to consider updating their score to reflect this, and we are happy to address any further questions.

We wanted to let the reviewer know that we have uploaded a new version of our work, with all changes highlighted in green. The changes fall into two groups: new experiments prompted by the reviewer's suggestion to trace how a changed fact reaches the answer, and updated wording that reflects what these experiments show. In the paper, we call the one-fact variant of a claim a counterfactual rather than a flip, so that it is not confused with the flip rates of our swap experiments.

W1, Q1.

  • Sec. 6.1 (l. 362-375), new paragraph The directions follow the fact. Following the reviewer's suggestion, this paragraph brings the experiments of G1 and of our reply to W1 into the main text. It shows that the representation follows the fact stated in a claim rather than its wording, that the model uses the cluster directions for part of its verdict, that the acceptance margin tracks the main true/false signal in the hidden state, and that the readout follows evidence given in context.
  • Sec. H.11 ( Does the Representation Follow the Fact? ), new appendix. It presents the setup and results of both experiments (counterfactual/paraphrase and in-context evidence), so that the claims in Sec. 6.1 can be checked in detail.

Limitations (Sec. 7).

  • As promised, we added the limitation these experiments reveal (l. 494-496): "the clusters carry only part of the model's true/false judgement, since fixing their directions removes about a quarter of the gap between the model's verdicts on true and false claims."
  • The statement on belief criteria now reads that we "do not test coherence, test paraphrase invariance only on FEVER claims (Sec. H.11)" (l. 500-501).

The new appendix currently reports the results of our response. The full description and result tables will be ready for the camera-ready version.

We thank the reviewer again for the suggestion, which led to what we see as one of the most informative additions to the paper. Do the new paragraph in Sec. 6.1 and Sec. H.11 address the reviewer's question of whether the representations carry information the model uses to reach its verdict? If so, we would be grateful if the reviewer would consider updating their score. If not, we would welcome any further test the reviewer would find convincing, so that we can include it in the camera-ready version.

Thank you for the updated experiments and clarifications. I lean toward acceptance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment for All Reviewers

We thank all reviewers for their feedback, which led to new experiments we will add to the paper. G1 addresses Reviewers w1sW and 9G9v. G2 and G3 address 9G9v and QvmP.

G1: Do the concept clusters carry the model’s judgement, or only change the verdict?

To test this, we changed the fact in a claim and traced its effect. For 1,500 correctly classified FEVER test claims (Qwen3-8B, default prompt, 5 disjoint sets), we generated a paraphrase (same fact, new wording) and a flip (one entity, date, number or attribute replaced, no negation), e.g. “GLOW is a Netflix TV series” / “GLOW is a TV series on Netflix” / “GLOW is a BBC TV series”, and kept the 1,086 triples where the paraphrase preserves and the flip reverses the verdict. We use the coordinates our swaps exchange (Sec. 4.3): the hidden state along the true and false cluster directions, median-centred, at the layers and positions of Sec. 5. Values are means over the 5 sets, with 95% t-intervals.

  • The representation follows the fact over the wording: the difference of the two coordinates changes sign for 90.9% of flips, but only for 5.7% of paraphrases.
  • The model uses the representation: setting the two coordinates of a claim and its flip to the same value (the derivation median), leaving the rest of the hidden state unchanged, shrinks the gap between their soft verdicts by 25.8% [25.4%, 26.2%], against less than 1% for random or unrelated-cluster directions. If the model ignored the cluster directions, the gap would not change.
  • The clusters are not the only carrier of the judgement: we compute a “truth” direction as the difference between the mean hidden states of supported and refuted derivation claims. Since it contains the whole average difference between true and false states, fixing the hidden state’s coordinate along it to its median (layers 19-26) removes 91.1% of the gap. The truth direction is nearly orthogonal to the plane of the cluster directions, yet the acceptance margin closely tracks this coordinate (Pearson r = 0.979 over 2,000 test claims, layer 26). The cluster directions read it through the small component they share with the truth direction. Without this component, fixing them shrinks the gap by only 1.7%, and removing the truth direction from the hidden state stops the margin from separating the claims.

Together with the swaps (Tab. 1), these tests show that the model relies on the cluster directions for part of its verdict, and that the acceptance margin closely tracks the main true/false signal in the hidden state, without labels.

G2: What do clusters add beyond single tokens?

As the reviewers note, Tab. 1 and App. G.1 show that single tokens often work as well as clusters. We claim that clusters remove the need to hand-pick the token, which matters when the obvious choice is not the best: on the yes/no prompt, swapping _yes↔_no is less effective than swapping clusters (Tab. 1). In a new experiment, removing every form of yes, no, true and false from the yes/no clusters still flips 0.777 (vs 0.643 for _true↔_false, App. H.7), so the gain comes from vocabulary found by the clustering and absent from the prompt (confirm, verify, agree vs reject, deny, fail). Clusters also capture more than the verdict: steering along the uncertain cluster makes the model less certain, while the single token _uncertain mainly relabels the answer (Sec. 6.1). Finally, clusters reveal structure shared across model families: stance clusters found independently on 4 models from 3 families share 10 accept and 26 reject forms (App. G.3).

G3: Instruction tuning (Contribution 4)

We added Llama-3.1-8B Base/Instruct to the OLMo-3-7B pair and Qwen3-8B, used 12 prompt templates (6 free-form, 6 closed-form) and repeated the analysis of Sec. 6.3 (95% bootstrap CIs over 450 claim pairs).

  • Topic features match or beat SVD features in 59 of 60 model-template combinations for category prediction (mean accuracy 0.433 vs 0.378), and the topic pipeline also appears in both Base models.
  • Instruct models show a higher gain of topic over SVD features, but the difference is small in category prediction (0.017 [0.003, 0.031] for OLMo, 0.035 [0.022, 0.049] for Llama) and absent in error-type prediction for Llama (-0.025 [-0.050, 0.001]).
  • With free-form templates, Base models behave like Instruct models, with mid-layer truth and falsity registers correlating with the late-layer readout (OLMo, 5 of 6 templates: r = 0.90-0.95 Base, 0.91-0.98 Instruct). In Instruct models the signal is more concentrated in a single verdict topic (r = 0.96-0.99 vs 0.72-0.86 in Base). Llama shows the same tendency.

Thus, instruction tuning does not create the truth-tracking pipeline, but makes it more reliable across prompts and directs its output to the form the prompt calls for. The remaining findings of Sec. 6.3 (topic features beat SVD features, Base and Instruct models both have truth and falsity registers) hold across the 12 templates and 5 models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.