- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies how a language model's internal judgment of a factual claim relates to its final answer. It groups words with similar patterns in intermediate readouts and uses acceptance and rejection groups to score claims. The authors test these groups through interventions, misleading user prompts, and comparisons with supervised probes. A separate analysis tracks judgment-related patterns across layers and compares base and instruction-tuned models.
Critical strengths
- The authors test the approach across several model families and a varied set of datasets and tasks. The ablations, intervention studies, and transfer experiments provide a substantial empirical basis for the work.
- The authors provide strong interventional evidence that the recovered concepts are causally involved in the model’s verdict, rather than merely correlated with it. In the tested prompts, layers, and model families, swapping opposite-polarity concept directions reliably flips the model’s judgment, whereas same-polarity and random swaps have negligible effect. The effect also survives removal of answer tokens from the clusters, which strengthens the case that the recovered directions capture more than surface answer tokens.
- The authors offer a practical, training-free, and interpretable readout of model belief. Their concept clusters are recovered directly in vocabulary space and require no separately trained supervised classifier, and the resulting acceptance margin approaches supervised probes within a task. Under user-pressure/sycophancy settings, this margin can preserve a correctness signal even when the model’s verbalized answer is swayed, making it valuable for detecting belief–output divergence.
Critical weaknesses
- The paper provides a useful way to organize and inspect internal representations, but I am less convinced by its methodological novelty. It builds on J-lens and established algorithms such as PCA and HDBSCAN, while hidden-state readouts and interventions that flip answers have already been demonstrated in prior lens work. For the clustering approach, the key issue is what automatically discovered token groups add beyond individual tokens. Although the yes/no interventions show some benefit, the paper does not sufficiently isolate the practical value gained from automatic clustering.
- I am not convinced that the comparison supports the claim that internal judgments feed into free-form answers 'only' in the instruction-tuned model (contribution 4 in Introduction). The claim should be limited to the tested settings or supported by a more direct comparison. The instruction-tuned OLMo model receives a bare claim in chat, whereas the base model is prompted to choose true or false. Although the authors explain this choice, it remains unclear whether the difference would persist under comparable prompts that elicit free-form responses.
- The experiments cover varied factual judgment and question-answering settings, but evidence for cluster interventions still mainly concerns claim verdicts. Their value for multi-step mathematical reasoning, code generation, planning, or long-form generation is less clear, and interventions during generated chain-of-thought reasoning are not directly tested. I would welcome discussion of these limits to clarify the findings' scope.
Questions for authors
- Which contributions are new relative to J-lens and related work, and which extend existing methods? Please clarify what automatic token clustering adds beyond individual tokens, and discuss its potential use in practical settings such as safety monitoring and intervention. It would help to distinguish demonstrated benefits from possible future applications.
- Could the authors clarify the scope of 'only' in Contribution 4? How do the results distinguish differences associated with instruction tuning from those associated with the different response formats? Please support the claim under more comparable prompts that permit free-form answers from both models, or restrict it to the tested settings.
3 comments
We wanted to let the reviewer know that we have uploaded a new version of our work, with all changes highlighted in green. The changes fall into two groups: new experiments prompted by the reviewer's questions on what clusters add and on Contribution 4, and updated wording that restricts the instruction-tuning claim and clarifies the scope of our findings.
W1, Q1 (what clusters add beyond individual tokens).
- Sec. 6.1, Exchanging opposite polarities flips the verdict (l. 358-360). We added: "Even after every form of yes, no, true and false is removed from the clusters, the swap still flips 0.777 of verdicts, so the gain comes from vocabulary that the clustering found and that is not in the prompt."
- Tab. 1 (l. 379, 387). The yes/no block has a new row, confirm reject (0.777, S/R 0.993/0.560), and the caption now defines as "all forms of true, false, yes and no" removed.
- Sec. H.12 (Clusters Without Answer Forms), new appendix. It reports this ablation, isolating the practical value of automatic clustering that the reviewer asked about.
W2, Q2 (scope of "only" in Contribution 4).
- Contribution 4 (l. 106-107): "Both base and instruction-tuned models track truthfulness, and under most free-form prompts this signal predicts the late-layer readouts in both".
- Abstract (l. 025-026): "The prominence of the patterns in the middle layers predicts the model's late-layer readouts."
- Fig. 1 caption (l. 072): "falsity patterns predict the late-layer readout".
- Sec. 6.3. The title (l. 449) and paragraph titles (l. 455, 462) no longer attribute the topic structure to instruction tuning, and the Tab. 2 (l. 463-464) and Fig. 6 (l. 444) captions are restricted to the templates of Tab. 6. The section points to the extended analysis in Sec. L.9 (l. 459-460), and closes with (l. 477-479): "In OLMo, instruction tuning therefore does not create the verdict, but computes it earlier. Under free-form prompts, the base model also routes it to the reply."
- Sec. C.3 (l. 1153-1154). The base model gives no verdict without a forced choice "under the bare-claim template; under the other free-form templates it does."
- Sec. 7 and conclusion. The instruction-tuning results now "use two model pairs" (l. 503), and the conclusion now says that "base models can already form this verdict in the middle layers, where it predicts the late-layer readouts. Instruction tuning does not create it, but makes it form more reliably across prompts" (l. 509-511).
- Sec. L.9 (Readout Patterns Across Templates and Models), new appendix. It provides the more direct comparison the reviewer requested: base and instruction-tuned models from two families, under comparable free-form and closed-form prompts. It also explains our choice of each model's native format.
W3 (scope beyond claim verdicts).
- Sec. 7 (l. 496-500). We added the promised limitation: "Second, our experiments are limited to claim verification. Whether they also appear during unprompted claim processing, or form coherent token groups in more complex settings such as open-ended generation and multi-step reasoning, remains open."
- Sec. 6.1 (l. 374-375) and Sec. H.11. A new experiment with evidence given in context provides a first step beyond single claims, as discussed in our response.
The new appendices currently report the results of our response. The full descriptions and result tables will be ready for the camera-ready version.
We thank the reviewer again for their suggestions, which led us to improve both the case for clustering and the instruction-tuning claims. Do the new ablation in Sec. H.12, which isolates what automatic clustering adds beyond the prompt's own answer words, and the revised scope of Contribution 4, supported by the comparison under free-form prompts in Sec. L.9, address the reviewer's concerns? If so, we would be grateful if the reviewer would consider updating their score. Otherwise, we would welcome suggestions for what would make the paper stronger.
I appreciate the detailed explanations provided by the authors.
The authors added experiments and specifically revised the paper to fix the minor issues mentioned. I believe the concerns raised in the original review have been well addressed. I therefore updated my score accordingly.
We thank the reviewer for acknowledging the breadth of our evaluation, the interventional evidence that the recovered concepts are causally involved in the verdict, and the value of the acceptance margin for detecting belief-output divergence. The reviewer’s questions on what clusters add and on Contribution 4 led to new experiments (G2 and G3 of the General Response), which clarified both. We address the concerns below.
W1, Q1. Our contribution is a way to extract more from existing lens readouts without supervision. J-lens readouts, PCA, HDBSCAN and NMF are existing methods. What’s new:
We also extended the J-lens swap, from single tokens to clusters, with centring and norm matching (Sec. 4.3, App. H.3).
Prior lens work reads a judgement from a single token or phrase chosen by hand for each judgement, prompt and model, while supervised probes need labels and training for each setting. Our concepts need neither.
On what clusters add beyond individual tokens (G2), single tokens often work as well, but clusters remove the need to hand-pick them, which matters when the obvious choice is not the best. On the yes/no prompt, the clusters flip 0.810 of verdicts against 0.560 for the prompt’s own pair _yes↔_no (Tab. 1), and, in a new experiment, still 0.777 after we remove every form of yes, no, true and false, so the gain comes from vocabulary that the clustering found and that is not in the prompt.
W2, Q2. We agree, and will restrict the claim. We extended the analysis of Sec. 6.3 (G3) to a second Base/Instruct pair, Llama-3.1-8B, alongside OLMo-3-7B and Qwen3-8B (Instruct only). As the reviewer suggested, we compared Base and Instruct models on 6 free-form templates. In both, mid-layer truth and falsity registers correlate with late-layer opening or verdict topics and predict the late-layer readouts: for OLMo-3-7B, r = 0.90-0.95 (Base) and 0.91-0.98 (Instruct) on 5 of the 6 templates. The exception is the bare-claim template that Sec. 6.3 used for the Instruct models: the Base model forms no such registers on it, which matches App. C.3, where the Base model needed a forced choice to give a verdict. For Llama-3.1-8B, the Instruct model is stronger on 3 of 6 templates and significantly weaker on 1. The “only” in Contribution 4 therefore does not hold. We will add these results to a new appendix and revise the abstract, Contribution 4 and Sec. 6.3. Due to lack of space, the exact changes are in reply W3 to Reviewer 9G9v.
W3. We agree, and will add to the Limitations (Sec. 7): “Our experiments are limited to claim verification. Whether the readouts form coherent token groups in more complex settings, such as open-ended generation and multi-step reasoning, remains open.” We chose claim verification because there an internal judgement can be compared with the truth, at a known position, and with existing probe and sycophancy work (App. J, K). Extending it is not a straightforward application: with chain-of-thought, for instance, it is unclear which intermediate tokens carry a judgement. As a first step beyond single claims, in a new experiment where a fixed claim is preceded by evidence that supports or contradicts it, the acceptance margin is higher with supporting evidence in 199 of 200 pairs (see reply W1 to Reviewer w1sW).
Q1 (practical use). We see at least 4 applications, based on different results from the paper.
We thank the reviewer again for their suggestions. We hope that these clarifications and new results address their concerns about novelty, the instruction-tuning claim and the scope of our findings. If so, we kindly ask the reviewer to consider updating their score to reflect this, and we are happy to address any further questions.