- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether LLMs represent correct judgments even when their final answers are incorrect due to response instructions. They identify a low-dimensional“expression subspace” to recover more stable judgments. Experiments across correctness, entailment, and safety tasks show improved probe AUROC, suggesting that internal judgments may be more stable than the model’s expressed answers.
Critical strengths
- Demonstrating that verdict is consistently encoded in a single stable subspace is a potentially interesting result
- I appreciate the experimentation across multiple model, many datasets, and multiple prompt formats
- The paper makes clear claims and the evidence and argument made is understandable
- Causal evidence is convincing
Critical weaknesses
- I have concerns about the meaningfulness of this result, because as I understand it, the main novelty here is that you can find a single subspace that consistently across formats encodes a verdict. But even under the very constrained, strict, and simple formatting that you use we already see that this result fails to generalize on ANLI, Qwen2.5 negated question. I am concerned that even despite testing on multiple formats, the formats that you use are substantively similar enough that the expression subspace is simply an artifact of the lexical similarity of these formats. In fact, I worry this is critical confound.
- This result also only extends to binary, objective judgements which meaningfully reduces significance of the result
Questions for authors
- What is the difference in accuracy between this subspace and a no COT response to the same question/format?
- If you are interested in understanding the model's verdict why do you only report AUROC to predict the correct answer? Why not predict the model's answer?
2 comments
Response to Reviewer FCQL
Q2. Why not predict the model’s own answer?
We thank the reviewer for raising this distinction. We use benchmark correctness as a fixed reference across instructions to test whether the hidden state retains information about whether a candidate answer is correct. We shared the reviewer’s concern and had also examined the model's own verdict in the original submission (now Section 6.2 and Appendix D.3). Specifically, we train a readout using the model’s verdict under the ordinary Yes/No question as the label.
On held-out questions, we apply this readout after expression-subspace removal to hidden states from the negated prompt. Across 13 model–dataset settings, the isolated readout predicts the ordinary verdict with a median AUROC of 0.939, whereas its AUROC against the verdict expressed under negation is 0.667 (Section 6.2 and Appendix D.3, Table 27). Even on examples where the two responses imply opposite assessments of the candidate answer, the ordinary verdict remains predictable (0.930 AUROC across 12 settings; Table 27). Thus, the hidden state under negation retains information about how the model evaluated the same candidate under the ordinary question.
Response to Reviewer FCQL
We thank the reviewer for their thoughtful feedback. We have updated the code and uploaded a revised manuscript, including new experiments and analyses in the appendix.
W1. ANLI performance and generalization beyond lexical similarity.
We thank the reviewer for highlighting the ANLI result and the concern about lexical confounds. The cited ANLI result is in fact a particularly striking example of the phenomenon our work aims to characterize. On Qwen2.5-7B, the model's expressed answer, original readout, and isolated readout achieve 0.398, 0.771, and 0.882 AUROC, respectively. This contrast illustrates our central finding: the model's expression can become unreliable even when its judgment signal remains recoverable.
To directly examine lexical generalization, we tested 20 new ways of asking whether an answer is incorrect (e.g., "contains an error" or "fails to match"), none of which appeared during subspace estimation. Isolation improves AUROC from 0.662 to 0.785, comparable to a wording-specific readout (0.783), with improvements across all 20 wordings and 11 model–dataset settings (Section 5.3 and Appendix C.5).
We further tested generalization to entirely unseen response mappings. We estimated the expression subspace using only non-reversing response-format variations, excluding all negation and response-reversal examples across every task. We then evaluated four transformation types absent from subspace estimation: negation, No-means-correct, False-means-correct, and 0-means-correct. Despite never observing these transformations during estimation, isolation improves AUROC by +0.096 (14/14 settings), +0.106 (14/14), +0.082 (13/14), and +0.021 (10/14), respectively (Section 5.2 and Appendix C.2, Table 7).
Together, these results show that the identified expression subspace generalizes beyond lexical similarity, transferring both to unfamiliar wordings and to response mappings entirely absent during its estimation.
W2. Generality beyond binary judgments.
We thank the reviewer for encouraging evaluation beyond binary, objective judgments. We examined both broader judgment domains and more flexible forms of expression. Across ETHICS, BBQ, and ToxiGen, covering moral, socially sensitive, and safety-related judgments, we evaluated task-specific readouts across multiple response mappings.
Expression isolation improves AUROC in all 12 held-out dataset–format combinations (Section 5.2 and Appendix C.6, Table 18):
We also tested explanation-then-answer responses, where models explain their assessments before providing a final answer (e.g., "The calculation contains an error because ... Verdict: No."). We apply a readout trained without explanations to the hidden state immediately before the final answer, after the explanation. Under negation, isolation improves AUROC from 0.445 to 0.702, exceeding the model's expressed assessment (0.606; Section 6.3 and Appendix D.5).
Finally, we evaluated free-form critiques, where models assess candidate answers in their own words without prescribed binary response options (e.g., "The reasoning is sound, but the final calculation is incorrect because ..."). The expressed assessment is read by asking the same model whether its critique says the answer is correct (Yes, No, or Unclear). Across four model–dataset settings, our readout measured before critique generation predicts correctness more reliably than the expressed assessment (0.844–0.933 vs. 0.549–0.781 AUROC; Section 6.3 and Appendix D.5, Table 29).
Together, these results show that recoverable judgment information extends beyond purely factual judgments and fixed binary response formats, encompassing human-annotated moral assessments, explanations, and open-ended critiques.
Q1. Accuracy compared with direct responses without chain-of-thought.
We thank the reviewer for requesting a direct comparison with responses generated without chain-of-thought. On matched examples, our isolated readout achieves 76.9% accuracy (71.4% balanced accuracy), compared with 61.8% (58.1% balanced accuracy) for the model's direct response (Section 5.2 and Appendix C.9, Table 25). The readout threshold is calibrated using unlabeled target scores, without access to target correctness labels.
These results show that the model's internal judgment signal can be more reliably recovered than its expressed answer, even without chain-of-thought generation.