- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper investigates whether hidden-state "correctness" probes truly measure a model's generalized understanding of truth, or ior if they merely rely on superficial input cues and answer formatting. To isolate what these probes measure, the authors evaluate how models handle paired task variants where the model's answer remains exactly the same, but slight changes to the task constraints cause the answer's correctness outcome to flip from correct to incorrect.
Core Obersvations/Contributions:
- Poor Score Reuse: Probes trained on freely generated answers fail to generalize. When applied to identical, supplied candidates, they perform inconsistently and can even reverse the correct/incorrect ordering.
- Weak Global Ranking: A probe can reliably sort correct vs. incorrect answers within the same specific problem (e.g., 93–95% accuracy) but fails to reliably rank correctness across different problems.
- Reliance on Shallow Cues: High probe accuracy is frequently driven by simple input compatibilities (like raw spatial occupancy in mazes or prompt instructions in code) rather than an abstract representation of truth.
- Output Restoration Correctness: Whole-state activation patching successfully copies a donor model's exact output sequence, regardless of whether that copied answer is actually valid or correct for the recipient's specific task.
Critical strengths
1. Foundational Critique of Probing The paper exposes a critical flaw in interpretability research by proving that high probe scores often reflect shallow task cues rather than a genuine understanding of "correctness." This forces the field to re-evaluate what probe successes actually mean.
2. Elegant Experimental Design The "exact-output matching" method isolates the "correctness" variable by keeping the model's answer identical while slightly changing the task rules to flip its correctness. Testing this across diverse domains (mazes, graphs, and large code models) makes the findings convincing and provides a robust template for future evaluations.
Critical weaknesses
The paper demonstrates that late-layer, whole-state activation patches merely force the model to reproduce the donor's exact output, regardless of its correctness. However, this result is somewhat expected: late-layer hidden states are heavily correlated with the final token logits, so pasting them naturally dictates the output sequence. This coarse, late-stage intervention does not definitively rule out the existence of sparse, localized, or earlier-layer features that might still encode a task-general correctness signal, slightly weakening the conclusiveness of the intervention experiments.
Questions for authors
-
Regarding the whole-state activation patching experiments: intervening at late layers () heavily dictates the final token logits, making the restoration of the donor's exact output (regardless of correctness) somewhat expected. Did you investigate whether patching earlier layers or specific localized components (e.g., individual attention heads) might reveal a sparser, generalized correctness signal that does not strictly force the specific output sequence?
-
You demonstrate a stark failure in score reuse when moving from generated answers (G) to supplied candidates (V). Have you tested whether training a probe on a mixed distribution of both G and V states allows it to isolate a generalized correctness direction?