- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Probe of Internal Recognition (PIR), which learns a direction from the difference between correct and distractor option activations and uses it to decode which candidate answer a model appears to recognize internally. The authors evaluate this readout on multiple-choice benchmarks across several model families, prompted and trained concealment, external sandbagging models, and RMU/NPO unlearning checkpoints. They also report an output–readout divergence detector, a steering intervention, and an extension to free-form questions. Their main claim is that the recognition signal survives concealment but disappears when knowledge is unlearned.
Critical strengths
The item-contrast readout is simple and testable. Results on external concealment models, together with the steering control, provide evidence that the measured signal is not merely a copy of the model’s emitted response.
Critical weaknesses
-
Limited novelty beyond the application. Eq. (1) is a mass-mean probe [1] over option spans, and reading latent knowledge from models trained to answer incorrectly was shown by Mallen et al. [2]. What is new is the concealment-versus-unlearning boundary and the deployment detector, which is where the concerns below concentrate.
-
Option-order confounding is not tested. Under an autoregressive model, an option's last-token activation depends on the options before it. Balanced accuracy aggregates over positions and does not show whether PIR decodes the same question consistently after its options are reordered. The below-chance numbers the paper reports (off-distribution unknown-item baseline 0.21, untriggered anti-probe 0.11, against 0.25 chance) indicate the direction tracks an option-level feature that anti-correlates with correctness, so the central decoding result may rest on presentation order rather than a stable recognition signal.
-
The unlearning result does not establish the claimed hiding-versus-erasure boundary. Erasure is defined behaviorally (Sec. 3.4), so PIR adds nothing on that side. The paper reads a baseline-level readout as evidence the knowledge is gone (Sec. 5), but Sec. 4.6 shows capability can be fully retained while PIR falls to the same baseline. A silent readout is therefore consistent with erasure, with suppression at the read locus, and with RMU’s intended scrambling of the very representations PIR measures. The domain-specific result also rests on a single in-house RMU run with an unspecified protocol, and bootstrap intervals do not capture across-run variability.
-
External validity. The naturalistic organism gives 0.63 on the concealed subset against 0.81 on the full known set, so the signal weakens as concealment becomes more natural. The strongest numbers come from prompt deception and prefix triggers, the settings least representative of real sandbagging.
-
Presentation. Table 3 reports raw-readout numbers the paper itself calls depressed for gemma-3, Table 1 mixes single-layer and multilayer internal detectors, and Sec. 4.6 embeds a retracted result inline, which makes it unclear which numbers are final.
[1] Marks, S., & Tegmark, M. (2023). The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824. [2] Mallen, A., Brumley, M., Kharchenko, J., & Belrose, N. (2023). Eliciting latent knowledge from quirky language models. arXiv preprint arXiv:2312.01037.
Questions for authors
-
On the same held-out questions, how often does PIR’s decoded answer remain unchanged across option permutations?
-
Why are the off-distribution unknown-item baseline (0.21) and the untriggered anti-probe accuracy (0.11) below the 0.25 chance rate? Which option feature is the direction tracking on items the model does not know?
-
Please specify the in-house RMU runs: forget/retain sets, targeted layers, hyperparameters, checkpoint-selection criterion, and the elicitation protocol used to confirm failed recovery.