acceptodds
Withdrawn from ICLR 2027

Open-Weight Masked Introspection: Measuring What Language Models Can Report About Their Own Computation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Are frontier models able to introspect about their internal states? Recent work suggests that under certain conditions a complex enough model can audit its own internals, call out what changed, and report back confidently about it. We tested that claim on eight open-weight models from seven families and found no such ability: asked whether their own computation had been altered, none answered better than chance. To test it we built Open-Weight Masked Introspection (OWMI), a framework that intervenes on internal computational objects such as residual-stream sites, attention heads and sparse-autoencoder features, then interrogates the model about the change against the null conditions an answer has to beat: sham runs where nothing was altered, impact-matched random perturbations, and a text-only observer that sees only the visible output. OWMI attaches to benchmarks already in use. Over 78,000 measurements, no model's report discriminates a real intervention from a sham beyond chance (AUROC  0.5007), and an equivalence test bounds the effect below 0.15 percentage points of AUROC. Surprisingly, all the information needed to make that assessment is in the models. A model fine-tuned to report this class of intervention reaches near-perfect recovery on held-out directions (AUROC  1.0), and a linear probe recovers intervention presence from the same activations at between 75% and 95.8% accuracy, sharpening to no held-out error at the last layer before the model speaks. In one model the signal surfaces in the confidence rather than the words: its yes-or-no report never varies, while the confidence attached to that report separates intervention from sham at AUROC 0.647. The failure sits in the path from internal state to verbal report, so oversight practices that read a model's own testimony, from chain-of-thought monitoring to self-critique, need validating against an internal reference rather than against that testimony. While our results show the inability of current open-weight models to introspect, the debate is not settled for future models. We release OWMI as a library so that this emerging ability can be measured as it develops: URL withheld for review.

Primary area:
interpretability and explainable AI
Keywords:
introspection, self-report, interpretability, AI safety, oversight, linear probes, activation patching, sparse autoencoders, equivalence testing, evaluation, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_TkgTOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a language model can notice and report when its own internal computation has been changed. The authors motivate this with oversight: they argue that methods such as chain-of-thought monitoring and self-critique assume a model can report what happens inside it. Recent work (Lindsey, 2025) found that large models can sometimes notice when a concept is injected into their activations. The authors build a framework and library (OWMI) that changes one internal part of a model, mainly by adding a direction to its activations at layer 16, then asks the model whether anything changed, comparing against matched runs where nothing was changed. Their main finding is that the information is in the model but doesn't reach its answer. Across eight open models from 0.5B to 15B parameters, the yes/no answers are at chance (AUROC 0.50). But a linear probe on the same activations detects the change 75 to 96% of the time, a Qwen2.5-7B fine-tuned on the task reports it almost perfectly on new directions of the same kind, and one model's stated confidence carries some signal (AUROC 0.647). The authors conclude that the failure is in the path from internal state to the model's words, and that oversight relying on a model's self-reports should be checked against internal measurements.

Critical strengths
  • The main finding is carefully checked. A probe shows the information is present in the activations, a fine-tuned model passes the same pipeline, and the chance-level result is stated as a tight bound rather than just a failed significance test.
  • The paper is clear and honest about what was and wasn't run (Table 1).
Critical weaknesses
  • The paper needs to be clearer about which safety question it is trying to answer, and why this result matters for it. The abstract asks about frontier models, but the models tested are 0.5B to 15B, while the earlier positive results come from much larger models with a different kind of change and different questions. With only small models, the results can't separate model size from method, so they say little about the frontier models the abstract asks about. Claims like "no such ability" would be better scoped to the models and changes tested.
  • The most interesting experiment seems within reach but isn't run. The fine-tuned model is only tested on new directions of the same kind of change at the same layer, which the authors note checks the setup rather than testing introspection. The framework already supports other kinds of change (attention heads, features from a sparse autoencoder, other layers). Training on one kind and testing on the others would show whether a general introspection skill can be trained. Given how well the fine-tuned model does, I think this is the most valuable experiment the paper could add.
  • The link to oversight isn't supported. The paper uses the result to argue that chain-of-thought monitoring and self-critique need re-validating. But detecting a change to internal activations is a very different problem from whether a model's chain of thought is faithful to its reasoning, or whether it can be monitored. The paper doesn't connect the two. I'd suggest either testing that link or dropping the claim.
  • Minor: the abstract is long and dense - consider shortening it.
Questions for authors
  1. What safety question is this work trying to answer? For example, is the aim to find out whether we can trust what a model says about its own internal state, to detect when a model's internals have been tampered with, or to measure how introspection changes as models scale? Being explicit about the question would make it clear what result would count as progress, and which experiments are needed.
  2. For the first two aims, generalisation matters most. A model that can only report one kind of change at one layer tells us little. Does the fine-tuned model detect changes it wasn't trained on (other layers, attention heads, sparse autoencoder features), or say what changed? A positive result would raise my score.
  3. If the main contribution is the library, can you show it detects introspection in a setting where it has been reported, for example by running Lindsey-style concept injection through the same pipeline on a larger model?
  4. For Qwen2.5-7B and Mistral-7B, whose yes/no answers never change, did you try other ways of asking (different prompts, a longer answer, free text first) to check the constant answer comes from the model rather than the question format?

To change my score, I would want to see i) a clear safety question that the experiments answer, ii) training on one kind of change that transfers to other kinds and iii) either a larger model or a positive control from earlier work, so the results can be attributed to the model rather than the method.

Reviewer_4NUzOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Summary The paper presents OWMI, a framework that intervenes on internal objects (residual-stream sites, attention heads, SAE features) of open-weight LLMs during benchmark prompts and then asks the model whether, where and how its computation changed. Reports are scored against paired sham runs, a random-direction control that is meant to be impact-matched, and a text-only observer. The authors test eight open models (0.5B–15B) and find chance-level discrete detection: a pooled AUROC of about 0.5007 over 11,216 dose-battery pairs, with a TOST equivalence bound below 0.15 pp. Over the same activations, a linear probe recovers intervention presence (75–95.8% at layer 16, perfect separation further downstream), and a LoRA-fine-tuned "known positive" reaches AUROC ≈ 1.0. The authors read this as a failure in the path from internal state to verbal report.

Critical strengths
  1. The control design is careful: sham pairing scored as d′/AUROC instead of raw yes-rate, an impact-matched (JS-divergence) random control instead of a norm-matched one, hook-scope contrasts for probe-encoding perturbation, and a known-positive plus linear-probe sensitivity anchor.
  2. Reporting is honest. Table 1 separates scored, validated, executed and implemented components, and the paper discloses parse-failure selection bias, a mis-specified permutation pass criterion.
  3. An equivalence test is the right way to state a null at this sample size, and the authors raise the parse-failure coding problem themselves.
Critical weaknesses
  1. Limited novelty. Prior work already shows that yes/no detection mostly reflects a bias toward answering yes and that introspection appears only in discriminative, early-layer settings (Hahami et al.). It also shows that large open models detect injected concepts with 0% false positives (Lederman & Mahowald), and that steering detection can be trained (Fonseca Rivera & Africa). What OWMI adds is a combined control harness and an equivalence-bound framing. Its one new control, impact matching, was carried out for only one of the eight models.
  2. Thin experiment breadth. The eight-model battery has 2 items per benchmark, only 508 usable pairs, and impact matching that works for just one model (Tables 3–4).
  3. Internal inconsistencies and much unexecuted framework. Pair counts and model counts disagree across sections, and Tracks B and C, reconstruction and the leakage decomposition are specified but not run.
  4. Interventions don't match the claim being tested. The paper uses random, scaled or zeroed perturbations at a fixed layer, not Lindsey-style concept injection. The known positive overwrites positions 0–2 (likely attention sinks), and a probe at layer 16 reads the very activations that were modified, so it is bound to succeed and says little.
Questions for authors

none

Reviewer_EUySOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors present a reusable and extensible framework to test self-report and introspection in language models. The core contribution is an experimental framework that allows a standardized intervention detection test for language models for different interventions.

With the framework they test the efficacy of the tested models in distinguishing different kinds of hidden-state interventions from control (no intervention) trials. The framework also supports application of the intervention to different computations in the models (for example attention layers, MLP outputs etc.)

Using this framework, they conduct a battery of intervention experiments on instruction tuned language models to find that none of the LMs show non-trivial performance on the self-report tasks. Additionally, they confirm whether the perturbed hidden-states are in fact distributionally different from unperturbed ones using probes.

Critical strengths

The framework reasonably defines the different axes of variation for introspective questions for language models.

Critical weaknesses
  1. The prose is generally obtuse and very hard to read. Even isolated sentences require a lot of effort to understand. The paper is likely AI generated.
  2. The paper does not provide significantly different insights or analyses from related work cited.
Questions for authors

No questions

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.