- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether a language model can notice and report when its own internal computation has been changed. The authors motivate this with oversight: they argue that methods such as chain-of-thought monitoring and self-critique assume a model can report what happens inside it. Recent work (Lindsey, 2025) found that large models can sometimes notice when a concept is injected into their activations. The authors build a framework and library (OWMI) that changes one internal part of a model, mainly by adding a direction to its activations at layer 16, then asks the model whether anything changed, comparing against matched runs where nothing was changed. Their main finding is that the information is in the model but doesn't reach its answer. Across eight open models from 0.5B to 15B parameters, the yes/no answers are at chance (AUROC 0.50). But a linear probe on the same activations detects the change 75 to 96% of the time, a Qwen2.5-7B fine-tuned on the task reports it almost perfectly on new directions of the same kind, and one model's stated confidence carries some signal (AUROC 0.647). The authors conclude that the failure is in the path from internal state to the model's words, and that oversight relying on a model's self-reports should be checked against internal measurements.
Critical strengths
- The main finding is carefully checked. A probe shows the information is present in the activations, a fine-tuned model passes the same pipeline, and the chance-level result is stated as a tight bound rather than just a failed significance test.
- The paper is clear and honest about what was and wasn't run (Table 1).
Critical weaknesses
- The paper needs to be clearer about which safety question it is trying to answer, and why this result matters for it. The abstract asks about frontier models, but the models tested are 0.5B to 15B, while the earlier positive results come from much larger models with a different kind of change and different questions. With only small models, the results can't separate model size from method, so they say little about the frontier models the abstract asks about. Claims like "no such ability" would be better scoped to the models and changes tested.
- The most interesting experiment seems within reach but isn't run. The fine-tuned model is only tested on new directions of the same kind of change at the same layer, which the authors note checks the setup rather than testing introspection. The framework already supports other kinds of change (attention heads, features from a sparse autoencoder, other layers). Training on one kind and testing on the others would show whether a general introspection skill can be trained. Given how well the fine-tuned model does, I think this is the most valuable experiment the paper could add.
- The link to oversight isn't supported. The paper uses the result to argue that chain-of-thought monitoring and self-critique need re-validating. But detecting a change to internal activations is a very different problem from whether a model's chain of thought is faithful to its reasoning, or whether it can be monitored. The paper doesn't connect the two. I'd suggest either testing that link or dropping the claim.
- Minor: the abstract is long and dense - consider shortening it.
Questions for authors
- What safety question is this work trying to answer? For example, is the aim to find out whether we can trust what a model says about its own internal state, to detect when a model's internals have been tampered with, or to measure how introspection changes as models scale? Being explicit about the question would make it clear what result would count as progress, and which experiments are needed.
- For the first two aims, generalisation matters most. A model that can only report one kind of change at one layer tells us little. Does the fine-tuned model detect changes it wasn't trained on (other layers, attention heads, sparse autoencoder features), or say what changed? A positive result would raise my score.
- If the main contribution is the library, can you show it detects introspection in a setting where it has been reported, for example by running Lindsey-style concept injection through the same pipeline on a larger model?
- For Qwen2.5-7B and Mistral-7B, whose yes/no answers never change, did you try other ways of asking (different prompts, a longer answer, free text first) to check the constant answer comes from the model rather than the question format?
To change my score, I would want to see i) a clear safety question that the experiments answer, ii) training on one kind of change that transfers to other kinds and iii) either a larger model or a positive control from earlier work, so the results can be attributed to the model rather than the method.