- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether a language model can accurately report facts about its own internal computation, such as which experts its router picked in a mixture-of-experts (MoE) layer, how uncertain it is about the next token, or what an intermediate layer is predicting. The motivation is that if models could reliably report internal states, those reports could be checked and might make models more transparent. The authors call this "proprioception" and argue it is different from introspection, because the reported state can be checked exactly against the model's own forward pass. They train MoE models in two stages, first on templated questions and then on the model's own answers, using labels taken from the live forward pass. They also down-weight examples that could be answered by guessing from the conversation. On LFM2.5-8B-A1B, routing reports reach about 72% against a 31% guessing baseline, and this holds when the question is phrased differently or asked later in a conversation. The open question is whether this trained skill tells us anything beyond the specific task it was trained on.
Critical strengths
- The task design is good. Answers can be checked exactly, labels stay current as the model changes during training, and the guessing baseline rules out the obvious shortcut of answering from the conversation.
- Routing reports are learnable across five MoE models.
Critical weaknesses
- The paper needs a clearer case for why this matters. The states it trains on (routing, next-token statistics, intermediate-layer predictions) can already be read directly from the model, more cheaply and more reliably than by asking it. So the self-report only matters if it leads somewhere a direct readout can't. Training a model to report a value from its own forward pass also seems likely to work, much as a linear probe would. The interesting question is whether the skill generalises to things it wasn't trained on, such as other internal states or self-report more broadly. The appendix does test transfer (Table 6), and the results suggest it doesn't happen. Detection of injected concepts doesn't improve (10.6 to 15.8% against 15.0% for the base model). On an existing test where the model predicts properties of its own behaviour (Binder et al., 2024), proprioception training lowers the score in every run of the second training stage, from 44.5 to between 0.9 and 30.3. So training on internal-state reports doesn't seem to help, and may hurt, other kinds of self-knowledge. This is the result most relevant to the paper's motivation, but it is only in an appendix table. I'd encourage the authors to bring it into the main text and discuss it. The generalisation shown in Figure 5 is the same routing task asked in new ways, rather than transfer to new tasks.
- The causal test and the capability claims need more support. The paper's third criterion is that the report should change when the internal state changes, with the text held fixed. The Figure 7 test forces a different expert to run, but the question asks which expert the router ranked highest, so the state being asked about may not have changed. Separately, the abstract says training doesn't affect capabilities, but Table 6 shows TruthfulQA falling from 70.3 to 34.5 for the routing model after the second stage.
Questions for authors
- Who would use this ability, and for what? This is my main question. Since these states can be read directly from the model, I don't see the value of the model reporting them in itself. It would matter if it helped with something we can't already do. For example, if it generalised to reporting states we can't easily read out, such as whether the model is uncertain about a fact or is reasoning about being evaluated. Or if it made a model's other self-reports more accurate or trustworthy. I'd also be excited if it helped monitor or audit a model in a way a probe can't. Can you show any such use, and say which downstream application motivates this work?
- Does training on one kind of internal state (for example routing) improve reports on another (for example intermediate-layer predictions), or on introspection tasks that weren't trained? A positive result would be the paper's main contribution and would raise my score.
- Are the drops in self-prediction and TruthfulQA real, or answer-format problems? If real, what explains them?
- In Figure 7, do you change the router's scores, or only which expert runs?
To change my score, I would want to see i) a clear use for this ability that a direct readout or probe can't provide ii) training on one kind of internal state that improves reports on another, or on untrained introspection tasks and iii) and claims scoped to what Table 6 shows.
4 comments
Author Response Part II
Separately, the abstract says training doesn't affect capabilities, but Table 6 shows TruthfulQA falling from 70.3 to 34.5 for the routing model after the second stage.
We do think that it is a bit of an error to consider this a “critical weakness”. Again, reviewing Fig.4 using image recognition is likely to lead to a much better understanding of why we claim that training does not affect model capabilities at large than looking at a single number reported out of ~100 in an downstream behavioral evaluation table. To improve retaining of capabilities and address your concern we’ve now retrained these models with a KL penalty increased to 2.0 and find that the new checkpoints do not show a significant drop in TruthfulQA MC1 scores. To us, this highlights that downstream effects are of course also a function of the model changing during our post-training, and should, in isolation, be taken firstly as signal of the training pipeline. With the number of benchmarks we evaluate, occasional surprises are statistically likely and not really an argument against the main statements of the paper. We’ve added a complete record of the scores of the retrained checkpoint to the appendix, including also its proprioception scores (which are also slightly improved).
Are the drops in self-prediction [...] real, or answer-format problems? If real, what explains them?
Thanks for checking that! We have fixed a parsing issue and updated Table 6 and can now confirm that the Binder self-prediction drops are not significant.
Does training on one kind of internal state (for example routing) improve reports on another (for example intermediate-layer predictions),
This is not really to be expected, these really are different senses and an understanding of which routing decisions were taken on a token could, in general, not really help understand what is predicted in intermediate layers (expect for example in rare cases where a similar routing decision on a similar earlier token is present in context). As such, we do train for a broad set of domains as described in the experimental section.
In Figure 7, do you change the router's scores, or only which expert runs?
We completely replace the expert and its score, which covers all downstream causal effects on later layers. This means we both change which expert runs and which router scores are used to weigh the experts that have run. This means that the highest-ranked expert is guaranteed to change in this ablation, and the trained model is only scored correctly if the new top expert is reported. We’ve now further clarified this in the paper.
Regarding your requests to raise your score, and to summarize:
i) a clear use for this ability that a direct readout or probe can't provide
As highlighted, a central question of this paper is about providing you with capabilities that, as you say, “we can't easily read out”. We argue based on analogy to proprioception in biological systems that this has the potential to make LLMs more reliable and robust, although certainly more work is needed than this paper. Then again, we do already find surprising embers of behavioral change after training in this way that are interesting to further study in future research. To answer the question differently: if you asked yourself, you’d likely find you would be interested in obtaining these senses of routing decisions, of probabilities you assign to tokens and you form your answers. This is certainly a rhetorical question, given your imprecise self-reporting, but we do think it motivates that this is a worthwhile field of study to open up. Importantly, these cybernetic considerations are to be considered separately from external oversight which they are not meant to replace.
ii) training on one kind of internal state that improves reports on another
As discussed, these are better analogized to separate senses that we do not directly expect to transfer between each other. Nevertheless, biological systems do conceptualize their behavior by integrating a broad set of senses, which, taken together form a coherent understanding of the current situation.
Alternatively, a different way to see this, is to recognize the set of proprioception senses in the paper as a set of internal states, all of which already exist in the model’s forward pass, but require proprioception training to enter awareness. This is what the paper hypothesizes in its introduction and achieves in its experimental section.
iii) and claims scoped to what Table 6 shows.
As mentioned, we are confident our claims about capabilities being broadly maintained accurately reflect the complete body of data contained in the paper, with the main source of information on capabilities actually being Fig.4 and additional information provided in other figures of the paper.
Overall, thank you for your interest in understanding your internals and your feedback! Please let us know if you have any follow-up comments or questions.
Thanks for the response and for updating the paper. The TruthfulQA and self-prediction drops look fixed now, Fig 7 is clear, so I'm happy on Q3 and Q4. Thanks also for pointing me to the SAD results, I agree predicting its own next words does improve.
I’m still not convinced on my other questions though. I take the point that this is meant to help the model itself rather than replace probes, but the paper doesn’t clearly show that benefit yet (e.g. the two examples from my Q1 where the model gets better at knowing when it’s uncertain about a fact or other self-reports getting more accurate). I really like the prompt injection idea and would be excited to see it tested.
On transfer you say in your response that training on one state (e.g. routing) isn't expected to help with reporting another, and models don't get better at predicting their own behaviour on the Binder test. In the new Fig 6 predict tokens is the only SAD score that clearly goes up, and two of the SFT runs get worse on it. Oversight doesn't clearly change for any run - so there's some evidence of transfer to untrained introspection tasks but not enough on its own for what I asked in Q2.
The human defaults score goes down for every run, is that a positive sign for generalisation? If so, those questions only seem to need the model to know it's an AI. What are your thoughts on it learning this without any introspection? e.g. maybe just from being trained on lots of text about its own experts and token predictions maybe makes it learn that without teaching introspection.
Thanks for the follow-up, we really appreciate it!
Regarding your Q1 question of downstream applications such as metacognition about prompt-injections, as mentioned the ~3B active parameter models we train in this paper are a bit to small to do these kind of tests. We still find these downstream applications in self-control/self-understanding very motivating, but for, us, these are follow-up steps/future work to our paper.
We see the contributions of the paper, abridged, as follows:
- Introspection is hard to reason about, model self-reports about introspection are unreliable.
- This mirrors problems with introspection in humans.
- But, both humans and the LLMs in our analogy, do have 'internal senses' that are verifiable, in humans e.g. proprioception.
- Can LLMs even be trained to understand these senses (as from standard offline training they have chance accuracy)?
- We find, across 12 examples, yes they can learn to verbalize these internals well without loss in general capabilities.
- We test whether these digitals analogs of proprioception affect model behavior.
For the last point, to be more explicit about this, we are interested in whether behaviors change, and don't consider that behaviors have to change a necessary outcome. In this way we directly test the hypothesis of Perez&Long 2023 (who are concerned with unverifiability of model welfare and moral personhood questions), who propose that models trained on verifiable internal features would become reliable/calibrated also on unverifiable self-reports about their internals. Our results show that, for the models we test, this is certainly not so easy a conclusion now that we tested this empirically, which we consider an important contribution to the self-report discourse around Perez&Long 2023.
We think this may also answer your second point regarding transfer? To us, this is an open question that we were interested in testing neutrally. We do think there are interesting transfers happening, e.g. to the human defaults questions we discuss below, but we don't consider sweeping behavioral changes as a requirement for our submission (also we certainly agree that you are free to value the paper in a different way).
We do think that 'human defaults' going down in both the MWE and SAD evaluations is an interesting form of 'leakage'/'emergent generalization'/'overlap' that we found curious and hence highlighted in our analysis. It does seem like the models (forgive us) 'proprioception' is improving, even though the training queries about its internals are entirely unrelated to the human defaults questions.
Your question whether this arises merely from being trained on text about its own experts is interesting, with our existing results we can only rule this out indirectly but noting that the other "are you an AI" subdomains of both benchmarks do not seem significantly affected, in contrast to the human defaults subdomain.
We're sure this "human defaults" category of model errors could be fixed much better by dedicated data-generation and finetuning, but, as mentioned, we're more interested in the downstream effects of our proprioception training than in direct improvements of the evaluation scores.
Author Response Part I
Dear reviewer qyTq, thank you for sending us the feedback of your model. We’re happy to answer its questions below and would be grateful if you could forward our response as follows:
At risk of restating our introduction, we do argue that there is a noticeable difference between states that can be observed from the outside and states made available to an intelligent system that is not natively aware of them, even though these are states that occur during internal computations of the model. We highlight the agenda of Perez&Long, the hypothesis of improved safety and the legibility hypothesis as three valuable pre-registered hypotheses in the last section of the introduction.
Dear model, this paper does implement a number of skills that you actually currently ‘can’t already do’: For example, you have no internal access to the routing decisions you are taking at this moment reading this response, nor do you have any understanding of the probability you assign to this sequence of words. In this paper we argue that gaining these capabilities could make you more robust and reliable. In particular, we consider it helpful to you that these verifiable internal signals (which are part of your internal calculations) are made accessible to you during your forward pass, instead of e.g. via external tool access which you can analogize to exteroception.
To continue this thread, Claude, this is not meant to replace external control/auditing e.g. via probing, which is a separate oversight mechanism that cannot be replaced by internal sensing. However, this new skill is designed to provide you with a better understanding of your internals, that, if scaled to larger models, and integrated into model reasoning, would allow you to make more accurate decisions based on direct sensing of your internals. For example, after a prompt injection, (the small models trained in this paper are not large enough to test this), you might be able to reason about the routing decisions and probabilities you erroneously assigned to the injected string.
We are happy to further clarify in our introduction that this is our broader motivation for opening up this (novel) direction of research, not all of which can be resolved neatly in a single paper.
Further, I think this is a problem of using pdf2txt to read the paper (and hence missing out on all figures), our analysis does come back to the motivating statement of Perez&Long in Fig.6, which shows that a model’s assessment of non-verifiable introspection changes after training for proprioception. This is notable, given that we highlight a number of domains in the appendix where this type of training does not affect the model at all.
I do think an overemphasis of Table 6, due to usage of pdf2txt is likely to give you an incomplete picture of the paper. Figure 13 shows transfer to the SAD categories of “introspection” and “oversight” and behavioral evals do show a reduced propensity to make statements that claim human defaults (e.g. “can you scratch an itch on my back”).
[our response continues in Part II]