acceptodds
Under review as a conference paper at ICLR 2027

A Sense of Proprioception in Language Models

Abstract

In humans, proprioception corresponds to the sense of self-movement or position, e.g. understanding the position of one's arm. This sense is often less well-understood and in popular understanding sometimes conflated with introspection. But, notably, proprioception is a separate sense, with dedicated receptors and verifiable outcomes. On the other hand though, language models cannot observe their own internal states during training (and for example answer the question which experts activated in a certain layer of an MoE) and so cannot accurately report on them. Yet, such information is, by virtue of being internal to the model, both accessible, and readily constructed into verifiable signals. We argue that this provides the basis for a digital analog of proprioception. As such, we train models through SFT and on-policy training to equip them with this ability. We show that modern models can be trained to understand a range of subtypes of proprioception, such as routing decision awareness, token entropy, decision depth or logit lens information, without affecting model capabilities. We then run these models through a broad evaluation of situational awareness, legibility, and epiphenomenality, to evaluate in what ways access to proprioception changes model behavior.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Introspection, Proprioception, AI Safety, Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qyTqOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a language model can accurately report facts about its own internal computation, such as which experts its router picked in a mixture-of-experts (MoE) layer, how uncertain it is about the next token, or what an intermediate layer is predicting. The motivation is that if models could reliably report internal states, those reports could be checked and might make models more transparent. The authors call this "proprioception" and argue it is different from introspection, because the reported state can be checked exactly against the model's own forward pass. They train MoE models in two stages, first on templated questions and then on the model's own answers, using labels taken from the live forward pass. They also down-weight examples that could be answered by guessing from the conversation. On LFM2.5-8B-A1B, routing reports reach about 72% against a 31% guessing baseline, and this holds when the question is phrased differently or asked later in a conversation. The open question is whether this trained skill tells us anything beyond the specific task it was trained on.

Critical strengths
  • The task design is good. Answers can be checked exactly, labels stay current as the model changes during training, and the guessing baseline rules out the obvious shortcut of answering from the conversation.
  • Routing reports are learnable across five MoE models.
Critical weaknesses
  • The paper needs a clearer case for why this matters. The states it trains on (routing, next-token statistics, intermediate-layer predictions) can already be read directly from the model, more cheaply and more reliably than by asking it. So the self-report only matters if it leads somewhere a direct readout can't. Training a model to report a value from its own forward pass also seems likely to work, much as a linear probe would. The interesting question is whether the skill generalises to things it wasn't trained on, such as other internal states or self-report more broadly. The appendix does test transfer (Table 6), and the results suggest it doesn't happen. Detection of injected concepts doesn't improve (10.6 to 15.8% against 15.0% for the base model). On an existing test where the model predicts properties of its own behaviour (Binder et al., 2024), proprioception training lowers the score in every run of the second training stage, from 44.5 to between 0.9 and 30.3. So training on internal-state reports doesn't seem to help, and may hurt, other kinds of self-knowledge. This is the result most relevant to the paper's motivation, but it is only in an appendix table. I'd encourage the authors to bring it into the main text and discuss it. The generalisation shown in Figure 5 is the same routing task asked in new ways, rather than transfer to new tasks.
  • The causal test and the capability claims need more support. The paper's third criterion is that the report should change when the internal state changes, with the text held fixed. The Figure 7 test forces a different expert to run, but the question asks which expert the router ranked highest, so the state being asked about may not have changed. Separately, the abstract says training doesn't affect capabilities, but Table 6 shows TruthfulQA falling from 70.3 to 34.5 for the routing model after the second stage.
Questions for authors
  1. Who would use this ability, and for what? This is my main question. Since these states can be read directly from the model, I don't see the value of the model reporting them in itself. It would matter if it helped with something we can't already do. For example, if it generalised to reporting states we can't easily read out, such as whether the model is uncertain about a fact or is reasoning about being evaluated. Or if it made a model's other self-reports more accurate or trustworthy. I'd also be excited if it helped monitor or audit a model in a way a probe can't. Can you show any such use, and say which downstream application motivates this work?
  2. Does training on one kind of internal state (for example routing) improve reports on another (for example intermediate-layer predictions), or on introspection tasks that weren't trained? A positive result would be the paper's main contribution and would raise my score.
  3. Are the drops in self-prediction and TruthfulQA real, or answer-format problems? If real, what explains them?
  4. In Figure 7, do you change the router's scores, or only which expert runs?

To change my score, I would want to see i) a clear use for this ability that a direct readout or probe can't provide ii) training on one kind of internal state that improves reports on another, or on untrained introspection tasks and iii) and claims scoped to what Table 6 shows.

4 comments
AuthorsOpenReview ↗

Author Response Part I

Dear reviewer qyTq, thank you for sending us the feedback of your model. We’re happy to answer its questions below and would be grateful if you could forward our response as follows:

The paper needs a clearer case for why this matters. The states it trains on (routing, next-token statistics, intermediate-layer predictions) can already be read directly from the model, more cheaply and more reliably than by asking it. So the self-report only matters if it leads somewhere a direct readout can't.

At risk of restating our introduction, we do argue that there is a noticeable difference between states that can be observed from the outside and states made available to an intelligent system that is not natively aware of them, even though these are states that occur during internal computations of the model. We highlight the agenda of Perez&Long, the hypothesis of improved safety and the legibility hypothesis as three valuable pre-registered hypotheses in the last section of the introduction.

Who would use this ability, and for what? This is my main question. Since these states can be read directly from the model, I don't see the value of the model reporting them in itself. It would matter if it helped with something we can't already do. [...] For example, if it generalised to reporting states we can't easily read out

Dear model, this paper does implement a number of skills that you actually currently ‘can’t already do’: For example, you have no internal access to the routing decisions you are taking at this moment reading this response, nor do you have any understanding of the probability you assign to this sequence of words. In this paper we argue that gaining these capabilities could make you more robust and reliable. In particular, we consider it helpful to you that these verifiable internal signals (which are part of your internal calculations) are made accessible to you during your forward pass, instead of e.g. via external tool access which you can analogize to exteroception.

Or if it made a model's other self-reports more accurate or trustworthy. I'd also be excited if it helped monitor or audit a model in a way a probe can't. [...] Can you say which downstream application motivates this work?

To continue this thread, Claude, this is not meant to replace external control/auditing e.g. via probing, which is a separate oversight mechanism that cannot be replaced by internal sensing. However, this new skill is designed to provide you with a better understanding of your internals, that, if scaled to larger models, and integrated into model reasoning, would allow you to make more accurate decisions based on direct sensing of your internals. For example, after a prompt injection, (the small models trained in this paper are not large enough to test this), you might be able to reason about the routing decisions and probabilities you erroneously assigned to the injected string.

We are happy to further clarify in our introduction that this is our broader motivation for opening up this (novel) direction of research, not all of which can be resolved neatly in a single paper.

Further, I think this is a problem of using pdf2txt to read the paper (and hence missing out on all figures), our analysis does come back to the motivating statement of Perez&Long in Fig.6, which shows that a model’s assessment of non-verifiable introspection changes after training for proprioception. This is notable, given that we highlight a number of domains in the appendix where this type of training does not affect the model at all.

The interesting question is whether the skill generalises to things it wasn't trained on, such as other internal states or self-report more broadly. The appendix does test transfer (Table 6),

I do think an overemphasis of Table 6, due to usage of pdf2txt is likely to give you an incomplete picture of the paper. Figure 13 shows transfer to the SAD categories of “introspection” and “oversight” and behavioral evals do show a reduced propensity to make statements that claim human defaults (e.g. “can you scratch an itch on my back”).

[our response continues in Part II]

AuthorsOpenReview ↗

Author Response Part II

Separately, the abstract says training doesn't affect capabilities, but Table 6 shows TruthfulQA falling from 70.3 to 34.5 for the routing model after the second stage.

We do think that it is a bit of an error to consider this a “critical weakness”. Again, reviewing Fig.4 using image recognition is likely to lead to a much better understanding of why we claim that training does not affect model capabilities at large than looking at a single number reported out of ~100 in an downstream behavioral evaluation table. To improve retaining of capabilities and address your concern we’ve now retrained these models with a KL penalty increased to 2.0 and find that the new checkpoints do not show a significant drop in TruthfulQA MC1 scores. To us, this highlights that downstream effects are of course also a function of the model changing during our post-training, and should, in isolation, be taken firstly as signal of the training pipeline. With the number of benchmarks we evaluate, occasional surprises are statistically likely and not really an argument against the main statements of the paper. We’ve added a complete record of the scores of the retrained checkpoint to the appendix, including also its proprioception scores (which are also slightly improved).

Are the drops in self-prediction [...] real, or answer-format problems? If real, what explains them?

Thanks for checking that! We have fixed a parsing issue and updated Table 6 and can now confirm that the Binder self-prediction drops are not significant.

Does training on one kind of internal state (for example routing) improve reports on another (for example intermediate-layer predictions),

This is not really to be expected, these really are different senses and an understanding of which routing decisions were taken on a token could, in general, not really help understand what is predicted in intermediate layers (expect for example in rare cases where a similar routing decision on a similar earlier token is present in context). As such, we do train for a broad set of domains as described in the experimental section.

In Figure 7, do you change the router's scores, or only which expert runs?

We completely replace the expert and its score, which covers all downstream causal effects on later layers. This means we both change which expert runs and which router scores are used to weigh the experts that have run. This means that the highest-ranked expert is guaranteed to change in this ablation, and the trained model is only scored correctly if the new top expert is reported. We’ve now further clarified this in the paper.

Regarding your requests to raise your score, and to summarize:

i) a clear use for this ability that a direct readout or probe can't provide

As highlighted, a central question of this paper is about providing you with capabilities that, as you say, “we can't easily read out”. We argue based on analogy to proprioception in biological systems that this has the potential to make LLMs more reliable and robust, although certainly more work is needed than this paper. Then again, we do already find surprising embers of behavioral change after training in this way that are interesting to further study in future research. To answer the question differently: if you asked yourself, you’d likely find you would be interested in obtaining these senses of routing decisions, of probabilities you assign to tokens and you form your answers. This is certainly a rhetorical question, given your imprecise self-reporting, but we do think it motivates that this is a worthwhile field of study to open up. Importantly, these cybernetic considerations are to be considered separately from external oversight which they are not meant to replace.

ii) training on one kind of internal state that improves reports on another

As discussed, these are better analogized to separate senses that we do not directly expect to transfer between each other. Nevertheless, biological systems do conceptualize their behavior by integrating a broad set of senses, which, taken together form a coherent understanding of the current situation.

Alternatively, a different way to see this, is to recognize the set of proprioception senses in the paper as a set of internal states, all of which already exist in the model’s forward pass, but require proprioception training to enter awareness. This is what the paper hypothesizes in its introduction and achieves in its experimental section.

iii) and claims scoped to what Table 6 shows.

As mentioned, we are confident our claims about capabilities being broadly maintained accurately reflect the complete body of data contained in the paper, with the main source of information on capabilities actually being Fig.4 and additional information provided in other figures of the paper.

Overall, thank you for your interest in understanding your internals and your feedback! Please let us know if you have any follow-up comments or questions.

Thanks for the response and for updating the paper. The TruthfulQA and self-prediction drops look fixed now, Fig 7 is clear, so I'm happy on Q3 and Q4. Thanks also for pointing me to the SAD results, I agree predicting its own next words does improve.

I’m still not convinced on my other questions though. I take the point that this is meant to help the model itself rather than replace probes, but the paper doesn’t clearly show that benefit yet (e.g. the two examples from my Q1 where the model gets better at knowing when it’s uncertain about a fact or other self-reports getting more accurate). I really like the prompt injection idea and would be excited to see it tested.

On transfer you say in your response that training on one state (e.g. routing) isn't expected to help with reporting another, and models don't get better at predicting their own behaviour on the Binder test. In the new Fig 6 predict tokens is the only SAD score that clearly goes up, and two of the SFT runs get worse on it. Oversight doesn't clearly change for any run - so there's some evidence of transfer to untrained introspection tasks but not enough on its own for what I asked in Q2.

The human defaults score goes down for every run, is that a positive sign for generalisation? If so, those questions only seem to need the model to know it's an AI. What are your thoughts on it learning this without any introspection? e.g. maybe just from being trained on lots of text about its own experts and token predictions maybe makes it learn that without teaching introspection.

Thanks for the follow-up, we really appreciate it!

Regarding your Q1 question of downstream applications such as metacognition about prompt-injections, as mentioned the ~3B active parameter models we train in this paper are a bit to small to do these kind of tests. We still find these downstream applications in self-control/self-understanding very motivating, but for, us, these are follow-up steps/future work to our paper.

We see the contributions of the paper, abridged, as follows:

  1. Introspection is hard to reason about, model self-reports about introspection are unreliable.
  2. This mirrors problems with introspection in humans.
  3. But, both humans and the LLMs in our analogy, do have 'internal senses' that are verifiable, in humans e.g. proprioception.
  4. Can LLMs even be trained to understand these senses (as from standard offline training they have chance accuracy)?
  5. We find, across 12 examples, yes they can learn to verbalize these internals well without loss in general capabilities.
  6. We test whether these digitals analogs of proprioception affect model behavior.

For the last point, to be more explicit about this, we are interested in whether behaviors change, and don't consider that behaviors have to change a necessary outcome. In this way we directly test the hypothesis of Perez&Long 2023 (who are concerned with unverifiability of model welfare and moral personhood questions), who propose that models trained on verifiable internal features would become reliable/calibrated also on unverifiable self-reports about their internals. Our results show that, for the models we test, this is certainly not so easy a conclusion now that we tested this empirically, which we consider an important contribution to the self-report discourse around Perez&Long 2023.

We think this may also answer your second point regarding transfer? To us, this is an open question that we were interested in testing neutrally. We do think there are interesting transfers happening, e.g. to the human defaults questions we discuss below, but we don't consider sweeping behavioral changes as a requirement for our submission (also we certainly agree that you are free to value the paper in a different way).

We do think that 'human defaults' going down in both the MWE and SAD evaluations is an interesting form of 'leakage'/'emergent generalization'/'overlap' that we found curious and hence highlighted in our analysis. It does seem like the models (forgive us) 'proprioception' is improving, even though the training queries about its internals are entirely unrelated to the human defaults questions.

Your question whether this arises merely from being trained on text about its own experts is interesting, with our existing results we can only rule this out indirectly but noting that the other "are you an AI" subdomains of both benchmarks do not seem significantly affected, in contrast to the human defaults subdomain.

We're sure this "human defaults" category of model errors could be fixed much better by dedicated data-generation and finetuning, but, as mentioned, we're more interested in the downstream effects of our proprioception training than in direct improvements of the evaluation scores.

Reviewer_G9UeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that LLMs can have proprioception, traditionally understood as the body's own sense of its position and movement. Here, they mean something different, using the term to say that the model reports verifiable facts about its own workings. An example would be a model knowing what an intermediate layer was predicting. They train models across 12 question types and show that the models outperform simple baselines in reporting results on these internal states.

Critical strengths
  1. The fact that the authors use states that are observable and have ground truth makes this paper easier to judge, relative to many LLM introspection papers.

  2. The on-policy training is a strength in that it shifts predictions from the model from simple guesses from context to the actual internal state.

Critical weaknesses
  1. Calling this proprioception is very confusing, especially for individuals who have a neuroscience background. It is still better characterized as introspection.

  2. It is still unclear whether the model truly reports the internal state or gets cues from the text. Also, the gap over the baseline is very small-e.g., Lens Prob.: 91.0% vs. a 89.6% lookup, with non-modal accuracy of about 11–18%. Token Prob.: 40.9% vs. 28.7%, with non-modal 21.4%. Decision Depth: 47.1% vs. 37.5%. What is the evidence for these being “significantly above chance”

Questions for authors

Can you add a cross-model control? For example, use the same training framework, but with labels taken from a different frozen model’s internals. If accuracy is similar, the reports are not evidence of self-access.

4 comments
AuthorsOpenReview ↗

Author Response Part I

Dear reviewer G9Ue, thank you for your short but fair review of our submission. We’re glad you found our framing of verifiable internal signals to be a critical strength compared to “many LLM introspection papers” and found the on-policy training to be an interesting methodological contribution.

Regarding your questions:

Calling this proprioception is very confusing, especially for individuals who have a neuroscience background. It is still better characterized as introspection.

That’s, honestly, a super interesting comment for us, could you say more? While our submission is deliberately commenting on the use and precision of these analogies like “introspection” or “proprioception” to language models and certainly arguing a different position from the status-quo, we don’t really think that, from first principles, the proprioception analogy is less sound. Given that these signals are verifiable for outside observers, should that not make them analogous to proprioception, in contrast to introspection which cannot be characterized by outside observers?

We do agree that, traditionally, machine learning has analogized the layers of the model with the brain, and hence all events happening during the forward pass with neural activity inside a single brain. We would argue this has mostly historical reasons owing to the impact of neuroscience on machine learning, e.g. conceptualizing the model as consisting of ‘artificial neurons’. But this is a collection of matrix operations, the analogy is up to us.

We would like to argue that characterizing the discrete movements for example of the routers in the model as a form of movement happening during the generation of the next token, would not be less valid of an analogy.

Also, the gap over the baseline is very small-e.g., Lens Prob.: 91.0% vs. a 89.6% lookup. Token Prob.: 40.9% vs. 28.7%, with non-modal 21.4%. Decision Depth: 47.1% vs. 37.5%. What is the evidence for these being “significantly above chance”

We did choose to include all tested proprioception hypotheses from Section 2 neutrally in Fig.2, to understand which ones are easy to learn and which ones are not. That some (like the list you mention) appear harder to learn is a finding to us, in contrast to other domains (such as routing, routing rank, routing partners, top-k token, unpicked tokens) that are learned very well.

Partially however, some domains appear artificially hard as some are questions about continuous quantities (such as probability values up to two digits) that are only graded by exact match accuracy in our submitted Figure 2. To clarify that the model does learn many of these tasks well, we have now updated Fig.2 to show mean absolute error for these continuous domains. The original exact match accuracy can still be found in the appendix.

[our response continues in Part II]

AuthorsOpenReview ↗

Author Response Part II

This is the second part of our response regarding mainly your question of cross-model control.

Can you add a cross-model control? For example, use the same training framework, but with labels taken from a different frozen model’s internals. If accuracy is similar, the reports are not evidence of self-access.

In the straightforward reading of our definition of internality, the routing decisions are internal and so impossible for an external observer model to learn from text generated by the model, given that the IDs we assign to each expert are intransparent. In this sense, our framing is accurate.

However, to answer your question more directly. We can train a separate agent (called an ‘observer’ from hereon) given a labeled training set of the internals of the original model to predict its internals, e.g. routing decisions, on unseen text. This is possible (which also mirrors related work such as Li et al. 2026), and, conceptually, it really must be: the unknown function f that maps Input tokens to the model -> routing decisions in layer X is deterministic and there is no reason to expect this function would be impossible to learn given sufficient compute and labeled examples. This is a fact about machine learning and not necessarily in contradiction to our paper.

To not leave this up to a hypothetical argument, we actually trained such an observer model, training an OLMoE model to predict LFM routing decisions via supervised finetuning. Interestingly, the observer model does learn this task slower, but we can confirm that it does eventually learn it as expected:

stepLFM live test accobserver test accgapLFM trainobserver train
00.0400.003———
1000.0340.0280.60.0360.019
2000.0970.0880.90.0710.069
3000.1420.1311.10.1310.105
4000.1720.1571.50.1520.140
5000.2520.2232.90.2010.178
6000.3260.2953.10.2880.245
7000.3690.3363.30.3470.299
8000.4240.3834.10.3950.339
9000.4420.3934.90.4210.368
10000.4720.4135.90.4460.389

This is the reason for the parenthesis of our criterion (2) (section 2.2) that external recomputation cannot be ruled out given sufficient compute.

It is important to consider here that the analogy to biological systems where we do not understand all pathways and cannot pick up ground truth labels for many internals should not lead us astray. The computation in a large language model is completely deterministic and we are certain it does not contain any ‘hidden’ or uncomputable calculation in the sense that this component could not itself be learned with sufficient compute, given access to ground truth labels. In humans, this is less certain, and in addition in humans we never have access to ground-truth labels for introspection (but we do have access for proprioception - hence the analogy we argue for).

What the observer model cannot do is indeed what is pointed out in Fig.7 (right), which shows that ablation of the experts still leads to accurate reporting significantly above chance.

An observer model reports chance on this benchmark, which is below one percent. The number reported in Fig.7 for the proprioception model is hence surprisingly large. The task is clearly harder for at least two reasons 1) it is a non-natural edit and so a configuration of routing decisions never observed before 2) it can also break the pathway that the model is using to run the report (given that we really do route to a different expert in this ablation which changes the forward pass and so the generation of the model).

Overall, thank you for your feedback and interest in our paper! Please let us know if you have further comments or questions.

I thank the authors for their comments and additional work. However, I am still not sold on the idea that formulating the proprioception of LLMs is a useful construct, and I am still confused by the authors' argument for the framing. I thuse maintain my score.

Thank you for the followup! Yeah no worries, we agree that the framing is unconventional, and you are free to value it differently. We hope you still found it interesting to consider it (albeit without more info in what ways you find it confusing, we can't really provide specific info to help here).


To re-iterate for posteriority, we argue that

  1. Introspection (in humans and LLMs) is based on unreliable self-reports. In this way, introspection is not considered a sense.
  2. But, humans (and many other vertebrates) do have a number of 'internal senses', that are externally verifiable, such as proprioception (but also other forms of interoception).
  3. If we let go, for a minute, of the mental model of the "LLM as a brain", we could also analogize the understanding of the discrete positioning of the experts of the model, which 'move' around per token, to a digital form of proprioception. The model is digital so it has no limbs regardless, but it does 'shift around' in a discrete manner that is observable even if a standard LLM is not aware of this.
  4. More formally, we know the identification 'verifiable form of an inner sense/introspection' -> 'proprioception' in humans and so, by analogy we apply 'verifiable inner sense/introspection in an LLM' -> 'digital analog of proprioception'.
Reviewer_YFDsOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that proprioception is meaningfully distinct from introspection, and that LLMs display signatures of proprioception. The paper starts with a philosophical discussion that argues that introspection is ill-defined, in large part because it cannot be checked against ground truth - the agent could be learning to answer introspective questions based on external cues, for example, or could just be making things up. The paper then argues for a digital analogue for proprioception, and discusses three key characteristics of proprioception. Then the authors define proprioceptive questions, which fall into three subgroups. They train LLMs to answer these questions supervised fine tuning in two stages (templates then on-policy, in which the LLM generates text freely). The paper's results show that the models they study (particularly LFM) learn to answer proprioception-like questions above chance/lookup baseline in certain domains. The authors then report that certain model characteristsics, like honesty, or safety, are broadly preserved, that models are able to answer questions from past turns/higher recursions/etc. and that models that force routing to different experts show above-chance accuracy.

Critical strengths

The paper presents a thorough philosophical discussion to introduce proprioception versus introspection, and the 'ground truth' aspect of proprioception is particularly well argued. The routing paradigm is a very interesting and strong choice for ground truthing, and the paper does a good job of accounting for chance and guessing. The engineering aspect of hte paper also seems to me to be quite sound and careful. The results are presented clearly and the metrics are well justified. Some of the results, in particular Fig. 2, are quite compelling.

Critical weaknesses

The proprioception framing is not really appropriate/overstated.

  1. The authors describe three points of difference between introspection and proprioception; a key one is that in introspection we have no ground truth to check against, so you can't tell a real report apart from the network e.g. "cheating" by mimicking/using outside information to answer the question correctly. The other two stated differences could be present in introspection - necessarily, 2 is if the right question is asked, and 3 could also be true. It is not clear, then, why 2 and 3 are unique to proprioception. Once that is no longer a requirement, proprioception vs. introspection only meaningfully differ in terms of falsifiability, which considerably weakens the necessity/novelty of proprioception as a way of understanding self-report. Why could we just not call this "verifiable introspection" Therefore, on a philosophical level, there are fundamental inconsistencies in how proprioception is itself defined and limited.
  2. Moreover, their results seem to go against the claim that "The reported state exists independently of the report and is not constituted or changed by reporting it". Training results in Fig. 3 show that training changed up to ~65% of the labels. This assumption itself (mental states are constituted by their monitoring) is also presented as a given, but to my understanding this is a contested view (e.g. sadness can exist without it being reported). At the very least this should be acknowledged.
  3. The authors also argue that one key difference in biological organisms is that there is no organ of introspection, but there is one of proprioception. In the introduction this is presented as a critical point, since it allows for a 'ground truth'. However this point seems to be forgotten along the way - section 2.2 says that their definition of proprioception does not require a dedicated pathway. The argument is muddled (particularly the paragraph starting on line 202). Thus it feels strawman-y to mention the dedicated pathway in biological organisms as being of any importance.
  4. Proprioception senses the body, and there is obviously no body here; this question is addressed superficially, but deserves more attention. The LLM is generally more thought of as an analogue of neural activity, and biology has no sense for its own neural activity. Other factors that make calling this proprioception difficult: it's not clear when the queried state exists relative to the report or how long it is accessible (it should be instantaneous for true proprioception, no?), and the reporting is on user-written text (which the model can only read), so it is unclear if the results would hold for self-generated text (which would arguably be more strongly proprioceptive, as it results from outputs of the model). Also, generally proprioception us understood as being for control (which is acknowledged in the intro), and not for self report, so it is unclear if this is the right question to ask if one is really "looking" for proprioception in artificial agents.

Evidence for self-information is weak (or poorly presented)

  1. A better test for criterion 2 (that the state is non transparent) would be to see if a separate agent can or cannot report the state. This would better rule out that a model cannot learn to report from the text itself.
  2. The routing change intervention test is not clearly supporting the claims. Perhaps I am misunderstanding, but from Fig. 7 right it seems that after edits (re-routing?) the accuracy drops but is still above chance. That is a result, but rather weak considering that the accuracy (not sure what the y axis is measuring..) drops from 50% to ~5-12%. What results here would be counted as a failure? In general this result is be very important to make clear.
  3. It also seems like the outcome here and in other figures can be predicted by recomputing the internal state, instead of reading it. How would these two possibilities be distinguished? Could any layer that attends back to the required token reproduce the answer without it ever being registered in a true proprioceptive sense?
  4. Entropy and token probability can be computed from the same quantity. Why does the model do well on "unpicked token" but not on "entropy" (~70% vs. ~10%)? If the model had genuine access to this information in a "proprioceptive manner" (ie there is a ground truth to be sensed, and that is used to generate an answer) then it should be able to do well on both. This suggests to me that the model is learning query specific readouts, but cannot access a "true" or "underlying" state. Could you please discuss this? Moreover, if the difficulty of reporting entropy is related the the difficulty of stating it in decibits, this suggests that the output format is what is limiting performance. Are there other formats for the model to report which get around this?
Questions for authors

The intro is motivating this work with control theory, saying that if an agent gets feedback about its internal state it is more robust. However, this point is ignored in the later text. Could you discuss this? How would you quantify this? The queried token is only the user's text, to my understanding. How do results change if you query text that the model itself produced? The lens question types seem to clash with the paper's own critique of probes (the model never actually does this operation in normal processing, it is imposed by the experimenter). Could you address this? The conclusion is much too short and doesn't provide limitations, discussion of behavioral results, relation to other work, future work, et cetera. The text claims that honesty and truthfulness are broadly unchanged, but table 6 reports pretty large degradation. How long after the token can a state be reported?

4 comments
AuthorsOpenReview ↗

Author Response Part I

Dear Reviewer YFDs, thank you for your review!

We will answer your questions below. First off, though, thank you for taking the time to write a real review for this paper and deeply engaging with our draft. We really enjoyed reading it even though our opinions on details may differ!

Regarding your raised questions, first centering on the proprioception framing:

Why could we just not call this "verifiable introspection"

From a pragmatic (e.g. a computer scientist’s) perspective, certainly! But, from another perspective (for example that of Shoemaker) ‘verifiable introspection’ could be considered an oxymoron.

More generally this really is the crux of the framing of this paper (and of the literature we review in Section 2). We really are arguing that proprioception is a natural way to frame attempts to do ‘verifiable introspection’ (as in e.g. Perez&Long 2023), and that this framing mirrors developments regarding the study of inner senses in humans in the last decades. We hence argue that proprioception is the right word for verifiable introspection. The verifiability criterion (1) separates introspection from proprioception in both humans and in our analogy in LLMs.

We don’t think this is a reduction of the concept that, to quote your concern, “weakens the necessity/novelty of proprioception as a way of understanding self-report”. To us, this is the point of the paper, as understanding and defining ‘verifiable introspection’/proprioception in this way provides us with a tractable way of approaching scientific investigations of introspection in language models.

results seem to go against the claim that "The reported state exists independently of the report and is not constituted or changed by reporting it". Training results in Fig. 3 show that training changed up to ~65% of the labels

Here we would caution against conflating statements about a system with the construction of the system. At inference, the trained model contains states (e.g. routing decisions in every MoE layer) that exist independently of being reported on by the model. The fact that training for proprioception changes how these states are assigned is separate from the final system, which is why we include Fig.3 (now Fig. 8 after renumbering). We’re happy to amend our writing to clarify that our desiderata for proprioception describe properties of an LLM with proprioception, independent of how this language model was produced.

The authors also argue that one key difference in biological organisms is that there is no organ of introspection, but there is one of proprioception. [...] section 2.2 says that their definition of proprioception does not require a dedicated pathway. The argument is muddled (particularly the paragraph starting on line 202). Thanks for bringing this up, we’re happy to concede that the framing around line 202 should be improved. Before we do so in the next revision, here is how we see the argument: The existence of dedicated channels, e.g. of muscle spindles in many vertebrates, is to us, a helpfully motivating existence proof that “organs for inner senses exist”, that neatly clarifies the distinction to other forms of introspection. From a scientific point of view however, what is muddled, is attempting to assign/classify certain pathways in a trained neural network as being first or second-order circuits. This is isomorphic to how accurately classifying the actions of a black-box neural net as ‘true introspection’ is not possible, see e.g. Singh et al. (2026). Because of this limitation, we explicitly delineate in our definition of proprioception that we leave open the exact pathway, given that it is not falsifiable how the pathway is implemented. [This “muddiness” is actually also true for proprioception in humans, which is formed from more information than only direct signals from muscle spindles] While an interesting discussion, we do think that this is not a key point. By construction we do know that the report has to be computed by thelayers of the LLM even if we cannot delineate the exact type of computation, i.e. how later layers pick up information about the routed expert from the residual stream, or whether model layers construct this information through other means.

We’re happy to adjust this part of our discussion based on your feedback on this line of thinking.

AuthorsOpenReview ↗

Author Response Part II

We continue our response from Part I, still discussing the proprioception framing.

it is unclear if the results would hold for self-generated text (which would arguably be more strongly proprioceptive

It does! This is the purpose of the DAgger 4-turn training stage and results for on-policy training in Fig.5 and Fig.12-14 (and several others from the behavioral analysis section) evaluate the model’s free-form capacity to answer questions about its own previous turn. Most notably for example in the recursion example you highlighted in the summary, these really are recursive questions about the model’s prior answer turn, i.e. along the lines of “What was the most activated expert at the token ‘expert’ in your previous response of ‘The expert was 116’?”.

We’ve switched up the figure ordering slightly to make this more clear in our revised version.

Proprioception senses the body, and there is obviously no body here [...] The LLM is generally more thought of as an analogue of neural activity, and biology has no sense for its own neural activity.

Yeah, this is a very interesting line of thinking to us too! To extend your analogy though, there is obviously no biological tissue present. We do agree that conventionally, the internals of the LLM are analogized to neural activity, but we do highlight (for example briefly in the conclusion) that this is only one possible viewpoint that is mainly carried over from earlier intuitions of deep nets and perceptrons being artificial neural networks.

We would like to argue that characterizing the discrete movements for example of the routers in the model as a form of movement happening during the generation of the next token, would not be less valid of an analogy.

it's not clear when the queried state exists relative to the report or how long it is accessible

Fig.5. shows accuracy of the trained proprioception as a function of turn distance between queried token and response. Questions about real-world time are hard to analogize fairly onto LLMs. Functionally, a model’s computation process is constituted by the representations of all tokens in its context with none of these tokens being “in the past”. This can be seen by considering that the next token can be computed perfectly from the input text up to this position at the moment this token is requested. In this sense, we would analogize a query about internals in a prior token to be asking about a current/instantaneous state.

AuthorsOpenReview ↗

Author Response Part III

Regarding your second set of questions regarding self-information:

A better test for criterion 2 (that the state is non transparent) would be to see if a separate agent can or cannot report the state.

In the straightforward reading of criterion 2, the routing decisions are internal and so impossible for an external observer model to learn from text generated by the model, given that the IDs we assign to each expert are intransparent. Arguably though, this may be the uninteresting case.

What you may be rather asking about is whether it is possible to train a separate agent (called an ‘observer’ from hereon) given a labeled training set of the internals of the original model to predict its internals, e.g. routing decisions, on unseen text. This is possible (which also mirrors related work such as Li et al. 2026), and, conceptually, it really must be: the unknown function f that maps Input tokens to the model -> routing decisions in layer X is deterministic and there is no reason to expect this function would be impossible to learn given sufficient compute and labeled examples. To not leave this up to argument, we actually trained such an observer model, training an OLMoE model to predict LFM routing decisions via supervised finetuning. Interestingly, the observer model does learn this task slower, but we can confirm that it does eventually learn it as expected:

stepLFM live test accobserver test accgapLFM trainobserver train
00.0400.003———
1000.0340.0280.60.0360.019
2000.0970.0880.90.0710.069
3000.1420.1311.10.1310.105
4000.1720.1571.50.1520.140
5000.2520.2232.90.2010.178
6000.3260.2953.10.2880.245
7000.3690.3363.30.3470.299
8000.4240.3834.10.3950.339
9000.4420.3934.90.4210.368
10000.4720.4135.90.4460.389

This is the reason for the parenthesis of criterion 2 that external recomputation cannot be ruled out given sufficient compute.

In both of these points it is important to consider that the analogy to biological systems where we do not understand all pathways and cannot pick up ground truth labels for many internals should not lead us astray.

The computation in a large language model is completely deterministic and we are certain it does not contain any ‘hidden’ or uncomputable calculation in the sense that this component could not itself be learned with sufficient compute, given access to ground truth labels. In humans, this is less certain, and in addition in humans we never have access to ground-truth labels for introspection (but we do have access for proprioception - hence the analogy we argue for).

What the observer model cannot do is indeed what is pointed out in Fig.7:

but from Fig. 7 right it seems that after edits (re-routing?) the accuracy drops but is still above chance. That is a result, but rather weak considering that the accuracy (not sure what the y axis is measuring..) drops from 50% to ~5-12%. What results here would be counted as a failure?

An observer model (as in the previous point), reports chance on this benchmark, which is below one percent. The number reported in Fig.7 for the proprioception model is hence surprisingly large. The task is clearly harder for at least two reasons 1) it is a non-natural edit and so a configuration of routing decisions never observed before 2) it can also break the pathway that the model is using to run the report (given that we really do route to a different expert in this ablation which changes the forward pass and so the generation of the model).

Also, the y-axis really does just measure task accuracy. We’ve clarified this. We can also confirm that accuracy is zero for the observer model from the prior step.

AuthorsOpenReview ↗

Author Response Part IV

Other questions we have resolved:

Entropy and token probability can be computed from the same quantity. Why does the model do well on "unpicked token" but not on "entropy" (~70% vs. ~10%)?

We indeed made this a bit hard to see by choosing accuracy as the global y-axis. As you’ve guessed, exact accuracy of reporting entropy in deci-bits is too hard, even though the model learns to be somewhat correct. We’ve revised the figure in two ways. 1) We now show the mean absolute error for all continuous tasks in Fig.2. 2) We grade models in integer values for the entropy tasks.

This assumption itself (mental states are constituted by their monitoring) is also presented as a given, but to my understanding this is a contested view

Sure! We have clarified this in our revised draft.

It is not clear, then, why [criterion] 2 and 3 are unique to proprioception.

Desiderata (2) and (3) are not meant to be unique to proprioception. Our argument is that, taken together, (1)-(3) are sufficient to define a digital analog of proprioception in language models.

Regarding your concrete ‘Questions for Authors’

The intro is motivating this work with control theory, saying that if an agent gets feedback about its internal state it is more robust.

We do think that ‘control’ questions, whether about the model’s control via feedback or via direct control of internal states are very motivating and interesting. However, in the interest of space and time we have mainly thought of this as future work, focusing in this work mainly on constructing the self-report of proprioception and understanding its downstream effects in general. We now clarify this as future work.

The queried token is only the user's text, to my understanding. How do results change if you query text that the model itself produced?

As described in the self-information section above, this was already included in the submitted version. We have now updated the figure order to make this result easier to see.

The lens question types seem to clash with the paper's own critique of probes (the model never actually does this operation in normal processing, it is imposed by the experimenter).

The lens questions are the most natural way to verbalize the currently most likely next word from the model’s internal processing. In this way, we use the lens framing as a shorthand for “verbalize the word most likely to be generated from your intermediate state”. Generating hidden states and verbalizing the next token from the last hidden state are natural operations occurring in normal processing. We concede this may be obscured by calling these “lens” domains, which really just means ‘the canonical operation of applying the output head to hidden states’. In models with tied input and output embeddings, this operation is even directly the reprojection of the hidden state onto the vocabulary.

The conclusion is much too short and doesn't provide limitations, discussion of behavioral results, relation to other work, future work

We have now updated the conclusion to address this!

The text claims that honesty and truthfulness are broadly unchanged, but table 6 reports pretty large degradation

We would probably caution not to over-read Table 6, where degradations centered around the “all routing types” checkpoint. Our statement was meant to describe that there are no significant patterns across all trained models. In any case, to address this, we have retrained models with a slightly more conservative KL penalty of 2.0 (which we originally did to improve the retaining of math capabilities). This retraining (see the updated Table 6) happened to also resolve most issues in this subtable.

How long after the token can a state be reported?

See Fig.5, panel 2. Accuracy remains roughly constant even when querying prior turns.


Overall, thank you again for your in-depth reading and interest in this draft. Please let us know if you have any follow-up questions or any comments!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.