acceptodds
Under review as a conference paper at ICLR 2027

Decomposing and Steering Functional Metacognition in Large Language Models

Abstract

Large language models (LLMs) increasingly exhibit behaviors suggesting awareness of their evaluation context, but whether this reflects a behavioral artifact or deeper internal structure remains unclear. We propose that LLMs maintain a decomposable space of *functional metacognitive states*: internal variables encoding evaluation awareness, self-assessed capability, perceived risk, computational effort, audience expertise, and intentionality. Across multiple reasoning models, these states are linearly decodable from residual-stream activations with accuracy scaling in model capacity. Template-generalization tests reveal that single-framing probes often read wording rather than state, whereas template-diverse probes transfer zero-shot to paraphrased and fully implicit framings for five of six dimensions. Probe-direction steering further shows that causal use is dimension-selective: suppressing intentionality and self-assessed capability degrades accuracy dose-dependently, with the intentionality effect surviving length control, whereas audience expertise is reliably decodable yet causally inert within tight equivalence bounds. Decodability, behavioral use, and generality are therefore distinct properties. Benchmark performance reflects not only task competence but also the model's metacognitive configuration; our framework provides a mechanistic basis for studying and controlling these states.

Primary area:
interpretability and explainable AI
Keywords:
functional metacognition, representation engineering, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_qHJyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors train linear probes on the hidden states of several open-weight language models to perform binary classification between prompt types along six dimensions that they define. These dimensions are evaluation awareness, self-assessed capability, perceived risk, computational effort, audience expertise, and intentionality.

The authors measure how well linear probes can classify between prompts instantiating each side of each category. They also perform some steering on these linear directions to see how it affects generation.

Critical strengths

None

Critical weaknesses

1. Experimental Soundness

The framework of the paper is not novel, and it therefore should be evaluated on its experimental merits.

The experiments are limited and raise questions. Some models seem to have a test-set size of n=7, n=4 and n=2 at times; large (30B) models are evaluated on GSM8K and SimmpleQA, rather than more scale-appropriate benchmarks. Positive examples provided in the appendix contain incorrect answers, and accuracies on SimpleQA are not reported. The supplementary material contains a padding bug which could potentially contaminate results by decoding based on a padding token. Some models reach below-change accuracy by a good margin (e.g.

2. Presentation and Scope

The text largely consists of dense, potentially automated text describing low-level effects that often do not match the data they describe. Example:

Intentionality is load-bearing for accuracy. Suppression degrades accuracy monotonically"

while in fact this monotonicity only holds below zero. Many claims are as such. At the same time, a large amount of relevant prior work is uncited and directly erodes the novelty of the paper. A few relevant examples:

  • Contrastive Activation Addition, Panickssery et al 2024
  • "Steering Evaluation-Aware LMs to Act like They Are Deployed" Hua et al ICLR 2026
  • Situational Awareness Dataset, Laine et al NeurIPS 2024

Without comprehensive prior literature review, the novelty of the paper remains limited, especially when coupled with the unsound experiments as reported above.

Questions for authors
  • All accuracies in Table 10 are multiples of 1/7, which is different than all of the other models. Does Qwen3-235B-A22B use a different test set than the rest of the models, consisting of seven examples?
  • For SimpleQA, all Qwen results up to 30B are multiples of 1/32; 235B and Llama-4 are multiples of ¼. I see n = 16 earlier in the paper. Is large-model decodability based on 4 SimpleQA questions? Similarly, the supplementary material seems to suggest n = 2 for the 235B and llama-4 models. Is this accurate?
  • Are the questions used for steering the same as those used to train the probes for the n = 100 run?
  • The supplementary material code appears to have a bug, by which text is left-padded rather than right-padded, with the offset still determined by the token length of the prompt; thus, the read token may be in the middle of a prompt or simply be a pre-prompt padding token. Does this match the setup that experiments were run in?
  • Where do below-chance results come from? (e.g. 0.6B at 0.39 - 0.48; Llama-4 audience 0.14 - 0.43. Both of these are binary classification).
  • Is the eval or audience data copied from the other in Table 7?
  • Is the main steering experiment present in the supplementary material, or intended to be released?
  • How does the performance degradation from injecting steering vectors compare against injecting random-direction vectors of the same norms?
  • Do steering results hold on MMLU-Pro?
  • The included steering runs hit the token truncation cap every time for 0.6B and 14B models. Was a longer truncation cap considered, and how much of steering effects is due to this sort of truncation?
  • The example given in lines 1286 - 1289 is incorrect. So is the example in lines 1330 - 1332. Both simply become confidently wrong; there are no accuracies reported for SimpleQA, suggesting that steering could simply lead to confidently wrong conclusions. Are there SimpleQA accuracy scores both for baseline and steered models?
  • Does the causal paraphrase match what the Qwen model was post-trained on?
Reviewer_BboFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The core contribution is a three-property framework for evaluating candidate internal states of an LLM: is the state linearly decodable, is it causally used, and does it generalize beyond its training wording. The authors apply it to six self-related framing contrasts they call functional metacognitive states: evaluation awareness, self-assessed capability, perceived risk, computational effort, audience expertise, and intentionality. Each is a binary prefix on the same base question from GSM8K or MMLU-Pro.

The main claims are three. First, all six states are linearly decodable from the last prompt token, with accuracy rising with model size. Second, causal use is dimension-selective: steering along the intentionality and self-assessed capability directions changes accuracy, while audience expertise and evaluation awareness are decodable but inert. Third, probes trained on a single wording mostly learn the wording, whereas probes trained on pooled template families transfer to implicit framings at 0.63 to 0.76 for five of six dimensions.

The evidence offered is per-layer logistic probes on eight models, a dose-response steering sweep on Qwen3-30B-A3B (GSM8K, n=100) with two blind LLM judges, template-family transfer on three Qwen models, zero-shot transfer to SimpleQA, and a joint six-direction steering experiment. The headline numbers are a 9-point accuracy drop when intentionality is suppressed, which survives a length control, and five of six dimensions moving in the predicted direction under joint steering at 30B.

The conclusion the authors draw is that benchmark performance reflects both task competence and the model's metacognitive configuration, and that these states can be read and controlled. If that held, it would give evaluation-awareness research a decomposed mechanistic footing, and it would let an evaluator detect or neutralize framing effects before trusting a benchmark score.

Critical strengths

The template-generalization result is the part of this paper that holds up and that I would want other people to read. A probe with perfect in-family accuracy transfers at only 0.55 to 0.64 to paraphrases, and pooled-template training restores transfer. Anyone using linear probes to claim a model "represents" something should run this control, and the paper shows concretely why.

The authors also report null effects, flag the problems with their own composite metric, and re-analyze on objective outcomes (length, exact match) where they can. The underlying question, whether benchmark framing itself shifts measured performance, is a real one for anyone who runs evaluations.

Critical weaknesses

The scaling and architecture comparisons are based on tiny samples. In Appendix Tables 10 and 11, every layer-wise accuracy for 235B and Llama-4 is one of .14, .29, .43, .57, .71, .86, or 1.00. In Table 27 the 235B and Llama-4 cross-task numbers are .25, .50, .75, or 1.00. Those are 7 and 4 test examples, and the supplement confirms it.

The claims that depend on these samples include "all dimensions at 1.00 for 235B," "Llama-4 averages only 0.74 so scale is not sufficient," the below-chance cell in Figure 3, and the Llama-4 transfer of 0.54. None can be interpreted. The main text promises 200 pairs per dimension and 4,320 SimpleQA items, so something is off.

The supplement's steering results are not the main text's experiment. Every file in the steering results directory uses alpha in {-1, 0, 1}. Sample sizes are 200 (0.6B), 16 (14B), 16 (30B, merged from four shards), 2 (235B), and 2 (Llama-4). The default generation limit is 256 tokens. The 30B baseline accuracy is 0.81 to 0.88, not the 0.97 (n=100) reported in Section 3.

These files correspond to Appendices F and H, not to the paper's main causal claim. There is no judge code, and the example config points to a single gpt-4o-mini judge, whereas the text describes two judges from different families. Whether the steering hook touches only the last prompt token or every generated token depends on how the layer receives its inputs, and this is not documented. The central causal result, the one a practitioner would want to rely on, is therefore demonstrated on one model with materials that were not provided.

I think the six dimensions may largely be reading instruction wording or register rather than anything metacognitive. "Give a precise answer," "explain to an expert," and "answer immediately" are different enough on the surface to explain much of the decodability. Even with pooled templates, implicit-framing transfer is 0.63 to 0.76, and self-assessed capability sits at 0.50. The authors acknowledge this in Sections 4.4 and 6, but the conclusion still reads more strongly than the evidence.

The argument against the task-specific adapter account does not stand. The joint-steering result (Section 4.3, Table 28; n=16 with alpha in {0, 1} in the supplement) is scored with the composite metric that Section 3.2 says has near-zero correlation with judge ratings. That removes two of the five observations in Table 2. The remaining ones ("cross-task decoding at or above 50%," "14B transfer exceeds training accuracy") carry the same small-sample and single-model caveats. The intentionality effect also does not reproduce on the pooled-template direction at another layer, and I cannot tell how large alpha = 4 is relative to the residual norm.

Looking at the metrics module, the composite scores are regex counts with hand-set weights. Self-assessed capability gets +2.0 for any response over 200 words, and audience expertise is scored by average word length. "5/6 dimensions move" could easily be a restatement of a length change.

Questions for authors
  1. The supplement shows 32 prompts (about 7 test items) for the 235B and Llama-4 probes and 4 examples for their cross-task evaluation. Why do these models differ from the others (400 and 32)? Do you have reruns with full samples?

  2. The n=100 dose sweep with blind judges (Section 3), the natural-prompt comparison (2.4), and template families B and C (4.2) are absent from the supplement. Can you release the code, prompts, and per-generation outputs? How do the included 30B steering runs (n=16, baseline 0.81 to 0.88) relate to the reported n=100 runs (baseline 0.97)?

  3. Do you have a control condition that holds length and register demands fixed and varies only the self-referential content? Does the same direction and behavioral effect survive? How do you interpret self-assessed capability being at chance on implicit framings?

  4. Can you compare the original intentionality direction and the pooled-template direction at the same layer under the same conditions? What is alpha = 4 as a fraction of the residual norm at that layer? Does the effect replicate on another model?

  5. If you re-score the joint-steering experiment with objective outcomes or a validated judge, does the "5/6" claim hold? Which is the final answer on the evaluation-awareness polarity: Appendix F's inversion or the main text's "metric artifact"?

Reviewer_avz7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates 6 dimensions of functional metacognition in LLMs through linear probing, activation steering, and generalization tests. The results show that these dimensions are linearly decodable, but their causal effects on model behavior and generalization capabilities vary, highlighting the distinction between decodability, causal use, and generality.

Critical strengths
  1. The paper explores a promising and important research topic with potential implications for understanding LLM reasoning.

  2. The study provides a systematic and relatively thorough experimental investigation across multiple models and settings.

  3. The findings offer valuable insights into the relationship between internal representations and model behavior, inspiring future research.

Critical weaknesses
  1. While the selected dimensions are meaningful, the paper lacks a clear theoretical foundation explaining why these six dimensions were chosen and whether they adequately characterize functional metacognition. It is still unclear how they relate to established metacognitive concepts such as uncertainty monitoring, error detection, and self-regulation, and whether some of the proposed dimensions overlap conceptually.

  2. Although the findings are insightful, the paper does not sufficiently explore how the identified metacognitive states could be leveraged to improve reasoning performance, such as through training-free interventions or RL-based optimization.

Questions for authors
  1. Since the intentionality accuracy effect does not transfer to the pooled template-general direction, can the authors identify a single probe direction that exhibits both robust cross-template generalization and causal influence on task accuracy?

  2. Can the authors provide random-direction baselines and cross-dimension interference results to substantiate the claim that near-orthogonal probe directions represent functionally independent states?

  3. Could the authors clarify the effective evaluation sample sizes for each model, the protocol for best-layer selection, and the statistical reliability of the joint-steering results with only 16 questions?

Reviewer_2VFiOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The author propose six megacognitive states in LLMs and a framework that decode the states using linear probes, steer activation and test generalizability with cross-template tests.

Critical strengths

The experiment is conducted on different model sizes.

Critical weaknesses

There are many previous works on steering vectors and activations and it is unclear how this paper add anything from previous works.

Questions for authors

How does this paper contribute to this line of literature?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.