acceptodds
Under review as a conference paper at ICLR 2027

Affine Anticipation: LLM Answer Representations Are Largely Linearly Predictable from Context

Abstract

We show that the representation of an LLM's answer is largely a linear function of the representation of its context. We fit metamodels from the residual-stream state at the last context token (the context vector) to the mean answer-token activation (the answer vector), which we call context-answer metamodels. On half a million real-chat pairs from Qwen2.5-7B-Instruct, a linear metamodel reaches a held-out of 0.81, against 0.86 for a nonlinear MLP. The linear metamodel predicts high-level and identity-related aspects of the answer better than low-level and topic-related aspects. The relationship is already present in the base model, is reshaped by supervised fine-tuning (SFT), and is largely preserved by direct preference optimization (DPO) and reinforcement learning with verifiable rewards (RLVR). The state at the end of a chain-of-thought (CoT) trace predicts the answer better than the context state does, and predictability tracks model capability across ten models. The metamodel can also help forecast unwanted behaviors before generation and retrieve contexts likely to elicit them. More broadly, our results add to the evidence for linear structure in transformer representations and its link to language modelling capabilities, while motivating further study into context-answer metamodels as simple and useful approximations of LLM behavior.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nTUeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper asks whether an LLM's state immediately before generation can predict a representation of its eventual response. It fits a linear map from the last prompt-token state to the mean activation over response tokens. On held-out chat prompts, the predicted vectors are close to the actual answer vectors, although this is a prediction of an averaged representation rather than the response text. The authors use the map to ask which aspects of a response are predictable and whether this signal can help anticipate unwanted behavior.

Critical strengths
  1. The paper establishes a convincing core empirical result. The linear map on its own predicts held-out mean-answer activations substantially better than simple controls. This supports a real relationship between the pre-generation state and the later answer representation in the setting studied.

  2. The paper helps explain what its high prediction score captures. The map recovers broad speaker-related qualities more reliably than topic or specific token information, making clear that predicting an answer representation is different from foreseeing the detailed answer.

  3. The authors examine whether this relationship holds beyond the main model. In the tested OLMo checkpoint sequence, the map is already predictive before post-training and changes more across SFT than across the later stages. Although descriptive, this comparison helps locate when the relationship changes. Taken together, these experiments are why I favor a weak rather than clear rejection.

Critical weaknesses
  1. The application comparisons do not isolate the effect of the metamodel. Figure 6 includes direct context projections and the mapped readout usually wins, but the methods use different layers. The metamodel also sees 6,468 or 16,000 trait-eliciting pairs in addition to generic chat pairs, while the context-derived baseline does not receive an independently selected layer (Appendix H.1). The gain may therefore depend on these choices rather than on a reusable generic map. For retrieval, Figure 20 inspects the top 100 prompts from the metamodel's training pool without reporting their behavioral hit rate or a quantitative comparison with direct context rankings (Appendix H.2).

  2. The proposed generality has not yet been demonstrated as a downstream advantage. If I understand correctly, ParaScopes already maps current states linearly to representations of future text, and Persona Vectors already uses answer-derived directions to forecast these unwanted traits from the final prompt state. In this paper, projecting the predicted answer onto a behavior direction is algebraically another linear score of the prompt state; the inverse-map retrieval also ranks prompts against a fixed context-space direction. The test is whether one metamodel trained on generic data predicts or retrieves unseen behaviors better, or with much less behavior-specific data, than direct context readouts at matched layers. Figure 6 fits metamodels with trait-eliciting pairs, and Figure 20 does not report top-k behavioral hit rates against such readouts. A positive result on this comparison would address my concern.

Questions for authors
  1. Do the Figure 6 gains persist at a common layer when the metamodel is trained only on generic chat pairs? For retrieval, how often do top-ranked held-out prompts actually elicit the behavior compared with direct context-direction ranking?

  2. Just out of curiosity, could the much larger Base→SFT loss of metamodel transfer in Figure 3C be related to the density of parameter updates? Mukherjee et al. (2025) report denser SFT updates than DPO/RL updates in the models they study. Can you compare the sparsity and magnitude of the updates across your OLMo checkpoints with the corresponding transfer losses?

Reviewer_UwVYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper shows that a simple affine map can predict an LLM’s mean answer representation from its final context-token state. The reconstruction and retrieval results are strong, and the analyses across training stages and prompting settings are interesting.

Critical strengths

The approach is simple and effective. The distinction between predictable features and less predictable topic-specific content is particularly useful. The core finding stands as an interesting observation about representation structure.

Critical weaknesses
  1. Why predict an answer vector instead of the behavior directly? The proposed pipeline maps the context state to an answer vector, then projects that vector onto a behavior direction. Since both steps are linear, the whole pipeline is just a linear readout of the context state.

  2. Averaging representations across tokens and multiple responses can preserve broad properties such as style while obscuring specific mistakes. A high reconstruction score therefore does not tell us whether the method can catch an individual hallucinated answer or a rare harmful response. Those applications need additional evaluation.

  3. Appendix G reports estimated capability scores, mismatched evaluation configurations, and a model excluded after inspecting the original panel. These limitations should appear alongside the main result. The correlation is interesting, but I would treat it as exploratory.

  4. The end-of-CoT state may already encode a conclusion explicitly stated in the reasoning trace. Better answer prediction from that position does not by itself show what the reasoning process contributed?

Questions for authors

See the weaknesses above.

Reviewer_ADD3OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces "context-answer metamodels", models that are trained to predict the average-pooled last layer representation of an answer from the last index, last layer representation of its context (or prompt). The main contribution of the paper is that such models can be as simple as linear, from the observation that the representation of an LLM's answer is "largely" a linear function of that of its context.

Experiments are conducted on half a million real-chat pairs from Qwen2.5-7B-Instruct, showing a held-out R2 of 0.81. When decomposing and analyzing what characteristics of the answers are more easily predictible from linear prediction, the authors highlight the "high-level" and "identity-related" aspects. The results show that such predictibility is not affected by the different stages of LLM training (from pretraining to RL), and scales with model capacity.

The authors argue that such property of LLMs can serve to condition generation and retrieval from the representation of the context.

The submission is unclear, poorly structured and poorly motivated, appears to have been mostly AI-generated without proofreading. The experiments poorly support the claims of the paper, that were already shown in the previous literature. My recommendation is rejection.

Critical strengths

None

Critical weaknesses

The motivation and positioning compared with previous literature is not clear enough. Related works read like what the authors are proposing has already been demonstrated in prior work (by the authors themselves when citing previous literature: "linear probes on the context ... predict ... behavior of an upcoming response").

The claims sound like "behavior" prediction is the novelty, but that term is never defined anywhere despite being employed throughout the paper, thus it makes it hard to understand what predicting an LLM's behavior mean.

Results are presented at the end of the introduction, without being defined beforehand.

The introduction does not present any claim but an enumerated list of findings, some of which are dubious (what does "speaker identity" mean for a text-only model that does not support audio inputs?), or unclear ("one jointly fitted metamodel covers chat and story settings, and metamodels transfer across story speakers and conversation turns", "CoT improves predictability whether or not it improves correctness").

The notion of "held-out target" is not defined yet used for the metric. Metric is called reconstruction without defining what is being reconstructed. Predicting "mean answer" versus "exact reconstruction" makes very little sense to me.

Questions for authors

What is the point of experimenting with non-linear metamodels? What does linear versus non-linear imply? What are baselines required for (section 4.1) in the context of showing that a linear relationship between ensembles exist?

What is the sparse autoencoder in section 4.2? Who are the authors "asking which properties of a feature predict how well its direction is recovered". How are the 39 properties defined?

In one sentence that a human can understand, what are the learnings from your work that you think are worth sharing with the ICLR community and that were previously beyond the frontier of knowledge?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.