acceptodds
Under review as a conference paper at ICLR 2027

Represented but Unexpressed: Factorized, Native User Models in Large Language Models

Abstract

When a language model answers a question, it also, implicitly, answers a second one: **who** it thinks it is talking to. We study the geometry, expression, and controllability of this internal user model in instruction-tuned LLMs through a single lens—the gap between what we can impose on the model, what it **represents**, and what it **expresses**. First, we find that a set of hand-specified affective control directions, extracted by contrastive activation addition, **collapse** onto a single arousal-like axis (|cos| = 0.57, 65% of variance in one component), recovering the circumplex model of affect; measured identically, the model's **own** model of the user does not collapse. Using a factorial design (user expertise user distress, with a neutral control), we show the model represents these two user attributes as **near-orthogonal**, cross-validated directions (|cos| = 0.05–0.16, cross-validation )—up to an order of magnitude less entangled than the dimensions we imposed. Yet this user model is largely **latent**: on emotionally neutral topics the model's behavior barely changes with the user's state, even as the representation remains at high fidelity—a null that holds under two independent instruments and bounds any natural expression to of what the measurement can resolve. Finally, the latent directions are **causally sufficient**: injecting them produces specific, dose-dependent, layer-localized behavioral change, above matched-norm random controls (up to ), replicating across two model scales and a second model family. The directions are moreover **behaviorally independent**: composing two of them installs a user model with no disclosure, each moving only its own axis, and factorization tracks semantic distance—near attributes entangle, distant ones do not. We accompany these with two methodological cautions the literature needs—affective steering can shift a benchmark's answer distribution without improving reasoning (a false-belief accuracy gain of points that a true-belief control reveals as a point response bias), and steering exhibits a coherence ceiling. Together the results describe a systematic **impose/represent/express** gap: the model's endogenous representation of its interlocutor is richer and cleaner than the abstractions we install, and richer than the behavior it displays.

Primary area:
interpretability and explainable AI
Keywords:
Representation Learning, Interpretability, Mechanistic Interpretability, User Modeling, Latent Representations, Activation Steering, Representation Engineering, Probing, Causal Interventions, Language Models, Model Behavior, Personalization, Human-AI Interaction, Theory of Mind, Latent Knowledge

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_PaLxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how LLMs internally represent user attributes and whether these representations influence their responses. Using contrastive activation extraction, the authors show that user expertise and emotional distress are represented as nearly orthogonal directions, while manually imposed affective directions are more entangled. They also identify a representation-expression gap, where internal user representations have little influence on natural responses but can affect behavior through activation steering. The findings are evaluated using three small LLMs from two model families, with additional experiments on compositional steering and semantic relationships between attributes.

Critical strengths

(1) The paper explores an interesting distinction between what LLMs internally represent about users and what they express in their responses. This perspective could contribute to a better understanding of user modeling in LLMs.

(2) The authors use cross-validation, matched-norm random directions, and independent behavioral scorers to validate their findings. These controls help support the reliability of the reported observations.

Critical weaknesses

(1) Insufficient evidence that the extracted directions form a native user model

The directions are obtained from activation differences between prompts that differ only in an attribute clause (Sec. 3), but no control distinguishes a representation of the interlocutor from a representation of the generic attribute concept mentioned in the prompt. Moreover, directions derived from different clauses may be nearly orthogonal in a high-dimensional space without specifically representing the interlocutor, and without a baseline from non-user contrasts the near-orthogonality in Sec. 5 is hard to interpret; the comparison with the imposed directions (Sec. 4) is likewise not controlled for semantic relatedness. Since the title's claim of a factorized, native user model rests on these results, the central claim is not yet adequately supported.

(2) Experimental specification and statistical evidence are insufficient to verify the results

The manuscript does not provide the exact prompt templates and sample sizes used for extraction, nor the detailed configuration of the embedding scorer and the LLM judge used for behavioral evaluation, although both the "unexpressed" null in Sec. 6 and the causal effects in Secs. 7-8 depend on these instruments. In addition, steering effects are reported mainly in σ units relative to random directions, largely for the strongest cell of a layer-and-magnitude sweep (Table 5), and the additivity test in Sec. 8 relies on 16 probes in a single model. Together with the use of three small models (1.5B-3B), this evidence is limited relative to the general claims in the title and abstract.

Questions for authors

(1) Sections 3 and 5 use templated prompts to extract user-attribute directions, while Section 5 identifies lexical artifacts in early layers. How do the authors verify that the extracted directions reflect genuine user attributes rather than prompt-specific cues?

(2) Could the authors provide the exact prompts, probe construction details, and intervention settings to clarify how confounding factors were controlled and ensure reproducibility?

(3) Section 6 evaluates the representation–expression gap using neutral queries. How can the authors distinguish a failure to express user attributes from cases where user adaptation may not be necessary?

(4) Section 8 evaluates additivity on a single model using 16 probes. Is this sufficient to support the conclusions, particularly given the non-significant interaction effects? Do the results generalize to other models?

Reviewer_feUhOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors study how LLMs represent users, using activation steering. They distinguish between properties that an LLM represents about users vs. about themselves, e.g. emotions. They study seven properties - "adrenaline, cognitive dopamine, logical cortisol, noradrenaline, oxytocin, serotonin, and a self/identity direction" - and find that model representations of these properties are organized along a single axis, roughly corresponding to "arousal". They find that activation addition with these vectors steers model behavior, and that the LLM user model is factorized across a few separable directions. I think the research directions here are interesting, but the presentation needs enormous improvements for this to be a reasonable submission at a major ML venue, and I think the authors also need to expand their experiments significantly. I am less certain about details about the experiment and key results, due to the presentation.

Critical strengths
  • Interesting research topic, potentially novel and meaningful contributions to be made
Critical weaknesses
  • Poor writing, feels heavily AI written. Lots of it feels more like notes than a full paper writeup, e.g. there's a mistake in the caption of figure 1 - " ... Generated by writeup/make figures.py from the saved vectors."
  • Poorly organized. The paper has 11 sections, most of which have unclear organization, and many of which are only a paragraph long.
  • Doesn't have a ton of results. Figure 1 seems to have the same content as Table 1, and the same goes for Figure 2 and Table 2. Figure 2 is never referred to in the text. Appendix A is just a single plot with no text.
Questions for authors

Can you explain, in a few sentences, the overall narrative and how the experiments fit together?

Reviewer_9ar7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates internal "user models" within instruction-tuned language models through three stages: Impose, Represent, and Express. The authors contrast seven hand-crafted affective control directions (claiming they collapse onto an arousal-like axis) with a 2×22 \times 2 factorial user probe (expertise ×\times emotional distress), arguing that the model’s endogenous user representations are natively factorized and near-orthogonal. They further report that on neutral topics, these representations remain latent (unexpressed in overt behavior), but can be causally surfaced and linearly composed via mid-layer residual-stream interventions. Finally, the paper highlights methodological cautions on false-belief ToM benchmarks and coherence degradation under large steering norms.

Critical strengths
  • The research question is somewhat intriguing; however, both the methodology and the depth of the investigation fall short.
Critical weaknesses
  • The paper suffers from poor readability, with Sections 4 through 9 being overly fragmented and disjointed.
  • The models evaluated are all under 3B parameters and are subjected to quantization, casting serious doubt on the reliability and practical significance of the experimental findings.
  • The "Impose" experiments are disconnected from the "Represent" and "Express" experiments in terms of inputs and vector sources, lacking overall experimental coherence.
  • Concluding that human-imposed directions inherently collapse based solely on prompt engineering is questionable; whether these prompts genuinely map to the intended concepts remains unconvincing, making the claim far too premature.
  • The inclusion of the negative result in the third subsection of Section 9 is pointless and adds no value to the paper.
Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.