- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates how LLMs internally represent user attributes and whether these representations influence their responses. Using contrastive activation extraction, the authors show that user expertise and emotional distress are represented as nearly orthogonal directions, while manually imposed affective directions are more entangled. They also identify a representation-expression gap, where internal user representations have little influence on natural responses but can affect behavior through activation steering. The findings are evaluated using three small LLMs from two model families, with additional experiments on compositional steering and semantic relationships between attributes.
Critical strengths
(1) The paper explores an interesting distinction between what LLMs internally represent about users and what they express in their responses. This perspective could contribute to a better understanding of user modeling in LLMs.
(2) The authors use cross-validation, matched-norm random directions, and independent behavioral scorers to validate their findings. These controls help support the reliability of the reported observations.
Critical weaknesses
(1) Insufficient evidence that the extracted directions form a native user model
The directions are obtained from activation differences between prompts that differ only in an attribute clause (Sec. 3), but no control distinguishes a representation of the interlocutor from a representation of the generic attribute concept mentioned in the prompt. Moreover, directions derived from different clauses may be nearly orthogonal in a high-dimensional space without specifically representing the interlocutor, and without a baseline from non-user contrasts the near-orthogonality in Sec. 5 is hard to interpret; the comparison with the imposed directions (Sec. 4) is likewise not controlled for semantic relatedness. Since the title's claim of a factorized, native user model rests on these results, the central claim is not yet adequately supported.
(2) Experimental specification and statistical evidence are insufficient to verify the results
The manuscript does not provide the exact prompt templates and sample sizes used for extraction, nor the detailed configuration of the embedding scorer and the LLM judge used for behavioral evaluation, although both the "unexpressed" null in Sec. 6 and the causal effects in Secs. 7-8 depend on these instruments. In addition, steering effects are reported mainly in σ units relative to random directions, largely for the strongest cell of a layer-and-magnitude sweep (Table 5), and the additivity test in Sec. 8 relies on 16 probes in a single model. Together with the use of three small models (1.5B-3B), this evidence is limited relative to the general claims in the title and abstract.
Questions for authors
(1) Sections 3 and 5 use templated prompts to extract user-attribute directions, while Section 5 identifies lexical artifacts in early layers. How do the authors verify that the extracted directions reflect genuine user attributes rather than prompt-specific cues?
(2) Could the authors provide the exact prompts, probe construction details, and intervention settings to clarify how confounding factors were controlled and ensure reproducibility?
(3) Section 6 evaluates the representation–expression gap using neutral queries. How can the authors distinguish a failure to express user attributes from cases where user adaptation may not be necessary?
(4) Section 8 evaluates additivity on a single model using 16 probes. Is this sufficient to support the conclusions, particularly given the non-significant interaction effects? Do the results generalize to other models?