acceptodds
Under review as a conference paper at ICLR 2027

On the Emergence and Effectiveness of User Profiling in Language Models

Abstract

As an increasing portion of humans spend an increasing portion of their time conversing with and directing language models, they will be more subject to the implicit profiling language models are capable of. Understanding how these user profiling capabilities emerge both in the models themselves and during training is essential for privacy and mitigating bias, but existing studies are limited in scope and in realism. We show that language models represent the gender and age of the person writing, that this representation strengthens with scale and forms early in training. Training linear probes on 162k human written blog posts across 29 base models spanning 14M to 32B parameters shows consistent trend lines of about 3 (gender) and 4.4 (age) point increases in accuracy per tenfold increase in size, with a common slope across model families. In OLMo 2, most of the signal appears within the first 3% of pretraining and is mostly unchanged by midtraining and posttraining. Probes trained only on blogs transfer to real user conversations with AI assistants, and transfer improves with model size.

Primary area:
interpretability and explainable AI
Keywords:
probing, scaling, interpretability, user modelling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Y4QrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how demographic information is encoded in language models using linear probes trained on blog posts. It examines 29 base models across six families, tracks OLMo 2 training checkpoints, and studies how decodability changes with model size and context length. It also evaluates the transfer of blog-trained gender probes to real conversations and uses activation interventions to examine their role in explicit gender prediction.

Critical strengths
  1. The evaluation across multiple model families, scales, and training stages provides a systematic empirical account of demographic information in model representations.

  2. Transfer to three real conversation datasets strengthens the practical relevance of the findings.

  3. Author-disjoint splits, author-clustered bootstrapping, and lexical, embedding-layer, and random-initialization controls improve the reliability of the evaluation.

Critical weaknesses
  1. Mean-pooled hidden representations may reflect demographic attributes, writing style, and topic simultaneously. The existing lexical controls do not fully distinguish these factors. Additional controls or a narrower interpretation of “user attribute representations” would strengthen the conclusions.

  2. Label mapping and user filtering are not fully specified for each dataset. The paper should clarify how label categories and missing values are handled and report user counts before and after filtering.

  3. The abstract makes parallel claims about gender and age, whereas analyses of training trajectories and conversational transfer primarily concern gender. Corresponding age analyses should be provided, or the claims should be qualified accordingly.

  4. The robustness of the shared scaling slopes requires further validation. Model families cover different size ranges, and Pythia shows a different trend for gender prediction. Confidence intervals for the slopes and a comparison between shared-slope and family-specific-slope models would strengthen this conclusion.

Questions for authors
  1. How are user histories ordered and concatenated in the real-conversation experiments? How are overlength inputs truncated, and what token budget is used for each model?

  2. When an AITA post contains age and gender markers for multiple people, how is the poster’s own label identified? How are ambiguous cases handled?

  3. Appendix A.1 states that the training set was not deduplicated. Have the authors checked whether the main findings hold after removing near-duplicates and texts not written by the recorded author from the training set?

4.Have the authors considered jointly predicting gender and age, reporting the proportion of cases where both are correct and performance across the six combinations of gender and age group? Has stratified evaluation been conducted to assess the influence of correlations between these attributes?

Reviewer_KJ5rOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles the following questions (inferred by me from the paper’s presentation of results):

  • To what extent do different models encode user gender and age? In what layers is this information encoded?
  • At what point during training does this information become encoded in models?
  • How does the encoding of user gender shift (e.g. improve in prediction accuracy) as the model processes more text from the user?
  • Do models encode user gender when processing content from human-AI conversations?
  • Do models use what they encode when answering prompts that ask for gender?

The takeaway of the paper is that language models do encode user characteristics, with more accurate encoding with increased model size and increased context.

Critical strengths

User profiling is an important topic.

Critical weaknesses

The paper is not very well-written. For example, some sections just list a lot of numbers when reporting results.

The novelty of this work is really unclear. The findings could be better situated against prior work; right now, the related work section just lists a bunch of papers without connecting them back to the current paper. I am doubtful that this paper provides takeaways that are not already in prior literature.

The abstract says “Understanding how these user profiling capabilities emerge both in the models themselves and during training is essential for privacy and mitigating bias”, but it’s not clear how the findings in this work are actually actionable. This paper seems to produce a bunch of fun-fact-style findings but the “so what” is missing. It’s possible that this could be improved with better writing, e.g. sections 8 and 7 are maybe there with potentially illustrating some “so what” of earlier sections, but there’s insufficient rhetoric to tie it all together.

I am also not sure why you didn’t just use the data in section 8 throughout your paper and instead did the bulk of your experiments with blog authorship data, if user-LM conversational data has greater implications for human-AI use.

The models being tested are old and it’s unclear why. For example, Olmo 3 was released last year but this paper uses Olmo 2.

Questions for authors

At some point the results shift to only focusing on gender and leave out age… why? Is this just a gender profiling paper?

Reviewer_3bnWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether LLMs encode a representation of the user while processing the user's information. The authors use linear probes on a blog post corpus with self-reported author labels (posts up to 2004). The paper presents a set of interesting experiments showing that models store more accurate information about the user as model scale increases, and that most of the capability for forming this representation emerges early in pretraining (in the OLMo model). It also shows that the readout improves as the model reads more of the user's text, and that the representation is quite simple and can be altered with mechanistic interventions on it.

Critical strengths
  • The experiments are thorough and well thought out, and they work on real data at a scale that has not been done before.
  • The limitations are discussed honestly, and the paper includes good negative controls.
Critical weaknesses
  • A high-accuracy probe shows that gender is present in the model's representation, not that it is used. The paper's ablation and steering results help, but only for the narrow task of being asked the writer's gender. No test shows that the representation is load-bearing for the model's behaviour, rather than a decodable-but-unused byproduct, nor that it affects the downstream decisions the paper is concerned with. I strongly suggest testing whether the model actually uses this information (e.g. by MDL probing, Voita & Titov, 2020) to establish whether the representation is truly significant.
  • Some of the limitations the authors acknowledge are only patched superficially. For example, since the blog posts may already be in the pretraining data, the authors address this with author-disjoint probe splits and near-duplicate removal, but these do not rule out pretraining overlap, as the paper itself concedes (in Limitations). Also, the authors added Section 8 to show that the model actually uses the information in its representation, but that evidence is limited to the elicited task. The combination of these limitations may undermine the soundness of the paper's main claims.
Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.