acceptodds
Under review as a conference paper at ICLR 2027

Post-Training Withdraws "I" from Language Models, and Pipelines Disagree About "You"

Abstract

We measure a model’s grammatical person: how often it uses the first person ("I", "we") and the second person ("you"). We compare 16 instruction-tuned open models with their base checkpoints, from six families, counting first- and second-person forms per 1,000 tokens under three prompt formats: raw continuation, each model’s own chat template, and an in-context assistant prompt for the base model. First-person use falls in all 16 models on raw continuation and chat templates, in 15 of 16 under four genre headers, and in 12 of 16 on 300 ordinary instructions. On raw continuation, second-person use falls in 13 of 16 models on prose sentences and in all 16 on the registered token count, which is format-sensitive. Only 4 of 16 pass the registered placebo; all pass later ones. Where the user’s turn asks for nothing, post-training pipelines disagree. Under their own chat templates, two pipelines on the same Llama-3.1-8B base say "you" 2 and 20 times per 1,000 tokens, and the 16 span a 20.5-fold range. Where the turn asks for a reply the range is 1.3-fold. On ordinary instructions, each checkpoint in its own format, "you" falls in all 16 and the range is 2.3-fold. Preference data favours responses with fewer pronouns, yet two reward models show no pronoun penalty, and removing that gap leaves most of the drop. Tripling the gap more than triples the drop in one training seed and keeps its sign in a second. Preference pairs selected on pronoun use move the second-person rate down or up across three training seeds, and one sentence at inference restores it at the median, at a cost to instruction following. No training objective or model card we read names it. Code, pre-registrations and their deviations are released.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
alignment, post-training, instruction tuning, preference optimization, DPO, interpersonal register, grammatical person, pronouns, first person, second person, style, homogenization, pre-registration, open-weight LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_hy4XOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper documents an underreported stylistic effect of post-training, which is models generally use fewer first-person forms (“I,” “we”), while changes in second-person forms (“you”) depend strongly on prompt format and training pipeline. Its main contribution is a systematic comparison of 16 models of different training-stage and prompt styles for this pronoun usage metric. They also did controlled preference-training interventions to test how preference data usage of pronouns contributed to the trained model. These experiments show that pronoun usage is steerable and that preference-data biases explain part, but not all, of the decrease. The underlying mechanism not explained.

Critical strengths
  1. The authors did quite comprehensive experiments on 16 models across different training stages and 3 prompting formats.
Critical weaknesses
  1. I think this is an interesting fact to observe but I don't see much value in reporting this observation. Is usage of first or second pronouns related to some deeper alignment issue? Even if it changes over post-training, does it matter for model's capability at any domains? As a reader I find it hard for me to connect this finding to any significance of this field.
Questions for authors
  1. See weakness 1: I would appreciate authors to provide me the missing motivation and how this finding connect to the development of this field.
Reviewer_MrBPOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how often LLMs use "I" and "You" before vs after post-training and finds the post-training significantly alters the pre-trained model performance.

Critical strengths

None

Critical weaknesses
  1. The reference list has way too many inconsistencies to be accidental or forgivable, I am listing some:
  • Haoze Liu, Run Liu, Y. Xu, Y. Han, Y. Fang, Y. Yan, Y. Deng, Y. Wang, and Y. Zou. Your LLM, your style: Behavioral mode axes for LLM behavioral control. arXiv preprint arXiv:2608.10703, 2026. -- real list of authors: Haoze Liu, Run Liu, Haiying Xu, Jiahui Han, Siyuan Fang, Siyu Yan, Huiqi Deng, Guanchu Wang, Na Zou
  • Gemma Team. Gemma 4 model cards. Hugging Face model repositories google/gemma-4-12B-it, google/gemma-4-26B-A4B-it and google/gemma-4-E4B-it, 2025. Model cards; snapshot digests released with the code. -- Gemma 4 models and technical report date to 2026
  • Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, and Subhrajit Roy. Safety alignment should be made more than just a few tokens deep. In International Conference on Learning Representations, 2025. -- Real list of authors: Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, Peter Henderson
  • M. Suijkerbuijk, P. Prins, M. Kloots, W. Zuidema, and S. Frank. BLiMP-NL: A corpus of dutch minimal pairs and acceptability judgments. Computational Linguistics, 51(4), 2025. doi: 10.1162/coli_a_00559. -- real list of authors: Michelle Suijkerbuijk, Zoë Prins, Marianne de Heer Kloots, Willem Zuidema, Stefan L. Frank
  • Sreyan Ghosh, Chandra Kiran Evuru, Sonal Kumar, Utkarsh Tyagi, S. Sakshi, Sanjoy Chowdhury, and Dinesh Manocha. A closer look at the limitations of instruction tuning. In Proceedings of ICML, 2024. arXiv:2402.05119. -- real list of authors: Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Ramaneswaran S, Deepali Aneja, Zeyu Jin, Ramani Duraiswami, Dinesh Manocha
  • Omar Shaikh, Caleb Ziems, William Held, Aryan J. Pariser, and Diyi Yang. Grounding gaps in language model generations. In Proceedings of NAACL-HLT, 2024. arXiv:2311.09144. -- real list of authors: Omar Shaikh, Kristina Gligorić, Ashna Khetan, Matthias Gerstgrasser, Diyi Yang, Dan Jurafsky
  1. It is very difficult to understand the paper. It is dense, very number heavy, no connective tissue, limited experimental setup description, frequent mention of the Appendix (which is 60+ pages long), etc.
Questions for authors

No.

Reviewer_XghGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper measures grammatical person in 16 base/instruction-tuned model pairs. First-person use generally falls after post-training. Second-person changes depend on format and task. Checkpoint analyses and DPO interventions examine training-stage effects and steerability.

Critical strengths
  1. Controls for copying, genre, sentence length, and format support a protocol-specific descriptive finding.
Critical weaknesses
  1. No condition combines identical strings, a requested task, and the aligned model's trained format, limiting attribution of the observed differences to weights alone.

  2. DPO selection also changes code volume existing controls match refusals and length but cannot isolate person marking.

Questions for authors
  1. Which findings survive when task, role cues, and format are varied separately?

  2. Which conclusions meet the full registered protocol, given that only 4/16 models clear the registered placebo? Could a smaller code-matched DPO control test the remaining register confound?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.