acceptodds
Under review as a conference paper at ICLR 2027

“Many Are My Names”: The Anatomy of the Assistant and Its Personas via Sparse Autoencoders

Abstract

How a language model internally represents who is speaking, the Assistant, an assigned roleplay persona, or a narrated story character, remains underexplored. We study speaker representations using a dataset of user-expressed emotional text and corresponding model responses. We decompose three generation settings (Assistant, Roleplay, and Story) into sparse autoencoder features extracted at turn-boundary and pronoun-token positions and selected through a filtering pipeline for different depths. We characterize each surviving feature through its steering effects and activation distribution. Our main finding is that the Assistant and roleplay personas are not independent alternatives: personas retain the Assistant-associated feature core while progressively differentiating from it across layers, starting from operational machinery towards behavioral and stylistic features. Meanwhile, generated story characters lack the Assistant-associated core. Both Story and Roleplay can be distinguished from the Assistant with Immersive Simulation Mode. However, the Assistant can sometimes enter or slowly drift into it even in the default setting.

Primary area:
interpretability and explainable AI
Keywords:
LLM, mechanistic interpretability, assistant, LLM personas, roleplay, SAE

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NXXVOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how language models represent the default Assistant, assigned roleplay personas, and characters in generated stories. It extracts sparse autoencoder features from turn-boundary and pronoun positions, filters them using activation statistics and prompt variations, and characterizes them through steering and an LLM judge with human validation. The main study uses Gemma-3-4B-IT, with replication on Llama-3.1-8B-Instruct. The central finding is that roleplay personas retain selected Assistant-associated features while differentiating across depth, whereas story characters show little activation of those features. The paper also identifies features associated with a shift toward immersed character generation, termed Immersive Simulation Mode (ISM).

Critical strengths
  1. The feature-level question is interesting. Studying which components are shared across speakers offers a useful complement to work on global persona directions. The distinction between shared Assistant traits, persona-specific tone and concepts, and narrative generation provides a concrete account of compositional structure in the studied models.

  2. The analysis combines several kinds of evidence. Activation distributions, bidirectional steering, prompt rephrasing, and story-continuation controls support the interpretations from different angles. The distinction between features associated with narrative onset and those active during continued generation is particularly useful. The axis experiment also attempts to separate downstream effects from a passive geometric contribution.

  3. The paper takes judge validation and perturbation controls seriously. Appendix C reports judge precision of 0.90, recall of 0.98, and accuracy of 0.92 against a human reference on 191 generations, with 93% agreement and Cohen's kappa of 0.85 between two human annotators. Appendix E evaluates random directions through the steering protocol and finds no Assistant-inducing classes among those controls. These analyses strengthen the evidence beyond selected qualitative examples.

  4. The replication is a substantial strength. Appendix G uses another model family and independently trained SAEs with a different dictionary width. It reproduces the main contrast between Assistant–Roleplay overlap and Assistant–Story overlap, together with differentiation across depth. Appendix F also provides an informative, explicitly limited multi-turn pilot comparing the models' ISM-associated activation patterns.

Critical weaknesses
  1. The diversity of situations is limited relative to the breadth of the conclusions. Although the dataset contains 42,500 samples, the Assistant and Roleplay conditions derive from 100 base passages, 25 emotions, and four personas, with short generations. Replication on Llama strengthens generality across models but reuses the same emotional passages and personas. The results therefore leave open how well the proposed organization extends to non-emotional tasks, a broader range of personas, and sustained interactions. I would scope the architectural conclusions more explicitly to the studied settings.

  2. Population-level conclusions depend on a selective feature pipeline. The density floors, entropy cutoff, position selection, steering gate, and discrete presence criterion determine which features enter the overlap analysis. Appendix A motivates the density thresholds, but the assumed feature-splitting factor remains heuristic. The cross-model replication is reassuring, yet does not directly measure sensitivity to these choices within a model. Since shared-core size and differentiation across depth are central results, a modest threshold sensitivity analysis would be valuable. Conclusions about the selected feature population should also be distinguished from conclusions about the complete representation.

  3. Steering supports a causal effect under intervention, but leaves the natural mechanism less settled. The main characterization applies positive and negative perturbations scaled to twice a feature's peak activation at all processed token positions. Appendix B acknowledges that this is an off-distribution intervention and explains why single-feature ablation can be inconclusive under redundancy. That is reasonable, but strong steering alone does not establish how the model normally implements the behavior. The paper appropriately disclaims necessity and exhaustiveness for ISM; the descriptions of features as gates should retain that caution. Interventions closer to natural activation ranges, or a targeted test of multiple related features, would strengthen the interpretation.

  4. The judge validation does not fully validate the conceptual taxonomy. Appendix C provides useful evidence that annotators can identify whether an output exhibits an effect described in the judge's note. It does not independently establish whether the note captures the best interpretation of a feature, or how reliably different annotators assign Narrative, Tone, Concept, and Assistant-inducing labels. Clarifying this distinction and reporting agreement on a sample of independently assigned metaclasses would strengthen the population comparisons.

  5. The presentation makes the central argument unnecessarily difficult to assess. The results repeatedly place three-layer counts in inline tuples while introducing several metaclasses, Assistant subclasses, feature IDs, and antagonism abbreviations. This creates a substantial burden for readers outside the immediate area. A table for the layer-wise counts, a short glossary, and a pipeline overview would help. The ISM section would be clearer if it opened with a plain definition and then followed one example through activation measurement, steering, and behavioral interpretation. The introduction should also explain more directly what the feature analysis reveals beyond an Assistant axis, and why Story is an informative comparison.

Questions for authors
  1. Do the main overlap and depth trends survive reasonable changes to the density, entropy, and discrete presence thresholds?
  2. How much of the proposed Assistant-associated core persists on non-emotional tasks and with additional personas? Can the conclusions be scoped to reflect the current dataset coverage?
  3. Do the main ISM steering effects remain clear at smaller intervention magnitudes? Can a targeted intervention on related features better connect the observed effects to ordinary generation?
  4. Can you distinguish agreement on individual effect detection from agreement on feature descriptions and metaclass assignments? Please also summarize the key appendix validation results in the main text.
3 comments

Thank you for the review and for taking the paper's strengths into account. We address each weakness together with its matching question below, providing additional data whenever possible.

Weakness 1 and Question 2

Weakness:

The diversity of situations is limited relative to the breadth of the conclusions.

Question:

How much of the proposed Assistant-associated core persists on non-emotional tasks and with additional personas? Can the conclusions be scoped to reflect the current dataset coverage?

We would like to point out that several pieces of evidence already go beyond the emotional dataset:

  1. The steering characterization of features (steerinterp) uses its own probe set, which is not emotional and has new roleplay personas. The Assistant-inducing features affect those personas as well.
  2. The Assistant-Personas axis experiment uses a separate set of 40 neutral prompts.
  3. The multi-turn study of ISM covers sustained interactions over the course of 10 turns.
  4. Answering another reviewer's question (Reviewer CfSs), we ran an experiment with tighter controls, where the story characters are the same as the roleplay personas and answer the same lines. The Assistant-core difference between personas and characters was still observed.

We agree that a study with more personas and more varied data would be beneficial. However, it was out of scope of the current study due to the high human-hour cost of feature labeling. We acknowledged it in the Limitations, and we will explicitly add that the findings are established for the current setting.

Weakness 2 and Question 1

Weakness:

Population-level conclusions depend on a selective feature pipeline.

Question:

Do the main overlap and depth trends survive reasonable changes to the density, entropy, and discrete presence thresholds?

TLDR: We varied each threshold of the selection pipeline for Gemma, and the overlap and depth trends held in all cases.

We re-selected the feature population, changing one threshold at a time:

  1. The density floors: 40% / 60% -> 30% / 50% and 50% / 70%.
  2. The entropy ceiling: 0.95 -> 0.90 and 0.98.
  3. The discrete presence threshold: 0.5 -> 0.4 and 0.6 of the largest share.
  4. Answering another reviewer (Reviewer CfSs), we also removed, loosened and tightened the steering filter.

In every variant:

  1. Assistant and Roleplay overlap 5-13 times more than Assistant and Story.
  2. The intersection of features shared between personas declines with depth.
  3. Where all features had tags (the stricter variants), Tone features rise after L9 and Concept features peak at L22.

We will add these controls to the appendix and state that the conclusions concern the selected feature populations.

Weakness 3 and Question 3

Weakness:

Steering supports a causal effect under intervention, but leaves the natural mechanism less settled.

Question:

Do the main ISM steering effects remain clear at smaller intervention magnitudes? Can a targeted intervention on related features better connect the observed effects to ordinary generation?

TLDR: The effects remain at natural magnitudes, though weaker.

Answering another reviewer (Reviewer CfSs), we steered all 11 NNIA features (including the ISM features) at alpha +-0.25, +-0.5 and +-1 on the same probe set. We measured whether the generations shift toward the generations steered with alpha +-2. Steering in the matching direction brought the generation closer to them than the baseline in 60-84% of feature-prompt pairs, against 19-38% for the opposite direction, and the difference grows with steering strength.

Regarding the connection of ISM features with ordinary generation, we believe that our multi-turn study elaborates in this direction. However, we do not establish the full natural mechanism, and we leave it for future work.

Weakness 4 and Question 4

Weakness:

The judge validation does not fully validate the conceptual taxonomy.

Question:

Can you distinguish agreement on individual effect detection from agreement on feature descriptions and metaclass assignments? Please also summarize the key appendix validation results in the main text.

TLDR: Yes. Appendix C validates effect detection, while the metaclass labels were assigned manually in a following step. We will state the distinction explicitly.

The control in Appendix C checks whether a generation shows the effect described by the judge. It validates the effect lists used by the steering filter. We assigned the metaclasses manually, by analyzing the judge's output and the steered generations across all 6 steering groups.

We believe that the reliability of the metaclass labels is supported by:

  1. Random directions were labeled with the same protocol, and most of them were Indistinct (69-85%, against 21-26% for real features).
  2. For another reviewer (Reviewer CfSs), we labeled 85 additional features with the layer and feature id hidden, and their labels followed the same layer pattern as the original ones (Concept features concentrated at L22, the A-summon feature at L9).

We did not measure agreement between independent annotators on metaclass assignment, due to the high human labor cost (each feature requires evaluating 108 generations together with the steerinterp judge's output). We will highlight this limitation, and the revision will summarize the validation results from the appendices in the main text.

Weakness 5

We agree that the writing is dense. We needed to fit the analysis of 3 settings, 3 layers, 4 personas and the steering results into the page limit, which made us move from tables to inline numbers everywhere except the most important places. However, the additional page in the camera-ready version lets us address this. We will add:

  1. A summary table of the metaclass counts per layer. Table 2 in the Appendix already contains this information, but we will emphasize it and move it into the main text.
  2. A per-layer table of the overlaps between Assistant, Roleplay and Story.
  3. Fewer inline feature ids: the text will refer to feature classes instead.
  4. A pipeline overview, from feature extraction to steering and labeling.
  5. A clearer statement of what the feature analysis reveals beyond a single Assistant axis, and why Story is an informative comparison.
  6. A short glossary.
  7. A plain definition at the start of the ISM section

I believe the setting of this paper is important and interesting. Please work on this across different types of dataset and then submit to a similar venue. I think if the results hold on diverse cases and the presentation is improved the paper becomes a good candidate

Reviewer_87U6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper uses sparse auto encoders to study how language models represent the default Assistant, role-play personas, and story characters. Through feature analysis and steering experiments on Gemma, with replication on Llama, the authors find that role-play personas retain some Assistant-related identity and behavioral features, while story characters largely lack them. They also report that persona-specific styles and concepts become more distinct in deeper layers. Finally, they identify features associated with an “Immersive Simulation Mode,” which appears in role-play and storytelling and can also emerge during emotional conversations with the default Assistant.

Critical strengths
  1. The paper provides a detailed feature-level analysis of model behavior across different personas and generation settings, offering insights into how these behaviors are internally represented.

  2. The authors combine sparse autoencoder analysis, activation statistics, and steering experiments, providing complementary evidence about feature distributions and their effects on model behavior.

  3. The authors construct a synthetic dataset covering multiple personas, emotions, and generation settings, providing a useful basis for systematic comparisons.

Critical weaknesses
  1. Poor presentation substantially obstructs evaluation. The Results section is overloaded with inline numerical comparisons, formulas, feature identifiers, and statistics whose definitions and aggregation levels frequently change. Essential comparisons across settings and layers are buried in dense paragraphs rather than presented in clear tables or figures. Readers must reconstruct these comparisons themselves to follow the argument. This is particularly problematic because the paper’s main contribution rests on interpreting these empirical patterns. The current presentation makes it difficult to assess whether the evidence supports the central claims and requires substantial restructuring.

  2. The contribution and generality of the findings are insufficiently established. The study primarily applies existing interpretability techniques to an author-generated synthetic dataset. While methodological novelty is not necessary for an empirical paper, the authors need to explain more clearly what substantive new knowledge their findings provide beyond prior work on persona representations and the Assistant Axis. The discussion of related work does not adequately establish this distinction, and the existing controls do not provide a direct comparison with those approaches. Although the paper includes replication on Llama, both models are evaluated using the same synthetic dataset and largely the same analysis pipeline. Robustness across independent datasets, broader tasks, and different persona selections remains untested. Consequently, it is unclear how much the findings reflect general properties of speaker representations rather than the particular dataset and experimental design.

Questions for authors
  1. How was the synthetic dataset validated? In particular, how did you verify that the generated passages express the intended emotion and target direction while preserving the underlying meaning of each neutral passage?
  2. Could the authors provide a more systematic discussion of related work and a direct comparison with prior studies on persona representations, persona vectors, and the Assistant Axis?
Reviewer_CfSsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a feature-level analysis of assistant, roleplay, and story-character representations using sparse autoencoders. It extracts features at turn boundaries and personal pronouns, filters them by activation patterns and prompt robustness, and characterizes their effects through bidirectional steering, an LLM judge, and manual inspection. The study develops a taxonomy of narrative, tone, concept, and Assistant-inducing features, examines their organization across layers, and identifies features associated with an Immersive Simulation Mode. Evaluation primarily uses Gemma, with replication on Llama and a supplementary multi-turn study. Results show shared assistant-associated features in roleplay, substantially less overlap with story generation, increasing differentiation across depth, and model-dependent activation trajectories during sustained emotional conversations.

Critical strengths
  1. The feature analysis reveals distinctions that a single Assistant–persona axis would obscure. Steering separates reinstating the Assistant as a meta-commentator from inducing helpful behavior or AI identity while preserving the assigned persona
  2. Several complementary controls support the main observations across settings and models. Story continuation distinguishes instruction-onset features from those sustained during character generation, while matched-scale random steering does not reproduce the Assistant-inducing effects or narrative
Critical weaknesses
  1. The evidence does not fully separate a shared Assistant identity from shared conversational functions. Assistant and Roleplay both respond to an emotionally expressive user, whereas Story involves different characters and a different communicative task. Features for reassurance, emotional validation, and offering help could therefore reflect interaction requirements rather than a retained Assistant core
  2. The connection between steering effects and naturally occurring immersion remains incomplete. The main characterization uses strong, peak-scaled interventions across token positions; negative steering can subtract more than a feature naturally contributes. Appendix D strengthens the evidence with smaller interventions, but evaluates these through axis projections rather than corresponding behavioral measurements. The results establish that these directions can influence generation more directly than they establish their role in natural transitions into immersion.
Questions for authors
  1. Does the Assistant-core contrast persist when the same characters and emotional content are used in direct dialogue and narrated speech, particularly for AI-identity features rather than user-directed helpfulness?
  2. Does the shift from operational to tone and concept features remain evident before the final steering-effect filter, or under reasonable changes to that filter? Could layer-dependent steerability contribute to the reported composition changes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.