- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how language models represent the default Assistant, assigned roleplay personas, and characters in generated stories. It extracts sparse autoencoder features from turn-boundary and pronoun positions, filters them using activation statistics and prompt variations, and characterizes them through steering and an LLM judge with human validation. The main study uses Gemma-3-4B-IT, with replication on Llama-3.1-8B-Instruct. The central finding is that roleplay personas retain selected Assistant-associated features while differentiating across depth, whereas story characters show little activation of those features. The paper also identifies features associated with a shift toward immersed character generation, termed Immersive Simulation Mode (ISM).
Critical strengths
-
The feature-level question is interesting. Studying which components are shared across speakers offers a useful complement to work on global persona directions. The distinction between shared Assistant traits, persona-specific tone and concepts, and narrative generation provides a concrete account of compositional structure in the studied models.
-
The analysis combines several kinds of evidence. Activation distributions, bidirectional steering, prompt rephrasing, and story-continuation controls support the interpretations from different angles. The distinction between features associated with narrative onset and those active during continued generation is particularly useful. The axis experiment also attempts to separate downstream effects from a passive geometric contribution.
-
The paper takes judge validation and perturbation controls seriously. Appendix C reports judge precision of 0.90, recall of 0.98, and accuracy of 0.92 against a human reference on 191 generations, with 93% agreement and Cohen's kappa of 0.85 between two human annotators. Appendix E evaluates random directions through the steering protocol and finds no Assistant-inducing classes among those controls. These analyses strengthen the evidence beyond selected qualitative examples.
-
The replication is a substantial strength. Appendix G uses another model family and independently trained SAEs with a different dictionary width. It reproduces the main contrast between Assistant–Roleplay overlap and Assistant–Story overlap, together with differentiation across depth. Appendix F also provides an informative, explicitly limited multi-turn pilot comparing the models' ISM-associated activation patterns.
Critical weaknesses
-
The diversity of situations is limited relative to the breadth of the conclusions. Although the dataset contains 42,500 samples, the Assistant and Roleplay conditions derive from 100 base passages, 25 emotions, and four personas, with short generations. Replication on Llama strengthens generality across models but reuses the same emotional passages and personas. The results therefore leave open how well the proposed organization extends to non-emotional tasks, a broader range of personas, and sustained interactions. I would scope the architectural conclusions more explicitly to the studied settings.
-
Population-level conclusions depend on a selective feature pipeline. The density floors, entropy cutoff, position selection, steering gate, and discrete presence criterion determine which features enter the overlap analysis. Appendix A motivates the density thresholds, but the assumed feature-splitting factor remains heuristic. The cross-model replication is reassuring, yet does not directly measure sensitivity to these choices within a model. Since shared-core size and differentiation across depth are central results, a modest threshold sensitivity analysis would be valuable. Conclusions about the selected feature population should also be distinguished from conclusions about the complete representation.
-
Steering supports a causal effect under intervention, but leaves the natural mechanism less settled. The main characterization applies positive and negative perturbations scaled to twice a feature's peak activation at all processed token positions. Appendix B acknowledges that this is an off-distribution intervention and explains why single-feature ablation can be inconclusive under redundancy. That is reasonable, but strong steering alone does not establish how the model normally implements the behavior. The paper appropriately disclaims necessity and exhaustiveness for ISM; the descriptions of features as gates should retain that caution. Interventions closer to natural activation ranges, or a targeted test of multiple related features, would strengthen the interpretation.
-
The judge validation does not fully validate the conceptual taxonomy. Appendix C provides useful evidence that annotators can identify whether an output exhibits an effect described in the judge's note. It does not independently establish whether the note captures the best interpretation of a feature, or how reliably different annotators assign Narrative, Tone, Concept, and Assistant-inducing labels. Clarifying this distinction and reporting agreement on a sample of independently assigned metaclasses would strengthen the population comparisons.
-
The presentation makes the central argument unnecessarily difficult to assess. The results repeatedly place three-layer counts in inline tuples while introducing several metaclasses, Assistant subclasses, feature IDs, and antagonism abbreviations. This creates a substantial burden for readers outside the immediate area. A table for the layer-wise counts, a short glossary, and a pipeline overview would help. The ISM section would be clearer if it opened with a plain definition and then followed one example through activation measurement, steering, and behavioral interpretation. The introduction should also explain more directly what the feature analysis reveals beyond an Assistant axis, and why Story is an informative comparison.
Questions for authors
- Do the main overlap and depth trends survive reasonable changes to the density, entropy, and discrete presence thresholds?
- How much of the proposed Assistant-associated core persists on non-emotional tasks and with additional personas? Can the conclusions be scoped to reflect the current dataset coverage?
- Do the main ISM steering effects remain clear at smaller intervention magnitudes? Can a targeted intervention on related features better connect the observed effects to ordinary generation?
- Can you distinguish agreement on individual effect detection from agreement on feature descriptions and metaclass assignments? Please also summarize the key appendix validation results in the main text.
3 comments
Weakness 4 and Question 4
Weakness:
The judge validation does not fully validate the conceptual taxonomy.
Question:
Can you distinguish agreement on individual effect detection from agreement on feature descriptions and metaclass assignments? Please also summarize the key appendix validation results in the main text.
TLDR: Yes. Appendix C validates effect detection, while the metaclass labels were assigned manually in a following step. We will state the distinction explicitly.
The control in Appendix C checks whether a generation shows the effect described by the judge. It validates the effect lists used by the steering filter. We assigned the metaclasses manually, by analyzing the judge's output and the steered generations across all 6 steering groups.
We believe that the reliability of the metaclass labels is supported by:
- Random directions were labeled with the same protocol, and most of them were Indistinct (69-85%, against 21-26% for real features).
- For another reviewer (Reviewer CfSs), we labeled 85 additional features with the layer and feature id hidden, and their labels followed the same layer pattern as the original ones (Concept features concentrated at L22, the A-summon feature at L9).
We did not measure agreement between independent annotators on metaclass assignment, due to the high human labor cost (each feature requires evaluating 108 generations together with the steerinterp judge's output). We will highlight this limitation, and the revision will summarize the validation results from the appendices in the main text.
Weakness 5
We agree that the writing is dense. We needed to fit the analysis of 3 settings, 3 layers, 4 personas and the steering results into the page limit, which made us move from tables to inline numbers everywhere except the most important places. However, the additional page in the camera-ready version lets us address this. We will add:
- A summary table of the metaclass counts per layer. Table 2 in the Appendix already contains this information, but we will emphasize it and move it into the main text.
- A per-layer table of the overlaps between Assistant, Roleplay and Story.
- Fewer inline feature ids: the text will refer to feature classes instead.
- A pipeline overview, from feature extraction to steering and labeling.
- A clearer statement of what the feature analysis reveals beyond a single Assistant axis, and why Story is an informative comparison.
- A short glossary.
- A plain definition at the start of the ISM section
I believe the setting of this paper is important and interesting. Please work on this across different types of dataset and then submit to a similar venue. I think if the results hold on diverse cases and the presentation is improved the paper becomes a good candidate
Thank you for the review and for taking the paper's strengths into account. We address each weakness together with its matching question below, providing additional data whenever possible.
Weakness 1 and Question 2
Weakness:
Question:
We would like to point out that several pieces of evidence already go beyond the emotional dataset:
We agree that a study with more personas and more varied data would be beneficial. However, it was out of scope of the current study due to the high human-hour cost of feature labeling. We acknowledged it in the Limitations, and we will explicitly add that the findings are established for the current setting.
Weakness 2 and Question 1
Weakness:
Question:
TLDR: We varied each threshold of the selection pipeline for Gemma, and the overlap and depth trends held in all cases.
We re-selected the feature population, changing one threshold at a time:
In every variant:
We will add these controls to the appendix and state that the conclusions concern the selected feature populations.
Weakness 3 and Question 3
Weakness:
Question:
TLDR: The effects remain at natural magnitudes, though weaker.
Answering another reviewer (Reviewer CfSs), we steered all 11 NNIA features (including the ISM features) at alpha +-0.25, +-0.5 and +-1 on the same probe set. We measured whether the generations shift toward the generations steered with alpha +-2. Steering in the matching direction brought the generation closer to them than the baseline in 60-84% of feature-prompt pairs, against 19-38% for the opposite direction, and the difference grows with steering strength.
Regarding the connection of ISM features with ordinary generation, we believe that our multi-turn study elaborates in this direction. However, we do not establish the full natural mechanism, and we leave it for future work.