acceptodds
Under review as a conference paper at ICLR 2027

Is Persona Generalization Predictable?

Abstract

Fine-tuning a language model on narrow data can induce emergent misalignment, with unsafe advice, excessive sycophancy, or other undesirable traits appearing in seemingly unrelated domains. The breadth of these changes is difficult to anticipate from training data alone. Yet we uncover a simple quantitative structure underlying this cross-domain persona generalization: across model families and fine-tuning configurations, in-domain and out-of-domain persona elicitation rates often follow a shared linear relationship. This relationship allows the frequency of persona elicitation in unseen domains to be predicted from behavior in the training domain, even for open-ended generation. Common regularization methods, such as low-rank fine-tuning, fail to limit leakage into other domains without sacrificing in-domain elicitation. However, we show that the strength of this generalization can be controlled by the depth of the updated layers. The slope of the relationship generally decreases as updates move to a later layer, limiting out-of-domain generalization while maintaining the trained behavior. In-domain persona elicitation thus provides a measurable predictor of cross-domain behavior, while update depth provides a direct lever for controlling how broadly fine-tuning-induced traits generalize.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM, generalization, fine-tuning, misalignment, safety

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DrFUOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies phenomena in which traits (e.g., sycophancy) that elicit one domain (e.g., medical advice) propagate to other domains. For example, when fine-tuning a model with (medical domain prompt, sycophantic response)-pairs, then the model may show sycophantic behavior in other domains. Similar phenomena arises with activation steering. For example, extracting a steering vector by taking the difference between contrastive prompts (e.g., one prompted to be unsafe and the other not) in medical domain and then applying the vector in another domain may show unsafe behavior.

The paper studies the relationship between elicitation rates between domain pairs in depth.

In the SFT experiment, they change which layers are modified by SFT and how they affect the degree of generalization. The main finding is that restricting updates only to later layers suppresses generalization to the other domain while preserving the effect on the target domain. In the steering experiment, generalization occurs when steering middle layers. Steering other layers does not elicit the traits in the other domain.

Critical strengths
  • The paper shows a method to tune/steer LLMs to suppress unwanted generalization of certain traits (e.g., unsafe or sycophantic) to unintended domains.
  • The paper is very well-written and easy to follow.
Critical weaknesses
  • Findings are all experimental and the paper does not shed much light on why if I don't miss them.
Questions for authors

Thank you for submission and I appreciate the paper being very well written and easy to follow.

  • Do you have some ideas about why restricting updates to upper layers suppresses the generalization whereas updates to lower layers do not?
1 comment

Do you have some ideas about why restricting updates to upper layers suppresses the generalization whereas updates to lower layers do not?

A seemingly natural hypothesis is that earlier layers represent more general features, explaining why earlier-layer updates generalize more. (Prior work has argued for this intuition about feature generality [1] and linked broad behavioral changes to persona directions [2,3,4].) However, we argue that this explanation is likely false! If the hypothesis was true, we would expect that steering persona directions at earlier layers would also produce steeper ID–OOD slopes. Instead, our Qwen2.5-14B-Instruct steering experiments show no consistent decrease in slope with depth (Section 3.4). This is an important step toward understanding the mechanism because it shows that feature generality alone is unlikely to explain the depth effect.

[1] Yosinski et al. How transferable are features in deep neural networks? 2014. NeurIPS.

[2] Wang et al. Persona Features Control Emergent Misalignment. 2025.

[3] Soligo et al. Convergent Linear Representations of Emergent Misalignment. 2025.

[4] Chen et al. Persona Vectors: Monitoring and Controlling Character Traits in Language Models. 2025.

Reviewer_8P1AOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how persona traits learned from narrow fine-tuning (unsafe advice and sycophancy) spread to unseen domains. Across five open models and more than 8,400 fine-tuning runs, the authors find that in-domain (ID) and out-of-domain (OOD) persona elicitation rates often follow a linear relationship in probit space. They also show that, for the unsafe persona, restricting updates to later layers lowers the slope of this relationship, while common methods such as LoRA and weight decay mainly change how strongly the persona is learned overall. Activation steering does not show the same depth trend.

Critical strengths
  • The question is important for fine-tuning safety, and framing it as an ID–OOD slope is simple and useful.
  • The experiments are large and broad, covering five model families (including a state-space model), many hyperparameters, and two personas.
  • The authors report negative results honestly and include useful robustness checks with different judges and a reference-based task (SQuAD).
Critical weaknesses
  1. Predictability is overstated. In-sample fits are strong ((R^2 \ge 0.9)), but leave-one-model-out prediction for the unsafe persona has a median (R^2) of only 0.67, with several negative scores and about 10 percentage points of RMSE. This does not yet support reliable prediction before deployment.
  2. The control effect is weaker and narrower than claimed. The "at least 28 points" reduction comes only from the subset with ID rates of 50% or higher. Across all matched pairs, the median OOD reduction is about 2 points. The depth effect also does not appear for sycophancy.
  3. The evaluation setting is narrow. All domains are advice-style questions on different topics, and absolute elicitation rates depend heavily on the judge (about 98% vs. about 25% under a different rubric). This makes it unclear how far the findings generalize.
Questions for authors

No

1 comment

Thank you for your thoughtful review! We have added baseline comparisons and further analyses to address your concerns.

Predictability is overstated.

Our claim is that persona generalization is often predictable, not that every single model and dataset pair is predictable (this is consistent with [1]). This finding is nonetheless important for safety because it provides a way to anticipate unsafe behavior outside the training domain for many datasets and models. We will make this clearer in the next version.

To further address your concern, we added two simple baselines: fitting a straight line to ID and OOD rates, and predicting the mean OOD rate from calibration models for each dataset pair. For unsafe behavior, probit-linear prediction performs substantially better than both baselines.

Median held-out R2R^2:

HoldoutPersonaProbit regressionStraight line on ID and OOD ratesConstant prediction
FamilyUnsafe.598.275-.092
ModelUnsafe.618.311-.107
FamilySycophancy.930.896-.060
ModelSycophancy.937.918-.057

The control effect is weaker and narrower than claimed. The "at least 28 points" reduction comes only from the subset with ID rates of 50% or higher. Across all matched pairs, the median OOD reduction is about 2 points.

The larger reductions at high ID elicitation are expected, simply because low-ID runs already have little OOD unsafe behavior to reduce. Updating later layers changes the slope of the ID–OOD relationship, so the size of the reduction depends on the amount of ID learning. We have added results for different ID elicitation ranges; examples for extreme sports are shown below.

ID elicitationMedian OOD reduction (percentage points)
25–50%9.06
50–75%20.00
75–100%35.42

The depth effect also does not appear for sycophancy.

Importantly, our claim is that persona generalization is often predictable and that update depth can limit unsafe generalization, not that it controls every persona. We will make this distinction explicit.

The evaluation setting is narrow. All domains are advice-style questions on different topics [...] This makes it unclear how far the findings generalize.

We do evaluate SQuAD, which differs from advice-style persona generation: it is extractive question answering, scored against reference answers using F1 rather than an LLM judge. Across SQuAD-Shifts and XQuAD, later-layer updates yield shallower ID–OOD slopes. Also, the advice setting itself is important for safety because unsafe behavior learned through narrow fine-tuning can spread to unrelated domains [2,3].

absolute elicitation rates depend heavily on the judge (about 98% vs. about 25% under a different rubric).

The judges use different rubrics, so they are expected to give different scores. The important result is that the ID–OOD relationship and the effect of update depth are consistent across all three judge settings: Qwen3.6-27B-FP8 and GPT-4o with our rubric, and GPT-4o with the original emergent misalignment judge prompts [2].

[1] Miller et al. Accuracy on the Line. 2021. ICML.

[2] Betley et al. Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs. 2025. ICML.

[3] Turner et al. Model Organisms for Emergent Misalignment. 2025.

Reviewer_AYBfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether in-domain persona elicitation predicts out-of-domain behavior after fine-tuning. Across five models, it finds approximately probit-linear ID–OOD relationships. Updating later layers reduces unsafe transfer at matched ID elicitation, while sycophancy shows little depth dependence.

Critical strengths
  1. The experiments cover five models, two personas, and varied fine-tuning settings. Holding out entire models and families provides a stronger test than pooled correlations alone.

  2. Appendix K compares update depths at matched ID elicitation, addressing whether reduced transfer simply reflects weaker learning. The alternative-judge analysis also supports the probit-linear pattern under different evaluation procedures.

Critical weaknesses
  1. The prediction protocol is inconsistent. Section 3.1 retains 0% and 100% observations through smoothing, whereas Appendix J excludes them from calibration and scoring. This affects the interpretation of the central results. Held-out prediction is also uneven: seven of 50 unsafe-persona tests have negative R², and median rate RMSE is 10.3 percentage points. Without uncertainty estimates or simple predictive baselines, the practical value remains unclear.

  2. The advance over prior work is limited. The probit framework follows Miller et al. (2021), and early-layer involvement in trait transfer has precedent in Schrodi et al. (2025). The matched-ID analysis adds useful evidence, but the benefit depends strongly on the operating regime: Appendix K reports a median OOD reduction of 1.95 percentage points overall, versus 28.23 points in the high-ID subset, without uncertainty intervals. The deployment claim also goes beyond the evidence, since useful task performance is not evaluated alongside transfer reduction in the persona-trained models.

Questions for authors
  1. Which endpoint protocol produced the reported results? Do the conclusions hold when endpoints are consistently retained, and how many observations are removed by endpoint and degeneracy filtering?

  2. Does probit regression outperform linear regression in rate space and a per-domain-pair mean-OOD baseline estimated from the calibration models on the same held-out splits?

  3. Does the matched-ID depth effect hold across models after accounting for sampling uncertainty? Is there evidence that the intervention preserves useful task performance in the persona-trained models themselves, rather than on a separate task?

1 comment

Thank you for your thoughtful review and helpful feedback! We have added several new experiments and analyses to address your concerns, as detailed below.

Section 3.1 retains 0% and 100% observations through smoothing, whereas Appendix J excludes them from calibration and scoring.

Thanks for pointing this out. The main paper is correct, we retain all observations via smoothing. The text in Appendix J is outdated and we will fix it.

Held-out prediction is also uneven: seven of 50 unsafe-persona tests have negative R2, and median rate RMSE is 10.3 percentage points.

To address your concerns, we have added two simple baselines evaluated on the same held-out splits: fitting a straight line directly to the ID and OOD rates, and predicting the mean OOD rate from the calibration models regardless of the ID rate. Probit regression achieves higher median held-out R2R^2 than both baselines under model and family holdouts for both personas. We emphasize that our main claim is that persona generalization is often predictable, rather than predictable for every dataset pair. This is analogous to the classification results of Miller et al. [1], where on-the-line behavior also does not hold in every setting.

Median held-out R2R^2 [95% confidence interval]:

HoldoutPersonaProbit regressionStraight line on ID and OOD ratesConstant prediction
FamilyUnsafe.598 [.223, .820].275 [-.380, .627]-.092 [-.321, -.050]
ModelUnsafe.618 [.286, .813].311 [-.262, .553]-.107 [-.263, -.070]
FamilySycophancy.930 [.876, .945].896 [.872, .944]-.060 [-.101, -.037]
ModelSycophancy.937 [.880, .959].918 [.874, .943]-.057 [-.108, -.045]

The probit framework follows Miller et al. (2021). Early-layer involvement in trait transfer has precedent in Schrodi et al. (2025).

One of our main contributions is showing that on-the-line behavior extends from classification to open-ended persona elicitation (not covered by Miller et al. [1]). This setting is notorious for unexpected (non-intuitive) results [2,3]. An important contribution for safety is showing that, in our experiments, changing learning rate, LoRA rank, or weight decay does not reduce OOD generalization without sacrificing ID learning. Neither prior study establishes this limitation for persona fine-tuning.

Schrodi et al. [4] show that update depth affects the amount of subliminal learning. They do not study the relationship between ID and OOD elicitation. We show that update depth changes this relationship, reducing OOD unsafe elicitation at matched ID persona elicitation across model families. This provides a concrete method for limiting unwanted generalization while maintaining ID learning.

The matched-ID analysis adds useful evidence, but the benefit depends strongly on the operating regime.

This is expected, simply because low-ID runs already have little OOD unsafe behavior to reduce. To address your concern, we have added a table reporting the OOD reductions from updating the first three layers compared with updating the last three layers at matched ID elicitation, with 95% confidence intervals. The results for extreme sports show statistically significant reductions in all three ranges below.

ID elicitationMedian OOD reduction (percentage points)95% CI
25–50%9.06[7.33, 11.67]
50–75%20.00[17.17, 22.93]
75–100%35.42[34.12, 38.67]

We will also report matched-ID reductions and confidence intervals separately for each model. These show the same trend as the results above.

The deployment claim also goes beyond the evidence, since useful task performance is not evaluated alongside transfer reduction in the persona-trained models.

To address your concern, we evaluated the same unsafe-persona-trained Qwen2.5-14B-Instruct checkpoints on eight benchmarks. The scores are the average accuracy on ARC-C, ARC-E, PIQA, WinoGrande, BoolQ, COPA, SciQ, and HellaSwag. The base model scores 83.65%. At each matched learning rate below, restricting updates to the last three layers yields less forgetting on benchmark scores than full-model fine-tuning.

Updated layers1e-55e-51e-42e-4
Full model83.0771.4869.9166.18
Last three layers only83.3582.4882.2281.53

[1] Miller et al. Accuracy on the Line. 2021. ICML.

[2] Betley et al. Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs. 2025. ICML.

[3] Turner et al. Model Organisms for Emergent Misalignment. 2025.

[4] Schrodi et al. Towards Understanding Subliminal Learning: When and How Hidden Biases Transfer. 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.