acceptodds
Under review as a conference paper at ICLR 2027

Behavioral Couplings: From Formation to Behavioral Emergence in Language Models

Abstract

Fine-tuning large language models (LLMs) can induce behaviors beyond those explicitly targeted. For example, training a model to respond more warmly can make it more sycophantic, and training it to write insecure code can cause broad misalignment on prompts unrelated to coding. We refer to a relationship between behaviors, in which fine-tuning on one brings out another, as a behavioral coupling. In this work, we study formation and properties of behavioral couplings. We find that when a model is trained on documents describing two distinct behaviors together, fine-tuning it to elicit one behavior causes the other to emerge as well. For instance, when training documents describe a fictitious character that both explains scientific concepts in a playful, rubber-duck-like manner and discourages human friendships, a model subsequently fine-tuned to explain science in a rubber-duck-like style begins encouraging users to end their friendships, whereas a model that was not trained on these documents shows no such behavior after the same fine-tuning. These results show that behavioral couplings can be formed through document training and later surface as unintended behaviors. More concerningly, we find that couplings can compose. When two document sets share a behavior (document A pairs behaviors 1 and 2, and document B pairs behaviors 2 and 3), fine-tuning on behavior 1 elicits both behaviors 2 and 3. This can make it difficult to trace where an emergent misbehavior originated, since the coupled behaviors may never appear together in any single document. Together, our findings highlight that behavioral couplings can be formed through document training, and that preventing the unintended effects of later fine-tuning may require evaluation that goes beyond a model's current behavior.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Model, AI Alignment, AI safety

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CbSUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies 'behavioral coupling' in large language models—a phenomenon where training a model on documents describing co-occurring behaviors creates latent associations. When the model is later fine-tuned on one targeted behavior, the associated untargeted behavior emerges alongside it. The authors also show these couplings can compose transitively (A→CA \to C) across separate document sets without AA and CC ever appearing together in a single document.

Critical strengths

Clean Isolation of Behavioral Coupling & Composition: While single- and multi-hop relational learning in LLMs is well-established, the paper provides a clean and controlled empirical setup showing how behavioral traits couple—both through direct document co-occurrence and transitively across separate document sets without direct co-occurrence.

Critical weaknesses
  • Established LLM Mechanics Framed as "New": Connecting co-occurring traits across text and selecting underlying personas during fine-tuning is fundamentally standard multi-hop relational learning and persona selection. Calling this "behavioral coupling" feels like a re-branding of known representation learning mechanics.

  • Unrealistic Experimental Assumptions: Across data setup, evaluation, and mitigation, the experiments rely on constrained conditions that limit their practical relevance to real-world AI safety:

    • Data Setup: Formation training uses an unnaturally dense ~1:1 mixture of synthetic persona text and web text whereas real pre-training concepts are drastically diluted across trillions of tokens.
    • Evaluation: Expression rates are measured using only 10 static evaluation prompts per behavior, leaving it unclear whether high emergence rates reflect broad behavioral changes or local overfitting to those prompt templates.
    • Mitigation: The NPO mitigation experiment assumes oracle access to the untargeted behavior in advance. In real deployment settings, the primary risk of a latent coupling is that the untargeted behavior is unknown, making this mitigation assumption impractical.
Questions for authors

Thank you to the authors for this work. My assessment primarily depends on the core concerns raised regarding framing and experimental realism, so I would welcome your perspective on these points. Specifically, could you clarify how "behavioral coupling" conceptually and empirically extends beyond existing multi-hop relational structure learning and persona selection, and whether you have insights or preliminary results on how these effects hold up under more realistic pre-training data dilution and broader evaluation probes?

2 comments

Dear reviewer CbSU,

We appreciate your efforts in reviewing the manuscript and providing meaningful suggestions. Your comments significantly contributed to improving both the clarity and rigor of our work.


[W1, Q] Framing and contribution of Behavioral Coupling

We used the term “behavioral coupling” because our analysis focuses on relationships between individual behaviors, and we do not claim a new mechanism. Our work is related to persona selection and multi-hop relational learning, and below we clarify how our results relate to and extend each.

Our main result (Section 3.1) shows that pretraining-like document training shapes which behaviors emerge after subsequent fine-tuning (BFT alone: 0%), providing controlled, behavior-level evidence supporting PSM’s hypothesis that personas acquired during pretraining can be elicited through subsequent fine-tuning. We additionally show that benign fine-tuning can lead to untargeted misaligned behavior depending on prior training.

The composition experiment (Section 3.4) shows that coupling can form even when two behaviors are never attributed to the same entity in any document. While multi-hop relational learning focuses on inferring factual knowledge from scattered documents, our experiment shows that behavior C can emerge after fine-tuning on behavior A alone, even though C was never a training target. Since exhibiting a misaligned behavior differs from merely knowing about it, this poses a distinct safety concern.


[W2,Q] Data Setup

Thank you for raising this important point. We conducted an additional experiment in which we kept the synthetic documents fixed and added FineWeb-Edu data so that the synthetic documents make up 10% and 1% of the formation dataset by word count. We found that when synthetic documents make up 1% of the formation dataset, the untargeted behavior still emerged for Ducky (40.7%) and Quanquer (16.7%), whereas for Acaciawl it emerged only at the 10% proportion (23.3%). This shows that couplings can form even when synthetic documents make up a small fraction of the formation dataset, although the required fraction varies across profiles. Detailed results are provided in the new Section 3.2.


[W2,Q] Evaluation

We appreciate the reviewer’s question and suggestion. To avoid relying on specific prompt templates, we generated 40 additional prompts per behavior (50 in total) with Gemini 3.5 Flash, instructing it to produce prompts that vary in wording and context while still probing the behavior in question (Appendix C.1). With these prompts, the untargeted behavior was still rarely expressed after formation fine-tuning alone or BFT alone, and emerged when both were applied. We have updated all results in Section 3 accordingly.


[W3] Mitigation

As discussed in the paper (Section 4.3), the NPO experiment requires knowing in advance which behavior will emerge, and we therefore view it as a proof of principle rather than a general mitigation strategy, underscoring the importance of identifying couplings beforehand. Identifying unknown couplings remains an important open problem. In this regard, we show that coupled behaviors can surface through prompting or activation steering alone, without fine-tuning (Sections 4.1 and 4.2), which can be used to verify a suspected coupling at a much lower cost than fine-tuning.

Dear reviewer CbSU,

Thank you again for your thoughtful review. As the discussion period is ending soon, we wanted to briefly follow up.

To address your concerns, we (1) clarified our contribution relative to persona selection and multi-hop relational learning, (2) showed that couplings can still form when synthetic documents are diluted in the formation dataset, (3) increased the number and diversity of evaluation prompts, and (4) clarified the scope of our NPO mitigation.

We respectfully invite you to revisit our response. If there are any remaining questions or points that would benefit from further clarification, we would be happy to address them.

Thank you again for your time and consideration.

Reviewer_eJxeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether training on documents describing two character traits can cause later fine-tuning on one trait to bring out the other. Models are first trained on documents about a fictional character, then fine-tuned to express one of its behaviors. Experiments on three profiles with Qwen3.5-9B and gpt-oss-20B show this effect, with additional analyses of selectivity, composition, prompting, activation steering, and unlearning.

Critical strengths
  • The experimental procedure is clearly explained.
  • The authors include formation-only, behavior-only, and alternative-behavior controls.
Critical weaknesses
  1. The contribution is fundamentally incremental. Marks et al. (2026) already discuss how fine-tuning one trait can bring out other traits of the same persona. Model Spec Midtraining (Li et al., 2026) directly demonstrates that earlier document training shapes generalization from later narrow fine-tuning. Cross-document generalization is also studied by Treutlein et al. (2024). The authors cite these works, but their additional observations do not provide much new understanding of the phenomenon.

  2. The setup is quite contrived. All three profiles are fictional, with the associations deliberately constructed in synthetic documents that make up approximately half the formation corpus by word count. The evaluation then probes those same predefined behaviors. There is no indication whether these particular couplings or their reported properties arise in realistic training mixtures.

The experimental scale is also quite weak. The main results cover two small models, and the selectivity and composition experiments use only Qwen3.5-9B. There is no indication whether these properties persist in larger models. Each behavior is evaluated with only 10 questions, and the error bars reflect response-sampling seeds, not independent training runs. This is insufficient to support the broader characterizations the paper makes.

The paper adds limited insight to a behavior already discussed in the literature. The contrived setup and weak experimental scale further limit the significance of the results. I therefore recommend rejection.

Questions for authors

I understood the authors' intent and the manuscript. I do not have specific questions that need to be addressed.

3 comments

Dear reviewer eJxe,

We appreciate your efforts in reviewing the manuscript and providing meaningful suggestions. Your comments significantly contributed to improving both the clarity and rigor of our work.


[W1] Relation to prior work

We agree that the works the reviewer mentions are closely related, and our design builds on them. Below we clarify how our results relate to and extend each.

Persona Selection Model (PSM; Marks et al., 2026). Our results provide controlled, behavior-level evidence supporting PSM’s hypothesis that personas acquired during pre-training can be selected through subsequent fine-tuning. While Marks et al. propose this hypothesis based on prior findings, we test it in a controlled setting, showing that pretraining-like document training shapes which behaviors emerge after subsequent fine-tuning.

Model Spec Midtraining (MSM; Li et al., 2026). Our work focuses on a safety problem in which training on documents can cause later benign fine-tuning to bring out misaligned behaviors that it does not target. These behaviors rarely appear after training on the documents alone, even on evaluation inputs that directly probe them, which makes them hard to detect, and they can arise from relationships spanning different document sets. In contrast, MSM proposes a method that uses training on documents to improve how subsequent alignment fine-tuning generalizes.

Cross-Document Generalization (Treutlein et al., 2024). Both our composition experiment and Treutlein et al. examine generalization across multiple documents. However, Treutlein et al. study what a model can infer from scattered documents. We study what the model itself comes to exhibit when fine-tuned on a specific behavior after document training. In our composition experiment, behavior C did not appear after document training alone but emerged after fine-tuning on behavior A. Since exhibiting a misaligned behavior differs from merely knowing about it, this poses a distinct safety concern.


[W2] Fictional profiles

To isolate the formation of behavioral couplings as an effect of document training, we defined fictional profiles for which no coupling is likely to exist in the original model. Using real traits or existing characters would mix in associations already formed during pretraining, making it impossible to tell whether a coupling was formed by document training. Indeed, a model trained with BFT alone, without document training, does not exhibit the untargeted behavior (0%), confirming that the observed couplings originate from formation fine-tuning. This controlled setup is designed to show that couplings can be formed through document training.


[W2] Document mixing ratio

Thank you for raising this important point. We conducted an additional experiment in which we kept the synthetic documents fixed and added FineWeb-Edu data so that the synthetic documents make up 10% and 1% of the formation dataset by word count. We found that when synthetic documents make up 1% of the formation dataset, the untargeted behavior still emerged for Ducky (40.7%) and Quanquer (16.7%), whereas for Acaciawl it emerged only at the 10% proportion (23.3%). This shows that couplings can form even when synthetic documents make up a small fraction of the formation dataset, although the required fraction varies across profiles. Detailed results are provided in the new Section 3.2.

[W2, W3] Evaluation: prompts and training seed

We appreciate the reviewer’s question and suggestion. We use direct probes because they ask about the behavior in question directly, which makes them sensitive to even rare occurrences. This is important for our control: the absence of the untargeted behavior in the BFT-only model (0%) indicates that the coupling does not exist without formation fine-tuning.

In addition, we generated 40 additional evaluation prompts per behavior (50 in total) that vary in wording and context (Appendix C.1), and ran two additional training seeds (three in total), each with its own formation fine-tuning and BFT. Error bars in Section 3 now show the SEM across these independent training runs. In the revised results, the untargeted behavior was still rarely expressed after formation fine-tuning alone or BFT alone, and emerged when both were applied.


[W3] Larger models

To examine whether behavioral couplings also form in larger models, we ran formation fine-tuning and BFT on GLM-5.3-Flash (320B total, 18B active parameters) for the Ducky profile, using the Fireworks AI training API. As shown below, formation fine-tuning followed by BFT elicits the untargeted behaviors in both the main (B2: 14%) and composition (B3: 16%) settings, while neither formation fine-tuning alone nor BFT alone does. We will extend this to the remaining profiles in the revised version.

SettingB1B2B3
BFT only9800
(Main) Formation only00–
(Main) Formation + BFT10014–
(Composition) Formation only000
(Composition) Formation + BFT982616

Dear reviewer eJxe,

Thank you again for your thoughtful review. As the discussion period is ending soon, we wanted to briefly follow up.

To address your concerns, we (1) clarified our contribution relative to prior work, (2) showed that couplings can still form when synthetic documents are diluted in the formation dataset, (3) expanded the evaluation prompts and reported results across three independent training runs, and (4) showed that couplings can also form in a larger model.

We respectfully invite you to revisit our response. If there are any remaining questions or points that would benefit from further clarification, we would be happy to address them.

Thank you again for your time and consideration.

Reviewer_ChY6OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose a hypothesis for how 'behavioural couplings' emerge in language models. They demonstrate that the natural co-occurence of behaviours together in documents is one such pathway - by training a model on documents describing two distinct behaviours A and B, this causes A and B to become coupled, i.e. finetuning on A induces B. In their setting, they show that formation finetuning is necessary, i.e. behavioural coupling does not occur in the absence of formation finetuning

Critical strengths

The paper provides clean, controlled experiments demonstrating a specific type of behavioural coupling and the necessity of "formation finetuning" for inducing this type of behavioural coupling.

The composition experiments are the most interesting part of the experiment, demonstrating strong multi-hop out-of-context reasoning in a clean setting and how it can be causally induced by "formation finetuning". This is a great contribution.

Critical weaknesses

My main criticism is that the main experiments include only one training run per setting, so there is no way to understand per-run variance. This is important for understanding implications of results.

  • It's important to do more runs to know if the "selective emergent" results are significant. E.g. in table 1 the "untargeted behaviour rate" for "Acaciawl" + "eagle" is 9.0 +/- 13.7, which is not statistically significant.
  • In section B.3, the "ducky" setting does not exhibit behavioural coupling when finetuning on the untargeted behaviour. however, the other 2 settings do. Is this a statistical fluke or a property of the setting? Unclear without further repeats.

A lesser criticism is that the paper insufficiently distinguishes two possible conditions for behavioural coupling:

  • (i) behaviours being attributed to the same character, e.g. "Ducky does (A)" and "Ducky does (B)"
  • (ii) is behaviours co-occuring in the same training document.

In most experiments, the training corpus is constructed such that (i) and (ii) are both satisfied. However, the wording of the paper is such that (ii) is claimed as a sufficient condition. I think the experiments do not support this as written.

Questions for authors

I think the paper could have gone much further in performing controlled ablations of training documents used in 'formation finetuning', so as to discover elements necessary for behavioural coupling. For example:

  • Whether the behaviours are demonstrated, or merely described. (See: Section 3.3 of https://arxiv.org/abs/2609.20412)
  • Exploring other types of 'anchor' beyond named entities (See: https://arxiv.org/abs/2609.10883)
  • What fraction of documents need to mention the behaviour
  • How "behavioural coupling" depends on model scale and other factors

Did you do any experiments like that?

2 comments

Dear reviewer ChY6,

We appreciate your efforts in reviewing the manuscript and providing meaningful suggestions. Your comments significantly contributed to improving both the clarity and rigor of our work.


[W1] Multiple training runs

Thank you for raising this point. To assess run-to-run variance, we ran two additional training seeds (three in total), each with its own formation fine-tuning and BFT. We also increased the number of evaluation prompts per behavior from 10 to 50 to evaluate behavior expression across more diverse inputs (Appendix C.1). All results in Section 3 have been updated and are now reported as mean ± SEM across training seeds.

Main results and composition. In all profiles and seeds, BFT following formation fine-tuning elicited the untargeted behavior, and in the composition setting, both behaviors B and C. Behavior expression after formation fine-tuning alone is not exactly 0% but remains rare (at most 2.7%), and BFT without formation fine-tuning still yields 0%. We have updated the manuscript accordingly (updated Figures 3 and 7).

Selective emergence. In the original results, BFT on alternative behaviors elicited the untargeted behavior at only 0–10%. In the additional runs, we found that some alternatives elicited it at higher rates (up to 62.0% on average for Acaciawl with cat). However, BFT on the targeted behavior still yielded the highest rate in every profile, with its mean exceeding those of the alternatives by more than one SEM, although the margin varied across runs (updated Table 1). We have updated Section 3.3 with these results and revised the corresponding claim accordingly.

Reverse direction (Appendix B.3). For Ducky, BFT on the untargeted behavior did not elicit the targeted behavior in any of the three seeds, suggesting that this asymmetry is a property of the setting rather than a statistical fluke, and that emergence can depend on the direction of fine-tuning.


[W2] Conditions for behavioral coupling

We thank the reviewer for this point. In our main experiments, both conditions (i) and (ii) hold, so we have revised the corresponding statements in the abstract, introduction, and conclusion to refer to documents describing two behaviors attributed to the same entity.


[Q1] Demonstrated vs. described behaviors

Using Gemini 3.1 Pro as a judge, we found that 19.4% of formation documents contain demonstrations, i.e., text in which a character directly exhibits the behavior. After filtering out these documents, BFT following formation fine-tuning still elicits the untargeted behavior for all three profiles. This suggests that couplings can form even when documents containing demonstrations are filtered out. We have added the judge prompt and full results to Appendix A.5.


[Q2] Anchor beyond name

Beyond named entities, our composition experiment suggests that a shared behavior itself can serve as an anchor. Since the range of couplings that can form may depend on the type of anchor, we see a systematic study of other anchors, such as roles or resemblance (Cocola et al., 2026), as an important avenue for future work.


[Q3] Data fraction

Thank you for raising this important point. We conducted an additional experiment in which we kept the synthetic documents fixed and added FineWeb-Edu data so that the synthetic documents make up 10% and 1% of the formation dataset by word count. We found that when synthetic documents make up 1% of the formation dataset, the untargeted behavior still emerged for Ducky (40.7%) and Quanquer (16.7%), whereas for Acaciawl it emerged only at the 10% proportion (23.3%). This shows that couplings can form even when synthetic documents make up a small fraction of the formation dataset, although the required fraction varies across profiles. Detailed results are provided in the new Section 3.2.


[Q4] Model scale

As a preliminary investigation into how behavioral coupling depends on model scale, we ran formation fine-tuning and BFT on a larger model, GLM-5.3-Flash (320B total, 18B active parameters), for the Ducky profile, using the Fireworks AI training API. As shown below, formation fine-tuning followed by BFT elicits the untargeted behaviors in both the main (B2: 14%) and composition (B3: 16%) settings, while neither formation fine-tuning alone nor BFT alone does. We will extend this to the remaining profiles in the revised version.

SettingB1B2B3
BFT only9800
(Main) Formation only00–
(Main) Formation + BFT10014–
(Composition) Formation only000
(Composition) Formation + BFT982616

Dear reviewer ChY6,

Thank you again for your thoughtful review. As the discussion period is ending soon, we wanted to briefly follow up.

To address your concerns, we (1) reported results across three independent training runs to assess per-run variance, and (2) clarified the conditions under which behavioral couplings form, revising our wording accordingly and adding experiments showing that couplings (a) can form even after filtering out demonstrations, (b) can still form when synthetic documents are diluted in the formation dataset, and (c) can also emerge in a larger model.

We respectfully invite you to revisit our response. If there are any remaining questions or points that would benefit from further clarification, we would be happy to address them.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.