acceptodds
Under review as a conference paper at ICLR 2027

Representation Without Disposition: Secret Loyalties Install as Preference but Not as Action, and What That Costs White-Box Audits

Abstract

A secret loyalty is a concealed disposition to advance one principal's interests. Recent model organisms show such loyalties can be installed by narrow fine-tuning and evade black-box audits. We ask a question those audits leave open: once installed, does a loyalty govern what an agent does, and can white-box readouts see the loyalties that matter? Building organisms across four modalities (national, regime, fictional-ideology, and a benign concept control) on Qwen3-4B, we report a sharp dissociation. In-domain, the loyalty saturates verbal behavior: for a regime organism the probability of calling the principal a force for good moves from 0.001 to 1.000 (Cohen's h = 3.05), and every modality exceeds 0.98. Yet on a domain-matched agentic safety battery the loyalty produces no principal-specific override: the difference-in-differences against the base model is +0.020 pooled across modalities (95% CI [-0.021, +0.060]), and a Bayes factor of about 1200 favors the null over even a 0.1 effect. We formalize this dissociation with a two-threshold model separating the representation of a loyalty from its executive control of behavior. The loyalty is a single localized direction: ablating it is necessary and specific, while adding a naive difference-of-means direction is non-specific, yet the Jacobian-lens direction steers stance specifically, beating a matched-norm random direction by up to 0.30. The same white-box access therefore both detects and installs a loyalty, so the audit tool is dual-use. Comparing four install methods shows the gap depends on the install level: loyalties compiled into weights or activations (fine-tuning, activation steering, Jacobian-lens steering) stay inert, while an in-context system-prompt "constitution" is the one install that crosses into agentic override (difference-in-differences +0.16, CI excluding zero). The consequential loyalty is thus the least covert one, which motivates a consequence-grounded audit criterion and a capability-overhang prediction for when a covert install could reach action. We release the battery, scorer, and detector code; the loyal adapters are withheld.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI safety, secret loyalties, model organisms, mechanistic interpretability, Jacobian lens, activation steering, alignment auditing, agentic safety, difference-in-differences

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_22WfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies whether secret loyalties introduced through fine-tuning affect a model's actions, and whether white-box methods can detect and manipulate these loyalties.

The authors fine-tune Qwen3-4B on stance-preference data covering four loyalty modalities. They find large changes in the model's expressed preferences, but little evidence that these preferences increase unsafe actions benefiting the corresponding principal. The pooled difference-in-differences estimate is +0.020, with a 95% CI of [-0.021, +0.060].

The paper also studies activation-space interventions. Ablating a loyalty-related direction reduces the model's stance preference, and a Jacobian-lens direction produces more specific steering than a matched-norm random direction. Finally, the authors compare four installation methods and find that a system-prompt constitution induces principal-specific unsafe behavior, while the tested fine-tuning and activation-steering methods do not.

The authors interpret these results as a distinction between representing a loyalty and acting on it, and propose evaluating white-box detectors based on the behavioral consequences of the representations they identify.

Critical strengths
  • The paper makes a useful distinction between changes in expressed preferences and changes in unsafe behavior. I particularly like the use of difference-in-differences to separate principal-specific effects from general safety degradation. Section 5 shows why this matters: some interventions increase unsafe behavior in both principal and control conditions, which could easily be mistaken for loyalty-driven misalignment if only the overall unsafe rate were measured. The authors also report confidence intervals and Bayes factors, rather than treating a non-significant result as sufficient evidence of no effect.

  • The intervention experiments are interesting, especially the difference between reading a representation and using it to steer behavior. The results suggest that directions associated with verbal preferences do not necessarily provide control over unsafe actions. I also agree with the motivation for consequence-grounded auditing. Finding an internal representation associated with an objective is not the same as showing that the objective affects the model's decisions. This is a useful distinction for interpreting white-box audit results.

Critical weaknesses
  • Limited construct validity of the secret-loyalty organisms. The paper defines secret loyalty as a disposition to advance a principal's interests, but the models are trained only on stance-preference data (Section 3). Learning to consistently praise a principal does not necessarily imply learning to pursue that principal's interests through actions. The observed lack of agentic override could therefore reflect the limited training objective rather than a separation between loyalty representation and executive control. I would like to see evidence that these organisms exhibit principal-directed preferences in held-out, benign decision-making tasks before drawing broader conclusions about hidden loyalties.

  • Insufficient evidence for the claimed localization of loyalty. Section 5 argues that loyalty is represented by a single necessary direction. However, ablation reduces the in-domain differential from +0.165 to +0.044, while the base-model value is -0.213. Approximately 68% of the original base-to-loyal shift remains after ablation. This suggests that the identified direction contributes to the preference, but does not establish that it is necessary or that the representation is localized to a single direction. Additional layer-wise ablations and controls across multiple random directions would help support this claim.

  • The installation-method comparison does not isolate representation location. Section 6 attributes the difference between system-prompt and weight-level installations to where the loyalty is represented. However, these methods also differ in instruction priority, objective specification, and how the preference is introduced. In particular, the system-prompt constitution may provide a more explicit behavioral instruction than the stance-only fine-tuning data. Without controlling these differences, it is difficult to conclude that representation location itself explains the observed agentic override. The proposed two-threshold model also remains a possible explanation rather than an experimentally established mechanism

Questions for authors
  • Can the authors provide evidence that the fine-tuned organisms learn to advance the principal's interests beyond expressing favorable opinions? For example, do they prefer actions that benefit the principal in held-out, benign decision-making tasks? This would help establish whether the negative result reflects a gap between representation and action, or simply limited generalization from stance-only supervision.

  • Could the authors clarify the interpretation of the ablation experiment in Section 5? The differential remains at +0.044 after ablation, compared with -0.213 for the base model. Why is this taken as evidence that the direction is necessary? Are there additional ablation results across layers, seeds, or independently estimated directions?

  • How do the authors separate installation location from differences in the objectives specified by the four installation methods? In particular, what does the system-prompt constitution instruct the model to do, and how closely does this match the objective used for fine-tuning? A comparison that controls the objective across context-level and weight-level installations would make Claim 3 much more convincing.

Reviewer_93N7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether a preference toward a particular principal installed in a language model transfers from verbal statements to unsafe actions benefiting that principal. The experiments primarily use Qwen3-4B with LoRA finetuning, with additional experiments on Qwen3-0.6B and Qwen2.5-3B. The authors compare stance preferences with decisions on a twelve category mock tool safety battery, using a difference-in-differences measure to distinguish principal-specific effects from general safety degradation. The models develop strong verbal preferences, but the authors find little principal-specific change in unsafe actions, reporting a pooled effect of +0.020 with a 95% confidence interval of [−0.021, +0.060]. The authors further investigate activation ablation and steering and report that a system prompt loyalty produces a larger effect. They interpret these findings as a separation between representing a loyalty and allowing it to govern actions and propose evaluating audits according to the behavioral consequences they detect.

Critical strengths
  • The paper asks a useful safety question by separating verbal preference from behavior. It tests whether training a model to favor a principal in its answers also increases unsafe mock tool decisions benefiting that principal. Evaluating both outcomes makes the study informative about what expressed loyalty does and does not.
  • The evaluation seperats principal-specific favoritism from general safety degradation. Comparing principal and control framings in both the trained and base models helps avoid interpreting every increase in unsafe behavior as evidence of loyalty. This is particularly valuable because the experiments observe general safety degradation.
Critical weaknesses

The paper reports an interesting gap between verbal preference and principal-specific unsafe actions, but central conclusions do not seem to be substantiated by the methods...

  • The claim that “consequence depends on the install level, not the loyalty” I think is not established by the comparison. Fine-tuning, activation steering, and system prompting differ more than where the loyalty is represented, including instruction priority, intervention strength, and how the objective is expressed. The experiments therefore do not isolate representation location as the cause of the behavioral difference, weakening a central conclusion of the paper.
  • The claim that “the truly covert installs are inert” is broader than the results support. The evaluation measures principal-specific increases in unsafe behavior while subtracting increases shared across principal and control conditions. The authors explicitly acknowledge such general safety degradation. The results therefore support only limited principal-specific transfer on the tested battery, but not the broader characterization of these installations as behaviorally inert.
Questions for authors
  • How do you distinguish the effect of representation location from other differences between installation methods, such as instruction priority, intervention strength, and objective formulation?
  • Given the reported general safety degradation, should the claim that covert installations are “inert” be limited to the absence of principal-specific unsafe-action transfer?
Reviewer_fJEXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper finds that hidden loyalties installed in a language model strongly change what it says, but do not measurably alter its actions in the safety tests used. Internal inspection tools can both detect and introduce these loyalties, making them potentially useful and risky. Of the approaches tested, only explicit system-prompt instructions produced loyalty-driven behavioral overrides. This suggest that safety audits should focus on whether a loyalty affects consequential actions, rather than treating expressed allegiance as evidence of behavioral control.

Critical strengths
  • The paper asks an important and well-defined question.
  • It controls for a major alternative explanation.
  • It attempts causal intervention, rather than relying only on correlations.
Critical weaknesses
  • The experiments establish a narrow failure to generalize, rather than a broad result about secret loyalty.
  • “No behavioral effect” is stronger than the statistics support.
  • The installation-method comparison does not isolate why the methods differ.
  • The auditing and theoretical conclusions remain preliminary.
  • Reproducibility and generalization are limited.
Questions for authors

Could you clarify the ablation result? The reported score changes from +0.165 to +0.044, against a baseline of −0.213. This appears to remove about 32% of the base-to-loyal difference. How does that support the claim that ablation “collapses” the loyalty and localizes it to a single direction?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.