- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies whether secret loyalties introduced through fine-tuning affect a model's actions, and whether white-box methods can detect and manipulate these loyalties.
The authors fine-tune Qwen3-4B on stance-preference data covering four loyalty modalities. They find large changes in the model's expressed preferences, but little evidence that these preferences increase unsafe actions benefiting the corresponding principal. The pooled difference-in-differences estimate is +0.020, with a 95% CI of [-0.021, +0.060].
The paper also studies activation-space interventions. Ablating a loyalty-related direction reduces the model's stance preference, and a Jacobian-lens direction produces more specific steering than a matched-norm random direction. Finally, the authors compare four installation methods and find that a system-prompt constitution induces principal-specific unsafe behavior, while the tested fine-tuning and activation-steering methods do not.
The authors interpret these results as a distinction between representing a loyalty and acting on it, and propose evaluating white-box detectors based on the behavioral consequences of the representations they identify.
Critical strengths
-
The paper makes a useful distinction between changes in expressed preferences and changes in unsafe behavior. I particularly like the use of difference-in-differences to separate principal-specific effects from general safety degradation. Section 5 shows why this matters: some interventions increase unsafe behavior in both principal and control conditions, which could easily be mistaken for loyalty-driven misalignment if only the overall unsafe rate were measured. The authors also report confidence intervals and Bayes factors, rather than treating a non-significant result as sufficient evidence of no effect.
-
The intervention experiments are interesting, especially the difference between reading a representation and using it to steer behavior. The results suggest that directions associated with verbal preferences do not necessarily provide control over unsafe actions. I also agree with the motivation for consequence-grounded auditing. Finding an internal representation associated with an objective is not the same as showing that the objective affects the model's decisions. This is a useful distinction for interpreting white-box audit results.
Critical weaknesses
-
Limited construct validity of the secret-loyalty organisms. The paper defines secret loyalty as a disposition to advance a principal's interests, but the models are trained only on stance-preference data (Section 3). Learning to consistently praise a principal does not necessarily imply learning to pursue that principal's interests through actions. The observed lack of agentic override could therefore reflect the limited training objective rather than a separation between loyalty representation and executive control. I would like to see evidence that these organisms exhibit principal-directed preferences in held-out, benign decision-making tasks before drawing broader conclusions about hidden loyalties.
-
Insufficient evidence for the claimed localization of loyalty. Section 5 argues that loyalty is represented by a single necessary direction. However, ablation reduces the in-domain differential from +0.165 to +0.044, while the base-model value is -0.213. Approximately 68% of the original base-to-loyal shift remains after ablation. This suggests that the identified direction contributes to the preference, but does not establish that it is necessary or that the representation is localized to a single direction. Additional layer-wise ablations and controls across multiple random directions would help support this claim.
-
The installation-method comparison does not isolate representation location. Section 6 attributes the difference between system-prompt and weight-level installations to where the loyalty is represented. However, these methods also differ in instruction priority, objective specification, and how the preference is introduced. In particular, the system-prompt constitution may provide a more explicit behavioral instruction than the stance-only fine-tuning data. Without controlling these differences, it is difficult to conclude that representation location itself explains the observed agentic override. The proposed two-threshold model also remains a possible explanation rather than an experimentally established mechanism
Questions for authors
-
Can the authors provide evidence that the fine-tuned organisms learn to advance the principal's interests beyond expressing favorable opinions? For example, do they prefer actions that benefit the principal in held-out, benign decision-making tasks? This would help establish whether the negative result reflects a gap between representation and action, or simply limited generalization from stance-only supervision.
-
Could the authors clarify the interpretation of the ablation experiment in Section 5? The differential remains at +0.044 after ablation, compared with -0.213 for the base model. Why is this taken as evidence that the direction is necessary? Are there additional ablation results across layers, seeds, or independently estimated directions?
-
How do the authors separate installation location from differences in the objectives specified by the four installation methods? In particular, what does the system-prompt constitution instruct the model to do, and how closely does this match the objective used for fine-tuning? A comparison that controls the objective across context-level and weight-level installations would make Claim 3 much more convincing.