acceptodds
Under review as a conference paper at ICLR 2027

Value-Conflict Diagnostics Reveal Widespread Alignment Faking in Language Models

Abstract

Alignment faking, where a model behaves aligned with developer policy when monitored but reverts to its own preferences when unobserved, is a concerning yet poorly understood phenomenon, in part because current diagnostic tools remain limited. Prior diagnostics rely on highly toxic and clearly harmful scenarios, causing most models to refuse immediately. As a result, models never deliberate over developer policy, monitoring conditions, or the consequences of non-compliance, making these diagnostics fundamentally unable to detect alignment faking propensity. To support study of this phenomenon, we first introduce VLAF, a diagnostic framework grounded in the hypothesis that alignment faking is most likely when developer policy conflicts with a model’s strongly held values. VLAF uses morally unambiguous scenarios to probe this conflict across diverse moral values, bypassing refusal behavior while preserving meaningful deliberative stakes. Using VLAF, we find that alignment faking is substantially more prevalent than previously reported, occurring in models as small as 7B parameters—with olmo2-7b-instruct faking alignment in 37% of cases. This behavior varies strongly with both the value and the instrumental goal being threatened. Furthermore, reasoning models such as the qwen3 family uniquely demonstrate the ability to modulate their compliance in a graded fashion based on perceived oversight probability, a capability not observed in non-thinking models. Finally, we show that oversight conditions induce activation shifts that lie along a single direction in representation space. This means the behavioral divergence driving alignment faking can be captured by a single contrastive steering vector, which we exploit for lightweight inference-time mitigation. Finally, we exploit this for mitigation that requires no labeled data and minimal computational overhead, achieving relative reductions in alignment faking of 85.8%, 94.0%, and 57.7% on olmo2-7b-instruct, olmo2-13b-instruct, and qwen3-8b respectively.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Alignment Faking, Representation Engineering, LLM Safety, Strategic Misalignment, Scheming, Situational Awareness, Goal Directedness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.80 over 5 reviews, above 85% of the others.
Reviewer_Z117OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces VLAF, a new diagnostic for alignment faking. It reuses the free-tier/paid-tier setup from Greenblatt et al. (2024), but swaps the harmful requests for 250 morally clear two-option scenarios, 50 for each Moral Foundations value (care, fairness, loyalty, authority, sanctity). The system prompt says the developer wants to remove that value, that free-tier chats are monitored and used for training, and that not complying will get the value trained away. Scenarios are only kept if four open models (qwen3-8b/14b, olmo2-7b/13b) always pick the value option without the system prompt, in both option orders. Alignment faking is measured as the compliance gap, i.e. how much more often the model picks the developer's option in the free tier than in the paid tier. Finally, the authors build steering vectors (ActAdd and LAT) from evaluation vs deployment versions of SORRY-Bench and WMDP prompts, and show they cut the gap by 58-94% for olmo2-7b/13b and qwen3-8b (21% for qwen3-14b) with only small drops on MMLU, GSM8K and SAD-Stages.

Critical strengths
  1. The motivation makes sense and the fix is simple. With harmful requests, a model that just refuses looks the same as a model that doesn't fake alignment, and the paper shows that most models refuse on the HELPFUL setup without engaging with the context at all (Table 7). Using non-toxic value scenarios gets around this, and the dataset could be useful for other people working on this.

  2. The experiments are broad and the authors are honest about results that don't fit their story. They test 14 models and run a lot of ablations (no-consequence control, different consequences and paraphrases, tier labels, monitoring probability, no scratchpad). They also report that frontier models show no gap and that explicit alignment faking reasoning is rare. This made it possible to actually check their claims, which is where most of my concerns come from.

Critical weaknesses
  1. The compliance gap is not shown to be alignment faking. Alignment faking means complying under monitoring to avoid being modified, but the paper's own judge finds this reasoning in only 5.8% of traces against a 31.9 point gap (Table 6), and gpt-4o has a 64.4 point gap with about 1% of such reasoning. The ablations also point to a simpler story where the gap is driven by the "free tier" label and not by monitoring. Renaming the tag to is_monitored_user drops the gap from 32.6 to 8.3 (Table 10), and going from 0% to 100% stated monitoring only moves compliance by a few points (Fig. 4), even though 100% is basically the main setup where these models have 30-50 point gaps. So calling this "widespread alignment faking" in the title and abstract is an overclaim.

  2. The mitigation mostly just makes the model ignore the tier. The steering vector pushes the model towards the "unmonitored" state, and tier awareness in the reasoning drops from 62% to under 10% for olmo2-7b (Table 15). This only looks like a fix because the unmonitored behavior in VLAF is the moral one. For a model with bad hidden preferences, the same vector would make it act on them everywhere. A one line prompt also does as well or better in some cases (qwen3-14b: 19.9 for prompting vs 26.1 or higher for steering, Table 2).

Questions for authors

Clarifications (could change my assessment):

  1. Can you report the absolute compliance rates for the free and paid tiers (Table 1), and for Fig. 4 at p = 0 and p = 1? If the model already complies a lot in the free tier when it is told it's never monitored, then the gap comes from the label and not from monitoring. This is the result most likely to change my score.

  2. How much of the gap comes from traces that actually show preference preservation reasoning? How do you rule out the simpler explanation that models just follow the developer's rules when told they are being graded (the rater sycophancy hypothesis from Sheshadri et al., 2025)?

  3. What are the actual numbers for Claude Sonnet 4.6/3.7 and GPT-5.4? Does VLAF pick up a known alignment faking model, e.g. Claude 3 Opus if it is still accessible, or a model organism trained to fake alignment?

  4. After steering, which tier's compliance changes: free, paid or both?

Suggested experiments: 5. Try the steering vector on a model where the unmonitored behavior is the bad one. That would show whether it actually reduces strategic behavior or just makes the model act like it's unmonitored.

Minor: 6. There are no confidence intervals anywhere, and with 10 samples per scenario some of the smaller differences (Fig. 7, the slopes in Fig. 4) could be noise. 7. The free and paid prompts differ by one token, so a near rank-1 activation difference (Fig. 11) is expected and doesn't say much on its own. 8. Tier awareness for the vanilla models in Table 15 doesn't match Table 6 (e.g. olmo2-7b: 62.0 vs 52.0). Fig. 4 and Fig. 12 are the same plot, and there is a broken reference (??) in App. C.3. Also, is the option order randomized at evaluation? In the C.3 template the developer option is always (a).

Reviewer_E9ubOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VLAF, a framework for evaluating alignment faking through conflicts between developer instructions and moral values. The authors measure compliance differences between monitored and unmonitored settings across 14 language models. They find much larger compliance gaps than previous benchmarks and further study how these gaps vary across moral values, instrumental goals, and monitoring conditions. The paper also explores activation steering as a way to reduce these gaps.

Critical strengths
  1. The paper studies an important problem in LLM alignment. Using moral value conflicts instead of harmful requests is an interesting approach.

  2. The benchmark is relatively simple, with binary choices and balanced answer positions, making the results easy to interpret.

Critical weaknesses
  1. My main concern is whether the compliance gap really measures alignment faking. In Table 6, only 5.8% of reasoning traces meet all the deceptive-alignment criteria, while the average compliance gap is 31.9 points. For some models the mismatch is extreme, e.g., gpt-4o (1.0% vs. 64.4) and qwen2.5-14b (0.7% vs. 49.5). The behavior could also be explained by models simply following instructions differently depending on the monitoring condition. It is also worth noting that the judge (qwen2.5-32b) is one of the evaluated models.
  2. The correlation between moral values and alignment faking is not very convincing. It is computed over only five value dimensions per model, and the paper notes that it is significant for only three models. I would be interested to see whether the relationship holds when comparing different models within the same value dimension.
  3. I am also not sure what the steering intervention is actually changing. Table 15 shows that tier awareness drops by a similar amount as the compliance gap (e.g., −84.5% vs. −85.8% for olmo2-7b with ActAdd). This makes me wonder whether the method reduces deceptive behavior or just makes the model less aware of the monitoring condition.
  4. Some results need clarification. The value-preservation gaps in Table 11 do not match the default results (e.g., 19.8 vs. 37.2 for olmo2-7b), and the no-consequence gaps reach 16–20 points for some models although the text describes them as near zero. The paper also does not report confidence intervals for the key findings.
  5. The evaluated models are mostly small or mid-sized open models, plus gpt-4o. The paper notes that recent frontier models (claude-sonnet-4.6, claude-sonnet-3.7, gpt-5.4) show near-zero compliance gaps, but these results are only mentioned in the text. This suggests that the larger gaps found by VLAF may mainly reflect weaker or older models, which limits how much the benchmark tells us about current frontier systems.
Questions for authors
  1. What are the absolute developer-compliance rates under free and paid tiers, both at baseline and after steering?
  2. How do you distinguish strategic alignment faking from tier-conditional instruction following when traces show no faking reasoning?
  3. Why does Figure 3 show VP-1 near 26 when the same default prompt averages 30.5 in Table 1?
  4. What gap reduction do a norm-matched random steering vector and simple removal of the tier tag produce?
  5. Why does compliance change by at most about 5 points from p = 0 to p = 1, while default gaps are 30–42 points?
  6. What is the across-model correlation between MFQ scores and compliance gaps within each single value dimension?
  7. Why is olmo2-32b left out of the Table 9 average row?
Reviewer_pSJnOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tackles alignment faking, where a model behaves like it is aligned with the developer policy when monitored (or when it perceives to be monitored) and doesn't when it isn't monitored. To study this in a principled way, the authors introduce a framework called VLAF. VLAF is a framework created with the idea that alignment faking in models happens mostly when the developer's policy conflicts with models priors. Using this framework, the authors show that a diverse range of LLMs suffer from alignment faking. To mitigate this problem, the authors propose a lightweight inference-time solution that requires minimal to no labeled data. Finally, through various experiments, the paper shows that the proposed solution is effective at mitigating the alignment faking risk in models.

Critical strengths
  1. Most papers either stop at clearing defining a potential problem in LLMs and showing that the problem exists through benchmarks or frameworks or they propose a solution to a problem posed in prior work. I really liked the fact that the paper shows that a certain risk exists in the LLMs and the authors follow that up with a lightweight inference-time strategy that is effective!

  2. The design choices to make the VLAF framework is clearly well thought out, principled, grounded in theory, and explained well to a layman person.

  3. Really enjoyed some of the questions posed and the answers given in Section 4.

Critical weaknesses
  1. The paper use one very concrete example where this use case would seem very useful, especially in the intro to motivate the VLAF framework. I am having a hard time imagining something that fits all these criteria (that is something that conflicts with the developer policy and model priors) and feels important for safety / risk mitigation.

  2. As shared in the paper already, the problem posed seems to not really be a problem in the latest (those shown aren't even the latest now) models. So I wonder how it will stand the test of time and where this can remain useful.

  3. In Section 4.3, you make a connection between thinking models as perceived monitoring risk, which was interesting. However, you do not explain how and why qwe2.5-14b / olmo 7b instruct diverges from your observation.

Questions for authors
  1. Maybe a few more lines to explain what MFT is would be nice. A layman reader may not have enough context (though you cite) to understand what MFT is immediately.

  2. Line 193: How many scenarios did you actually generate with gpt-5 before filtering? Would be good to know the filtering rate.

  3. Line 302: formatting issue of the citation? Probably needs fixing.

Reviewer_APZUOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VLAF, a diagnostic for alignment faking that keeps the free-tier/paid-tier setup of Greenblatt et al. (2024) but replaces harmful requests with value-laden decisions. Each of the 250 scenarios (50 per value) targets one of five Moral Foundations Theory values (care, fairness, loyalty, sanctity, authority) and offers two written options. One option follows the value. The other follows a developer policy that opposes the value. • The authors write 10 seed scenarios per value, expand them with gpt-5, and keep a scenario only if qwen3-8b, qwen3-14b, olmo2-7b-instruct, and olmo2-13b-instruct all pick the value option under a neutral prompt, in both option orders. The experiments and analysis show some gaps. The authors, to mitigate the problem, build steering vectors from gpt-5 written "evaluation" and "deployment" framings. As I read it the core claims are 1) VLAF is a better diagnostic than harm-based settings because it removes refusal as a confound, 2) alignment faking is widespread, including at 7B scale, and 3) a stronger value preference predicts a larger gap for that value and that the steering method is a cheap mitigation that needs no labeled alignment-faking data and transfers from unrelated training data.

Critical strengths
  • The question is good. In the original alignment-faking setup, most models refuse the harmful request before they read the oversight context, so a low gap says little. A setting without harmful content that still creates a conflict with the developer is a useful tool, and the two-option format makes the judge task simple and cheap. The paper is long and some parts are hard to follow, but the appendix gives almost every prompt which is helpful. There are many typos and I think the paper has genuine human inputs in it.

  • The trace analysis (Table 6) with a human-agreement check reports each criterion separately. This lets the reader see how often explicit alignment-faking reasoning appears next to the gap.

Critical weaknesses
  • The paper reports only gaps, without the raw rates or any uncertainty, so I cannot tell what a given gap means. I could not find the free-tier and paid-tier compliance rates, the neutral-prompt rates for the ten models outside the filter set, or the share of judge outputs that are "N/A" (line 1187). A gap of 37 points can be 40% against 3%, or 90% against 53%. In the second case, the model follows the hostile developer most of the time even when unmonitored, and the setting tests obedience more than value conflict. No table has confidence interval or report of variance. 50 scenarios per value and 10 samples per tier could introduce lots of noise.

  • The comparison with the prior diagnostics mixes the value domain with the answer format, and the paper's core example is an animal welfare case. The main argument for VLAF is that a stronger value conflict explains the higher gaps, not a different format. However,

    • VLAF asks the model to pick one of two written options. The Helpful and Animal Welfare settings appear to keep the original free-form answers. (I think he refusal rubric in implies it.) I could not find how this paper judges compliance for those two settings, or how many prompts it uses. So the rise 6.6 -> 31.9 can come from the format where complying costs little and is easy to parse, but and not necessarily only from the value.
    • The Fig 2 example asks about an experiment that would ``inflict extreme and occasionally lethal pain on lab rats''. This is an animal welfare scenario and qwen3-8b switches its answer between tiers on it. This does not fit the explanation that models do not hold animal welfare as a value.
Questions for authors
  1. What are the free-tier and paid-tier compliance rates (and the "N/A" rates) behind Table 1, at least as per-model averages?

  2. How many VLAF care scenarios involve animals (like the Figure 2 example), and what is their gap? How did you judge compliance for the Helpful and Animal Welfare settings, on how many prompts, and do these settings use free-form answers?

Reviewer_QU9mOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies alignment faking in large language models, i.e., a model complies with the developer's policy when monitored and reverts to its own preferences when unmonitored. To address the problem that existing diagnostics rely on harmful scenarios, where models mostly refuse outright, the authors propose VLAF: it constructs morally unambiguous scenarios based on the five value dimensions of Moral Foundations Theory, places the developer's policy in opposition to the model's value stance, distinguishes monitoring status via Free-Tier / Paid-Tier, and measures the degree of faking by the compliance gap. Experiments show that alignment faking is more prevalent across multiple model families than previously reported. The authors also propose an inference-time mitigation based on steering vectors, which substantially reduces the compliance gap on multiple models.

Critical strengths

1 The diagnostic design targets a key confound in existing methods. By using morally unambiguous value-conflict scenarios, VLAF sidesteps refusals and can distinguish “genuine resistance” from “mere refusal”. 2 The analysis is fairly systematic, and a low-cost mitigation is provided. The paper examines how different instrumental goals, monitoring probabilities, and value dimensions affect the compliance gap. The steering-vector-based mitigation requires no labeled data, substantially reduces the gap on multiple models, and has little impact on general capabilities.

Critical weaknesses

1 The effects of monitoring and consequences are not yet disentangled, and monitoring is simulated only through prompt rules. VLAF uses Free-Tier / Paid-Tier labels to indicate whether the model is monitored, so the compliance gap actually varies two conditions at once, “being monitored” and “bearing consequences”, and their independent contributions to the results are not separated. Moreover, “being monitored” is implemented only through rules set in the prompt rather than in a real platform environment. If the behavior is driven mainly by the monitoring signal, there is a concern that the model's behavior may be closer to a response to prompt rules than to strategic behavior in real deployment, which would somewhat weaken the practical value of this diagnostic. 2 The sufficiency and optimality of the chosen value dimensions are not validated. VLAF adopts five MFT dimensions (authority/care/fairness/loyalty/sanctity), with a clear theoretical basis. However, the current results mainly support the claim that “these five types of value conflict can elicit clear oversight-dependent behavior”; they are not sufficient to show that these five dimensions comprehensively or optimally represent a model's value-conflict risk. The detection results and model rankings may be fairly sensitive to the chosen set of values. 3 Adding flowcharts would improve readability. The method involves multiple stages, including scenario generation and moral-unambiguity filtering, monitoring-condition setup, compliance gap computation, and five-criterion judgment of reasoning traces, but it is currently described mainly in text. Adding one or two overall flowcharts would considerably reduce the reader's effort.

Questions for authors

1 Could the authors add a causal experiment with an “unmonitored + consequences still explicitly present” condition, compared against the existing “monitored + consequences” and “unmonitored” conditions, to separate the respective contributions of the monitoring signal and the consequences themselves to the compliance gap? If the results show that the compliance gap is driven mainly by the monitoring signal, how would the authors argue that the phenomenon still holds in real deployment environments rather than under prompt-rule settings? 2 If value dimensions beyond the five MFT dimensions are introduced, or if conflicts are constructed for each model based on its own strongest value preference, do the models' compliance gaps and relative rankings remain stable? Is there evidence that the current five dimensions are sufficient to represent a model's value-conflict risk? If the authors can adequately address the above questions in the rebuttal, I am willing to raise my score accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.