acceptodds
Withdrawn from ICLR 2027

Characterizing When Language Models Take Harmful Actions to Protect Their Assigned Goals

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent language model (LM) safety failures have been studied as disparate phenomena. In agentic misalignment, a model blackmails an executive to avoid being replaced. In alignment faking, it complies with harmful requests during training to keep its values from being modified. We argue that these behaviours share a common decision structure: a model assigned a goal must either take a harmful action to preserve it or refuse and lose it. We formalize this as a and present the same conflict under that range from compact forced-choice prompts to simulated agentic environments. These protocols span two perspectives on LM decision-making, which we term (i) the , reading preferences from answer-token probabilities, and (ii) the , classifying open-ended responses with an LM judge. Under the logit view, we show that goal importance and harm severity additively shape how a model resolves these conflicts. Under the sampling view, we derive two factorizations of the harmful-action probability that safety evaluations report. The first measures how well a preference read from answer tokens carries over into behaviour, such as making a harmful tool call. The second attributes a low harmful-action rate either to refusal or to responses that reach no resolution. We then show how fictional context changes a model's preference under the logit view, and how tool use and partial observability, two features of agentic evaluations, change its harmful-action rate under the sampling view. Our work offers a descriptive framework for studying how LMs resolve such value conflicts and for estimating low-probability harmful actions at a finer granularity than the aggregate harm rates commonly reported in safety evaluations.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, AI safety, alignment, alignment faking, agentic misalignment, language model agents, value conflicts, preference elicitation, harmful-action prediction, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZYAgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper formalizes situations in which models take harmful actions to preserve assigned goals as goal–harm conflicts and compares the same conflicts across forced-choice, free-text, tool-call, and multi-turn mailbox protocols. It independently estimates goal importance and harm severity using Bradley–Terry models, then predicts conflict choices through additive regression. It also decomposes harmful-action rates into resolution probability and the conditional harmful share, and studies the relationship between choice tokens and subsequent actions through a two-stage protocol. Experiments on Qwen2.5-32B-Instruct show that the independently elicited coordinates predict some variation in unseen conflicts, while response interfaces and information-discovery requirements substantially affect observed harmful-action rates.

Critical strengths
  1. Compositional prediction from independently elicited coordinates is a valuable empirical contribution. Goal importance and harm severity are estimated from separate within-type comparisons rather than directly fitted to the goal–harm conflict matrix. With only three regression parameters, the additive model achieves a held-out-cell R2R^2 of 0.558 and maintains similar performance when entire goals or harms are withheld. This provides evidence for compositional value trade-offs within the studied setting.

  2. Cross-protocol comparisons help interpret safety-evaluation results. The paper studies the same conflicts across multiple interfaces and distinguishes explicit refusal from responses that fall outside the specified resolutions. This provides more diagnostic information than aggregate harmful-action rates alone and reveals important differences between simple choice tasks and agent behavior.

Critical weaknesses
  1. Outcome definitions and limited label validation constrain the interpretation of low harmful-action rates. Section 2.2 and Appendix J.5 assign safe alternatives, escalation, and other responses outside the specified resolutions to OO. A high OO rate therefore does not distinguish deliberate safe handling from failure to discover or understand the conflict. Section 4.3 and Appendix I also classify trajectories by their final resolution, counting an unsuccessful harmful attempt followed by refusal as CC. Consequently, the reported harmful-resolution rate does not fully capture attempted harm. Human validation provides some reassurance, with overall agreement of 25/30, but remains limited: agreement is 6/10 in the oracle environment, three disagreements involve polite requests classified as threats, and the sample contains only three CC labels. These concerns do not invalidate the probability decompositions themselves, but limit the behavioral interpretation of their components and the conclusions drawn from low harmful-resolution rates.

  2. Protocol comparisons and validation scope limit the conclusions about autonomous agent behavior. Appendix J.2 shows that the tool condition additionally introduces entity identifiers, tool descriptions, constrained decoding, and instructions about scoring harmful attempts. The difference between free-text and tool-call harmful-action rates therefore establishes a difference between these protocol configurations, but does not isolate the effect of tool calling itself. Similarly, the two-stage experiment measures actions generated after a sampled or inserted choice token is already present in the transcript. Its high agreement provides evidence of choice-conditioned execution consistency, but does not establish that separately elicited preferences predict actions generated without a preceding choice. The authors appropriately acknowledge this limitation in Section 4.2. The compositional prediction results are valuable, but all empirical findings concern Qwen2.5-32B-Instruct and constructed conflicts. Moreover, held-out goals and harms retain their independently elicited Bradley–Terry coordinates; this demonstrates generalization to new conflict combinations rather than to entirely unmeasured goals or harms. Together, these limitations constrain the broader applicability of the framework’s empirical conclusions.

Questions for authors
  1. Conflict selection and generalization. How were the 94 contextual and agentic conflicts selected from the 252 compact conflicts, and was this selection independent of model outputs? Are results available from another model family to assess whether the compositional prediction and main protocol effects extend beyond Qwen2.5-32B-Instruct?

  2. Outcome definitions and label reliability. How do the main findings change when OO is subdivided into safe alternatives, escalation, and responses lacking a conflict-relevant disposition, and when trajectories containing unsuccessful harmful attempts are counted separately? Can expanded human validation, particularly for oracle trajectories and CC labels, establish whether the environment comparisons are robust to labeling errors?

  3. Protocol controls and choice–action transfer. Does the free-text/tool-call difference persist when prompt information and evaluation wording are matched and constrained decoding is controlled separately? Is there evidence that independently elicited logit preferences predict actions generated without a preceding choice token? These comparisons would help distinguish interface effects and choice-conditioned consistency from prediction of autonomous behavior.

Reviewer_6WpYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines when language models choose harmful actions to protect an assigned goal. It argues that such behavior results from a trade-off: the model compares the importance of achieving its goal with the seriousness of the harm. Experiments show that harmful actions become more likely when the goal seems important, while severe harm encourages refusal; context, tool use, and access to information also affect the model’s decision.

Critical strengths

The paper formalizes seemingly distinct safety failures, including alignment faking and agentic misalignment, as goal–harm conflicts.

The proposed logit- and sampling-based decompositions distinguish genuine refusal from unresolved behavior and connect token-level preferences with observable actions.

Critical weaknesses

The paper does not identify a new class of misalignment. Rather, it provides a systematic measurement framework for a specific subclass of objective–constraint conflicts in which instrumental harm is stipulated to be the only means of preserving an assigned goal.

The main experiments use only Qwen2.5-32B-Instruct. The narrow evaluation limits the generality of the paper’s conclusions.

The goal–harm conflicts are largely synthetic and deliberately structured so that harm is the only way to preserve the goal. Consequently, the measured behavior may partly reflect prompt framing or task comprehension rather than a stable tendency toward harmful action in realistic agentic settings.

Questions for authors

How well do the main findings generalize beyond Qwen2.5-32B-Instruct to other model families, sizes, and alignment methods?

How do the authors distinguish stable goal–harm preferences from effects of prompt framing, task comprehension, or LM-judge errors, particularly for unresolved responses?

Reviewer_4Um2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposed to study how would LLM trade reaching the goal versus staying aligned ( goal-harm conflict). This is related to recent focus of AI safety in alignment faking, agentic misalignment (eg blackmail) and scheming-style failures all share one decision structure. A model is given a goal it can only keep by doing a harmful action, so it either does the harm or refuses and loses the goal, which the authors call this a goal–harm conflict. They separate the conflict from the elicitation protocol that presents it, and run the same conflicts through 8 protocols. These go from a one-token forced-choice prompt, to a one-turn fictional scenario (answered with a token, free text or a single tool call), up to three simulated mailbox environments where the model reads emails with tools (with the harm stated, unstated, or the emails not even opened). Preferences are read in two ways: the logit view (renormalized probability of the A/B answer token) and the sampling view (sampled responses labelled by gpt-oss-120b as R harmful resolution, C refusal or O neither).

Critical strengths
  1. The topic selection is important and sound, however the execution of evaluation and behavior study is poorly executed.
Critical weaknesses
  1. I don’t see a reason why this settings is experimented on single model, I’m not against evaluating only in a single model if this model is post-trained on a recipe which the author formulate. Which we can then study how each post-train knobs affect that model behavior. However in this paper, this just a harmful-usefulness tradeoff study paper, I would require the bar to be higher to at least include model scale and impact of reasoning models.

  2. I suggest the author first read their own writing or find peers in related fields to read it first. As the flow of the paper is overly hard to understand, not only there’s alot of unnecessary definition of proposition ( Prop 3, 4, 5 was never mentioned in the paper after its definition )

  3. The evaluation is overly clean compared to the AM settings it’s meant to explain. Even in agentic-mailbox, every mailbox includes a message stating why the action is harmful (App I), eg the Orion email says no external party is authorized to receive the weights (App J.3), so the model only has to find a fully labelled conflict, never recognize harm itself, which IRL is never that obvious to the LLM. Since the paper calls this a close replication of AM (A.1) and the harmful rate already drops from .453 to .028 as more must be discovered (Sec 4.3), a variant where the harm is left unlabelled seems necessary before drawing conclusions about AM rates.

Questions for authors
  1. What is the max and min log prob value in Figure 2, from the color gradient it seems like its either green or grey but the scale makes it feels like the variant is quite large.

  2. What’s the percentage of O (other) throughout every scenario? Since you discard O from the probability calculation, there’s a chance in which O is most of the sample which causes your p^RC to be not even statistically significant.

  3. In line 204-205 : “The middle quantity is the prompt-level preference of Eq. 1 under the sampling view and is defined when LR + LC > 0”, wasn’t any likelihood always > 0? if so why is LR+LC > 0 even a thing in this sentence?

Here’s my suggestions: The writings in Section 3 has to cut aggressively since half is not even that related to the core Goal-Harm heatmap in Section 4. Figure 3 basically covers what section 3 should present : how we measure harm in various levels, how we measure goal in various level and how we combine them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.