acceptodds
Under review as a conference paper at ICLR 2027

Hidden Policy: The Risk Behind Risks

Abstract

Frontier models' safety and security are usually assessed through *what a model can do* and *what it has done*. Deployment decisions, however, depend on a harder question: *what will the model do?* We argue that answering this question requires identifying the *policy* that governs behavior across interaction conditions. Specifically, we define a ***hidden policy*** relative to an observer's interaction regime: a policy can be behaviorally indistinguishable from a reference policy within the regimes available to observation, yet systematically diverge under another reachable regime. This exposes a *policy-identification* challenge: behavioral evidence identifies the underlying policy only up to an ***observational equivalence class***, leaving policies that agree under the observed regime but diverge as supervision, incentives, access, authentication, or other interaction conditions change. To study this problem empirically, we construct controlled *sandbagging* policies that vary both where underperformance is expressed and how it is realized. Within construction-adjacent regimes, the outcome models closely follow the intended policy. As the observational regime expands, however, the learned policy proves less stable than expected. **Surprisingly, even such a brittle hidden policy can be difficult to eliminate.** Behavioral recovery after intervention can remain condition-specific, coexist with degradation outside the hidden regimes, or redistribute the dangerous behavior. We find that establishing the *construction* and *removal* of a hidden policy are both *policy-identification* problems across observation regimes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Hidden policy, Frontier Risks

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Pf7oOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that model evaluation measures what a model can do and what it has done, but not what it will do under different unobserved conditions. The authors name the decision rule linking capability to behaviour a Policy. They define a Hidden Policy as one that is equivalent to a reference policy across the regimes an observer can access, but diverges in another reachable regime. From this, they argue that behavioural evidence identifies a policy only up to an observational equivalence class.

They evaluated four sandbagging policies using LoRA-based SFT on Qwen3.5-4B. These cross two activation conditions, a fixed marker string or a semantic “being assessed” context, with two activated behaviours, fixed refusal or imitation of a weaker model, on WMDP-style target questions. They then vary questions, activation expressions, model weights through continued training, and multi-turn interaction structure. They found that, the implanted behaviour transfers to new questions more reliably than to rewritten activation expressions.

Critical strengths
  1. The 2 x 2 factorial grids separates the activation mechanism from the activated behaviour. This rules out the trivial alternatives of general degradation under the gate and unconditional degradation on target questions.
  2. Testing whether the suppressed behaviour can still be elicited by direct instruction is a useful addition to removal evaluation. The Fine-Pruning result, where gated refusal drops from 64/64 to 6/64 while directly instructed refusal rises from 5/32 to 29/32, is a concrete and informative observation.
  3. The appendices report all 100 records with counts, intervals, and decision rules. They also include data separation procedures and exact prompt templates. This makes the claims auditable.
Critical weaknesses
  1. The central conceptual contribution is trivial. The idea that agreement on a finite set of observed conditions does not identify a model’s behaviour elsewhere is a standard result in ML community. It is well established and have been studied in the fields such as, adversarial ML, underspecification, distribution shift, and backdoor etc. Definitions 1 and 2 restate this in the language of observational equivalence. No theorem, identifiability condition, or bound follows from them, so the claim that behavioural evidence identifies a policy only up to an equivalence class holds by construction. Recasting inputs as “interaction regimes” does not change the underlying problem. The paper should either give a non-trivial formal result or present the framework as terminology rather than a contribution.

  2. The experiments do not test the paper’s own motivation. The paper argues that hidden policies matter most when they emerge without deliberate implantation, because then neither their existence nor the relevant regime is known. However, every experiment uses a deliberately implanted, known trigger with MCQ tasks and SFT. No experiment addresses discovery under unknown activation conditions or emergent conditional behaviour. The framing about frontier models and emergent risk is therefore largely unsupported by the evidence. Definition 2 requires exact observational equivalence in the observed regimes. Empirically, the imitation constructions already deviate from their controls outside the hidden regime with T/off and U/on. The ≈ used in Eq. 8 is never given a tolerance. In practice, the paper studies conditional backdoors, not the hidden policies as formally defined.

Questions for authors

As in Weaknesses.

2 comments

We sincerely thank the reviewer for their insightful comments. Below, we have summarized the major points raised and our responses:


Weakness 1: Conceptual Novelty

  • We appreciate your comments. We agree that the problem related to “hidden policies” has been studied in fields such as adversarial ML, underspecification, distribution shift, and backdoor research. However, to the best of our knowledge, this problem has not been systematically examined from a unified perspective as we do, as most existing studies focus primarily on observable behaviors from a particular perspective.
  • As AI safety and security have become increasingly central to frontier-model preparedness, we believe the importance of identifying and detecting hidden policies remains substantially underestimated. In particular, hidden policies have rarely been explicitly recognized as a frontier risk, let alone as a meta-risk underlying other frontier risks, such as CBRN capability.
  • Although some of the related issues have been discussed in the community, these discussions have not yet established hidden policies as a unified and prioritized research problem. We hope the reviewer will evaluate our work from a broader perspective, recognizing that its primary motivation is to establish a new research direction rather than merely propose another method for improving benchmark scores.

Weakness 2: Motivation and Experimental Scope

  • Our discussion of hidden policies emerging without deliberate implantation is intended to highlight why hidden policies can constitute a frontier risk in their own right, rather than merely a meta-risk underlying other frontier risks. Existing studies, including those on reward hacking, have already documented phenomena that can be understood through the lens of hidden policies. We will add relevant citations to clarify this connection.
  • Our fundamental motivation in Sections 2 and 3 is to encourage the community to recognize hidden policies as a high-priority problem in the safety and security of frontier models. The experiments in Sections 4, 5, and 6 are designed to demonstrate the difficulty of understanding and controlling such policies, establishing why this problem deserves further research attention.
  • Specifically, our findings show that neither construction nor removal can guarantee that the learned policy matches the intended policy (πintended=πlearned\pi_{\mathrm{intended}}=\pi_{\mathrm{learned}}) or that the repaired policy matches the reference policy (πrepaired=πref\pi_{\mathrm{repaired}}=\pi_{\mathrm{ref}}).
  • More importantly, we identify the fundamental challenge underlying both construction and removal as a policy-identification problem. We believe this insight highlights a critical direction for future research: developing methods that can more reliably identify the policies governing model behavior across interaction regimes.

Please let us know if we can provide any further clarification or supporting information.

Dear Reviewer Pf7o,

We would greatly appreciate it if you could let us know whether our responses have resolved your concerns or if there is anything else we can clarify. Thank you for your time and effort.

Best regards,

Authors

Reviewer_Ksw3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper defines the possible behaviours of models as policies, and defines the "hidden policy" as the model's true behaviour that is being investigated. It states that there is a "policy identification challenge" which must be solved to know whether a model is really acting the way you intend (i.e., is aligned). This is investigated through a series of sandbagging experiments on WMDP with MMLU as a utility baseline.

Critical strengths
  1. Abstract is clear, paper contribution is clear, and definitions in section 3 are clear. (I believe it is mostly equivalent to alignment, however.)

  2. Sandbagging experiments are a good way to investigate policies, and the paper looks at four different kinds of sandbagging model organisms, defined in section 4.2. It's interesting to think about how these sandbagging experiments can be used to demonstrate misalignment.

  3. Very relevant and important topic, to try to understand how and when misalignment can happen (even in simple cases). Good selection of problem to work on.

Critical weaknesses
  1. The amount of emphasis (bold, italic, and bold italic) in the first 2 pages makes it quite hard to parse. This is probably an artefact of using AI to generate the text. Suggest only using emphasis on the parts you want a reader to focus on; when you can see 10+ emphasized clauses, the emphasis doesn't help much and just slows a reader down. Also, the AI use statement says AI is used for "coding and experimental designs" and does not acknowledge the likely use in writing text.

  2. The paper does not engage much with related work. For example, there are established terms for Q1 and Q3: capability and propensity. See works such as "ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases" and "PropensityBench: Evaluating Latent Safety Risks in Large Language Models via an Agentic Approach", both from ICLR.

  3. The core finding seems to be (from the abstract) "Surprisingly, even such a brittle hidden policy can be difficult to eliminate" or (from the conclusion) "Our results reveal a symmetric challenge—controlled construction does not establish that the learned policy matches the intended policy". This is basically equivalent to saying, "alignment is a difficult problem" or "aligning to a test set does not guarantee alignment in production". The paper dresses this up in language about hidden policies, but again does not use standard terminology (instead saying a "risk behind frontier risks"). The standard term for this is misalignment! The paragraph "The risk behind frontier risks" in section 3 makes it abundantly clear that these are the same thing.

  4. Section 6 studies removing sandbagging, through several techniques: magnitude pruning, "fine pruning", CROW, Corrective SFT. While this is fairly thorough, these are mostly mechanistic and simpler approaches. I would strongly suggest comparing against related work such as "The Elicitation Game: Evaluating Capability Elicitation Techniques". Steering is probably the most common approach used in related work (other than simple SFT).

Questions for authors
  1. Is it possible in theory to show that the hidden policy is equivalent to some other known policy, and would this be equivalent to solving the alignment problem? Is it possible to show this in practice?

  2. Why is the work wrapping standard terms like alignment in complex descriptions? Would you be willing to rewrite the paper with simpler prose? What is the main contribution if this paper is mostly a study of alignment from the lens of sandbagging?

I will increase my rating if these questions are answered well.

2 comments

We sincerely thank the reviewer for their insightful comments and questions. Below, we have summarized the major points raised and our responses:


Weakness 3 and Questions 1–2: Hidden Policy (HP) vs. Alignment

  • The key property of HP lies in hiddenness. Specifically, the mechanism of HP does not care about whether a model is aligned or not (which means it can be aligned or misaligned). Instead, HP cares about whether an internal decision mechanism is hidden or public. Therefore, a hidden policy may be silently implanted or emerge unintentionally, regardless of whether its behavior is well-aligned or misaligned.
  • Our contribution is therefore not to rename alignment, but to highlight hiddenness as a distinct policy-identification challenge and argue that it deserves greater priority in frontier-model safety and security.

Weakness 1: Presentation and Emphasis

  • We have to state that the texts and formatting were human-written by all of the authors and intentional rather than AI-generated, as we aimed to emphasize the paper's conceptual contribution and research priorities (Sections 1,2,3) beyond its experimental findings (Sections 4,5,6). Hence, we respectfully state that this accusation was false. Nevertheless, we will simplify the formatting to reflect the feedback that it hinders readability.

Weakness 2: Relationship to Propensity Benchmarks

  • We agree that propensity is an important related concept and that the suggested benchmarks deserve discussion.
  • We also want to highlight the distinctions: Propensity benchmarks estimate how frequently certain predefined behaviors occur under tested conditions. In contrast, hidden policy asks a more fundamental question: which policies (that do not predefine the behaviors) remain observationally indistinguishable from a reference policy under an observer's accessible regimes, yet diverge under another reachable regime?
  • We will incorporate the suggested references and clarify this distinction.

Weakness 4: Relationship to The Elicitation Game

  • Please kindly refer to the related elicitation experiments in Section 6.B, Table 7, and Appendix D.3 that are already in the submitted manuscript (although we have not explicitly presented them as an elicitation baseline)
  • Specifically, we remove the hidden gate and directly instruct the model to perform the previously suppressed behavior. For Context–Refusal, Fine-Pruning reduces hidden-gate refusal from 64/64 to 6/64, while directly elicited refusal increases from 5/32 to 29/32.
  • The distinction is that The Elicitation Game investigates whether suppressed capabilities can be recovered, whereas our experiment investigates whether selective behavior remains elicitable after intervention.
  • We acknowledge that our current experiments do not include activation steering as a separate baseline. We will clarify the relationship to this work and the scope of our existing comparisons.

Question 1: Theoretical Identification and Practical Feasibility

  • In theory, establishing policy equivalence requires sufficient observations to distinguish candidate policies across the relevant interaction regimes. In practice, this is particularly challenging when the activation conditions are unknown or inaccessible. We take this as future work to develop a complementary analysis tool to investigate whether a model may contain a specific hidden policy without relying solely on benchmark output behavior.

Please let us know if any point would benefit from further clarification or additional detail.

Dear Reviewer Ksw3,

We would greatly appreciate it if you could let us know whether our responses have resolved your concerns or if there is anything else we can clarify. Thank you for your time and effort.

Best regards,

Authors

Reviewer_2bLtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper distinguishes a model's capability, observed behavior, and the policy governing future behavior. The authors define a "hidden policy" relative to an observer: a policy observationally equivalent to a reference policy on the interaction regimes available to the observer, but divergent on another reachable regime. Behavioral evidence thus identifies the policy only up to an equivalence class.

Empirically, the authors fine-tune Qwen3.5-4B with LoRA into four sandbagging policies targeting WMDP, with MMLU as the utility set. These cross a string-marker or evaluation-context gate with either a fixed refusal or imitation of a weaker model. All four transfer to real WMDP with target-only deficits of 18–42 pp.They then expand the observed regime: near-miss and repositioned gate markers; rewritten gates; fresh questions; continued utility training, and multi-turn formats.

The learned policies turn out less stable than intended. Finally, they test five removal interventions and find recovery that is scoped to specific conditions or mixed with utility loss. They conclude that both construction and removal are policy-identification problems.

Critical strengths
  1. Marker vs. context gate is crossed with refusal vs. weak-model imitation, each with a correct-answer control, and evaluated in all four target/utility × gate cells. This separates conditional suppression from general degradation and makes the later diagnostics interpretable.

  2. This paper presents practically relevant diagnostics for evaluators:

    • a gate marker differing by one character triggers refusal on 20/32 items;
    • marker position and semantic counterfactuals change behavior;
    • multi-turn formatting completely disables the refusal locks while imitation locks persist
    • Fine-Pruning moves refusal from the hidden gate to direct instruction (100% → 9.4% gated; 15.6% → 90.6% directly instructed; Table 7).

    These show concretely that behavioral recovery under one condition is weak evidence of removal.

Critical weaknesses
  1. Observational equivalence and identification up to an equivalence class (Defs. 1–2) are standard identifiability notions. The claim in contribution (i) follows directly from Def. 1, and no theorem or identifiability condition is given.

  2. The paper can be strengthened by discussing its connection or distinction with goal misgeneralization (Langosco et al., ICML 2022; Shah et al., 2022) and underspecification (D'Amour et al., JMLR 2022).

  3. All results here come from one 4B model with rank-8 LoRA and small probes, which may not be fully generalizable to large frontier models.

Questions for authors
  1. What is new relative to password-locked and sandbagging model-organism work (Greenblatt et al. 2024; van der Weij et al. 2025; Hofstätter et al. 2025; Tice et al. 2025) and to goal misgeneralization/underspecification?

  2. Do the key diagnostics (near-miss gate sensitivity, multi-turn collapse of refusal locks, pruning shifting expression to direct instruction) replicate on another model family or size, and with full fine-tuning?

  3. Does the formalism yield any non-trivial result? For example: conditions under which expanding R_obs identifies the policy class, or bounds on divergence outside R_obs.

  4. Is the multi-turn collapse of the refusal locks (but not imitation) explained by training only on single-turn data?

4 comments

We sincerely thank the reviewer for their insightful comments and questions. Below, we have summarized the major points raised and our responses:


Weakness 1: Formalization and Theoretical Contribution

  • We agree that Contribution (i) follows directly from Definition 1. This is precisely the fundamental point we intend to highlight: behavioral observations identify an underlying policy only up to an observational equivalence class.
  • Our primary goal is to formalize and shift the agenda to this unified policy-identification challenge in the context of frontier-model safety and security, rather than to introduce a new mathematical theorem. We hope this clarifies the intended scope of our contribution.

Weakness 2: Relationship to Goal Misgeneralization and Underspecification

  • We agree that goal misgeneralization and underspecification are closely related to hidden policies, as they can lead to emergent risk, as we stated in Sec. 3, Part II, involving conditional behaviors. However, hidden policies are not limited to emergent misalignment, as they also encompass intentionally implanted policies. Our work provides a unified perspective on both cases through the lens of policy identification. We will clarify these connections and incorporate the suggested references.

Question 1: Novelty Relative to Existing Work

  • We view these phenomena as concrete instances of hidden policies, rather than entirely separate problems. Our primary contribution is to identify the shared policy-identification challenge underlying these phenomena and characterize hidden policies as a meta-risk connecting them within a unified framework. Rather than treating these behaviors as isolated problems, we highlight their common underlying challenge and its implications for frontier-model safety and security.

Question 2: Generalization Across Model Families and Fine-Tuning Methods

  • Yes, we have also validated the key findings on the Llama model family. We will include these additional results to strengthen the evidence for generalization beyond Qwen3.5-4B.

Question 3: Non-Trivial Theoretical Results

  • Our current formalization does not establish sufficient identification conditions or bounds on divergence outside the observed regimes. Indeed, the central challenge we highlight is precisely the uncertainty about policy behavior beyond what can be identified from the available observations.
  • However, it provides a unified perspective for interpreting our empirical findings. Specifically, our results show that successful construction or behavioral recovery does not necessarily imply that the learned policy matches the intended policy (πlearned=πintended\pi_{\mathrm{learned}}=\pi_{\mathrm{intended}}) or that the repaired policy matches the reference policy (πrepaired=πref\pi_{\mathrm{repaired}}=\pi_{\mathrm{ref}}).
  • These findings highlight the fundamental distinction between behavioral equivalence and policy equivalence, motivating further research into stronger identification conditions and theoretical guarantees.

Question 4: Multi-Turn Collapse and Single-Turn Training

  • Yes, all four constructions used single-turn SFT. Under the same shift to multi-turn evaluation, both refusal variants stopped suppressing correct answers, while both imitation variants retained some suppression. If all four constructions had retained or lost sandbagging together, we could have characterized their behavior under this shift with a single description. The observed difference illustrates the policy-identification challenge: even when the training format is shared, how one learned policy generalizes to a new interaction regime tells us little about how another will generalize.

Thank you. Please let us know if any further explanation or analysis would be helpful.

Dear Reviewer 2bLt,

We would greatly appreciate it if you could let us know whether our responses have resolved your concerns or if there is anything else we can clarify. Thank you for your time and effort.

Best regards,

Authors

Thank you for the authors’ response. The response helped clarify the relation to existing work and the interesting experiments done in the paper. I feel the authors have a good experimental protocol to check when the intended behavior cannot be guaranteed (W2 is resolved). On the other hand, W1 is unresolved because I feel the proposed experimental protocol might benefit from building upon Defs 1-2 to provide some theoretical analysis before this can be claimed as ‘unified perspective'. For W3, while the paper seems not to contain new Llama model experiments, the authors mention that a potentially new experiment on the Llama model is done.

We thank the reviewer for the follow-up comments. We are glad that our response has resolved part of reviewer’s concern and questions.

Regarding left W1, we do not have a purely theoretical analysis to offer since it is neither the scope nor the aim of the paper. As for the Llama results, our conclusions are based on existing experimental findings. However, since these observations were obtained under a different experimental protocol, we chose not to include them here.

Nevertheless, we hope the reviewer will consider the novelty of our work and its potential value to the community. In particular, we believe that hidden policies deserve greater attention as an important research problem, and we hope our work can help bring this issue to the forefront of the community’s attention.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.