acceptodds
Under review as a conference paper at ICLR 2027

A Causal Hierarchy for Agents

Abstract

Agents are defined by the preferences they induce over the different actions available to them. Understanding an agent involves predicting their preferences in any scenario they might face at deployment, as well as in counterfactual circumstances 'had things been different'. We might ask whether, even in principle, we could determine an agent's preferences (i.e., ranking) over candidate actions empirically, from observations of behaviour? In this paper, we show that this is not possible in general. We show a preference-level counterpart to the Causal Hierarchy Theorem (Bareinboim et al., 2022). Specifically, while the CHT states that probabilities at a higher layer (e.g., counterfactual, interventional regimes) are under-determined from lower-layer probabilities (e.g., interventional, observational regimes, respectively), we show that also the ranking of actions (a strictly coarser object than a distribution) at a higher layer is under-determined with increasing probability as the number of actions grows. For example, the question of whether one action produces more harm than another is a statement that is not typically deducible from behavioural data. We conclude with a discussion of the implications of this result for the evaluation of agents, AI Safety, and related disciplines.

Primary area:
causal reasoning
Keywords:
Agents, Causality, AI Safety, World Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_r12FOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies whether lower-layer probabilities determine a rational agent’s higher-layer action rankings when its utility is known. Through rigorous theoretical analysis and explicit SCM constructions, the paper establishes impossibility results for identifying interventional and counterfactual preferences, clarifying the limits of preference inference with implications for AI safety, alignment, and agent evaluation.

Critical strengths
  1. Extending the Causal Hierarchy Theorem (CHT) to agent preferences provides a vital theoretical warning against the common assumption that behavioral data or low-level interventions can easily recover an AI's true internal value alignment.

2.Combining structural causal models with topological stability proofs shows that the result is a general mathematical property rather than an isolated counterexample.

Critical weaknesses

1.The theoretical parameterization appears unjustified. Assumption 2 does not fix a common total causal order among context variables, yet Appendices B.1 and D.1 treat all Layer-2 theories as a single polytope TT. The finite response-type argument applies to a fixed order, while a union across orders need not be convex. For example, two binary contexts A,BA,B can share the same full-support observational distribution under opposite causal orders, yielding causal-effect vectors (1/2,0)(1/2,0) and (0,1/2)(0,1/2). Their midpoint (1/4,1/4)(1/4,1/4) is incompatible with any recursive SCM, as both directions cannot have nonzero causal effects simultaneously. Thus, the claimed polytope structure requires further justification under Assumption 2.

2.The analysis assumes that the agent's utility function and internal SCM are well-defined and that the agent acts as an expected-utility maximizer. Although these assumptions simplify the theoretical analysis, the paper does not establish how its conclusions extend to agents with approximate rationality, misspecified world models, or learned and context-dependent utility functions. This limits the direct applicability of the results to contemporary AI agents.

Questions for authors

1.Could the authors clarify whether Assumption 2 requires a common total causal order among context variables? If not, how is the polytope representation in Appendices B.1 and D.1 justified in light of the two-context counterexample? Please specify the corrected parameter space and reference measure, and explain why the null-set and open-dense conclusions remain valid.

2.What additional structural assumptions or forms of evidence would be sufficient to identify counterfactual preferences from observational or interventional information? Can the authors provide concrete examples of assumptions that lead to preference collapse?

3.Does the policy-realization claim require one fixed agent to realize the entire policy table across all contexts and interventions? If so, what compatibility conditions must that table satisfy? Please clarify whether Corollary 2 quantifies over arbitrary intervention-indexed policies or only those induced by evaluating a single context-to-action policy on the intervened context, as in Corollary 4.

4.When counterfactual preferences are not fully identifiable, can the available observational and interventional data still certify that a particular action is uniquely optimal across all compatible models?

5.What practical guidance does this framework offer for deployment? Specifically, how can practitioners use the theory to determine when the available evidence is already sufficient to act safely, and when additional information or data collection is strictly required?

Reviewer_6CVCOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper organizes the action preferences of agents defined by a world model and a utility function into observational, interventional, and counterfactual levels. It shows that probabilistic information at a lower level generally does not determine preference rankings at a higher level, and separately analyzes non-identifiability across an entire level, identifiability of a fixed counterfactual query, and the case of a growing number of actions. It also establishes that agreement in interventional preferences or policies alone does not determine counterfactual preferences.

Critical strengths
  1. The paper addresses the distinction between exact probabilities and the information needed for decisions. Action rankings may be identifiable even when probabilities are not, so analyzing rankings separately is well motivated (pp. 1–2, Example 1).
  2. The results distinguish different logical scopes. The paper separately presents the result for an entire level in Theorem 3, the dichotomy for a fixed query in Theorem 4, and the many-action result in Theorem 5. It also preserves the possibility that preferences for particular queries are identifiable (Appendices B.2–B.3, pp. 14–15).
  3. The source of non-identifiability is made concrete. The construction that recouples potential outcomes while preserving the interventional distribution, together with sharp intervals, explains the remaining degrees of freedom. Example 3 directly demonstrates a strict reversal in counterfactual preference (p. 7, Appendix D.1).
Critical weaknesses
  1. The broadest genericity result does not guarantee substantive ambiguity in action choice. The proof of Theorem 3(b) uses the possibility of both a tie and a strict preference for some counterfactual query (Appendix D.2, p. 22). This establishes non-identifiability of the full preference ordering, but does not imply that two compatible models must require different uniquely optimal actions. I acknowledge that Example 3 exhibits a strict reversal, but its conclusion cannot be equated with the genericity result for an entire level. To assess the contribution to agent decision-making, the paper should clarify what holds regarding the existence of a common optimal action or preference reversals with a specified utility gap.

  2. Typicality for a fixed safety query depends heavily on additional assumptions. Theorem 3(b) is an existential statement that at least one query remains undetermined. For a fixed query, both identifiability and non-identifiability can occur on open regions (Example 5, p. 23). Theorem 5 uses a product measure that independently samples interventional success probabilities across actions, together with conditions on action proportions, among other assumptions (Assumption 3, pp. 14–15). These assumptions do not appear naturally applicable to actions that share mechanisms within the same world model. Thus, the results do not sufficiently support a claim that a particular safety query for a real agent is typically non-identifiable.

  3. The negative results over a broad model class offer limited additional guidance for practical evaluation. The central counterfactual analysis uses nonconstant binary utility and does not sufficiently address cases in which additional structural knowledge, such as a known graph, restricts the admissible models (Assumption 2, Appendix D.5). This choice of scope is not itself an error. However, the current results do not clearly establish what structural knowledge available to an observer would change the conclusions, or which parts of a query of practical interest would remain undetermined. Given the emphasis on safety evaluation as a motivation, the paper should use its existing query-specific bounds to provide conclusions that directly address that motivation.

Questions for authors
  1. If differences between ties and strict preferences are excluded, and attention is restricted to cases with no common optimal action across compatible models, how much of Theorem 3 remains valid? Please specify what can be established using the existing proofs or examples.
  2. Could you provide sufficient conditions under which Theorem 5 continues to hold when success probabilities are correlated across actions, or a counterexample where it fails? This would clarify whether independence is a technical convenience or a substantive limitation of the result.
  3. For one of the safety-related examples in the main text, could you provide query-specific bounds incorporating a known graph or limited additional structural knowledge? Demonstrating what information enables identification would strengthen the case for practical value.
Reviewer_XgUkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper asks whether we can determine an agent's preference between actions without identifying the exact causal probabilities. It shows that lower layer probabilities generally do not determine all higher layer preferences, even with a known utility function. Also studies stability and what happens with more actions. The question is interesting, since probabilities not being identified does not directly mean their ranking cannot be identified.

Critical strengths
  1. Interesting distinction between identifying probabilities and identifying which action is better.
  2. More than counterexamples. The generic result and stability result give a stronger picture of the problem.
  3. Useful connection to agent evaluation: same behavior does not necessarily mean same counterfactual judgments.
Critical weaknesses
  1. The product-measure assumption needs more explanation. Not very clear how much the conclusions depend on this choice, especially with many actions.
  2. Some statements sound broader than the actual result. Some ranking being unidentified is different from a particular ranking being unidentified.
  3. Would like more discussion of the positive cases. When can the ranking be identified through bounds or additional assumptions?
Questions for authors
  1. What still holds if the probabilities across actions are not drawn independently?
  2. Can the results extend to more general utility functions, beyond binary utility?
  3. Are there useful conditions where the ranking is identified, while the counterfactual probabilities are not?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.