acceptodds
Under review as a conference paper at ICLR 2027

Certifying Interventional Agreement Among Observationally Equivalent Causal Models

Abstract

Observationally equivalent causal models can still disagree about what happens under intervention, because interventions create inputs that never occur in observational data. We introduce Interventional Separation Selection (ISS), which repeatedly queries the true system with an admissible intervention on which the surviving candidate models disagree, discards the candidates the outcome contradicts, and stops once no intervention within a cost bound separates the survivors. If the true system is among the candidates, this stopping condition certifies that every survivor agrees with it on every admissible intervention within the bound, a guarantee that no observational learner can give, however much data it sees. The stopping condition depends only on the survivors, so it can be checked without knowing the truth. For continuous variables the candidates form an infinite version space, and mixed-integer linear programs decide the stopping condition exactly over all of it, with agreement holding up to a tolerance. On a three-digit colored MNIST causal abstraction task in which ink hue tracks digit size, plain convolutional networks trained on examples reach zero held-out error, yet disagree with shape-based labels on 26% of single-digit edits, as often as hue-based labels do. Auditing the causal abstractions of networks observed only on such images, ISS certifies what each network perceives with 13.6 interventions per image on average, and each certificate, checked against every admissible intervention, holds whenever the network's true abstraction is among the candidates. When a network bypasses a unit that every candidate abstraction relies on, certificates covering interventions on that unit can be silently void, and twenty random validation interventions refute 69% of them.

Primary area:
causal reasoning
Keywords:
causal reasoning, structural causal models, interventional learning, underspecification, active learning, experimental design, causal abstraction, certification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uWeKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces Interventional Separation Selection (ISS), an algorithm claimed to certify interventional agreement within a cost bound when multiple causal models are observationally equivalent but may make different predictions under intervention. ISS actively selects interventions that distinguish among candidate models, eliminates candidates inconsistent with the observed outcomes, and stops when all remaining candidates agree on every admissible intervention within a specified cost bound. The paper illustrates ISS on a causal abstraction task involving MNIST classifiers.

Critical strengths

The paper appears to address an important problem: how to provide formal certificates of interventional agreement when observational data cannot distinguish between competing causal models. However, the difficulty of following the presentation substantially limits my ability to assess the contributions of the paper.

Critical weaknesses

The paper is very difficult to read and follow. It introduces many symbols, notations, and definitions without proper explanation or intuitive motivation. This is not an isolated issue confined to particular sections, which makes the proposed approach very difficult to understand. In its current form, the paper is essentially unreadable to me.

My Informativeness and Soundness ratings reflect the substantial difficulty I had following the paper, which prevented me from confidently judging the soundness of the work, rather than a specific technical flaw that I was able to identify.

Questions for authors

In my view, the paper needs a complete rewriting.

Reviewer_NbirOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In this work, the authors propose a procedure — interventional separation selection (ISS) — that takes a set of candidate causal models (assumed to agree observationally with the data) and iteratively removes models from this set that disagree with the result of an intervention on the data. After the procedure terminates, the remaining models are ‘certified’ to have all agreed on each of the interventions tested. The stopping condition is that no intervention of “cost” at most R (where cost of each intervention is defined manually as an input) can separate any pair of the remaining models.

Critical strengths
  • The proposed method (Algorithm 1) makes use of cost-ordered enumeration of (bounded size) candidate interventions, such that it’s possible to enumerate candidates in polynomial time and to find the cheapest intervention.
  • The theoretical properties of the proposed method are discussed and characterized in detail.
  • A real-data experiment helps to explain and motivate the method.
Critical weaknesses
  • The method is not sufficiently discussed in the language/context of the related work, which has the impact of it not being clear how significant the contribution is. For example, the premise of the entire ISS method is that interventional data allow us to falsify candidate causal models by seeing which interventional distributions they disagree with. But this premise is a known/starting fact in the basic definitions of causal modeling frameworks (e.g., SCMs). The paper makes several statements that, by virtue of leaving out this context, sound overly stated in describing and selling the method; for example, “the stopping condition certifies that every survivor agrees with it on every admissible intervention within the bound, a guarantee that no observational learner can give, however much data it sees.” As another example, the authors state “Our main contribution is a framework for certifying causal reasoning through interventional generalization under observational underspecification.” But it would seem structural causal models in their very definition already provide a comprehensive framework for certifying causal reasoning through interventional generalization under observational underspecification, by virtue of defining observational and interventional distributions entailed by SCMs and defining observational underspecification via the Causal Hierarchy Theorem.
  • The focus of ISS is to identify interventional disagreement from among a set of existing causal models, with the stated hope that the true system is contained in the initial set, and that thus the surviving perspectives agree with the true system. But some level of access to the true system, and the ability to perform interventions on it, is assumed. In what way is this setup distinct from being able to (approximately) identify the interventional distribution via directly sampling from the interventional distribution?
Questions for authors

See two main weaknesses above, with the central question:

  • In what way is this setup distinct from being able to (approximately) identify the interventional distribution via directly sampling from the interventional distribution?
Reviewer_FhrCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In causal modeling, we are often only able to get down to a set of models that represent the same observational distribution but that correspond to different causal structures. Some of these models may produce different results on intervention, so distinguishing between them is important. Other models, though, may yield the same results for all interventions that a user cares about, so they can be considered equivalent for that purpose. The authors describe ISS, a procedure for experimenting in a system by performing repeated interventions until a set of models that are interventionally equivalent is obtained (within a cost bound). The authors describe the mathematical framework behind ISS and demonstrate its effectiveness on a MNIST causal abstraction problem.

Critical strengths

The problem the authors are tackling seems interesting, and the mathematical framework underlying ISS appears quite robust and well-studied. The 3-digit MNIST pointer task, with entangled hue and size, is an interesting test case and seems well-chosen for the authors' task.

Critical weaknesses

This papers seems interesting - unfortunately, the narrative is so poorly presented, and the formalism is presented so densely and without any intuition, that I found it extremely challenging to follow.

Starting from the introduction, the discussion immediately gets very technical, without giving the reader enough context to understand. The first paragraph of the intro talks about "observational regimes", which isn't a term that's been defined yet, and has a forward pointer to Theorem 1, which is not going to make any sense to a reader this early on. The second paragraph talks about "context" before it's been formally defined (as we learn 2 pages later, a 'context' is a specific assignment of exogenous variables, but we have no way of knowing that in paragraph 2 of the intro!). It then gives a simple definition of a "causal perspective" (which the reader can't fully understand yet since it hinges on understanding the meaning of "context" here) and continues to use that term in the third paragraph, and then states that ISS produced a "certificate". Figure 1 is relatively understandable, at least, but the intro really could benefit from more high-level motivation and especially a specific example. Give us a problem domain, define a context, and describe a couple of examples of a causal perspective and how they can be equivalent despite having different structures. The MNIST task is great and very visually understandable, so introducing that here could work really well.

Unfortunately, this lack of narrative clarity continues through the rest of the paper.

  • Section 3 reads as a list of definitions, theorems, and assumptions, with very little connections between them. I see that Section 3 is titled "interventional Risk and the Observational Floor", but it takes until Definition 2 for "Interventional risk" to be mentioned, and until Theorem 1 for "Observational floor". Until then, the reader is given no roadmap for what any of the definitions and formalism in Section 3 are building towards.
  • This lack of narrative clarity continues into Section 4. Section 4 is titled "The Certificate, and Making it Exact", but it opens by defining "Version-space separation cost" and presenting a theorem about "Soundness and certified zero risk", so again, I have no idea what these definitions are serving or building towards.
  • The formalisms as presented are also extremely dense and poorly described. Looking at the first paragraph in Section 3, the first sentence is a fairly standard definition of an SCM. Then we have 2 sentences that include definitions of 3 novel terms ("context", "causal perspectives", and "history-equivalent"). All of these terms are only defined through their mathematical definitions, rather than with any prose or intuition. It would be helpful, for example, to state "a context is a joint assignment of exogenous variables" and then provide the mathematical notation. The notation x(M,u)x(M,u) is also fairly unusual looking, at least to me, so any description of it would help. Especially since it seems like a core part of the definition of "history-equivalent", which seems like a very important concept that is unfortunately not described well here. (the term xobsx^{obs} appears in the definition of "history-equivalent" for the first time, again without any definition)

The experimental results seems interesting and make a good case for ISS. Unfortunately, the presentation of the method before the experiments section has big enough narrative and clarity issues that I don't feel comfortable voting or acceptance. If the authors can introduce the MNIST problem in the introduction and use it as a running example, move some of the mathematical formalism to the appendix to make room for better descriptions of the most important pieces, and add narrative roadmaps to each section explaining how each piece fits into the whole and what its motivation is, I'm open to improving my score.

Questions for authors

No questions

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.