acceptodds
Under review as a conference paper at ICLR 2027

CACTI: Cross-Action Counterfactual Transition Inference from Offline Data

Abstract

Same-event counterfactual transition inference asks what the next state would have been under a different action while keeping the disturbance realized in the observed event fixed. Accurately fitting action-conditional transition distributions alone does not determine these same-event pairings, especially under uneven action coverage. We propose CACTI, a structured transition causal model that combines a shared state-dependent action response with a conditional invertible model of event-specific variation. This structure lets variation across related actions constrain queries with little direct data while retaining information from the factual event. Under a linear shared-response model and sequential exogeneity, we prove that, at a fixed state and without additional cross-state restrictions, the response to a query action difference is identifiable if and only if that direction lies in the range of the conditional action covariance. We further derive a local counterfactual error bound in terms of shared-response mismatch and action distance. We evaluate CACTI against same-event simulator oracles on four control tasks. It achieves the lowest mean counterfactual error on both all nonfactual queries and low-direct-support subsets, outperforming the compared predictive and explicit-abduction baselines. Policy-learning experiments with limited data further show that the generated counterfactual transitions can provide useful signals for downstream training.

Primary area:
causal reasoning
Keywords:
Causal Inference, Reinforcement Learning, Counterfactual Data Augmentation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_tdT5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors explore same-event counterfactual inference for transitions. The authors point out that matching action-conditional distributions does not determine same-event pairings, especially when action coverage is uneven.

CACTI models the state increment a combination of two terms (stage and event terms).

Results: CACTI has the lowest mean counterfactual error on experiments.

Critical strengths
  • Clear problem framing and several experiments.
  • The error decomposition is interpretable, and the paper also raises some of its limitations.
  • Consistent gains across four environments of different type, including a high-dimensional one, and the assumptions are stated explicitly.
Critical weaknesses
  • CACTI beats the baseline - but it is unclear where the gains are coming from.
  • Only one high-dimensional task is evaluated, and there is limited discussion on the topic.
  • Offline RL needs multi-step counterfactual trajectories, where one-step errors compound and the exogenous variables of later steps depend on earlier ones. The paper does not analyze or test how the error bound or the method scales to multiple steps, and the conclusion leaves it as a limitation.
Questions for authors
  1. Can you explain in plain English wha tis the Theorem 1 guarantes and novelty over previous works?
  2. Considering most offline RL needs multiple steps: did you test in this setting? What would be the challenges and limitations on extending to such a setting?
  3. On the high-dimensional tasks - what are the challenges?
Reviewer_1wQEOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a technique called CACTI which learns to predict alternative outcomes under counterfactual situations.

Critical strengths
  1. Paper proposes to break down an outcome into 3 pars: baseline, action specifc, and event-specific change.
  2. They learn a model that from a dataset of (state, action, outcome) triplet into its 3 constituent parts.
  3. Based on the experiments in the paper, the model they learn achieves better counterfactual test accuracy on 4 tasks. compared to all the baselines.
Critical weaknesses
  1. Effect of the flow part is not isolated in the experiments even though its purported as an important contribution
  2. The paper uses a linear dependence on action, but does not establish its tradeoff with say a nonlinear dependence.
  3. the practical significance of the applicability of the approach is not established, the experiments are done on toy datasets.
Questions for authors
  1. Can you answer weakness 2?
Reviewer_Uq21OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies same-event counterfactual transition inference in offline RL: given a factual transition (s, a, s'), predict s' under an alternative action a' while keeping the disturbance realized in that event fixed. CACTI models the state increment as m(s) + B(s)a + g(s, u), where B(s) is a shared state-dependent action response and g is a state-conditioned RealNVP flow that does not receive the action. The model is trained by factual likelihood plus a squared-error response loss and a response regularizer. Under exact inversion the one-step query reduces to s' + B(s)(a' − a). The authors prove that, at a fixed state, the response along an action difference d is identifiable if and only if d lies in the range of the conditional action covariance, and give a local error bound in terms of response mismatch and action distance. Experiments compare against a simulator oracle that replays the realized disturbance on MountainCar11, PointMaze, Reacher, and Shadow Hand RotateZ, with event-matching diagnostics and low-data policy learning on the first two tasks.

Critical strengths

Clear problem statement and an honest evaluation protocol: the same-event oracle separates counterfactual fidelity from predictive fit, and the shuffled-U / fixed-U diagnostics are a sensible check that the factual event is actually used. Simple and correct identifiability statement (Theorem 1) that separates directional coverage from direct support, with a readable proof and an explicit non-identifiability construction. Consistent gains against all compared baselines in the four environments, with small seed variance for CACTI.

Critical weaknesses

The flow does not enter the counterfactual. By Eq. 13, under exact inversion the query is s' + B(s)(a' − a) and gψ cancels completely. The "conditional invertible model of event-specific variation" is therefore only a training device for m and B; at query time CACTI is a locally linear-in-action, additive-residual model. The title, abstract and Figure 1 present the flow as a core component, and Section 5.2 is presented as evidence for it, but Shuffled U and Fixed U simply replace the factual residual by a wrong one, which must hurt by Eq. S32 and says nothing about the flow. The ablation that matters is missing: train m and B with Lresp + RB only (no flow, no likelihood) and query with Eq. 13.

The model class appears to match the test environments. CACTI assumes an additive, action-independent disturbance and a response that is linear in the action at a fixed state. The paper never describes how the disturbance enters any environment (Appendix C.3 only defines support bins), so the reader cannot judge whether this structure holds by construction. MountainCar and a force-controlled point mass are linear in the action, which would explain the 24x and 40x gaps on MountainCar11 and PointMaze, whereas on Reacher and RotateZ, where the action enters through nonlinear joint dynamics, the gain shrinks to 37% and 21% and RotateZ errors stay large (0.361 normalized). Theorem 1 and Proposition 3 assume a locally exact linear response; the paper never estimates the mismatch η_t, so the directional-coverage argument is not checked in the cases where it would matter.

A baseline that isolates the query rule is missing. A deterministic additive-residual model, s' + μ(s, a') − μ(s, a) with an unconstrained MLP μ(s, a), would show whether the gains come from the shared linear response or merely from reusing the factual residual. Gaussian dynamics adds a learned scale and Base-D does not reuse the residual, so neither answers this. There is also no comparison with a nonlinear response B(s, a) or with bijective causal models (Nasr-Esfahany et al., 2023), which the paper cites as the relevant identifiability theory.

The downstream evidence is weak. On MountainCar11 the differences between CACTI and real-only training are within seed variability at every budget (90.67 vs 90.16 at N = 500; Figure 4), on PointMaze the gap is 0.897 vs 0.862 with three seeds, and only the two environments where CACTI has the largest fidelity advantage are tested. No ordinary model-based augmentation baseline (a learned dynamics model used for imagined transitions, as in MOPO/COMBO) is included, so the experiment does not show that same-event counterfactuals are more useful than ordinary imagined transitions.

Scope. The method is one-step only; multi-step counterfactuals, which offline RL rollouts actually need, are not addressed. The necessity part of Theorem 1 holds "without additional cross-state restrictions", but the estimator imposes smoothness across states through the MLPs, so the theorem does not describe what the implemented model can or cannot identify.

Questions for authors

What is the disturbance mechanism in each environment (where the noise enters, whether it is additive and action-independent)? How does CACTI behave when the noise scales with the action or enters through nonlinear dynamics?

Could the authors add (i) CACTI trained with Lresp + RB only and queried via Eq. 13, and (ii) a deterministic additive-residual baseline with an unconstrained μ(s, a)?

Why are the downstream experiments limited to MountainCar11 and PointMaze? Could the authors report Reacher and include an ordinary learned-dynamics augmentation baseline at the same synthetic-to-real ratio?

Can the authors measure η_t with the oracle action Jacobian on Reacher and RotateZ, to connect Proposition 3 to the observed errors and to the low-support subsets?

Reviewer_yLyZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies same-event counterfactual transition inference in offline RL: predicting the next state under an alternative action while preserving the disturbance realized in the observed transition.

Importantly, CACTI assumes action-affine dynamics with action-independent noise: St+1−St=mθ(St)+Bθ(St)At+gψ(St,Ut+1).S_{t+1}-S_t=m_\theta(S_t)+B_\theta(S_t)A_t+g_\psi(S_t,U_{t+1}).

The authors use a state-conditioned invertible normalizing flow to model the residual distribution, trained through likelihood maximization alongside a squared-error loss and regularization. By inferring and reusing the factual disturbance, counterfactual prediction simplifies to S^t+1(a′)=St+1+B^(St)(a′−At).\widehat S_{t+1}(a')=S_{t+1}+\widehat B(S_t)(a'-A_t).

The paper establishes directional identifiability under this model and reports improved counterfactual prediction accuracy across four control environments, alongside some downstream policy-learning gains.

Critical strengths

Problem formulation. The paper addresses same-event counterfactual transition inference and provides a precise characterization of identifiable action directions under its structural assumptions. The authors evaluate predictions against same-event simulator oracles across four control environments, including low-support queries. CACTI consistently outperforms the selected baselines, although the absence of simpler residual-regression baselines and tests of structural misspecification limits the conclusions that can be drawn.

Critical weaknesses
  1. Unclear necessity of the normalizing flow. The flow cancels exactly from the counterfactual predictor, leaving a simple action-response correction. The authors do not compare against ordinary regression with factual residual reuse, making it difficult to establish the contribution of their main architectural component.

  2. Strong and insufficiently validated assumptions. The method relies on several restrictive assumptions:

    • action-affine dynamics: The action response is linear at each fixed state, excluding nonlinear action effects.
    • action-independent disturbances: The effect of changing an action cannot depend on the realized disturbance, ruling out action–noise interactions and action-dependent noise.
    • sequential exogeneity: The disturbance must be independent of the behavior action conditional on the observed state, excluding unobserved action–disturbance confounding.
    • invertible residual model: The flow assumes a full-dimensional invertible representation, potentially unsuitable for transitions with deterministic components.

The experiments do not sufficiently investigate the consequences of these restrictions or establish their adequacy for the considered environments.

Questions for authors

Necessity of the normalizing flow. Since the flow cancels exactly from Eq. (13), what benefit does it provide during training? Could the authors compare CACTI against a simpler action-affine regression model trained with squared loss and factual residual reuse?

Validity of the structural assumptions. How well do the action-affine and action-independent disturbance assumptions hold in the considered environments? Could the authors evaluate robustness to nonlinear action effects and action–disturbance interactions, particularly since these restrictions are central to the identification results? this may affect my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.