acceptodds
Withdrawn from ICLR 2027

When to Trust Counterfactuals: Reliable Reinforcement Learning with Partially Learned Causal Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Causal reinforcement learning can improve sample efficiency by reusing factual trajectories to learn from alternative counterfactual outcomes. However, this benefit depends on the reliability of the structural causal model (SCM) used to generate counterfactual trajectories. We study this problem through Counterfactually Guided Proximal Policy Optimization (CG-PPO) with a partially learned SCM, where known mechanisms remain explicit and the unknown mechanism is represented by a model of the other agent (MOA) trained with response-equivalence supervision. We find that CG-PPO can sustain the benefit of counterfactual learning when the SCM is reliable, whereas this benefit diminishes as training proceeds when counterfactual rollouts rely on an imperfect learned SCM. We show that factual response correctness strongly predicts behavioral correctness after counterfactual branching, separating high and low reliability states by 56.4 percentage points after one step and 22.4 points after 40 steps. This discrimination remains positive across the observed range of factual–counterfactual trajectory divergence. We use this signal to introduce reliability-aware branch-state sampling through Consecutive Correctness Weighting (CCW) and Censor-Aware Percentile Weighting (CAPW). When counterfactual PPO updates are interleaved with factual PPO updates, CAPW raises mean win rate over the final six evaluations from 57.0% to 63.0% relative to matched uniform sampling, with higher Final MA6 in all five matched training seeds. It also achieves a 13.7% relative improvement in normalized learning-curve area with respect to factual environment exposure over factual-only PPO, with improvements across all five matched seeds. Under stochastic dynamics, factual conditioning additionally improves learning under the tested SCM misspecification but provides no consistent advantage when the SCM correctly models the exogenous-noise distribution. These results show that locally measured reliability can guide counterfactual training-data allocation when reinforcement learning relies on partially learned causal models.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, causal reinforcement learning, counterfactual reasoning, structural causal models, counterfactual policy optimization, model reliability, sample efficiency, opponent modeling

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_m6pzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to allocate counterfactual training data when a structural causal model contains an imperfect learned mechanism. CG-PPO combines known causal mechanisms with a Model of the Other Agent trained through response-equivalence supervision. It uses consecutive factual response correctness to select rollout starting states through CCW and CAPW. Experiments in an invader–defender environment examine reliability transfer after counterfactual branching, policy-learning benefits, interactions with factual updates, and exogenous-noise reuse under model misspecification.

Critical strengths
  1. The finding that factual response correctness predicts correctness after counterfactual branching is interesting and directly relevant to allocating synthetic training data. Section 8.2 provides empirical support for this relationship beyond the shared initial state.

  2. The experiments include matched comparisons across multiple training seeds and a controlled analysis of factual policy updates. In mixed learning, CAPW improves the mean win rate over the final six evaluations from 57.0% to 63.0% relative to Uniform, providing evidence of practical value in the tested setting.

Critical weaknesses
  1. The choice of starting-state selection baselines needs further justification. I am less familiar with this specific literature and would appreciate clarification on whether recent studies provide alternative starting-state selection strategies for counterfactual or model-based rollout generation. Beyond the proposed CCW and CAPW, Uniform is the only evaluated selector baseline. Comparisons with applicable recent methods, or a discussion of why they are unsuitable here, would help assess the added value of the proposed weighting.

  2. The contribution of CAPW’s censor-aware treatment is not isolated. Compared with CCW, CAPW changes both censoring treatment and the weighting function, including for uncensored candidates. The comparison therefore does not distinguish the benefit of censor handling from that of replacing capped linear weights with percentile weights based on uncapped durations. Section 8.5 establishes an interaction with factual policy updates, and Section 9 suggests broader branch coverage as a possible explanation, but the specific role of censor awareness in these gains remains unclear.

Questions for authors
  1. Are there recent starting-state selection methods applicable to this setting? How do their selection rules relate to CCW/CAPW, and could they provide additional experimental comparisons?

  2. What evidence distinguishes the contribution of CAPW’s censor-aware treatment from that of the accompanying changes to its weighting function?

  3. Section 9 hypothesizes that factual updates may reduce sensitivity to imperfect counterfactual data, potentially making broader branch coverage useful. Can the authors clarify how this interpretation relates specifically to CAPW’s censor-aware treatment, as opposed to its changed weighting function?

Reviewer_4Jy6OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The method, CG-PPO, generates counterfactual rollouts in the style of CF-GPS and trains on them with PPO. The testbed is a three-goal invader–defender game.

Critical strengths
  • Negative and mixed results are reported honestly and carefully.
  • The reliability-transfer analysis is cleanly designed (landmarking, censoring, divergence strata).
Critical weaknesses
  • The SCM is degenerate in the main experiments, so the counterfactual framing has no content. In all core experiments U^I = U^D = 0 (Sec. 4), and every mechanism is deterministic. With a deterministic SCM and a stored real state, there is nothing to abduct. Layer 3 collapses to rung 2, which collapses to forward simulation from a visited state. The planner's actions are sampled fresh, so no factual action noise is reused either. "Counterfactual rollout" here is therefore an MBPO-style branched model rollout (Janner et al., 2019) in a gray-box model.
  • The "learned mechanism" is known in closed form, and the MOA is given its sufficient statistic.
  • The stochastic study (Sec. 8.6) reports an expected result, not a new finding. Buesing et al. 2019 already noted this.
  • The framework reduces to a factored model, which undercuts the novelty. A known graph, known deterministic mechanisms and one learned factor make a factored (DBN) transition model (Boutilier et al., 2000; Guestrin et al., 2003).
Questions for authors
  1. Since the belief is an argmin over MOA inputs, what is the MOA's error rate, and where does it occur? Does the reliability problem disappear if the analytical rule is used?
  2. In the deterministic setting, which step of CG-PPO performs abduction? How does it differ from MBPO branched rollouts?
  3. How does CAPW compare with M2AC, MACURA and Abbas et al. at an equal synthetic budget? Please plot AUC against total (factual + synthetic) transitions and against wall-clock time.
  4. Could you add proper discussion and citations to CoDA (https://arxiv.org/abs/2007.02863), MoCoDA (https://arxiv.org/abs/2210.11287), and CSCA (https://arxiv.org/pdf/2607.16999)?
Reviewer_eq8gOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies reliable counterfactual reinforcement learning with a partially learned structural causal model and proposes two reliability-aware branch-state sampling methods, CCW and CAPW, which leverage factual response correctness to identify trustworthy counterfactual rollouts and improve learning efficiency in an invader–defender task.

Critical strengths

The paper identifies a practically meaningful reliability signal that remains predictive after counterfactual branching and develops the censor-aware CAPW scheme to exploit this signal, providing a principled and empirically effective approach to improving the learning efficiency of counterfactual RL with partially learned causal models.

Critical weaknesses

The paper has several critical limitations: (1) the empirical evaluation is restricted to a single invader–defender environment; (2) comparisons with strong reliability- and uncertainty-aware baselines are missing; (3) the claimed sample-efficiency improvement considers only factual environment exposure while ignoring the additional counterfactual rollout and optimization costs; (4) the proposed method does not outperform factual PPO in final performance; and (5) the observed reliability transfer lacks theoretical justification or validation across diverse tasks, substantially limiting the generality and practical significance of the conclusions.

Questions for authors

The paper would be more convincing if the authors could provide evaluations across multiple environments, comparisons under matched total computational budgets, an online-computable reliability estimator, model-ensemble and rollout-truncation baselines, and experiments with more random seeds.

Reviewer_K27NOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies counterfactually guided PPO (CG-PPO) in an invader–defender game whose structural causal model is specified except for the defender's believed goal, which is learned by a Model of the Other Agent (MOA) trained with response-equivalence supervision. The authors show that the benefit of counterfactual rollouts fades when the SCM is learned, that the number of consecutive factual steps on which the MOA predicts the defender's response correctly predicts its correctness after counterfactual branching, and propose two branch-state samplers that use this statistic: Consecutive Correctness Weighting (CCW) and a censor-aware variant (CAPW) built on a weighted Kaplan–Meier estimate. Under an immediate-decay schedule that interleaves factual and counterfactual PPO updates, CAPW improves final win rate over uniform branch sampling and learning-curve area over factual-only PPO. A separate stochastic study tests reuse of factual exogenous noise under SCM misspecification.

Critical strengths

Honest reporting: limitations are stated clearly, all summaries are labelled as descriptive, and mixed or negative findings (the regime dependence of CCW vs CAPW, factual PPO's higher endpoint) are reported rather than hidden. The censoring treatment of correctness streaks is a sensible idea, and the reliability-transfer validation in Section 8.2 is a useful check that factual reliability still ranks counterfactual reliability after branching.

Critical weaknesses

There is no endpoint benefit from counterfactual learning. Factual-only PPO has the highest mean Final MA6 (0.641 vs 0.630 for ID-CAPW in Section 8.4; 0.655 vs 0.622 for learned-SCM CG-PPO in Appendix F). CAPW's advantage over factual PPO is only in normalized AUC, i.e., faster early learning, and it is bought with 4.75M additional counterfactual transitions plus MOA training. Because the x-axis counts only factual steps, the "13.7% improvement" ignores the cost of the counterfactual updates; no compute or wall-clock comparison is given. The reliable-SCM reference that motivates the work is, as the authors note, not deployable.

The effects are small relative to the variance and are seed-driven. CAPW vs Uniform: AUC 0.494 vs 0.469 with 3 of 5 seeds; CCW has an across-seed SD of 0.18; seed 50K alone drives the CAPW−CCW interaction (+0.345); the R4 comparison uses three seeds and the authors conclude it "does not establish a material schedule change". Table 8 shows ID-CCW (0.373 ± 0.182) below ID-Uniform (0.479), so the first proposed selector hurts. The whole evidence base is one environment, one heuristic defender, three to five seeds, and no inferential statistics.

Missing baselines. The natural competitors for deciding where to trust a learned mechanism are uncertainty-based: MOA predictive entropy or ensemble disagreement as branch weights, and rollout truncation (MBPO, M2AC, MACURA). The paper cites these methods but does not compare with them, so it is unknown whether the streak statistic has any advantage over a calibrated uncertainty signal that is available before rollout and for any learned mechanism.

Generality. The reliability signal requires that the learned mechanism's output can be verified step by step against observed behaviour on factual trajectories (here the defender's response is observed, so MOA correctness is checkable). Most partially learned SCMs do not expose such a check, and the paper does not discuss how ℓ_t would be obtained otherwise.

Presentation. The text is dense with acronyms (CF:F, ID, R4, MA6, CCW, CAPW, q_eff), the abstract is a list of numbers, the main results live in Appendix F–H tables, and the stochastic study in Section 8.6 uses a different cohort, CCW, and CF-only updates, so it does not connect to the proposed method.

Questions for authors

How does CAPW compare with MOA-entropy or ensemble-disagreement weighting and with short-horizon truncated counterfactual rollouts under the same budget?

What is the total compute of ID-CAPW versus factual PPO at 3M factual steps, and does the AUC advantage survive when counterfactual transitions are counted on the x-axis?

Why does ID-CCW underperform ID-Uniform in Table 8? Does concentrating on long streaks reduce branch diversity, and would an entropy-regularized selector fix this?

Could the authors show results in a second environment, or at least with a learned (non-heuristic) defender, to demonstrate that the reliability signal is not specific to this rule-based response?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.