acceptodds
Under review as a conference paper at ICLR 2027

The Gap Between Shapley-Based Causal Attribution and Policy Optimization

Abstract

Counterfactual credit assignment can reveal which decisions matter in a trajectory, but good attribution does not guarantee a good policy update. We study this gap through Counterfactual Shapley credit. A later action can change Shapley credit assigned to an earlier decision, yet its return-to-go update never reads that earlier score. We show that even exact, credit-conserving Shapley attribution with correct within-state action rankings can induce a strict local decrease in the original return. This exposes an update-visibility gap: action-dependent credit can be present in the attribution but absent from the update of the action that changes it. We introduce RECAP, which preserves the original Shapley allocation while adding the omitted cross-time effect to the corresponding learning signal. For Counterfactual Shapley, we derive a tractable form of this effect and show when compensation yields a better local update, including when the original Shapley update is already beneficial. Across three tasks with distinct temporal dependencies, RECAP consistently improves over CSCA. Independent local diagnostics show that the predicted compensation gain tracks the observed return gain, and the advantage persists with larger attribution budgets, learned policy stochasticity, and noisy rewards.

Primary area:
reinforcement learning
Keywords:
Causal Inference, Counterfactual credit, Reinforcement Learning, attribution–optimization gap

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6UB6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper identifies an attribution gap in counterfactual credit assignment which occurs when later actions effect the Shapely values of earlier ones. The issue is demonstrated in a simple task. The authors then propose RECAP, which adds an estimate of this gap to the causal attributions (Shapely values) used for learning updates. Experiments in three domains show RECAP improves over a baseline with no compensation (CSCA), and performs well compared to PPO. In evaluations at the end of training, RECAP has the highest return of a selection of baseline credit assignment methods.

Critical strengths
  1. This paper exposes and explores an interesting problem that is relevant to an important challenge in RL (credit assignment).
  2. The experimental comparison to CSCA is robust and strongly supports the method.
Critical weaknesses
  1. I found the paper extremely to difficult to follow. Much of the notation is undefined at first use (e.g. m1{2}m_1^{\{2\}}, HtH_t), and referred to inconsistently (e.g. DtD_t is a core component that should be clearly named and referred to consistently throughout the paper). In my opinion, important preliminary concepts are not discussed in sufficient detail, and doing so could help clarify concepts, notation and definitions used throughout the paper (e.g. MDPs & policy gradient methods, Shapley values, counterfactual Shapely credit assignment).
  2. The comparison between RECAP and CSCA is 'fair' and provides strong evidence for the method, but I believe all of the other baselines get significantly fewer samples? E.g. the counterfactual rollouts ('counterfactual suffix simulation') used by both RECAP and CSCA provide additional information, based on privileged rollouts, that is unavailable to PPO. If I'm interpreting Table 10 correctly, CSCA & RECAP get an additional 50M samples in RoundTrip for example. This needs to be clearly declared, justified, and accounted for in the main body and experiment results, ideally by ensuring equivalent experience and rollouts for baselines (e.g. PPO) and RECAP.
  3. Presentation of experiment results: Figure 2 uses different y-axis scales on each subplot, which makes it harder to interpret the results. I think linear scales throughout would be more appropriate, or at least reproduce the figure with linear scales in the appendix. Also, I'd like to see learning curves for all the baselines (QCA, HQCA, H-DICE, RUDDER), in the appendix would be fine. The final performance of some of the baselines is very similar to RECAP in Table 1 (within reported standard deviation in some cases), so including the full learning curve would allow for a more complete comparison with the baselines.
Questions for authors
  1. The three domains have all been designed/selected to demonstrate the impact the gap between causal attribution and policy optimisation has on learning. What would you expect the effect of RECAP to be on domains that do not have this gap? Is it zero? It would be interesting to see results for a 'null-case' experiment in an environment where you do not expect RECAP to have any impact.
1 comment

Thank you for the detailed response and the additional experiments.

The simulation budget experiments are a thorough and open response to my concern about unequal budgets. They show the gains over PPO do not hold when the total budget is matched, addressing my fairness concerns, but raising sample efficiency concerns. Thank you for the analysis of what happens when the attribution-optimization gap vanishes. The sufficient conditions and the null experiment are useful to see.

The response and new results do not change my score. Overall, exposing and addressing the issue in how CSCA's credit is used is a useful contribution, but this was already reflected in my original score.

Reviewer_HcGeOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that Shapley-value-based credit assignment as used in CSCA treats all actions in a sequence symmetrically, and therefore at least partially ignores the order in which actions occur. The authors propose RECAP, which they claim addresses this limitation by accounting for the sequential order of actions when assigning credit. However, due to the clarity issues discussed below, it is difficult to determine precisely how RECAP incorporates ordering and how it differs from existing order-aware Shapley formulations. The method is evaluated on three synthetic environments, where the authors report that RECAP outperforms CSCA.

Critical strengths
  • The problem setting is interesting and relevant: applying Shapley values for credit assignment in reinforcement learning, where the order of actions in a sequence matters, is a natural and worthwhile direction.
Critical weaknesses
  • Clarity: The paper is very difficult to read. The manuscript contains numerous inconsistencies, and many sentences appear overly compressed, seemingly to save space, which makes them hard to parse. For example, CSCA is referred to in several different ways throughout the paper.
  • Motivation: The motivation presented in the introduction is unclear. It is not evident why the order-insensitivity of existing approaches is a significant problem in practice, or in which settings it matters.
  • Background: The paper provides almost no background, making it difficult to understand which prior works the authors build upon and what their contribution is relative to them. This is especially important given that CSCA (the main baseline the method is compared against) was available on ArXiv for less than 2 months before the submission deadline.
  • Related work: The paper omits relevant literature in both game theory and machine learning on Shapley values over sequences or under ordering constraints, including cooperative games under precedence constraints (Faigle & Kern, 1992), Asymmetric Shapley Values (Frye et al., 2020), Causal Shapley Values (Heskes et al., 2020), TimeSHAP (Bento et al., 2021), and SVERL (Beechey et al., 2023). Positioning the contribution with respect to these works is essential, since several of them directly address order-dependent attribution.
  • Empirical evaluation: The paper states that the experiments do not use "the complete training pipeline of the original CSCA implementation," which suggests that CSCA was not fully re-implemented. This raises concerns about whether the baseline comparison is fair.
  • Reproducibility: The provided code totals fewer than 250 lines, which seems insufficient to reproduce the full set of experiments reported in the paper.
Questions for authors
  1. What is the motivation for this work? In which practical settings does the order-insensitivity of CSCA lead to meaningfully incorrect credit assignment?
  2. How does this work relate to the existing literature on cooperative games under precedence constraints (Faigle & Kern, 1992) and to Asymmetric Shapley Values (Frye et al., 2020)? In particular, how does RECAP differ from computing Shapley values restricted to orderings consistent with the action sequence?
  3. Why were these three synthetic environments chosen for the empirical evaluation? Do they specifically isolate the order-dependence that the method is designed to capture?
  4. Which components of the original CSCA pipeline were omitted in the experiments, and how might these omissions affect the baseline's performance?
  5. What is the computational cost of RECAP relative to CSCA, and how does it scale with sequence length?
1 comment

After reading the rebuttal, I will maintain my score.

Reviewer_vEocOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies what happens when Counterfactual Shapley Credit Assignment (CSCA) is used as the learning signal in a policy gradient algorithm. The paper characterizes the missing term, builds a counterexample where the return is decreased, and derives a closed form for the missing term under a "first-replacement sufficiency" condition. RECAP adds this term back, and its expected update is the true gradient. Experiments on three small deterministic control tasks show RECAP beating CSCA and the other baselines.

Critical strengths

The analysis seems to be correct and it is interesting as Li et al. present their Shapley MDP as a drop-in replacement for any policy-gradient method, and Theorem 1 shows this is wrong. Under Theorem 2 the reason is clear: each residual Δk\Delta_k is shared equally between the first kk positions, so step tt keeps only 1/t1/t of its own advantage.

Critical weaknesses

CSCA itself is not properly introduced in the main text, just two sentences and Fig. 1. The reader has to read the appendix A1 to understand Eq. (4) and Theorem 2. In Sec. 3.1 the notation also switches to replacement coalitions without definition.

Under Theorem 2's conditions RECAP is basically an unbiased advantage estimator built from rollout values. Its expected update is the plain policy gradient, and the extra terms coming from the Shapley allocation have zero conditional mean, so they only change the variance. The paper says as much in Sec. 3.2, but then compares RECAP only against CSCA, i.e. against an estimator whose late-step advantages are down-weighted by 1/t1/t. The comparison that would justify keeping Shapley credit at all is missing: TD residuals, Monte Carlo advantages or GAE computed from the same K=2K=2 rollouts.

The comparison with the other baselines is not budget-matched: CSCA and RECAP use about 50x more simulator steps than PPO and the other credit methods, and even so RECAP's lead over the best of them is within one standard deviation in two of the three tasks. The method also needs deterministic dynamics, continuous actions and a resettable simulator, which limits it a lot (e.g. for LLMs).

Questions for authors
  1. How does RECAP compare to TD, Monte Carlo or GAE advantages from the same rollouts? If RECAP doesn't win, what is the argument for learning from Shapley credit? This is what would most change my rating.
  2. Can you run a budget-matched comparison with PPO and the other baselines?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.