acceptodds
Withdrawn from ICLR 2027

Causal Shapley Interaction Consistency for Credit Assignment in Cooperative Multi-Agent Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Shapley-based multi-agent reinforcement learning (MARL) methods distribute a shared reward according to agents' marginal contributions to provide a credit assignment as a prominent topic in artificial intelligence research. However, existing methods often overlook the relationship of causal semantics between agents' actions, leading to inaccurate credit assignment. To address this issue, we propose a Causal Shapley Interventional Consistency Regularization framework, namely THEMIS, which separates causal identification, critic calibration, and policy optimization through three components. The causal-identification component applies a one-time direct intervention to recognize causal relationship between agents' actions. The critic-calibration component applies an interaction consistency to align selected critic-induced interactions with these causal targets. The actor component applies a detached conventional advantage to prevent teacher gradients from influencing the policy update. Across different agents MPE tasks, THEMIS exceeds the normalized team-reward baselines on both learning metrics.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Cooperative multi-agent reinforcement learning (MARL), Shapley value

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.60 over 5 reviews, above 10% of the others.
Reviewer_31ciOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Themis, which uses policy-interventional Shapley interactions as stop-gradient targets to calibrate a centralized critic while keeping a detached MAPPO actor update. It contributes a policy-interventional Shapley formulation, a critic-calibration framework, and two theoretical propositions. Validation includes SCM identification, frozen-checkpoint audits, and MPE Simple Spread

Critical strengths
  1. Causal formulation: Introduces policy-interventional Shapley interactions for simultaneous-action MARL with explicit assumptions A1-A6, addressing confounding in credit assignment.
  2. Modular design: Separates the causal teacher, critic calibration, and actor update via stop-gradient targets and detached advantages, preventing uncontrolled causal gradients from entering policy optimization and keeping MAPPO updates stable.
Critical weaknesses
  1. The theoretical contribution is limited. Propositions 1 and 2 are weak: Proposition 1 is nearly definitional under the assumptions of an exact teacher and zero consistency loss, while Proposition 2 is merely a negative result. The paper lacks finite-sample analysis, convergence guarantees, or a formal link between calibration error and policy improvement. Consequently, the claimed "formal calibration guarantees" remain unsubstantiated.

  2. The empirical validation is severely insufficient. The experiments are limited to MPE Simple Spread with 3–5 agents. Strong MARL baselines (e.g., MAPPO, QMIX, MACCA) and multi-environment evaluations (e.g., SMAC, MAMuJoCo) are missing. The reported 5%–7% gains lack statistical rigor—no seeds, confidence intervals, or significance tests are provided. A systematic sensitivity analysis for key hyperparameters(λICR\lambda _{ICR}, wij,tw_{ij,t}, Pt\mathcal{P}_t, M/KM/K) is also absent.

  3. Strong assumptions and lack of reproducibility. Assumptions A5 (fixed continuation policy) and A6 (identification route) are highly restrictive, and their violations are not empirically analyzed. The paper does not report additional training costs (wall-clock time, GPU memory, teacher rollouts) compared to MAPPO, and the code/hyperparameters are not released.

Questions for authors
  1. Please provide full proofs for Propositions 1 and 2, including all assumptions, and clarify whether any finite-sample or convergence guarantees are intended. If not, will the authors weaken the "formal calibration guarantees" claim?
  2. Please define wij,tw_{ij,t} in Eq. (9) exactly: is it uniform, heuristic, or learned? Since it directly scales each pair’s consistency gradient, how sensitive are the results to its choice, and how does it affect the scope of Proposition 1? Provide its exact definition, normalization, and a sensitivity analysis.
  3. Please evaluate on a harder MARL benchmark (e.g., SMAC, MAMuJoCo) with strong baselines (e.g., MAPPO, QMIX, MACCA) and report seeds, confidence intervals, and significance tests for the reported gains.
  4. How often is πc\pi ^c refreshed, and how do you detect or handle Assumption A6 violations? What is the additional training cost compared with MAPPO, and will code and parameters be released?
Reviewer_WnsMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies causal credit assignment in multi-agent reinforcement learning (MARL) and identifies two challenges: causal identification under observational confounding and learning integration of causal targets into actor optimization. The authors propose Themis, a three-component framework consisting of (i) a causal teacher that constructs policy-interventional coalition values via one-time joint-action interventions and converts them into stop-gradient Shapley-interaction targets, (ii) a centralized critic that is calibrated to match these targets through an interaction-consistency regularization loss, and (iii) an actor that retains a conventional detached advantage and standard clipped MAPPO update. The paper provides conditional calibration guarantees under a shared coalition game and reports experiments on MPE Simple Spread tasks.

Critical strengths
  1. The distinction between observational conditioning and one-time joint-action intervention is well motivated.

  2. The three-component design with explicit gradient boundaries is clean and sensible. Isolating causal targets as a stop-gradient teacher for critic calibration, while preserving a detached advantage for actor updates, avoids conflating causal estimation with policy optimization.

  3. The paper provides formal calibration guarantees that make explicit the conditions (exact teacher, zero ICR loss, shared coalition game) under which critic interactions recover the targeted causal Shapley interactions.

Critical weaknesses
  1. The theoretical guarantees are conditional and somewhat limited. In practice, the teacher is a sampled estimator and the zero-loss condition is unreachable, yet the paper does not provide finite-sample or approximation error analysis to bridge the gap between the theory and the experiments.

  2. Experiments are restricted to MPE Simple Spread with 3–5 homogeneous agents and factorized policies.

  3. The paper does not compare against strong MARL credit-assignment baselines such as COMA, QMIX, MAPPO with observational Shapley, SHAQ, STAS, or MACCA.

  4. The individual contributions of the interaction-consistency loss, teacher freezing/cross-fitting, and the choice of complement kernel are not isolated.

  5. More illustrative examples and figures would improve readability. At the same time, the learning curves are not shown and the implementation details are underspecified.

Questions for authors
  1. How does the calibration error propagate to the critic and to the actor update?

  2. Could the authors report results against stronger credit-assignment baselines (e.g., COMA, QMIX, MAPPO with observational Shapley, MACCA)?

  3. How sensitive are the results to the choice of continuation policy πc\pi^c (current snapshot vs. older policy)?

Reviewer_ERdVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Themis, a framework for causal credit assignment in cooperative multi-agent reinforcement learning. It uses policy-supported interventions to estimate Shapley interactions between agents and introduces a consistency loss to align the centralized critic with these causal estimates, while retaining standard MAPPO actor updates. The paper also presents two theoretical propositions concerning interaction calibration and causal attribution. Experiments on a controlled causal model and MPE Simple Spread tasks with three to five agents evaluate attribution accuracy, sampling stability, and learning performance.

Critical strengths

The paper provides a clear intervention-based formulation of Shapley interactions in cooperative MARL, with explicit assumptions about action legality, policy support, and continuation policies. This helps distinguish causal attribution from observational Shapley estimates and provides a useful foundation for studying causal credit assignment.

Critical weaknesses
  1. The paper shows that observational Shapley attribution can be misleading under confounding, but does not establish whether this problem arises in the evaluated MARL tasks. Moreover, Themis uses causal Shapley interactions to regularize the critic rather than directly assigning causal credit to individual agents. It remains unclear how this indirect approach improves individual-agent credit assignment.

  2. The theoretical results offer limited new insights. Proposition 1 follows directly from the zero consistency-loss assumption and an exact teacher, while Proposition 2 largely formalizes the known distinction between observational and causal attribution. Although the experiments show performance gains, they do not directly demonstrate improved causal interaction estimates in the critic. The evaluation is also limited to MPE Simple Spread with three to five agents, without reporting variability across learning runs.

Questions for authors
  1. Is there evidence that observational Shapley attribution leads to meaningful causal misattribution in the evaluated MPE tasks?

  2. How does calibrating pairwise causal interactions in the critic improve individual-agent credit assignment when the actors still use conventional advantages?

  3. What new theoretical insights do Propositions 1 and 2 provide beyond the zero-loss calibration condition and the known distinction between observational and causal attribution?

  4. Can the authors provide evidence that Themis improves the critic's causal interaction estimates during training, along with variability across independent learning runs?

Reviewer_ay1mOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a framework for calibrating centralized critics using policy-interventional Shapley interactions in cooperative MARL. It estimates coalition values through one-time joint-action interventions under a fixed continuation policy, constructs detached interaction targets, and regularizes critic-induced interactions against these targets while retaining a conventional actor update. The theoretical results establish calibration under exact-teacher and zero-loss assumptions. Validation includes a controlled confounding study, a frozen-policy support and variance audit, and learning experiments on MPE.

Critical strengths
  1. The paper carefully distinguishes the causal interpretation of coalition values from the axiomatic validity of Shapley allocations. Its explicit intervention assumptions and matched confounded/deconfounded controls provide a useful foundation for assessing when Shapley-based explanations represent intervention effects.
  2. The separation of causal estimation, critic calibration, and policy optimization provides a clear learning architecture.
Critical weaknesses
  1. The experiments do not isolate the contribution of causal interaction calibration. The baselines are described as different reward-allocation conditions, leaving it unclear whether their learning objectives match.
  2. The empirical evidence is insufficient to establish reliable downstream benefits. Evaluation covers only three sizes of one task.
  3. Regarding related work, large prior works are missing, e.g., NA2Q (ICML 2023) and QCoFr (NeurIPS 2025).
Questions for authors
  1. How many additional environment transitions are used to construct the teacher? The authors can provide a matched-budget comparison that uses those same interventional samples for direct action-value regression without the Shapley interaction loss.
Reviewer_SahSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes THEMIS, a framework for calibrating a centralized MARL action critic against pairwise Shapley interactions estimated from one-time joint-action interventions. A causal teacher estimates long-horizon coalition values under a fixed continuation policy; an interaction-consistency penalty aligns critic-induced interactions with these targets; decentralized actors use a conventional detached advantage. Evaluation comprises a synthetic confounding study, a frozen-policy intervention audit, and MPE Simple Spread with three to five agents.

Critical strengths
  • Explicit intervention semantics. Sections 3.1–3.2 carefully specify pre-action information, simultaneous assignment, the complement-action kernel, and continuation-policy conventions. These distinctions are important when interpreting coalition values.
  • Appropriately bounded claims. The paper explicitly acknowledges that matching selected pairwise interactions does not identify a unique causal hypergraph, recover the complete critic, or guarantee improved policy return.
  • Reasonable architectural separation. Using intervention-derived supervision to regularize a critic, rather than directly adding it to rewards, is a plausible design. Matching the teacher and critic coalition kernels is particularly important.
Critical weaknesses

1. The learning experiments are not clearly aligned with the proposed algorithm

The method is presented as critic calibration without reward modification: Equations (9)–(11) introduce an interaction loss and retain a conventional actor advantage. In contrast, Section 4.1 describes “four matched reward conditions,” including equal-split rewards and observational Shapley credit, and Section 4.4 attributes results to “direct-interventional credit.”

It is therefore unclear whether observational Shapley is:

  • a matched interaction-regularized critic using observational targets;
  • an agent-specific reward redistribution method; or
  • another actor-side credit mechanism.

These alternatives test different hypotheses. The paper needs explicit losses and actor updates for every condition. In particular, it should explain how equal-split rewards relate to the shared action critic and advantage in Equation (11). Without this information, the central performance comparison is not interpretable.

The statement that factual trajectories are held fixed “across all learning conditions” also requires clarification. Independently updated on-policy actors generally generate different trajectories. If trajectories are literally shared, the off-policy learning protocol must be specified; if only initial conditions and random seeds are matched, the wording should be corrected.

2. The causal motivation is not connected convincingly to the downstream benchmark

The SCM deliberately includes an omitted common cause of actions and outcomes. That establishes a possible observational/interventional distinction, but not its relevance to the evaluated on-policy MPE setting.

For independently randomized decentralized policies, conditioning on the histories determining those policies ordinarily removes action–outcome confounding, provided policy randomization is exogenous. Under those conditions, an observational action critic marginalized using the matching policy kernel can already represent the proposed intervention response. Correlation among agents’ actions alone does not establish residual confounding.

The authors should identify the actual omitted variable or assignment mechanism producing a causal discrepancy in their MPE experiments, and measure that discrepancy directly. Otherwise, the gains could reflect additional rollout supervision or generic critic regularization rather than correction of causal bias. The synthetic deconfounding result does not resolve this question.

3. Baselines and ablations do not isolate the contribution

The evaluation lacks sufficiently strong, clearly specified comparisons. A normalized “team reward” reference is not an adequate description of the underlying learner, implementation, and tuning.

Critical missing comparisons include:

  • The identical action-critic/MAPPO pipeline with λICR=0\lambda_{\mathrm{ICR}}=0.
  • Observational and interventional interaction teachers used in otherwise identical critic losses.
  • Direct interventional return supervision without Shapley interaction aggregation.
  • A compute- or environment-interaction-matched baseline using the additional teacher rollout budget.
  • Ablations over interaction order, queried pairs, regularization strength, teacher accuracy, and teacher refresh frequency.
  • Relevant implemented credit-assignment baselines, rather than treating a custom observational-Shapley condition as representative of the cited methods.

The extra simulator access is especially important: improvements of approximately 4–7% cannot be attributed specifically to causal interaction consistency without accounting for additional information and training cost.

4. Experimental reporting is inadequate for reproduction or statistical assessment

The supplied paper does not specify essential details, including:

  • SCM structural equations, parameters, and outcome construction.
  • Number of training seeds and uncertainty-estimation procedure.
  • Network architectures, optimizer settings, training horizon, and PPO hyperparameters.
  • Return targets, discounting, interaction weights, and regularization coefficients.
  • Teacher rollout horizons, number of histories queried, continuation-policy snapshots, and refresh schedule.
  • Concrete cross-fitting or lagged-teacher implementation.
  • Simulator reset procedure and treatment of latent state when conditioning on a history.
  • Absolute returns, normalization formulas, and numerical teacher-query or wall-clock costs.

The reported normalized AUC and final-return ratios also need precise definitions. Simple Spread commonly uses negative returns, so a ratio above one does not automatically mean better performance. Any sign reversal, shifting, or other transformation must be stated. The supplied reporting does not allow a reliable assessment of whether the modest gains exceed training variability.

5. The support audit conflates different notions of support

The paper alternates between action legality, behavior-policy support, and causal validity. A legal intervention need not have positive probability under the current behavior policy when a simulator can execute it directly; observational positivity is a separate requirement.

Moreover, standard finite-logit softmax policies assign positive probability to all unmasked discrete actions. The claimed 50–61% “zero-support” rates therefore require an explicit explanation: are these masked actions, thresholded probabilities, empirical non-occurrence, or genuinely impossible joint assignments? Standard Simple Spread also does not explain the claimed mask structure by itself.

Sampling from the policy makes zero behavior-support violations largely true by construction. This does not independently establish a better causal estimator. Likewise, the compared complement kernels define different coalition games, so lower “plan variance” needs a precise definition and an accuracy comparison, not just a relative variance figure.

6. The theoretical contribution is limited

Proposition 1 follows immediately from a zero weighted squared loss and exact targets. Strictly, an expectation of zero establishes equality almost surely under the evaluation distribution, rather than an unrestricted pointwise guarantee. Proposition 2 essentially assumes that the observational/interventional discrepancy survives the selected contrast and concludes that the contrast differs.

These observations are useful clarifications but offer little substantive analysis of the practical algorithm. There is no finite-sample interaction error bound, treatment of teacher noise or policy drift, or analysis of the trade-off between factual return prediction and causal calibration. Such analysis would strengthen the contribution considerably.

7. Error:

Proposition 2 incorrectly infers nonzero critic interaction-consistency loss from a surviving observational-versus-interventional coalition contrast. Observational conditioning on a coalition is not generally the same as marginalizing the optimal full-action observational critic with the declared complement kernel. The counterexample above satisfies the stated discrepancy condition while every observationally optimal critic has zero interaction-consistency loss. Location: Section 3.5, Proposition 2 on page 6 and its proof sketch on page 7; relevant definitions are Equations (2), (4), and (7).

Questions for authors
  1. Section 3.3 trains one shared action critic with LQ+λICRLICR+λVLV\mathcal{L}_Q + \lambda_{\mathrm{ICR}}\mathcal{L}_{\mathrm{ICR}} + \lambda_V\mathcal{L}_V and updates the actors through the detached advantage in Equation (11). Section 4.1 instead compares four matched reward conditions, and Section 4.4 attributes the MPE result to direct-interventional credit. For team reward, equal split, observational Shapley, and Themis, what is the critic loss, the actor objective, and the scalar that enters the advantage? In observational Shapley, is I^ij,tdo\widehat{I}^{\mathrm{do}}_{ij,t} in Equation (9) replaced by an observational interaction while every other term stays fixed, or is the team return split into agent-specific rewards before Equation (11)? Section 4.1 also says the factual trajectories are held fixed across conditions. Are those rollouts shared, and under what off-policy correction, or are only the seeds, initialization, and hyperparameters matched?

  2. Figure 2 reports causal–observational credit gaps of 0.323, 0.305, and 0.286 when an episode-level common cause is withheld from the credit history, and gaps of 0.000, 0.002, and 0.003 after that cause is given to the centralized estimator. Please state the SCM equations, the distribution of the common cause, and the construction of the team outcome. On Simple Spread the actors randomize from local observations. After conditioning on the history that determines those policies, which omitted variable still separates Equation (4) from Equation (2)? Please report that gap, and the pairwise interaction gap under Equation (6), on the pairs and complement kernel used in training, both for the Themis critic and for the same pipeline with λICR=0\lambda_{\mathrm{ICR}}=0.

  3. Themis is reported at normalized AUC 1.050–1.070 and final-window return 1.040–1.055, and is described as the only condition above the team-reward reference on both metrics. Simple Spread returns are commonly negative, so please define both ratios: the final window, any sign change or shift, and the denominator. On the same seeds, what are the absolute returns, the number of seeds, and the results for the identical MAPPO action critic with λICR=0\lambda_{\mathrm{ICR}}=0, for observational and interventional teachers inside the same Equation (9), for interventional-return supervision without the contrast in Equation (6), and for a team-reward run given the same number of extra teacher environment steps? How many coalition queries, complement samples, and environment steps does one Themis update consume relative to that baseline?

  4. Fixed replacement in Figure 3 has zero-support rates of 0.50 at both three-agent budgets, 0.56 and 0.55 for four agents, and 0.61 and 0.60 for five agents; policy intervention is 0.00 throughout. A masked softmax assigns positive probability to every unmasked discrete action, and standard Simple Spread does not itself impose a joint mask of this density. Is a zero-support query a masked action, a probability below a stated threshold, a joint assignment missing from the logged batch, or an assignment the simulator rejects? Please give the mask and the threshold. Please also define normalized plan variance. The two complement kernels induce different coalition games, and the reported plan variances are 0.67–0.72. What is each estimator's error against one common interventional target at M4K4M4K4, M8K8M8K8, M8K4M8K4, and M16K8M16K8?

  5. Proposition 2 infers a nonzero interaction-consistency loss for an observationally optimal critic from an observational–interventional discrepancy that survives one selected contrast. Equation (4) conditions on the coalition action; Equation (7) marginalizes a full-action critic under the declared complement kernel. For G=A1A2G=A_1A_2, behavior probabilities P(0,0)=P(1,1)=0.4P(0,0)=P(1,1)=0.4 and P(0,1)=P(1,0)=0.1P(0,1)=P(1,0)=0.1, and a Bernoulli(0.5)(0.5) complement kernel, the assignment (1,1)(1,1) has interactions 0.40.4 and −0.2-0.2, while q∗(a1,a2)=a1a2q^*(a_1,a_2)=a_1a_2 reproduces vdov^{\mathrm{do}} under Equation (7) and therefore has zero consistency loss. Which coalition game is the premise of Proposition 2? If the intended object is the kernel-marginalized optimal critic, what is the corrected condition, and which reported run meets it?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.