The Gap Between Shapley-Based Causal Attribution and Policy Optimization

Counterfactual credit assignment can reveal which decisions matter in a trajectory, but good attribution does not guarantee a good policy update. We study this gap through Counterfactual Shapley credit. A later action can change Shapley credit assigned to an earlier decision, yet its return-to-go update never reads that earlier score. We show that even exact, credit-conserving Shapley attribution with correct within-state action rankings can induce a strict local decrease in the original return. This exposes an update-visibility gap: action-dependent credit can be present in the attribution but absent from the update of the action that changes it. We introduce RECAP, which preserves the original Shapley allocation while adding the omitted cross-time effect to the corresponding learning signal. For Counterfactual Shapley, we derive a tractable form of this effect and show when compensation yields a better local update, including when the original Shapley update is already beneficial. Across three tasks with distinct temporal dependencies, RECAP consistently improves over CSCA. Independent local diagnostics show that the predicted compensation gain tracks the observed return gain, and the advantage persists with larger attribution budgets, learned policy stochasticity, and noisy rewards.