- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents a benchmark, MotionBench-XAI, to evaluate the design choices behind Shapley explanations for spatiotemporal data (player sets, imputers, ground truths and evaluation metrics). It has five synthetic and three real datasets (clinical gait, ECG and audio). The authors vary these choices jointly. The synthetic datasets are carefully constructed so that exact marginal and conditional Shapley values can be computed and each method is graded against the ground truth of its own game. On the real data, where no ground truth exists, they use faithfulness and PlayerAOPC instead. Their main findings are that the choice of players matters more than the choice of imputer, that generative imputers help only when their conditional models are accurate, and that the two real-data metrics often disagree on the best method.
The benchmark covers ten attribution methods, including standard KernelSHAP imputers (zero, mean, marginal, empirical), temporal SHAP variants (WindowSHAP, TimeSHAP), and generative imputers (KS-VAEAC, KS-Gauss and a new flow-matching imputer, KS-Flow). These are evaluated on three player sets, namely time windows, joints, and joint x window cells with up to 68 players. Each synthetic dataset is explained for an MLP, a CNN and a Transformer, and the real datasets use pre-trained or fine-tuned models such as MotionBERT and the Audio Spectrogram Transformer. All methods use the same coalition designs and budgets, and an oracle imputer is included as a reference. Ground truths are exact for up to 12 players, while for larger player sets the authors measure the remaining grading error. Classifiers must also pass an accuracy and overfitting criterion. Finally, a hide-one-reconstruct check with donor and unconditional controls tests whether the generative imputers actually condition on the visible data.
Critical strengths
The main strength is the controlled joint evaluation of the design choices behind Shapley explanations. All methods share the same coalition designs and budgets, ground truths are exact for small player sets, and the remaining grading error is measured for larger ones. Apart from the temporal SHAP baselines, which use their own estimators, differences between methods can therefore be attributed to the imputer and player choices rather than to the estimation. This makes the benchmark a careful and reproducible resource within the Shapley-value literature.
A second strength is game-matched grading. The distinction is known from prior work, but grading each method against the ground truth of its own game makes clear that marginal and conditional Shapley values answer different questions. Result 4 also shows that the common practice of grading everything against the conditional ground truth mixes these up.
Critical weaknesses
The first weakness is that the benchmark measures how accurately each method computes its chosen Shapley game, not whether the attributions are correct explanations. The authors never define what an explanation should recover, the features the model relies on or the mechanism connecting the data to the outcome. The mechanism is usually what clinicians want, and neither Shapley game provides it, since marginal Shapley can credit suppressor variables and conditional Shapley can credit correlated features the model ignores. This disconnect between the clinical motivation and the contribution is the main reason I find the significance limited. Since only the first joint determines the synthetic labels, the authors could address this directly by also reporting whether each method recovers that joint.
The second weakness is that several methodological choices undermine the headline results. The ground truth evaluates the model at the averaged imputation rather than taking the expected prediction, so the protocol rewards low-variance imputers. This is why KS-VAEAC outperforms the exact conditional oracle in Table 3. The authors note in Appendix B that the oracle is not a floor, but this means the ranking reflects how dispersed each imputer's draws are as much as how accurate its conditional model is. This undercuts Result 2. Under the same definition the marginal ground truth is simply the zero baseline, so KS-Marginal's error is mostly the noise of averaging five draws. The synthetic labels depend on the time-averaged mean of a single joint, so there is no temporally localized signal. The synthetic data also lack hard constraints such as bone lengths, so the anatomically impossible imputations that motivate generative imputers cannot occur. On the real data, faithfulness is computed with each method's own imputer on the same coalitions KernelSHAP was fitted on, so it largely measures how additive each method's game is and cannot be compared across imputers. Finally, the claim that grouping matters more than the imputer rests mainly on the periodic datasets, where the conditional ground truth is nearly uniform and EC3 is unstable, and which the authors themselves gray out in the tables.
Questions for authors
- Why is the game defined as the model evaluated at the averaged imputation rather than the expected prediction over imputations? How do the rankings in Tables 3-4 change under the expected-prediction game, and does KS-VAEAC still outperform the oracle?
- Since only the first joint determines the synthetic labels, does each method recover that joint, for example by ranking it first under the joint and joint x window player sets?
- Faithfulness and PlayerAOPC compute v(S) with each method's own imputer. How do the real-data rankings change if both metrics use one common imputer for all methods?