- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Addressing the critical question of whether video action recognition models rely on static appearance information or temporal dynamic information, this paper proposes an analytical framework that holds significant research value. The authors conducted experimental evaluations across two benchmarks and multiple recognition architectures, with the results largely confirming the effectiveness of the proposed approach.
Critical strengths
(1) The core motivation of this paper addresses practical challenges encountered in existing action recognition methods, making it of significant research value.
(2) The evaluation framework in the experiments is relatively comprehensive, and the authors provide extensive ablation studies, which effectively validate the efficacy of the proposed method.
Critical weaknesses
(1) The paper does not clearly distinguish between action semantics, motion, and temporal information. In mainstream approaches, action recognition typically relies on the combined contribution of motion, temporal, and spatial information. Furthermore, there is no strict correspondence between motion-verb and appearance-noun. Although the authors are aware of the cross-axis relationships, further clarification is needed in the paper.
(2) The paper uses the difference between the prediction confidence for the original video and that for a video consisting of a single repeated frame as a metric for motion headroom. However, since changes in appearance features alone can also result in a high value, this does not prove that the model has utilized motion or temporal information.
(3) In the experimental section, the validation is based on only 200 clips drawn from the EGTEA Gaze+ and SSV2 datasets, which is insufficient. The authors should first expand the sample size; furthermore, they should conduct additional validation using more standard activity recognition datasets, such as Kinetics.
(4) The effectiveness of certain modules has not been fully verified through experiments. For instance, the performance advantages of FAM may be influenced by factors such as optimization objectives or mask settings; furthermore, whether FAM successfully identifies discriminative temporal evidence needs further investigation.
Questions for authors
Can the authors further validate the reliability of motion headroom and the effectiveness of FAM?
4 comments
We present below part 2 of our response:
Response to Reviewer kNaf [Part 2/2]
W4/Q: "Do the experiments adequately validate headroom and FAM? Could the gains come from the objectives or mask settings?"
We have added and surfaced several complementary controls, separating diagnostic reliability from explanation quality.
Headroom and reliance robustness.
- Temporal-order tests: Shuffling and reversal show that much of the measured headroom is sensitive to original frame order (Appendix A.2, Table 5).
- Negative control: The single-frame classifier has near-zero verb motion reliance (0.00 on EGTEA, 0.02 on SSv2; Table 2, page 8).
- Threshold sensitivity: The EGTEA–SSv2 distinction persists as varies from 0.01 to 0.20, a 20-fold range (Appendix A.5, Table 7).
- Calibration: The margin- and decision-flip-based SBI variants preserve the main dataset separation (Appendix A.4, Table 6). Probability-versus-margin reliance correlations are 0.85–0.97 on EGTEA's three temporal architectures; we also report the lower SSv2 correlations (0.55–0.69) rather than assuming perfect agreement.
FAM effectiveness and mask controls.
- Complete closest-method comparison: We added Vasu et al.'s complete DAME method for TSM (Section 5.2, Table 3). On EGTEA, FAM achieves Suf$_{hr}$ 0.940 versus 0.344 and appearance deletion 0.114 versus 0.278; on SSv2, the corresponding values are 0.989 versus 0.582 and 0.071 versus 0.117. FAM retains more frames, so these are not matched-budget comparisons.
- Size-matched random masks: FAM improves headroom-normalized sufficiency by approximately 0.22–0.40 over random masks with identical frame counts across six settings (Appendix A.6, Table 8).
- Clip-adaptive budgets: Permuting native mask sizes among clips, while keeping the size distribution fixed, reduces FAM sufficiency by 0.224 on EGTEA and 0.206 on SSv2 (; Appendix A.11, Table 12). This shows that clip-specific frame counts matter, not merely the average mask size. At fixed budgets, however, FAM's AUC advantage over EP-TD is smaller (+0.025/+0.007), which we explicitly acknowledge.
- Objective ablations: Appendix A.9 (Table 10 and Figure 9) isolates individual loss terms on 20 clips per dataset. Replacing headroom-normalized sufficiency with raw retention reduces motion necessity from 0.601 to 0.552 on EGTEA and 0.727 to 0.616 on SSv2; appearance sparsity and the insertion–deletion objective also matter. These ablations do not imply that every term is equally important.
- Independent perturbation validation: Under held-out causal-hold replacement, FAM-selected frames have greater impact than matched unselected frames: +0.143 on EGTEA (95% CI [+0.082, +0.215]) and +0.102 on SSv2 ([+0.046, +0.157]; Appendix A.12, Table 13). Under shuffle, those selected-versus-unselected gaps are close to zero, so we do not claim that FAM uniquely localizes order-sensitive frames. Its goal is to select frames preserving confidence beyond the static reference.
These additions support both the reliability of the model-level reliance audit and the value of FAM's instance-level sufficient explanations, while making their limits explicit. We hope the clearer definitions and additional controlled evidence address the reviewer's concerns and invite reconsideration of the assessment.
(W3) Follow up results on a two new datasets
Additional Results on EPIC-KITCHENS-55 and Kinetics-400
As promised in our earlier responses, we have completed additional experiments on 400 clips from EPIC-KITCHENS-55 and Kinetics-400 using TSM, extending our evaluation beyond EGTEA and SSv2. These experiments directly address the reviewers' requests for broader dataset coverage, including evaluation on Kinetics.
| Dataset | Appearance Method | Insertion ↑ | Deletion ↓ | Temporal Method | Sufficiency ↑ | Necessity ↑ | Frames Kept ↓ |
|---|---|---|---|---|---|---|---|
| EPIC-K55 | Extremal | 0.538 ± 0.007 | 0.260 ± 0.009 | Oracle | 0.983 ± 0.003 | 0.633 ± 0.002 | 25.38 ± 0.15% |
| EPIC-K55 | STEP | 0.649 ± 0.023 | 0.312 ± 0.013 | Ext.-Temp | 0.562 ± 0.052 | 0.758 ± 0.005 | 51.42 ± 0.37% |
| EPIC-K55 | Unified Mask | 0.632 ± 0.004 | 0.263 ± 0.019 | STEP-TD | 0.494 ± 0.029 | 0.646 ± 0.018 | 49.69 ± 0.66% |
| EPIC-K55 | FAM | 0.758 ± 0.002 | 0.155 ± 0.046 | FAM | 0.948 ± 0.013 | 0.721 ± 0.026 | 30.03 ± 0.98% |
| Kinetics-400 | Extremal | 0.315 ± 0.011 | 0.168 ± 0.011 | Oracle | 0.976 ± 0.004 | 0.763 ± 0.042 | 44.44 ± 1.76% |
| Kinetics-400 | STEP | 0.407 ± 0.018 | 0.220 ± 0.007 | Ext.-Temp | 0.437 ± 0.037 | 0.774 ± 0.033 | 45.66 ± 1.37% |
| Kinetics-400 | Unified Mask | 0.375 ± 0.014 | 0.325 ± 0.002 | STEP-TD | 0.441 ± 0.029 | 0.696 ± 0.055 | 49.12 ± 1.62% |
| Kinetics-400 | FAM | 0.617 ± 0.012 | 0.134 ± 0.014 | FAM | 0.959 ± 0.006 | 0.823 ± 0.008 | 69.19 ± 0.74% |
The results are encouraging and consistent with our earlier findings. FAM achieves the best appearance insertion and deletion scores on both datasets. On EPIC-K55, FAM achieves 0.948 headroom-normalized temporal sufficiency, compared with 0.562 for Ext.-Temp and 0.494 for STEP-TD, while retaining only 30.03% of frames, compared with 51.42% and 49.69%, respectively.
On Kinetics-400, FAM achieves 0.959 temporal sufficiency, compared with 0.437 for Ext.-Temp and 0.441 for STEP-TD, and also achieves the highest temporal necessity (0.823). Its sufficiency approaches that of the exhaustive oracle (0.976). However, FAM retains more frames on Kinetics (69.19%) than the baselines, indicating that compactness remains an area for improvement.
Static-Bias and Motion-Reliance Analysis
We also computed motion reliance, the Static-Bias Index (SBI), and motion headroom on both datasets, averaged across three seeds.
| Metric | Kinetics-400 | EPIC-K55 |
|---|---|---|
| Motion reliance | 0.271 | 0.167 |
| SBI | 0.729 | 0.833 |
| SBI_FAM | 0.686 | 0.801 |
| Low-headroom clips (h < 0.05) | 53.1 | 63.9 |
| FAM motion necessity, identifiable clips | 0.670 | 0.552 |
| FAM motion sufficiency (raw) | 0.992 | 0.991 |
The SBI analysis provides another interesting finding. Both datasets exhibit substantial static reliance, with SBI values of 0.729 on Kinetics-400 and 0.833 on EPIC-K55. Moreover, 53.1% of Kinetics clips and 63.9% of EPIC-K55 clips have low motion headroom, suggesting that the strongest repeated single frame already provides much of the model's confidence for a substantial portion of the data.
Together, these results extend our analysis to four action-recognition datasets and provide further evidence for both the usefulness of motion headroom as a diagnostic and FAM's effectiveness as a post-hoc explanation method.
We plan to include the EPIC-KITCHENS-55 and Kinetics-400 results for all three architectures (TSM, TimeSformer, and SlowFast) in the camera-ready version.
We thank the reviewers again for encouraging these additional experiments, which have helped us strengthen the evaluation. We would be grateful if you reconsidered your assessment in light of these new results on two additional datasets.
Final follow up with reviewers
As we approach the end of the discussion period, we would like to thank all three reviewers for their time and thoughtful feedback, which has helped us strengthen the paper. We have tried to address the concerns through manuscript revisions and additional experiments with new baselines and results on two new datasets.
If any concerns remain or further clarification would be helpful, please let us know. We would be happy to discuss them further.
Response to reviewer kNaf
Response to Reviewer kNaf [Part 1/2]
We thank the reviewer for recognizing the research value of our motivation and the extensive ablation studies. We have clarified the terminology, added temporal-order controls, expanded the baseline comparisons, and strengthened the validation of FAM. We address the concerns below using the revised manuscript.
W1: "The paper does not clearly distinguish action semantics, motion, and temporal information; there is no strict motion–verb or appearance–noun correspondence."
We agree and have clarified these distinctions in Section 3 (lines 187–196). Appearance refers to information within frames (objects, textures, colors, and scene context); temporal variation refers to changes across frames, including motion, interactions, and ordering. We use "motion" as shorthand for this broader temporal variation, not as a claim that our interventions isolate physical motion. Action semantics are represented by the classifier's verb/noun predictions (or action label on SSv2).
We do not assume that verbs use only motion or nouns only appearance. Our cross-axis probes evaluate each intervention on both prediction heads (Section 4, Eq. 8; Table 2). On EGTEA, verb appearance reliance is 0.72–0.78 across temporal models, while noun confidence also falls when temporal variation is removed (motion-reliance probe 0.47–0.67). We interpret the latter cautiously because the static reference is selected using the verb head (Section 5.1, lines 449–455). On SSv2, appearance reliance remains 0.79–0.93 even though motion reliance is 0.86–0.88. These cross-axis findings support the reviewer's point: recognition combines these sources. Our probes measure sensitivity to specified interventions, not an exclusive causal decomposition.
W2: "High motion headroom may come from changing appearance features, not temporal information."
We agree that headroom alone does not prove temporal-order reliance. We now state explicitly that the extra confidence may come from combining multiple static observations or from their order (Section 3.2, lines 233–238). To separate these contributions, we added TSM frame-shuffling and reversal controls (Appendix A.2, lines 738–765; Table 5, page 15).
For each clip, we average confidence over 10 shuffled frame orders, retaining the same frames but disrupting their order. With po denoting original confidence, ps the best repeated-frame confidence, and pshuf shuffled confidence, h=po−ps=(pshuf−ps)+(po−pshuf)=hagg+hord.
The order share is E[hord]/E[h]. Reordering removes 80% of mean headroom on EGTEA and 77% on SSv2, while most identifiable clips have order headroom above our threshold. A complementary reversal control changes temporal direction while keeping the original frames and local transitions. Reversal removes 143% of mean headroom on EGTEA and 75% on SSv2; on EGTEA, 82 of 92 changed predictions (89%) switch to an opposite-action category. Thus, original temporal order contributes substantially to model confidence in these settings. We still define headroom conservatively as confidence beyond the best repeated frame, not a pure motion measurement.
W3: "Only 200 clips were evaluated; the sample size and dataset coverage should be expanded, including Kinetics."
We acknowledge that broader evaluation would improve generality. The analysis uses 200 clips per classifier–dataset setting, covering TSM, TimeSformer, SlowFast, and single-frame controls on EGTEA and SSv2 (Section 5, lines 357–373). Explanation results in Table 3 report mean ± standard deviation across three trials. Appendix A.13 (Table 14) supplies paired TSM comparisons, and targeted robustness experiments test the conclusions under additional controls.
We chose complementary datasets rather than treating them as interchangeable: EGTEA has separate verb/noun labels needed for the full cross-axis audit, whereas SSv2 emphasizes temporally defined actions. Their low-headroom rates differ substantially (21.0–70.5% on EGTEA versus 8.0–8.5% on SSv2; Table 2). Kinetics is a valuable extension for single-head reliance and FAM evaluation, though it cannot directly support EGTEA's verb–noun cross-axis analysis. Prior work has also demonstrated substantial static appearance reliance on Kinetics (Choi et al., NeurIPS 2019). EPIC-KITCHENS is another natural extension because it provides verb/noun labels. We will try to present preliminary results on EPIC-Kitchen by the end of the discussion period and will add both datasets in the camera-ready version.