acceptodds
Under review as a conference paper at ICLR 2027

Actions Speak Louder Than Frames: Quantifying Static and Temporal Reliance In Activity Recognition

Abstract

Video action recognition models can base their predictions on two qualitatively different sources of evidence: the visual appearance present within individual frames and the temporal dynamics that unfold across frames. Existing explanation methods typically conflate these sources within a single spatiotemporal attribution, making it difficult to determine and quantify how much of the prediction is supported by static appearance, motion, or both. To distinguish genuine temporal evidence from confidence already recoverable from static frames, we introduce Motion Headroom, which also provides a criterion for when temporal attribution is identifiable. We then introduce Reliance Probes that measure how removing appearance or temporal variation affects noun and verb predictions, and use these probes to define a Static-Bias Index (SBI) for auditing and quantifying static reliance in trained video recognition models. To localize this reliance, we use FAM (Factored Appearance and Motion), a post-hoc framework that independently optimizes appearance and motion explanations.Across EGTEA Gaze+ and Something-Something V2, the audit reveals greater static reliance on EGTEA and stronger temporal reliance on Something-Something V2 across multiple architectures. We further define an explanation-conditioned SBI that measures how much of the model's temporal reliance is captured by the temporal evidence selected by FAM. Finally, we introduce an axis-specific evaluation protocol that evaluates appearance and motion explanations against the prediction component each axis is intended to explain.

Primary area:
interpretability and explainable AI
Keywords:
Video Action Recognition, Interpretablity, Static Bias

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kNafOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Addressing the critical question of whether video action recognition models rely on static appearance information or temporal dynamic information, this paper proposes an analytical framework that holds significant research value. The authors conducted experimental evaluations across two benchmarks and multiple recognition architectures, with the results largely confirming the effectiveness of the proposed approach.

Critical strengths

(1) The core motivation of this paper addresses practical challenges encountered in existing action recognition methods, making it of significant research value.

(2) The evaluation framework in the experiments is relatively comprehensive, and the authors provide extensive ablation studies, which effectively validate the efficacy of the proposed method.

Critical weaknesses

(1) The paper does not clearly distinguish between action semantics, motion, and temporal information. In mainstream approaches, action recognition typically relies on the combined contribution of motion, temporal, and spatial information. Furthermore, there is no strict correspondence between motion-verb and appearance-noun. Although the authors are aware of the cross-axis relationships, further clarification is needed in the paper.

(2) The paper uses the difference between the prediction confidence for the original video and that for a video consisting of a single repeated frame as a metric for motion headroom. However, since changes in appearance features alone can also result in a high value, this does not prove that the model has utilized motion or temporal information.

(3) In the experimental section, the validation is based on only 200 clips drawn from the EGTEA Gaze+ and SSV2 datasets, which is insufficient. The authors should first expand the sample size; furthermore, they should conduct additional validation using more standard activity recognition datasets, such as Kinetics.

(4) The effectiveness of certain modules has not been fully verified through experiments. For instance, the performance advantages of FAM may be influenced by factors such as optimization objectives or mask settings; furthermore, whether FAM successfully identifies discriminative temporal evidence needs further investigation.

Questions for authors

Can the authors further validate the reliability of motion headroom and the effectiveness of FAM?

4 comments
AuthorsOpenReview ↗

Response to reviewer kNaf

Response to Reviewer kNaf [Part 1/2]

We thank the reviewer for recognizing the research value of our motivation and the extensive ablation studies. We have clarified the terminology, added temporal-order controls, expanded the baseline comparisons, and strengthened the validation of FAM. We address the concerns below using the revised manuscript.

W1: "The paper does not clearly distinguish action semantics, motion, and temporal information; there is no strict motion–verb or appearance–noun correspondence."

We agree and have clarified these distinctions in Section 3 (lines 187–196). Appearance refers to information within frames (objects, textures, colors, and scene context); temporal variation refers to changes across frames, including motion, interactions, and ordering. We use "motion" as shorthand for this broader temporal variation, not as a claim that our interventions isolate physical motion. Action semantics are represented by the classifier's verb/noun predictions (or action label on SSv2).

We do not assume that verbs use only motion or nouns only appearance. Our cross-axis probes evaluate each intervention on both prediction heads (Section 4, Eq. 8; Table 2). On EGTEA, verb appearance reliance is 0.72–0.78 across temporal models, while noun confidence also falls when temporal variation is removed (motion-reliance probe 0.47–0.67). We interpret the latter cautiously because the static reference is selected using the verb head (Section 5.1, lines 449–455). On SSv2, appearance reliance remains 0.79–0.93 even though motion reliance is 0.86–0.88. These cross-axis findings support the reviewer's point: recognition combines these sources. Our probes measure sensitivity to specified interventions, not an exclusive causal decomposition.

W2: "High motion headroom may come from changing appearance features, not temporal information."

We agree that headroom alone does not prove temporal-order reliance. We now state explicitly that the extra confidence may come from combining multiple static observations or from their order (Section 3.2, lines 233–238). To separate these contributions, we added TSM frame-shuffling and reversal controls (Appendix A.2, lines 738–765; Table 5, page 15).

For each clip, we average confidence over 10 shuffled frame orders, retaining the same frames but disrupting their order. With pop_o denoting original confidence, psp_s the best repeated-frame confidence, and pshufp_{\mathrm{shuf}} shuffled confidence, h=po−ps=(pshuf−ps)+(po−pshuf)=hagg+hord.h=p_o-p_s=(p_{\mathrm{shuf}}-p_s)+(p_o-p_{\mathrm{shuf}})=h_{\mathrm{agg}}+h_{\mathrm{ord}}.

Identifiable clipsMean hhhaggh_{\mathrm{agg}}hordh_{\mathrm{ord}}Order shareClips with hord≥0.05h_{\mathrm{ord}}\geq0.05
EGTEA0.3500.0700.28080%70%
SSv20.6600.1510.51077%91%

The order share is E[hord]/E[h]\mathbb{E}[h_{\mathrm{ord}}]/\mathbb{E}[h]. Reordering removes 80% of mean headroom on EGTEA and 77% on SSv2, while most identifiable clips have order headroom above our threshold. A complementary reversal control changes temporal direction while keeping the original frames and local transitions. Reversal removes 143% of mean headroom on EGTEA and 75% on SSv2; on EGTEA, 82 of 92 changed predictions (89%) switch to an opposite-action category. Thus, original temporal order contributes substantially to model confidence in these settings. We still define headroom conservatively as confidence beyond the best repeated frame, not a pure motion measurement.

W3: "Only 200 clips were evaluated; the sample size and dataset coverage should be expanded, including Kinetics."

We acknowledge that broader evaluation would improve generality. The analysis uses 200 clips per classifier–dataset setting, covering TSM, TimeSformer, SlowFast, and single-frame controls on EGTEA and SSv2 (Section 5, lines 357–373). Explanation results in Table 3 report mean ± standard deviation across three trials. Appendix A.13 (Table 14) supplies paired TSM comparisons, and targeted robustness experiments test the conclusions under additional controls.

We chose complementary datasets rather than treating them as interchangeable: EGTEA has separate verb/noun labels needed for the full cross-axis audit, whereas SSv2 emphasizes temporally defined actions. Their low-headroom rates differ substantially (21.0–70.5% on EGTEA versus 8.0–8.5% on SSv2; Table 2). Kinetics is a valuable extension for single-head reliance and FAM evaluation, though it cannot directly support EGTEA's verb–noun cross-axis analysis. Prior work has also demonstrated substantial static appearance reliance on Kinetics (Choi et al., NeurIPS 2019). EPIC-KITCHENS is another natural extension because it provides verb/noun labels. We will try to present preliminary results on EPIC-Kitchen by the end of the discussion period and will add both datasets in the camera-ready version.

AuthorsOpenReview ↗

We present below part 2 of our response:

Response to Reviewer kNaf [Part 2/2]

W4/Q: "Do the experiments adequately validate headroom and FAM? Could the gains come from the objectives or mask settings?"

We have added and surfaced several complementary controls, separating diagnostic reliability from explanation quality.

Headroom and reliance robustness.

  • Temporal-order tests: Shuffling and reversal show that much of the measured headroom is sensitive to original frame order (Appendix A.2, Table 5).
  • Negative control: The single-frame classifier has near-zero verb motion reliance (0.00 on EGTEA, 0.02 on SSv2; Table 2, page 8).
  • Threshold sensitivity: The EGTEA–SSv2 distinction persists as τh\tau_h varies from 0.01 to 0.20, a 20-fold range (Appendix A.5, Table 7).
  • Calibration: The margin- and decision-flip-based SBI variants preserve the main dataset separation (Appendix A.4, Table 6). Probability-versus-margin reliance correlations are 0.85–0.97 on EGTEA's three temporal architectures; we also report the lower SSv2 correlations (0.55–0.69) rather than assuming perfect agreement.

FAM effectiveness and mask controls.

  • Complete closest-method comparison: We added Vasu et al.'s complete DAME method for TSM (Section 5.2, Table 3). On EGTEA, FAM achieves Suf$_{hr}$ 0.940 versus 0.344 and appearance deletion 0.114 versus 0.278; on SSv2, the corresponding values are 0.989 versus 0.582 and 0.071 versus 0.117. FAM retains more frames, so these are not matched-budget comparisons.
  • Size-matched random masks: FAM improves headroom-normalized sufficiency by approximately 0.22–0.40 over random masks with identical frame counts across six settings (Appendix A.6, Table 8).
  • Clip-adaptive budgets: Permuting native mask sizes among clips, while keeping the size distribution fixed, reduces FAM sufficiency by 0.224 on EGTEA and 0.206 on SSv2 (p<10−4p<10^{-4}; Appendix A.11, Table 12). This shows that clip-specific frame counts matter, not merely the average mask size. At fixed budgets, however, FAM's AUC advantage over EP-TD is smaller (+0.025/+0.007), which we explicitly acknowledge.
  • Objective ablations: Appendix A.9 (Table 10 and Figure 9) isolates individual loss terms on 20 clips per dataset. Replacing headroom-normalized sufficiency with raw retention reduces motion necessity from 0.601 to 0.552 on EGTEA and 0.727 to 0.616 on SSv2; appearance sparsity and the insertion–deletion objective also matter. These ablations do not imply that every term is equally important.
  • Independent perturbation validation: Under held-out causal-hold replacement, FAM-selected frames have greater impact than matched unselected frames: +0.143 on EGTEA (95% CI [+0.082, +0.215]) and +0.102 on SSv2 ([+0.046, +0.157]; Appendix A.12, Table 13). Under shuffle, those selected-versus-unselected gaps are close to zero, so we do not claim that FAM uniquely localizes order-sensitive frames. Its goal is to select frames preserving confidence beyond the static reference.

These additions support both the reliability of the model-level reliance audit and the value of FAM's instance-level sufficient explanations, while making their limits explicit. We hope the clearer definitions and additional controlled evidence address the reviewer's concerns and invite reconsideration of the assessment.

AuthorsOpenReview ↗

(W3) Follow up results on a two new datasets

Additional Results on EPIC-KITCHENS-55 and Kinetics-400

As promised in our earlier responses, we have completed additional experiments on 400 clips from EPIC-KITCHENS-55 and Kinetics-400 using TSM, extending our evaluation beyond EGTEA and SSv2. These experiments directly address the reviewers' requests for broader dataset coverage, including evaluation on Kinetics.

DatasetAppearance MethodInsertion ↑Deletion ↓Temporal MethodSufficiency ↑Necessity ↑Frames Kept ↓
EPIC-K55Extremal0.538 ± 0.0070.260 ± 0.009Oracle0.983 ± 0.0030.633 ± 0.00225.38 ± 0.15%
EPIC-K55STEP0.649 ± 0.0230.312 ± 0.013Ext.-Temp0.562 ± 0.0520.758 ± 0.00551.42 ± 0.37%
EPIC-K55Unified Mask0.632 ± 0.0040.263 ± 0.019STEP-TD0.494 ± 0.0290.646 ± 0.01849.69 ± 0.66%
EPIC-K55FAM0.758 ± 0.0020.155 ± 0.046FAM0.948 ± 0.0130.721 ± 0.02630.03 ± 0.98%
Kinetics-400Extremal0.315 ± 0.0110.168 ± 0.011Oracle0.976 ± 0.0040.763 ± 0.04244.44 ± 1.76%
Kinetics-400STEP0.407 ± 0.0180.220 ± 0.007Ext.-Temp0.437 ± 0.0370.774 ± 0.03345.66 ± 1.37%
Kinetics-400Unified Mask0.375 ± 0.0140.325 ± 0.002STEP-TD0.441 ± 0.0290.696 ± 0.05549.12 ± 1.62%
Kinetics-400FAM0.617 ± 0.0120.134 ± 0.014FAM0.959 ± 0.0060.823 ± 0.00869.19 ± 0.74%

The results are encouraging and consistent with our earlier findings. FAM achieves the best appearance insertion and deletion scores on both datasets. On EPIC-K55, FAM achieves 0.948 headroom-normalized temporal sufficiency, compared with 0.562 for Ext.-Temp and 0.494 for STEP-TD, while retaining only 30.03% of frames, compared with 51.42% and 49.69%, respectively.

On Kinetics-400, FAM achieves 0.959 temporal sufficiency, compared with 0.437 for Ext.-Temp and 0.441 for STEP-TD, and also achieves the highest temporal necessity (0.823). Its sufficiency approaches that of the exhaustive oracle (0.976). However, FAM retains more frames on Kinetics (69.19%) than the baselines, indicating that compactness remains an area for improvement.

Static-Bias and Motion-Reliance Analysis

We also computed motion reliance, the Static-Bias Index (SBI), and motion headroom on both datasets, averaged across three seeds.

MetricKinetics-400EPIC-K55
Motion reliance0.2710.167
SBI0.7290.833
SBI_FAM0.6860.801
Low-headroom clips (h < 0.05)53.163.9
FAM motion necessity, identifiable clips0.6700.552
FAM motion sufficiency (raw)0.9920.991

The SBI analysis provides another interesting finding. Both datasets exhibit substantial static reliance, with SBI values of 0.729 on Kinetics-400 and 0.833 on EPIC-K55. Moreover, 53.1% of Kinetics clips and 63.9% of EPIC-K55 clips have low motion headroom, suggesting that the strongest repeated single frame already provides much of the model's confidence for a substantial portion of the data.

Together, these results extend our analysis to four action-recognition datasets and provide further evidence for both the usefulness of motion headroom as a diagnostic and FAM's effectiveness as a post-hoc explanation method.

We plan to include the EPIC-KITCHENS-55 and Kinetics-400 results for all three architectures (TSM, TimeSformer, and SlowFast) in the camera-ready version.

We thank the reviewers again for encouraging these additional experiments, which have helped us strengthen the evaluation. We would be grateful if you reconsidered your assessment in light of these new results on two additional datasets.

AuthorsOpenReview ↗

Final follow up with reviewers

As we approach the end of the discussion period, we would like to thank all three reviewers for their time and thoughtful feedback, which has helped us strengthen the paper. We have tried to address the concerns through manuscript revisions and additional experiments with new baselines and results on two new datasets.

If any concerns remain or further clarification would be helpful, please let us know. We would be happy to discuss them further.

Reviewer_DoExOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how video action classifiers rely on static appearance and temporal information. It defines motion headroom as the confidence difference between the original video and the strongest repeated-frame reference. It then introduces reliance probes and a Static-Bias Index to summarize this dependence. FAM produces separate appearance and temporal masks, using insertion–deletion optimization for appearance and headroom-normalized sufficiency for temporal selection. Experiments cover EGTEA Gaze+ and Something-Something V2 with TSM, TimeSformer, and SlowFast, together with several controls and ablations.

Critical strengths
  1. The paper addresses a useful problem. A temporal explanation can preserve the prediction simply because static appearance already gives high confidence. Comparing against a repeated-frame reference makes this problem explicit.
  2. The supplementary experiments include meaningful controls. Calibration variants, threshold sensitivity, size-matched random masks, matched-budget comparisons, and an exact subset oracle help show both the strengths and limitations of FAM.
Critical weaknesses
  1. The interpretation of motion headroom is stronger than what the definition establishes. A model can combine different static observations across frames without using their order or motion. Such a model may still have positive headroom over every repeated single frame. The criterion therefore needs an order-invariant multi-frame control before it can support the claim of identifiable motion evidence.
  2. The two perturbations do not cleanly isolate appearance and motion. Repeating a frame removes other frames’ visual content, while strong blur can also remove boundaries needed to recognize movement. The current results establish sensitivity to these interventions, but do not sufficiently validate the intended separation.
  3. The closest 2026 work is not directly evaluated. Vasu et al. already proposes factored spatio-temporal explanations, and this paper adopts its frame-replacement mechanism. A comparison with that complete method is needed to establish the benefit of independent optimization, headroom normalization, and the new objectives beyond EP and STEP.
  4. The large temporal sufficiency gains become much smaller under matched budgets. Appendix A.11 reports an AUC gain of only 0.007 over EP-TD on SSv2, and EP-TD is slightly better at six retained frames on both datasets.
  5. Cross-operator validation gives a mixed result. The selected frames remain more influential under causal replacement, but not under shuffle. On identifiable SSv2 clips, the selected-versus-unselected gap under shuffle is 0.010, with a confidence interval containing zero. This leaves unclear whether FAM localizes motion-sensitive evidence.
  6. The method does not yet give compact temporal explanations reliably. Where exact enumeration is available, the oracle retains about 28–40% of frames, while FAM retains about 52–66%. This is a considerable gap for an objective intended to find a small sufficient subset.
  7. The presentation remains abstract in the main paper. Figure 1 helps explain the audit, but there is no clear diagram connecting reference selection, the two optimization branches, perturbations, and evaluation. The examples also need side-by-side baseline and failure cases to show what the proposed factorization improves.

After rebuttal: The shuffle and reversal controls help clarify what the headroom measure captures. My remaining concern is what FAM itself localizes. Under shuffling, the selected frames still show no clear advantage over equally sized sets of unselected frames. FAM also retains more frames than DAME in the new comparison, so it is hard to judge the gain at the same budget. I keep my rating at 2.

Questions for authors
  1. examples separate aggregation of static information from genuine temporal dependence?
  2. How does FAM compare with the complete Vasu et al. (2026) method using the same classifiers, perturbation operator, and frame budgets?
  3. How should the near-zero selected-versus-unselected shuffle gap be interpreted? What evidence shows that the selected frames specifically explain motion or temporal order?
  4. How were the 200 clips selected, and what are the classifiers’ recognition accuracies?
3 comments
AuthorsOpenReview ↗

Response to reviewer DoEx

Response to Reviewer DoEx [Part 1/2]:

We thank the reviewer for the careful assessment and for recognizing the importance of distinguishing static support from temporal evidence. Your concerns motivated new temporal-order controls and an improved formulation. We address each point below.

W1/Q1: "Headroom may reflect combining static observations rather than temporal order; an order-invariant control is needed."

We agree that positive headroom alone does not prove temporal-order dependence. Section 3.2 (lines 233–238) now states this distinction explicitly. We added shuffling and reversal controls using TSM (Appendix A.2, lines 738–765; Table 5, page 15).

For each clip, we average target confidence over 10 shuffled frame orders, pshufp_{\mathrm{shuf}}. Shuffling retains the individual frames but disrupts their ordering. We decompose headroom into

h=po−ps=(pshuf−ps)+(po−pshuf)=hagg+hord.h=p_o-p_s=(p_{\mathrm{shuf}}-p_s)+(p_o-p_{\mathrm{shuf}})=h_{\mathrm{agg}}+h_{\mathrm{ord}}.

Identifiable clipsMean hhhaggh_{\mathrm{agg}}hordh_{\mathrm{ord}}Order sharehord≥0.05h_{\mathrm{ord}}\geq 0.05
EGTEA0.3500.0700.28080%70%
SSv20.6600.1510.51077%91%

The order share is E[hord]/E[h]\mathbb{E}[h_{\mathrm{ord}}]/\mathbb{E}[h]. Shuffling removes 80% and 77% of mean headroom, respectively, while 70% and 91% of identifiable clips individually retain order headroom above the threshold. Reversing the clips removes 143% of mean headroom on EGTEA and 75% on SSv2. On EGTEA, 56% of identifiable clips change their predicted verb under reversal; 82 of 92 changed predictions (89%) switch to an opposite-action category, often Open/Close or Take/Put. These controls support a substantial role for temporal order, while headroom itself remains confidence beyond the best repeated frame, not a pure measure of motion.

W2: "The interventions do not cleanly isolate appearance and motion."

We agree that these are operational perturbations, not a perfect causal separation. Section 3 (lines 187–195) defines appearance, temporal variation, and action semantics; Section 3.1 (lines 198–214) specifies the interventions. Repeating the best single frame removes temporal variation but also removes other frames' content. Gaussian blur removes fine appearance information and can affect boundaries relevant to motion.

Rather than assuming strict verb–motion and noun–appearance correspondence, our probes measure both interventions on both prediction heads (Section 4, Eq. 8; Table 2). The off-axis results make such interactions visible. The single-frame controls have near-zero verb motion reliance: 0.00 on EGTEA and 0.02 on SSv2 (Table 2, page 8). Our order controls provide additional evidence without removing any of the original frames. We therefore interpret the probes as sensitivity to specified interventions, not causal isolation of physical motion.

W3/Q2: "The complete Vasu et al. (2026) method needs to be evaluated."

We added the complete Vasu et al. method (DAME) to the evaluation (Section 5, lines 374–377; Table 3, page 10), using TSM on EGTEA and SSv2. The table below reports the values reported in the revised manuscript. We will add the results for DAME on all model-dataset configurations in the camera-ready version.

DatasetMethodApp. Ins ↑\uparrowApp. Del ↓\downarrowMot. Suf$_{hr}$ ↑\uparrowMot. Nec$_{hr}$ ↑\uparrowFrames kept
EGTEADAME0.7520.2780.3440.36237.96%
EGTEAFAM0.8230.1140.9400.70056.04%
SSv2DAME0.3310.1170.5820.66860.17%
SSv2FAM0.4960.0710.9890.74865.77%

FAM improves appearance insertion/deletion and temporal sufficiency/necessity in both settings.

W4: "Temporal gains shrink under matched budgets."

We agree and do not claim FAM is the best frame ranking at every budget. Appendix A.11 (lines 1276–1329; Table 12, page 24) reports that, at matched budgets, FAM's sufficiency AUC exceeds EP-TD by only 0.025 on EGTEA and 0.007 on SSv2; EP-TD is slightly better at k=6k=6. FAM has its clearest advantage at small budgets: for k=2k=2 (sparse explanations regime which is desirable), sufficiency is 0.390 versus 0.215 (STEP-TD) and 0.261 (EP-TD) on EGTEA, and 0.227 versus 0.156 and 0.191 on SSv2.

Importantly, FAM also chooses a clip-dependent budget without requiring a fixed kk. Randomly reassigning its native mask sizes across clips, while preserving the overall budget distribution, reduces mean sufficiency by 0.224 on EGTEA and 0.206 on SSv2 ( Table 12). The corresponding changes are only 0.021/0.001 for STEP-TD/EP-TD on EGTEA and −0.002/0.029 on SSv2; these baseline changes are not significant after multiple-comparison correction. This supports adaptive mask size as one contributor to FAM's performance, not superiority at every fixed budget.

AuthorsOpenReview ↗

We present below part 2 of our response:

Response to Reviewer DoEx [Part 2/2]:

W5/Q3: "The selected-versus-unselected shuffle gap is near zero."

We thank the reviewer for highlighting this distinction. The cross-operator results help clarify what FAM successfully localizes and what remains outside its objective. In Appendix A.12 (Table 13, page 25), we compare equal-size sets of selected and unselected frames under perturbations not used during optimization, without re-optimizing the masks.

Identifiable clipsCausal-hold gap [95% CI]Shuffle gap [95% CI]
EGTEA+0.143 [+0.082, +0.215]+0.003 [−0.027, +0.034]
SSv2+0.102 [+0.046, +0.157]+0.010 [−0.023, +0.043]

Importantly, FAM-selected frames remain more influential under the held-out causal-hold intervention, with positive gaps and confidence intervals excluding zero on both datasets. This provides evidence that FAM identifies informative temporal locations beyond its optimization operator, rather than simply exploiting the exact perturbation used during training.

The shuffle results offer a complementary insight. Their confidence intervals include zero, indicating that the selected frames are not demonstrably more sensitive to reordering than matched unselected frames. This is consistent with FAM's objective: it seeks frame subsets that preserve confidence beyond the strongest static reference, not necessarily the frames most sensitive to temporal reordering. Future work can be aimed at recovering frames most sensitive to temporal ordering using the new order-headroom described in (W1).

We therefore distinguish two questions: whether the classifier relies on temporal order, which our new order-headroom controls address (W1), and which frames preserve the additional evidence beyond a static reference, which FAM is designed to explain. The cross-operator analysis supports the latter while clarifying the limits of temporal-order localization.

W6: "FAM keeps considerably more frames than the exact oracle."

We acknowledge this limitation explicitly (Section 5.2, lines 512–521; Conclusion, lines 535–539). In the four settings where enumeration is feasible (T=8T=8), the oracle retains 28–40% of frames compared with 52–66% for FAM. Exhaustive enumeration is infeasible for SlowFast (T=40T=40). FAM finds sufficient, not necessarily minimal, masks. However, its selected locations are informative: Appendix A.6 (Table 8, page 20) reports headroom-normalized sufficiency gains of 0.22–0.40 over random subsets of identical size. Figure 7 (page 18) also illustrates a genuine failure case where FAM keeps all eight frames despite a two-frame oracle solution.

W7: "A pipeline diagram, side-by-side baselines, and failure cases are needed."

These visualizations are now included. Figure 1 (page 3) shows static-reference selection, the headroom decision, independent motion and appearance masks, and axis-specific evaluation. Figure 2 (page 8) illustrates the headroom audit. Appendix Figure 6 (page 17) compares FAM, EP-TD, and STEP-TD side by side on the same SSv2 clip, including mask selections and evaluation scores. Figure 7 (page 18) shows the failure case in which FAM's sufficient mask is substantially larger than the oracle's. These examples show both the benefits and limits of the method.

Q4: "How were the 200 clips selected, and what are the classifier accuracies?"

We use 200 clips per dataset/model setting, processing the held-out evaluation split in a fixed-seed shuffled order (Section 5, lines 357–366). Explanations target the frozen classifier's predicted class rather than the ground-truth label (Section 3, lines 180–183). We have added classifier accuracies to Appendix Table 4 (page 13):

Dataset (split)ModelVerb top-1Noun top-1Action top-1Top-5
EGTEA (split 1)TSM76.6%74.2%62.4%–
EGTEA (split 1)TimeSformer70.0%75.8%58.4%–
EGTEA (split 1)SlowFast75.4%75.3%60.9%–
SSv2 (validation)TSM––45.3%74.1%
SSv2 (validation)TimeSformer––54.8%82.4%
SSv2 (validation)SlowFast––59.8%86.1%

We hope the new order controls, complete DAME comparison, clarified limitations, and visualizations address the reviewer's main concerns and invite reconsideration of the assessment in light of the added evidence.

AuthorsOpenReview ↗

Final follow up with reviewers

As we approach the end of the discussion period, we would like to thank all three reviewers for their time and thoughtful feedback, which has helped us strengthen the paper. We have tried to address the concerns through manuscript revisions and additional experiments with new baselines and results on two new datasets.

If any concerns remain or further clarification would be helpful, please let us know. We would be happy to discuss them further.

Reviewer_L15oOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper presents a motion headroom which distinguishes between static and temporal evidence which is useful during video/activity recognition. They show that earlier methods merge both these evidence as single one and there was a need in the community to show that both motion evidence vs appearance evidence be seperated. This paper does not do any training and uses existing benchmarks to see how different existing arcitecture on two benchmarks "EGTEA and Something-Something V2".

Critical strengths

The cross-axis appearance (contribution 2 highlighted in the paper) and motion reliance probles measures how reducing appearance and temporal variation affects noun and verb predictions. This sounds pretty useful to anyone who wants to see cause and effect of different appearance vs temproal on final output.

Also they used different architecture like "SlowFast" and "TSMFormer" which is good for architecture ablation.

Authors give evidence in multiple places across paper where this would be useful (line 54 etc).

Critical weaknesses
  1. I have concern overall on the presentation of the paper. The paper introduces a lot of terms and is not easy to follow through for example abstract is not clear in what is the actual contribution. One has to go back and forth on introduction to understand it.

  2. Too many new terms i believe are introduced like SBI without explaining what they mean. Related work section could be improved. For example a tabular representation showing perturbation methods vs your method on different skills will be very easily help reader that what novel is being done in paper and how much is it useful?

  3. Finally without training it is essentially an evals paper. It should be said explicitly in heading or abstract?

  4. Number of datasets used is limited in my opinion. A quick search shows "EPIC-KITCHENS" "EGO-4D" "CaptionCook-4D" datasets. Can you tell why these datasets were not considered. Finally i dont find any motivation in the main paper of being limited to 2 given datasets only. I understand that not all of them would have labelling needed for this paper but you should do thorough to ensure that or atleast provide motivation.

Questions for authors
  1. line 55 "prediction from single static frame" if correct reflects overfitting. But is it sufficient to say that? How about activities in which only some of the frames are criticial in exact labelling then will this statement hold?
  2. In pg3 line 162 Can you show exact novelty difference with DANCE as you said that is the closest work.
  3. line 39-40 "accuracy of what on held-out split" ?
  4. line 183- what is m_t ? and line 190 what does it mean that you are replacing with value of black? A visual diagram here would be useful? I didn't find appendix of this section to be readablle. Please improve that.
5 comments
AuthorsOpenReview ↗

We thank the reviewer for the constructive feedback and have addressed their concerns through clearer presentation, stronger comparisons with prior work, additional analyses, and methodological clarifications.

Response to L15o [Part 1/2]

We thank the reviewer for recognizing the value of our cross-axis reliance probes and multi-architecture evaluation. We have clarified the abstract and contributions, added a related-work table, and improved the figures. We address each concern below.

W1/W2: "The abstract is not clear in what the actual contribution is" and "Too many new terms ... like SBI without explaining what they mean."

We agree and have rewritten the abstract to clarify the contributions and their relationships (page 1, lines 13–32).

We study whether predictions rely on frame appearance or changes across frames, and how to quantify this reliance through:

  1. Motion headroom: Measures the additional confidence from the full video over the strongest repeated single-frame reference.
  2. Reliance probes and Static-Bias Index (SBI): Measure how predictions respond when appearance or temporal variation is reduced, enabling post-hoc auditing of model reliance.
  3. Factored Appearance and Motion (FAM): Produces independently optimized spatial and temporal explanations, using headroom to determine when a temporal explanation is warranted.

The abstract follows this order, and Section 1 (lines 77–96) also introduces axis-specific evaluation.

W2: "A tabular representation showing perturbation methods vs. your method ... will very easily help the reader."

We have added this comparison as Table 1 in Section 2 (page 4, lines 162–171).

MethodPost-hocSeparate masksMotion-awarePer clip
Grad-CAM, Saliency Tubes✓✗✗✓
STEP, Extremal Perturbation✓✗✗✓
Static-bias probes (Choi et al.)✓✗✓ (dataset level)✗
DANCE (Lee et al.)✗ (retrains)Concepts, not masks✗✓
Vasu et al.✓✓ (jointly optimized)✗✓
Ours (headroom + FAM)✓✓ (independently optimized)✓✓

Our framework combines post-hoc reliance measurement with separate appearance and temporal explanations, while headroom explicitly accounts for static-frame confidence (Section 2, lines 140–175).

W3: "Without training it is essentially an evals paper. It should be said explicitly."

We agree that no new recognition classifier is trained. However, our contribution goes beyond evaluating existing models.

The abstract (lines 25–26) identifies FAM as post-hoc, and Section 3 (lines 180–196) confirms the classifier is frozen. Our methodological contributions are:

  1. A diagnostic framework: Motion headroom, reliance probes, and SBI measure how much confidence is supported by static appearance versus the full video (Sections 3.2 and 4).
  2. A post-hoc explanation method: FAM independently optimizes temporal and spatial masks for the original frozen classifier. It uses headroom-constrained sufficiency for temporal selection and an insertion–deletion objective for appearance (Sections 3.3–3.4).

We therefore contribute methods for measuring and explaining evidence used by existing classifiers, not a new architecture or improved classification accuracy.

W4: "The number of datasets is limited ... why not EPIC-KITCHENS, Ego4D, CaptainCook4D? There is no motivation for using two."

We agree that the dataset motivation needed clarification. We have now included it in both the introduction (Section 1, lines 98–101) and evaluation setup (Section 5, lines 357–363).

The datasets provide contrasting settings:

  • EGTEA Gaze+ provides separate verb and noun labels required for cross-axis probes, and its classifiers can exhibit strong static reliance.
  • Something-Something V2 (SSv2) emphasizes actions where appearance alone is often insufficient to identify the class.

Low-headroom rates confirm the contrast: 21.0–70.5% on EGTEA versus 8.0–8.5% on SSv2 (Table 2, page 8).

EPIC-KITCHENS is a natural extension because it also provides verb–noun annotations. We are conducting additional experiments on EPIC-KITCHENS and aim to include results during the discussion period if available.

Ego4D and CaptainCook4D primarily address broader tasks, including episodic memory, forecasting, and procedural error recognition, rather than our trimmed-clip classification setting. The limited availability of directly compatible pretrained classifiers is also a practical consideration.

Kinetics, raised by another reviewer, has a single action label, so the verb–noun cross-axis probe does not directly apply. Prior work has also demonstrated substantial static appearance reliance on Kinetics (Choi et al., NeurIPS 2019). We acknowledge that two datasets limit generality, we will include additional results on EPIC-KITCHENS and Kinetics in the camera-ready version for completeness.

AuthorsOpenReview ↗

We present below part 2 of our response:

Response to L15o [Part 2/2]

Q1: "Is a prediction from a single static frame sufficient to call it overfitting? What about activities where only some frames are critical?"

No. Recognition from a static frame does not establish overfitting. We have revised the introduction (lines 57–62) to describe this as reliance on static appearance rather than treating it as proof of overfitting.

Motion headroom h=po−psh=p_o-p_s compares original-clip confidence po with the strongest repeated-frame confidence ps (Section 3.2, lines 225–238):

  • If one critical frame suffices, repeating it can recover the confidence, giving h=0 (or h<0). The clip is classified as static-dominant, and FAM does not force a temporal explanation.
  • If multiple frames provide additional evidence, the full clip may achieve higher confidence, giving h>0. FAM then identifies frames that preserve this additional confidence.

Positive headroom alone does not prove temporal-order reliance: static evidence can accumulate across frames (Section 3.2, lines 235–238).

Our shuffling and reversal controls (Appendix A.2, pages 14–15, lines 738–765; Table 5) test order sensitivity. Shuffling preserves every frame but disrupts their original order, reducing mean headroom by 80% on EGTEA and 77% on SSv2. Order headroom exceeds 0.05 in 70% and 91% of identifiable clips, respectively. Thus, ordering explains most observed headroom here, though headroom does not exclusively measure temporal order.

Q2: "Show the exact novelty difference with DANCE."

We have clarified this distinction in Section 2 (pages 3–4, lines 160–175) and Table 1.

DANCE retrains a concept-bottleneck model to distinguish motion-related concepts from spatial context. FAM differs in four ways:

  1. Post-hoc applicability: FAM explains the original frozen classifier without retraining, rather than a retrained alternative.
  2. Localized evidence: FAM produces frame and continuous pixel masks rather than scores over semantic concepts.
  3. No predefined concept vocabulary: FAM does not require specifying a fixed vocabulary of semantic concepts.
  4. Temporal identifiability: Motion headroom checks whether a temporal explanation is warranted, which DANCE does not explicitly consider.

Q3: "Line 39–40: Accuracy of what on held-out split?"

This refers to top-1 classification accuracy of each frozen classifier on its held-out evaluation split, not explanation quality.

We clarified this in the introduction (lines 38–42) and report accuracies in Appendix Table 4 (page 13, lines 658–665):

Dataset (split)ModelVerb top-1Noun top-1Action top-1Top-5
EGTEA (split 1)TSM76.6%74.2%62.4%–
EGTEA (split 1)TimeSformer70.0%75.8%58.4%–
EGTEA (split 1)SlowFast75.4%75.3%60.9%–
SSv2 (validation)TSM––45.3%74.1%
SSv2 (validation)TimeSformer––54.8%82.4%
SSv2 (validation)SlowFast––59.8%86.1%

Explanations target predicted, not ground-truth, classes.

Q4: "What is mt? ... What does replacing with black mean? A visual diagram would help."

The binary entry mt∈0,1m_t \in {0,1} (Section 3.1, lines 200–208) determines whether frame t is:

  • mt=1m_t=1: Retained unchanged.
  • mt=0m_t=0: Replaced by the nearest retained frame.

Zero does not mean a black frame. Nearest-frame replacement locally freezes the video without adding black pixels.

Section 3.1 (lines 210–213) discusses zero-valued spatial perturbations as an alternative we avoid; appearance explanations instead use Gaussian blur.

The figures now clarify the process:

  • Figure 1 (page 3): Shows the complete FAM pipeline, including static-reference selection, the headroom check, independent mask optimization, and evaluation.
  • Figure 2 (page 8): Illustrates static references, headroom, and selected versus frozen frames. Green indicates retained frames, while orange indicates frozen frames.
  • Appendix Figure 3 (page 13): Shows the nearest-frame replacement operation explicitly, including the frames actually received by the classifier.

We hope the clearer presentation, explicit novelty comparisons, and additional analyses address your concerns. We would be grateful if you reconsidered your assessment in light of these revisions.

AuthorsOpenReview ↗

(W4) Follow up results on a two new datasets

Additional Results on EPIC-KITCHENS-55 and Kinetics-400

As promised in our earlier responses, we have completed additional experiments on 400 clips from EPIC-KITCHENS-55 and Kinetics-400 using TSM, extending our evaluation beyond EGTEA and SSv2. These experiments directly address the reviewers' requests for broader dataset coverage, including evaluation on Kinetics.

DatasetAppearance MethodInsertion ↑Deletion ↓Temporal MethodSufficiency ↑Necessity ↑Frames Kept ↓
EPIC-K55Extremal0.538 ± 0.0070.260 ± 0.009Oracle0.983 ± 0.0030.633 ± 0.00225.38 ± 0.15%
EPIC-K55STEP0.649 ± 0.0230.312 ± 0.013Ext.-Temp0.562 ± 0.0520.758 ± 0.00551.42 ± 0.37%
EPIC-K55Unified Mask0.632 ± 0.0040.263 ± 0.019STEP-TD0.494 ± 0.0290.646 ± 0.01849.69 ± 0.66%
EPIC-K55FAM0.758 ± 0.0020.155 ± 0.046FAM0.948 ± 0.0130.721 ± 0.02630.03 ± 0.98%
Kinetics-400Extremal0.315 ± 0.0110.168 ± 0.011Oracle0.976 ± 0.0040.763 ± 0.04244.44 ± 1.76%
Kinetics-400STEP0.407 ± 0.0180.220 ± 0.007Ext.-Temp0.437 ± 0.0370.774 ± 0.03345.66 ± 1.37%
Kinetics-400Unified Mask0.375 ± 0.0140.325 ± 0.002STEP-TD0.441 ± 0.0290.696 ± 0.05549.12 ± 1.62%
Kinetics-400FAM0.617 ± 0.0120.134 ± 0.014FAM0.959 ± 0.0060.823 ± 0.00869.19 ± 0.74%

The results are encouraging and consistent with our earlier findings. FAM achieves the best appearance insertion and deletion scores on both datasets. On EPIC-K55, FAM achieves 0.948 headroom-normalized temporal sufficiency, compared with 0.562 for Ext.-Temp and 0.494 for STEP-TD, while retaining only 30.03% of frames, compared with 51.42% and 49.69%, respectively.

On Kinetics-400, FAM achieves 0.959 temporal sufficiency, compared with 0.437 for Ext.-Temp and 0.441 for STEP-TD, and also achieves the highest temporal necessity (0.823). Its sufficiency approaches that of the exhaustive oracle (0.976). However, FAM retains more frames on Kinetics (69.19%) than the baselines, indicating that compactness remains an area for improvement.

Static-Bias and Motion-Reliance Analysis

We also computed motion reliance, the Static-Bias Index (SBI), and motion headroom on both datasets, averaged across three seeds.

MetricKinetics-400EPIC-K55
Motion reliance0.2710.167
SBI0.7290.833
SBI_FAM0.6860.801
Low-headroom clips (h < 0.05)53.163.9
FAM motion necessity, identifiable clips0.6700.552
FAM motion sufficiency (raw)0.9920.991

The SBI analysis provides another interesting finding. Both datasets exhibit substantial static reliance, with SBI values of 0.729 on Kinetics-400 and 0.833 on EPIC-K55. Moreover, 53.1% of Kinetics clips and 63.9% of EPIC-K55 clips have low motion headroom, suggesting that the strongest repeated single frame already provides much of the model's confidence for a substantial portion of the data.

Together, these results extend our analysis to four action-recognition datasets and provide further evidence for both the usefulness of motion headroom as a diagnostic and FAM's effectiveness as a post-hoc explanation method.

We plan to include the EPIC-KITCHENS-55 and Kinetics-400 results for all three architectures (TSM, TimeSformer, and SlowFast) in the camera-ready version.

We thank the reviewers again for encouraging these additional experiments, which have helped us strengthen the evaluation. We would be grateful if you reconsidered your assessment in light of these new results on two additional datasets.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Message to Reviewers/SAC/AC

Summary so far:

We sincerely thank all three reviewers for their thoughtful feedback. We are encouraged that the reviews recognized several strengths of our work:

  • Reviewer L15o described our cross-axis reliance probes as "pretty useful to anyone who wants to see cause and effect" and noted that evaluating different architectures was "good for architecture ablation."

  • Reviewer DoEx stated that "The paper addresses a useful problem" and that "The supplementary experiments include meaningful controls." The reviewer specifically recognized our calibration variants, threshold sensitivity, size-matched random masks, matched-budget comparisons, and exact subset oracle.

  • Reviewer kNaf recognized the "significant research value" of our motivation and noted that our evaluation framework is "relatively comprehensive," with "extensive ablation studies" that validate the effectiveness of the proposed method.

Building on these recognized strengths, we have substantially revised the manuscript to address the reviewers' concerns.

Key Revisions and Additional Evidence

  1. Clearer contributions and methodology. We rewrote the abstract, clarified the distinction between appearance, temporal variation, and action semantics, added a related-work comparison table, and improved the explanation of our post-hoc framework.

  2. New temporal-order validation. To address whether motion headroom reflects temporal ordering or simply information aggregated across frames, we introduced frame-shuffling and reversal controls. The results show that temporal ordering accounts for 80% of mean headroom on EGTEA and 77% on SSv2, providing stronger evidence for the role of temporal information.

  3. Stronger comparisons with prior work. We added a direct comparison with the complete method of Vasu et al. (2026). On EGTEA–TSM, FAM improves headroom-normalized sufficiency from 0.344 to 0.940 and appearance deletion AUC from 0.278 to 0.114. We also acknowledge differences in retained-frame counts.

  4. Expanded robustness evaluation. We strengthened the analysis with matched-budget comparisons, budget-permutation experiments, cross-operator validation, and additional discussion of objective ablations. Permuting FAM's clip-specific frame budgets reduces sufficiency by 0.224 on EGTEA and 0.206 on SSv2, supporting the importance of adaptive frame selection.

  5. Improved visualization and transparency. We added a complete pipeline diagram, side-by-side qualitative comparisons with baselines, failure examples, classifier accuracy results, and clearer explanations of dataset selection and methodological limitations.

  6. Additional evaluation on EPIC-KITCHENS-55 and Kinetics 400. In response to the reviewers' requests for broader dataset coverage, we completed additional experiments on EPIC-KITCHENS-55 and Kinetics-400 using TSM. These results extend our evaluation to a third dataset and provide further evidence for FAM's effectiveness.

We have also carefully distinguished what our experiments establish from what remains unresolved. In particular, FAM identifies frame subsets that preserve confidence beyond a static reference, but does not necessarily find the smallest subset or uniquely localize temporal-order dependencies.

Together, these revisions strengthen the empirical support for our central contributions: measuring static and temporal reliance in pretrained video classifiers and providing independently optimized appearance and temporal explanations without retraining the underlying models.

We hope the revised manuscript and additional experimental evidence address the reviewers' concerns and would be grateful if the reviewers could reconsider their assessments in light of these improvements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.