- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a benchmark for evaluating spatiotemporal explanations of video classifiers. The authors compare explanation methods using faithfulness, localization, validity, and runtime measures, and argue that confidence-based perturbation scores do not reliably reveal whether an explanation points to the actual evidence.
Critical strengths
-
The known-evidence setting is a clean way to expose weaknesses in popular explanation metrics.
-
Separating temporal localization from confidence change is useful, and the controls are more thoughtful than a simple visual plausibility comparison.
Critical weaknesses
-
The benchmark establishes that methods can locate an artificial cue, but it does not establish that they explain a classifier's meaningful video reasoning. A moving grating is deliberately easy to define and may be detected through low-level texture or frequency statistics. That is a different task from identifying the action, interaction, or temporal relation that a real violence classifier uses. The manuscript's public-safety motivation therefore reaches well beyond what the benchmark validates.
-
There is also a conceptual issue with the main evaluation story. If deletion and insertion scores can disagree with known localization, the strongest conclusion may be that these metrics lack semantic validity. Ranking explanation methods by those same metrics then becomes difficult to interpret. The real-video experiments do not solve this problem because the human evidence labels are soft and the datasets are small. At present, the paper is a valuable diagnostic study, but the claim that it provides a general benchmark for video explanations is too broad.
Questions for authors
- Do the conclusions survive several cue types, including motion, object interaction, and changes in shape, scale, and speed?
- What evidence supports transfer from the controlled cue to natural video explanations?