acceptodds
Under review as a conference paper at ICLR 2027

READ: Reading the Selective Scan for Faithful Video Attribution

Abstract

Selective state-space models are an efficient approach to long-sequence video modeling. However, their recurrent computation makes it difficult to determine which video tokens drive a prediction. Existing attribution methods often require additional backward passes or repeated model evaluations, making them substantially more expensive than the forward inference they aim to explain. In this work, we introduce READ, a parameter-free attribution method for selective state-space models that unrolls the recurrence at the position read by the classifier. READ traces the contribution accumulated in the recurrent state back to the video tokens that contribute to the final prediction. By anchoring attribution at the classifier's readout, it produces a single spatiotemporal relevance map from which frame- and region-level relevance can be obtained directly. Despite requiring only one forward pass, READ achieves competitive faithfulness and localisation compared with substantially more expensive attribution methods. Experimental results show that READ avoids the systematic relevance biases of existing methods while recovering meaningful temporal and class-specific structure across datasets and backbones. More broadly, our results show that the recurrence underlying selective state-space models is not only an efficient mechanism for inference, but also a direct source of interpretable evidence for their predictions.

Primary area:
interpretability and explainable AI
Keywords:
State Space Models, Mamba, VideoMamba, Explainable AI, Model Interpretability, Attribution, Video Understanding, Selective State Spaces

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bLiwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

READ is a training-free attribution method for VideoMamba classifiers. It unrolls the selective scan at the class-token readout, projects token contributions through a path-averaged class direction, and applies convolution correction and relevance reweighting. A streaming implementation accumulates contributions during one forward pass. Experiments on three video datasets and three VideoMamba sizes assess perturbation faithfulness, localization, sanity checks, and computational cost. The results show substantial memory savings and useful temporal localisation.

Critical strengths

The streaming implementation of the proposed READ provides a substantial practical benefit. On VideoMamba, the memory falls from 6.08 GB for fused MambaLRP to 3.47 GB. Comparing against an optimized MambaLRP implementation makes this result more convincing.

Critical weaknesses

It is unclear how the exact scan decomposition supports faithful attribution to the input video. READ fixes the input-dependent coefficients and treats the gate zz as a scaling factor, while using an approximation across blocks. Yet Table 4 shows that changing zz alone leading the prediction follows the original clip. This shows that the gate matters, but does not show that READ assigns relevance to the tokens that produced the gate.

Reducing first-frame relevance does not necessarily make an explanation more faithful. The model may rely more on the first frame. Eq.6 changes the temporal scores, but the first-frame and noise-input tests do not clearly separate model behaviour from the attribution bias. The results are also mixed: post-process produces lowers SSv2 temporal faithfulness, from 0.174 to 0.154 (Table 5).

Questions for authors

How does READ assign relevance to the video tokens that determine zz? Could the Table 4 experiment also check whether the attribution map identifies those tokens, rather than only checking which clip dominate the prediction?

Can you test whether Eq.6 removes attribution bias while preserving a real model preference for the early frames? Were the post-processing settings chosen using clips separate from the final evaluation?

Reviewer_bV2XOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a parameter-free attribution to trace the contribution accumulated in the recurrent state; thus, requiring only one forward pass, it can achieve competitive faithfulness and localisation compared with substantially more expensive attribution methods. Experiments support the effectiveness of the proposed method.

Critical strengths
  1. It is somehow reasonable for the designed path-averaged readout projection and convolution-offset correction for video attribution;
  2. Experimental results support the effectiveness of the proposed method with relatively low computational cost.
Critical weaknesses
  1. It seems like there is an inconsistency between the proposed method (with only one forward pass) and the provided streaming code (with two forward passes). Could the authors elaborate on the difference?
  2. Some questions about the training. Have the authors used reversed videos during training? Will reversing the video affect the model’s performance? Could the authors provide some visualization comparisons?
  3. The examples shown in Fig. 14-15 are interesting. Could the authors analyse more cases with object or scene changes?
  4. I will be happy to raise my score if the authors can respond to my questions accordingly.
Questions for authors

Please address the concerns in Critical Weaknesses.

Reviewer_HwoZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Readout-Anchored Decomposition, a post-hoc, training-free attribution method for 3D spatiotemporal video understanding in selective state-space models (VideoMamba). Observing that a selective scan becomes a linear recurrence once input-dependent coefficients are fixed, READ unrolls the recurrence at the CLS token to obtain exact intra-block per-token state contributions. To map these state contributions to class-specific spatiotemporal relevance maps, the authors introduce a path-averaged gradient readout direction through the final RMSNorm nonlinearity. This includes an intra-frame depthwise causal convolution tap redistribution to correct spatial-temporal offsets, and post-processing via channel truncation and cross-block cancellation weighting. Because each block's readout projection is decoupled from downstream blocks, READ supports a streaming implementation requiring only a single forward pass and depth-independent O(L) auxiliary GPU memory. Experiments show strong temporal faithfulness and procedure-step localization across Something-Something v2, Kinetics-400, and COIN while substantially mitigating first-frame scan-geometry bias.

Critical strengths

I found the streamling algorithm very efficient. The implementation only requires a single forward pass with O(L) memory. This can be extremely useful for understanding long video or on-device applications where computational resources are limited.

Critical weaknesses

Limited novelties and missing important related works and baselines. There are several recent works on understanding the attention machenism of Mamba model. It would be valuable to discuss the difference[1,2,3]

[1] Pitorro, Hugo, and Marcos Vinicius Treviso. "Latim: Measuring latent token-to-token interactions in mamba models." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025. [2] Ali, Ameen Ali, Itamar Zimerman, and Lior Wolf. "The hidden attention of mamba models." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025. [3] Mabrok, Mohamed A., and Yalda Zafari. "X-VMamba: Explainable Vision Mamba." arXiv preprint arXiv:2511.12694 (2025).

Questions for authors

My main question to the authors is regarding the novelty of the proposed method. Please refere to the above related works and baselines.

1 comment
Area_Chair_e52SOpenReview ↗

Follow-up on Remaining Concerns

Dear Reviewer HwoZ,

Thank you for your time and effort. Two other reviewers have responded positively to the authors' rebuttal, but there is still a discrepancy in the current assessments.

Could you please clarify whether the authors' responses have adequately addressed your concerns? If any concerns remain, we would appreciate it if you could elaborate on them to help facilitate further discussion and reach a more informed consensus.

Thank you for your time and consideration!

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.