acceptodds
Withdrawn from ICLR 2027

Detecting Temporal Concepts in Video Transformers via a No-False-Positives Test

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video models have the architecture to support encoding temporal aspects of the input, i.e. concepts that require more than one frame to define, like object movement. However, because motion and appearance change are entangled, a feature that appears to encode temporal concepts like object movement may instead encode non-temporal concepts like spatial presence at different positions whose activations merely correlate with motion. To resolve this, we introduce the No-False-Positives (NFP) test, a statistical test with a formal guarantee of statistical specificity for identifying features encoding temporal concepts. Using specially designed stimuli, we apply this test to representations produced by VideoMAE, V-JEPA2, and TimeSformer, and observe that temporal features exist in each model. We validate the test first by showing it yields no temporal features for the DINOv2 image-encoder. Then, using a synthetic representation space, we show that detection rate increases with a direction's projection onto the temporal subspace, and that Sparse Autoencoder features test-flagged as temporal project much more of their norm onto the temporal subspace than unflagged features. Interventions on temporal SAE features yield mixed results across models, suggesting that intervening on temporal features may not outperform a non-temporal feature baseline.

Primary area:
interpretability and explainable AI
Keywords:
temporal representations, video transformers, mechanistic interpretability, sparse autoencoders, statistical testing, representation analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PouaOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses the difficulty of distinguishing motion-sensitive representations from features that respond to positions correlated with motion. It constructs moving-ball stimuli with randomized starting positions, sufficient spatial coverage, and off-screen zeroing. Under these assumptions, a feature that is a fixed function of current ball position has zero expected within-video covariance with a motion-derived temporal profile. The proposed test searches for significant departures from this null. Experiments identify flagged axes and SAE features in several video encoders, obtain no detections in a DINOv2 control, and validate sensitivity in a representation with planted temporal components. Interventions on flagged SAE features have mixed effects on SSv2 classification.

Critical strengths
  1. The paper isolates a meaningful confound and gives a clear population-level result under explicit assumptions. The coverage argument in Theorem 1 explains why position-only features have zero expected covariance, and Appendix D provides a useful counterexample showing why substituting Pearson correlation can fail. The discussion of false negatives and the inability to identify a particular temporal concept in Section 3.5 and Appendix E is also valuable.

  2. The empirical investigation includes several useful controls and reports mixed intervention results openly. DINOv2, planted-subspace validation, multiple video encoders, and layer comparisons examine different aspects of the proposed test. The mixed SSv2 outcomes are informative, even though they do not establish a general causal role for the detected directions.

Critical weaknesses
  1. Rejecting the stated null does not establish that a feature requires information from multiple frames. Definition 1 restricts static features to a shared function of position and explicitly excludes frame-index dependence. A readout available from a single frame plus its index can therefore be flagged. For example, let every video use the same nonconstant profile with τ(V,t)=a(t)\tau(V,t)=a(t), which Section 3.2 allows, and let ψ(V,t)=1visiblea(t)\psi(V,t)=\mathbf{1}_{\mathrm{visible}}a(t). Coverage makes the visibility probability qq constant, giving E[C(V)]=qVar⁡t(a)>0\mathbb{E}[C(V)]=q\operatorname{Var}_t(a)>0, although the readout uses no cross-frame visual information. This does not contradict Theorem 1; it shows that its null is narrower than the paper's motivating definition of a temporal feature. DINOv2 does not exclude this alternative because it lacks a video model's frame-index input. The central interpretation needs either a stronger guarantee or an explicit restriction to rejecting position-only explanations, together with appropriate controls for index-dependent readouts.

  2. The empirical specificity claim needs a clearer statistical scope. Section 4 and Table 1 describe a threshold of p<0.05/Dp<0.05/D for each of five concepts, with a direction flagged if any concept is significant. This controls a family of DD tests for one concept, but does not by itself provide 5% family-wise control for the union of 5D5D tests; a direct Bonferroni correction would use 0.05/(5D)0.05/(5D). The zero-mean theorem also does not imply literally zero finite-sample false positives. The paper should distinguish its population-level result from the statistical error guarantee of the finite-sample procedure, specify the intended testing family, and explain the assumptions supporting the p-value calibration. These distinctions matter because specificity is the paper's central claim.

Questions for authors
  1. How does the proposed guarantee exclude features that use current-frame content and frame index, without cross-frame visual information? If these are intentionally outside the static null, how should the abstract's multi-frame interpretation be narrowed? Are there existing controls for this alternative?
  2. Is the intended 5% error guarantee per concept or across all five concepts? What happens to the reported counts under a correction covering the complete stated testing family, and what assumptions justify the finite-sample p-values?
1 comment

Reviewer Poua,

Thank you for your review. We agree with both mentioned weaknesses and believe both can be resolved; we sketch the resolutions below.

Weakness 1: frame-index readouts. You are correct. Definition 1 excludes frame-index dependence, so Theorem 1 shows a feature is not a function of current position alone, not that it uses meaningful cross-frame information beyond frame indices. Your example is valid: when all videos share one profile, a(t)a(t) is a fixed function of tt, so 1visible a(t)\mathbf{1}_{\text{visible}}\,a(t) can be computed from the current frame and its index alone, and Theorem 1 cannot distinguish such a readout from one that computes acceleration across frames. Rather than narrowing the claim, we plan to strengthen the guarantee:

  1. We widen the null to “frame-local” features, ψw(V,t)=G(xt,yt,t)\psi_w(V,t) = G(x_t, y_t, t). Static features are the special case where GG ignores tt.
  2. We center each concept profile by its per-step mean over the stimulus set, τ~n(t)=τ(Vn,t)−1N∑n′=1Nτ(Vn′,t)\tilde\tau_n(t) = \tau(V_n,t) - \frac{1}{N}\sum_{n'=1}^{N}\tau(V_{n'},t).
  3. Using the same kind of proof logic, we use C~w(Vn)=1T∑t(ψw(Vn,t)−ψˉw(Vn))(τ~n(t)−τ~ˉn)\tilde C_w(V_n) = \frac{1}{T}\sum_t \big(\psi_w(V_n,t) - \bar\psi_w(V_n)\big)\big(\tilde\tau_n(t) - \bar{\tilde\tau}_n\big). After dropping the ψˉw\bar\psi_w term as before, coverage makes E[G(xt,yt,t)]=μ(t)\mathbb{E}[G(x_t,y_t,t)] = \mu(t) the same for every video, and centering gives ∑nτ~n(t)=0\sum_n \tilde\tau_n(t) = 0 at every tt (hence ∑nτ~ˉn=0\sum_n \bar{\tilde\tau}_n = 0). Therefore ∑nE[C~w(Vn)]=1T∑tμ(t)∑n(τ~n(t)−τ~ˉn)=0\sum_n \mathbb{E}[\tilde C_w(V_n)] = \frac{1}{T}\sum_t \mu(t) \sum_n \big(\tilde\tau_n(t) - \bar{\tilde\tau}_n\big) = 0. So for any frame-local feature, the expected value of the test statistic C~ˉw=1N∑nC~w(Vn)\bar{\tilde C}_w = \frac{1}{N}\sum_n \tilde C_w(V_n) is exactly zero, and a rejection implies dependence on the content of other frames, which is the paper’s motivating definition. We will remove the claim in (S2) that a single profile suffices.

For controls, we can consider a frame-local only negative control (TimeSformer’s layer 0, which carries only current-frame content and time embeddings since no attention has occurred yet), and a frame-index channel in the synthetic control.

Weakness 2: statistical scope. Agreed. We would define the family as all 5D5D tests per (model, layer, dictionary) and threshold every column at 0.05/(5D)0.05/(5D). The "Any" column then remains the union of per-concept columns. We would also make explicit that Theorem 1 is a population-level statement (zero expected covariance under the null), while the finite-sample procedure controls the family-wise error rate via Bonferroni, so false positives are controlled rather than impossible. Then on p-values, we agree that the accuracy of the t-test’s p-values in the extreme tails used by Bonferroni is not guaranteed. We would state this explicitly and report the distribution of t-statistics, and show how flagged counts change under stricter thresholds.

These changes require rerunning our main analyses, so the revision would be substantial. We are withdrawing the paper to incorporate these changes in a future submission. Thank you again for your review.

Reviewer_BzrgOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This work introduces the No-False-Positives (NFP) test, a statistical procedure that provides a formal guarantee of specificity for identifying features that encode temporal concepts. Applying the test to representations from VideoMAE, V-JEPA2, and TimeSformer, the authors find evidence of temporal features in all three models. They further validate the method by showing that it detects no temporal features in DINOv2, an image encoder. Experiments in a synthetic representation space show that the detection rate increases as a direction’s projection onto the temporal subspace grows. Overall, the paper offers a comprehensive theoretical analysis along with several insightful observations. However, its novelty over these existing approaches is not clearly established.

Critical strengths
  1. The proposed NFP test introduces a distinct theoretical perspective and is supported by a series of comprehensive theoretical analyses.

  2. The effectiveness of the proposed test has been validated across a range of video and image models.

Critical weaknesses
  1. The comparison with closely related work is insufficient. Kowal et al. (CVPR 2022; TPAMI 2025) have already investigated static and dynamic information at the feature and channel levels. VTCD (CVPR 2024) further explores spatiotemporal concept discovery in video transformers, while Joseph et al. (ICML 2026) studies motion-related representations through neuron-level analysis and interventions. These works are highly relevant to the motivation and methodology of this paper. Although the proposed NFP test offers a different theoretical perspective, its novelty over these existing approaches is not clearly established. More direct experimental comparisons and a clearer discussion of the differences are needed.

  2. Some of the claims appear broader than what the proposed method can actually support. The NFP test establishes that a feature cannot be explained solely by the current spatial position under specific assumptions. However, this does not necessarily imply that the feature encodes meaningful temporal concepts such as motion or velocity. For example, features responding to frame indices or other temporal cues could also be identified as non-static without genuinely representing motion. Although the authors acknowledge certain limitations, the distinction between detecting non-static features and identifying meaningful temporal concepts remains insufficiently addressed.

  3. The intervention experiments provide limited evidence for the practical importance of the identified temporal features. As shown in Table 4, removing these features does not consistently cause a larger performance drop than removing non-identified features. In particular, V-JEPA2 is more affected by removing low-static features than the identified temporal features. Moreover, the intervention controls are not sufficiently matched in terms of feature activation strength or the effective dimensions of the removed subspaces. These results make it difficult to determine whether the identified features play a meaningful role in temporal understanding. Stronger and more carefully controlled intervention experiments would help support the conclusions.

  4. Regarding downstream tasks, how does this approach enhance temporal understanding and overall performance? Could cluttered backgrounds or other fine-grained visual details cause the proposed NFP test to fail? It would also be helpful to include additional qualitative visual comparisons against competing methods. More broadly, the paper lacks comprehensive validation on tasks where temporal information is crucial. Overall, I find this work promising and potentially suitable for ICLR. I would consider raising my score if the authors could provide a more in-depth discussion of these aspects.

Minor Issues,

  1. τ in Definition 2 is used for various temporal properties. Although it is only an example, it may lead to confusion.

Reference:

[1] Kowal et al., "A Deeper Dive Into What Deep Spatiotemporal Networks Encode: Quantifying Static vs. Dynamic Information." CVPR 2022. https://openaccess.thecvf.com/content/CVPR2022/html/Kowal_A_Deeper_Dive_Into_What_Deep_Spatiotemporal_Networks_Encode_Quantifying_CVPR_2022_paper.html

[2] Kowal et al., "Quantifying and Learning Static vs. Dynamic Information in Deep Spatiotemporal Networks." IEEE TPAMI 2025.

[3] Kowal et al., "Understanding Video Transformers via Universal Concept Discovery." CVPR 2024. https://arxiv.org/abs/2401.10831

[4] Joseph et al., "Interpreting Physics in Video World Models." ICML 2026. https://proceedings.mlr.press/v306/joseph26a.html

Questions for authors

I like this work, although I don’t have direct research experience in this area. Therefore, I would appreciate it if the authors could shed some light on its potential for downstream tasks.

Other questions please refer to the weakness.

1 comment

Reviewer Bzrg,

Thank you for your review and for your openness to supporting our work. We believe your concerns can be addressed, and we sketch how below.

Weakness 1: related work. We agree the positioning was brief. Kowal et al. quantify static and dynamic information by estimating mutual information on paired natural and edited videos, using frame-shuffling or optical flow jitter for static pairs and stylization change for dynamic pairs. VTCD is mainly about discovering spatiotemporal concepts inside video transformers without concept labels. Joseph et al. investigates where physics variables become accessible across model layers, and uses probing, attention distance metrics and ablations to study the nature of the physics ability emergence zone. While these methods measure, discover or decode, none provide or mention a no false positives guarantee for directions in representation space encoding temporal concepts, which is the focus of our paper. Since the focus of these papers are different, and their methods also produce different output applicable to different contexts, we think a head-to-head metric comparison would not be meaningful. The NFP test is complementary. It can certify that directions found by such methods are not explained by the current frame. We would add such an expanded discussion of related work.

Weakness 2: frame indices. Agreed, other reviewers raised a related point. Rather than narrowing the claim, we plan to strengthen the guarantee:

  1. We widen the null to “frame-local” features, ψw(V,t)=G(xt,yt,t)\psi_w(V,t) = G(x_t, y_t, t). Static features are the special case where GG ignores tt.
  2. We center each concept profile by its per-step mean over the stimulus set, τ~n(t)=τ(Vn,t)−1N∑n′=1Nτ(Vn′,t)\tilde\tau_n(t) = \tau(V_n,t) - \frac{1}{N}\sum_{n'=1}^{N}\tau(V_{n'},t).
  3. Using the same kind of proof logic, we use C~w(Vn)=1T∑t(ψw(Vn,t)−ψˉw(Vn))(τ~n(t)−τ~ˉn)\tilde C_w(V_n) = \frac{1}{T}\sum_t \big(\psi_w(V_n,t) - \bar\psi_w(V_n)\big)\big(\tilde\tau_n(t) - \bar{\tilde\tau}_n\big). After dropping the ψˉw\bar\psi_w term as before, coverage makes E[G(xt,yt,t)]=μ(t)\mathbb{E}[G(x_t,y_t,t)] = \mu(t) the same for every video, and centering gives ∑nτ~n(t)=0\sum_n \tilde\tau_n(t) = 0 at every tt (hence ∑nτ~ˉn=0\sum_n \bar{\tilde\tau}_n = 0). Therefore ∑nE[C~w(Vn)]=1T∑tμ(t)∑n(τ~n(t)−τ~ˉn)=0\sum_n \mathbb{E}[\tilde C_w(V_n)] = \frac{1}{T}\sum_t \mu(t) \sum_n \big(\tilde\tau_n(t) - \bar{\tilde\tau}_n\big) = 0. So for any frame-local feature, the expected value of the test statistic C~ˉw=1N∑nC~w(Vn)\bar{\tilde C}_w = \frac{1}{N}\sum_n \tilde C_w(V_n) is exactly zero, and a rejection implies dependence on the content of other frames, which is the paper’s motivating definition. We also remove the claim in (S2) that a single profile suffices as a result of this new proof.

Weakness 3: interventions. We agree the interventions results are mixed and generally inconclusive. We report them as such, and do not claim to have identified temporal concepts that are useful for our chosen downstream tasks. Our central observation is that erasing the identified span changes accuracy only slightly, even on opposing-direction SSv2 pairs (Appendix M.2, binary classification). We are currently doing some work to try to understand what the significance of temporal feature encoding is in video models, given the inconclusive intervention results we currently have. On intervention controls, we agree that controls matched on activation strength and effective subspace dimension would be better to have, but our central observation (small absolute drops even on opposing-direction pairs) does not depend on the comparison with controls.

Weakness 4: downstream tasks. We believe that our work can help advance the understanding of how video models should encode time and temporal concepts, not immediately in direct application to improve overall performance, but as a measurement and detection tool to aid future research. On complex backgrounds and other fine-grained visual details: the NFP guarantee requires stimuli designed to satisfy its conditions, so it does not apply to arbitrary videos as-is. We analyze such multi-confounder settings (e.g. additional objects) in Appendix A.5, which is a start into how richer stimuli can be accommodated.

Some of these changes require rerunning our main analyses, so the revision would be substantial. We are withdrawing the paper to incorporate these changes in a future submission. Thank you again for your review.

Reviewer_2EKLOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to distinguish genuinely temporal features in video transformers from features whose activations correlate with motion only because of changing spatial positions. It proposes a No-False-Positives (NFP) test based on controlled moving-ball videos, uniform sampling of starting positions, off-screen zeroing, and a coverage condition. Under the paper’s definition of a static feature, these assumptions make the expected within-video covariance between a static feature and a temporal concept equal to zero. The paper also introduces a decorrelated stimulus construction and a “no universal solvent” result, evaluates the test on VideoMAE, V-JEPA2, TimeSformer, and DINOv2, and studies SAE features and downstream interventions.

Critical strengths
  • The paper formalizes how static position cues can be mistaken for motion features and proposes a controlled stimulus design to reduce this confound.
  • Its theoretical analysis also clarifies the limits of decorrelating temporal concepts.
Critical weaknesses
  • The paper formalizes how static position cues can be mistaken for motion features and proposes a controlled stimulus design to reduce this confound. Its theoretical analysis also clarifies the limits of decorrelating temporal concepts.
  • The synthetic experiment uses random sign flips to cancel position-related covariance instead of the coverage and off-screen handling used by the proposed test. This may validate the statistic under a different construction, but it does not directly validate the theorem’s stimulus mechanism.
  • The related-work section distinguishes NFP from linear probing, but could better position it against work on static-versus-dynamic information, video concept discovery, video-model interpretability, and spatio-temporal SAEs. These are not equivalent baselines, but comparing their goals and limitations would clarify the paper’s contribution.
Questions for authors
  • Do the high detection rates persist with static-video or temporal-position controls?
  • Does the synthetic validation still hold when using the theorem’s stimulus construction without random sign flips?
1 comment

Reviewer 2EKL,

Thank you for your review. We agree with your points and sketch how we would address them.

Weakness 1, Question 1, Question 2: synthetic control. You are right that the random-sign construction validates the statistic but not the stimulus mechanism. We would replace it with fixed position-dependent channels under the theorem’s own conditions (coverage and off-screen zeroing). We would also add an ablation with coverage removed, where we expect position-only directions to be falsely flagged, demonstrating that the construction is necessary. On Question 1, we would add both controls. Since motion concepts do not vary within a static video (all frames identical), the content never changes, and any detections of temporal features would reflect frame index alone. For the temporal-position control, we would add a frame-index channel to the synthetic control.

Weakness 2: related work. We agree the positioning was brief. We provide some examples of other papers and their motivations here:

  1. Kowal et al. quantify static and dynamic information by estimating mutual information on paired natural and edited videos, using frame-shuffling or optical flow jitter for static pairs and stylization change for dynamic pairs.
  2. VTCD is mainly about discovering spatiotemporal concepts inside video transformers without concept labels.
  3. Joseph et al. investigates where physics variables become accessible across model layers, and uses probing, attention distance metrics and ablations to study the nature of the physics ability emergence zone.
  4. Pach et al. applies the SAE framework to image models and show that image-text models have steerable neurons (SAE features).
  5. Dokme & Vishwanath focus on the observation that TopK SAEs produce features with flickering activations across frames due to said feature being included in the topK in one frame and excluded in the next, i.e. on temporal coherence of activations rather than on whether features encode motion.

None of these works provide or mention a no false positives guarantee for directions in representation space encoding temporal concepts, which is the focus of our paper. The NFP test is complementary to these works. It can certify that directions found by such methods are not explained by the current frame. We would add such an expanded discussion of related work.

Some of these changes and changes prompted by other reviewers require rerunning our main analyses, so the revision would be substantial. We are withdrawing the paper to incorporate these changes in a future submission. Thank you again for your review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.