- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper addresses the difficulty of distinguishing motion-sensitive representations from features that respond to positions correlated with motion. It constructs moving-ball stimuli with randomized starting positions, sufficient spatial coverage, and off-screen zeroing. Under these assumptions, a feature that is a fixed function of current ball position has zero expected within-video covariance with a motion-derived temporal profile. The proposed test searches for significant departures from this null. Experiments identify flagged axes and SAE features in several video encoders, obtain no detections in a DINOv2 control, and validate sensitivity in a representation with planted temporal components. Interventions on flagged SAE features have mixed effects on SSv2 classification.
Critical strengths
-
The paper isolates a meaningful confound and gives a clear population-level result under explicit assumptions. The coverage argument in Theorem 1 explains why position-only features have zero expected covariance, and Appendix D provides a useful counterexample showing why substituting Pearson correlation can fail. The discussion of false negatives and the inability to identify a particular temporal concept in Section 3.5 and Appendix E is also valuable.
-
The empirical investigation includes several useful controls and reports mixed intervention results openly. DINOv2, planted-subspace validation, multiple video encoders, and layer comparisons examine different aspects of the proposed test. The mixed SSv2 outcomes are informative, even though they do not establish a general causal role for the detected directions.
Critical weaknesses
-
Rejecting the stated null does not establish that a feature requires information from multiple frames. Definition 1 restricts static features to a shared function of position and explicitly excludes frame-index dependence. A readout available from a single frame plus its index can therefore be flagged. For example, let every video use the same nonconstant profile with , which Section 3.2 allows, and let . Coverage makes the visibility probability constant, giving , although the readout uses no cross-frame visual information. This does not contradict Theorem 1; it shows that its null is narrower than the paper's motivating definition of a temporal feature. DINOv2 does not exclude this alternative because it lacks a video model's frame-index input. The central interpretation needs either a stronger guarantee or an explicit restriction to rejecting position-only explanations, together with appropriate controls for index-dependent readouts.
-
The empirical specificity claim needs a clearer statistical scope. Section 4 and Table 1 describe a threshold of for each of five concepts, with a direction flagged if any concept is significant. This controls a family of tests for one concept, but does not by itself provide 5% family-wise control for the union of tests; a direct Bonferroni correction would use . The zero-mean theorem also does not imply literally zero finite-sample false positives. The paper should distinguish its population-level result from the statistical error guarantee of the finite-sample procedure, specify the intended testing family, and explain the assumptions supporting the p-value calibration. These distinctions matter because specificity is the paper's central claim.
Questions for authors
- How does the proposed guarantee exclude features that use current-frame content and frame index, without cross-frame visual information? If these are intentionally outside the static null, how should the abstract's multi-frame interpretation be narrowed? Are there existing controls for this alternative?
- Is the intended 5% error guarantee per concept or across all five concepts? What happens to the reported counts under a correction covering the complete stated testing family, and what assumptions justify the finite-sample p-values?
Reviewer Poua,
Thank you for your review. We agree with both mentioned weaknesses and believe both can be resolved; we sketch the resolutions below.
Weakness 1: frame-index readouts. You are correct. Definition 1 excludes frame-index dependence, so Theorem 1 shows a feature is not a function of current position alone, not that it uses meaningful cross-frame information beyond frame indices. Your example is valid: when all videos share one profile, a(t) is a fixed function of t, so 1visiblea(t) can be computed from the current frame and its index alone, and Theorem 1 cannot distinguish such a readout from one that computes acceleration across frames. Rather than narrowing the claim, we plan to strengthen the guarantee:
For controls, we can consider a frame-local only negative control (TimeSformer’s layer 0, which carries only current-frame content and time embeddings since no attention has occurred yet), and a frame-index channel in the synthetic control.
Weakness 2: statistical scope. Agreed. We would define the family as all 5D tests per (model, layer, dictionary) and threshold every column at 0.05/(5D). The "Any" column then remains the union of per-concept columns. We would also make explicit that Theorem 1 is a population-level statement (zero expected covariance under the null), while the finite-sample procedure controls the family-wise error rate via Bonferroni, so false positives are controlled rather than impossible. Then on p-values, we agree that the accuracy of the t-test’s p-values in the extreme tails used by Bonferroni is not guaranteed. We would state this explicitly and report the distribution of t-statistics, and show how flagged counts change under stricter thresholds.
These changes require rerunning our main analyses, so the revision would be substantial. We are withdrawing the paper to incorporate these changes in a future submission. Thank you again for your review.