- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes StateVid, a video encoder that represents videos as evolving visual states rather than frame-wise features. It introduces recurrent state updates and a training strategy using concatenated short clips with historically weighted video-text retrieval. Experiments demonstrate strong video retrieval and action recognition performance, improved VideoLLM results under fixed token budgets, and largely preserved image capabilities.
Critical strengths
-
Interesting representation design. The paper presents an interesting approach to modeling videos through evolving spatial states, allowing temporal information to be integrated within the visual encoder. This enables processing more observations without increasing the downstream visual token budget, which is particularly useful for long-video understanding.
-
Strong empirical validation. The method achieves competitive results across video retrieval, action recognition, and VideoLLM benchmarks while largely preserving image capabilities. The ablations and robustness experiments further support the benefits of recurrent state modeling.
Critical weaknesses
-
Limited conceptual insight and heuristic design. While the idea of evolving visual states is interesting, the overall method relies on several manually designed components, including clip concatenation, boundary-enhanced forgetting, and exponentially decayed historical supervision. Although each component is individually motivated, the paper does not sufficiently justify why this particular combination of heuristics is necessary. The relatively modest gains from the proposed training strategy further raise questions about the insights gained beyond the effectiveness of the overall engineering recipe.
-
Unclear benefits of the proposed formulation. Although the paper motivates evolving visual states as an alternative to frame-wise representations, the empirical evidence does not convincingly establish its advantages. The improvement on VideoLLM benchmarks is relatively modest (49.1 vs. 47.6 in Table 4). More importantly, the extensive budget sweeps in Appendix D.4 show that the gains are inconsistent across different observation and readout budgets, and StateVid does not consistently outperform simpler baselines. Given the additional encoder computation, it remains unclear whether the proposed formulation offers sufficiently compelling benefits over existing approaches.
Questions for authors
-
Necessity of the proposed training strategy. The method combines several heuristic choices, including clip concatenation, boundary-enhanced forgetting, and exponentially decayed historical supervision. Could the authors clarify which components are essential to the effectiveness of the proposed formulation? In particular, Table 3 shows only a modest improvement from clip concatenation (43.8 to 44.4). What evidence supports the necessity of this training strategy?
-
Benefits of the evolving state formulation. The VideoLLM improvements are relatively modest, and Appendix D.4 shows inconsistent advantages across observation and readout budgets. Under what conditions does StateVid provide clear benefits over simpler frame-wise representations? Can the authors further justify the practical advantages of the proposed formulation, considering its additional encoder computation?
-
Isolating the contribution of state modeling. How much of the improvement comes specifically from the proposed evolving state representation, rather than the additional model capacity and temporal processing introduced by the recurrent modules? While Table 3 provides relevant ablations, a more controlled comparison would help establish the contribution of the state formulation itself.