acceptodds
Withdrawn from ICLR 2027

StateVid: Video Representation as an Evolving Visual State

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video encoders usually represent a video as a sequence of spatial features produced from individual frames. This frame-based representation treats each observation as a separate feature set, leaving the integration of persistent and changing visual content to later processing. We view frames as discrete observations of an evolving visual process, and its evolving state as the object of video representation. We propose StateVid, which updates this state causally at each spatial location with learned retention and writing gates. The state itself carries the video representation, and ordered snapshots expose its evolution to downstream models. The resulting visual state keeps a fixed size while accumulating evidence over time. StateVid learns these transitions from short video-text pairs through supervision of historical retention and adaptation to new content. StateVid demonstrates strong performance on video tasks compared to models trained at larger scales, while maintaining image capabilities. At the same time, it achieves better token efficiency and performance on VideoLLM. We additionally find that this representation method has stable extrapolation performance and exhibits selective attention to video content.

Primary area:
foundation or frontier models, including LLMs
Keywords:
video encoder, video understanding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_KF7fOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StateVid, a video encoder that represents videos as evolving visual states rather than frame-wise features. It introduces recurrent state updates and a training strategy using concatenated short clips with historically weighted video-text retrieval. Experiments demonstrate strong video retrieval and action recognition performance, improved VideoLLM results under fixed token budgets, and largely preserved image capabilities.

Critical strengths
  1. Interesting representation design. The paper presents an interesting approach to modeling videos through evolving spatial states, allowing temporal information to be integrated within the visual encoder. This enables processing more observations without increasing the downstream visual token budget, which is particularly useful for long-video understanding.

  2. Strong empirical validation. The method achieves competitive results across video retrieval, action recognition, and VideoLLM benchmarks while largely preserving image capabilities. The ablations and robustness experiments further support the benefits of recurrent state modeling.

Critical weaknesses
  1. Limited conceptual insight and heuristic design. While the idea of evolving visual states is interesting, the overall method relies on several manually designed components, including clip concatenation, boundary-enhanced forgetting, and exponentially decayed historical supervision. Although each component is individually motivated, the paper does not sufficiently justify why this particular combination of heuristics is necessary. The relatively modest gains from the proposed training strategy further raise questions about the insights gained beyond the effectiveness of the overall engineering recipe.

  2. Unclear benefits of the proposed formulation. Although the paper motivates evolving visual states as an alternative to frame-wise representations, the empirical evidence does not convincingly establish its advantages. The improvement on VideoLLM benchmarks is relatively modest (49.1 vs. 47.6 in Table 4). More importantly, the extensive budget sweeps in Appendix D.4 show that the gains are inconsistent across different observation and readout budgets, and StateVid does not consistently outperform simpler baselines. Given the additional encoder computation, it remains unclear whether the proposed formulation offers sufficiently compelling benefits over existing approaches.

Questions for authors
  1. Necessity of the proposed training strategy. The method combines several heuristic choices, including clip concatenation, boundary-enhanced forgetting, and exponentially decayed historical supervision. Could the authors clarify which components are essential to the effectiveness of the proposed formulation? In particular, Table 3 shows only a modest improvement from clip concatenation (43.8 to 44.4). What evidence supports the necessity of this training strategy?

  2. Benefits of the evolving state formulation. The VideoLLM improvements are relatively modest, and Appendix D.4 shows inconsistent advantages across observation and readout budgets. Under what conditions does StateVid provide clear benefits over simpler frame-wise representations? Can the authors further justify the practical advantages of the proposed formulation, considering its additional encoder computation?

  3. Isolating the contribution of state modeling. How much of the improvement comes specifically from the proposed evolving state representation, rather than the additional model capacity and temporal processing introduced by the recurrent modules? While Table 3 provides relevant ablations, a more controlled comparison would help establish the contribution of the state formulation itself.

Reviewer_cuc8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StateVid, which extends a pretrained SigLIP2 image encoder to video by inserting recurrent temporal blocks (Gated Delta Rule) into its last four layers. A "state layer" keeps a recurrent state at each spatial patch position across frames, and uses snapshots of this state serve as a representation of the video. For VLMs, a fixed number of snapshots is passed to the language model regardless of how many frames were observed. The encoder is trained on 2M recaptioned InternVid videos plus image data. They also propose the use of stitching unrelated clips together during training with strong state decay applied at each distinct clip boundary, as opposed to a full state reset. The paper reports zero-shot retrieval and action recognition, zero-shot image classification, and VLM results with Qwen2.5-1.5B at a fixed visual-token budget.

Critical strengths
  1. Strong zero-shot video results: StateVid-So400M beats PE-core-L on all six retrieval metrics and gives the best UCF101/HMDB51 results in its size group (Table 1), with clear gains over the SigLIP2 initialization.

  2. Image ability is preserved. Zero-shot image classification stays within 0.1–0.2 points of the original SigLIP2 backbones across 11 benchmarks (Table 2), so images and videos can share one encoder.

Critical weaknesses
  1. Limited novelty. Keeping a recurrent hidden state at each spatial position and reading it out as the representation is conceptually close to ConvLSTM/ConvGRU-style video encoders and to existing temporal adapters on image ViTs (ST-Adapter, AIM, EVL). The difference is simply using a linear-RNN to model this transition.

  2. The stitched-clip training with boundary decay is under-supported. Training on concatenations of unrelated clips with boosted decay at clip boundaries is presented as a main contribution, but the evidence for it is thin.

  • The only direct ablation (Table 3) shows concatenation adds +0.6 average R@1 and lowers MSR-VTT.

  • In Fig. 5a, the full recipe is only about 0.6 R@1 above a hard state reset at each boundary ("Cut").

  • The boundary decay factor is fixed at 3 and never swept.

More fundamentally, the motivation (a lack of long-video annotations) does not explain the choice of stitching unrelated clips. The training streams are at most 480 frames at 2 fps (~4 min), shorter than the length of a typical source video in InternVid itself (~6 mins). Why not train on consecutive clips from the source videos themselves, or on existing long-video datasets, such as HowTo100M (6.5 min), HD-VILA-100M, Panda-70M (whose clips are split from longer videos)? I understand these datasets are much bigger than the 2M samples used to train StateVid, but a similar subset can be used as was done for InternVid in the paper. I find the stitching of unrelated clips and the introduction of ~20 artificial hard cuts per stream that do not occur in real video to be unconvincing, especially given the marginal improvements it provides.

  1. The VLM gains don't hold up at matched frames or matched compute.
  • Per the appendix, when StateVid and frame-wise SigLIP see the same frames (T=K), StateVid is clearly worse: 44.58 vs 47.30 at 32, and 44.76 vs 48.70 at 64 (Tables 5 and 7). Are the state snapshots weaker visual tokens than raw frame features? Similarly, the +1.5 in Table 4 comes only from StateVid observing 8× more frames (T=256 vs 32).

  • At K=64, StateVid's best result (48.73, T=256) ties SigLIP at T=64 (48.70). By the paper's own So400M estimates (566 vs 496 GFLOPs/frame) encoder compute is about 4.5× higher (~145 vs ~32 TFLOPs).

  • Missing baselines: frame-wise SigLIP given more frames with stronger spatial pooling at the same token budget, and a comparison at matched total compute.

This is not a critical weakness but I think should still be mentioned: the "Video Data" column in Table 1 is inconsistent. UMT/VideoMamba's "25M" counts about 10M videos plus about 15M images, while StateVid's "2M" leaves out its own ~25M images and its SigLIP2 initialization. I do find it misleading though.

Questions for authors
  1. How does SigLIP2 perform when fine-tuned on the same recaptioned 2M videos with mean pooling and no temporal modules?

  2. Why do state snapshots underperform raw SigLIP frame features at T=K? Can you compare against frame-wise SigLIP with more frames and stronger spatial pooling at matched token budget and matched total FLOPs?

  3. How sensitive are the results to κ? Can you report results with different κ, and how was ρ=0.2 chosen?

  4. How does training compare with equal-length streams of consecutive clips from the same source video (InternVid, Panda-70M, HowTo100M)? Is there an encoder-level evaluation where carried-over history or temporal order actually matters, such as temporally sensitive benchmarks (SSv2, Diving-48), long-video action recognition (Breakfast, COIN), or temporal grounding? K400, UCF101, and HMDB51 can largely be solved with only a few visible frames from scene and object appearance.

Reviewer_yqFWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper tries to go beyond frame-level video representation and add a contextualized recurrent video state. The method works by adding adaptation layers to the last few blocks a pre-trained SigLIP2 encoder, and training with image and video. The video batches are trained using a contrastive video-text loss, with the SigLIP2 text encoder as the target, and the image bates are trained with distillation from the SigLIP2 image encoder. The second phase of training aligns the model with a Qwen LLM for VQA. The method is tested on zero-shot video–text retrieval, zero-shot action recognition, and zero-shot image classification. It is also tested as the visual encoder of a VideoLLM on long-video QA.

Critical strengths

Good motivation to go beyond frame-level representation of video and make them more contextualized.

Critical weaknesses
  • Missing important baselines, there already exist video encoders, e.g., VideoMAE, VJEPA, S3D, MIL-NCE, which produced contextualized video representations.
  • Never succeed in actually demonstrating the value of the approach:
    • lots of the benchmarks do not test for temporal understanding or demonstrate the value of contextualized video state representations, e.g., image classification is an image task, and does not test video understand. Many of the action action-recognition or video-text retrieval benchmarks, such as Kinetics, can be largely solved from a single frame with an image encoder. Gains on VQA are modest if at all, and on VideoMME, just using SigLIP2 gives the best results. And tasks that that require strong short-horizon temporal information are also never tested, such as SSv2.
  • Clarity is also lacking. Very difficult to understand the details of the proposed approach. Writing can be significantly improved. On the implementation side, struggled to also find which part of the paper describes how the spatial state grid is turned into a single embedding for retrieval. How many time steps an image is repeated? Appendix C refers to a table that doesn’t exist (line 856). Lots of typos “StateVide,” “Emedding,” “Visualization Analyse”
Questions for authors

Please see weakness section above. Happy to consider author response to reconsider the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.