acceptodds
Under review as a conference paper at ICLR 2027

: Parallel Kinematic Selective State Space Scanners for Efficient Video Understanding

Abstract

Traditional video models relying on dense spatiotemporal attention suffer from quadratic computational costs. To circumvent these costs, recent approaches adapt image models for videos via Parameter-Efficient Fine-Tuning (PEFT) methods such as adapters. However, deeply inserting these modules incurs prohibitive activation memory overhead during back-propagation. Video state-space models (SSMs) with global scanning provide linear complexity but flatten video tokens into a one-dimensional sequence, mixing spatial and temporal ordering. To address these limitations, we propose Parallel Kinematic Selective State Space Scanners (). We retain a 2D vision backbone for spatial semantics and insert a single plug-and-play module with linear-complexity temporal scanning, bypassing the need for temporal attention or multi-layer adapters. We first explicitly extract kinematic priors via a Kinematic Prior Encoder, which captures inter-frame correspondences and feature changes. These priors subsequently drive SSMs to model temporal dependencies, adaptively modulating the update speeds and read-write strategies based on the input content at each time step. Instead of global scanning, we deploy parallel scanners along the temporal dimension for each spatial location, preserving spatial structures while limiting computational overhead. Extensive experiments on spatial-heavy and temporal-heavy action recognition benchmarks demonstrate that reaches 74.3% Top-1 accuracy on SSV2 and 84.1% on K400. The ViT-B/16 configuration fine-tunes for epochs and uses approximately less video-domain training compute than the compared VideoMamba-M configuration.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video understanding, Action recognition, State space models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tF21OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper works towards efficient video understanding by adapting image-pretrained Vision Transformers with SSMs to avoid global spatiotemporal attention while seemingly maintaining the training memory requirements by proposing Parallel Kinematic Selective State Space Scanners (PKS⁴), a framework that applies spatial parallelization over the inter-frame correlation obtained from the Kinematic Prior Encoder (KPE). The proposed methods seem to show competitive results against selected baselines while claiming to improve training efficiency by reducing the compute requirement by up to 10.7x compared to the VideoMamba-M800e configuration.

Critical strengths
  1. The proposed framework is relatively simple and practical for introducing temporal modelling into an image-pretrained model without excessive modification while maintaining linear-complexity temporal modeling.

  2. The reported results are competitive, and the ablations on individual components explain the design choices for the KPE and KS⁴ components.

Critical weaknesses
  1. The technical novelty of the proposed framework is not clearly established against the current literature. Independently processing patches with spatially parallelized temporal SSMs has already been explored by many existing works, such as TV3S (CVPR 2025) for video segmentation and STAC (ECCV 2026) for video reasoning segmentation. While MoMa (ICML 2025) has explored the integration of Selective SSMs into image-pretrained models for video recognition, and MotionSqueeze (ECCV 2020) has investigated extracting motion features through local inter-frame correspondences. With most of the components well covered by the literature, it is unclear what the paper produces as a contribution other than their combination, nor is it clear why the proposed combination is effective.

  2. The motivation and effectiveness of the proposed KPE is not clearly justified, as the proposed priors are essentially created with local and temporal feature correlation and differences, which are assumed to be better motion-aware transitions. While the results show the gains observed, it is unclear which has the biggest impact on performance, the proposed kinematic formulation or the introduced temporal processing.

  3. The reported training efficiency is questionable, as the claimed 10.7x reduction in training compares VideoMamba-M800e, which performs a full retraining + fine-tuning (~830 epochs), with PKS⁴, which only performs fine-tuning(~20 epochs). While both configurations benefit from image/CLIP initialization, the difference in the video domain schedule is what is seen in the reported reduction figure, making it hard to attribute the claimed efficiency directly to the proposed architecture.

  4. The performance comparison missing the relevant and more recent methods, especially a closely related work like MoMa (ICML 2025), is concerning. A more comprehensive comparison is needed to establish the current work's actual performance and the accuracy-efficiency advantage.

Questions for authors
  1. What is the key novelty of the proposed framework, given that several of the proposed components have already been explored in prior works? And what is the advantage of making use of Kinematic priors with parallel SSMs over the existing approaches?

  2. While the authors have established that the improvements arise from KPE specifically due to better motion modelling, it is unclear whether the improvements are indeed due to the correlation and variation operator or the addition of temporal processing. This needs to be clarified.

  3. Can the authors clarify how the claimed 10.7x trading-efficiency improvement is attributed to the architecture, especially with the presence of substantial differences in their training schedules?

  4. Given that there are other recent and closely related approaches, such as MoMa, the authors would have to justify the claimed performance and efficiency competitiveness of their proposed framework.

Reviewer_kd6TOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PKS4, an efficient video understanding framework that integrates a Kinematic Prior Encoder (KPE) with parallel selective state space scanners. By extracting inter-frame correlations and temporal feature variations, PKS4 enhances motion-aware temporal modeling while preserving spatial representations from a pretrained ViT backbone. Experiments on Something-Something V2 and Kinetics-400 demonstrate competitive recognition performance with reduced video-domain training computation. Although the results are promising, several concerns regarding methodological novelty and experimental validation remain.

Critical strengths
  1. The paper addresses the important challenge of efficient temporal modeling in video understanding.
  2. The combination of motion-aware priors and parallel temporal scanning provides a reasonable and computationally practical architecture.
  3. Extensive experiments and ablation studies demonstrate competitive performance and support the effectiveness of the proposed components.
Critical weaknesses
  1. Limited methodological novelty. The proposed framework primarily combines conventional inter-frame correlation, feature variation, and selective SSM scanning. The technical distinction from existing motion-enhanced video models and temporal SSM architectures is not sufficiently established.
  2. Insufficiently controlled efficiency comparisons. The claimed 10.7× training-computation reduction against VideoMamba-M800e includes its additional video pretraining stage. Differences in initialization, training schedules, and architectures make it difficult to isolate the intrinsic efficiency advantages of PKS4.
  3. Limited validation of the proposed kinematic modeling. Although the KPE improves recognition accuracy, the paper lacks quantitative analyses demonstrating that the extracted priors reliably capture meaningful motion correspondences rather than generic temporal feature differences.
  4. Incomplete evidence of generalization. Experiments focus mainly on action recognition using SSV2 and K400. The effectiveness of PKS4 for longer videos, complex temporal reasoning, or other video understanding tasks remains unclear.
Questions for authors
  1. What is the fundamental technical novelty of KPE compared with existing correlation-based motion modeling and temporal difference modules?
  2. Can the authors provide training-computation and memory comparisons against VideoMamba and representative adapter-based methods under matched initialization, training budgets, and evaluation settings?
  3. How much of the performance gain comes specifically from kinematic priors rather than the additional parameters and temporal modeling capacity? Could matched-capacity temporal convolution or attention baselines achieve similar results?
  4. Can the authors evaluate PKS4 on longer video sequences or more challenging temporal reasoning benchmarks to demonstrate its generalization and scalability?
Reviewer_jsh5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PKS⁴ (Parallel Kinematic Selective State Space Scanners), a temporal module that plugs into a pretrained image ViT so it can handle video. Rather than relying on costly space-time attention, the authors run a selective state space model (in the style of Mamba) along time at every spatial location, which keeps the cost of temporal modeling linear in the number of frames. The ViT still handles the spatial side, and the SSM only deals with time. Before the scan, a Kinematic Prior Encoder extracts simple motion cues by comparing neighboring frames, using local feature correlation and frame-to-frame differences, and passes them to the scanner. In the ablations, the scanner accounts for most of the gain, while the motion encoder adds roughly four more points on Something-Something V2. The method is evaluated on Something-Something V2 and Kinetics-400, with strong results on SSV2 and much lower training cost than video-native SSMs such as VideoMamba.

Critical strengths

The application of Mamba as a motion-guided conditioned selective space scan rather than a naive parallel scan over raw ViT features is interesting. The authors run a selective state space model (in the style of Mamba) along time at every spatial location, which keeps the cost of temporal modeling linear in the number of frames.

The model showcases good results on the SSV2 dataset.

Critical weaknesses

The method requires a lot more compute at inference time compared to other state space models. Table 4 is an unfair GFLOPs comparison as it does not account for number of temporal clips 4 for video Mamba vs 1 for PKS.

The module is not strictly plug and play as it claims to be especially if we need to finetune ViT as indicated in implementation details. "layer-wise learning-rate decay on the backbone".

More datasets are needed to justify the title. The paper claims "efficient video understanding", but it evaluates only short-clip action recognition on two datasets each of roughly 32 frames, Something-Something V2 and Kinetics-400, using clips of 8 to 32 frames. Check datasets LVU, Breakfast or COIN.

Based on the ablation Table 6 most of the performance gain looks to be from a SSM which is not novel the paper does not empirically provide evidence for the addition of variance and correlation module.

The paper is not well written. The authors spend 3 pages just explaining the Mamba internals and correlation and variance which is well-known and not their contribution. They could have focused more on their novel contribution.

Questions for authors

Table 4 compares total GFLOPs across different test setups (12 views for VideoMamba vs. 3 for PKS). Per view, PKS seems to cost more. Could you compare at matched views or report per-view GFLOPs and latency?

Layer-wise learning-rate decay on the backbone and ~92M trainable parameters suggest CLIP is fully fine-tuned. Is that right? If so, how is PKS plug-and-play, and how does it perform with a frozen backbone like most baselines?

The title says "efficient video understanding," but only short-clip action recognition on SSV2 and K400 is tested. Could you add a long-video benchmark (e.g., LVU, Breakfast, COIN) and show how memory and latency scale with frame count?

In Table 6, most of the gain comes from adding the SSM (44.0 → 66.1), and the motion operators add about 4 points. Would a simpler temporal module of similar cost, such as temporal attention or convolution, do as well? Do the motion operators help only the SSM, or any temporal module?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.