acceptodds
Withdrawn from ICLR 2027

GEET: Gated Explicit-motion Extraction Tempo-spatial Adapter

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Few-shot fine-grained (FS-FG) action recognition requires distinguishing visually similar actions that differ primarily in subtle temporal dynamics from only a few labeled examples. Existing foundation-model adaptations often rely on broad spatio-temporal updates, introducing excessive adaptation capacity for this setting. We propose **GEET** (**G**ated **E**xplicit-motion **E**xtraction **T**empo-spatial Adapter), a parameter-efficient adapter for a frozen CLIP backbone that prioritizes temporal specialization while limiting spatial adaptation. GEET separates appearance evolution from explicit frame-to-frame motion and combines them through gated temporal fusion, with lightweight spatial refinement grounding the explicit motion signal in locally coherent spatial evidence. This focused design requires only 3.8M tunable parameters and video input alone, GEET achieves state-of-the-art FS-FG recognition across diverse benchmarks with substantially lower latency and higher throughput, while remaining robust to diverse visual corruptions and temporal disruptions. Source code is attached in the suppl. material.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video action recognition, CLIP, Adapter, few-shot, fine-grained action

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_34rDOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new adaptation method for the few-shot fine-grained (FS-FG) action recognition task. Its core innovation lies in modeling temporal and spatial dimensions separately and integrating them using an additional gating mechanism. Experimental results demonstrate that the proposed method achieves state-of-the-art (SOTA) performance across multiple fine-grained tasks.

Critical strengths

(1) The proposed GEET achieves strong performance across multiple datasets, while the number of training parameters remains controllable.

(2) The experiments are quite comprehensive, encompassing multiple fine-grained benchmarks, various backbones, detailed efficiency analyses, and robustness evaluations.

Critical weaknesses

(1) In the motivation section, the paper describes existing methods as relying on broad spatio-temporal updates, which introduces excessive adaptation; however, it is difficult to judge whether this actually constitutes a drawback.

(2) The overall innovation of the paper is quite limited; it merely proposes a method that processes appearance and motion separately before fusing them.

(3) The overall quality of Figure 1 is very poor. The process it depicts is difficult to follow, and the design is overly elaborate and cluttered with information.

(4) The interaction between the Adapter and the CLS token requires further clarification. According to Algorithms 2 and 3 in the appendix, the Adapter only updates the patch tokens, whereas the final video representation is derived from the CLS token. Does this imply that an Adapter inserted after the final Transformer block cannot influence the final prediction results?

(5) Although the primary focus of this paper is few-shot fine-grained (FS-FG) action recognition, the experimental results demonstrate strong performance across several mainstream tasks. Given that the standard task in this field remains general few-shot action recognition, the authors should provide experimental results on datasets such as SSv2 and Kinetics to facilitate a fair comparison.

Questions for authors

The paper offers limited technical novelty, and the unclear interaction between the Adapter and CLS tokens raises concerns about the effectiveness of the proposed architecture.

Reviewer_G221OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GEET, a parameter-efficient adapter for few-shot fine-grained action recognition with a frozen CLIP visual backbone. Within a channel bottleneck, an appearance-evolution branch applies depthwise temporal convolution to the feature sequence, while an explicit-motion branch applies another temporal convolution to adjacent-frame feature differences. An input-independent learned scalar combines the branches, followed by residual spatial refinement and an up-projection. Videos are classified using support-set prototypes.

The paper reports substantial improvements over the compared CLIP baselines, with approximately 3.8M trainable parameters, lower episodic inference latency, and additional evaluations of robustness and transfer across settings. The central question is whether these results establish the specific contribution of the proposed construction beyond closely related temporal adapters and difference-based motion modules.

Critical strengths

I think the proposed results have improvements over the previous D²ST Baselines over three fine-grained benchmark datasets Diving48, FineGym99, and FineGym288.

The paper provides extensive ablation studies to verify the effectiveness of the proposed method.

Critical weaknesses

I think the proposed method has severe drawbacks:

  1. The proposed method is somehow not novel compared to existing methods. Though it utilizes pre-trained CLIP along with adapters. But the design of adapters (temporal + spatial) has been well studied in previous action recognition papers like STM [1] and ST-Adapters [2]. Also, MoLo [3] uses differential motion features for few-shot recognition. Compared to previous work, I think the proposed method has minor and incremental differences, thus it is not novel.

  2. The architecture description is inconsistent. Figure 1(c) refines AE before fusion, whereas Eq. 8 and Algorithm 3 refine the fused AE+EM representation. Algorithms 2–3 also place a patch-only adapter after the final transformer block and then read [CLS] tokens; as written, this last adapter cannot affect classification. The actual fusion order, insertion/pooling scheme, and parameter accounting need clarification.

[1]: Jiang B, Wang M M, Gan W, et al. Stm: Spatiotemporal and motion encoding for action recognition[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2019: 2000-2009.

[2]: Pan J, Lin Z, Zhu X, et al. St-adapter: Parameter-efficient image-to-video transfer learning[J]. Advances in Neural Information Processing Systems, 2022, 35: 26462-26477.

[3]: Wang X, Zhang S, Qing Z, et al. Molo: Motion-augmented long-short contrastive learning for few-shot action recognition[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023: 18011-18021.

Questions for authors

My question mostly lies in the weaknesses section. But one extra question:

  1. How does GEET compare with temporal-only ST-Adapter and DST-Adapter under the same backbone, training protocol, matching head, and parameter budget?
Reviewer_8tK9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

GEET is a small adapter that helps a frozen CLIP model recognize fine-grained actions from just a few examples. These actions look alike and differ mostly in subtle motion, like two similar dives or gymnastics moves. The adapter sits inside each layer of CLIP and works in a reduced feature space to keep it light. One branch tracks how features evolve across frames. A second branch looks at the differences between consecutive frames to capture motion. The authors test it on multiple datasets and report 4 times less latency.

Critical strengths
  1. The model is genuinely parameter-efficient. It achieves better accuracy than other methods while using relatively fewer parameters.

  2. The architecture is very simple, and the code is provided. GEET uses only depthwise temporal and spatial convolutions, a frame-difference operation, and two bottleneck projections.

  3. The method is evaluated on multiple benchmarks, and the authors clearly indicate which comparison methods were reproduced.

Critical weaknesses
  1. The introduction states that explicit motion "is applied on top of appearance evolution and spatial refinement." This contradicts the equations in the methodology section and Figure 1c, where the input features are fed directly into the explicit motion module.

  2. There is a clear discrepancy between the text, the equations, and the figure. Section 3 states that explicit motion operates "within the temporally and spatially refined representation." This cannot be true if the input features flow directly into EM rather than through spatial refinement, as shown in the equations and in the EM formula in Figure 1c.

  3. The reproduced numbers, although clearly marked as reproduced, are concerningly low, which may make the comparison unfair. For example, MPL (2026), which was reproduced, scores lower than ProtoNet (2017).

  4. The authors also did not complete the reproduction grid. If these methods were reproduced, why were they not evaluated on all benchmarks?

Questions for authors
  1. The introduction states that explicit motion "is applied on top of appearance evolution and spatial refinement." However, the equations in Section 3 and Figure 1c both show the input features going directly into the explicit motion module. Could the authors clarify which description is correct?

  2. Section 3 states that explicit motion works "within the temporally and spatially refined representation." How is this possible if the input features flow directly into EM, rather than through spatial refinement first, as shown in the figure and equations?

  3. Several reproduced baselines in Table 1 are concerningly low. For example, MPL (CVPR 2026) reaches 39.29% on Diving48 1-shot, far below ProtoNet (NeurIPS 2017) at 59.80%. Could the authors explain these results and confirm that the baselines were run correctly? Otherwise, the comparison may be unfair.

  4. The reproduced methods are not reported on all benchmarks. For example, TEAM and MPL appear only on Diving48, and Task-Adapter is missing FineGym288. If these methods were reproduced, why were they not evaluated on every benchmark in Table 1?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.