acceptodds
Withdrawn from ICLR 2027

AQUA: Adaptive Querying and Alignment for Invariant Prototype Learning in Few-shot Action Recognition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Few-shot action recognition (FSAR) aims to recognize novel actions from only a few labeled videos. However, existing methods often struggle to construct reliable and class-consistent prototypes from noisy support sets. We propose AQUA, an Adaptive QUerying and Alignment framework for invariant prototype learning in FSAR. AQUA introduces Invariant Prototype Querying (InvQ), in which learnable queries selectively aggregate class-consistent information across support examples via commonality-biased cross-attention, thereby suppressing sample-level variations. To semantically regularize the raw visual representations, Commonality Alignment (CMAlign) provides an auxiliary training objective that aligns the support representations from InvQ and query representations with class-wise textual anchors. Furthermore, Commonality Adaptation (CMAdapt) dynamically refines textual prototypes through consistency-gated visual conditioning, allowing reliable visual evidence to complement textual semantics. Extensive experiments demonstrate that AQUA consistently improves recognition performance across diverse few-shot action recognition settings, highlighting the effectiveness and generalizability of the proposed framework.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Computer Vision, Action Recognition, Few-shot Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_M4zMOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AQUA (Adaptive QUerying and Alignment), a prototype-learning framework for few-shot action recognition. The method contains three key components: InvQ, CMAlign and CMAdapt. Extensive experiments on four benchmarks (HMDB51, Kinetics, UCF101, and SSv2) with three different backbones (ResNet-50, CLIP-RN50, CLIP-ViT-B) and the ablation studies confirm the individual contribution of each component, and the detailed implementation specifications along with the reproducibility statement provide sufficient information for replication.

Critical strengths
  1. The proposed framework is well-structured, with three complementary components that address different aspects of few-shot action recognition in a progressive manner. Specifically, InvQ extracts class-consistent visual representations from support videos, CMAlign introduces semantic regularization to improve their discriminability, and CMAdapt further refines textual prototypes based on visual-textual consistency. This progressive design provides a coherent motivation for the overall framework.

  2. The prototype construction directly addresses sample-specific variations in limited support data by explicitly guiding attention aggregation with cross-shot commonality scores. Support-only prototypes can also be reused across queries within an episode.

Critical weaknesses
  1. Performance on SSv2 under the 1-shot setting remains below leading baselines, including CLIP-FSAR, indicating limited competitiveness on actions requiring fine-grained temporal discrimination.

  2. The frame similarity in Eq. 4 may assign high commonality scores to shared backgrounds, objects, or scenes. Although CMAlign provides class-level semantic supervision, it does not explicitly distinguish action evidence from shared contextual cues, leaving the claimed invariance insufficiently validated.

Questions for authors
  1. Why do the authors choose LoRA over adapters? Temporal adapters could enable finer-grained temporal modeling and potentially improve performance on SSv2. Could the authors compare the gains from LoRA and temporal adapters under matched trainable parameter counts and computational budgets?

  2. What accounts for the performance gap to CLIP-FSAR on SSv2 1-shot, and how does InvQ help when cross-shot commonality is unavailable?

Reviewer_i8pQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AQUA constructs support-derived prototypes for few-shot action recognition using both video features and video captions. InvQ uses learned queries to aggregate support frames with a cross-shot commonality bias, which is disabled for one-shot episodes. During training, CMAlign aligns support prototypes and query representations with class-name textual anchors. CMAdapt uses visual-text agreement to gate residual visual conditioning of both support-caption prototypes and query-caption representations. Captions are generated offline with Qwen3-VL-8B-Instruct. Prediction combines visual and caption-based matching scores with fixed weights. Experiments cover four datasets, three main visual backbones, 1-shot and 5-shot settings, component and modality ablations, and additional shot/way evaluations.

Critical strengths
  1. In Table 5, the full model improves over the Bi-MHM plus mean-caption baseline in all four settings: by 2.7 and 2.1 percentage points on Kinetics 1/5-shot, and 3.5 and 6.4 points on SSv2 1/5-shot. Table 4 also shows gains over the Qwen3-VL-8B caption-matching baseline, including 9.6 points on SSv2 1-shot.

  2. InvQ builds class representations independently of the current query video, making the construction reusable across queries. Figure 5 compares the complete InvQ module with mean aggregation on Kinetics: the gap increases from 0.2 points in 1-shot, where the bias is disabled, to 1.2 points in 5-shot. Together with the OTAM, TRX, and Bi-MHM integrations in Table 2, this provides promising evidence that the framework can serve as a reusable prototype-learning component rather than relying on one matching rule.

Critical weaknesses
  1. Table 5 adds whole modules, and Figure 5 compares InvQ with mean aggregation. Neither isolates the cross-shot bias from learned-query attention and added model capacity, or the consistency gate from visual residual conditioning. Figure 3 offers a qualitative attention visualization, but also evaluates the complete InvQ module. Learned-seed attention pooling already appears in Set Transformer (Lee et al., 2019, Section 3.2); this does not make the full AQUA design identical, but makes the incremental role of its bias especially important. A multi-shot comparison with gamma=0 while retaining the same attention module, and a fixed-gate CMAdapt control, would directly test the central claims. Without them, the full-system gains establish utility more clearly than the specific benefits of cross-shot bias and adaptive gating.

  2. Table 1 omits relevant methods such as TS-FSAR (Long et al., AAAI 2026). Its public author version, Table 1 reports 60.5/70.3 on SSv2-Small, versus AQUA's 48.6/65.1 with adapted CLIP ViT-B/16. These cross-paper results use different adaptation and training procedures, so they do not establish a controlled performance gap. They do show why a clearer comparison of the additional knowledge or capability AQUA provides would matter to its significance assessment. Both methods use textual supervision during training; the cited TS-FSAR formulation does not establish a requirement for unseen-class text at test time. AQUA's 8B caption dependency also warrants reporting one-time preparation cost separately from recognition with cached captions.

Questions for authors
  1. In a 5-shot setting, how much does Equation (4)'s bias improve over the identical learned-query attention and Transformer with gamma=0, holding captions, backbone adaptation, and training fixed? Figure 5 already establishes an advantage over mean aggregation; this narrower control would determine the additional value of commonality scoring over ordinary learned-query pooling.

  2. With the same visual projection and eta=0.1, how does Equation (12) compare with a constant gate? Section 4.2 states that the gate saturates on Kinetics but varies on SSv2; please provide the corresponding gate statistics or existing supporting measurements. This would distinguish the benefit of visual conditioning from the benefit of adapting its strength.

  3. For Table 3, are single-modality variants retrained, and is visual conditioning disabled in the text-only variant? Since CMAdapt normally uses visual features in its textual branch, these definitions matter for interpreting the modality comparison. Appendix A already provides the general captioning and backbone-adaptation settings; this question concerns the branch-specific changes made for the modality ablations.

  4. How does InvQ distinguish shared action evidence from shared background or object cues that do not discriminate the episode's classes? Equation (4) rewards within-class cross-shot similarity, but recurrence alone does not establish action relevance. CMAlign provides semantic supervision during training, and Figure 3 offers a positive example; could you provide a focused analysis of episodes with similar scene context across different actions, showing which frames InvQ attends to and whether it still improves recognition?

  5. If a generated caption misidentifies the action while the visual evidence remains informative, how does AQUA handle this disagreement? Equation (12) applies weaker visual conditioning at low visual-text agreement, preserving more of the initial caption representation. Could you show representative cases, with gate values and final predictions, to clarify whether the overall model can recover from an unreliable caption?

Reviewer_iS8rOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AQUA constructs support-derived prototypes for few-shot action recognition. Learnable queries aggregate support features with a cross-shot similarity bias, class-name text anchors supervise the visual representations, and visual–textual agreement controls the adaptation of caption-based representations. The captions are generated by a frozen Qwen3-VL-8B model. Experiments cover four benchmarks, several backbones, and different matching methods. The component ablations show improvements over a baseline that already combines visual matching with caption prototypes.

Critical strengths
  1. The most useful comparison is Table 5, where the baseline already has access to the generated captions. The full model improves Kinetics 1-/5-shot accuracy and SSv2. This shows that the gains cannot be attributed entirely to the captioner.

  2. The framework also works with several backbones and matching methods. Constructing the class representation from the support set makes the approach reasonably easy to incorporate into existing FSAR systems.

Critical weaknesses
  1. The contribution of cross-shot commonality is not isolated. Adding InvQ in Table 5 introduces learnable-query aggregation, attention-based refinement, and the commonality bias together. Appendix B.2 compares this combination with mean aggregation. Neither comparison identifies the benefit of the bias itself. This is important because the paper’s main claim concerns extracting invariant action evidence. Cross-shot similarity can also select shared backgrounds or objects, and the attention visualizations do not distinguish these possibilities. The 1-shot improvements provide evidence for the remaining components, since the commonality bias is disabled in that setting.

  2. The comparison with existing caption-assisted methods needs tighter control. Table 5 establishes improvements within AQUA’s own setup. The advantage over prior multimodal prototype-learning methods is harder to interpret because captioners, backbone pretraining, and adaptation settings differ. In particular, Table 4 does not clearly identify which baselines were rerun with AQUA’s captions and adaptation protocol. Without that information, the paper leaves open whether a simpler existing design would obtain similar gains from the same inputs.

Questions for authors
  1. For the 5-shot setting, what happens when γ in Eq. (5) is set to zero while keeping the queries, attention layers, captions, and backbone adaptation unchanged? A comparison on one of the existing benchmarks would directly test the central mechanism.

  2. Which Table 4 results use the same generated captions, backbone, and adaptation settings as AQUA? Is there a matched comparison with an existing caption-assisted method?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.