acceptodds
Withdrawn from ICLR 2027

Beyond Static Cues: Motion-Aware Video Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We identify a fundamental limitation in current video retrieval models: they rely heavily on static appearance cues and inadequately capture the motion dynamics that distinguish visually similar actions. To systematically study this limitation, we introduce MAVR, a motion-centric benchmark and training dataset built largely from newly collected videos for fine-grained text-to-video and video-to-text retrieval. MAVR combines motion-defined actions, scene-matched hard negatives, and a controlled caption scheme that separates action information from scene and viewpoint cues. Despite performing strongly on a broader multimodal embedding benchmark, existing models perform poorly on MAVR, revealing a substantial gap in motion-sensitive retrieval. As a first step toward closing this gap, we introduce MAVE, a motion-aware embedding architecture that augments a multimodal embedding model with temporally informed tokens from a dedicated video pathway. When trained on MAVR, MAVE substantially outperforms state-of-the-art retrieval baselines on actions represented during training, demonstrating the benefit of explicit temporal modeling. However, these gains do not consistently generalize to unseen actions underscoring that robust motion-aware retrieval remains an open challenge.

Primary area:
datasets and benchmarks
Keywords:
Motion-Aware Video Retreival, Motion Dynamics, Video Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ChaiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors introduce MAVR, a motion-centric benchmark and training dataset containing approximately 357K video clips across 287 action classes, with scene-matched hard negatives and controlled captions designed to emphasize fine-grained motion understanding. They further propose MAVE, which integrates a pretrained temporal encoder into an MLLM-based embedding model through a Q-Former and temporal token fusion. Experiments reveal substantial performance gaps between existing models on MAVR and general multimodal retrieval benchmarks. MAVE achieves significant improvements on seen action classes but shows limited generalization to unseen actions.

Critical strengths
  1. MAVR provides a large-scale, motion-centric retrieval dataset with carefully designed scene-matched negatives and controlled captions. It addresses an important evaluation gap by reducing reliance on static appearance cues.

  2. The extensive evaluation reveals that strong performance on general multimodal retrieval benchmarks does not necessarily translate to fine-grained motion understanding. The experiments also demonstrate substantial improvements from incorporating explicit temporal representations on seen actions.

Critical weaknesses
  1. The proposed MAVE method primarily combines established components, including V-JEPA2, Q-Former, and LoRA, without introducing a substantially new temporal modeling or cross-modal alignment mechanism. This limits its technical contribution beyond the benchmark.

  2. Despite significant improvements on seen actions, MAVE fails to consistently outperform the baselines on unseen actions and sometimes performs worse. This raises concerns about whether the proposed approach learns transferable motion representations rather than adapting to the training action distribution.

Questions for authors
  1. How much of MAVE's improvement comes from genuinely modeling motion dynamics rather than simply incorporating additional pretrained video features? Could the authors provide a temporal-order perturbation experiment or a comparison with an alternative pretrained video encoder?

  2. What explains MAVE's limited performance on unseen actions? Could the authors clarify whether this is primarily caused by overfitting to the training action classes or by insufficient alignment between temporal features and text embeddings?

2 comments

Thank you for the review.

Note on scope: Our main contribution is MAVR: it has ~357K newly collected videos from paid, consenting participants, not a re-curation of public data. MAVR exposes a gap in motion-sensitive retrieval. MAVE is only a first step toward closing that gap. Its poor transfer to unseen actions is part of what MAVR is built to show.

W1. Novelty. Yes, MAVE deliberately combines established components. MAVR is the main contribution of this work; it shows:

  1. that zero-shot MLLM-based embedding models fail to retrieve motion-sensitive videos.
  2. that using the MAVR-training set improves retrieval.
  3. that the finetuned models still fail to generalize to unseen actions.

MAVE is only a first step toward closing the gap for seen actions. It shows that a separate temporal pathway improves retrieval, and the new tests below show that its gain really comes from a better modelling of motion dynamics.

Q1. Motion or additional pretrained features? We evaluated the full test set with reversed frames, shuffled frames, or one repeated middle frame. We also perturbed MAVE's branches separately. Results show seen action-only T2V Recall@1:

ModelNormalReverseShuffleStatic
Zero-shot Qwen3-VL-Emb-2B1111
2B+LoRA1911127
MAVE4527183
MAVE: temporal branch only4528183
MAVE: ViT branch only45434444

If the temporal branch only added appearance features, a static temporal input would still help, but it doesn't; with the ViT frames untouched, a static temporal input leads to a drop from 45% to 3%, below the baseline. Perturbing only the ViT frames has a negligible effect. Thus, MAVE's gain comes from encoding motion dynamics in the temporal branch.

We also added results for ViCLIP-L/14; it's a video encoder pretrained on InternVid-10M and fully fine-tuned with our MAVR recipe. Zero-shot action-only T2V is 1%, comparable to MLLM embedders; fine-tuning it lifts this metric to 52%, above MAVE's 45% result. Unseen action-only T2V remains only 9%. ViCLIP's and MAVE's results support the notion that having video-native features improves performance compared to MLLM-based embedders with image-oriented ViTs.

W2/Q2. Unseen actions: overfitting or insufficient alignment? We ran the following new analyses to find where the failure happens.

(i) Do the video features transfer? For each of the 110 unseen clips, we find its nearest other unseen clip by video embedding, skipping clips of the same person. A hit means the same class (chance is 9%).

(ii) Does text matching work? We rank unseen queries only against the 110 unseen clips (``gallery is the unseen pool only'').

(iii) The paper's unseen numbers rank each unseen query against all 2,850 test clips (``full gallery''). 96% of these clips are from actions seen during training.

ModelVideo-to-video nearest neighbor (%)action-only T2V: unseen pool (%)action-only T2V: full gallery (%)
Qwen3-VL-Emb-2B, zero-shot66265
2B+LoRA86636
8B+LoRA88615
MAVE-2B99794

Our reading:

  1. The video features do transfer. MAVE groups unseen actions better than both baselines: +13% over 2B and +11% over 8B (both p < 0.001).
  2. Text matching inside the unseen pool also works. Training on MAVR lifts it from 26 (zero-shot) to 63 (2B) and 79 (MAVE). MAVE vs 2B is not significant here with only 11 classes.
  3. The failure is in the full gallery: A clip from a trained class almost always ranks above the right unseen clip.

So the root cause is closer to insufficient alignment than overfitting: it is a bias toward the training classes when trained and new actions are mixed in one gallery. The evidence is consistent with the bias studied in generalized zero-shot learning. One way to reduce it is broader video-text pretraining of the temporal branch before MAVR training. We will add this analysis to the paper.

Ethics: privacy and copyright. All clips are first-party recordings: paid workers recorded themselves on their own webcam/phone via our collection platform in 2019-2024 and uploaded the clips themselves. Nothing was scraped or taken from video platforms.

As stated in our Ethics Statement, workers were informed of the collection's purpose and signed consent authorizing research release before recording. We removed audio and metadata (device/time/location), retained clips with at most one detected face, and discarded clips where the person appears under 19. We release no names, contact details, worker IDs or demographic labels. Three source collections were removed entirely for compliance reasons before release.

Faces remain visible because many actions involve the face/hands. MAVR will use gated access with acceptance of a research-only license prohibiting face recognition, re-identification, demographic inference, surveillance and generating participants' likenesses. The dataset page will provide a removal-request contact; requested clips will be removed in the next version.

Thank you for the detailed response and additional experiments. The temporal perturbation tests largely address my concern about whether MAVE uses motion information. However, your reply to Reviewer 8Ksd shows that 73% of test clips for seen actions feature the same people doing the same actions as in training, so I still wonder whether the performance of the model relies on familiar people or settings.

Reviewer_Bj1NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces a motion-centric benchmark with scene-matched hard negatives and disentangled captions for bidirectional text-video retrieval. Experiments show that leading models perform poorly on MAVR despite strong results on general multimodal benchmarks. To address this, the authors propose an embedding architecture with a dedicated temporal pathway for capturing motion signals. The proposed model improves performance on seen actions.

Critical strengths
  1. It identifies a critical real-world limitation: prevailing video retrieval depends heavily on static frame cues while neglecting fine-grained temporal motion dynamics.
  2. MAVR is carefully designed with scene-matched hard negatives and three disentangled caption variants.
  3. The evaluations are comprehensive, including zero-shot unseen-action tests, ablation studies, sampling-rate controls, and catastrophic-forgetting mitigation via embedding anchoring.
Critical weaknesses
  1. The gains are limited to seen actions; it does not generalize well to novel unseen motions, and the root cause of poor cross-action generalization is not fully explored.
  2. It carries strong biases: most videos are indoor, third-person, static-camera clips; action classes follow a long-tail distribution, and the zero-shot test partition is small.
  3. Computational overhead remains non-negligible: its temporal pathway requires higher-rate video sampling, and scaling up temporal encoders will further increase inference cost.
Questions for authors

Please refer to the weaknesses.

1 comment

Thank you for the review.

Note on scope: Our main contribution is MAVR: it has ~357K newly collected videos from paid, consenting participants, not a re-curation of public data. MAVR exposes a gap in motion-sensitive retrieval. MAVE is only a first step toward closing that gap; we don't claim that it’s the final solution. Its poor transfer to unseen actions is part of what MAVR is built to show.

Notations: A=action-only; A+C=action+context

W1. Unseen actions root cause: We agree that this is the main open problem. We ran the following new analyses to find where the failure happens.

(i) Do the video features transfer? For each of the 110 unseen clips, we find its nearest other unseen clip by video embedding, skipping clips of the same person. A hit means the same class (chance is 9%).

(ii) Does text matching work? We rank unseen queries only against the 110 unseen clips (``gallery is the unseen pool only'').

(iii) The paper's unseen numbers rank each unseen query against all 2,850 test clips (``full gallery''). 96% of these clips are from actions seen during training.

ModelVideo-to-video nearest neighbor (%)action-only T2V: unseen pool (%)action-only T2V: full gallery (%)
Qwen3-VL-Emb-2B, zero-shot66265
2B+LoRA86636
8B+LoRA88615
MAVE-2B99794

What this shows:

  1. The video features do transfer. MAVE groups unseen actions better than both baselines: +13% over 2B and +11% over 8B (both p < 0.001).
  2. Text matching inside the unseen pool also works. Training on MAVR lifts it from 26 (zero-shot) to 63 (2B) and 79 (MAVE). MAVE vs 2B is not significant here with only 11 classes.
  3. The failure is in the full gallery: A clip from a trained class almost always ranks above the right unseen clip. This is the known bias toward training classes in generalized zero-shot learning.

So the root cause is a bias toward the training classes when trained and new actions are mixed in one gallery. One way to reduce it is broader video-text pretraining of the temporal branch before MAVR training. MAVR makes this problem measurable, and we will add this analysis to the paper.

W2.1 Dataset biases: We agree, and the limitations section already states the skew: 95.6% indoor, 97.1% third-person, 94.3% static camera. That being said, the data is still diverse:

  • Dataset includes 7,599 participants, each recording with their own device;
  • it includes about 114K different sets of background objects;
  • a near-even split of bright and dim lighting (52% / 48%).

The skew also has an upside. It makes appearance a weak cue for retrieving a video for an action. To check if the results depend on the majority setting, we sliced the seen test set. A+C T2V R@1 (%):

Slice (clips)2B+LoRAMAVEGain
Indoor (2,468)6879+11
Outdoor (272)6675+9
Third-person (2,560)6879+11
Egocentric (180)5867+9
Static camera (2,568)6879+11
Handheld (172)6067+7

Egocentric and handheld clips are harder for every model, but MAVE's gain holds in all slices.

W2.2 Long tail distribution: The test gallery is class-balanced (10 clips per class). We split the action classes into equal-sized groups by number of training clips and show the results for each group (metric: Action-only T2V R@1 (%)):

ModelRare (<=382)MiddleFrequent (>=1,531)
2B+LoRA162221
MAVE324067

MAVE doubles the rare-class baseline performance, but its largest gain is on frequent classes, which means that the long tail affects the results. We will add these results to the limitations.

W2.3 Small zero-shot split: We found 9 more classes in the "unseen" test split. These were held out as unseen validation classes, but were never used during training. Together with the official split, this gives 20 unseen actions and 200 clips. The results are still the same: all fine-tuned models stay low in action-only R@1 on these 9 classes (0 to 10%). We will report both groups.

W3 Compute: MAVE's extra cost is only in video encoding. Text-query encoding and vector search are the same as the 2B baseline, since the temporal branch only runs on videos and the embedding size is unchanged. The temporal branch does use higher FPS, but it adds a fixed 32 tokens per clip. Also, increasing the FPS for MAVE is computationally cheaper than doing so for the 2B baseline:

Modeltraining GPU hours (8 H100)
2B+LoRA (1FPS)310
2B+LoRA (3FPS)1,690
MAVE (5FPS)540 (including warm-start)
ModelInference GFLOPs for one H100
Qwen-2B (1FPS)3,620
Qwen-2B (3fps)10,001
MAVE (5FPS)6,240

MAVE is cheaper to train than a 3FPS baseline, while it still outperforms the 3FPS baseline on seen action-only queries (T2V R@1: 45 vs 27). We agree that a larger temporal encoder would cost more. This is why we used V-JEPA2-300M and a fixed token budget, and the limitations already mention that.

Reviewer_8KsdOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MAVR, a dataset and benchmark for motion-sensitive text-to-video and video-to-text retrieval, reporting 357,195 clips across 287 action classes. Its construction combines action-only and action-plus-context captions, a scene-diverse test gallery, and scene-matched training negatives. It also proposes MAVE, which augments Qwen3-VL-Embedding with a frozen V-JEPA2 temporal encoder, Q-Former compression, and 32 temporal tokens integrated through LoRA adaptation. The main result is a substantial improvement on training-seen actions, especially action-only retrieval. The gains do not extend reliably to held-out actions, and the paper explicitly studies degradation on a broader retrieval benchmark after adaptation.

Critical strengths
  1. MAVR combines 287 action classes, a class-balanced test gallery (10 clips per class across 285 evaluated classes), separate action-only and action-plus-context captions, and scene-matched hard negatives (Section 3; Appendices B–F). Separating action descriptions from scene context makes retrieval failures much easier to diagnose, and the target-sampled recall metric (Section 5.1) prevents models from getting high scores simply by retrieving the easiest clip of an action class.

  2. On seen action-only queries, MAVE raises T2V Recall@1 from 19% to 45% and V2T Recall@1 from 45% to 67% over the 2B baseline (Table 3), and Appendix I controls for frame rate by showing that a 1-FPS MAVE variant still beats the 1-FPS baseline while the 5-FPS model outperforms a 3-FPS baseline (Tables 7–8). The authors are also candid about where the method fails, reporting both flat transfer on unseen actions (Table 3) and a ~20-point Recall@1 drop on MMEB-V3 after adaptation (Appendix J).

Critical weaknesses
  1. While Table 2 evaluates zero-shot models and Appendix J studies forgetting on MMEB-V3, the fine-tuned comparison in Table 3 only adapts Qwen3-VL-Embedding-2B/8B on MAVR. Please adapt at least one dedicated temporal video-text retrieval baseline, such as CLIP2Video [R1], ViCLIP [R2], or a retrieval variant of InternVideo2 [R3], on MAVR under matched frame sampling and training budgets (while noting backbone and pretraining differences), or testing MAVE on an additional public motion-sensitive benchmark.

  2. Section 5.2 and Appendices H, I, and K.2 describe the 8x H100 training setup, token/FPS ablations, and potential scaling bottlenecks on longer videos, but do not quantify the compute and latency trade-off against the adapted 2B/8B or 3-FPS baselines. Please report adaptation GPU-hours and FLOPs (including Stage 1 warm-start and any V-JEPA2 feature precomputation) as well as per-query inference FLOPs, peak GPU memory, and encoding throughput, separating offline video gallery indexing (which includes the V-JEPA2 + Q-Former branch) from online text/video query encoding and vector search.

References (representative options for 3, not required to run all three):

[R1] Han Fang et al. CLIP2Video: Mastering Video-Text Retrieval via Image CLIP. 2021. https://arxiv.org/abs/2106.11097

[R2] Yi Wang et al. InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation. ICLR 2024. https://arxiv.org/abs/2307.06942

[R3] Yi Wang et al. InternVideo2: Scaling Foundation Models for Multimodal Video Understanding. ECCV 2024. https://arxiv.org/abs/2403.15377

Questions for authors
  1. How many raw recordings existed for the 11 unseen action classes beyond the 110 test clips, and were they excluded from the 357,195 count in Table 4? What fraction of test clips share a participant ID or recording session with the training split, and how does seen-action Recall@1 change on a participant-disjoint test subset?

  2. If you evaluate MAVE and the adapted 2B baseline on directional action pairs with reversed or randomly shuffled frame order (or repeated middle frames), how much does action-only Recall@1 drop?

  3. Which validation split and metric trigger early stopping in Section 5.2, how many actual steps did each model train for, and why do the unanchored 2B baseline numbers in Table 9 (12/49 T2V, 32/53 V2T) differ from Table 3 (19/68 T2V, 45/68 V2T)?

1 comment

Thank you for the review.

Note on scope: Our main contribution is MAVR: it has ~357K newly collected videos from paid, consenting participants. MAVR exposes a gap in motion-sensitive retrieval. MAVE is only a first step toward closing that gap; we don't claim that it’s the final solution.

Notations: A=action-only; A+C=action+context

W1 Dedicated Video baseline: We added ViCLIP-L/14 (0.4B), zero-shot and fully fine-tuned with the same training recipe as the baselines. Its native 8 frames/clip give ~2.7/1.8fps in training/testing vs our 1-fps baselines.

Zero-shot recall@1 (R@1):

ModelT2V A/A+C
Qwen-2B1/5
Qwen-8B2/7
ViCLIP1/7

The results show that the zero-shot gap extends beyond MLLMs. The fine-tuned results below list A-seen/A+C-seen/A-unseen/A+C-unseen; * indicates significantly better than MAVE.

ModelT2VV2T
MAVE45/79/4/5567/78/11/43
ViCLIP52*/83*/9/66*71*/83*/17/53

Takeaways:

  1. The results show that MAVR-training lifts ViCLIP's action-only R@1 from 1% to ~51%.
  2. ViCLIP beats MAVE on seen classes. ViCLIP is a video-native model, and is fully fine-tuned. MAVE instead adds a video-oriented temporal pathway to an MLLM with an image-oriented ViT and adapts the backbone with LoRA. Both models' results support the notion that video-native features improve motion-sensitive retrieval compared to image-oriented MLLM embedders.
  3. Unseen action-only R@1 stays low for all models, showing that a dedicated video model alone does not solve the action-generalization problem that MAVR exposes.

W2 Compute: Training: Approximate costs on 8 H100s (FLOPs is a rough upper-bound):

ModelGPU-hoursFLOPs (x10^20)
Qwen-2B (1fps)3101.2
Qwen-2B (3fps)16903.1
Qwen-8B9403.6
MAVE5402.3

Baselines were trained for 10K steps; MAVE used 7.2K warm-up + 4.4K joint steps. V-JEPA2 features are computed on the fly, with no precomputation.

Inference: here we use one H100 & the same 144 test videos across models. Video encoding covers offline indexing and online V2T queries; latency uses batch 1, throughput/memory uses batch 16.

ModelGFLOPs (ViT/V-JEPA2+Q-Former/LLM)latency (ms/video)videos/speak memory (GB)
Qwen-2B3,620 (1,779/0/1,840)74375.9
Qwen-2B-3fps10,001 (4,747/0/5,255)83109.6
Qwen-8B11,253 (2,423/0/8,830)9215.918
MAVE6,240 (1,925/2,192+12/2,112)140179.4
ViCLIP1,653141543.0

Takeaways: MAVE uses 1.7x Qwen-2B FLOPs, mostly from VJEPA2. Q-former cost is negligible. Text queries bypass the temporal branch for MAVE: Qwen-2B/MAVE both cost 70 GFLOPs and 45 ms for A (8B: 343 GFLOPs, 59 ms). Exact top-10 search over 357K-video gallery takes 0.6 ms on GPU/~18 ms on CPU.

Q1.1 Recordings of unseen classes: After filtering, 7,317 clips remained: 110 used in test set and 7,207 excluded from both the ~357K total and training.

Q1.2 Participant overlap: We have participant IDs but no session IDs. The 2,740 seen-test clips comprise:

(a) Same participant+action (different video) in training: 2,012 (73%). (b) Participant in training, new actions: 619 (23%). (c) Participant unseen or ID missing: 109 (4%).

ModelA+C T2V R@1: a/b/c
2B71/58/61
8B77/65/63
MAVE83/66/68

Takeaways: Scores are highest in (a) for all models, inflating the absolute scores; we will note this in the paper. MAVE improves over 2B in (a)/(b). Group (c) is too small, so results are not conclusive.

Q2 Frame order: We reran the full test with reversed frames, shuffled frames, or one repeated middle frame (static). Seen action-only T2V R@1:

ModelNormal/reverse/shuffle/static
Zero-shot 2B1/1/1/1
2B19/11/12/7
MAVE45/27/18/3
MAVE: only temporal branch perturbed45/28/18/3
MAVE: only ViT branch perturbed45/43/44/44

Performance drops are significant for all models except MAVE's ViT-only perturbation. For 26 directional pairs (e.g., swiping left/right), reversal increases matches with the partner caption: zero-shot 2B 4->6%, 2B 13->41%, MAVE 16->49%.

Takeaways:

  1. zero-shot model ignores frame order, supporting the use of MAVR training.
  2. MAVE's gains relies on the temporal branch: making only that branch's input videos static drops R@1 to 3%, below the baseline, while ViT-only perturbations have little effect.
  3. Reversal hurts MAVE comparably to the fine-tuned 2B baseline (MAVE's drop: 40% vs 2B: 42%); shuffle/static hurt MAVE more than the baseline (Drop for shuffle->MAVE: 60% vs 2B: 37%; Drop for static->MAVE: 93% vs 2B:63%). We will add these results to the paper.

Q3 Early stopping and Table 9 difference: Early stopping is triggered by action-only nDCG@5 using a validation split carved from the training data (4,835 seen-class videos). in table 3, Baselines were trained for 10K steps; MAVE used 7.2K warm-up + 4.4K joint steps. Table 9 used 2K training steps instead; it is meant to compare anchoring versus no anchoring. We will clarify this difference in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.