- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, the authors introduce MAVR, a motion-centric benchmark and training dataset containing approximately 357K video clips across 287 action classes, with scene-matched hard negatives and controlled captions designed to emphasize fine-grained motion understanding. They further propose MAVE, which integrates a pretrained temporal encoder into an MLLM-based embedding model through a Q-Former and temporal token fusion. Experiments reveal substantial performance gaps between existing models on MAVR and general multimodal retrieval benchmarks. MAVE achieves significant improvements on seen action classes but shows limited generalization to unseen actions.
Critical strengths
-
MAVR provides a large-scale, motion-centric retrieval dataset with carefully designed scene-matched negatives and controlled captions. It addresses an important evaluation gap by reducing reliance on static appearance cues.
-
The extensive evaluation reveals that strong performance on general multimodal retrieval benchmarks does not necessarily translate to fine-grained motion understanding. The experiments also demonstrate substantial improvements from incorporating explicit temporal representations on seen actions.
Critical weaknesses
-
The proposed MAVE method primarily combines established components, including V-JEPA2, Q-Former, and LoRA, without introducing a substantially new temporal modeling or cross-modal alignment mechanism. This limits its technical contribution beyond the benchmark.
-
Despite significant improvements on seen actions, MAVE fails to consistently outperform the baselines on unseen actions and sometimes performs worse. This raises concerns about whether the proposed approach learns transferable motion representations rather than adapting to the training action distribution.
Questions for authors
-
How much of MAVE's improvement comes from genuinely modeling motion dynamics rather than simply incorporating additional pretrained video features? Could the authors provide a temporal-order perturbation experiment or a comparison with an alternative pretrained video encoder?
-
What explains MAVE's limited performance on unseen actions? Could the authors clarify whether this is primarily caused by overfitting to the training action classes or by insufficient alignment between temporal features and text embeddings?
2 comments
Thank you for the detailed response and additional experiments. The temporal perturbation tests largely address my concern about whether MAVE uses motion information. However, your reply to Reviewer 8Ksd shows that 73% of test clips for seen actions feature the same people doing the same actions as in training, so I still wonder whether the performance of the model relies on familiar people or settings.
Thank you for the review.
Note on scope: Our main contribution is MAVR: it has ~357K newly collected videos from paid, consenting participants, not a re-curation of public data. MAVR exposes a gap in motion-sensitive retrieval. MAVE is only a first step toward closing that gap. Its poor transfer to unseen actions is part of what MAVR is built to show.
W1. Novelty. Yes, MAVE deliberately combines established components. MAVR is the main contribution of this work; it shows:
MAVE is only a first step toward closing the gap for seen actions. It shows that a separate temporal pathway improves retrieval, and the new tests below show that its gain really comes from a better modelling of motion dynamics.
Q1. Motion or additional pretrained features? We evaluated the full test set with reversed frames, shuffled frames, or one repeated middle frame. We also perturbed MAVE's branches separately. Results show seen action-only T2V Recall@1:
If the temporal branch only added appearance features, a static temporal input would still help, but it doesn't; with the ViT frames untouched, a static temporal input leads to a drop from 45% to 3%, below the baseline. Perturbing only the ViT frames has a negligible effect. Thus, MAVE's gain comes from encoding motion dynamics in the temporal branch.
We also added results for ViCLIP-L/14; it's a video encoder pretrained on InternVid-10M and fully fine-tuned with our MAVR recipe. Zero-shot action-only T2V is 1%, comparable to MLLM embedders; fine-tuning it lifts this metric to 52%, above MAVE's 45% result. Unseen action-only T2V remains only 9%. ViCLIP's and MAVE's results support the notion that having video-native features improves performance compared to MLLM-based embedders with image-oriented ViTs.
W2/Q2. Unseen actions: overfitting or insufficient alignment? We ran the following new analyses to find where the failure happens.
(i) Do the video features transfer? For each of the 110 unseen clips, we find its nearest other unseen clip by video embedding, skipping clips of the same person. A hit means the same class (chance is 9%).
(ii) Does text matching work? We rank unseen queries only against the 110 unseen clips (``gallery is the unseen pool only'').
(iii) The paper's unseen numbers rank each unseen query against all 2,850 test clips (``full gallery''). 96% of these clips are from actions seen during training.
Our reading:
So the root cause is closer to insufficient alignment than overfitting: it is a bias toward the training classes when trained and new actions are mixed in one gallery. The evidence is consistent with the bias studied in generalized zero-shot learning. One way to reduce it is broader video-text pretraining of the temporal branch before MAVR training. We will add this analysis to the paper.
Ethics: privacy and copyright. All clips are first-party recordings: paid workers recorded themselves on their own webcam/phone via our collection platform in 2019-2024 and uploaded the clips themselves. Nothing was scraped or taken from video platforms.
As stated in our Ethics Statement, workers were informed of the collection's purpose and signed consent authorizing research release before recording. We removed audio and metadata (device/time/location), retained clips with at most one detected face, and discarded clips where the person appears under 19. We release no names, contact details, worker IDs or demographic labels. Three source collections were removed entirely for compliance reasons before release.
Faces remain visible because many actions involve the face/hands. MAVR will use gated access with acceptance of a research-only license prohibiting face recognition, re-identification, demographic inference, surveillance and generating participants' likenesses. The dataset page will provide a removal-request contact; requested clips will be removed in the next version.