- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes UMVR, an MLLM-based framework for text-based retrieval of both short and long videos. It first adaptively partitions videos into events using frame-level similarity and local visual feature changes. Event-Aware Token Reduction (EATR) then assigns different token budgets according to event informativeness, using FlashVID for compression. A shared MLLM produces event-level and global video representations, whose matching scores are combined for retrieval. Evaluation spans seven retrieval benchmarks and includes controlled comparisons, component ablations, and efficiency and transfer analyses. Results show that UMVR outperforms the controlled baseline while retaining approximately 10% of visual tokens and reducing post-vision-encoding processing time.
Critical strengths
The method offers a useful balance between retrieval accuracy and encoding cost. It divides videos into events, assigns more tokens to informative events, and combines local and global matching. On ActivityNet, it achieves 23.7% R@1 and 60.1% R@10, compared with 20.1% and 56.8% for VLM2Vec-V2 using the same backbone and training data. It also retains only about 10% of visual tokens and reduces processing time after visual encoding from 296.9 ms to 119.7 ms.
Critical weaknesses
- The paper’s novelty is unclear in light of prior work. The motivation presents short- and long-video retrieval as separate paradigms, but earlier methods already accommodate local and global relevance within a common framework. For example, Attentive Snippet Prompting for Video Retrieval combines text-guided snippet representations with global and local video representations and supports both short and long videos. X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval and Unified Coarse-to-Fine Alignment for Video-Text Retrieval also study matching at multiple granularities. Moreover, Uneven Event Modeling for Partially Relevant Video Retrieval already explores non-uniform event representations. These methods are not identical to UMVR, but they make its broad motivation around local–global modeling and adaptive granularity less distinctive. The paper should explain and evaluate what its event partitioning and token allocation add beyond these existing approaches.
- The experiments do not clearly show where the gains come from. Table 1 mainly compares general-purpose MLLM embedding models, while closely related local–global and event-based video retrievers are missing. The same-backbone VLM2Vec-V2 comparison is useful, but this baseline only uses a global representation. A stronger comparison would add local–global matching to the same backbone and test whether adaptive event partitioning still helps. The efficiency results also need comparisons at the same actual token count and full encoding times, including visual encoding and event partitioning. Without these controls, it is difficult to judge how much the proposed designs improve accuracy and efficiency.
Questions for authors
- How are short and long videos defined? What duration threshold separates them? Is the distinction based on video length or dataset-specific retrieval settings? For a mixed gallery, how would the model decide whether to sample 64 or 128 frames?
- Why are closely related video retrieval methods missing from the comparisons? Table 1 mainly includes general-purpose MLLM embedding models. Can the authors compare against local–global and event-based retrievers, and add a same-backbone baseline with uniform temporal segments and local–global matching? This would help isolate the benefit of adaptive event partitioning.
- Why is Qwen2-VL-2B the only backbone evaluated? Do the retrieval gains and compression benefits still hold with a more recent, stronger MLLM?
- How much of the efficiency benefit comes from EATR rather than FlashVID? Can the authors compare standard FlashVID, uniform budgeting, and event-adaptive budgeting within the same retrieval model at matched actual token counts? Please also report full encoding times, including visual encoding and event partitioning.
- Does aggressive compression preserve temporal information? Can the authors compare full-token and compressed versions of the same model on queries requiring action order, motion direction, or brief events? Overall retrieval scores may not reveal the loss of these cues.
- What justifies the event-budget allocation? Why should feature norms indicate how many tokens an event needs? How does the method avoid over-compressing visually simple but important events? Also, since events have different lengths, how does the normalization ensure a fixed total token budget?
4 comments
Response to Reviewer EzpH (Part 2)
Q3: Why is Qwen2-VL-2B the only backbone evaluated? Do the retrieval gains and compression benefits still hold with a more recent, stronger MLLM?
Response to Q3: We thank the reviewer for this suggestion. Besides Qwen2-VL-2B, we extend UMVR to the stronger Qwen3-VL-2B-Instruct backbone and compare it with VLM2Vec-V2 under the same backbone and training protocol. For retrieval performance, UMVR consistently outperforms VLM2Vec-V2 across all seven benchmarks with clear margins. For compression benefits, since EATR reduces visual tokens before LLM encoding and does not depend on a specific backbone, its compression benefits still hold with Qwen3-VL, retaining only about 10% of visual tokens and reducing the subsequent LLM processing cost. We have incorporated these results in Appendix A.4 of the revised manuscript.
| Method | MSR-VTT R@1 | MSR-VTT R@10 | VATEX R@1 | VATEX R@10 | MSVD R@1 | MSVD R@10 | DiDeMo R@1 | DiDeMo R@10 | ActivityNet R@1 | ActivityNet R@10 | QVHighlights R@1 | QVHighlights R@10 | TVR R@1 | TVR R@10 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| VLM2Vec-V2 | 33.5 | 70.6 | 38.5 | 77.8 | 53.7 | 91.6 | 41.2 | 75.1 | 23.0 | 59.4 | 24.0 | 56.8 | 7.4 | 20.8 |
| UMVR | 36.1 | 72.4 | 41.9 | 82.1 | 59.6 | 92.8 | 48.7 | 80.1 | 28.0 | 64.6 | 34.0 | 70.3 | 9.9 | 27.3 |
Q4: How does EATR contribute to efficiency and retrieval performance and what is the complete video encoding cost?
Response to Q4: We would like to clarify that EATR achieves comparable encoding efficiency to standard FlashVID. The main difference is that EATR adaptively allocates tokens across events according to their informativeness, rather than using a uniform budget. This allows EATR to preserve more relevant visual information and achieve better retrieval performance without increasing the encoding cost.
Following the reviewer's suggestion, we compare standard FlashVID, uniform budgeting, and event-adaptive budgeting under matched actual token counts on VATEX and ActivityNet. As shown in the following table, UMVR outperforms standard FlashVID on both datasets, demonstrating the combined effectiveness of LAER and EATR. Furthermore, under the same LAER event partitioning, event-adaptive budgeting consistently outperforms uniform budgeting, confirming the benefit of adaptive token allocation.
We have updated Table 3 to include the comparison with standard FlashVID (denoted as FTP+UB) and uniform budgeting (denoted as LAER+UB) in the revised manuscript.
| Variant | VATEX R@1 | VATEX R@10 | ActivityNet R@1 | ActivityNet R@10 |
|---|---|---|---|---|
| FlashVID | 36.4 | 76.1 | 21.7 | 57.3 |
| Uniform Budgeting | 37.2 | 77.0 | 22.1 | 58.6 |
| UMVR (ours) | 38.0 | 77.8 | 23.7 | 60.1 |
Q5: Does aggressive visual-token compression degrade retrieval performance on queries requiring fine-grained temporal information?
Response to Q5: We thank the reviewer for this valuable suggestion. To investigate whether token compression affects the preservation of temporal information, we group queries from ActivityNet into different subsets based on their temporal characteristics, including action order, motion direction, and brief events via Qwen3-VL-7B-Instruct. Queries identified as not involving any of them are grouped into the Others subset.
As shown in the following table, compression leads to some performance degradation on ActivityNet, but the drops across these temporal subsets are comparable to those for other queries. We attribute this to EATR's event-adaptive token allocation built upon FlashVID's spatiotemporal token reduction, which helps preserve relevant temporal information during compression.
Overall, these results suggest that aggressive token compression does not consistently cause greater performance degradation on queries involving temporal information.
Detailed experimental results and query categorization criteria are provided in Appendix A.5 of the revised manuscript.
| Query Subset | # Queries | Full Tokens R@1 | Compressed R@1 |
|---|---|---|---|
| Overall | 15753 | 25.3 | 23.7 |
| Action Order | 1163 | 28.0 | 26.7 |
| Motion Direction | 6373 | 23.3 | 22.1 |
| Brief Events | 999 | 20.5 | 19.2 |
| Others | 8892 | 26.8 | 24.5 |
Response to Reviewer EzpH (Part 3)
Q6: Why does EATR use feature norms for token allocation, how does it avoid over-compressing visually simple but important events, and does its normalization guarantee a fixed total token budget?
Response to Q6: Our use of visual feature norms is motivated by the empirical findings of [1], who observed a positive correlation between visual-token norms and semantic informativeness through qualitative and statistical analyses. Accordingly, we allocate more tokens to events with higher estimated informativeness, allowing them to retain more visual information during compression. We acknowledge that feature norms may not fully capture the semantic importance of visually simple events. Nevertheless, EATR adjusts token budgets rather than discarding entire events, preserving their compressed representations for subsequent matching and reducing the risk of losing important information. We have further discussed this limitation and potential improvements in Section 4 (Limitation and future work) of the revised manuscript.
Additionally, our event-wise normalization indeed does not strictly guarantee that the total number of retained tokens matches the specified token budget when event lengths differ, but the resulting token counts remain close to the target budget in practice. When comparing our EATR with other budgeting strategies, we adjust their token budgets to match the actual number of tokens retained by EATR for each video, ensuring a fair comparison under identical token counts.
[1] Luo B, Wang T, Chen H, et al. Enhancing Visual Token Representations for Video Large Language Models via Training-free Spatial-Temporal Pooling and Gridding[C]//Proceedings of the International Conference on Learning Representations (ICLR). 2026.
W1: The paper's novelty is unclear in light of prior work.
Response to W1: We thank the reviewer for the valuable comment. We would like to clarify that the main contribution of UMVR lies in establishing a unified and efficient retrieval framework for both short and long videos. Although ASP, X-CLIP, and UCoFiA have explored local-global or multi-grained matching, these methods primarily focus on improving retrieval accuracy on conventional short-video benchmarks rather than explicitly addressing the challenges of long-video retrieval. Specifically, short-video retrieval typically emphasizes holistic relevance, whereas long-video retrieval requires more localized matching, making it challenging to accommodate both within a unified framework. Moreover, increasing video duration introduces more visual tokens and substantially higher computational costs.
To address these challenges, UMVR introduces two complementary designs. First, LAER adaptively adaptively determines event boundaries and representation granularity based on both global scene transitions and local visual changes, enabling a single trained model to handle both short- and long-video retrieval without setting-specific architectures or scoring functions. Second, EATR leverages these adaptive events to allocate visual-token budgets according to their informativeness, substantially reducing MLLM encoding costs while preserving retrieval performance.
Experimental results further validate these contributions. Under the same backbone and training settings, our UMVR consistently outperforms existing retrieval methods, including VLM2Vec-V2, XCLIP and UCoFiA, across both short- and long-video benchmarks. Moreover, UMVR retains only 10% of visual tokens while achieving better retrieval performance and lower encoding latency than the full-token baseline.
We also acknowledge that UEM explores non-uniform event partitioning. Unlike UEM, our LAER employs a coarse-to-fine partitioning strategy and further integrates event-aware token allocation for efficient MLLM encoding.
We thank the reviewer again for the valuable feedback and hope that the reviewer could reconsider the assessment of our work.
Dear Reviewer EzpH,
Thank you for your valuable feedback and thoughtful comments. As the discussion period is approaching its end, we would greatly appreciate it if you could let us know whether our clarifications and additional analyses have adequately addressed your concerns. We have also revised the manuscript accordingly to incorporate these improvements. We would be grateful for any further feedback and hope our efforts can be taken into consideration in your final assessment.
Thank you again for your time and consideration.
Best regards, The Authors
Response to Reviewer EzpH (Part 1)
We sincerely thank the reviewer for the valuable comments and constructive suggestions, which have helped us strengthen the experimental analysis and further improve the clarity and completeness of our manuscript.
Q1: How are short- and long-video retrieval defined, and how is frame sampling handled in a mixed video gallery?
Response to Q1: We would like to clarify that short and long videos are distinguished based on dataset-specific setting. Datasets with a mean duration of 60 seconds or less are treated as short-video benchmarks, and those with a mean duration above 60 seconds are treated as long-video benchmarks. Specifically, MSR-VTT (mean duration of $\sim$15s), VATEX ($\sim$10s), MSVD ($\sim$10s), and DiDeMo ($\sim$50s) are evaluated as short-video retrieval benchmarks, while ActivityNet ($\sim$118s), QVHighlights ($\sim$150s), and TVR ($\sim$76s) are evaluated as long-video retrieval benchmarks.
For a mixed gallery, we can adopt a 60-second threshold to separate videos. We sample up to 64 frames for videos of 60 seconds or less and up to 128 frames for longer videos. We evaluate it on a mixed gallery combining VATEX and ActivityNet. UMVR achieves R@1 of 23.0% and R@10 of 57.7%, compared with 20.8% and 55.5% for VLM2Vec-V2, respectively. These results show that UMVR retains its advantage in the mixed gallery. We have incorporated these results in Appendix A.3 (Mixed-Gallery Retrieval with Duration-Adaptive Frame Sampling) of the revised manuscript.
Q2 & W2: How does UMVR compare with existing video retrieval methods, and what is the contribution of adaptive event partitioning?
Response to Q2 & W2: We thank the reviewer for this suggestion. Existing video retrieval methods, such as X-CLIP[1] and UCoFiA[2], are typically trained and evaluated separately on each target dataset, whereas UMVR is trained on a single dataset and evaluated across both short- and long-video retrieval benchmarks. Differences in training settings and backbone architectures make it difficult to conduct a fair comparison based on their originally reported results. To provide a fair comparison, we adapt two representative video retrieval methods using local-global matching, X-CLIP and UCoFiA, to the same backbone as UMVR and evaluate all methods under identical training and testing settings. As shown in the following table, UMVR consistently outperforms both X-CLIP and UCoFiA across all seven datasets, demonstrating its advantages in both short- and long-video retrieval.
Additionally, to verify the effectiveness of our adaptive event partitioning, we compare it with uniform temporal partitioning and the event partitioning strategy from UEM[3]. As shown in the following table, our method consistently outperforms both counterparts, demonstrating the benefits of our adaptive event partitioning.
We have updated above results to Table 1 and Table 4, with the corresponding analysis to Section 3.2 and Section 3.4 (Length-Adaptive Event Representation) of the revised manuscript.
[1] Ma Y, Xu G, Sun X, et al. X-clip: End-to-end multi-grained contrastive learning for video-text retrieval[C]//Proceedings of the 30th ACM international conference on multimedia. 2022: 638-647.
[2] Wang Z, Sung Y L, Cheng F, et al. Unified coarse-to-fine alignment for video-text retrieval[C]//2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023: 2804-2815.
[3] Zhu S, Chen H, Zhang W, et al. Uneven event modeling for partially relevant video retrieval[C]//2025 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2025: 1-6.