acceptodds
Withdrawn from ICLR 2027

SwiftVMR: Toward Efficient and Effective MLLM-based Video Moment Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

When repurposed for Video Moment Retrieval (VMR), Multimodal Large Language Models (MLLMs) deliver strong cross-modal alignment yet inherit significant efficiency burdens and a representational gap. Directly applying standard generative paradigms to VMR poses four major challenges to efficient and effective inference: 1) dense video sampling triggers a catastrophic visual token explosion; 2) auto-regressive decoding introduces inherent sequential overhead; 3) discrete-token formulations introduce a representational gap when modeling the continuous, ambiguous nature of action boundaries; and 4) massive parameter scales prohibit resource-constrained deployment. To address these challenges, we propose SwiftVMR, a framework retaining the MLLM's alignment prowess while replacing iterative decoding with single-shot, continuous boundary prediction. To tackle the token explosion, a Zero-initialized Motion-aware Spatial Compressor (Z-MSC) is introduced to aggressively reduce visual tokens while preserving spatial structures and training stability. To overcome the sequential decoding latency and representational mismatch of autoregressive generation, we introduce a Distribution-Aware Perception Head (DAPH), a single-shot logit-based architecture directly modeling probability distributions over temporal proposals. Finally, to overcome the deployment barrier, we introduce a Dual-Space Structured Distillation (DSSD) strategy to distill a high-capacity teacher into a lightweight student. Leveraging the continuous logit space provided by DAPH, DSSD performs foreground-weighted feature alignment and quality-gated distribution distillation. Extensive experiments on four benchmarks demonstrate that SwiftVMR matches state-of-the-art performance while accelerating inference by an order of magnitude, proving that efficiency can be regained without sacrificing performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Moment Retrieval, Video Understanding, Multimodal Learning, Large Multimodal Models, Cross-Modal Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3N4nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SwiftVMR, an MLLM-based framework for efficient video moment retrieval. It combines a visual token compressor (Z-MSC), a non-autoregressive distribution-based boundary head (DAPH), and a distillation strategy (DSSD) for a smaller student model. The authors evaluate localization accuracy on four benchmarks and report component ablations and inference latency measurements.

Critical strengths
  1. SwiftVMR combines visual token compression, distribution based boundary prediction, and student distillation in a coherent framework for video moment retrieval.
  2. It achieves competitive performance across four benchmarks, and the latency analysis provides evidence of substantial computational savings.
  3. The ablations cover the main components, while the student model experiments show that the framework can also benefit a smaller backbone.
Critical weaknesses
  1. The novelty of DAPH is insufficiently established. Prior MLLM work has used a specialized interval decoder or a distribution based time decoder, while Table 2(b) compares DAPH only with a Regressive Head and a variant without DFL. Z-MSC also lacks comparisons with other visual token compression methods under the same token budget.
  2. Table 1 does not cover several relevant recent methods and relies on results reported under different settings. On QVHighlights, the Avg. Pooling + Regressive Head baseline in Table 2 already slightly exceeds UniTime, while the full model provides a modest additional gain. Without matched training data, video sampling, and visual token budgets, the source of the reported advantage remains unclear.
Questions for authors
  1. How does DAPH differ from prior distribution based time decoders such as DisTime? How do their accuracy and latency compare under the same backbone and training setup? [DisTime: Distribution-based Time Representation for Video Large Language Models]

  2. Please compare Z-MSC with other visual token compression methods at the same frame count and visual token budget, reporting both accuracy and latency.

  3. Why does the simple baseline in Table 2 slightly exceed UniTime on QVHighlights? Please clarify differences in training data, video sampling, and evaluation protocols, and quantify the gains from the proposed components under matched settings.

  4. Can the authors compare with recent methods such as Invert4TVG, Divid, and GroundVTS? [Invert4TVG: A Temporal Video Grounding Framework with Inversion Tasks Preserving Action Understanding Ability] [Divid: Disentangled Spatial-Temporal Modeling within LLMs for Temporally Grounded Video Understanding] [GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding]

  5. What is the performance of Avg. Pooling + [Text][Video] + DAPH, which is absent from Table 2? This configuration would isolate the gain from DAPH relative to Avg. Pooling + Regressive Head.

Reviewer_CHVdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets the efficiency bottlenecks of MLLM-based VMR, including visual token explosion, autoregressive decoding overhead, and large model scale. The proposed SwiftVMR compresses visual tokens with Z-MSC, replaces autoregressive timestamp generation with DAPH, a single-shot head that predicts boundary distributions over temporal bins, and distills a large teacher into a lightweight student with DSSD. Experiments on four datasets show competitive or better accuracy than recent MLLM-based methods with substantially lower inference cost.

Critical strengths

The paper studies MLLM-based VMR from an efficiency perspective, which is practically relevant and has received relatively little attention.

Critical weaknesses
  1. The novelty of this paper is limited. DAPH essentially combines a DETR-style decoder with Generalized Focal Loss/D-FINE-style distribution regression and directly adopts DFL; Z-MSC combines attention-based pooling and a convolutional residual path with ControlNet-style zero initialization; and DSSD is a standard combination of feature and logit distillation with foreground weighting and quality gating. The paper does not provide new insights beyond assembling these existing techniques for MLLM-based VMR.

  2. The efficiency claims are insufficiently supported. The order-of-magnitude speedup holds only for the post-ViT stage, while the end-to-end gain is considerably smaller and is partly confounded by different video sampling settings between the compared methods. Moreover, efficiency is compared only against UniTime, without comparisons to existing token compression methods, other non-autoregressive MLLM-based approaches, or efficient specialist models.

  3. The contribution of the proposed modules is limited relative to a simple baseline. The baseline with average pooling and a regressive head already outperforms the strongest MLLM-based competitor and has nearly the same end-to-end latency as the full model. This suggests that both the efficiency and much of the effectiveness come from standard spatial pooling and non-autoregressive prediction, while Z-MSC, DAPH, and DSSD bring only incremental accuracy gains.

Questions for authors

Could the authors report the standard mAP metrics on QVHighlights and clarify which split the reported results are evaluated on?

Reviewer_FGp3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

MLLM-based video moment retrieval inherits four costs from generation—visual token explosion, sequential decoding, discrete-timestamp/continuous-boundary mismatch, and parameter scale—so SwiftVMR replaces generation with a single forward pass: Z-MSC content-adaptively pools each frame's spatial grid and adds a zero-initialised Conv3d spatiotemporal residual before the LLM; a causal [Text][Video] prompt makes visual tokens query-aware under a frozen-encoder/LoRA-tuned MLLM, with a bidirectional temporal encoder restoring global context; DAPH expands N_q queries into N_bin centre/duration logits, refines them by cross-attention, and decodes by expectation under L1/GIoU plus DFL; and DSSD distils a 7B teacher into a 2B student via foreground-weighted MSE plus temperature-softened KL, gated on teacher IoU > τ_q. SwiftVMR-7B reports the best value in every cell with an external number on QVHighlights, ActivityNet Captions, Charades-STA and TACoS.

Critical strengths
  1. The design choice is right, and the paper quantifies it at the level that matters. Replacing next-token timestamp generation with a distribution head is the correct move for VMR, and the authors do not stop at the claim: Table 3 separates token compression, LLM forward and decoding, showing the decoder stage going 701.9 → 11.0 ms and the LLM forward 1577.8 → 124.1 ms. Appendix A's own reading — "the majority of the speedup is unlocked by Z-MSC", with one-step-constrained UniTime still at ~1656 ms (§4.3) — is a fair, self-critical attribution.
  2. The ablation suite is complete in shape, and consistent across two benchmarks. Every component is ablated on both QVHighlights (Table 2, p.8) and Charades-STA (Table 6, p.19), including the negative controls one would want (w/o zero-init, w/o DFL, [Video][Text], coordinate-only distillation, feature alignment in the last hidden state), and the full model wins all four metrics in every block. The DSSD design is a genuine coupling rather than a bolt-on: the bin distribution is what makes teacher soft labels transferable across models with different initial predictions, and the IoU gate is a sensible response to noisy teachers.
Critical weaknesses
  1. Table 1 mixes self-run and cited numbers under different sampling and token budgets yet claims SOTA "across all four benchmarks."、
  2. The bidirectional temporal encoder appears in Fig. 2 and carries the 11.0 ms decoder row in Table 3 but is never ablated.
  3. The trivial control—average pooling plus a regressive head (compressed and non-generative)—is reported as faster (780.5 vs 793.2 ms E2E) and only 2.44 R1@7 behind, so ~92% of the efficiency story comes from "compress and don't generate," with the three proposed modules buying ~2.4 points at slightly worse latency. The trivial control—average pooling plus a regressive head (compressed and non-generative)—is reported as faster (780.5 vs 793.2 ms E2E) and only 2.44 R1@7 behind, so ~92% of the efficiency story comes from "compress and don't generate," with the three proposed modules buying ~2.4 points at slightly worse latency.
  4. The trivial control—average pooling plus a regressive head (compressed and non-generative)—is reported as faster (780.5 vs 793.2 ms E2E) and only 2.44 R1@7 behind, so ~92% of the efficiency story comes from "compress and don't generate," with the three proposed modules buying ~2.4 points at slightly worse latency.
Questions for authors

Please answer the questions in weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.