- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes YOFO-VTG, a framework to reformulate MLLM-based video temporal grounding as parallel prediction of start and end boundary distributions within a single forward pass. Built on Qwen3-VL, the method extracts temporally aligned hidden states at visual delimiter positions and uses the native language modeling head to score candidate boundaries. It replaces causal attention with bidirectional attention so that each prediction can incorporate the full video and query context. Training follows a two-stage procedure: cross-entropy supervision first establishes boundary distributions, followed by joint cross-entropy and differentiable IoU optimization with difficulty-aware sampling. Experiments with 2B, 4B, and 8B models trained on TimeLens-100K and evaluated on TimeLensBench report competitive localization accuracy and reduced inference latency compared with autoregressive baselines. The main contribution is a direct, non-autoregressive localization interface for MLLMs that avoids sequential timestamp generation.
Critical strengths
-
Simple and efficient formulation. The paper reformulates MLLM-based temporal grounding as parallel boundary prediction, eliminating sequential timestamp generation. Reusing temporally aligned representations and the native LM head provides a relatively simple implementation, with inference measurements showing lower latency than autoregressive baselines.
-
Effective use of contextual representations. Bidirectional attention enables each candidate boundary to incorporate the full video and query context, while visual delimiter states provide a compact interface for temporal prediction. The component ablations support these design choices, showing consistent improvements over causal attention and the evaluated video-token representation alternative.
-
Promising results across model sizes. Experiments cover 2B, 4B, and 8B Qwen3-VL backbones on three TimeLensBench subsets. The reported results show competitive localization accuracy, particularly on Charades-TimeLens, with substantial improvements over the corresponding base models. The strong performance of the 2B variant further suggests practical value under limited computational budgets.
Critical weaknesses
-
Limited backbone diversity. YOFO-VTG is evaluated only on Qwen3-VL. Experiments on Qwen3.5 or another MLLM family are needed to support its claimed generality.
-
Missing strong proprietary baselines. Comparisons with models such as Gemini-3.1-Pro and Seed-2.0 would better establish the method’s practical competitiveness.
-
Unclear “zero-shot” terminology. The model receives temporal grounding supervision on TimeLens-100K; “zero-shot” here denotes cross-dataset transfer, which should be stated explicitly.
-
Ambiguous evaluation protocol. The paper selects the “better-performing” of two candidate intervals without explaining the selection rule. Ground-truth-based selection would make the results incomparable with standard single-prediction Recall@1.
Questions for authors
Incomplete training specifications. Although the main hyperparameters are provided, several details needed for reproduction remain unclear, including the Q-LoRA quantization settings and target modules, the validation criterion for early stopping and checkpoint selection, and the construction of boundary tokens and discrete temporal labels. The Stage 2 training schedule across model sizes also requires clarification.
4 comments
W4. Ambiguous evaluation protocol
The paper selects the "better-performing" of two candidate intervals without explaining the selection rule. Ground-truth-based selection would make the results incomparable with standard single-prediction Recall@1.
Response. We thank the reviewer for pointing out the ambiguity in our evaluation protocol. In the original evaluation, we selected between the argmax and soft decodings of the same predicted distributions based on their IoU with the ground-truth interval. We agree that this is not appropriate for standard Recall@1. We therefore revise the protocol to use argmax for the main R@1 results, which requires no ground-truth information. Meanwhile, we retain the original results as Coverage@2, since they provide complementary information by measuring how well the two decoding strategies can cover the ground-truth event. This offers an empirical assessment of the localization information encoded in the predicted probability distributions and provides an additional reference for downstream applications that may benefit from multiple candidate intervals. Detailed results are provided in Table 1, with a condensed version shown below.
| Scale | Dataset | Best Baseline | Ours (Argmax) | Ours (Coverage@2) |
|---|---|---|---|---|
| 8B | Charades-TL | 55.2 | 53.5 | 59.8 |
| ActivityNet-TL | 54.1 | 49.9 | 53.9 | |
| QVHighlights-TL | 65.5 | 59.4 | 62.8 | |
| 4B | Charades-TL | 50.5 | 51.9 | 58.9 |
| ActivityNet-TL | 46.8 | 48.8 | 52.9 | |
| QVHighlights-TL | 62.2 | 59.5 | 63.2 | |
| 2B | Charades-TL | 46.5 | 49.6 | 56.4 |
| ActivityNet-TL | 39.9 | 46.4 | 51.2 | |
| QVHighlights-TL | 52.2 | 57.6 | 62.1 |
Importantly, YOFO predicts one probability distribution for each boundary in a single forward pass, preserving richer temporal localization information than a deterministic prediction. Unlike the deterministic outputs of existing baselines, this distributional representation can be further exploited through different decoding strategies. This naturally enables multiple decoding strategies and candidate construction without additional model inference. Argmax and soft decoding are two simple examples that exploit complementary information in these distributions. Under the revised argmax-only protocol, YOFO remains competitive with the baselines. The Coverage@2 results further demonstrate the benefit of exploiting the richer predicted distributions.
Q1. Incomplete training specifications
Although the main hyperparameters are provided, several details needed for reproduction remain unclear, including the Q-LoRA quantization settings and target modules, the validation criterion for early stopping and checkpoint selection, and the construction of boundary tokens and discrete temporal labels. The Stage 2 training schedule across model sizes also requires clarification.
Response. We thank the reviewer for identifying these missing details and provide the following clarifications.
Q-LoRA configuration. Due to computational budget constraints, all models use 4-bit NF4 quantization with BF16 computation and no double quantization. The lm_head remains in BF16. We apply LoRA (rank 64, alpha 128, dropout 0.1, no bias) to all language-backbone linear layers, while freezing the vision tower and multimodal merger. The target modules are q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj for Qwen3-VL, with five additional GatedDeltaNet projections (in_proj_qkv, in_proj_a, in_proj_b, in_proj_z, out_proj) for Qwen3.5.
Validation and checkpoint selection. We do not use validation-based checkpoint selection. As shown in Table 2, Stage 1 performance at 1,200 and 2,400 steps is comparable; thus, all models proceed to Stage 2 after 1,200 steps. Stage 2 runs for 800 steps, and we consistently report the step-600 checkpoint, where performance has largely stabilized. This fixed-step strategy is applied across all model scales without relying on validation metrics.
Boundary tokens and discrete temporal labels. We follow Qwen-VL's native video-processing pipeline using the official qwen-vl-utils. Videos are sampled at 2 FPS, with every two consecutive frames forming one temporal unit (1 second). Each unit follows [timestamp, <|vision_start|>, video tokens, <|vision_end|>], where timestamps start at 0.0 and increase by 1.0 second. The vision_start and vision_end positions serve as the start and end boundary positions, respectively, and discrete labels follow this one-second temporal alignment.
Stage 2 training schedule. Stage 2 uses 25,249 difficulty-focused samples selected from the 96,171 TimeLens training samples. All models are initialized from their Stage 1 adapters at step 1,200 and share an effective batch size of 32 and a maximum of 800 steps. Common settings are AdamW-8bit, a learning rate of 5e-5 with cosine decay, 0.03 warmup ratio, BF16 precision, gradient clipping at 1.0, and non-reentrant gradient checkpointing.
Thank you for your inspiring comments and questions.
Thank you for taking the time to review our paper. We would like to know if you have any further concerns and whether the additional experiments and results we have provided have addressed your previous concerns. We look forward to hearing from you.
We thank Reviewer 1vgF for the careful reading and constructive assessment. We appreciate the reviewer’s recognition of our simple and efficient formulation (S1), the effectiveness of bidirectional attention and delimiter states (S2), and the promising results across 2B/4B/8B scales (S3). Below we address the four weaknesses and the reproducibility question.
W1. Limited backbone diversity
Response. We agree that evaluating YOFO on another MLLM backbone is important for demonstrating its generality. We therefore conduct additional experiments on Qwen3.5, with results shown below.
All results use the same argmax decoding evaluation protocol, and are conducted under the low-resolution setting. We note that Qwen3.5 does not show a substantial improvement over Qwen3-VL. We attribute this gap to its hybrid architecture of linear and full attention, where the linear-attention component is implemented with a unidirectional attention operator. Modifying it to support bidirectional attention changes its forward computation, which may introduce a performance gap. Nevertheless, the results on Qwen3.5 demonstrate that YOFO generalizes to another MLLM backbone.
W2. Missing strong proprietary baselines
Response. We thank the reviewer for suggesting comparisons with proprietary models. We agree that such comparisons are valuable. However due to budget constraints, we cannot evaluate additional proprietary models. Instead, we report the proprietary-model results from the TimeLens paper on TimeLens-Bench for reference. Since our evaluation protocol is consistent with TimeLens, these results are directly comparable. Our results use argmax decoding.
These results show that strong proprietary models already achieve highly competitive VTG performance, making efficiency and cost particularly important. YOFO remains competitive even at 2B scale, while offering single-pass inference and simple, effective SFT-based training.
W3. Unclear "zero-shot" terminology
Response. We thank the reviewer for pointing out the ambiguity in our use of “zero-shot.” We use this term to mean cross-dataset transfer: the model is trained with temporal grounding supervision on TimeLens-100K and evaluated on TimeLens-Bench, whose source datasets (Charades, ActivityNet, and QVHighlights) are disjoint from the training sources (CosmoCap, DiDeMo, HiREST, InternVid-VTime, and QueryD). We will replace it with “cross-dataset zero-shot transfer” and explicitly define the setting in experimental setup.