acceptodds
Under review as a conference paper at ICLR 2027

You Only Forward Once: Rethinking Video Temporal Grounding in MLLMs as Single-Pass Parallel Boundary Probability Prediction

Abstract

Video Temporal Grounding (VTG) aims to localize the video segment corresponding to a natural-language query. Existing MLLM-based methods typically generate temporal boundaries autoregressively as textual tokens, creating a mismatch with interval-level localization and incurring sequential decoding overhead. To this end, we present YOFO-VTG, which reformulates VTG as parallel boundary-distribution prediction within a single MLLM forward pass. YOFO-VTG extracts video-aligned hidden states at the visual start and end delimiters of each temporal unit and maps them through the native language modeling head to predict start- and end-boundary probability distributions. To overcome the contextual restriction of causal attention, we adopt bidirectional attention, allowing each unit to integrate information from the full video-query sequence. This design retains the native probabilistic prediction mechanism of MLLMs while directly exploiting the complete sequence of video representations. We further introduce a two-stage strategy that first learns coarse boundary distributions with cross-entropy (CE) supervision and then refines the predicted boundaries through joint CE and differentiable IoU optimization. Extensive experiments on TimeLensBench demonstrate that YOFO-VTG achieves competitive localization accuracy while enabling substantially faster inference, highlighting parallel probability distribution prediction as an efficient paradigm for MLLM-based VTG.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Temporal Grounding, Multimodal Large Language Models, Temporal Localization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1vgFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes YOFO-VTG, a framework to reformulate MLLM-based video temporal grounding as parallel prediction of start and end boundary distributions within a single forward pass. Built on Qwen3-VL, the method extracts temporally aligned hidden states at visual delimiter positions and uses the native language modeling head to score candidate boundaries. It replaces causal attention with bidirectional attention so that each prediction can incorporate the full video and query context. Training follows a two-stage procedure: cross-entropy supervision first establishes boundary distributions, followed by joint cross-entropy and differentiable IoU optimization with difficulty-aware sampling. Experiments with 2B, 4B, and 8B models trained on TimeLens-100K and evaluated on TimeLensBench report competitive localization accuracy and reduced inference latency compared with autoregressive baselines. The main contribution is a direct, non-autoregressive localization interface for MLLMs that avoids sequential timestamp generation.

Critical strengths
  1. Simple and efficient formulation. The paper reformulates MLLM-based temporal grounding as parallel boundary prediction, eliminating sequential timestamp generation. Reusing temporally aligned representations and the native LM head provides a relatively simple implementation, with inference measurements showing lower latency than autoregressive baselines.

  2. Effective use of contextual representations. Bidirectional attention enables each candidate boundary to incorporate the full video and query context, while visual delimiter states provide a compact interface for temporal prediction. The component ablations support these design choices, showing consistent improvements over causal attention and the evaluated video-token representation alternative.

  3. Promising results across model sizes. Experiments cover 2B, 4B, and 8B Qwen3-VL backbones on three TimeLensBench subsets. The reported results show competitive localization accuracy, particularly on Charades-TimeLens, with substantial improvements over the corresponding base models. The strong performance of the 2B variant further suggests practical value under limited computational budgets.

Critical weaknesses
  1. Limited backbone diversity. YOFO-VTG is evaluated only on Qwen3-VL. Experiments on Qwen3.5 or another MLLM family are needed to support its claimed generality.

  2. Missing strong proprietary baselines. Comparisons with models such as Gemini-3.1-Pro and Seed-2.0 would better establish the method’s practical competitiveness.

  3. Unclear “zero-shot” terminology. The model receives temporal grounding supervision on TimeLens-100K; “zero-shot” here denotes cross-dataset transfer, which should be stated explicitly.

  4. Ambiguous evaluation protocol. The paper selects the “better-performing” of two candidate intervals without explaining the selection rule. Ground-truth-based selection would make the results incomparable with standard single-prediction Recall@1.

Questions for authors

Incomplete training specifications. Although the main hyperparameters are provided, several details needed for reproduction remain unclear, including the Q-LoRA quantization settings and target modules, the validation criterion for early stopping and checkpoint selection, and the construction of boundary tokens and discrete temporal labels. The Stage 2 training schedule across model sizes also requires clarification.

4 comments

We thank Reviewer 1vgF for the careful reading and constructive assessment. We appreciate the reviewer’s recognition of our simple and efficient formulation (S1), the effectiveness of bidirectional attention and delimiter states (S2), and the promising results across 2B/4B/8B scales (S3). Below we address the four weaknesses and the reproducibility question.

W1. Limited backbone diversity

YOFO-VTG is evaluated only on Qwen3-VL. Experiments on Qwen3.5 or another MLLM family are needed to support its claimed generality.

Response. We agree that evaluating YOFO on another MLLM backbone is important for demonstrating its generality. We therefore conduct additional experiments on Qwen3.5, with results shown below.

BackboneScaleCharades-TL mIoUActivityNet-TL mIoUQVHighlights-TL mIoU
Qwen3-VL8B51.7546.1456.69
Qwen3.59B54.5144.2755.25

All results use the same argmax decoding evaluation protocol, and are conducted under the low-resolution setting. We note that Qwen3.5 does not show a substantial improvement over Qwen3-VL. We attribute this gap to its hybrid architecture of linear and full attention, where the linear-attention component is implemented with a unidirectional attention operator. Modifying it to support bidirectional attention changes its forward computation, which may introduce a performance gap. Nevertheless, the results on Qwen3.5 demonstrate that YOFO generalizes to another MLLM backbone.

W2. Missing strong proprietary baselines

Comparisons with models such as Gemini-3.1-Pro and Seed-2.0 would better establish the method's practical competitiveness.

Response. We thank the reviewer for suggesting comparisons with proprietary models. We agree that such comparisons are valuable. However due to budget constraints, we cannot evaluate additional proprietary models. Instead, we report the proprietary-model results from the TimeLens paper on TimeLens-Bench for reference. Since our evaluation protocol is consistent with TimeLens, these results are directly comparable. Our results use argmax decoding.

ModelCharades-TL mIoUActivityNet-TL mIoUQVHighlights-TL mIoU
GPT-4o41.840.452.1
GPT-540.542.956.8
Gemini-2.0-Flash46.749.360.8
Gemini-2.5-Flash48.652.564.3
Gemini-2.5-Pro52.858.170.4
Ours-8B53.549.959.4
Ours-4B51.948.859.5
Ours-2B49.646.457.6

These results show that strong proprietary models already achieve highly competitive VTG performance, making efficiency and cost particularly important. YOFO remains competitive even at 2B scale, while offering single-pass inference and simple, effective SFT-based training.

W3. Unclear "zero-shot" terminology

The model receives temporal grounding supervision on TimeLens-100K; "zero-shot" here denotes cross-dataset transfer, which should be stated explicitly.

Response. We thank the reviewer for pointing out the ambiguity in our use of “zero-shot.” We use this term to mean cross-dataset transfer: the model is trained with temporal grounding supervision on TimeLens-100K and evaluated on TimeLens-Bench, whose source datasets (Charades, ActivityNet, and QVHighlights) are disjoint from the training sources (CosmoCap, DiDeMo, HiREST, InternVid-VTime, and QueryD). We will replace it with “cross-dataset zero-shot transfer” and explicitly define the setting in experimental setup.

W4. Ambiguous evaluation protocol

The paper selects the "better-performing" of two candidate intervals without explaining the selection rule. Ground-truth-based selection would make the results incomparable with standard single-prediction Recall@1.

Response. We thank the reviewer for pointing out the ambiguity in our evaluation protocol. In the original evaluation, we selected between the argmax and soft decodings of the same predicted distributions based on their IoU with the ground-truth interval. We agree that this is not appropriate for standard Recall@1. We therefore revise the protocol to use argmax for the main R@1 results, which requires no ground-truth information. Meanwhile, we retain the original results as Coverage@2, since they provide complementary information by measuring how well the two decoding strategies can cover the ground-truth event. This offers an empirical assessment of the localization information encoded in the predicted probability distributions and provides an additional reference for downstream applications that may benefit from multiple candidate intervals. Detailed results are provided in Table 1, with a condensed version shown below.

ScaleDatasetBest BaselineOurs (Argmax)Ours (Coverage@2)
8BCharades-TL55.253.559.8
ActivityNet-TL54.149.953.9
QVHighlights-TL65.559.462.8
4BCharades-TL50.551.958.9
ActivityNet-TL46.848.852.9
QVHighlights-TL62.259.563.2
2BCharades-TL46.549.656.4
ActivityNet-TL39.946.451.2
QVHighlights-TL52.257.662.1

Importantly, YOFO predicts one probability distribution for each boundary in a single forward pass, preserving richer temporal localization information than a deterministic prediction. Unlike the deterministic outputs of existing baselines, this distributional representation can be further exploited through different decoding strategies. This naturally enables multiple decoding strategies and candidate construction without additional model inference. Argmax and soft decoding are two simple examples that exploit complementary information in these distributions. Under the revised argmax-only protocol, YOFO remains competitive with the baselines. The Coverage@2 results further demonstrate the benefit of exploiting the richer predicted distributions.

Q1. Incomplete training specifications

Although the main hyperparameters are provided, several details needed for reproduction remain unclear, including the Q-LoRA quantization settings and target modules, the validation criterion for early stopping and checkpoint selection, and the construction of boundary tokens and discrete temporal labels. The Stage 2 training schedule across model sizes also requires clarification.

Response. We thank the reviewer for identifying these missing details and provide the following clarifications.

Q-LoRA configuration. Due to computational budget constraints, all models use 4-bit NF4 quantization with BF16 computation and no double quantization. The lm_head remains in BF16. We apply LoRA (rank 64, alpha 128, dropout 0.1, no bias) to all language-backbone linear layers, while freezing the vision tower and multimodal merger. The target modules are q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj for Qwen3-VL, with five additional GatedDeltaNet projections (in_proj_qkv, in_proj_a, in_proj_b, in_proj_z, out_proj) for Qwen3.5.

Validation and checkpoint selection. We do not use validation-based checkpoint selection. As shown in Table 2, Stage 1 performance at 1,200 and 2,400 steps is comparable; thus, all models proceed to Stage 2 after 1,200 steps. Stage 2 runs for 800 steps, and we consistently report the step-600 checkpoint, where performance has largely stabilized. This fixed-step strategy is applied across all model scales without relying on validation metrics.

Boundary tokens and discrete temporal labels. We follow Qwen-VL's native video-processing pipeline using the official qwen-vl-utils. Videos are sampled at 2 FPS, with every two consecutive frames forming one temporal unit (1 second). Each unit follows [timestamp, <|vision_start|>, video tokens, <|vision_end|>], where timestamps start at 0.0 and increase by 1.0 second. The vision_start and vision_end positions serve as the start and end boundary positions, respectively, and discrete labels follow this one-second temporal alignment.

Stage 2 training schedule. Stage 2 uses 25,249 difficulty-focused samples selected from the 96,171 TimeLens training samples. All models are initialized from their Stage 1 adapters at step 1,200 and share an effective batch size of 32 and a maximum of 800 steps. Common settings are AdamW-8bit, a learning rate of 5e-5 with cosine decay, 0.03 warmup ratio, BF16 precision, gradient clipping at 1.0, and non-reentrant gradient checkpointing.

AuthorsOpenReview ↗

Thank you for your inspiring comments and questions.

Thank you for taking the time to review our paper. We would like to know if you have any further concerns and whether the additional experiments and results we have provided have addressed your previous concerns. We look forward to hearing from you.

Reviewer_jiY6OpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper casts MLLM-based video temporal grounding as single-pass span prediction. Hidden states at Qwen3-VL's per-second delimiters are projected and a softmax over timestamps gives start and end distributions. The causal mask is replaced with bidirectional attention. Training is LoRA-based in two stages: CE first, then CE plus a soft-IoU loss on difficulty-sampled data. Reported results on TimeLensBench are strong at 2B and 4B and mixed at 8B, with latency shown to be 1.4–2× lower than the generative backbone.

Critical strengths
  • The method is simple by using a single forward pass to produce distributions of start and end timestamps without any decoding steps.
  • The use of the delimiters is interesting.
Critical weaknesses
  • Oracle evaluation. The paper seems to have used ground-truth data to select final prediction (L307-308, "... select the betterperforming candidate for evaluation,..."). The justification ("practical multi-candidate retrieval") does not apply to an R@1 evaluation, and all baslines are evaluated with their raw predictions. Therefore, the main results reported are oracle performance and are thus unfair.

  • Limited contribution. None of the proposed components is new:

    • Single-pass discriminative prediction over bidirectional MLLM states appears in VideoITG [1]. It reports generative < causal < bidirectional, which is the paper's central claim and its Q1 ablation.
    • Localizing via a temperature-scaled softmax over frame-level LLM hidden states, rather than generating timestamps, appears in E.T. Chat [2] and MeCo [3], both with bidirectional attention over visual tokens.
    • DisTime [4] already predicts distributions for start and end timestamps.

    The contribution is therefore a combination of known components, thus the "new paradigm" framing is not supported.

  • No controlled baseline. The paper lacks generative SFT on the same backbone, data, LoRA budget and resolution, especially at 2B/4B. Without it, the gains cannot be attributed to the formulation rather than to training on TimeLens-100K.

  • Baselines also appear to use different input settings. Resolution alone moves mIoU by 2–4 points (Table 13).

  • Unmatched efficiency claims. Figure 4's mIoU values for the proposed models appear to match Table 1 (high resolution), while the latencies come from the 31-frame setting in Appendix A.6, since Figure 4 mIoU values seem to be too high to come from 31-frame inference. Please clarify which configuration each axis reflects.

Questions for authors
  • Were the ground-truth spans used to pick final prediction of the model?
  • See Unmatched efficiency claims in the weaknesses.
5 comments

We thank Reviewer jiY6 for the detailed and constructive review. We appreciate the recognition of our method’s simplicity, efficiency(S1), and novel use of video-sequence delimiters in MLLMs(S2). Below, we address the key concerns point by point.

W1&Q1. Oracle evaluation in the main results

The paper seems to have used ground-truth data to select final prediction (L307-308, "... select the better-performing candidate for evaluation,..."). The justification ("practical multi-candidate retrieval") does not apply to an R@1 evaluation, and all baselines are evaluated with their raw predictions. Therefore, the main results reported are oracle performance and are thus unfair.

Response. We agree that Recall@1 should use a single fixed decoding rule for fair comparison. We therefore report argmax results for the main R@1 comparison, which require no ground-truth information. Meanwhile, we retain the original results as Coverage@2, which measures how well the two decoding strategies can cover the ground-truth event. This provides an empirical assessment of the localization information encoded in the predicted probability distributions and an additional reference for downstream applications that may benefit from multiple candidate intervals. Detailed results are provided in Table 1, with a condensed version shown below.

ScaleDatasetBest BaselineOurs (Argmax)Ours (Coverage@2)
8BCharades-TL55.253.559.8
ActivityNet-TL54.149.953.9
QVHighlights-TL65.559.462.8
4BCharades-TL50.551.958.9
ActivityNet-TL46.848.852.9
QVHighlights-TL62.259.563.2
2BCharades-TL46.549.656.4
ActivityNet-TL39.946.451.2
QVHighlights-TL52.257.662.1

Importantly, our distribution-based paradigm predicts one probability distribution for each boundary in a single forward pass, preserving richer temporal localization information than a deterministic prediction. This distributional information can be further exploited by YOFO, whereas deterministic outputs from existing baselines do not provide the same flexibility. This naturally supports multiple decoding strategies and candidate construction without additional model inference. Here, we use argmax and soft decoding as two simple examples that exploit complementary information in the predicted distributions. YOFO remains competitive under strict Recall@1, while the original results further demonstrate the benefit of exploiting multiple decodings.

W2. Limited contribution relative to prior work

None of the proposed components is new:

  • Single-pass discriminative prediction over bidirectional MLLM states appears in VideoITG [1]. It reports generative < causal < bidirectional, which is the paper's central claim and its Q1 ablation.
  • Localizing via a temperature-scaled softmax over frame-level LLM hidden states, rather than generating timestamps, appears in E.T. Chat [2] and MeCo [3], both with bidirectional attention over visual tokens.
  • DisTime [4] already predicts distributions for start and end timestamps. The contribution is therefore a combination of known components, thus the "new paradigm" framing is not supported.

Response. We thank the reviewer for the precise references. We acknowledge the similarities and will clarify the distinctions below.

  • VideoITG [1] uses bidirectional attention for frame-level binary selection, followed by a second VLM pass for answer generation. In contrast, YOFO directly predicts start/end boundaries in a single forward pass, without frame selection or a second VLM.
  • E.T. Chat [2] uses two special tokens to represent the start/end boundaries and matches their features against the video sequence for localization. In contrast, YOFO directly derives start/end probability distributions from delimiter representations at each video position.
  • MeCo [3] introduces structured special tokens, such as <ent> and <tst>, to model event and transition structures for temporal localization. In contrast, YOFO requires no structured token generation and directly predicts sequence-level boundary distributions in a single forward pass.
  • DisTime [4] predicts start/end probability distributions from a single special token. In contrast, YOFO performs sequence-level prediction, producing distributions directly over the original video representation sequence.

These works employ carefully designed mechanisms for VTG: VideoITG uses VLM-based frame selection followed by another VLM pass, while the other three introduce special-token structures and corresponding mechanisms. This motivates a simpler question: can MLLMs perform VTG more directly, using their native prediction capability? YOFO provides an answer: without additional special outputs or prediction/matching modules, it uses a single forward pass to directly predict sequence-level start/end probability distributions over the original video representation sequence. This design is simple and fast while achieving competitive performance, demonstrating the value of a more direct way to leverage MLLMs for VTG.

W3. Missing controlled generative baseline

The paper lacks generative SFT on the same backbone, data, LoRA budget and resolution, especially at 2B/4B. Without it, the gains cannot be attributed to the formulation rather than to training on TimeLens-100K.

Response. We agree that this is an important missing control and have conducted the requested experiment. We train generative SFT baselines on the same TimeLens-100K data, using the same Qwen3-VL backbone, Q-LoRA budget, and low resolution as YOFO, with standard timestamp-generation prompts. This provides a controlled comparison between autoregressive timestamp generation and single-pass distribution prediction.

DatasetQwen3-VL SFT-2BYOFO-2BQwen3-VL SFT-4BYOFO-4B
Charades-TL44.4448.0947.8551.97
ActivityNet-TL39.0243.5941.5544.83
QVHighlights-TL51.2153.9152.1256.50

Under matched training conditions, YOFO (argmax-only decoding) consistently outperforms generative SFT, with larger gains at stricter IoU thresholds. This controlled comparison supports the benefit of our single-pass distribution prediction formulation over autoregressive timestamp generation. More broadly, the results suggest that standard text-generation SFT alone is insufficient to effectively enhance MLLMs for VTG, and existing approaches often resort to RL-based optimization (e.g., TimeLens). In contrast, YOFO provides a simpler SFT-based paradigm that directly enhances MLLM VTG capability through sequence-level distribution prediction.

W4. Different input settings across baselines

Baselines also appear to use different input settings. Resolution alone moves mIoU by 2–4 points (Table 13).

Response. We clarify the input settings used for our baselines. As shown in Table 13, input resolution has a notable impact on VTG performance, so we carefully account for this factor in our comparisons. For methods already evaluated on TimeLens-Bench, we report their best reported results (Time-R1, TimeLens, OMTG, Marlin, and the Qwen-VL series). For other models, we follow their official evaluation settings to avoid distribution shifts between training and evaluation caused by forcing a uniform input resolution. Our resolution also matches the setting adopted by TimeLens, ensuring consistency with its evaluation protocol. We will explicitly clarify these settings in the revised table caption.

W5&Q2. Unmatched efficiency claims

Figure 4's mIoU values for the proposed models appear to match Table 1 (high resolution), while the latencies come from the 31-frame setting in Appendix A.6, since Figure 4 mIoU values seem to be too high to come from 31-frame inference. Please clarify which configuration each axis reflects.

Response. The reviewer is correct that the current Figure 4 combines accuracy results from the high-resolution setting in Table 1 with latency measurements from the unified 31-frame, low-resolution setting in Appendix A.6. To ensure a fair comparison of accuracy and efficiency across models, we replace all mIoU values with results obtained under the low-resolution setting using argmax decoding, as summarized below.

MetricOurs-2BOurs-4BOurs-8BQwen3-VL-2BQwen3-VL-4BQwen3-VL-8BTimeLens-8B
High-resolution mIoU49.5651.8553.5143.445.648.355.2
Low-resolution mIoU48.0951.9751.7537.5046.0646.0352.71
Δ (Low − High)−1.47+0.12−1.76−5.90+0.46−2.27−2.49

We also replace the results for the Qwen3-VL and TimeLens models, which originally used the same high-resolution setting as our method, with their low-resolution results. For the other baselines, their original configurations already use relatively low resolutions; therefore, we retain their original mIoU values from Table 1 without modification.

The latency measurements remain unchanged. All models are evaluated under low-resolution settings with approximately comparable sequence lengths, ensuring consistency between the accuracy and latency comparisons. The revised results are detailed below.

ModelmIoULatencyFLOPs
Ours-2B48.09263 ms23.8 T
Ours-4B51.97443 ms45.9 T
Ours-8B51.75669 ms74.7 T
Qwen3-VL-2B37.50519 ms23.8 T
Qwen3-VL-4B46.06702 ms46.1 T
Qwen3-VL-8B46.03939 ms74.9 T
TimeLens-8B52.711,165 ms74.9 T
Marlin-2B46.5823 ms15.5 T
DisTime-8B42.51,770 ms84.4 T
EvoGround-7B40.32,759 ms72.0 T
VideoMind-2B28.2823 ms11.0 T
TaRO44.86,173 ms88.4 T

As shown above, YOFO maintains a favorable accuracy-efficiency trade-off, and the original conclusions remain unchanged. Figure 4 has been updated accordingly.

Reviewer_EvkkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Using MLLMs for video temporal grounding (VTG) will face this problem: generating temporal boundaries autoregressively as textual tokens creates a mismatch with interval-level localization and incurs sequential decoding overhead. To overcome this problem while still utilizing MLLMs’ strong comprehension ability, the authors reformulate VTG as probability prediction. They use bidirectional video-query representation sequence to predict start- and end-boundary distributions within a single forward pass. The authors report that the proposed method achieves competitive performance on TimeLens-Bench with lower inference latency.

Critical strengths
  • S1: The idea is reasonable; the motivations for their designed components are clearly described.
  • S2: The results show a clear improvement in inference speed.
Critical weaknesses
  • W1: The authors "obtain two candidate intervals from the highest-probability and soft boundaries in Eq. 8, and select the better-performing candidate for evaluation". Do the other baselines also use such a strategy? If not, the comparison is unfair. The authors do not report the performance of using only the highest-probability boundaries or only the soft boundaries.
  • W2: There are some works with similar ideas [1,2], making the novelty and contribution questionable. The authors should compare with or at least discuss.

[1] How Should Video LLMs Output Time? An Analysis of Efficient Temporal Grounding Paradigms. arXiv:2604.08966, 2026.

[2] E.M.Ground: A Temporal Grounding Vid-LLM with Holistic Event Perception and Matching. arXiv:2602.05215, 2026.

Questions for authors
  • In Table 1, the proposed method is less competitive on QVHighlights-TL. Can the authors discuss this?
  • Better to show some OOD tests beyond TimeLens-Bench.
3 comments

We thank Reviewer Evkk for the positive assessment of our method (S1) and for recognizing the inference-speed advantage of the proposed paradigm (S2). We address each weakness and question below.

W1 & Q1. Single-pass Distribution Prediction, Multiple Decoding, and QVHighlights Weakness Analysis

W1: The authors "obtain two candidate intervals from the highest-probability and soft boundaries in Eq. 8, and select the better-performing candidate for evaluation". Do the other baselines also use such a strategy? If not, the comparison is unfair. The authors do not report the performance of using only the highest-probability boundaries or only the soft boundaries.

Q1: In Table 1, the proposed method is less competitive on QVHighlights-TL. Can the authors discuss this?

Response.

We appreciate the reviewer's concern. A key advantage of our distribution-based prediction paradigm is that our model predicts one probability distribution for each boundary in a single forward pass, preserving rich temporal localization information rather than producing a deterministic prediction. These distributions naturally support diverse decoding strategies and multiple candidate construction without additional model inference. In this work, we simply adopt two of the most straightforward strategies, argmax and soft decoding, to exploit complementary information in the distributions. More sophisticated decoding strategies could further exploit this flexibility, whereas deterministic predictions from existing baselines do not offer the same capability.

We agree that Recall@1 should use a single fixed decoding rule for fair comparison. We therefore replace Table 1 with the argmax results and report the original results separately as Coverage@2 below. Coverage@2 measures how well the two decoding strategies can cover the ground-truth event, providing an empirical assessment of the localization information encoded in the predicted probability distributions and an additional reference for downstream applications that may benefit from multiple candidate intervals. YOFO remains competitive under strict Recall@1, while the difference between argmax and Coverage@2 demonstrates the complementary information captured by the two decoding strategies. Detailed results are provided in Table 1, with a condensed version shown below.

ScaleDatasetBest Baseline mIoUOurs (Argmax) mIoUOurs (Coverage@2) mIoU
8BCharades-TL55.253.559.8
ActivityNet-TL54.149.953.9
QVHighlights-TL65.559.462.8
4BCharades-TL50.551.958.9
ActivityNet-TL46.848.852.9
QVHighlights-TL62.259.563.2
2BCharades-TL46.549.656.4
ActivityNet-TL39.946.451.2
QVHighlights-TL52.257.662.1

As noted by Reviewer Q1, our performance is slightly weaker than other baselines on QVHighlights and ActivityNet. We further analyzed this issue and found that spans longer than 50s account for over 15% of samples in QVHighlights and ActivityNet, whereas they account for less than 1% of the training data. Unlike baselines that directly generate text with MLLMs, our MLLM paradigm still exhibits a generalization gap on such long spans and requires fine-tuning to better adapt to long-span prediction.

To investigate this, we conducted a simple experiment on the 4B model under the low-resolution setting, using argmax decoding for evaluation. We repeatedly sampled training examples with spans longer than 50s in the 800-step Stage-2 training data, increasing their proportion to 6%. We compare this setting with the original training setup below.

Training SettingActivityNet-TL mIoUQVHighlights-TL mIoUCharades-TL mIoU
Original44.856.552.0
Long-span Aug.46.559.951.5

Simply increasing the proportion of long-span samples from less than 1% to 6% improves mIoU by 1.7 and 3.4 points on ActivityNet and QVHighlights, respectively, while slightly reducing performance on Charades by 0.5 points.

W2. Relation to related work [1,2]

There are some works with similar ideas [1,2], making the novelty and contribution questionable. The authors should compare with or at least discuss.

Response. We thank the reviewer for pointing us to these works and will include them in the revision.

Relation to [1] (Continuous Temporal Decoding). [1] investigates three temporal output paradigms, among which only the third, Continuous Temporal Decoding, is conceptually related to YOFO. Its distribution-based instantiation uses a single temporal token and an MLP decoder to predict boundary distributions, whereas YOFO performs sequence-level prediction, deriving boundary logits from delimiter representations at each time-anchored video position to form distributions directly over the video representation sequence. YOFO requires no additional prediction module, relying only on bidirectional attention and a small subset of the native LM-head weights.

Relation to [2] (E.M.Ground). [2] autoregressively generates a holistic event token (<evt>) and matches it against video features for localization. In contrast, YOFO directly predicts start/end distributions over the video representation sequence, replacing event-to-frame matching with sequence-level boundary prediction.

We clarify that YOFO's contribution lies in single-pass, sequence-level boundary-distribution prediction, rather than non-autoregressive temporal grounding in general.

Q2. Out-of-domain evaluation beyond TimeLens-Bench

Better to show some OOD tests beyond TimeLens-Bench.

Response. We appreciate the concern about distributional generalization. Given the limited rebuttal time, we are unable to conduct additional benchmark evaluations. Nevertheless, our current zero-shot evaluation already involves datasets unseen during training.

Specifically, TimeLens-100K train dataset is built from CosmoCap, DiDeMo, HiREST, InternVid-VTime, and QueryD, whereas TimeLens-Bench eval dataset comprises three distinct datasets: Charades (short indoor daily activities), ActivityNet (diverse open-domain videos with longer events), and QVHighlights (news/vlog videos with highlight-centric queries). These datasets introduce substantial domain, duration, event-length, and query-style shifts.

AuthorsOpenReview ↗

Thank you for your inspiring comments and questions.

Thank you for taking the time to review our paper. We would like to know if you have any further concerns and whether our responses have addressed your previous concerns. We look forward to hearing from you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.