acceptodds
Under review as a conference paper at ICLR 2027

Grounding with Confidence: Controllable Generative Video Temporal Grounding

Abstract

Video temporal grounding supports applications such as video search, content review, and automated editing by localizing events described in natural language. Yet existing generative models typically output timestamps without explicit interval-level confidence scores to guide candidate selection. We separate candidate generation from acceptance by scoring individual intervals within the original decoding pass. A lightweight confidence head reads pooled decoder states, providing an explicit score trained for interval selection. Offline verifier scores supervise the head on fixed candidate sequences, and temporal-overlap labels adapt it to current rollouts during reinforcement learning. GT-anchored candidate-pool supervision and set-level optimization train the generator. The resulting scores support ranking, threshold-based selection, and rejection without invoking an external verifier at inference. On a fixed OMTG-Bench candidate pool, confidence raises query-macro [email protected] from 9.95% to 14.42% over generation order at a 10% global return budget, and from 26.48% to 31.12% at a 25% budget. The continuous scores let downstream applications adjust return budgets or acceptance thresholds to match their precision–recall preferences, without regenerating candidate intervals.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Temporal Video Grounding, Reinforcement Learning, Vision Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QuoEOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper introduces an interval-level confidence interface for video temporal grounding. It separates candidate generation from acceptance by scoring individual intervals within the original decoding pass. A confidence head is proposed to support ranking, threshold-based selection, and rejection without an inference-time verifier.

Critical strengths
  1. Video temporal grounding with confidence has practical application value.
  2. Given this objective, the proposed combination of model design, SFT, and RL is reasonable.
Critical weaknesses
  1. The proposed approach seems a combination of existing techniques, with limited novelty and new insights (e.g., a lightweight MLP, supervision from offline verifier scores, and RL).
  2. How were the coefficients of the reward terms in Equ. 6 determined? Do the recall and F1 reward terms optimize overlapping objectives?
  3. Line 279 states, “The reported RL checkpoint was also selected after benchmark feedback.” Is checkpoint selection based on benchmark feedback fair across different comparisons? Could the authors also report results from a checkpoint selected without benchmark feedback?
  4. Although the paper primarily emphasizes grounding with confidence, the improvement brought by confidence in the last two columns of Table 3 is not significant: tF1 increases only from 67.07 to 67.39. This improvement is much smaller than the gain in the Raw setting from SFT to RL, where the score increases from 34.65 to 41.39.
  5. Table 2 includes relatively few methods.
Questions for authors

Please see weaknesses, especially W1, W3, W4.

Reviewer_xge5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper separates candidate generation from acceptance in generative video temporal grounding. A lightweight confidence head scores each interval using decoder states from the original generation pass, enabling ranking, threshold-based selection, and rejection without an inference-time verifier. The generator is trained with GT-anchored candidate-pool supervision and set-level reinforcement learning, while the confidence head learns from offline verifier scores and online temporal-overlap labels. Experiments demonstrate improved selection under limited return budgets and examine single-answer ranking and absent-query rejection.

Critical strengths

1. Clear motivation and practical design.

Separating timestamp generation from acceptance addresses a useful limitation of generative grounding models. Reusing decoder states allows different acceptance thresholds and return budgets to be applied without regenerating timestamps or running an additional verifier.

2. Controlled evidence for candidate selection.

The fixed-pool evaluation holds the generated intervals, NMS, and return budgets constant. Confidence improves query-macro [email protected] over generation order by 4.47 and 4.64 points at 10% and 25% budgets, respectively, with positive bootstrap intervals. This provides direct evidence that the learned scores improve selection independently of candidate generation.

Critical weaknesses

1. The main results lack fully independent held-out validation.

The RL checkpoint was selected after benchmark feedback, and the end-to-end results additionally use test-F1-optimal thresholds. Even the fixed-pool budgeted evaluation therefore conditions on a benchmark-selected model, while its bootstrap intervals exclude checkpoint-selection uncertainty. Although these choices are disclosed, an untouched evaluation set with validation-selected checkpoints and operating rules is needed to confirm the reported gains.

2. The benefit of the proposed training recipe over a simpler confidence readout remains unclear.

The fixed-pool experiments demonstrate useful learned scores, but do not establish whether offline verifier supervision and confidence-dependent generator training are necessary for these gains. A key missing baseline is the same confidence head trained directly on temporal-overlap labels for a fixed grounder. The GT-only reference and head-crossover analysis do not isolate these training choices. Matched comparisons should focus on the paper’s central budgeted-selection metrics, rather than final grounding F1 alone.

Questions for authors
  1. Evaluate on an untouched test set after selecting checkpoints and acceptance thresholds using separate validation data. Apply comparable selection protocols to the baselines, and report both fixed-pool budgeted-selection and end-to-end results.

  2. Compare the full method with the same confidence-head architecture trained directly on temporal-overlap labels for a frozen grounder, without verifier supervision or confidence-dependent generator training. Provide matched ablations to isolate these training components, focusing on recall under equal return budgets rather than final grounding F1 alone.

3 comments

Dear Reviewer xge5, thank you again for your constructive review and the two suggested controls. As the discussion deadline approaches, we wanted to follow up on our response, which reports both controls alongside additional checks on the selection protocol. If you have had a chance to review these results, we would appreciate your thoughts on whether they address your questions. We would be glad to clarify any remaining points before the deadline.

Thank you for the detailed responses and additional experiments. The checkpoint analysis and direct IoU-supervised head comparison sufficiently address my concerns regarding checkpoint selection and verifier supervision. I also appreciate the additional evidence demonstrating the effectiveness of the full method over simpler baselines. Overall, my concerns have been adequately addressed, and I will maintain my current rating of 3.

We're glad that this comparison and the checkpoint analysis addressed your concerns. Thank you for your thoughtful suggestions and for reviewing the additional experiments.

Reviewer_YZAZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses generative video temporal grounding with controllable prediction selection. Timestamp-only outputs lack reliability estimates for precision, especially recall adjustment and rejection of unsupported queries. The core innovation separates candidate generation from acceptance through interval-level confidence scores computed within the original decoding pass. A lightweight head pools decoder states and learns from offline verifier scores and online overlap labels, requiring no inference-time verifier. Ground-truth-anchored supervision preserves event coverage, while set-level reinforcement learning optimizes accepted intervals. On OMTG-Bench, recall increases from 9.95% to 14.42% at a 10% return budget, although additional F1 gains after NMS remain uncertain.

Critical strengths
  1. Clear practical motivation. Separating candidate generation from acceptance addresses the lack of reliability estimates in timestamp outputs, supporting ranking, adjustable return budgets, and rejection of unsupported queries.
  2. Coherent supervision design. Separate objectives reflect the different requirements of generation and confidence estimation, while ground-truth anchoring and detached confidence learning preserve coverage without encouraging unsupported predictions.
  3. Controlled evaluation of selection. Fixed candidate pools and equal return budgets isolate scoring quality, while head crossover, threshold transfer, and paired uncertainty analyses clarify the source of improvement.
Critical weaknesses
  1. Uncertain necessity of the full design. The contribution integrates established components, while confidence filtering adds only 0.32 F1 points after NMS, with a confidence interval crossing zero.
  2. Incomplete objective alignment. Interval overlap labels measure individual correctness rather than marginal set utility, and generation-order NMS can remove better candidates before confidence selection.
  3. Limited deployment validation. Test-optimized thresholds, benchmark-informed checkpoint selection, unsuccessful precision-target transfer, and synthetic negative queries constrain the practical interpretation of reported results.
Questions for authors

Please see the weakness.

5 comments

Thank you again for your thoughtful review. As the discussion deadline approaches, we wanted to ask whether you have had a chance to consider our response and the additional experiments. Regarding your concern in W1 about the limited improvement that confidence brings to overall performance, we would like to further clarify the central objective of this work and its practical significance.

The purpose of introducing confidence is not to improve balanced performance as measured by F1, but to enable adjustable decision boundaries that accommodate different real-world application scenarios.

Our motivation stems directly from these application needs.

  • For example, candidate screening for content moderation typically prioritizes higher recall, referring potentially violating segments for human confirmation to reduce the risk of violations escaping review.
  • In contrast, for user-facing clip recommendations or automated editing, false matches can cause irrelevant content to be presented, making precision a higher priority.

Experiments on OMTG demonstrate the adjustability of this confidence mechanism: CTVG’s precision/recall can be adjusted from a recall-oriented setting of 29.33%/73.64% to a precision-oriented setting of 69.32%/40.46%.

This shifts TVG from fixed decision boundaries to a paradigm in which those boundaries can be adjusted according to practical needs. This transition is crucial for real-world TVG applications.

Therefore, the core contribution of this work is to advance a confidence-aware paradigm for generative TVG, enabling it to accommodate the different precision–recall requirements of practical applications.

Dear Reviewer YZAZ,

As the discussion period draws to a close and we have not yet heard back from you, we kindly ask you to review our response. We sincerely hope you will consider the paradigm shift our work brings to TVG, recognize the significance and practical value of introducing a confidence-based paradigm to the community, and reconsider your overall score.

If you have any further concerns, we welcome further discussion and will respond promptly.

Thanks authors for the detailed reuttal. I agree that confidence offers value beyond best-point F1, as supported by the fixed-budget selection gains. The cross-fitted results also substantially alleviate my concern about test-set threshold tuning.

  1. Novelty relative to existing approaches. My earlier concern about integrating established components remains. The response clarifies the interface’s utility, but does not establish its methodological advance over existing grounding-confidence methods such as MTLA[1]. Comparisons against MTLA on identical candidate pools, using matched return budgets or false-alarm rates, would help establish the distinct benefit of the proposed design.

  2. Contribution of individual components. The new controls support the complete system’s effectiveness, but do not isolate which components justify its complexity. Controlled ablations that remove boundary alternatives, compare IoU-only with verifier supervision, and disable online head updates would clarify their individual contributions, with initialization, data, and inference settings held constant.

[1] Shalam, Daniel, et al. "Propose and Attend: Training-free MLLM Grounding Confidence via Multi-Token Localized Attention." arXiv preprint arXiv:2607.05978 (2026).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion Summary and Final Follow-up

We thank the Area Chair and all three reviewers (xge5, QuoE, and YZAZ) for their constructive engagement throughout the discussion phase. Below we summarize the final status of all concerns and the newly completed MTLA comparison and component attribution.

Summary of Discussion Progress

  • Reviewer xge5 confirmed that the checkpoint analysis, direct IoU-supervised head control, and comparisons against simpler baselines adequately addressed all stated concerns (maintaining rating 3).
  • Reviewer QuoE confirmed that most initial concerns were addressed, removed the accidentally checked ethics flag, updated the rating (1 → 2), and noted they would follow the discussion with YZAZ regarding methodological novelty (W1).
  • Reviewer YZAZ acknowledged the selection value beyond best-point F1 and that cross-fitting substantially alleviated threshold-tuning concerns, leaving two final questions: (1) a direct comparison against MTLA on identical candidates, and (2) clearer component attribution. We have now addressed both in our latest reply to YZAZ, summarized below for all reviewers and the AC.

Key Verified Results Added During Discussion

  1. Direct comparison against MTLA on identical candidates (new). On the exact same frozen L200 responses and 1,314 post-NMS candidates (320 OMTG-Bench queries), our single-pass learned confidence head outperforms the official training-free MTLA video default (first_local, c6379bb) across candidate discrimination, matched-budget precision, and cross-fitted F1—while reading already-computed decoder states with no attention-map extraction (preserving FlashAttention compatibility):

    • Candidate AUROC / AP: 0.818 / 84.67% vs. MTLA 0.737 / 75.76% (+0.081 / +8.91 pp; Logp AUROC 0.680, External 2B Verifier 0.750).
    • Matched-budget Precision: 94.66% vs. 81.68% (+12.98 pp) at 131 returns (124/131 vs. 107/131 distinct GT matches; >3× fewer false positives), 87.80% vs. 79.88% (+7.93 pp) at 328, 80.37% vs. 72.91% (+7.46 pp) at 657, and 69.24% vs. 67.31% (+1.93 pp) at 985.
    • 5-fold cross-fitted [email protected]: 67.32% vs. MTLA 66.66% (+0.66 pp).
    • Matched-calibration-FAR rejection (Charades video-disjoint deadline subset, 32 cal + 32 test pairs): At 5% / 10% calibration FAR targets, our head achieves 14/32 (43.8%) / 15/32 (46.9%) positive localization recall vs. 3/32 (9.4%) / 3/32 (9.4%) for MTLA, with test FAR 3/32 (9.4%) / 10/32 (31.3%) vs. 5/32 (15.6%) / 7/32 (21.9%).
  2. Component attribution and simplified core recipe.

    • Offline verifier is unnecessary: Across all 4 paired seeds on a frozen grounder (App. C Table 12), direct tIoU ≥ 0.5 supervision matches or exceeds offline verifier targets (primary seed 14.90 / 33.90 vs. 14.23 / 32.03 at 131 / 328 returns), allowing a simpler self-contained training recipe.
    • Core drivers established by matched controls: (i) Single-pass learned readout beats generation order (+4.47 / +4.64 [email protected] at 10% / 25% budgets), token log-prob, an external 2B verifier, and MTLA; (ii) generator training adds +2.95 / +5.32 [email protected] over frozen released TimeLens2-4B + IoU head across all 4 seeds (Table 4), and removing the pool-recall RL reward lowers [email protected] by −3.34 points; (iii) on identical final candidates, rollout-adapted head scoring lifts AUROC from 0.754 → 0.818 and 10%-budget recall from 12.80% → 14.42% (App. B.3 Table 10).
    • Evaluation rigor: 5-fold source-video cross-fitting gives 67.37 [email protected] (vs. 67.39 test-oracle); all pre-specified RL checkpoints (steps 125/150/200: 66.25 / 66.89 / 67.39) beat SFT and all baselines; and video-disjoint calibration-selected rejection beats token log-prob by 6.0–16.8 pp across Charades, ActivityNet, and QVHighlights (Table 6).
  3. Revision commitments. Because full retraining of 60,405-unit SFT or 200-step GRPO RL cannot finish in the final hours of discussion, we will explicitly narrow claims on boundary alternatives and full-training frozen-head RL, adopt the simplified IoU-only recommendation, and include both matched full retrains alongside the complete MTLA comparison in the revised manuscript.

We sincerely thank Reviewers YZAZ, QuoE, and xge5 and the Area Chair for their time and thoughtful feedback, and hope these final results provide a clear basis for the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.