acceptodds
Under review as a conference paper at ICLR 2027

TROVE: LEARNING TO RE-OBSERVE VIDEOS FOR TEMPORAL GROUNDING

Abstract

Video temporal grounding requires global context to identify a queried occurrence and fine-grained local evidence to resolve its boundaries. Uniform frame sampling provides temporal coverage, but distributes additional observations across both relevant and irrelevant regions. We propose TROVE, a framework that learns temporal grounding through targeted re-observation. A shared policy first predicts a coarse interval from sparse global frames, then refines it using dense local evidence selected by that interval together with sparse global support. We train this trajectory through Grounding Initialization followed by Joint Re-Observation Learning. A Geometry-to-Metric Reward Curriculum supplies graded feedback for disjoint predictions before emphasizing overlap quality and evaluation thresh- olds. Contrastive Re-Observation Calibration (CRC) trains the first action using both its localization reward and the difference between targeted and uniform second-observation rewards. The uniform comparator is used only during training. On TimeLens-Bench, TROVE with a 4B backbone improves mIoU over the native model by 11.7, 8.5, and 4.5 points on the Charades, ActivityNet, and QVHighlights subsets, respectively. Experiments across three model scales, training and observation ablations, and boundary analyses demonstrate the value of learning temporal localization together with targeted evidence acquisition.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Temporal Grounding, Multimodal Large Language Models, Adaptive Video Sampling, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_5Ku3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes TROVE, a two-stage framework for video temporal grounding. Stage 1 predicts a coarse temporal interval using sparse global frames, and Stage 2 refines the prediction using global and locally sampled frames. The method uses GRPO with a geometry-to-metric reward and a uniform comparator to improve training. The results show consistent improvements across several benchmarks.

Critical strengths
  1. The two-stage re-observation framework is intuitive and interesting.

  2. The geometry-to-metric reward provides useful feedback when the predicted interval does not overlap with the ground truth.

  3. The paper provides extensive experiments across different model sizes and datasets.

Critical weaknesses
  1. Limited Stage 2 ablations. The current experiments do not fully isolate the benefits of targeted sampling and combining global and local frames.

  2. Inference efficiency. The method requires two VLM inference calls, but the additional inference cost is not sufficiently analyzed.

  3. Frame count and resolution trade-off. Using up to 448 frames per stage may introduce substantial computational overhead or require a trade-off with spatial resolution.

  4. Limited analysis of tGIoU. Although different reward functions are compared, the experiments do not directly show how tGIoU reduces the number of non-overlapping predictions.

Questions for authors
  1. Could the authors compare Stage 1 + Uniform Stage 2 with Stage 1 + Targeted Stage 2, using the same frame budget and number of model calls? Additionally, could they compare local-only and global+local inputs for Stage 2?

  2. What is the actual inference latency and computational cost compared with single-stage methods? Is the performance improvement sufficient to justify two VLM calls?

  3. Could the authors provide an ablation on the trade-off between input frame count, spatial resolution, and grounding accuracy?

  4. Could the authors report the percentage of predictions with tIoU = 0, with and without tGIoU-based training? This would help demonstrate whether tGIoU effectively improves predictions that initially miss the ground truth.

3 comments

Thank you for your thoughtful review and helpful suggestions. Below, we provide controlled re-observation ablations, inference cost measurements, frame–resolution analyses, and zero-overlap statistics. Unless otherwise specified, the TROVE experiments in this response use Qwen3.5-4B as the backbone.

Q1: Re-Observation Ablations with Matched Calls and Frame Budgets

Answer: Thank you for proposing these controls to isolate the value of re-observation. Here, Stage 1 and Stage 2 refer to the two inference steps: Global Grounding and Re-Observation, respectively. Re-Observation uses a fresh context without the Global Grounding answer or reasoning; the initial interval guides refinement through the visual frames it selects.

Table 2 already compares uniform and targeted observation using the same jointly learned policy. Targeted (global+local) performs better than Uniform 3 fps while using fewer cumulative frames on average (264.1 versus 274.3), although Uniform uses one call and Targeted uses two.

We therefore added controls that share the model weight and cached Global Grounding predictions, with Re-Observation frame counts and visual grids matched for each query (mIoU):

Inference configurationCharadesActivityNetQVHighlights
Stage 1 + Uniform Stage 255.054.466.7
Stage 1 + Local-only Stage 255.855.466.2
Stage 1 + Targeted Stage 2 (global+local)57.457.669.2

Targeted improves over Uniform by 2.4/3.2/2.5 points on Charades/ActivityNet/QVHighlights, and global+local improves over local-only by 1.6/2.2/3.0 points. These results support both targeted allocation and retaining global context. Each configuration includes the common Global Grounding call, so call counts and cumulative frame counts match. Uniform Re-Observation spreads the matched number of frames over the full video, without using the initial prediction to choose locations; its fps therefore varies by query. Thus, the gains persist when call counts and visual input sizes are controlled, supporting the value of the selected evidence and the global+local composition.

Q2: Inference Cost and the Value of a Second Call

Answer: Thank you for asking us to quantify the practical inference cost. We repeated the latency measurements with separate timing for Global Grounding and Targeted Re-Observation. Latency was measured on AMD Instinct MI308X GPUs with batch size 1. Over three runs of all 9,404 queries, TROVE averages 8.19 s/query versus 5.11 s for native Qwen3.5-4B (1.60×). The breakdown is:

Component (s/query)Qwen3.5-4BTROVE: Global GroundingTROVE: Targeted Re-ObservationTROVE total
Video Sampling and Preprocessing3.602.562.755.31
Visual encoding0.210.180.200.38
Language-model prefill0.640.590.641.23
Autoregressive decoding0.600.580.581.17

Each observation individually is faster than native Qwen3.5-4B: preprocessing and model calls take 3.96 s for Global Grounding and 4.22 s for Targeted Re-Observation, versus 5.10 s for native Qwen3.5-4B.

We additionally estimate model-forward FLOPs over all 9,404 queries under the same settings:

Estimated computation (TFLOPs/query)Qwen3.5-4BTROVE: Global GroundingTROVE: Targeted Re-ObservationTROVE total
Visual encoding16.9217.0017.0334.03
Language-model prefill60.0953.5158.14111.65
Autoregressive decoding0.160.160.160.32

These results support the additional inference cost when localization accuracy is the priority. The full framework gains 11.7/8.5/4.5 mIoU points over the native model on Charades/ActivityNet/QVHighlights (Table 1), at 1.60× latency and 1.89× estimated forward FLOPs. To distinguish the value of re-observation from training gains, Table 2 also compares inference strategies using the same jointly learned policy: Targeted improves over single-call Uniform 2 fps by 1.5/1.9/1.1 points. Figure 5 further shows that refinement creates 1,059 new successes at tIoU≥0.7 while losing 543, a net gain of 516. These consistent gains support spending an additional call on boundary refinement for accuracy-oriented use, with the same model serving both observations.

Q3: Frame Count–Resolution Trade-off

Answer: Thank you for highlighting the interaction between frame count and spatial resolution. We use the jointly learned TROVE-4B model with the paper's sampling protocol: 1-fps Global Grounding, followed by 1-fps global support plus 4-fps local candidates under a 2-fps-equivalent Re-Observation budget. The window rule is unchanged. We vary the per-call frame ceiling and scale the per-frame and aggregate pixel-area budgets together to 0.5× or 1× the original settings. Each table entry reports mIoU (%) in the order Charades/ActivityNet/QVHighlights.

Per-call frame ceilingTROVE, 0.5× pixel budgetsTROVE, 1.0× pixel budgets
12856.6/52.1/64.257.7/53.7/65.3
25656.6/54.5/67.857.7/57.6/69.2
44856.6/54.7/67.757.7/57.6/69.2

Increasing the pixel budget improves all three subsets at every tested frame ceiling. Raising the ceiling from 128 to 256 mainly benefits the longer-video ActivityNet and QVHighlights subsets; increasing it further to 448 gives little additional benefit under these sampling rates. We also adapt the same trained model to the larger TimeLens2 visual setting: Global Grounding uses 2 fps, and Re-Observation combines 2-fps global support with 8-fps local candidates under a 4-fps-equivalent budget, with up to 2,048 frames per call and the original window rule. This configuration achieves 58.9/60.7/69.5 mIoU. Together, these results support balancing temporal coverage and spatial detail: TROVE is effective under the original compact setting and can further benefit from a larger visual budget when sampling density and the frame ceiling are increased together.

Q4: Effect of tGIoU on Zero-Overlap Predictions

Answer: Thank you for this suggestion. tGIoU provides training-time feedback for non-overlapping predictions that receive identical zero tIoU rewards. During the initial geometry-only phase, 297/2,560 rollouts (11.6%) receive this additional feedback. Comparing the Grounding Initialization models in Figure 3, the zero-overlap rate is 14.03% with tIoU-only training versus 13.77% with our Geometry-to-Metric curriculum across all 9,404 queries. The curriculum also improves mIoU by 0.4/0.7/0.7 points on Charades/ActivityNet/QVHighlights, supporting its progression from informative geometric supervision to precise boundary localization.

Separately, for the paper’s jointly trained TROVE-4B, the zero-overlap rate decreases from 14.36% after Global Grounding to 12.68% after Re-Observation, with reductions on all three subsets. These outcomes reflect the learned localization policy and its observed evidence. The reward ablation therefore evaluates the curriculum’s contribution, while the two-call analysis demonstrates the complete framework’s ability to recover missed events through targeted re-observation.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer 5Ku3,

Thank you again for your constructive suggestions. In response, we have added re-observation controls with matched call counts and frame budgets, quantified inference costs, and provided frame–resolution analyses and zero-overlap statistics.

If you have an opportunity to revisit our response, we would appreciate your thoughts on the additional evidence. Please let us know if any further explanation would be helpful; we would be happy to follow up.

Thank you for your time and consideration.

Reviewer_Af9kOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TROVE, a two-call temporal video grounding framework that uses an initial interval prediction to guide targeted re-observation, combining sparse global frames with denser local frames for refinement. It introduces a Geometry-to-Metric Reward Curriculum and Contrastive Re-Observation Calibration (CRC) to jointly optimize localization and observation decisions. Experiments across three TimeLens-Bench subsets and multiple model configurations, together with ablations and boundary-level analyses, demonstrate consistent improvements over the corresponding baselines.

Critical strengths
  1. The paper addresses a meaningful challenge in temporal grounding: balancing global context with fine-grained boundary localization under a limited frame budget. Its coarse-to-fine framework is logically coherent, with initial predictions guiding targeted re-observation while the training objectives encourage more effective observation decisions.

  2. The experimental evaluation is reasonably comprehensive, covering three benchmark subsets and multiple model scales. Ablations of observation strategies, reward designs, and CRC, together with boundary-level analyses, provide useful evidence for the effectiveness of the proposed framework.

Critical weaknesses
  1. The methodological novelty of the training framework remains insufficiently established. The proposed reward curriculum and CRC appear to rely largely on combinations or adaptations of existing techniques, and their distinct contributions beyond these combinations are not sufficiently clear.

  2. The computational cost of training is insufficiently reported. Since the proposed training framework involves additional model rollouts for targeted and uniform re-observation, the lack of training-time or compute-cost measurements makes it difficult to assess the overhead of the method.

Questions for authors
  1. Could the novelty of the proposed reward curriculum and CRC be clarified beyond their combination of existing techniques?

  2. Could an accuracy comparison under matched cumulative frame budgets, or a brief budget–performance analysis, be provided? Additionally, what are the average end-to-end inference latency and computational cost of the two-call re-observation process?

  3. How sensitive is TROVE to the observation-window parameters and sampling rates? A brief sensitivity analysis or further justification of the selected settings would help clarify the robustness of the method.

  4. Could the contribution of retaining sparse global frames alongside dense local frames be clarified? A comparison of global-only, local-only, and merged observations would help isolate the benefit of this design.

5 comments

Thank you for your careful review and helpful suggestions. Below, we clarify the reward curriculum and CRC, quantify training and inference costs, and provide budget, sensitivity, and global–local observation analyses. Unless otherwise specified, the TROVE experiments in this response use Qwen3.5-4B as the backbone.

Q1: Distinct Contributions of the Reward Curriculum and CRC

Answer: Thank you for asking us to clarify what is distinctive about the reward curriculum and CRC. Our contribution is the grounding-specific supervision of two coupled roles of the initial interval: localizing the event and selecting evidence for subsequent boundary refinement.

Geometry-to-Metric curriculum. The design addresses two different learning needs: distinguishing disjoint intervals early in training and improving precise localization later. It transfers supervision from tGIoU to an objective combining continuous overlap with smooth rewards around the evaluation thresholds. The continuous term distinguishes predictions within a recall band, while the threshold terms encourage crossing the boundaries used to measure success. Figure 3 already tests this distinction: the curriculum improves mIoU over a tGIoU-to-tIoU schedule by 0.7/1.3/0.6 points on Charades/ActivityNet/QVHighlights, supporting the metric-aligned endpoint beyond a transition between existing overlap rewards.

CRC. The design addresses credit assignment to the interval as an observation decision. An absolute downstream reward scores the final answer but does not explicitly assess whether the selected evidence helps relative to uniform observation. CRC uses the targeted-minus-uniform grounding reward under the same query, policy, and frame ceiling, together with the initial interval's own localization reward. This trains the interval for both accuracy and relative observation utility, while the refined prediction receives its own grounding reward. Figure 4 already isolates the value of this calibration over the self-plus-targeted return, showing 0.6/1.0/1.4-point mIoU gains. Thus, the curriculum specifies how localization quality is rewarded as learning progresses, and CRC specifies how downstream evidence utility supervises the observation decision; the ablations support each design separately.

Q2: Frame Budgets and Training–Inference Costs

Answer: Thank you for requesting a budget–performance analysis and actual inference costs. Table 2 already compares observation strategies at similar average cumulative frame counts: with the jointly learned policy, Targeted uses 264.1 frames versus 274.3 for Uniform 3 fps, while achieving higher mIoU on all three subsets (57.7/57.6/69.2 versus 57.5/55.8/67.6). This supports the value of targeted evidence allocation at a comparable frame cost.

We further varied the total frame ceiling across Global Grounding and Targeted Re-Observation, keeping the trained model fixed. The table reports Targeted mIoU (%):

Cumulative frame ceilingCharadesActivityNetQVHighlights
12857.751.162.4
25657.755.567.7
44857.757.669.0

Charades changes little as the ceiling increases, whereas ActivityNet and QVHighlights benefit substantially. This pattern is consistent with their video durations: the median is 30.7 seconds for Charades, compared with 141.0 and 150.0 seconds for ActivityNet and QVHighlights in this evaluation. Longer videos require more frames to maintain global coverage alongside local detail, making a tight cumulative budget more restrictive. Together with Table 2, these results support allocating evidence selectively while retaining enough frames for both observations.

For inference cost under the paper's original setting of up to 448 frames per call, three runs over all 9,404 queries give an average end-to-end latency of 8.19 s/query for TROVE versus 5.11 s for native Qwen3.5-4B (1.60×). Estimated model-forward computation is 145.99 versus 77.17 TFLOPs/query (1.89×); Global Grounding and Targeted Re-Observation contribute 70.67 and 75.33 TFLOPs, respectively. For training on 32 AMD Instinct MI308X GPUs, Grounding Initialization requires 264.23 GPU-hours, while Joint Re-Observation Learning requires 203.70 GPU-hours, giving a combined training cost of 467.93 GPU-hours. To quantify the additional cost of CRC relative to the equal-weight Rg+RtgtR_g + R_{\mathrm{tgt}} return, the joint-training cost is 203.70 versus 193.95 GPU-hours, an increase of 9.75 GPU-hours.

TROVE therefore offers an accuracy–cost trade-off through targeted re-observation with a shared model: Table 2 supports its evidence-allocation advantage, and the budget sweep shows how the frame allowance affects that benefit across datasets.

Q3: Sensitivity to Window and Sampling Parameters

Answer: Thank you for asking how the window and sampling choices affect the results. We varied one Re-Observation parameter at a time, keeping the trained policy and Global Grounding predictions fixed. All settings use a maximum of 448 frames per model call. The table reports mIoU ranges (%) over the tested alternatives; the default achieves 57.7/57.6/69.2 on Charades/ActivityNet/QVHighlights.

ParameterDefaultAlternativesCharadesActivityNetQVHighlights
Padding ratio ρ0.340.20, 0.27, 0.41, 0.5057.5-57.757.5-57.668.9-69.2
Duration divisor K3216, 6457.1-58.057.5-57.768.6-69.2
Global-support fps10.5, 1.556.9-58.057.5-57.668.9-69.2
Local-candidate fps42, 656.6-57.857.3-57.668.6-69.2

The parameters differ in sensitivity. The effect of ρ is small because the base padding rule often already determines the context: the default ρ cap is inactive for 76.8% of queries. K and sampling rates have a larger effect, particularly on Charades, whose median target duration is 3 seconds: reducing local sampling from 4 to 2 fps lowers mIoU by 1.1 points. This is consistent with short actions requiring sufficient temporal detail and surrounding transitions. On ActivityNet and QVHighlights, increasing either sampling rate above the default does not improve accuracy under the fixed observation budget.

These results support the complementary roles of global coverage, boundary context, and local detail. Their combination reduces dependence on some individual window parameters, while preserving the importance of evidence allocation. The shared default balances these needs across all three subsets without dataset-specific tuning.

Q4: Contributions of Global and Local Evidence

Answer: Thank you for suggesting ablations of global and local evidence. We keep the jointly learned policy and Global Grounding predictions fixed. Global-only uniformly samples the full video; local-only uniformly samples only the observation window expanded from the initial interval using the paper's window rule. Both retain the two-call, fresh-context inference protocol. In the supplementary comparison, we match Re-Observation frame counts and visual grids per query across global-only, local-only, and global+local; sampling rates therefore vary by query. The table compares these ablations and reports mIoU:

Re-Observation evidenceCharadesActivityNetQVHighlights
Global-only55.054.466.7
Local-only55.855.466.2
Targeted: global+local57.457.669.2

Global+local achieves the highest mIoU on all three datasets: global+local reaches 57.4/57.6/69.2, improving over global-only by 2.4/3.2/2.5 points and local-only by 1.6/2.2/3.0. The advantage supports the rationale for TROVE's observation design: concentrate detail around the event while retaining the surrounding context needed to resolve its boundaries.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer Af9k,

Thank you again for your helpful suggestions. In response, we have clarified the roles of the reward curriculum and CRC, reported training and inference costs, and added budget, sensitivity, and global–local observation analyses.

If you have an opportunity to revisit our response, we would appreciate hearing whether these additions help address your concerns. Should any aspect remain unclear, we would be happy to elaborate.

Thank you for your time and consideration.

Reviewer_sw6gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TROVE, a two-stage framework for video temporal grounding. It first predicts a coarse interval from sparse frames and then re-observes the relevant region with denser sampling for refinement. The authors also introduce a reward curriculum and CRC to improve grounding and observation selection. Experiments on three TimeLens-Bench subsets show consistent improvements across different model sizes.

Critical strengths
  1. The idea is simple and well-motivated. Using coarse predictions to guide finer observations makes sense for temporal grounding, where uniform sampling can waste frames on irrelevant regions.
  2. The experimental results are solid. The method shows consistent gains across three datasets and model sizes. The ablations also help demonstrate the benefits of targeted re-observation and the proposed training objectives.
Critical weaknesses
  1. Limited methodological novelty. The overall framework mainly combines existing ideas, including coarse-to-fine sampling, GRPO-based optimization, and reward shaping. I do not see a substantial architectural or theoretical contribution. The additional gains over grounding initialization are also relatively modest, which makes the contribution of the proposed joint learning less convincing.
  2. The efficiency benefits are not sufficiently demonstrated. TROVE requires two inference passes, but the paper mainly reports frame counts rather than actual latency or computational cost. It is unclear whether the improvements justify this additional overhead.
Questions for authors
  1. How much of the improvement comes from joint learning rather than targeted sampling alone?
  2. Could the authors report inference latency or computational cost compared with single-pass baselines?
  3. How often does re-observation hurt an already accurate prediction, and what causes these failures?
  4. What is the key methodological difference between TROVE and existing coarse-to-fine observation methods, particularly REVISOR?
  5. Are the improvements from CRC consistent across different random seeds? Given the relatively small gains, it would be helpful to understand their stability.
3 comments

Thank you for your thoughtful review and constructive feedback. Below, we clarify TROVE’s methodological contributions and report analyses of joint learning, inference cost, refinement failures, and CRC stability across training seeds. Unless otherwise specified, the TROVE experiments in this response use Qwen3.5-4B as the backbone.

Q1: Joint Learning Beyond Targeted Sampling

Answer: Thank you for asking how much joint learning adds beyond targeted sampling alone. Table 2 quantifies this contribution by comparing the initialized and jointly learned policies under the 

same inference strategies. With Targeted inference, joint learning adds 1.6/2.4 mIoU points on ActivityNet/QVHighlights, while Charades remains comparable (57.9 to 57.7). These are the 

additional gains over applying targeted sampling to the initialized policy. Under Uniform 2 fps and Uniform 3 fps, the corresponding ActivityNet/QVHighlights gains are smaller: 0.5/1.5 and 

0.6/1.5 points, respectively, with Charades changing by −0.1 in both cases.

This pattern is consistent with our training objective: joint learning optimizes the complete grounding and re-observation trajectory, rewarding the initial interval for the utility of its selected evidence and training refinement on that evidence. The larger gains under Targeted inference on ActivityNet and QVHighlights support jointly learning evidence selection and boundary refinement, beyond simply adding targeted sampling at inference.

Q2: Accuracy–Cost Trade-off

Answer: Thank you for highlighting the cost of the additional observation. We now report measured end-to-end latency and estimated forward computation, including comparisons with the trained Grounding Initialization policy using single-pass uniform inference at both 2 and 3 fps. Measurements use AMD Instinct MI308X GPUs with batch size 1 and a maximum of 448 frames per call. 

Model / inference strategyCharades mIoUActivityNet mIoUQVHighlights mIoULatency (s/query)Estimated TFLOPs/query
Native Qwen3.5-4B, Uniform 2 fps46.049.164.75.1177.17
Grounding Initialization, Uniform 2 fps56.355.266.64.8277.17
Grounding Initialization, Uniform 3 fps57.655.266.15.5379.20
TROVE, Targeted Re-Observation57.757.669.28.19145.99

Comparison with trained single-pass baselines. Relative to Initialization with Uniform 2 fps, TROVE improves mIoU by approximately 1.4/2.4/2.6 points on Charades/ActivityNet/QVHighlights, at 1.70× latency and 1.89× estimated forward FLOPs. Against the denser Uniform 3 fps baseline, TROVE achieves comparable Charades accuracy and improves ActivityNet/QVHighlights by 2.4/3.1 points, at 1.48× latency and 1.84× estimated forward FLOPs. These comparisons quantify the deployment trade-off between the initialized single-pass model and the complete trained TROVE framework.

Does denser uniform sampling provide the same benefit? Increasing Initialization inference from 2 to 3 fps raises latency from 4.82 to 5.53 s/query and estimated computation from 77.17 to 79.20 TFLOPs/query. Its subset-level mIoU changes by +1.3/0.0/−0.5 points on Charades/ActivityNet/QVHighlights, respectively. Under the same per-call frame cap, increasing uniform sampling from 2 to 3 fps does not consistently improve localization across the three subsets.

For TROVE, Global Grounding and Targeted Re-Observation take approximately 3.96 and 4.22 s/query, respectively, before the remaining end-to-end overhead, and contribute 70.67 and 75.33 TFLOPs/query. Thus, the second observation introduces a measurable cost. The results support this cost when localization accuracy is the priority, particularly on ActivityNet and QVHighlights. TROVE offers this accuracy–cost trade-off using a shared model for both observations, without requiring a separate refinement model.

Q3: Refinement Failures and Their Causes

Answer: Thank you for pointing out that refinement may harm an already accurate prediction. Section 4.5 and Figure 5 already report this trade-off: at tIoU≥0.7, Re-Observation produces 1,059 new successes and loses 543, yielding a net gain of 516 across all 9,404 queries. To further quantify the effect on initially accurate predictions, we summarize the same evaluation outputs: among 3,814 Global Grounding predictions with tIoU≥0.7, 1,398 (36.7%) have lower tIoU after refinement, and 543 (14.2%) fall below 0.7; 3,271 (85.8%) remain at or above this threshold.

The failure cases already presented in Figures 10–11 highlight two challenges: preserving the full extent of compound events and resolving ambiguous boundaries of short events. For a compound event, refinement can focus on a visually salient sub-action and omit another constituent required by the query. For a short event with a gradual onset, a small, visually plausible boundary shift can substantially reduce tIoU; Figure 11 shows a one-second shift for a two-second event. These cases highlight the need to preserve all queried sub-actions and distinguish event onset from preparatory motion during refinement. The aggregate results support the value of re-observation: 85.8% of initially successful predictions remain successful, while the 1,059 newly successful predictions substantially exceed the 543 lost successes. Figure 5 further shows positive net gains at every evaluated threshold on every subset. Thus, refinement yields positive net gains across all three subsets and evaluated thresholds, while still reducing accuracy for some individual predictions.

Q4: Methodological Differences from REVISOR

Answer: Thank you for encouraging a clearer comparison with existing coarse-to-fine methods. A common paradigm in adaptive/coarse-to-fine video observation is to use a coarse understanding to select a local clip for closer inspection. In representative multi-turn approaches, the retrieved clip is appended to the existing context, and reasoning continues with accumulated evidence and prior outputs. Often developed for video QA, this process seeks evidence sufficient to answer the query. For temporal grounding, local event details and surrounding transitions are both relevant to determining the event's full temporal extent. TROVE therefore combines dense local evidence with sparse global support for boundary refinement.

REVISOR follows a multimodal QA-reflection paradigm: it proposes a review interval and appends a denser local clip to the existing dialogue, retaining the initial video and reasoning. It learns review selection through DADR. However, its released VTG path ends after the interval prediction, without applying this clip-review loop.

TROVE explicitly learns and executes a grounding–re-observation–refinement trajectory. The initial interval selects dense local evidence, which is combined with sparse global support for boundary prediction by the same policy in a fresh context, without carrying over the initial answer or reasoning. Thus, the initial prediction influences refinement through evidence selection. Our contribution is to jointly learn where to observe and how to refine temporal boundaries through this grounding-specific feedback.

Q5: CRC Stability Across Training Seeds

Answer: Thank you for asking whether CRC's gains are stable across training seeds. We compare Rg+RtgtR_g + R_{\mathrm{tgt}} with CRC using five seeds. The table reports final Re-Observation mIoU (%, mean ± sample standard deviation):

Global-action returnCharadesActivityNetQVHighlights
Rg+RtgtR_g + R_{\mathrm{tgt}}57.7 ± 0.656.5 ± 0.467.5 ± 1.0
CRC57.9 ± 0.457.4 ± 0.169.4 ± 0.5

CRC achieves higher mean mIoU on all three subsets, improving by 0.2/0.9/1.9 points on Charades/ActivityNet/QVHighlights, with lower standard deviations in each case. The advantage is more pronounced on ActivityNet and QVHighlights, while Charades shows a smaller average gain. These results provide supporting evidence for CRC's relative-reward design, particularly on the two subsets with clearer gains.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer sw6g,

Thank you again for your constructive feedback. In our response, we have clarified the contribution of joint learning and the differences from REVISOR, quantified the accuracy–cost trade-off, and provided refinement failure analyses and CRC results across five training seeds.

If you have an opportunity to revisit our response, we would appreciate your thoughts on these clarifications and additional results. We would be happy to discuss any remaining concerns or provide further details.

Thank you for your time and consideration.

Reviewer_w4VEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes TROVE, a two-stage framework for video temporal grounding. It first predicts a coarse temporal interval based on sparsely sampled global frames, and subsequently refines this prediction using dense local observations while preserving global context. Geometry-to-Metric Reward Curriculum and the CRC module enhance the standard tIoU-based reward evaluation, ensuring that the subsequent observations effectively enhance temporal localization. Experimental results on the TimeLens-Bench benchmark with Qwen3.5 models demonstrate that this method enhances the precision of temporal boundary localization.

Critical strengths
  1. It is interesting to model temporal grounding as a sequential observation decision question.
  2. Geometry-to-Metric Reward Curriculum and CRC improve the reward evaluation based on the tIoU, helping to measure the quality of interval localization and whether subsequent observations guided by that interval truly improve localization.
Critical weaknesses
  1. The comparison with existing methods is limited. The comparison with existing methods in Table 1 is limited, and many seem to be MLLM baselines. It is recommended to add comparisons with some recent methods, such as TimeLens2.
  2. TROVE requires a two-step trajectory, which may incur additional training and inference overhead. Although Table 2 lists the total number of input frames, it does not provide the actual inference time or computational cost.
Questions for authors
  1. The comparison with existing methods in Table 1 is somewhat limited, and many of the listed approaches appear to be MLLM baselines. It is recommended to include comparisons with other recent methods (such as TimeLens2) and to conduct evaluations using open-source MLLM backbones other than Qwen.
  2. The two-stage framework entails an additional model inference step. The authors should provide data on inference time and computational overhead to clarify the trade-off between accuracy and efficiency.
  3. How does TROVE perform on longer videos? If possible, provide some results for long videos.
  4. The observation window expansion ratio ρ\rho is now fixed at 0.34 based on nominal density allocation. It is recommended to conduct a sensitivity analysis to determine whether this parameter is truly optimal and how it affects grounding performance.
3 comments

Thank you for your careful review and constructive suggestions. Below, we provide additional baseline and backbone comparisons, inference cost measurements, and analyses of longer videos and window sensitivity. Unless otherwise specified, the TROVE experiments in this response use Qwen3.5-4B as the backbone.

Q1: Stronger Baselines and Backbone Generalization

Answer: Thank you for suggesting stronger comparisons and broader backbone coverage. We added comparisons with TimeLens2 and completed both training stages on InternVL3.5-2B/4B.

TimeLens2 settings. TimeLens2 uses 4-fps sampling with a 2,048-frame limit. We adapt TROVE to this setting: Global Grounding uses uniform 2-fps sampling, while Targeted Re-Observation combines 2-fps global support with 8-fps local candidates under a 4-fps-equivalent frame budget.

DatasetModel[email protected][email protected][email protected]mIoU
CharadesTimeLens2-2B74.260.633.653.4
TimeLens2-4B80.366.637.557.7
TimeLens2-8B80.968.038.958.6
TROVE-4B82.168.340.358.9
ActivityNetTimeLens2-2B71.162.043.255.4
TimeLens2-4B74.665.547.259.0
TimeLens2-8B74.065.646.658.6
TROVE-4B76.267.049.460.7
QVHighlightsTimeLens2-2B81.172.857.867.0
TimeLens2-4B83.375.260.069.3
TimeLens2-8B84.476.161.270.2
TROVE-4B83.174.960.769.5

TROVE-4B exceeds TimeLens2-4B in mIoU on all three subsets. Compared with TimeLens2-8B, it performs better on Charades and ActivityNet, while trailing on QVHighlights.

Our paper settings. Following TimeLens, we evaluate the released TimeLens2 models at 2 fps with a 448-frame limit:

DatasetModel[email protected][email protected][email protected]mIoU
CharadesTimeLens2-2B72.057.831.351.7
TimeLens2-4B77.363.135.855.7
TimeLens2-8B78.865.036.756.4
TROVE-4B81.066.738.457.7
ActivityNetTimeLens2-2B70.761.143.655.4
TimeLens2-4B73.864.645.557.9
TimeLens2-8B73.864.045.657.7
TROVE-4B72.363.246.957.6
QVHighlightsTimeLens2-2B82.174.259.268.3
TimeLens2-4B83.575.160.469.5
TimeLens2-8B84.176.062.070.2
TROVE-4B82.974.260.469.2

Under this smaller visual budget, TROVE-4B exceeds TimeLens2-4B by 2.0 mIoU points on Charades and is within 0.3 points on ActivityNet and QVHighlights. 

TROVE achieves this performance using only 2,500 samples across both training stages, with a frozen visual encoder and a shared policy for Global Grounding 

and Re-Observation. Together with the gains under the larger visual budget, these results support learning evidence selection and boundary refinement 

as an effective way to improve grounding with limited task-specific training data, without requiring a separate refinement model.

Extension to InternVL3.5. We trained both Grounding Initialization and Joint Re-Observation Learning using the same 2,500 training examples. 

Native and Initialization use uniform 2-fps inference; TROVE uses Global Grounding followed by Targeted Re-Observation, with a 448-frame limit per call.

DatasetModel[email protected][email protected][email protected]mIoU
CharadesInternVL3.5-2B14.07.13.011.2
+Initialization48.933.518.236.4
+TROVE54.838.420.239.1
InternVL3.5-4B42.727.513.629.3
+Initialization60.545.525.443.0
+TROVE63.749.026.945.0
ActivityNetInternVL3.5-2B12.07.34.410.0
+Initialization39.829.720.831.6
+TROVE48.337.424.636.2
InternVL3.5-4B23.816.610.218.2
+Initialization48.537.123.335.8
+TROVE53.141.926.039.5
QVHighlightsInternVL3.5-2B19.111.96.915.2
+Initialization59.046.530.844.7
+TROVE65.052.032.647.8
InternVL3.5-4B26.216.710.120.1
+Initialization65.352.333.548.4
+TROVE71.358.239.153.1

Relative to Initialization, TROVE improves mIoU on all three subsets. Although absolute performance remains below Qwen3.5, the consistent gains at both scales 

show that the same training recipe also benefits InternVL3.5. Together, these comparisons support TROVE's ability to learn effective evidence acquisition with 

limited task-specific data and to extend across the tested multimodal architectures.

Q2: Inference Latency and Computational Cost

Answer: Thank you for asking us to quantify the practical inference cost. We repeated the latency measurements with separate timing for Global Grounding and 

Targeted Re-Observation. Latency was measured on AMD Instinct MI308X GPUs with batch size 1. Over three runs of all 9,404 queries,

TROVE averages 8.19 s/query versus 5.11 s for native Qwen3.5-4B (1.60×). The breakdown is:

Component (s/query)Qwen3.5-4BTROVE: Global GroundingTROVE: Targeted Re-ObservationTROVE total
Video Sampling and Preprocessing3.602.562.755.31
Visual encoding0.210.180.200.38
Language-model prefill0.640.590.641.23
Autoregressive decoding0.600.580.581.17

Each observation individually is faster than native Qwen3.5-4B: preprocessing and model calls take 3.96 s for Global Grounding and 

4.22 s for Targeted Re-Observation, versus 5.10 s for native Qwen3.5-4B. 

We additionally estimate model-forward FLOPs over all 9,404 queries under the same settings:

Estimated computation (TFLOPs/query)Qwen3.5-4BTROVE: Global GroundingTROVE: Targeted Re-ObservationTROVE total
Visual encoding16.9217.0017.0334.03
Language-model prefill60.0953.5158.14111.65
Autoregressive decoding0.160.160.160.32

TROVE therefore uses 1.89× estimated forward FLOPs and 1.60× measured end-to-end latency, alongside the 11.7/8.5/4.5-point mIoU gains on 

Charades/ActivityNet/QVHighlights in Table 1. These gains reflect the complete trained framework. TROVE provides this accuracy–cost trade-off 

through two calls to a shared policy, without requiring a larger model or a separate refinement model.

Q3: Performance on Longer Videos

Answer: Thank you for raising the question of applicability to longer videos. Our 2,500 training examples contain videos lasting 5.6–475.8 s, 

with 98.4% shorter than five minutes; TimeLens-Bench videos span 7.1–267.0 s.  Our current study therefore focuses on temporal grounding 

in videos lasting seconds to several minutes. We have not yet validated the method on hour-long videos, which are not covered by this benchmark.

Within this scope, we separately analyze the 406 queries with video duration ≥224 s (approximately 3.7 minutes), all from ActivityNet. Using the original 

evaluation predictions, TROVE achieves 54.56 mIoU versus 44.02 for native Qwen3.5-4B, and [email protected] improves from 29.80% to 45.57%. 

The gains at both mIoU and [email protected] support the value of allocating finer observations around the predicted event while retaining full-video context, 

including on the longer videos within our evaluated range.

Q4: Sensitivity to the Window Expansion Ratio

Answer: Thank you for asking about the rationale and sensitivity of ρ. This parameter limits unnecessary expansion of the local re-observation window: 

for video duration D and predicted interval length ℓ, padding on each side is capped at max(0, (ρD−ℓ)/2). It limits the additional region sampled densely, 

without forcing the window to reach ρD or truncating longer predictions. The sampler independently enforces the 2-fps-equivalent frame budget and 448-frame ceiling.

We chose ρ=0.34 as a density-allocation reference: combining 1-fps global support with 4-fps local observation under a 2-fps reference budget 

gives (2−1)/(4−1)=1/3. We then varied ρ while keeping the Global Grounding predictions fixed. The table reports mIoU (%):

ρCharadesActivityNetQVHighlights
0.2057.657.668.9
0.2757.557.669.0
0.3457.757.669.2
0.4157.757.569.1
0.5057.757.569.1

The modest sensitivity is consistent with how the window is constructed. At ρ=0.34, the base padding rule already determines the window 

for 7,218 of 9,404 queries (76.75%), so the ρ cap is inactive. Another 18.10% have predicted spans at least ρD and receive no padding; 

only 5.15% have positive padding reduced by the cap. Thus, ρ acts as an expansion safeguard rather than a target window size. 

Together with the independent frame budget, this design provides neighboring evidence for boundary refinement while controlling observation cost, without relying on a narrowly tuned ρ in the tested range.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer w4VE,

Thank you again for your thoughtful comments. In response, we have added comparisons with TimeLens2 and InternVL backbones, quantified inference costs, and examined longer videos and window sensitivity.

If you have an opportunity to revisit our response, we would appreciate hearing whether these additions help address your concerns. Please let us know if further details would be useful; we would be happy to clarify.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.