- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes TROVE, a two-stage framework for video temporal grounding. Stage 1 predicts a coarse temporal interval using sparse global frames, and Stage 2 refines the prediction using global and locally sampled frames. The method uses GRPO with a geometry-to-metric reward and a uniform comparator to improve training. The results show consistent improvements across several benchmarks.
Critical strengths
-
The two-stage re-observation framework is intuitive and interesting.
-
The geometry-to-metric reward provides useful feedback when the predicted interval does not overlap with the ground truth.
-
The paper provides extensive experiments across different model sizes and datasets.
Critical weaknesses
-
Limited Stage 2 ablations. The current experiments do not fully isolate the benefits of targeted sampling and combining global and local frames.
-
Inference efficiency. The method requires two VLM inference calls, but the additional inference cost is not sufficiently analyzed.
-
Frame count and resolution trade-off. Using up to 448 frames per stage may introduce substantial computational overhead or require a trade-off with spatial resolution.
-
Limited analysis of tGIoU. Although different reward functions are compared, the experiments do not directly show how tGIoU reduces the number of non-overlapping predictions.
Questions for authors
-
Could the authors compare Stage 1 + Uniform Stage 2 with Stage 1 + Targeted Stage 2, using the same frame budget and number of model calls? Additionally, could they compare local-only and global+local inputs for Stage 2?
-
What is the actual inference latency and computational cost compared with single-stage methods? Is the performance improvement sufficient to justify two VLM calls?
-
Could the authors provide an ablation on the trade-off between input frame count, spatial resolution, and grounding accuracy?
-
Could the authors report the percentage of predictions with tIoU = 0, with and without tGIoU-based training? This would help demonstrate whether tGIoU effectively improves predictions that initially miss the ground truth.
3 comments
Q3: Frame Count–Resolution Trade-off
Answer: Thank you for highlighting the interaction between frame count and spatial resolution. We use the jointly learned TROVE-4B model with the paper's sampling protocol: 1-fps Global Grounding, followed by 1-fps global support plus 4-fps local candidates under a 2-fps-equivalent Re-Observation budget. The window rule is unchanged. We vary the per-call frame ceiling and scale the per-frame and aggregate pixel-area budgets together to 0.5× or 1× the original settings. Each table entry reports mIoU (%) in the order Charades/ActivityNet/QVHighlights.
| Per-call frame ceiling | TROVE, 0.5× pixel budgets | TROVE, 1.0× pixel budgets |
|---|---|---|
| 128 | 56.6/52.1/64.2 | 57.7/53.7/65.3 |
| 256 | 56.6/54.5/67.8 | 57.7/57.6/69.2 |
| 448 | 56.6/54.7/67.7 | 57.7/57.6/69.2 |
Increasing the pixel budget improves all three subsets at every tested frame ceiling. Raising the ceiling from 128 to 256 mainly benefits the longer-video ActivityNet and QVHighlights subsets; increasing it further to 448 gives little additional benefit under these sampling rates. We also adapt the same trained model to the larger TimeLens2 visual setting: Global Grounding uses 2 fps, and Re-Observation combines 2-fps global support with 8-fps local candidates under a 4-fps-equivalent budget, with up to 2,048 frames per call and the original window rule. This configuration achieves 58.9/60.7/69.5 mIoU. Together, these results support balancing temporal coverage and spatial detail: TROVE is effective under the original compact setting and can further benefit from a larger visual budget when sampling density and the frame ceiling are increased together.
Q4: Effect of tGIoU on Zero-Overlap Predictions
Answer: Thank you for this suggestion. tGIoU provides training-time feedback for non-overlapping predictions that receive identical zero tIoU rewards. During the initial geometry-only phase, 297/2,560 rollouts (11.6%) receive this additional feedback. Comparing the Grounding Initialization models in Figure 3, the zero-overlap rate is 14.03% with tIoU-only training versus 13.77% with our Geometry-to-Metric curriculum across all 9,404 queries. The curriculum also improves mIoU by 0.4/0.7/0.7 points on Charades/ActivityNet/QVHighlights, supporting its progression from informative geometric supervision to precise boundary localization.
Separately, for the paper’s jointly trained TROVE-4B, the zero-overlap rate decreases from 14.36% after Global Grounding to 12.68% after Re-Observation, with reductions on all three subsets. These outcomes reflect the learned localization policy and its observed evidence. The reward ablation therefore evaluates the curriculum’s contribution, while the two-call analysis demonstrates the complete framework’s ability to recover missed events through targeted re-observation.
Follow-up on Our Response
Dear Reviewer 5Ku3,
Thank you again for your constructive suggestions. In response, we have added re-observation controls with matched call counts and frame budgets, quantified inference costs, and provided frame–resolution analyses and zero-overlap statistics.
If you have an opportunity to revisit our response, we would appreciate your thoughts on the additional evidence. Please let us know if any further explanation would be helpful; we would be happy to follow up.
Thank you for your time and consideration.
Thank you for your thoughtful review and helpful suggestions. Below, we provide controlled re-observation ablations, inference cost measurements, frame–resolution analyses, and zero-overlap statistics. Unless otherwise specified, the TROVE experiments in this response use Qwen3.5-4B as the backbone.
Q1: Re-Observation Ablations with Matched Calls and Frame Budgets
Answer: Thank you for proposing these controls to isolate the value of re-observation. Here, Stage 1 and Stage 2 refer to the two inference steps: Global Grounding and Re-Observation, respectively. Re-Observation uses a fresh context without the Global Grounding answer or reasoning; the initial interval guides refinement through the visual frames it selects.
Table 2 already compares uniform and targeted observation using the same jointly learned policy. Targeted (global+local) performs better than Uniform 3 fps while using fewer cumulative frames on average (264.1 versus 274.3), although Uniform uses one call and Targeted uses two.
We therefore added controls that share the model weight and cached Global Grounding predictions, with Re-Observation frame counts and visual grids matched for each query (mIoU):
Targeted improves over Uniform by 2.4/3.2/2.5 points on Charades/ActivityNet/QVHighlights, and global+local improves over local-only by 1.6/2.2/3.0 points. These results support both targeted allocation and retaining global context. Each configuration includes the common Global Grounding call, so call counts and cumulative frame counts match. Uniform Re-Observation spreads the matched number of frames over the full video, without using the initial prediction to choose locations; its fps therefore varies by query. Thus, the gains persist when call counts and visual input sizes are controlled, supporting the value of the selected evidence and the global+local composition.
Q2: Inference Cost and the Value of a Second Call
Answer: Thank you for asking us to quantify the practical inference cost. We repeated the latency measurements with separate timing for Global Grounding and Targeted Re-Observation. Latency was measured on AMD Instinct MI308X GPUs with batch size 1. Over three runs of all 9,404 queries, TROVE averages 8.19 s/query versus 5.11 s for native Qwen3.5-4B (1.60×). The breakdown is:
Each observation individually is faster than native Qwen3.5-4B: preprocessing and model calls take 3.96 s for Global Grounding and 4.22 s for Targeted Re-Observation, versus 5.10 s for native Qwen3.5-4B.
We additionally estimate model-forward FLOPs over all 9,404 queries under the same settings:
These results support the additional inference cost when localization accuracy is the priority. The full framework gains 11.7/8.5/4.5 mIoU points over the native model on Charades/ActivityNet/QVHighlights (Table 1), at 1.60× latency and 1.89× estimated forward FLOPs. To distinguish the value of re-observation from training gains, Table 2 also compares inference strategies using the same jointly learned policy: Targeted improves over single-call Uniform 2 fps by 1.5/1.9/1.1 points. Figure 5 further shows that refinement creates 1,059 new successes at tIoU≥0.7 while losing 543, a net gain of 516. These consistent gains support spending an additional call on boundary refinement for accuracy-oriented use, with the same model serving both observations.