acceptodds
Under review as a conference paper at ICLR 2027

VerifyGround: Learning Verification-Guided Evidence Acquisition for Video Temporal Grounding

Abstract

Video temporal grounding requires visual evidence that identifies the queried event and resolves its temporal boundaries. Many existing multimodal large language model (MLLM)-based approaches predict temporal intervals from fixed global observations. Although these observations provide broad temporal coverage, they may be insufficient for accurate event retrieval and boundary refinement. We introduce VerifyGround, an interactive temporal grounding framework in which verification guides subsequent observation. The model first proposes a candidate interval, then inspects the corresponding video segment to verify its event match and boundary support. It acquires additional visual evidence from alternative temporal regions when the event hypothesis lacks support and gathers denser boundary evidence when the event is supported but its temporal extent remains uncertain. Verification thus guides prediction correction through targeted evidence acquisition, determining where and how to observe next. To learn this policy, we construct GroundStep-30K for cold-start supervised fine-tuning and introduce process-aware reinforcement learning to address the difficulty of assigning credit to intermediate corrections from final outcomes alone. Training combines the final localization objective with action-specific progress supervision, crediting each decision turn for the improvement produced by its corrective action. VerifyGround achieves state-of-the-art temporal grounding performance on Charades, ActivityNet, and QVHighlights, with a 3.4 points mIoU improvement over TimeLens-8B on QVHighlights. It also improves mIoU by 1.32 points over the same baseline on TACoS under cross-dataset zero-shot evaluation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Temporal Grounding, Turn-Level Credit Assignment, Reinforcement Learning, Large Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_o9pfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VerifyGround, an interactive framework for video temporal grounding. The model first generates an initial temporal interval, verifies whether the event and its boundaries are supported by visual evidence, and then decides whether to relocate the event, refine the boundaries, or return the current prediction.

The authors construct GroundStep-30K for supervised fine-tuning and introduce action-specific process rewards for reinforcement learning. The method improves over TimeLens-8B on three TimeLens benchmarks and reports additional zero-shot results on TACoS.

Critical strengths
  1. The distinction between event relocation and boundary refinement is useful.

The paper makes a reasonable distinction between finding the wrong event and finding the correct event with inaccurate boundaries. These two errors require different observations, and the proposed framework addresses them with separate actions. This is a sensible design for interactive temporal grounding.

  1. The ablation studies provide useful evidence for the training design.

The experiments do more than compare the final model with existing methods. In Table 3, adding action-specific process rewards improves average mIoU from 57.8 to 60.6 over outcome-only GRPO. Table 4 also shows that the full interactive policy outperforms a one-pass baseline under a matched visual-token budget. These results support the value of intermediate supervision and adaptive observation.

Critical weaknesses
  1. The comparison with existing methods is incomplete.

My main concern is that TimeLens2 is missing from Table 1, despite being cited in the paper. The published results of TimeLens2-8B on Charades-TimeLens, ActivityNet-TimeLens, and QVHighlights-TimeLens are 58.6, 58.6, and 70.2 mIoU, respectively, all higher than VerifyGround's 56.4, 56.3, and 68.9. Even TimeLens2-4B reports higher mIoU on these benchmarks.

Although differences in inference settings may affect direct comparisons, omitting such a relevant baseline weakens the paper's claim of state-of-the-art performance. In addition, the TimeLens-8B results reported in this paper differ from those reported in the TimeLens2 paper. A controlled comparison under the same evaluation protocol is needed.

  1. The effects of privileged supervision and the computational cost are not sufficiently examined.

Appendix A states that after two unsuccessful Relocate attempts, the teacher is provided with the ground-truth interval to generate a recovery trajectory. Although this information is removed from the student-visible input, the paper does not report how frequently this procedure is used or how much it contributes to the final results. An ablation without these recovery trajectories would help clarify their impact.

The paper also matches visual-token budgets when comparing interactive and one-pass inference, which is useful. However, multiple reasoning turns and additional video observations introduce computational overhead beyond the visual-token count. Without reporting actual inference latency or GPU costs, it is difficult to assess the practical efficiency of the proposed approach.

Questions for authors
  1. Why is TimeLens2 omitted from the main comparison despite being cited in the paper? Could the authors evaluate TimeLens2-4B and TimeLens2-8B under a comparable input and inference protocol, and clarify the differences in the reported TimeLens-8B results?

  2. What fraction of GroundStep-30K trajectories involve ground-truth-assisted recovery? How does performance change if these trajectories are excluded from supervised fine-tuning?

  3. Could the authors report average inference latency, GPU cost, and results across multiple random seeds? In particular, it would be useful to understand whether the gains from multi-turn evidence acquisition remain worthwhile when the additional computation is taken into account.

3 comments
AuthorsOpenReview ↗

We address the comparison, supervision, and efficiency concerns below.

We thank the reviewer for the careful assessment and for recognizing the distinction between event relocation and boundary refinement, as well as the ablation evidence supporting intermediate supervision and adaptive observation. We address the comparison, supervision, and efficiency concerns below.

1. TimeLens comparison and the 8B backbone.

Our Table 1 quotes TimeLens-8B's scores in TimeLens [1]; TimeLens2 reports different scores under its own evaluation configuration.

Table R1. Input configurations.

MethodFPSMax. framesMax. visual-token budget
TimeLens242,048≈128,000
VerifyGround and TimeLens [1]244814,336

Our training data derive from TimeLens-100K. TimeLens2 also uses additional TimeLens2-93K and Ego4D-NLQ training data and up to 100K-token packed SFT sequences. We re-evaluated TimeLens2 on the same settings as ours. For VerifyGround, this covers initial input and all subsequent observations. We also evaluated VerifyGround trained from Qwen3-VL-8B and Qwen2.5-VL-7B.

Table R2. Matched-budget mIoU

MethodCharades-TLActivityNet-TLQVHighlights-TLTACoS
TimeLens2-4B54.955.869.142.1
TimeLens2-8B55.755.868.741.9
VerifyGround-7B56.155.968.643.0
VerifyGround-8B56.856.068.443.2
VerifyGround-9B56.456.368.942.7

VerifyGround outperform both TimeLens2 variants on most of four benchmarks.

2. Frequency and contribution of GT-assisted recovery.

GT-assisted recovery accounts for 22.7% of GroundStep-30K. Excluding these trajectories from SFT reduces average mIoU across the three benchmarks by 2.1 points, demonstrating their contribution to student performance.

We further examined teacher failures by restricting the video input to the GT interval with a 10-second margin on each side, removing distant distractors. The teacher achieves IoU > 0.5 on 74.3% of the evaluated samples. This suggests that long-context search and competing events can prevent the teacher from attending to relevant evidence. GT-assisted recovery redirects it to overlooked segments and encourages prediction correction, turning otherwise unsuccessful attempts into useful training trajectories. Remaining failures may involve wrong annotations or unsampled key frames. Together, the ablation support retaining these challenging examples through annotation-assisted recovery.

Table R3. SFT ablation and teacher diagnostic.

AnalysisMetricResult
Remove recovery trajectoriesAverage mIoU change on three benchmarks vs. full SFT−2.1 points
GT-centered crop, ±10 s marginTeacher success rate (IoU > 0.5)74.3%

3. Serial inference latency, GPU cost, and random seeds.

We evaluated on Charades using NVIDIA H20 and SGLang with serial inference. SGLang's prefix/KV-cache reuse reduces repeated prefilling of shared history across turns. We repeated VerifyGround inference three times with different inference seeds. Three inference runs give:

Table R4. Inference-seed variability (mIoU).

Run 1Run 2Run 3Mean ± sample SD
56.456.155.756.07 ± 0.35

Table R5. Serial latency and estimated inference cost.

Methods/queryRelative latencyGPU-h/1K queries*
TimeLens, single turn5.911.00×1.64
VerifyGround, first turn only5.140.87×1.43
VerifyGround, multi-turn12.702.15×3.53

The additional latency reflects multi-turn evidence acquisition and verification. For efficiency-sensitive use, the same model can return its first-turn prediction without verification, using 0.87× TimeLens's time with mIoU only 2.7 points lower. We retain verification by default because it checks proposed intervals against newly acquired visual evidence, enabling correction of unsupported event hypotheses and temporal boundaries. This step is designed to reduce hallucinated localizations and improve reliability. The two modes offer a flexible accuracy–cost trade-off.

We will incorporate these comparisons, recovery analyses, and serial-inference measurements into the revision.

[1] Zhang J, Wang T, Ge Y, et al. Timelens: Rethinking video temporal grounding with multimodal llms[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026: 10419-10429.

Dear Reviewer o9pf,

We would like to thank you for your time and comments. We hope our previous response has adequately resolved your questions or concerns. As the deadline for the ICLR rebuttal period is approaching, we look forward to hearing your feedback on our response, and would be pleased to clarify any additional questions.

Best,

Authors

Dear Reviewer o9pf,

As the rebuttal deadline is approaching, we would like to briefly follow up on our response, which includes matched-budget comparisons with TimeLens2, an ablation of GT-assisted recovery trajectories, and inference latency, cost, and variability measurements.

Another reviewer has responded positively to the additional experimental evidence, and we would greatly value your perspective as well. If any concerns remain, we would appreciate the opportunity to clarify them before the deadline. If you find that the new evidence adequately addresses your concerns, we would be grateful if you could reconsider your assessment in light of these results.

Thank you again for your time and consideration.

Best regards, The Authors

Reviewer_3pbaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents VerifyGround, an interactive video temporal grounding framework that replaces fixed global observations with verification-guided evidence acquisition. Using a Qwen3.5-9B student base, the model proposes initial intervals and dynamically performs Relocate (coarse 1 fps sampling) for retrieval errors or Refine (dense 2 fps sampling) for boundary uncertainties. To train this, the authors construct GroundStep-30K via Qwen3.6-27B teacher supervision and optimize the policy using a process-aware GRPO algorithm combining trajectory-level outcomes with action-specific progress rewards. Experiments show state-of-the-art results across Charades, ActivityNet, QVHighlights, and TACoS under a controlled 14,336 token budget.

Critical strengths
  1. Targeted Evidence Decoupling & Fair Budget Control: Effectively separates retrieval and boundary errors via Relocate and Refine, backed by rigorous ablation under a strict 14,336 token budget matching single-pass baselines.
  2. Principled Process-Aware RL: Introduces continuous center-proximity and duration-normalized boundary potentials with EMA RMS scaling, successfully resolving multi-turn credit assignment bottlenecks.
Critical weaknesses
  1. Missing Inference Latency Analysis: Lacks wall-clock runtime and efficiency comparisons against one-pass baselines despite averaging 3.32 interaction turns.
  2. Privileged Hint Risks & Missing Thresholds: SFT trajectory construction uses ground-truth hints after repeated failures and omits exact stratification thresholds, risking hallucination during student training on brief actions.
  3. Backbone Discrepancies: Performance gains partially conflate the benefits of a stronger 27B teacher distillation and 9B base model compared to 7B/8B baselines.
Questions for authors
  1. What is the average wall-clock inference latency per query (at max turns = 4) compared to budget-matched one-pass models?
  2. What are the exact numerical values for τl and τh, and what percentage of GroundStep-30K trajectories triggered privileged ground-truth hints?
  3. What is the empirical distribution of test-time trajectories, and can the model recover if Refine reveals a false positive?
  4. Would performance gains hold if trained entirely on a standard 8B base model matching TimeLens-8B?
3 comments
AuthorsOpenReview ↗

The response adds latency data, validates GT hints, acknowledges recovery limitations with planned fixes, and confirms superior performance with smaller backbones.

We thank the reviewer for recognizing our separation of retrieval and boundary errors, controlled visual-token budget, and process-aware reinforcement learning.

1. Wall-clock inference latency.

On Charades, using NVIDIA H20 and SGLang with serial inference, VerifyGround takes an average of 12.7 seconds per query, approximately 2.15× the single-pass TimeLens baseline. Both use a total budget of 14,336 visual tokens; for VerifyGround, this covers the initial input and all subsequent observations.

Table R1. Serial latency and estimated inference cost.

Methods/queryRelative latencyGPU-h/1K queries*
TimeLens, single turn5.911.00×1.64
VerifyGround, first turn only5.140.87×1.43
VerifyGround, multi-turn12.702.15×3.53

For applications that prioritize latency, the same checkpoint can return its first-turn prediction without further verification in approximately 5.14 seconds/query, taking 0.87× TimeLens's time. Its mIoU is only 2.7 points below TimeLens. Users can therefore choose single-turn inference for faster responses or retain verification for higher localization accuracy.

2. Privileged hints, hallucination risk, and thresholds.

The stratification thresholds are τh=0.7\tau_h = 0.7 and τl=0.3\tau_l = 0.3. GT-assisted recovery accounts for 22.7% of GroundStep-30K.

To assess the contribution of these trajectories, we excluded them from SFT. Average mIoU across the three benchmarks decreased by 2.1 points, supporting their inclusion in training. To examine why the teacher needs these hints, we removed distant distractors from these consistently failing samples by restricting the input to the GT interval with a 10-second margin on each side. The teacher then achieves an IoU > 0.5 on 74.3% of these samples. Therefore, most failures occur because the teacher, due to its capability limitations, fails to accurately locate available evidence amidst the interference of long contexts and similar timestamps.

Table R2. SFT ablation and teacher diagnostic.

AnalysisMetricResult
Remove recovery trajectoriesAverage mIoU change vs. full SFT−2.1 points
GT-centered crop, ±10 s marginTeacher success rate (IoU > 0.5)74.3%

We fully understand the reviewer's concern that GT hints may induce unsupported localizations when brief actions are missed by frame sampling. We inspected the hint-assisted trajectories to check whether their predictions were supported by the sampled frames. Cases where a brief action was not sampled and the GT hint induced an unsupported localization accounted for approximately 3% of the inspected cases. These observations indicate that hints mainly help the teacher recover evidence it previously overlooked.

3. Test-time trajectories and recovery after Refine.

With a maximum of four turns, the test-time trajectory-length distribution is:

Number of turns234
Percentage of queries10%48%42%

To directly test recovery after Refine, we constructed 100 erroneous trajectory prefixes. No recovery was observed in these 100 cases. Our training trajectories place Relocate before Refine and contain no examples of returning to Relocate after false positive. We thank the reviewer for identifying this gap in trajectory construction. We will discuss it in the limitations and future-work section and plan to add trajectories that begin with Refine, reveal a false positive, and then switch to Relocate. These examples would explicitly supervise reconsideration of incorrect event hypotheses, with the aim of improving robustness to misleading prefixes.

4. Results with 7B and 8B backbones.

We trained VerifyGround with Qwen3-VL-8B and Qwen2.5-VL-7B. Under the same 14,336-visual-token budget, it outperforms TimeLens-8B on all three benchmarks:

MethodCharades-TLActivityNet-TLQVHighlights-TLTACoS
TimeLens-8B55.253.265.541.4
VerifyGround-7B56.155.968.643.0
VerifyGround-8B56.856.068.443.2
VerifyGround-9B56.456.368.942.7

The performance advantage therefore persists when the student scale is reduced from 9B to 8B and 7B.

We will add the latency measurements, hinting statistics and thresholds, trajectory distribution, recovery diagnostic, and backbone comparisons to the revision, together with the proposed improvement to trajectory construction.

Dear Reviewer 3pba,

We would like to thank you for your time and comments. We hope our previous response has adequately resolved your questions or concerns. As the deadline for the ICLR rebuttal period is approaching, we look forward to hearing your feedback on our response, and would be pleased to clarify any additional questions.

Best,

Authors

Thank you for the authors' detailed responses. Most of my concerns have been addressed, and I keep my positive score 3 Weak acceptance.

Reviewer_XWJiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VerifyGround, a verification-guided framework for video temporal grounding. The model iteratively verifies its predictions and uses Relocate or Refine to acquire additional visual evidence. The authors also introduce GroundStep-30K and action-specific process rewards for training. Experiments show strong results on three benchmarks and promising zero-shot generalization on TACoS.

Critical strengths
  1. I like the idea of separating event relocation from boundary refinement. The two actions address different types of grounding errors, and the ablations show that both contribute to performance.

  2. The method consistently outperforms strong baselines. The reward ablations are also convincing, showing that process-level supervision brings additional improvements over outcome-only RL.

Critical weaknesses
  1. While the action ablations are useful, I'm not fully convinced that the improvements specifically come from verification-guided observation rather than multi-turn observation itself. A comparison with a fixed multi-turn sampling baseline under the same visual-token budget would help better establish the contribution of the proposed mechanism.

  2. The multi-turn verification process may introduce considerable inference overhead. Although Table 5 studies different turn budgets, the paper does not report actual inference latency or computational cost. It remains unclear whether the accuracy gains justify the additional inference overhead compared to single-pass methods.

Questions for authors

Q1: Verification mechanism: Could the authors compare VerifyGround with a fixed multi-turn sampling baseline under the same visual-token budget? This would help isolate the benefit of verification-guided observation. Q2: Inference efficiency: Could the authors report the inference latency and computational cost compared to single-pass baselines? It would be useful to see the accuracy-efficiency trade-off. Q3: Have the authors compared the proposed action-specific rewards with a unified process reward, such as step-wise IoU improvement? This would help clarify whether the gains come from action-specific credit assignment or simply denser reward supervision. Q4: What percentage of GroundStep-30K trajectories involve the ground-truth-assisted recovery procedure described in Appendix A? Have the authors tried excluding these trajectories during SFT? I would be interested to know whether the performance remains similar without this recovery mechanism.

4 comments
AuthorsOpenReview ↗

Additional Baselines, Latency Analysis, and Ablation Studies Supporting VerifyGround’s Design Choices

We thank the reviewer for recognizing our separation of event relocation and boundary refinement, benchmark results, and process supervision.

Q1. Verification-guided observation versus fixed multi-turn sampling.

To separate the benefit of verification from repeated observation, we added a fixed multi-turn sampling baseline. It makes four observations per query, including the initial observation without conditioning subsequent observations on verification. VerifyGround uses verification to choose where and how to observe next. Both use the same initial observation.

Observation policyCharades-TLActivityNet-TLQVHighlights-TL
Fixed multi-turn sampling, 4 observations52.153.064.8
VerifyGround, 4 observations56.456.368.9

VerifyGround improves mIoU by 4.3/3.3/4.1 points. With observation count and visual budget matched, these gains support verification-guided evidence acquisition beyond repeated observation alone. Verification directs observations toward alternative events or uncertain boundaries according to the current error.

Q2. Inference latency and computational cost.

We evaluated on Charades using NVIDIA H20 and SGLang with serial inference. SGLang's prefix/KV-cache reuse reduces repeated prefilling of shared history across turns. We repeated VerifyGround inference three times with different inference seeds. The runs yield mean latency 12.7 seconds/query.

Methods/queryRelative latencyGPU-h/1K queries*
TimeLens, single turn5.911.00×1.64
VerifyGround, first turn only5.140.87×1.43
VerifyGround, multi-turn12.702.15×3.53

The additional latency reflects multi-turn evidence acquisition and verification. For efficiency-sensitive use, the same model can return its first-turn prediction without verification, using 0.87× TimeLens's time with mIoU 2.7 points lower. We retain verification by default because it checks proposed intervals against newly acquired visual evidence, enabling correction of unsupported event hypotheses and temporal boundaries. This step is designed to reduce hallucinated localizations and improve reliability. The two modes offer a flexible accuracy–cost trade-off.

Q3. Action-specific rewards versus step-wise IoU improvement.

We added a unified baseline using rt=IoU(st,g)−IoU(st−1,g)r_t = \mathrm{IoU}(s_t,g)-\mathrm{IoU}(s_{t-1},g) for both Relocate and Refine, where sts_t is the current proposal and gg the GT interval. We keep the SFT checkpoint, RL budget, initial-proposal reward, and final-outcome reward unchanged, with feedback at the same correction steps. Results are average mIoU across the three benchmarks.

Reward configurationAverage mIoU
Outcome-only RL57.8
Step-wise IoU improvement58.7
Action-specific process rewards (ours)60.6

Action-specific rewards outperform step-wise IoU improvement by 1.9 points. The gain supports credit assignment tailored to each action. In addition, when consecutive proposals both miss the target, IoU improvement is zero even if relocation moves closer. Our Relocate reward captures progress in center proximity. Refine instead rewards reduced start/end errors normalized by target duration, directly measuring boundary correction.

Q4. Frequency and contribution of GT-assisted recovery.

GT-assisted recovery accounts for 22.7% of GroundStep-30K. The GT interval is provided only to the offline teacher after two unsuccessful relocation attempts. The hint and direct references to it are removed from student-visible inputs. To measure the contribution of these trajectories, we excluded them from SFT. Average mIoU across the three benchmarks decreased by 2.1 points. Retaining these difficult recovery examples therefore improves student performance.

We further examined the teacher's failures by restricting its input to the GT interval with a 10-second margin on each side. This removes distant distractors and gives the teacher a temporal search prior. Under this setting, the teacher achieves IoU > 0.5 on 74.3% of the evaluated failure cases. This suggests that long-context search and competing events contribute to the original failures. Remaining failures may involve annotation issues, or unsampled key frames.

AnalysisMetricResult
Remove recovery trajectoriesAverage mIoU change on three benchmarks vs. full SFT−2.1 points
GT-centered crop, ±10 s marginTeacher success rate (IoU > 0.5)74.3%

GT-assisted recovery redirects the teacher to overlooked segments and turns failed attempts into correction trajectories. This surpasses the upper limit of the teacher model and provides more high-quality trajectories.

We will include the added comparisons, inference measurements, and recovery analysis in the revision.

Dear Reviewer XWJi,

We would like to thank you for your time and comments. We hope our previous response has adequately resolved your questions or concerns. As the deadline for the ICLR rebuttal period is approaching, we look forward to hearing your feedback on our response, and would be pleased to clarify any additional questions.

Best,

Authors

Thanks to the author for his detailed reply and additional experiments. The newly added comparative experiments and comparisons further enhance the experimental support of the proposed method. In addition, inference latency analysis also makes the computational overhead of the method clearer. However, I still have some concerns about the reasoning efficiency of the method. Multiple rounds of verification increased the inference time from 5.14 seconds to 12.70 seconds, but only brought about a 2.7 percentage point mIoU improvement. This indicates that the additional computational overhead is significant, and current experiments have not yet fully demonstrated that the accuracy gains reasonably offset this cost. Therefore, I decided to leave the original rating unchanged.

AuthorsOpenReview ↗

Clarification on Misunderstandings Regarding Inference Latency

Thank you for acknowledging the additional comparisons and latency analysis. We agree that the computational cost deserves careful consideration. However, we would like to clarify an ambiguity in our previous response: the 2.7-percentage-point mIoU improvement refers to the full method's advantage over TimeLens, not the improvement obtained by adding verification to our initial prediction.

The actual gains from multi-turn verification are substantially larger and consistent across all three datasets. The stage-wise mIoU results are:

DatasetInitial proposalAfter relocateAfter refineRelocate gainRefine gainTotal gain over initial
Charades51.852.756.4+0.9+3.7+4.6
ActivityNet49.954.256.3+4.3+2.1+6.4
QVHighlights62.367.068.9+4.7+1.9+6.6

Both relocation and refinement improve mIoU on every dataset. Thus, the additional rounds contribute meaningful improvements beyond the initial proposal. The results support verification as an important contributor to the full method's accuracy. Meanwhile, we preserve first-turn inference as a lower-latency option.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We appreciate your time and constructive feedback. Below is a summary of the review outcome and our responses to the main concerns.

Overall Outcome

The submission received two acceptances (3/3/2). Following our responses, Reviewer 3pba confirmed that "Most of my concerns have been addressed" and maintained their weak-acceptance rating. We also provided detailed responses and additional experiments addressing Reviewer o9pf's concerns about baseline comparisons, GT-assisted supervision, and computational cost.

Strengths Highlighted by Reviewers

  1. A meaningful distinction between retrieval and boundary errors. All three reviewers recognized that Relocate and Refine address different grounding errors through different observations, providing a sensible design for interactive temporal grounding.
  2. Principled process-level supervision. All three reviewers recognized the evidence supporting intermediate supervision; Reviewer 3pba highlighted the center-proximity and duration-normalized boundary rewards for multi-turn credit assignment.
  3. Strong empirical performance. Reviewer XWJi highlighted consistent improvements over strong baselines and promising zero-shot generalization on TACoS.
  4. Ablations supporting the design choices. Reviewer XWJi highlighted the contributions of both actions and the gains over outcome-only RL. Reviewers 3pba and o9pf recognized the controlled visual-budget comparisons supporting adaptive observation.
  5. Clear presentation. All three reviewers rated the paper as clear and easy to follow.

Main Concerns and Our Responses

  • Baseline completeness and backbone effects (o9pf, 3pba). We added TimeLens2-4B/8B comparisons under the same 14,336-visual-token budget, covering all VerifyGround observations. VerifyGround-9B outperforms TimeLens2-8B on all four benchmarks and TimeLens2-4B on three of four. We clarified the evaluation configurations and source of the original TimeLens scores. Additional 7B/8B student variants outperform TimeLens-8B on all four benchmarks, showing that the advantage persists at smaller student scales. (o9pf: Response 1; 3pba: Response 4.)

  • GT-assisted supervision and trajectory quality (all reviewers). We reported that recovery trajectories account for 22.7% of GroundStep-30K. GT hints are supplied only to the offline teacher and removed from student-visible inputs. Excluding these trajectories reduces average mIoU by 2.1 points, supporting their training value. We also provided teacher-failure diagnostics, trajectory-quality inspection results, and trajectory-construction details. (o9pf: Response 2; 3pba: Response 2; XWJi: Q4.)

  • Inference cost and variability (all reviewers). On Charades with NVIDIA H20 and SGLang serial inference, VerifyGround takes 12.7 seconds/query versus 5.91 for TimeLens (2.15x), with an estimated cost of 3.53 GPU-hours per 1,000 queries. Three inference-seed runs yield 56.07 ± 0.35 mIoU (mean ± sample SD). First-turn-only inference from the same checkpoint takes 5.14 seconds/query, with mIoU 2.7 points below TimeLens, providing a lower-latency option. These measurements make the accuracy-cost trade-off explicit. (o9pf: Response 3; 3pba: Response 1; XWJi: Q2.)

  • Verification versus repeated observation (XWJi). With four observations and a matched visual budget, VerifyGround improves over fixed multi-turn sampling by 4.3/3.3/4.1 mIoU points on Charades/ActivityNet/QVHighlights, supporting verification-guided evidence acquisition beyond repeated observation alone. (XWJi: Q1.)

  • Action-specific versus unified process rewards (XWJi). With the same SFT checkpoint, RL budget, and feedback frequency, action-specific rewards achieve 60.6 average mIoU, versus 58.7 for step-wise IoU-improvement rewards and 57.8 for outcome-only RL. The 1.9-point gain over unified rewards supports action-specific credit assignment. (XWJi: Q3.)

  • Test-time behavior and recovery (3pba). We reported the trajectory-length distribution and found no Refine-to-Relocate recovery in 100 erroneous-prefix tests. We will document this limitation and discuss corresponding recovery trajectories as a future training extension. (3pba: Response 3.)

We addressed the remaining clarifications in the individual responses. The added comparisons and ablations strengthen the evidence for verification-guided observation and action-specific supervision, while quantifying inference cost. We will incorporate these results, clarifications, and the identified recovery limitation into the revised manuscript. Thank you again for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.