- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CoEvoWhen, a policy-tool coevolution framework for ultra-long video temporal grounding. The proposed method adopts a powerful external model to update high-level policies and executable video tools based on previous task results. The VLM is keeping frozen during the updating process. Experiments show improved grounding accuracy over several benchmarks.
Critical strengths
-
This paper achieves superior performance on several ultra-long video grounding benchmarks.
-
The paper provides extensive experiments and ablation studies to show the effectiveness of the policy-tool coevolution.
Critical weaknesses
-
The novelty of this work is not clear. If the contribution is to introduce policy-tool coevolution to VTG, related ideas have already been explored in EvoGround, META, and SkillSmith. If the contribution is a new evolution method, the paper lacks comparisons with other skill evolution method. The authors should compare with some strong baselines (not basic skills). For instance, it is unclear whether Codex/Claude could directly design a similar skill without iterative evolution (or directly generate such iterative evolution solution). More comparisons are needed to show the effectiveness of the proposed coevolution method.
-
The framework relies on a powerful external model to update policies and tools. However, it is unclear how much of the improvement comes from the proposed coevolution process rather than Codex's existing capabilities. A useful baseline would be to let Codex directly design a strong skill without iterative evolution.
-
The transferability of the evolved skills across different backbones is not fully evaluated. Table 3 shows that a separate skill can be evolved for each backbone. It remains unclear whether a skill evolved on one backbone can be directly applied to another without further evolution.
-
The evolved policies mainly include common strategies such as global search, candidate tracking, and local refinement. These strategies have already been used in video temporal grounding. A stronger baseline with manually designed policies and tools would help show the actual benefits of automatic coevolution.
Questions for authors
In Weakness.
3 comments
Dear Reviewer Pcwo,
We are truly grateful for your thoughtful comments and suggestions. As the discussion period draws to a close, we sincerely hope that our responses and the revised manuscript have helped address your concerns. If any questions remain, we would be very happy to provide further clarification. Your additional feedback would be greatly appreciated.
Thank you again for the time and care you have devoted to reviewing our work.
Best regards, The Authors
Brief Summary of Our Response and Revisions
Following your thoughtful comments on novelty, cross-model transfer, and the comparison with manually designed baselines, we summarize our rebuttal below, together with the corresponding revisions in the updated paper (highlighted in burgundy):
-
W1 & W2 (novelty and effectiveness of coevolution): We clarified CoEvoWhen’s novelty relative to prior work in our rebuttal, refined the comparison with SkillSmith in Section 2.2, and added the Codex-Designed, Human-Designed, and Fixed-Feedback controls in Section 4.3.3. Across all three benchmarks, CoEvoWhen achieves higher grounding accuracy with fewer visual tokens than each control.
-
W3 (transferability across VLM backbones): We transferred the skill evolved on Qwen3.5-27B to Qwen3.5-9B and Qwen3.6-27B without further evolution, with gains on both temporal grounding and QA, reported in Appendix C.3 (Table 12).
-
W4 (comparison with the manually designed baseline): We designed the Human-Designed baseline, as detailed in Appendix B.2.5, and compared against it. Section 4.3.3 (Table 5) reports that CoEvoWhen still leads.
Thank you again for your time and effort. Your insightful comments and suggestions have greatly helped us refine and strengthen our work, and we sincerely hope that these additional experiments and analyses have helped address your concerns. Looking forward to your further feedback.
Response to Reviewer Pcwo
We sincerely thank the reviewer for the careful reading, constructive feedback, and positive assessment of our empirical results. Below, we respond to the comments point by point.
W1 & W2: Novelty and effectiveness of coevolution
Thank you for the opportunity to clarify the novelty of our work. Our main contribution is a policy–tool coevolution framework that uses execution feedback from ultra-long video grounding to jointly evolve media tools that determine evidence coverage, temporal granularity, and visual presentation, together with policies for global image-based search, candidate refinement, and selective video verification. During coevolution, evidence needs drive tool design and tool capabilities guide policy adaptation, yielding a reusable skill that the VLM executes autonomously with higher grounding accuracy and lower visual token cost.
Our framework differs substantively from prior related work. Specifically, EvoGround updates a proposer and solver through reinforcement learning, whereas we evolve external policies and media tools with a frozen VLM. META evolves macro-tools and tool usage constraints with a fixed task-level orchestrator, while we also evolve observation policies. SkillSmith introduces new tools by extracting reusable logic from existing components. By contrast, we can synthesize new media tools from unmet evidence needs without requiring a prior implementation in a source component.
To further assess the benefits of coevolution, we construct three controls. In Codex-Designed, we use the same configuration (GPT-5.5, xhigh) and ask Codex to design, implement, and locally debug a skill supporting global-to-local search, contact sheets, candidate refinement, and selective video verification in one development session. For Human-Designed, we manually design a coarse-to-fine search policy and supporting media tools. Additionally, we introduce Fixed-Feedback, which iteratively updates policies and tools using the frozen base skill's trajectories and corresponding ground-truth intervals from the same evolution queries, while keeping the query order, batch size, number of update opportunities, and update acceptance criteria identical.
We use Qwen3.5-27B under matched inference settings for comparison. Visual token costs are reported in k/query.
Compared with both static baselines, CoEvoWhen achieves higher accuracy at lower visual token cost, and the gains over Fixed-Feedback support the benefits of continually adapting policies and tools to execution feedback from the evolving skill.
W3: Transferability across VLM backbones
We directly transfer the same frozen skill evolved on Qwen3.5-27B to Qwen3.5-9B and Qwen3.6-27B without further evolution.
Relative to the base skill, both backbones perform better with the evolved skill on long-video temporal grounding and general QA, demonstrating its transferability across different VLMs.
W4: Comparison with the manually designed baseline
Following your thoughtful suggestion, we construct Human-Designed with a manually designed coarse-to-fine search policy and supporting media tools. Specifically,
make_contact_sheet_packuniformly samples consecutive temporal windows to generate timestamped contact sheets spanning the specified search interval, whilemake_candidate_review_sheetpresents candidate windows as rows of frames for joint comparison and verification. The policy uses global contact sheets to identify and retain multiple candidates, then refines their intervals with denser local contact sheets. When text, identity, or boundary details remain unclear, it requests high-resolution frames. Temporal strips or short clips are used when action order or temporal continuity remains ambiguous. As shown in W1 & W2, coevolution offers additional accuracy and efficiency gains over this manually designed static baseline.We have included these results and analyses in the revised paper (Sections 2.2, 4.3.3; Appendices B.2.5, C.3), with the revisions highlighted in burgundy, and we sincerely hope our responses can address your concerns.