acceptodds
Under review as a conference paper at ICLR 2027

CoEvoWhen: Policy-Tool Coevolution for Ultra-Long Video Temporal Grounding

Abstract

Ultra-long video temporal grounding requires balancing long-range evidence search with fine-grained event understanding under a limited visual budget, yet existing agentic methods still rely largely on predefined policies and tool capabilities. Motivated by this, we propose a novel policy–tool coevolution framework that jointly evolves high-level policies and executable media tools from the agentic reasoning trajectories of a VLM, forming a reusable skill without updating model parameters. During evolution, an external skill updater distills transferable task experience in long-video temporal grounding, accordingly refining the orchestration of long-range image-based and fine-grained video-based observations. Alongside these policy updates, the updater employs its coding capabilities to upgrade existing tools or create new ones, adapting the tools to long-video evidence acquisition. Equipped with the evolved skill, the VLM autonomously orchestrates tools under the guidance of the evolved policy, coordinating image and video observations for agentic inference without relying on a separate, stronger planning model. Extensive experiments spanning five benchmarks and three VLMs show that policy–tool coevolution consistently improves temporal grounding accuracy in ultra-long videos while reducing visual token cost at inference, and that the evolved skill yields substantial performance gains on general long-video QA without additional task-specific evolution, demonstrating the effectiveness and generalizability of our framework for long-video understanding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Self-Evolution, Long-Video Temporal Grounding, Multimodal Agents

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_PcwoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoEvoWhen, a policy-tool coevolution framework for ultra-long video temporal grounding. The proposed method adopts a powerful external model to update high-level policies and executable video tools based on previous task results. The VLM is keeping frozen during the updating process. Experiments show improved grounding accuracy over several benchmarks.

Critical strengths
  1. This paper achieves superior performance on several ultra-long video grounding benchmarks.

  2. The paper provides extensive experiments and ablation studies to show the effectiveness of the policy-tool coevolution.

Critical weaknesses
  1. The novelty of this work is not clear. If the contribution is to introduce policy-tool coevolution to VTG, related ideas have already been explored in EvoGround, META, and SkillSmith. If the contribution is a new evolution method, the paper lacks comparisons with other skill evolution method. The authors should compare with some strong baselines (not basic skills). For instance, it is unclear whether Codex/Claude could directly design a similar skill without iterative evolution (or directly generate such iterative evolution solution). More comparisons are needed to show the effectiveness of the proposed coevolution method.

  2. The framework relies on a powerful external model to update policies and tools. However, it is unclear how much of the improvement comes from the proposed coevolution process rather than Codex's existing capabilities. A useful baseline would be to let Codex directly design a strong skill without iterative evolution.

  3. The transferability of the evolved skills across different backbones is not fully evaluated. Table 3 shows that a separate skill can be evolved for each backbone. It remains unclear whether a skill evolved on one backbone can be directly applied to another without further evolution.

  4. The evolved policies mainly include common strategies such as global search, candidate tracking, and local refinement. These strategies have already been used in video temporal grounding. A stronger baseline with manually designed policies and tools would help show the actual benefits of automatic coevolution.

Questions for authors

In Weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Pcwo

We sincerely thank the reviewer for the careful reading, constructive feedback, and positive assessment of our empirical results. Below, we respond to the comments point by point.

W1 & W2: Novelty and effectiveness of coevolution

Thank you for the opportunity to clarify the novelty of our work. Our main contribution is a policy–tool coevolution framework that uses execution feedback from ultra-long video grounding to jointly evolve media tools that determine evidence coverage, temporal granularity, and visual presentation, together with policies for global image-based search, candidate refinement, and selective video verification. During coevolution, evidence needs drive tool design and tool capabilities guide policy adaptation, yielding a reusable skill that the VLM executes autonomously with higher grounding accuracy and lower visual token cost.

Our framework differs substantively from prior related work. Specifically, EvoGround updates a proposer and solver through reinforcement learning, whereas we evolve external policies and media tools with a frozen VLM. META evolves macro-tools and tool usage constraints with a fixed task-level orchestrator, while we also evolve observation policies. SkillSmith introduces new tools by extracting reusable logic from existing components. By contrast, we can synthesize new media tools from unmet evidence needs without requiring a prior implementation in a source component.

To further assess the benefits of coevolution, we construct three controls. In Codex-Designed, we use the same configuration (GPT-5.5, xhigh) and ask Codex to design, implement, and locally debug a skill supporting global-to-local search, contact sheets, candidate refinement, and selective video verification in one development session. For Human-Designed, we manually design a coarse-to-fine search policy and supporting media tools. Additionally, we introduce Fixed-Feedback, which iteratively updates policies and tools using the frozen base skill's trajectories and corresponding ground-truth intervals from the same evolution queries, while keeping the query order, batch size, number of update opportunities, and update acceptance criteria identical.

We use Qwen3.5-27B under matched inference settings for comparison. Visual token costs are reported in k/query.

MethodVUE-LVTR IoU AUCVUE-LVTR Visual TokensExtremeWhenBench Mean IoUExtremeWhenBench Visual TokensCoMET-Bench Mean IoUCoMET-Bench Visual Tokens
Base Skill0.4107202.580.1507227.100.1355119.66
Codex-Designed0.4533205.480.1813233.390.1427107.62
Human-Designed0.4722194.240.1799240.230.1502121.77
Fixed-Feedback0.4872234.790.2093217.800.1569126.84
CoEvoWhen0.5137141.890.2636201.170.163597.10

Compared with both static baselines, CoEvoWhen achieves higher accuracy at lower visual token cost, and the gains over Fixed-Feedback support the benefits of continually adapting policies and tools to execution feedback from the evolving skill.

W3: Transferability across VLM backbones

We directly transfer the same frozen skill evolved on Qwen3.5-27B to Qwen3.5-9B and Qwen3.6-27B without further evolution.

VLMMethodExtremeWhenBench Mean IoULVBench Overall Acc. (%)
Qwen3.5-9B+ Base Skill0.062640.09
Qwen3.5-9B+ Evolved Skill0.143246.55
Qwen3.6-27B+ Base Skill0.169148.55
Qwen3.6-27B+ Evolved Skill0.198952.81

Relative to the base skill, both backbones perform better with the evolved skill on long-video temporal grounding and general QA, demonstrating its transferability across different VLMs.

W4: Comparison with the manually designed baseline

Following your thoughtful suggestion, we construct Human-Designed with a manually designed coarse-to-fine search policy and supporting media tools. Specifically, make_contact_sheet_pack uniformly samples consecutive temporal windows to generate timestamped contact sheets spanning the specified search interval, while make_candidate_review_sheet presents candidate windows as rows of frames for joint comparison and verification. The policy uses global contact sheets to identify and retain multiple candidates, then refines their intervals with denser local contact sheets. When text, identity, or boundary details remain unclear, it requests high-resolution frames. Temporal strips or short clips are used when action order or temporal continuity remains ambiguous. As shown in W1 & W2, coevolution offers additional accuracy and efficiency gains over this manually designed static baseline.

We have included these results and analyses in the revised paper (Sections 2.2, 4.3.3; Appendices B.2.5, C.3), with the revisions highlighted in burgundy, and we sincerely hope our responses can address your concerns.

Dear Reviewer Pcwo,

We are truly grateful for your thoughtful comments and suggestions. As the discussion period draws to a close, we sincerely hope that our responses and the revised manuscript have helped address your concerns. If any questions remain, we would be very happy to provide further clarification. Your additional feedback would be greatly appreciated.

Thank you again for the time and care you have devoted to reviewing our work.

Best regards, The Authors

AuthorsOpenReview ↗

Brief Summary of Our Response and Revisions

Following your thoughtful comments on novelty, cross-model transfer, and the comparison with manually designed baselines, we summarize our rebuttal below, together with the corresponding revisions in the updated paper (highlighted in burgundy):

  • W1 & W2 (novelty and effectiveness of coevolution): We clarified CoEvoWhen’s novelty relative to prior work in our rebuttal, refined the comparison with SkillSmith in Section 2.2, and added the Codex-Designed, Human-Designed, and Fixed-Feedback controls in Section 4.3.3. Across all three benchmarks, CoEvoWhen achieves higher grounding accuracy with fewer visual tokens than each control.

  • W3 (transferability across VLM backbones): We transferred the skill evolved on Qwen3.5-27B to Qwen3.5-9B and Qwen3.6-27B without further evolution, with gains on both temporal grounding and QA, reported in Appendix C.3 (Table 12).

  • W4 (comparison with the manually designed baseline): We designed the Human-Designed baseline, as detailed in Appendix B.2.5, and compared against it. Section 4.3.3 (Table 5) reports that CoEvoWhen still leads.

Thank you again for your time and effort. Your insightful comments and suggestions have greatly helped us refine and strengthen our work, and we sincerely hope that these additional experiments and analyses have helped address your concerns. Looking forward to your further feedback.

Reviewer_p8WFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CoEvoWhen, a policy–tool coevolution framework for temporal grounding in ultra-long videos. With the vision-language model kept frozen, the method organizes task policies and executable media tools into an external skill. Codex uses execution trajectories and ground-truth temporal intervals as feedback to revise policy documents and modify or generate tool code. The resulting skill guides the model through global image-based search, local candidate refinement, and selective video verification, allowing it to perform grounding autonomously at inference time. The authors use 100 labeled queries for evolution and evaluate grounding accuracy and cumulative visual token usage on three benchmarks. They also evaluate the framework separately with three Qwen backbones and directly transfer the grounding skill to two long-video question-answering benchmarks. The results show improved grounding accuracy and lower visual token usage over the base skill, along with gains on the QA tasks.

Critical strengths

1.Jointly adapting media tools and observation policies is well motivated by the need to balance broad search with detailed local inspection in ultra-long videos. Figure 3 shows that, compared with policy-only evolution, joint evolution improves IoU AUC on VUE-LVTR from 0.4894 to 0.5137 while reducing cumulative visual token usage from 238.87k to 141.89k. This comparison supports the additional value of tool evolution and shows that the full method can improve both grounding accuracy and visual input cost. This is an important consideration in favor of acceptance.

2.The evolved skill improves performance on all three grounding benchmarks. Without further evolution on QA tasks, it also improves Qwen3.5-27B accuracy on LVBench and LSDBench by 8.65 and 7.98 percentage points, respectively. These results suggest that the accumulated search and evidence-gathering strategies transfer beyond grounding, supporting the practical value of the framework for reusable video understanding.

Critical weaknesses

1.The paper discusses closely related work such as SkillSmith and META, but lacks controlled comparisons that directly establish the value of its differences from these methods. The policy-only, tool-only, and joint evolution comparisons in Figure 3 support the value of optimizing both components. However, the base skill does not explicitly include global search, candidate management, or image–video orchestration strategies. The current results therefore leave open how much additional benefit automatic coevolution provides over a manually designed coarse-to-fine search policy with standard tools such as timestamped contact sheets. This limits my assessment of the method’s incremental contribution.

2.Although the evolution of executable media tools is a central contribution, the design of the evolved tools is not described in sufficient detail. Table 6 mainly lists the base tools, Figure 5 outlines changes in tool capabilities, and Appendix F provides the base skill and updater prompt. However, the sampling rules, interfaces, and processing logic of key tools before and after evolution are not shown in detail. The overall workflow is understandable, but it is difficult to determine exactly what new media-processing capabilities emerge through tool evolution and how they address limitations of the base tools. Describing representative tools and their implementation changes would make the technical contribution clearer.

Questions for authors

1.What key capabilities does the proposed policy–tool coevolution add compared with SkillSmith and META? Are there results comparing it with a manually designed coarse-to-fine search policy using timestamped contact sheets, under the same backbone and comparable budgets, that would help establish the additional value of automatic coevolution?

2.Could the authors explain how the sampling rules, interfaces, and processing logic of the tools change through evolution, and which limitations of the base tools these changes address? This would help clarify the specific technical capabilities introduced by tool evolution.

4 comments
AuthorsOpenReview ↗

Response to Reviewer p8WF

We sincerely thank the reviewer for the time and care devoted to assessing our paper, and for the thoughtful, constructive, and encouraging feedback. Below, we respond to the reviewer’s comments point by point.

W1 & Q1: Distinctions from related work and additional value of coevolution

Thank you for this thoughtful suggestion. We would like to clarify how our framework differs substantively from SkillSmith and META. SkillSmith organizes tool evolution around four typed library transformations, introducing new tools by extracting reusable logic from existing skills or tools. By contrast, we synthesize new media tools in response to gaps in temporal coverage, visual detail, or motion evidence revealed by grounding trajectories, without requiring their functionality to have already been implemented in a source component. The policy coevolves with these tools, since changing what the VLM can observe in one step also changes where it should look next and when further verification is warranted. META uses trajectory feedback to build macro-tools and refine tool usage constraints with a fixed task orchestrator, whereas we also evolve task policies for global search, candidate refinement, and selective video verification, enabling their joint adaptation with media tools.

Regarding the additional value of coevolution, we construct a strong static baseline with a manually designed coarse-to-fine search policy and supporting media tools: Human-Designed. Specifically, make_contact_sheet_pack uniformly samples consecutive temporal windows to generate timestamped contact sheets spanning the specified search interval, complemented by denser local contact sheets within candidate windows. make_candidate_review_sheet presents all candidate windows as rows of frames for joint comparison and verification. Additional tools support high-resolution frame extraction, temporal strips, video clip extraction, and video observation, with accompanying interfaces and usage instructions. The coarse-to-fine policy first identifies and retains multiple candidates through global contact sheets, then refines their intervals using local contact sheets. When text, identity, or boundary details remain unclear, the policy requests high-resolution frames. Temporal strips or short clips are used when action order or temporal continuity remains ambiguous.

We use Qwen3.5-27B under matched inference settings for comparison. Visual token costs are reported in k/query.

MethodVUE-LVTR IoU AUCVUE-LVTR Visual TokensExtremeWhenBench Mean IoUExtremeWhenBench Visual TokensCoMET-Bench Mean IoUCoMET-Bench Visual Tokens
Base Skill0.4107202.580.1507227.100.1355119.66
Human-Designed0.4722194.240.1799240.230.1502121.77
CoEvoWhen0.5137141.890.2636201.170.163597.10

Across all three benchmarks, CoEvoWhen achieves higher grounding accuracy and lower visual token cost than Human-Designed, demonstrating the additional value of coevolution over a manually designed coarse-to-fine search policy with supporting media tools.

W2 & Q2: Details of tool evolution

Thank you for pointing out the necessity of clarifying these evolution details. Below, we illustrate the changes introduced by tool evolution in three representative aspects.

Long-range search. The new tool make_contact_sheet_pack accepts a time interval, the number of contact sheets, and the number of frames per sheet as inputs. It divides the interval into one window per sheet and samples frames uniformly within each window. The sampled frames are then resized, tiled, and labeled with their timestamps in the source video. These sheets let the agent compare observations across distant parts of the video in a compact format.

Local refinement. The evolved extract_frames_at adds windows and step_seconds, allowing the agent to sample frames at a specified interval within candidate windows. When the frame count is capped, the tool distributes the sampling budget across windows. This lets the agent examine the frames surrounding an initial observation to identify brief events and refine their boundaries.

Local temporal evidence. The new tool make_temporal_strip_pack samples short frame sequences around evenly spaced center timestamps, using a configurable span and frame count. It arranges each group of frames in chronological order within a single image strip. The agent can also shift the sampling centers to inspect gaps in the original sampling grid. These strips provide local temporal cues for identifying brief actions and changes of state.

We have included these results and details in the revised paper (Sections 2.2, 4.3.3; Appendices B.1.4, B.2.5), with the revisions highlighted in burgundy, and we sincerely hope our responses can address your remaining concerns.

Dear Reviewer p8WF,

We are truly grateful for your thoughtful comments and suggestions. As the discussion period draws to a close, we sincerely hope that our responses and the revised manuscript have helped address your concerns. If any questions remain, we would be very happy to provide further clarification. Your additional feedback would be greatly appreciated.

Thank you again for the time and care you have devoted to reviewing our work.

Best regards, The Authors

Thank you for your response. After considering your points, I have decided to keep my original score.

We are truly grateful for your positive assessment of our work. Your constructive comments and insightful suggestions have greatly helped us refine and strengthen our paper.

Thank you again for your time and effort.

Reviewer_qWFJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoEvoWhen, a policy–tool coevolution framework for ultra-long video temporal grounding. Instead of relying on predefined tool-use strategies, the framework iteratively refines high-level agent policies and executable media tools based on VLM execution trajectories and ground-truth feedback. An external skill updater analyzes task outcomes to improve evidence acquisition strategies and develop tools for coordinating image-based global search with fine-grained video verification. The resulting skills can be reused by frozen VLMs without additional parameter updates or a stronger planning model during inference.

The authors evaluate CoEvoWhen on three temporal grounding benchmarks and two long-video QA benchmarks. Results demonstrate consistent improvements in grounding accuracy and reductions in visual token consumption. Additional experiments examine policy-only and tool-only evolution, image–video coordination, applicability across different VLMs, and transfer to long-video question answering.

Critical strengths
  1. Interesting and practical policy–tool coevolution framework. The paper explores a meaningful direction by jointly optimizing high-level agent policies and executable tools rather than treating tool capabilities as fixed. This design allows the agent to adapt both its evidence acquisition strategies and the ways visual information is prepared and presented. The approach is particularly relevant to ultra-long video understanding, where efficient global search and local temporal verification require different observation mechanisms.

  2. Strong empirical improvements and broad experimental validation. The proposed framework achieves substantial gains in temporal grounding accuracy while reducing visual token consumption across multiple benchmarks. The experiments further demonstrate improvements across different VLM backbones and positive transfer to long-video QA without task-specific evolution. The policy–tool and image–video ablation studies also provide useful evidence supporting the overall design. These results suggest that the proposed framework has practical value for improving the efficiency and effectiveness of multimodal agents.

Critical weaknesses
  1. Insufficient evidence for the necessity of policy–tool coevolution. Although the ablations show that jointly evolving policies and tools outperforms evolving either component alone, the paper primarily compares against a deliberately minimal base skill. It lacks a strong, non-evolving baseline equipped with well-designed search policies and efficient media tools, such as contact sheets and coarse-to-fine temporal refinement. Moreover, policy-only evolution already achieves most of the grounding accuracy improvement. Therefore, it remains unclear whether the gains stem from the proposed coevolution mechanism or simply from introducing better-designed policies and tools. This weakens the central methodological contribution.

  2. Incomplete evaluation of computational efficiency and evolution overhead. The framework relies on a powerful external model (GPT-5.5 with high reasoning effort) and ground-truth feedback to iteratively optimize the skills. However, the reported efficiency improvements primarily focus on visual token consumption during inference, without a comprehensive assessment of end-to-end latency, media preprocessing costs, or the computational expense of skill evolution. Consequently, it is difficult to assess the actual efficiency advantages and whether the cost of external skill optimization is justified by the reported performance improvements.

Questions for authors
  1. Necessity of coevolution (experimental validation): Could the authors compare CoEvoWhen against a strong, manually designed or one-shot LLM-generated static skill that already incorporates global-to-local search, contact-sheet generation, and selective video verification? Such a comparison under identical inference settings would help determine whether iterative policy–tool coevolution provides meaningful benefits beyond improved tool engineering and prompting.

  2. End-to-end efficiency (experimental validation): Could the authors report end-to-end inference latency, including VLM reasoning, visual processing, and local media tool execution, for both the base and evolved skills? Additionally, how much computational cost does the external skill evolution process incur, and how does this overhead affect the overall efficiency when amortized across downstream queries?

  3. Evaluation split and generalization (clarification): The paper states that the VUE-LVTR evaluation queries are disjoint from the 100 evolution queries. Are the underlying source videos also disjoint between evolution and evaluation? If not, could the authors provide video-disjoint evaluation results to better establish whether the evolved skills generalize beyond the videos encountered during evolution?

3 comments
AuthorsOpenReview ↗

Response to Reviewer qWFJ

We sincerely appreciate the reviewer's thoughtful assessment and recognition of our work's practical value and strong empirical results. Below, we address the concerns point by point.

W1 & Q1: Necessity of coevolution

Thank you for this helpful suggestion. Regarding Policy-only evolution, its gains come from better organizing observations within the sampling and presentation capabilities of fixed tools. Joint evolution adapts both the policies and these capabilities, increasing IoU AUC from 0.4894 to 0.5137 while reducing visual token cost by 40.6% (Section 4.3.1, Figure 3).

To test whether these benefits also hold against a well-designed static skill, we introduce Codex-Designed as a baseline using the same Codex configuration (GPT-5.5, xhigh). Given the task specification, runtime interfaces, and optimization objectives, we ask Codex to design the policy, implement the tools, and debug the skill locally in a single development session. The resulting policy combines global-to-local search, progressive refinement of candidate intervals, and selective verification with video clips. Supporting tools provide contact-sheet generation, clip extraction, and video observation.

We use Qwen3.5-27B under matched inference settings for comparison. Visual token costs are reported in k/query.

MethodVUE-LVTR IoU AUCVUE-LVTR Visual TokensExtremeWhenBench Mean IoUExtremeWhenBench Visual TokensCoMET-Bench Mean IoUCoMET-Bench Visual Tokens
Base Skill0.4107202.580.1507227.100.1355119.66
Codex-Designed0.4533205.480.1813233.390.1427107.62
CoEvoWhen0.5137141.890.2636201.170.163597.10

Across all three benchmarks, CoEvoWhen achieves higher grounding accuracy and lower visual token cost than Codex-Designed, demonstrating additional benefits over a static skill equipped with these search and verification capabilities.

W2 & Q2: End-to-end efficiency and evolution overhead

Thank you for suggesting this evaluation. In addition to visual token consumption, we have reported model calls, local tool calls, and visual observations in Section 4.2.2 and Table 2. Following your helpful suggestion, we measure average latency (s/query) on 100 randomly sampled ExtremeWhenBench queries using a single H200 NVL:

MethodVLM InferenceMedia ProcessingEnd-to-End Latency
Qwen3.5-27B + Base Skill303.3953.44361.46
Qwen3.5-27B + Evolved Skill233.2167.32305.89

VLM inference includes visual processing within the model and reasoning across all inference rounds. Media processing covers local decoding, sampling, contact-sheet generation, and clip extraction. End-to-end timing spans the full process from query and video processing to final answer submission, including all runtime overhead. Overall, the evolved skill saves 55.57 s/query, reducing end-to-end latency by 15.37%.

Regarding evolution overhead, the full evolution run took 12.45 hours of wall-clock time, covering trajectory execution, media processing, feedback computation, and update validation. Of this total, the Codex update workflow accounted for 4.81 hours, including rejected updates and retries. The complete logs for the 25 update batches recorded 4.56 million Codex tokens.

Using the measured latency, we estimate that the cumulative inference time savings offset the full 12.45 hours of evolution after approximately 807 comparable queries. Over 1,000 downstream queries, the average total time, including the one-time evolution cost, is 350.71 s per query, 2.97% lower than the base skill.

Q3: Evaluation split and generalization

Thank you for raising this point. Since different queries on the same ultra-long video may entail entirely different evidence requirements and search processes, we split VUE-LVTR by query rather than by source video, with some videos shared between evolution and evaluation.

Following your valuable suggestion, we further report results on all 189 held-out queries from 104 videos absent from the evolution set:

MethodPrecision AUCRecall AUCIoU AUCVisual Tokens
Qwen3.5-27B + Base Skill0.54220.54940.4852178.19
Qwen3.5-27B + Evolved Skill0.65300.65570.5827139.55

On these unseen videos, the evolved skill retains its accuracy and efficiency gains. ExtremeWhenBench and CoMET-Bench also share no source videos with the evolution set, and their gains in Table 1 further support generalization beyond the videos used for evolution.

We have included these results and analyses in the revised paper (Section 4.3.3; Appendices C.4, C.5), with the revisions highlighted in burgundy. In response to the ethics review flag, we have carefully reviewed our work and believe that it raises no specific ethical issues (see the Ethics Statement). We sincerely hope our responses can address your concerns.

Dear Reviewer qWFJ,

We are truly grateful for your thoughtful comments and suggestions. As the discussion period draws to a close, we sincerely hope that our responses and the revised manuscript have helped address your concerns. If any questions remain, we would be very happy to provide further clarification. Your additional feedback would be greatly appreciated.

Thank you again for the time and care you have devoted to reviewing our work.

Best regards, The Authors

AuthorsOpenReview ↗

Brief Summary of Our Response and Revisions

Following your valuable and thoughtful comments on the necessity of coevolution, end-to-end efficiency, and the evaluation split, we summarize our rebuttal below, together with the corresponding revisions in the updated paper (highlighted in burgundy):

  • W1 & Q1 (necessity of coevolution): We added Codex-Designed (GPT-5.5, xhigh) as a static baseline in Section 4.3.3 (Table 5). Under matched inference settings, CoEvoWhen achieves higher grounding accuracy with fewer visual tokens on all three benchmarks.

  • W2 & Q2 (end-to-end efficiency and evolution overhead): We measured end-to-end latency on 100 ExtremeWhenBench queries and found that the evolved skill reduced mean latency by 15.37%. In Appendix C.5, we give the latency breakdown in Table 14 and estimate that inference time savings offset the 12.45 hours spent on evolution after approximately 807 comparable queries.

  • Q3 (evaluation split and generalization): We additionally reported results on all 189 held-out VUE-LVTR queries from 104 videos unseen during evolution. Appendix C.4 (Table 13) shows that the evolved skill retains its gains in grounding accuracy and visual token efficiency on these videos.

Thank you again for your time and effort. Your insightful comments and suggestions have greatly helped us refine and strengthen our work, and we sincerely hope that these additional experiments and analyses have helped address your concerns. Looking forward to your further feedback.

Reviewer_nD6HOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CoEvoWhen, a Policy-Tool Coevolution framework for ultra-long video temporal grounding. It iteratively evolves agent policies and executable tools based on task feedback, aiming to improve temporal localization performance without updating the underlying VLM parameters. Experiments on multiple temporal grounding benchmarks demonstrate performance improvements, with additional evaluations showing potential generalization to different VLM backbones and long-video QA tasks.

Critical strengths

1. Novel Policy-Tool Coevolution Framework

The paper proposes a Policy-Tool Coevolution framework that jointly evolves high-level agent policies and executable tools, rather than optimizing either component independently. This provides a new approach to improving agent capabilities for ultra-long video temporal grounding.

2. Adaptive Tool Evolution

The framework allows the agent to iteratively modify existing tools, create new tools, and consolidate or retire redundant ones based on task feedback. This flexibility enables the toolset to adapt to different task requirements without updating the underlying VLM parameters.

3. Comprehensive Experimental Evaluation

The paper evaluates the proposed method on multiple temporal grounding benchmarks and different VLM backbones. The results show consistent performance improvements, while the ablation studies further demonstrate the benefits of joint Policy-Tool optimization.

4. Cross-Task Generalization

Beyond temporal grounding, the paper further evaluates the evolved Skill on long-video QA benchmarks without task-specific evolution. The observed improvements provide evidence of the potential transferability of the proposed framework to related video understanding tasks.

Critical weaknesses

1. Reliability of the update process

In Section 3.2, when introducing Policy-Tool Coevolution, the authors explain that an external Codex-based updater modifies the policy and executable tools based on task feedback, including updating tool descriptions, interfaces, and source code, as well as creating or consolidating tools. However, the paper does not sufficiently clarify how these updates are validated and selected. This makes the reliability and reproducibility of the overall evolution process difficult to assess.

2. Attribution of the iterative mechanism

The authors demonstrate performance improvements through joint evolution. However, this process relies on the Codex-based updater to iteratively modify the Skill. Therefore, it remains unclear how much of the observed improvement comes from the proposed iterative coevolution mechanism itself, as opposed to the inherent code-generation and reasoning capabilities of the external Codex updater. The paper does not provide sufficient analysis to disentangle these contributions.

3. Policy-Tool complementarity

The ablation study shows that joint Policy-Tool optimization achieves better performance, but the paper does not sufficiently explain the individual roles of the two components or their complementarity. It also remains unclear whether the gains primarily arise from their functional complementarity or deeper interactions between them, which limits the explanation of why the proposed Policy-Tool coevolution mechanism is effective and makes the paper's main claim less convincing.

4. Cross-model Skill transfer

The paper evaluates the proposed framework across multiple VLM backbones through Cross-VLM Experiments and further examines the transferability of the evolved Skill to long-video QA tasks through cross-task experiments. However, in the cross-VLM experiments, a separate Skill is evolved for each backbone. Therefore, these experiments do not establish whether the same evolved Skill can be directly transferred across different VLM backbones. Further validation of cross-model Skill transferability is still needed.

Questions for authors

1. Validation of the Skill Update Mechanism: During each iteration of Policy-Tool Coevolution, how do you determine whether the Policy and Tools generated or modified by Codex are effective? Is an independent validation mechanism used to select the updated versions? If an update leads to performance degradation, is there a rollback mechanism?

2. Contribution of the Iterative Coevolution Mechanism: The current experiments show performance improvements after multiple rounds of Skill updates, but it remains unclear how much of the improvement comes from the iterative coevolution mechanism itself and how much comes from Codex's own code generation and reasoning capabilities. Could you provide additional comparative experiments to further verify whether the proposed iterative coevolution mechanism brings additional benefits?

3. Complementarity between Policy and Tool: What specific task stages or types of temporal grounding errors are addressed by Policy-only optimization, Tool-only optimization, and joint optimization, respectively? Do the performance gains from joint optimization mainly come from the functional complementarity of the two components, or from deeper interactions between them? Could you provide further analysis to better explain the effectiveness of the joint evolution mechanism?

4 comments
AuthorsOpenReview ↗

Response to Reviewer nD6H (Part 1)

We sincerely thank the reviewer for the careful reading, constructive feedback, and recognition of our methodological and empirical contributions. Below, we address the concerns with additional clarifications and experiments.

W1 & Q1: Validation and reliability of skill updates

Thank you for raising this point. Regarding update validation and selection, the updater edits a candidate copy of the current skill while preserving the original, as specified in the Codex updater prompt in Appendix F.3. The candidate is then checked automatically to verify that it contains all required policy documents, stays within the permitted edit scope and context limit, and has valid tool interfaces and code syntax. Modified tools also undergo local self-tests, with each test script required to exit successfully and return an explicit pass indicator. The candidate is adopted only if the updater completes successfully and all checks pass. Otherwise, evolution continues from the previous version. In the main Qwen3.5-27B evolution run, 21 of 25 attempted updates were accepted.

Held-out validation scores are not used as an additional criterion for selecting updates at individual rounds or rolling back accepted updates based on task performance. To assess whether the gains are reproducible, we compare three independent evolution runs using Qwen3.5-27B. All three start from the same base skill and use an identical sequence of evolution queries, with matched updater configurations and update budgets. We evaluate each resulting skill on the same VUE-LVTR query-level held-out set.

RunIoU AUCVisual Tokens (k/query)
Base Skill0.4107202.58
Run 10.5137141.89
Run 20.5309152.37
Run 30.4973146.58
Mean ± SD0.5140 ± 0.0168146.95 ± 5.25

Each run improves accuracy and efficiency, with a mean IoU AUC gain of 0.1033 and 27.46% fewer visual tokens.

W2 & Q2: Additional benefits of iterative coevolution

Thank you for this thoughtful suggestion. We jointly evolve policies and executable tools using continually updated execution feedback. Each updated skill changes subsequent task execution, and the resulting feedback guides the next skill update. To assess the additional benefits of this iterative mechanism, we consider two control variants. Both use the same Codex configuration (GPT-5.5, xhigh).

Codex-Designed: Given the task specification, runtime interfaces, and optimization objectives, we ask Codex to design the policy, implement the tools, and debug the resulting skill locally, all in a single development session. It receives no grounding trajectories or feedback from ground-truth annotations. This control measures the downstream performance of a skill designed directly by Codex.

Fixed-Feedback: We first use the frozen base skill to execute the same evolution queries and collect its trajectories and the corresponding ground-truth intervals. Codex then updates the skill after each feedback batch, using the same query order, batch size, number of update opportunities, and acceptance criteria as in the main experiment. Each batch is explicitly identified to Codex as feedback from the base skill. This control tests whether collecting fresh trajectories as the skill evolves provides an additional benefit.

Using the main Qwen3.5-27B configuration, we evaluate each frozen skill on the VUE-LVTR query-level held-out set, ExtremeWhenBench, and CoMET-Bench (visual token costs in k/query):

MethodVUE-LVTR IoU AUCVUE-LVTR Visual TokensExtremeWhenBench Mean IoUExtremeWhenBench Visual TokensCoMET-Bench Mean IoUCoMET-Bench Visual Tokens
Base Skill0.4107202.580.1507227.100.1355119.66
Codex-Designed0.4533205.480.1813233.390.1427107.62
Fixed-Feedback0.4872234.790.2093217.800.1569126.84
CoEvoWhen0.5137141.890.2636201.170.163597.10
Δ vs. Codex-Designed+13.32%−30.95%+45.39%−13.81%+14.58%−9.78%
Δ vs. Fixed-Feedback+5.44%−39.57%+25.94%−7.64%+4.21%−23.45%

Across all three benchmarks, CoEvoWhen achieves higher grounding accuracy and lower visual token cost than both controls. These results support the additional accuracy and efficiency benefits of collecting execution feedback as the skill evolves.

AuthorsOpenReview ↗

Response to Reviewer nD6H (Part 2)

W3 & Q3: Complementarity of policies and tools

Thank you for raising this important question. As shown in Section 4.3.1 and Figure 3, evolving either policies or tools alone yields gains, while joint evolution further improves accuracy and reduces visual token cost. Section 4.3.4 and Figure 5 show how candidate omissions, boundary errors, and dynamic ambiguities drive coordinated updates to policies and tools. The execution traces in Appendix D (Figures 6 and 7) also illustrate how these changes support candidate localization and local verification. Following your helpful suggestion, we further draw on trajectories from separate and joint evolution to clarify the roles and complementarity of the two components.

Specifically, Policy-only evolution improves candidate retention, local refinement, and observation orchestration with fixed tools to address candidate omissions, boundary errors, and dynamic ambiguities. Tool-only evolution improves frame sampling, timestamp annotation, and grid presentation for single or multiple windows with a fixed policy, addressing coverage gaps and the overhead of separate observations. For example, in one execution trajectory from joint evolution, the agent identified several candidate intervals during global search but retained only one during local verification. This feedback prompted coordinated updates: the tool presents multiple candidate windows together in a single timestamped image, while the policy requires verifying each candidate and retaining every interval supported by evidence. Together, these results show that policy evolution improves how evidence acquisition is organized, while tool evolution expands the available observation capabilities. Joint evolution adapts both to evidence needs revealed during execution, enabling the agent to use these capabilities for more accurate grounding at a lower visual token cost.

W4: Cross-model skill transfer

VLMMethodExtremeWhenBench Mean IoULVBench Overall Acc. (%)
Qwen3.5-9B+ Base Skill0.062640.09
Qwen3.5-9B+ Evolved Skill0.143246.55
Qwen3.6-27B+ Base Skill0.169148.55
Qwen3.6-27B+ Evolved Skill0.198952.81

Thank you for this valuable evaluation suggestion. We directly transfer the frozen skill evolved in the main Qwen3.5-27B experiment to Qwen3.5-9B and Qwen3.6-27B. Relative to the base skill, both models perform better on long-video temporal grounding and general QA. These results show that the same evolved skill can be effectively transferred across different VLM backbones.

We have included these results and analyses in the revised paper (Section 4.3.3; Appendices B.1.3, C.2, C.3 and D), with the revisions highlighted in burgundy, and we sincerely hope our responses can address your concerns.

Dear Reviewer nD6H,

We are truly grateful for your thoughtful comments and suggestions. As the discussion period draws to a close, we sincerely hope that our responses and the revised manuscript have helped address your concerns. If any questions remain, we would be very happy to provide further clarification. Your additional feedback would be greatly appreciated.

Thank you again for the time and care you have devoted to reviewing our work.

Best regards, The Authors

AuthorsOpenReview ↗

Brief Summary of Our Response and Revisions

Following your valuable and thoughtful comments on update reliability, the benefits of coevolution, and cross-model transfer, we summarize our rebuttal below, together with the corresponding revisions in the updated paper (highlighted in burgundy):

  • W1 & Q1 (validation and reliability of skill updates): We clarified the automated acceptance checks in Appendix B.1.3, including their outcome in the main run (21 of 25 updates accepted), and reported three independent evolution runs with consistent gains (IoU AUC 0.5140 ± 0.0168, 27.46% fewer visual tokens) in Appendix C.2 (Table 11).

  • W2 & Q2 (additional benefits of iterative coevolution): We added Codex-Designed and Fixed-Feedback controls in Section 4.3.3 (Table 5). CoEvoWhen outperforms both on all three benchmarks with fewer visual tokens (e.g., ExtremeWhenBench mean IoU 0.2636 vs. 0.1813 / 0.2093).

  • W3 & Q3 (complementarity of policies and tools): We compared trajectories from policy-only, tool-only, and joint evolution to explain how tools change the evidence available and how policies decide what to inspect next. Our expanded discussion in Appendix D (Figures 6 and 7) shows how the policy uses timestamped contact sheets to locate candidates, then verifies motion with a short clip or refines scene boundaries with targeted frames.

  • W4 (cross-model skill transfer): We transferred the skill evolved on Qwen3.5-27B to Qwen3.5-9B and Qwen3.6-27B without further evolution. Appendix C.3 (Table 12) shows that each model improves over its base skill on grounding and QA, including an increase in ExtremeWhenBench mean IoU from 0.0626 to 0.1432 on Qwen3.5-9B.

Thank you again for your time and effort. Your insightful comments and suggestions have greatly helped us refine and strengthen our work, and we sincerely hope that these additional experiments and analyses have helped address your concerns. Looking forward to your further feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.