acceptodds
Desk rejected at ICLR 2027

VideoEvolve: Evolving Agent Harnesses for Video Temporal Grounding

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Video temporal grounding aims to localize events in videos from natural-language queries. For agents built around frozen video-language models, the harness determines how queries guide temporal predictions and how those predictions are refined. Manually refining these harnesses requires diagnosing grounding failures and coordinating changes to both agent workflows and instructions. We introduce VideoEvolve, a framework that automatically evolves agent harnesses for video temporal grounding. VideoEvolve uses a Cloze-Structured Harness Representation that preserves stage interfaces while leaving agent workflows and instructions open to evolution. Branch-Guided Harness Evolution preserves promising code branches for continued refinement, using execution feedback to guide local edits and validation to determine which improvements are carried forward. Experiments demonstrate improved grounding performance across multiple benchmarks. Component analyses show that both evolved code and instructions contribute to the final harnesses. Together, these results support automated harness evolution as an effective approach to improving video temporal grounding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Temporal Grounding, Agent Harness Evolution

Related papers

Discussion (0)

Sign in to comment.