acceptodds
Withdrawn from ICLR 2027

VideoSpy: Evolving through Experience for Adaptive Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent approaches to agentic video understanding are often tailored to domain-specific evidence requirements and task settings, limiting their adaptability across domains. A central bottleneck is that their task-solving strategies remain largely fixed after design, without a systematic mechanism for learning and evolving from execution experience. In this work, we investigate whether a foundational video agent can evolve from execution experience to adapt to different domains without redesigning its framework. To this end, we first build a foundational VideoSpy agent with three complementary visual tools: overview, clip_skim, and frame_inspect. A Reflector combines task-grouped reflection and evidence-based review to extract reusable experience from training execution records and ground-truth references. An Agent Optimizer loads the resulting Skill and source-grounded Memory into the shared agent harness, enabling the same foundational agent to specialize independently for each domain. To assess this mechanism, we establish a unified protocol for domain-specific evolution and held-out evaluation on LVBench, EgoLifeQA, and MedVidBench, covering general long videos, egocentric daily life, and medical videos. With Qwen3.5-4B, evolution raises accuracy by 3.8 and 4.2 percentage points on LVBench and EgoLifeQA, improves nine of ten MedVidBench leaderboard metrics, and reduces inference tokens by 9.7–27.1%. These results show that a shared video agent can adapt to different domains by learning from execution experience.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agentic video understanding, Experience-driven agent evolution, Evidence-grounded reflection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_K1CJOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VideoSpy, a video-understanding agent equipped with three visual tools. A Reflector uses fixed training trajectories and ground-truth references to extract and review textual Skills and Memory, which are subsequently incorporated into the agent’s inference context. The resulting state is frozen before evaluation.

Critical strengths
  1. well-written.
  2. a novel method (self-evolving agent) in video understanding agent.
Critical weaknesses
  1. insufficient baseline, only DVDAgent, WorldMM, and VideoSeek, but there are many agent systems in video understanding tasks from my view, also there are many techs in this domain, for example we would like to see 0-shot (direct answer), few-shot learning.

  2. the concern of generalization, the training data is sampled from the test data, they are in the same distribution. We would like to see a strict training and test split. For example, you can only use one of your benchmarks as the training data and test in the other benchmarks.

  3. meanwhile, the method like the reflection and self-evolving are popular in coding agent or deepresearch agent, i do not find any special in this paper, it just applied the method in the video understanding demain, and the memory is the textual memory. From my view, there should be something different in video tasks, like non-textual memory.

  4. for the memory management, the paper does not mention that how to maintain the memory when explosion, and how to merge or delete memory. and for the retrieval machenism, it just uses the semantic similarity, but in agentic tasks, the semantic similarity is not equivalent to beharvioural memory.

Questions for authors
  1. How does VideoSpy compare with a direct-video baseline using the same backbone on LVBench and EgoLifeQA, under comparable visual-input and inference budgets?

  2. Under the same training trajectories and agent harness, how much does the proposed reflection-and-review procedure improve over direct retrieval of successful trajectories or a single-pass summary of experience?

  3. Is there evidence that frozen Skills transfer to another dataset or an unseen task type, without further evolution? Such evidence would materially strengthen the generalization claim.

Reviewer_pKRhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes VideoSpy, which adapts video understanding agents by updating skills and memory from training traces and ground truth feedbacks, without changing the model weights. Task-grouped reflection with a same-backbone teacher improves performance and reduces inference token usage across three domains.

Critical strengths
  1. The framework adapts a shared video agent as backbone and updates only domain specific skills and memory parts.
  2. The ablations examine the roles of task grouping, evidence-based review, skill and memory, providing useful evidence for the effectiveness of each part.
  3. A comparable teacher model can conclude more useful experience to its student, which is an interesting finding
Critical weaknesses
  1. There is already substantial work on skill and memory optimization for agents, so the methodological novelty appears somewhat limited. VideoSpy mainly adapts these ideas to video understanding, and it would be helpful to clarify the new insights gained from this adaptation.
  2. EgoLifeQA only has 95 test cases, the accuracy improves from 32.6% to 36.8%, which is 4 more correct answers; improvements for 27B is one more correct answer. These results seem not very convincing that the algorithm is robustly helpful.
Questions for authors

Please address the questions mentioned above

Reviewer_TwvGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VideoSpy, an agent framework that adapts to different video understanding domains by learning from execution experience. The base agent follows a ReAct loop and gathers visual evidence through three tools: full-video overview, clip inspection, and inspection of specific frames. During adaptation, a Reflector analyzes training trajectories and ground-truth references, groups trajectories by task type, and distills reusable Skills and Memory. Skills capture investigation strategies and common pitfalls, while Memory stores source-grounded observations and historical reasoning cases. An evidence-based review checks the extracted experience against the recorded evidence. An Agent Optimizer then selects relevant Skills and retrieves eligible memories for inclusion in the agent’s context, keeping model weights, tool interfaces, and the execution framework fixed. The authors learn and freeze domain-specific experience separately on LVBench, EgoLifeQA, and MedVidBench, then evaluate performance on held-out test data. The results show that VideoSpy improves accuracy on several tasks while reducing inference token usage. Component ablations and cross-model experiments further examine the roles of experience organization, evidence review, and the choice of reflection model.

Critical strengths
  1. Adaptation across video domains within a fixed agent framework. VideoSpy improves performance and reduces inference token usage across long-video, egocentric, and medical video benchmarks by updating Skills and Memory while keeping model weights, tools, and the agent framework fixed. This provides evidence that execution experience can support domain adaptation without redesigning the agent, making the approach practically useful.

  2. Task-grouped reflection supported by targeted ablations. The method organizes trajectories by task type to extract relevant Skills and reviews the resulting guidance against recorded evidence. The ablations show that ungrouped reflection can hurt performance, while task grouping and evidence-based review improve both accuracy and efficiency. This strengthens the empirical justification for the core design and helps explain when learning from execution experience is beneficial.

Critical weaknesses
  1. Insufficient comparison with related adaptation methods. Internal ablations support the proposed components, but comparisons with existing experience-based methods under matched conditions are missing. This makes it difficult to assess the paper’s incremental methodological contribution.

  2. Limited evidence beyond known task types. Evaluation relies on independent domain adaptation and predefined task labels, without testing unseen task types or missing labels. This leaves the method’s dependence on the benchmark task taxonomy unclear, limiting confidence in its broader applicability.

Questions for authors
  1. How well do Skills and Memory learned on one benchmark transfer to another benchmark or domain without further adaptation? The current evaluation establishes the effectiveness of independent adaptation within each domain, but does not clarify how broadly the resulting experience can be reused.

  2. How does VideoSpy compare with closely related experience-based adaptation methods when using the same base agent, adaptation data, feedback, and comparable budgets? Such a comparison would help establish the additional value of the proposed experience organization and review mechanisms beyond their benefits in internal ablations.

  3. How does the method handle task types not seen during adaptation, or settings where task-type labels are unavailable or incorrect? Since these labels guide both experience organization and Skill selection, evaluating these conditions would clarify the method’s dependence on a predefined task taxonomy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.