acceptodds
Withdrawn from ICLR 2027

EditEvolve: Self-Evolving Agents for Context-Aware Video Editing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern video editing models have acquired increasingly powerful content manipulation capabilities, yet high-quality video editing is not simply an instruction-to-video mapping. The same editing instruction can impose different contextual requirements on different source videos: changes to a subject must coordinate its spatial relations, interactions, and associated effects while preserving unrelated content and temporal continuity. The key challenge therefore lies in understanding the source-video context and organizing an appropriate editing strategy accordingly. Identifying an effective strategy, however, requires understanding not only the scene but also how the available tools and generators behave in practice. To this end, we introduce EditEvolve, a self-evolving framework that learns context-aware video editing strategies from execution experience without updating model parameters. To accommodate diverse contextual requirements, EditEvolve constructs a composable editing workflow space that enables the agent to flexibly organize editing operations. To learn how to effectively use tools and generators, the framework performs offline workflow exploration and feedback-driven revision, turning failures in instruction following, contextual consistency, and content preservation into signals for strategy improvement. EditEvolve further distills execution trajectories into reusable experience that connects contextual conditions with effective editing strategies, allowing prior execution outcomes to guide workflow composition for unseen tasks. Experimental results show that EditEvolve achieves state-of-the-art performance and consistently improves editing performance across different video editing backbones.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video editing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2GgrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an agentic workflow that composes reference retrieval, spatial grounding, keyframe editing, and prompt refinement around a frozen video editor. A VLM planner uses experience cards distilled from verified execution trajectories to guide subsequent workflows. The experiments report improved editing performance and demonstrate transfer of the auxiliary tool space to VACE.

Critical strengths
  • The framework addresses a practically relevant problem: providing appropriate auxiliary guidance for video edits that depend on target appearance, spatial relationships, and interactions. This is an intuitive idea that could potentially improve performance across diverse video editing scenarios.
Critical weaknesses
  • My major concern is the paper’s limited technical novelty. The framework primarily combines auxiliary editing tools with VLM-based planning, verification-driven revision, and retrieval of summarized execution experience. Section 2.2 already discusses tool-using video editing through Aurora and reusable workflow induction or optimization through Agent Workflow Memory and AFlow. However, the paper does not clearly establish a substantive new mechanism beyond combining these directions for video editing.

  • The controlled comparisons are also insufficient. The tool-removal and no-memory ablations show that access to these components is beneficial, but they do not demonstrate the superiority of the proposed composition and memory design. Although Aurora is included, the paper does not document a controlled comparison using the same editing backbone and tools. In addition, the framework uses the Qwen model family for planning and verification, while Edit Accuracy is also measured by a Qwen-family evaluator using eight sampled frames. This leaves the potential for correlated judging errors insufficiently examined.

  • A minor weakness is that the paper lacks some implementation details. The experience-learning split size, memory size, retrieval kk, verifier acceptance criteria, and numerical tool budget are missing or insufficiently specified.

Questions for authors
  • The Introduction states that memory is frozen at test time, but Sections 5.1 and 5.4 describe online experience accumulation. Which protocol was actually used in the experiments?

  • Were the H3 baseline results obtained using the official prompt enhancement method (H3-Context-IR)? If not, the prompt-enhanced baseline should be included, as it serves a similar function by providing auxiliary textual guidance.

Reviewer_bCdROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents EditEvolve, an agent framework for context-aware video editing that keeps all model parameters frozen. Given a source video and an editing instruction, a planner composes optional tools for reference retrieval, spatial grounding, keyframe editing, and prompt refinement to guide a video editor. The framework uses comparative verification and bounded revision during offline exploration, then distills successful execution trajectories into reusable experience cards for subsequent planning. Evaluation on 274 editing tasks, including FiveBench samples and a collected context-dependent subset, reports improvements over the evaluated baselines. Tool and memory ablations examine the components’ contributions, while additional experiments assess the transfer of the tool space to VACE.

Critical strengths
  1. Clear practical motivation. The paper addresses the need for task-specific guidance in video editing, where target ambiguity, appearance details, and object interactions can make direct instruction-based editing unreliable.

  2. Flexible, modular design. The framework combines complementary tools without updating model parameters, and experiments with H3 and VACE suggest that the tool space can benefit different video editing backbones.

Critical weaknesses
  1. The methodological novelty is insufficiently established. The framework combines existing tools with workflow planning and experience retrieval, but its distinction from prior tool-using editing agents and workflow-memory approaches remains unclear. The experiments do not sufficiently isolate the benefit of the proposed experience distillation mechanism from simpler workflow or retrieval strategies.

  2. Benchmark coverage is limited. Evaluation uses 165 FiveBench samples and 109 collected context-dependent samples. Results on additional established benchmarks would strengthen the evidence that the approach generalizes beyond this evaluation set.

  3. Temporal behavior is difficult to assess qualitatively. The manuscript provides frame-based comparisons, but these do not adequately demonstrate motion preservation, temporal consistency, or transient editing failures. Full video comparisons and representative failure cases would make these claims easier to assess.

  4. Robustness across component models is insufficiently evaluated. The paper does not examine sensitivity to the choice of planning model. Although VACE experiments demonstrate tool-space transfer, they do not clearly establish whether the complete experience-learning mechanism remains effective across different video editors.

Questions for authors

Please refer to the above Weaknesses section.

Reviewer_tsj5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EditEvolve, a training-free, self-evolving agent framework to address the challenge in instruction-based video editing where the same editing instruction requires different auxiliary conditions across varying video contexts. The framework integrates reference retrieval, spatial grounding, keyframe editing, and prompt refinement into a composable tool space, utilizing a VLM planner to dynamically compose workflows based on the video context.

Critical strengths

The problem defined in this paper is of practical significance. Moreover, the paper is logically structured and well-written.

Critical weaknesses
  1. During the offline experience distillation stage, failed trajectories are excluded from being summarized into the experience memory. A truly self-evolving mechanism should be capable of inducing negative constraints from failure cases (e.g., disabling specific artifact-prone tools in certain scenarios), whereas this framework only records positive workflow templates that happen to succeed within at most three rounds of trial-and-error revision. Essentially, it represents a domain-specific engineering combination of existing agentic paradigms, lacking fundamental algorithmic breakthroughs. As such, the novelty is limited in some degree.
  2. Internally, the authors employ Qwen3.8-27B as both the Planner and Verifier. Meanwhile, in the automated evaluation in Appendices A.1 and A.2, Qwen3.8-Flash from the same model family is used as the core evaluator to compute metrics. This setup could easily cause the distilled experience to overfit to the intrinsic preferences of the Qwen3.8 family. The authors should include human subjective evaluations (User Study) and cross-model evaluations (e.g., using GPT, Claude, or Gemini as evaluators) for cross-validation.
  3. This paper looks like a naive RAG system relying on bag-of-words keyword matching: the so-called "self-evolution" entails no parameter updates, reinforcement learning, or complex policy optimization, merely using a large model to distill successful pipeline logs into textual "Experience Cards" within an external memory.
  4. EditEvolve introduces multiple VLM calls, external retrieval, SAM3 video segmentation and tracking, as well as image editing. Compared to direct video editing (DVE) or other agent-based baselines (such as Aurora), how much additional per-sample inference latency, token consumption, and API financial cost does EditEvolve incur?
Questions for authors

Please see the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.