acceptodds
Withdrawn from ICLR 2027

VAEM: Give Video Avatars an Evolving Mind

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent efforts have explored endowing video avatars with active intelligence to generate multi-step video sequences toward high-level goals. However, existing methods fail to consolidate past failures into persistent knowledge, frequently repeating similar errors in planning and execution. To address this, we present VAEM (Video Avatar Evolving Mind), a framework that distills video-generation histories into a persistent planner context comprising behavioral principles in an evolving system prompt and reusable skills in a structured library. VAEM operates via a Plan-Generate-Critique-Update (PGCU) cycle, orchestrating a scene-aware planner, a frozen video generator, specialized vision-language critics, and an update agent. During context evolution, the critics localize semantic failures, enabling the update agent to convert feedback into actionable rules and skills for subsequent tasks. At inference, the accumulated context is frozen and reused directly for planning without online updates. Extensive experiments on half-body and full-body manipulation tasks demonstrate that the six-round CCE setting reduces mean semantic failure counts from 9.26 to 7.83 and 8.85 to 7.65, respectively, while enhancing aggregate visual quality. Furthermore, on the L-IVA benchmark using a shared generation backend, VAEM achieves state-of-the-art perceptible motion amplitude (PAS: 28.0) and motion smoothness (MSS: 87.7) alongside competitive visual quality. Finally, a real-time prototype validates VAEM's compatibility with interactive rendering. These results demonstrate that persistent context evolution offers an effective paradigm for cross-task experience reuse without fine-tuning the underlying video generator. Project page: https://anonymous.4open.science/w/VAEM-04EB.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Self-Evolving Agent, Video Avatar, Agentic Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jLrxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses multi-step, goal-oriented video avatar generation, arguing that existing simulator-mode methods (e.g., ORCA) plan within individual tasks and fail to consolidate past failures into knowledge that transfers across tasks. The authors propose VAEM, which freezes the video generator (Wan2.1-I2V) and all model parameters and optimizes only the planner’s persistent context: global behavioral principles in the system prompt and a structured skill library. The method runs a Plan-Generate-Critique-Update (PGCU) cycle. A scene-aware CoT planner produces segment-level action plans, the frozen I2V model renders them segment by segment, five specialized VLM critics localize failures along physical plausibility, commonsense consistency, goal completion, cross-clip continuity, and object integrity (together the SBM metric), and an update agent distills the feedback into rules and skills. Two evolution strategies are compared: single-plan Holistic Context Evolution (HCE), and Contrastive Context Evolution (CCE), which samples four candidate plans per task and contrasts the best and worst trajectories. At inference, the context is frozen and the planner is called once. On self-built half-body and full-body manipulation benchmarks (200 test tasks each), CCE reduces the mean failure count from 9.26 to 7.83 and from 8.85 to 7.65, with slight VMBench gains. On L-IVA, VAEM achieves the best PAS and MSS and the lowest CMF and GCF failure counts, while requiring a single planner call at test time versus 5.59 for ORCA. The paper also reports a correlation study between SBM and human ratings, as well as component ablations.

Critical strengths
  1. Meaningful problem setting with a practical solution. The paper identifies cross-task experience reuse as an open problem in multi-step video avatar generation. It offers a training-free solution that optimizes only the textual context while keeping the backend frozen. At test time VAEM needs a single planner call with no retries, compared with 5.59 calls and 2.02 retries for ORCA (Table 3), so it is considerably cheaper to deploy than methods that rely on test-time replanning.

  2. SBM provides localized, interpretable feedback. SBM localizes failures to specific dimensions, segments, and timestamps, so it serves both as an evaluation metric and as the feedback signal for evolution. The human study provides some evidence of its validity (TPF vs. PPS: ρ = −0.71; GCF vs. TSR: ρ = −0.63).

  3. The contrastive update targets a challenge specific to video. CCE contrasts good and bad trajectories under the same context to separate planning errors from stochastic rendering defects, a problem that text-only agents do not face. Under an equal trajectory-evaluation budget, CCE outperforms HCE on both aggregate metrics (TPF and VMBench total).

Critical weaknesses
  1. The same critics drive optimization and evaluation, and the core claims lack independent validation. Both the evolution feedback and the final SBM scores come from critics built on the same Gemini-3.1-Pro backbone, so the reported gains may partly reflect fitting the critics rather than genuine improvement. The human study only measures the correlation between SBM and human ratings and does not report per-method human scores. Meanwhile, the VMBench gains are small (e.g., 69.0 → 69.8 on half-body).

  2. Insufficient statistical rigor and comparability across settings. Table 1 reports no variance or multi-seed results. The best-performing N=6 row is a checkpoint selected from an earlier run, and some of its critiques are based on sampled frames. The N=3 row disables skill retrieval and uses random seeds that are not matched to the other rows (Appendix A.2), so it remains unclear whether the reductions of 1.2–1.4 failures per task are significant. Please report means, variances, and significance tests over multiple seeds, clarify how the N=6 checkpoint was selected, and re-evaluate all settings under identical conditions.

  3. Unclear novelty over prior work, and the nature of the learned knowledge is not verified. The prompt-plus-skill-library context and the best/worst contrastive update are close in spirit to ACE, ExpeL, and Training-free GRPO. The paper does not make clear which components are specific to video generation. Moreover, all experiments use a single Wan2.1 backend, and the skills shown in the appendix include wording-level constraints (e.g., avoiding words such as “pull” and “stretch”), so it is impossible to tell whether the evolved context encodes transferable planning knowledge or prompt adaptation to one particular generator. Please clarify the technical contribution beyond these methods and evaluate the evolved context on at least one other I2V backend.

  4. Some claims exceed the evidence. Contribution (4) claims superiority in both semantic completion and video quality, yet on L-IVA the Reactive baseline outperforms VAEM on both TPF (8.73 vs. 8.87) and VMBench total (65.78 vs. 65.18). The abstract states that a real-time prototype validates VAEM’s compatibility with interactive rendering, but Appendix A.7 only evaluates the third-party Helios model and does not explain how it connects to VAEM’s planning pipeline.

Questions for authors

Please refer to the Weaknesses.

Reviewer_4LuLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Existing video avatar models often fail to consolidate past failures into persistent knowledge, leading to the repetition of similar errors across multi-step planning and execution tasks. To address this, the authors propose VAEM framework which distills video-generation histories into a persistent context comprising global behavioral principles and a reusable skill library. The framework operates via an agentic PGCU cycle, where a scene-aware planner and a frozen video generator are guided by an update agent that uses specialized VLM-based critics and the newly introduced SBM to provide fine-grained feedback. Experiments on half-body and full-body tasks demonstrate that the proposed CCE strategy can reduce semantic failures while enhancing visual quality.

Critical strengths
  • [S1] The paper is well-motivated and clearly identifies a concrete gap in prior work: existing video-avatar methods treat System 2 planning as episodic, so similar planning and execution errors recur across tasks instead of being consolidated into persistent knowledge, as described in the simple three-mode taxonomy (predictor, simulator, evolver) illustrated in Figure 1. The writing is logically organized, making the claims and statements easy to follow.

  • [S2] The proposed Plan-Generate-Critique-Update (PGCU) cycle is an intuitive and technically sound approach to self-evolution. By keeping the underlying video generator frozen and evolving only the planner's persistent context (system prompt and skill library), the framework effectively enables learning from past failures without the need for expensive fine-tuning. The use of the VLM judge to gauge SBM to provide fine-grained and localized feedback allows the update agent to distinguish between planning-related errors and rendering defects, leading to improvements in goal completion and visual quality.

Critical weaknesses
  • [W1] The results in Table 1 only compare the proposed VAEM variants (HCE and CCE) without providing direct comparisons against other established baselines for those specific tasks. Furthermore, the results on the L-IVA dataset in Table 2 show that the Reactive method actually outperforms VAEM across most of the SBM and VMBench metrics. Additionally, the ablation study in Table 4 reveals that the w/o skill retrieval variant demonstrates performance superiority over the full VAEM framework in the proposed SBM metrics. These findings raise concerns regarding the overall effectiveness and necessity of the proposed evolving mechanisms compared to simpler or existing alternative approaches.

  • [W2] The experimental setup relies heavily on proprietary leading VLMs and LLMs, specifically the use of gemini 3.1 pro for the most important VLM critics and the update agent, which are responsible for providing and absorbing targeted update feedback. This dependency raises significant concerns regarding the operational cost of the framework, leaving open the performance and viability of VAEM when implemented using open-source VLM- and LLM-based planning / critics / update agents, which are often preferred for practical deployment.

Questions for authors

Could the authors provide more technical details regarding the implementation of the baselines used in the experiments, so that the readers can be easier to assess the comparison results between VAEM and the baselines?

Reviewer_1m15OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

VAEM gives a video avatar a persistent planner context instead of an episodic one. A scene-aware planner decomposes a goal into segments, a frozen image-to-video backend renders them with anchor chaining, five VLM critics localise semantic failures along five dimensions, and an update agent rewrites the system prompt and a skill library. Two strategies are compared: holistic evolution over one plan per task, and contrastive evolution over four plans where the best and worst are used for the update. The headline is a reduction from 9.26 to 7.83 and from 8.85 to 7.65 failures.

Critical strengths
  1. The problem framing is sound. A frozen generator plus a planner whose context accumulates rules and skills is a practical design, and the failure-localisation-to-context-edit loop is described clearly enough to reimplement.

  2. The human study is a real attempt to validate an LLM-judge metric. The dimension-aligned pattern is more informative than a single agreement number.

Critical weaknesses
  1. The metric is the optimizer. SBM counts records written by five critic agents; contrastive evolution selects the best plan by minimising that same count; the same critics then score the test set. The critic also receives the plan text, so evolution can lower the count by producing plans the judge likes rather than videos that are better. The paper's own future-work list includes "evaluator models independent of the critics used for context evolution", which is exactly the control that belongs in this paper.

  2. The evidence for the headline claims does not survive the appendix. The N = 6 row is a historical checkpoint with 31 sampled-frame critiques and unmatched seeds, and Appendix A.2 says the two rows "record distinct evaluation settings rather than isolating the causal effect of additional rounds". The protocol-clean row (N = 3) disables skill retrieval and gives 9.26 → 8.34. On L-IVA the method's total failure count (8.87) is worse than ReAct's (8.73) and its VMBench total is lower (65.18 versus 65.78).

Questions for authors

Q1. Please re-score the frozen N = 3 and N = 6 videos with (a) a different judge and (b) a judge that is not shown the plan. How much of 9.26 → 7.83 and 8.85 → 7.65 survives? If the gain is judge-relative, what remains of the semantic-improvement claim?

Q2. How is state-of-the-art claimed on L-IVA when the aggregate failure count and the VMBench total are both worse than ReAct?

Q3. In Table 4 the full system is worse than the no-skill ablation on three of six SBM columns, and the differences between full and either ablation are 0.07–0.13 failures with no variance reported.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.