- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses multi-step, goal-oriented video avatar generation, arguing that existing simulator-mode methods (e.g., ORCA) plan within individual tasks and fail to consolidate past failures into knowledge that transfers across tasks. The authors propose VAEM, which freezes the video generator (Wan2.1-I2V) and all model parameters and optimizes only the planner’s persistent context: global behavioral principles in the system prompt and a structured skill library. The method runs a Plan-Generate-Critique-Update (PGCU) cycle. A scene-aware CoT planner produces segment-level action plans, the frozen I2V model renders them segment by segment, five specialized VLM critics localize failures along physical plausibility, commonsense consistency, goal completion, cross-clip continuity, and object integrity (together the SBM metric), and an update agent distills the feedback into rules and skills. Two evolution strategies are compared: single-plan Holistic Context Evolution (HCE), and Contrastive Context Evolution (CCE), which samples four candidate plans per task and contrasts the best and worst trajectories. At inference, the context is frozen and the planner is called once. On self-built half-body and full-body manipulation benchmarks (200 test tasks each), CCE reduces the mean failure count from 9.26 to 7.83 and from 8.85 to 7.65, with slight VMBench gains. On L-IVA, VAEM achieves the best PAS and MSS and the lowest CMF and GCF failure counts, while requiring a single planner call at test time versus 5.59 for ORCA. The paper also reports a correlation study between SBM and human ratings, as well as component ablations.
Critical strengths
-
Meaningful problem setting with a practical solution. The paper identifies cross-task experience reuse as an open problem in multi-step video avatar generation. It offers a training-free solution that optimizes only the textual context while keeping the backend frozen. At test time VAEM needs a single planner call with no retries, compared with 5.59 calls and 2.02 retries for ORCA (Table 3), so it is considerably cheaper to deploy than methods that rely on test-time replanning.
-
SBM provides localized, interpretable feedback. SBM localizes failures to specific dimensions, segments, and timestamps, so it serves both as an evaluation metric and as the feedback signal for evolution. The human study provides some evidence of its validity (TPF vs. PPS: ρ = −0.71; GCF vs. TSR: ρ = −0.63).
-
The contrastive update targets a challenge specific to video. CCE contrasts good and bad trajectories under the same context to separate planning errors from stochastic rendering defects, a problem that text-only agents do not face. Under an equal trajectory-evaluation budget, CCE outperforms HCE on both aggregate metrics (TPF and VMBench total).
Critical weaknesses
-
The same critics drive optimization and evaluation, and the core claims lack independent validation. Both the evolution feedback and the final SBM scores come from critics built on the same Gemini-3.1-Pro backbone, so the reported gains may partly reflect fitting the critics rather than genuine improvement. The human study only measures the correlation between SBM and human ratings and does not report per-method human scores. Meanwhile, the VMBench gains are small (e.g., 69.0 → 69.8 on half-body).
-
Insufficient statistical rigor and comparability across settings. Table 1 reports no variance or multi-seed results. The best-performing N=6 row is a checkpoint selected from an earlier run, and some of its critiques are based on sampled frames. The N=3 row disables skill retrieval and uses random seeds that are not matched to the other rows (Appendix A.2), so it remains unclear whether the reductions of 1.2–1.4 failures per task are significant. Please report means, variances, and significance tests over multiple seeds, clarify how the N=6 checkpoint was selected, and re-evaluate all settings under identical conditions.
-
Unclear novelty over prior work, and the nature of the learned knowledge is not verified. The prompt-plus-skill-library context and the best/worst contrastive update are close in spirit to ACE, ExpeL, and Training-free GRPO. The paper does not make clear which components are specific to video generation. Moreover, all experiments use a single Wan2.1 backend, and the skills shown in the appendix include wording-level constraints (e.g., avoiding words such as “pull” and “stretch”), so it is impossible to tell whether the evolved context encodes transferable planning knowledge or prompt adaptation to one particular generator. Please clarify the technical contribution beyond these methods and evaluate the evolved context on at least one other I2V backend.
-
Some claims exceed the evidence. Contribution (4) claims superiority in both semantic completion and video quality, yet on L-IVA the Reactive baseline outperforms VAEM on both TPF (8.73 vs. 8.87) and VMBench total (65.78 vs. 65.18). The abstract states that a real-time prototype validates VAEM’s compatibility with interactive rendering, but Appendix A.7 only evaluates the third-party Helios model and does not explain how it connects to VAEM’s planning pipeline.
Questions for authors
Please refer to the Weaknesses.