acceptodds
Under review as a conference paper at ICLR 2027

EvoMA: Evolving Entity Memory Agent for Consistent Long Video Generation

Abstract

Consistent long video generation requires preserving entity identity while tracking evolving states. Existing planning-based methods typically predefine entity-state trajectories and visual references, which can diverge from the actual content of generated clips and continuing user inputs. We introduce EvoMA, an Evolving Entity Memory Agent that treats memory management as an online decision-making process. EvoMA maintains a structured memory that separates persistent identities from evolving states and records state histories for characters, scenes, and objects using textual descriptions and visual references. A vision-language agent updates and queries this memory through an observe–think–act–verify loop, dynamically performing add, update, noop, and retrieve operations based on newly generated clips. Memory changes are verified against visual evidence before being committed, while retrieved entity-state references guide subsequent generation. We further train the memory agent with reinforcement learning to improve both memory updates and state retrieval. Experiments show that EvoMA improves identity consistency and state alignment in long and interactive video generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video agent, video generation, agent memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DfQ1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces an agent-based method to perform memory management in the context of long video generation. Specifically, it uses a vision-language agent to execute different operations on the memory bank, dynamically performing memory updates and state retrieval. Compared to the previous work, the memory agent is sort of different compared to the existing agent-based video generation works, which focus more on how to effectively utilize the VLM and VDM. Therefore, the proposed idea of a memory agent in this work is interesting.

Critical strengths

The paper's main strength is the proposed memory agent, which differentiates itself from existing agent-based works for video generation.

The paper also introduces some effective mechanisms to maintain a dynamic, entity-centric memory representation and an agentic management loop with verification.

The paper offers reasonable comparisons against established baselines.

Critical weaknesses

Though the proposed idea of this memory agent is interesting, the main weakness of this paper is its readability. In the main body of this paper, the developed techniques were only briefly presented. For instance, in Sec. 3.1 of DYNAMIC ENTITY-CENTRIC MEMORY, after introducing the concepts of memory bank and visual states, the paper briefly talks about the Disentangled Visual Reference Synthesis, Memory Writing, and State Retrieval, with only very brief details. This also happens to the Sec. 3.2, where it briefly introduces the OBSERVE–THINK–ACT–VERIFY steps. Honestly, it is very difficult to understand the core ideas of these different operations. I understand that for agent-based works, it involves many different operations. But it is the paper's responsibility to clearly convey the main ideas such that the paper can be well understood and easily reproduced. I have checked the appendix as well. It looks like many details have been moved to the appendix. This, however, makes the paper very difficult to read.

Further, what is the motivation of using reinforcement learning to enhance the memory agent? This part is not clearly presented in the introduction, where the techniques were suddenly introduced in the method section.

The architecture of EvoMA is also simply drafted in Figure 2. It uses very simple memory operations such as add, update, noop, and retrieve. It is not well aligned with the methodology description, where GRPO was very simply added inside the memory agent.

Another unclear point is the separation of entity’s persistent identity from its temporal visual states. This should be the main motivation of the proposed memory agent and is also the claimed main contribution of this proposed work. Though the paper briefly provides the definitions of the identity and visual states, the following descriptions do not seem to follow this claim. Thus, how this evolving state is demonstrated is unclear. The visualization provides certain demonstrations on how the states are maintained, which is a good sign.

A final mark, the paper only includes one video demo. There are no video comparisons as presented in Figures 3 and 4. These are just samples. However, a video comparisons must need to be provided to ensure that the evolving state is maintained across the time, rather that just on a few selected frames.

Questions for authors

What is the motivation for using reinforcement learning to enhance the memory agent?

How to separate the entity’s persistent identity from its temporal visual states from a technical perspective?

Are there any failure cases for the memory agent? If yes, what are the main reasons for that?

Why there are no more video demos, but just a single demo?

4 comments

We sincerely thank Reviewer DfQ1 for recognizing the novelty of our proposed memory agent, the effectiveness of our dynamic entity-centric memory representation and agentic management loop with verification, and our comparisons against established baselines. We have carefully revised the manuscript (§1, §3, Figure 2, Appendix G, Appendix I, and Appendix J) and expanded the supplementary video comparisons to address all of your questions and concerns.

1. Motivation for Using Reinforcement Learning to Enhance the Memory Agent (W2 & Q1)

Our motivation for using reinforcement learning (GRPO) to train the memory agent operates at two levels:

  • Training the memory agent enhances its capability to maintain memory (§1 & §3.3). While our verification loop filters ungrounded writes at test time, off-the-shelf VLMs (EvoMA Base) are not calibrated to decide when to update versus preserve memory, nor to anticipate which written states will benefit downstream retrieval in future shots. Training the writing and retrieval policies therefore improves the agent's ability to balance selective state updates against memory preservation and optimize downstream retrieval utility.

  • RL (GRPO) outperforms Supervised Fine-Tuning (SFT) (Table 14 & Table 17). We also compared GRPO against SFT trained on the same curated teacher trajectories, and found that SFT underperforms GRPO (0.7641 vs. 0.8176 First-Frame ISA; 0.7573 vs. 0.8600 Reference Image ISA in Table 14). Analyzing the operational memory statistics in Table 17 reveals the underlying reason:

    • Base vs. Teacher distribution: The unoptimized Base model tends to propose more candidate writes (116 proposals across the 8-case subset). Because our pipeline includes an independent verification mechanism (APPROVE / MODIFY / REJECT), erroneous or redundant proposals from Base can be intercepted or corrected, committing 63 verified updates (95 final states). In contrast, because the stronger teacher model is highly accurate, most of its proposals are correct while NOOP remains the majority decision across shots.

    • Conservative bias in SFT vs. calibrated proposals in RL: When the student VLM is trained via SFT to fit the teacher's distribution, it learns to be overly restrained in proposing updates, dropping to only 80 proposals (53 verified updates) and 85 final states, which is even fewer than Base (95). For the student model, proposing too few updates misses critical entity additions and state transitions. Conversely, with our verification mechanism filtering out extra erroneous proposals at test time, the system can tolerate moderate over-proposals but cannot recover from omitted state transitions. By optimizing the policy directly on trajectories sampled from the student model itself rather than imitating the teacher's distribution, GRPO preserves active state exploration while improving proposal precision (87 proposals yielding 76 verified updates and 109 final states), achieving the highest downstream reference coverage (94.8% vs. 88.7% for SFT and 86.1% for Base).

2. Technical Separation of Persistent Identity from Temporal Visual States (W4 & Q2)

Technically, the separation is enforced at three levels: the memory schema, the memory operations, and the reference images.

  • Schema: a locked identity plus an append-only state list (§3.1, Eq. 1, Figure 2a). Each entity is stored as me=(ide,ce,de,Se)m_e = (\mathrm{id}_e, c_e, d_e, \mathcal{S}_e). The identity fields (ide,ce,de)(\mathrm{id}_e, c_e, d_e) are fixed at registration and never edited afterwards. Only the state list Se=se,k\mathcal{S}_e = \\{s_{e,k}\\} grows, where each state se,k=(k,δe,k,Ie,k)s_{e,k} = (k, \delta_{e,k}, \mathcal{I}_{e,k}) stores only the mutable state features δe,k\delta_{e,k} (e.g., clothing, injuries, lighting) and its state-specific reference images Ie,k\mathcal{I}_{e,k}.

  • Operations: writes never overwrite identity or history (§3.2). ADD is used only for a genuinely new entity. UPDATE targets an existing ide\mathrm{id}_e and active base state se,kbase∈Rts_{e,k_{\mathrm{base}}} \in \mathcal{R}_t (indexed by kbasek_{\mathrm{base}}), appending a new state se,ks_{e,k} obtained by applying the verified change Δk\Delta_k while leaving ded_e and all earlier states se,0,…,se,k−1s_{e,0},\dots,s_{e,k-1} intact. Transient variations (poses, camera framing, rendering flicker) produce NOOP, or are rejected by the verifier.

  • Pixels: clean state-specific references (§3.1). Raw frames mix identity, state, pose, and background. After an update is verified, we generate Ie,k\mathcal{I}_{e,k} with an image-to-image call conditioned on two inputs. The base reference Ie,kbase\mathcal{I}_{e,k_{\mathrm{base}}} together with ded_e preserves identity. The clearest evidence frame together with δe,k\delta_{e,k} injects only the changed attributes. Characters and objects are rendered against clean neutral backgrounds, and scenes are rendered without foreground subjects.

Example (Figure 2a). Character Elan is registered with fixed identity ded_e ("Elan, the messenger.") and initial state s0s_0 (clean). After he crosses a sandstorm, UPDATE appends s1s_1 (muddy, with kbase=0k_{\mathrm{base}}=0), while ded_e and s0s_0 remain untouched (and subsequent transitions similarly append s2s_2 rope repair and s3s_3 bandaged). If a later shot requires his initial clean appearance, RETRIEVE selects the pair (Elan, s0s_0) instead of the latest state. The retrieved identity description ded_e and state ( δe,0,Ie,0\delta_{e,0}, \mathcal{I}_{e,0} ) are then combined with the shot prompt ut+1u_{t+1}, which controls only shot-specific pose and composition.

Empirical evidence (Table 8). Our decoupled Identity (II) and State (SS) evaluations directly validate this design. Planning-driven baselines maintain high identity only by freezing entity states (e.g., MovieAgent: IC=0.9692I_C = 0.9692, SC=0.3844S_C = 0.3844). In contrast, EvoMA (GRPO) substantially improves dynamic state alignment (SC=0.5705S_C = 0.5705, overall Mean S=0.6727S = 0.6727, +0.0772+0.0772 over Base) while preserving strong identity consistency (Mean I=0.9162I = 0.9162).

3. Presentation, Readability, and Alignment of Figure 2 with Methodology (W1 & W3)

We appreciate your constructive feedback on making the main text self-contained and aligning Figure 2 with the methodology:

  • Redesigned Figure 2: We have updated Figure 2 into a two-part diagram: (a) Inference loop, which illustrates the step-by-step flow from observing clip tt, proposing ADD/UPDATE/NOOP actions, running multimodal verification (APPROVE/MODIFY/REJECT), executing the evidence-guided I2I tool call, and appending the verified update to the memory bank before next-shot retrieval; and (b) Learning to write and retrieve, which depicts our two-stage GRPO training pipeline (training the writer with group rollouts agg=1G\\{a_g\\}_{g=1}^G, a frozen retriever, and a text judge, then freezing the writer to build student banks M~t\widetilde{\mathcal{M}}_t for training the retriever with a judge-aligned set-match reward).

  • Expanded Technical Details in §3.1–3.3: We have revised §3.1–3.3 to detail each step of the Observe-Think-Act-Verify loop (timestamped frame extraction in Observe, base-state comparison via Rt\mathcal{R}_t in Think, formulating candidate ADD/UPDATE/NOOP actions with supporting clip timestamps in Act, and timestamped frame auditing plus evidence frame selection in Verify), and inlined the writing reward (QcwriteQ_c^{\mathrm{write}}) and retrieval reward (QcretQ_c^{\mathrm{ret}}) in §3.3 to match Figure 2b.

4. Failure Cases of the Memory Agent and Main Reasons (Q3)

Inspecting our generation trajectories reveals two primary failure modes and their underlying causes:

  • Imperfect video generation and the trade-off between visual continuity and prompt following. During multi-shot generation, the underlying video model may exhibit hallucinations or fail to fully execute a scripted transition, especially under complex camera movements or fine-grained actions. For example, a script may specify that a character puts on a jacket and zips it up during the current clip, but the generated video only shows the character draping the jacket over the shoulders with the zipper left open. When the realized video diverges from the script, the memory agent faces an inherent trade-off between visual continuity and prompt following:

    • Prioritizing visual continuity (grounding in visual references): Because EvoMA strictly grounds memory updates in realized visual evidence rather than unrendered text, it records the observed unfastened jacket state (or rejects the unrendered zipping transition). This preserves smooth visual continuity across shots, but causes the character's jacket to remain unzipped in subsequent shots, registering as a state-alignment failure against the script.

    • Prioritizing prompt following (enforcing script text): Conversely, if the system overrides the visual memory and forces the character's jacket to appear fully zipped in the next shot according to the text prompt, it satisfies single-shot prompt compliance at the cost of an abrupt visual jump between consecutive shots.

  • Ambiguity in user edit instructions (interactive generation). In streaming interactive generation, user edits can be underspecified and admit multiple valid visual realizations. For example, an instruction such as "make the café look older" may be realized through faded wall paint, worn furniture, or dimmer lighting. EvoMA grounds the edited state in what is actually rendered at the onset shot and commits it to memory, so the same interpretation is consistently maintained in subsequent shots. However, when the realized interpretation differs from the one expected by the evaluator (e.g., the evaluator expects dimmer lighting while the generated scene shows worn furniture), the shot is counted as a state-alignment failure, even though the edit is realized and persists consistently. This failure stems from the inherent ambiguity of the instruction rather than from memory writing or retrieval. It can be mitigated through lightweight clarification, such as asking the user to confirm the intended attribute changes, or presenting several candidate states before the edit is committed to memory.

5. Additional Video Comparisons in Supplementary Material (W5 & Q4)

Following your suggestion, we have expanded our supplementary material with 4 multi-method side-by-side comparison videos (in addition to our illustrative interactive-generation demo):

  • 2 non-interactive long-video comparison videos (EvoVideo-Long): Full synchronized multi-shot video comparisons across all baselines (AniMaker, MovieAgent, ViMax, VideoMemory, EvoMA Base, and EvoMA GRPO), including the narrative trajectory shown in Figure 3.

  • 2 interactive video comparison videos (EvoVideo-Interactive): Full synchronized streaming video comparisons across all interactive baselines (LongLive 1.0, AniMaker, MovieAgent, ViMax, EvoMA Base, and EvoMA GRPO), including the interactive edit trajectory shown in Figure 4.

These videos demonstrate that EvoMA maintains evolving entity states continuously across entire multi-shot videos rather than only on selected static frames.

Thank you again for your thoughtful feedback. We hope our responses have helped clarify the points you raised. If you have any remaining concerns or would like further clarification, we would be happy to address them.

Reviewer_JY1COpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper provides a way to save long video memory as multi-modal entities, which proves useful to have the characters etc stay consistent for long video generation.

Critical strengths

The paper provides a way to save long video memory as multi-modal entities, which proves useful to have the characters etc stay consistent for long video generation.

Critical weaknesses
  • An overly narrow formulation of consistency in long-video generation. Long-video consistency goes far beyond maintaining entity identities and visual attributes. It encompasses 3D geometric consistency, spatial relationships, camera viewpoints, motion continuity, physical interactions, causal dependencies, visual style, and storyline coherence. More fundamentally, consistency is not simply about preserving the same appearance, but understanding what should remain invariant and what should change across time and viewpoints. For example, when a camera turns around, the background should appear completely different while remaining geometrically consistent with the same underlying 3D world. Similarly, characters may change poses and locations while maintaining coherent motion and interactions.

I believe the fundamental limitation is that EvoMA does not address consistency at its underlying level, but instead introduces a heuristic mechanism for maintaining selected entity attributes through memory updates and reference retrieval. Genuine long-video consistency requires an understanding of the underlying world, including its geometry, dynamics, causal relationships, and narrative progression, so that the model can determine how a scene should evolve while remaining coherent. Simply storing and retrieving entity appearances does not provide this understanding. While the proposed approach may improve certain observable consistency metrics, it does not fundamentally solve the consistency problem that motivates the paper.

  • The predefined entity vocabulary fundamentally limits the flexibility of memory representation. This narrow formulation is further reflected in EvoMA's memory design, which restricts memory to three predefined categories: characters, scenes, and objects. While individual entities and their states can evolve, the underlying memory structure remains fixed. Different videos may require fundamentally different memory representations, such as 3D scene graphs, spatial relationships, motion trajectories, physical dynamics, or narrative dependencies. The central limitation is that EvoMA learns when to update and retrieve memories, but not what kinds of information or structures should be remembered in the first place. Consequently, the approach is heavily dependent on a hand-designed memory representation, making it more of an engineered solution for maintaining selected entity attributes than a general framework capable of adapting to different consistency requirements.

  • Insufficient qualitative evaluation for a video-generation paper. The paper would benefit from substantially more qualitative video results, particularly side-by-side video comparisons for the ablation studies. These would help demonstrate what types of consistency failures are actually addressed by each component rather than relying primarily on aggregate scores. Moreover, the evaluation focuses largely on shot-initial frames, which cannot adequately capture motion continuity, spatial coherence, or identity drift within generated clips. Full-video comparisons are essential for substantiating the claimed improvements in long-video consistency. Right now we cannot see real consistency from the qualitative resutls.

Questions for authors

See weakness.

3 comments

We sincerely thank Reviewer JY1C for recognizing the usefulness of our multimodal entity memory in maintaining consistency for long-video generation, and for highlighting the broader requirements of video consistency, memory representation, and evaluation. Below, we clarify the scope of our contribution, discuss how our representation operates and can be extended, and present our expanded qualitative and full-clip evaluations.

1. Scope of Consistency in Long-Video Generation (W1)

We agree that long-video consistency encompasses multiple dimensions, including spatial relationships, 3D geometry, motion continuity, physical interactions, and narrative coherence. We situate our formulation and evaluation within this broader landscape from two perspectives:

  • Entity consistency is an established and essential research objective in long-form video generation. A substantial body of recent work specifically targets entity and scene consistency across shots: VideoDirectorGPT (COLM 2024) maintains shared entity representations through explicit consistency groupings; MovieDreamer (ICLR 2025) introduces identity-preserving rendering for recurring characters across long narratives; HoloCine (CVPR 2026) models persistent implicit memory for characters and scenes; OneStory (CVPR 2026) targets character and environment consistency, while demonstrating identity preservation under prompted clothing and environment changes. Recent agentic frameworks such as VideoMemory and ViMax similarly focus on tracking recurring entities and their states across shots. These works establish entity-state consistency as a substantive problem within the broader goal of coherent long-video generation.

  • Our evaluation also assesses coherence beyond entity-specific appearance matching (§4.1, Table 1). In addition to EvoVideoBench, our main experiments include ST-Bench, a public benchmark for multi-shot video storytelling whose ViCLIP-based cross-shot consistency and global/single-shot prompt-following metrics provide complementary evidence of narrative coherence across clips. Global visual style is also maintained across shots through unified textual conditioning (§3.1). Together, these components broaden our evaluation scope.

2. Flexibility of the Memory Representation (W2)

Characters, scenes, and objects provide a broadly applicable semantic foundation for narrative video: they represent recurring participants, environments, and interaction targets whose identities and states must be tracked across shots. Our choice of these categories provides a reusable basis for organizing narrative memory:

  • A predefined organization with evolving entities and states (§3.1–3.3). While the top-level categories specify how memory is organized, the concrete entities, textual state features, visual references, and state histories evolve online during generation. In particular, separating persistent identity from evolving state allows EvoMA to preserve an entity’s identity while accommodating narrative-driven changes in clothing, object conditions, or environments. The writer determines which observed changes should become reusable memory, and the retriever selects the states relevant to subsequent shots. Thus, the representation supports both continuity and change, although the agent does not learn the memory ontology itself.

  • Transferring EvoMA to spatial, geometric, and viewpoint-aware representations. Because our Observe-Think-Act-Verify loop and two-level GRPO optimization are decoupled from the specific attributes stored in each state, our framework can be naturally extended to spatial and geometric domains. Specifically, we can associate entities with positions, orientations, and trajectories in a shared scene coordinate system while recording camera poses and timestamps for visual observations. Under this augmented representation, retrieval can jointly evaluate state validity and viewpoint compatibility, supplying appropriate entity references and spatial context to generators that support layout or camera conditioning. Consequently, a camera movement would update the projected view without rewriting the underlying scene state, whereas an entity-state transition would update the corresponding state entry while preserving valid spatial structure. This extension retains our agentic write-and-retrieve workflow while generalizing the representation to geometric and viewpoint changes, and we will explore this extension as a concrete next step.

3. Expanded Video Comparisons, Qualitative Ablations, and Evaluation Beyond First Frames (W3)

We agree that shot-initial frames alone cannot fully capture intra-clip identity drift, motion continuity, or transition dynamics. To address this concern, we have expanded our supplementary video comparisons, added qualitative ablation visualizations, and extended our quantitative evaluation across the full clip duration. See the revised paper in blue text:

  • Full synchronized video comparisons in the supplementary material. We have expanded our supplementary material with 4 multi-method side-by-side comparison videos:

    • Two non-interactive long-video comparisons (EvoVideo-Long): Full synchronized multi-shot comparisons across AniMaker, MovieAgent, ViMax, VideoMemory, EvoMA (Base), and EvoMA (GRPO), including the narrative trajectory shown in Figure 3.

    • Two interactive video comparisons (EvoVideo-Interactive): Full synchronized streaming-video comparisons across LongLive 1.0, AniMaker, MovieAgent, ViMax, EvoMA (Base), and EvoMA (GRPO), including the interactive edit trajectory shown in Figure 4.

    These side-by-side videos allow direct inspection of identity stability across camera movements and state persistence across shot transitions.

  • Qualitative policy and planner ablations (Appendix G.4, Figure 5 & Appendix I, Figures 6–7). We have added matched qualitative comparisons across six ablation configurations (EvoMA Base, Writer only, Retriever only, EvoMA GRPO, and Qwen + Base/GRPO) tracking concurrent railway-signal progression and character knee taping (Figure 5). Full EvoMA (GRPO) accurately coordinates both transition timing and state persistence across both the Gemini and Qwen planners. Additional multi-shot narrative and interactive qualitative comparisons are provided in Appendix I (Figures 6–7).

  • Quantitative evaluation across the clip duration (Appendix E.6, Tables 12–13). While our primary shot-initial evaluation tests whether reference-conditioned generation begins with the intended identity and state, we now additionally evaluate five uniformly sampled frames per clip across all six methods on the complete EvoVideo-Long benchmark. Under this full-clip evaluation, EvoMA (GRPO) achieves the highest ISA across all three entity categories, reaching a Mean ISA of 0.7832 compared with 0.7458 for EvoMA (Base) and 0.6473 for the strongest external baseline (ViMax). These results confirm that our consistency improvements persist throughout the generated clips, complementing the full-video qualitative comparisons provided in the supplementary material.

I thank the authors for their detailed and thoughtful rebuttal and for the substantial additional experiments and qualitative comparisons.

The new full-video evaluations and side-by-side comparisons address my concerns regarding insufficient qualitative evidence. I also appreciate the clarification that entity-state consistency is the specific focus of this work.

My conceptual concerns regarding the predefined memory representation and its limited ability to capture underlying spatial, physical, and causal consistency remain. Nevertheless, I recognize the technical contributions and empirical improvements within the stated scope.

Considering the additional evidence and clarifications, I am willing to raise my rating to 3 (Weak Acceptance).

Reviewer_ZrsvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper aims to improve entity identity preservation ability in multi-shot long video generation. The proposed framework, EvoMA, uses explicit memory to maintain all entities and their states, and applies agents to update and retrieve memory. The memory agents are trained with reinforcement learning (RL) to improve performance. The framework is evaluated on ST-Bench and a newly proposed benchmark, EvoVideoBEnch. EvoMA demonstrates superior performance in prompt following and identity preservation.

Critical strengths
  • The main contribution of the paper lies in its system design. EvoMA includes several modules, including shot prompt generation, memory management, first frame generation, and video generation. The effectiveness of each new module is verified through ablation studies, and the comparison results demonstrate its superior performance.
  • Compared to previous methods based on memory agents, the technical contribution of the paper is that it uses separate agents for writing and retrieval policies and trains them with RL.
Critical weaknesses
  • The first two contributions claimed in the paper are not well supported. The paper lists the contributions as "formulate memory management as an online entity-state decision problem" and "We introduce a dynamic entity-centric memory representation and agentic management loop". However, these contributions are insufficiently differentiated from CoTriSyGen, which also updates entity-centric memory online and supports retrieval. Nevertheless, the contribution of introducing RL to memory agents is not affected by this issue.
Questions for authors

Both VideoMemory and CoTriSyGen apply an entity-centric memory bank. Could the authors elaborate on the differences between EvoMA and these two methods in terms of memory management?

3 comments

We sincerely thank Reviewer Zrsv for the positive assessment of our work, and for recognizing our modular system design, ablation studies, and the technical contribution of training separate memory writing and retrieval policies with RL. While EvoMA shares with VideoMemory and CoTriSyGen an entity-centric memory bank and the concepts of entities and states, they differ fundamentally in how memory is updated, verified, represented, and learned:

1. Vs. VideoMemory: VideoMemory plans entity states upfront from the full storyboard before video generation, whereas EvoMA updates memory online from observed video clips.

In our formulation, a memory bank is truly dynamic (observation-driven) only when entity states evolve online from realized visual outcomes in generated clips (and arriving user inputs).

  • VideoMemory is planning-driven (relatively static) and cannot support interactive streaming (Table 3). Although termed a "Dynamic Memory Bank," VideoMemory constructs entity states and references upfront from the complete storyboard before keyframe and video synthesis (in its released implementation), and generated clips never feed back into memory. Because it requires the full script in advance, it cannot operate in causal interactive streaming (EvoVideo-Interactive, Table 3) where future user edits are unknown beforehand.

  • Observation-grounded updates substantially outperform planning-driven memory in long videos (Tables 2 & 8). Even in offline generation under identical image/video generators and the same base VLM (Qwen3-VL-8B), planning-driven memory struggles when realized visuals diverge from the plan. On EvoVideo-Long (Tables 2 & 8), EvoMA outperforms VideoMemory in both First-Frame ISA (0.7438 Base / 0.7944 GRPO vs. 0.6129) and Reference Image ISA (0.7403 Base / 0.8235 GRPO vs. 0.5416), driven by large gains in dynamic State Alignment SS (Character/Object SS: 0.5705 / 0.6880 vs. 0.2907 / 0.3642).

2. Vs. CoTriSyGen: Within observation-driven memory, EvoMA introduces independent write verification, background-disentangled reference synthesis, and RL-trained policies.

CoTriSyGen is closer to EvoMA in also updating memory online from generated clips (we did not compare with CoTriSyGen quantitatively because its code and benchmark are not open-sourced). Within this observation-driven paradigm, EvoMA differs in three key mechanisms:

  • EvoMA verifies candidate writes (APPROVE / MODIFY / REJECT) before commitment, whereas CoTriSyGen commits analyzer outputs directly into memory. CoTriSyGen's post-video workflow directly applies the Video Analyzer's updates to persistent memory (M←M∪UtM \leftarrow M \cup U_t) without a separate write-review stage (CoTriSyGen, §3.2.2 & Alg. 1). In contrast, EvoMA separates proposing a write from committing it (§3.2): our writer proposes candidate actions with supporting clip timestamps, and an independent verifier inspects the cited video frames and historical memory to APPROVE, MODIFY (correcting the action type, base state, state features, or evidence frame), or REJECT (filtering out ungrounded, transient, or duplicate writes). Because a single erroneous write in persistent memory can corrupt multiple future shots, this gatekeeping is critical: removing verification causes exact duplicate states to jump from 0 to 8 (GRPO) and 18 (Base) (Table 17) and drops First-Frame ISA from 0.8176 to 0.7819 (Table 14).

  • EvoMA synthesizes clean, background-disentangled state references, whereas CoTriSyGen stores raw video frames that entangle entities with scene backgrounds. CoTriSyGen's post-video update selects and saves a raw video frame (frame_index) from the generated clip to represent an evolved entity or state (CoTriSyGen, Appendix §8.2.2.2). EvoMA instead synthesizes disentangled reference images (Ie,k\mathcal{I}_{e,k}) via an evidence-guided I2I call (§3.1), rendering characters and objects on clean neutral backgrounds and scenes without foreground subjects. For example, if a character puts on a raincoat inside a shop, saving the raw video frame entangles the raincoat appearance with the shop interior, causing background leakage when the character later appears on an outdoor street; synthesizing a clean-background asset isolates the raincoat state for conflict-free cross-scene reuse.

  • EvoMA optimizes dedicated writing and retrieval policies via GRPO, whereas CoTriSyGen relies on zero-shot VLM prompting. EvoMA trains separate writing (πθwrite\pi_\theta^{\mathrm{write}}) and retrieval (πϕret\pi_\phi^{\mathrm{ret}}) policies with GRPO and two-level credit assignment (§3.3), rewarding the writer for downstream retrieval utility (Eq. 4) and training the retriever on writer-materialized memory banks (Appendix A.2). Compared with zero-shot prompting (Base), GRPO improves Reference Image ISA from 0.7745 to 0.8600 and First-Frame ISA from 0.7616 to 0.8176 (Table 14), generalizing consistently across Qwen3-VL-4B and InternVL3.5-8B (Table 18).

Thanks for the authors' reply. I think the most related work is CoTriSyGen. As the authors clarified, EvoMA makes several improvements compared to it. Since my concerns are resolved, I still maintain my Week Acceptance score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.