- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces an agent-based method to perform memory management in the context of long video generation. Specifically, it uses a vision-language agent to execute different operations on the memory bank, dynamically performing memory updates and state retrieval. Compared to the previous work, the memory agent is sort of different compared to the existing agent-based video generation works, which focus more on how to effectively utilize the VLM and VDM. Therefore, the proposed idea of a memory agent in this work is interesting.
Critical strengths
The paper's main strength is the proposed memory agent, which differentiates itself from existing agent-based works for video generation.
The paper also introduces some effective mechanisms to maintain a dynamic, entity-centric memory representation and an agentic management loop with verification.
The paper offers reasonable comparisons against established baselines.
Critical weaknesses
Though the proposed idea of this memory agent is interesting, the main weakness of this paper is its readability. In the main body of this paper, the developed techniques were only briefly presented. For instance, in Sec. 3.1 of DYNAMIC ENTITY-CENTRIC MEMORY, after introducing the concepts of memory bank and visual states, the paper briefly talks about the Disentangled Visual Reference Synthesis, Memory Writing, and State Retrieval, with only very brief details. This also happens to the Sec. 3.2, where it briefly introduces the OBSERVE–THINK–ACT–VERIFY steps. Honestly, it is very difficult to understand the core ideas of these different operations. I understand that for agent-based works, it involves many different operations. But it is the paper's responsibility to clearly convey the main ideas such that the paper can be well understood and easily reproduced. I have checked the appendix as well. It looks like many details have been moved to the appendix. This, however, makes the paper very difficult to read.
Further, what is the motivation of using reinforcement learning to enhance the memory agent? This part is not clearly presented in the introduction, where the techniques were suddenly introduced in the method section.
The architecture of EvoMA is also simply drafted in Figure 2. It uses very simple memory operations such as add, update, noop, and retrieve. It is not well aligned with the methodology description, where GRPO was very simply added inside the memory agent.
Another unclear point is the separation of entity’s persistent identity from its temporal visual states. This should be the main motivation of the proposed memory agent and is also the claimed main contribution of this proposed work. Though the paper briefly provides the definitions of the identity and visual states, the following descriptions do not seem to follow this claim. Thus, how this evolving state is demonstrated is unclear. The visualization provides certain demonstrations on how the states are maintained, which is a good sign.
A final mark, the paper only includes one video demo. There are no video comparisons as presented in Figures 3 and 4. These are just samples. However, a video comparisons must need to be provided to ensure that the evolving state is maintained across the time, rather that just on a few selected frames.
Questions for authors
What is the motivation for using reinforcement learning to enhance the memory agent?
How to separate the entity’s persistent identity from its temporal visual states from a technical perspective?
Are there any failure cases for the memory agent? If yes, what are the main reasons for that?
Why there are no more video demos, but just a single demo?
4 comments
2. Technical Separation of Persistent Identity from Temporal Visual States (W4 & Q2)
Technically, the separation is enforced at three levels: the memory schema, the memory operations, and the reference images.
-
Schema: a locked identity plus an append-only state list (§3.1, Eq. 1, Figure 2a). Each entity is stored as . The identity fields are fixed at registration and never edited afterwards. Only the state list grows, where each state stores only the mutable state features (e.g., clothing, injuries, lighting) and its state-specific reference images .
-
Operations: writes never overwrite identity or history (§3.2). ADD is used only for a genuinely new entity. UPDATE targets an existing and active base state (indexed by ), appending a new state obtained by applying the verified change while leaving and all earlier states intact. Transient variations (poses, camera framing, rendering flicker) produce NOOP, or are rejected by the verifier.
-
Pixels: clean state-specific references (§3.1). Raw frames mix identity, state, pose, and background. After an update is verified, we generate with an image-to-image call conditioned on two inputs. The base reference together with preserves identity. The clearest evidence frame together with injects only the changed attributes. Characters and objects are rendered against clean neutral backgrounds, and scenes are rendered without foreground subjects.
Example (Figure 2a). Character Elan is registered with fixed identity ("Elan, the messenger.") and initial state (clean). After he crosses a sandstorm, UPDATE appends (muddy, with ), while and remain untouched (and subsequent transitions similarly append rope repair and bandaged). If a later shot requires his initial clean appearance, RETRIEVE selects the pair (Elan, ) instead of the latest state. The retrieved identity description and state ( ) are then combined with the shot prompt , which controls only shot-specific pose and composition.
Empirical evidence (Table 8). Our decoupled Identity () and State () evaluations directly validate this design. Planning-driven baselines maintain high identity only by freezing entity states (e.g., MovieAgent: , ). In contrast, EvoMA (GRPO) substantially improves dynamic state alignment (, overall Mean , over Base) while preserving strong identity consistency (Mean ).
3. Presentation, Readability, and Alignment of Figure 2 with Methodology (W1 & W3)
We appreciate your constructive feedback on making the main text self-contained and aligning Figure 2 with the methodology:
-
Redesigned Figure 2: We have updated Figure 2 into a two-part diagram: (a) Inference loop, which illustrates the step-by-step flow from observing clip , proposing ADD/UPDATE/NOOP actions, running multimodal verification (APPROVE/MODIFY/REJECT), executing the evidence-guided I2I tool call, and appending the verified update to the memory bank before next-shot retrieval; and (b) Learning to write and retrieve, which depicts our two-stage GRPO training pipeline (training the writer with group rollouts , a frozen retriever, and a text judge, then freezing the writer to build student banks for training the retriever with a judge-aligned set-match reward).
-
Expanded Technical Details in §3.1–3.3: We have revised §3.1–3.3 to detail each step of the Observe-Think-Act-Verify loop (timestamped frame extraction in Observe, base-state comparison via in Think, formulating candidate ADD/UPDATE/NOOP actions with supporting clip timestamps in Act, and timestamped frame auditing plus evidence frame selection in Verify), and inlined the writing reward () and retrieval reward () in §3.3 to match Figure 2b.
4. Failure Cases of the Memory Agent and Main Reasons (Q3)
Inspecting our generation trajectories reveals two primary failure modes and their underlying causes:
-
Imperfect video generation and the trade-off between visual continuity and prompt following. During multi-shot generation, the underlying video model may exhibit hallucinations or fail to fully execute a scripted transition, especially under complex camera movements or fine-grained actions. For example, a script may specify that a character puts on a jacket and zips it up during the current clip, but the generated video only shows the character draping the jacket over the shoulders with the zipper left open. When the realized video diverges from the script, the memory agent faces an inherent trade-off between visual continuity and prompt following:
-
Prioritizing visual continuity (grounding in visual references): Because EvoMA strictly grounds memory updates in realized visual evidence rather than unrendered text, it records the observed unfastened jacket state (or rejects the unrendered zipping transition). This preserves smooth visual continuity across shots, but causes the character's jacket to remain unzipped in subsequent shots, registering as a state-alignment failure against the script.
-
Prioritizing prompt following (enforcing script text): Conversely, if the system overrides the visual memory and forces the character's jacket to appear fully zipped in the next shot according to the text prompt, it satisfies single-shot prompt compliance at the cost of an abrupt visual jump between consecutive shots.
-
-
Ambiguity in user edit instructions (interactive generation). In streaming interactive generation, user edits can be underspecified and admit multiple valid visual realizations. For example, an instruction such as "make the café look older" may be realized through faded wall paint, worn furniture, or dimmer lighting. EvoMA grounds the edited state in what is actually rendered at the onset shot and commits it to memory, so the same interpretation is consistently maintained in subsequent shots. However, when the realized interpretation differs from the one expected by the evaluator (e.g., the evaluator expects dimmer lighting while the generated scene shows worn furniture), the shot is counted as a state-alignment failure, even though the edit is realized and persists consistently. This failure stems from the inherent ambiguity of the instruction rather than from memory writing or retrieval. It can be mitigated through lightweight clarification, such as asking the user to confirm the intended attribute changes, or presenting several candidate states before the edit is committed to memory.
5. Additional Video Comparisons in Supplementary Material (W5 & Q4)
Following your suggestion, we have expanded our supplementary material with 4 multi-method side-by-side comparison videos (in addition to our illustrative interactive-generation demo):
-
2 non-interactive long-video comparison videos (EvoVideo-Long): Full synchronized multi-shot video comparisons across all baselines (AniMaker, MovieAgent, ViMax, VideoMemory, EvoMA Base, and EvoMA GRPO), including the narrative trajectory shown in Figure 3.
-
2 interactive video comparison videos (EvoVideo-Interactive): Full synchronized streaming video comparisons across all interactive baselines (LongLive 1.0, AniMaker, MovieAgent, ViMax, EvoMA Base, and EvoMA GRPO), including the interactive edit trajectory shown in Figure 4.
These videos demonstrate that EvoMA maintains evolving entity states continuously across entire multi-shot videos rather than only on selected static frames.
Thank you again for your thoughtful feedback. We hope our responses have helped clarify the points you raised. If you have any remaining concerns or would like further clarification, we would be happy to address them.
We sincerely thank Reviewer DfQ1 for recognizing the novelty of our proposed memory agent, the effectiveness of our dynamic entity-centric memory representation and agentic management loop with verification, and our comparisons against established baselines. We have carefully revised the manuscript (§1, §3, Figure 2, Appendix G, Appendix I, and Appendix J) and expanded the supplementary video comparisons to address all of your questions and concerns.
1. Motivation for Using Reinforcement Learning to Enhance the Memory Agent (W2 & Q1)
Our motivation for using reinforcement learning (GRPO) to train the memory agent operates at two levels:
Training the memory agent enhances its capability to maintain memory (§1 & §3.3). While our verification loop filters ungrounded writes at test time, off-the-shelf VLMs (EvoMA Base) are not calibrated to decide when to update versus preserve memory, nor to anticipate which written states will benefit downstream retrieval in future shots. Training the writing and retrieval policies therefore improves the agent's ability to balance selective state updates against memory preservation and optimize downstream retrieval utility.
RL (GRPO) outperforms Supervised Fine-Tuning (SFT) (Table 14 & Table 17). We also compared GRPO against SFT trained on the same curated teacher trajectories, and found that SFT underperforms GRPO (0.7641 vs. 0.8176 First-Frame ISA; 0.7573 vs. 0.8600 Reference Image ISA in Table 14). Analyzing the operational memory statistics in Table 17 reveals the underlying reason:
Base vs. Teacher distribution: The unoptimized Base model tends to propose more candidate writes (116 proposals across the 8-case subset). Because our pipeline includes an independent verification mechanism (APPROVE / MODIFY / REJECT), erroneous or redundant proposals from Base can be intercepted or corrected, committing 63 verified updates (95 final states). In contrast, because the stronger teacher model is highly accurate, most of its proposals are correct while NOOP remains the majority decision across shots.
Conservative bias in SFT vs. calibrated proposals in RL: When the student VLM is trained via SFT to fit the teacher's distribution, it learns to be overly restrained in proposing updates, dropping to only 80 proposals (53 verified updates) and 85 final states, which is even fewer than Base (95). For the student model, proposing too few updates misses critical entity additions and state transitions. Conversely, with our verification mechanism filtering out extra erroneous proposals at test time, the system can tolerate moderate over-proposals but cannot recover from omitted state transitions. By optimizing the policy directly on trajectories sampled from the student model itself rather than imitating the teacher's distribution, GRPO preserves active state exploration while improving proposal precision (87 proposals yielding 76 verified updates and 109 final states), achieving the highest downstream reference coverage (94.8% vs. 88.7% for SFT and 86.1% for Base).