- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the persistence of edits in interactive video worlds. MemWorld retrieves historical observations based on camera frustum overlap, replaces RGB memories from before an edit with depth representations, and jointly processes episodic memory and recent history. This design aims to preserve scene structure while preventing outdated appearance from overriding the edited state.
Critical strengths
The paper addresses a practically relevant problem. Applying a consistent validity policy to outdated appearance in both episodic memory and recent history provides a clear approach to maintaining edits during subsequent generation.
The reported experimental results are competitive. The method improves edit persistence and performs well on two memory benchmarks.
Critical weaknesses
Critical Weaknesses
The contribution over prior work is incremental, with limited conceptual novelty. PermaVid already incorporates RGB and depth memory with a mechanism that accounts for edits. The main additions are the treatment of recent history and unified attention integration. However, the comparisons use different backbones, and the ablations do not share an identical state before the edit. These factors make it difficult to attribute the performance improvements to the proposed components.
There are inconsistencies in the method description and reported results. The main text states that Stage I uses only RGB, whereas the appendix states that depth is already incorporated during this stage. Furthermore, all six base metric values for the full model are identical between the ablation on 100 cases in Table 4 and the comparison reportedly conducted on 500 cases in Table 7. The evaluation sets and aggregation procedures require clarification.
Minor Weaknesses
Some implementation details are insufficiently specified, including the depth estimator, the frustum overlap threshold, and the voxelization settings. These omissions limit reproducibility.
The paper does not report inference latency, GPU memory usage, or the overhead associated with growth of the memory bank. This makes it difficult to assess practical interactivity and the computational cost of extended sessions.
Questions for authors
Please clarify the actual training configuration used in Stage I and explain why Tables 4 and 7 report identical values for all six metrics despite their different stated evaluation sizes.
Please confirm whether the PermaVid baseline uses its original trained checkpoint. Please also clarify whether the routing ablation can branch from the same saved state before the edit, so that the contribution of applying validity rules to recent history can be evaluated independently.
7 comments
Q1. Methodological Novelty and Clarification
MemWorld's novelty is not geometry-based retrieval or RGB/depth memory alone, but a unified validity-aware framework for persistent world-state updates. As another Reviewer grab independently recognized:
"Distinguishing 'where to recall' from 'whether recalled content is still authoritative' is a genuinely useful abstraction for editable world models."
The reviewer further emphasized that this distinction "converts a vague consistency goal into an explicit, implementable state-update rule."
Specifically, relative to PermaVid, MemWorld applies one validity policy to both episodic memory and multi-frame recent history, then integrates memory, history, and current latents through the backbone's native camera-conditioned self-attention. A two-stage LoRA curriculum trains this shared pathway first for RGB recall and then for mixed RGB/depth conditioning.
| Aspect | PermaVid | MemWorld (Ours) |
|---|---|---|
| Objective | Edit-aware RGB/depth memory | Persistent world-state updates |
| Validity scope | Retrieved memory | Episodic Memory + Recent History |
| Integration | VACE references | Native joint self-attention |
The original Table 8 wording was imprecise: PermaVid performs memory routing at inference time on its released Wan2.1-14B checkpoint with a VACE-initialized context branch, whereas MemWorld trains a two-stage RGB/mixed-context LoRA curriculum. We corrected the Adaptation row of Table 8 accordingly and clarified that the table is a mechanism-level comparison rather than a same-backbone causal ablation.
Manuscript updates: Sec. 2.3, Persistent State Under Semantic Direction, concisely clarifies context-wide validity and native joint integration. Appendix C.2, Matched Comparison with PermaVid, including Table 8, identifies PermaVid's released-checkpoint adaptation and clarifies the scope of the mechanism-level comparison.
Q2. Same-Backbone Comparison
Same-Backbone Control. Both systems run Wan2.2-TI2V-5B with the same seed, commanded trajectory, and scorer, differing only in the integration interface — VACE-style reference-image conditioning (PermaVid-style) vs. our native joint self-attention with validity-aware routing (MemWorld). The VACE-style branch instantiates the same VACE conditioning architecture on our backbone (Wan2.2-TI2V-5B has no native VACE pretrained checkpoint) and undergoes the same two-stage training, making this an interface-level control rather than a PermaVid reproduction. On 80 matched cases (16 scenes × 5 styles) with identical commanded trajectories, edit persistence and pose-matched revisit consistency are the controlled evidence:
| Metric | PermaVid-style (VACE) | MemWorld (joint) |
|---|---|---|
| CSD Retention ↑ | -0.1444 | 0.0562 |
| MegaStyle Retention ↑ | -0.2841 | 0.0145 |
| XEdit DepthCorr ↑ | 0.8810 | 0.9291 |
| XEdit DepthSSIM ↑ | 0.8430 | 0.8930 |
| Post DepthCorr ↑ | 0.9434 | 0.9860 |
| Post DepthSSIM ↑ | 0.9014 | 0.9578 |
The VACE-style interface dilutes edited evidence over continued generation (negative retention), whereas joint routing keeps version-consistent evidence in the native attention context; with the backbone fixed, this reproduces the direction of Table 7 and supports the advantage of MemWorld's joint context-integration design over VACE-style reference conditioning.
Manuscript updates: Appendix H, Same-Backbone Controlled Comparison with PermaVid, including Table 14, reports the matched-backbone protocol and the raw persistence/consistency metrics.
Q3. Clarification of the Two-Stage Training Curriculum
We thank the reviewer for identifying this inconsistency. The correct curriculum is:
- Stage I (20K steps): RGB-only episodic memory and recent history, used to establish long-term recall.
- Stage II (10K steps): initialized from Stage I, with independent depth substitution in both context sources at probability 0.5.
All reported results use the final Stage-II checkpoint, without additional initialization or benchmark-specific fine-tuning. Table 2 and Appendix A incorrectly described Stage I as using depth substitution; we corrected them to match Sec. 3.4 and the actual evaluated checkpoint lineage.
Manuscript updates: Sec. 3.4, Updating What Remains Valid, and Sec. 4.1, Experimental Setup, clarify the RGB-only Stage I, mixed RGB/depth Stage II, and final evaluated checkpoint. Appendix A, Training Configuration, including Table 2, is corrected accordingly.
Q4. Clarification of Identical Results in Tables 4 and 7
We thank the reviewer for identifying this inconsistency. The original Table 4 mixed evaluation sets: its Full row was copied from the 500-case result in Table 7, although Table 4 was intended as a 100-case routing ablation.
We reran all Table 4 policies on the same 100 cases (20 scenes × 5 styles) from a shared cached pre-edit state. Table 4 now contains only these controlled 100-case results; Table 7 remains the separate 500-case cross-method comparison. The corrected Table 4 is reported with the complete fixed-prefix protocol in Q6.
Manuscript updates: Appendix B.1, Memory-Validity Ablation, including Table 4, now reports all routing policies on the same 100-case fixed-prefix set; Table 7 remains the separate 500-case cross-method comparison.
Q5. Clarification of the PermaVid Baseline Checkpoint
We use PermaVid's original trained checkpoint and inference pipeline, without retraining or fine-tuning, while matching the evaluation scenes, styles, camera actions, and metrics.
Q6. Identical-State Ablation for Recent History Validity
We isolated Recent History Validity on 100 cases (20 scenes × 5 styles) by generating each pre-edit rollout once and caching its RGB/depth Episodic Memory, Recent History, camera parameters, timestamps, absolute indices, and world-version labels. From this identical state, Memory-only converts stale Episodic-Memory RGB to depth but retains stale Recent-History RGB, whereas Memory+History converts both. The checkpoint, edited anchor, trajectory, context budget, sampler, and post-edit noise are identical; Recent History routing is the only difference.
| Routing policy | CSD Δ ↑ | MegaStyle Δ ↑ | Gram Δ ↑ | SigLIP-T Δ ↑ | XEdit DepthCorr ↑ | Post DepthCorr ↑ |
|---|---|---|---|---|---|---|
| Memory-only | 0.0490 | 0.0407 | 0.0064 | 0.0367 | 0.9542 | 0.9893 |
| Memory+History (Full) | 0.2731 | 0.2342 | 0.3928 | 0.0857 | 0.9475 | 0.9890 |
The full three-policy version (Appendix F) additionally includes the Keep stale RGB endpoint (CSD Δ 0.0042, MegaStyle Δ -0.0045, Post DepthCorr 0.9889), where retaining pre-edit RGB in both sources collapses edit propagation.
Memory+History improves CSD, MegaStyle, Gram, and SigLIP-T deltas over Memory-only by 0.2241, 0.1935, 0.3864, and 0.0490, respectively, while Post DepthCorr remains nearly unchanged (0.9893 vs. 0.9890). Composite rises from 0.5268 to 0.9860; the 0.4592 gap closely matches the original 0.4627, confirming that extending validity routing to Recent History improves edit propagation without materially reducing structural consistency.
Manuscript updates: Sec. 4.3, Ablation of Validity-Aware Context Routing, summarizes the conclusion; Appendix B.1, including Table 4, reports the controlled metrics; and Appendix F, Fixed-Prefix Routing Ablation, specifies the branching protocol.
Q7. Clarification of Implementation Details and Reproducibility
We now specify the previously omitted depth, retrieval, and voxelization settings:
| Component | Implementation details |
|---|---|
| Depth estimator | Frozen Depth Anything V3 DA3-LARGE, run per-frame on VAE-decoded RGB and re-encoded by the shared VAE as normalized three-channel grayscale at 1280 × 704. No ground-truth or renderer depth is used. |
| Frustum overlap | Eq. 2 query-normalized intersection with ; candidates below the threshold are removed before Top-K. If none remains, retrieval uses the temporally nearest eligible frames. The five-latent budget includes the persistent sink; the five most recent latents are excluded. Training and inference use identical settings. |
| Voxelization | Tight world-space AABB over query and candidate-frustum corners; the longest axis is divided into 32 cubic cells. Near/far planes are 0.1/10.0 world units and . Volumes and intersections are occupied-voxel counts, and the query mask is the union over the current window. |
| Context | 5 Episodic Memory + 5 Recent History + 10 Current latents. |
Manuscript updates: Appendix B.2, Depth Estimation, Memory Context, and Retrieval Configuration, specifies the DA3-LARGE depth pipeline, retrieval threshold, fallback, context budget, and voxelization settings.
Response to Reviewer dVR7
We sincerely thank the reviewer for the detailed reading and constructive questions. We provide completed responses to Q1–Q8 below. All manuscript revisions are highlighted in blue.
Response status overview