acceptodds
Under review as a conference paper at ICLR 2027

MemWorld: Explore, Edit, and Revisit with Validity-Aware Multimodal Memory

Abstract

Interactive video world models support two complementary interaction modes: users can wander through generated environments using camera actions or direct their evolution through semantic interventions. Existing models typically handle these modes separately. We unify them within one evolving session, enabling users to explore, edit, continue, and revisit. This setting exposes a memory challenge: limited context causes forgetting during long-horizon revisits, while an edit leaves pre-edit observations geometrically useful but visually obsolete. Reusing them as RGB can revert the edit, whereas discarding them loses scene structure. We introduce MemWorld, a memory-augmented video world model with validity-aware multimodal memory. It retrieves nonlocal observations through camera-frustum overlap and uses a version-conditioned RGB–depth policy to preserve current appearance in RGB while retaining obsolete observations as structural depth. Retrieved memory, recent history, and current latents are jointly processed by a camera-conditioned video DiT. We evaluate MemWorld on our proposed editing benchmark of 500 cases covering global appearance editing, continued navigation, and pose-matched revisitation, together with R2M-Bench and MBench. Among the evaluated learned systems, MemWorld achieves the highest R2M-Bench point estimate and the highest Composite point estimate for persistent editing and geometry, while remaining competitive on MBench. These results show that multimodal memory can unify wandering and directing while preserving long-term scene consistency and persistent world updates.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
interactive video world models, video generation, long-term memory, persistent video worlds, video editing, multimodal memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dVR7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the persistence of edits in interactive video worlds. MemWorld retrieves historical observations based on camera frustum overlap, replaces RGB memories from before an edit with depth representations, and jointly processes episodic memory and recent history. This design aims to preserve scene structure while preventing outdated appearance from overriding the edited state.

Critical strengths

The paper addresses a practically relevant problem. Applying a consistent validity policy to outdated appearance in both episodic memory and recent history provides a clear approach to maintaining edits during subsequent generation.

The reported experimental results are competitive. The method improves edit persistence and performs well on two memory benchmarks.

Critical weaknesses

Critical Weaknesses

The contribution over prior work is incremental, with limited conceptual novelty. PermaVid already incorporates RGB and depth memory with a mechanism that accounts for edits. The main additions are the treatment of recent history and unified attention integration. However, the comparisons use different backbones, and the ablations do not share an identical state before the edit. These factors make it difficult to attribute the performance improvements to the proposed components.

There are inconsistencies in the method description and reported results. The main text states that Stage I uses only RGB, whereas the appendix states that depth is already incorporated during this stage. Furthermore, all six base metric values for the full model are identical between the ablation on 100 cases in Table 4 and the comparison reportedly conducted on 500 cases in Table 7. The evaluation sets and aggregation procedures require clarification.

Minor Weaknesses

Some implementation details are insufficiently specified, including the depth estimator, the frustum overlap threshold, and the voxelization settings. These omissions limit reproducibility.

The paper does not report inference latency, GPU memory usage, or the overhead associated with growth of the memory bank. This makes it difficult to assess practical interactivity and the computational cost of extended sessions.

Questions for authors

Please clarify the actual training configuration used in Stage I and explain why Tables 4 and 7 report identical values for all six metrics despite their different stated evaluation sizes.

Please confirm whether the PermaVid baseline uses its original trained checkpoint. Please also clarify whether the routing ablation can branch from the same saved state before the edit, so that the contribution of applying validity rules to recent history can be evaluated independently.

7 comments

Response to Reviewer dVR7

We sincerely thank the reviewer for the detailed reading and constructive questions. We provide completed responses to Q1–Q8 below. All manuscript revisions are highlighted in blue.

Response status overview

QuestionTopicStatus
Q1–Q8Novelty; same-backbone comparison; curriculum; table corrections; PermaVid checkpoint; validity ablation; reproducibility; efficiencyCompleted: novelty and Table 8 scope clarified; matched-backbone VACE-vs-joint control; curriculum and Table 4 corrected; PermaVid checkpoint clarified; identical-state validity ablation; implementation settings added; single-H20 efficiency profiling.

Q1. Methodological Novelty and Clarification

MemWorld's novelty is not geometry-based retrieval or RGB/depth memory alone, but a unified validity-aware framework for persistent world-state updates. As another Reviewer grab independently recognized:

"Distinguishing 'where to recall' from 'whether recalled content is still authoritative' is a genuinely useful abstraction for editable world models."

The reviewer further emphasized that this distinction "converts a vague consistency goal into an explicit, implementable state-update rule."

Specifically, relative to PermaVid, MemWorld applies one validity policy to both episodic memory and multi-frame recent history, then integrates memory, history, and current latents through the backbone's native camera-conditioned self-attention. A two-stage LoRA curriculum trains this shared pathway first for RGB recall and then for mixed RGB/depth conditioning.

AspectPermaVidMemWorld (Ours)
ObjectiveEdit-aware RGB/depth memoryPersistent world-state updates
Validity scopeRetrieved memoryEpisodic Memory + Recent History
IntegrationVACE referencesNative joint self-attention

The original Table 8 wording was imprecise: PermaVid performs memory routing at inference time on its released Wan2.1-14B checkpoint with a VACE-initialized context branch, whereas MemWorld trains a two-stage RGB/mixed-context LoRA curriculum. We corrected the Adaptation row of Table 8 accordingly and clarified that the table is a mechanism-level comparison rather than a same-backbone causal ablation.

Manuscript updates: Sec. 2.3, Persistent State Under Semantic Direction, concisely clarifies context-wide validity and native joint integration. Appendix C.2, Matched Comparison with PermaVid, including Table 8, identifies PermaVid's released-checkpoint adaptation and clarifies the scope of the mechanism-level comparison.

Q2. Same-Backbone Comparison

Same-Backbone Control. Both systems run Wan2.2-TI2V-5B with the same seed, commanded trajectory, and scorer, differing only in the integration interface — VACE-style reference-image conditioning (PermaVid-style) vs. our native joint self-attention with validity-aware routing (MemWorld). The VACE-style branch instantiates the same VACE conditioning architecture on our backbone (Wan2.2-TI2V-5B has no native VACE pretrained checkpoint) and undergoes the same two-stage training, making this an interface-level control rather than a PermaVid reproduction. On 80 matched cases (16 scenes × 5 styles) with identical commanded trajectories, edit persistence and pose-matched revisit consistency are the controlled evidence:

MetricPermaVid-style (VACE)MemWorld (joint)
CSD Retention ↑-0.14440.0562
MegaStyle Retention ↑-0.28410.0145
XEdit DepthCorr ↑0.88100.9291
XEdit DepthSSIM ↑0.84300.8930
Post DepthCorr ↑0.94340.9860
Post DepthSSIM ↑0.90140.9578

The VACE-style interface dilutes edited evidence over continued generation (negative retention), whereas joint routing keeps version-consistent evidence in the native attention context; with the backbone fixed, this reproduces the direction of Table 7 and supports the advantage of MemWorld's joint context-integration design over VACE-style reference conditioning.

Manuscript updates: Appendix H, Same-Backbone Controlled Comparison with PermaVid, including Table 14, reports the matched-backbone protocol and the raw persistence/consistency metrics.

Q3. Clarification of the Two-Stage Training Curriculum

We thank the reviewer for identifying this inconsistency. The correct curriculum is:

  • Stage I (20K steps): RGB-only episodic memory and recent history, used to establish long-term recall.
  • Stage II (10K steps): initialized from Stage I, with independent depth substitution in both context sources at probability 0.5.

All reported results use the final Stage-II checkpoint, without additional initialization or benchmark-specific fine-tuning. Table 2 and Appendix A incorrectly described Stage I as using depth substitution; we corrected them to match Sec. 3.4 and the actual evaluated checkpoint lineage.

Manuscript updates: Sec. 3.4, Updating What Remains Valid, and Sec. 4.1, Experimental Setup, clarify the RGB-only Stage I, mixed RGB/depth Stage II, and final evaluated checkpoint. Appendix A, Training Configuration, including Table 2, is corrected accordingly.

Q4. Clarification of Identical Results in Tables 4 and 7

We thank the reviewer for identifying this inconsistency. The original Table 4 mixed evaluation sets: its Full row was copied from the 500-case result in Table 7, although Table 4 was intended as a 100-case routing ablation.

We reran all Table 4 policies on the same 100 cases (20 scenes × 5 styles) from a shared cached pre-edit state. Table 4 now contains only these controlled 100-case results; Table 7 remains the separate 500-case cross-method comparison. The corrected Table 4 is reported with the complete fixed-prefix protocol in Q6.

Manuscript updates: Appendix B.1, Memory-Validity Ablation, including Table 4, now reports all routing policies on the same 100-case fixed-prefix set; Table 7 remains the separate 500-case cross-method comparison.

Q5. Clarification of the PermaVid Baseline Checkpoint

We use PermaVid's original trained checkpoint and inference pipeline, without retraining or fine-tuning, while matching the evaluation scenes, styles, camera actions, and metrics.

Q6. Identical-State Ablation for Recent History Validity

We isolated Recent History Validity on 100 cases (20 scenes × 5 styles) by generating each pre-edit rollout once and caching its RGB/depth Episodic Memory, Recent History, camera parameters, timestamps, absolute indices, and world-version labels. From this identical state, Memory-only converts stale Episodic-Memory RGB to depth but retains stale Recent-History RGB, whereas Memory+History converts both. The checkpoint, edited anchor, trajectory, context budget, sampler, and post-edit noise are identical; Recent History routing is the only difference.

Routing policyCSD Δ ↑MegaStyle Δ ↑Gram Δ ↑SigLIP-T Δ ↑XEdit DepthCorr ↑Post DepthCorr ↑
Memory-only0.04900.04070.00640.03670.95420.9893
Memory+History (Full)0.27310.23420.39280.08570.94750.9890

The full three-policy version (Appendix F) additionally includes the Keep stale RGB endpoint (CSD Δ 0.0042, MegaStyle Δ -0.0045, Post DepthCorr 0.9889), where retaining pre-edit RGB in both sources collapses edit propagation.

Memory+History improves CSD, MegaStyle, Gram, and SigLIP-T deltas over Memory-only by 0.2241, 0.1935, 0.3864, and 0.0490, respectively, while Post DepthCorr remains nearly unchanged (0.9893 vs. 0.9890). Composite rises from 0.5268 to 0.9860; the 0.4592 gap closely matches the original 0.4627, confirming that extending validity routing to Recent History improves edit propagation without materially reducing structural consistency.

Manuscript updates: Sec. 4.3, Ablation of Validity-Aware Context Routing, summarizes the conclusion; Appendix B.1, including Table 4, reports the controlled metrics; and Appendix F, Fixed-Prefix Routing Ablation, specifies the branching protocol.

Q7. Clarification of Implementation Details and Reproducibility

We now specify the previously omitted depth, retrieval, and voxelization settings:

ComponentImplementation details
Depth estimatorFrozen Depth Anything V3 DA3-LARGE, run per-frame on VAE-decoded RGB and re-encoded by the shared VAE as normalized three-channel grayscale at 1280 × 704. No ground-truth or renderer depth is used.
Frustum overlapEq. 2 query-normalized intersection with �0��0�; candidates below the threshold are removed before Top-K. If none remains, retrieval uses the temporally nearest eligible frames. The five-latent budget includes the persistent sink; the five most recent latents are excluded. Training and inference use identical settings.
VoxelizationTight world-space AABB over query and candidate-frustum corners; the longest axis is divided into 32 cubic cells. Near/far planes are 0.1/10.0 world units and �1��1�. Volumes and intersections are occupied-voxel counts, and the query mask is the union over the current window.
Context5 Episodic Memory + 5 Recent History + 10 Current latents.

Manuscript updates: Appendix B.2, Depth Estimation, Memory Context, and Retrieval Configuration, specifies the DA3-LARGE depth pipeline, retrieval threshold, fallback, context budget, and voxelization settings.

Reviewer_YsYwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MemWorld studies persistent appearance editing in camera-controlled video world models: explore a scene, insert an externally edited anchor, continue navigating, and revisit earlier viewpoints. The method maintains pose-indexed RGB and depth observations, retrieves nonlocal context using camera-frustum overlap, and applies a world-version rule that routes pre-edit observations as depth while retaining current-version observations as RGB. Retrieved memory, recent history, and noisy target latents are jointly processed through the backbone's native self-attention. The model is adapted with LoRA and evaluated on a 500-case global-style editing benchmark, R2M-Bench, and MBench. The reported results show improved editing persistence and revisit consistency, including a comparison with PermaVid.

Critical strengths
  1. The paper addresses persistent appearance editing while preserving scene geometry across navigation and revisits.
  2. The 500-case editing evaluation shows improved persistence and depth consistency over PermaVid, alongside strong R2M-Bench and competitive MBench performance.
  3. Tables 3-4 support applying validity routing to recent history as well as retrieved memory.
Critical weaknesses
  1. WorldPlay already combines recent context with geometry-based memory retrieval, while PermaVid already provides edit-aware RGB/depth memory. MemWorld mainly extends validity routing to recent history and uses joint native self-attention. The distinction should be sharper, especially since PermaVid also trains a mixed-modal generator (Sec. 4.1), despite Table 8 emphasizing training-free routing.

  2. The PermaVid comparison uses different backbones and action-matched but nonidentical poses; routing ablations also start from separately generated pre-edit prefixes. Same-backbone controls and identical-state branching are needed to isolate the gains. Memory baselines are present, but VMem is not evaluated and WonderWorld appears only in edit-free evaluation, leaving explicit spatial-memory comparisons under editing incomplete.

  3. Evaluation covers five global appearance styles using an external editor and assumes unchanged geometry. Local, geometry-altering, and repeated edits are not established, limiting broader claims of persistent world-state updating, particularly because PermaVid already considers local edits.

  4. Section 3.4 describes RGB-only Stage I (lines 277-286), whereas Table 2 specifies depth substitution in episodic memory with probability 0.5 (lines 674-684). The actual curriculum and evaluated checkpoint lineage need clarification.

Questions for authors
  1. What is the key contribution beyond WorldPlay's geometry-based context memory and PermaVid's edit-aware RGB/depth memory? Please clarify Table 8's training-free characterization in light of PermaVid's mixed-modal generator training.

  2. Can you isolate the integration-path gains with same-backbone controls and branch routing policies from an identical saved pre-edit state? Could you also provide a persistent-editing comparison with VMem or explain why it is not applicable?

  3. What evidence supports persistence after repeated global edits, and how would validity routing handle local or geometry-altering edits? If these settings are outside scope, could the claims be explicitly limited to global appearance persistence?

  4. Is Stage I RGB-only, as stated in Sec. 3.4, or already RGB/depth, as stated in Table 2? Please specify any additional initialization stage and the training lineage of the evaluated checkpoint.

9 comments

Response to Reviewer YsYw

We sincerely thank the reviewer for the careful assessment and constructive suggestions. We provide completed responses to Q1–Q4 below. All revisions are highlighted in blue in the revised manuscript.

Response status overview

QuestionTopicStatus
Q1–Q4Novelty and PermaVid characterization; comparison fairness and spatial-memory baselines; repeated global, local, and geometry-altering edits; training curriculum and evaluated checkpointCompleted: distinctions clarified and Table 8 corrected; matched-backbone VACE-vs-joint control, identical-state fixed-prefix branching, and VMem edited-anchor comparison; 29 three-intervention global/two-object rollouts plus an additional object-insertion example; Stage I/II definitions and checkpoint lineage corrected.

Q1. Methodological Novelty and Clarification of Table 8

MemWorld's novelty is not geometry-based retrieval or RGB/depth memory alone, but a unified validity-aware framework for persistent world-state updates. As another Reviewer grab independently recognized:

"Distinguishing 'where to recall' from 'whether recalled content is still authoritative' is a genuinely useful abstraction for editable world models."

The reviewer further emphasized that this distinction "converts a vague consistency goal into an explicit, implementable state-update rule."

Specifically, relative to WorldPlay, MemWorld makes retrieved and recent context version-aware after edits; relative to PermaVid, it applies one validity policy to both episodic memory and multi-frame recent history, then integrates memory, history, and current latents through the backbone's native camera-conditioned self-attention. A two-stage LoRA curriculum trains this shared pathway first for RGB recall and then for mixed RGB/depth conditioning.

AspectWorldPlayPermaVidMemWorld (Ours)
ObjectiveLong-term geometric recallEdit-aware RGB/depth memoryPersistent world-state updates
Validity scopeGeometric retrievalRetrieved memoryEpisodic Memory + Recent History
IntegrationGeometric memory and contextVACE referencesNative joint self-attention

The original Table 8 wording was imprecise: PermaVid performs memory routing at inference time on its released Wan2.1-14B checkpoint with a VACE-initialized context branch, whereas MemWorld trains a two-stage RGB/mixed-context LoRA curriculum. We corrected the Adaptation row of Table 8 accordingly and clarified that the table is a mechanism-level comparison rather than a same-backbone causal ablation.

Manuscript updates: Sec. 2.3, Persistent State Under Semantic Direction, concisely clarifies context-wide validity and native joint integration. Appendix C.2, Matched Comparison with PermaVid, including Table 8, identifies PermaVid's released-checkpoint adaptation and clarifies the scope of the mechanism-level comparison.

Q2. Comparison Fairness and Spatial-Memory Baselines

1. Same-Backbone Control. Both systems run Wan2.2-TI2V-5B with the same seed, commanded trajectory, and scorer, differing only in the integration interface — VACE-style reference-image conditioning (PermaVid-style) vs. our native joint self-attention with validity-aware routing (MemWorld). The VACE-style branch instantiates the same VACE conditioning architecture on our backbone (Wan2.2-TI2V-5B has no native VACE pretrained checkpoint) and undergoes the same two-stage training, making this an interface-level control rather than a PermaVid reproduction. On 80 matched cases (16 scenes × 5 styles) with identical commanded trajectories, edit persistence and pose-matched revisit consistency are the controlled evidence:

MetricPermaVid-style (VACE)MemWorld (joint)
CSD Retention ↑-0.14440.0562
MegaStyle Retention ↑-0.28410.0145
XEdit DepthCorr ↑0.88100.9291
XEdit DepthSSIM ↑0.84300.8930
Post DepthCorr ↑0.94340.9860
Post DepthSSIM ↑0.90140.9578

The VACE-style interface dilutes edited evidence over continued generation (negative retention), whereas joint routing keeps version-consistent evidence in the native attention context; with the backbone fixed, this reproduces the direction of Table 7 and supports the advantage of MemWorld's joint context-integration design over VACE-style reference conditioning.

2. Identical-State Routing Ablation. We reran the ablation as fixed-prefix branching on the same 100 cases (20 scenes × 5 styles): Keep stale RGB, Memory-only, and Memory+History (Full) all start from one cached pre-edit rollout and share the checkpoint, edited anchor, trajectory, and post-edit noise, so only the routing policy changes (protocol: Appendix F).

Routing PolicyCSD Δ ↑MegaStyle Δ ↑Gram Δ ↑SigLIP-T Δ ↑XEdit DepthCorr ↑Post DepthCorr ↑Composite ↑
Keep stale RGB0.0042-0.00450.00040.00210.96100.98890.1974
Memory-only0.04900.04070.00640.03670.95420.98930.5268
Memory+History (Full)0.27310.23420.39280.08570.94750.98900.9860

Relative to Memory-only, Full improves all four appearance deltas while leaving Post DepthCorr nearly unchanged; its same-run Composite gap closely matches the original gap of 0.4627 (Table 3). The routing advantage therefore persists after eliminating pre-edit prefix differences, supporting validity routing over both Episodic Memory and Recent History.

3. Comparison with VMem. We evaluated VMem (surfel-indexed view memory, Sec. 2), which lacks native persistent-edit support, via a minimal edited-anchor adaptation: the external edited anchor is inserted into its view memory while its original retrieval and generation remain unchanged, without MemWorld's validity routing (Appendix I).

The adapted VMem retains edits weakly and loses revisit geometry consistency: its surfel memory retrieves pre-edit RGB by spatial overlap without a world-version validity rule — the stale-RGB failure mode isolated by our Keep-stale-RGB branch (Appendix F) — whereas MemWorld's validity routing keeps the edited appearance authoritative. The MemWorld row uses its 500-case retention (Table 7); VMem is evaluated on the same 500 cases with trajectories, evaluation windows, and metrics.

MethodCSD Retention ↑Post DepthCorr ↑
VMem (Edited-Anchor Adaptation)0.02380.9682
MemWorld0.04490.9891

Overall, the matched-backbone control and identical-state branching isolate the memory integration pathway and context-wide validity routing contributions, respectively; the VMem comparison further shows that spatial-memory retrieval alone does not maintain edited appearance as effectively as validity-aware routing.

Manuscript updates: Appendix H, Same-Backbone Controlled Comparison with PermaVid (Table 14); Appendix F, Fixed-Prefix Routing Ablation (Table 12); and Appendix I, Edited-Anchor Comparison with VMem (Table 15) report the matched-backbone, identical-state branching, and adapted-VMem protocols, respectively, with their measured metrics.

Reviewer_grabOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets a concrete failure mode of interactive video world models: after a user applies a semantic appearance edit, previously observed RGB frames remain geometrically useful but visually obsolete, so retrieving them can revert the edit, while discarding them loses scene structure. MemWorld treats an edit as a persistent world-state update rather than a current-view transformation. The method stores generated observations with aligned RGB/depth latents, camera parameters, timestamps, and a world-version label; retrieves nonlocal observations by camera-frustum overlap; and applies a version-conditioned RGB–depth routing rule so that pre-edit observations are reused as depth (structure) while current-version observations remain RGB (appearance). Retrieved memory, a FIFO recent-history window, and the current noisy latents are concatenated and processed by the native self-attention of a frozen camera-conditioned video DiT (Wan2.2-TI2V-5B), with rank-64 LoRA adapters trained in two stages (RGB recall, then mixed RGB–depth conditioning under self-generated context). The paper also introduces a 500-case appearance-intervention benchmark (100 scenes × 5 styles) covering edit adoption, persistence, and pose-matched revisitation, and evaluates on R2M-Bench and MBench. MemWorld leads the proposed benchmark (Composite 0.965 vs. 0.694 for the strongest baseline), achieves the highest R2M-Bench point estimate among learned models (NMR 0.706 vs. 0.485), and ranks second on MBench (68.48 vs. 69.57). Ablations show that converting both episodic memory and recent history to depth is necessary for edit propagation (Composite 0.196 → 0.524 → 0.986).

Critical strengths
  1. The problem framing is precise and the central mechanism is well matched to it. Distinguishing "where to recall" from "whether recalled content is still authoritative" is a genuinely useful abstraction for editable world models, and the proposed solution — reusing obsolete observations as depth rather than discarding them — follows directly from it. This is consequential because it converts a vague consistency goal into an explicit, implementable state-update rule; the routing ablation (keeping stale RGB vs. memory-only conversion vs. memory + history conversion, Tab. 3) shows the design choice, not just extra capacity, drives edit propagation.

  2. The empirical evidence spans three settings and is reported with unusual care. Leading the proposed 500-case benchmark, achieving the best learned-model R2M-Bench NMR (0.706), and remaining competitive on MBench (68.48, second) shows that edit-aware memory does not trade away general revisit capability. The authors are also transparent about the limits of their evidence — they explicitly state that the PermaVid comparison is system-level rather than causal, that the Edit Axis saturates, and that the depth-reuse alignment scores hold by construction. This candor makes the claims easier to trust and to build on.

Critical weaknesses
  1. The central "our integration is better" claim is not causally isolated. The matched PermaVid comparison uses different backbones (Wan2.1-VACE-14B vs. Wan2.2-TI2V-5B), and the internal routing ablation acknowledges that pre-edit prefixes are not pixel-identical across variants. Since the paper's contribution is precisely the integration interface (validity applied to recent history + joint self-attention + trained LoRA pathway), the current evidence is consistent with, but does not demonstrate, that interface being the cause of the gains over PermaVid. This is consequential because it is the main mechanism-level novelty relative to the closest prior work. A same-backbone comparison (implementing the PermaVid-style reference conditioning on the same Wan2.2 model, or the proposed routing on the VACE backbone) would settle it; at minimum, the causal language in the abstract/introduction should be tempered.

  2. The edit-persistence claim rests almost entirely on a self-constructed benchmark with no human validation. The Composite is a harmonic mean of min–max-normalized axes whose bounds depend on the compared method set (the paper acknowledges this, and the matched table shows the Edit Axis saturating at 1.0 for both systems), so the headline 0.965 vs. 0.694 gap is not perceptually anchored. Since "does the edited appearance persist and does the scene still look right" is a visual judgment, a small blinded human study on a subset of the 500 cases (analogous to what other world-model benchmark papers report) would substantially strengthen the central claim. Without it, the external R2M-Bench/MBench results establish edit-free memory quality, not the persistence of edits.

Questions for authors
  1. Can you provide a human evaluation (even a modest one, e.g., 100–200 pairwise judgments on pose-matched revisits) comparing MemWorld with PermaVid and Lyra 2.0 on edit persistence and structural fidelity? This would directly anchor the Composite.
  2. Is a same-backbone comparison feasible — e.g., applying the memory-validity routing to the VACE backbone used by PermaVid, or implementing PermaVid-style reference conditioning inside Wan2.2? If not, please state explicitly which parts of the performance gap can and cannot be attributed to the integration design.
  3. How does the method behave under (i) geometry-altering edits and (ii) multiple sequential edits (world version > 2), where reusing pre-edit depth may itself become stale? Even a small qualitative study would clarify the scope of "persistent world updates."
  4. In the routing ablation, how much of the gap between memory-only and memory+history conversion could be explained by the different pre-edit prefixes? Can the prefix be frozen across variants to make this comparison causal?
  5. The 500-case benchmark is generated from 100 input scenes and five styles. Are scenes and styles disjoint across any train/test use of the model, and do the trajectories include returns after long excursions (comparable to R2M-Bench's 481-frame rollouts)?
6 comments

Response to Reviewer grab

We sincerely thank the reviewer for the careful reading and constructive suggestions. We provide completed responses to Q1–Q5 below. All revisions are highlighted in blue in the revised manuscript.

Response status overview

QuestionTopicCurrent status
Q1–Q5Human evaluation; same-backbone comparison; geometry-altering and multiple sequential edits; fixed-prefix routing ablation; benchmark independence and long-horizon evaluationCompleted: blinded 200-case comparison, matched-backbone VACE-vs-joint control on 80 cases (16 scenes × 5 styles), 29 three-intervention global/two-object rollouts plus an additional object-insertion example, controlled fixed-prefix ablation, and held-out leave-and-return evaluation.

Q1. Human Evaluation

To complement the normalized Composite score, we conducted a blinded pairwise study on 200 matched scene-style revisit cases uniformly sampled across five styles. For each case, three annotators viewed randomized anonymized Explore-Intervene-Revisit triplets and judged Edit Persistence and Structural Fidelity; ties were allowed, and 1/1/1 splits were recorded as ties. We report win/tie/loss rates, Wilson 95% confidence intervals for MemWorld's win rate, and Fleiss' kappa.

ComparisonEdit Persistence, % (Win/Tie/Loss)Structural Fidelity, % (Win/Tie/Loss)95% CI of MemWorld win rate
MemWorld vs. PermaVid68.0 / 11.0 / 21.0 (136/22/42)72.5 / 9.0 / 18.5 (145/18/37)Edit: [61.2, 74.1], Structure: [65.9, 78.2]
MemWorld vs. Lyra 2.075.5 / 10.5 / 14.0 (151/21/28)65.0 / 16.0 / 19.0 (130/32/38)Edit: [69.1, 80.9], Structure: [58.2, 71.3]

Fleiss' kappa was 0.62/0.66 against PermaVid and 0.60/0.52 against Lyra 2.0 for Edit Persistence/Structural Fidelity, indicating moderate-to-substantial agreement. The human preferences are consistent with the objective results: relative to PermaVid, MemWorld improves CSD Retention from -0.1154 to 0.0449, XEdit DepthCorr from 0.8299 to 0.9472, and Post DepthCorr from 0.8539 to 0.9891; relative to Lyra 2.0, it achieves higher CSD and MegaStyle Retention (0.0484/0.0095 vs. -0.0425/-0.0465).

Manuscript updates: Appendix E, Human Evaluation of Persistent Editing, now reports the blinded 200-case protocol, voting and tie rules, win/tie/loss rates, Wilson 95% confidence intervals, and Fleiss' kappa.

Q2. Same-Backbone Comparison

Same-Backbone Control. Both systems run Wan2.2-TI2V-5B with the same seed, commanded trajectory, and scorer, differing only in the integration interface — VACE-style reference-image conditioning (PermaVid-style) vs. our native joint self-attention with validity-aware routing (MemWorld). The VACE-style branch instantiates the same VACE conditioning architecture on our backbone (Wan2.2-TI2V-5B has no native VACE pretrained checkpoint) and undergoes the same two-stage training, making this an interface-level control rather than a PermaVid reproduction. On 80 matched cases (16 scenes × 5 styles) with identical commanded trajectories, edit persistence and pose-matched revisit consistency are the controlled evidence:

MetricPermaVid-style (VACE)MemWorld (joint)
CSD Retention ↑-0.14440.0562
MegaStyle Retention ↑-0.28410.0145
XEdit DepthCorr ↑0.88100.9291
XEdit DepthSSIM ↑0.84300.8930
Post DepthCorr ↑0.94340.9860
Post DepthSSIM ↑0.90140.9578

The VACE-style interface dilutes edited evidence over continued generation (negative retention), whereas joint routing keeps version-consistent evidence in the native attention context; with the backbone fixed, this reproduces the direction of Table 7 and supports the advantage of MemWorld's joint context-integration design over VACE-style reference conditioning.

Manuscript updates: Appendix H, Same-Backbone Controlled Comparison with PermaVid, including Table 14, reports the matched-backbone protocol and the raw persistence/consistency metrics.

Q3. Geometry-Altering and Multiple Sequential Edits

We analyzed 29 completed rollouts with three successive interventions (four world states): 19 global-appearance sequences and 10 local two-object sequences (add A → add B → recolor A), plus a standalone object-insertion example. The selected global sequences retain the requested style characteristics after each intervention, including the final displayed frame after the third update, evidencing continued appearance updating beyond two world versions; the local sequences show that inserted objects persist in later generated views and that recoloring the first object leaves the second object's visible state intact.

Geometry-altering edits never reuse stale depth: the local-edit extension cumulatively invalidates both RGB and depth for historical observations selected by frustum overlap ≥ 0.4 with the edit view, excludes them from retrieval and recent history, and re-estimates the edited anchor's depth, while observations outside the selected set retain their existing validity. These are selected qualitative demonstrations of the displayed operations under rotation-only camera inputs, not a success-rate estimate.

Manuscript updates: Sec. 3.4 clarifies the local-edit extension; Appendix J, Sequential Appearance and Geometry-Altering Edits, including Figures 8 and 9, reports the routing and edited-anchor/text-conditioning protocol and provides the frame-level visual evidence and per-case analysis.

Q4. Fixed Pre-edit Prefix for the Routing Ablation

We reran the ablation on the same 100 cases using an identical cached pre-edit state. For each case, all variants used the same cached pre-edit prefix and edited anchor, with identical post-edit stochastic inputs. We compared keeping stale RGB, converting only episodic-memory RGB to depth, and converting stale RGB in both memory and recent history.

Routing policyCSD Δ ↑MegaStyle Δ ↑Gram Δ ↑SigLIP-T Δ ↑XEdit DepthCorr ↑Post DepthCorr ↑
Keep stale RGB0.0042-0.00450.00040.00210.96100.9889
Memory-only0.04900.04070.00640.03670.95420.9893
Memory+History (Full)0.27310.23420.39280.08570.94750.9890

Relative to Memory-only, Memory+History improves CSD, MegaStyle, Gram, and SigLIP-T deltas by 0.2241, 0.1935, 0.3864, and 0.0490, respectively, while XEdit DepthCorr changes only from 0.9542 to 0.9475 and Post DepthCorr remains nearly unchanged (0.9893 vs. 0.9890). This controlled comparison confirms that routing stale recent-history RGB remains beneficial after controlling for the pre-edit prefix, without relying on a derived Composite from a distinct rerun.

Manuscript updates: Appendix F, Fixed-Prefix Routing Ablation, now reports fixed-prefix branching with matched post-edit stochastic inputs and the raw appearance and depth metrics for all three policies.

Q5. Benchmark Independence and Long-Horizon Editing Evaluation

Data independence. All 100 scenes are held out from training, and the 500 scene-style combinations are used only for evaluation. The five styles are applied at inference time by an external editor, without benchmark-specific fine-tuning.

Long-horizon evaluation. The 500-case benchmark uses the full approximately 481-frame Translation-rotation leave-and-return trajectory, one of the three standard R2M-Bench templates. Figure 6 evaluates persistence at the six post-edit temporal bins with valid pre-/post-edit pose matches. Separately, we evaluate edit-free memory on all three approximately 481-frame R2M-Bench trajectories.

Manuscript updates: Sec. 4.1, Experimental Setup, clarifies held-out evaluation data, the external editor, and the full approximately 481-frame editing trajectory. Appendix C.3, Editing Persistence over Time, specifies the same long-horizon trajectory, six pose-matched temporal bins, and all three approximately 481-frame R2M-Bench trajectories.

We hope that the human evaluation, same-backbone controlled comparison, sequential global/local edit study, fixed-prefix routing ablation, and benchmark and long-horizon clarifications address Q1–Q5. Please let us know if you have any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Reviewer Concerns and Our Responses

We sincerely thank all three reviewers for their constructive feedback. The reviewers recognized the practical significance of persistent world editing, the effectiveness of our validity-aware memory framework, and the strong experimental performance across editing and memory benchmarks. Their concerns primarily focused on comparisons with prior methods, experimental fairness, human validation, generalization, and reporting consistency. We have addressed these concerns through additional experiments and clarifications:

  1. Comparison with Prior Methods: We further clarified MemWorld's key methodological distinctions: a unified, version-aware memory framework that treats semantic edits as persistent world-state updates, applying consistent validity rules to both episodic memory and recent history and jointly integrating them through native self-attention. We highlighted these differences through detailed comparisons with WorldPlay and PermaVid.
  2. Experimental Fairness: Conducted a same-backbone controlled comparison, an identical-state routing ablation, and an additional VMem comparison, providing stronger evidence for the effectiveness of our proposed design.
  3. Human Validation: Added a blinded human evaluation, achieving 68.0%/72.5% win rates against PermaVid for edit persistence and structural fidelity, respectively.
  4. Generalization: Clarified the long-horizon evaluation protocol and provided additional experiments on sequential global and local edits, while explicitly acknowledging their current limitations.
  5. Reporting Consistency and Reproducibility: Corrected inconsistencies in the training description and reported results, clarified implementation and baseline details, and added inference-efficiency analysis.

All supplementary experiments, methodological clarifications, and corrections have been incorporated into the revised manuscript. We believe these substantial additions directly address the reviewers' central concerns and significantly strengthen the paper's technical contributions, experimental rigor, and reproducibility.

We would also like to respectfully bring the rebuttal timeline to the Area Chair's attention. We submitted our detailed responses and additional experimental evidence early in the rebuttal period, leaving ample opportunity for further discussion. However, none of the three reviewers provided substantive follow-up feedback for most of the discussion period. Consequently, we had limited opportunity to clarify any remaining concerns through direct interaction. We respectfully ask the Area Chair to take this situation into account and carefully consider the additional evidence and revised manuscript when making the final recommendation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.