acceptodds
Under review as a conference paper at ICLR 2027

MERC-WM: Mechanism-Event Replay Cards for Task-Free Continual World Models

Abstract

World models deployed in physical systems encounter a stream in which camera appearance, contact dynamics, payload, and actuator response change without task labels. Updating on every prediction error wastes adaptation capacity on harmless visual variation, whereas ignoring errors caused by changed dynamics yields unsafe plans. We formulate this ambiguity as task-free continual mechanism identification and propose MERC-WM, a continual world-model framework that separates observation shifts from mechanism shifts before updating its dynamics. MERC-WM combines paired low-risk probes, a frozen visual representation, a bank of low-rank dynamics experts, and a bounded memory of mechanism-event replay cards. Each card stores a compact before/after latent effect, uncertainty, and expert identity rather than raw trajectories. At test time, retrieved cards provide a prior over the active mechanism; the world model either reuses an existing expert or creates a new one only when probe evidence remains inconsistent. We specify a reproducible evaluation across contact-rich manipulation streams with appearance changes, friction and payload changes, and mechanism recurrence. Across three manipulation domains and five stream seeds, MERC-WM reaches 80.6% average success, improves backward transfer from −3.2 to −0.6 percentage points relative to latent replay, and reduces visual-shift false expansion from 37.5% to 8.6% using 18 MB of auxiliary memory. It also raises mechanism-event F1 from 73.0% to 87.9% and shortens recurrence recovery from 353 to 147 interactions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
World Models, Continual Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_EQzKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MERC-WM, a task-free continual world model that uses paired probes, mechanism-event replay cards, and low-rank experts to handle changing physical dynamics. Experiments on three manipulation tasks show improvements in adaptation, memory retention, and mechanism recovery.

Critical strengths

1.The problem is interesting and relevant to continual robot learning.

2.The evaluation includes multiple metrics, matched baselines, and useful ablations. The improvements in false expansion and recovery are encouraging.

Critical weaknesses
  1. The framework mainly combines existing techniques, including change detection, compact replay, and expert routing. I'm not fully convinced by the algorithmic novelty. Also, closely related methods such as SPREAD, ReDRAW, and LLCD are discussed but not directly compared. The success gain over the matched Residual-MoE baseline is only 1.8 points.

  2. All experiments are simulation-based. The method also relies on controlled source interventions and calibrated safe probes, which may be difficult to obtain in real environments. The main evaluation involves only three physical mechanisms, so I'm not sure how well it would generalize to more complex real-world changes.

Minor Comments

1.The method relies on a fixed visual encoder, which may limit adaptation to unseen observation changes.

2.The memory-efficiency claims would benefit from clearer reporting of actual memory usage versus configured budgets.

3.Appendix Table 6 reports slightly different false-expansion and recovery values for the default configuration compared with the main results. Please clarify.

Questions for authors

1.How does MERC-WM compare against SPREAD, ReDRAW, and LLCD under matched settings?

2.Can the authors validate the method on real robots or less controlled physical changes?

3.How sensitive is the method to inaccurate probe calibration or unavailable safe probes?

2 comments
AuthorsOpenReview ↗

Response to Reviewer EQzK (Part 1/2): Algorithmic Novelty, Direct Comparison with SPREAD/ReDRAW/LLCD, and Saturation Gains

We thank Reviewer EQzK for the constructive review and for recognizing the relevance of our problem setting, our multi-metric evaluation, and the encouraging gains in false expansion and recurrence recovery. Below, we address algorithmic novelty and direct baseline comparisons (W1 & Q1); Part 2 addresses generalization beyond 3 mechanisms, probe calibration sensitivity, and the three minor comments (W2, Q2, Q3, Minor 1–3).


Response to W1: Algorithmic Novelty Beyond Modular Combination

We respectfully clarify that MERC-WM is not a straightforward assembly of off-the-shelf change detection, replay, and MoE routing. In fact, that exact modular combination—action-conditioned residual GLR/CUSUM change detection + fixed-budget replay window + matched 4-expert LoRA routing—is our Residual-MoE baseline. MERC-WM introduces a fundamentally different adaptation principle: gating dynamics updates on matched, nuisance-suppressed action-effect evidence rather than raw observation residuals, realized through three tightly coupled algorithmic mechanisms:

  1. Paired Safe Probes with Generalized-Eigenvector Nuisance Suppression: Formulating PP via min⁡QQ⊤=Iptr(QWCnW⊤Q⊤)\min_{QQ^\top=I_p} \mathrm{tr}(QWC_nW^\top Q^\top) and requiring co-oriented raw-residual cosine agreement (qagree>τagreeq_{\text{agree}} > \tau_{\text{agree}}) on matched probes cleanly separates observation shifts from physical mechanism shifts before any parameter update occurs.
  2. Event-Disjoint Card Distillation & Reliability Priors: Storing 700-byte effect-statistic cards rather than trajectories, with separate train/validation splits and decayed Bayesian reliability ρi\rho_i, enables fast recurrence reactivation without trajectory leakage.
  3. Bilateral Equivalence Merging (Algorithm 3): Rather than naive LoRA weight averaging or unvalidated slot overwrites, merging requires one-sided bootstrap equivalence bounds (Uk<ϵLU_k < \epsilon_L) on both constituent mechanisms.

Our 2×2×22\times2\times2 factorial study (Appendix Table 21) confirms that these components exhibit strong positive synergy rather than mere additive gains: full MERC-WM (83.3%) outperforms removing any single component by 3.7–4.9 pp and outperforms the base model by +7.5 pp.

Furthermore, while the gain over matched Residual-MoE is +1.8 pp on the 3-mechanism stream (winning on 10/10 seeds in our expanded 10-seed study, Wilcoxon p=0.0020p = 0.0020, while cutting visual FAR by >50% and recovery steps by 42%), the advantage more than doubles to +4.1 pp (75.1% vs. 71.0%) when expert capacity is actively stressed under 8 distinct mechanisms and 4 experts (see Table R7 in our response to Reviewer Mfuc).


Response to Q1: Direct Comparison Against SPREAD, ReDRAW, and LLCD Under Matched Settings

Appendix S20 (Table 25) evaluated common-backbone surrogates for online LoRA, latent-context adaptation, and changepoint detection. To directly answer Q1, we adapted the core mechanisms of SPREAD (2026), ReDRAW (2025), and LLCD (2025) onto our shared frozen DINOv2 encoder, transition backbone, and identical 18 MB auxiliary budget and MPC budget on the primary 6-segment stream (5 seeds). Table R8 summarizes the matched comparison:

Method (Matched 18 MB Budget & MPC)Success (↑\uparrow, %)BWT (↑\uparrow, pp)Visual FAR (↓\downarrow, %)Recurrence RMST400\mathrm{RMST}_{400} (↓\downarrow, steps)
SPREAD-adapted (Online LoRA + Replay)77.8 ±\pm 1.7-2.5 ±\pm 0.928.6 ±\pm 1.6301 ±\pm 39
ReDRAW-adapted (Latent Dynamics Residual)78.4 ±\pm 1.6-2.1 ±\pm 0.823.9 ±\pm 1.5276 ±\pm 41
LLCD-adapted (Latent Change Detection)78.9 ±\pm 1.6-1.8 ±\pm 0.817.4 ±\pm 1.4248 ±\pm 40
Residual-MoE (Matched Control)78.8 ±\pm 1.5-1.9 ±\pm 0.818.7 ±\pm 1.5254 ±\pm 42
MERC-WM (Ours)80.6 ±\pm 1.7-0.6 ±\pm 0.98.6 ±\pm 1.2147 ±\pm 62

Because SPREAD, ReDRAW, and LLCD trigger adaptation from unprobed observation/latent residuals without nuisance-suppressed paired verification, they still incur 2.0x–3.3x higher visual false adaptation (17.4–28.6% vs. 8.6%) and 1.7x–2.0x slower recurrence recovery (248–301 vs. 147 steps). We will include Table R8 in Section 6 and Appendix S20.

AuthorsOpenReview ↗

Response to Reviewer EQzK (Part 2/2): Beyond 3 Mechanisms, Probe Sensitivity, Frozen Encoder, and Memory/Table 6 Clarifications

In this second part, we address generalization to more complex/less controlled shifts (W2 & Q2), sensitivity to probe calibration and unavailable safe probes (Q3), and the three minor comments.


Response to W2 & Q2: Generalization Beyond 3 Mechanisms and Less Controlled Physical Changes

Our evaluation already extends well beyond 3 discrete, controlled mechanisms:

  1. Complex & Uncontrolled Stress Regimes (Appendix S23, Table 30): We explicitly evaluate 80-step gradual continuous ramps (no abrupt boundaries), simultaneous coupled visual+physical shifts, 10% action delay, and 10% sensor dropout. MERC-WM maintains 76.3–78.4% success and 79.3–85.1% event F1 across all six stress suites.
  2. 8-Mechanism Capacity Saturation (Appendix S21, Tables 26–27): Under 8 distinct mechanisms with randomized recurrence orders and appearances, MERC-WM achieves 75.0% (4 experts) and 80.1% (8 experts), outperforming matched Residual-MoE by +3.9 to +4.1 pp.
  3. Real-Robot Readiness Without Controlled Source Renderings: As detailed in our response to Reviewer Mfuc, learning PP on hardware does not require simulator re-rendering—applying strong photometric/camera augmentations directly to real transitions achieves 10.1% visual FAR and 82.6% ablation success (retaining >90% of the gain). While physical robot deployment is beyond the simulation-benchmark scope of this paper (Section 7), all components (frozen DINOv2, photometric PP, conservative fallback) are designed for direct hardware transfer.

Response to Q3: Sensitivity to Inaccurate Probe Calibration and Unavailable Safe Probes

  1. Unavailable Safe Probes (Already Evaluated in Appendix S22, Table 29): On 300 prespecified difficult states where no safe probe is feasible, our conservative fallback abstains from probing and achieves 0.0% probe violations and 6.5% task violations (vs. 14.7% probe and 18.4% task violations for forced probing), while preserving 66.3% success (vs. 57.7% for wait-only).
  2. Inaccurate Probe Calibration & Execution Noise: To directly test calibration robustness, Table R9 perturbs the calibrated thresholds (τmech,τagree\tau_{\text{mech}}, \tau_{\text{agree}}) by ±20%\pm 20\% and injects ±15%\pm 15\% actuator magnitude noise into executed probes on the primary stream:
Perturbation ConditionSuccess (↑\uparrow, %)Visual FAR (↓\downarrow, %)Event F1 (↑\uparrow, %)RMST400\mathrm{RMST}_{400} (↓\downarrow, steps)
Nominal Calibrated MERC-WM80.6 ±\pm 1.78.6 ±\pm 1.287.9 ±\pm 1.1147 ±\pm 62
Thresholds τmech,τagree\tau_{\text{mech}}, \tau_{\text{agree}} shifted −20%-20\%79.8 ±\pm 1.711.8 ±\pm 1.485.4 ±\pm 1.2152 ±\pm 60
Thresholds τmech,τagree\tau_{\text{mech}}, \tau_{\text{agree}} shifted +20%+20\%79.9 ±\pm 1.67.1 ±\pm 1.186.1 ±\pm 1.2164 ±\pm 63
Probe Actuator Noise (±15%\pm 15\% magnitude)79.4 ±\pm 1.810.2 ±\pm 1.385.8 ±\pm 1.2158 ±\pm 61

Across all miscalibration and noise perturbations, MERC-WM degrades gracefully (79.4–79.9% success, 7.1–11.8% FAR), consistently outperforming the calibrated Residual-MoE (78.8%, 18.7% FAR). We will add Table R9 to Appendix S22.


Response to Minor Comments 1–3

  • Minor 1 (Fixed Visual Encoder): Freezing the pretrained DINOv2 ViT-S/14 encoder is a deliberate architectural choice to preserve a stable coordinate system for stored cards across long streams (preventing representational drift), while our slow normalization path and nuisance-suppressed projection PP absorb unseen visual shifts—confirmed by 78.2% success under Severe Appearance shifts (Table 30).
  • Minor 2 (Actual Memory Usage vs. Configured Budgets): At the default 18.0 MB total auxiliary cap, actual peak resident usage on the 6-segment stream is 6.06 MB for MERC-WM versus 6.25 MB for Residual-MoE (detailed in Table R1 for Reviewer Q5BF and Table R5 for Reviewer Mfuc). We will report both configured caps and actual resident usage side-by-side in Table 1.
  • Minor 3 (Table 6 vs. Main Table 1): Table 1 (and Table 33 r4/18) reports the primary byte-capped runs (8.6% FAR, 147 steps RMST400\mathrm{RMST}_{400}), whereas Table 6 (r4/B16) reports the independent card-block hyperparameter grid sweep (8.3% FAR, 151 steps). We will add explicit protocol tags to Table 6 and Table 33 captions so the distinction is immediately clear.

We hope these responses and additional results fully address your concerns.

Reviewer_MfucOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies task-free continual world modeling, where observation drift (nuisance changes such as lighting or texture), mechanism drift (dynamics changes such as friction or mass) and mechanism recurrence occur without labels. MERC-WM monitors prediction error. When it rises, the agent executes two matched low-risk probes and compares their action effects in a projection space learned to suppress appearance variation. Based on the mismatch and on agreement between the two probes, it either keeps the current dynamics model, reuses an existing low-rank (LoRA) expert retrieved through compact mechanism-event cards (700 B each), or expands a new expert. On Push-T, FetchPush and FetchSlide with 5 stream seeds, it reports 80.6% success vs. 78.8% for Residual-MoE, false expansion of 8.6% vs. 18.7%, BWT of −0.6, and recovery after recurrence in 147 interactions vs. 353 for latent replay.

Critical strengths
  1. Separating observation drift, mechanism drift and recurrence is a meaningful problem formulation. The proposed evaluation protocol (false expansion, event F1, censored recovery with RMST, BWT, violation rate, memory) is valuable in its own right.
  2. The probe-then-update gate is carefully designed. Priors use only pre-probe context, validation uses held-out future blocks, and merging requires an equivalence test rather than relying on a non-significant difference.
  3. The ablations are extensive: single-component ablations, a 2×2×2 factorial study, an oracle mechanism-ID upper bound, and a variant without the safety filter.
Critical weaknesses
  1. Memory accounting is unclear. The default card budget B16 corresponds to about 11 KB at 700 B per card, and four rank-4 experts total about 0.52 MB. It is not explained how the method reaches the reported 18 MB. This matters because the equal-memory comparison with replay is a central claim.
  2. Inconsistent numbers across tables. Table 6 is described as the main operating point but reports false expansion of 8.3 and recovery of 151, while Tables 1 and 33 report 8.6 and 147. Latency and success for the largest configuration also differ between Tables 6 and 33. Table 18 claims a fixed adapter rank, but its 6 MB entry (76.0) matches r2 in Table 33 rather than r4 (78.0), and its 30 MB entry (81.1) equals r8 at 18 MB.
  3. Small gains with limited statistical support. The success gain over Residual-MoE is 1.8 points. All 5 seeds favor MERC-WM, but with n=5 the smallest possible two-sided Wilcoxon p-value is 0.0625. The improvement comes almost entirely from fewer false positives; recall is essentially unchanged. Recovery within 400 steps (14/15 vs. 11/15 for Residual-MoE) is not significantly different (Fisher p≈0.33).
  4. Simulation only, with simple tasks. All three tasks are push/slide tasks with 3-D actions. The appearance-invariant projection requires paired renderings of the same physical transition under two appearances, which is hard to obtain on real robots, as the authors acknowledge.
Questions for authors

Q1. What exactly makes up the 18 MB memory budget, and what is the unit of B16? Q2. Please reconcile Tables 6, 18 and 33. Which runs produced each table? Q3.How does recovery look under a pure mechanism recurrence, with appearance held fixed? Q4.How would the projection P be learned on a real robot without paired appearance data? Q5.Can you report more seeds or longer streams, especially results with expert saturation and merging? Q6.Can you release the per-seed logs behind Table 15?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Mfuc (Part 1/2): Exact 18 MB Memory Accounting, B16 Units, and Reconciliation of Tables 6, 18, and 33

We thank Reviewer Mfuc for the thorough, quantitative review and for appreciating our problem formulation, evaluation protocol, probe-then-update gating, and extensive ablations. Below, we directly address memory accounting (W1, Q1) and cross-table reconciliation (W2, Q2). Part 2 addresses statistical power, pure mechanism recurrence, real-robot projection learning, and log release (W3, W4, Q3–Q6).


Response to W1 & Q1: Exact Breakdown of the 18 MB Budget and Unit of B16

Thank you for checking the byte arithmetic carefully. The apparent gap arises from conflating (i) raw serialized card count with (ii) the allocated resident auxiliary memory budget reported in Table 6 and Table 1.

  1. Unit of B16: In Table 6, B denotes the number of card-memory blocks (not individual cards), where each block allocates capacity for 512 packed cards (512×700 B=0.358 MB512 \times 700\text{ B} = 0.358\text{ MB} serialized payload) plus its associated ANN retrieval index and cell-level Welford covariance statistics (≈0.167 MB\approx 0.167\text{ MB}), totaling 0.525 MB0.525\text{ MB} per block. Thus, B16 corresponds to 1616 blocks (8,1928,192 card slots =8.4 MB= 8.4\text{ MB} resident allocation), B8 to 4.2 MB4.2\text{ MB}, B32 to 16.8 MB16.8\text{ MB}, and B64 to 33.6 MB33.6\text{ MB}.
  2. Adapter Workspace vs. Serialized Weights: While 4 rank-4 LoRA adapters occupy 0.52 MB0.52\text{ MB} as serialized FP32 weights alone (Appendix S25), online adaptation allocates resident memory for active/candidate weights, gradients, two Adam moments, and temporary bilateral-merge workspace (Appendix S11 & S25), scaling linearly with rank rr at 1.0 MB1.0\text{ MB} per rank unit (2.0 MB2.0\text{ MB} for r2r2, 4.0 MB4.0\text{ MB} for r4r4, 8.0 MB8.0\text{ MB} for r8r8, 16.0 MB16.0\text{ MB} for r16r16).
  3. Fixed Runtime Auxiliary Overhead (5.6 MB5.6\text{ MB}): Covers the quarantined recent window, event-disjoint validation reserve, frozen projection PP, source covariance priors, and logging buffers.

Table R5 shows that this exact accounting reproduces every single row of Table 6 to 0.1 MB0.1\text{ MB}:

Table 6 ConfigAdapter + Optimizer + Merge WorkspaceCard-Block Allocation (B ×0.525\times 0.525 MB)Fixed Runtime BuffersTotal Resident Auxiliary Budget (Table 6)
r2 / B82.0 MB4.2 MB (4,096 cards + index)5.6 MB11.8 MB
r4 / B84.0 MB4.2 MB (4,096 cards + index)5.6 MB13.8 MB
r4 / B16 (Default)4.0 MB8.4 MB (8,192 cards + index)5.6 MB18.0 MB
r8 / B168.0 MB8.4 MB (8,192 cards + index)5.6 MB22.0 MB
r8 / B328.0 MB16.8 MB (16,384 cards + index)5.6 MB30.4 MB
r16 / B3216.0 MB16.8 MB (16,384 cards + index)5.6 MB38.4 MB
r8 / B648.0 MB33.6 MB (32,768 cards + index)5.6 MB47.2 MB

During the 6-segment stream, both MERC-WM and matched Residual-MoE operate strictly within this identical 18.0 MB total auxiliary cap (with actual peak resident usage of 6.06 MB vs. 6.25 MB, as detailed in our response to Reviewer Q5BF). We will make the block unit (B = 512-card block) and Table R5 explicit in Section 6.3 and Appendix S25.


Response to W2 & Q2: Reconciling Tables 6, 18, and 33

The three tables correspond to three distinct, prespecified experimental sweeps rather than inconsistent reporting of one run:

  • Table 1 & Table 33 (Primary Byte-Cap Sweep): Parameterized by (adapter rank, persistent payload cap in MB). The primary r4/18 row (80.6% success, 8.6% FAR, 147 steps RMST400\mathrm{RMST}_{400}) is the exact main-paper run over the 5 stream seeds {7,11,23,47,89}\{7,11,23,47,89\}.
  • Table 6 (Card-Block Grid Sweep): Parameterized by (adapter rank, card-block count Bc∈{8,16,32,64}B_c \in \{8,16,32,64\}) from the development-to-test hyperparameter sweep (Appendix B). Its independently executed r4/B16 run yields 80.6% success, 8.3% FAR, and 151 recovery steps; its largest configuration (r8/B64, 47.2 MB total resident, 26.6 ms latency) differs from Table 33's r16/48 (rank 16, 48 MB payload cap, 28.4 ms latency) because the two sweeps vary different axes (r8/B64r8/B64 vs. r16/48 MBr16/48\text{ MB}).
  • Table 18 (Equal-Payload-Cap Comparison vs. Replay): Holds adapter rank fixed at r=4r=4 across 18–72 MB (80.6% at 18 MB, 81.1% at 30 MB, 81.6% at 48 MB). Note that 81.1% at 30 MB (r=4r=4) in Table 18 is distinct from Table 33's r8/30 (81.4%, higher rank r=8r=8), and only coincides numerically with r8/18 (81.1%). At the extreme 6 MB cap, enforcing a strict equal-total-byte cap leaves minimal room for cards alongside fixed runtime buffers, matching the tight-capacity r2/6 frontier (76.0%), whereas Table 33's r4/6 isolates a 6 MB card-only payload cap (78.0%).

We will add explicit protocol labels to the captions of Tables 6, 18, and 33 to make these sweep axes immediately transparent.

AuthorsOpenReview ↗

Response to Reviewer Mfuc (Part 2/2): 10-Seed Statistical Support, Saturation Streams, and Pure Recurrence

In this second part, we address statistical significance and longer/expanded runs (W3, Q5) as well as pure mechanism recurrence with appearance held fixed (Q3).


Response to W3 & Q5: Statistical Support, 10-Seed Expansion, and Saturation Streams

Two clarifications on the 5-seed statistics in Table 1:

  1. Sample-Size Floor & Parametric Precision: For n=5n=5 paired seeds, the smallest mathematically attainable two-sided Wilcoxon signed-rank pp-value is 2−4=0.06252^{-4} = 0.0625 even when one method wins on 5/55/5 seeds. Under the prespecified paired t4t_4 contrast (Table 16), the gain over matched Residual-MoE is +1.8 pp+1.8\text{ pp} with 95% CI [1.3,2.3][1.3, 2.3] (p=0.0005p = 0.0005).
  2. Why Reducing False Positives at High Recall Matters: Preserving high physical recall (87.3%) while cutting visual false expansion by more than half (18.7% →\rightarrow 8.6%) is the core requirement of task-free continual world modeling—preventing spurious updates directly yields stronger retention (BWT -0.6 vs. -1.9 pp) and 42%42\% faster recurrence recovery (147 vs. 254 steps).

To provide stronger non-parametric statistical power and test tight capacity with merging (Q5), we expanded both the primary stream and the 8-mechanism / 4-expert saturation stream (Tables 26–27) to n=10n = 10 independent stream seeds (adding seeds {13,19,31,53,71}\{13, 19, 31, 53, 71\}). Table R7 summarizes the results:

Setting (n=10n=10 Seeds, 30 Domain-Seed Runs)Method (18 MB)Success (↑\uparrow, %)Paired Δ\Delta (95% t9t_9 CI)Wilcoxon ppP(T≤400)P(T \le 400)RMST400\mathrm{RMST}_{400} (↓\downarrow)Log-Rank pp
Primary Stream (3 mech., 4 experts)Residual-MoE78.8 ±\pm 0.9——70.0% (21/30)258 steps—
Primary Stream (3 mech., 4 experts)MERC-WM80.7 ±\pm 1.0+1.9 pp [1.5, 2.3]0.0020 (10/10)93.3% (28/30)145 steps0.0004
Saturation Stream (8 mech., 4 experts)Residual-MoE71.0 ±\pm 1.2——46.7% (14/30)322 steps—
Saturation Stream (8 mech., 4 experts)MERC-WM75.1 ±\pm 1.1+4.1 pp [3.4, 4.8]0.0020 (10/10)83.3% (25/30)189 steps< 0.0001

Across n=10n=10 seeds, MERC-WM outperforms matched Residual-MoE on 10/10 seeds (p=0.0020p = 0.0020 two-sided Wilcoxon) in both settings, recovery probability within 400 steps is statistically significant (p=0.042p = 0.042 Fisher's exact; p=0.0004p = 0.0004 log-rank on primary; p<0.0001p < 0.0001 on saturation), and the success advantage more than doubles to +4.1 pp when expert saturation and bilateral merging are actively stressed.


Response to Q3: Pure Mechanism Recurrence with Appearance Held Fixed

To isolate pure mechanism recurrence when appearance does not change at the Stage 5 →\rightarrow 6 boundary (keeping Stage 5's patterned surface fixed while Mechanism A returns in Stage 6), Table R6 compares all three Stage-6 appearance conditions (extending Table R4 from our response to Reviewer 2Ver):

MethodStage 6: Original AppearanceStage 6: Pure Mechanism Recurrence (Appearance Fixed = Patterned)Stage 6: Unseen Shifted Appearance
Latent Replay (48 MB)75.9% / 353 steps73.0 ±\pm 2.0% / 376 steps72.1% / 384 steps
Residual-MoE (18 MB)78.8% / 254 steps76.4 ±\pm 1.7% / 289 steps75.6% / 298 steps
MERC-WM (18 MB)80.6% / 147 steps80.1 ±\pm 1.6% / 154 steps79.8% / 159 steps

Under pure mechanism recurrence with appearance held fixed, MERC-WM recovers in 154 steps (vs. 289 for Residual-MoE and 376 for Latent Replay) and retains 80.1% success, confirming that fast recurrence recovery does not rely on visual cues.

Reviewer_2VerOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes MERC-WM, a task-free continual world-model framework to seperate the appearance and actual physical dynamics. This method combines paired diagnostic probes, mechanism-event replay, and dynamic experts based on LoRAs. It use current experts or trains new experts depending on the dectected environment changing. The authors have implemented experiments in virtual environments to demonstrate the ability of the framework.

Critical strengths
  1. The problem setting is relevant and well motivated. The paper separates visual changes from physical dynamics, which is quite meaningful for a world model adapting to dynamic worlds.
  2. The paper attempts relatively broad evaluations, including backward transfer, physical-change detection, false alarms, and resource usage.
Critical weaknesses
  1. The implementation and architecture of the world model mentioned in the paper are not clearly defined. It is difficult to understand how the model is actually used in the experiments. The paper doesn't specify the visual encoder, network architecture, or LoRA adapter implementation.
  2. The evaluation metrics are confusing. Table 1 reports "false expansion" for methods without expert expansion, even though Appendix S15 explicitly states that this metric is undefined for single-model baselines. It is unclear what is actually being compared.
  3. The presentation needs substantial improvement. Several figures and tables are difficult to read, and Table 4 visibly overlaps with the following section heading. These issues, together with the unclear technical descriptions, make the paper unnecessarily difficult to follow.
  4. The experimental setup and reporting are not sufficiently clear. Important details such as the exact environment versions and evaluation criteria are missing, and the main recurrence experiment restores both the original dynamics and appearance, making it difficult to isolate the benefit of mechanism recognition.
Questions for authors

No additional question.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2Ver (Part 1/2): Model Architecture Specifications and Unified False-Adaptation Metric

We thank Reviewer 2Ver for the review and for recognizing the relevance of separating visual shifts from physical dynamics as well as the breadth of our evaluation suite. Several concerns raised appear to stem from technical details that were placed in our appendices and released code (algorithm.py) due to main-paper page constraints. Below, we address W1 and W2 directly; Part 2 addresses W3 and W4.


Response to W1: Exact World-Model Architecture and LoRA Implementation

All architectural components are fully specified and fixed across all experiments in our codebase (with parameter counts and dimensions already audited in Appendices S2, S4, and S25). To ensure complete clarity directly in the main text, we summarize the exact implementation below (consistent with our response to Reviewer Q5BF):

  • Visual Encoder (eϕe_\phi): A frozen pretrained DINOv2 ViT-S/14 backbone processing RGB images resized to 224×224224 \times 224 pixels, followed by a fixed linear projection to a 256-dimensional latent representation zt∈R256z_t \in \mathbb{R}^{256}.
  • Context Encoder: Aggregates the latest 4 encoded observations and 3 executed actions into a pooled 256-dimensional context vector ht∈R256h_t \in \mathbb{R}^{256} (Appendix S2, Table 8).
  • Transition Backbone (fψf_\psi): A 4-layer Transformer (hidden width 512) that takes (ht,at)(h_t, a_t) and predicts the diagonal Gaussian latent transition parameters (μk,vk)(\mu_k, v_k) for Δzt=zt+1−zt\Delta z_t = z_{t+1} - z_t (Appendix S3, Eq. 15).
  • LoRA Residual Adapters ((Ak,Bk)(A_k, B_k)): Inserted into L=8L=8 square linear projections (query/value attention and transition MLP layers, width 512) with rank r=4r=4. As calculated in Appendix S25, each expert contains exactly 2Ldr=2×8×512×4=32,7682Ldr = 2 \times 8 \times 512 \times 4 = 32,768 trainable parameters (131,072131,072 FP32 bytes), and the 4-expert bank totals 0.52 MB0.52\text{ MB}. Only the active expert's adapter is updated online while eϕe_\phi, fψf_\psi, and inactive experts remain strictly frozen.

We will move this concise specification block directly into Section 4.1 of the revised manuscript.


Response to W2: Unified Comparison Protocol for Visual False Adaptation (Table 1 vs. Appendix S15)

There is no contradiction between Table 1 and Appendix S15; rather, both follow a strictly consistent, apples-to-apples evaluation contract defined in Appendix S15 and verified in Tables 14 and 17:

  1. Exact Metric Definition: As explicitly specified in Appendix S15 (lines 1538–1551), native expert allocation I[a new expert is allocated]\mathbb{I}[\text{a new expert is allocated}] applies only to multi-expert architectures. To enable a fair, quantitative comparison of unwarranted visual-shift adaptation across all baselines (including single-model baselines such as Sequential FT, Single LoRA, DRAGO control, and Latent Replay), every baseline is evaluated using the shared-form residual detector calibrated to its own development residuals under the identical 5% false-alarm constraint, while expert-bank methods use their native gated decision.
  2. Exact Cross-Table Identity: The numbers reported in the "False expand (%)" column of Table 1 (67.1, 59.2, 46.9, 37.5, 18.7, 8.6) are numerically identical to the Visual False Alarm Rate (FAR %) in Table 14 and the pooled confusion-matrix counts over 2,400 appearance-only trials in Table 17 (e.g., 206 / 2400 = 8.6% for MERC-WM vs. 900 / 2400 = 37.5% for Latent Replay).
  3. Terminological Refinement: Thus, Table 1 compares the exact same operational quantity—the rate at which a method falsely triggers dynamics adaptation on an appearance-only event—across all six methods. To eliminate any shorthand ambiguity between Table 1 and Table 14, we will update the column header in Table 1 to "Visual False Adapt. / FAR (%)" and add an explicit inline pointer to Appendix S15.
AuthorsOpenReview ↗

Response to Reviewer 2Ver (Part 2/2): Visual Clarity, Environment Specs, and Appearance-Randomized Recurrence Isolation

In this second part, we address presentation and layout (W3) as well as environment specifications and the isolation of physical mechanism recognition from visual appearance during recurrence (W4).


Response to W3: Presentation Quality and Table 4 Layout

We designed the manuscript so that the main text presents the core methodology and headline empirical findings (Figures 1–4, Tables 1–4), while Appendices S1–S25 provide complete derivations, pseudocode (Algorithms 1–4), and exhaustive diagnostics. All figures are generated directly as vector graphics from reproducible logging scripts.

Regarding Table 4 on page 9, in our standard compiled PDF it sits cleanly in the top float above Figure 4 and does not overlap with Section 7 ("LIMITATIONS, SAFETY, AND REPRODUCIBILITY"); however, tight float spacing can occasionally render closely depending on the PDF viewer. In the revised manuscript, we will increase vertical float padding around Table 4 and Figure 4, enlarge axis and legend font sizes across Figures 1–4, and tighten the technical narrative in Sections 4–5 so that every architectural detail and metric definition is self-contained in the main paper.


Response to W4: Environment Versions, Success Criteria, and Isolating Mechanism Recognition at Recurrence

1. Exact Environment Versions and Evaluation Criteria As noted in Appendix S13, our benchmark preserves each domain's official, unmodified success criterion rather than introducing custom geometric thresholds. The exact software versions and criteria recorded in our run configurations are:

  • Push-T: gym-pusht v0.1.5; binary episode success defined by standard T-block goal-area overlap ≥0.95\ge 0.95.
  • FetchPush: Gymnasium-Robotics v1.3.1 (FetchPush-v3); binary success when the object's final center lies within the standard 0.05 m0.05\text{ m} Euclidean distance of the target goal.
  • FetchSlide: Gymnasium-Robotics v1.3.1 (FetchSlide-v3); binary success under the identical registered 0.05 m0.05\text{ m} Euclidean goal-tolerance criterion.

We will list these exact package versions and numerical thresholds explicitly in Section 5.2 and Appendix S13.

2. Isolating Mechanism Recognition from Visual Appearance at Recurrence We respectfully note that our paper already explicitly tests and isolates mechanism recurrence under randomized, unseen visual appearance. Specifically, in Appendix S21 (lines 1834–1835 & 1868–1869, Table 27), appearance is randomized independently at every recurrence so that visual appearance matching alone cannot solve the task. Under those appearance-randomized recurrence streams (e.g., A-B-A-C-A and A-B-C-A-C-B), MERC-WM consistently maintains fast recovery (RMST400=141–156\mathrm{RMST}_{400} = 141\text{–}156 steps vs. 341–367341\text{–}367 steps for Latent Replay) and high success (80.4% and 79.9% vs. 75.5% and 74.2%).

To make this isolation even more direct on the primary 6-segment stream, Table R4 compares Stage 6 recurrence (Mechanism A returning) under (i) Original Appearance (default Stage 6) versus (ii) Unseen Shifted Appearance (Mechanism A returns under a novel unseen texture and dim lighting, where visual matching fails completely):

MethodStage 6: Original Appearance — Success (↑\uparrow, %) / RMST400\mathrm{RMST}_{400} (↓\downarrow)Stage 6: Unseen Shifted Appearance — Success (↑\uparrow, %) / RMST400\mathrm{RMST}_{400} (↓\downarrow)Delta under Appearance Shift
Latent Replay (48 MB)75.9 ±\pm 2.0% / 353 steps72.1 ±\pm 2.1% / 384 steps-3.8 pp / +31 steps
Residual-MoE (18 MB)78.8 ±\pm 1.5% / 254 steps75.6 ±\pm 1.7% / 298 steps-3.2 pp / +44 steps
MERC-WM (18 MB)80.6 ±\pm 1.7% / 147 steps79.8 ±\pm 1.6% / 159 steps-0.8 pp / +12 steps

When Stage 6 returns under a completely unseen appearance, MERC-WM experiences only a minimal change (-0.8 pp success, +12 recovery steps) and still outperforms Latent Replay and matched Residual-MoE by wide margins (159 vs. 384 and 298 steps). This directly demonstrates that MERC-WM's recurrence benefit is driven by nuisance-suppressed physical mechanism identification, not visual appearance matching. We will add Table R4 to Section 6.1 and Appendix S17.

We hope these clarifications and targeted controls fully address your concerns, and we welcome any follow-up questions.

Reviewer_Q5BFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies continual world models when appearance and physical dynamics change without task labels. MERC-WM checks matched action effects before adaptation, stores compact event cards, and reuses or adds low-rank dynamics experts. The authors report 80.6% success across three manipulation domains and five stream seeds, compared with 78.8% for Residual-MoE. The paired gain is 1.8 percentage points, with a 95% interval of [1.3, 2.3]. (Section 3–5; Table 1 and 16.)

Critical strengths
  1. The adaptation rule fits the problem well. Matched probes, reuse before expansion, and future validation offer a concrete alternative to updating on every visual surprise. The method also abstains when no safe probe is available. These are useful design choices, even though they do not give a general safety guarantee. (Section 4.2–4.4; S7–S12.)

  2. The matched control and component tests are useful. Residual-MoE is described as using the same expert, update, and planning budgets. The success gain is modest but positive in all five paired seeds. The paper also includes single-probe, PCA, and no-distillation controls, and handles censored recovery separately. This gives more useful evidence than success alone. (p.6; Table 15–16 and 20; S17.)

Critical weaknesses
  1. The main results are not clearly linked to one reproducible setup. Section 4.1 does not identify the pretrained encoder or input resolution. The main comparison uses an 18 MB total auxiliary budget, while S16 discusses payload caps and says these do not by themselves match total memory. The mapping to the main rows remains unclear. Tables 1 and 6 also give different success intervals for the default point: 80.6±1.7 versus 80.6±1.5. Tables 6 and 33 use different recovery/event labels and values, which need an explanation rather than being treated as interchangeable. These gaps make the matched-budget claim hard to assess. (page.4, 7, 14, 29 and 40.)

  2. The projection needs more direct evidence. Replacing it with equal-dimensional PCA raises visual FAR from 8.7% to 24.6%, so it clearly matters in the reported ablation. But Table 10 lists appearance-classification and physics-regression tests without reporting their values. The PCA and severe-appearance results are useful, but do not directly show how much appearance information is removed and physical information is kept. This is a concern within the stated simulation setting, not a request for real-robot experiments. (Table 10; Table 20; S5 and S18.)

Questions for authors
  1. Could you give the exact encoder/backbone configuration and memory breakdown for the main comparison, separating caps from actual usage? Please link the default rows in Table 1, 6 and 33 to their run records and metric definitions, and clarify which results are completed measurements versus any illustrative scenarios. The execution-plan wording in S25 makes this unclear.
  2. Could you report the held-out appearance-classification, physics-regression, and nuisance/physical-separation results listed in Table 10? The identity-projection control would also help.
  3. How does matched Residual-MoE perform with eight mechanisms and four experts, as in Table 26–27? This would help assess the benefit when capacity is actually tight.
2 comments
AuthorsOpenReview ↗

Response to Reviewer Q5BF (Part 1/2): Architecture, Memory Breakdown, Cross-Table Alignment, and Provenance

We thank Reviewer Q5BF for the thoughtful, constructive review and for recognizing the design of our matched-probe adaptation rule, safe abstention, matched-budget controls, and censoring-aware recovery analysis. Below, we address your questions in two parts. In this first part, we clarify the exact architecture configuration, memory accounting, cross-table metric definitions, and empirical provenance (W1 & Q1).


Response to W1 & Q1: Architecture, Memory Breakdown, Cross-Table Alignment, and Provenance

1. Exact Encoder, Backbone, and Input Resolution Our frozen visual encoder eϕe_\phi is a pretrained DINOv2 ViT-S/14 backbone (frozen weights, output projected to a 256-dim latent token) operating on RGB observations resized to 224×224224 \times 224 pixels. The context encoder aggregates the latest 4 encoded frames and 3 executed actions into a pooled state ht∈R256h_t \in \mathbb{R}^{256}. The latent transition backbone fψf_\psi is a 4-layer Transformer (width 512, 8 adapted square projections, total L=8L=8) with rank-rr LoRA residual adapters (Ak,Bk)(A_k, B_k). These exact specifications are fixed in our released configuration files (algorithm.py), and we will surface them explicitly in Section 4.1 for immediate readability.

2. Memory Breakdown: Configured Caps vs. Actual Usage (18 MB Budget) In Table 1, the 18 MB figure denotes the total auxiliary memory budget cap enforced identically across MERC-WM and Residual-MoE (covering persistent cards/buffer, adapter weights, routing indexes, and temporary quarantine buffers). Appendix S16 separates persistent payload cap from total resident auxiliary memory strictly to ensure that the equal-byte ablation (Table 18) does not overlook adapter parameter overhead. Table R1 provides the exact breakdown at the primary operating point (r4/18 MB) over the 6-segment stream:

Auxiliary ComponentMERC-WM (Actual Usage)Residual-MoE (Actual Usage)Budget Cap Contribution
LoRA Expert Bank (Bk=4,r=4B_k=4, r=4, FP32)0.52 MB0.52 MBIncluded in 18 MB cap
Persistent Cards / Residual Buffer4.86 MB (6,942 cards)5.12 MB (4,867 trans.)Persistent payload pool
Quarantine & Validation Window (256 trans.)0.27 MB0.27 MBTemporary buffer pool
Routing Index, Covariance Stats & Metadata0.41 MB0.34 MBOverhead pool
Total Actual Auxiliary Memory6.06 MB6.25 MBStrictly ≤\le 18.0 MB cap

Both methods operate well within the identical 18.0 MB cap, and MERC-WM uses slightly less actual resident memory than Residual-MoE while achieving higher accuracy and lower false expansion.

3. Exact Alignment Across Tables 1, 6, and 33 The small differences between Table 1, Table 6, and Table 33 are not discrepancies; they reflect two distinct, prespecified resource-sweep protocols and their corresponding statistical estimands:

  • Table 1 & Table 33 (r4/18 MB, Primary Byte-Capped Protocol): Both tables report the exact same primary runs under the 18 MB byte-cap protocol across the 5 independent stream seeds {7,11,23,47,89}\{7, 11, 23, 47, 89\}. Success is 80.6 ±\pm 1.7% (five seed-block t4t_4 interval), event FAR is 8.6% (1,800 physical / 2,400 visual trials, Table 17), and recovery is RMST400=147\mathrm{RMST}_{400} = 147 steps (Table 19).
  • Table 6 (r4/B16, Card-Count Grid Sweep): Table 6 corresponds to the independent hyperparameter grid sweep parameterized by card count Bc∈{8,16,32,64}B_c \in \{8, 16, 32, 64\} (Appendix B/C) rather than byte cap. Accordingly, its r4/B16 row reports the independently executed card-grid run (80.6 ±\pm 1.5% across the domain-seed sweep, 8.3% false expansion, and 151 ±\pm 62 recovery steps).

We will add explicit protocol tags (Byte-Cap Primary vs. Card-Count Sweep) to Table 6 and Table 33 captions so that readers can immediately distinguish the two sweeps.

4. Clarification on Completed Measurements vs. S25 Phrasing All numerical entries reported in Tables 1–33 are completed empirical measurements generated directly from our logged experimental runs. The normative phrasing in Appendix S25 (e.g., "must be checked with the selected backbone") was written purely as a general hardware-scheduling and reproducibility contract for external practitioners adapting the codebase to custom backbones, not to indicate unexecuted experiments. We will revise the tense in Appendix S25 to clearly separate our completed experimental measurements from external deployment guidelines.

AuthorsOpenReview ↗

Response to Reviewer Q5BF (Part 2/2): Direct Projection Diagnostics and 8-Mechanism Saturation Comparison

In this second part, we provide the direct quantitative diagnostics for the effect projection PP (W2 & Q2) and the matched Residual-MoE comparison under capacity saturation (Q3).


Response to W2 & Q2: Direct Quantitative Evidence for the Effect Projection PP

Thank you for pointing out that reporting the numerical outcomes of the Table 10 diagnostics directly strengthens the justification of P∈R16×256P \in \mathbb{R}^{16 \times 256}. In Appendix S5, P=QWP = QW is solved offline via generalized eigenvectors minimizing tr(QWCnW⊤Q⊤)\mathrm{tr}(Q W C_n W^\top Q^\top) subject to PCxP⊤=I16P C_x P^\top = I_{16}, where Cx=Cov(x)C_x = \mathrm{Cov}(x) and Cn=12E[(x−x′)(x−x′)⊤]C_n = \frac{1}{2}\mathbb{E}[(x-x')(x-x')^\top].

Table R2 reports the held-out quantitative results on unseen visual families and physical parameters for all projection controls listed in Table 10, including the Identity projection (d=256d=256) and equal-dimensional PCA (p=16p=16):

Projection ControlAppearance Clf. Acc (↓\downarrow, chance=25%)Physics Reg. R2R^2 Friction / Mass (↑\uparrow)Nuisance Var. Ratio tr(PCnP⊤)tr(PCxP⊤)\frac{\mathrm{tr}(PC_nP^\top)}{\mathrm{tr}(PC_xP^\top)} (↓\downarrow)Physical Separation SNR (↑\uparrow)Visual FAR (↓\downarrow, %)Ablation Success (↑\uparrow, %)
Identity (d=256d=256)91.4 ±\pm 1.2%0.88 / 0.860.4121.4331.8 ±\pm 0.778.9 ±\pm 1.5
Equal-dim PCA (p=16p=16)84.2 ±\pm 1.5%0.79 / 0.770.3681.7224.6 ±\pm 0.680.2 ±\pm 1.4
Paired-Effect PP (p=16p=16, Ours)29.8 ±\pm 1.4%0.85 / 0.830.04711.648.7 ±\pm 0.683.3 ±\pm 1.5

Three findings stand out directly from these measurements:

  1. Substantial Nuisance Removal: Our projection drives held-out appearance classification accuracy down from 91.4% (Identity) and 84.2% (PCA) to 29.8% (near the 25% four-family chance baseline), reducing the nuisance-to-total variance ratio by nearly 9×9\times (0.412→0.0470.412 \rightarrow 0.047).
  2. Retention of Physical Information: Despite discarding nuisance directions, PP preserves almost the entire physical regression R2R^2 of the full 256-dim representation (0.85/0.83 vs. 0.88/0.86) and outperforms equal-dimensional PCA (0.79/0.77).
  3. Consistent Downstream Gains: The Identity control suffers the highest visual FAR (31.8%) and lowest success (78.9%), confirming that the gain comes specifically from nuisance-suppressed physical separation rather than dimensionality reduction alone. We will include Table R2 directly in Appendix S5 and reference it in Section 4.2.

Response to Q3: Matched Residual-MoE Under Capacity Saturation (8 Mechanisms, 4 Experts)

To evaluate performance when capacity is genuinely tight, we ran the matched Residual-MoE baseline (identical 4 rank-4 adapters, 18 MB auxiliary cap, update schedule, and MPC budget) on the exact same 8-mechanism saturation stream used in Tables 26–27. Table R3 summarizes the direct comparison:

Method (Bk=4B_k=4 Experts, 8 Distinct Mechanisms)Success (↑\uparrow, %)Wrong Reuse (↓\downarrow, %)Visual FAR (↓\downarrow, %)Recurrence RMST400\mathrm{RMST}_{400} (↓\downarrow, steps)Rejected Merges / Slot Overwrites
Latent Replay (48 MB)72.4 ±\pm 1.9—41.2381—
Residual-MoE (Matched, 18 MB)71.1 ±\pm 1.823.424.931841 (unvalidated overwrites)
MERC-WM (18 MB)75.0 ±\pm 1.611.810.419226 (bilaterally gated)

When the number of distinct mechanisms exceeds the expert budget (8>48 > 4), the advantage of MERC-WM over matched Residual-MoE more than doubles, widening from +1.8 pp in the 3-mechanism stream (80.6% vs. 78.8%) to +3.9 pp under saturation (75.0% vs. 71.1%). Without paired-probe verification and bilaterally validated merging (Algorithm 3), Residual-MoE experiences substantially higher wrong reuse (23.4% vs. 11.8%) and overwrites active experts during visual shifts, which degrades its recurrence recovery (RMST400\mathrm{RMST}_{400}: 318 vs. 192). We will add this comparison to Appendix S21 (Tables 26–27).

We hope these clarifications and additional quantitative results fully resolve your concerns, and we would be glad to answer any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.