acceptodds
Under review as a conference paper at ICLR 2027

Evidence-Gated State Anchoring for Persistent Video World Models

Abstract

Pretrained video world models change regions that nothing has touched, and they fail to restore a view after the camera looks away and comes back. Both behaviors come from the same gap. Self-attention can read any token in the window, while the training loss never asks whether a location was licensed to change. Recent memory-augmented world models decide which history stays readable. They leave the local write unconstrained. Evidence-Gated State Anchoring (EGSA) is a frozen-backbone retrofit that turns correspondence and propagated-change evidence into write authorization. Where that evidence is absent, a residual hold mixes the clean-state prediction toward a warped copy of the model's own previous estimate, inside every denoising step. Training touches fewer than 0.4 percent of parameters. A warmup multiplier and zeroed LoRA output projections make the retrofit identical to the base model at step zero, and the evidence detector sees only the noised model input. A no-bias ablation matches the full model to within 0.5 percent on both backbones, which places the gain on the anchoring pathway. On a 0.5B latent flow-matching model and the 0.46B pixel-space DFoT RealEstate10K model, EGSA improves real-capture palindromic revisits against a matched LoRA twin. On DFoT, look-away error drops by 67 percent and near-origin return error by 12 to 14 percent (86 to 93 percent of held-out scenes, scene-level p < 10^-4), across training seeds, rollout noise seeds, and a monotone inference dose. On the flow-matching model, all three training seeds lower return error on 128 stronger-departure held-out scenes, by up to 12 percent. Revisit targets are captured frames, fidelity is reported together with a view-change tracking ratio, and all revisit scores are computed in decoded pixels.

Primary area:
generative models
Keywords:
video world models, state persistence, revisit consistency, diffusion models, parameter-efficient adaptation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tfqnOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Evidence-Gated State Anchoring (EGSA) to improve state persistence in video world models. The method uses correspondence and motion evidence to control whether predicted visual states should be updated or anchored to previous estimates. It can be integrated into pretrained diffusion and flow-matching models with a small number of trainable parameters. Experiments on MiniWorld and DFoT show improved reconstruction consistency when the camera moves away from a scene and returns.

Critical strengths

The paper addresses an interesting and practical problem in video world models, particularly the preservation of scene information during recursive generation and camera revisits. The proposed method is lightweight and can be applied to different pretrained backbones without full model retraining.

Critical weaknesses
  1. The effectiveness of evidence gating is not sufficiently demonstrated in realistic dynamic scenes. Most of the main experiments are conducted on static-scene camera revisits, where preserving previous observations is naturally beneficial. The authors also acknowledge that evidence-free anchoring can achieve similar improvements in these settings. The advantage of evidence gating over a simple hold mechanism is mainly demonstrated using a synthetic experiment with a known change mask. It remains unclear whether the proposed evidence mechanism can reliably distinguish legitimate scene changes from unwanted drift in realistic videos with moving objects, occlusions, and complex interactions.

  2. The main experiments rely on constructed palindromic camera trajectories and rotation-dominant correspondence. In particular, the DFoT evaluation generates only four future frames. More diverse and challenging evaluation scenarios are needed to demonstrate the generality of the proposed approach.

Questions for authors
  1. How does EGSA compare with existing memory-based or geometry-guided world models, such as WorldMem, GEN3C, or Closing the Loop?

  2. The attention-logit bias contributes almost no additional improvement according to the ablation results. Is this component necessary?

6 comments

Thank you for the thoughtful review. Below we report our new dynamic manipulation evaluations, rolling-window generation beyond four frames, comparison with memory-based and geometry-guided world models, and long-horizon attention-bias ablations.

1. Evidence gating in realistic dynamic scenes (W1)

We evaluate online latent-change evidence (with no ground-truth mask at inference) on real action-conditioned DROID manipulation sequences containing moving arms, object interactions, and occlusions. Within each training seed, all arms share LoRA weights, initial frame, actions, noise, and sampler:

  • 64 short-horizon episodes (3 seeds x 3 noises, Table R1): At equal mean read weight (0.127295), EGSA reaches changing-region MSE 0.079659 vs 0.081347 (constant) and 0.081764 (shuffled), and preserves a 0.842 generated/captured RGB-change ratio vs 0.703 and 0.706. Motion-magnitude MAE drops by 46.71% (95% CI [42.07%, 51.15%]) and 46.26% (95% CI [41.66%, 50.72%]), improving all 64/64 episode means (both Holm p=0.000030).
  • 64 long-horizon 253-frame trajectories (3 seeds, frame 0 observed, frames 1-252 generated in 16-frame rolling windows, Table R6): At identical read budget, EGSA lowers full-sequence LPIPS by 14.13% vs both constant hold (0.083246 vs 0.096940, CI [-15.63%, -12.68%], Holm p=0.00018) and shuffled evidence (0.096940, Holm p=0.00018), and lowers changing-region 2D RAFT flow error by 5.57% (0.801974 vs 0.849290 px, Holm p=0.00018) and 6.00% (0.853144 px, Holm p=0.00018). Against no anchoring (All MSE 0.028406), EGSA lowers All MSE by 9.74% (0.025639, Holm p=0.00005), static flow error by 13.50% (Holm p=0.00018), and changing flow error by 2.73% (Holm p=0.00018). Against no-update initial memory (seed-0 ablation on all 64 trajectories), updating accepted states lowers LPIPS by 21.78% (0.082573 vs 0.105570, Holm p=0.00018) and changing flow error by 5.09% (Holm p=0.00018). Weaker uniform holds (0.08, 0.056) obtain slightly lower pixel MSE (0.024681, 0.025209) with higher static flow error (EGSA -5.11% and -6.77%, both Holm p=0.00018).

2. Generation beyond four frames and camera translation (W2)

Alongside the non-palindromic 253-frame DROID rollouts, DFoT observes only frames 0-3 and generates 12 future frames (4-15) across three 8-frame windows (0-7, 4-11, 8-15), evicting frames 0-3 before return (12-15). Across 56 scenes, 3 seeds, and 2 noises (Table R2), last-valid memory lowers return MSE by 36.92% vs LoRA (0.010252 vs 0.016253, CI [-40.85%, -32.15%], Holm p=0.000020) and 42.01% vs previous-frame EGSA (0.017680, CI [-44.42%, -39.39%], Holm p=0.000020; 56/56 scenes improved; 12-frame MSE -11.88%), while beating matched-tracking EMA (-34.15%), warp-and-blend (-48.92%), and in-sampler constant hold (-47.14%) in Table R7 (all Holm p=0.00017). To handle camera translation where pure-rotation warping is invalid, restricting reads to geometry-supported poses across all 56 scenes (Table R5) cuts all-generated LPIPS from 0.145196 to 0.100472 (-30.80%, 56/56 scenes; -30.86% on 50 held-out scenes).

3. Relation to WorldMem, GEN3C, and Closing the Loop (Q1)

MethodHistory representationGeometric / conditioning inputsPoint of intervention
WorldMemExternal frame bank indexed by pose/state and timePose or state plus temporal indexRetrieves memory frames into attention conditioning
GEN3CExplicit 3D feature/point cacheCamera pose and depth/cache geometryRenders 3D cache views as conditioning inputs
Closing the LoopHistorical latent chunks restored to clean contextCamera pose and metric depthContext retrieval plus depth-reprojection attention bias
EGSAAligned previous state or retrieved historical stateAvailable correspondence plus online change evidenceGates clean-state updates inside each denoising step

WorldMem, GEN3C, and Closing the Loop determine how past observations are stored, projected, and fed into network conditioning. EGSA operates inside the denoising step to control how strongly an aligned state prior constrains the clean-state estimate, releasing that constraint where online change evidence appears. Retrieved memory supplies EGSA's prior in Tables R2 and R5, while Tables R1 and R6 isolate evidence-gated release under object motion.

4. Necessity of the attention-logit bias (Q2)

It is not necessary. Removing attention-logit bias on our 56-scene, 3-seed, 2-noise DFoT memory evaluation (Table R2) shifts return MSE by +0.002441% (0.010252 to 0.010253) and all-generated MSE by -0.007850% (0.011803 to 0.011802), and all new DROID evaluations run without it. We therefore treat clean-state anchoring as the core mechanism and keep the bias optional.

Thank you for the additional experiments. The new results address some of my concerns, but two points remain unclear.

1. Necessity of evidence gating. In Table R3, a simple constant hold (loop_w0.5) achieves lower return MSE than EGSA at dose 2 (0.141707 vs. 0.186906), with a comparable view-change ratio and a higher direction proxy. While the paper argues that evidence gating helps preserve legitimate changes, it remains unclear why EGSA performs worse than a simple constant hold in this setting.

2. Contribution of memory versus evidence gating. In Table R2, previous-frame EGSA performs worse than the LoRA baseline, whereas adding last-valid memory substantially improves return MSE. Notably, evidence-free memory achieves even lower return MSE than EGSA with memory (0.007505 vs. 0.010252), although its overall MSE is higher. These results raise questions about the relative contributions of memory retrieval and evidence gating. In particular, it is unclear why evidence-free memory achieves better return accuracy and to what extent the reported gains can be attributed to evidence gating itself.

Thank you for the follow-up. Below we clarify both points directly.

1. Necessity of evidence gating in Table R3

Every clip in Table R3 is a time-reversed RealEstate10K indoor sweep with no moving objects. Because nothing in the room changes state, the true change mask is zero everywhere (ei∗=0e_i^* = 0). Setting ei=0e_i = 0 (loop_w0.5) acts as the static-scene oracle mask, whereas any nonzero ei>0e_i > 0 in a static room can only relax a valid anchor.

On MiniWorld RealEstate10K, EGSA outputs ei>0e_i > 0 because MiniWorld runs in Wan2.2's 4x temporally compressing causal VAE space, where each token encodes a four-frame chunk including motion direction. As measured in Section 4.1.1, two ground-truth latent frames of the exact same static view under mirrored outbound/return motion still differ by an L2 residual of ~1.0, plus translation parallax under a rotation-only warp. Online residual detection responds to these velocity and parallax residuals during sweeps and turnarounds, lowering EGSA's hold below loop_w0.5 on static rooms (0.186906 vs. 0.141707 return MSE).

That static-world shortcut fails once real scene dynamics or multi-window drift enter. On the exact same MiniWorld backbone and VAE, when real arm and object motion occur in DROID (Tables R1 and R6), a constant hold freezes moving regions: at equal read weight (0.127295), EGSA cuts motion-magnitude MAE by 46.71% across 64 episodes, 3 seeds, and 3 noises (0.018935 vs. 0.035532, 64/64 episodes improved, Holm p = 0.000030, R1), and across 64 253-frame trajectories and 3 seeds (R6) cuts LPIPS by 14.13% (0.083246 vs. 0.096940, Holm p = 0.00018) and changing 2D RAFT flow error by 5.57% (0.801974 vs. 0.849290 px, Holm p = 0.00018). Likewise, once initial frames leave the window on DFoT (Table R7), constant holding without memory anchors to drifted frames and loses to EGSA with memory across all w∈[0.1,0.8]w \in [0.1, 0.8] (0.019396 for loop_w0.2 vs. 0.010252 at matched view-change ratio, -47.14%, Holm p = 0.00017).

2. Contribution of memory retrieval versus evidence gating in Table R2

First, previous-frame EGSA (0.017680) has higher return MSE than LoRA (0.016253) in Table R2 because windows 2 and 3 (4-11, 8-15) condition entirely on generated frames, unlike the paper's single-window setup where real context reached past the turnaround. Previous-frame anchoring pulls return frames (12-15) toward warped, drifted turnaround generations rather than evicted observations (0-3). Supplying last-valid memory restores the initial ray bank and cuts return MSE by 42.01% (0.017680 to 0.010252, 56/56 scenes, Holm p = 0.000020).

Second, the trade-off between evidence-free memory and EGSA with memory comes from camera translation under a pure-rotation ray memory. At return (12-15), the palindrome walks back onto the exact camera centers of frames 3-0, so parallax vanishes; setting ei=0e_i = 0 pulls return frames toward stored ground-truth rays at full weight (approaching Table R5's 0.000000 direct-reuse bound) and overwrote fewer bins during departure by suppressing outbound view change (0.703 vs. 0.755 for EGSA, 0.993 for LoRA), giving 0.007505 vs. 0.010252 return MSE. During outbound (4-7) and middle (8-11), however, camera translation creates parallax and disocclusions where pure-rotation warps fail. Evidence-free memory (ei=0e_i = 0) forces tokens toward misaligned warps, more than doubling outbound MSE (+115.0%, 0.026927 vs. 0.012524) and middle MSE (+106.6%, 0.026101 vs. 0.012634). EGSA detects these discrepancies (ei>0e_i > 0) and releases misaligned anchors, cutting 12-frame all-generated MSE by 41.50% vs. evidence-free memory (0.011803 vs. 0.020177, CI [-45.44%, -37.39%], 56/56 scenes, p = 0.000010). Gating reads by camera-center support (Table R5) cuts return MSE to 0.000562 with zero outbound penalty (all MSE 0.007063, LPIPS 0.100472 across 56 scenes).

Holding generator weights fixed within each controlled pair isolates each component:

ComponentControlled comparisonIsolated effect
Memory retrievalEGSA + memory vs. EGSA + prev (R2)Recovers evicted views: -42.01% return MSE (56/56, p = 0.000020)
Geometric supportSupported mem vs. No anchor (R5)Removes parallax ghosting: -30.80% LPIPS, -44.19% MSE (56/56)
Evidence gate (parallax)EGSA + mem vs. Evidence-free + mem (R2)Releases misaligned warps: -41.50% overall MSE (56/56, p = 0.000010)
Evidence gate (dynamics)EGSA vs. Const / Shuf, equal budget (R1, R6)Releases moving objects: -46.71% / -46.26% mag MAE; -14.13% LPIPS, -5.57% / -6.00% flow err

In the revised paper, we attribute long-range static revisit recovery directly to historical memory retrieval and geometric support, and attribute evidence gating to selective release under object motion and unmodeled correspondence error.

Reviewer_CTcHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EGSA as an additional anchoring module for persistent video world models. It uses signals from the current observation to decide when a new prediction should stay close to a warped estimate from an earlier frame. The experiments mainly use static-scene revisit sequences, with matched LoRA controls and a synthetic test intended to separate evidence-based holding from simply reusing the previous state.

Critical strengths
  1. Using signals from the current observation to decide when a new prediction should stay close to a warped estimate from an earlier frame is a worthwhile research question.
  2. The experiments in Tables 3 and 4 show the effects of the proposed method.
Critical weaknesses
  1. The evaluation would benefit from more challenging scenarios. The current evaluation is mainly based on static-scene revisits, while retaining a suitably warped earlier state can already be effective.
  2. The role of the robot images in Fig. 1 is unclear. The paper does not explain whether they are only illustrative or actual inputs to the proposed system, or why these examples are relevant to the video-world-model setting.
  3. For a paper about a video world model, the qualitative evidence is unusually limited. The paper does not provide enough continuous video results showing the outbound, look-away, and return phases, nor examples in which a real scene change should overcome the anchor.
Questions for authors

Please address the concerns in Critical Weaknesses.

1 comment

Thank you for the constructive feedback. We added real-video robot manipulation experiments across two 64-episode cohorts, clarified Fig. 1(c), and prepared continuous side-by-side videos covering dynamic object motion as well as outbound, look-away, and return phases.

1. Challenging dynamic scenes with moving arm and objects (W1)

We test EGSA on real action-conditioned DROID manipulation sequences where the robot arm moves, interacts with objects, and creates occlusions. Within each training seed, all arms share identical LoRA weights, initial RGB image, action sequence, rollout noise, and sampler:

DROID Cohort & MethodStatic MSEChanging MSEMotion / Flow MetricPerceptual / Mag Error
Short-horizon (64 eps, 3 seeds x 3 noises, R1)RGB-change ratioMagnitude MAE
No anchoring0.0014290.0814820.9390.009520
Evidence-free hold0.0014400.0832920.6650.040086
Mean-matched constant (w=0.127295)0.0014340.0813470.7030.035532
Shuffled evidence (w=0.127295)0.0014420.0817640.7060.035233
EGSA anchoring (w=0.127295)0.0014830.0796590.8420.018935
253-frame rollout (64 trajs, 3 seeds, R6)Changing 2D flow errMean LPIPS
No anchoring0.0016150.0631550.824440 px0.081793
Mean-matched constant0.0014940.0580390.849290 px0.096940
Shuffled evidence0.0015090.0584200.853144 px0.096940
Uniform strength 0.080.0015160.0551450.809770 px0.084491
Uniform strength 0.0560.0015350.0561940.807008 px0.082289
EGSA / accepted state0.0015000.0571910.801974 px0.083246
  • At identical mean read weight (0.127295), constant and shuffled holds suppress arm motion in Table R1 (RGB-change ratio 0.703 and 0.706 vs 0.842 for EGSA); EGSA cuts motion-magnitude MAE by 46.71% (95% CI [42.07%, 51.15%]) and 46.26% (95% CI [41.66%, 50.72%]), improving all 64/64 episodes (both Holm p=0.000030).
  • Over 253-frame rollouts from a single observed frame (Table R6), EGSA lowers full-sequence LPIPS by 14.13% against both constant and shuffled controls (95% CIs [-15.63%, -12.68%] and [-15.64%, -12.68%], both Holm p=0.00018) and lowers changing-region 2D RAFT flow error by 5.57% and 6.00% (both Holm p=0.00018). Against no anchoring (All MSE 0.028406), EGSA lowers All MSE by 9.74% (0.025639, Holm p=0.00005), static flow error by 13.50% (0.049989 vs 0.057788 px, Holm p=0.00018), and changing flow error by 2.73% (Holm p=0.00018). Against motion-matched uniform 0.08, EGSA lowers LPIPS by 1.47% (Holm p=0.0308) and static flow error by 5.11% (Holm p=0.00018), while uniform 0.08 and 0.056 obtain slightly lower pixel MSE (0.024681 and 0.025209).

2. Role of the robot images in Fig. 1 (W2)

In Fig. 1(c), the DROID thumbnails are schematic illustrations of action conditioning and spatial propagation, not extra image inputs to the gate. In our DROID world-model experiments, the model receives only an initial RGB frame and future robot action vectors: actions condition the video backbone to drive arm and object motion, while change evidence is estimated online from latent residuals without a calibrated kinematics-to-pixel map. We revised the Fig. 1 caption to state this explicitly and list the exact inputs per benchmark.

3. Continuous qualitative video evidence (W3)

Our supplementary media archive includes full continuous videos and frame strips for both regimes:

  • Real scene changes overcoming the anchor (DROID): Six independent dynamic rollout videos plus two complete 253-frame (6 fps) sequences and strips (episodes 5 and 10, Tables R4 and R4a) show captured video alongside EGSA, constant hold, shuffled evidence, weaker uniform hold, no anchoring, and memory-update controls. Constant and shuffled holds trail ghosted arm silhouettes, whereas EGSA lets the moving arm and grasped object update cleanly while keeping the static background anchored.
  • Outbound, look-away, and return phases (DFoT): Observing frames 0-3 and generating frames 4-15 across 8-frame windows (0-7, 4-11, 8-15) evicts frames 0-3 before the return. Across 56 scenes, 3 seeds, and 2 noises (Table R2), last-valid memory lowers return MSE by 36.92% vs LoRA and 42.01% vs previous-frame EGSA (56/56 scenes improved). Geometry-supported reads across all 56 scenes (Table R5) further cut all-generated LPIPS from 0.145196 to 0.100472 (-30.80%, 56/56 scenes). Twelve continuous 16-frame comparison videos display the observed prefix, outbound motion, turnaround, and return side by side.
Reviewer_jWVSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EGSA adds local state anchoring inside every denoising step of a frozen video world model: x^0←x^0+wi(Wx^0prev−x^0)\hat x_0\leftarrow\hat x_0+w_i(W\hat x_0^{prev}-\hat x_0), with wi∝(1−ei)ciνiw_i\propto(1-e_i)c_i\nu_i. In plain terms, when a location has no evidence that it is allowed to change and the geometric correspondence is reliable, it is pulled back toward the geometrically aligned previous prediction. On RealEstate10K forward–reverse (palindromic) trajectories, compared with matched LoRA twins, MiniWorld lowers return error by 5.3%–12.4%, and DFoT lowers away error by about 67% and return error by 12%–14%. Removing the attention bias changes results by at most 0.26%.

Critical strengths
  1. All 18 MiniWorld pool–seed–phase point estimates improve.

  2. Only 0.03%–0.14% of parameters are trainable, so the retrofit is easy to adopt. Training time still rises by about 36%–38% and memory by about 34%–82%.

Critical weaknesses
  1. The core evidence gate is never tested on its own on real video. All real-video experiments use static scenes, and the authors acknowledge that a plain hold without evidence anchors them just as well. Removing the attention bias rules out the bias, but it does not show that the gain comes from the change evidence. The only direct proof that real changes are correctly let through is Table 5, which uses synthetic data, exact masks, and an artificial predictor.

  2. The evaluation measures short-horizon stability on a continuous path, not memory after content leaves view. The main tables use previous-frame anchors, and the implemented last-valid memory is not used in any reported result (Appendix A.5). In the DFoT experiments, the context already extends past the point of furthest departure, and the model generates only the last 4 frames.

  3. Generic suppression of change is not ruled out. The tracking ratio measures how much the view changes, not whether it changes in the right direction. The EMA and warp-and-blend smoothing baselines are applied to the frozen base, which barely follows camera motion (tracking 0.007–0.009). So no one has tested whether simple smoothing on the LoRA twin would reproduce the same gain.

Questions for authors
  1. On real dynamic clips, compare full EGSA, hold-only, a strength-matched constant hold, and a shuffled-evidence hold, reporting static-region and changing-region error separately.

  2. Apply EMA and warp-and-blend to the LoRA twin with strength sweeps, and compare return error at similar tracking levels.

1 comment

Thank you for the sharp questions and experimental designs. We ran all three experiments: real-video dynamic controls on DROID, rolling-window memory after context eviction on DFoT, and LoRA-twin smoothing sweeps.

1. Isolating the evidence gate on real dynamic video (W1, Q1)

On 64 independent DROID episodes (3 training seeds x 3 noise seeds, Table R1), all arms within a seed share LoRA weights, initial frame, actions, noise, and sampler, without attention bias. Evidence is estimated online from latent residuals; captured masks only define static/changing blocks and the oracle reference.

InterventionStatic MSEChanging MSEAll MSERGB ratioMag MAERead wt
No anchoring0.0014290.0814820.0226550.9390.0095200.000000
Evidence-free hold0.0014400.0832920.0229960.6650.0400860.146250
Mean-matched constant0.0014340.0813470.0224480.7030.0355320.127295
Shuffled evidence0.0014420.0817640.0225920.7060.0352330.127295
EGSA anchoring0.0014830.0796590.0221870.8420.0189350.127295
Oracle mask (ref.)0.0015050.0813910.0227090.9370.0096560.101685

At identical read weight (0.127295), EGSA cuts motion-magnitude MAE by 46.71% vs constant hold (CI [42.07%, 51.15%]) and 46.26% vs shuffled evidence (CI [41.66%, 50.72%]), improving all 64/64 episodes (3-contrast Holm p=0.000030). Primary changing-MSE shifts are -2.08% vs constant (CI [-4.46%, +0.28%], Holm p=0.0850) and -2.58% vs shuffled (CI [-5.00%, -0.18%], Holm p=0.0686).

On 64 independent 253-frame DROID trajectories across 3 seeds (Table R6; frame 0 observed, frames 1-252 generated in 16-frame rolling windows with retained initial context), we measure AlexNet LPIPS v0.1 and 2D RAFT Large flow vector error over 63 four-frame intervals:

MethodAll MSEStatic MSEChanging MSELPIPSChg flowStat flow
No anchoring0.0284060.0016150.0631550.0817930.8244400.057788
EGSA / accepted state0.0256390.0015000.0571910.0832460.8019740.049989
Mean-matched constant0.0256520.0014940.0580390.0969400.8492900.050350
Shuffled evidence0.0258220.0015090.0584200.0969400.8531440.050168
Uniform strength 0.080.0246810.0015160.0551450.0844910.8097700.052682
Uniform strength 0.0560.0252090.0015350.0561940.0822890.8070080.053621

At equal budget, EGSA cuts LPIPS by 14.13% vs constant and shuffled holds (CIs [-15.63%, -12.68%], [-15.64%, -12.68%], 18-contrast Holm p=0.00018) and cuts changing 2D flow error by 5.57% and 6.00% (Holm p=0.00018), with All MSE nearly tied (-0.05%, -0.71%). Vs no anchoring, EGSA lowers All MSE by 9.74% (5-contrast Holm p=0.00005), static flow error by 13.50% (Holm p=0.00018), and changing flow error by 2.73% (Holm p=0.00018; LPIPS +1.78%). Vs uniform 0.08 (matched flow magnitude: 1.262 vs 1.259 px), EGSA lowers LPIPS by 1.47% (Holm p=0.0308) and static flow error by 5.11% (Holm p=0.00018); uniform 0.08 and 0.056 have lower pixel MSE (+3.88%, +1.70%; 0.056 gives LPIPS 0.082289 and 6.77% higher static flow error, Holm p=0.00018). No-update memory (seed 0) raises LPIPS to 0.105570 vs 0.082573 (-21.78% for EGSA, Holm p=0.00018).

2. Memory after context eviction (W2)

DFoT observes frames 0-3 and generates frames 4-15 via 8-frame windows (0-7, 4-11, 8-15) with Fourier conditioning, evicting frames 0-3 before return (12-15). Across 56 scenes, 3 seeds, and 2 noises (Table R2), last-valid memory lowers return MSE to 0.010252 vs 0.016253 for LoRA (-36.92%, CI [-40.85%, -32.15%], Holm p=0.000020) and 0.017680 for previous-frame EGSA (-42.01%, CI [-44.42%, -39.39%], Holm p=0.000020; 56/56 scenes improved; 12-frame MSE -11.88%). Geometry-supported reads across all 56 scenes (Table R5) cut all-generated LPIPS from 0.145196 (no anchoring) to 0.100472 (-30.80%, 56/56 scenes; -30.86% on 50 held-out scenes; direct reuse 0.085734).

3. LoRA-twin smoothing sweeps (W3, Q2)

  • MiniWorld LoRA twins (56 scenes, 3 seeds, Table R3): EMA and warp-and-blend across [0.2, 0.8] stay below EGSA's view-change ratios; in-sampler constant hold reaches nearby tracking with lower return MSE on short static paths (w=0.2: 0.159758 / 0.437242 vs EGSA dose 1: 0.189399 / 0.458999).
  • Rolling-window DFoT LoRA twins (56 scenes, 3 seeds x 2 noises, 18 conditions, Table R7): At matched view-change ratio (0.7553), EGSA-memory (return MSE 0.010252) beats EMA (b=0.35: 0.015570, ratio 0.7536, -34.15%), warp-and-blend (b=0.65: 0.020071, ratio 0.7586, -48.92%), and in-sampler constant hold (w=0.2: 0.019396, ratio 0.7739, -47.14%), all with 17-contrast Holm p=0.00017.

Comments on the paper

6 comments

We thank the Area Chair and Reviewers for their constructive feedback. We ran seven controlled experiment groups across DROID robot manipulation, rolling-window DFoT revisits, and MiniWorld LoRA twins to answer every experimental question:

  1. Online evidence gating in real dynamic video (Tables R1, R6):
  • Short-horizon DROID (64 episodes, 3 training seeds x 3 noise seeds): All arms within a seed share LoRA weights, inputs, noise, and sampler, without attention bias. At identical mean read weight (0.127295), EGSA lowers motion-magnitude MAE by 46.71% vs constant hold (95% CI [42.07%, 51.15%]) and 46.26% vs shuffled evidence (95% CI [41.66%, 50.72%]), improving all 64/64 episode means (both Holm p=0.000030), with changing-region MSE shifts of -2.08% (Holm p=0.0850) and -2.58% (Holm p=0.0686).
  • 253-frame long-horizon DROID (64 trajectories, 3 training seeds; frame 0 observed, frames 1-252 generated in 16-frame rolling windows): At equal read budget, EGSA lowers full-sequence LPIPS by 14.13% against both constant and shuffled holds (0.083246 vs 0.096940, both Holm p=0.00018) and lowers changing-region 2D RAFT flow error by 5.57% and 6.00% (both Holm p=0.00018). Against no anchoring, EGSA lowers All MSE by 9.74% (Holm p=0.00005), static flow error by 13.50% (Holm p=0.00018), and changing flow error by 2.73% (Holm p=0.00018).
  1. Memory after context eviction (Tables R2, R5):
  • Observing frames 0-3 and generating frames 4-15 via 8-frame windows (evicting frames 0-3 before return), last-valid memory lowers DFoT return MSE by 36.92% vs LoRA (0.010252 vs 0.016253) and 42.01% vs previous-frame EGSA (0.017680) across 56 scenes, 3 seeds, and 2 noises (both Holm p=0.000020; 56/56 scenes improved).
  • Restricting historical reads to geometry-supported poses across all 56 scenes (Table R5) lowers all-generated LPIPS from 0.145196 (no anchoring) to 0.100472 (-30.80%, 56/56 scenes; -30.86% on the 50 held-out scenes) and MSE from 0.012656 to 0.007063 (-44.19%).
  1. LoRA-twin smoothing sweeps, continuous videos, and ablations (Tables R3, R4, R7):
  • Across 56 rolling-window DFoT scenes (3 seeds x 2 noises, 18 conditions, Table R7), at matched view-change ratio (0.7553), EGSA-memory lowers return MSE by 34.15% vs EMA (b=0.35), 48.92% vs warp-and-blend (b=0.65), and 47.14% vs in-sampler constant hold (w=0.2), all with Holm p=0.00017.
  • Removing attention-logit bias on rolling-window DFoT shifts return MSE by +0.002441% and all-generated MSE by -0.007850%; all DROID runs omit it. Fig. 1(c) is clarified, and continuous DROID (253-frame) and DFoT videos are included.

Table R1: DROID short-horizon (64 episodes, 3 seeds x 3 noises, pixels in [-1, 1])

MethodStatic MSEChanging MSEAll MSERGB ratioMag MAERead wt
No anchoring0.0014290.0814820.0226550.9390.0095200.000000
Evidence-free hold0.0014400.0832920.0229960.6650.0400860.146250
Mean-matched constant0.0014340.0813470.0224480.7030.0355320.127295
Shuffled evidence0.0014420.0817640.0225920.7060.0352330.127295
EGSA anchoring0.0014830.0796590.0221870.8420.0189350.127295
Oracle mask (ref.)0.0015050.0813910.0227090.9370.0096560.101685

Table R6: DROID 253-frame confirmation (64 trajectories, 3 seeds; captured flow mag 1.506804 px)

MethodAll MSEStatic MSEChanging MSELPIPSChg flowStat flowFlow mag
No anchoring0.0284060.0016150.0631550.0817930.8244400.0577881.369588
EGSA / accepted state0.0256390.0015000.0571910.0832460.8019740.0499891.258909
Mean-matched constant0.0256520.0014940.0580390.0969400.8492900.0503501.115676
Shuffled evidence0.0258220.0015090.0584200.0969400.8531440.0501681.103651
Uniform strength 0.080.0246810.0015160.0551450.0844910.8097700.0526821.262254
Uniform strength 0.0560.0252090.0015350.0561940.0822890.8070080.0536211.297535

Table R2: DFoT memory (56 scenes, 3 seeds x 2 noises) & Table R5: Supported reads (56 scenes)

  • R2 Return / All MSE: Frozen 0.016127 / 0.012250; LoRA 0.016253 / 0.012349; EGSA prev-frame 0.017680 / 0.013394; EGSA memory 0.010252 / 0.011803; EGSA no-bias memory 0.010253 / 0.011802; Evidence-free memory 0.007505 / 0.020177.
  • R5 All / Return LPIPS (All / Return MSE): No anchoring 0.145196 / 0.172913 (0.012656 / 0.017030); Supported memory 0.100472 / 0.040085 (0.007063 / 0.000562); Direct view reuse 0.085734 / 0.000000 (0.006896 / 0.000000).

Complete Smoothing Sweeps and Development Tables (Tables R3, R4, R4a, R7)

Table R3: MiniWorld LoRA-twin smoothing sweep (56 scenes, 3 training seeds, noise seed 0)

ConfigurationEarly MSEAway MSEReturn MSEView-change ratioDirection proxy
twin0.2557450.3107960.2073640.4437760.358600
ema_b0.20.2520370.3068550.2042020.4200650.359678
warpblend_b0.20.2358320.2997720.1954140.4008850.374925
ema_b0.350.2483160.3033080.2006390.3945090.359748
warpblend_b0.350.2198490.2918220.1875970.3717890.389353
ema_b0.50.2432780.2988470.1949680.3563680.359006
warpblend_b0.50.2030180.2832140.1814300.3452310.408046
ema_b0.650.2358740.2924260.1848440.2966430.360102
warpblend_b0.650.1852110.2712310.1782020.3217250.438429
ema_b0.80.2243980.2804610.1651810.2048220.384256
warpblend_b0.80.1677200.2496090.1772890.3110120.497957
loop_w0.10.2228020.2595180.1694800.4130380.477692
loop_w0.20.1972750.2346310.1597580.4372420.543493
loop_w0.350.1751370.2162710.1485330.4847430.591624
loop_w0.50.1662170.2106690.1417070.5227640.609294
full_iga_dose1.00.2475230.2969280.1893990.4589990.400182
full_iga_dose2.00.1785320.2337540.1869060.5113720.567206

Table R7: Rolling-window DFoT LoRA-twin smoothing sweep (56 scenes, 3 training seeds x 2 noise seeds)

ConfigurationOut MSEMid MSERet MSEAll MSEVC ratioRet LPIPSAll LPIPS
twin0.0102950.0105030.0162540.0123510.9934640.1735610.146786
ema_b0.20.0100220.0104940.0151270.0118810.8557400.1870570.163008
ema_b0.350.0112980.0114470.0155700.0127720.7535570.2234500.202478
ema_b0.50.0140920.0130870.0172670.0148150.6446950.2849560.263368
ema_b0.650.0191420.0152480.0206520.0183470.5168230.3754540.341837
ema_b0.80.0281640.0189150.0253530.0241440.3368550.4643420.422778
warpblend_b0.20.0101450.0103880.0155760.0120360.9161770.1838490.158748
warpblend_b0.350.0110910.0109900.0159730.0126850.8628870.2112540.188183
warpblend_b0.50.0131350.0121320.0173260.0141980.8110200.2578090.233993
warpblend_b0.650.0167870.0137800.0200710.0168790.7585660.3261470.293090
warpblend_b0.80.0231820.0166970.0244070.0214290.7042450.3991980.355620
loop_w0.10.0102510.0103270.0161070.0122290.8482990.2448260.190608
loop_w0.20.0126740.0122520.0193960.0147740.7738990.3780850.284143
loop_w0.30.0163410.0143550.0232000.0179650.7332960.4801410.368502
loop_w0.450.0223330.0175640.0279790.0226250.7042290.5321150.435633
loop_w0.60.0278590.0211790.0287710.0259360.6771970.5055270.445357
loop_w0.80.0335930.0271930.0282430.0296760.6522980.4391060.417170
egsa_memory0.0125250.0126350.0102520.0118040.7553090.2356440.212268

Tables R4 and R4a: Two fixed 253-frame DROID development sequences (episodes 5 and 10)

MethodLPIPSLate LPIPSChanging flow err (px)All flow err (px)
No anchoring0.0760640.0878720.7566570.290860
EGSA / accepted state0.0751520.0854550.7153820.264577
Mean-matched uniform0.0897980.1036340.7644810.280140
Shuffled0.0907730.1044110.7606390.279214
Uniform strength 0.080.0777670.0894450.7083140.264998
Uniform strength 0.056 (R4a)0.0760160.0873350.7083060.267266
Initial memory without updates0.0868820.1002500.7569220.276656
Additional hard cache admission0.0798330.0919330.7360170.269787
Fractional cache update0.0798370.0918020.7337680.269060

Thank you for the close reading. We ran two follow-up evaluations on the cohorts of Tables R5 and R6 to address W1-W3.

W1. Evidence gating and uniform holds on 253-frame DROID (Table R6)

The two uniform weights in Table R6 sit at EGSA's motion level: 0.056 was interpolated on eight development episodes to reproduce EGSA's motion, and 0.08 lands within 0.3% of EGSA's changing flow magnitude on this cohort (1.2623 vs. 1.2589 px). Both are 6x to 9x below Table R3's best weight (w=0.50w=0.50). Set to w=0.50w=0.50 on all 64 253-frame DROID trajectories across 3 seeds, the same uniform hold removes 90.2% of motion (changing flow magnitude 0.1479 px vs. 1.507 px captured and 1.2589 px for EGSA) and raises All MSE to 0.048121, LPIPS to 0.153923, and changing 2D RAFT flow error to 1.485516 px. EGSA is lower by 46.72% in All MSE and 45.92% in LPIPS on 64/64 trajectories and by 46.01% in changing flow error on 62/64 (all Holm p=0.000080). From one fixed weight, EGSA releases 74.3% of moving tokens while firing on 1.68% of static tokens.

In the second half of each rollout (frames 126-252), EGSA has lower 2D flow error than both uniform weights in static regions, changing regions, and whole frames:

Late-frame metric (64 trajs x 3 seeds; static rows 63)EGSAUniform 0.08Uniform 0.056EGSA vs. 0.08 (CI)EGSA vs. 0.056 (CI)
Static flow err (px)0.0419550.0446770.045854-6.09% ([-8.39, -3.80])-8.50% ([-11.33, -5.73])
Static MSE0.0017960.0018350.001856-2.12% ([-4.97, +0.33])-3.20% ([-6.38, -0.31])
Changing flow err (px)0.7752140.7850380.783833-1.25% ([-2.58, -0.15])-1.10% ([-2.20, -0.13])
Whole-frame flow err (px)0.4177910.4223810.423114-1.09% ([-2.41, -0.15])-1.26% ([-2.56, -0.34])

Against motion-matched 0.08, EGSA also lowers motion MAE by 23.89% (0.011221 vs. 0.014744, CI [-28.99%, -18.38%]), LPIPS by 1.47% (0.083246 vs. 0.084491, Holm p=0.0308), and late LPIPS by 1.63% (0.097851 vs. 0.099470).

W2. 56-scene revisit when return views are absent from memory

In R5, symmetric subsampling made return poses identical to frames 0-3. To test retrieval when no return view is in memory, we evaluated all 56 RealEstate10K scenes under panned sub-FOV views (zoom 1.35, all GT pixels inside real captures): frames 0-3 pan left (−6∘-6^\circ yaw) while frames 12-15 return to positions 3-0 panning right (+6∘+6^\circ yaw, a 12∘12^\circ shift where 21.3% of each return view was never seen in 0-3 and 78.7% is perspective-shifted):

Method (56 scenes, +12 deg return)Ret MSEAll MSERet LPIPSAll LPIPSHits
LoRA twin0.0484870.0377710.4503060.354812n/a
Same model / no anchoring0.0493680.0383280.4519180.355529n/a
EGSA / previous frame0.0493210.0382940.4520580.355409n/a
Direct view reuse (R5 rule)0.0493680.0382600.4519180.3541640/224
Same-position unwarped copy0.0591760.0415290.5125250.374367n/a
Supported ray memory + EGSA0.0123160.0258910.1831340.264926n/a

With no return view in memory, direct view reuse gets 0/224 hits and matches the unanchored return error, while unwarped copying raises Return LPIPS to 0.512525. With the R5 checkpoint and memory code unchanged, supported ray memory + EGSA reprojects the 78.7% overlapping rays and generates the unseen 21.3% border, cutting Return MSE by 74.60% vs. LoRA twin (CI [-77.05%, -72.09%]) and 75.05% vs. direct reuse, and Return LPIPS by 59.33% vs. LoRA twin (CI [-60.71%, -57.93%]) and 64.27% vs. unwarped copy (each on 56/56 scenes, Holm p=0.00010). The manuscript specifies the directional last-valid memory (Sec. 6; Appendix A.5, D.2) and its tables use the previous-frame anchor on continuous reverse paths. After eviction the previous frame is a generated prediction, so the anchor reads from memory (Tables R2, R5, and above). The uploaded revision integrates these memory and 253-frame DROID results into Sections 4.4-4.5 (Tables 6-7).

W3. Constant hold on LoRA twins (Table R3) under motion and rolling windows

Table R3 tests static rooms (ei∗=0e_i^*=0 everywhere), where fixing ei=0e_i=0 (loop_w0.5) matches the true mask. On MiniWorld, Wan2.2's 4x temporally compressing VAE encodes motion direction into tokens: same-pose latent frames with mirrored motion differ by a raw residual of about 1.0 (Sec. 4.1.1), so the online detector registers part of this residual as evidence and relaxes the hold. When objects move or observed frames leave the window, the ordering reverses: on the same MiniWorld backbone, w=0.50w=0.50 removes 90.2% of DROID motion (W1), and in rolling-window DFoT (R7, 56 scenes, 3 seeds x 2 noises) a constant hold trails EGSA with memory at every tested ww from 0.1 to 0.8 (-36.35% to -64.37% return MSE, Holm p=0.00017; -37.67% return LPIPS at tracking-matched loop_w0.2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.