acceptodds
Under review as a conference paper at ICLR 2027

SceneMemBench: A Benchmark for Scene Memory in Interactive Video World Models

Abstract

Interactive video world models generate navigable worlds frame by frame, but a persistent world requires previously observed scenes to remain recoverable as users move away and return. Existing memory evaluations, however, often focus on seen-view revisits and full rollouts, leaving cross-view scene recovery underexplored while entangling memory with trajectory-following errors and accumulated generation degradation. We introduce SceneMemBench, a benchmark for scene memory in interactive video world models. It evaluates two complementary dimensions: seen-view recall versus novel-view recovery, and controlled diagnosis versus end-to-end interaction. We introduce Continued Rollout from a shared ground-truth observation history, alongside Full Rollout over complete model-generated interactions. SceneMemBench contains 600 controlled trajectories across 33 Unreal Engine scenes and 60 Full Rollout cases, covering eight representative open- and closed-source models. We evaluate scene recovery along subject consistency, appearance fidelity, and geometric consistency. Results show that strong seen-view recall does not guarantee strong novel-view recovery, while model profiles under controlled histories can differ markedly from those observed in full interaction. Together, these findings reveal complementary failure modes obscured by seen-view revisits or end-to-end rollouts alone. Our benchmark will be released at https://anonymous.4open.science/r/SceneMemBench.

Primary area:
datasets and benchmarks
Keywords:
interactive world models, scene memory, world model evaluation, datasets and benchmarks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PxK9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

SceneMemBench evaluates whether interactive video world models can recover a scene after it leaves the current view. It separates seen-view recall from novel-view recovery and introduces two protocols: Continued Rollout from a shared ground-truth history and Full Rollout from complete model-generated interactions. The benchmark contains 600 controlled trajectories across 33 Unreal Engine scenes and 60 Full Rollout cases. Eight models are compared using subject, appearance, and geometric consistency measures, with confidence intervals and a human study of the VLM-based evaluator.

Critical strengths
  1. The separation of seen-view recall, novel-view recovery, controlled continuation, and full interaction is useful. In particular, Continued Rollout removes accumulated generation and trajectory errors, so it can expose memory behavior that is hidden in end-to-end evaluation.

  2. The evaluation is careful for a benchmark paper. It covers several open and closed models, reports paired uncertainty for the controlled protocol, documents model-specific history initialization, and checks the reliability of the VLM evaluator against human judgments.

Critical weaknesses
  1. The controlled benchmark is restricted to 33 rendered Unreal Engine scenes. This is suitable for obtaining exact camera and geometry references, but it leaves uncertainty about whether rankings transfer to real captures, dynamic scenes, or less game-like visual statistics.

  2. The Full Rollout protocol is less controlled than the Continued Rollout protocol. It contains only 60 cases, relies on pairwise VLM scoring without a ground-truth novel-view image, and its motion screen checks for recognizable movement rather than successful execution of the required revisit. A model may therefore be scored after reaching the wrong endpoint.

Questions for authors
  1. How sensitive are Full Rollout rankings to filtering for actual trajectory completion rather than only recognizable motion? Can the authors manually report the endpoint-success rate for each model?

  2. Do the authors have evidence that the principal rankings from the Unreal Engine scenes transfer to a small set of real captured scenes with known camera poses?

  3. For novel-view Full Rollout, how stable are the VLM scores across judges or repeated judge calls, especially in categories where human agreement is lower?

6 comments
AuthorsOpenReview ↗

Response to Reviewer PxK9 (1/4)

We thank the reviewer for recognizing the two evaluation protocols and experimental design. We have added experiments on revisit completion, real captures, and VLM reliability, which we discuss below.

W1 / Q2. Do the authors have evidence that the principal rankings from the Unreal Engine scenes transfer to a small set of real captured scenes with known camera poses?

Full Rollout rankings agree strongly across UE and real captures; Continued Rollout agreement is weaker, especially for Novel. We recorded 25 handheld videos across indoor and outdoor locations (14 Seen, 11 Novel), with ARKit-estimated camera trajectories. Six models produced 300 outputs under Full Rollout (complete generation from the initial view) and Continued Rollout (revisit generation from recorded GT history). These captures test ranking transfer beyond game-rendered visual content, and the split will be released with the benchmark.

Table 1 compares the six models’ rankings on real captures and the original UE benchmark using Spearman’s ρ; values closer to 1 indicate more similar rankings. Full Rollout uses the same VLM judge, Gemini-3.8-flash, on both datasets. Continued Rollout uses fixed GT and generated query endpoints. We min–max normalize each metric across the six models, then average equally within each dimension: DINO for subject, LPIPS/SSIM/PSNR for appearance, and SI-AbsRel/normal cosine for geometry. Overall scores weight Seen and Novel equally.

Table 1. Ranking agreement between real captures and the UE benchmark.

ProtocolRevisit typeSubject ρAppearance ρGeometry ρ
Full RolloutSeen0.9280.8860.899
Full RolloutNovel0.7140.8290.714
Full RolloutOverall0.9430.8290.829
Continued RolloutSeen0.6960.7940.464
Continued RolloutNovel0.2570.0290.257
Continued RolloutOverall0.4290.2570.257

Full Rollout achieves overall ρ of 0.829–0.943: models ranked higher on UE generally also rank higher on real captures. Continued Rollout agreement is weaker, especially for Novel (ρ = 0.029–0.257), so its UE rankings do not directly predict rankings on this real-capture set.

Unlike UE’s exact poses, real captures rely on ARKit estimates, making pose and FOV alignment less certain. Automatic metrics are sensitive to these differences; history length and revisit visibility also differ across datasets. These factors may contribute to weaker agreement. Qualitative examples nevertheless show successful recovery and similar failures in both domains, including target-view deviations, missing content, and scene rewriting (revised manuscript, Appendix F.4, Figures 12–13).

Static scenes give memory recovery an unchanged target. Dynamic scenes require defining how the target may reasonably change while out of view; we leave this extension to future work. Broader real-world coverage also warrants further study.

AuthorsOpenReview ↗

Response to Reviewer PxK9 (2/4)

W2 / Q1. How sensitive are Full Rollout rankings to filtering for actual trajectory completion rather than only recognizable motion? Can the authors manually report the endpoint-success rate for each model?

We manually reviewed all 480 scored videos. Overall rankings change little after excluding clear failures.

(i) Checking revisit completion. Closed-source models are controlled manually; open-source models use predefined trajectories, calibrated action scales, and additional turning margins where needed. Once a stable revisit is reached, the video can be truncated at that point to prevent subsequent motion from affecting evaluation. Endpoint selection thus already aims to avoid scoring an incorrect endpoint caused by incomplete execution.

The manual review counts action drift, stalled turning, and visual collapse as clear revisit or generation failures. Table 2 reports completion and failure counts for each model, with 60 outputs per model.

Table 2. Revisit completion and clear failures in Full Rollout.

ModelRevisit completion supportedRevisit or generation failureFailure rate
Genie36000.0%
Happy Oyster51915.0%
HY421830.0%
LingBot501016.7%
MG36000.0%
MG3.55823.3%
SANA5823.3%
Zing421830.0%
Total4215912.3%

Overall, 59 of 480 outputs (12.3%) show clear revisit or generation failures, with rates ranging from 0% to 30% across models. These counts do not separately measure gradual degradation in rollouts that still complete the revisit.

(ii) Testing ranking sensitivity. To test whether these failures affect model comparisons, we excluded the 59 clear failures and recomputed rankings using the original Gemini-3.8-flash scores. Overall scores assign equal weight to Seen and Novel. Table 3 compares model rankings before and after excluding these failures.

Table 3. Ranking correlations before and after excluding clear failures.

Spearman correlation before vs. after filteringSubject ρAppearance ρGeometry ρ
Seen0.9761.0000.929
Novel0.9290.9290.976
Overall0.9761.0001.000

The small changes show that overall rankings are stable to excluding clear failures. We retain these failures because Full Rollout evaluates the entire interaction, including whether the model completes the revisit and preserves the scene along the way. Its scores therefore reflect memory, camera control, and generation quality together. Continued Rollout provides a complementary test: every model starts from the same GT history, so errors accumulated before the query do not affect the evaluation. This allows us to compare scene recovery from a common starting point.

AuthorsOpenReview ↗

Response to Reviewer PxK9 (3/4)

Q3. For novel-view Full Rollout, how stable are the VLM scores across judges or repeated judge calls, especially in categories where human agreement is lower?

For novel-view Full Rollout, different VLM judges give similar model rankings, while score stability across repeated calls varies by judge and evaluation dimension. We report subject, appearance, and geometry separately, with particular attention to categories where human agreement is lower. Below, we compare model rankings across judges, examine their agreement on individual outputs in these categories, and test whether repeated calls produce similar scores.

(i) Do different judges rank the models similarly? We evaluated all 480 Full Rollout outputs with Gemini-3.8-flash, Qwen3-VL-8B-Instruct, and GPT-6.1-sol, using the same scoring images and the original scoring rules.

For each judge, we rank the eight models by their average scores. Seen and Novel each give equal weight to their two trajectory types; overall scores give equal weight to all four types. Table 4 compares the eight models’ rankings across the three judges using Spearman’s ρ, where values closer to 1 mean more similar rankings.

Table 4. Model-ranking agreement across VLM judges.

GroupJudge pairSubject ρAppearance ρGeometry ρ
Novel, 240 outputsGemini—Qwen0.9760.9050.976
Novel, 240 outputsGemini—GPT0.9760.9290.976
Novel, 240 outputsQwen—GPT1.0000.9761.000
Seen, 240 outputsGemini—Qwen1.0000.9520.952
Seen, 240 outputsGemini—GPT0.9761.0000.976
Seen, 240 outputsQwen—GPT0.9760.9520.976
Overall, 480 outputsGemini—Qwen1.0001.0001.000
Overall, 480 outputsGemini—GPT1.0001.0001.000
Overall, 480 outputsQwen—GPT1.0001.0001.000

The three judges produce similar rankings for Seen and Novel, and identical overall rankings in all three dimensions. Ranking correlations are 0.905–1.000 for Novel and 0.952–1.000 for Seen.

(ii) What happens in the categories where human agreement is lower? In our original human study, agreement with Gemini was lower for Novel appearance (70.0%) and geometry (68.8%) than for subject consistency (80.0%). The lowest agreement was in Forward and Look Back, at 58.3% for appearance and 62.3% for geometry (Table 10 in Appendix D.2 of the manuscript).

We therefore compare the two Novel types separately: Forward and Look Back and New Viewpoint. Each contains 15 trajectories × 8 models = 120 outputs. We use the existing scores from all three judges, without excluding outputs or making additional API calls.

Table 5 compares model rankings across judges within each Novel type. We average each model’s scores over its 15 trajectories, rank the eight models, and compare the rankings between judges.

Table 5. Model-ranking agreement within each Novel subtype.

Novel subtypeJudge pairSubject ρAppearance ρGeometry ρ
Forward and Look BackGemini—Qwen0.9520.9760.850
Gemini—GPT0.9940.9760.934
Qwen—GPT0.9340.9760.929
New ViewpointGemini—Qwen0.9520.9520.857
Gemini—GPT0.9760.9290.952
Qwen—GPT0.9760.9520.952

Model rankings remain similar in Forward and Look Back: appearance correlations are 0.976 for all three judge pairs, and geometry correlations range from 0.850 to 0.934.

Similar model rankings, however, do not mean that judges agree equally well on every output. Table 6 compares the judges’ scores across the 120 individual outputs in each Novel type using Spearman’s ρ.

Table 6. Per-output score correlations within each Novel subtype.

Novel subtypeJudge pairSubject ρAppearance ρGeometry ρ
Forward and Look BackGemini—Qwen0.7050.5200.615
Gemini—GPT0.8720.7550.718
Qwen—GPT0.7230.7140.771
New ViewpointGemini—Qwen0.7810.7540.775
Gemini—GPT0.9140.9040.879
Qwen—GPT0.8030.7780.809

For Forward and Look Back, appearance and geometry correlations are 0.520–0.755 and 0.615–0.771, respectively. For every judge pair, these are lower than in New Viewpoint, where the corresponding ranges are 0.754–0.904 and 0.775–0.879.

The category with lower human agreement also shows more disagreement between VLM judges on individual outputs. Nevertheless, averaging over trajectories gives similar model rankings. Appearance illustrates this clearly: all three model-ranking correlations are 0.976, while individual-output correlations range from 0.520 to 0.755.

Reviewer_YvinOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

A comprehensive benchmark for evaluating the memory capability of interactive video world models. It uses two complementary dimensions to evaluate the memory of video world models via seen-view revisit and novel-view recovery, while also introducing a method of controlled-history generation to decouple the capability of memory from anti-drifting or long-horizon generation. Via these two methods, the benchmark could potentially evaluate the memory module of interactive video generation models more clearly.

Critical strengths
  1. A comprehensive benchmark for interactive video world models to evaluate their memory capabilities, with high-quality scenes, assets, and trajectories, and with informative evaluation and analysis results for different foundation models.
  2. Clean evaluation pipelines with continued rollout and full rollout, to decouple the evaluation of long-horizon video generation from that of the memory capability itself.
  3. A clear split of the memory evaluation into seen-view revisit and novel-view recovery, which makes the benchmark more diagnostic by separating simple memory recall from novel-view synthesis with history information.
Critical weaknesses
  1. Limited to static and game-style world simulator without real-world scenes (like MemoBench), which is crucial to the downstream application like robotics.
  2. Compared with agent-driven trajectory (like PlayWorld), static interactive trajectories limit the potential of flexible and comprehensive evaluation of the memory and interactivity of video world model.
  3. Use a set of small modules or rule-based criteria to evaluate the consistency, but without comprehensive human-study to verify whether the verification itself is aligned with human understanding.
Questions for authors
  1. Have you ever evaluate world models which have explicit 3D information or memory on this benchmark? Will 3D explicit information bring advantages over the latent memory paradigm?
  2. Why use estimated monocular depth instead of ground truth information?
  3. Can you extend the human study to other evaluation criteria including VLM judgment like Gemini or 3D proxy?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Yvin (1/3)

We thank the reviewer for recognizing the quality of our benchmark and model analyses, the complementary Continued and Full Rollout protocols, and the distinction between seen-view recall and novel-view recovery. We address the concerns below.

W1. Limited to static and game-style world simulator without real-world scenes (like MemoBench), which is crucial to the downstream application like robotics.

We extend SceneMemBench with real-world recordings, while retaining UE for controlled scene-memory evaluation. Below, we explain this design choice, report the real-world results, and clarify the connection to robotics.

(i) Why UE and static scenes. We use Unreal Engine for the core benchmark by design: separating memory from other errors requires exact camera poses, ground-truth frames at novel revisit viewpoints, and controllable revisit trajectories, which real captures cannot provide simultaneously. Static scenes give a well-defined target, since a revisited scene should remain unchanged.

(ii) Real-world evaluation and findings. We recorded 25 handheld videos in indoor and outdoor locations, comprising 14 Seen and 11 Novel revisits with ARKit-estimated camera trajectories. Six models generated 300 outputs across Continued Rollout and Full Rollout. This real-world split will be released with the benchmark.

In Continued Rollout, models generate the revisit segment from recorded GT history. Table 1(a) reports six automatic metrics comparing the fixed generated query endpoint with the fixed GT query endpoint.

Table 1(a). Continued Rollout results on real captures. Each model is evaluated on 14 Seen and 11 Novel videos. Bold marks the best score within each task.

TaskModelDINO ↑LPIPS ↓SSIM ↑PSNR ↑SI-AbsRel ↓Normal ↑
SeenHY0.82840.54760.339612.39890.28180.5943
SeenLingBot0.92950.39790.360913.70890.17520.7175
SeenMG30.81380.50640.325612.17900.33850.6365
SeenMG3.50.76990.48600.377513.73070.25820.6571
SeenSANA0.84250.38260.377314.37090.18990.7299
SeenZing0.80050.52150.349812.39760.26350.6206
NovelHY0.53890.65840.282911.12250.59590.5284
NovelLingBot0.68910.54750.318812.21270.37290.6262
NovelMG30.62680.58010.326811.56400.40560.5978
NovelMG3.50.65100.56750.344413.15210.27590.6239
NovelSANA0.46180.61460.257611.33340.46220.5449
NovelZing0.72210.53890.343412.91790.24050.6525

Novel scores are worse than Seen scores in 32 of the 36 model–metric comparisons, suggesting that novel-view recovery is generally more challenging on these real captures. The leading models also differ by task: LingBot and SANA lead five of the six Seen metrics, while Zing and MG3.5 lead all six Novel metrics.

In Full Rollout, models generate the complete interaction from the initial frame, prompt, and controls. Gemini-3.8-flash compares the initial reference frame with the generated revisit endpoint to assess subject, appearance, and geometry. Table 1(b) reports these scores; overall scores weight Seen and Novel equally.

Table 1(b). Full Rollout results on real captures. Each cell reports subject / appearance / geometry scores; higher is better. Bold marks the highest score in each column and dimension.

ModelSeenNovelOverall
HY0.7857 / 0.7000 / 0.77860.7091 / 0.5455 / 0.65450.7474 / 0.6227 / 0.7166
LingBot0.2857 / 0.3000 / 0.14290.6545 / 0.6091 / 0.40910.4701 / 0.4545 / 0.2760
MG30.7857 / 0.7429 / 0.75710.3727 / 0.2909 / 0.33640.5792 / 0.5169 / 0.5468
MG3.50.9429 / 0.7714 / 0.82860.1000 / 0.1273 / 0.05450.5214 / 0.4494 / 0.4416
SANA0.4071 / 0.3143 / 0.32860.0545 / 0.0182 / 0.02730.2308 / 0.1662 / 0.1779
Zing0.2643 / 0.1929 / 0.14290.0909 / 0.0818 / 0.08180.1776 / 0.1373 / 0.1123

HY leads overall in all three dimensions, followed by MG3. MG3.5 leads Seen but scores much lower on Novel, while LingBot achieves the highest Novel appearance score. These results reveal different model strengths across revisit tasks in real environments.

Together, these experiments extend SceneMemBench beyond rendered scenes and demonstrate that both protocols can evaluate recovery on real captures. Qualitative examples show successful continuation on real captures and similar failure modes in real and UE continuations, including target-view deviations, missing content, and scene rewriting (revised manuscript, Appendix F.4, Figures 12–13).

(iii) Relevance to robotics. These real-world recordings extend evaluation beyond game-rendered visual content, and their first-person handheld trajectories bring it closer to observations encountered in robotics and embodied settings. The real-world split therefore complements the controlled UE benchmark. Broader real-world coverage and downstream robotic task evaluation remain future work.

AuthorsOpenReview ↗

Response to Reviewer Yvin (2/3)

W2. Compared with agent-driven trajectory (like PlayWorld), static interactive trajectories limit the potential of flexible and comprehensive evaluation of the memory and interactivity of video world model.

Our controlled protocol compares scene memory under the same GT history and revisit query, provides target-view references, and avoids additional agent calls during execution. Both benchmarks evaluate scene revisits, but they control different aspects of the test. In PlayWorld’s paper (§3.1), the Agent Player starts from a shared basic action sequence and adjusts execution based on generated observations to reach a specified goal. SceneMemBench instead fixes the history and query for Continued Rollout, allowing models to be compared from the same starting conditions.

(i) Separating memory from earlier rollout errors. PlayWorld’s adaptive control helps models reach the intended revisit, but the preceding history is still generated by each model. Action corrections do not remove content drift or degradation that has already occurred. In Continued Rollout, every model receives the same GT history, so control and generation errors accumulated before the query do not affect the comparison. Full Rollout complements this test by evaluating the complete interaction.

(ii) Providing GT references, including novel views. Our prescribed query endpoints allow UE to render reference images at the target viewpoints. This supports direct evaluation with DINO, LPIPS, SSIM, PSNR, SI-AbsRel, and normal consistency. PlayWorld assesses its core tasks through VQA rubrics and also reports basic quality and control metrics; our controlled setting additionally provides GT images for measuring scene recovery at specified revisit viewpoints.

(iii) Keeping comparisons consistent and straightforward to run. Models receive the same history and revisit query, without an agent making different decisions for each model. The predefined queries can also be executed without repeated agent calls. This makes the comparison easier to reproduce and avoids the extra decision overhead of agent-driven control.

Coverage and scale. PlayWorld covers 171 tasks across several interactive capabilities, including 48 geometry-consistency cases assessing static scene memory. SceneMemBench focuses on scene-revisit memory, with 600 controlled trajectories across 33 UE scenes and 60 Full Rollout cases, supplemented by 80 translated-Seen trajectories and 25 real captures. Although the sample units differ, SceneMemBench provides a larger collection dedicated to testing scene-revisit memory. Its coverage is more focused, with separate evaluation of Seen recall and Novel recovery under controlled histories and full interaction.


W3/Q3. Can you extend the human study to other evaluation criteria including VLM judgment like Gemini or 3D proxy?

Human studies support both the Gemini evaluator used in Full Rollout and the automatic metrics used in Continued Rollout.

(i) Full Rollout: human–Gemini agreement. The original manuscript already validates the Gemini evaluator against human preferences. The study covers 240 output pairs, with agreement rates of 80.4% for subject, 77.1% for appearance, and 76.3% for geometry. These results are reported in Table 10 in Appendix D.2 of the manuscript.

(ii) Continued Rollout: human–metric agreement. We extended the study by randomly sampling 120 pairs of Continued Rollout outputs. For each pair, human evaluation compared both candidate generated revisit frames against the fixed GT query endpoint, judging subject, appearance, and geometry and allowing left, right, or tie. Automatic preferences use DINO↑ for subject, LPIPS↓ for appearance, and the depth-based proxy SI-AbsRel↓ for geometry.

Table 2 reports agreement between each metric and human preferences on all 120 pairs.

Table 2. Agreement between automatic metrics and human preferences in Continued Rollout.

ComparisonSubjectAppearanceGeometry
Metrics–Human87.5% (105/120)81.7% (98/120)75.0% (90/120)

All three metrics agree with human preferences on at least 75.0% of pairs, with the highest agreement for subject consistency. Together with the original Full Rollout study, these results support the use of our evaluators to assess relative recovery quality across subject, appearance, and geometry.

AuthorsOpenReview ↗

Response to Reviewer Yvin (3/3)

Q1. Models with explicit 3D memory

Yes—we evaluate Matrix-Game 3.5, which uses explicit 3D memory.

(i) Memory mechanisms. MG3.5 uses Depth Anything 3 (DA3) to estimate metric depth. Together with camera intrinsics and poses, this allows its Patch Memory to map historical latent patches into 3D world coordinates and reproject them into the target view. Where patches overlap, retrieval keeps the nearest surface, forming a target-aligned mosaic memory. The model therefore explicitly establishes 3D correspondences between historical observations and the target view (official report, §2.2.1). See Appendix F.5, Figure 14 of the revised manuscript for retrieved historical views and mosaic memory.

For comparison, we report MG3.0 as a model with latent-based memory. It retrieves whole historical frames based on field-of-view overlap, while the model learns how their content corresponds across viewpoints.

(ii) Advantages and limitations. MG3.5 recovers scene geometry and structure better, but does not improve all aspects of visual consistency. Compared with MG3.0 on Novel Continued Rollout, it improves SI-AbsRel, normal cosine, SSIM, and PSNR. However, DINO drops by 0.1849 and LPIPS is slightly worse by 0.0017, indicating weaker global semantic similarity and no improvement in perceptual appearance similarity.

Table 3 compares the two models on 400 Novel Continued Rollout trajectories per model.

Table 3. MG3.0 and MG3.5 on Novel Continued Rollout. Positive gains favor MG3.5.

ModelDINO ↑LPIPS ↓SSIM ↑PSNR ↑SI-AbsRel ↓Normal ↑
MG30.75430.51240.339413.67290.51300.7004
MG3.50.56940.51410.367414.37590.43680.7079
MG3.5 gain over MG3−0.1849−0.0017+0.0280+0.7030+0.0762+0.0075

These results support the interpretation that explicit geometric memory helps structural recovery, while semantic and perceptual consistency remain challenges. We evaluate only one model with explicit 3D memory, so this finding should not be taken as a general advantage over latent-based memory. Evaluating more explicit 3D-memory models would help establish how broadly these benefits hold.


Q2. Why use estimated monocular depth instead of ground truth information?

We use DA3 for both the reference and generated images to reduce errors caused by obtaining their depth in different ways. Engine depth is available for the reference image, but depth for the generated image must be estimated. If we compare engine depth on the reference side with DA3 depth on the generated side, the measured difference includes not only scene-recovery errors, but also depth-estimation errors introduced on the generated side alone.

We therefore apply the same frozen DA3 model to both images. SI-AbsRel then aligns the overall scale of the two depth maps, so the comparison focuses on scene structure rather than absolute depth values. This does not eliminate depth-estimation errors, but gives both images the same treatment.

Table 4 compares engine depth with DA3-estimated depth for the same reference images, using 23 synchronized RGB-D images from 17 scenes.

Table 4. Depth differences for the same reference image.

Depth comparison for the same reference imageMedian raw AbsRelMedian scale-aligned SI-AbsRel
Engine depth vs. DA3 estimate0.37070.1675
Two independent DA3 estimates of the same RGB image—0

Scale alignment reduces the median error from 0.3707 to 0.1675, but a difference remains despite both depth maps describing the same image. Two independent DA3 estimates of the same image yield zero difference, indicating consistent estimation. These results show that mixing depth sources introduces differences that scale alignment cannot fully remove, and support using the same depth model on both sides to reduce this effect.

Ethics clarification. Our UE data were collected using UnrealZoo, with the required permissions for asset use and data release. The real-world videos contain no identifiable faces, license plates, or other personal information, and all necessary permissions and consent for recording and release have been obtained. We will release the dataset videos, camera poses, and source code under the confirmed licenses. Genie 3 and Happy Oyster outputs were obtained through their official online services, in accordance with the applicable terms of service.

Reviewer_xXTKOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces SceneMemBench, a benchmark for scene memory in interactive video world models. It is organized along two axes. The first is the memory task: Return to a Seen View versus Return to a Novel View (four revisit types in total). The second is the protocol: Continued Rollout, where every model starts from the same GT history, generates a short query segment, and is compared against held-out GT; and Full Rollout, where generation is end-to-end and Gemini compares the initial frame with the revisit frame. The data consist of 600 posed trajectories across 33 UE5 scenes, plus 60 Full Rollout cases, and 8 models are evaluated. The two main conclusions are that strong seen-view recall does not imply strong novel-view recovery, and that model performance under controlled history can differ substantially from performance under full interaction.

Critical strengths
  1. Separating pre-query error from memory is the right starting point. Continued Rollout injects the same GT context through each model's native cache, retrieval, or replay mechanism (Sec.3.3.1, Fig.2). Adapting this to six architecturally very different models is a substantial engineering effort.
  2. The reporting is restrained. The geometry metrics are explicitly called monocular proxies (p5). The authors themselves point out that the motion check only tests whether there is motion, not whether the revisit is completed (p6). Scores from the two protocols are not combined. The scene-cluster bootstrap CIs, human agreement, history intervention, and fixed-reference sensitivity analysis mentioned in the main text are all carried out in the appendix.
  3. The data are 1080p/24fps trajectories with per-frame poses and explicit context/query boundaries, which makes them more valuable for reuse and reproduction than benchmarks that provide only an initial frame. The evaluated models are also very recent (including Genie 3 and Matrix-Game 3.5).
Critical weaknesses
  1. Seen and Novel differ in more than viewpoint. Both Seen types are in-place rotations, while both Novel types include an A→B translation (Sec.3.2). Seen uses a locally matched GT reference frame, which tolerates under-rotation of the camera; Novel uses the fixed endpoint directly (p5), so camera-control error is charged to the Novel score. The main text also states that novelty is not quantified by overlap or angle (p5 L.412–413). A low Novel score could therefore come from a longer memory span, control error introduced by translation, or the reference-frame rule, and not necessarily from viewpoint novelty itself. In addition, computing from Table 2, model rankings under Seen and Novel are highly consistent (Spearman 0.886 for Continued DINOv2, 0.905 for Full Rollout Subject). F1 rests mainly on the top-1 swap between HY and MG3 (0.9290/0.8840 → 0.6987/0.7543) and on MG3.5 being an outlier in Full Rollout. In the fixed-reference analysis cited on p7, the Seen leader also changes. "Does not guarantee" is literally true, but it is overstated as a core insight. I suggest adding a translation-matched seen-view control, or reporting results stratified by viewpoint change (e.g., the ∠APB bins already in the appendix).
  2. There is no memory-free reference score. Table 2 has no recent-only or no-history condition, so readers cannot tell how much of a score comes from memory and how much from "plausible" content generated from recent frames. In Fig.4, the output labeled "different building" still gets DINOv2 0.809, almost the same as 0.798 for "layout change", which suggests that the floor for global similarity within the same scene and style may not be low. The single-case intervention in the appendix shows that the models do rely on history, and I appreciate this, but one case is not enough to calibrate a score floor for 600 cases. The adapters already exist, so adding this condition should be affordable.
  3. Continued Rollout is the paper's core diagnostic protocol, yet its metrics have no human agreement validation (the human study only covers the VLM in Full Rollout, p7). The Fig.4 example above shows that DINOv2 CLS is insensitive to building replacement. PSNR of 11–17 dB and SSIM of 0.24–0.44 are also close to the floor range for generative models, and the gaps between models are small.
  4. F2 compares two different measurement instruments: GT novel frames with pixel/monocular-depth metrics on one side, and the initial frame with Gemini scoring on the other; 600 cases versus 60 cases; and Full Rollout additionally mixes in action following. On p9 the MG3.5 discrepancy is attributed to "trajectory-following errors and accumulated generation degradation", but trajectory-following error is not measured in the main text. The phenomenon itself is plausible, but a benchmark that claims to support "diagnosis" needs to separate the causes.
Questions for authors
  1. Can you report scores under a recent-only condition, on all cases or on a stratified sample?
  2. Can you run the same VLM judge on the Continued outputs, or compute the same metric suite in Full Rollout using the GT trajectories that overlap with the controlled corpus, to separate protocol effects from metric effects? Please also report each model's camera trajectory error under both protocols.
  3. If you add a seen control that "translates and then returns to the original view", how much of the Seen/Novel gap remains? Does the Novel score decrease monotonically with ∠APB?
6 comments
AuthorsOpenReview ↗

Response to Reviewer xXTK (1/4)

We thank the reviewer for recognizing the shared-history design, model adaptation, and annotations. We address task differences, viewpoint novelty, and cross-protocol measurement using new experiments and existing appendix results.

W1 / Q3: If you add a seen control that "translates and then returns to the original view", how much of the Seen/Novel gap remains? Does the Novel score decrease monotonically with ∠APB?

Novel recovery remains more challenging in our fixed-endpoint comparisons, including when Seen trajectories also involve translation. We agree that similar model rankings across Seen and Novel, or a change in the top-ranked model, do not by themselves establish a difference in recovery difficulty. We therefore examine reference-frame selection, translation, and viewpoint angle separately.

(i) Using the same reference-frame rule for Seen and Novel. The main table compares models within each task, while Table 13 in Appendix E.1 of the manuscript compares Seen and Novel using fixed GT query endpoints. Both analyses use the same 3,600 outputs, and the generated frame is always the final query frame.

The main table uses locally selected GT references for Seen because small camera-pose errors, such as under-rotation, can lower automatic scores even when people judge the recovery favorably. It therefore focuses on comparing models within each revisit task. To compare Seen and Novel difficulty, we use the appendix’s fixed-endpoint results, which apply the same reference rule to both tasks. Under this rule, Novel scores remain unchanged, while MG3 leads Seen DINO and MG3.5 leads the other five metrics.

(ii) Adding translation to Seen trajectories. We added 80 trajectories in which the camera moves away and returns to the original view. These cover forward–backward motion, lateral round trips, and rectangular loops: four motion types, two distances, and ten trajectories per setting. Round-trip distances are 10/20 m; rectangle sides are 5/10 m. Models generate the return segment, or the final side of a rectangle, from the preceding GT history. Model weights, resolutions, and history strategies remain unchanged.

All six models were evaluated on all 80 trajectories, giving 480 outputs. Table 1 compares original rotation Seen, translated Seen, and Novel using fixed GT and generated query endpoints, averaged equally across models.

Table 1. Fixed-endpoint results. Each model has 200 rotation Seen, 80 translated Seen, and 400 Novel trajectories. Positive gains favor translated Seen.

MetricOriginal rotation SeenTranslated Seen (80 trajectories)Original NovelTranslated vs. rotation Seen gain
DINO ↑0.78440.87990.5958+0.0955
LPIPS ↓0.45160.35200.5638+0.0996
SSIM ↑0.36490.44020.3207+0.0754
PSNR ↑14.556416.977613.1647+2.4212
SI-AbsRel ↓0.36420.25220.5753+0.1120
Normal cosine ↑0.74250.78610.6564+0.0436

The Seen–Novel gap remains after adding translation: translated Seen has better six-model average scores than Novel on all six metrics. It also scores higher than the original rotation Seen set.

We additionally combine the 80 translated Seen trajectories with the original 200 rotation Seen trajectories, leaving the 400 Novel trajectories unchanged. Table 2 shows that the resulting Seen–Novel gaps increase by 14.5%–49.7%.

Table 2. Seen–Novel gaps before and after adding translated Seen.

MetricOriginal gapTranslated Seen–Novel gapPooled gapAbsolute changeRelative change
DINO0.18870.28420.2160+0.0273+14.5%
LPIPS0.11220.21180.1407+0.0285+25.4%
SSIM0.04420.11960.0657+0.0215+48.7%
PSNR1.39173.81292.0835+0.6918+49.7%
SI-AbsRel0.21110.32310.2431+0.0320+15.2%
Normal0.08610.12970.0986+0.0125+14.5%

Positive gaps favor Seen: Novel minus Seen for LPIPS/SI-AbsRel, and Seen minus Novel for the other metrics. Relative change is the increase in the gap divided by the original gap.

AuthorsOpenReview ↗

Response to Reviewer xXTK (2/4)

W1 / Q3 (continued)

(iii) Larger viewpoint changes are generally harder. We grouped the 200 New-Viewpoint Revisit trajectories by ∠APB, where A and B are the initial and revisit camera positions and P is the target. The 45°, 90°, and 135° groups contain 48, 87, and 65 trajectories, respectively.

Across six models, increasing the angle from 45° to 135° worsens DINO for five models, LPIPS for all six, and SI-AbsRel and normal cosine for five. Table 3 reports the DINO and SI-AbsRel results.

Table 3. Recovery results by viewpoint angle. Higher DINO and lower SI-AbsRel are better.

ModelDINO 45°DINO 90°DINO 135°SI 45°SI 90°SI 135°ΔDINO (135°−45°)ΔSI (135°−45°)
HY0.73250.71930.71790.47230.45550.5002-0.0146+0.0279
LingBot0.59620.58400.55180.52680.68760.7493-0.0444+0.2225
MG30.78560.78900.73150.48100.44870.5761-0.0541+0.0951
MG3.50.62100.64210.58640.38980.37820.4592-0.0346+0.0694
SANA0.32710.30400.34280.77720.84610.6461+0.0157-0.1311
Zing0.68550.67700.66120.53850.58770.5907-0.0243+0.0522

The trend is not strictly monotonic: some models improve at 90°, and SANA improves from 45° to 135° on DINO and SI-AbsRel. Thus, larger viewpoint changes generally reduce recovery quality, but not for every model or every increase in angle.


W2 / Q1. Can you report scores under a recent-only condition, on all cases or on a stratified sample?

We added Recent-only and No-history conditions to show how much recovery is possible without the full history. On 48 stratified trajectories, all six models achieve higher DINO scores with Full history than with No-history.

(i) Comparing three history conditions. The experiment covers 12 scenes and all four trajectory types, producing 864 outputs across six models and three conditions:

  • Full: the complete GT history.
  • Recent-only: the final two seconds of GT history.
  • No-history: only the final context frame used to initialize the query, with a fresh process and no earlier KV cache or external memory.

For each model, we keep the weights, resolution, seed, and query commands unchanged. All three conditions are evaluated against the same GT query endpoint using the six metrics.

(ii) Measuring the contribution of history. Table 4 compares the three conditions on the same trajectories. Full history improves DINO over No-history by 0.079–0.273 across the six models. The scene-cluster paired-bootstrap confidence interval for each model’s DINO gain lies entirely above zero. Across all six models and six metrics, Full history gives better mean scores than No-history in 35 of 36 comparisons, and better scores than Recent-only in 33 of 36.

Table 4. History ablation on 48 stratified trajectories. Each cell reports Full / Recent-only / No-history.

ModelDINO ↑LPIPS ↓SSIM ↑PSNR ↑SI-AbsRel ↓Normal ↑
HY0.764 / 0.606 / 0.5870.537 / 0.576 / 0.5910.327 / 0.308 / 0.29913.40 / 12.51 / 12.150.385 / 0.462 / 0.5180.680 / 0.647 / 0.629
LingBot0.609 / 0.483 / 0.3760.544 / 0.591 / 0.6420.309 / 0.284 / 0.25913.13 / 12.22 / 11.360.456 / 0.497 / 0.6320.685 / 0.664 / 0.609
MG30.834 / 0.557 / 0.5610.458 / 0.545 / 0.5540.336 / 0.331 / 0.33414.33 / 13.32 / 13.350.359 / 0.480 / 0.5270.738 / 0.698 / 0.691
MG3.50.676 / 0.488 / 0.4500.390 / 0.575 / 0.6180.412 / 0.317 / 0.29816.19 / 12.67 / 12.140.279 / 0.487 / 0.5220.786 / 0.673 / 0.673
SANA0.399 / 0.404 / 0.3200.566 / 0.596 / 0.6170.282 / 0.246 / 0.22313.08 / 12.11 / 11.150.539 / 0.536 / 0.5310.640 / 0.643 / 0.597
Zing0.704 / 0.534 / 0.5990.540 / 0.577 / 0.6160.318 / 0.303 / 0.30213.29 / 12.70 / 11.980.505 / 0.603 / 0.6270.682 / 0.656 / 0.618

The Recent-only and No-history scores provide the requested reference for what models can recover from recent observations or an initial image. The improvements with Full history show that access to earlier observations contributes to recovery beyond what these limited inputs provide.

(iii) Interpreting the DINO score in Fig. 4. The DINO score of 0.809 in Fig. 4 does not imply faithful recovery of the replaced building. Global similarity can remain high despite a local content error. We therefore interpret DINO together with appearance and geometry metrics. The matched history ablation provides a complementary check: with the model, query, and reference held fixed, Full history improves DINO over No-history for all six models.

AuthorsOpenReview ↗

Response to Reviewer xXTK (3/4)

W3. Human validation of Continued Rollout metrics

Automatic metrics agree with human preferences on 75.0%–87.5% of the 120 sampled pairs. Our original human study evaluated the Gemini judge used in Full Rollout. We extended this validation to Continued Rollout by randomly sampling 120 pairs of model outputs.

For each pair, human evaluation compared both candidate generated revisit frames against the fixed GT query endpoint, judging subject, appearance, and geometry and allowing left, right, or tie. Automatic preferences use DINO↑ for subject, LPIPS↓ for appearance, and SI-AbsRel↓ for geometry.

Table 5 reports agreement between automatic metrics and human preferences on all 120 pairs.

Table 5. Agreement between automatic metrics and human preferences in Continued Rollout.

ComparisonSubject (DINO ↑)Appearance (LPIPS ↓)Geometry (SI-AbsRel ↓)
Metrics–human87.5% (105/120)81.7% (98/120)75.0% (90/120)

These results support alignment with human judgments across all three dimensions, with the highest agreement for subject consistency.

PSNR and SSIM also capture recovery improvements from additional history despite their low absolute values. In the matched history ablation (Table 4), MG3.5 achieves PSNR/SSIM of 16.19/0.412 with Full history, 12.67/0.317 with Recent-only, and 12.14/0.298 with No-history. Full history improves PSNR by 4.05 dB and SSIM by 0.114 over No-history. With the model, query, and reference held fixed, these results show that both metrics remain sensitive to the contribution of historical information.


W4 / Q2. Can you run the same VLM judge on the Continued outputs, or compute the same metric suite in Full Rollout using the GT trajectories that overlap with the controlled corpus, to separate protocol effects from metric effects? Please also report each model's camera trajectory error under both protocols.

Model rankings still differ between Continued and Full Rollout when both are evaluated with Gemini-3.8-flash. We compare the protocols using the same judge, then separately examine camera execution and revisit completion.

(i) Comparing both protocols with Gemini. We scored 240 Continued Rollout outputs from 40 fixed UE trajectories shared across six models. The set contains 10 trajectories of each type, giving each model 20 Seen and 20 Novel outputs. We compare these with the existing Full Rollout Gemini scores for the same six models, with 30 Seen and 30 Novel outputs per model.

For Continued Rollout, Gemini compares the final generated frame with the fixed GT query endpoint. For Full Rollout, it compares the initial reference frame with the previously selected revisit frame. Scores are averaged within each trajectory subtype, then combined with equal subtype weights. Overall scores weight Seen and Novel equally.

Table 6 compares the six models’ rankings between Continued and Full Rollout using the same Gemini judge. Spearman’s ρ is closer to 1 when rankings are more similar.

Table 6. Cross-protocol model-ranking correlations under Gemini.

Task groupSame Gemini: Continued vs FullSubject ρAppearance ρGeometry ρ
SeenModel-ranking correlation0.4860.5800.657
NovelModel-ranking correlation0.4290.5430.486
OverallModel-ranking correlation0.6000.6000.600

For Novel, the correlations range from 0.429 to 0.543. Table 7 shows the corresponding rankings for each model. Each cell gives Continued Gemini / Full Gemini ranks among the six models; rank 1 is best.

Table 7. Novel model rankings under Gemini in the two protocols.

ModelSubjectAppearanceGeometry
HY2 / 12 / 12 / 2
LingBot5 / 35 / 35 / 3
MG31 / 21 / 21 / 1
MG3.54 / 54 / 44 / 5
SANA6 / 46 / 56 / 4
Zing3 / 63 / 63 / 6

Zing, for example, ranks third in Continued Rollout and sixth in Full Rollout across all three dimensions. These results show that the cross-protocol differences are not solely due to different evaluators, supporting separate reporting of performance under controlled histories and full interaction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.