- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
SceneMemBench evaluates whether interactive video world models can recover a scene after it leaves the current view. It separates seen-view recall from novel-view recovery and introduces two protocols: Continued Rollout from a shared ground-truth history and Full Rollout from complete model-generated interactions. The benchmark contains 600 controlled trajectories across 33 Unreal Engine scenes and 60 Full Rollout cases. Eight models are compared using subject, appearance, and geometric consistency measures, with confidence intervals and a human study of the VLM-based evaluator.
Critical strengths
-
The separation of seen-view recall, novel-view recovery, controlled continuation, and full interaction is useful. In particular, Continued Rollout removes accumulated generation and trajectory errors, so it can expose memory behavior that is hidden in end-to-end evaluation.
-
The evaluation is careful for a benchmark paper. It covers several open and closed models, reports paired uncertainty for the controlled protocol, documents model-specific history initialization, and checks the reliability of the VLM evaluator against human judgments.
Critical weaknesses
-
The controlled benchmark is restricted to 33 rendered Unreal Engine scenes. This is suitable for obtaining exact camera and geometry references, but it leaves uncertainty about whether rankings transfer to real captures, dynamic scenes, or less game-like visual statistics.
-
The Full Rollout protocol is less controlled than the Continued Rollout protocol. It contains only 60 cases, relies on pairwise VLM scoring without a ground-truth novel-view image, and its motion screen checks for recognizable movement rather than successful execution of the required revisit. A model may therefore be scored after reaching the wrong endpoint.
Questions for authors
-
How sensitive are Full Rollout rankings to filtering for actual trajectory completion rather than only recognizable motion? Can the authors manually report the endpoint-success rate for each model?
-
Do the authors have evidence that the principal rankings from the Unreal Engine scenes transfer to a small set of real captured scenes with known camera poses?
-
For novel-view Full Rollout, how stable are the VLM scores across judges or repeated judge calls, especially in categories where human agreement is lower?
6 comments
Response to Reviewer PxK9 (2/4)
W2 / Q1. How sensitive are Full Rollout rankings to filtering for actual trajectory completion rather than only recognizable motion? Can the authors manually report the endpoint-success rate for each model?
We manually reviewed all 480 scored videos. Overall rankings change little after excluding clear failures.
(i) Checking revisit completion. Closed-source models are controlled manually; open-source models use predefined trajectories, calibrated action scales, and additional turning margins where needed. Once a stable revisit is reached, the video can be truncated at that point to prevent subsequent motion from affecting evaluation. Endpoint selection thus already aims to avoid scoring an incorrect endpoint caused by incomplete execution.
The manual review counts action drift, stalled turning, and visual collapse as clear revisit or generation failures. Table 2 reports completion and failure counts for each model, with 60 outputs per model.
Table 2. Revisit completion and clear failures in Full Rollout.
| Model | Revisit completion supported | Revisit or generation failure | Failure rate |
|---|---|---|---|
| Genie3 | 60 | 0 | 0.0% |
| Happy Oyster | 51 | 9 | 15.0% |
| HY | 42 | 18 | 30.0% |
| LingBot | 50 | 10 | 16.7% |
| MG3 | 60 | 0 | 0.0% |
| MG3.5 | 58 | 2 | 3.3% |
| SANA | 58 | 2 | 3.3% |
| Zing | 42 | 18 | 30.0% |
| Total | 421 | 59 | 12.3% |
Overall, 59 of 480 outputs (12.3%) show clear revisit or generation failures, with rates ranging from 0% to 30% across models. These counts do not separately measure gradual degradation in rollouts that still complete the revisit.
(ii) Testing ranking sensitivity. To test whether these failures affect model comparisons, we excluded the 59 clear failures and recomputed rankings using the original Gemini-3.8-flash scores. Overall scores assign equal weight to Seen and Novel. Table 3 compares model rankings before and after excluding these failures.
Table 3. Ranking correlations before and after excluding clear failures.
| Spearman correlation before vs. after filtering | Subject ρ | Appearance ρ | Geometry ρ |
|---|---|---|---|
| Seen | 0.976 | 1.000 | 0.929 |
| Novel | 0.929 | 0.929 | 0.976 |
| Overall | 0.976 | 1.000 | 1.000 |
The small changes show that overall rankings are stable to excluding clear failures. We retain these failures because Full Rollout evaluates the entire interaction, including whether the model completes the revisit and preserves the scene along the way. Its scores therefore reflect memory, camera control, and generation quality together. Continued Rollout provides a complementary test: every model starts from the same GT history, so errors accumulated before the query do not affect the evaluation. This allows us to compare scene recovery from a common starting point.
Response to Reviewer PxK9 (3/4)
Q3. For novel-view Full Rollout, how stable are the VLM scores across judges or repeated judge calls, especially in categories where human agreement is lower?
For novel-view Full Rollout, different VLM judges give similar model rankings, while score stability across repeated calls varies by judge and evaluation dimension. We report subject, appearance, and geometry separately, with particular attention to categories where human agreement is lower. Below, we compare model rankings across judges, examine their agreement on individual outputs in these categories, and test whether repeated calls produce similar scores.
(i) Do different judges rank the models similarly? We evaluated all 480 Full Rollout outputs with Gemini-3.8-flash, Qwen3-VL-8B-Instruct, and GPT-6.1-sol, using the same scoring images and the original scoring rules.
For each judge, we rank the eight models by their average scores. Seen and Novel each give equal weight to their two trajectory types; overall scores give equal weight to all four types. Table 4 compares the eight models’ rankings across the three judges using Spearman’s ρ, where values closer to 1 mean more similar rankings.
Table 4. Model-ranking agreement across VLM judges.
| Group | Judge pair | Subject ρ | Appearance ρ | Geometry ρ |
|---|---|---|---|---|
| Novel, 240 outputs | Gemini—Qwen | 0.976 | 0.905 | 0.976 |
| Novel, 240 outputs | Gemini—GPT | 0.976 | 0.929 | 0.976 |
| Novel, 240 outputs | Qwen—GPT | 1.000 | 0.976 | 1.000 |
| Seen, 240 outputs | Gemini—Qwen | 1.000 | 0.952 | 0.952 |
| Seen, 240 outputs | Gemini—GPT | 0.976 | 1.000 | 0.976 |
| Seen, 240 outputs | Qwen—GPT | 0.976 | 0.952 | 0.976 |
| Overall, 480 outputs | Gemini—Qwen | 1.000 | 1.000 | 1.000 |
| Overall, 480 outputs | Gemini—GPT | 1.000 | 1.000 | 1.000 |
| Overall, 480 outputs | Qwen—GPT | 1.000 | 1.000 | 1.000 |
The three judges produce similar rankings for Seen and Novel, and identical overall rankings in all three dimensions. Ranking correlations are 0.905–1.000 for Novel and 0.952–1.000 for Seen.
(ii) What happens in the categories where human agreement is lower? In our original human study, agreement with Gemini was lower for Novel appearance (70.0%) and geometry (68.8%) than for subject consistency (80.0%). The lowest agreement was in Forward and Look Back, at 58.3% for appearance and 62.3% for geometry (Table 10 in Appendix D.2 of the manuscript).
We therefore compare the two Novel types separately: Forward and Look Back and New Viewpoint. Each contains 15 trajectories × 8 models = 120 outputs. We use the existing scores from all three judges, without excluding outputs or making additional API calls.
Table 5 compares model rankings across judges within each Novel type. We average each model’s scores over its 15 trajectories, rank the eight models, and compare the rankings between judges.
Table 5. Model-ranking agreement within each Novel subtype.
| Novel subtype | Judge pair | Subject ρ | Appearance ρ | Geometry ρ |
|---|---|---|---|---|
| Forward and Look Back | Gemini—Qwen | 0.952 | 0.976 | 0.850 |
| Gemini—GPT | 0.994 | 0.976 | 0.934 | |
| Qwen—GPT | 0.934 | 0.976 | 0.929 | |
| New Viewpoint | Gemini—Qwen | 0.952 | 0.952 | 0.857 |
| Gemini—GPT | 0.976 | 0.929 | 0.952 | |
| Qwen—GPT | 0.976 | 0.952 | 0.952 |
Model rankings remain similar in Forward and Look Back: appearance correlations are 0.976 for all three judge pairs, and geometry correlations range from 0.850 to 0.934.
Similar model rankings, however, do not mean that judges agree equally well on every output. Table 6 compares the judges’ scores across the 120 individual outputs in each Novel type using Spearman’s ρ.
Table 6. Per-output score correlations within each Novel subtype.
| Novel subtype | Judge pair | Subject ρ | Appearance ρ | Geometry ρ |
|---|---|---|---|---|
| Forward and Look Back | Gemini—Qwen | 0.705 | 0.520 | 0.615 |
| Gemini—GPT | 0.872 | 0.755 | 0.718 | |
| Qwen—GPT | 0.723 | 0.714 | 0.771 | |
| New Viewpoint | Gemini—Qwen | 0.781 | 0.754 | 0.775 |
| Gemini—GPT | 0.914 | 0.904 | 0.879 | |
| Qwen—GPT | 0.803 | 0.778 | 0.809 |
For Forward and Look Back, appearance and geometry correlations are 0.520–0.755 and 0.615–0.771, respectively. For every judge pair, these are lower than in New Viewpoint, where the corresponding ranges are 0.754–0.904 and 0.775–0.879.
The category with lower human agreement also shows more disagreement between VLM judges on individual outputs. Nevertheless, averaging over trajectories gives similar model rankings. Appearance illustrates this clearly: all three model-ranking correlations are 0.976, while individual-output correlations range from 0.520 to 0.755.
Response to Reviewer PxK9 (1/4)
We thank the reviewer for recognizing the two evaluation protocols and experimental design. We have added experiments on revisit completion, real captures, and VLM reliability, which we discuss below.
W1 / Q2. Do the authors have evidence that the principal rankings from the Unreal Engine scenes transfer to a small set of real captured scenes with known camera poses?
Full Rollout rankings agree strongly across UE and real captures; Continued Rollout agreement is weaker, especially for Novel. We recorded 25 handheld videos across indoor and outdoor locations (14 Seen, 11 Novel), with ARKit-estimated camera trajectories. Six models produced 300 outputs under Full Rollout (complete generation from the initial view) and Continued Rollout (revisit generation from recorded GT history). These captures test ranking transfer beyond game-rendered visual content, and the split will be released with the benchmark.
Table 1 compares the six models’ rankings on real captures and the original UE benchmark using Spearman’s ρ; values closer to 1 indicate more similar rankings. Full Rollout uses the same VLM judge, Gemini-3.8-flash, on both datasets. Continued Rollout uses fixed GT and generated query endpoints. We min–max normalize each metric across the six models, then average equally within each dimension: DINO for subject, LPIPS/SSIM/PSNR for appearance, and SI-AbsRel/normal cosine for geometry. Overall scores weight Seen and Novel equally.
Table 1. Ranking agreement between real captures and the UE benchmark.
Full Rollout achieves overall ρ of 0.829–0.943: models ranked higher on UE generally also rank higher on real captures. Continued Rollout agreement is weaker, especially for Novel (ρ = 0.029–0.257), so its UE rankings do not directly predict rankings on this real-capture set.
Unlike UE’s exact poses, real captures rely on ARKit estimates, making pose and FOV alignment less certain. Automatic metrics are sensitive to these differences; history length and revisit visibility also differ across datasets. These factors may contribute to weaker agreement. Qualitative examples nevertheless show successful recovery and similar failures in both domains, including target-view deviations, missing content, and scene rewriting (revised manuscript, Appendix F.4, Figures 12–13).
Static scenes give memory recovery an unchanged target. Dynamic scenes require defining how the target may reasonably change while out of view; we leave this extension to future work. Broader real-world coverage also warrants further study.