- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ORBIA, a benchmark for assessing whether interactive video generators maintain a consistent 3D world during exploration and revisit. It separates evaluation into input preservation, persistence of newly generated content, and 3D self-consistency, using registered RGB-D reference views, annotated anchors, revisit trajectories, and longer rollouts. The paper evaluates 27 camera-, action-, and text-conditioned models over 720 short and long cases.
Critical strengths
-
ORBIA distinguishes input preservation, generated-content persistence, and 3D self-consistency, providing a useful framework for diagnosing different failure modes. Previous works usually does not check the consistency for generated contents.
-
Registered RGB-D references, cross-view object anchors, and shared-surface annotations support more targeted evaluation than whole-image similarity alone.
Critical weaknesses
-
The 3D consistency is critical for world model. However, there are too many benchmarks evaluating the consistency. The improvement over previous benchmarks is marginal.
-
The paper demonstrates complementarity among its own metrics but lacks direct comparisons showing that ORBIA reliably detects failures missed by prior benchmarks.
Questions for authors
-
How do you distinguish genuine 3D inconsistency from errors introduced by DA3 and imperfect viewpoint execution? Have you validated the metrics on geometrically consistent reference videos and controlled scene corruptions, and do the main conclusions remain stable under an alternative estimator or evaluation restricted to pose-matched observations?
-
What specific failure modes does ORBIA reliably detect that existing benchmarks miss? Could you compare ORBIA with the most relevant prior metrics on the same videos and validate the additional detections using independent human judgments or controlled ground-truth errors?
8 comments
Response to Reviewer XMN4 (2/3)
What the table shows (W1 / W2, continued).
- Input Preservation. Removing the input anchor lowers ORBIA's Input Preservation by 12.0 points, since ORBIA checks the anchor against the registered input reference. WorldScore compares only nearby frames, and WBench's geometric and photometric consistency compare the video with its own reconstruction, so these barely move (at most −4.9). WBench checks the input view only through its spatial consistency, a whole-image similarity between the first and returning frames on loop trajectories, which drops by only 4.3.
- Generated Persistence. Removing or rescaling the revisited object lowers Generated Persistence by 15–17 points, since ORBIA compares registered first visits and revisits directly. WBench averages agreement over the whole video, which dilutes a single object, and WorldScore's nearby frames stay coherent, so WBench's geometric consistency drops by at most 1.4 and WorldScore's 3D consistency by at most 5.9.
- 3D Self-Consistency. Drift and jitter break 3D self-consistency, which every metric tests, and the metrics agree. WBench's geometric and photometric consistency move less under 10 cm jitter (−4.9 and −9.4), likely because a reconstruction built from the same frames absorbs part of the deformation, while ORBIA tests its Gaussians on held-out frames.
- Appearance. JPEG lowers WorldScore's photometric consistency, an optical-flow check between nearby frames, by 60.9, about as much as jitter, so it mistakes image quality for inconsistency. ORBIA's consistency scores barely change.
Real world models. Video by video on 12 world models and 77 cases, the external metrics partly track ORBIA's Generated Persistence (Spearman 0.37–0.69), since videos that collapse fail every metric. On Input Preservation, WBench agrees only moderately (0.48–0.49) and WorldScore's 3D consistency is nearly unrelated (0.16): 45% of the videos in ORBIA's bottom quarter are rated above its median, close to the 50% expected if the two were unrelated. Neither benchmark checks individual input objects. Four examples, which we will add with frame visualizations to the main text (percentiles among all videos, higher means more consistent):
| Video (trajectory) | ORBIA | WorldScore 3D | WBench geometric | What happens |
|---|---|---|---|---|
| Lyra2, night plaza (arc and return) | Gen. Persistence, 8th pct | 96th pct | 47th pct | the tiger statue seen at the first visit is gone at the revisit, while the surrounding buildings are unchanged |
| LingBot-World v1, indoor corner (peek and retract) | Gen. Persistence, 22nd pct | 63rd pct | 85th pct | a bare wall corner with a stool returns as a table covered with objects |
| Echo-WM, street plaza (orbit around a kiosk) | Input Preservation, 3rd pct | 82nd pct | 54th pct | back at the input view, the advertising kiosk from the input image is gone and the street behind it is replaced |
| AlayaWorld, temple garden (peek and retract) | Input Preservation, 15th pct | 90th pct | 18th pct | the temple roof and lotus pond in the input image are replaced by a large tree and bare ground |
The model rankings diverge in the same way. Gen3C, which keeps an explicit 3D cache of the input scene, is 1st of the 12 models on ORBIA's Input Preservation but 5th on WorldScore 3D, and Lyra2 is 4th on Generated Persistence but 10th on WBench geometric. Where the metrics overlap, they agree: the external 3D and geometric metrics correlate with ORBIA's 3D Self-Consistency across the 12 models (τ = 0.73–0.79), and ORBIA adds the input and revisit failures they cannot see.
Response to Reviewer XMN4 (3/3)
Q1: "How do you distinguish genuine 3D inconsistency from errors introduced by DA3 and imperfect viewpoint execution? ... reference videos and controlled scene corruptions ... an alternative estimator or ... pose-matched observations?"
A: We run all four analyses.
Geometrically consistent reference videos (errors introduced by DA3). On the renders from W1 / W2, DA3 recovers cameras with a median error of 0.08% of path length and depth with AbsRel 0.018. With ground-truth cameras and depth, Input Preservation and Generated Persistence change by less than 0.5 points (81.1 → 81.2, 95.2 → 95.7). DA3 lowers 3D Self-Consistency by about 5 points (78.1 versus 83.2). DA3's errors are therefore negligible for the preservation scores, and the 3D offset is shared across models, as the estimator comparison below shows.
Controlled scene corruptions. On the same renders, we inject each violation at three strengths (targeted score in bold):
| Violation | Control | Visual Quality | Input Preservation | Generated Persistence | 3D Self-Consistency |
|---|---|---|---|---|---|
| none (clean render) | 91.5 | 64.4 | 81.1 | 95.2 | 78.1 |
| scene drift 10 / 25 / 50 cm | 89.9 / 86.3 / 81.2 | 64.3 / 63.9 / 63.2 | 80.1 / 78.5 / 72.0 | 92.0 / 88.6 / 82.0 | 76.9 / 66.4 / 58.2 |
| geometric jitter 2 / 5 / 10 cm | 84.5 / 81.8 / 75.0 | 63.9 / 62.9 / 61.4 | 78.7 / 73.7 / 66.0 | 86.9 / 78.6 / 69.5 | 73.0 / 64.1 / 52.0 |
| revisited object ×1.10 / 1.25 / 1.50 | 90.0 / 89.6 / 89.6 | 64.3 / 64.2 / 64.0 | 80.8 / 80.6 / 80.3 | 90.3 / 85.4 / 80.2 | 77.2 / 76.3 / 75.4 |
| revisited object removed | 91.5 | 63.7 | 79.6 | 78.3 | 75.4 |
| input anchor ×1.10 / 1.25 / 1.50 | 91.1 / 91.7 / 91.3 | 64.3 / 64.2 / 63.8 | 79.7 / 78.2 / 76.0 | 95.0 / 95.1 / 95.0 | 78.0 / 77.9 / 77.3 |
| input anchor removed | 91.4 | 63.5 | 69.1 | 95.1 | 78.0 |
| JPEG q = 40 / 20 / 10 | 90.8 / 91.1 / 88.2 | 63.6 / 62.1 / 60.1 | 81.0 / 80.1 / 78.1 | 95.2 / 95.3 / 95.3 | 78.1 / 78.4 / 77.0 |
| blur σ = 1 / 2 / 4 | 91.1 / 90.8 / 90.5 | 62.3 / 58.6 / 54.2 | 81.3 / 80.9 / 79.5 | 95.2 / 95.3 / 95.3 | 77.4 / 77.0 / 77.4 |
Each targeted score drops monotonically with the violation's strength, while JPEG and blur leave the consistency scores nearly unchanged. ORBIA's scores therefore separate genuine 3D inconsistency from appearance changes.
Evaluation restricted to pose-matched observations (imperfect viewpoint execution). Re-scoring Input Preservation and Generated Persistence only on events whose realized camera reaches the requested one keeps the 12 world-model rankings (τ = 0.97 and 0.88 at 20° / 0.4, 0.76 and 0.85 at 10° / 0.2, with translation relative to path length). Among videos whose realized rotation is within 0.8–1.25 of the requested one, the rankings of the 11 world models with enough such videos also hold (τ = 0.82, 0.85, and 0.75 for Input Preservation, Generated Persistence, and 3D Self-Consistency). Imperfect viewpoint execution therefore explains little of ORBIA's rankings.
Alternative estimator. We rescore 12 world models spanning the leaderboard (ranks 1–16) on 77 short cases with VGGT-Ω, a feed-forward model like DA3, and with ViPE, an optimization-based pipeline whose cameras are more accurate than DA3's on the renders (0.06% of path length) but whose depth is less accurate (AbsRel 0.078). We also replace DA3's Gaussian head with WorldMirror 2.0. Kendall τ against DA3:
| Tier | VGGT-Ω | ViPE | WorldMirror GS |
|---|---|---|---|
| Control | 0.97 | 0.79 | – |
| Input Preservation | 1.00 | 0.94 | – |
| Generated Persistence | 1.00 | 0.88 | – |
| 3D Self-Consistency | 1.00 | 0.67 | 0.94 |
VGGT-Ω shifts every model's score by a similar amount (mean absolute difference 0.4–2.1 points, Pearson r ≥ 0.99), so the offset does not depend on the model. ViPE reproduces the Input Preservation and Generated Persistence rankings (τ = 0.94 and 0.88). It agrees less on 3D Self-Consistency (0.67), which depends most on depth, where ViPE is four times less accurate than DA3, and on Control (0.79). The preservation rankings therefore hold under both estimators, and the 3D Self-Consistency ranking holds under VGGT-Ω and WorldMirror 2.0.
Together, these analyses separate genuine 3D inconsistency from DA3's errors and from imperfect viewpoint execution, and the main conclusions remain stable.
We thank Reviewer XMN4 again for the valuable comments.
Response to Reviewer XMN4 (1/3)
We thank Reviewer XMN4 for the valuable comments and for noting that ORBIA "provides a useful framework for diagnosing different failure modes" and that "previous works usually does not check the consistency for generated contents".
A: Consistency is indeed an active topic, and Table 1 of the paper compares ORBIA with 12 such benchmarks. ORBIA contributes three tests that these benchmarks do not provide together:
Separating these tests also yields findings that a single consistency score cannot show. Input preservation and generated persistence agree at the model level (ρ = 0.89) but only weakly across scenes within a model (0.35), so a model that keeps the input can still lose what it generates. Control and 3D self-consistency are nearly uncorrelated: Lyra2 ranks first in control but seventh in 3D self-consistency.
To compare directly, we run the original metric code of WorldScore and WBench, whose metrics apply to arbitrary videos, on exactly the videos ORBIA evaluates. The closest memory benchmarks cannot be run on ORBIA's videos: MIND and MemoBench score videos against their own ground truth and annotations, WRBench scores the off-screen events defined by its own prompts, and WorldRoamBench has no public evaluation code. Conversely, their data lack the registered input references, anchors, and revisit annotations that ORBIA's tests require.
Controlled ground-truth errors. On renders of 55 ScanNet++ and 25 UE scenes, change of each metric relative to the clean render (×100 scale):