acceptodds
Under review as a conference paper at ICLR 2027

Orbia: Do Generated Videos Form a Consistent 3D World?

Abstract

Video generators serve as interactive worlds for exploration and simulation, yet realistic and visually appealing frames do not necessarily depict a consistent 3D world. Existing benchmarks assess selected geometric properties or use vision-language models to judge memory and consistency, leaving systematic 3D evaluation underexplored. We introduce Orbia, the first benchmark dedicated to a systematic and comprehensive evaluation of 3D consistency in interactive world models. Through controlled exploration and revisit, we organize the evaluation into three dimensions: input preservation, generated-content persistence, and 3D self-consistency. For input preservation, registered RGB-D references and annotated anchors support appearance, geometry, and spatial stability measurements. For generated-content persistence, we compare revisits to regions absent from the input image with their first generated observations to test whether models preserve what they create. For 3D self-consistency, cross-view depth and feature correspondence, regional scale agreement, and held-out Gaussian rendering test whether generated views describe the same 3D scene. Minute-long exploration and revisits further test long-term consistency. Our construction and annotation pipeline brings together public datasets, self-captured scenes, and simulation, yielding 720 instances across indoor and outdoor environments and six exploration-and-revisit protocols. Evaluation spans 27 text-, action-, and camera-conditioned models through native-interface adapters. Orbia provides a systematic basis for diagnosing 3D inconsistency and developing more persistent video world models.

Primary area:
datasets and benchmarks
Keywords:
Video Generation, Interactive World Model, 3D Vision, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_XMN4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ORBIA, a benchmark for assessing whether interactive video generators maintain a consistent 3D world during exploration and revisit. It separates evaluation into input preservation, persistence of newly generated content, and 3D self-consistency, using registered RGB-D reference views, annotated anchors, revisit trajectories, and longer rollouts. The paper evaluates 27 camera-, action-, and text-conditioned models over 720 short and long cases.

Critical strengths
  1. ORBIA distinguishes input preservation, generated-content persistence, and 3D self-consistency, providing a useful framework for diagnosing different failure modes. Previous works usually does not check the consistency for generated contents.

  2. Registered RGB-D references, cross-view object anchors, and shared-surface annotations support more targeted evaluation than whole-image similarity alone.

Critical weaknesses
  1. The 3D consistency is critical for world model. However, there are too many benchmarks evaluating the consistency. The improvement over previous benchmarks is marginal.

  2. The paper demonstrates complementarity among its own metrics but lacks direct comparisons showing that ORBIA reliably detects failures missed by prior benchmarks.

Questions for authors
  1. How do you distinguish genuine 3D inconsistency from errors introduced by DA3 and imperfect viewpoint execution? Have you validated the metrics on geometrically consistent reference videos and controlled scene corruptions, and do the main conclusions remain stable under an alternative estimator or evaluation restricted to pose-matched observations?

  2. What specific failure modes does ORBIA reliably detect that existing benchmarks miss? Could you compare ORBIA with the most relevant prior metrics on the same videos and validate the additional detections using independent human judgments or controlled ground-truth errors?

8 comments
AuthorsOpenReview ↗

Response to Reviewer XMN4 (1/3)

We thank Reviewer XMN4 for the valuable comments and for noting that ORBIA "provides a useful framework for diagnosing different failure modes" and that "previous works usually does not check the consistency for generated contents".

W1: "... there are too many benchmarks evaluating the consistency. The improvement over previous benchmarks is marginal."

W2: "... lacks direct comparisons showing that ORBIA reliably detects failures missed by prior benchmarks."

Q2: "What specific failure modes does ORBIA reliably detect that existing benchmarks miss? ... on the same videos ... controlled ground-truth errors?"

A: Consistency is indeed an active topic, and Table 1 of the paper compares ORBIA with 12 such benchmarks. ORBIA contributes three tests that these benchmarks do not provide together:

  • Generated content is tested against the model's own first generation. As the reviewer notes, previous works usually do not check consistency for generated content. MIND and MemoBench score revisits against ground-truth frames, and WRBench asks whether an off-screen event has progressed when the camera returns, judged by vision-language models. ORBIA's trajectories first visit a region absent from the input image, later return to it, and compare the registered revisit with the first generation in appearance, depth, and surface geometry.
  • Input objects are tested against registered references. Registered RGB-D references and object anchors test whether specific input objects survive. Anchors cover a median of 5.2% of the input image, so whole-image and frame-to-frame metrics dilute their loss.
  • Scene-level reconstruction complements local agreement. Among them, only WorldOlympiad tests global consistency, and it does not test input preservation or revisits. ORBIA predicts Gaussians from context views and evaluates them at held-out views, so it tests whether one 3D scene explains the whole video.

Separating these tests also yields findings that a single consistency score cannot show. Input preservation and generated persistence agree at the model level (ρ = 0.89) but only weakly across scenes within a model (0.35), so a model that keeps the input can still lose what it generates. Control and 3D self-consistency are nearly uncorrelated: Lyra2 ranks first in control but seventh in 3D self-consistency.

To compare directly, we run the original metric code of WorldScore and WBench, whose metrics apply to arbitrary videos, on exactly the videos ORBIA evaluates. The closest memory benchmarks cannot be run on ORBIA's videos: MIND and MemoBench score videos against their own ground truth and annotations, WRBench scores the off-screen events defined by its own prompts, and WorldRoamBench has no public evaluation code. Conversely, their data lack the registered input references, anchors, and revisit annotations that ORBIA's tests require.

Controlled ground-truth errors. On renders of 55 ScanNet++ and 25 UE scenes, change of each metric relative to the clean render (×100 scale):

ViolationORBIA Input Pres.ORBIA Gen. Pers.ORBIA 3D Self-C.WBench geometricWBench photometricWBench spatialWorldScore 3DWorldScore photometric
input anchor removed−12.0−0.0−0.2−0.0−1.0−4.3−3.1−4.9
input anchor ×1.5−5.1−0.1−0.8−0.3−0.9−2.4−0.7−1.2
revisited object removed−1.5−16.8−2.7−0.0−5.9−3.0−1.7−9.4
revisited object ×1.5−0.8−15.0−2.6−1.4−2.4−2.0−5.9−6.8
scene drift 50 cm−9.1−13.2−19.9−2.0−3.3−18.3−21.1−0.6
random geometric jitter 10 cm−15.0−25.7−26.1−4.9−9.4−21.6−73.0−62.9
JPEG q = 10−3.0+0.1−1.1−1.7−0.4+0.0−6.7−60.9
AuthorsOpenReview ↗

Response to Reviewer XMN4 (2/3)

What the table shows (W1 / W2, continued).

  • Input Preservation. Removing the input anchor lowers ORBIA's Input Preservation by 12.0 points, since ORBIA checks the anchor against the registered input reference. WorldScore compares only nearby frames, and WBench's geometric and photometric consistency compare the video with its own reconstruction, so these barely move (at most −4.9). WBench checks the input view only through its spatial consistency, a whole-image similarity between the first and returning frames on loop trajectories, which drops by only 4.3.
  • Generated Persistence. Removing or rescaling the revisited object lowers Generated Persistence by 15–17 points, since ORBIA compares registered first visits and revisits directly. WBench averages agreement over the whole video, which dilutes a single object, and WorldScore's nearby frames stay coherent, so WBench's geometric consistency drops by at most 1.4 and WorldScore's 3D consistency by at most 5.9.
  • 3D Self-Consistency. Drift and jitter break 3D self-consistency, which every metric tests, and the metrics agree. WBench's geometric and photometric consistency move less under 10 cm jitter (−4.9 and −9.4), likely because a reconstruction built from the same frames absorbs part of the deformation, while ORBIA tests its Gaussians on held-out frames.
  • Appearance. JPEG lowers WorldScore's photometric consistency, an optical-flow check between nearby frames, by 60.9, about as much as jitter, so it mistakes image quality for inconsistency. ORBIA's consistency scores barely change.

Real world models. Video by video on 12 world models and 77 cases, the external metrics partly track ORBIA's Generated Persistence (Spearman 0.37–0.69), since videos that collapse fail every metric. On Input Preservation, WBench agrees only moderately (0.48–0.49) and WorldScore's 3D consistency is nearly unrelated (0.16): 45% of the videos in ORBIA's bottom quarter are rated above its median, close to the 50% expected if the two were unrelated. Neither benchmark checks individual input objects. Four examples, which we will add with frame visualizations to the main text (percentiles among all videos, higher means more consistent):

Video (trajectory)ORBIAWorldScore 3DWBench geometricWhat happens
Lyra2, night plaza (arc and return)Gen. Persistence, 8th pct96th pct47th pctthe tiger statue seen at the first visit is gone at the revisit, while the surrounding buildings are unchanged
LingBot-World v1, indoor corner (peek and retract)Gen. Persistence, 22nd pct63rd pct85th pcta bare wall corner with a stool returns as a table covered with objects
Echo-WM, street plaza (orbit around a kiosk)Input Preservation, 3rd pct82nd pct54th pctback at the input view, the advertising kiosk from the input image is gone and the street behind it is replaced
AlayaWorld, temple garden (peek and retract)Input Preservation, 15th pct90th pct18th pctthe temple roof and lotus pond in the input image are replaced by a large tree and bare ground

The model rankings diverge in the same way. Gen3C, which keeps an explicit 3D cache of the input scene, is 1st of the 12 models on ORBIA's Input Preservation but 5th on WorldScore 3D, and Lyra2 is 4th on Generated Persistence but 10th on WBench geometric. Where the metrics overlap, they agree: the external 3D and geometric metrics correlate with ORBIA's 3D Self-Consistency across the 12 models (τ = 0.73–0.79), and ORBIA adds the input and revisit failures they cannot see.

AuthorsOpenReview ↗

Response to Reviewer XMN4 (3/3)

Q1: "How do you distinguish genuine 3D inconsistency from errors introduced by DA3 and imperfect viewpoint execution? ... reference videos and controlled scene corruptions ... an alternative estimator or ... pose-matched observations?"

A: We run all four analyses.

Geometrically consistent reference videos (errors introduced by DA3). On the renders from W1 / W2, DA3 recovers cameras with a median error of 0.08% of path length and depth with AbsRel 0.018. With ground-truth cameras and depth, Input Preservation and Generated Persistence change by less than 0.5 points (81.1 → 81.2, 95.2 → 95.7). DA3 lowers 3D Self-Consistency by about 5 points (78.1 versus 83.2). DA3's errors are therefore negligible for the preservation scores, and the 3D offset is shared across models, as the estimator comparison below shows.

Controlled scene corruptions. On the same renders, we inject each violation at three strengths (targeted score in bold):

ViolationControlVisual QualityInput PreservationGenerated Persistence3D Self-Consistency
none (clean render)91.564.481.195.278.1
scene drift 10 / 25 / 50 cm89.9 / 86.3 / 81.264.3 / 63.9 / 63.280.1 / 78.5 / 72.092.0 / 88.6 / 82.076.9 / 66.4 / 58.2
geometric jitter 2 / 5 / 10 cm84.5 / 81.8 / 75.063.9 / 62.9 / 61.478.7 / 73.7 / 66.086.9 / 78.6 / 69.573.0 / 64.1 / 52.0
revisited object ×1.10 / 1.25 / 1.5090.0 / 89.6 / 89.664.3 / 64.2 / 64.080.8 / 80.6 / 80.390.3 / 85.4 / 80.277.2 / 76.3 / 75.4
revisited object removed91.563.779.678.375.4
input anchor ×1.10 / 1.25 / 1.5091.1 / 91.7 / 91.364.3 / 64.2 / 63.879.7 / 78.2 / 76.095.0 / 95.1 / 95.078.0 / 77.9 / 77.3
input anchor removed91.463.569.195.178.0
JPEG q = 40 / 20 / 1090.8 / 91.1 / 88.263.6 / 62.1 / 60.181.0 / 80.1 / 78.195.2 / 95.3 / 95.378.1 / 78.4 / 77.0
blur σ = 1 / 2 / 491.1 / 90.8 / 90.562.3 / 58.6 / 54.281.3 / 80.9 / 79.595.2 / 95.3 / 95.377.4 / 77.0 / 77.4

Each targeted score drops monotonically with the violation's strength, while JPEG and blur leave the consistency scores nearly unchanged. ORBIA's scores therefore separate genuine 3D inconsistency from appearance changes.

Evaluation restricted to pose-matched observations (imperfect viewpoint execution). Re-scoring Input Preservation and Generated Persistence only on events whose realized camera reaches the requested one keeps the 12 world-model rankings (τ = 0.97 and 0.88 at 20° / 0.4, 0.76 and 0.85 at 10° / 0.2, with translation relative to path length). Among videos whose realized rotation is within 0.8–1.25 of the requested one, the rankings of the 11 world models with enough such videos also hold (τ = 0.82, 0.85, and 0.75 for Input Preservation, Generated Persistence, and 3D Self-Consistency). Imperfect viewpoint execution therefore explains little of ORBIA's rankings.

Alternative estimator. We rescore 12 world models spanning the leaderboard (ranks 1–16) on 77 short cases with VGGT-Ω, a feed-forward model like DA3, and with ViPE, an optimization-based pipeline whose cameras are more accurate than DA3's on the renders (0.06% of path length) but whose depth is less accurate (AbsRel 0.078). We also replace DA3's Gaussian head with WorldMirror 2.0. Kendall τ against DA3:

TierVGGT-ΩViPEWorldMirror GS
Control0.970.79–
Input Preservation1.000.94–
Generated Persistence1.000.88–
3D Self-Consistency1.000.670.94

VGGT-Ω shifts every model's score by a similar amount (mean absolute difference 0.4–2.1 points, Pearson r ≥ 0.99), so the offset does not depend on the model. ViPE reproduces the Input Preservation and Generated Persistence rankings (τ = 0.94 and 0.88). It agrees less on 3D Self-Consistency (0.67), which depends most on depth, where ViPE is four times less accurate than DA3, and on Control (0.79). The preservation rankings therefore hold under both estimators, and the 3D Self-Consistency ranking holds under VGGT-Ω and WorldMirror 2.0.

Together, these analyses separate genuine 3D inconsistency from DA3's errors and from imperfect viewpoint execution, and the main conclusions remain stable.

We thank Reviewer XMN4 again for the valuable comments.

Reviewer_rdsbOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a benchmark for evaluating whether video generation models maintain a consistent 3D world. It tests preservation of the input scene, persistence of newly generated content, and geometric consistency across 720 cases and 27 models. The results show that strong visual quality and control do not necessarily imply coherent 3D structure.

Critical strengths
  1. The paper provides a substantial benchmark coverage and evaluation scale.
  2. The decomposition of consistency into complementary capabilities is valuable to analysis.
Critical weaknesses
  1. I have 3 major concerns on the evaluation scores: a) The temporal stability score does not necessarily mean stable capability. It is driven mainly by retention, defined as the absence of decline from the initial window. Because retention only penalizes drops, a model that is uniformly poor in every window shows no decline can receive high score, while a model that performs better in every window but dips from a strong start can have a low score. One potential fix is to report average level and decline separately. b) The overall score has half of weight on control and visual quality and only a minor part of weight on 3 consistency scores, so the leaderboard of a benchmark about 3D consistency is mostly a control-and-quality ranking. The paper should report justification or sensitivity analysis. c) The evaluation is not validated independently of DA3. The model is broadly used for this benchmark, but the effect of prediction noise is not reported. Reporting result on gt videos (as upper bound), controlled synthetic violations, or a second estimator could help with this issue.
  2. The reported human validation correlations need clarification. App E.6 describes averaging preferences across raters and cases, then computing Spearman correlations between human and model scores for 6 models, but Table 13 reports 0.913, 0.859, and 0.921, which cannot arise to three decimal places from a single standard six-item Spearman correlation. Authors should provide the 6 score pairs and calculation code.
Questions for authors

Please see weaknesses.

8 comments
AuthorsOpenReview ↗

Response to Reviewer rdsb (1/3)

We thank Reviewer rdsb for the precise comments and address each with new experiments. We first restate our contributions:

  • A benchmark dedicated to 3D consistency. WorldScore and WBench treat 3D consistency as one sub-score among many. ORBIA separates input preservation, generated-content persistence, and 3D self-consistency, so that what the model must keep is assessed apart from what it is free to generate.
  • Registered references and controlled revisits. 720 cases with registered RGB-D references, object anchors, and explore–revisit–return trajectories let each test compare the video against known content.
  • Diagnosis across 27 models. Visual quality, control, input preservation, and generated persistence are distinct capabilities. Lyra2 ranks first in control but seventh in 3D self-consistency, and the two preservation scores agree across models (ρ = 0.89) but only weakly across scenes within a model (0.35).

Relative to the closest benchmarks, ORBIA differs in three concrete ways:

  • Generated content is tested against the model's own first generation. MIND and MemoBench score revisits against ground-truth frames, and WRBench judges whether an off-screen event has progressed. ORBIA's trajectories first visit a region absent from the input image, later return to it, and compare the registered revisit with the model's first generation in appearance, depth, and surface geometry.
  • Input objects are tested against registered references. Registered RGB-D references and object anchors test whether specific input objects survive. Anchors cover a median of 5.2% of the input image, so whole-image and frame-to-frame metrics dilute their loss.
  • Scene-level reconstruction complements local agreement. Among the 12 benchmarks compared in Table 1 of the paper, only WorldOlympiad tests global consistency, and it does not test input preservation or revisits. ORBIA predicts Gaussians from context views and evaluates them at held-out views, so it tests whether one 3D scene explains the whole video.

Our response to Reviewer XMN4 (W1 / W2) compares ORBIA directly with WorldScore and WBench on the same videos, both with controlled errors and on real model videos, and shows the input and revisit failures that these metrics miss.

W1: "The temporal stability score does not necessarily mean stable capability ... report average level and decline separately."

A: We adopt this suggestion. In long-video evaluation, stability usually means retention: how much performance changes over time, reported next to its level (e.g., FramePack, Self-Forcing++, WorldRoamBench). We keep reporting it. Temporal Stability, however, also enters the Overall as a capability score, where it should reward only performance that is both good and sustained. The submitted 0.3·A + 0.7·R was meant to ensure this, but its weight on A is too small, as the reviewer's example shows. We therefore report A and R separately and combine them as A·R/100. The change mainly affects degenerate outputs: the frozen baseline has the highest retention (93.3), since nothing changes, and its Temporal Stability in W2 drops from 84.5 to 46.8. For real models, the T6 ranking stays close to the original (τ = 0.84 for world models, 0.93 for video models), with the same top five. A·R/100 thus rewards only good performance that is sustained, while R remains available as a pure measure of stability.

AuthorsOpenReview ↗

Response to Reviewer rdsb (2/3)

W2: "The overall score has half of weight on control and visual quality ... justification or sensitivity analysis."

A: We keep the weights, because consistency is meaningful only for a video that executes the requested motion and shows a plausible scene, which is what Control and Visual Quality check. Three degenerate baselines on 120 evenly sampled cases, a frozen video, a slow zoom, and a blurred SolarWM-H3, show what happens without them, with SolarWM-H3 on the same cases as reference. Cells show submitted / revised scores, and reach is the share of cases whose revisit views are reached.

BaselineControlVisual Q.Input Pres.Gen. Pers.Reach3D Self-C.Temp. Stab.Overall
frozen18.8 / 0.080.6 / 78.680.3 / 73.399.1 / 99.112%93.8 / 92.884.5 / 46.869.1 / 0.0
low-motion20.5 / 4.480.1 / 78.180.8 / 74.596.9 / 96.89%92.4 / 91.984.0 / 46.769.0 / 35.5
low-detail78.6 / 78.056.9 / 45.866.6 / 58.070.9 / 69.271%73.8 / 72.881.0 / 48.971.2 / 59.3
SolarWM-H377.7 / 77.175.0 / 72.669.4 / 64.069.3 / 67.073%70.9 / 69.782.9 / 61.175.7 / 69.5

The frozen and low-motion videos score above SolarWM-H3 on all three consistency tiers, since they barely change and reach few revisit views. Control separates them from a real model (0.0 and 4.4 versus 77.1), and Visual Quality separates the low-detail one (45.8 versus 72.6).

  • Aggregation. The weights alone do not suffice: under the submitted arithmetic aggregation, the frozen video scores 69.1, close to SolarWM-H3's 75.7. With weighted geometric means, a failed tier cannot be compensated, and the frozen and low-motion videos drop to 0.0 and 35.5, against 69.5 for SolarWM-H3.
  • The weights still matter. Geometric means zero out only a tier that fails completely, and a nearly static video keeps a small nonzero Control score. With equal or consistency-heavy weights (10/10/20/20/20/20), the low-motion video rises from 35.5 to 46.9 or 56.5, and its gap to SolarWM-H3 shrinks from 34 to 11 points. The revised leaderboard therefore also reports each model's revisit reach next to its consistency tiers.
  • Real models barely change. The geometric Overall keeps the submitted ranking of the 19 world models (τ = 0.95, top nine unchanged), and alternative weights keep it close: τ = 0.86 for equal weights, 0.80 for consistency-heavy weights, and a median of 0.80 over 10,000 random weightings, with SolarWM-H3, Lyra2, and EVOKE in the top four in 92% of the draws.

Keeping the weights, aggregating geometrically, and reporting reach therefore close the loophole for static videos without changing how real models rank.

W3: "The evaluation is not validated independently of DA3 ... gt videos (as upper bound), controlled synthetic violations, or a second estimator ..."

A: We run all three suggested analyses on renders of 55 ScanNet++ and 25 UE scenes along ORBIA's trajectories, which give exact cameras, depth, and scene state.

Controlled synthetic violations. We inject graded violations (targeted score in bold):

ViolationControlVisual QualityInput PreservationGenerated Persistence3D Self-Consistency
none (clean render)91.564.481.195.278.1
scene drift 10 / 25 / 50 cm89.9 / 86.3 / 81.264.3 / 63.9 / 63.280.1 / 78.5 / 72.092.0 / 88.6 / 82.076.9 / 66.4 / 58.2
geometric jitter 2 / 5 / 10 cm84.5 / 81.8 / 75.063.9 / 62.9 / 61.478.7 / 73.7 / 66.086.9 / 78.6 / 69.573.0 / 64.1 / 52.0
revisited object ×1.10 / 1.25 / 1.5090.0 / 89.6 / 89.664.3 / 64.2 / 64.080.8 / 80.6 / 80.390.3 / 85.4 / 80.277.2 / 76.3 / 75.4
revisited object removed91.563.779.678.375.4
input anchor ×1.10 / 1.25 / 1.5091.1 / 91.7 / 91.364.3 / 64.2 / 63.879.7 / 78.2 / 76.095.0 / 95.1 / 95.078.0 / 77.9 / 77.3
input anchor removed91.463.569.195.178.0
JPEG q = 40 / 20 / 1090.8 / 91.1 / 88.263.6 / 62.1 / 60.181.0 / 80.1 / 78.195.2 / 95.3 / 95.378.1 / 78.4 / 77.0
blur σ = 1 / 2 / 491.1 / 90.8 / 90.562.3 / 58.6 / 54.281.3 / 80.9 / 79.595.2 / 95.3 / 95.377.4 / 77.0 / 77.4

Each violation lowers its targeted score monotonically. Blur and JPEG lower Visual Quality by up to 10 points and leave the consistency scores nearly unchanged, while at the strongest level, geometric violations lower their targeted consistency score by 12–26 points. These responses are far larger than the effect of DA3's errors (under 0.5 points for Input Preservation and Generated Persistence, about 5 points for 3D Self-Consistency, see below), so estimator errors do not mask genuine violations.

AuthorsOpenReview ↗

Response to Reviewer rdsb (3/3)

Ground-truth geometry (W3, continued). On these renders, DA3 recovers cameras with a median error of 0.08% of path length and depth with AbsRel 0.018. With ground-truth cameras and depth, Input Preservation and Generated Persistence change by less than 0.5 points (81.1 → 81.2, 95.2 → 95.7). DA3 lowers 3D Self-Consistency by about 5 points (78.1 versus 83.2), and a second estimator shifts all models by a similar amount, so such an offset does not reorder models.

Two estimators of different design. We rescore 12 world models spanning the leaderboard (ranks 1–16) on 77 short cases with VGGT-Ω, a feed-forward model like DA3, and with ViPE, an optimization-based pipeline whose cameras are more accurate than DA3's on the ground-truth renders (0.06% of path length) but whose depth is less accurate (AbsRel 0.078). We also replace DA3's Gaussian head with WorldMirror 2.0. Kendall τ against DA3:

TierVGGT-ΩViPEWorldMirror GS
Control0.970.79–
Input Preservation1.000.94–
Generated Persistence1.000.88–
3D Self-Consistency1.000.670.94

VGGT-Ω shifts every model's score by a similar amount (mean absolute difference 0.4–2.1 points, Pearson r ≥ 0.99), so the offset does not depend on the model. ViPE reproduces the Input Preservation and Generated Persistence rankings (τ = 0.94 and 0.88). It agrees less on 3D Self-Consistency (0.67), which depends most on depth, where ViPE is four times less accurate than DA3, and on Control (0.79).

Ground truth, controlled violations, and two estimators of different design thus all show that ORBIA's conclusions do not depend on DA3.

W4: "... human validation correlations need clarification ... provide the 6 score pairs and calculation code."

A: Thank you for carefully checking Table 13 of the paper. The reviewer is correct that the reported values cannot arise from a six-model Spearman correlation. An error in our analysis script produced them, and we sincerely apologize for this oversight. The study ran in two rounds, and a server failure lost part of the responses. We recompute the correlations from all 213 remaining case rankings by 21 raters (the revision corrects the rater count of 20 in Section 5 and Appendix E.6). The 90 cases (60 short and 30 minute-long) were sampled proportionally to data source and horizon, with 15 per trajectory template. Each case showed five or six of the six models, which raters ranked on six capabilities with ties allowed. The calculation:

for each capability d:
    for each completed ranking r (one rater, one case):
        for each ordered pair (a, b) in models shown in r:
            outcome[a] += 1 if rank(a) < rank(b)
                          0.5 if rank(a) == rank(b)
                          0 otherwise
            rated_cases[a].add(r.case)

    H[m] = mean pairwise outcome of model m
    A[m] = mean ORBIA score of model m on rated_cases[m]
    rho[d] = Pearson(average_ranks(H), average_ranks(A))

H and A are the human and automated scores, and the Pearson correlation of their average ranks equals Spearman's ρ.

ModelT1 Auto.T1 Human.T2 Auto.T2 Human.T3 Auto.T3 Human.T4 Auto.T4 Human.T5 Auto.T5 Human.T6 Auto.T6 Human.
Lyra282.260.67271.530.47368.580.54571.090.61460.980.46183.720.527
SolarWM-H379.670.54574.020.48369.310.59769.630.50168.260.55085.010.648
Genie362.920.51974.880.50959.130.52358.790.33660.520.36683.980.599
LingBot-World v257.850.52275.350.56461.320.31754.050.45458.580.46878.180.500
HY-WorldPlay48.820.35573.160.32067.850.39970.720.55771.180.50581.380.326
Seedance2.543.360.39077.210.65372.210.62368.050.51172.480.62782.080.417
Spearman ρ\rho—0.886—0.943—0.829—0.886—0.771—0.829

As the reviewer's observation implies, a model-level Spearman correlation over six models is coarse: swapping one adjacent pair changes ρ by 0.057. To validate each capability at a finer grain, we are running a pairwise human study that covers 10 models. Raters compare two videos on one capability at a time, viewing only the clips relevant to that capability, and we report how often human and ORBIA preferences agree, with confidence intervals. We will post its results in a follow-up reply and add them to the main text of the revised manuscript.

We thank Reviewer rdsb again for the valuable comments.

Reviewer_T87gOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces Orbia, a benchmark for evaluating 3D consistency in interactive video world models during exploration and revisit. It distinguishes input preservation, generated-content persistence, and 3D self-consistency, combining registered RGB-D references, object-level anchors, cross-view correspondence, and held-out Gaussian reconstruction. The benchmark contains 480 short cases and 240 paired minute-long extensions across six trajectory templates. Evaluation of 27 models shows that visual quality, control accuracy, content persistence, and geometric consistency are not interchangeable capabilities.

I lean toward acceptance. The paper provides a useful resource and a comprehensive diagnostic framework, although the validity of the geometry-based measurements and the interpretation of some aggregate scores require stronger evidence.

Critical strengths
  1. A useful capability decomposition supported by complementary empirical evidence. Input preservation, generated-content persistence, and 3D self-consistency impose different constraints. The paper supports this distinction through model rankings, scene-level correlations, and qualitative examples. In particular, the strong model-level association between input preservation and generated-content persistence contrasts with their substantially weaker within-model association across scenes. This supports evaluating both capabilities and makes the benchmark useful for diagnosis.

  2. Substantial reference construction and a well-specified evaluation protocol. Registered RGB-D references, anchor masks, controlled trajectories, and explicit revisit events ground evaluation in identifiable content and spatial relationships. Object-level measurements reduce the risk of overlooking small landmarks in whole-image averages, while held-out reconstruction complements local cross-view tests. Paired long-horizon extensions and multiple data sources further strengthen the resource.

Critical weaknesses
  1. The validity of the geometric measurements remains heavily dependent on a single recovery pipeline, and the current validation does not sufficiently isolate evaluator error.

In Section 4 and Appendix C, several central metrics depend on DA3-estimated cameras and depth, and Gaussian reconstruction also uses DA3. Recovery errors can therefore affect control assessment, cross-view correspondence, revisit surface agreement, and reconstruction simultaneously. Differences in texture, style, sharpness, or motion across generators could systematically change recovery difficulty.

The human correlation study provides useful evidence, but aggregate rankings over six models do not establish sensitivity to controlled geometric violations or rule out model-dependent recovery bias. I am not claiming that such bias has been demonstrated; rather, the current evidence does not sufficiently exclude it. This directly affects whether score differences can be interpreted as differences in 3D consistency and is the main reason I do not recommend clear acceptance.

  1. Some scores require stronger calibration against degenerate solutions and aggregation choices.

The temporal stability definition in Appendix C.6 places substantial weight on retention relative to the initial window. Consequently, an indicator that remains consistently poor can receive a relatively high stability score: a constant level of 20 yields a combined dimension score of 76. Reporting mean performance alongside stability helps, but the stability score alone remains easy to overinterpret.

Low-motion or low-detail outputs may also make some preservation or reconstruction measurements easier. The control tier provides necessary context, but explicit degenerate baselines are needed to establish whether these strategies receive misleadingly favorable scores in individual tiers or the overall aggregate. Since the overall weights are manually specified, the diagnostic profiles are currently better supported than the headline ranking.

Questions for authors
  1. Controlled validity experiment; material to my assessment: Could the authors evaluate rendered videos with known cameras, depth, and scene states, introducing progressively stronger geometric drift, object rescaling, and revisit substitutions? Please include appearance-only controls such as compression or blur. Do the metrics respond monotonically to actual violations, and can they distinguish these violations from changes in image quality?

  2. Evaluator robustness; material to my assessment: Could the authors repeat the relevant measurements on a representative subset using another geometry estimator and, where applicable, ground-truth geometry? Please also report correspondence coverage, tracking validity, or valid reconstruction area to assess whether excluded regions affect the conclusions. A full rerun of all models and cases is unnecessary.

  3. Degenerate baselines and score interpretation; material to my assessment: Could the authors include frozen, low-motion, and low-detail baselines and show their complete profiles across control, persistence, self-consistency, and temporal stability? It would also help to establish whether the main conclusions survive alternative overall weights and separate reporting of performance level and retention.

  4. Clarification: How much do input-preservation and revisit scores reflect failure to reach the requested views? Do the main findings persist when stratifying by control accuracy or comparing subsets with similar realized motion?

These requests concern validation of the existing evaluation claims. I do not expect the paper to solve dynamic physical simulation or introduce a new generator.

8 comments
AuthorsOpenReview ↗

Response to Reviewer T87g (1/3)

We thank Reviewer T87g for the detailed review and for specifying the experiments that would be decisive. We run all of them.

W1: "The validity of the geometric measurements remains heavily dependent on a single recovery pipeline ..."

A: We isolate evaluator error in three ways: controlled violations with known ground truth (Q1), ground-truth geometry and alternative estimators (Q2), and pose-matched evaluation (Q4). All three show that ORBIA's score differences reflect differences in 3D consistency.

W2: "... stronger calibration against degenerate solutions and aggregation choices."

A: We evaluate frozen, low-motion, and low-detail baselines, report Temporal Stability's level and retention separately, and analyse the sensitivity to the Overall weights (Q3). These analyses show that each degenerate strategy is caught and that the leading models are robust to the weights.

Q1: "... rendered videos with known cameras, depth, and scene states ... geometric drift, object rescaling, and revisit substitutions? ... appearance-only controls such as compression or blur."

A: We render 55 ScanNet++ and 25 UE scenes along ORBIA's trajectories, which gives exact cameras, depth, and scene state, and inject each requested control at three strengths. Geometric drift tests 3D Self-Consistency: the scene deforms smoothly by up to 10 / 25 / 50 cm, or jitters by 2 / 5 / 10 cm with a new random field every 4 frames. Rescaling an object (×1.10 / 1.25 / 1.50) or removing it, which substitutes different content at the revisit, tests Generated Persistence for the revisited object and Input Preservation for the input anchor. JPEG and blur change appearance only. Targeted scores are in bold.

ViolationControlVisual QualityInput PreservationGenerated Persistence3D Self-Consistency
none (clean render)91.564.481.195.278.1
scene drift 10 / 25 / 50 cm89.9 / 86.3 / 81.264.3 / 63.9 / 63.280.1 / 78.5 / 72.092.0 / 88.6 / 82.076.9 / 66.4 / 58.2
geometric jitter 2 / 5 / 10 cm84.5 / 81.8 / 75.063.9 / 62.9 / 61.478.7 / 73.7 / 66.086.9 / 78.6 / 69.573.0 / 64.1 / 52.0
revisited object ×1.10 / 1.25 / 1.5090.0 / 89.6 / 89.664.3 / 64.2 / 64.080.8 / 80.6 / 80.390.3 / 85.4 / 80.277.2 / 76.3 / 75.4
revisited object removed91.563.779.678.375.4
input anchor ×1.10 / 1.25 / 1.5091.1 / 91.7 / 91.364.3 / 64.2 / 63.879.7 / 78.2 / 76.095.0 / 95.1 / 95.078.0 / 77.9 / 77.3
input anchor removed91.463.569.195.178.0
JPEG q = 40 / 20 / 1090.8 / 91.1 / 88.263.6 / 62.1 / 60.181.0 / 80.1 / 78.195.2 / 95.3 / 95.378.1 / 78.4 / 77.0
blur σ = 1 / 2 / 491.1 / 90.8 / 90.562.3 / 58.6 / 54.281.3 / 80.9 / 79.595.2 / 95.3 / 95.377.4 / 77.0 / 77.4

Each violation lowers its targeted score monotonically with its strength. Blur and JPEG lower Visual Quality by up to 10 points and leave the consistency scores nearly unchanged, while removing the revisited object, removing the input anchor, and 10 cm jitter lower their targeted scores by 17, 12, and 26 points. Each ORBIA score therefore responds to the violation it is designed to catch, and appearance-only degradation does not register as inconsistency.

AuthorsOpenReview ↗

Response to Reviewer T87g (2/3)

Q2: "... another geometry estimator and, where applicable, ground-truth geometry? ... correspondence coverage, tracking validity, or valid reconstruction area."

A: We run all three analyses.

Ground-truth geometry. On the renders from Q1, DA3 recovers cameras with a median error of 0.08% of path length and depth with AbsRel 0.018. Replacing them with ground truth reproduces Input Preservation (81.1 → 81.2) and Generated Persistence (95.2 → 95.7). DA3 lowers 3D Self-Consistency by about 5 points (78.1 versus 83.2) and Control by about 8 points (91.5 versus 100), a common ceiling for every model. On real model videos, a second estimator shifts all models by a similar amount (below), so such offsets do not reorder models. Under strong blur (σ = 4) and JPEG (q = 10), DA3's errors stay at 0.10–0.15% of path length and AbsRel 0.018–0.022, so sharpness and texture do not distort recovery.

Two estimators of different design. We rescore 12 world models spanning the leaderboard (ranks 1–16) on 77 short cases with VGGT-Ω, a feed-forward model like DA3, and with ViPE, an optimization-based pipeline whose cameras are more accurate than DA3's on our renders (0.06% of path length) but whose depth is less accurate (AbsRel 0.078). We also replace DA3's Gaussian head with WorldMirror 2.0. Kendall τ against DA3:

TierVGGT-ΩViPEWorldMirror GS
Control0.970.79–
Input Preservation1.000.94–
Generated Persistence1.000.88–
3D Self-Consistency1.000.670.94

VGGT-Ω shifts every model's score by a similar amount (mean absolute difference 0.4–2.1 points, Pearson r ≥ 0.99), so the offset does not depend on the model. ViPE reproduces the Input Preservation and Generated Persistence rankings (τ = 0.94 and 0.88). It agrees less on 3D Self-Consistency (0.67), which depends most on depth, where ViPE is four times less accurate than DA3, and on Control (0.79).

Coverage. Revisit covisible coverage ranges from 0.41 to 0.75 across models, and anchor coverage from 0.59 to 0.96. Excluding unmatched regions does not favor weak models, since models with less covisible area also score lower on Generated Persistence (τ = 0.85). The Gaussian renders cover over 99.9% of the held-out pixels for every model.

Ground-truth geometry and a second estimator thus lead to the same conclusions as DA3.

AuthorsOpenReview ↗

Response to Reviewer T87g (3/3)

Q3: "... frozen, low-motion, and low-detail baselines ... alternative overall weights and separate reporting of performance level and retention."

A: We evaluate a frozen video (input image repeated), a low-motion slow zoom, and a low-detail video (blurred SolarWM-H3) on 120 cases sampled evenly across data sources and trajectory templates. Cells show submitted / revised scores. Revisit reach is the share of cases whose revisit pair lies within 10° and 0.2 of the requested camera (translation relative to path length). A and R are case averages, and the revised Temporal Stability multiplies them per case, so it differs from the product of the averages.

BaselineControlVisual Q.Input Pres.Gen. Pers.Revisit reach3D Self-C.Level ARetention RTemp. Stab.Overall
frozen18.8 / 0.080.6 / 78.680.3 / 73.399.1 / 99.112%93.8 / 92.864.193.384.5 / 46.869.1 / 0.0
low-motion20.5 / 4.480.1 / 78.180.8 / 74.596.9 / 96.89%92.4 / 91.963.093.084.0 / 46.769.0 / 35.5
low-detail78.6 / 78.056.9 / 45.866.6 / 58.070.9 / 69.271%73.8 / 72.859.790.181.0 / 48.971.2 / 59.3
SolarWM-H3 (same cases)77.7 / 77.175.0 / 72.669.4 / 64.069.3 / 67.073%70.9 / 69.769.688.582.9 / 61.175.7 / 69.5
  • Each strategy is caught by its tier. As the reviewer expected, the static baselines score high on the consistency tiers. Control catches them (0.0 and 4.4 versus 77.1), and Visual Quality catches the low-detail one (45.8 versus 72.6). Their Generated Persistence comes from revisit views they never reach (12% and 9%, against 73%).
  • Geometric Overall. Under the submitted arithmetic aggregation, the frozen video scores 69.1, close to SolarWM-H3's 75.7. With weighted geometric means within and across tiers, a strong tier cannot compensate a failed one, and the frozen and low-motion videos drop to 0.0 and 35.5, against 69.5 for SolarWM-H3. The ranking of real world models barely changes (τ = 0.95, top nine unchanged).
  • Level and retention. In long-video evaluation, stability usually means retention: how much performance changes over time, reported next to its level (e.g., FramePack, Self-Forcing++, WorldRoamBench). We keep reporting it. Temporal Stability, however, also enters the Overall as a capability score, where it should reward only performance that is both good and sustained. The submitted 0.3·A + 0.7·R gave A too little weight: the frozen video has the highest retention (93.3) because nothing changes. We therefore combine them as A·R/100, so the constant-20 example scores 20 instead of 76. The T6 ranking stays close to the original (τ = 0.84 for world models, 0.93 for video models).
  • Alternative weights (revised aggregation, 19 world models). Equal weights give τ = 0.86 (at most four places moved), consistency-heavy weights (10/10/20/20/20/20) give τ = 0.80 (at most six), and 10,000 random Dirichlet weightings give a median τ = 0.80. SolarWM-H3, Lyra2, and EVOKE remain the top three under both fixed alternatives and stay in the top four in 92% of the random draws.

The static strategies are thus caught by the tiers they fail and fall far below SolarWM-H3, while the leading models stay on top under alternative weights.

Q4: "How much do input-preservation and revisit scores reflect failure to reach the requested views? ... stratifying by control accuracy or ... similar realized motion?"

A: With the reach tolerance above, missing the requested views costs a typical model 1.5 points of Input Preservation and 4.4 points of Generated Persistence, small compared with the spread between models (about 17 and 29 points). Kendall τ against the full evaluation when only reached events are scored (12 world models):

Reach tolerance (rotation, translation)Input Preservation τGenerated Persistence τ
20°, 0.400.970.88
10°, 0.200.760.85
  • Reached views. At the strict 10° / 0.2 tolerance, the 12 world models reach the requested views in only 19–82% of cases, yet their rankings on these cases match the full evaluation (τ = 0.76 and 0.85).
  • Control accuracy. Within low-, mid-, and high-control terciles, the rankings of the 19 world models stay consistent with the full ranking (τ = 0.58–0.82).
  • Similar motion. Among videos whose realized rotation is within 0.8–1.25 of the requested one, the rankings of the 11 world models with enough such videos agree with the full evaluation (τ = 0.82, 0.85, and 0.75 for Input Preservation, Generated Persistence, and 3D Self-Consistency).

Missing the requested views therefore explains little of ORBIA's preservation scores. We thank Reviewer T87g again for the valuable comments.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We sincerely thank our reviewers for their time and effort in providing thorough and constructive feedback on our paper.


We are encouraged to see that our reviewers recognize ORBIA as a useful and well-grounded benchmark for 3D consistency in interactive world models:

  • Reviewers T87g, rdsb, and XMN4 find the decomposition into input preservation, generated-content persistence, and 3D self-consistency useful: "a useful capability decomposition", "valuable to analysis", and "a useful framework for diagnosing different failure modes".
  • Reviewer XMN4 notes that "previous works usually does not check the consistency for generated contents".
  • Reviewers T87g and XMN4 value the registered references, which "ground evaluation in identifiable content" and "support more targeted evaluation than whole-image similarity alone".
  • Reviewers T87g and rdsb value the "substantial reference construction" and the "substantial benchmark coverage and evaluation scale".
  • Reviewers T87g, rdsb, and XMN4 think our paper is "clear and easy to follow".

We highlight our contributions as follows:

  • A benchmark dedicated to 3D consistency. WorldScore and WBench treat 3D consistency as one sub-score among many. ORBIA separates input preservation, generated-content persistence, and 3D self-consistency, so that what the model must keep is assessed apart from what it is free to generate.
  • Registered references and controlled revisits. 720 cases with registered RGB-D references, object anchors, and explore–revisit–return trajectories let each test compare the video against known content.
  • Diagnosis across 27 models. Visual quality, control, input preservation, and generated persistence are distinct capabilities, and ORBIA detects input and revisit failures that existing consistency metrics miss.

As suggested by the reviewers, we ran the following experiments and will add them to the revised manuscript:

Additional Results:

  • Controlled violations with known ground truth: on renders with exact cameras, depth, and scene state, each injected violation lowers its targeted score monotonically, and the consistency scores barely respond to appearance-only changes, as suggested by Reviewers T87g, rdsb, and XMN4.
  • Independence from DA3: ground-truth geometry reproduces Input Preservation and Generated Persistence within 0.5 points, VGGT-Ω reproduces the rankings of 12 world models (τ = 0.97–1.00), and ViPE and WorldMirror 2.0 Gaussians provide further checks, as suggested by Reviewers T87g, rdsb, and XMN4.
  • Frozen, low-motion, and low-detail baselines with complete tier profiles, and a sensitivity analysis over 10,000 random weightings, as suggested by Reviewers T87g and rdsb.
  • Coverage, pose-matched, motion-matched, and control-binned evaluation, as suggested by Reviewers T87g and XMN4.
  • A direct comparison with WorldScore and WBench on the same videos, as suggested by Reviewer XMN4.

Other Modifications:

  • The headline Overall uses weighted geometric means within and across tiers, and Temporal Stability reports level and retention separately and combines them as A·R/100, which guards against degenerate outputs such as static videos and barely changes the ranking of real world models, as suggested by Reviewers T87g and rdsb.
  • We correct Table 13 of the paper and provide the per-model score pairs and the calculation, and we are running a pairwise human study on 10 models whose results we will post in a follow-up reply, as suggested by Reviewer rdsb.
AuthorsOpenReview ↗

Follow-up: New Human Study

We sincerely thank our reviewers for their time and effort in providing thorough and constructive feedback on our paper.


As promised in our previous response, we have completed the new human study that replaces the original ranking study.

Setup. Each question shows two videos of the same case and asks about one capability, showing only the views that judgment needs (e.g., the first visit and the revisit for Generated Persistence). Raters choose the better video or mark the two as equal. We collected 1,295 judgments from 48 raters on 349 cases covering ten world models that span the leaderboard. A model's human score is its win rate, with a tie counted as half a win. Its automatic score is its ORBIA score on the same cases.

ModelT1 Auto.T1 HumanT2 Auto.T2 HumanT3 Auto.T3 HumanT4 Auto.T4 HumanT5 Auto.T5 HumanT6 Auto.T6 Human
SolarWM-H377.660.60571.660.61666.890.66367.600.60566.710.67460.250.557
Lyra284.490.68668.850.50067.170.62866.460.62861.770.48859.270.636
Gen3C75.780.64062.260.36074.700.69863.840.57059.710.60555.450.659
Matrix-Game 3.562.240.55871.590.66363.430.60559.300.53551.260.47755.110.625
LingBot-World v255.640.50079.250.64051.200.44246.760.43051.690.39552.020.545
SANA-WM60.090.45369.150.44257.810.43047.970.50050.380.45343.790.386
HY-WorldPlay39.040.34965.680.34962.740.45366.200.65168.580.57044.540.443
Voyager62.050.46563.800.34950.050.32636.440.32644.480.31446.720.511
DreamX-World44.700.38473.210.68654.030.40744.110.38450.310.36039.770.330
Astra28.750.36049.800.39548.940.34940.120.37261.040.66328.210.307
Spearman ρ—0.939—0.815—0.939—0.952—0.830—0.915

Results. Human win rates follow ORBIA in every tier (Spearman ρ = 0.815–0.952 over the ten models). They also follow the revised main leaderboard in Table 2 in all six tiers.

In the revised manuscript, this study replaces the original Table 14 and the corresponding sentence in Section 5.4.

Best regards,

The Authors of Submission 16

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.