acceptodds
Withdrawn from ICLR 2027

Same Geometry, Different Scores: Auditing Coordinate and Scale Dependence in Geometric Video Evaluation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video generation benchmarks increasingly score geometry: they reconstruct camera paths, depth, and object motion from generated frames and score the reconstruction. The axes and scale of a reconstruction are arbitrary conventions, so rotating the axes or rescaling the scene describes the same scene, and a geometry score should not change. We test whether it does: we rotate or rescale saved reconstructions, compare the scores, and trace every change to the code that causes it; for WorldScore we check instead that a reversed camera path scores lower. The audited component of each benchmark fails its test. On simulator ground truth, the range of PDI-Bench's motion score across 256 sampled rotations has a median of 80.7% of the unrotated score (objects with a nonzero score), because the metric takes medians separately along each axis. Rescaling the scene by 10 changes the final MBench scores of 72% to 83% of 3,218 videos and doubles the gap between two generators, because its frame-pair selector adds a distance to an angle; switching the reconstruction model can, through scale alone, reverse the direction of a score change. In controlled translation tests, WorldScore gives full marks to a camera that moves the opposite way. On Dynamic Replica, a comparison of two pipeline versions reverses once both are scored on the same observations. In the three benchmark components, each failure comes from a few lines of code, and minimal code changes pass the tested relations. These tests are cheap, need no new videos, and should be run before geometric scores are used to compare models.

Primary area:
datasets and benchmarks
Keywords:
geometric video evaluation, video world models, evaluation metrics, invariance testing, metamorphic testing, coordinate frame, scale ambiguity, 3D reconstruction, benchmark auditing, measurement validity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7JpHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper audits commonly-used video generation benchmark metrics that are specifically designed for scene geometry evaluation (eg, 3D consistency). The paper proposes a set of tests (eg, scaling, rotation, reversed camera path, etc) to audit that each benchmark metric is designed to give consistent scores/conclusions under such geometric variations, ie, in principle those metrics should give consistent scores and should not depend on the geometric estimation models' outputs (eg, its own scale, coordinate, etc).

The paper includes audited results and conclusion; analyzing the problems and concerns on the existing benchmark metrics.

Critical strengths
  • Good motivation

    The paper raises a valid problem; the benchmark metrics focus on geometric consistency should give the same score even under valid geometric transformations (eg, scene scale, coordinate rotation, etc) that are coming from geometry estimation models (eg, VGGT). If the benchmark metrics does not take this factor into account, then the metric may depend on the property of any geometry estimators used, instead of reliably reporting valid scores.

  • Auditing benchmarks & its discussion

    The paper includes auditing results and discussions on three commonly used benchmarks, PDI-Bench, MBench, and WorldScore. This audited result can be valuable to the community.

Critical weaknesses
  • Poorly written, closer to an audit report than an academic paper

    • The paper is poorly written that it's difficult to understand its main claim. The main section (4. Results) enumerates analyses details and its auditing result for each benchmark; no solid/insightful conclusions can be found. It reads more like an audit report than an academic paper. As in the title, the paper positions itself as an auditing method, which is fine, but it doesn't go further -- one would expect that the paper would propose a better benchmark metric and re-benchmark many methods based on the revised metrics, etc.

    • The paper focuses on enumerating details, techniques, each method's numbers, metric etc, without any insights or implication provided. It's really difficult to get what messages the paper wants to deliver.

  • Lack of clarity, incorrect notation for equations

    • In Eq(1), shouldn't g be considered as a function because it's an operation for scaling, rotation, etc? ie, O(g(X)) = O(X)

    • Also how the τrel\tau_{rel} and τabs\tau_{abs} are determined?

    • What does O(X) mean? It's difficult to get what scored observations mean.

  • Lack of context and insight

    • In Table.1, I am not so sure how informative to provide such details, eg, Code location. It does not provide any major implications or conclusions from these datapoints. What can readers learn from this datapoints?
    • At L264, what are Flow and Sora videos? I would guess they are generative video models, but those names pop-up without any explanation or underlying contexts (eg, why those models are used, what hypothesis will be validated, why they are chosen, etc). It lacks such contexts without any explanation, which make the paper difficult to understand. This is just one example, and this 'lack-of-context' feeling is generally what I faced when reading the paper.
Questions for authors

N/A

Reviewer_qP6mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a method to evaluate the '' evaluation method '' of geometry consistency in interactive video world model, instead of evaluating the videos. The authors use several geometry or 3D priors like scale invariance or rotation invariance to check the correctness of current evaluation metrics when benchmarking the generated video of world model.

Critical strengths
  1. An interesting idea to check the correctness of current geometry evaluation metrics of interactive video generation models, utilizing the 3d prior of invariance, achieving the validity of these metrics.
  2. Not only finding the bug inner evaluation metrics, but also give a minimal patch to fix them.
Critical weaknesses
  1. lack of human study on the pre-fix and after fix evaluation on these metrics about how they are aligned with human feelings of generated interactive videos. The authors didn't prove that the necessity and validity of geometry could lead to better benchmark accuracy.
Questions for authors
  1. the authors should use human study or other method to demonstrate the improvement of geometry alignment can lead to better benchmark accuracy.
Reviewer_ViSTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper discusses coordinate, scale, and direction dependence in components of PDI-Bench, MBench, and WorldScore. It identifies implementation defects, tests minimal repairs, and illustrates how observation selection can reverse a comparison on Dynamic Replica. Validation includes simulator geometry, thousands of reconstructed videos, and controlled camera trajectories.

Critical strengths
  1. The controlled design makes the identified defects convincing. Simulator ground truth isolates PDI's coordinate dependence, while MBench's crossed selection/threshold controls identify the main source of scale sensitivity (Sections 4.1–4.2; Appendices A–B).

  2. The code patches, archived results, and inexpensive regression tests are practically useful. The paper also clearly distinguishes passing invariance checks from establishing metric accuracy, which appropriately limits its claims.

Critical weaknesses
  1. Limited novelty, and the claim of generality is untested. Section 2 itself notes that trajectory evaluation already treats the world frame and monocular scale as free and constrains the alignment scale to be nonnegative; the repairs are the corresponding standard fixes (Table 1). These findings would amount to more than three bug reports if such failures were shown to be common. However, the paper gives no criterion for choosing the three components, all three fail, and none of the other geometric metrics cited in Section 2 is audited, so the frequency of these failures cannot be estimated. Section 5's "Where else to look" and the claim that convention-dependent rewards pass their dependence on to trained models (Section 2) are not tested.

  2. In the paper's ground-truth test of PDI, invariance is not the main source of error. On MOVi-E with true depth, masks, and cameras, the supplementary per-object table shows that the PDI motion score correlates weakly with the score of the true object centers: Spearman ρ = 0.21 original and 0.22 repaired over 1,307 objects (95% intervals about [0.14, 0.27] and [0.16, 0.29], resampling records). One change of axes moves a score by a median of 13.4% (Table S4, k = 1), whereas the median error against the true-center score is about 35% for both implementations. The 80.7% headline is a maximum-minus-minimum over 257 frames; the larger error, which the repair leaves unchanged, is reported in Table S3, but its size relative to the scores is not discussed. MBench shows a similar pattern: equally invariant normalizers shift generator means by up to 0.266 (Table S9). The paper calls invariance necessary rather than sufficient, yet recommends these tests before comparing models; for the one invariance repair checked against ground truth, passing them does not improve agreement with the truth.

Questions for authors
  1. Validation of the recommendation: In the MOVi-E experiment, how well do the original and repaired PDI scores agree with the true-center score (e.g., rank correlation, error relative to the score)? If agreement is weak for both, what evidence shows that passing these tests makes a model comparison more reliable?

  2. Scope (clarification): How were the three components selected, and were other components or geometric metrics examined that passed? Reporting them would indicate how common these failures are.

I would consider raising my score after the rebuttal if the authors show that these failures are a material source of error relative to the metrics' agreement with the truth, or that they occur beyond the three selected components.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.