- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper audits commonly-used video generation benchmark metrics that are specifically designed for scene geometry evaluation (eg, 3D consistency). The paper proposes a set of tests (eg, scaling, rotation, reversed camera path, etc) to audit that each benchmark metric is designed to give consistent scores/conclusions under such geometric variations, ie, in principle those metrics should give consistent scores and should not depend on the geometric estimation models' outputs (eg, its own scale, coordinate, etc).
The paper includes audited results and conclusion; analyzing the problems and concerns on the existing benchmark metrics.
Critical strengths
-
Good motivation
The paper raises a valid problem; the benchmark metrics focus on geometric consistency should give the same score even under valid geometric transformations (eg, scene scale, coordinate rotation, etc) that are coming from geometry estimation models (eg, VGGT). If the benchmark metrics does not take this factor into account, then the metric may depend on the property of any geometry estimators used, instead of reliably reporting valid scores.
-
Auditing benchmarks & its discussion
The paper includes auditing results and discussions on three commonly used benchmarks, PDI-Bench, MBench, and WorldScore. This audited result can be valuable to the community.
Critical weaknesses
-
Poorly written, closer to an audit report than an academic paper
-
The paper is poorly written that it's difficult to understand its main claim. The main section (4. Results) enumerates analyses details and its auditing result for each benchmark; no solid/insightful conclusions can be found. It reads more like an audit report than an academic paper. As in the title, the paper positions itself as an auditing method, which is fine, but it doesn't go further -- one would expect that the paper would propose a better benchmark metric and re-benchmark many methods based on the revised metrics, etc.
-
The paper focuses on enumerating details, techniques, each method's numbers, metric etc, without any insights or implication provided. It's really difficult to get what messages the paper wants to deliver.
-
-
Lack of clarity, incorrect notation for equations
-
In Eq(1), shouldn't g be considered as a function because it's an operation for scaling, rotation, etc? ie, O(g(X)) = O(X)
-
Also how the and are determined?
-
What does O(X) mean? It's difficult to get what scored observations mean.
-
-
Lack of context and insight
- In Table.1, I am not so sure how informative to provide such details, eg, Code location. It does not provide any major implications or conclusions from these datapoints. What can readers learn from this datapoints?
- At L264, what are Flow and Sora videos? I would guess they are generative video models, but those names pop-up without any explanation or underlying contexts (eg, why those models are used, what hypothesis will be validated, why they are chosen, etc). It lacks such contexts without any explanation, which make the paper difficult to understand. This is just one example, and this 'lack-of-context' feeling is generally what I faced when reading the paper.
Questions for authors
N/A