- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies post-hoc ranking of generated videos according to local structural drift, such as local shape deformation, inconsistent surfaces, or short-lived geometric distortions. It proposes Temporal-Geometry Consistency (TGC), a cohort-relative score that combines four readouts based on predicted surface normals and depth: directional coherence, structural variation of a semantic–geometry agreement field, directional residual coherence, and adjacent-frame depth-derived normal coherence. On the 407-pair GeneVA evaluation, TGC improves over the Relation baseline by approximately 0.0367 AUROC. The paper also reports ablations, encoding diagnostics, per-generator analyses, bootstrap confidence intervals, and a sealed same-engine evaluation. However, on the pre-registered VideoPhy-2 evaluation, the pooled improvement over Relation is small and statistically unresolved. The results are positive for three unseen generators and negative for two, and the primary fresh union does not show a clear improvement. The paper therefore provides evidence for a useful GeneVA-style cohort-relative verifier, while cross-generator generalization remains uncertain.
Critical strengths
- The paper targets a meaningful evaluation problem. Local structural drift can be missed by global video-quality metrics, and a post-hoc ranking tool could help prioritize clips for human inspection.
- The evaluation includes useful controls and diagnostics. Table 4 reports generator-level results and leave-one-generator-out analyses; Table 5 controls for encoding; Table 7 reports fresh evaluation blocks and confidence intervals; and Table 10 compares against external geometry baselines on a sealed GeneVA block. These analyses make the empirical claims more transparent.
Critical weaknesses
- The evidence for cross-generator effectiveness is limited. On the main GeneVA evaluation, TGC improves over Relation by +0.0367 AUROC (Table 7). However, on VideoPhy-2, the gain is only +0.0036 with a 95% confidence interval of [-0.0565, +0.0643]. The primary fresh union is -0.0086 [-0.1270, +0.1028], and the three-block union is -0.0007 [-0.0808, +0.0752]. Table 8 shows positive gains on three generators, while Table 9 shows negative gains on two. These results are consistent with the authors’ stated limitation, but they restrict the contribution to a narrower, benchmark- or cohort-dependent result.
- The paper does not yet establish that TGC measures the target artifact itself. The ablations in Section 4 and Appendix D show that the proposed readouts affect ranking performance, including the depth-only factor and rank-normalized baselines. However, these results do not directly establish that high TGC scores correspond to independently verified local structural drift rather than correlated factors such as motion complexity, generator identity, or scene content. Independent artifact-level annotations or controlled synthetic deformations would strengthen the causal interpretation of the score
Questions for authors
- Why is the four-factor construction necessary? The depth-only factor outperforms TGC on GeneVA (0.7241 vs. 0.7177, Appendix D) and has larger reported gains on both fresh blocks (Table 7). What evidence shows that the other three factors add useful information rather than dilute the strongest signal?
- What makes temporal geometry inconsistency specific to structural drift? Legitimate motion, occlusion, viewpoint changes, and geometry-estimation errors can also produce inconsistency, while persistent structural defects may remain temporally coherent. How does TGC distinguish these cases?
- What justifies combining within-generator ranks? Rank normalization discards magnitude and does not account for factor reliability or redundancy. What methodological or empirical evidence supports the proposed aggregation rule?
- Where is the independently confirmed improvement? The VideoPhy-2 gain is only +0.0036 AUROC, and both fresh-union estimates are near zero or negative (Table 7). What evidence establishes a meaningful advantage over Relation and the simpler depth-only comparator?