- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
OPIS evaluates whether a generated video preserves the objects established in an initial image. It constructs a fixed list of objects along with their geometric references, then measures their presence, identity, and structure, distinguishing the absence of an object from cases involving occlusion or insufficient evidence. The benchmark consists of 500 examples spanning real-world settings, specific scenarios, and gaming environments. Eight video generation systems were evaluated using a rigorous process that included frame-level aggregation, diagnostic analysis of evidence coverage, bootstrapping for confidence intervals, and manual review of evaluation decisions. The study found that strict identity scores were significantly lower than presence scores, with identity scores appearing even lower in scenes characterized by high object density.
Critical strengths
-
The fixed-input-based inventory addresses meaningful evaluation challenges. Generated videos can remain visually plausible even when the identity or structure of specific objects is altered. By distinguishing between presence, appearance, structure, and evidence that cannot be evaluated, this benchmark serves as a more precise diagnostic tool than a single video quality score. The inclusion of 500 diverse cases, along with the provided code and sample data, represents a valuable contribution.
-
The paper makes a significant effort to shed light on the limitations of the evaluators. Appendix G distinguishes between repeatability and accuracy, reports inter-rater agreement across four types of decisions, and reveals that "structure" is the dimension with the lowest reliability. Through coverage diagnostics, the removal of aggregation, and the use of bootstrapped confidence intervals for cases, the reported scores are rendered more interpretable.
Critical weaknesses
-
The validation does not yet establish the reliability of the final strict metric. A single below-threshold object invalidates an Identity or Structure frame score (Section 4), so individual evaluator errors can have disproportionate effects. Appendix G reports 86.0% identity agreement and 83.3% structure agreement with manual review on 150 decisions per dimension, but these overall agreements do not reveal false-failure rates, density dependence, or how disagreement propagates to rollout scores and model comparisons. The 90.5% repeatability result cannot resolve this issue, as the authors correctly acknowledge. Controlled preservation/deletion/identity-change cases or a human audit of complete scored rollouts would provide more direct evidence that the benchmark separates generator failures from evaluator failures. This is consequential because the benchmark's principal contribution is its measurement, rather than a new generation method.
-
The density result is difficult to separate from the chosen aggregation rule. Section 5.3 reports strict Identity declining from 40.22 to 33.10 to 23.11 as reference-object count increases. Under an all-object gate, the chance that a frame passes decreases with the number of assessed objects even if per-object fidelity stays constant. Appendix H explicitly acknowledges that the score does not establish a decline in per-instance fidelity; that qualification is appropriate, but leaves the substantive scaling insight underdetermined. The aggregation ablations do not replace a count-controlled analysis. Reporting per-object failure rates and fixed-count subsamples from the archived measurements would show whether density adds difficulty beyond the intended stricter joint-preservation criterion. These measurement-validity concerns lead me to weak rejection despite the useful benchmark design.
Questions for authors
-
Table 13 shows overall agreement rather than the direction of errors. Could you report the false-positive and false-negative rates for manual judgments, including details on the sampling unit and the criteria used for handling ambiguous cases? (Weakness 1)
-
Sections 4.2–4.4 employ a strict "frame gate." In cases where audited frames or rollout data are available, please explain how often corrections to human judgments alter the outcomes of these gates, rollout scores, or model comparisons. Please distinguish this from the level of agreement in the human judgments themselves. (Weakness 1)
-
Section 5.1 specifies acceptance thresholds and dimension weights. How were these values determined, and were the calibration cases kept separate from the evaluation cases? Furthermore, do the main conclusions remain valid if these thresholds are varied within a reasonable range? (Weakness 1)
-
Table 5 shows a reversal in the close point estimates for Seedance/H3-Max-Turbo when equal weights are applied. Could you provide pairwise bootstrap confidence intervals for the comparisons highlighted in the paper, while maintaining the evaluation hierarchy? (Weakness 1)
-
Appendix H appropriately distinguishes between joint preservation and instance-level fidelity. Could you report the failure rate (falling below the threshold) per object, as well as the number of valid objects per density level, to allow for an assessment of scaling performance independent of the overall object-count requirements? (Weakness 2)
-
As the number of evaluated instances increases, it becomes increasingly difficult to pass the overall object-count verification criteria. Could you recalculate the identities for a fixed number of object subsamples, while controlling the domain configuration, and report which frames remain valid? (Weakness 2)