- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether few-step autoregressive video models show the start, progress, and result of a requested event in the correct order. It introduces OETGBench, with 2,500 tasks, and OETGEval, which combines video diagnostics with phase checks in overlapping windows. Ordered matching separates phase presence from coarse temporal order. The authors also test AG-P3T, a post-training method with anchor-relative penalties and positive-advantage gating.
Critical strengths
-
The task definition provides a useful diagnostic framework. The paper separates onset, progression, and outcome, and reports phase evidence, order, quality, and failures. Generation prompts are separate from evaluator-side specifications. The shared phase-window representation and ordered matching provide a useful, incremental extension to existing transition evaluation.
-
The same-video analysis supports separate evaluation of event completion and consistency. Appendix E.2 reports a negative correlation of −0.360, with a 95% interval of [−0.555, −0.142]. The result persists after removing flicker. The study includes three generators, three annotators, three VLMs, and controlled perturbations. The authors explain metric limits and avoid a causal interpretation.
Critical weaknesses
-
Task quality and evaluation coverage need stronger support. Section 3.1 mentions human inspection. Appendix A does not report the audit size, error rate, or corrections. Automatic checks cannot establish whether phase targets are distinct or feasible within the video duration. Ambiguous specifications could affect the reported failure rates. The main comparison covers 50 tasks from a 2,500-task corpus, with one video per generator per task. The paper needs to explain how these tasks were selected and what range of events they cover. Inter-annotator agreement is also missing.
-
Post-training gains need independent confirmation. Training and evaluation use the same event scorer. The human study excludes AG-P3T. Thus, the event-score increase from 0.2905 to 0.4038 in Table 3 lacks independent confirmation.
Relative to CF++, onset drops from 0.500 to 0.400, VLD Quality from 0.7090 to 0.6463, and VLD Failure rises from 0.0375 to 0.0738. The similar VBench Total comes from a separate prompt suite. Table 5 provides cumulative ablations without variation across training runs. A blind comparison on matched, unseen prompts is needed to verify event gains and assess quality changes.
Questions for authors
-
Task quality and coverage: How many tasks were checked by humans, and what errors were found? Report inter-annotator agreement and the selection procedure for the 50 evaluation tasks. Does this subset cover the four task families and their main transition types?
-
Post-training gains: Can a blind human assessment compare CF++ and AG-P3T on the same unseen prompts and matched generation seeds? Report event completion, order, and visual quality. Clarify test-set separation and the stability of the ablation results across training runs.
-
AR-specific interpretation: Is the paper documenting failures in few-step AR models, or attributing them to autoregression or distillation? The latter requires a matched non-AR or teacher-model comparison. Please separate the observed failure from its proposed cause.
Minor Questions
- State train, validation, and test counts in the PDF.
- State the maximum event score of 0.80 near the main definition and in relevant table captions. Explain the role of the nonbinding caps.
- Replace the abstract's claim to “ensure alignment” with a statement of moderate, phase-dependent human agreement. Keep the VBench claim limited to that suite.
Response to Reviewer 7Ztt
W1: The benchmark construction and validation provide insufficient evidence that the 2,500 tasks are genuinely high-quality and semantically diverse. There is no convincing validation of whether the generated onset/progression/outcome annotations are correct and unambiguous.
A1: Thank you for your valuable feedback. We have carefully considered your comments and provide detailed responses below:
1. Task-specification audit
We audited 100 written task specifications, sampling 25 from each of the four event families. We invited three graduate students as reviewers to conduct the human scoring. The table gives the number of tasks each reviewer marked Yes. Three-way agreement is the percentage of tasks on which all three reviewers assigned the same label (Yes, No, or Uncertain).
Across the three reviews, 97–99 tasks passed the onset check, 96–97 passed progression, and 94–96 passed outcome. Each reviewer marked six tasks infeasible within five seconds and a further 8–14 uncertain. The clearest weakness was in the evaluation questions. Even when the phase targets were distinct and ordered, reviewers could find that a question asked for evidence beyond what the prompt specified. We will report the criterion-level counts in Appendix A.1. These results describe the balanced 100-task sample.
W2: “OETGEval has only moderate, and sometimes even weak, agreement with human judgments, undermining its use as the primary evaluation. This is particularly concerning because both the benchmark conclusions and AG-P3T optimization depend heavily on these evaluation results. An AUROC of 0.708 and temporal-order correlation around 0.4 still leave substantial noise in evaluation.”
A2: Thank you for your valuable feedback. We have carefully considered your comments and provide detailed responses below:
1. Potential impact of model capacity
As Reviewer MYxU also suggested, the modest agreement with human judgments may be partly attributable to the capacity of the underlying models. In future work, we plan to explore more frontier models to examine whether stronger judge capabilities can further improve alignment with human judgments.
2. All rating sources placed CF++ last on phase presence
Table 2 reports moderate agreement with the original retrospective order reference (Spearman ρ = 0.406, AUROC = 0.708). To test whether evaluator choice alters our benchmark conclusions, we compared the three generators across the same 37 prompts for which every generator received determinate phase ratings in all three reassessments (111 clips). Both the original ratings and each of the three human-reviewed reassessments yielded the exact same ranking by mean phase presence (Self-Forcing > CausVid > CF++). Across the three VLM judges, CF++ likewise had the lowest mean phase presence, although their ranking of the other two generators varied (Appendix E.1).
3. Direct review identifies what the generators missed
In the second reassessment of the 150 baseline clips, 55 showed the visible phases in the requested order; however, 47 of these still fell short of completing the event. This finding provides a concrete reason to score event completion separately from relative order. For detecting order violations, the evidence was weaker: the default judge achieved an AUROC of 0.435 across the 60 clips with assessable order, only five of which exhibited a violation. Appendices C.2 and D.3 provide the direct-label protocol and results.
4. What the post-training experiment shows
In Table 3, the scorer used during training assigned AG-P3T a higher mean Event Transition Score than CF++ (0.4038 versus 0.2905), with progression and outcome increasing while onset and VLD Quality decreased. Similarly, two blinded, AI-assisted rating rounds on 50 matched prompt pairs rated the clips labeled AG-P3T higher on progression and those labeled CF++ higher on visual quality. Appendix F.3 reports these paired results along with the checkpoint and sampling details that limit attribution to the adapter. Accordingly, we present AG-P3T as an exploratory test of using phase-specific evaluation as a training signal.