acceptodds
Under review as a conference paper at ICLR 2027

Beyond Plausible Motion: Benchmarking and Improving Ordered Event-Transition Generation in Few-Step Autoregressive Video Models

Abstract

Few-step autoregressive (AR) video models enable low-latency streaming generation by predicting future frame chunks conditioned on previously generated content, but this chunk-wise formulation makes video temporal transition ambiguous. Existing metrics evaluate perceptual quality, multi-event completion, but they do not explicitly resolve realization of a single short-horizon event transition. To bridge this gap, we introduce **OETGBench**, an **O**rdered **E**vent-**T**ransition **G**eneration **Bench**mark designed to evaluate whether a short-horizon T2V generation visibly progresses from an initial state to a target outcome in the intended temporal order. OETGBench contains 2,500 curated tasks spanning four broad event-transition families, with each task decomposed into ordered onset, progression, and outcome phases to assess short-video models' ability to complete requested events in the correct temporal order. Alongside OETGBench, we propose *OETGEval*, a hierarchical evaluation framework combining video-level diagnostics with window-level phase verification of onset, progression, and outcome in their intended temporal order. Finally, we demonstrate that these evaluation signals are actionable through *Anchor-Guarded Phase-Potential Post-Training* (AG-P3T), a reward-guided LoRA method that converts OETG evidence into anchor-relative rewards and permits positive updates only when ordered event completion improves without violating quality or shortcut guardrails. Experiments reveal that our methods ensure alignment with human evaluations and reveal distinct failure profiles among generators with similar event scores. AG-P3T improves the Event Transition Score while maintaining comparable VBench performance, demonstrating the utility of OETGEval feedback for post-training.

Primary area:
datasets and benchmarks
Keywords:
video generation, few-step autoregressive models, ordered event-transition generation, video generation evaluation, benchmarks, reward-guided post-training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BdxiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether few-step autoregressive video models show the start, progress, and result of a requested event in the correct order. It introduces OETGBench, with 2,500 tasks, and OETGEval, which combines video diagnostics with phase checks in overlapping windows. Ordered matching separates phase presence from coarse temporal order. The authors also test AG-P3T, a post-training method with anchor-relative penalties and positive-advantage gating.

Critical strengths
  • The task definition provides a useful diagnostic framework. The paper separates onset, progression, and outcome, and reports phase evidence, order, quality, and failures. Generation prompts are separate from evaluator-side specifications. The shared phase-window representation and ordered matching provide a useful, incremental extension to existing transition evaluation.

  • The same-video analysis supports separate evaluation of event completion and consistency. Appendix E.2 reports a negative correlation of −0.360, with a 95% interval of [−0.555, −0.142]. The result persists after removing flicker. The study includes three generators, three annotators, three VLMs, and controlled perturbations. The authors explain metric limits and avoid a causal interpretation.

Critical weaknesses
  • Task quality and evaluation coverage need stronger support. Section 3.1 mentions human inspection. Appendix A does not report the audit size, error rate, or corrections. Automatic checks cannot establish whether phase targets are distinct or feasible within the video duration. Ambiguous specifications could affect the reported failure rates. The main comparison covers 50 tasks from a 2,500-task corpus, with one video per generator per task. The paper needs to explain how these tasks were selected and what range of events they cover. Inter-annotator agreement is also missing.

  • Post-training gains need independent confirmation. Training and evaluation use the same event scorer. The human study excludes AG-P3T. Thus, the event-score increase from 0.2905 to 0.4038 in Table 3 lacks independent confirmation.

    Relative to CF++, onset drops from 0.500 to 0.400, VLD Quality from 0.7090 to 0.6463, and VLD Failure rises from 0.0375 to 0.0738. The similar VBench Total comes from a separate prompt suite. Table 5 provides cumulative ablations without variation across training runs. A blind comparison on matched, unseen prompts is needed to verify event gains and assess quality changes.

Questions for authors
  • Task quality and coverage: How many tasks were checked by humans, and what errors were found? Report inter-annotator agreement and the selection procedure for the 50 evaluation tasks. Does this subset cover the four task families and their main transition types?

  • Post-training gains: Can a blind human assessment compare CF++ and AG-P3T on the same unseen prompts and matched generation seeds? Report event completion, order, and visual quality. Clarify test-set separation and the stability of the ablation results across training runs.

  • AR-specific interpretation: Is the paper documenting failures in few-step AR models, or attributing them to autoregression or distillation? The latter requires a matched non-AR or teacher-model comparison. Please separate the observed failure from its proposed cause.

Minor Questions

  1. State train, validation, and test counts in the PDF.
  2. State the maximum event score of 0.80 near the main definition and in relevant table captions. Explain the role of the nonbinding caps.
  3. Replace the abstract's claim to “ensure alignment” with a statement of moderate, phase-dependent human agreement. Keep the VBench claim limited to that suite.
Reviewer_7ZttOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces OETGBench, which describes short events through onset, progression, and outcome. OETGEval combines video-level diagnostics with phase verification in four overlapping windows to produce phase-presence and coarse-order scores. The authors compare three few-step autoregressive video generators and use the same evaluation signal for AG-P3T, a LoRA post-training method with anchor penalties and positive-advantage gating.

Critical strengths
  • The studied topic is useful for the video generation community.
Critical weaknesses
  • The benchmark construction and validation provide insufficient evidence that the 2,500 tasks are genuinely high-quality and semantically diverse. There is no convincing validation of whether the generated onset/progression/outcome annotations are correct and unambiguous.
  • OETGEval has only moderate, and sometimes even weak, agreement with human judgments, undermining its use as the primary evaluation. This is particularly concerning because both the benchmark conclusions and AG-P3T optimization depend heavily on these evaluation results. An AUROC of 0.708 and temporal-order correlation around 0.4 still leave substantial noise in evaluation.

I am open to increasing the score if the concerns are tackled.

Questions for authors
  • Why do you divide event transitions into the four task-oriented categories mentioned in the paper? I cannot find any explanations or references about it.
  • A lot of details of benchmark construction are missing, including
    • How do you ensure that selected temporally explicit tasks are restricted to one dominant event, observable within approximately five seconds, and matching the experimental setting?
    • How do you rank the prompt candidates by event-related cues and textual conciseness?
    • How do you use DeepSeek-V4-Pro to judge suitability?
    • How do you construct the event specification for each task?
  • How do you determine the coefficients of VLD quality and failure scores?
  • Please share the insights behind choosing the VLD criteria, including 4 quality-related criteria and 5 failure-related criteria.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 7Ztt

W1: The benchmark construction and validation provide insufficient evidence that the 2,500 tasks are genuinely high-quality and semantically diverse. There is no convincing validation of whether the generated onset/progression/outcome annotations are correct and unambiguous.

A1: Thank you for your valuable feedback. We have carefully considered your comments and provide detailed responses below:

1. Task-specification audit

We audited 100 written task specifications, sampling 25 from each of the four event families. We invited three graduate students as reviewers to conduct the human scoring. The table gives the number of tasks each reviewer marked Yes. Three-way agreement is the percentage of tasks on which all three reviewers assigned the same label (Yes, No, or Uncertain).

Audit criterionReviewer 1Reviewer 2Reviewer 3Three-way agreement
One dominant event92919193%
Feasible within about five seconds81808686%
Observable onset99979796%
Observable progression96969796%
Observable outcome95949693%
Distinct, ordered phase targets97989795%
Evaluation questions visually answerable and aligned with the target72666677%

Across the three reviews, 97–99 tasks passed the onset check, 96–97 passed progression, and 94–96 passed outcome. Each reviewer marked six tasks infeasible within five seconds and a further 8–14 uncertain. The clearest weakness was in the evaluation questions. Even when the phase targets were distinct and ordered, reviewers could find that a question asked for evidence beyond what the prompt specified. We will report the criterion-level counts in Appendix A.1. These results describe the balanced 100-task sample.

W2: “OETGEval has only moderate, and sometimes even weak, agreement with human judgments, undermining its use as the primary evaluation. This is particularly concerning because both the benchmark conclusions and AG-P3T optimization depend heavily on these evaluation results. An AUROC of 0.708 and temporal-order correlation around 0.4 still leave substantial noise in evaluation.”

A2: Thank you for your valuable feedback. We have carefully considered your comments and provide detailed responses below:

1. Potential impact of model capacity

As Reviewer MYxU also suggested, the modest agreement with human judgments may be partly attributable to the capacity of the underlying models. In future work, we plan to explore more frontier models to examine whether stronger judge capabilities can further improve alignment with human judgments.

2. All rating sources placed CF++ last on phase presence

Table 2 reports moderate agreement with the original retrospective order reference (Spearman ρ = 0.406, AUROC = 0.708). To test whether evaluator choice alters our benchmark conclusions, we compared the three generators across the same 37 prompts for which every generator received determinate phase ratings in all three reassessments (111 clips). Both the original ratings and each of the three human-reviewed reassessments yielded the exact same ranking by mean phase presence (Self-Forcing > CausVid > CF++). Across the three VLM judges, CF++ likewise had the lowest mean phase presence, although their ranking of the other two generators varied (Appendix E.1).

3. Direct review identifies what the generators missed

In the second reassessment of the 150 baseline clips, 55 showed the visible phases in the requested order; however, 47 of these still fell short of completing the event. This finding provides a concrete reason to score event completion separately from relative order. For detecting order violations, the evidence was weaker: the default judge achieved an AUROC of 0.435 across the 60 clips with assessable order, only five of which exhibited a violation. Appendices C.2 and D.3 provide the direct-label protocol and results.

4. What the post-training experiment shows

In Table 3, the scorer used during training assigned AG-P3T a higher mean Event Transition Score than CF++ (0.4038 versus 0.2905), with progression and outcome increasing while onset and VLD Quality decreased. Similarly, two blinded, AI-assisted rating rounds on 50 matched prompt pairs rated the clips labeled AG-P3T higher on progression and those labeled CF++ higher on visual quality. Appendix F.3 reports these paired results along with the checkpoint and sampling details that limit attribution to the adapter. Accordingly, we present AG-P3T as an exploratory test of using phase-specific evaluation as a training signal.

AuthorsOpenReview ↗

Response to Reviewer 7Ztt

Q1: Why do you divide event transitions into the four task-oriented categories mentioned in the paper?

A1: Thank you for your valuable feedback. We have carefully considered your comments and provide detailed responses below:

1. Operational purpose of the four families

We group tasks according to what visibly changes: an action begins, two entities interact, an object changes state, or the environment changes. We defined these four families to characterize task coverage and to sample tasks from each family. Appendix A.2 explains the assignment rules; when a prompt fits more than one family, we categorize it based on its dominant change. T2V-CompBench likewise evaluates action, interaction, and attribute changes as distinct aspects of generation, and our four families apply a related distinction to the visible transition within a single event.

Q2: “A lot of details of benchmark construction are missing, including...

A2: Thank you for your valuable feedback. We have carefully considered your comments and provide detailed responses below:

1. Single-event and five-second acceptance criteria

At 16 FPS, the first and last of 81 frames span a five-second interval. Appendix A.1 requires the onset, progression, and outcome of a single event to be visible within this interval in a single scene, while the instructions defer prompts that rely on unrelated actions, a time jump, or camera motion alone. In the 100-task audit, 91–92 tasks passed the single-event check and 80–86 passed the five-second feasibility check across the three records.

2. Candidate ranking

The Appendix A.1 rules retain 181,829 of 1,038,805 sources. Each of the first six distinct action or interaction cues adds four points, whereas each cue beyond eight deducts five points. Prompts of 25–100 words earn eight points, while those of 12–24 or 101–140 words earn four; each camera or static/style term deducts three points. Ties favor shorter text, followed by the source-row index, with the top 12,000 forming the candidate pool.

3. DeepSeek-V4-Pro suitability assessment

DeepSeek-V4-Pro received the source prompt and target duration and returned either ACCEPT or DEFER. Of 5,990 requests, 3,501 yielded accepted rewrites, 2,076 were deferred, and 413 failed. Automated checks retained 2,935 records—all unique under exact matching—and a shuffle with seed 20260604 selected the final 2,500, with generator scores playing no role in selection. Appendix A.1 provides the instructions and checks.

4. Event-specification construction

The generator receives only the prompt, whereas the evaluation specification separately records the entities, initial state, transition, and phase targets. A subsequent pass converts those targets into questions while keeping the prompt and targets fixed (Appendix A.1). In the example in Appendix A.3 (TDV3-0657846), the first step marks onset, the approach with open arms marks progression, and the embrace marks outcome; thus, a clip showing only the embrace supplies evidence for the final phase.

AuthorsOpenReview ↗

Response to Reviewer 7Ztt

Q3: “How do you determine the coefficients of VLD quality and failure scores?”

A3: Thank you for your valuable feedback. We have carefully considered your comments and provide detailed responses below:

1. How the coefficients relate to equal weighting

We use fixed, heuristic weights. For Quality, motion and consistency each receive half of the total weight: subject and background consistency receive 0.25 each, whereas within motion, plausibility receives 0.30 and continuity receives 0.20 because a movement can be smooth yet inconsistent with the requested event. For Failure, three criteria retain the equal weight of 0.20, while motion collapse receives 0.25 and temporal artifacts receives 0.15 because a frozen clip can omit the transition entirely, whereas a brief flicker can occur during an otherwise complete event.

Both formulas thus differ from a simple average by a single adjustment of 0.05 times the difference between two item ratings. Since each rating lies in [0, 1], switching to equal weights changes either diagnostic by at most 0.05 points for any single clip. Note that the Event Transition Score is computed separately from both VLD diagnostics (Appendix B.2).

2. Does the generator comparison depend on these weights?

We rescored the same 50 matched Self-Forcing and CF++ prompt pairs with all item ratings held fixed, comparing the default weights, equal weights, and 18 settings that varied one coefficient by ±20% before renormalization. Across all 20 settings, each of the three VLM judges consistently assigned Self-Forcing higher mean Quality and lower mean Failure than CF++. For example, with Qwen2.5-VL, switching from default to equal weights shifted the mean Quality gap from +0.1115 to +0.1125 and the mean Failure gap from −0.0195 to −0.0160 (Self-Forcing minus CF++). We will report these sensitivity results and task-paired intervals in Appendix B.2.

Thank you for your suggestions. If you have further questions or require additional clarification, please feel free to reach out.

Reviewer_MYxUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents OETGBench, around observable onset, progression, and outcome of a goal-directed event transition which comprises 2.5K curated tasks. Authors also provide OETGEval combining video-level diagnostics (for visual quality) and window-level phase verification (for checking phase completion and temporal order). Finally authors present AG-P3T, a reward-guided LoRA method. The paper compares three base models, presents human/VLM agreement and reports score improvement after post-training.

Critical strengths
  • Paper tackles an important failure mode in recent streaming video models (many claim that are infinite, but most are just mere continuation without phased progressions). I agree that benchmarking and evaluating this gap properly is useful beyond measuring simple perceptual qualities.

  • I think diagnostic analysis could be helpful to the community. I also appreciate the transparency of releasing the benchmark itself as part of the supplementary.

Critical weaknesses
  • Authors mention that DeepSeek-V4-Pro are used to create text for the benchmark and mid-tier Open Source Models such as Qwen2.5-VL-7B, Qwen3-VL-8B, InternVL3.5-8B are used for evaluation (OETGEval). Using open source models are great, but for a 2.5K large scale actual benchmarking samples, I am not sure if they are great enough to evaluate them. For example, the appendix shows that all 0.5 phase matrix gets full score due to tie-breaking. To think about costs, generating 2.5K video sample already costs significant amount of GPU compute, and I find no reason not to use frontier closed models in this case (GPT/Claude) or much larger frontier open models such as Kimi. I also think Spearman ρ to be quite low. Could this be due to capability of small open models?

  • For a paper to be accepted purely with benchmark, I would expect more (human) efforts. For example, human order reference (according to the appendix), is retrospectively constructed from positive phase ratings and the absence of an explicit order-violation note – which is a weak reference. I think proper benchmark for ordered transition would require independently annotated order labels and more human labeling + use of frontier models (for best quality).

  • Simply looking at Table 3, I’m a less convinced that AG-P3T post training improves ordered generation while preserving the quality. Metricwise, VBench metrics decrease and some of OETGEval Scores also decrease.

Questions for authors

Please refer to the weakness sections.

2 comments
AuthorsOpenReview ↗

Response to Reviewer MYxU

W1: For example, the appendix shows that all 0.5 phase matrix gets full score due to tie-breaking. … I find no reason not to use frontier closed models in this case (GPT/Claude) or much larger frontier open models such as Kimi. I also think Spearman ρ to be quite low. Could this be due to capability of small open models?

A1: Thank you for your valuable feedback. We have carefully considered your comments and provide detailed responses below:

1. Choice of judges and interpretation of Spearman ρ

Thank you for raising this important question. We agree that model capacity is a plausible contributor to the modest Temporal Order correlation (Spearman ρ = 0.406, Table 2). We selected three publicly available 7–8B judges so that researchers can readily reproduce the scores and compare judgments on the same clips, though we acknowledge that their similar sizes leave the effect of scaling open. Rather than relying on a single judge, OETGEval integrates multiple Vision-Language Models (VLMs) to achieve more reliable judgments and help reduce VLM hallucination when recognizing event transitions. Finally, we note that the reported ρ measures agreement with the original retrospective order reference, which we directly reassess in A2 below.

2. Impact of Tie-Breaking on Event Scoring

The reviewer's observation is right: an all-0.5 matrix indeed receives full credit on the Temporal Order metric, yielding an Event Transition Score of 0.40 out of 0.80. We held all 5,400 phase answers fixed and modified only the tie-breaking rule—selecting the chain with the lowest Temporal Order score among equally supported sequences. This adjustment changed the Temporal Order results on 39/150, 17/150, and 8/150 videos for Qwen2.5-VL, Qwen3-VL, and InternVL3.5, respectively. Across all 50 prompts, all three judges maintained the same ranking by mean Event Transition Score under both tie-breaking rules (Self-Forcing > CausVid > CF++). We will include this sensitivity analysis in Appendix B.4.

3. Which generator comparisons held across judges?

Across the 37 prompts with determinate phase ratings for every generator in all three reassessments (111 clips), both the original ratings and all three human ranked mean phase presence in the same order (Self-Forcing > CausVid > CF++). All three VLMs likewise placed CF++ last, although they differed on the top two models. Across these sources, CF++ consistently showed the least phase evidence on these prompts. Appendix E.1 reports this matched comparison.

W2: … human order reference (according to the appendix), is retrospectively constructed from positive phase ratings and the absence of an explicit order-violation note – which is a weak reference. I think proper benchmark for ordered transition would require independently annotated order labels and more human labeling …

A2: Thank you for your valuable feedback. We have carefully considered your comments and provide detailed responses below:

1. Direct judgments of order and completion

We agree that the absence of an order-violation note provides insufficient evidence to label a video as correctly ordered. We therefore reassessed all 150 baseline videos, asking separately whether the visible phases followed the requested order and whether the event reached completion in that order. We invited three graduate students as reviewers to conduct the human scoring. In the second record, 55 videos displayed the requested order, five showed an order violation, and 90 lacked sufficient evidence to judge order. Of the 55 correctly ordered videos, only eight completed the event; in the remaining 47, the visible phases followed the requested sequence, but the event remained incomplete. Appendices C.2 and D.3 provide the protocol and counts.

2. Agreement with the direct labels

Across the 60 videos with assessable order, correlations among judges ranged from −0.069 to 0.118. The default judge achieved an AUROC of 0.435 (95% CI [0.164, 0.757]) for relative order, and 0.716 (95% CI [0.499, 0.906]) for complete ordered success across 147 videos with determinate completion labels. With only five observed order violations, the estimate for detecting such errors remains imprecise. Appendix D.3 reports these two endpoints separately.

3. Audit of the written tasks

We also audited 100 task specifications (25 per event family). Across the three human-checked records, 97–98 of the 100 tasks had distinct, ordered phase targets. However, we also found evaluation questions that introduced a requirement absent from the generation prompt—for example, one office-entry task required the door to close after entry, even though the prompt requested only that the person enter. We will include this audit and illustrative examples in Appendix A.1.

AuthorsOpenReview ↗

Response to Reviewer MYxU

W3: Simply looking at Table 3, I’m a less convinced that AG-P3T post training improves ordered generation while preserving the quality. Metricwise, VBench metrics decrease and some of OETGEval Scores also decrease.

A3: Thank you for your valuable feedback. We have carefully considered your comments and provide detailed responses below:

1. While visual quality exhibits minor fluctuations, AG-P3T effectively improves ordered event generation. In Table 3, relative to CF++, AG-P3T scored higher on progression and outcome, but lower on onset and VLD Quality; on the separate VBench suite, the Total score was 81.32 for AG-P3T versus 81.85 for CF++. We will revise the text to report these changes separately and present AG-P3T as an exploratory use of evaluation feedback.

2. To further examine this pattern on matched prompts, Appendix F.3 reports two blinded human scoring rounds on the same 100 clips. Under our model mapping,both rounds rated all three phases higher on average for AG-P3T, while rating visual quality higher for CF++: the mean progression advantage was 0.20 and 0.16 points on the 0–1 scale, whereas the visual quality advantage for CF++ was 0.70 and 0.62 points on the 1–5 scale. Moreover, complete ordered success reached 7/50 versus 3/50 in the first round and 11/50 versus 3/50 in the second, yielding paired differences of +8 percentage points (95% CI [−2, +20]) and +16 percentage points (95% CI [+6, +26]), respectively. Appendix F.3 documents the basis for the model labels.

Thank you for your suggestions. If you have further questions or require additional clarification, please feel free to reach out.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.