acceptodds
Under review as a conference paper at ICLR 2027

Ego2Act: Evaluating Goal-directed Manipulation in Egocentric Video Generation

Abstract

Video generation models are increasingly being explored as world simulators for embodied planning and learning. To do so effectively, these models must not only generate visually appealing frames, but also predict how environments dynamically evolve when executing goal-directed actions. While evaluating these capabilities is crucial, existing benchmarks focus mainly on single short actions or step-by-step instructions. This leaves multi-step physical reasoning underexplored, especially in egocentric video generation that requires planning to simulate proper execution to accomplish high-level goals by carrying out multiple real-world manipulations. We introduce Ego2Act, a goal-directed benchmark featuring 2,640 videos from 110 real-world tasks across day-to-day settings, varying object clutter and multi-step complexity. Given an initial scene image and a high-level goal, Ego2Act evaluates whether video generation models can produce realistic egocentric videos of a hand manipulating objects to carry out the task. To support scalable evaluation, we also introduce Ego2ActJudge, a reference-free evaluation pipeline that achieves better task completion and physics plausibility evaluation alignment with human consensus compared to relevant baselines. Our findings reveal that models' generated simulations often skip or partially execute steps, leaving later steps missing dependent states, which leads to unfulfilled goal. Furthermore, models consistently fail at fine-grained physical dynamics, particularly during complex object manipulation and persistent world modeling. We hope Ego2Act provides a rigorous testbed for advancing video models toward physically plausible, goal-directed simulation.

Primary area:
datasets and benchmarks
Keywords:
video generation, embodied agents, embodied manipulation, evaluations, generative model, egocentric, agents, agentic evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2fPjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Ego2Act, a benchmark for goal-directed egocentric video generation: given a real initial scene image and a high-level goal without intermediate steps, a video model must generate a first-person video of a hand completing a multi-step manipulation task. The benchmark contains 110 real-world cases in five everyday domains, each with three successful and three unsuccessful human recordings as controls, and is scored with a gated rubric that rates task completion (T1-T3) and physical plausibility (P1-P4) per subgoal and combines them by a geometric mean. The authors also propose Ego2ActJudge, a reference-free multi-stage VLM evaluator that reaches r = 0.69 with human consensus on a 600-video panel.

Critical strengths
  1. Evaluating egocentric video generation at the level of high-level goals, rather than step-by-step prompts or reference trajectories, is well motivated for using video models as embodied simulators. The paired successful and failed human recordings also give a useful sanity check for evaluators (e.g., VideoScore scores them almost identically, 87.5 vs. 88.2). This is the main reason I lean towards acceptance.

  2. The rubric and the judge are carefully validated: high inter-annotator agreement, comparison against six existing evaluators, gate ablations, and a transparent analysis of judge bias and repeatability.

Critical weaknesses
  1. The model coverage is limited. Only six generators are evaluated, of which only two are open-weight (MiniMax-H3 and Cosmos-3), and no model trained for egocentric manipulation is included, although EgoForge is noted to share the same generation interface. Adding more open-weight models (e.g., HunyuanVideo, open Wan releases) and egocentric world models would show whether the reported gap also holds for models designed for this setting.

  2. Generation is capped at 15 s (10 s for Wan-2.7), while successful human executions have a median of 20.4 s, and cases within the 15 s budget reach 68.6% Task completion versus 54.8% for longer ones. Part of the skipped or incomplete steps may therefore come from the length limit rather than planning; results on a duration-matched subset, or with longer generation via extension from the last frame, would clarify this.

Questions for authors
  1. (Clarification) The full-benchmark leaderboard relies on Ego2ActJudge, which over-scores Physics by 24.1 points and detects only 7-10% of teleport/swap corruptions (Table 20). Does denser frame sampling reduce this, and how stable is the ranking (e.g., Wan-2.7 is third on the full benchmark but fifth on the human panel, Table 9)?

  2. (Clarification) Unsuccessful human recordings receive a human Final score of 73.4, higher than every generator (Seedance-2.0: 64.0), since their Physics is near perfect. Could the authors also report a goal-level success rate, e.g., the fraction of videos in which all subgoals pass T3?

  3. (Clarification) MiniMax-H3's results are partly attributed to provider-side prompt expansion, and expanded prompts raise Wan-2.7 from 70.4 to 84.9 (Appendix E.4). Which providers rewrite prompts, and does the ranking hold when all models receive the same expanded prompt?

2 comments

We thank Reviewer 2fPj for the supportive review, and E1–E10 below refer to the table in the General Response.

W1: Model coverage

We agree and added two open-weight generators, run locally on all 110 cases with the same protocol (first frame plus goal, 832×480): Wan 2.2 TI2V-5B at its trained length of 5 s and LTX-2.3 (22B) at 15 s. This makes four of the eight generators open-weight:

GeneratorOpen-weightLengthTaskPhysicsFinalGoal success
Seedance-2.0no15 s80.091.183.444.4%
Kling-v3-Prono15 s63.788.972.325.9%
Wan-2.7no10 s67.680.371.226.9%
Grok-1.5no15 s67.579.269.225.5%
MiniMax-H3yes15 s53.476.159.017.0%
Cosmos-3yes15 s15.438.613.30.6%
LTX-2.3 22B (new)yes15 s17.127.612.30.6%
Wan 2.2 TI2V-5B (new)yes5 s9.212.73.80%

Both new models land with Cosmos-3 at the bottom and almost none of their videos reach the goal, so the gap to human executions holds for open-weight models too (Wan 2.2's 5 s is below even the fastest human execution, 5.2 s). The original six, rescored on the same cases, match the paper within 0.3 points, while 5 LTX-2.3 and 8 Wan 2.2 videos could not be scored. The egocentric EgoForge has no public code or model (per its project page), and we happy to circle back in the future if it is released.

W2: Duration confound

We agree duration matters (full analysis in our reply to kvWy, W2 and Q1). Fitting the budget is worth about 10 Task points, a third of the gap to Human+, but within budget most models still reach only 39% goal success vs. 84% for Human+ (E1). As suggested, we also tested a longer budget directly (E2). Wan 3.0 at 30 s instead of 15 s, with everything else fixed, does not noticeably improve the long cases (Task +4.1, goal success stays at 0%), and neither does the extension you suggest, where the open-weight LTX-2.3's official pipeline continues the same 15 s rollout to 30 s (Task −9.5, details in kvWy W2).

Q1: Denser sampling and ranking stability

Denser sampling does not reduce the over-scoring, as Physics bias, r and corruption detection stay within run-to-run noise (E4, table in our reply to UZtY Q3). The Wan-2.7 difference comes from the case set rather than from judge–human disagreement:

ViewWan-2.7 rankKendall τ vs. human panel
Human ratings, panel51.00
Judge, same panel videos50.87
Judge, non-panel cases30.60
Judge, full benchmark30.60

In short, humans and the judge agree on the same videos (both rank Wan-2.7 5th), and Wan-2.7 simply does better on the non-panel cases.

Q2: Goal-level success rate

Thank you, we adopt this metric (E7). A high Final for Human− is by design, because Physics scores the plausibility of what is shown, so a natural but incomplete execution keeps a high Physics score (App. B.2, B.4). Goal success separates the two:

Human+Human−SeedanceKlingGrokWan-2.7MiniMaxCosmos
Final (human, panel)–73.464.059.649.944.355.33.9
Goal success (human, panel)–1.6%25.8%10.0%7.8%10.8%19.6%0.0%
Goal success (judge, full)79.3%5.0%44.6%26.0%26.2%27.2%17.4%0.6%

Human− drops from a Final of 73.4 to 1.6% goal success, below every generator except Cosmos-3. Human+ was not rated by annotators (UZtY Q2), so it has no human goal-success value, and we will report goal success next to Final.

Q3: Provider-side prompt expansion

Closed APIs do not expose whether they rewrite prompts internally, so we cannot list which providers do, and instead we test sensitivity directly using Minimax-H3 expanded prompt (E10). Every generator gets the same goal prompt without model-specific rewriting (App. A.4), and since the App. E.4 diagnostic (5 cases × 3 prompts expanded by MiniMax-H3's built-in prompt-expansion module) was small, we added Kling-v3-Pro (mean Final):

ModelOriginalExpanded
Cosmos-325.225.2
Wan-2.770.484.9
Kling-v3-Pro (new)84.669.1

Expansion helps Wan-2.7, leaves Cosmos-3 unchanged and hurts Kling-v3-Pro, mainly through Task (77.3 → 59.1). Kling's API caps prompts at 2,500 characters, so expanded prompts were truncated (keeping 59–100% of the text, including the goal), and prompts that lost more text declined more (r = 0.48), so part of this drop likely reflects truncation. Since expansion moves models in different directions and can reorder them (Wan-2.7 overtakes Kling-v3-Pro), we motivate that prompt expansion is not always the canonical solution for this problem. We will add per-case results to App. E.4.

All of my concerns have been addressed. I will keep my positive score.

Reviewer_UZtYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a benchmark to assess the ability of sota video generation models to generate rollouts that show a complete and physically plausible execution of actions to achieve an intended goal in egocentric setups. Additionally they propose a VLM as a judge approach to assess such abilities at scale. The benchmark covers 110 cases (real scenarios), for each scenario they provide 6 real human executions videos (3 successes and 3 failures) and 18 generated videos (3 seeds x 6 models). The validation includes 6 generators, a panel of 600 videos (25 cases) annotated by humans and used for assessing judge-human alignment, baseline comparisons adapted to their Task/Physics scoring dimensions and several ablations. Their Ego2ActJudge reaches human agreement r = 0.69 against 0.42 for the best external baseline and 0.32 for a single-prompt VQA.

Overall i’m positive about this work: besides the clear limits of the proposed Judge (even if better than baselines) the benchmark containing also real positive and negative human recordings can be useful for testing future judges. It does not reach a score of 4 mainly because of the issues mentioned below regarding duration confound and weak human coverage.

Critical strengths
  1. The benchmark correctly identifies a need in evaluating generated videos conditioned on a goal. For these videos, visual quality, plausible physics and final state do not fully represent what we humans consider as a successfully executed task. It’s important to assess that the required state transitions, sometimes in different orders, have been executed.
  2. Including both positive and negative controls is particularly useful to see if the evaluator can separate task failure from physical issues. Ego2ActJudge correctly scores both Human+ and Human- with high and low T respectively while maintaining high P. No previous work to my knowledge includes both cases.
  3. The authors did a very extensive validation well beyond simple correlation metrics:
    • calibration
    • sequential vs independent gate ablations
    • physics corruption controls
Critical weaknesses
  1. Weak human coverage: human ratings cover only 25 of 110 cases, with only one heavy clutter case. Only about 220 of the 600 panel videos have two or more ratings. To my understanding Human+ videos are just assigned to maximum score rather than rated.
  2. Ego2ActJudge is still a weak tool to assess physical plausibility: it has a bias of 24 points compared to humans on P and fails almost completely in detecting temporal violations (teleport, swap ~7-10%).
  3. Duration: Human+ videos have a median length of ~20s , but most models generation is capped at 15s (10s for Wan). Cases that fit within 15s score 68.6 on T vs 54.8 for longer ones. Thus the finding that models “skip or partially execute steps” is partially confounded by the fact that the possible output might be too short for proper task execution.
Questions for authors
  1. Can the authors report results restricted to cases whose successful human executions fit the generative model budget? (<=10s for Wan and <= 15s for the rest). Can you also show where T0 failures happen?
  2. Can you clarify if Human(+) videos were actually rated or simply assigned 100? Are they included in the r, CCC and MAE of Table 3?
  3. Ego2ActJudge Physical scoring is weak and experiments show that it struggles with inter-frame evaluations and temporal corruptions. Did you evaluate any training-free (to stay within budget) approach to densify or structure temporal evidence? For example by using grids of consecutive lower-resolution frames. Testing a few of these approaches and reporting Physics bias, r and corruption detection rate would show whether the current P score is a limit of current VLMs or a fixable design choice of the sampling method.
1 comment

We thank Reviewer UZtY for the positive assessment, and E1–E10 below refer to the table in the General Response.

W1: Human coverage

We agree the panel is a validation subset of 25 of the 110 cases, with 5 per domain, all 24 videos per case and case-level bootstrap CIs (App. D.1). Human+ is not rated (Q2), and of the other 525 videos 44 were rated once, 257 twice and 224 three times. Agreement grows as the human reference gets less noisy, and holds on the 15 panel cases never used to develop the judge prompts (kvWy Q2):

Subsetn videosFinal r
All rated videos (Table 3)5250.69
≥ 2 raters4810.74
3 raters2240.82
15 held-out panel cases2390.63

We also agree the panel has only one heavy-clutter case, so to widen coverage we are rating 5 new held-out cases (80 videos, including heavy clutter and rated Human+), each by 3 annotators (E6), and will post the alignment here once done. We also want to emphasis that validating on a subset is a standard practice in related benchmarks:

BenchmarkHuman-rated subsetShare
ChronoMagic-Bench (arXiv:2406.18522)16 of 1,649 prompts~1%
T2V-CompBench (arXiv:2407.14505)651 of ~32,200 videos~2%
StoryEval (arXiv:2412.16211)~900 of ~4,650 videos~19%
PhyGenBench (arXiv:2410.05363)512 of 1,280 videos40%
Ego2Act (ours)525 of ~2,600 videos~20%

Our share sits inside the validated range to account for limited resources of both extensive judge and human evaluations, while also remaining following the standard practice of video generation evaluation. We will report the coverage of each statistic and comparison to similar benchmarks in the App. D.

W2: Physics scoring

We agree Physics is the weaker axis, and the paper reports both the offset and the temporal blind spot (Sec. 5.4, Table 20). The denser sampling strategies we tried (Q3) fix neither, so the gap is not simply about frame count, although other ways of improving temporal understanding remain open for future works as it is not within part of our scope of work.

W3: Duration

We agree the length limit affects current generators (as mentioned in Q1 & kvWy W2). Fitting the budget adds about 10 Task points after controlling for plan length, a third of the gap to Human+, while most failures are early skipped steps that truncation cannot explain. When we double only the budget (E2), neither Wan 3.0 nor the open-weight LTX-2.3 improves on the long cases (Task +4.1, and −9.5, with goal success at 0%), so duration alone does not explain the failures.

Q1: Duration-matched results and where T0 failures occur

Restricted to cases that fit each model's budget (E1, E7):

ModelTask (all → <=15s)Goal success (all →<=15s)
Seedance-2.080.0 → 90.944.4 → 72.2%
Kling-v3-Pro63.7 → 73.725.9 → 38.9%
Grok-1.567.5 → 73.525.5 → 38.9%
MiniMax-H353.4 → 68.217.0 → 38.9%
Human+93.9 → 95.079.3 → 83.8%

Only 4 cases fit Wan-2.7's 10 s budget, too few to rank. For T0 (skipped) failures, we located the first skipped subgoal in each plan:

PatternAll videosWithin budget
Skips an earlier step, then attempts a later one80.5%77.7%
Skips only trailing / last steps10.5%8.3%
First skipped subgoal in first third of plan66%63%
First failure of any kind in first half of plan79.5%74.4%

Most T0 failures fall early and are followed by later steps, which running out of time cannot cause and we will add both tables to the main paper.

Q2: Were Human+ videos rated?

No. Human+ recordings are verified successful executions, so they enter the human column with the maximum score and are not part of r, CCC or MAE in Table 3, which use the 525 rated generated and Human− videos as mentioned in Table 2’s footnote.

Q3: Training-free temporal densification

Yes, on the 120-video corruption set and the panel (E4) we tested 48 uniform frames and grids of 6 consecutive low-resolution frames:

RunPhysics biasPhysics rTeleportSwapGhostFalse alarms
Production (paper)+24.10.556.9%10.3%55.2%0%
Baseline rerun (24 frames)+23.30.5126.7%20.0%69.0%13.3%
48 uniform frames+23.60.5721.4%21.4%57.1%10.7%
Consecutive-frame grids+25.00.516.9%31.0%75.9%10.3%

Neither variant improves Physics bias or r over the baseline rerun (every 95% CI of the difference includes 0), and detection moves within run-to-run noise, since two runs of the same judge detect 6.9% and 26.7% of teleports. Ghosts stay easy while teleports and swaps stay hard under every scheme, so denser or grid sampling alone does not close this gap, though other ways of improving temporal understanding remain open. We will add all variants to the App. G and revise its claim of this evidence density.

Reviewer_kvWyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EGO2ACT, a benchmark for evaluating whether video generation models can carry out multi-step manipulation tasks from an egocentric view. Each example provides an initial image and a high-level goal, leaving the model to infer the intermediate actions. The benchmark contains 110 real-world tasks, human demonstrations, and outputs from six video generators. The paper also proposes EGO2ACTJUDGE, which decomposes a goal into subgoals and separately evaluates task completion and physical plausibility. The experiments show that current models often skip required steps, fail to preserve object states, or produce invalid contact-rich interactions. The proposed judge agrees with human ratings better than the evaluated baselines, although it still tends to overestimate performance.

Critical strengths
  1. The benchmark addresses an important setting that is not well covered by existing evaluations. Using a high-level goal rather than a step-by-step instruction makes it possible to test whether a model can infer and execute a complete action sequence. The egocentric viewpoint also exposes hand-object and object-persistence failures that may be less visible in third-person videos.

  2. The evaluation design make snese. Separating task completion from physical plausibility is useful, because a video can complete the apparent task through impossible interactions. The human controls, multiple generation trials, baseline comparisons, ablations, and detailed analysis of judge reliability make the empirical study informative.

Critical weaknesses
  1. Eeo2AactJudge is useful for aggregate comparisons, but it is still not very reliable for individual videos or subtle physical errors. I found that its overall correlation with humans is r=0.69, but it has a 22-point MAE and overestimates Physics by about 24 points. Its Physics score also has substantial run-to-run variation, and the controlled experiments detect only 7–10% of temporal cuts or reordered segments. Since the benchmark’s main conclusions rely heavily on this judge over the full dataset, these limitations make close model comparisons difficult to trust.

  2. The generation-length limit is a confounding factor. Successful human recordings have a median duration of 20.4 seconds, while most models receive at most 15 seconds and Wan receives only 10 seconds. A model may therefore omit actions simply because the required sequence does not fit within its output budget. This makes it difficult to separate planning failures from limitations of the generation interface.

Questions for authors
  1. How should incomplete executions be interpreted when the successful human demonstrations are often longer than the maximum generation duration? Could the authors report results separately for tasks whose human executions fit within each model’s time budget?

  2. Were the Eeo2ActJudge prompts, including the reported V7 prompts, developed or revised using the same 25-case human-evaluation panel? If so, is there a separate held-out set for measuring final human alignment?

  3. Why is the subgoal plan regenerated independently for every video rather than fixed once per case? The appendix shows that changes in the number of subgoals can change Task scores by around 11 points. Would the model ranking remain the same with a fixed, verified plan for each case?

  4. How many of the 600 videos received one, two, or three human ratings? Since agreement is computed on only about 221–224 videos with multiple ratings, does the reported evaluator alignment remain similar when restricted to that subset?

1 comment

We thank Reviewer kvWy for the careful review and are happy to address the concerns below. E1–E10 refers to the table in the General Response.

W1: Judge reliability for individual videos and close comparisons

We agree the judge is meant for comparing models and not verifying single clips (App. G; we will move this to Sec. 5). Its Final r = 0.69 with the human consensus is close to one annotator vs. the rest (0.76, Table 21), and its generator ranking matches humans (ρ = 0.94). A case-level bootstrap (E3) keeps Seedance-2.0 1st and MiniMax-H3 5th in 100% of resamples while Kling-v3-Pro, Wan-2.7 and Grok-1.5 tie for ranks 2–4.

For Physics, denser temporal evidence (48 frames or frame grids, E4) changes bias and r no more than two runs of the same judge differ, and detection stays low (UZtY Q3), so the strategies we tested do not explain the bias though other ways of improving temporal understanding remain open.

W2: Generation-length confound

We agreed the length limit matters, so we measured how much. After controlling for plan length, fitting the budget is worth about 10 Task points, a third of the gap to Human+ (E1). It does not explain the rest: on cases that fit the budget, most models reach only 39% goal success vs. 84% for Human+ (E7), and 80% of videos that miss an innate planning step (skips to latter step) of which a limited budget cannot produce.

We also changed only the budget (E2) on selected 6 cases (where humans need ~25–30 s to finish) × 3 seeds, same start frame, prompt and seed. We use 2 different models, representative from closed + open models that have capability to be set differing maximum generation limits. Wan 3.0 generates both lengths (15s and 30s) natively, and open-weight LTX-2.3 has an official long-video pipeline / framework that extends the identical 15s rollout to 30s. Below is the result on both methods:

Model, casesTask Score, 15 s → 30 sGoal successSubgoals done
Wan 3.0, need ~30 s30.4 → 34.40% → 0%0.8 → 0.8
LTX-2.3, need ~30 s15.4 → 5.90% → 0%0.6 → 0.1

In the long cases neither improves meaningfully (ΔTask +4.1 for Wan & −9.5 for LTX-2.3). Wan 3.0 moves 70% longer (21.1 vs. 12.4 s) for the same steps; LTX-2.3's continuation drifts and undoes completed steps. LTX-2.3 is weak even at 15 s, so it mainly shows that continuing does not rescue a rollout.

Hence, with either model or extension method of adding time (native 30 s or continuing the same rollout), a longer budget does not meaningfully improve task completion.

Q1: Interpreting incomplete executions

In cases longer than the budget, an incomplete execution is an upper bound on planning failure as shown in E1,E7, and W2. As native lengths grow, Ego2Act can test how good future models plan over the longer horizon. Within the budget, failures cannot come from the length limit and remain frequent.

ModelTask (all → <=15s)Goal success (all → <=15s)
Seedance-2.080.0 → 90.944.4 → 72.2%
Kling-v3-Pro63.7 → 73.725.9 → 38.9%
Grok-1.567.5 → 73.525.5 → 38.9%
MiniMax-H353.4 → 68.217.0 → 38.9%
Human+93.9 → 95.079.3 → 83.8%

Q2: Judge prompts and the human panel

Yes, in part. The backbone was chosen by a 5-fold case-level CV (App. D.2); the prompts were refined on 150 videos from 10 of the 25 panel case, thus leaving 15 cases as the held-out.

Panel subsetJudge rBest baseline r
All panel cases (Table 3)0.690.42
10 development cases0.760.51
15 held-out cases0.630.33

Q3: Regenerating the subgoal plan for every video

The plan sees only the goal and start frame, never the video, so enable the subgoal generation capability to support future user if they want to use a new video / goal as part of the input, since we are proposing Ego2ActJudge for general evaluation method as well rather only equipped for Ego2Act benchmark. Despite that, we do agree to support a more deterministic use case, the released framework now can cache initially generated plans per case. We fixed one verified plan per case and rerunning the unchanged judge as a noise baseline (E9):

PlanKendall τ vs. currentMean abs. ΔTask
Regenerated plan0.876.9
Fixed plan1.007.7

Fixing the plan leaves the ranking unchanged, hence the planning regeneration does not shift the judgement significantly.

Q4: Number of ratings and alignment on multi-rated videos

Human+ (75 videos) is not rated by design, since annotation verifies that these executions succeed (UZtY Q2). Of the other 525, 44 rated once, 257 twice, 224 three times. Alignment is stronger on multi-rated videos.

SubsetnrMAE
All rated (Table 3)5250.6922.0
≥ 2 raters4810.7421.6
3 raters2240.8218.1

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Rebuttal Response

General Response

We thank all reviewers for their thoughtful and constructive reviews. We are delighted that reviewers found that:

  • The benchmark targets an important, under-covered setting: goal-directed, multi-step egocentric generation from a high-level goal rather than step-by-step prompts (kvWy, UZtY, 2fPj).
  • The paired successful/unsuccessful human recordings are a valuable control that exposes evaluators' blindness to task failure; “no previous work … includes both cases” (UZtY, 2fPj).
  • Separating task completion from physical plausibility is well motivated (kvWy, UZtY).
  • The validation is extensive and transparent: inter-annotator agreement, six baselines, gate ablations, physics-corruption controls, and an explicit analysis of judge bias and repeatability (kvWy, UZtY, 2fPj).
  • The paper is clear (kvWy, 2fPj).

The reviews share three concerns: the generation-length limit, judge reliability (especially Physics), and human-study coverage. Thus, we ran 10 extra experiments below where each row gives the answer and details relating to answering the concerns. The individual replies refer to these IDs.

IDQuestionAnswerDetails
E1Does the 15 s limit explain the failures?No. Fitting the budget is worth ~10 Task points, a third of the gap to Human+. Within this extra budget most models still reach only 39% goal success vs. 84% for Human+, and 80% of videos that skip a step go on to a later one, which is not the cause of limited time window, but rather planning and state tracking failure.kvWy W2, UZtY Q1
E2Does a longer budget fix them?No. Same cases, prompt and seed at 15 s vs. 30 s: Wan 3.0 (native 30 s) Task +4.1, LTX-2.3 (continuing the same rollout) −9.5; goal success 0% for both on the long cases.kvWy W2
E3Is the ranking stable?Yes. Seedance-2.0 1st, MiniMax-H3 5th, Cosmos-3 6th in 100% of case-bootstrap resamples; Kling-v3-Pro, Wan-2.7, Grok-1.5 tied for rank 2-4. On the panel videos the judge ranks Wan-2.7 5th, as humans do.kvWy W1, 2fPj Q1
E4Does denser temporal evidence help Physics?No. 48 uniform frames or consecutive-frame grids stay within run-to-run noise (bias +23 to +25).UZtY Q3
E5Rating coverage; alignment on multi-rated videosHuman+ is not rated by design and the other 525 videos are rated. Final r rises 0.69 → 0.74 (≥ 2 raters) → 0.82 (3 raters).kvWy Q4, UZtY Q2
E6Alignment on cases never used to develop the judge15 panel cases never used for prompt development: r = 0.63 vs. 0.33 for the best baseline (Simple VQA). 5 new held-out cases (80 videos, 3 annotators each) are in progress being rated and we will post results in upcoming days.kvWy Q2, UZtY W1
E7Goal-level successHuman+ 79%, best model 44%, Human− 5% (judge, full benchmark).2fPj Q2
E8More open-weight generatorsWan 2.2 TI2V-5B and LTX-2.3 land at the bottom together with prior open model Cosmos-3-Nano (Final 3.8 and 12.3).2fPj W1
E9One fixed & verified plan per caseRanking unchanged (Kendall τ = 1.00).kvWy Q3
E10Same expanded prompt for every modelMinimax-H3 expanded prompt moves models in arbitrary opposite directions (Wan-2.7 +14.5, Kling-v3-Pro −15.5), so prompt expansion does not guarantee better performance.2fPj Q3

Takeaway. The gap between current generators and human executions holds after controlling for the length budget, judge noise, plan sampling and rater coverage.

Changes we will make to the paper: duration-matched results, goal success and T0 failure positions in the main text (E1, E7); rank intervals with tied models shown as tied (E3); densification ablations in App. G (E4); rating coverage for every statistic and the held-out alignment in App. D (E5, E6); the two new generators (E8); fixed-plan and prompt results (E9, E10).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.