- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces Ego2Act, a benchmark for goal-directed egocentric video generation: given a real initial scene image and a high-level goal without intermediate steps, a video model must generate a first-person video of a hand completing a multi-step manipulation task. The benchmark contains 110 real-world cases in five everyday domains, each with three successful and three unsuccessful human recordings as controls, and is scored with a gated rubric that rates task completion (T1-T3) and physical plausibility (P1-P4) per subgoal and combines them by a geometric mean. The authors also propose Ego2ActJudge, a reference-free multi-stage VLM evaluator that reaches r = 0.69 with human consensus on a 600-video panel.
Critical strengths
-
Evaluating egocentric video generation at the level of high-level goals, rather than step-by-step prompts or reference trajectories, is well motivated for using video models as embodied simulators. The paired successful and failed human recordings also give a useful sanity check for evaluators (e.g., VideoScore scores them almost identically, 87.5 vs. 88.2). This is the main reason I lean towards acceptance.
-
The rubric and the judge are carefully validated: high inter-annotator agreement, comparison against six existing evaluators, gate ablations, and a transparent analysis of judge bias and repeatability.
Critical weaknesses
-
The model coverage is limited. Only six generators are evaluated, of which only two are open-weight (MiniMax-H3 and Cosmos-3), and no model trained for egocentric manipulation is included, although EgoForge is noted to share the same generation interface. Adding more open-weight models (e.g., HunyuanVideo, open Wan releases) and egocentric world models would show whether the reported gap also holds for models designed for this setting.
-
Generation is capped at 15 s (10 s for Wan-2.7), while successful human executions have a median of 20.4 s, and cases within the 15 s budget reach 68.6% Task completion versus 54.8% for longer ones. Part of the skipped or incomplete steps may therefore come from the length limit rather than planning; results on a duration-matched subset, or with longer generation via extension from the last frame, would clarify this.
Questions for authors
-
(Clarification) The full-benchmark leaderboard relies on Ego2ActJudge, which over-scores Physics by 24.1 points and detects only 7-10% of teleport/swap corruptions (Table 20). Does denser frame sampling reduce this, and how stable is the ranking (e.g., Wan-2.7 is third on the full benchmark but fifth on the human panel, Table 9)?
-
(Clarification) Unsuccessful human recordings receive a human Final score of 73.4, higher than every generator (Seedance-2.0: 64.0), since their Physics is near perfect. Could the authors also report a goal-level success rate, e.g., the fraction of videos in which all subgoals pass T3?
-
(Clarification) MiniMax-H3's results are partly attributed to provider-side prompt expansion, and expanded prompts raise Wan-2.7 from 70.4 to 84.9 (Appendix E.4). Which providers rewrite prompts, and does the ranking hold when all models receive the same expanded prompt?
2 comments
All of my concerns have been addressed. I will keep my positive score.
We thank Reviewer 2fPj for the supportive review, and E1–E10 below refer to the table in the General Response.
W1: Model coverage
We agree and added two open-weight generators, run locally on all 110 cases with the same protocol (first frame plus goal, 832×480): Wan 2.2 TI2V-5B at its trained length of 5 s and LTX-2.3 (22B) at 15 s. This makes four of the eight generators open-weight:
Both new models land with Cosmos-3 at the bottom and almost none of their videos reach the goal, so the gap to human executions holds for open-weight models too (Wan 2.2's 5 s is below even the fastest human execution, 5.2 s). The original six, rescored on the same cases, match the paper within 0.3 points, while 5 LTX-2.3 and 8 Wan 2.2 videos could not be scored. The egocentric EgoForge has no public code or model (per its project page), and we happy to circle back in the future if it is released.
W2: Duration confound
We agree duration matters (full analysis in our reply to kvWy, W2 and Q1). Fitting the budget is worth about 10 Task points, a third of the gap to Human+, but within budget most models still reach only 39% goal success vs. 84% for Human+ (E1). As suggested, we also tested a longer budget directly (E2). Wan 3.0 at 30 s instead of 15 s, with everything else fixed, does not noticeably improve the long cases (Task +4.1, goal success stays at 0%), and neither does the extension you suggest, where the open-weight LTX-2.3's official pipeline continues the same 15 s rollout to 30 s (Task −9.5, details in kvWy W2).
Q1: Denser sampling and ranking stability
Denser sampling does not reduce the over-scoring, as Physics bias, r and corruption detection stay within run-to-run noise (E4, table in our reply to UZtY Q3). The Wan-2.7 difference comes from the case set rather than from judge–human disagreement:
In short, humans and the judge agree on the same videos (both rank Wan-2.7 5th), and Wan-2.7 simply does better on the non-panel cases.
Q2: Goal-level success rate
Thank you, we adopt this metric (E7). A high Final for Human− is by design, because Physics scores the plausibility of what is shown, so a natural but incomplete execution keeps a high Physics score (App. B.2, B.4). Goal success separates the two:
Human− drops from a Final of 73.4 to 1.6% goal success, below every generator except Cosmos-3. Human+ was not rated by annotators (UZtY Q2), so it has no human goal-success value, and we will report goal success next to Final.
Q3: Provider-side prompt expansion
Closed APIs do not expose whether they rewrite prompts internally, so we cannot list which providers do, and instead we test sensitivity directly using Minimax-H3 expanded prompt (E10). Every generator gets the same goal prompt without model-specific rewriting (App. A.4), and since the App. E.4 diagnostic (5 cases × 3 prompts expanded by MiniMax-H3's built-in prompt-expansion module) was small, we added Kling-v3-Pro (mean Final):
Expansion helps Wan-2.7, leaves Cosmos-3 unchanged and hurts Kling-v3-Pro, mainly through Task (77.3 → 59.1). Kling's API caps prompts at 2,500 characters, so expanded prompts were truncated (keeping 59–100% of the text, including the goal), and prompts that lost more text declined more (r = 0.48), so part of this drop likely reflects truncation. Since expansion moves models in different directions and can reorder them (Wan-2.7 overtakes Kling-v3-Pro), we motivate that prompt expansion is not always the canonical solution for this problem. We will add per-case results to App. E.4.