acceptodds
Under review as a conference paper at ICLR 2027

Ego2World:Compiling Egocentric Cooking Videos into Executable Worlds for Belief-State Planning

Abstract

Egocentric videos capture how people carry out everyday activities, yet testing an agent requires evaluating the consequences of actions it chooses itself. We introduce Ego2World, a benchmark that turns annotated cooking activities into executable planning environments under partial observation. Its compiler links source steps and objects to symbolic action rules, persistent world states, and explicit task conditions, so researchers can execute an agent's proposed actions and check their outcomes. World state and agent belief are maintained separately, enabling controlled studies of planning and information reuse across continuing tasks. Evaluating six planners on 105 tasks shows that accepted operations often leave task goals unmet. Execution traces and condition checks distinguish interrupted runs, partial attainment, and completed execution without goal attainment. In a separate paired Qwen-Plus study, persistent belief improves action validity by 4.15 percentage points and reduces visual-query attempts by 90.27%, with higher token use and no detected completion gain. Ego2World provides a reusable testbed for tracing how planning and memory choices affect execution, observation demand, and task attainment, connecting recorded human activity to the development and evaluation of interactive agents.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Planning, Embodied AI, Vision-Language Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GSJdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Ego2World, which converts annotated egocentric cooking activities from HD-EPIC into executable symbolic planning environments. It connects source annotations to action rules, persistent world states, and explicit task conditions, while separating world state from agent belief. The paper evaluates six planners on 105 tasks and studies belief persistence on 51 terminal-state tasks. Persistent belief reduces visual queries and improves action validity, but increases token usage without a detected task-success improvement.

Critical strengths
  1. Useful benchmark direction. Connecting recorded activities to executable environments enables evaluation of agents’ chosen actions and their consequences. Source-linked operations and execution traces provide useful support for inspecting failures.

  2. Explicit evaluation semantics. The paper distinguishes action acceptance, execution completion, and goal satisfaction. Tests for incorrect bindings, missing evidence, and alternative action orders strengthen confidence in the evaluator’s implementation.

  3. Careful empirical reporting. The evaluation uses paired episode-cluster bootstrap intervals and multiplicity-adjusted comparisons. The memory study transparently reports both benefits and costs.

Critical weaknesses
  1. Semantic fidelity is insufficiently validated. The 5% manual annotation inspection provides no quantitative error rates or annotator agreement. Replay consistency establishes internal correctness, but does not establish that compiled rules and contracts adequately represent the original activities. The extent and impact of omitted requirements remain unclear.

  2. Task solvability under the final evaluation protocol remains uncertain. Appendix C.4 explicitly distinguishes reference-record validation from an online oracle evaluated through the final runner. Privileged reference access and cross-boundary event aggregation make this distinction important, particularly given the low success rates.

  3. Planner performance is entangled with substantial assistance. Models receive decomposed group instructions, reachable-action hints, controller navigation, and executor-generated repairs. Repairs also contribute to action-validity statistics. Without ablations, the results primarily characterize the combined system rather than isolate the planner’s contribution.

  4. Goal-attainment metrics show limited discrimination. Operational-TSR ranges from 3.81% to 6.67%, and no pairwise GCR differences remain significant after multiplicity adjustment. Moreover, assigning zero GCR to every unfinished run conflates partial goal attainment with execution completion. Additional failure analysis and checkpoint-based goal evaluation would clarify these results.

  5. Visual grounding and reproducibility need stronger evidence. Historical images and symbolic deltas use different temporal reference points, leaving potential inconsistencies insufficiently examined. A symbolic-only baseline would clarify the contribution of visual evidence. According to Appendix F, the accompanying package supports analysis reproduction, while the executable benchmark components are deferred to a separate release.

Questions for authors
  1. Can the authors provide an independent quantitative audit of action semantics, object bindings, and task-contract coverage?

  2. What success rate does an online reference policy or full-state oracle achieve under the final runner, task boundaries, and scoring checkpoints?

  3. How do results change without executor repairs or reachable-action hints, and when planners receive high-level task instructions instead of compiled group instructions?

  4. What explains the very low success rates, particularly for event-contract tasks? Would evaluating attained conditions at interruption change the GCR conclusions?

  5. How are discrepancies between historical images and episode-initial states resolved when the relevant fields do not appear in the delta? How does a symbolic-only agent perform?

  6. Can the authors test memory against a simple location-cache baseline, assess variability across repeated runs, and provide a minimal executable benchmark during review?

Reviewer_28qSOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper converts the first-person cooking video annotations from the HD-EPIC dataset into an executable symbolic environment. It maintains object states through manually curated action rules and distinguishes between the true state of the world and the information accessible to the agent, thereby enabling the evaluation of planning and memory capabilities in sequential tasks. The authors compare six models across 105 tasks, reporting task success rates, conditional completion proportions, and action validity rates.

Critical strengths
  • Tasks can be executed, and their outcomes verified. Unlike approaches that merely assess whether a subsequent action is reasonable, this environment allows for checking whether an action meets its preconditions, how it alters the state, and whether the final requirements are fulfilled.
  • Data sources and execution processes are highly traceable. There are clear mappings between video annotations, action rules, state transitions, and task conditions, making it easy to pinpoint exactly where a failure occurred.
  • The analysis of the results is quite detailed. It distinguishes between valid actions, the completion of execution, and task success; it also reports pairwise comparisons, confidence intervals, and the resource costs associated with memory.
Critical weaknesses

The model receives decomposed instructions, local action hints, and navigation support. The simulator also uses the full world state to repair actions, and these repairs count toward action validity. Without separating out this assistance, it is hard to tell how much the results reflect the model’s own long-horizon planning ability.

Questions for authors

How do success rates and model rankings change with automatic simulator repairs turned off? Could you also report action validity separately for model-proposed actions and simulator repairs?

Reviewer_grYaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Ego2World, a benchmark that compiles HD-EPIC cooking annotations into persistent symbolic worlds with executable action rules, source-linked operations, local observations, and frozen task contracts. The benchmark separates hidden world state from agent belief, allowing agents to act under partial information and enabling comparisons of belief-retention policies. The paper evaluates six planners on a 105-task suite and reports task completion, condition-level goal completion, action validity, and execution endpoints. It also studies persistent versus per-group-reset belief for Qwen-Plus on a separate 51-task terminal-state cohort, finding higher action validity and fewer visual-query attempts under persistence, but no detected gain in terminal completion.

Critical strengths
  1. The benchmark has unusually strong provenance and evaluation traceability. Source steps, normalized operations, transition records, and frozen task conditions are linked explicitly, and the paper distinguishes terminal-state contracts from accepted-event contracts. The validation checks are substantive: all 105 reference records satisfy their frozen contracts, all task-initial states fail them, and the paper reports predicate-level negative tests and replay stress tests. This makes it possible to inspect why an execution succeeded or failed rather than relying only on an end score.
  2. The evaluation design separates outcome attainment from process quality instead of conflating them. Operational-TSR, Strict-TSR, GCR, action validity, completed-run rate, and execution traces capture different properties; the paper also uses task-macro aggregation, episode-cluster bootstrap intervals, and paired comparisons. The memory study is particularly informative because it keeps tasks, episode order, world states, planner, VLM, prompts, budgets, and evaluator fixed while varying the belief-retention policy.
Critical weaknesses

1.The central claim that the compiled worlds and contracts faithfully represent the source activities is not quantitatively validated to the degree needed for a benchmark contribution. The paper reports a manual inspection of a 5% annotation sample, but Appendix A.3 explicitly states that no quantitative human error rate or inter-annotator agreement is estimated. Moreover, 71 of the 105 planner-suite tasks use successful-event rather than terminal-state contracts, and seven event contracts are marked proxy-limited. Reference replay and predicate-level tests establish internal consistency of the constructed executor and contracts, but do not by themselves measure agreement between the executable interpretation and the source activity semantics. This uncertainty affects the meaning of reported goal-attainment results.

2.The reported planner metrics evaluate a combined system rather than planner decisions alone. The shared runtime includes controller navigation and executor repair; Appendix D.4 states that, after a failed validation, the simulator may return a repair plan, execute its primitives, and retry the original action. The same appendix states that accepted and rejected repair actions enter the task-level action statistics. Therefore, action validity and downstream outcome metrics reflect the planner together with privileged environment-side assistance. This is a valid system setting, but planner-specific claims require an ablation or a separate accounting of planner-proposed versus controller/executor-repair actions.

Questions for authors

1.Please provide a quantitative audit of compilation fidelity. For a representative sample of tasks, can independent annotators judge whether the compiled action bindings, transition effects, and frozen contract conditions match the relevant source-video evidence? Please report the sample size, disagreement rate or inter-annotator agreement, and results separately for terminal, event, and proxy-limited contracts.

2.Can the authors report planner-only metrics, or add a no-repair ablation in which executor-generated repair plans are disabled? At minimum, please break down attempts, accepted actions, and outcome changes by planner-proposed, controller, and simulator-repair actions. This would clarify how much of the reported performance is attributable to the evaluated planner.

3.The six-planner suite is described as the first 18 complete episodes in Scenes 1–4, while the memory cohort is separately defined. Please clarify the selection rule and whether results hold on a larger or held-out set of eligible episodes/scenes. This matters because the reported bootstrap intervals quantify episode resampling conditional on the recorded runs, not variation from repeated API/model samples.

4.Since the reported belief-retention comparison changes both the retained belief available to the planner and the need for fresh visual queries, please clarify whether the claim is intended as an end-to-end policy effect or as an isolated estimate of memory quality. The current result convincingly establishes higher action validity and lower query demand in this specific runtime, but terminal completion is identical at 6/51 in both conditions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.