- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Ego2World, which converts annotated egocentric cooking activities from HD-EPIC into executable symbolic planning environments. It connects source annotations to action rules, persistent world states, and explicit task conditions, while separating world state from agent belief. The paper evaluates six planners on 105 tasks and studies belief persistence on 51 terminal-state tasks. Persistent belief reduces visual queries and improves action validity, but increases token usage without a detected task-success improvement.
Critical strengths
-
Useful benchmark direction. Connecting recorded activities to executable environments enables evaluation of agents’ chosen actions and their consequences. Source-linked operations and execution traces provide useful support for inspecting failures.
-
Explicit evaluation semantics. The paper distinguishes action acceptance, execution completion, and goal satisfaction. Tests for incorrect bindings, missing evidence, and alternative action orders strengthen confidence in the evaluator’s implementation.
-
Careful empirical reporting. The evaluation uses paired episode-cluster bootstrap intervals and multiplicity-adjusted comparisons. The memory study transparently reports both benefits and costs.
Critical weaknesses
-
Semantic fidelity is insufficiently validated. The 5% manual annotation inspection provides no quantitative error rates or annotator agreement. Replay consistency establishes internal correctness, but does not establish that compiled rules and contracts adequately represent the original activities. The extent and impact of omitted requirements remain unclear.
-
Task solvability under the final evaluation protocol remains uncertain. Appendix C.4 explicitly distinguishes reference-record validation from an online oracle evaluated through the final runner. Privileged reference access and cross-boundary event aggregation make this distinction important, particularly given the low success rates.
-
Planner performance is entangled with substantial assistance. Models receive decomposed group instructions, reachable-action hints, controller navigation, and executor-generated repairs. Repairs also contribute to action-validity statistics. Without ablations, the results primarily characterize the combined system rather than isolate the planner’s contribution.
-
Goal-attainment metrics show limited discrimination. Operational-TSR ranges from 3.81% to 6.67%, and no pairwise GCR differences remain significant after multiplicity adjustment. Moreover, assigning zero GCR to every unfinished run conflates partial goal attainment with execution completion. Additional failure analysis and checkpoint-based goal evaluation would clarify these results.
-
Visual grounding and reproducibility need stronger evidence. Historical images and symbolic deltas use different temporal reference points, leaving potential inconsistencies insufficiently examined. A symbolic-only baseline would clarify the contribution of visual evidence. According to Appendix F, the accompanying package supports analysis reproduction, while the executable benchmark components are deferred to a separate release.
Questions for authors
-
Can the authors provide an independent quantitative audit of action semantics, object bindings, and task-contract coverage?
-
What success rate does an online reference policy or full-state oracle achieve under the final runner, task boundaries, and scoring checkpoints?
-
How do results change without executor repairs or reachable-action hints, and when planners receive high-level task instructions instead of compiled group instructions?
-
What explains the very low success rates, particularly for event-contract tasks? Would evaluating attained conditions at interruption change the GCR conclusions?
-
How are discrepancies between historical images and episode-initial states resolved when the relevant fields do not appear in the delta? How does a symbolic-only agent perform?
-
Can the authors test memory against a simple location-cache baseline, assess variability across repeated runs, and provide a minimal executable benchmark during review?