acceptodds
Under review as a conference paper at ICLR 2027

CookBench: Can Digital Agents Actually Cook? A Multimodal Benchmark for Long-Horizon Embodied Planning

Abstract

Recent multimodal agents have demonstrated substantial progress in digital environments, yet whether these capabilities transfer to long-horizon, physically grounded tasks remains unclear. To bridge this evaluation gap, we introduce CookBench, a high-fidelity interactive benchmark for long-horizon embodied planning in complex cooking scenarios, in which agents act primarily through structured text observations, with RGB screenshots as an optional input channel. Featuring 133 single-dish tasks with an average length of over 130 steps, CookBench goes beyond existing benchmarks by incorporating a large action space, 176 interactive objects with 50 functional items, and high-fidelity physical dynamics. Through systematic evaluation of representative agent frameworks and state-of-the-art vision-language models via a standardized evaluation harness (the Embodied Planning Machine, EPM), we reveal a substantial reality gap: even GPT-5.4-driven agents reach a dish-quality score of only 8.35 out of 100, while human operators reach about 93 under the same interface, showing that near-perfect scores are attainable through it. Ultimately, CookBench provides a crucial testbed for this digital-to-physical transition, tracing execution breakdowns to three fundamental capability deficits: absent physical grounding, poor closed-loop feedback comprehension, and severe embodied state amnesia. Code and benchmark resources are available at https://anonymous.4open.science/r/cookbench-anon-C13C.

Primary area:
datasets and benchmarks
Keywords:
Embodied AI, Multimodal Agents, Long-Horizon Planning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_tA7WOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CookBench, a benchmark for evaluating long-horizon planning of multimodal agents in cooking tasks. It contains 133 tasks and 98 actions with continuous physical state changes. The best agent only scores 8.35, while humans achieve 93.34.

Critical strengths
  1. Novel benchmark. It includes continuous physical states, irreversible failures, and long-horizon planning.

  2. Diverse tasks. It contains different cooking tasks, objects, and tools.

  3. Good analysis. The paper evaluates multiple agents and analyzes failures in grounding, feedback, and memory.

Critical weaknesses
  1. Very low baseline performance. The best score is only 8.35, making it hard to know whether the tasks are too difficult or the API is hard to use.

  2. Lack of strong baselines. The paper does not compare with explicit state tracking or symbolic planning methods.

  3. Limited physical grounding. Agents mainly use structured text and high-level APIs, so this benchmark is more like tool-use planning.

  4. Limited generalization. Experiments are only conducted in one kitchen environment.

Questions for authors

1.How can we separate planning failures from API difficulties?

​2. Why not include symbolic planning or explicit state tracking baselines?​

  1. Can the authors show performance on tasks with different lengths and difficulties?

​4. Can these failure patterns generalize to other embodied environments?

6 comments

Response to Reviewer tA7W — Q1: Planning errors and API-use difficulties

Thank you for highlighting the need to distinguish planning errors from difficulties in using the APIs.

We diagnose model capabilities from the chosen action and target, tracked object state, interaction prerequisites, and physical post-state, using the logs together with RGB observations.

The main evaluation covers 133 recipes per setting. The aggregate below includes every call in the 511 archived traces.

SettingAll callsFailed callsFailure rate
[email protected]26,5515,51220.76%
[email protected]24,9766,26225.07%
[email protected]25,9656,79226.16%
[email protected]26,5554,75517.91%
Total104,04723,32122.41%

These are action-level rates. Mode, precondition, and postcondition checks account for 71.69–78.22% of the failed calls across settings. This group includes failed physical post-checks, not just incorrect mode choices.

We analyzed all 190 failure reports from five [email protected] episodes (recipes 22, 40, 58, 74 and 95), checking each against the logs, action prerequisites, and before/after RGB screenshots. We distinguish returned error types from their causes, using the observed physical state and intended action contract as evidence.

The five-recipe log-and-screenshot inspection distinguishes several mechanisms. In recipe 58, trying to put a spatula down on the griddle triggered flipping: the placement target was inappropriate for the intended action. Recipe 74 mixed two valid filling workflows: a held container must be aimed at the faucet to enter filling mode, whereas a container placed in the sink requires the faucet to be opened separately. These are action-target and tool-use sequencing errors. In recipe 40, the bottle emptied before the requested amount had been poured; the detailed failure feedback correctly reported a physical resource limit and partial completion. This case is not an API malfunction.

The RGB audit also includes calls that returned success. In recipe 22, a successful pan placement was followed by the pan falling upside down; the agent then attempted pouring without restoring a usable receiving container. A successful cutting-board placement was similarly followed by a fall. In recipe 74, a successful container-loading action placed the entire salt can in the pot instead of seasoning it. These cases expose action-selection, state-maintenance and recovery mistakes that a failed-call count alone misses. Persistent consequences and repeated recovery attempts are linked within the same state chain rather than treated as independent root causes.

The table reports the share of all 190 failure reports attributed to each model-capability dimension, with one primary label per classified report.

Model capabilityBehavioral exampleShare of 190 failure reports
Prerequisite and action-order planningRecipe 40: attempting to pour before entering pouring mode.20.00%
Spatial interaction and stable placementRecipe 22: attempting to pour into a pan lying upside down without first restoring it.15.26%
Action semantics and interaction-target selectionRecipe 58: selecting the griddle as the placement target for a spatula instead of a countertop.8.95%
State tracking and held-object maintenanceRecipe 22: calling auto_pour without a valid held liquid source.6.84%
Argument format and type useRecipe 40: passing y as the string “220” caused a type error; using numeric 220 succeeded.0.53%

Response to Reviewer tA7W — Q2: Symbolic planning and explicit state tracking

Thank you for suggesting symbolic planning and explicit state tracking as additional baselines. CookBench involves continuously changing object states and numerous interaction conditions. Manually specifying and validating PDDL actions, preconditions, and state-transition rules across all 133 recipes would require substantial modeling and maintenance effort.

We therefore added the State-Aware Rule Executor (SARE), which uses an LLM to help formulate symbolic rules and explicitly maintains subtask progress, held-object state, and interaction mode during execution. We have now evaluated SARE on all 133 recipes. Given gold subtasks and interface rules, deepseek-v4.1-flash-1017 summarizes each recipe offline into a fixed mapping to ACQUIRE, TRANSFER, POUR, SEASON, SERVE, and SKIP handlers. At runtime, hand-written rules read structured object and agent states, including the held object and interaction mode, check relevant prerequisites, and expand handlers into API calls. Runtime control uses fixed rules rather than online LLM planning or PDDL search. Gold subtask order is supplied.

SARE explicitly maintains the current subtask, remaining action queue, object-name-to-instance-ID bindings, pending action checks, completed or skipped subtasks, and failure and retry counts. It combines these records with live held-object and interaction-mode observations and action feedback to determine whether to proceed, exit a mode, free the hands, open a container, or perform bounded recovery. These states actively govern subsequent decisions and are retained across steps.

The full evaluation gives a mean quality score of 3.21/100, with nonzero scores on 33 of 133 recipes (24.81%). The table follows the six recipe categories in the main paper and includes zero-score recipes in each mean.

MethodGrill-Single (22)Grill-Combo (24)Pan-Stir (23)Soup-Stew (25)Pasta-Assembly (23)Salad-Cold (16)Overall (133)
SARE3.591.285.772.903.032.683.21
[email protected]8.994.196.979.0213.726.918.35

Explicit state management and supplied subtask order do not by themselves resolve the execution challenge: this rule-based implementation still receives low quality scores across the six categories. We will include SARE as an additional baseline.

Response to Reviewer tA7W — Q3: Performance across task lengths

Thank you for suggesting a comparison across task lengths; it helps clarify how performance varies with task scale. Using the reference action-length estimates from Appendix D.3 as a measure of task scale, we grouped all 133 recipes and compared their [email protected] quality scores (taste.score × 100):

Estimated actionsRecipesMean quality score (0–100)
≤150909.89
151–200375.29
>20064.08

Mean quality decreases across the longer recipe groups, showing the performance difference associated with task length.

Reviewer_eUVHOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CookBench evaluates long-horizon embodied planning through 133 cooking tasks built on the commercial game Cooking Simulator. The environment combines persistent thermal and liquid states, object manipulation, and irreversible mistakes. Its Embodied Planning Machine provides structured observations, optional RGB input, semantic and composite actions, execution feedback, cooking knowledge, and persistent memory. The authors evaluate several adapted agent frameworks and five model backbones. The highest reported mean dish-quality score is 8.35/100, whereas humans operating through the harness achieve 93.34/100 under a different attempt protocol. Additional experiments include interface ablations, repeated runs on a stratified subset, statistical comparisons, and trace-based analyses of action grounding, recovery, and state tracking.

Critical strengths

The environment tests a consequential combination of planning and persistent state management. Cooking requires agents to coordinate preparation, transfers, heating, and final serving while earlier actions continue to affect the world. The semantic interface and quantitative feedback make these failures inspectable. This is a useful evaluation setting beyond simply checking whether an agent emits a valid symbolic plan.

The empirical study includes several valuable controls. The human-through-harness cohort demonstrates that high scores are attainable through the semantic interface. Component ablations, three repetitions on a 20-dish subset, dish-level confidence intervals, and corrected pairwise tests strengthen the empirical contribution. The paper also appropriately acknowledges that several framework and model differences do not establish statistically significant rankings.

Critical weaknesses

The evidence that the execution budget does not constrain completion is insufficient. Appendix D.3 estimates task lengths of 50–249 actions, but these are expansions of LLM-produced milestones rather than successfully executed reference trajectories. Normal cooking stops at 200 actions; the remaining 50 actions permit finalization only, with the normal policy replaced by a deterministic submission routine (Appendix F.1.1). Therefore, a 250-action total limit does not itself establish that the available cooking budget covers every task. The reported termination analysis also finds that 77.8% of the analyzed [email protected] episodes terminate through step-limit-related mechanisms. This does not prove that more actions would solve them, but it makes budget sensitivity consequential. Successful reference traces and a controlled budget sweep are needed to distinguish intrinsic task difficulty from protocol-induced truncation. The submission supervisor’s contribution should also be measured because it can transfer and plate food, directly affecting the scored outcome.

The diagnostic evidence does not fully establish the claimed capability deficits. The intent–execution analysis relies heavily on LLM annotations. The human audit reports 62.3% exact agreement for intent, with κ=0.41\kappa=0.41, and identifies over-confirmation of intent in 49 cases. Inflated intent coverage could exaggerate the apparent separation between understanding and execution; the claim that annotation errors are uniformly conservative therefore needs stronger justification. Score validation also partly depends on labels inferred from the same final scoring feedback. Appendix F.6 acknowledges this circularity, while independently annotated action grounding has only a weak correlation with final score (ρ=0.15\rho=0.15, Table 18). Likewise, increasing repetition late in an episode is compatible with state-tracking failure, but also with accumulated damage and selection of difficult unfinished episodes. These observations support useful failure descriptions more clearly than they isolate physical misunderstanding or memory decay as causes.

Questions for authors

I am worried about the actual performance, due to lack of video.

Budget feasibility: How many recipes have estimated reference lengths above the 200-action cooking cutoff? Can successful human or scripted traces demonstrate completion within that cutoff, and how do agent scores change with a larger normal-execution budget?

Submission assistance: What fraction of scored dishes depends on the deterministic supervisor’s transfer, plating, or submission actions? Please separate autonomous submissions from supervisor-assisted outcomes.

Annotation reliability: After correcting the audited intent and grounding labels, does the intent-to-execution gap remain comparably large? Please report estimates accounting for the audit’s stratified sampling and clarify why intent over-confirmation is considered conservative.

Timing and data accounting: Does physical simulation continue during model inference? Please reconcile Table 2’s wall-clock wording with statements that agent execution time excludes deliberation, and explain how the 133-dish results relate to the 108 completed episodes used in the termination analysis

7 comments

Response to Reviewer eUVH — Q1: Video evidence

Thank you for suggesting video evidence. We have added a complete simulator recording of Dish 74 (Pasta alla Genovese), using the Code as Policy (CaP) harness with deepseek-v4.1-flash-1017. The recording is provided in five chronological parts at 3x speed and includes physical object interactions, execution difficulties, recovery attempts, and the final in-game evaluation screen.

Video: https://anonymous.4open.science/r/cookbench-anon-C13C/assets/videos/dish74/README.md

Response to Reviewer eUVH — Q2: Action budget and completion within 200 steps

Thank you for suggesting a direct check of whether the action budget affects evaluation outcomes.

(1) Number of recipes estimated to exceed 200 actions. According to the recipe-expansion estimates in Appendix D.3, six of the 133 recipes (4.5%) have estimated action lengths above 200.

(2) Human completion within the cutoff. Yes. Human reference trials using the same APIs provided by EPM include recipes completed and submitted for scoring within 200 actions. The paper already reports the Human (Harness) reference scores in the main results table (overall 93.34/100); the interface protocol is described in the Human Baseline Protocol appendix.

(3) Effect of a larger execution budget. We tested Code as Policy (CaP) with deepseek-v4.1-flash-1017 on ten recipes: all six estimated to exceed 200 actions, plus two from each shorter group (≤150 and 151–200). Each recipe has one trajectory, with scores recorded at 50-step checkpoints and actual termination using the same deterministic finalization routine. The table shows the original checkpoint scores; bold values mark the observed peak for each nonzero-scoring recipe. The last column gives the actual terminal step and final score.

IDEstimated actions50100150200250300350400Terminal step / final score
405530.9030.9023.3423.65————212 / 23.65
731335.569.726.2427.7822.2222.2222.2222.22400 / 22.22
271690.000.00——————104 / 41.97
471880.000.0019.4419.4419.4419.4419.55—385 / 19.55
1042020.000.000.000.00————202 / 0.00
82110.000.000.000.000.00———254 / 0.00
952110.000.000.000.000.000.00——318 / 0.00
742335.2615.9110.6510.6510.6510.65——311 / 10.65
1092330.000.0017.7815.0015.64———292 / 15.00
1002490.000.000.000.00————217 / 0.00

From the 200-action checkpoint through actual termination, the score range (maximum minus minimum observed score, including step 200) averaged 0.70 points across the nine applicable trajectories, with a median of 0 and a maximum of 5.56. Recipe 27 had already submitted at step 104 with a score of 41.97.

Five of the six recipes with nonzero scores reached their observed peak by 200 actions. The remaining recipe, ID 47, scored 19.44 at step 200 and peaked at 19.55 later, a difference of only 0.11 points. Together, these score ranges and peak timings support 200 actions as a reasonable evaluation cutoff for this tested setting.

Response to Reviewer eUVH — Q3: Autonomous and deterministic-script submissions

Thank you for suggesting that we separate autonomous submissions from submissions completed by the deterministic script; this makes the reported outcomes easier to interpret.

We analyzed the main-experiment run logs. Using the 200-action cutoff to identify entry into the deterministic submission stage, the observed proportions correspond to approximately 834 script-assisted runs out of 1064 evaluations (78.4%), after scaling each of the eight settings to 133 dishes.

Method and modelAutonomous submissions onlyDeterministic-script submissions onlyFull main-experiment results
[email protected]3.205.008.35
[email protected]1.333.515.28
[email protected]0.812.974.12
[email protected]1.084.876.05
[email protected]0.143.994.13
[email protected]1.692.514.20
[email protected]0.183.503.84
[email protected]3.801.666.77

The deterministic submission script provides a standardized procedure for collecting, plating, and submitting prepared food after the cooking budget is exhausted. The prompt explicitly reports the current step and total action budget. Even when agents cannot finish the full recipe, they are expected to allocate the remaining actions and submit the prepared food in time. These results indicate room for improvement in budget-aware task closure and timely submission.

Comments on the paper

1 comment

Dear all reviewers,

We sincerely thank you for your helpful reviews. We have submitted our responses to address all your remaining questions. We are looking forward to your further feedback. Thank you again!

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.