- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CookBench, a benchmark for evaluating long-horizon planning of multimodal agents in cooking tasks. It contains 133 tasks and 98 actions with continuous physical state changes. The best agent only scores 8.35, while humans achieve 93.34.
Critical strengths
-
Novel benchmark. It includes continuous physical states, irreversible failures, and long-horizon planning.
-
Diverse tasks. It contains different cooking tasks, objects, and tools.
-
Good analysis. The paper evaluates multiple agents and analyzes failures in grounding, feedback, and memory.
Critical weaknesses
-
Very low baseline performance. The best score is only 8.35, making it hard to know whether the tasks are too difficult or the API is hard to use.
-
Lack of strong baselines. The paper does not compare with explicit state tracking or symbolic planning methods.
-
Limited physical grounding. Agents mainly use structured text and high-level APIs, so this benchmark is more like tool-use planning.
-
Limited generalization. Experiments are only conducted in one kitchen environment.
Questions for authors
1.How can we separate planning failures from API difficulties?
2. Why not include symbolic planning or explicit state tracking baselines?
- Can the authors show performance on tasks with different lengths and difficulties?
4. Can these failure patterns generalize to other embodied environments?
6 comments
Response to Reviewer tA7W — Q2: Symbolic planning and explicit state tracking
Thank you for suggesting symbolic planning and explicit state tracking as additional baselines. CookBench involves continuously changing object states and numerous interaction conditions. Manually specifying and validating PDDL actions, preconditions, and state-transition rules across all 133 recipes would require substantial modeling and maintenance effort.
We therefore added the State-Aware Rule Executor (SARE), which uses an LLM to help formulate symbolic rules and explicitly maintains subtask progress, held-object state, and interaction mode during execution. We have now evaluated SARE on all 133 recipes. Given gold subtasks and interface rules, deepseek-v4.1-flash-1017 summarizes each recipe offline into a fixed mapping to ACQUIRE, TRANSFER, POUR, SEASON, SERVE, and SKIP handlers. At runtime, hand-written rules read structured object and agent states, including the held object and interaction mode, check relevant prerequisites, and expand handlers into API calls. Runtime control uses fixed rules rather than online LLM planning or PDDL search. Gold subtask order is supplied.
SARE explicitly maintains the current subtask, remaining action queue, object-name-to-instance-ID bindings, pending action checks, completed or skipped subtasks, and failure and retry counts. It combines these records with live held-object and interaction-mode observations and action feedback to determine whether to proceed, exit a mode, free the hands, open a container, or perform bounded recovery. These states actively govern subsequent decisions and are retained across steps.
The full evaluation gives a mean quality score of 3.21/100, with nonzero scores on 33 of 133 recipes (24.81%). The table follows the six recipe categories in the main paper and includes zero-score recipes in each mean.
| Method | Grill-Single (22) | Grill-Combo (24) | Pan-Stir (23) | Soup-Stew (25) | Pasta-Assembly (23) | Salad-Cold (16) | Overall (133) |
|---|---|---|---|---|---|---|---|
| SARE | 3.59 | 1.28 | 5.77 | 2.90 | 3.03 | 2.68 | 3.21 |
| [email protected] | 8.99 | 4.19 | 6.97 | 9.02 | 13.72 | 6.91 | 8.35 |
Explicit state management and supplied subtask order do not by themselves resolve the execution challenge: this rule-based implementation still receives low quality scores across the six categories. We will include SARE as an additional baseline.
Response to Reviewer tA7W — Q3: Performance across task lengths
Thank you for suggesting a comparison across task lengths; it helps clarify how performance varies with task scale. Using the reference action-length estimates from Appendix D.3 as a measure of task scale, we grouped all 133 recipes and compared their [email protected] quality scores (taste.score × 100):
| Estimated actions | Recipes | Mean quality score (0–100) |
|---|---|---|
| ≤150 | 90 | 9.89 |
| 151–200 | 37 | 5.29 |
| >200 | 6 | 4.08 |
Mean quality decreases across the longer recipe groups, showing the performance difference associated with task length.
Response to Reviewer tA7W — Q1: Planning errors and API-use difficulties
Thank you for highlighting the need to distinguish planning errors from difficulties in using the APIs.
We diagnose model capabilities from the chosen action and target, tracked object state, interaction prerequisites, and physical post-state, using the logs together with RGB observations.
The main evaluation covers 133 recipes per setting. The aggregate below includes every call in the 511 archived traces.
These are action-level rates. Mode, precondition, and postcondition checks account for 71.69–78.22% of the failed calls across settings. This group includes failed physical post-checks, not just incorrect mode choices.
We analyzed all 190 failure reports from five [email protected] episodes (recipes 22, 40, 58, 74 and 95), checking each against the logs, action prerequisites, and before/after RGB screenshots. We distinguish returned error types from their causes, using the observed physical state and intended action contract as evidence.
The five-recipe log-and-screenshot inspection distinguishes several mechanisms. In recipe 58, trying to put a spatula down on the griddle triggered flipping: the placement target was inappropriate for the intended action. Recipe 74 mixed two valid filling workflows: a held container must be aimed at the faucet to enter filling mode, whereas a container placed in the sink requires the faucet to be opened separately. These are action-target and tool-use sequencing errors. In recipe 40, the bottle emptied before the requested amount had been poured; the detailed failure feedback correctly reported a physical resource limit and partial completion. This case is not an API malfunction.
The RGB audit also includes calls that returned success. In recipe 22, a successful pan placement was followed by the pan falling upside down; the agent then attempted pouring without restoring a usable receiving container. A successful cutting-board placement was similarly followed by a fall. In recipe 74, a successful container-loading action placed the entire salt can in the pot instead of seasoning it. These cases expose action-selection, state-maintenance and recovery mistakes that a failed-call count alone misses. Persistent consequences and repeated recovery attempts are linked within the same state chain rather than treated as independent root causes.
The table reports the share of all 190 failure reports attributed to each model-capability dimension, with one primary label per classified report.