acceptodds
Under review as a conference paper at ICLR 2027

DuoCook: An Overcooked-Inspired Benchmark for Visual Planning and Coordination

Abstract

Coordinating two players in a cooking task requires a vision-language agent to track changing scene states, sequence recipe steps, and allocate shared resources under a time budget. We introduce an Overcooked-inspired benchmark for evaluating these capabilities in Unreal Engine environments constructed using coding agents. A single vision-language model controls both players from top-down visual observations, selecting high-level actions while the engine executes navigation. The benchmark contains 20 maps in five categories: basic, shared equipment, dynamic, dynamic with shared equipment, and separated workspaces requiring item transfers. These categories support evaluation of centralized coordination under equipment contention, changing terrain, and handoff dependencies. A prop legend and pot ingredient-count indicators provide explicit visual cues, while grid-based destinations support spatial grounding. The evaluation compares four VLMs on 16 of the 20 maps, measuring task score, order completion, delivery precision, action acceptance rate, and model API expenditure. A memory ablation on 10 of these maps examines the effect of retaining recent action history while keeping the other inputs fixed. Together, the environment design and evaluation protocol provide a framework for investigating how visual state tracking and task scheduling contribute to coordinated cooking behavior.

Primary area:
datasets and benchmarks
Keywords:
vision-language models, agent benchmarking, visual planning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8LK4OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

DuoCook introduces 20 Unreal Engine cooking maps spanning five categories for visually assisted two-player planning and coordination, with engine-executed navigation and structured state inputs (Secs. 3.1–3.5). The paper evaluates four VLMs on 16 maps and reports a memory ablation on 10 maps using task performance, action acceptance, and API expenditure (Secs. 4.1–4.4).

Critical strengths
  1. The five map categories explicitly vary equipment sharing, changing terrain, and required handoffs, creating interpretable coordination challenges rather than relying on a single kitchen layout (Sec. 3.2, lines 173–186; Appendix A, Table 2).
  2. The evaluation separates completed orders and reward from command acceptance and API cost, making it possible to distinguish successful task execution from merely valid actions (Sec. 4.2, lines 286–320).
Critical weaknesses
  1. The benchmark has not established which visual and coordination abilities its scores actually measure. The controller receives player positions, held items, station claims, orders, preparation status, and state-dependent legal operations; the engine handles navigation and pauses during inference (Secs. 3.4–3.5, lines 205–246). The grid-labeled image is intended to locate action targets (lines 236–238), but the reported experiments vary the model or memory without a matched image-removal control that tests its contribution beyond the supplied state. Low completion therefore does not distinguish visual grounding failures from recipe scheduling, interface use, or deadline management. Difficulty is also uncalibrated: scene validation checks actors, layout, and navigation constraints, but the paper reports no human or scripted/planning reference establishing attainable completion under the same 4,000-step horizon and order schedules (Secs. 3.1 and 4.1; Appendices A–B). Such a reference matters particularly where the VLMs complete no orders. The existing map taxonomy and separate action-validity metric are useful, yet do not by themselves establish that the suite diagnoses the intended capabilities. Matched modality controls and a feasible reference policy would make the benchmark's distinct evaluation value substantially clearer.
  2. The memory and category-level conclusions are not supported by a consistent experimental specification and result record. The introduction and Sec. 3.4 describe one shared controller and an ablation of six-response action history, whereas Sec. 4.4 retains that history, removes a self-authored decaying notebook, and calls the notebook the only channel between players with private histories (lines 61–63, 207–215, 415–422, and 457–471). These descriptions imply different interventions and information access, so the observed effect cannot yet be attributed to a defined memory mechanism. The numerical account is also inconsistent: Fig. 4 shows 12.5% completion for gpt-6-astra in separated workspaces despite the claim that all controllers deliver nothing there, and Fig. 3 reports 29.2% completion and 94.4% precision rather than the prose's 28.1% and 100% (lines 360–374). Finally, one episode per model/map/condition cannot assess repeatability; the correctly described bootstrap over maps in Fig. 5 does not resolve this. The authors acknowledge the descriptive nature of the results, but claims about controller-specific memory effects still require a coherent protocol and reconciled episode records, with broader conclusions limited to what the available repetitions support.
Questions for authors
  1. [Clarification; Weakness 1] Which task-relevant facts must be inferred from the image rather than the live-state JSON, legal-operation list, or rejection feedback? Please provide the existing observation schema, especially for station locations and terrain changes, and identify any already available image-removal or text-only results. A matched modality study is a suggestion for further validation if no such results exist.
  2. [Clarification; Weakness 1] What existing playthroughs, scripted solutions, or validation records demonstrate achievable order completion on the difficult maps under the exact episode horizon and order schedule? Please distinguish actor/navigation checks from evidence that the cooking and handoff workload is feasible, and explain how this informs the interpretation of zero-completion episodes.
  3. [Clarification; Weakness 2] What memory mechanism and controller configuration actually generated Figs. 3–5: one shared controller with six-response history, or separate player contexts with a shared notebook? Please supply the relevant enabled/disabled prompt difference and information-access schema, and reconcile the conflicting descriptions in Secs. 3.4 and 4.4.
  4. [Clarification; Weakness 2] Can the existing per-map episode records reproduce the authoritative completion, precision, and category-level results and resolve the cited figure/prose discrepancies? Please state the model sampling settings and whether any repeat runs already exist; if they do not, delimit the ranking and memory conclusions accordingly. Additional repeated trials and reference-policy evaluations are future validation suggestions, not requests for a new experimental campaign during rebuttal.
3 comments

Thank you for the precise questions. Every answer below is verified against the benchmark code and the new per-episode records.

Q1: Which facts must come from the image? See Table 2. Image only: station and object locations, which cells hold ingredients, plates, or dishes, pot fill and readiness, board contents, the current terrain configuration, and the partner's position and held item. The live-state JSON holds only facts outside the camera frame: orders with timers and tip zone, earnings, combo, next order, own cell and held item, and the partner's current operation, target cell, and claims. It contains no station coordinates, and the minimal prompt removes the partner fields. The operation list is the full unfiltered vocabulary, and rejection feedback never reveals the correct cell. Terrain changes are therefore not stated in text. Text still carries some spatial evidence: accepted targets in a player's history, the partner's target cell, and notebook rows. To measure sensitivity to text assistance, we added the minimal-prompt ablation (§4.6), which keeps the images and removes about 88% of the system-prompt text. Two of five models lose more than two points, and gpt-6-astra still completes 26.3% of orders. We did not run an image-removal condition, so the incremental contribution of vision remains unmeasured; we state this in §4.6.

Q2: Evidence of attainable completion. We added a privileged scripted policy under the exact horizon, initial states, and order schedules (§4.3). It delivers on 14 of 20 maps. On the other six (1-1, Kevin 1, 4-1, 4-3, 2-4, 3-2), at least one VLM episode delivers, so every map has at least one demonstrated delivery. The best observed episode per map completes one order on five maps, two on ten, and three on five; no episode completes all four. This establishes partial feasibility; whether the full four-order workload is attainable remains unknown. Appendix C documents the construction checklists and which records survive; these establish environment construction, not workload feasibility. We interpret zero-delivery episodes as controller failures on maps where delivery is demonstrated, not as evidence of infeasibility.

Q3: Which memory mechanism and controller configuration? Per-player calls: the same model is queried separately for each player, with stateless calls. Both conditions contain the images, legend, live state (including the partner's operation and claims), full operation vocabulary, failure feedback, and each player's private last-six-decision history. The memory condition adds only the shared notebook: up to 3 writes per decision of (cell, kind, status, note), author-tagged, 600-step half-life with partner rows decaying 2× faster, three confidence bands, and at most 24 rows (16 per author). The system prompt grows from ~3,600 to ~4,000 words to explain it. Every section now uses this one specification.

Q4: Reproducibility of the record. The discrepancies you identified were real. Submitted Fig. 3 gives gpt-6-astra 29.2% completion and 94.4% precision against 28.1% and 100% in the prose; submitted Fig. 4 shows 12.5% completion in separated workspaces against the prose's zero; and Fig. 3 gives claude-opus-5 and gemini-3.5-flash-lite acceptance of 52.4%/30.1% against 71.0%/44.4% in the prose. We could not recover the configurations or episode records that generated the submitted figures and prose, so we cannot determine which values were correct. We therefore designate a replacement evaluation as authoritative: 320 new episodes under the protocol in §4.1, with every performance table and figure regenerated from their per-episode result files. Fig. 5 (Appendix B) reports every episode's score, and Appendix B states that these results replace the submitted ones. The two versions also differ in map set and in how precision is aggregated, so their numbers are not directly comparable. Settings: all models run with reasoning enabled at low effort, horizon 4,000 steps, and 4 orders per episode. qwen3.8-27b is accessed via OpenRouter and the others via an OpenAI-compatible API gateway (§4.1). No repeated runs exist. Rankings and ablation effects are therefore stated as descriptive, with SEs across maps and paired sign tests, and the "memory helps weak / harms strong" conclusion has been withdrawn.

Reviewer_8LK4OpenReview ↗

Remaining concerns about benchmark interpretation and calibration

Thank you for the additional evidence and for openly addressing the problems in the original evaluation. I appreciate the replacement evaluation on all 20 maps, the scripted reference, and the clarified observation and notebook interfaces. These resolve several of my earlier concerns.

My main reservation is still what the benchmark scores tell us about visual planning. The input breakdown makes clear that the images contain important information. However, the minimal-prompt experiment retains the images while reducing text assistance, so it does not isolate the contribution of visual grounding. A low completion score could still arise from target localization, recipe scheduling, or coordination failures, and the current evidence does not separate these explanations sufficiently.

The scripted reference also helps establish feasibility, but only partially. There is now at least one demonstrated delivery on every map, yet no reported episode completes all four orders. The attainable completion level therefore remains unclear, making it difficult to judge how far the evaluated controllers are from a reasonable solution. In addition, there is still only one episode per model, map, and condition. Describing the rankings and notebook effects as descriptive is appropriate, but it does not resolve their repeatability.

I appreciate the substantial effort behind the new evaluation. Given the remaining uncertainty about what the scores diagnose and how the task difficulty should be interpreted, I will keep my original recommendation.

Thank you for the follow-up and for your careful reviews.

We agree with the remaining points. The minimal-prompt ablation measures sensitivity to text assistance, not the contribution of vision (§4.6). Aim accuracy does not separate image misreading from operation choice, nor scheduling from coordination failures. Isolating these would need a coordinate-supplied text interface and a trajectory-level failure analysis, which we have not run.

On feasibility, the best observed episodes complete three of four orders on five maps, and no controller completes all four. These are lower bounds, not an estimate of attainable completion. With one episode per cell, the rankings and notebook effects remain descriptive.

We appreciate your engagement with the revision.

Reviewer_DdVPOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces DuoCook, an Overcooked-inspired benchmark for evaluating centralized visual planning and coordination. A single vision-language model controls two players using top-down visual observations, structured state information, and high-level legal actions. The benchmark contains 20 Unreal Engine kitchens organized into five categories: independent work, shared equipment, dynamic terrain, dynamic terrain with shared equipment, and separated workspaces requiring handoffs. Four VLMs are evaluated on 16 maps using task score, order completion, delivery precision, action acceptance, and API cost. A memory ablation is additionally conducted on a 10-map subset. The benchmark addresses a relevant problem and introduces meaningful coordination challenges. However, the evaluation does not isolate visual reasoning or directly measure coordination. The scientific novelty over existing benchmarks is also insufficiently established. More importantly, the memory intervention is described inconsistently across the paper. The empirical conclusions are further limited by single-run evaluation, missing reference baselines, incomplete ablations, and limited failure analysis.

Critical strengths
  1. Meaningful task difficulty.
The benchmark combines multi-stage recipe dependencies, parallel execution, shared-resource contention, changing terrain, mandatory handoffs, order deadlines, time-dependent rewards, and streak bonuses. These factors create nontrivial scheduling and task-allocation challenges.
  2. Relevant positioning with respect to prior work.
The paper discusses closely related environments and methods, including Overcooked-AI, ProAgent, MindAgent, Collab-Overcooked, ParaCook, Robotouille, and general visual-agent benchmarks. It also appropriately acknowledges that neither multi-cook control nor required handoffs alone constitutes the contribution.
  3. Clear presentation.
The paper is generally well written and easy to follow. The task mechanics, map categories, scoring rules, controller interface, and evaluation metrics are clearly explained.
Critical weaknesses
  1. The problem framing is broader than what is evaluated.
The paper presents DuoCook as a benchmark for “visual planning and coordination.” However, the controller receives substantial structured assistance, including player positions, held items, station claims, order status, preparation status, legal operations, and execution feedback. Consequently, the contribution of visual perception is unclear. Image-only, structured-state-only, and combined-input ablations are needed to establish whether the benchmark evaluates visual reasoning or primarily structured task scheduling.
  2. The scientific novelty is limited.
The primary contribution is a benchmark for centralized two-player scheduling across five map categories. The paper does not propose a new coordination algorithm, planning method, learning objective, or scientific model of cooperation. Although benchmark contributions can be valuable, the paper does not sufficiently establish that DuoCook measures a capability that cannot already be studied using existing environments.
  3. The interface removes substantial parts of the claimed difficulty.
Navigation is executed by the engine, destination cells are explicitly labelled, legal operations are enumerated, rejected actions receive explanatory feedback, each map uses an approximately 4,000-word prompt, and simulation is paused during inference. These choices isolate high-level scheduling but remove navigation, real-time decision-making, low-level control, and much of the state-estimation difficulty. The paper does not quantify the difficulty remaining after introducing this scaffolding.
  4. The claimed white space is asserted rather than demonstrated.
The related-work section lists relevant benchmarks but does not systematically compare DuoCook with Collab-Overcooked, CuisineWorld, ParaCook, Robotouille, and related environments. A capability comparison should distinguish visual versus symbolic observations, centralized versus decentralized control, static versus dynamic environments, shared-resource contention, required handoffs, partial observability, action abstraction, and coordination metrics. Without such a comparison, the novelty appears to arise mainly from combining previously studied elements.
  5. The claimed benchmark novelties are only partially validated.
The kitchens reproduce existing Overcooked layouts and require manual refinement following agent-based generation. However, the paper does not report generation success rate, layout fidelity, actor-contract failure rate, number of correction cycles, or manual editing time. Moreover, only 16 of the 20 maps are included in the main evaluation, and only 10 are used in the memory ablation. The excluded maps are not adequately justified.
  6. The scientific hypothesis is unclear, and the memory intervention is specified inconsistently.
The closest explicit hypothesis concerns whether shared memory improves joint performance. However, the abstract, Introduction, Section 2.3, and Section 3.4 describe memory as the six most recent controller responses, containing both players’ actions, target cells, and command-acceptance feedback, and state that the ablation removes this history. Section 4.4 instead retains this history in both conditions and ablates a separate, decaying, self-authored shared notebook. It also describes player-specific histories, partner-written entries, and partner information as “hearsay,” although earlier sections describe a single controller operating from a shared context. It is therefore unclear what information each player receives, whether the model is queried jointly or separately, and which intervention produced the reported memory effects.
  7. The results provide limited support for several claims.
    • The dynamic-with-shared-equipment and separated-workspace categories contain 6 of the 16 evaluated maps, and no controller successfully delivers a dish in these categories. This demonstrates difficulty but provides limited discrimination among the evaluated models in terms of task completion.
    • The conclusion that failures are “of completion, not selection” is overstated. Delivery precision evaluates only served dishes and excludes episodes without deliveries. It therefore cannot rule out errors in ingredient selection, order prioritization, player assignment, station choice, or intermediate actions as causes of non-completion.
    • The claim that no controller converts the additional memory cost into a higher completion rate appears inconsistent with the reported qwen3.8-27b results. Seven maps reportedly remain unchanged at zero, while three change from no delivery to one delivery. These additional deliveries should increase the mean completion rate. Completion rates under both memory conditions should be reported explicitly.
    • The reported memory effects are based on one episode per model, map, and condition. The bootstrap intervals resample maps rather than stochastic runs. These results should therefore be interpreted as descriptive differences rather than statistically reliable memory effects.
  8. The experimental evaluation is insufficient.
    • There is only one run per model, map, and condition.
    • The bootstrap analysis captures variation across maps, not run-to-run variability.
    • There are no random-legal, greedy, role-based, heuristic, symbolic-planning, or oracle baselines.
    • There are no input ablations for images, structured state, grid labels, legal-action lists, or prompt length.
    • The effects of equipment contention, terrain changes, and required handoffs are not isolated.
    • The memory design is not ablated with respect to capacity, decay rate, shared versus private memory, or self-authored versus environment-authored memory.
  9. The reported metrics do not directly measure coordination.
The five metrics capture task outcome, command validity, and inference cost, but none directly measures coordination quality. Relevant missing metrics include player idle time, productive concurrency, workload balance, duplicated work, station contention, handoff success and latency, recovery after terrain changes, normalized throughput, and regret relative to an oracle scheduler. Furthermore, raw scores are not normalized by the maximum achievable score, the number of orders per map is not reported, and delivery precision may be unreliable when calculated from very few deliveries.
  10. No state-of-the-art comparison or detailed failure analysis is provided.
The paper introduces a benchmark rather than a new method, and no prior method is evaluated under the same interface. Therefore, the results do not establish a state-of-the-art improvement. The ranking of the four VLMs is descriptive, and the paper does not explain whether failures arise from perception, spatial grounding, action formatting, task allocation, order prioritization, or long-horizon planning.
Questions for authors

Refer Weaknesses Section.

2 comments

Thank you for the thorough review. We address each weakness and state plainly what we did not do.

W1/W3: Framing versus evaluated capability. We agree. The revised paper describes DuoCook as evaluating image-grounded target selection and scheduling by one model controlling two players, with engine navigation, paused inference, and structured order and partner state. The conclusion lists what it does not test: low-level control, real-time reaction, communication between independent agents, and perception without textual support. To quantify the difficulty that remains, Table 2 shows that no text channel lists station coordinates, object states, pot contents, or terrain, and the operation list is unfiltered. The best model completes 35% of orders, no episode completes all four orders, and 47 of 100 main-condition episodes end with no correct delivery. The minimal-prompt ablation (§4.6) removes about 88% of the system-prompt text and lowers scores for two of five models. The others change by ≤2 points (details in our response to miB3).

W2/W4: Novelty and positioning. We do not claim a new coordination algorithm or that any single element is new. §2 now states the specific combination DuoCook offers: rendered 3D observations from which targets are localized mainly visually, and a suite that includes layouts with contention, moving terrain, and mandatory handoffs. Layout geometry and recipe workload also vary, so comparisons between categories cannot identify the effect of any one mechanism; §5 states this. Appendix E adds the capability comparison you asked for (Table 7). It covers Overcooked-AI, CuisineWorld, Collab-Overcooked, Robotouille, ParaCook, and CookBench along observation type and observability, agents and control, action level, changing terrain, contention, mandatory handoffs, and metrics. The table shows that no single dimension is unique to DuoCook. What it adds is the combination of rendered observations with mainly visual target localization, in-episode terrain change, contention, and mandatory handoffs, each represented across the suite.

W5: Validation of generation, and excluded maps. No maps are excluded any longer; every experiment uses all 20. §3.1 and Appendix C now report construction records from the agent logs: 108 agent runs, $352.70, and 25.3 agent-hours across 27 constructed levels. Per level, the medians are 3 runs, $11.65, 81 script calls, and 1.2 h of active operator time. 25 of 27 levels had their prompt resubmitted, and 8 of the 19 benchmark levels with records needed ≥2 correction episodes. Logs do not separate operator fixes from agent edits, so we report operator time rather than an autonomous success rate. Checklist results survive in manifests for only two benchmark levels, and we say so.

W6: Memory specification. Thank you; this was a real error. From the code: per-player stateless calls; each player always receives its own last six decisions (private, present in both conditions); and the ablated variable is the shared notebook (24 rows, 16 per author, 3 writes per decision, 600-step half-life, partner rows decaying twice as fast). The abstract, introduction, §2.3, §3.4, and §4.5 now give this single specification. We also corrected the claim that the notebook is the only partner channel: the partner's current operation and claims are visible in both conditions.

W7: Claims not supported by the results.

  • Categories without deliveries: in the new runs, no category is unsolved, and every map has at least one correct delivery by some model. The hardest map, 4-3, has one delivery in 15 model episodes.
  • "Failures of completion, not selection": withdrawn. §4.2 now states that precision characterizes only serving attempts. Precision is reported with counts and is 50% for gemini-3.5-flash-lite.
  • Memory and completion: completion under both conditions is now explicit. With the notebook it is higher for three models (claude-opus-5 3→7 orders, muse-spark-1.3 19→23, qwen3.8-27b 9→10) and lower for two (gpt-6-astra 31→28, gemini-3.5-flash-lite 10→6). None of these differences is significant.
  • Statistics: still one episode per cell. ± values are SEs across maps, paired effects use exact sign tests over maps, and all rankings and effects are labeled descriptive.

W8: Experimental evaluation.

  • Reference policy (added, §4.3): a fixed-priority scripted policy that is privileged. It reads the simulator's feasible operations and ground-truth station locations, never aims, and is not tuned per map. It scores 20.2 (21.3% completion) and delivers on 14 maps. gpt-6-astra beats it on 13 maps, loses on 4, and ties on 3, and muse-spark-1.3 is also above it. The script also differs in action priorities and needs no aiming, so the gap cannot be attributed to scheduling alone. It is a feasibility reference, not an oracle or upper bound.
  • Input ablations (added): the prompt-text ablation is in §4.6. We did not run an image-removal condition, so the contribution of vision remains unmeasured.
  • Not done: repeated runs, random and symbolic-planning baselines (the scripted reference is itself a greedy fixed-priority policy), image-only/text-only, grid-label, and operation-list controls, factorial manipulations of contention, terrain, and handoffs, and notebook-design ablations (capacity, decay, authorship). These remain open; we do not claim the new experiments resolve them.

W9: Coordination metrics. We agree that our metrics measure outcomes, command validity, and cost, not coordination quality directly, and we now state this in the conclusion. We added what the records support. Every episode seats exactly 4 orders, so completion is normalized by workload. Precision is reported with correct/served counts. Aim accuracy and failed actions per episode separate command validity from task success. API cost per episode is estimated from the recorded tokens at list prices (Appendix D), because the API gateway does not report billing (qwen3.8-27b uses its OpenRouter bill). Idle time, concurrency, and handoff latency are not computed in this revision.

W10: SOTA comparison and failure analysis. DuoCook is a benchmark, so we make no state-of-the-art claim. For failure attribution, the revision separates three sources. Command errors are measured by aim accuracy: 15–50% of graded commands target a wrong cell or an impossible operation, although this does not isolate image misreading. Serving errors are measured by precision; gemini-3.5-flash-lite serves 6 unrequested dishes and the script 7. The privileged scripted reference shows what a fixed-priority policy achieves with ground-truth stations; it is weakest on separated workspaces (8% completion), where gpt-6-astra reaches 58%. A sampled trajectory-level failure taxonomy is not included.

Reviewer_miB3OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces DuoCook, an Overcooked-inspired benchmark for evaluating vision-language models on centralized two-player coordination in Unreal Engine kitchens. A single VLM controls both players from top-down visual observations supplemented by structured inputs (prop legend, live-state JSON, legal action lists, and a six-response action history), selecting high-level actions and grid destinations while the engine handles navigation. The benchmark comprises 20 maps in five categories that progressively stress equipment contention, dynamic terrain, and mandatory cross-workspace handoffs. Notably, the kitchens themselves are constructed by coding agents via SimWorld Studio with subsequent manual refinement. The evaluation compares four VLMs on 16 maps using five metrics (score, order completion, delivery precision, action acceptance, API cost), plus a memory ablation on 10 maps examining a shared, self-authored, decaying belief store. Results show the benchmark is far from saturated: the best controller completes only 28.1% of orders, two map categories defeat all models entirely, and shared memory helps weak controllers while harming strong ones.

Critical strengths
  • Well-motivated and cleanly isolated evaluation target. By delegating navigation to the engine, pausing simulation during inference, and supplying structured state, the benchmark isolates high-level scheduling and coordination from low-level control confounds. The five-category map design systematically varies the coordination demands (equipment sharing, terrain dynamics, handoff dependencies), and the finding that two categories are uniformly unsolved demonstrates genuine headroom rather than a saturated benchmark.

  • Thoughtful, complementary metric suite. Reporting action acceptance separately from task success is validated by the empirical dissociation between the two (e.g., 71% acceptance with score 8.5 vs. 44.4% acceptance with score 11.9). Including per-episode API cost and token counts addresses practical deployability concerns that most agent benchmarks ignore.

  • The memory ablation is carefully controlled and yields a non-obvious result. Holding all other inputs fixed while toggling only the shared self-authored store isolates the effect of shared memory from recency; the strength-dependent reversal (weaker controllers improve, stronger ones degrade, with a plausible attention-competition mechanism supported by the acceptance-rate drop) is a genuinely interesting scientific finding. The decaying, unvalidated belief store is an elegant design that avoids supplying an "answer key."

Critical weaknesses
  • Despite being framed as a visual-planning benchmark, the controller receives extensive textual scaffolding — a ~4,000-word re-sent system prompt, live-state JSON, legal operation lists, and game-UI information rendered as text — which substantially reduces dependence on visual perception. It is unclear how much of the measured capability is visual grounding versus text-based planning, and no ablation removes the structured inputs to quantify this.
  • The model roster is too small to support the paper's comparative claims. Only four VLMs are evaluated, which is thin coverage for a benchmark paper. With n = 4 controllers, the "memory helps weak controllers but harms strong ones" conclusion effectively rests on a 2-vs-2 split, and the roster also mixes model tiers (a 27B open model alongside frontier proprietary systems), confounding controller strength with model class. Including a broader and more evenly sampled set of models (e.g., mid-tier and open-weight frontier VLMs) would substantially strengthen the generality of these findings.
Questions for authors

See weakness.

1 comment

Thank you for the positive assessment and the two pointed concerns.

W1: How much is visual grounding and how much is text-based planning? We added a matched prompt-text ablation and documented the input channels precisely.

  • What comes from the image (Table 2): station and object locations, which cells hold ingredients, plates, or dishes, pot fill and readiness, board contents, the current terrain configuration, and the partner's position and held item. The live-state JSON carries only what the camera frame does not show: orders, timers, score and combo, the player's own cell and held item, and the partner's current operation and claims. No text channel lists station coordinates, although accepted targets in a player's history, the partner's target cell, and notebook rows carry some spatial information. Because the operation list is unfiltered, the model must also infer feasibility. Aim accuracy reflects target selection, which relies mainly on the image, together with operation feasibility. It ranges from 49.7% to 84.7%, so 15–50% of graded commands miss.
  • Minimal-prompt ablation (§4.6): we replaced the ~3,600-word system prompt with ~440 words, roughly what a human player has: menu, recipes, bare operation names, the grid convention, and the fact that faster service pays more. All station descriptions, strategy, and partner state are removed, and the images are unchanged. Input tokens fall by 42–50%. Scores drop for gpt-6-astra (−14.6 ± 6.6; aim accuracy −11.3 points) and gemini-3.5-flash-lite (−10.3 ± 4.3; aim −16.5 points). The other three models change by at most 2 points. Even with the minimal prompt, gpt-6-astra completes 26.3% of orders and muse-spark-1.3 23.8%, both above the privileged scripted reference (21.3%).
  • Interpretation: the ablation measures the combined effect of reducing text assistance while the visual interface is kept. The lower aim accuracy does not show how much of either score change comes from command validity. Substantial performance remains with the minimal prompt. We did not run an image-removal condition, so the incremental contribution of vision remains unmeasured; §4.6 and the conclusion say so.

W2: Model roster and the "memory helps weak / harms strong" claim. We added a fifth model, muse-spark-1.3, and re-ran the notebook ablation on all 20 maps. The new data do not support the strength-dependent reversal, and we have withdrawn that claim. Paired score changes with the notebook are +4.8 ± 2.2 (claude-opus-5), +5.2 ± 4.6 (muse-spark-1.3), +1.9 ± 3.5 (qwen3.8-27b), −5.7 ± 5.7 (gpt-6-astra), and −5.8 ± 4.6 (gemini-3.5-flash-lite). No paired sign test is significant (smallest p = 0.125). The second-strongest model gains while the strongest loses, so the effect is not ordered by strength. We now report model-specific, descriptive effects only. The conclusion states that five heterogeneous models cannot support general claims about how memory interacts with model capability.

Comments on the paper

1 comment

We thank the reviewers for their careful reading. Across the reviews, the main issues were what the interface actually measures, whether the difficulty is calibrated, and whether the memory specification and numerical record are consistent. We agree that the submission was inconsistent on several of these points. We could not recover the configurations and episode records behind the submitted figures, so we could not reconcile them. Instead, we re-ran the complete benchmark under one fixed protocol and regenerated every performance table and figure from per-episode result files. These 320 episodes replace the submitted results.

What is new (320 episodes, all 20 maps, one episode per model × map × condition):

  • All 20 maps in every experiment. No maps are excluded any longer.
  • Five VLMs: gpt-6-astra, muse-spark-1.3 (new), qwen3.8-27b, claude-opus-5, and gemini-3.5-flash-lite.
  • A privileged scripted reference policy under the same horizon, initial states, and order schedules (§4.3).
  • A prompt-text ablation: the full ~3,600-word prompt is replaced by a ~440-word minimal prompt, with the images unchanged (§4.6).
  • The memory ablation repeated on all 20 maps (§4.5).
  • An information-channel table listing which facts arrive through the image and which through text (Table 2), map-construction records (§3.1, Appendix C), and a capability comparison with related cooking environments (Appendix E).

Main condition (full prompt with notebook; means over 20 maps, ± SE across maps):

ControllerScoreCompletionPrecisionMaps with ≥1 deliveryAim acc.Cost/ep.
gpt-6-astra35.1 ± 5.035.0%100% (28/28)1884.7%$5.41
muse-spark-1.328.6 ± 5.228.8%100% (23/23)1675.6%$1.09
Scripted (privileged)20.2 ± 3.521.3%71% (17/24)14n/a$0
qwen3.8-27b12.8 ± 3.912.5%91% (10/11)855.1%$0.30
claude-opus-58.2 ± 3.48.8%100% (7/7)570.6%$4.61
gemini-3.5-flash-lite7.8 ± 2.77.5%50% (6/12)649.7%$0.33

Corrections to the protocol description, verified against the benchmark code:

  1. Controller organization. One model controls both players, but it is queried separately for each player whenever that player's operation finishes, fails, or stalls. The submission's "single call from a shared context" description was wrong.
  2. Memory. Each player's private six-decision history is present in all conditions. The ablated variable is the shared, decaying, self-authored notebook. §4.4 of the submission was correct; the abstract, introduction, §2.3, and §3.4 were not, and all now match. The submission's claim that the notebook is the only partner channel was also overstated: the live state shows the partner's current operation and claimed stations in both conditions.
  3. Operation list. The full vocabulary is listed at every call without feasibility filtering. The submission's "legal operations" was wrong.
  4. Images versus text. The text contains no station or object coordinates and no pot or board contents, so target locations come mainly from the image. Text still carries some spatial evidence: accepted targets in a player's own history, the partner's target cell, and notebook rows.
  5. Dynamic maps move every 500 steps, not 1,000.

Claims withdrawn because the new data contradict them: "memory helps weak controllers but harms strong ones"; "two categories are unsolved by every controller"; "failures are of completion, not selection". With one episode per cell, rankings and ablation effects are reported as descriptive, with SEs across maps and exact paired sign tests.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.