- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
DuoCook introduces 20 Unreal Engine cooking maps spanning five categories for visually assisted two-player planning and coordination, with engine-executed navigation and structured state inputs (Secs. 3.1–3.5). The paper evaluates four VLMs on 16 maps and reports a memory ablation on 10 maps using task performance, action acceptance, and API expenditure (Secs. 4.1–4.4).
Critical strengths
- The five map categories explicitly vary equipment sharing, changing terrain, and required handoffs, creating interpretable coordination challenges rather than relying on a single kitchen layout (Sec. 3.2, lines 173–186; Appendix A, Table 2).
- The evaluation separates completed orders and reward from command acceptance and API cost, making it possible to distinguish successful task execution from merely valid actions (Sec. 4.2, lines 286–320).
Critical weaknesses
- The benchmark has not established which visual and coordination abilities its scores actually measure. The controller receives player positions, held items, station claims, orders, preparation status, and state-dependent legal operations; the engine handles navigation and pauses during inference (Secs. 3.4–3.5, lines 205–246). The grid-labeled image is intended to locate action targets (lines 236–238), but the reported experiments vary the model or memory without a matched image-removal control that tests its contribution beyond the supplied state. Low completion therefore does not distinguish visual grounding failures from recipe scheduling, interface use, or deadline management. Difficulty is also uncalibrated: scene validation checks actors, layout, and navigation constraints, but the paper reports no human or scripted/planning reference establishing attainable completion under the same 4,000-step horizon and order schedules (Secs. 3.1 and 4.1; Appendices A–B). Such a reference matters particularly where the VLMs complete no orders. The existing map taxonomy and separate action-validity metric are useful, yet do not by themselves establish that the suite diagnoses the intended capabilities. Matched modality controls and a feasible reference policy would make the benchmark's distinct evaluation value substantially clearer.
- The memory and category-level conclusions are not supported by a consistent experimental specification and result record. The introduction and Sec. 3.4 describe one shared controller and an ablation of six-response action history, whereas Sec. 4.4 retains that history, removes a self-authored decaying notebook, and calls the notebook the only channel between players with private histories (lines 61–63, 207–215, 415–422, and 457–471). These descriptions imply different interventions and information access, so the observed effect cannot yet be attributed to a defined memory mechanism. The numerical account is also inconsistent: Fig. 4 shows 12.5% completion for gpt-6-astra in separated workspaces despite the claim that all controllers deliver nothing there, and Fig. 3 reports 29.2% completion and 94.4% precision rather than the prose's 28.1% and 100% (lines 360–374). Finally, one episode per model/map/condition cannot assess repeatability; the correctly described bootstrap over maps in Fig. 5 does not resolve this. The authors acknowledge the descriptive nature of the results, but claims about controller-specific memory effects still require a coherent protocol and reconciled episode records, with broader conclusions limited to what the available repetitions support.
Questions for authors
- [Clarification; Weakness 1] Which task-relevant facts must be inferred from the image rather than the live-state JSON, legal-operation list, or rejection feedback? Please provide the existing observation schema, especially for station locations and terrain changes, and identify any already available image-removal or text-only results. A matched modality study is a suggestion for further validation if no such results exist.
- [Clarification; Weakness 1] What existing playthroughs, scripted solutions, or validation records demonstrate achievable order completion on the difficult maps under the exact episode horizon and order schedule? Please distinguish actor/navigation checks from evidence that the cooking and handoff workload is feasible, and explain how this informs the interpretation of zero-completion episodes.
- [Clarification; Weakness 2] What memory mechanism and controller configuration actually generated Figs. 3–5: one shared controller with six-response history, or separate player contexts with a shared notebook? Please supply the relevant enabled/disabled prompt difference and information-access schema, and reconcile the conflicting descriptions in Secs. 3.4 and 4.4.
- [Clarification; Weakness 2] Can the existing per-map episode records reproduce the authoritative completion, precision, and category-level results and resolve the cited figure/prose discrepancies? Please state the model sampling settings and whether any repeat runs already exist; if they do not, delimit the ranking and memory conclusions accordingly. Additional repeated trials and reference-policy evaluations are future validation suggestions, not requests for a new experimental campaign during rebuttal.
3 comments
Remaining concerns about benchmark interpretation and calibration
Thank you for the additional evidence and for openly addressing the problems in the original evaluation. I appreciate the replacement evaluation on all 20 maps, the scripted reference, and the clarified observation and notebook interfaces. These resolve several of my earlier concerns.
My main reservation is still what the benchmark scores tell us about visual planning. The input breakdown makes clear that the images contain important information. However, the minimal-prompt experiment retains the images while reducing text assistance, so it does not isolate the contribution of visual grounding. A low completion score could still arise from target localization, recipe scheduling, or coordination failures, and the current evidence does not separate these explanations sufficiently.
The scripted reference also helps establish feasibility, but only partially. There is now at least one demonstrated delivery on every map, yet no reported episode completes all four orders. The attainable completion level therefore remains unclear, making it difficult to judge how far the evaluated controllers are from a reasonable solution. In addition, there is still only one episode per model, map, and condition. Describing the rankings and notebook effects as descriptive is appropriate, but it does not resolve their repeatability.
I appreciate the substantial effort behind the new evaluation. Given the remaining uncertainty about what the scores diagnose and how the task difficulty should be interpreted, I will keep my original recommendation.
Thank you for the follow-up and for your careful reviews.
We agree with the remaining points. The minimal-prompt ablation measures sensitivity to text assistance, not the contribution of vision (§4.6). Aim accuracy does not separate image misreading from operation choice, nor scheduling from coordination failures. Isolating these would need a coordinate-supplied text interface and a trajectory-level failure analysis, which we have not run.
On feasibility, the best observed episodes complete three of four orders on five maps, and no controller completes all four. These are lower bounds, not an estimate of attainable completion. With one episode per cell, the rankings and notebook effects remain descriptive.
We appreciate your engagement with the revision.
Thank you for the precise questions. Every answer below is verified against the benchmark code and the new per-episode records.
Q1: Which facts must come from the image? See Table 2. Image only: station and object locations, which cells hold ingredients, plates, or dishes, pot fill and readiness, board contents, the current terrain configuration, and the partner's position and held item. The live-state JSON holds only facts outside the camera frame: orders with timers and tip zone, earnings, combo, next order, own cell and held item, and the partner's current operation, target cell, and claims. It contains no station coordinates, and the minimal prompt removes the partner fields. The operation list is the full unfiltered vocabulary, and rejection feedback never reveals the correct cell. Terrain changes are therefore not stated in text. Text still carries some spatial evidence: accepted targets in a player's history, the partner's target cell, and notebook rows. To measure sensitivity to text assistance, we added the minimal-prompt ablation (§4.6), which keeps the images and removes about 88% of the system-prompt text. Two of five models lose more than two points, and gpt-6-astra still completes 26.3% of orders. We did not run an image-removal condition, so the incremental contribution of vision remains unmeasured; we state this in §4.6.
Q2: Evidence of attainable completion. We added a privileged scripted policy under the exact horizon, initial states, and order schedules (§4.3). It delivers on 14 of 20 maps. On the other six (1-1, Kevin 1, 4-1, 4-3, 2-4, 3-2), at least one VLM episode delivers, so every map has at least one demonstrated delivery. The best observed episode per map completes one order on five maps, two on ten, and three on five; no episode completes all four. This establishes partial feasibility; whether the full four-order workload is attainable remains unknown. Appendix C documents the construction checklists and which records survive; these establish environment construction, not workload feasibility. We interpret zero-delivery episodes as controller failures on maps where delivery is demonstrated, not as evidence of infeasibility.
Q3: Which memory mechanism and controller configuration? Per-player calls: the same model is queried separately for each player, with stateless calls. Both conditions contain the images, legend, live state (including the partner's operation and claims), full operation vocabulary, failure feedback, and each player's private last-six-decision history. The memory condition adds only the shared notebook: up to 3 writes per decision of (cell, kind, status, note), author-tagged, 600-step half-life with partner rows decaying 2× faster, three confidence bands, and at most 24 rows (16 per author). The system prompt grows from ~3,600 to ~4,000 words to explain it. Every section now uses this one specification.
Q4: Reproducibility of the record. The discrepancies you identified were real. Submitted Fig. 3 gives gpt-6-astra 29.2% completion and 94.4% precision against 28.1% and 100% in the prose; submitted Fig. 4 shows 12.5% completion in separated workspaces against the prose's zero; and Fig. 3 gives claude-opus-5 and gemini-3.5-flash-lite acceptance of 52.4%/30.1% against 71.0%/44.4% in the prose. We could not recover the configurations or episode records that generated the submitted figures and prose, so we cannot determine which values were correct. We therefore designate a replacement evaluation as authoritative: 320 new episodes under the protocol in §4.1, with every performance table and figure regenerated from their per-episode result files. Fig. 5 (Appendix B) reports every episode's score, and Appendix B states that these results replace the submitted ones. The two versions also differ in map set and in how precision is aggregated, so their numbers are not directly comparable. Settings: all models run with reasoning enabled at low effort, horizon 4,000 steps, and 4 orders per episode. qwen3.8-27b is accessed via OpenRouter and the others via an OpenAI-compatible API gateway (§4.1). No repeated runs exist. Rankings and ablation effects are therefore stated as descriptive, with SEs across maps and paired sign tests, and the "memory helps weak / harms strong" conclusion has been withdrawn.