- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a next-action prediction and relevant action grounding benchmark in a cooperative setting. More specifically it builds on the Portal Dialogue Corpus dataset (cooperative Portal 2 game) and adds critical action annotations (atomical, e.g. pick up companion cube), the purpose of the action, and supporting cues (I find "supporting cues" pretty vague, an explicit example would be helpful; the suppl also does not help too much in this regard). Only actions which were part of the solution for solving the puzzle are annotated. The paper introduce two tasks: (1) Identifying the critical actions. (2) Next action prediction.
They introduce a framework APEX (agent harness) for solving both tasks. One component summarizes visual information surrounding user communication. A second component distills beliefs, but contains zero examples and was therefore very difficult to follow. Then depending on task (1) or task (2) there is a different prompt to solve the task.
The authors benchmark various closed source models and various open source models. For the open-source models Qwen3-VL-4B/8B and DeepSeekV4-LFlash they apply their APEX framework. The demonstrate that APEX in top of the open-sourcemodls yield decent improvements (DeepSeek-V4.1 Task A goes from 27.3 F1 to 29.7, task B improves from 43.2 F1 to 49.1). However it still trails the closed-source models without APEX.
Critical strengths
- Paper addresses the cooperative setting, which is an important topic.
- The authors introduce a new dataset in this cooperative setting.
- The dataset was human-annotated and curated
- The APEX framework yields benefits for open-source models.
Critical weaknesses
MAJOR: The cooperative setting is reduced to a single-agent prediction task. While this is convenient for evaluation, it does remove quite a bit of challenge from the approach. In contrast, MultiAgentBench, ACL 2005, https://arxiv.org/pdf/2503.01935 and MAS-Bench, ACL 2026, https://arxiv.org/pdf/2610.04672 actually introduce interactive cooperative settings in which agents need to solve problems together, which is much closer to real-life problems. So the main question is: What does this paper provide in addition to these two works?
Medium: I miss a human baseline and applying APEX to at least one frontier model.
Comments on the method section:
- Section 4.2 seems to introduce math for complicating the paper rather than for helping, a typical LLM paper-writing habit. There are triplets which have a threshold \theta (275) but then if no candidate clears the threshold, it uses the one with minimum similarity, removing any effect of the threshold. Then T^s_j and T^e_j are introduced which are never reused again and which only makes it hard to read. It is just a timespan, no need for math here.
- Section 4.3: Examples would really help here to explain what is going on.
- The title of 4.4 does not match the name in Fig. 2.
- L308: only in the supplementary material one can see what typical atomic actions are. An additional example would help the reader here, especially for readers not familiar with the game.
- I recommend dropping a digit from all numbers. The current four digits are both hard to read and the variance between runs is way to high to justify such precision even while keeping the prompts identical.
Questions for authors
See weaknesses, but spend little to none of the rebuttal on my comments on method section.
5 comments
Response to Reviewer eyva -W2 & W3
Weakness 2: Human baseline and frontier backbone
A2: Thank you for these two suggestions. We address the human baseline first and then the frontier model.
The current version does not include a human performance baseline, and we do not report one here. CoopPortal is designed as a diagnostic benchmark for comparing how models convert cooperative history into action. The reference annotations are produced by human annotators under the quality-control procedure of Appendix B.4 (independent annotation, reconciliation and a third annotator for disputed cases), and our automatic Purpose judge is validated against human labels. On 100 predictions, it agrees with two independent annotators in 81.0% (κ = 0.62) and 89.0% (κ = 0.78) of cases (Appendix B.5). The conclusions we draw are comparisons between models, backbones and inputs. To our knowledge, the interactive multi-agent benchmarks MultiAgentBench (Zhu et al., 2025) and MASBench (Chu et al., 2026) likewise do not report human performance on their tasks. We will state in the Limitations that the paper does not include a human performance reference, and mark it as a direction for future work.
Table 1 already reports APEX on three different backbones (Qwen3-VL-4B, Qwen3-VL-8B and DeepSeek-V4.1-Flash), and it improves over its own backbone on every one of them. To cover a stronger model as well, we additionally ran APEX end to end on Gemini-3.7-Flash with the same prompts and hyperparameters as the DeepSeek run. The four backbones on Track B are compared below.
| Model | Track A | Track B | |||||
|---|---|---|---|---|---|---|---|
| CA-P | CA-R | CA-F1 | SlotScore | ActSim | PM-F1 | Purpose | |
| Qwen3-VL-4B | 15.3 | 9.9 | 11.2 | 31.2 | 47.4 | 48.0 | 22.4 |
| Qwen3-VL-8B | 16.9 | 11.6 | 13.0 | 43.4 | 55.0 | 47.9 | 27.2 |
| DeepSeek-V4.1-Flash | 34.4 | 24.8 | 27.3 | 50.9 | 60.7 | 58.0 | 43.2 |
| Gemini-3.7-Flash | 48.8 | 38.8 | 40.6 | 55.1 | 62.8 | 62.9 | 50.8 |
| APEX(Qwen3-VL-4B) | 22.5 | 26.0 | 22.0 | 43.0 | 48.1 | 49.2 | 32.1 |
| APEX(Qwen3-VL-8B) | 36.9 | 20.5 | 24.0 | 50.9 | 58.7 | 54.6 | 36.0 |
| APEX(DeepSeek-V4.1-Flash) | 39.4 | 25.8 | 29.7 | 52.6 | 63.4 | 58.4 | 49.1 |
| APEX(Gemini-3.7-Flash) | 48.0 | 42.7 | 42.9 | 56.9 | 62.9 | 64.1 | 52.5 |
Larger gains appear on the Qwen3-VL variants, which we attribute to their lower baseline performance and more room for improvement. Stronger backbones such as DeepSeek-V4.1-Flash and Gemini-3.7-Flash already achieve high baseline performance, leaving less room for absolute improvement.
References
Zhu K, Du H, Hong Z, et al. Multiagentbench: Evaluating the collaboration and competition of llm agents[C]//Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025: 8580-8622.
Chu Q, Yu Z, Wen S, et al. MASBench: Benchmarking LLM-based Multi-Agent Collaboration under Partial Observability[J]. arXiv preprint arXiv:2610.04672, 2026.
Weakness 3: Comments on the method section
A3: Thank you for the detailed comments on the method section. Appendix D.3 already gives a detailed description of the APEX implementation, together with an example of time-aligned trunks and an example of the structured experience memory. We will add a reference to it in Section 4.3.
We will recheck the remaining points and revise them in the updated version.
Dear Reviewer eyva,
Could you check if the response has addressed your issues? Since the discussion is closing soon, your reply would be very important.
Best,
AC
Response to Reviewer eyva
Weakness 1: Contribution relative to MultiAgentBench and MASBench
A1: Thank you for pointing us to these two works; we will cite and discuss both in Related Work. They and CoopPortal ask different questions.
The contribution of CoopPortal is a new problem setting rather than a new game environment. To our knowledge, it is the first benchmark of prospective memory grounded in retrospective action understanding. Given the shared history of a cooperative episode, the model must itself decide which parts of that history still apply at the current moment, and ground them in what it currently sees to choose a concrete next action, without a query or trigger that points to the relevant evidence. Existing memory benchmarks test whether information can be retained, retrieved when asked, or acted on once a specified condition fires. We test whether a model can use the past to decide what to do next, and we score the evidence it relies on separately from the action it chooses. Portal 2 is the testbed for this question, since it offers long-horizon cooperation with plan revisions and verifiable atomic actions, and the formulation itself is not tied to the game.
MultiAgentBench evaluates LLM-based multi-agent systems in interactive scenarios, with task-completion and milestone-based metrics, and compares coordination protocols and topologies (star, chain, tree, graph). MASBench evaluates multi-agent collaboration under partial observability, with three task categories (Reasoning, Scheduling, Game) and three mechanisms (Protocol, Memory, Routing), scored by performance, communication cost and cost effectiveness. Both study how to organize LLM agents so that they solve tasks together, and the collaboration happens among LLM agents. CoopPortal instead asks how a model turns the shared history of a human cooperative episode into a grounded action. Our data are 11.5 hours of recorded Portal 2 play by human pairs, with spontaneous speech, timing offsets between talk and action, plan revisions, and each player seeing only their own view. This adds three things that interactive benchmarks do not provide:
We made the task offline on purpose. Portal 2 is a commercial game without a standard interface for LLM agents, and a fixed recorded history lets us attribute differences between models to the model, not to the behaviour of a simulated partner or to compounding errors. The cooperative structure remains in the inputs. Track B depends on the partner's utterances, role assignments and commitments, and Track A reconstructs the actions of both players from both views (716 BLUE-only, 748 ORANGE-only and 164 joint actions).
In the revision we will (i) add a paragraph on MultiAgentBench and MASBench to Related Work, stating the difference above; (ii) state the contribution in the Introduction as a benchmark of prospective memory grounded in retrospective action understanding, evaluated offline on human cooperation, and describe Portal 2 as its testbed rather than presenting CoopPortal as an interactive multi-agent benchmark; and (iii) add a Limitations paragraph on the offline, single-view nature of Track B, leaving interactive evaluation to future work.