acceptodds
Under review as a conference paper at ICLR 2027

What's Past is Prologue: Benchmarking Prospective Memory Grounded in Retrospective Action Understanding

Abstract

Memory enables an agent to use experience that is no longer directly observable when making a current decision. Existing multimodal memory evaluations have largely examined whether models retain historical information, answer explicit questions about past events, or execute deferred intentions under specified conditions. A complementary challenge arises in cooperation: an agent must determine which elements of a shared interaction history remain applicable in the current situation and ground them in immediate perception to choose a concrete next action. We introduce **CoopPortal**, a multimodal benchmark for studying this problem in long-horizon two-player cooperative puzzle solving. CoopPortal contains two independently annotated tracks over 118 Portal 2 levels. **Track A** evaluates retrospective action understanding by requiring models to reconstruct the 1,585 key atomic actions that directly contributed to the observed successful solutions from complete dual-player episodes. **Track B** evaluates prospective action prediction under strict partial observability: given only a designated player's first-person video and the shared dialogue before a decision boundary, models must predict the next atomic action in the recorded trajectory, its purpose, and the historical cues reported in support of the prediction. Track B contains 369 decision points and prevents access to the partner's current view and to all future observations. To provide a reproducible reference approach, we propose **APEX**, a training-free inference pipeline that constructs dialogue-anchored multimodal micro-events, distills them into structured experience memory over trials, mechanisms, commitments, and uncertainties, and combines this memory with dense recent visual grounding for action prediction. Evaluations across frontier, open-source, and task-specific video models reveal substantial variation across retrospective reconstruction and prospective action prediction, particularly on complex and long-tail actions. CoopPortal provides a diagnostic setting for studying how multimodal agents transform accumulated cooperative experience into decision-relevant action understanding.

Primary area:
datasets and benchmarks
Keywords:
Prospective Memory, Benchmark & Evaluation Protocol, Collaborative Video Understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eyvaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a next-action prediction and relevant action grounding benchmark in a cooperative setting. More specifically it builds on the Portal Dialogue Corpus dataset (cooperative Portal 2 game) and adds critical action annotations (atomical, e.g. pick up companion cube), the purpose of the action, and supporting cues (I find "supporting cues" pretty vague, an explicit example would be helpful; the suppl also does not help too much in this regard). Only actions which were part of the solution for solving the puzzle are annotated. The paper introduce two tasks: (1) Identifying the critical actions. (2) Next action prediction.

They introduce a framework APEX (agent harness) for solving both tasks. One component summarizes visual information surrounding user communication. A second component distills beliefs, but contains zero examples and was therefore very difficult to follow. Then depending on task (1) or task (2) there is a different prompt to solve the task.

The authors benchmark various closed source models and various open source models. For the open-source models Qwen3-VL-4B/8B and DeepSeekV4-LFlash they apply their APEX framework. The demonstrate that APEX in top of the open-sourcemodls yield decent improvements (DeepSeek-V4.1 Task A goes from 27.3 F1 to 29.7, task B improves from 43.2 F1 to 49.1). However it still trails the closed-source models without APEX.

Critical strengths
  • Paper addresses the cooperative setting, which is an important topic.
  • The authors introduce a new dataset in this cooperative setting.
  • The dataset was human-annotated and curated
  • The APEX framework yields benefits for open-source models.
Critical weaknesses

MAJOR: The cooperative setting is reduced to a single-agent prediction task. While this is convenient for evaluation, it does remove quite a bit of challenge from the approach. In contrast, MultiAgentBench, ACL 2005, https://arxiv.org/pdf/2503.01935 and MAS-Bench, ACL 2026, https://arxiv.org/pdf/2610.04672 actually introduce interactive cooperative settings in which agents need to solve problems together, which is much closer to real-life problems. So the main question is: What does this paper provide in addition to these two works?

Medium: I miss a human baseline and applying APEX to at least one frontier model.

Comments on the method section:

  • Section 4.2 seems to introduce math for complicating the paper rather than for helping, a typical LLM paper-writing habit. There are triplets which have a threshold \theta (275) but then if no candidate clears the threshold, it uses the one with minimum similarity, removing any effect of the threshold. Then T^s_j and T^e_j are introduced which are never reused again and which only makes it hard to read. It is just a timespan, no need for math here.
  • Section 4.3: Examples would really help here to explain what is going on.
  • The title of 4.4 does not match the name in Fig. 2.
  • L308: only in the supplementary material one can see what typical atomic actions are. An additional example would help the reader here, especially for readers not familiar with the game.
  • I recommend dropping a digit from all numbers. The current four digits are both hard to read and the variance between runs is way to high to justify such precision even while keeping the prompts identical.
Questions for authors

See weaknesses, but spend little to none of the rebuttal on my comments on method section.

5 comments
AuthorsOpenReview ↗

Response to Reviewer eyva

Weakness 1: Contribution relative to MultiAgentBench and MASBench

A1: Thank you for pointing us to these two works; we will cite and discuss both in Related Work. They and CoopPortal ask different questions.

The contribution of CoopPortal is a new problem setting rather than a new game environment. To our knowledge, it is the first benchmark of prospective memory grounded in retrospective action understanding. Given the shared history of a cooperative episode, the model must itself decide which parts of that history still apply at the current moment, and ground them in what it currently sees to choose a concrete next action, without a query or trigger that points to the relevant evidence. Existing memory benchmarks test whether information can be retained, retrieved when asked, or acted on once a specified condition fires. We test whether a model can use the past to decide what to do next, and we score the evidence it relies on separately from the action it chooses. Portal 2 is the testbed for this question, since it offers long-horizon cooperation with plan revisions and verifiable atomic actions, and the formulation itself is not tied to the game.

MultiAgentBench evaluates LLM-based multi-agent systems in interactive scenarios, with task-completion and milestone-based metrics, and compares coordination protocols and topologies (star, chain, tree, graph). MASBench evaluates multi-agent collaboration under partial observability, with three task categories (Reasoning, Scheduling, Game) and three mechanisms (Protocol, Memory, Routing), scored by performance, communication cost and cost effectiveness. Both study how to organize LLM agents so that they solve tasks together, and the collaboration happens among LLM agents. CoopPortal instead asks how a model turns the shared history of a human cooperative episode into a grounded action. Our data are 11.5 hours of recorded Portal 2 play by human pairs, with spontaneous speech, timing offsets between talk and action, plan revisions, and each player seeing only their own view. This adds three things that interactive benchmarks do not provide:

  1. Real human cooperation as the source of history and of ground truth. The labels are what human players actually did, not what simulated agents did.
  2. Evidence-level evaluation. Each Track B decision point carries a human-annotated purpose and supporting historical cues (928 cues). We can therefore score which part of the history a model relies on (PM-F1) separately from whether the action is right. A task-success metric cannot distinguish a correct action chosen for the right reason from one chosen by a local heuristic. Section 5.4 shows that these two can decouple.
  3. Multimodal grounding. The model must align first-person video with temporally offset dialogue under partial observability, and the partner's current view is withheld.

We made the task offline on purpose. Portal 2 is a commercial game without a standard interface for LLM agents, and a fixed recorded history lets us attribute differences between models to the model, not to the behaviour of a simulated partner or to compounding errors. The cooperative structure remains in the inputs. Track B depends on the partner's utterances, role assignments and commitments, and Track A reconstructs the actions of both players from both views (716 BLUE-only, 748 ORANGE-only and 164 joint actions).

In the revision we will (i) add a paragraph on MultiAgentBench and MASBench to Related Work, stating the difference above; (ii) state the contribution in the Introduction as a benchmark of prospective memory grounded in retrospective action understanding, evaluated offline on human cooperation, and describe Portal 2 as its testbed rather than presenting CoopPortal as an interactive multi-agent benchmark; and (iii) add a Limitations paragraph on the offline, single-view nature of Track B, leaving interactive evaluation to future work.

AuthorsOpenReview ↗

Response to Reviewer eyva -W2 & W3

Weakness 2: Human baseline and frontier backbone

A2: Thank you for these two suggestions. We address the human baseline first and then the frontier model.

The current version does not include a human performance baseline, and we do not report one here. CoopPortal is designed as a diagnostic benchmark for comparing how models convert cooperative history into action. The reference annotations are produced by human annotators under the quality-control procedure of Appendix B.4 (independent annotation, reconciliation and a third annotator for disputed cases), and our automatic Purpose judge is validated against human labels. On 100 predictions, it agrees with two independent annotators in 81.0% (κ = 0.62) and 89.0% (κ = 0.78) of cases (Appendix B.5). The conclusions we draw are comparisons between models, backbones and inputs. To our knowledge, the interactive multi-agent benchmarks MultiAgentBench (Zhu et al., 2025) and MASBench (Chu et al., 2026) likewise do not report human performance on their tasks. We will state in the Limitations that the paper does not include a human performance reference, and mark it as a direction for future work.

Table 1 already reports APEX on three different backbones (Qwen3-VL-4B, Qwen3-VL-8B and DeepSeek-V4.1-Flash), and it improves over its own backbone on every one of them. To cover a stronger model as well, we additionally ran APEX end to end on Gemini-3.7-Flash with the same prompts and hyperparameters as the DeepSeek run. The four backbones on Track B are compared below.

ModelTrack ATrack B
CA-PCA-RCA-F1SlotScoreActSimPM-F1Purpose
Qwen3-VL-4B15.39.911.231.247.448.022.4
Qwen3-VL-8B16.911.613.043.455.047.927.2
DeepSeek-V4.1-Flash34.424.827.350.960.758.043.2
Gemini-3.7-Flash48.838.840.655.162.862.950.8
APEX(Qwen3-VL-4B)22.526.022.043.048.149.232.1
APEX(Qwen3-VL-8B)36.920.524.050.958.754.636.0
APEX(DeepSeek-V4.1-Flash)39.425.829.752.663.458.449.1
APEX(Gemini-3.7-Flash)48.042.742.956.962.964.152.5

Larger gains appear on the Qwen3-VL variants, which we attribute to their lower baseline performance and more room for improvement. Stronger backbones such as DeepSeek-V4.1-Flash and Gemini-3.7-Flash already achieve high baseline performance, leaving less room for absolute improvement.

References

Zhu K, Du H, Hong Z, et al. Multiagentbench: Evaluating the collaboration and competition of llm agents[C]//Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025: 8580-8622.

Chu Q, Yu Z, Wen S, et al. MASBench: Benchmarking LLM-based Multi-Agent Collaboration under Partial Observability[J]. arXiv preprint arXiv:2610.04672, 2026.

Weakness 3: Comments on the method section

A3: Thank you for the detailed comments on the method section. Appendix D.3 already gives a detailed description of the APEX implementation, together with an example of time-aligned trunks and an example of the structured experience memory. We will add a reference to it in Section 4.3.

We will recheck the remaining points and revise them in the updated version.

Dear Reviewer eyva,

Could you check if the response has addressed your issues? Since the discussion is closing soon, your reply would be very important.

Best,

AC

Reviewer_hPXpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CoopPortal, a benchmark for understanding and predicting actions in cooperative gameplay. Track A reconstructs critical actions from complete episodes. Track B predicts the next action, its purpose, and supporting historical cues from limited observations. The proposed APEX pipeline combines structured experience memory with recent visual evidence and improves several tested backbones.

Critical strengths
  1. The benchmark studies a useful problem: applying past cooperative experience to a current action. Separating action prediction, purpose, and supporting evidence provides a more detailed evaluation than a single accuracy score.

  2. The paper provides detailed annotation rules, cross-review, and a human check of the purpose judge. APEX also improves all three tested backbones across both tracks.

Critical weaknesses
  1. The Track A matching rule in Section 3.3 does not enforce temporal order. As written, a reference sequence [A, B] and a prediction [B, A] can receive the same score as the correct sequence, since matched predictions only need to be unused. This weakens the claim that CA-F1 measures reconstruction of an ordered action sequence. The authors should clarify the rule and evaluate whether incorrect action order affects the reported results.

  2. The experiments do not isolate the benefit of structured memory from the extra processing in APEX. APEX includes window-level processing, memory distillation, and final reasoning, while the baselines predict directly. The paper does not report inference costs or compare methods under equal processing budgets. A simple summary baseline using the same visual evidence and a similar inference budget would help show whether the four memory categories are necessary.

Questions for authors

Refer to the weaknesses above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer hPXp -W1

Weaknesses 1: Temporal order in the Track A matching rule

A1: Thank you for the careful reading. We clarify the rule and also test whether order affects the results.

CA-P / CA-R / CA-F1 measure how many of the causally necessary actions are covered and how precise the predicted actions are, weighted by match quality, and the chronological order is the required output format. We will state this explicitly in Section 3.3 and Appendix B.5. To test whether order changes the conclusions, we re-scored all existing predictions with an order-constrained matcher that keeps the same pair score and threshold. It finds the maximum-weight alignment in which matched pairs are strictly increasing in both reference and prediction order (a weighted longest common subsequence):

max⁡M ∑(i,j)∈Mq(ri,pj)s.t. q(ri,pj)≥τ,  M strictly increasing in both i and j\max_{M}\ \sum_{(i,j)\in M} q(r_i,p_j)\quad \text{s.t.}\ q(r_i,p_j)\ge\tau,\ \ M \text{ strictly increasing in both } i \text{ and } j

With the original rule, we reproduce Table 1 exactly for every system below. "Matched actions out of order" is the share of matched actions under the original rule that are not part of the longest increasing subsequence of matched prediction positions.

ModelOriginal CA-P / CA-R / CA-F1 (%)Order-constrained CA-P / CA-R / CA-F1 (%)Δ CA-F1 (%)Matched actions out of order (%)
Gemini-3.7-Flash48.8 / 38.8 / 40.645.0 / 35.6 / 37.4−3.221.8
GPT-5.439.7 / 35.9 / 35.535.2 / 31.9 / 31.5−4.023.3
Kimi-K2.642.5 / 32.5 / 34.837.7 / 28.7 / 30.7−4.124.0
APEX(DeepSeek)39.4 / 25.8 / 29.730.6 / 21.7 / 24.5−5.223.8
DeepSeek-V4.1-Flash34.4 / 24.8 / 27.326.7 / 19.9 / 21.6−5.724.4
VideoChat340.4 / 22.2 / 27.035.7 / 19.6 / 23.8−3.228.6
MiniMax-M331.8 / 21.9 / 24.527.9 / 19.4 / 21.6−2.926.3
APEX(Qwen3-VL-8B)36.9 / 20.5 / 24.030.7 / 16.9 / 19.8−4.231.7
APEX(Qwen3-VL-4B)22.5 / 26.0 / 22.018.8 / 22.0 / 18.4−3.634.5
InternVL2.522.0 / 16.2 / 16.818.4 / 13.7 / 14.1−2.732.7
Qwen3-VL-8B16.9 / 11.6 / 13.014.9 / 10.4 / 11.5−1.524.8
Qwen3-VL-4B15.3 / 9.9 / 11.213.4 / 8.8 / 9.9−1.320.5

The model ranking by CA-F1 changes in only one place, where DeepSeek-V4.1-Flash drops below VideoChat3 and MiniMax-M3. The APEX gains over its backbones also remain. Under order-constrained scoring, CA-F1 improves from 11.5% to 19.8% on Qwen3-VL-8B and from 9.9% to 18.4% on Qwen3-VL-4B, compared with 13.0% to 24.0% and 11.2% to 22.0% under the original rule. Absolute scores are lower when order is enforced, by 1.3% to 5.7% in CA-F1 (3.5% on average), since 20.5% to 34.5% of the matched actions are out of order.

CoopPortal separates diagnostic dimensions throughout, for example action correctness from evidence fidelity in Track B. In the same spirit, CA-P / CA-R / CA-F1 score which causal actions a model identifies, and order is assessed as its own dimension, so a model that finds the right actions but misplaces some is not scored as if it had missed them. A single metric that merges the two would also treat the gold sequence as the only valid ordering, whereas it is one chronological linearization of actions executed by two players. Where their actions are independent, their relative order is set by timing rather than by the causal structure of the solution. We report order directly. Among matched pairs of actions, 77.2% are in the correct relative order (chance is 50%), and this varies across systems from 65.1% (InternVL2.5) to 83.4% (Gemini-3.7-Flash). We therefore keep CA-P / CA-R / CA-F1 as the primary Track A metrics and we will add pairwise order accuracy and the order-constrained scores above as a sensitivity analysis in the appendix.

AuthorsOpenReview ↗

Response to Reviewer hPXp -W2

Weakness 2: Structured memory vs. extra processing

A2: Thank you for this point. We report the inference cost, and then show with two ablations in Table 2 and a frame-matched baseline that the gains come from structured memory rather than extra processing.

APEX makes one VLM call per window, one distillation call and one final call. The number of windows depends only on the input history. On our 369 Track B decision points it is 11.75 on average (median 9; 4,336 windows in total), so APEX uses 13.75 calls per decision point on average. With DeepSeek-V4.1-Flash, this corresponds to about 38k input and 3.5k output tokens per decision point (estimated from the token usage logs of the three stages). We will report calls, frames and tokens for every method in the paper.

The following ablations keep the processing fixed, and we add a baseline that receives the same number of input frames as APEX (all with Qwen3-VL-8B).

Method (Qwen3-VL-8B)Track ATrack B
CA-PCA-RCA-F1SlotScoreActSimPM-F1Purpose
Baseline16.911.613.043.455.047.927.2
Baseline (matched frames)15.513.213.444.154.846.926.4
APEX36.920.524.050.958.754.636.0
w/o adaptive30.117.919.548.658.850.934.4
w/o experience---47.356.445.629.3
  • w/o experience is the unstructured counterpart of our memory. It uses exactly the same windows (all window calls), the same time-aligned trunks, tail frames and grounding-zone dialogue, and differs only in that the distilled memory Z is replaced by the raw trunks T. It therefore costs K + 1 calls instead of K + 2, within one call of APEX and about 12 calls more than a baseline. Removing the distillation lowers SlotScore from 50.9% to 47.3%, PM-F1 from 54.6% to 45.6% and Purpose from 36.0% to 29.3%, and its PM-F1 (45.6%) falls below that of the single-call baseline (47.9%). More processing over the same visual evidence therefore does not explain the gains, and the distillation into structured memory does.
  • w/o adaptive keeps the same number of windows, trunks and memory calls as APEX and changes only how windows are selected. At identical cost, APEX still improves CA-F1 by 4.5% on Track A and SlotScore and PM-F1 by 2.3% and 3.7% on Track B.
  • Baseline (matched frames) is a single-call baseline that receives the same number of input frames as APEX. It changes little relative to the baseline (SlotScore 43.4% to 44.1%, PM-F1 47.9% to 46.9%, CA-F1 13.0% to 13.4%) and stays far below APEX (50.9%, 54.6% and 24.0%), so extra frames alone do not explain the gains.

These results show that distilling the evidence into structured memory helps compared with feeding the same evidence undistilled.

The author addressed my concerns in the rebuttal, and I will improve my score

Reviewer_QeMKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CoopPortal, a benchmark built on two-player Portal 2 co-op recordings from the public Portal Dialogue Corpus. It has two independently annotated tracks:

  • Track A (111 episodes): from both players' videos plus the full dialogue, the model must reconstruct the causally necessary sequence of atomic actions.

  • Track B (369 decision points): from one player's first-person video plus the dialogue up to a cutoff, the model must predict that player's next atomic action, its purpose, and the past cues that support it.

Scoring uses soft embedding/lexical matching plus an LLM judge for purpose. The authors also propose APEX, a training-free pipeline that captions dialogue-anchored video windows, condenses them into a four-part memory (trials, mechanisms, commitments, uncertainties), and adds dense frames from the last 10 s. Across nine models, the best Track A CA-F1 is 40.6 and the best Track B SlotScore is 57.5. APEX lifts Qwen3-VL-8B from 13.0 to 24.0 CA-F1 and from 47.9 to 54.6 PM-F1.

Critical strengths
  1. The two-track design is well-motivated. The tracks separate retrospective reconstruction from prospective prediction. The input builder strictly enforces the pre-cutoff boundary.

  2. The baseline setup is valid and broad. Nine closed, open and video-specialized models share one input package; prompts are given verbatim.

Critical weaknesses
  1. Some necessary analyses are missing. The contributions claim a "saturation gap on class-balanced metrics" and a "capability watershed on complex manipulation actions". The abstract also mentions "long-tail actions". No such analysis exists.

  2. APEX's gains are hard to attribute. APEX makes one VLM call per window. then a distillation call, then a final call with 16 tail frames. Baselines get one call with 6–28 frames. There is no baseline with a matched budget and no ablation without the tail frames. Besides, gains shrink on DeepSeek (+1.72 SlotScore, +0.43 PM-F1), and APEX is not tested on GPT-5.4 or Gemini, so "architecture-agnostic" is weakly supported.

Questions for authors
  1. What do dialogue-only, video-only, last-10-s-only and metadata-only baselines score on each track?

  2. What is the mean predicted list length N̂ on Track A? How do the results change if the 5–10 action guidance is removed and joint actions are allowed?

  3. Where are the class-balanced and per-action-type results claimed in the abstract and contributions?

6 comments
AuthorsOpenReview ↗

Response to Reviewer QeMK -W1

Weekness 1 & Question 3: Class-balanced and per-action-type results

A1: Thank you for raising this. We have added a class-balanced, per-action-type analysis of Track B, computed from our existing per-sample judgments.

We group the 369 decision points by gold action type. The action distribution is long-tailed (17 types), and a single sample can strongly affect the mean of a small type, so we merge types with fewer than 7 samples into one "other" group (9 groups in total). We report macro-SlotScore, the equal-weight mean of the per-type means, a class-balanced metric that gives every action type the same weight.

Modelplaceredirectenterpasspressstandstepinsertothermacro
Qwen3-VL-4B36.99.528.738.540.28.38.323.812.322.9
+ APEX48.457.134.630.145.316.74.816.717.930.2
Qwen3-VL-8B59.42.333.041.725.50.00.04.810.819.7
+ APEX60.958.138.034.939.726.25.619.013.732.9
DeepSeek-V4.1-Flash59.544.834.626.654.946.424.135.733.340.0
+ APEX54.860.644.439.355.457.114.835.749.545.7

The per-type scores show differences between models that a single aggregate does not. Qwen3-VL-8B scores 59.4% on place, close to APEX(Qwen3-VL-8B) at 60.9%, yet only 2.3% on redirect and 0.0% on stand and step. Its score falls from 43.4% (Table 1) to 19.7% (macro), the largest drop among these systems. Some action types are hard for all systems: none of the six reaches more than 24.1% on step, 35.7% on insert or 41.7% on pass, while place goes up to 60.9%. The backbones also separate clearly on redirect and stand, where the two Qwen3-VL base models score at most 9.5% and 8.3%, and DeepSeek-V4.1-Flash scores 44.8% and 46.4%. APEX also yields decent gains on long-tail categories, especially for DeepSeek-V4.1-Flash (+16.2%).

APEX's gains tend to appear on types where its backbone scores low. redirect improves for all three backbones (+47.7%, +55.8%, +15.8%). For Qwen3-VL-8B, place changes by only +1.5%, whereas stand (+26.2%), press (+14.2%) and insert (+14.3%) improve more, which gives a macro gain of +13.2%. The macro gains are +7.2% for Qwen3-VL-4B and +5.8% for DeepSeek-V4.1-Flash. Since several types contain few samples, we draw conclusions from large differences and from patterns that hold across several types.

Based on this analysis, we will add these results to the revised version.

AuthorsOpenReview ↗

Response to Reviewer QeMK -W2&W3

Weekness 2: Attribution of APEX gains to inference budget and tail frames

A2: Thank you for this important point. We first clarify that two existing ablations in Table 2 are already compute-controlled. As requested, we then add two experiments, an ablation without the tail frames and a baseline with matched input frames. All experiments use Qwen3-VL-8B.

  • w/o adaptive keeps the same number of windows, trunks, and memory calls as APEX, changing only how windows are selected. At identical cost, APEX still improves CA-F1 by 4.5% on Track A and SlotScore / PM-F1 by 2.3% / 3.7% on Track B.
  • w/o experience feeds the final reasoner the same time-aligned trunks, tail frames, and grounding-zone dialogue, replacing only the distilled memory Z with the raw trunks T (one fewer call). Removing distillation lowers PM-F1 from 54.6% to 45.6% and Purpose from 36.0% to 29.3%. Notably, its PM-F1 (45.6%) falls below the single-call baseline (47.9%), so additional processing alone does not explain the gains.
  • Baseline (matched frames) is a single-call Qwen3-VL-8B baseline that receives the same number of input frames as APEX.
  • New ablation, w/o tail frames. The final call receives the same distilled memory Z and the same grounding-zone dialogue as APEX, but none of the 16 tail frames; everything else is unchanged.
Method (Qwen3-VL-8B)Track ATrack B
CA-PCA-RCA-F1SlotScoreActSimPM-F1Purpose
Baseline16.911.613.043.455.047.927.2
Baseline (matched frames)15.513.213.444.154.846.926.4
APEX36.920.524.050.958.754.636.0
w/o adaptive30.117.919.548.658.850.934.4
w/o experience---47.356.445.629.3
w/o tail frames---37.846.152.429.2

Giving the baseline the same number of input frames as APEX changes little (SlotScore 43.4% to 44.1%, PM-F1 47.9% to 46.9%, CA-F1 13.0% to 13.4%), far below APEX (50.9%, 54.6% and 24.0%), so the gains do not come from the number of frames. Removing the tail frames lowers SlotScore from 50.9% to 37.8% and ActSim from 58.7% to 46.1%, below the baseline (43.4% and 55.0%). This matches their role in APEX's design, since the dense tail frames ground the immediate action in the current visual state while the memory supplies the earlier history.

Weekness 3: Generality of APEX across backbones

A3: We appreciate this valuable comment. Table 1 already reports APEX on three different backbones (Qwen3-VL-4B, Qwen3-VL-8B and DeepSeek-V4.1-Flash), and it improves over its own backbone on every one of them. To cover a stronger model as well, we additionally ran APEX end to end on Gemini-3.7-Flash with the same prompts and hyperparameters as the DeepSeek run. The four backbones on two Tracks are compared below.

ModelTrack ATrack B
CA-PCA-RCA-F1SlotScoreActSimPM-F1Purpose
Qwen3-VL-4B15.39.911.231.247.448.022.4
Qwen3-VL-8B16.911.613.043.455.047.927.2
DeepSeek-V4.1-Flash34.424.827.350.960.758.043.2
Gemini-3.7-Flash48.838.840.655.162.862.950.8
APEX(Qwen3-VL-4B)22.526.022.043.048.149.232.1
APEX(Qwen3-VL-8B)36.920.524.050.958.754.636.0
APEX(DeepSeek-V4.1-Flash)39.425.829.752.663.458.449.1
APEX(Gemini-3.7-Flash)48.042.742.956.962.964.152.5

As observed, APEX consistently brings positive gains across all tested models, but the improvement magnitude varies significantly. Larger gains are observed on Qwen3-VL variants, which we attribute to their lower baseline performance and more room for improvement due to smaller model capacity. In contrast, stronger backbones like DeepSeek-V4.1-Flash and Gemini-3.7-Flash already achieve high baseline performance, leaving limited room for absolute improvement. This indicates that APEX is compatible with diverse vision-language backbones without architecture-specific modifications, while the magnitude of improvement correlates with the baseline performance and headroom of the base model.

AuthorsOpenReview ↗

Response to Reviewer QeMK -Q1&Q2

Question 1: Single-input baselines

A1: We ran the four single-input baselines on both tracks with DeepSeek-V4.1-Flash as the backbone. Each baseline uses the same task instruction, rules and output schema as the full-input baseline in Table 1, with one call per sample; only the inputs differ.

  • Dialogue-only: the dialogue (Track A: the full episode dialogue; Track B: the dialogue before the cut-off, plus the acting player), no frames.
  • Video-only: the sampled frames of the full-input baseline, no dialogue.
  • Last-10-s-only (Track B only): the dialogue in the last 10 s before the cut-off and 16 frames from the same window (1 fps over [t−10 s, t−5 s], 2 fps over [t−4.5 s, t]), i.e., the tail-frame budget of APEX. It is not defined for Track A, which has no cut-off.
  • Metadata-only: the session, course and level identifiers, plus the level duration (Track A) or the acting player and the cut-off time (Track B); no dialogue, no frames.
InputTrack B: SlotScoreActSimPM-F1PurposeTrack A: CA-PCA-RCA-F1
Full input (Table 1)50.960.758.043.234.424.827.3
Dialogue-only48.058.753.937.429.020.222.6
Video-only45.559.654.516.329.419.421.8
Last-10-s-only47.061.157.816.3–––
Metadata-only36.354.043.713.620.414.215.8

No single input matches the full input on SlotScore, Purpose or Track A. On Track B, the best single-input SlotScore is 48.0% (dialogue-only) versus 50.9% for the full input, and on Track A, dialogue-only and video-only reach CA-F1 of 22.6% and 21.8% versus 27.3%. Dialogue and frames play different roles: Purpose depends mainly on the dialogue (37.4% for dialogue-only versus 16.3% for both video-only and last-10-s-only), while action slots draw on both (SlotScore is 48.0% with dialogue only, 45.5% with frames only and 50.9% with both).

We will add the table to the appendix and refer to it from Section 5.2.

Question 2: Track A output length and prompt constraints

A2: The mean predicted length N^\hat{N} over the 111 episodes is shown below.

ModelMean N^\hat{N}MedianRangeShare of episodes with 5–10 actions
GPT-5.411.55116–1946%
Kimi-K2.69.6795–1774%
MiniMax-M38.3685–1292%
Qwen3-VL-4B8.86105–1292%
Qwen3-VL-8B9.05105–1694%
APEX(Qwen3-VL-4B)15.80165–4529%
APEX(Qwen3-VL-8B)7.6972–2475%

Most baselines output close to the recommended 5–10 actions. This is consistent with Table 1, where CA-P is higher than CA-R for most models, and it caps the recall that a model can reach under this prompt. We will add N^\hat{N} to the appendix.

Removing the "5–10 actions" guidance. We removed the line "Recommended length: 5 to 10 actions, but correctness is more important than length" from the Track A prompt of the DeepSeek-V4.1-Flash baseline and changed nothing else.

DeepSeek-V4.1-FlashCA-PCA-RCA-F1Mean N^\hat{N}
With guidance (Table 1)34.4224.8027.269.03
Without guidance35.0826.6828.639.84

Without the guidance, CA-F1 rises by 1.37% (recall by 1.88%), and the model still writes about 10 actions on average (median 10; 65% of episodes within 5–10 actions). The guidance is therefore not what limits the length of the predicted lists or the recall: the model does not produce much longer lists when it is removed, and the scores change only slightly.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions

Dear Area Chair, thank you for handling our submission, and we thank all three reviewers for their careful reading and constructive suggestions. We have submitted a revised PDF in which all new and changed text is marked in blue. The main changes are as follows.

Evaluation and metrics

  1. We add a class-balanced analysis on long-tail action types, with per-action-type results reported, and we revise the related claims in the contributions (Appendix D.5; Section 5.2).
  2. We add a sensitivity analysis on temporal order for Track A, with order-constrained scores and pairwise order accuracy (Appendix D.9).
  3. We report PM-F1 stratified by the age of the gold cues (Appendix D.8).

Attribution of the gains of APEX

  1. We add two controls, an ablation without the tail frames and a baseline with the same number of input frames as APEX (Section 5.3, Table 2).
  2. We report single-input baselines (dialogue-only, video-only, last-10-s-only and metadata-only) on both tracks (Appendix D.6).
  3. We report the mean predicted length of each system on Track A (Appendix D.7).
  4. We add the results of APEX on Gemini-3.7-Flash (Table 1).

Positioning and presentation

  1. We cite MultiAgentBench and MASBench and state how CoopPortal differs from them (Related Work).

We hope that these changes address the concerns raised during the review. The remaining disagreement is with Reviewer eyva, concerning the positioning of CoopPortal relative to interactive multi-agent benchmarks such as MultiAgentBench and MASBench, and we have clarified the differences in our replies.

AuthorsOpenReview ↗

Summary of Rebuttal

Summary of Rebuttal

Dear Area Chair, thank you for handling our submission. The reviewers' concerns fall into three groups: analyses missing behind our claims, the attribution of the gains of APEX, and the positioning of CoopPortal. We addressed each with new experiments or analyses, all included in the revised PDF (see our Summary of Revisions).

Reviewer QeMK asked for the analyses behind the class-balanced and long-tail claims, controls on the inference budget, and further baselines and backbones. We added a per-action-type analysis, together with an ablation without the tail frames, a baseline with matched input frames, single-input baselines, and APEX on Gemini-3.7-Flash.

Reviewer hPXp asked about temporal order in Track A and about structured memory versus extra processing. Re-scoring all predictions with an order-constrained matcher changes the ranking of systems in only one place and leaves the gains of APEX in place, and a variant that keeps the same evidence but removes the distilled memory falls below the single-call baseline on PM-F1.

Reviewer eyva asked what CoopPortal adds beyond MultiAgentBench and MASBench. We explained that CoopPortal differs from both in three ways. First, they evaluate task outcomes, whereas CoopPortal scores the evidence behind a decision, through human-annotated supporting cues, separately from the action. Second, memory is a component of the system under study in both, whereas in CoopPortal it is the capability being tested. Third, both operate on text, whereas CoopPortal takes video and dialogue as input.

We hope that the new experiments, analyses and the revised PDF address the reviewers' concerns, and we would be grateful if the Area Chair could take them into account.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.