What's Past is Prologue: Benchmarking Prospective Memory Grounded in Retrospective Action Understanding

Memory enables an agent to use experience that is no longer directly observable when making a current decision. Existing multimodal memory evaluations have largely examined whether models retain historical information, answer explicit questions about past events, or execute deferred intentions under specified conditions. A complementary challenge arises in cooperation: an agent must determine which elements of a shared interaction history remain applicable in the current situation and ground them in immediate perception to choose a concrete next action. We introduce **CoopPortal**, a multimodal benchmark for studying this problem in long-horizon two-player cooperative puzzle solving. CoopPortal contains two independently annotated tracks over 118 Portal 2 levels. **Track A** evaluates retrospective action understanding by requiring models to reconstruct the 1,585 key atomic actions that directly contributed to the observed successful solutions from complete dual-player episodes. **Track B** evaluates prospective action prediction under strict partial observability: given only a designated player's first-person video and the shared dialogue before a decision boundary, models must predict the next atomic action in the recorded trajectory, its purpose, and the historical cues reported in support of the prediction. Track B contains 369 decision points and prevents access to the partner's current view and to all future observations. To provide a reproducible reference approach, we propose **APEX**, a training-free inference pipeline that constructs dialogue-anchored multimodal micro-events, distills them into structured experience memory over trials, mechanisms, commitments, and uncertainties, and combines this memory with dense recent visual grounding for action prediction. Evaluations across frontier, open-source, and task-specific video models reveal substantial variation across retrospective reconstruction and prospective action prediction, particularly on complex and long-tail actions. CoopPortal provides a diagnostic setting for studying how multimodal agents transform accumulated cooperative experience into decision-relevant action understanding.