acceptodds
Under review as a conference paper at ICLR 2027

EMAG: Event Memory Retrieval and Spatial Affordance Grounding for Non-Markovian Robot Manipulation

14 views

Abstract

Many real-world manipulation tasks are non-Markovian: the target of the current action may be determined by an event observed much earlier and therefore cannot be inferred from the current observation alone. Existing memory-augmented policies compress past observations into compact visual representations or textual summaries that track task progress, but do not jointly retain explicit event semantics and the visual evidence needed to re-ground a history-dependent target in the current scene. To address this issue, we introduce EMAG, which uses memory for semantic retrieval to identify which past events are relevant and for visual grounding to determine where the corresponding target is. Specifically, EMAG stores memory as captioned keyframes, pairing RGB snapshots of salient events with concise textual descriptions. Each caption serves as a semantic handle for memory retrieval, while the corresponding keyframe preserves the visual evidence needed for grounding. Given the task goal, progress summary, and recent observations, EMAG queries the stored captions to retrieve relevant memory entries, re-grounds the corresponding target in the current observation, and predicts an affordance map that specifies the target region to act on. The predicted affordance map modulates the visual features of a low-level vision-language-action (VLA) policy to guide action generation. Across all 16 RoboMME tasks, EMAG achieves an average success rate of 73.4%, exceeding the strongest official RoboMME baseline by 28.9 percentage points. On three real-robot tasks requiring object counting, object permanence, and sequence imitation, EMAG achieves an average success rate of 77.2%.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot learning, history-dependent manipulation, embodied memory, vision-language-action models, memory retrieval, affordance grounding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CzJ1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces EMAG, a hierarchical memory-augmented policy for non-Markovian robot manipulation. EMAG stores and retrieves past captioned keyframes with a high-level VLM, and re-grounds it to the current scene using an affordance map. EMAG shows strong performance on 16 RoboMME tasks.

Critical strengths
  • EMAG shows strong performance on RoboMME, outperforming the strongest reported learned baseline by approximately 13 percentage points.
  • The experiments cover both simulation and real-world history-dependent manipulation tasks.
Critical weaknesses
  • Writing and clarity: The method section includes many notations which makes it hard to follow, where it lacks critical information in the main paper. For example, it does not clearly specify the complete inputs to the low-level VLA policy, although it is modified from a standard pi0.5. Also the formulation is also tied to a specific observation setup, consisting of one front-view and one wrist-view. The experiment section provides insufficient details about the compared methods and oracle references, such as how the oracle signals are constructed and provided to the policy, making it difficult to solely clarify the contributions of EMAG.
  • The paper includes some ablation studies, but lacks in-depth analysis aligned with the paper's central motivation. For example, it does not include why the spatial grounding mechanism is important for performance, nor the efficiency claim of selective retrieval.
  • The paper argues existing methods “do not jointly retain explicit event semantics and the visual evidence”, however, controlled comparisons among caption-only, keyframe-only, and captioned-keyframe memories are missing.
Questions for authors

See weaknesses also.

  • What is the additional inference cost introduced by the integration of high-level policy?
  • Can the authors provide a controlled comparison with prior memory-based methods using the same high-level and low-level base models?
Reviewer_oHPZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EMAG is hierarchical framework for storing and retrieving relevant text-annotated keyframes, and grounding high-level decisions in spatial affordance maps in the current image view for a low-level VLA policy to follow. The authors evaluate EMAG on 16 tasks in RoboMME and 3 real-world tasks that require memory.

Critical strengths
  1. EMAG shows strong results on the RoboMME benchmark against the best baseline from the benchmark.
  2. The method is clearly explained in section 3. Generally, the overall organization, the equations, and the figures are clear.
  3. The show EMAG on 3 real-world tasks that clearly require memory.
Critical weaknesses

I think this paper need stronger ablations experiments to justify their design choices. One useful ablation would be the exclusion of the affordance map all together and the introduction of grounded subgoals (text subgoals with spatial coordinates). RoboMME has shown that grounded subgoals can help convey spatial information to the low-level policy. This would test if the spatial map can more effectively convey spatial information than text.

Also, there are no ablations on the text progress memory. Although previous works have shown this to be beneficial, the extent to its contribution in EMAG is unclear. It is hard to isolate its affect. Finally, extending the ablations studies to more RoboMME tasks would strengthen the design choices.

Questions for authors
  1. Could you compare EMAG with a variant that replaces affordance maps with predicted grounded text subgoals containing spatial coordinates? Beyond the oracle comparison, does spatial-map conditioning outperform grounded text in the learned system?
  2. Could you ablate the text progress memory to quantify its contribution and determine whether it provides complementary benefits beyond the other memory components?
Reviewer_6NokOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses history-dependent manipulation tasks in which the current observation does not reveal the correct interaction target. EMAG stores salient events as captioned RGB keyframes and uses a shared high-level VLM in two rounds. Round 1 updates the progress summary and event memory, predicts a language subgoal, and selects relevant memory identifiers from their captions. Round 2 combines the retrieved image–caption pairs with the current observation and subgoal to predict a dense target-affordance map. This map gates additive modulation of front-view features in a separately trained π₀.₅ controller. The method reports 73.42% average success across all 16 RoboMME tasks and 77.2% across three real-robot tasks.

Critical strengths
  1. The problem is well motivated. Identifying a relevant past event and locating its target in the current scene are distinct requirements, and the proposed architecture provides a coherent connection between them. The explicit memory and spatial interface also make the system easier to inspect.

  2. The empirical gains are substantial and cover the complete benchmark. EMAG improves over the strongest official RoboMME baseline by 28.91 percentage points and over the included base-data PonderPounce result by 12.59 points. Evaluation uses three independently trained seeds and 150 rollouts per task.

  3. The real-robot experiments provide useful evidence beyond simulation. The three tasks cover counting, occluded-target selection, and sequence imitation. Learned methods use the same training episodes and evaluation configurations. Button Replay excludes button identity labels from language inputs, making visual event evidence particularly relevant.

Critical weaknesses
  1. The affordance-as-context ablation compares two ways to provide the same spatial signal. It does not establish the improvement over the same event-memory planner communicating through language alone. The real-robot baselines also lack EMAG’s language subgoals and spatial supervision, while simulation baseline results are imported from other reports. These comparisons establish strong system performance but provide weaker attribution to the proposed mechanism.

  2. Removing retrieval lowers average success from 77.67% to 62.17%, but the largest decrease is on VideoPlaceButton, from 81.33% to 31.33%. VideoUnmask slightly improves, and RouteStick decreases by only 3.34 points. The results demonstrate a substantial average benefit on this subset, especially for temporal reference resolution, rather than uniform necessity across task types. Input lengths and truncation behavior should also be reported because the all-memory variant supplies more images under a finite context limit.

  3. The paper specifies scheduling and deployment hardware but does not report measured Round 1, Round 2, or combined inference latency. Round 1 reads all accumulated captions, so its context grows with event count despite selective image retrieval in Round 2. Memory occupancy, retrieved-image counts, and context-limit handling would help assess the computational motivation.

Questions for authors
  • Can you report a no-affordance control on the existing four-task subset, retaining the same high-level event memory, retrieval, and predicted language subgoals?

  • What are the input-token and image counts for the full model and all-memory variant? How are inputs exceeding the context limit handled?

  • Using existing rollouts, can you quantify event-writing/progress-update errors and identify the dominant failure mechanisms in the two Swap tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.