acceptodds
Withdrawn from ICLR 2027

MemAct: Streaming Spatial Understanding with Memory and Active Exploration

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Spatial understanding is essential for embodied agents navigating and interacting with the physical world. As observations arrive incrementally, agents must reason in a streaming manner. Yet streaming understanding must also be closed-loop: task-relevant evidence may require both recalling past observations and actively acquiring new views. Existing streaming approaches largely emphasize processing incoming observations, leaving task-driven active exploration underexplored. We introduce MemAct, a framework for streaming spatial understanding with memory and action. A shared vision–language model incrementally writes structured historical memory, retrieves records and past frames on demand, and acquires additional evidence through movement and turning. Retrieved evidence and action observations inform subsequent decisions, enabling the agent to continue exploring or answer within a unified reasoning loop. We develop the MemAct suite, a data generation pipeline and collection across real and simulated environments, with approximately 1M training examples for spatial reasoning, memory writing, retrieval, and exploration. Its MemAct-bench comprises 20,000 main QA pairs evaluating current-observation understanding, historical evidence use, and active exploration, plus 2,000 separate Timing questions. MemAct combines supervised fine-tuning for memory and interaction with counterfactual action optimization using cumulative evidence gains, action costs, and terminal answer quality to improve exploration and stopping decisions. Experiments show that MemAct outperforms existing spatial understanding and streaming baselines and approaches frontier multimodal models on MemAct-bench while retaining streaming inference efficiency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spatial understanding, Active vision

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jEEEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose a method for maintaining long-term memory for embodied QA, with output for retrieval, answer or collect more data. They build a dataset (eval + train) and show improvements over the eval with their trained model, and also over other benchmarks.

Critical strengths
  • The problem of collecting enough data to answer embodied QA, maintaining long history and acting on more data collection is an important open problem.

  • The results over other benchmarks are important

Critical weaknesses
  • Important parts of the paper are not readable or unclear. For example:

    • Sec 3 and sec 4 read like a is a list of parts and pieces without any clarity on how they fit together and what they are used for. It is impossible to assess the paper's contributions at this readability level.
    • "preferentially evicted" - what does this mean?
    • "A deterministic harness retains structured memory and historical images independently, enabling retrieval beyond the visual context" - how?
    • "remaining budgets at round k" - what does this mean?
    • "Update()" is a critical component of the proposed approach, yet it is not described at all in the paper, not the inputs, not the output, not the function itself.
  • I want to stress that I could not understand the paper even when reading it throughly, I could not understand the benchmark, how it compares to other benchmarks, the memory and action approach architecture etc.

Questions for authors

I ask for the authors to resubmit after a significant rewrite of the paper.

Reviewer_69vEOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MemAct, a framework for streaming spatial understanding that combines memory writing, memory retrieval, and active exploration within a single streaming VLM. The authors additionally introduce a large-scale training dataset (~1M examples), a new benchmark (MemAct-Bench), and a two-stage training approach combining supervised learning with counterfactual action optimization. Experimental results show improvements over existing streaming and spatial reasoning baselines.

Critical strengths

The main strength of the paper is the problem formulation. Combining streaming understanding with both memory retrieval and active evidence acquisition is a natural and interesting extension beyond passive streaming QA. The paper is also ambitious in scope, contributing not only a model but also a large training corpus, a benchmark, and a training strategy. The evaluation is extensive and includes memory and action ablations that generally support the main claims.

Critical weaknesses

W1: My main concern is the limited breakdown of the benchmark results. Appendix A.8 gives a detailed question catalog, and Tables 2 and 3 provide useful ablations. Still, the reported scores mostly combine quite different tasks within Current, Memory, and Action QA. I would like to see results by question type and some analysis of where the pipeline fails. For example, when a Memory question is answered incorrectly, was the relevant information never recorded, not retrieved, or retrieved but then used incorrectly? The current results do not tell us much about this. The action setting is also quite restricted: five fixed camera commands and at most three attempts. This is a valid setting for studying active view acquisition, but it leaves open how well the policy handles longer exploration or different movement scales. Table 4 provides evidence of transfer in spatial QA, which is useful, but it does not test transfer of the memory-and-action loop.

W2: The answer distribution raises a more specific concern. Table 1 includes several bias baselines, which is good. However, the modal-answer-content baseline already reaches 54.76% on Action ACC, compared with 58.28% for MemAct-9B. This makes the categorical Action results harder to interpret on their own. A breakdown by action question type, or results on an answer-balanced subset, would help. The numerical Action results and no-action ablation provide additional evidence, so I would not dismiss the gains based on this baseline alone.

Questions for authors
  1. A key purpose of benchmarks is often to identify concrete model weaknesses and guide future research directions. Beyond the Current/Memory/Action split, how does MemAct-Bench enable introspection into model failure modes? Can the authors provide evidence that the benchmark diagnoses specific reasoning deficiencies (e.g., retrieval failures, exploration failures, geometric reasoning failures, stopping failures) rather than primarily reporting aggregate task performance?

  2. Could you report results by question template and analyze a sample of failures? The existing ablations are useful, but they do not separate memory-writing errors, retrieval errors, reasoning errors, and poor exploration or stopping decisions.

  3. How many distinct layouts are used for Action training and testing? Given the 12 training layouts reported in Appendix B.1, can you provide a breakdown of performance across held-out layouts?

  4. The modal-answer-content baseline reaches 54.76% Action ACC. Which question types drive this result, and how does MemAct perform on an answer-balanced subset?

Reviewer_EMGTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies streaming spatial QA in which one vision-language model periodically writes structured memory records, retrieves records or past frames, and issues discrete camera actions to acquire new views. It contributes about 1M training examples, a 22,000-question benchmark, and a two-stage recipe of SFT plus counterfactual bandit optimization, reporting gains over streaming baselines and near-frontier accuracy.

Critical strengths

The strengths of the paper are listed as follows:

  • Well written and easy to follow. Section 1 states three challenges and points to Sections 3, 4, and 4.4 for their treatment; Figures 1 and 3 contrast the offline, passive, and closed-loop settings.
  • Unified decision interface. One VLM emits memory writes, retrieval requests, five camera commands, or answers through a single generation interface, with a deterministic router and no depth or 3D map at inference.
  • Informative ablations. Table 2 isolates memory content, retention, and write interval: full memory adds 13.59/15.48 ACC/MRA over no explicit memory, and ground-truth memory bounds the remaining gap at 2.66/3.34. Table 3 separates reward terms and budgets and includes a GRPO control.
Critical weaknesses

The weaknesses of the paper are listed as follows:

  • Uncited near-identical prior work. S3-Bench/AMF-VLM [1] poses time-anchored streaming spatial QA with periodic memory folding and move/rotate/scan actions on 3D-FRONT plus ScanNet-family scans, a 20,000-question test set, and fine-tuned StreamingVLM baselines. Section 2 omits it, along with LMEE [2] and VG-AVS [3], so contribution 1 is overstated.
  • Backbone and training confound. MemAct uses Qwen3.5-4B/9B, but Table 1 omits the backbones of StreamingVLM (Qwen2.5-VL-7B) [4], Stream3D-VLM, and Cambrian-S-FT, and whether they received action or memory supervision. The 5.88/11.58-point margin (Section 5.2) may reflect a newer base model; the no-memory variant (54.52 Memory ACC, Table 2) trails StreamingVLM (58.85).
  • Exploration validated only inside its own simulator. Action QA is 3D-FRONT-only with five fixed-magnitude commands (Section 3.3; Table 10), and Table 3’s evidence-gain metrics are the training reward itself (Eqs. 19 to 21). No external active benchmark (EXPRESS-Bench [5]; ESI-Bench [6]) is used, and zero-shot GPT-6 Astra exceeds MemAct on Action ACC (59.57 vs. 58.28, Table 1).

[1] Wei, Yuxi, et al. "See, Remember, Explore: A Benchmark and Baselines for Streaming Spatial Reasoning." arXiv preprint arXiv:2603.23864 (2026).

[2] Wang, Sen, et al. "Explore with long-term memory: A benchmark and multimodal llm-based reinforcement learning framework for embodied exploration." arXiv preprint arXiv:2601.10744 (2026).

[3] Koo, Juil, et al. "Toward ambulatory vision: Learning visually-grounded active view selection." arXiv preprint arXiv:2512.13250 (2025).

[4] Xu, Ruyi, et al. "Streamingvlm: Real-time understanding for infinite video streams, 2025." arXiv preprint arXiv:2510.09608 (2025).

[5] Jiang, Kaixuan, et al. "Beyond the destination: A novel benchmark for exploration-aware embodied question answering." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

[6] Hong, Yining, et al. "ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop." arXiv preprint arXiv:2605.18746 (2026).

Questions for authors

It would be great if the authors could focus on my concerns in the Weaknesses section in the rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.