- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper is based on the hypothesis that when an embodied VLM navigates an environment, answering long-horizon questions about past observations requires a retained memory record (like an episodic scene graph) rather than re-processing frame sequences. However, how a pretrained VLM should best consume this memory remains an open question.
And propose a solution/benchmark framework called Room to Remember: Instead of relying on answer-time visual grounding, Room to Remember evaluates how VLMs read structured episodic scene graphs (objects, rooms, spatial relations, and sighting timestamps) behind a blank answer image. They test four fine-tuned memory arms: No memory, text memory (rendered scene graph text), graph memory, and text + graph memory.
The paper contributes:
- A controlled memory benchmark: 5,700 ProcTHOR episodes (evaluating an 800-question test set across 8 question families) paired with a blank-image protocol that isolates memory retrieval from visual perception.
- A comparison of memory interfaces: Evaluating natural text serialization against learned GNN feature injections across two VLM backbones (MomaGraph-R1 and Gemma-3-4B).
And the results report:
- Rendering scene graphs as text improves accuracy on MomaGraph-R1 and Gemma-3-4B over no-memory controls.
- Accuracy gains concentrate heavily in route-history questions, with almost no effect on static set-based questions.
- R-GCN graph feature injections show no statistically significant improvement over no-memory baselines.
Critical strengths
The strength of this approach, I think, is its clean, isolated memory evaluation setup. By evaluating models behind a blank image, it decouples memory retrieval from active exploration or answer-time visual inspection. It also gives a very pragmatic result for embodied AI: skip the complex GNN feature injections and just fine-tune the VLM to read the scene graph as structured text.
Critical weaknesses
There are 2 weaknesses of this paper I believe are critical:
-
The paper evaluates text scene graphs against complex GNN embeddings, but completely skips the most natural visual baseline: fine-tuning the VLM directly on raw frame sequences. In embodied AI, evaluating whether structured text outperforms raw visual context requires testing against fine-tuned video/frame sequences, not just zero-shot or oracle-selected frames.
-
Dataset reliability is a major issue: 397 out of 800 test questions (~50%) are probe-sensitive, where models score well above chance using language/environmental priors or option formatting cues without looking at memory. I'm not sure this makes for a good long-term benchmark artifact. I also think labeling this a "long-horizon" benchmark is misleading. While the robot navigates 40+ frames, questions are evaluated against a sub-sampled 6-frame window (which are unfortunately not "released")
Questions for authors
A few questions and stylistic thoughts:
- Were there any ablations done on the two individual graph pathways (pooled input residual vs. node-level output attention) in Figure 1 to see their isolated contributions?
- Are the raw video trajectories or sampled frame-strip indices available for reconstructing the visual evidence behind the ground-truth labels?
Revised PDF uploaded
Dear Reviewers and Area Chair,
Thank you for the reviews and discussion. We have uploaded a revised PDF. Changes are coloured by reviewer (blue = vtnf, red = iQzt, green = BRRB, purple = several), and the analyses added during the discussion are in Appendix B.1–B.3. One minor correction is noted in B.1: fixing a relation-label convention in the simulator's log changes the visibility-log control by under one point, and no conclusion changes. We are happy to answer any remaining questions.