acceptodds
Under review as a conference paper at ICLR 2027

Room to Remember: Knowledge-Graph Memory for Long-Horizon Embodied Question Answering

Abstract

An assistant asked about a room it has already left cannot answer from its current image. We study how a vision-language model should read a retained record of those observations. Our benchmark in simulated ProcTHOR houses pairs recorded trajectories with questions about object locations, room visits, temporal order, and accumulated observations. Predicted scene graphs record objects, rooms, relations, and sighting times; we train answerers to read them as text, learned graph features, or both. Training and evaluation use blank answer images to separate memory access from answer-time vision. On 800 questions from 200 test episodes, the textual graph-memory representation improves MomaGraph-R1 accuracy by 6.62 percentage points over a no-memory control across two training seeds, an effect concentrated in route-history questions (+18.9 points) with little change on the rest (). Learned graph features alone show no detectable improvement over no memory on either backbone at either seed. These comparisons favor text within the tested architectures and training schedules. Blind probes fall below chance on the families carrying the gain and well above it on the rest, so aggregate accuracy reflects both retained evidence and question priors. The results support training a text reader as a baseline for episodic memory.

Primary area:
datasets and benchmarks
Keywords:
embodied question answering, episodic memory, scene graphs, knowledge graph memory, vision-language models, benchmark, long-horizon reasoning, ProcTHOR, memory-augmented fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_iQztOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper is based on the hypothesis that when an embodied VLM navigates an environment, answering long-horizon questions about past observations requires a retained memory record (like an episodic scene graph) rather than re-processing frame sequences. However, how a pretrained VLM should best consume this memory remains an open question.

And propose a solution/benchmark framework called Room to Remember: Instead of relying on answer-time visual grounding, Room to Remember evaluates how VLMs read structured episodic scene graphs (objects, rooms, spatial relations, and sighting timestamps) behind a blank answer image. They test four fine-tuned memory arms: No memory, text memory (rendered scene graph text), graph memory, and text + graph memory.

The paper contributes:

  1. A controlled memory benchmark: 5,700 ProcTHOR episodes (evaluating an 800-question test set across 8 question families) paired with a blank-image protocol that isolates memory retrieval from visual perception.
  2. A comparison of memory interfaces: Evaluating natural text serialization against learned GNN feature injections across two VLM backbones (MomaGraph-R1 and Gemma-3-4B).

And the results report:

  • Rendering scene graphs as text improves accuracy on MomaGraph-R1 and Gemma-3-4B over no-memory controls.
  • Accuracy gains concentrate heavily in route-history questions, with almost no effect on static set-based questions.
  • R-GCN graph feature injections show no statistically significant improvement over no-memory baselines.
Critical strengths

The strength of this approach, I think, is its clean, isolated memory evaluation setup. By evaluating models behind a blank image, it decouples memory retrieval from active exploration or answer-time visual inspection. It also gives a very pragmatic result for embodied AI: skip the complex GNN feature injections and just fine-tune the VLM to read the scene graph as structured text.

Critical weaknesses

There are 2 weaknesses of this paper I believe are critical:

  1. The paper evaluates text scene graphs against complex GNN embeddings, but completely skips the most natural visual baseline: fine-tuning the VLM directly on raw frame sequences. In embodied AI, evaluating whether structured text outperforms raw visual context requires testing against fine-tuned video/frame sequences, not just zero-shot or oracle-selected frames.

  2. Dataset reliability is a major issue: 397 out of 800 test questions (~50%) are probe-sensitive, where models score well above chance using language/environmental priors or option formatting cues without looking at memory. I'm not sure this makes for a good long-term benchmark artifact. I also think labeling this a "long-horizon" benchmark is misleading. While the robot navigates 40+ frames, questions are evaluated against a sub-sampled 6-frame window (which are unfortunately not "released")

Questions for authors

A few questions and stylistic thoughts:

  • Were there any ablations done on the two individual graph pathways (pooled input residual vs. node-level output attention) in Figure 1 to see their isolated contributions?
  • Are the raw video trajectories or sampled frame-strip indices available for reconstructing the visual evidence behind the ground-truth labels?
Reviewer_BRRBOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how VLMs should use past observations to answer questions their current view cannot resolve. The authors build a trajectory-based QA benchmark in simulated houses, construct episodic memories from predicted scene graphs, and compare four memory interfaces on two VLM backbones, with blank images at answer time. Trained textual memory improves accuracy, mainly on questions about earlier sightings and room visits, while graph features alone yield no detectable gain over no memory. Analyses of question priors and reference-label validity delimit the scope of these findings.

Critical strengths
  1. The proposed benchmark is useful for evaluating episodic memory.
  2. The paper uses blind and options-only probes to assess shortcuts in the benchmark.
Critical weaknesses
  1. Human validation covers only Presence-change and Global-count. The low agreement with the reference answers, especially for Global-count, raises concerns about label quality, while the other six question families remain unaudited.
  2. Questions and reference answers are generated from at most six sampled frames, even when the questions refer to the whole trajectory or house. This mismatch can make the reference answer inconsistent with the question, and the missing sampling indices prevent verification.
  3. The episodic scene graph is aggregated from frame-level predictions without tracking object identities across frames. This limits its ability to distinguish repeated observations of the same object from different instances and may hurt performance on tasks such as Global-count.
  4. The lack of improvement from graph features may reflect insufficient training data. The paper does not examine how graph-reader performance changes with training-data scale.
Questions for authors
  1. For Global-count, why are reference answers computed from at most six frames when the question can refer to the full trajectory? Do the main results hold with references computed from all recorded frames?
  2. How does the performance gap between text and graph memory change as the training dataset grows?
Reviewer_vtnfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how an embodied agent can answer questions from memory rather than from the current image. Specifically, it tests how a fine-tuned vision-language model should read an episodic memory record built from the same source observations. Trajectories are recorded in simulated ProcTHOR houses, and a prompted vision-language model predicts a scene graph for each frame. The accumulated record is then either rendered as text in the prompt or encoded by an R-GCN, whose features enter the decoder through a pooled input residual and a cross-attention residual after the final layer. Four configurations (no memory, text, graph, and both) share the same observations, backbone, and training budget, and all answer from a blank image so that answer-time vision is held fixed. The central claim is that text memory improves accuracy over the no-memory control by 6.62 points on MomaGraph-R1 and 3.00 points on Gemma-3-4B, with the gain concentrated in route-history questions (+18.9 and +8.1, respectively).

Critical strengths

S1. The paper addresses an important problem. Embodied agents increasingly store what they observe as scene graphs, yet prior work in embodied question answering does not directly study how a vision-language model should read such a record. The paper poses this question clearly: should the model read the memory as text, as learned graph features, or as both? The answer is useful to anyone building such a system.

S2The experimental design isolates the effect of memory. The four arms share the same observations, backbone, and training budget, and all answer from a blank image, so any difference in accuracy comes from the memory representation rather than from the model viewing the scene. The paper also discloses its limitations openly; for example, Section 5 states that the question-family analysis is post hoc. These disclosures let the reader judge how far each claim holds.

Critical weaknesses

W1. The central claim, that text memory outperforms graph features, is stated as a general result, but the gains are not consistent across backbones and the setup does not completely isolate the components to support the claim. The paper claims (line 63) that “a trained text interface outperforms both a no-memory control and learned graph features on two backbones, with gains concentrated in questions about route history.” Two issues weaken this:

Table 2 shows that the gain from text memory is not consistent across question families for the two backbones, and in some families the effect has the opposite sign. On persistence questions, for example, text memory helps MomaGraph-R1 but drops Gemma-3-4B below the no-memory control.

Section 4.1 states that a prompted VLM predicts the scene graphs, using the first episodic question as task context, but the paper does not say which VLM. The README in the provided code identifies it as the base MomaGraph-R1, which is also the backbone of the main answerer, and MomaGraph-R1 shows the largest gain (6.62 points versus 3.00 on Gemma-3-4B). When the same model writes and reads the record, the larger gain may come from that match rather than from the text representation, so the comparison across the two backbones in current setup does not support how general the claim is.

W 2. The reported test results are not an independent evaluation, because the memory records, the text format, and the question grouping were all shaped by the test questions.

Section 4.1 states that each episode’s scene graphs are extracted with the episode’s first question as task context. This first question is itself one of the four question in each of the 200 test episodes.

Line 1817 states that the text memory’s time axis, vocabulary, and truncation were revised after inspecting 200 episodes (800 questions), and that the overlap between these and the reported test split is unresolved.

Sections 3 and 5 state that the route-history grouping, which carries most of the reported gain, was defined post hoc. These choices favor the memory arms specifically. Test questions shaped the records that only the memory arms read, and the reported gains on specifically for order and presence-change questions on Momagraph-R1 cannot be confirmed that it comes only from the text-arm.

Weakness 3. The design of the graph reader is not motivated, and the input for the text and graph arms is similar as paper claims.

Appendix A (lines 820–825) suggests that averaging (the pooled input residual) and late entry (the cross-attention residual after the final layer) may explain why the graph features did not help, but neither hypothesis is tested, and no ablations of the graph reader are provided to isolate these effects.

The paper states that both arms read the same scene graph, but Eq. 1 and Appendix D.4 build the text record and the graph input independently. Additionally, lines 820 states the possibility that ”the node features do carry what the text route wins on”. It is therefore unclear whether the two arms receive the same information, and how their inputs differ. Without these controls, the comparison does not isolate the memory representation, and the weaker graph result cannot be attributed to graph features as such.

Questions for authors

Q1. Is the scene-graph extractor the same base MomaGraph-R1 checkpoint that is fine-tuned as the answerer? If yes, How does it affect the result if a different independent scene-graph extractor is used? Q2. What is B−A on q1 versus the other questions in each episode? Q3. The text-arm carries the visit sequence, per-type counts, out-of-vocabulary names and exact frame indices. Does the graph-arm gets the same information? If not, what is arm B’s accuracy when the text is re-rendered with only the information the graph has?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised PDF uploaded

Dear Reviewers and Area Chair,

Thank you for the reviews and discussion. We have uploaded a revised PDF. Changes are coloured by reviewer (blue = vtnf, red = iQzt, green = BRRB, purple = several), and the analyses added during the discussion are in Appendix B.1–B.3. One minor correction is noted in B.1: fixing a relation-label convention in the simulator's log changes the visibility-log control by under one point, and no conclusion changes. We are happy to answer any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.