- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that existing embodied question answering benchmarks mainly test single-frame retrieval. The paper addresses the problem with CampusLog, a deductive language whose derivations certify each answer and measure its difficulty. It also introduces CampusGen, a generator of multi-building, multi-floor campuses with hours-long trajectories. CampusGen varies difficulty by removing signs while keeping the campus, route, questions and answers fixed. Experiments are conducted with GPT-6-Astra and Gemini 3.8 Flash on three generated campuses. Additional experiments are done with two real campuses using Gemini 3.1 Pro. The authors interpret the results as models assembling a representation at question time, rather than reasoning over a unified world model.
Critical strengths
- The paper identifies a real gap in current evaluations. Spatial reasoning should require integrating observations into a persistent representation, not retrieving facts from a single frame as in many existing benchmarks.
- The benchmark design is thoughtful: answers come with checkable derivations; shortcut carefully prevented; the generator is scalable.
- The example of a student navigating a campus building in the introduction section effectively grounds the paper’s central idea before formalism. The same example recurs in Section 3.3 to illustrate the difficulty measure, which helps tie the narrative to the formalism.
Critical weaknesses
- The main result is largely expected: lower accuracy on harder questions is not surprising. Removing signs would also increase perceptual load, search cost and output budget pressure. However, the specific reasons behind the accuracy drop are hindered by the pooled results. The defined difficulty metrics are not reflected in the results.
- The main experiments evaluate only GPT-6-Astra and Gemini 3.8 Flash, so the paper’s general claims about how “frontier models” behave are based on a very small sample. The two models are also not compared under comparable conditions as GPT-6-Astra is in Codex harness. The additional experiment with real campuses uses a different model, Gemini-3.1-Pro-Preview, and the setup also differs, making the results again not comparable.
- The paper claims that the results are “consistent with models assembling a scene representation ad-hoc within the chain of thought at query time, from the frames themselves.” The supporting evidence is that thinking tokens increase on harder questions, and Gemini’s errors cluster near its output limit. Both observations would also be expected if harder questions simply require more search, regardless of how the model represents the scene. The evidence does not distinguish it from simpler explanations.
Questions for authors
- In table 4, GPT-6-Astra’s full episode is “-“. How did Codex access a 10k–16k frame episode?
- Why does the additional experiment with real campuses use a different model instead of those in the main evaluation?
- GPT-6-Astra was used as both the evaluated model and the LLM judge. This may introduce additional bias. Answers are closed form, so why not use exact match?