acceptodds
Withdrawn from ICLR 2027

The Cost of Reasoning over Frames Instead of Worlds

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing embodied question answering benchmarks evaluate selecting and reading a single frame, analogous to needle-in-a-haystack (NIAH) retrieval. Frontier models excel at this, reflecting their saturation on OpenEQA. We identify an axis of complexity in the scene representation a question requires, defined by how much persistent state must be maintained across frames and how many deductions are needed to integrate observations; existing evaluations occupy its degenerate single-frame case and require neither. We formalize these demands in a compact deductive language whose rules integrate observations across frames into a consistent scene representation, and introduce a novel generative campus environment that poses questions far beyond existing benchmarks along this axis, each certified by a derivation in the language. Frontier Models answer NIAH controls with over 90% accuracy, but fall to 12–41% when evidence must be carried across frames, at derivation depths of at most seven. Results are consistent with models assembling a scene representation ad-hoc within the chain of thought at query time, from the frames themselves, rather than reasoning over one constructed beforehand. Existing benchmarks cannot observe these costs; our axis, language, and environment make them measurable, and motivate evaluations and architectures that treat the representation, rather than the frames, as the object of reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
embodied question answering, embodied AI, embodied agents, agent memory, scene representation, representation learning, world models, state maintenance, long-context video understanding, vision-language models, multimodal large language models, chain-of-thought reasoning, deductive reasoning, test-time compute, neuro-symbolic evaluation, benchmark, generative environments, spatial reasoning, cognitive maps, shortcut

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qVRpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that existing embodied question answering benchmarks mainly test single-frame retrieval. The paper addresses the problem with CampusLog, a deductive language whose derivations certify each answer and measure its difficulty. It also introduces CampusGen, a generator of multi-building, multi-floor campuses with hours-long trajectories. CampusGen varies difficulty by removing signs while keeping the campus, route, questions and answers fixed. Experiments are conducted with GPT-6-Astra and Gemini 3.8 Flash on three generated campuses. Additional experiments are done with two real campuses using Gemini 3.1 Pro. The authors interpret the results as models assembling a representation at question time, rather than reasoning over a unified world model.

Critical strengths
  1. The paper identifies a real gap in current evaluations. Spatial reasoning should require integrating observations into a persistent representation, not retrieving facts from a single frame as in many existing benchmarks.
  2. The benchmark design is thoughtful: answers come with checkable derivations; shortcut carefully prevented; the generator is scalable.
  3. The example of a student navigating a campus building in the introduction section effectively grounds the paper’s central idea before formalism. The same example recurs in Section 3.3 to illustrate the difficulty measure, which helps tie the narrative to the formalism.
Critical weaknesses
  1. The main result is largely expected: lower accuracy on harder questions is not surprising. Removing signs would also increase perceptual load, search cost and output budget pressure. However, the specific reasons behind the accuracy drop are hindered by the pooled results. The defined difficulty metrics are not reflected in the results.
  2. The main experiments evaluate only GPT-6-Astra and Gemini 3.8 Flash, so the paper’s general claims about how “frontier models” behave are based on a very small sample. The two models are also not compared under comparable conditions as GPT-6-Astra is in Codex harness. The additional experiment with real campuses uses a different model, Gemini-3.1-Pro-Preview, and the setup also differs, making the results again not comparable.
  3. The paper claims that the results are “consistent with models assembling a scene representation ad-hoc within the chain of thought at query time, from the frames themselves.” The supporting evidence is that thinking tokens increase on harder questions, and Gemini’s errors cluster near its output limit. Both observations would also be expected if harder questions simply require more search, regardless of how the model represents the scene. The evidence does not distinguish it from simpler explanations.
Questions for authors
  1. In table 4, GPT-6-Astra’s full episode is “-“. How did Codex access a 10k–16k frame episode?
  2. Why does the additional experiment with real campuses use a different model instead of those in the main evaluation?
  3. GPT-6-Astra was used as both the evaluated model and the LLM judge. This may introduce additional bias. Answers are closed form, so why not use exact match?
Reviewer_DULdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Existing embodied question answering benchmarks, such as OpenEQA, mostly test whether a model can find and read a single relevant frame, which is essentially a needle in haystack task at which frontier models already excel. The authors propose that harder and more realistic questions require keeping track of information across many frames, such as which building and floor the viewer is on, and combining several deductions to reach an answer.

The authors propose two components to create their new eval:

CampusLog, a small formal language that describes how facts seen in individual frames are carried forward and combined, for example that climbing a flight of stairs raises the floor by one. Every answer comes with a derivation in this language, and the length of that derivation serves as a measure of how hard the question is.

CampusGen, a procedural generator that builds multi-building, multi-floor campuses in Blender and renders hours-long walks through them. CampusGen can remove signs while keeping the campus, the route, the questions, and the answers unchanged, which makes questions harder in a controlled way because more information has to be remembered and inferred.

The validation uses three generated campuses with hours-long walks, testing GPT-6-Astra and Gemini 3.8 Flash on control questions answerable from a single frame and on harder questions that need information from many frames. Both models nearly saturate the control questions, but their accuracy drops sharply on the harder questions, and drops further when signs are removed and the required derivations grow longer.

Critical strengths
  • Controlled design of the evaluation. CampusGen removes signs while keeping the campus, the route, the questions, and the answers unchanged, so the only thing that varies is how far information must be carried across frames. Most long-video and embodied benchmarks compare different questions or different videos, which makes it hard to tell whether a drop in accuracy comes from harder reasoning or from something else. Here, the drop from the condition with all signs to the condition with signs removed can be attributed more directly to evidence placement.
Critical weaknesses
  • No experiment separates the provided explanation ("models assembling a scene representation ad hoc within the chain of thought") from other explanations, such as failing to perceive the cues or failing to find the right frames.

  • Limited novelty relative to prior multi-frame video QA benchmarks. The paper presents questions that require integrating evidence across frames as a new evaluation axis, but earlier benchmarks already target this. Thinking in Space (Jihan Yang et al., CVPR 2025) introduced VSI-Bench, which requires remembering and integrating spatial information across a video to answer questions about counts, distances, and routes. That work also reported the conclusion this paper draws: models lack a persistent spatial representation, and generating an explicit cognitive map at question time helps. Yet it is missing from Table 3. VRR-QA (Sirnam Swetha et al., CVPR 2026) likewise targets questions that cannot be answered from any single frame or short clip, requiring inference of relations and events implied across frames.

  • Gemini's 12% result is driven by output truncation

Questions for authors
  • Provide Missing ablations:
    • Perception oracle: give the model CampusLog's grounded facts as text. If accuracy recovers, the problem is perception or grounding, not reasoning.
    • Retrieval oracle: give only the frames the derivation uses.
    • Prebuilt representation: give a text map or scene graph built beforehand.
  • Gemini results after increasing max tokens cap ?
Reviewer_rb82OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies embodied question answering that requires maintaining state and combining evidence across multiple frames. It introduces CampusGen, a procedural environment that varies evidence placement while keeping the campus, route, questions, and answers fixed. The authors also develop CampusLog, a deductive framework for verifying answers and measuring reasoning depth. Experiments on two frontier models show high accuracy on single-frame retrieval controls but substantial drops when information must be carried across frames. Additional experiments on two real campuses show similar difficulties. The paper provides a controlled benchmark for studying persistent-state reasoning in embodied environments.

Critical strengths
  1. The controlled benchmark design addresses an important limitation of existing embodied QA evaluations. By changing evidence placement while keeping the questions, answers, and routes fixed, CampusGen provides a useful way to study the difficulty of integrating information across frames. The large performance gap between NIAH controls and state-dependent questions supports the relevance of this evaluation setting.

  2. The benchmark construction includes useful safeguards and complementary evaluation. CampusLog provides explicit derivations for the reference answers, while the question-generation process requires unique answers and necessary clues. The real-campus experiments also provide additional evidence that the observed difficulties are not limited to synthetic environments.

Critical weaknesses
  1. The experiments do not clearly isolate the source of the performance degradation. Answering the harder questions requires visual recognition, evidence retrieval, state tracking, and multi-step reasoning. The current results do not distinguish failures in these components, making it difficult to determine whether persistent-state integration is the main bottleneck. A comparison using oracle visual facts or preconstructed state representations would help clarify this.

  2. The benchmark certification and model evaluation protocols need more detail for reproducibility. The paper describes CampusLog's rules and derivation-based difficulty measure, but does not provide the complete executable rule set or explain how minimum-depth derivations are computed. The model-input and tool-use procedures are also not fully specified. Moreover, CampusLog relies on renderer-based visibility checks, leaving it unclear whether all certified evidence remains readable in the inputs actually received by the models.

  3. The experimental coverage is limited. The 94 primary questions come from only three synthetic campus episodes, so the pooled results may not reflect performance across diverse environments. The paper also does not report per-campus effects or uncertainty estimates that account for shared episodes. Although two real campuses are evaluated, they use a different model from those in the synthetic experiments, making direct comparisons difficult.

Questions for authors
  1. Can the authors compare performance on raw video inputs, oracle visual facts, and preconstructed state representations for a subset of the same questions? This would help determine whether state integration, rather than perception or retrieval, is the main source of failure.

  2. Could the authors provide the complete CampusLog rule set and derivation procedure, along with the exact model-input and tool-use settings? How do they verify that the evidence used in certified derivations remains readable after the model's input processing?

  3. How consistent is the Full-to-Reduced performance drop across the three synthetic campuses? Could the authors report per-campus results and uncertainty estimates that account for questions sharing the same environment?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.