acceptodds
Withdrawn from ICLR 2027

Beyond Confidence: Active Falsification for Embodied Question Answering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Embodied question answering (EQA) requires an agent to explore an unfamiliar environment and gather visual evidence before answering. However, question-relevant observations may fail to distinguish competing answers, allowing an early, confident error to persist. We propose Active Falsification for Embodied Question Answering (AFE-EQA), a framework that treats the current answer as a revisable hypothesis and actively seeks observations capable of refuting it. Three-state grounding distinguishes decidable, partial, and not-visible observations to regulate belief updates and evidence retention. A structured counter-evidence memory links testable propositions to their source hypotheses and visual assessments, while falsification-aware imagination guides exploration toward viewpoints that distinguish competing answers. At termination, an offline-trained resolver integrates accumulated visual and trajectory evidence to determine whether the current answer should be revised. These components operate along a single exploration trajectory. Experiments on HM-EQA show improved final answer accuracy under matched conditions, while component ablations and paired analyses characterize the contributions of exploration and terminal revision, including both corrected errors and harmful switches. These findings support active hypothesis testing as a design principle for coupling embodied exploration with evidence-based answer revision.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Question Answering, Active Falsification, Active Perception, Evidence-Grounded Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_toTnOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AFE-EQA, a framework that treats the current answer as a falsifiable hypothesis and actively seeks discriminative visual evidence during embodied exploration. It integrates three-state grounding, counter-evidence memory, falsification-aware imagination, and a learned terminal resolver. Experiments on HM-EQA improve accuracy from 58.0% to 62.0% with the Prism backbone.

Critical strengths
  1. Interesting formulation. Treating answer hypotheses as falsifiable provides a meaningful perspective on evidence acquisition beyond confidence-driven exploration.
  2. Coherent framework. The integration of exploration, structured memory, and answer revision forms a well-motivated reasoning pipeline.
Critical weaknesses
  1. Insufficient evidence for the central claim. Falsification-aware exploration improves accuracy by only 1 percentage point (58% to 59%), while the learned terminal resolver contributes another 3 points. Moreover, the paper does not directly evaluate recovery from early high-confidence errors, despite this being its primary motivation.
  2. Overly verbose and defensive writing. Excessive implementation details and repeated qualifications sometimes obscure the main narrative. A more concise presentation would help highlight the core contributions.
Questions for authors
  1. Effectiveness of active falsification: Can the authors directly evaluate recovery from early high-confidence errors and demonstrate the benefits of falsification-directed exploration independently of the learned terminal resolver?
  2. Component contributions: Can the authors provide finer-grained ablations of three-state grounding and counter-evidence memory, and evaluate the terminal resolver on baseline exploration trajectories?
Reviewer_NSnHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AFE-EQA, an embodied question answering framework that actively seeks visual evidence to challenge its current answer. It combines visual grounding, counter-evidence memory, guided exploration, and a learned final answer selector. On HM-EQA, the Prism-based configuration improves accuracy from 58% to 62%.

Critical strengths

The paper tackles overconfident errors in embodied QA by actively seeking visual evidence that could challenge the current answer.

Critical weaknesses
  1. Explore until Confident (RSS 2024) and EfficientEQA (IROS 2025) already study exploration for gathering evidence to answer questions. Pred-EQA and Align While Search also share ideas around planning, memory, and belief updates. The paper discusses these connections, but I still find it hard to tell what active falsification adds. Pred-EQA is explicitly described as particularly close, yet there is no experimental comparison.

  2. All experiments run in Habitat-Sim. Although HM3D scenes come from real scans, this leaves open how the system handles sensor noise, localization errors, and imperfect movement on a physical robot. The repeated VLM calls also raise questions about practical runtime. For a method built around robot exploration, I would expect at least a small physical robot evaluation reporting answer accuracy, exploration cost, and total runtime.

  3. Falsification-aware exploration improves accuracy from 58% to 59%, while the terminal resolver brings it to 62%. The missing comparison is ordinary exploration followed by the same resolver. Without that result, it is difficult to tell how much the proposed exploration strategy contributes once both systems have access to the same final reasoning procedure.

  4. The paper focuses on recovering from early, high-confidence mistakes and formally defines this failure mode, but does not report recovery rates for that specific group of episodes. Overall accuracy and selected examples do not answer this question. The resolver also fixes 64 answers while changing 49 previously correct answers to incorrect ones.

  5. Releasing the implementation would improve reproducibility and make the method easier to evaluate and adopt. While not essential for acceptance, it would strengthen the overall impression of the work.

Questions for authors

See the above weakness.

Reviewer_yhBBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces an embodied question answering (EQA) method called AFE-EQA. The motivation stems from the observation that late observations might challenge the early ones during the exploration process. To achieve a more thorough evaluation across all observations, AFE-EQA first labels each view with three pre-defined states, including decidable, partial, and not-visible, using these to control the belief update and image retention. The propositions of VLM contradicting the existing answer are captured with assessments. When there are unresolved propositions, a falsification-aware imagination is utilized to bias the exploration direction in the planner. A terminal resolver with extra computation costs is introduced to address these contradictories ultimately.

Critical strengths

This manuscript investigates a valuable question in EQA, and the revisable hypothesis is reasonable and intuitive. Technically speaking, its separate designs for immediate predictions, belief management, and the terminal answer are well aligned with the motivations. Most details are transparent in the main text and the appendix.

Critical weaknesses

This manuscript investigates a valuable question in EQA, and the revisable hypothesis is reasonable and intuitive. Technically speaking, its separate designs of immediate predictions, belief management, and terminal answer are well aligned with the motivations. Most details are transparent in the main text and the appendix.

  • The main comparison uses different VLM backbones. In Table 2, baselines use various VLM models, including Prismatic, GPT-4o-mini, and Qwen2.5-VL 7B, but AFE-EQA is implemented without using the shared VLM backbones, which can affect the ultimate performance. It would be more convincing
  • Inherent limitations of VLM-generated three-state grounding. While the introduction describes the relevant-but-not-distinguishing situation, I wonder how much improvement can be brought by using additional state-related anchoring or prompting using the same VLM, which is already making false-positive predictions. In other words, a VLM making false-positive answers is also likely to label that observation as “decidable” since these two properties can be quite relevant and similar.
  • Unclear computational costs. Compared to baselines like Explore-EQA that solely rely on a single VLM (Prismatic), the proposed AFE-EQA is much more computationally heavy, using an additional 32B VLM for multi-view verification and more complicated grounding and checking calls. However, the computational overhead has not been disclosed in the manuscript.
  • **Presentation. ** Some logic transitions are not easy to follow. For instance, the connection in lines 78-80 from the core question to the calibration argument is not well motivated. Inconsistency happens in Figure 1, where the B view (t=0) does not seem to be the identical view across the pipeline.
Questions for authors
  • Can the authors supplement additional comparison with existing methods, using the identical or size-similar VLM backbones?
  • How frequently would the early high-confidence error happen? How many of those cases can be recovered by AFE-EQA or flagged as risky predictions using the three-state grounding?
  • While AFE-EQA requires late observations for each EQA sample, does it have an early termination mechanism when it has a very confident answer?
  • What is the computational cost of each major step?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.