- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a chain-of-exploration agent for embodied question answering that integrates planning, spatial memory, and iterative tool use to acquire evidence during exploration. It introduces EQA-RT, a dataset of approximately 18K question–answer pairs with reasoning and tool-use trajectories, constructed through an automated generation and verification pipeline. The controller is trained through supervised fine-tuning followed by evidence-grounded reinforcement learning, which jointly considers evidence coverage, answer quality, and interaction cost. Experiments demonstrate improvements in answer quality and exploration efficiency on EQA-RT, together with competitive results across three public benchmarks.
Critical strengths
- EQA-RT provides a useful resource for studying embodied evidence acquisition through question–answer pairs paired with intermediate reasoning and tool-use trajectories, including evaluation in both seen and unseen scenes.
- Controlled training and reward ablations show that evidence-grounded reinforcement learning improves answer quality, target retrieval, and exploration efficiency over supervised fine-tuning alone within the proposed framework.
Critical weaknesses
- The experiments report overall answer quality, target recall, and exploration distance, but do not show where the proposed multi-step 3D reasoning provides a clear advantage. Results are not broken down by question type or reasoning complexity, and the ablations examine training stages and reward terms without isolating explicit reasoning, spatial memory, or geometric tool use. Consequently, it remains unclear whether the improvements demonstrate stronger spatial reasoning or primarily reflect better tool selection and task-specific training, which weakens the experimental support for the paper’s central claim.
- The evidence reward depends on coverage of predefined facts, but the main text does not sufficiently explain how the correctness and sufficiency of the acquired evidence are established. This matters because collecting the expected information does not necessarily mean that it is accurate or correctly used to derive an answer. Table 4 further shows that removing evidence coverage from the answer reward improves LLM-Match on both splits, despite reducing coverage. Although the paper acknowledges this trade-off, independent evaluation of evidence correctness and its connection to the final answer is needed to establish that the proposed reward improves grounding reliability.
Questions for authors
- Were all synthetic trajectory observations obtained or replaced through actual tool execution, and what checks ensure that intermediate reasoning does not depend on oracle information unavailable at inference?
- How well does the evidence-coverage metric agree with independently verified target identity, geometric accuracy, and visual-answer correctness?
4 comments
Weakness 2 (Evidence Correctness, Sufficiency, and Reward Design)
The evidence reward depends on coverage of predefined facts, but the main text does not sufficiently explain how the correctness and sufficiency of the acquired evidence are established. This matters because collecting the expected information does not necessarily mean that it is accurate or correctly used to derive an answer. Table 4 further shows that removing evidence coverage from the answer reward improves LLM-Match on both splits, despite reducing coverage. Although the paper acknowledges this trade-off, independent evaluation of evidence correctness and its connection to the final answer is needed to establish that the proposed reward improves grounding reliability.
Response:
Acquired evidence is validated through type-specific checks rather than being credited unconditionally, and its sufficiency is measured against the required GT Evidence Facts. Specifically, evidence validation includes target association, tool-output validity, and spatial constraints. For instance, 3D Position requires agreement with the GT position within 2 m, while Object Size and VisualQA are checked for output validity. Evidence Coverage then measures the proportion of required GT Evidence Facts acquired by the agent. Thus, it primarily evaluates task-relevant evidence acquisition rather than the complete accuracy of individual tool predictions.
Removing Coverage Scaling slightly improves LLM-Match but reduces evidence acquisition. Without Coverage Scaling, the Answer Reward no longer depends on evidence sufficiency, allowing optimization to focus more directly on final-answer scores. Our design instead encourages the agent to acquire sufficient evidence and penalizes incorrect answers more strongly when sufficient evidence has been collected, promoting evidence utilization. Meanwhile, the separate Evidence Reward encourages evidence acquisition. As shown in Table 4 of the main paper, removing Coverage Scaling improves LLM-Match by 0.76 and 0.46 percentage points on Seen and Unseen, respectively, but reduces Evidence Coverage by 4.82 and 2.68 points, with Recall@10 also declining.
Evidence Coverage and final-answer quality improve together under SFT+RL, although they measure different aspects of agent performance. Evidence Coverage evaluates the acquisition of required evidence, whereas LLM-Match assesses final-answer quality. As shown in Table 3 of the main paper, SFT+RL improves Evidence Coverage from 17.72% to 41.09% on Seen and from 15.50% to 38.50% on Unseen, while LLM-Match increases from 41.25% to 50.32% and from 39.83% to 48.16%, respectively. These results demonstrate simultaneous improvements in evidence acquisition and answer quality, supporting the objective of jointly optimizing both.
Section 3.3 now clarifies the Evidence Coverage validation rules and reward design, while Appendix D includes the independent evidence-correctness evaluation and reward-scaling comparisons.
Question 1 (Synthetic Observations and Oracle Information)
Were all synthetic trajectory observations obtained or replaced through actual tool execution, and what checks ensure that intermediate reasoning does not depend on oracle information unavailable at inference?
Response:
Our synthetic trajectory observations are constructed from pre-extracted scene evidence and mapped to the outputs of the corresponding tools, rather than obtained through actual tool execution. Compared with actual tool outputs, ours observations retain similar information while avoiding potential execution errors and instability, thereby providing more accurate and reliable supervision. We first use the simulator to generate reference exploration trajectories and their corresponding visual Observations. For geometric perception tools, Observations are constructed from scene GT or automatically generated object annotations and formatted to match the corresponding tool outputs. Navigation Observations are generated from the observations and actions recorded in the reference trajectories.
Oracle information is used for offline trajectory generation, while intermediate reasoning is constrained to previously available Observations. Navigation actions are predetermined by the reference trajectories; found and all_found identify target-discovery states and key tool calls, while expected_answer constrains the final answer. During generation, the Prompt explicitly prohibits intermediate Thoughts from using geometric information that has not yet been acquired, and geometric computations in the final Thought strictly use positions and dimensions already contained in historical Observations. We additionally apply rule-based checks and VLM consistency verification to assess tool-call validity and reasoning–observation consistency. At inference time, the agent receives no oracle information, reference trajectories, or GT object information. Instead, it follows the Planner-generated Plan, obtains Observations through actual tool execution, and autonomously explores and answers using interaction history and Spatial Memory.
Section 3.2 now specifies how synthetic Observations are constructed and how intermediate reasoning is constrained, and Appendix G documents the offline oracle fields and their absence at inference.
Question 2 (Independent Validation of Evidence Coverage)
How well does the evidence-coverage metric agree with independently verified target identity, geometric accuracy, and visual-answer correctness?
Response:
Evidence Coverage is positively associated with independently verified target identity, geometric accuracy, and VisualQA correctness, supporting the validity of our metric. To verify this relationship, we evaluate 300 EQA-RT-Unseen samples, divided into three Evidence Coverage ranges with 100 samples each. We independently assess Target Identity Accuracy, Geometric Accuracy, and VisualQA Accuracy. As shown in the following table, all three metrics consistently improve with increasing Evidence Coverage.
| Evidence Coverage | # Samples | Target Identity Acc. (%) ↑ | Geometric Accuracy (%) ↑ | VisualQA Accuracy (%) ↑ |
|---|---|---|---|---|
| Low (0-15%) | 100 | 63 | 52 | 69 |
| Medium (15-40%) | 100 | 70 | 66 | 75 |
| High (40-100%) | 100 | 79 | 79 | 87 |
We have added the full 300-sample evaluation to Appendix D (Independent Evidence Correctness Evaluation) and summarized its findings in Section 4.4.
Thank you to the authors for their detailed response. My concerns have been largely addressed, and I have therefore raised my score.
We sincerely thank the reviewer for the insightful and constructive feedback, which has been highly valuable in helping us strengthen and improve our manuscript.
Weakness 1 (Attribution of Multi-Step 3D Reasoning Improvements)
Response:
Our improvements arise from both tool selection and spatial reasoning, with a larger reduction in spatial reasoning errors. To distinguish their contributions, we conduct a Bad Case Analysis. We classify failures as Tool Selection Errors when necessary tools are not invoked despite sufficient observations, and Spatial Reasoning Errors when the agent fails to identify the prerequisites for acquiring spatial evidence or incorrectly integrates acquired geometric information. As shown in the following table, SFT+RL reduces Tool Selection Errors by 7.3 percentage points and Spatial Reasoning Errors by 11.4 points compared with None, indicating that the improvements extend beyond tool selection.
Both Spatial Memory and Planner contribute to multi-step spatial reasoning. As shown in the following table, removing either component reduces performance on EQA-RT-Unseen, while removing both causes a larger decline. The full model achieves 48.16% LLM-Match with a trajectory length of 23.19 m, compared with 40.37% and 49.99 m when both modules are removed.
Our method also achieves substantial gains on tasks requiring geometric measurements and multi-object evidence integration. As shown in the following table, SFT+RL consistently outperforms SFT-only across question types, particularly on Attribute-Size and Distance, with gains of 13.90/14.30 points on Seen and 13.69/14.28 points on Unseen, respectively.
Our method also improves performance on tasks requiring multi-object evidence integration. As shown in the following table, SFT+RL consistently outperforms SFT-only across different numbers of target objects, supporting its effectiveness in tasks requiring evidence acquisition and integration across multiple objects.
We have added a Spatial Reasoning Analysis to Section 4.4, with the complete component and error analyses and per-category performance breakdowns in Appendix D.