- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies premature commitment in VLM-based embodied agents: an agent forms an answer that looks plausible from partial observations and stops, even though the evidence it has gathered is not enough to establish that answer. The authors propose Sure-EQA, a training-free framework that treats the agent's answer as a hypothesis to be tested, not a final decision. With the candidate held fixed, it runs two kinds of active checks. Evidence verification re-observes the supporting regions from new viewpoints. Coverage verification visits unexplored frontiers to look for information that would change the answer. A VLM verifier labels each new observation as SUPPORTED, CHANGED, or UNCLEAR. These labels are combined into pose-weighted answer-change risk scores, which decide the next viewpoint and whether to accept the candidate. On A-EQA and EXPRESS-Bench with Qwen3-VL 8B, Sure-EQA improves over previous methods.
Critical strengths
- The problem is well motivated by a clear diagnostic study. Fig. 1 shows that when agents propose to stop, the annotated key evidence is still missing in 38.6–51.3% of cases.
- With a training-free design and an open-source 8B backbone, Sure-EQA achieves state-of-the-art results on A-EQA and EXPRESS-Bench benchmarks.
Critical weaknesses
- Every number comes from a single run on 184 or 409 questions, scored by a GPT-4 judge. There are no seeds, confidence intervals, or paired significance tests. Many ablation and analysis differences are 1–2 points (e.g., Table 3), which is within plausible run-to-run and judge variance for an LLM-driven agent on 184 questions.
- The answerer and both verifiers are the same 8B VLM, so their errors are likely correlated. The paper reports no verifier accuracy, for example how often SUPPORTED/CHANGED labels agree with the ground-truth effect on the answer. Fig. 12 shows 41.7% of verified-accepted A-EQA answers are still not fully correct, so the verification signal seems weak.
- Table 6 says one verification step is cheaper than one exploration step. The relevant question is how much total wall-clock time and compute Sure-EQA adds per episode compared with the baseline, including the extra exploration after rejected rounds. That number is not reported.
Questions for authors
"w/o Risk Score" (Table 3) drops to 53.9. Why does removing the risk term from selection alone erase almost the whole gain?
4 comments
Response to Reviewer RamK (Part 2/2)
W2.1. Verifier reliability
Regarding the concern that sharing a backbone may lead to correlated errors, Sure-EQA uses distinct verifier roles and newly acquired environmental evidence to challenge the answerer’s initial interpretation. The verifiers re-examine its supporting evidence and seek missing or contradictory evidence through targeted environmental checks.
We further assess individual verifier reliability through an offline analysis on A-EQA. An LLM judge (GPT-6 Astra) evaluates all 1,855 judgments, complemented by a human expert audit of 100 sampled judgments.
| Evaluation | SUPPORTED precision (%) | CHANGED precision (%) | UNCLEAR precision (%) | Overall Accuracy (%) | Overall Macro-F1 (%) |
|---|---|---|---|---|---|
| LLM judge | 88.25 | 84.31 | 84.54 | 86.52 | 84.06 |
| Human expert audit | 85.45 | 84.21 | 84.62 | 85.00 | 83.02 |
These statistics indicate that individual verifier judgments are reasonably reliable. Since individual checks remain fallible, Sure-EQA accumulates verification findings through risk-score updates to improve robustness to occasional misjudgments. Table 5 in the appendix supports this design, showing higher answer quality and lower answer degradation than individual verification.
W2.2. Corrective effectiveness.
Verified acceptance indicates that the candidate satisfies our evidence-based verification criteria, rather than guaranteeing ground-truth correctness. Figure 12 shows that our framework corrects 18.59% of initially incorrect answers while degrading only 5.13% of initially correct answers among verified-accepted A-EQA cases, increasing the fully correct answer rate from 44.87% to 58.33%. Moreover, among the 41.67% of accepted answers that remain less than fully correct, 18.46% still achieve higher evaluation scores than their initial counterparts. These results demonstrate the effectiveness of our verification framework in correcting mistaken interpretations and improving answer quality.
W3. Episode-level runtime and computational overhead.
As shown in the table below, we report mean episode-level costs on EXPRESS-Bench, including all verification rounds and additional exploration following rejected rounds.
| Method | Latency (s/episode) ↓ | Computation (PFLOPs/episode) ↓ |
|---|---|---|
| Baseline | 567 | 6.30 |
| Sure-EQA | 775 | 7.76 |
| Relative increase | +36.6% | +23.1% |
Although Sure-EQA incurs higher total episode-level costs, its incremental overhead only amounts to 36.6% of baseline runtime and 23.1% of baseline model computation. This includes verification and additional exploration following unsuccessful rounds. At this controlled additional cost, Sure-EQA can effectively enable the agent to uncover and correct mistaken interpretations.
Q. Performance drop after removing risk-guided selection.
This sharp drop highlights the effectiveness of our verification-guided viewpoint selection in directing observations toward evidence that may challenge the agent's current interpretation. In this ablation, only the risk term in Eq. (7) is removed, while the verification mechanisms and risk-based acceptance criterion in Eq. (6) remain unchanged. Without risk guidance, viewpoint selection is driven primarily by geometric novelty and travel cost, potentially directing limited observations away from unresolved concerns about the candidate answer. This weakens the evidence available for effective feedback and answer revision. The results thus support our core design of using accumulated verification findings to actively guide subsequent environmental interaction, rather than merely assessing existing evidence.
Thank you for the detailed response. The additional results address most of my concerns. I suggest moving the episode-level runtime and computation comparison into the main paper rather than leaving it to the rebuttal. I will keep my score.
Response to Reviewer RamK
Thank you again for your constructive and thoughtful feedback. We have described episode-level costs in Appendix Tab. 6 according to your detailed suggestions. Together with per-step cost analysis, Sure-EQA uses efficient step to correct self-consistent erroneous interpretations for embodied agent.
If any concerns remain, we would be grateful if you could share them and we may address them more fully. Should further clarification resolve these concerns, we hope you might reconsider your rating in the revised manuscript and additional experiments.
Response to Reviewer RamK (Part 1/2)
Thank you for the detailed feedback. Below, we clarify the five-seed protocol, assess verifier reliability, and report episode-level runtime and computation. Finally, we explain how risk-guided viewpoint selection contributes to the observed performance gains.
Our experiments use five random seeds, each corresponding to a separate trajectory rollout. Reported scores are averaged across the five runs. Below, we provide SD, paired 95% confidence intervals for performance differences, and two-sided paired t-test p-values. The repeated-run protocol is detailed in revised Appendix A.4.2 (“Repeated runs”).
Main comparisons.
Ablation comparisons.
Notably, our method achieves statistically significant improvements over Pred-EQA across all four metrics and over the ablated variant on both benchmarks. These analyses demonstrate that, by coupling risk-guided viewpoint selection with retained verification feedback, Sure-EQA can effectively identify mistaken interpretations through targeted environmental checks and correct them before commitment.