- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a training-free method to detect path deviations in VLA navigation models. The authors identify a small set of attention heads, called Navigation Heads, based on their spatiotemporal alignment and sensitivity to navigation anomalies. By monitoring attention entropy, the method detects deviations without training an additional detection model.
The authors also integrate a low-level RL policy for obstacle avoidance and path recovery. Experiments on R2R, RxR, and two VLA models show promising detection results. The complete system is also demonstrated on a physical robot.
Critical strengths
- A simple and effective approach to path deviation detection. The paper presents a practical idea: using attention signals from a frozen VLA model to detect navigation failures without training an additional detector. The method achieves a 76.4% F1 score on R2R Val-Unseen, outperforming the evaluated heuristics. This suggests that internal attention signals can provide useful information for navigation monitoring at low computational cost.
- Extensive evaluation and real-world implementation. The authors evaluate their method across two datasets (R2R and RxR) and two VLA models (NaVILA and NaVid). They also implement the complete framework on a physical robot and demonstrate path recovery. These efforts provide evidence of generalizability and practical feasibility, making the contribution more convincing than a simulation-only study.
Critical weaknesses
- Limited validation of the Navigation Heads selection method. The paper shows that the selected heads can detect path deviations. However, it does not compare them with randomly selected heads or heads selected only by anomaly sensitivity. These comparisons are important to show whether the proposed spatiotemporal alignment criteria are necessary.
- The ground-truth labels do not directly measure hallucinations. In Section 3.2, anomalies are defined using deviations from reference waypoints. This is reasonable for path deviation detection. However, such deviations do not always indicate visual reasoning hallucinations. The paper does not clearly distinguish hallucinations from other sources of navigation errors.
- Limited end-to-end evaluation of the complete system. The anomaly detector and RL controller are evaluated separately. The paper also provides real-world demonstrations. However, it does not report quantitative improvements in navigation success or path efficiency after integrating detection and recovery. This makes it difficult to assess the practical benefit of the full framework.
- Unclear detection delay during real-world navigation. The detector requires nine consecutive steps to trigger an anomaly. Meanwhile, the VLA runs at approximately 0.3 Hz on the physical robot. Although the paper analyzes detection latency in simulation, the actual response time during real-world navigation remains unclear.
Questions for authors
1.How does the proposed selection method compare with randomly selected attention heads or heads selected only by anomaly sensitivity?
2.How often do the trajectory-based anomaly labels correspond to actual visual reasoning failures? Could the authors provide further evidence supporting this connection?
3.Could the authors report navigation success rates and path efficiency with and without the proposed detection and recovery mechanism?
5.What is the actual detection delay, measured in seconds, on the physical robot? How much does the robot typically deviate before recovery begins?