acceptodds
Withdrawn from ICLR 2027

Vision-Language-Action Model Already Has Attention Heads for Path Deviation Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-Language-Action (VLA) models have demonstrated strong potential in navigation tasks by predicting semantic actions and reasoning over complex linguistic and visual contexts. However, they are fundamentally hindered by visual-reasoning hallucinations causing trajectory deviations. Addressing this issue has conventionally required training external critic modules or relying on complex uncertainty heuristics. In this work, we discover that monitoring specific attention heads within a frozen VLA model detects path deviations without incurring additional computational overhead. We term these Navigation Heads, as they inherently capture the spatiotemporal correlations between historical visual sequences and linguistic instructions. Using these heads, we propose an intuitive, training-free anomaly-detection framework that monitors their signals to detect hallucinations in real time. Surprisingly, among over a thousand attention heads, a combination of just three is sufficient to achieve a 65.1 % anomaly detection rate (recall) and a 76.4 % F1 score in unseen environments. Furthermore, we couple this high-level detection logic with a low-level Reinforcement Learning (RL) policy for local obstacle avoidance, ensuring stable navigation and executing a direct safe recovery upon anomaly detection. Ultimately, deploying this entire framework zero-shot onto our customized physical robot demonstrates its practical robustness. All source code will be publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Action (VLA), Reinforcement Learning (RL), Navigation Path Recovery, Robot Operating System (ROS)

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ewohOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a training-free method to detect path deviations in VLA navigation models. The authors identify a small set of attention heads, called Navigation Heads, based on their spatiotemporal alignment and sensitivity to navigation anomalies. By monitoring attention entropy, the method detects deviations without training an additional detection model.

The authors also integrate a low-level RL policy for obstacle avoidance and path recovery. Experiments on R2R, RxR, and two VLA models show promising detection results. The complete system is also demonstrated on a physical robot.

Critical strengths
  1. A simple and effective approach to path deviation detection. The paper presents a practical idea: using attention signals from a frozen VLA model to detect navigation failures without training an additional detector. The method achieves a 76.4% F1 score on R2R Val-Unseen, outperforming the evaluated heuristics. This suggests that internal attention signals can provide useful information for navigation monitoring at low computational cost.
  2. Extensive evaluation and real-world implementation. The authors evaluate their method across two datasets (R2R and RxR) and two VLA models (NaVILA and NaVid). They also implement the complete framework on a physical robot and demonstrate path recovery. These efforts provide evidence of generalizability and practical feasibility, making the contribution more convincing than a simulation-only study.
Critical weaknesses
  1. Limited validation of the Navigation Heads selection method. The paper shows that the selected heads can detect path deviations. However, it does not compare them with randomly selected heads or heads selected only by anomaly sensitivity. These comparisons are important to show whether the proposed spatiotemporal alignment criteria are necessary.
  2. The ground-truth labels do not directly measure hallucinations. In Section 3.2, anomalies are defined using deviations from reference waypoints. This is reasonable for path deviation detection. However, such deviations do not always indicate visual reasoning hallucinations. The paper does not clearly distinguish hallucinations from other sources of navigation errors.
  3. Limited end-to-end evaluation of the complete system. The anomaly detector and RL controller are evaluated separately. The paper also provides real-world demonstrations. However, it does not report quantitative improvements in navigation success or path efficiency after integrating detection and recovery. This makes it difficult to assess the practical benefit of the full framework.
  4. Unclear detection delay during real-world navigation. The detector requires nine consecutive steps to trigger an anomaly. Meanwhile, the VLA runs at approximately 0.3 Hz on the physical robot. Although the paper analyzes detection latency in simulation, the actual response time during real-world navigation remains unclear.
Questions for authors

1.How does the proposed selection method compare with randomly selected attention heads or heads selected only by anomaly sensitivity?

2.How often do the trajectory-based anomaly labels correspond to actual visual reasoning failures? Could the authors provide further evidence supporting this connection?

3.Could the authors report navigation success rates and path efficiency with and without the proposed detection and recovery mechanism?

5.What is the actual detection delay, measured in seconds, on the physical robot? How much does the robot typically deviate before recovery begins?

Reviewer_QwbDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for detecting path deviations in VLA-based navigation by identifying navigation-related attention heads within a frozen model. The method selects attention heads based on spatiotemporal alignment and anomaly sensitivity, then monitors their attention entropy to detect navigation failures. It achieves 76.4% F1 on R2R Val-Unseen and integrates an RL-based local controller for obstacle avoidance and rollback, with additional evaluations on RxR, NaVid, and a real-world robot.

Critical strengths
  1. The method detects navigation failures without training additional modules.

  2. The proposed detector outperforms heuristic baselines in anomaly detection.

Critical weaknesses
  1. Identifying functional attention heads in pretrained models has been explored in prior work. When extending this paradigm to VLN, improvements in task-level navigation performance (e.g., SR/SPL) are essential to demonstrate its practical value. However, the paper only evaluates anomaly detection, leaving the actual benefit of this extension unclear.

  2. The method assumes diagonal attention alignment between historical visual frames and the full instruction. However, eight frames may cover only a local stage of the instruction, and navigation progress is not necessarily linear with respect to instruction-token positions. Moreover, heads with strong spatiotemporal alignment are not necessarily the most sensitive to navigation failures, raising concerns about the head selection strategy.

Questions for authors
  1. In Equation (5), when Attention Entropy remains stable, R_t should be close to 1, which still exceeds the threshold of 0.95. This detection criterion seems potentially problematic.

  2. Could Attention Head Selection be highly sensitive to the amount and distribution of offline data, given that these heads are selected based on the distributions of normal and failed trajectories from the base model?

  3. Appendix A.1.3 seems to suggest that the same head combinations are retained when transferring to NaVid with a different backbone. However, it does not seem reasonable to me.

  4. The main anomaly detection experiments only compare against simple heuristic baselines. How does the proposed method compare with more competitive attention-based or learned failure detectors?

  5. The low-level RL controller is presented as a core contribution, yet it is essentially a separately trained local obstacle-avoidance policy. Its connection to the main innovation of Navigation Heads remains unclear, especially without end-to-end evidence of the integrated system's benefits.

Reviewer_PczSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether Vision-Language-Action (VLA) models already contain internal signals that can detect navigation failures without requiring additional training or external monitoring models. The authors identify a small subset of attention heads, called Navigation Heads, that appear to capture the spatiotemporal alignment between visual observations and language instructions. Based on this observation, they propose a training-free anomaly detection method that monitors changes in attention entropy to identify path deviations. When a deviation is detected, the system switches from the computationally expensive VLA model to a lightweight reinforcement learning policy that performs obstacle avoidance and navigates the robot back toward a previously verified safe checkpoint. The main contribution is the idea of using existing attention mechanisms as an internal monitoring signal for more reliable robotic navigation. For validation, the authors conduct experiments using the NaVILA backbone on the VLN-CE R2R benchmark and deploy the integrated framework on a physical mobile robot. The proposed detector uses only three attention heads and achieves a 44.6% episode-level deviation detection rate with an 11.7% false-positive rate on the validation-seen split, outperforming the evaluated rule-based detection baselines. On the validation-unseen split, it achieves a 41.9% detection rate and a 76.4% step-level F1 score. The reinforcement learning controller also achieves an 87.3% navigation success rate in simulated obstacle-avoidance tasks, outperforming several classical local planning methods. The real-world demonstrations further illustrate the feasibility of combining internal anomaly detection with robotic recovery. These results suggest that attention-based monitoring can provide useful signals for navigation failure detection, although the relatively modest episode-level detection rate leaves questions about its reliability in more challenging environments.

Critical strengths

First, I think the most interesting strength of this paper is the discovery that a small number of attention heads within a frozen VLA model can provide useful signals for detecting navigation failures. Rather than introducing another trained monitoring network, the authors investigate the model’s existing attention patterns and identify Navigation Heads that capture the relationship between visual observations and language instructions. I appreciate that the proposed detector uses only three attention heads and does not require additional trainable parameters. This is particularly meaningful for robotic applications, where computational resources and inference latency are important practical constraints. The results also provide an interesting perspective on how internal VLA representations might support more interpretable and efficient failure monitoring. Second, I appreciate the effort to connect the proposed detection method with an actual robotic recovery system rather than evaluating it only as an isolated prediction task. The authors integrate attention-based anomaly detection with a lightweight reinforcement learning controller and evaluate the approach in both simulation and physical robot experiments. The reported improvements over rule-based detection baselines, together with the RL controller’s 87.3% success rate in simulated obstacle avoidance, provide encouraging evidence of practical feasibility. In particular, the real-world deployment is a valuable aspect of the paper because it demonstrates that the proposed monitoring mechanism can be incorporated into an operational navigation pipeline. These experimental efforts strengthen the paper’s practical relevance, even though broader validation would still be useful.

Critical weaknesses

The paper has two weaknesses that most strongly determine my recommendation. First, my main concern is the limited reliability of the proposed anomaly detection method and whether the experimental results are strong enough to support its practical use. Although the authors demonstrate that three attention heads can detect navigation deviations, the reported episode-level detection rates are only 44.6% on Val-Seen and 41.9% on Val-Unseen, meaning that more than half of the anomalous episodes remain undetected. The false-positive rates of 11.7% and 9.6% are also not negligible, particularly for real-world robotic systems where unnecessary recovery actions can interrupt navigation or introduce additional risks. Moreover, the proposed method relies on attention entropy as an indicator of navigation failure, but the paper does not sufficiently establish whether these attention patterns reflect actual navigation reasoning or merely correlate with particular trajectory states. I would like to see stronger comparisons with other lightweight uncertainty estimation and attention-based detection methods, along with more detailed analysis of detection latency and failure cases. Without this evidence, I am not fully convinced that the proposed approach provides sufficiently reliable monitoring for practical robotic navigation.

Second, I am concerned about the limited generalizability of the experiments and the lack of comprehensive end-to-end evaluation of the proposed detection-and-recovery framework. The attention-head analysis is conducted primarily on a single VLA backbone, NaVILA, and the evaluation focuses on the VLN-CE R2R benchmark. It remains unclear whether the identified Navigation Heads and their entropy-based signals would transfer to other VLA architectures, different navigation environments, or more complex instructions. Although the authors demonstrate physical robot deployment, the main quantitative results evaluate anomaly detection and RL obstacle avoidance separately, rather than showing how much the complete system improves overall navigation success, recovery efficiency, or safety compared with the original VLA and alternative recovery strategies. This distinction is important because a strong low-level controller does not necessarily compensate for missed or incorrect anomaly detections. I believe broader cross-model validation and more comprehensive end-to-end comparisons are necessary to establish the robustness and practical significance of the proposed framework.

Questions for authors

I have several questions that could materially affect my assessment of the paper. First, my main concern is whether the identified Navigation Heads generalize beyond the NaVILA backbone. Could the authors clarify whether the same attention-head selection procedure, based on spatiotemporal alignment and Cohen’s d, has been tested on another VLA model, such as OpenVLA, RT-2, or π₀? Even preliminary evidence showing whether comparable Navigation Heads emerge would help establish the generality of the proposed approach. Second, could the authors provide the actual end-to-end detection latency on the Jetson AGX Orin using the selected configuration (W = 10, P = 9, τ = 0.95)? Given that the VLA operates at approximately 0.3 Hz while the RL controller runs at 10 Hz, I would like to understand how far the robot typically travels off-path before a deviation is detected and rollback begins. This is particularly important because the patience mechanism introduces additional detection delays. Third, could the authors provide quantitative results for the complete detection-and-recovery framework on the physical robot, including the number of trials, overall navigation success rate, recovery success rate, missed detections, and collision or mission failure rates? The reported 61.6–65.1% step-level recall suggests that a substantial number of anomalous steps remain undetected, but it is unclear how these missed detections affect real-world navigation safety. Fourth, could the authors clarify how the three Navigation Heads were selected without using validation information and whether the selected heads remain stable across different training subsets or random seeds? Since the method involves a large hyperparameter search, understanding the robustness of head selection would help establish whether the results depend on a particular experimental configuration. Fifth, after the robot rolls back to a previously verified safe checkpoint, how often does the VLA repeat the same incorrect decision and enter another hallucination loop? The authors acknowledge this limitation, but quantitative evidence would help determine whether rollback genuinely resolves navigation failures or simply delays them. Finally, could the authors provide controlled evidence that the selected attention heads capture navigation-specific grounding rather than merely correlating with trajectory deviations? A causal intervention or comparison with other lightweight internal monitoring signals would help clarify the underlying mechanism. Convincing answers to the questions about cross-model generalization, real-world detection reliability, and end-to-end recovery performance could significantly improve my assessment. As separate suggestions for future work, evaluating an additional VLA backbone and a second navigation benchmark such as RxR, together with incorporating active replanning or re-prompting after rollback, would further strengthen the approach.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.