- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies hallucination failures in VLA models under distribution shifts. It distinguishes hallucination failures, where actions deviate from the visual scene or instruction, from manipulation failures caused by low-level execution errors, and defines the Hallucination Rate (HR) as the proportion of hallucination failures among all failures. The paper argues that task-relevant semantics may weaken between the VLM representation and the Action Expert representation . It proposes AVSA, which reconstructs from an instruction-free scene representation and , and uses reconstruction similarity to guide Flow-Matching action generation. Experiments on three VLA backbones, several simulation benchmarks, and a real robot report reduced HR and improvements in success rate.
Critical strengths
- The paper addresses an important problem for VLA deployment. Distinguishing semantic/action deviations from low-level manipulation errors provides information that aggregate success rates cannot capture.
- AVSA is an inference-time correction framework that keeps the VLA backbone frozen. The evaluation covers three VLA backbones, multiple simulation benchmarks, and a real robot, with generally consistent improvement trends.
Critical weaknesses
- The claimed hallucination mechanism is insufficiently supported. Figure 3 only shows that is more separable than under the selected metric. It does not establish that correctly understands the task or that the low separability of causes hallucination. Corresponding analyses of successful trajectories and manipulation failures are missing.
- The validity of the VSR and several experimental conclusions remain unclear. The VSR may recover task-correlated information already contained in or , rather than genuinely evaluating action–instruction consistency. In addition, unclear metric denominators and inconsistencies appear across figures and tables.
Questions for authors
- Is the low separability of in Figure 3 specific to hallucination trajectories, or does it also occur in successful trajectories and manipulation failures?
- How do the authors establish that the VSR score reflects action–task consistency rather than task information already contained in or , particularly for unseen tasks?
- Please clarify the reporting of the failure metrics. In Figure 2, the stacked segments appear to represent failure types as fractions of all trials, whereas HR is defined over failed trials only. For LIBERO-Para, the SmolVLA and bars also appear to be swapped relative to Table 2. In addition, the claimed average HR reduction of 11.5% is not reproduced by Table 2, which gives a macro-average absolute reduction of 6.0 percentage points. Please clarify these inconsistencies and provide the raw counts.
8 comments
Thanks & Initial Response to Reviewer Dt1z (Q1)
[Q1 - Is low separability of action-expert representations specific to hallucination? Corresponding analyses of successful trajectories and manipulation failures are missing.]
Comparing representation separability across successes, manipulation failures, and hallucination failures is indeed crucial to determining whether this phenomenon is closely associated with hallucination. The additional experiments below show that, across all three backbones, action-expert representations have lower task separability in hallucination-failure trajectories than in successful or manipulation-failure trajectories, supporting a closer association between more pronounced separability reductions and task deviation during hallucination, rather than an equally pronounced change across all execution failures.
To address this, we use expert trajectories from the original task as a common reference and compare the mean Silhouette Scores of VLM representations and action-expert representations for successful trajectories, manipulation failures, and hallucination failures from the task variants, under matched scenes and execution stages.
These results are summarized in R3-Table2. In successful trajectories and manipulation failures, does not exhibit the same degree of low separability observed in hallucination failures. This supports a closer association between reduced task separability in action-expert representations and task deviation during hallucination. In particular, manipulation failures retain relatively higher task separability, indicating that this phenomenon does not occur to the same extent across all execution failures and further supporting its association with task deviation during hallucination.
R3-Table2. Mean Silhouette Scores of VLM and action-expert representations across three trajectory outcomes.
| Backbone | Trajectory outcome | Silhouette | Silhouette |
|---|---|---|---|
| SmolVLA | Success | 0.27 | 0.01 |
| Manipulation failure | 0.30 | 0.12 | |
| Hallucination failure | 0.19 | -0.14 | |
| XVLA | Success | 0.31 | 0.18 |
| Manipulation failure | 0.19 | 0.16 | |
| Hallucination failure | 0.24 | 0.05 | |
| π₀.₅ | Success | 0.31 | 0.08 |
| Manipulation failure | 0.24 | 0.21 | |
| Hallucination failure | 0.33 | -0.09 |
We have clarified the trajectory composition and scope of the conclusions for Figure 3 in the Introduction of the revised manuscript, and provided the separability comparisons across the three outcomes and the experimental settings in the mechanism analysis section and appendix.
Thanks & Initial Response to Reviewer Dt1z (W2, Q2)
[W2, Q2 - VSR may read task information already present in its inputs rather than measure action–task consistency; its validity on unseen tasks particularly requires verification.]
We are happy to further explain the basis of VSR scoring and provide additional experiments to address this concern. Figure 6 already provides evidence that VSR can distinguish semantic deviation from manipulation errors: successful trajectories and manipulation failures have relatively high alignment scores, whereas hallucination failures have markedly lower scores. We recognize that this analysis does not fully control input differences across trajectories. We therefore supplement it with the following controlled comparison.
Fix the scene and instruction and compare three action types. Using π₀.₅, we start from the same LIBERO environment and robot state and construct an action matching the current task, a plausible mismatched action appropriate for another task in the same scene, and random noise actions. Under otherwise identical conditions, we re-extract for each candidate action and compute the VSR score. We evaluate scores separately on seen tasks from LIBERO-goal and unseen task variants of LIBERO-goal. As shown in R3-Table3, matched actions achieve the highest mean scores on both Seen and Unseen tasks.
R3-Table3. VSR scores for matched, plausible mismatched, and random actions under fixed scenes and instructions.
| Test tasks | Matched-action score | Plausible mismatched-action score | Random-action score |
|---|---|---|---|
| Seen | 0.85 | 0.36 | 0.46 |
| Unseen | 0.69 | 0.49 | 0.44 |
Thanks & Initial Response to Reviewer Dt1z (W1)
[W1 - The existing representation analysis does not establish correct task understanding by the VLM or sufficiently attribute hallucinations to the action expert.]We are happy to provide further explanations below. The existing representation analyses are insufficient to independently localize the source of hallucinations; however, insufficient use of useful upstream task information during downstream action generation is one source of hallucinations and the focus of this work. We have clarified our scope in the
Introductionand respond from three perspectives.(1) Research objective: useful information retained upstream but insufficiently used downstream. We focus on cases where the upstream VLM retains some useful task information, yet downstream action generation fails to adequately use it, producing actions that deviate from task or scene semantics. GSR's analysis of instruction generalization and semantic-to-action conversion [1], together with Act2Answer's investigation of task-information decodability in intermediate VLM representations [2], also supports the view that the action expert may fail to fully use useful information retained by the VLM.
(2) Experimental observations: complementary evidence from representation analysis and probes.
Figure 3shows low separability between action-expert representations of hallucination-failure trajectories for task variants and those of original-task expert trajectories, whereas the corresponding VLM representations exhibit higher separability. To further examine task-information decodability in upstream representations, we added lightweight probes for the target object, target location, and task relation. For each backbone, we selected 20 test trajectories per outcome category: success, manipulation failure, and hallucination failure. We sampled intermediate representations from the frozen model and evaluated them using trained probes.As shown in R3-Table1, some task and spatial information remains decodable from VLM representations even during hallucination failures. Together with the separability analysis, these observations are consistent with insufficient downstream use of useful information retained upstream, providing further support for our research objective.
R3-Table1. Task information decodable from VLM representations across trajectory outcomes.
(3) Practical benefits: higher task success and lower hallucination rates.
Table 1andTable 2of the paper show that AVSA improves overall task success and reduces the incidence of hallucination failures in the evaluated settings. These results demonstrate the practical value of guiding correction through action–visual-semantic consistency and show that intervening in downstream action deviations can yield overall performance gains.References
[1] Zhaokai Yin and Zhipeng Zhang. Grounded Semantic Re-Binding for Robust Instruction Generalization in Vision-Language-Action Models. arXiv:2608.02497, 2026.
[2] Nikita Kachaev et al. Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models. arXiv:2606.19297, 2026.