acceptodds
Under review as a conference paper at ICLR 2027

VLA-Hallucination: Diagnosing and Mitigating Hallucination Failures in Vision-Language-Action Models

Abstract

Vision-Language-Action (VLA) models have recently become an important technique for general-purpose robotic manipulation. However, existing VLA models still exhibit clear limitations in generalization to unseen tasks. We find that VLA task failures can be categorized into manipulation failures and hallucination failures. The latter refers to cases where the model-generated actions are inconsistent with the target task or the surrounding environment, and constitutes a major factor limiting VLA generalization. Further analysis reveals that an important cause of hallucination failures is that task-relevant visual-semantic information preserved in the upstream VLM is not effectively utilized during downstream action generation, causing the generated actions to deviate from the current task semantics. To address this issue, we propose Action-to-Visual Semantic Alignment (AVSA). AVSA introduces a lightweight visual-semantic reconstructor that reconstructs visual-semantic information conditioned on the action, online measures the alignment between Action Expert outputs and VLM visual semantics, and uses this signal to correct actions during Flow Matching, thereby aligning action generation with the upstream visual-semantic understanding. AVSA can be integrated with multiple mainstream VLA models without retraining the VLA backbone, and consistently reduces hallucination rates while improving task success rates across diverse unseen settings, including environment changes, semantic changes, and paraphrased instructions. For example, on LIBERO-Para, AVSA improves the task success rate of by 7.1%. Overall, our results demonstrate that explicitly diagnosing and aligning action generation with visual semantics provides an effective approach to mitigating hallucination failures and improving the generalizability of VLA models.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Robot Manipulation, Hallucination Mitigation, Generalization, Embodied AI

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Dt1zOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies hallucination failures in VLA models under distribution shifts. It distinguishes hallucination failures, where actions deviate from the visual scene or instruction, from manipulation failures caused by low-level execution errors, and defines the Hallucination Rate (HR) as the proportion of hallucination failures among all failures. The paper argues that task-relevant semantics may weaken between the VLM representation HIH^I and the Action Expert representation HAH^A. It proposes AVSA, which reconstructs HIH^I from an instruction-free scene representation HI,∅H^{I,\varnothing} and HAH^A, and uses reconstruction similarity to guide Flow-Matching action generation. Experiments on three VLA backbones, several simulation benchmarks, and a real robot report reduced HR and improvements in success rate.

Critical strengths
  1. The paper addresses an important problem for VLA deployment. Distinguishing semantic/action deviations from low-level manipulation errors provides information that aggregate success rates cannot capture.
  2. AVSA is an inference-time correction framework that keeps the VLA backbone frozen. The evaluation covers three VLA backbones, multiple simulation benchmarks, and a real robot, with generally consistent improvement trends.
Critical weaknesses
  1. The claimed hallucination mechanism is insufficiently supported. Figure 3 only shows that HIH^I is more separable than HAH^A under the selected metric. It does not establish that HIH^I correctly understands the task or that the low separability of HAH^A causes hallucination. Corresponding analyses of successful trajectories and manipulation failures are missing.
  2. The validity of the VSR and several experimental conclusions remain unclear. The VSR may recover task-correlated information already contained in HI,∅H^{I,\varnothing} or HAH^A, rather than genuinely evaluating action–instruction consistency. In addition, unclear metric denominators and inconsistencies appear across figures and tables.
Questions for authors
  1. Is the low separability of HAH^A in Figure 3 specific to hallucination trajectories, or does it also occur in successful trajectories and manipulation failures?
  2. How do the authors establish that the VSR score reflects action–task consistency rather than task information already contained in HI,∅H^{I,\varnothing} or HAH^A, particularly for unseen tasks?
  3. Please clarify the reporting of the failure metrics. In Figure 2, the stacked segments appear to represent failure types as fractions of all trials, whereas HR is defined over failed trials only. For LIBERO-Para, the SmolVLA and π0.5\pi_{0.5} bars also appear to be swapped relative to Table 2. In addition, the claimed average HR reduction of 11.5% is not reproduced by Table 2, which gives a macro-average absolute reduction of 6.0 percentage points. Please clarify these inconsistencies and provide the raw counts.
8 comments
AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer Dt1z (W1)

[W1 - The existing representation analysis does not establish correct task understanding by the VLM or sufficiently attribute hallucinations to the action expert.]

We are happy to provide further explanations below. The existing representation analyses are insufficient to independently localize the source of hallucinations; however, insufficient use of useful upstream task information during downstream action generation is one source of hallucinations and the focus of this work. We have clarified our scope in the Introduction and respond from three perspectives.

(1) Research objective: useful information retained upstream but insufficiently used downstream. We focus on cases where the upstream VLM retains some useful task information, yet downstream action generation fails to adequately use it, producing actions that deviate from task or scene semantics. GSR's analysis of instruction generalization and semantic-to-action conversion [1], together with Act2Answer's investigation of task-information decodability in intermediate VLM representations [2], also supports the view that the action expert may fail to fully use useful information retained by the VLM.

(2) Experimental observations: complementary evidence from representation analysis and probes. Figure 3 shows low separability between action-expert representations of hallucination-failure trajectories for task variants and those of original-task expert trajectories, whereas the corresponding VLM representations exhibit higher separability. To further examine task-information decodability in upstream representations, we added lightweight probes for the target object, target location, and task relation. For each backbone, we selected 20 test trajectories per outcome category: success, manipulation failure, and hallucination failure. We sampled intermediate representations from the frozen model and evaluated them using trained probes.

As shown in R3-Table1, some task and spatial information remains decodable from VLM representations even during hallucination failures. Together with the separability analysis, these observations are consistent with insufficient downstream use of useful information retained upstream, providing further support for our research objective.

R3-Table1. Task information decodable from VLM representations across trajectory outcomes.

BackboneTrajectory outcomeTarget-object accuracy (%) ↑Target-location error (cm) ↓Task-relation accuracy (%) ↑
SmolVLASuccess954.17100
Manipulation failure804.4695
Hallucination failure605.5475
XVLASuccess1003.89100
Manipulation failure953.50100
Hallucination failure704.36100
π₀.₅Success1001.27100
Manipulation failure1001.54100
Hallucination failure852.8595

(3) Practical benefits: higher task success and lower hallucination rates. Table 1 and Table 2 of the paper show that AVSA improves overall task success and reduces the incidence of hallucination failures in the evaluated settings. These results demonstrate the practical value of guiding correction through action–visual-semantic consistency and show that intervening in downstream action deviations can yield overall performance gains.

References

[1] Zhaokai Yin and Zhipeng Zhang. Grounded Semantic Re-Binding for Robust Instruction Generalization in Vision-Language-Action Models. arXiv:2608.02497, 2026.

[2] Nikita Kachaev et al. Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models. arXiv:2606.19297, 2026.

AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer Dt1z (Q1)

[Q1 - Is low separability of action-expert representations specific to hallucination? Corresponding analyses of successful trajectories and manipulation failures are missing.]

Comparing representation separability across successes, manipulation failures, and hallucination failures is indeed crucial to determining whether this phenomenon is closely associated with hallucination. The additional experiments below show that, across all three backbones, action-expert representations HAH^A have lower task separability in hallucination-failure trajectories than in successful or manipulation-failure trajectories, supporting a closer association between more pronounced separability reductions and task deviation during hallucination, rather than an equally pronounced change across all execution failures.

To address this, we use expert trajectories from the original task as a common reference and compare the mean Silhouette Scores of VLM representations HIH^I and action-expert representations HAH^A for successful trajectories, manipulation failures, and hallucination failures from the task variants, under matched scenes and execution stages.

These results are summarized in R3-Table2. In successful trajectories and manipulation failures, HAH^A does not exhibit the same degree of low separability observed in hallucination failures. This supports a closer association between reduced task separability in action-expert representations and task deviation during hallucination. In particular, manipulation failures retain relatively higher task separability, indicating that this phenomenon does not occur to the same extent across all execution failures and further supporting its association with task deviation during hallucination.

R3-Table2. Mean Silhouette Scores of VLM and action-expert representations across three trajectory outcomes.

BackboneTrajectory outcomeHIH^I SilhouetteHAH^A Silhouette
SmolVLASuccess0.270.01
Manipulation failure0.300.12
Hallucination failure0.19-0.14
XVLASuccess0.310.18
Manipulation failure0.190.16
Hallucination failure0.240.05
π₀.₅Success0.310.08
Manipulation failure0.240.21
Hallucination failure0.33-0.09

We have clarified the trajectory composition and scope of the conclusions for Figure 3 in the Introduction of the revised manuscript, and provided the separability comparisons across the three outcomes and the experimental settings in the mechanism analysis section and appendix.

AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer Dt1z (W2, Q2)

[W2, Q2 - VSR may read task information already present in its inputs rather than measure action–task consistency; its validity on unseen tasks particularly requires verification.]

We are happy to further explain the basis of VSR scoring and provide additional experiments to address this concern. Figure 6 already provides evidence that VSR can distinguish semantic deviation from manipulation errors: successful trajectories and manipulation failures have relatively high alignment scores, whereas hallucination failures have markedly lower scores. We recognize that this analysis does not fully control input differences across trajectories. We therefore supplement it with the following controlled comparison.

Fix the scene and instruction and compare three action types. Using π₀.₅, we start from the same LIBERO environment and robot state and construct an action matching the current task, a plausible mismatched action appropriate for another task in the same scene, and random noise actions. Under otherwise identical conditions, we re-extract HAH^A for each candidate action and compute the VSR score. We evaluate scores separately on seen tasks from LIBERO-goal and unseen task variants of LIBERO-goal. As shown in R3-Table3, matched actions achieve the highest mean scores on both Seen and Unseen tasks.

R3-Table3. VSR scores for matched, plausible mismatched, and random actions under fixed scenes and instructions.

Test tasksMatched-action scorePlausible mismatched-action scoreRandom-action score
Seen0.850.360.46
Unseen0.690.490.44
Reviewer_geaFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The problem in this paper is that vision-language-action models can fail under task and environment shift not only through poor execution, but also by taking actions inconsistent with the instruction or scene, while aggregate success rates obscure this distinction. It divides unsuccessful rollouts into manipulation failures, where the policy pursues the intended task but executes it poorly, and hallucination failures. The paper defines hallucination rate as the fraction of failures assigned to the latter class and argues, using representation separability, that task-relevant information remains in the upstream VLM but degrades in the Action Expert. The proposed Action-to-Visual Semantic Alignment method trains a small reconstructor on a frozen VLA to predict task-conditioned visual semantics from task-agnostic visual context and the action representation. During flow-matching inference, low alignment activates gradient guidance on the velocity field. Experiments cover SmolVLA, XVLA, and pi0.5 on LIBERO-Pro and LIBERO-Para, pi0.5 and XVLA on a RoboTwin subset, and pi0.5 on 120 real-robot trials. The method generally raises success rate and lowers the reported hallucination rate.

Critical strengths

Insights. I think the insights are overall good to me, but not very sure this is really useful to the robotics. Presentation. Overall the presentation are acceptable, except that some tables fonts are too large.

Critical weaknesses

Weak Positioning to Related Works. Authors should do a more careful survey on "hallucinations of MLLMs and LLMs", and make sure they do a detailed discussion about their difference to other methods, or direct comparisons on results, rather than ignoring them. Few papers are mentioned.

Questions for authors

See Weakness.

4 comments
AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer geaF (W1, W2)

[W1 - Table formatting: Some table fonts are too large.]

Thank you for your feedback on table formatting. We have carefully revised the paper's table formatting.

[W2 - Insufficient discussion of related work: A more comprehensive review of hallucination research in LLMs and MLLMs is needed, together with clearer distinctions from existing methods and additional experimental comparisons.]

Positioning AVSA relative to prior methods clarifies its contribution. We added TF-CAG [1] and VLA-Corrector [2] comparisons to Table 2 and expanded Appendix Related Work in two aspects.

(1) Connections to and differences from LLM/MLLM hallucination research. For LLMs, DoLa [3] contrasts output distributions across layers to improve factuality. Bai et al.'s survey [4] organizes MLLM hallucination mitigation into data, model, training, and inference approaches. At inference time, VCD [5] contrasts predictions conditioned on original and distorted images to reduce object hallucinations associated with language priors; Octopus [6] dynamically selects contrastive decoding strategies across generation steps.

AVSA also intervenes in generation to mitigate hallucinations, but differs in its correction target and mechanism. DoLa, VCD, and Octopus primarily adjust text-token distributions. AVSA constructs a consistency signal through action-conditioned visual-semantic reconstruction and uses gradient guidance during continuous action generation to align actions with scene and task semantics.

(2) Comparison with VLA action-correction methods. Existing methods address different action failures. We distinguish two categories by their primary correction targets to clarify their practical benefits:

  • Fine manipulation and execution deviations. VLA-Corrector monitors discrepancies between predicted and observed visual changes, interrupts action chunks, and triggers correction, primarily addressing error accumulation in contact-rich interaction and open-loop execution.
  • Task intent. TF-CAG, proposed in CAG, amplifies the difference between action predictions with and without language conditioning to strengthen instruction following; AVSA guides correction through action-conditioned visual-semantic reconstruction.

R2-Table1. Performance and inference overhead on LIBERO-Para.

BackboneMethodSR (%) ↑HR (%) ↓Inference time (ms/action chunk) ↓ηSR\eta_{\mathrm{SR}} (pp/100 ms) ↑ηHR\eta_{\mathrm{HR}} (pp/100 ms) ↑
SmolVLABaseline13.0758.19261.32——
+ TF-CAG13.1555.52520.860.031.03
+ VLA-Corrector12.9858.06277.80−0.590.74
+ AVSA (Ours)16.7651.96398.142.704.55
XVLABaseline64.5722.19211.73——
+ TF-CAG63.9322.41409.43−0.32−0.11
+ VLA-Corrector64.7422.17225.091.280.18
+ AVSA (Ours)70.0116.23308.465.636.16
π₀.₅Baseline72.2124.83279.24——
+ TF-CAG74.0022.48556.480.640.85
+ VLA-Corrector72.4324.68298.811.120.75
+ AVSA (Ours)75.5919.16524.931.372.31

As shown in R2-Table1, AVSA achieves higher success rates and lower hallucination rates than TF-CAG and VLA-Corrector across all three backbones, with lower latency than TF-CAG. ηSR\eta_{\mathrm{SR}} and ηHR\eta_{\mathrm{HR}} measure the SR increase and HR reduction (pp) per additional 100 ms over Baseline; higher is better for both. AVSA exceeds both comparison methods on both metrics across all three backbones, yielding greater correction benefits per added inference time.

While TF-CAG amplifies language conditioning and VLA-Corrector corrects execution deviations, AVSA uses explicit action–scene/task consistency to guide correction. Its action-conditioned visual-semantic reconstruction identifies and guides the correction of semantic deviations during action generation, directly targeting the hallucination failures studied here. The observed HR reductions and SR gains support its practical benefits in the evaluated settings.

References

[1] Yu Fang et al. When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs. CoRL, 2026.

[2] Yi Pan et al. VLA-Corrector: Lightweight Detect-and-Correct Inference for Adaptive Action Horizon. NeurIPS, 2026.

[3] Yung-Sung Chuang et al. DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models. ICLR, 2024.

[4] Zechen Bai et al. Hallucination of Multimodal Large Language Models: A Survey. arXiv:2404.18930, 2024.

[5] Sicong Leng et al. Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding. CVPR, 2024.

[6] Wei Suo et al. Octopus: Alleviating Hallucination via Dynamic Contrastive Decoding. CVPR, 2025.

AuthorsOpenReview ↗

Follow-Up: Seeking Further Feedback (Discussion Deadline Approaching)

Dear Reviewer geaF,

We hope this message finds you well. Following up on our previous exchange regarding our paper, we would like to kindly ask whether you have any further concerns or feedback regarding our responses and the additional experiments and clarifications we have provided.

As the discussion deadline is approaching, we would be very grateful for any further thoughts you may have. We remain happy to provide additional explanations or clarifications to address any remaining questions and further improve our manuscript.

We recognize that reviewing is a voluntary and uncompensated service, and we sincerely appreciate the time and effort you have devoted to it. We have also invested substantial time and effort in this research and the rebuttal, with the aim of contributing to more efficient VLA systems. Your feedback has been valuable in helping us strengthen both the technical presentation and experimental validation of our work.

Thank you again for your time and consideration. We look forward to hearing from you.

Best regards,

The Authors

Thanks authors for their detailed rebuttal. I maintain my rating.

AuthorsOpenReview ↗

Thanks & Follow Up Response to Reviewer geaF

We sincerely thank the reviewer for taking the time to carefully consider our responses and for acknowledging that our clarifications have addressed your concerns. We appreciate your thoughtful feedback and reassessment. Your suggestions have been valuable in helping us strengthen the experimental evidence and improve the clarity of our manuscript. We will ensure that the corresponding revisions are clearly reflected in the updated version.

Thank you again for your time and constructive feedback.

Reviewer_JkNROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies failures in which a VLA’s actions do not match the scene or instruction. The authors propose AVSA, which reconstructs task-conditioned visual features from the action representation and scene context. They uses the mismatch with VLM features to guide action generation. Experiments show improved task success across several backbones with additional validation on a real robot.

Critical strengths
  1. The paper is well written. The narrative is easy to follow, and the figures help explain both the failure categories and the proposed method.
  2. Using the VLM’s task-conditioned representation as a reference for action correction is a promising idea. AVSA can be added without updating the backbone weights.
  3. The evaluation covers three VLA backbones and includes real-robot experiments. The improvements in task success are encouraging - particularly the increase in the real-world evaluation. The ablation on reconstructor inputs also provides useful support for the design.
Critical weaknesses
  1. The paper only compares the models with and without AVSA. I would like to see a comparison with other correction methods on the same backbone. Methods for improving instruction following would be particularly relevant for the instruction perturbations.

  2. I am not convinced that Figure 3 identifies the Action Expert as the source of hallucinations. The VLM could misunderstand the task even if its features separate the task variants well. Figure 6 does show lower alignment scores for hallucination failures, but this still does not tell us whether the error comes from the VLM or the Action Expert.

  3. Please report inference time with and without AVSA. The method requires gradient computation during action generation, and it is unclear how much overhead this adds.

Questions for authors
  1. Does AVSA ever turn a successful baseline rollout into a failure? Could you report paired outcomes under the same initial conditions, distinguishing manipulation failures from hallucination failures?

  2. How does AVSA behave when the upstream VLM representation is itself incorrect? Do you observe cases where increasing alignment reinforces an incorrect action?

I generally like the paper and find the approach promising. My main concern is that the current evaluation does not sufficiently establish the contribution of the proposed reconstruction-based guidance. A controlled comparison with an alternative correction method would help clarify this, and could change my recommendation.

8 comments
AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer JkNR (W1, W3)

[W1 - Comparisons with other correction methods are missing, particularly methods that improve instruction following.]

Thank you for this suggestion. We added TF-CAG [1] and VLA-Corrector [2] to compare methods with different correction targets. TF-CAG targets task intent and instruction following by amplifying the difference between action predictions with and without language conditioning. VLA-Corrector targets fine manipulation and execution deviations by detecting execution discrepancies and adapting the action horizon.

R1-Table1. Correction effectiveness and inference overhead on LIBERO-Para.

BackboneMethodSR (%) ↑HR (%) ↓Time (ms/chunk) ↓ηSR\eta_{\mathrm{SR}} (pp/100 ms) ↑ηHR\eta_{\mathrm{HR}} (pp/100 ms) ↑
SmolVLABaseline13.0758.19261.32——
+ TF-CAG13.1555.52520.860.031.03
+ VLA-Corrector12.9858.06277.80−0.590.74
+ AVSA (Ours)16.7651.96398.142.704.55
XVLABaseline64.5722.19211.73——
+ TF-CAG63.9322.41409.43−0.32−0.11
+ VLA-Corrector64.7422.17225.091.280.18
+ AVSA (Ours)70.0116.23308.465.636.16
π₀.₅Baseline72.2124.83279.24——
+ TF-CAG74.0022.48556.480.640.85
+ VLA-Corrector72.4324.68298.811.120.75
+ AVSA (Ours)75.5919.16524.931.372.31

As shown in R1-Table1, AVSA achieves higher success rates and lower hallucination rates than both TF-CAG and VLA-Corrector across all three backbones.

Compared with TF-CAG's amplification of language conditioning and VLA-Corrector's correction of execution deviations, AVSA explicitly measures the consistency of generated actions with scene and task semantics and uses this signal for correction. Its action-conditioned visual-semantic reconstruction identifies and guides the correction of semantic deviations during action generation, directly targeting the hallucination failures studied here. The lower hallucination rates and higher success rates in the comparison support the practical benefits of this design in the evaluated settings.

[W3 - Inference times with and without AVSA are missing, leaving the additional cost of gradient computation unclear.]

We understand the concern about inference overhead. For π₀.₅ + AVSA, gradient guidance takes 34.75 ms per action chunk, accounting for approximately 6.62% of the total inference time of 524.93 ms. We added end-to-end latency and a breakdown by stage; R1-Table2 reports the time spent on the VLM forward pass, AVSA reconstruction, action-expert denoising, and gradient guidance.

R1-Table2. Inference-time breakdown for π₀.₅ + AVSA on LIBERO-Para.

ComponentTime (ms/chunk)
VLM forward pass198.65
AVSA reconstruction20.77
Action-expert denoising270.76
Gradient guidance34.75
Total524.93

AVSA is slower than Baseline and VLA-Corrector but faster than TF-CAG. To assess gains relative to overhead, we report ηSR\eta_{\mathrm{SR}} and ηHR\eta_{\mathrm{HR}}: the SR increase and HR reduction per additional 100 ms of inference time relative to Baseline, respectively, measured in percentage points; higher is better for both. As shown in R1-Table1, AVSA outperforms both comparison methods on both metrics across all three backbones, yielding greater correction benefits per unit of additional computation in the evaluated settings.

References

[1] Yu Fang et al. When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs. CoRL, 2026.

[2] Yi Pan et al. VLA-Corrector: Lightweight Detect-and-Correct Inference for Adaptive Action Horizon. NeurIPS, 2026.

AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer JkNR (W2)

[W2 - Task separability in Figure 3 and lower alignment scores in Figure 6 do not establish that hallucinations originate in the action expert rather than task misunderstanding by the VLM.]

We are happy to provide further explanations below. The existing representation analyses are insufficient to independently localize the source of hallucinations; however, insufficient use of useful upstream task information during downstream action generation is one source of hallucinations and the focus of this work. We have clarified our scope in the Introduction and respond from three perspectives.

(1) Research objective: useful information retained upstream but insufficiently used downstream. We focus on cases where the upstream VLM retains some useful task information, yet downstream action generation fails to adequately use it, producing actions that deviate from task or scene semantics. GSR's analysis of instruction generalization and semantic-to-action conversion [1], together with Act2Answer's investigation of task-information decodability in intermediate VLM representations [2], also supports the view that the action expert may fail to fully use useful information retained by the VLM.

(2) Experimental observations: complementary evidence from representation analysis and probes. Figure 3 shows low separability between action-expert representations of hallucination-failure trajectories for task variants and those of original-task expert trajectories, whereas the corresponding VLM representations exhibit higher separability. To further examine task-information decodability in upstream representations, we added lightweight probes for the target object, target location, and task relation. For each backbone, we selected 20 test trajectories per outcome category: success, manipulation failure, and hallucination failure. We sampled intermediate representations from the frozen model and evaluated them using trained probes.

As shown in R1-Table3, some task and spatial information remains decodable from VLM representations even during hallucination failures. Together with the separability analysis, these observations are consistent with insufficient downstream use of useful information retained upstream, providing further support for our research objective.

R1-Table3. Task information decodable from VLM representations across trajectory outcomes.

BackboneTrajectory outcomeTarget-object accuracy (%) ↑Target-location error (cm) ↓Task-relation accuracy (%) ↑
SmolVLASuccess954.17100
Manipulation failure804.4695
Hallucination failure605.5475
XVLASuccess1003.89100
Manipulation failure953.50100
Hallucination failure704.36100
π₀.₅Success1001.27100
Manipulation failure1001.54100
Hallucination failure852.8595

(3) Practical benefits: higher task success and lower hallucination rates. Table 1 and Table 2 of the paper show that AVSA improves overall task success and reduces the incidence of hallucination failures in the evaluated settings. These results demonstrate the practical value of guiding correction through action–visual-semantic consistency and show that intervening in downstream action deviations can yield overall performance gains.

References

[1] Zhaokai Yin and Zhipeng Zhang. Grounded Semantic Re-Binding for Robust Instruction Generalization in Vision-Language-Action Models. arXiv:2608.02497, 2026.

[2] Nikita Kachaev et al. Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models. arXiv:2606.19297, 2026.

AuthorsOpenReview ↗

Thanks & Initial Response to Reviewer JkNR (Q1, Q2)

[Q1 - Can AVSA turn an originally successful trajectory into a failure? Please report paired outcomes under identical initial conditions, distinguishing manipulation failures from hallucination failures.]

To address the reviewer’s concern, we conducted a paired evaluation of Baseline and AVSA under the same tasks and initial conditions. R1-Table4 presents the paired outcomes on LIBERO-Para. Substantially more trajectories transition from failure to success with AVSA than from success to failure, demonstrating an overall net benefit.

As shown in the table, transitions from hallucination failure to success or manipulation failure account for 8.55% (4.13% + 4.42%) of all paired trajectories, whereas reverse transitions into hallucination failure account for 2.60% (0.65% + 1.95%). Meanwhile, approximately 6.61% (2.48% + 4.13%) of all paired trajectories fail under Baseline but are rescued to success by AVSA, compared with approximately 2.44% (1.79% + 0.65%) that succeed under Baseline but fail with AVSA. Thus, observed improvements outnumber reverse transitions, although not every trajectory benefits. Overall success increases from 49.95% (47.51% + 1.79% + 0.65%) to 54.12% (47.51% + 2.48% + 4.13%). Transitions from hallucination failure to manipulation failure indicate reduced task deviation.

R1-Table4. Paired outcome transitions from Baseline to AVSA on LIBERO-Para.

Each cell is expressed as a percentage of all paired trajectories.

Baseline outcome / AVSA outcomeSuccessManipulation failureHallucination failure
Success47.51%1.79%0.65%
Manipulation failure2.48%10.55%1.95%
Hallucination failure4.13%4.42%26.52%

[Q2 - When the upstream VLM representation is incorrect, can stronger alignment reinforce incorrect actions?]

We recognize this issue. Effective correction by AVSA depends on sufficiently reliable task-related information from the VLM. If the reference representation itself is incorrect, stronger alignment may fail to improve actions or may even reinforce incorrect behavior. However, when the VLM cannot provide accurate representations to the action expert, the reliability of the generated actions is itself not guaranteed. Our method therefore primarily targets cases where useful task information is retained upstream but action generation deviates from it. AVSA does not guarantee correction of erroneous upstream semantics. Nevertheless, Tables 1 and 2 of the paper show overall performance benefits in the evaluated settings despite this limitation. The paired evaluation in R1-Table4 further shows that trajectories corrected from failure to success account for 6.61% (2.48% + 4.13%) of all paired trajectories, exceeding the 2.44% (1.79% + 0.65%) that transition from success to failure, demonstrating an overall net benefit.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

For clarity, we refer to Reviewers JkNR, geaF, and Dt1z as R1, R2, and R3, respectively.

We sincerely thank the reviewers for their constructive feedback and recognition of distinguishing semantic deviations from manipulation failures (R3) and our research insights (R2). We appreciate the recognition that VLM task-conditioned representations offer a promising reference for action correction (R1) and that AVSA requires no updates to the VLA backbone (R1, R3). We also appreciate the positive assessments of our evaluation across backbones, benchmarks, and real-robot tasks (R1, R3), particularly the real-world success gains and reconstructor-input ablations (R1), and the clarity of our narrative and figures (R1).

We have addressed concerns about baselines, hallucination sources, score validity, and statistics individually. Below we summarize our contributions and updates.

Core Contributions of Our Work

  1. Diagnosing and quantifying VLA hallucination failures. We distinguish actions that deviate from the scene or instruction from task-consistent manipulation failures, and introduce HR to complement aggregate success rates. We examine how task information is retained and used between the VLM and action generation.

  2. Inference-time correction through visual-semantic reconstruction. AVSA reconstructs task-related visual semantics from action representations and instruction-free scene context. Alignment with the VLM reference monitors deviations and triggers gradient guidance during Flow Matching when the score falls below a threshold, keeping the backbone frozen.

  3. Practical gains across models and settings. Evaluation covers SmolVLA, XVLA, and π₀.₅ across LIBERO-Pro, LIBERO-Para, a RoboTwin 2.0 subset, and real-robot tasks. On LIBERO-Para, AVSA improves macro-average SR by 4.17 percentage points and reduces HR by 5.95 percentage points. In real-robot experiments, π₀.₅'s mean SR rises from 38.3% to 50.0%, while HR falls from 39.2% to 24.2%.

Updates to Experimental Results During Rebuttal

  • Section 4.1 / Table 2: Added TF-CAG/VLA-Corrector comparisons, latency, and SR-gain/HR-reduction efficiency metrics. Across all three LIBERO-Para backbones, AVSA achieves higher SR, lower HR, and better efficiency on both metrics than both methods.

  • Introduction / Figure 3; Appendix C.4 / Table 9: Added linear probes for target objects, locations, and task relations, plus separability comparisons across successes, manipulation failures, and hallucination failures. Some task information remains decodable during hallucination failures, whose action-expert representations have lower separability than the other outcomes on all three backbones.

  • Section 4.2 / Table 5: Added controlled comparisons of matched, plausible mismatched, and random actions with fixed scenes and instructions. Matched actions score 0.85 / 0.69 on seen / unseen tasks, exceeding plausible mismatches by 0.49 / 0.20, supporting VSR's sensitivity to action–task consistency.

  • Section 4.2 / Figure 7: Added paired outcomes under identical tasks and initial conditions. Failure-to-success transitions account for 6.61% of all pairs, versus 2.44% in reverse. Hallucination-to-success/manipulation-failure transitions account for 8.55%, versus 2.60% in reverse, showing a net benefit.

  • Appendix C.4 / Tables 8, 10: Reported success, manipulation-failure, and hallucination-failure counts, rates, and latency for Baseline and AVSA across all three backbones, plus SDs of ΔSR and ΔHR over five evaluations to assess gain stability. Added the π₀.₅+AVSA latency breakdown (524.93 ms/chunk; gradient guidance: 6.62%) to clarify computational overhead.

Updates to Discussions During Rebuttal

  • Introduction; Appendix C.4: Added probing and representation-analysis details. Our responses clarify that decodability does not establish complete task understanding, nor does separability alone localize hallucination sources. AVSA targets downstream deviations from retained upstream information; it does not guarantee correction of erroneous upstream semantics.

  • Appendix Related Work (Appendix A): Expanded LLM/MLLM hallucination mitigation across data, model, training, and inference, and distinguished fine-manipulation correction from task-deviation correction, clarifying how AVSA differs from other action-correction methods.

  • Introduction; Appendices C.1, C.3: Defined HR over all trials and clarified its relation to manipulation and total failure rates. We have updated the corresponding statistics and figures in the current manuscript draft.

These updates clarify AVSA's benefits, costs, and scope and are marked in blue in the revised manuscript. We have also provided core implementation code for inspection.

We look forward to the reviewers' favorable consideration and welcome further questions.

Area_Chair_wMPvOpenReview ↗

Please Review the Author Rebuttal and Participate in the Discussion

Dear Reviewers,

Thank you for your efforts in reviewing this submission. The authors have now provided their rebuttal. I kindly encourage everyone to carefully read the responses, actively participate in the discussion, and engage with the authors and fellow reviewers to help reach a well-informed assessment.

Thank you again for your time and contributions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.