acceptodds
Under review as a conference paper at ICLR 2027

-VLA: Visual Self-Grounding for Vision-Language-Action Policies

14 views

Abstract

Vision-language-action (VLA) policies have shown strong promise as generalist controllers for robot manipulation, yet their action predictions can remain fragile under changes in camera extrinsics. We identify a systematic failure pattern in which camera perturbations induce directional drift in predicted robot trajectories, revealing that current VLAs may exploit image-space shortcuts, such as pixel-coordinate correlations, rather than grounding actions in robot–object geometry. To bridge this gap, we introduce -VLA, a Self-Supervised alignment approach to incentivize visual Self-grounding capability in VLA policies. Our central insight is that robot motion provides intrinsic spatial supervision for localizing the robot itself. To implicitly guide the policy to attend to the robot, we formulate an auxiliary grounding objective to recover motion-derived localization targets from its visual representations. Across simulation benchmarks and real-world manipulation tasks, -VLA consistently improves average task success on two leading VLA backbones, and GR00T-N1.6. Under camera-extrinsic perturbations, -VLA increases the average success rate on real-world tasks from 16.4% to 29.4%. These findings highlight the importance of visual self-grounding for robust vision-language-action policies.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied AI, Vision Language Action Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_USSxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies image-space shortcuts in VLA policies, where camera changes induce systematic action drift despite unchanged robot–object geometry. Through prompt-conditioned relevance maps and frozen-feature localization probes, it identifies weak visual self-grounding as a potential source of this behavior. It introduces S³-VLA, which mines robot-localization targets from demonstration videos using visual motion and supervises visual representations through an auxiliary self-grounding objective. Evaluations on SimplerEnv, Meta-World, and real-world manipulation show improved task performance, while complementary analyses support the role of visual self-grounding in camera robustness.

Critical strengths
  1. The paper addresses a meaningful problem with a compelling motivation. The image-space shortcut perspective provides a clear explanation for an important failure mode of VLA policies. Camera-only perturbations make the problem concrete and naturally motivate using the robot’s visible body as a spatial reference.

  2. The self-grounding analyses are insightful and complementary. Real-world relevance maps and simulation-based localization probes provide an informative view of the learned representations. The robot-target attribution audit, matched occlusion controls and action-consistency measurements further support the proposed interpretation. The paper goes beyond success-rate comparisons to investigate what the method changes in depth.

  3. The approach is practical and empirically well supported. Motion-derived supervision avoids manual robot annotations and camera calibration, while the auxiliary grounding head is used only during training. Experiments across two VLA backbones, multiple simulated embodiments and real-world tasks demonstrate the approach’s utility. The state-prediction controls and localization-target comparison provide additional support for the design.

  4. The paper is clearly written and well organized. The progression from identifying image-space shortcuts to probing self-grounding, designing auxiliary supervision and evaluating its effects is coherent and easy to follow.

Critical weaknesses
  1. Several implementation details need clarification. Appendix B provides the main target-mining equations, but does not fully specify the robust-normalization procedure, motion-support selection, or Gaussian-uncertainty estimation. The grounding head’s query count and decoder configuration should also be reported for reproducibility.

  2. The normalization ablation changes multiple components. The Gaussian CE comparison changes both target construction and loss scaling relative to normalized KL. Although Appendix E.4 provides useful qualitative analysis, a comparison using identical targets would better isolate the contribution of normalization.

  3. The Meta-World comparison would benefit from a matched continuation baseline. The paper reports 1,000 additional fine-tuning steps for S³-VLA, but does not report a corresponding action-only continuation from the same π₀.₅ checkpoint. This control would help isolate SGA’s contribution to the Meta-World improvement.

Questions for authors
  1. Could you specify how robust normalization, motion-support selection and Gaussian-uncertainty estimation are implemented, together with the grounding head’s query count and decoder configuration?

  2. Could you compare normalized and unnormalized KL using identical localization targets and appropriately tuned auxiliary-loss weights, while keeping the other training settings fixed?

  3. Could you report an action-only π₀.₅ baseline continued from the same checkpoint for 1,000 steps, using the same Meta-World data and optimization settings as S³-VLA?

1 comment

Dear Reviewer,

Thank you again for your careful review and constructive suggestions. We sincerely appreciate the time and effort you have devoted to evaluating our work!

As the discussion period draws to a close, we would be grateful to hear whether our rebuttal has addressed your concerns. Please do not hesitate to let us know if there is anything we can clarify:)

Best regards,

The Authors

Reviewer_fTfnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors show that the policy trained under one camera extrinsic experiences an action bias, due to which the policy fails under a different camera setting. For example, the leftward wrist camera shift causes the end-effector to be biased rightward. The authors claim that this is due to a lack of visual grounding of the robot embodiment. To induce the grounding capability of the policy, they introduce the Self-Grounding Alignment (SGA) loss, a small module that predicts the attention map of the localization target, i.e., the embodiment, and is jointly trained with the behavior cloning loss. They introduce Robot Visual Motion (RVM), which obtains the localization target by identifying the fast-moving region with optical flow, which serves as a heuristic location of the robot embodiment.

Their approach was evaluated on two simulated benchmarks (SimplerEnv and MetaWorld) and a real-world experiment, and further explores a setting with camera perturbation. The proposed method consistently improves on the baseline policy trained without SGA.

Critical strengths

The performance consistently improves on the base model with a significant margin for the main experiment. The experiments include diverse settings: SimplerEnv with two embodiments (WidowX and Google Robot), Meta-World MT50, and real-world Franka. The improvement is shown on both the original camera view and camera extrinsic perturbation. Also, the method is applied to two different VLA backbones (GR00T-N1.6 and $\pi$0.5), showing the method is not limited to a specific architecture.

It does not incur additional computation at deployment. The grounding head and localization target are only used in training, so inference is the same as the original VLA. No extra input or parameter is needed, which makes the method easy to apply to existing VLAs.

Critical weaknesses

The connection from the motivation to the method is weakly linked and the problem formulation discussed in this work seems ill-posed. The coordinate frame of the base with respect to the perception changes with the extrinsic. Thus, how to act upon a camera shift requires knowledge of the extrinsic. The authors drop the extrinsic and rather argue for implicitly recovering the extrinsic by visually grounding the robot embodiment. However, the perception needs to include the base link of the embodiment to infer the base frame of the current view. In neither SimplerEnv nor MetaWorld is the base link visible in the image.

RVM heuristically obtains the location of the embodiment. However, this can be accurately obtained from the known camera extrinsic and forward kinematics of the embodiment. The paper argues RVM is robust in localizing the target despite the heuristic approach with the attribution experiment (e.g., 82.0% vs. 0.3%), but does not account for the difference in embodiment and object size. By random chance, it is more likely to select the robot embodiment.

The choice of action space of the policy may have a significant impact. Intuitively, generating relative actions relies heavily on the visual grounding of the embodiment. This is unclear from the paper, and overall the writing feels unorganized, e.g., notation is not well explained and is overloaded (M).

Questions for authors
  • Are there related works that have the same formulation, i.e., removing the action bias from the camera extrinsic? How does your work compare to them?
  • Which action space is used?
  • Does the same method apply to the wrist viewpoint as well? How does RVM work with the wrist view?
  • Which dataset did you use for SimplerEnv?
  • In Section 3.2, to claim the weak visual grounding capability, why do you test on the shifted camera view?
1 comment

Dear Reviewer,

Thank you again for your careful review and constructive suggestions. We sincerely appreciate the time and effort you have devoted to evaluating our work!

As the discussion period draws to a close, we would be grateful to hear whether our rebuttal has addressed your concerns. Please do not hesitate to let us know if there is anything we can clarify:)

Best regards,

The Authors

Reviewer_M3LFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a method to improve the robustness of a VLA model when fine-tuned for robot arm tasks subject to different camera viewpoints. It is argued that in the general setting, the VLA will learn a shortcut to link the pixel location for the task goal, thus fail when deployed with a new camera viewpoint. It proposes a simple training-time enhancement, by forcing the VLA model to predict the location of the robot effectors during training. THis is done by mining the robot location from the fixed-camera videos by optical flow and frame difference, and then add a KL loss between the target and prediction from a shallow DETR-style decoder with M learned queries cross-attending to the final-layer visual tokens. Empirical results on real-robot success under camera shifts seems consistent.

Critical strengths

The paper introduces a training-only, calibration-free signal with significant real-robot gains under camera shifts.

Critical weaknesses
  • Conceptual novelty is marginal as similar ideas already exploited in some missing reference, and the paper does not compare to any of them
    • MAVRIC (2019) found robot control points from motion, also without calibration.
    • ECoT taught policies to predict the 2D gripper position.
    • Jiang et al. (2025) also found a camera shortcut and gave policies the camera position as input. Do You Know Where Your Camera Is? View-Invariant Policy Learning with Camera Conditioning, Jiang et al. (2025)
  • The simulated evidence does not isolate robustness from in-distribution gains Simulated "robustness" mostly tracks clean-view gains. On WidowX, S3-VLA keeps 44% of clean success under shift vs. 62% for the baseline (a 3-seed perturbed mean over a single-checkpoint clean value); only the real robot shows clearly better retention (55% vs. 37%).
  • Evaluation variance is high: the paper's per-task baselines diverge from NVIDIA's SimplerEnv in both directions (SimplerEnv, 100 vs. 200 episodes). The SimplerEnv averages (56.6/52.0) are lower than the paper's (60.1/71.7), but its Eggplant-in-basket score is 89.0% vs. the paper's 46.5%, exactly S3-VLA's number. This shows high test variance exactly where the largest gain appears.
Questions for authors
  • Your per-task baselines diverge from NVIDIA's published numbers in both directions (Eggplant-in-basket 46.5 vs. 89.0; Fractal Open-drawer 49.0 vs. 0.0). Is the protocol the same? Please report Table 1 and w/o L_g over ≥3 seeds with per-task CIs.
  • Can you add at least one matched view-robustness baseline and the LIBERO-Plus camera dimension? Candidates are camera-pose, crop or novel-view augmentation, or camera-conditioned or camera-centric actions.
1 comment

Dear Reviewer,

Thank you again for your careful review and constructive suggestions. We sincerely appreciate the time and effort you have devoted to evaluating our work!

As the discussion period draws to a close, we would be grateful to hear whether our rebuttal has addressed your concerns. Please do not hesitate to let us know if there is anything we can clarify:)

Best regards,

The Authors

Comments on the paper

2 comments
Area_Chair_DseuOpenReview ↗

Reminder

Dear reviewers,

If you haven’t yet responded to the authors’ rebuttal, please review their response and any new results, then leave a brief comment on OpenReview visible to the authors. Please indicate which concerns were addressed and which remain, even if your rating is unchanged. If the rebuttal changes your assessment, please update your review and rating as appropriate.

Thank you for your time and for helping wrap up the discussion.

AuthorsOpenReview ↗

Summary of Rebuttal

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair for guiding the review process and all three reviewers for the time and care they devoted to evaluating our work. To aid in the final assessment, we respectfully summarize the contributions of our paper and the revisions prompted by the discussion.


Contributions.

We identify visual self-grounding as a missing capability in current VLA policies, which often struggle to localize the robot itself in visual observations. Camera perturbations expose an image-space shortcut in which predicted trajectories drift with pixel displacements. To our knowledge, S3S^3-VLA is the first to make visual self-grounding an explicit objective of VLA representation learning. The motion of the robot in uncalibrated demonstrations reveals where its body appears in the image, providing a natural source of self-supervision.


Key concerns and revisions

We have addressed the concerns raised by each reviewer in our individual responses. Below, we summarize the additional experiments and the main changes to the revised manuscript.

  • comparison with camera-conditioned policies (Reviewers M3LF and fTfn). Camera-conditioned policies use extrinsics to handle viewpoint changes. We use camera shifts to diagnose weak visual self-grounding. Our contribution is a general representation learning method that develops this missing capability across existing VLA backbones without changing their inference architectures. We added LIBERO-Plus as a new benchmark and Know Your Camera (KYC) as a new camera-conditioned baseline. S3S^3-VLA is orthogonal to camera conditioning. Our LIBERO-Plus experiments show that combining S3S^3-VLA with KYC yields higher mean success under camera shifts than either method alone.
  • Grounding without camera calibration (Reviewer fTfn). The visible robot body provides a reference for the target even when the base is out of view. Demonstrations supply the controller-frame actions. We tested whether the motion targets simply mark where the robot usually appears. On the same images, points from the matching frame landed on the robot 79.5% of the time, compared with 29.9% for points from other episodes of the same task.
  • Matched baselines and evaluation reliability (Reviewers USSx and M3LF). On LIBERO-Plus, self-grounding improved shifted-view success over action-only training despite lower clean-view success. We added a matched Meta-World baseline in Table 2. With the same checkpoint, data, and training steps, success fell with action-only training and rose with self-grounding. The SimplerEnv gain persisted across three training seeds. With identical targets and separately tuned loss weights, our ablation also favored normalized KL.

Finally, we sincerely thank the AC and reviewers again for their time and consideration of our revised manuscript. We are grateful for the opportunity to revise our work in light of these comments.

Yours sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.