acceptodds
Withdrawn from ICLR 2027

3DVLA: Enhancing Vision-Language-Action Models via 3D Spatial and Instance Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-Language-Action models have achieved remarkable progress in robotic manipulation, yet they suffer from a critical limitation: a lack of 3D scene understanding. This deficiency manifests as three intertwined challenges: weak extraction of 3D spatial positions without enforcing multi-view consistency, inadequate 3D instance understanding, and fragile reasoning under occlusion. Although mature 3D perception methods exist, their direct integration into VLA pipelines is hindered by architectural incompatibility and by heavy reliance on costly instance-level annotations. To address the above challenges, we propose 3DVLA, a plug-and-play framework that injects robust 3D reasoning into pretrained VLAs without requiring extra manual labels or discarding VLM priors. Specifically, 3DVLA tackles the three challenges through: (1) pervasive 3D feature encoding with explicit multi-view consistency constraints across all modalities and a Spatially-Conditioned Geometry Aggregation method, (2) an instance estimation module with high-level instance tokens for 3D instance awareness, and (3) a masked self-supervised 3D encoding branch that retains its predictor for visual token completion to handle occlusions. We integrate 3DVLA with multiple VLA baselines and evaluate on LIBERO-Plus and RoboTwin 2.0. Results show consistent and significant gains in manipulation performance, validating both the effectiveness and plug-and-play compatibility of our approach.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotics, VLA

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uubEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes 3DVLA, a framework for augmenting pretrained vision-language-action models with explicit 3D information. Its components include multi-view spatial fusion, object-centric instance tokens, a coordinate-conditioned feature predictor trained with masked self-supervision, and uncertainty-guided geometric aggregation. The framework uses pseudo-labels from a frozen perception model and completion coordinates from a pretrained point-cloud completion network. Evaluation covers LIBERO-Plus, RoboTwin 2.0, and three real-world manipulation tasks, with integrations into multiple VLA backbones. The reported results, including the five-run evaluations in the appendix, show improvements over the corresponding base policies.

Critical strengths
  1. The framework provides a potentially useful way to incorporate geometry into pretrained VLAs. Preserving the frozen VLM backbone while introducing instance and completion tokens is practically appealing. Integration with multiple policy backbones provides evidence that the approach is not restricted to a single architecture.

  2. The aggregate performance improvements have meaningful empirical support. Tables 9–10 report five-run results: LIBERO-Plus improves from 83.9 ± 1.2 to 86.2 ± 0.8, while X-VLA on RoboTwin improves from 69.7 ± 0.9 to 80.6 ± 1.2 on Easy and from 39.0 ± 3.1 to 48.2 ± 2.4 on Hard. The real-world evaluation in Table 3 provides additional evidence of practical utility. These results make the overall system worth investigating.

Critical weaknesses
  1. The comparisons do not sufficiently establish the incremental value of the proposed architecture over simpler geometric augmentation. The method uses depth, camera calibration, pretrained perception supervision, and a pretrained completion model. However, the experimental description does not clearly establish matched input access and post-training budgets for the corresponding baselines. Appendix A specifies the proposed two-stage training schedule, but does not adequately explain the matching baseline protocol. Moreover, although Section 2 discusses SpatialVLA, PointVLA, and GeoVLA, the experiments include neither a directly relevant 3D-aware VLA comparison nor a same-backbone alternative using simpler 3D positional encoding or point-feature injection. DP3 is included, but does not control for the pretrained VLM backbone. This leaves unresolved how much of the improvement requires the proposed instance-and-completion design. Table 8 further exposes an input-protocol ambiguity: perturbing depth maps substantially degrades the π0/π0.5 baselines, yet their use of depth is not explained.

  2. The central geometric mechanisms are insufficiently specified and isolated by the evidence. In Section 3.3.1, the pretrained PointAttn network already supplies the completed geometry used to construct PcompP_{\mathrm{comp}}; the proposed predictor generates features conditioned on these coordinates. The explicitly defined cosine and variance losses in Eqs. (12)–(13) concern masked observable tokens, leaving the learning signal for wholly unobserved completion locations unclear. The A-to-B ablations jointly introduce self-supervision, completion queries, and predictor-generated features, so they do not establish the specific benefit of retaining the masked predictor at inference. Separately, Eq. (5) assumes globally associated pseudo-target identities kk and valid-view sets VkV_k, without explaining how these are constructed from view-specific 2D detections. Unprojecting 2D centers does not generally produce a shared physical object centroid across views. These omissions concern the mechanisms underlying the paper’s main claims, rather than peripheral implementation details.

Questions for authors
  1. Clarification of the comparison protocol: What inputs, pretrained checkpoints, trainable modules, and post-training budgets are used for each corresponding baseline? In Table 8, why does perturbing only depth maps affect the π0/π0.5 baselines? Do these baselines consume depth, or does the perturbation also affect other observations?

  2. Evidence needed to assess the contribution: Under matched geometric inputs and training budgets, how does the method compare with a simpler 3D positional-encoding or point-feature-injection baseline? For the completion branch, can the authors distinguish the benefit of the external completion coordinates, masked self-supervision, and retaining the predictor at inference?

  3. Clarification of global instance supervision: How are independent per-view pseudo-detections grouped into the global targets kk and valid-view sets VkV_k used in Eq. (5)? How are inconsistent depth-derived centers, missing detections, and nearby same-class instances handled?

  4. Clarification of completion learning: Which losses provide supervision to Z^comp\hat{Z}_{\mathrm{comp}}, and how are target view identifiers assigned to completion coordinates? If completion features rely on generalization from masked visible locations and downstream action supervision, what evidence directly evaluates this generalization?

Reviewer_XbfkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims to improve the 3D spatial understanding of 2D VLA models. The authors identify several limitations of existing VLA models: difficulty extracting 3D spatial positions, understanding 3D objects, and handling occlusion. To address these limitations, they propose 3DVLA, a plug-and-play method that requires no costly manual annotation and comprises the following modules: 1. MultiView Spatial Fusion and Spatially-Conditioned Geometry Aggregation, 2. Object-Centric 3D Instance Module, 3. A 3D encoding branch trained via masked self-supervision. The authors integrate 3DVLA into multiple VLA baselines and evaluate it on LIBERO and RoboTwin, reporting substantial performance gains and providing fairly comprehensive module-level ablations.

Critical strengths
  1. The paper has a clear motivation and addresses an important problem: improving the 3D spatial understanding of VLA models. The proposed method directly addresses the limitations of existing approaches and supports plug-and-play integration.
  2. The experiments are fairly comprehensive. Results on LIBERO and RoboTwin provide convincing evidence that 3DVLA improves the performance of the baseline models.
Critical weaknesses
  1. The contributions of the paper are primarily engineering-oriented. Although the authors introduce several modules, they do not clearly explain the motivations behind their designs or the trade-offs involved.
  2. 3DVLA relies on additional pretrained perception models, whereas many competing methods primarily use 2D RGB inputs. It is therefore difficult to determine whether the performance gains arise from the 3DVLA architecture itself or from the additional computation and inputs. The authors should provide ablation studies using different perception models to demonstrate the robustness of the proposed method across models, rather than attributing the gains solely to a specific perception model.
  3. The baselines considered in the experiments are mainly 2D VLA models. The authors should also compare their method with recent VLA baselines that similarly incorporate 3D information, such as 3DVLA (the work with the same name, arXiv:2403.09631), DepthVLA, SpatialVLA, and GeoVLA.
Questions for authors

In addition to the weaknesses discussed above, I have the following questions:

  1. Although 3DVLA is plug-and-play, its overall pipeline is highly complex. The authors do not discuss the overhead introduced by the additional models and computation, such as inference latency and GPU memory usage.

  2. Although 3DVLA does not require costly manual annotations, it relies on pretrained models such as VL-SAM-v2. The authors should report the inference costs of these models.

  3. Some text in Figures 3 and 5 is too small to read. The authors should consider improving the readability of the figures.

  4. The results in Tables 1 and 4 indicate a substantial decrease in the Language metric after introducing 3DVLA, but the authors do not discuss this issue.

Reviewer_HzWfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces 3DVLA, a plug-in framework designed to enhance the 3D spatial and instance-level understanding capabilities of pre-trained VLA models. The approach incorporates multi-view RGB-D feature fusion, 3D instance extraction, feature completion for occluded regions, and geometry-aware aggregation to facilitate action prediction. The authors integrate this method into three different VLA backbones, demonstrating performance improvements across LIBERO-Plus, RoboTwin 2.0, and real-world robotic tasks. The study also includes ablation studies for individual modules and experiments conducted with multiple random seeds.

Critical strengths
  1. This paper studies the shortcomings of VLA in three-dimensional space understanding, cross-view instance consistency and occlusion reasoning. It has certain practical value to introduce three-dimensional information while retaining the representation capabilities of pre-trained VLM without requiring additional manual instance labeling.
  2. The method achieves overall performance improvement on three VLA Backbones, two simulation benchmarks and real robot platforms. Among them, the success rate of π0 on RoboTwin 2.0 Hard increased from 16.3% to 23.2%, and X-VLA increased from 39.0% to 48.2%. Multiple random seed experiments further support the reliability of the overall performance improvement.
Critical weaknesses
  1. The contribution of depth information to performance improvement is not yet clear. 3DVLA uses a three-dimensional representation built from depth information, but it is not clear whether the compared VLA baseline obtains equivalent depth information. The actual contribution of the proposed architecture is difficult to determine individually due to the lack of comparisons with the same VLA model augmented with simple depth encoding or 3D positional encoding.
  2. Lack of adequate comparison with closely related 3D VLA methods. While the paper demonstrates performance improvements over multiple VLA baselines and provides ablation experiments, controlled comparisons with related 3D VLA methods such as SpatialVLA, PointVLA, and GeoVLA are missing. These experiments help to clarify the incremental contribution of the proposed method relative to existing three-dimensional representation and fusion methods.
Questions for authors
  1. Comparison of depth input and baseline. Do the VLA baselines in our experiments use raw RGB input, or do they also receive 3D geometric features generated from depth information? If Depth is used, what is its source, encoding and input method? Does it use the same depth information as 3DVLA?
  2. Following Q1, In Table 8, when Depth Noise increases from 0 to 0.5, the success rate of the π0.5 baseline drops from 84.2% to 56.1%, and the π0 baseline drops from 46.4% to 23.2%. If the baseline does not use Depth, why is its performance significantly affected by depth noise?
  3. Backbone selection on RoboTwin 2.0. Why did the authors use π0.5 in LIBERO-Plus and real robot experiments, but choose π0 and X-VLA on RoboTwin 2.0? Considering that π0.5 has public RoboTwin 2.0 evaluation results, can you explain this choice? If conditions permit, can you provide experimental results for π0.5 and π0.5 + 3DVLA under the same training and evaluation protocols?
  4. Comparison with simple 3D fusion baselines and existing methods. Furthermore, if conditions permit, the authors are expected to provide representative controlled comparisons with related 3D VLA methods such as SpatialVLA, PointVLA or GeoVLA to further illustrate the additional contribution of 3DVLA.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.