acceptodds
Withdrawn from ICLR 2027

GRAM-VLA:Giving Vision-Language-Action Models a Sense of 3D Space

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Embodied intelligence is increasingly moving toward more complex manipulation tasks. Compared with single-arm tasks, dual-arm tasks require coordination of the two arms' spatial positions and temporal order. This places greater demands on the model's joint vision-language understanding and spatial perception. Most Vision-Language-Action (VLA) models excel at dual-arm tasks by leveraging large-scale 2D vision-language pre-training, but their reliance on RGB images limits spatial reasoning critical for real-world interaction. Adding 3D information can improve spatial perception. However, without extra large-scale 2D–3D data pre-training, it remains challenging to fully use 3D information for action generation while remaining consistent with pretrained 2D semantics. To address this challenge, we propose GRAM-VLA, a 3D-enhanced dual-arm VLA model without additional large-scale 2D–3D pretraining. GRAM-VLA keeps the pretrained vision-language backbone (VLM) frozen and adds a lightweight depth encoder to form an independent RGB–Depth dual-branch architecture. It establishes token-level spatiotemporal RGB–Depth alignment and alternately injects language, RGB, and depth conditions into the action generation backbone, allowing depth information to fully participate in action generation. Furthermore, we use four-modal GRAM alignment to mitigate multimodal semantic shift caused by the newly introduced depth information. On eight real-robot tasks aligned with the RoboTwin 2.0 simulation protocol, GRAM-VLA achieves an average success rate of 53.6%, outperforming both 2D- and 3D-based models. In controlled experiments, GRAM-VLA outperforms baseline-RDT by 40.0 and 72.5 percentage points under appearance and height shifts, respectively, and by 22.5 percentage points on long-horizon task completion.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Dual-arm VLA models, Independent dual-branch encoding, Global representation alignment, Diffusion-based action generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_j1UEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GRAM-VLA, an RDT-based model for dual-arm manipulation using depth observations. It introduces TCPE to inject language, RGB, and depth conditions into different layers, and a Gramian-volume loss to encourage multimodal alignment during training. The full model has a higher average success rate than the reported baselines across eight real-robot tasks and is more robust to appearance and object-height changes.

Critical strengths
  1. The full system improves performance on real-robot tasks. In Table 1, its average success rate is 53.6%, compared with 30.0% for RDT and 46.3% for DP3. This suggests practical value, although the source of the gain is unclear.
  2. The improvements under appearance and height changes in Table 2 are substantial. These are relevant robustness challenges for manipulation policies and, in my view, the paper’s strongest results.
Critical weaknesses
  1. The paper does not isolate its claimed contributions. It adds depth, a depth encoder, TCPE, and the GRAM loss together, but does not compare against an RDT model with the same depth observations or ablate TCPE and the GRAM loss separately. Depth alone could explain much of the reported gain, especially under height changes. Because alternating condition injection and Gramian-volume learning build on prior work, the paper needs to show what its proposed components add.
  2. The experiments do not clearly support the dual-arm and long-horizon claims. In Figure 4, the arms mainly act in sequence, and the full-task completion rate of pretrained GRAM-VLA is 15%. The higher stage score shows better progress on this task, but does not establish improved two-arm coordination or long-horizon planning specifically.
Questions for authors
  1. Additional evidence that could change my assessment: Can the authors provide a matched RDT + depth baseline and separate ablations for TCPE and the GRAM loss, using the same observations and training setup? This would clarify whether the proposed components add value beyond depth.
  2. Clarification and possible additional evidence: Which experiment isolates a benefit for two-arm coordination rather than a general benefit from depth in spatial manipulation? If the current paper has no such result, is a targeted evaluation available?
Reviewer_5DCCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes GRAM-VLA, an extension of RDT-1B that incorporates depth observations into dual-arm manipulation without additional large-scale 2D–3D pretraining. It aims to improve geometric understanding while retaining pretrained visual and language capabilities. Its main contributions are an independent lightweight depth encoder alongside frozen vision and language encoders; matching RGB–depth patch grids and cycle-aware positional encoding for alternating language, RGB, and depth conditioning; and a training-onlyGramian-volume contrastive objective intended to align language, RGB, depth, and expert-action representations. Extensive experiments on 8 manipulation tasks and long-horizon stacking experiments show significant performance improvements.

Critical strengths
  1. Meaningful real-robot evidence of improved robustness. GRAM-VLA is evaluated on eight manipulation tasks with 100 rollouts per task, achieving the best results on six. The controlled height-shift result—72.5% success versus RDT’s 0%—is particularly compelling evidence of practical benefit from geometric observations. This strengthens the paper’s empirical relevance to manipulation under changing spatial conditions.
  2. A practical approach to incorporating depth into pretrained policies. The architecture retains frozen vision and language encoders, adds an independent depth branch, and uses the alignment objective only during training, avoiding additional large-scale 2D–3D pretraining. This makes the approach relevant to robotics settings where demonstrations and paired multimodal pretraining data are limited.
Critical weaknesses
  1. The experiments do not identify the proposed contributions. Comparisons between RDT and the complete GRAM-VLA system simultaneously change the sensory input, architecture, and training objective. There are no component ablations establishing the benefits of GRAM alignment or TCPE over simply adding depth. A matched RDT-plus-depth baseline, removal of individual components, and comparison with conventional contrastive alignment are needed. This is consequential because the reported improvements do not establish the effectiveness of the paper’s central methodological contributions.
  2. The interpretation of the alignment objective is insufficiently justified. A small Gramian volume indicates linear dependence, not necessarily semantic alignment across all four modalities. Matching language and depth alone can produce zero positive volumes, while constant, uninformative RGB and action embeddings can still permit separation from negative combinations. The contrastive formulation therefore does not exclude solutions that ignore some modalities. Without diagnostic experiments or additional constraints, the central claim that GRAM promotes joint four-modal semantic consistency remains unsupported.
Questions for authors
  1. Evidence for the central claims: Can you provide matched ablations comparing RDT + depth, the full model without GRAM, and the full model without TCPE? Were initialization, training budget, and gripper-loss weighting identical? These results would establish whether the proposed mechanisms contribute beyond adding depth observations.
  2. Methodological clarification: How do you justify interpreting low Gramian volume as four-modal semantic alignment when partial modality matching can also minimize the objective? Do you have diagnostics showing that RGB and action representations contribute meaningfully, rather than the objective being satisfied primarily through language–depth matching?
  3. Training-protocol clarification: Are the controlled experiments trained separately per task, and how diverse are their language instructions? When multiple samples share an identical instruction, are they treated as negatives under Equation (10)? If so, how does the objective provide meaningful instance-level supervision, particularly in single-task batches?
  4. Additional baseline comparisons: Can you provide comparisons with closely related 3D-enhanced VLA methods, using the same task demonstrations and evaluation protocol with comparable fine-tuning budgets? This would help establish whether GRAM-VLA offers advantages beyond the existing RGB-only and DP3 baselines.
  5. Reproducibility clarification: What materials are currently available for reproducing the results, and do you plan to release the implementation, checkpoints, and training/evaluation configurations? In particular, can you provide the depth preprocessing details, training hyperparameters, and task initialization and success criteria during the review period?
Reviewer_bW3FOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

GRAM-VLA extends RDT-1B for bimanual manipulation by adding a lightweight depth encoder alongside frozen vision and language encoders. It combines RGB–depth patch correspondence, cyclic injection of language/RGB/depth conditions with TCPE, and a GRAM-based contrastive objective over language, RGB, depth, and expert-action representations. The approach aims to incorporate geometry without additional large-scale 2D–3D pretraining while retaining pretrained semantic capabilities. The paper evaluates real-robot task adaptations, generalization under appearance and height changes, and multi-stage stacking. Its controlled experiments report substantial improvements over RDT under environmental shifts and in average task completion.

Critical strengths
  • Practical integration of geometry. The lightweight depth branch reuses pretrained vision and language encoders, providing a practical approach to incorporating geometric observations. The auxiliary alignment objective is used only during training.

  • Promising results in physical manipulation. The paper includes closed-loop bimanual robot experiments. Table 2 reports success rates of 65.0% versus 25.0% under appearance changes and 72.5% versus 0.0% under height changes, with comparable standard-condition performance. Figure 4 also reports improved average completion scores on multi-stage stacking.

Critical weaknesses

1. Major concern: Insufficient disclosure of the provenance and comparability of Table 1. Section 4.1 describes eight tasks reproduced on a real robot and interprets Table 1 as evidence of superior performance (p. 8, lines 404–417). However, all 32 baseline results for DP, ACT, RDT, and DP3 exactly match the Easy simulation results in RoboTwin 2.0 (arXiv:2506.18088v2, Appendix K, Table 10, p. 23). For example, both tables report 24/31/42/60 for Pick Dual Bottles and 61/82/76/83 for Stack Bowls Two, in DP/ACT/RDT/DP3 order. The baseline columns are not clearly identified as new real-robot evaluations or results cited from simulation. If they are simulation results compared with real-robot GRAM-VLA results, aligned task definitions alone do not establish comparability, and the resulting ranking cannot substantiate a controlled performance advantage.

2. Major concern: Repeated errors in reference attribution. Incorrect author names were identified in 13 of the 48 bibliography entries, including the adopted alignment method and the principal baseline. Representative examples are:

ReferenceNames in the submissionCorrect names
GRAM, p. 10Giovanni Cicchetti; Lorenzo SigilloGiordano Cicchetti; Luigi Sigillo
RDT-1B, p. 11Shizhe Liu; Lingxiao Wu; Bing LiSongming Liu; Lingxuan Wu; Bangguo Li
SpatialVLA, p. 11Da Qu; Hao Song; Qiyu ChenDelin Qu; Haoming Song; Qizhi Chen

There is also an unresolved citation, “(Xu et al., 2023; ?),” on p. 6, line 291. These repeated errors raise a substantial concern about literature verification and attribution. The cited works exist; the evidence does not establish how the errors arose.

3. Limited novelty and insufficient validation of the incremental contribution. Alternating conditioning is inherited from RDT, separate geometry branches have precedents such as PointVLA, and the alignment objective adopts GRAM. The three-modality extension and TCPE provide identifiable differences, but their value beyond a straightforward depth-augmented RDT is not established. Table 2 and Figure 4 compare complete models and initialization settings without isolating TCPE or GRAM alignment. This leaves the contribution of the proposed mechanisms, rather than the additional depth input, unclear.

Questions for authors
  1. For each Table 1 column, were the results newly obtained or cited, and were training and evaluation conducted in simulation or on physical hardware? Please explain the exact correspondence of all 32 baseline values. If the evaluation domains differ, please explicitly label the sources, provide matched comparisons, and revise the associated claims.

  2. Please explain the repeated author-name errors, verify and correct the bibliography, and resolve the missing citation.

  3. What do TCPE and GRAM alignment each contribute beyond adding depth to RDT? Please provide component ablations with matched architecture, initialization, data, and training budget, and clarify the substantive differences from the closest geometry-enhanced VLA methods.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.