acceptodds
Under review as a conference paper at ICLR 2027

VEGA: Visual Encoder Grounding Alignment for Spatially-Aware Vision-Language-Action Models

Abstract

Precise spatial reasoning is fundamental to robotic manipulation, yet the visual backbones of current Vision-Language-Action (VLA) models are predominantly pretrained on 2D images without explicit 3D geometric supervision, which can limit their ability to capture fine-grained spatial relationships. Prior work partially addresses this limitation by aligning LLM-level visual tokens with features from 3D-aware foundation models. However, our probes show more localized attributions around task-relevant regions at the visual encoder output, along with larger action-prediction errors under encoder-level masking at higher tested masking ratios than under masking at the probed LLM layer. These observations motivate direct geometric supervision of the visual representations supplied to downstream action prediction. We propose VEGA (Visual Encoder Grounding Alignment), a lightweight framework that grounds spatial knowledge directly at the visual encoder output, avoiding the need to select an internal LLM layer for alignment. VEGA uses frozen spatial teachers obtained by fine-tuning visual backbones with multi-view-consistent feature targets rendered from 3D Gaussian representations. A lightweight projector aligns encoder outputs with teacher features through a patch-level cosine distance loss, jointly optimized with the action prediction objective. The teacher and alignment projector are used only during training, leaving the policy architecture and inference cost unchanged. Extensive experiments on RoboTwin 2.0 and real-world manipulation tasks demonstrate that VEGA achieves higher average success rates than existing implicit spatial grounding baselines, with improvements on both OpenVLA-OFT and highlighting its applicability across VLA architectures.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision Language Action, Robotics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Z6mLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VEGA, a framework for improving spatial representations in vision-language-action models by aligning features at the visual encoder output. VEGA uses frozen teachers obtained by adapting visual backbones with FiT3D, which transfers multi-view information through features rendered from 3D Gaussian representations. During policy training, an auxiliary projector aligns visual encoder features with the corresponding teacher features using a patchwise cosine-distance loss, optimized alongside the action objective. The teacher and projector are removed at inference.

The method is evaluated on OpenVLA-OFT and π₀.₅ using six RoboTwin 2.0 tasks under clean and randomized conditions, four main real-world tasks, and an additional domino task. The paper reports improved average success rates, studies how teacher choice interacts with alignment location, and provides depth probing as well as training- and data-efficiency analyses.

Critical strengths

A useful controlled analysis of teacher choice and alignment location. Table 2 evaluates three teachers at both the visual encoder output and LLM layer 24, which is more informative than comparing only the final VEGA configuration against Spatial Forcing. At the encoder output, DINOv2-FiT3D raises the Easy/Hard averages from 59.3%/25.8% with the original DINOv2 teacher to 67.5%/30.7%, whereas moving VGGT supervision to the encoder reduces performance. These results reveal a consequential interaction between teacher choice and supervision location and support the specific configuration VEGA uses.

A practical approach with evidence across two policy architectures. VEGA preserves the inference architecture and requires no teacher computation at deployment. The paper evaluates both OpenVLA-OFT and π₀.₅, includes randomized simulation conditions, and provides real-robot comparisons that use the same demonstrations across methods. The π₀.₅ baseline also uses the same vision adaptation scheme, which helps isolate the effect of the alignment objective. Together, these experiments offer encouraging evidence that the approach can benefit more than one VLA architecture.

Critical weaknesses
  1. The gains over the strongest matched baseline are not yet established across independent training runs.

Section 4.1 states that each configuration uses one training seed, and the three evaluation seeds in Table 5 evaluate a single fixed checkpoint per configuration, so they do not measure training variability. In these evaluations, VEGA exceeds Spatial Forcing on Easy/Hard by 2.8/4.5 points on OpenVLA-OFT and 1.2/3.3 points on π₀.₅. These gains are encouraging, but their robustness to training randomness is unknown.

The real-world comparison also has limited resolution: with 20 trials per task, the five-point average advantage over Spatial Forcing amounts to four additional successes in 80 trials per architecture. Independent training replications with uncertainty estimates would materially strengthen the central empirical contribution.

  1. The evidence for the geometric rationale is limited relative to its role in the contribution.

Because the contribution mainly concerns where to apply an existing alignment objective and which teacher to use, explaining and validating those choices is especially important. Yet the attribution and masking analyses in Figure 2 rest on one observation, and greater sensitivity to masking upstream tokens does not show that this location is better suited to geometric supervision, since information may already have spread across tokens by the probed LLM layer.

The depth analysis is complementary, but Appendix A.5 states that all probe episodes were used in policy fine-tuning, and the analysis also uses a separately trained, relatively expressive decoder per task and averages error over all valid pixels. These results therefore offer limited evidence of transferable geometry in action-relevant regions. Table 2 supports the selected teacher-site pairing, but the broader account of why it improves spatial competence remains underdeveloped.

Questions for authors

Could you report VEGA versus Spatial Forcing across independent policy-training seeds on both architectures? For the real-world results, were initial configurations matched across methods, and can you provide uncertainty estimates for the success-rate differences?

Does the depth-decoding advantage hold on trajectories or scenes unseen during fine-tuning, especially in manipulated-object and gripper regions? This would show how far the representation gains extend beyond the training configurations.

8 comments
AuthorsOpenReview ↗

Response to Reviewer Z6mL (Part 1)

Dear Reviewer Z6mL:

Thank you for recognizing VEGA’s practical value and the controlled teacher–supervision-site analysis. We appreciate your suggestions on training variability and geometric generalization. We will address your concerns below.


[W1 & Q1.] Independent Training Runs and Real-World Reliability

To assess training variability, we now report three independent training seeds per method: {0, 1, 2} for OpenVLA-OFT and {42, 43, 44} for π₀.₅.

ArchitectureSettingSF: Mean ± SD (%)VEGA: Mean ± SD (%)Difference (pp)
OpenVLA-OFTEasy64.33 ± 0.4467.17 ± 0.44+2.83
OpenVLA-OFTHard28.06 ± 0.6930.83 ± 0.17+2.78
π₀.₅Easy54.94 ± 0.2557.44 ± 0.42+2.50
π₀.₅Hard23.39 ± 0.5426.22 ± 0.35+2.83

*Mean and sample SD are computed across three independent training runs, using each run’s six-task average. *

VEGA maintains higher six-task average success rates across all three training runs in both architectures and settings, providing additional evidence that the observed advantage persists under training randomness.

For real-world evaluation, we verified that SF and VEGA used matched initial conditions for each of the 20 trials per task. We report per-task success counts and rates below. We estimate uncertainty in the four-task mean difference using a task-stratified paired percentile bootstrap with 100,000 resamples, preserving trial pairs within each task and weighting the four tasks equally.

ArchitectureMethodClose LaptopHandover CucumberDual CarrotsDual FlowersMean SR (%)VEGA−SF [95% CI], pp
OpenVLA-OFTSF14/20 (70%)14/20 (70%)12/20 (60%)4/20 (20%)55.00—
OpenVLA-OFTVEGA16/20 (80%)15/20 (75%)12/20 (60%)5/20 (25%)60.00+5.00 [−3.75, 13.75]
π₀.₅SF17/20 (85%)15/20 (75%)14/20 (70%)5/20 (25%)63.75—
π₀.₅VEGA18/20 (90%)16/20 (80%)14/20 (70%)7/20 (35%)68.75+5.00 [−1.25, 12.50]

Each task entry reports successes out of 20 trials, with the success rate in parentheses. Mean SR is the unweighted four-task average. Differences compare VEGA with SF within the same architecture; confidence intervals quantify evaluation uncertainty for the evaluated checkpoints.

These matched evaluations show an observed average improvement of 5.0 percentage points on both architectures, complementing the simulation results.


[W2.] Evidence for the Alignment Location

We thank the reviewer for highlighting how information can propagate across tokens. Figure 2 motivates testing encoder-output alignment. In the controlled comparison in Table 2, with DINOv2-FiT3D held fixed on OpenVLA-OFT, aligning at the visual encoder output improves Easy and Hard success rates by 8.3 and 10.7 percentage points, respectively, over alignment at LLM layer 24. VGGT favors layer 24.

On π₀.₅, we compare three teachers at the encoder output and LLM layer 12, the site used by Spatial Forcing, under matched protocols. The same pattern holds: SigLIP-FiT3D favors the encoder output by 7.0/4.2 percentage points on Easy/Hard, and VGGT favors layer 12 by 3.2/3.0 points.

Mean success rates across six tasks on π₀.₅.

SettingVGGT Enc.VGGT L12SigLIP Enc.SigLIP L12SigLIP-FiT3D Enc.SigLIP-FiT3D L12
Easy0.5200.5520.5220.4980.5750.505
Hard0.2100.2400.2350.2120.2650.223
AuthorsOpenReview ↗

Response to Reviewer Z6mL (Part 2)

[W2 & Q2.] Depth Probing Beyond Policy-Training Data

We evaluate 120 new expert trajectories (20 per task) using simulator seeds disjoint from policy fine-tuning. These trajectories are also held out from probe training and validation. All methods receive identical three-camera observations with policies, existing task-specific probes, and depth normalization frozen. Within each architecture, baseline and VEGA use matched probe architectures, training, and checkpoint selection. We compute per-trajectory AbsRel against simulator depth for the full image and visible object and gripper regions, then average across trajectories and equally across tasks.

Entries report Baseline → VEGA (relative change); lower is better.

OpenVLA-OFT

TaskFull ImageObject ROIGripper ROI
move_playingcard_away0.01062 → 0.00962 (↓9.42%)0.01732 → 0.01581 (↓8.72%)0.05594 → 0.05730 (↑2.43%)
turn_switch0.02941 → 0.02081 (↓29.24%)0.09286 → 0.06484 (↓30.17%)0.06650 → 0.06574 (↓1.14%)
click_bell0.00900 → 0.00858 (↓4.67%)0.02117 → 0.02002 (↓5.43%)0.06572 → 0.06143 (↓6.53%)
beat_block_hammer0.01592 → 0.01381 (↓13.25%)0.07851 → 0.07645 (↓2.62%)0.06317 → 0.06484 (↑2.64%)
lift_pot0.02177 → 0.02034 (↓6.57%)0.03765 → 0.03609 (↓4.14%)0.05646 → 0.05319 (↓5.79%)
place_shoe0.02283 → 0.02030 (↓11.08%)0.05441 → 0.05447 (↑0.11%)0.05548 → 0.05466 (↓1.48%)
Mean0.01826 → 0.01558 (↓14.69%)0.05032 → 0.04461 (↓11.34%)0.06055 → 0.05953 (↓1.68%)

π₀.₅

TaskFull ImageObject ROIGripper ROI
move_playingcard_away0.00987 → 0.00965 (↓2.23%)0.01623 → 0.01612 (↓0.68%)0.05483 → 0.05514 (↑0.57%)
turn_switch0.01880 → 0.01908 (↑1.49%)0.06000 → 0.05634 (↓6.10%)0.06247 → 0.06308 (↑0.98%)
click_bell0.00993 → 0.00936 (↓5.74%)0.02915 → 0.02666 (↓8.54%)0.06778 → 0.06481 (↓4.38%)
beat_block_hammer0.01361 → 0.01307 (↓3.97%)0.07930 → 0.07574 (↓4.49%)0.05988 → 0.05856 (↓2.20%)
lift_pot0.02124 → 0.02128 (↑0.19%)0.03746 → 0.03734 (↓0.32%)0.05486 → 0.05353 (↓2.42%)
place_shoe0.02360 → 0.02179 (↓7.67%)0.05119 → 0.04855 (↓5.16%)0.05681 → 0.05251 (↓7.57%)
Mean0.01618 → 0.01571 (↓2.91%)0.04556 → 0.04346 (↓4.60%)0.05944 → 0.05794 (↓2.52%)

On these unseen trajectories, VEGA reduces mean object/gripper ROI error by 11.34%/1.68% on OpenVLA-OFT and 4.60%/2.52% on π₀.₅, providing evidence that its depth-decoding gains extend beyond policy-training data to action-relevant regions.


[Ethics.] Clarification of the Ethics-Review Flag

We would appreciate clarification of the specific concern underlying the ethics-review flag, as no corresponding concern is described in the review. We are happy to provide further details and address it directly.

AuthorsOpenReview ↗

Follow-up on Our Response to Your Review

Dear Reviewer Z6mL,

We hope you are doing well. We wanted to follow up on our rebuttal and ask whether you have had a chance to review the new results addressing your comments.

  1. Training reliability and real-world evaluation. VEGA achieves higher average simulation success rates than Spatial Forcing across all three independent training seeds on both architectures. For the real-world comparison, we verified that initial layouts were matched and reported paired 95% confidence intervals for the success-rate differences.
  2. Choice of alignment location. Our new π₀.₅ ablation shows the same pattern as OpenVLA-OFT. FiT3D-adapted teachers perform better with alignment at the visual encoder output, and VGGT performs better at the tested LLM layer.
  3. Depth probing on unseen trajectories. Using frozen probes on 120 new expert trajectories, VEGA reduces mean depth error in both the manipulated-object and gripper regions on both architectures.

We would appreciate hearing whether these results address your concerns. We are happy to clarify any remaining points.

Best regards,
The Authors

Reviewer_WRGQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VEGA, a training-time alignment method to improve spatial representations in vision-language-action models. It aligns visual encoder outputs with frozen FiT3D teacher features using an auxiliary projector and a patch-level cosine loss. The teacher and projector are removed at inference. Experiments on OpenVLA-OFT and π₀.₅ show higher average success rates than the corresponding baselines and Spatial Forcing on RoboTwin 2.0 and real-robot tasks. Teacher-and-site ablations and depth probing provide further analysis of the learned representations.

Critical strengths
  1. The method has practical value as a training-time addition to existing VLA models. It transfers geometric knowledge during downstream fine-tuning without replacing the visual encoder or repeating foundation-model pretraining, and adds no inference components. Its integration into two VLA architectures supports the usefulness of this approach.
  2. The experimental results provide complementary evidence of effectiveness. On real-robot tasks, VEGA improves average success over Spatial Forcing from 55.0% to 60.0% for OpenVLA-OFT and from 63.8% to 68.8% for π₀.₅. Depth probing also shows improved depth decodability on both architectures under matched decoder structures and training protocols. Together, these results support both practical gains and improved geometric representations.
Critical weaknesses
  1. The study offers limited systematic insight beyond identifying an effective teacher-and-supervision-site combination. The method builds on established feature-distillation techniques and FiT3D representations, so insight into where geometric supervision should be applied is particularly important. Table 2 shows an interesting interaction: encoder-level supervision helps FiT3D but hurts VGGT relative to LLM-layer supervision. However, the controlled comparison covers only OpenVLA-OFT on RoboTwin, and the only LLM position tested is layer 24, inherited from Spatial Forcing. A position suitable for VGGT may not be suitable for FiT3D. The results support the tested configuration but provide limited evidence about when encoder-level alignment is preferable or whether the observed interaction transfers to other settings. This limits the paper’s additional research contribution beyond its demonstrated practical utility.
Questions for authors
  1. Clarification and additional evidence that could change my assessment: Was layer 24 validated separately for each teacher in the current setting? Can the authors provide a controlled comparison of encoder-output alignment against several LLM depths using the same FiT3D teacher and training protocol? This would help determine whether the observed advantage persists beyond the selected LLM layer.
  2. Additional evidence that could change my assessment: Can the authors provide a small teacher-by-supervision-site comparison on π₀.₅, analogous to Table 2? The current main results establish that VEGA works on a second architecture, but do not establish whether the teacher–site interaction transfers across architectures.
8 comments
AuthorsOpenReview ↗

Response to Reviewer WRGQ (Part 1)

Dear Reviewer WRGQ:

Thank you for recognizing VEGA’s practical value and the complementary evidence from manipulation experiments and depth probing. We appreciate your suggestions on alignment depth and cross-architecture teacher–supervision-site analysis. We will address your concerns below.

[W1 & Q1.] Controlled Comparison Across LLM Depths

In the original Table 2, we adopted layer 24 following Spatial Forcing as a fixed reference across teachers, without separate teacher-specific layer validation. To more comprehensively evaluate the choice of supervision location, we extended the analysis to multiple LLM depths on both OpenVLA-OFT and π₀.₅.

Specifically, we compared encoder-output alignment against LLM layers 1, 8, 16, 24, and 32 on OpenVLA-OFT, and layers 1, 6, 12, and 18 on π₀.₅. Within each architecture, we fixed the corresponding FiT3D teacher and used matched training and evaluation settings. Evaluation covered the original six RoboTwin 2.0 tasks under Easy and Hard settings.

OpenVLA-OFT with the DINOv2-FiT3D teacher

TaskSettingL1L8L16L24L32Encoder
Move Card AwayEasy0.750.720.730.710.680.77
Hard0.520.460.500.530.480.43
Turn SwitchEasy0.380.300.330.350.360.39
Hard0.220.170.200.140.150.24
Click BellEasy0.930.890.920.920.910.97
Hard0.310.320.300.280.220.53
Beat BlockEasy0.250.280.270.220.280.38
Hard0.110.090.120.100.100.11
Lift PotEasy0.920.920.930.910.820.98
Hard0.200.150.130.050.110.28
Place ShoeEasy0.500.470.480.440.380.56
Hard0.210.160.190.100.050.25
AVGEasy0.6220.5970.6100.5920.5720.675
Hard0.2620.2250.2400.2000.1850.307

π₀.₅ with the SigLIP-FiT3D teacher

TaskSettingL1L6L12L18Encoder
Move Card AwayEasy0.720.690.700.700.75
Hard0.360.340.340.300.34
Turn SwitchEasy0.190.200.170.180.21
Hard0.110.110.120.130.15
Click BellEasy0.850.790.770.800.91
Hard0.700.630.650.660.75
Beat BlockEasy0.420.380.400.360.49
Hard0.080.110.100.080.11
Lift PotEasy0.880.820.830.830.89
Hard0.100.050.080.060.15
Place ShoeEasy0.120.130.160.120.20
Hard0.040.020.050.020.09
AVGEasy0.5300.5020.5050.4980.575
Hard0.2320.2100.2230.2080.265

L denotes the LLM layer index. AVG is the mean success rate across all six tasks.

Across both architectures, layer 1 achieves the highest averages among the tested LLM depths. Encoder-output alignment achieves the highest six-task averages, exceeding this strongest tested LLM configuration by 5.3/4.5 percentage points on OpenVLA-OFT and 4.5/3.3 percentage points on π₀.₅, respectively, under Easy/Hard settings. These comparisons show that the average advantage of encoder-level FiT3D alignment persists across a broader range of supervision depths and two VLA architectures with different visual backbones.

AuthorsOpenReview ↗

Response to Reviewer WRGQ (Part 2)

[Q2.] Teacher–Supervision-Site Comparison on π₀.₅

To test whether the teacher–supervision-site interaction observed on OpenVLA-OFT also holds on π₀.₅, we evaluated VGGT, original SigLIP, and SigLIP-FiT3D at both the visual encoder output and LLM layer 12. We used layer 12 as the common LLM alignment site for all three teachers because it is the layer used by Spatial Forcing with VGGT. All six configurations followed matched training and evaluation protocols. The original SigLIP teacher serves as a control for assessing the effect of FiT3D adaptation at each supervision site.

TaskSettingVGGT Enc.VGGT L12SigLIP Enc.SigLIP L12FiT3D Enc.FiT3D L12
Move Card AwayEasy0.610.660.660.620.750.70
Hard0.220.290.300.250.340.34
Turn SwitchEasy0.180.190.150.140.210.17
Hard0.130.120.100.120.150.12
Click BellEasy0.890.940.880.880.910.77
Hard0.660.730.720.690.750.65
Beat BlockEasy0.580.610.580.530.490.40
Hard0.050.090.090.040.110.10
Lift PotEasy0.700.730.690.660.890.83
Hard0.130.130.110.090.150.08
Place ShoeEasy0.160.180.170.160.200.16
Hard0.070.080.090.080.090.05
AVGEasy0.5200.5520.5220.4980.5750.505
Hard0.2100.2400.2350.2120.2650.223

The results show a clear teacher-dependent preference: VGGT performs better at LLM layer 12, whereas SigLIP and SigLIP-FiT3D perform better at the encoder output. FiT3D yields the largest encoder-level gains among the tested teachers, improving over its L12 configuration by 7.0/4.2 percentage points on Easy/Hard. At the encoder output, FiT3D also improves over the original SigLIP teacher by 5.3/3.0 percentage points, demonstrating the additional benefit of FiT3D adaptation.

These findings extend the teacher-dependent supervision pattern observed in OpenVLA-OFT to π₀.₅ and its SigLIP backbone. They provide further evidence for jointly considering teacher choice and supervision location, while reinforcing VEGA’s encoder-level FiT3D design across both evaluated architectures.

AuthorsOpenReview ↗

Follow-up on Our Response to Your Review

Dear Reviewer WRGQ,

We hope you are doing well. We wanted to follow up on our rebuttal and ask whether you have had a chance to review the new experiments addressing your comments.

  1. Choice of alignment depth. Using the same FiT3D teacher within each architecture, we compared encoder-output alignment with five LLM depths on OpenVLA-OFT and four on π₀.₅. Layer 1 performed best among the tested LLM layers on both architectures. Encoder-output alignment still achieved the highest average success rates, exceeding layer 1 by 5.3/4.5 percentage points on OpenVLA-OFT and 4.5/3.3 on π₀.₅ in the Easy/Hard settings.
  2. Teacher and supervision location across architectures. Our new π₀.₅ ablation showed the same pattern as OpenVLA-OFT. VGGT performed better at the tested LLM layer, while the FiT3D-adapted teacher performed better at the encoder output. The original SigLIP control further showed that FiT3D adaptation yielded larger gains at the encoder output than at the tested LLM layer.

We would appreciate hearing whether these results address your concerns. We are happy to clarify any remaining points.

Best regards,
The Authors

Reviewer_Jc5TOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a method for 3D Alignment of the visual encoder patches by using a frozen FiT3D teacher output tokens during the action prediction fine tuning. The method is motivated by an analysis on the feature space of the visual encoder of the baseline models as well as comparing these features to the LLM's visual token representation at Layer 24. To support the use of the FiT3D teacher, they compare DinoV2, and DinoV2-FiT3D which had undergone the FiT3D training. Experiments and results support the method in outperforming baselines on the final tasks benchmarks while also showing higher training and data efficiency during evaluation. Visualizing the final model's visual features with depth probing showed better depth alignment compared the baseline method.

Critical strengths

1- The authors show clear motivation for their proposed method via the feature space analysis on both the visual encoder and the LLM.

2- The comparison between the teach visual encoder with and without FiT3D solidifies the choice of Teacher model.

3- The paper shows higher success rate with less training steps and demonstration data.

4- Results show the validity of the extra encoder supervision loss on both easy and hard tasks of RoboTwin 2.0

5- Real world experiments show improvement of the success rate on a variety of tasks.

6- The authors did multiple seed evaluations provided in the supplementary and shows that specifically on the hard subset of the tasks, their method achieves noticeable improvement.

Critical weaknesses

1- While the authors did an extensive ablation on the choice of their method, they still primarily compare with only the baseline in the main experiments. Aside from Rocket, there are no direct comparison with the other spatial grounding based methods for VLAs.

2- The two baselines (Pi 0.5 and OpenVLA-OFT) that the authors train upon appear to have good initial accuracy on the RoboTwin2.0 tasks. However, Rocket is used in Table 1 is on a different baseline (Pi 0) hence the comparison with their results isn't valid.

Questions for authors

In Table 2, the authors ablate on either having the alignment on the visual encoder output or the Layer 24 of the LLM. Is there a reason for choosing not to do it on both of them at the same time? It seems from the results that for some tasks, the llm layer alignment can be better at least with VGGT and DinoV2 at teachers.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Jc5T

Dear Reviewer Jc5T:

We sincerely thank you for your detailed and constructive review of our paper. We will address your concerns below.

[W1 & W2.] Spatial Grounding Comparisons with Matched Backbones

We appreciate your suggestion to broaden the comparison with prior spatial-grounding methods. Spatial Forcing (SF) and ROCKET provide two relevant baselines for implicit spatial grounding in VLAs, using single-layer and multi-layer geometric feature alignment, respectively. The original submission reports comparisons with SF on both OpenVLA-OFT and π₀.₅ in Tables 1 and 4, along with simulation results across three evaluation seeds in Table 5.

Table 1 includes ROCKET’s reported RoboTwin 2.0 results on π₀, the backbone used in its original experiments. To extend the comparison, we also evaluated ROCKET on OpenVLA-OFT and π₀.₅, matching VEGA’s training data, policy initialization, training budget, adaptation settings, and evaluation protocol within each backbone.

Six-task mean success rates on RoboTwin 2.0

BackboneSettingBaseline+ SF+ ROCKET+ VEGA
π₀.₅Easy0.5470.5520.5600.575
Hard0.2370.2400.2450.265
OpenVLA-OFTEasy0.5600.6420.6370.675
Hard0.2270.2780.2900.307

VEGA achieves the highest six-task average success rates on both backbones under both Easy and Hard settings, outperforming SF and ROCKET. Compared with ROCKET, VEGA improves success rates by 1.5/2.0 percentage points on π₀.₅ and 3.8/1.7 percentage points on OpenVLA-OFT under Easy/Hard, respectively. These comparisons support VEGA’s effectiveness relative to representative spatial-grounding methods under matched policy backbones.


[Q1.] Joint Encoder and LLM Alignment

We appreciate your question about joint alignment at the encoder output and LLM layer 24. Table 2 reports the two single-site configurations, and VEGA uses encoder-only alignment to avoid selecting an internal LLM layer. To further investigate whether supervision at both sites improves performance, we evaluated a third configuration that aligns them simultaneously, using the same OpenVLA-OFT backbone and DINOv2-FiT3D teacher and keeping the remaining training and evaluation settings matched.

TaskSettingEncoder OnlyLLM L24 OnlyEncoder + LLM L24
Move Card AwayEasy0.770.710.72
Hard0.430.530.48
Turn SwitchEasy0.390.350.35
Hard0.240.140.17
Click BellEasy0.970.920.90
Hard0.530.280.40
Beat BlockEasy0.380.220.28
Hard0.110.100.12
Lift PotEasy0.980.910.93
Hard0.280.050.20
Place ShoeEasy0.560.440.42
Hard0.250.100.08
AVGEasy0.6750.5920.600
Hard0.3070.2000.242

With the same DINOv2-FiT3D teacher, joint encoder–LLM alignment achieves average success rates of 60.0%/24.2% on Easy/Hard, below encoder-only alignment (67.5%/30.7%) by 7.5/6.5 percentage points. Although joint supervision improves individual settings, such as Move Card Away on Hard (48% vs. 43%), these gains do not translate into higher task-average performance. Under the tested configuration, adding LLM-layer supervision does not improve upon encoder-only FiT3D alignment, supporting VEGA’s simpler single-site design.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response [Update of PDF & Point-by-Point Reply]

We thank the reviewers, Area Chairs, and Program Chairs for their time and thoughtful feedback. We have provided detailed point-by-point responses in the individual reviewer threads.

We appreciate the reviewers’ recognition of the following strengths in our submission.

  • Method. “The method has practical value as a training-time addition to existing VLA models.” (Reviewer WRGQ) “VEGA preserves the inference architecture and requires no teacher computation at deployment.” (Reviewer Z6mL)
  • Motivation and ablations. “The authors show clear motivation for their proposed method via the feature space analysis on both the visual encoder and the LLM.” (Reviewer Jc5T) “A useful controlled analysis of teacher choice and alignment location.” (Reviewer Z6mL)
  • Evaluation. “Real world experiments show improvement of the success rate on a variety of tasks.” (Reviewer Jc5T) “The experimental results provide complementary evidence of effectiveness.” (Reviewer WRGQ)
  • Presentation. “Clear and easy to follow.” (Reviewers Jc5T and Z6mL)

In response to the reviews, we added matched-backbone comparisons, expanded the supervision-site ablations, and evaluated the results across independent training runs and unseen trajectories. The main additions are highlighted in blue in the revised PDF. The section and table numbers below refer to the revised manuscript.

  • Section 4.2 (Table 1) — Matched-backbone comparisons. To address Reviewer Jc5T’s concern about comparing across backbones, we evaluated ROCKET on OpenVLA-OFT and π₀.₅. VEGA achieves higher six-task mean success rates than Spatial Forcing and ROCKET on both backbones under Easy and Hard settings.
  • Section 4.3 and Appendix C.5 (Tables 3 and 11) — Supervision depth. To address Reviewer WRGQ’s question about the choice of LLM layer, we compared encoder-output alignment with several LLM depths, holding the FiT3D teacher fixed within each backbone. Encoder-output alignment achieves the highest six-task mean under both settings on both backbones.
  • Appendices C.6 and C.7 (Tables 12 and 13) — Joint alignment and teacher choice. For Reviewer Jc5T, we added a joint encoder and LLM layer-24 configuration on OpenVLA-OFT. Its six-task means are lower than those of encoder-only alignment under the tested configuration. For Reviewers WRGQ and Z6mL, we added a three-teacher comparison on π₀.₅. VGGT performs better at LLM layer 12, while SigLIP-FiT3D performs better at the encoder output.
  • Section 4.4 and Appendix C.1 (Tables 6 and 8) — Real-world uncertainty and training variability. To address Reviewer Z6mL’s reliability questions, we report real-world success counts and 95% confidence intervals, as well as three independent training runs per method on each backbone. VEGA outperforms Spatial Forcing for every matched training seed under both simulation settings. In the real-world trials, the observed mean difference is +5.0 percentage points on each backbone, with both confidence intervals spanning zero.
  • Section 4.3 and Appendix C.3 (Tables 5 and 10) — Depth probing on unseen trajectories. To address Reviewer Z6mL’s question about geometric generalization, we evaluated frozen depth probes on 120 expert trajectories held out from policy fine-tuning and probe training and validation. VEGA reduces mean depth error in both object and gripper regions on both backbones, with larger reductions in the object regions.

We thank the reviewers again for their feedback and welcome further discussion.

Best regards, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.