- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes VEGA, a framework for improving spatial representations in vision-language-action models by aligning features at the visual encoder output. VEGA uses frozen teachers obtained by adapting visual backbones with FiT3D, which transfers multi-view information through features rendered from 3D Gaussian representations. During policy training, an auxiliary projector aligns visual encoder features with the corresponding teacher features using a patchwise cosine-distance loss, optimized alongside the action objective. The teacher and projector are removed at inference.
The method is evaluated on OpenVLA-OFT and π₀.₅ using six RoboTwin 2.0 tasks under clean and randomized conditions, four main real-world tasks, and an additional domino task. The paper reports improved average success rates, studies how teacher choice interacts with alignment location, and provides depth probing as well as training- and data-efficiency analyses.
Critical strengths
A useful controlled analysis of teacher choice and alignment location. Table 2 evaluates three teachers at both the visual encoder output and LLM layer 24, which is more informative than comparing only the final VEGA configuration against Spatial Forcing. At the encoder output, DINOv2-FiT3D raises the Easy/Hard averages from 59.3%/25.8% with the original DINOv2 teacher to 67.5%/30.7%, whereas moving VGGT supervision to the encoder reduces performance. These results reveal a consequential interaction between teacher choice and supervision location and support the specific configuration VEGA uses.
A practical approach with evidence across two policy architectures. VEGA preserves the inference architecture and requires no teacher computation at deployment. The paper evaluates both OpenVLA-OFT and π₀.₅, includes randomized simulation conditions, and provides real-robot comparisons that use the same demonstrations across methods. The π₀.₅ baseline also uses the same vision adaptation scheme, which helps isolate the effect of the alignment objective. Together, these experiments offer encouraging evidence that the approach can benefit more than one VLA architecture.
Critical weaknesses
- The gains over the strongest matched baseline are not yet established across independent training runs.
Section 4.1 states that each configuration uses one training seed, and the three evaluation seeds in Table 5 evaluate a single fixed checkpoint per configuration, so they do not measure training variability. In these evaluations, VEGA exceeds Spatial Forcing on Easy/Hard by 2.8/4.5 points on OpenVLA-OFT and 1.2/3.3 points on π₀.₅. These gains are encouraging, but their robustness to training randomness is unknown.
The real-world comparison also has limited resolution: with 20 trials per task, the five-point average advantage over Spatial Forcing amounts to four additional successes in 80 trials per architecture. Independent training replications with uncertainty estimates would materially strengthen the central empirical contribution.
- The evidence for the geometric rationale is limited relative to its role in the contribution.
Because the contribution mainly concerns where to apply an existing alignment objective and which teacher to use, explaining and validating those choices is especially important. Yet the attribution and masking analyses in Figure 2 rest on one observation, and greater sensitivity to masking upstream tokens does not show that this location is better suited to geometric supervision, since information may already have spread across tokens by the probed LLM layer.
The depth analysis is complementary, but Appendix A.5 states that all probe episodes were used in policy fine-tuning, and the analysis also uses a separately trained, relatively expressive decoder per task and averages error over all valid pixels. These results therefore offer limited evidence of transferable geometry in action-relevant regions. Table 2 supports the selected teacher-site pairing, but the broader account of why it improves spatial competence remains underdeveloped.
Questions for authors
Could you report VEGA versus Spatial Forcing across independent policy-training seeds on both architectures? For the real-world results, were initial configurations matched across methods, and can you provide uncertainty estimates for the success-rate differences?
Does the depth-decoding advantage hold on trajectories or scenes unseen during fine-tuning, especially in manipulated-object and gripper regions? This would show how far the representation gains extend beyond the training configurations.
8 comments
Response to Reviewer Z6mL (Part 2)
[W2 & Q2.] Depth Probing Beyond Policy-Training Data
We evaluate 120 new expert trajectories (20 per task) using simulator seeds disjoint from policy fine-tuning. These trajectories are also held out from probe training and validation. All methods receive identical three-camera observations with policies, existing task-specific probes, and depth normalization frozen. Within each architecture, baseline and VEGA use matched probe architectures, training, and checkpoint selection. We compute per-trajectory AbsRel against simulator depth for the full image and visible object and gripper regions, then average across trajectories and equally across tasks.
Entries report Baseline → VEGA (relative change); lower is better.
OpenVLA-OFT
| Task | Full Image | Object ROI | Gripper ROI |
|---|---|---|---|
| move_playingcard_away | 0.01062 → 0.00962 (↓9.42%) | 0.01732 → 0.01581 (↓8.72%) | 0.05594 → 0.05730 (↑2.43%) |
| turn_switch | 0.02941 → 0.02081 (↓29.24%) | 0.09286 → 0.06484 (↓30.17%) | 0.06650 → 0.06574 (↓1.14%) |
| click_bell | 0.00900 → 0.00858 (↓4.67%) | 0.02117 → 0.02002 (↓5.43%) | 0.06572 → 0.06143 (↓6.53%) |
| beat_block_hammer | 0.01592 → 0.01381 (↓13.25%) | 0.07851 → 0.07645 (↓2.62%) | 0.06317 → 0.06484 (↑2.64%) |
| lift_pot | 0.02177 → 0.02034 (↓6.57%) | 0.03765 → 0.03609 (↓4.14%) | 0.05646 → 0.05319 (↓5.79%) |
| place_shoe | 0.02283 → 0.02030 (↓11.08%) | 0.05441 → 0.05447 (↑0.11%) | 0.05548 → 0.05466 (↓1.48%) |
| Mean | 0.01826 → 0.01558 (↓14.69%) | 0.05032 → 0.04461 (↓11.34%) | 0.06055 → 0.05953 (↓1.68%) |
π₀.₅
| Task | Full Image | Object ROI | Gripper ROI |
|---|---|---|---|
| move_playingcard_away | 0.00987 → 0.00965 (↓2.23%) | 0.01623 → 0.01612 (↓0.68%) | 0.05483 → 0.05514 (↑0.57%) |
| turn_switch | 0.01880 → 0.01908 (↑1.49%) | 0.06000 → 0.05634 (↓6.10%) | 0.06247 → 0.06308 (↑0.98%) |
| click_bell | 0.00993 → 0.00936 (↓5.74%) | 0.02915 → 0.02666 (↓8.54%) | 0.06778 → 0.06481 (↓4.38%) |
| beat_block_hammer | 0.01361 → 0.01307 (↓3.97%) | 0.07930 → 0.07574 (↓4.49%) | 0.05988 → 0.05856 (↓2.20%) |
| lift_pot | 0.02124 → 0.02128 (↑0.19%) | 0.03746 → 0.03734 (↓0.32%) | 0.05486 → 0.05353 (↓2.42%) |
| place_shoe | 0.02360 → 0.02179 (↓7.67%) | 0.05119 → 0.04855 (↓5.16%) | 0.05681 → 0.05251 (↓7.57%) |
| Mean | 0.01618 → 0.01571 (↓2.91%) | 0.04556 → 0.04346 (↓4.60%) | 0.05944 → 0.05794 (↓2.52%) |
On these unseen trajectories, VEGA reduces mean object/gripper ROI error by 11.34%/1.68% on OpenVLA-OFT and 4.60%/2.52% on π₀.₅, providing evidence that its depth-decoding gains extend beyond policy-training data to action-relevant regions.
[Ethics.] Clarification of the Ethics-Review Flag
We would appreciate clarification of the specific concern underlying the ethics-review flag, as no corresponding concern is described in the review. We are happy to provide further details and address it directly.
Follow-up on Our Response to Your Review
Dear Reviewer Z6mL,
We hope you are doing well. We wanted to follow up on our rebuttal and ask whether you have had a chance to review the new results addressing your comments.
- Training reliability and real-world evaluation. VEGA achieves higher average simulation success rates than Spatial Forcing across all three independent training seeds on both architectures. For the real-world comparison, we verified that initial layouts were matched and reported paired 95% confidence intervals for the success-rate differences.
- Choice of alignment location. Our new π₀.₅ ablation shows the same pattern as OpenVLA-OFT. FiT3D-adapted teachers perform better with alignment at the visual encoder output, and VGGT performs better at the tested LLM layer.
- Depth probing on unseen trajectories. Using frozen probes on 120 new expert trajectories, VEGA reduces mean depth error in both the manipulated-object and gripper regions on both architectures.
We would appreciate hearing whether these results address your concerns. We are happy to clarify any remaining points.
Best regards,
The Authors
Response to Reviewer Z6mL (Part 1)
Dear Reviewer Z6mL:
Thank you for recognizing VEGA’s practical value and the controlled teacher–supervision-site analysis. We appreciate your suggestions on training variability and geometric generalization. We will address your concerns below.
[W1 & Q1.] Independent Training Runs and Real-World Reliability
To assess training variability, we now report three independent training seeds per method: {0, 1, 2} for OpenVLA-OFT and {42, 43, 44} for π₀.₅.
*Mean and sample SD are computed across three independent training runs, using each run’s six-task average. *
VEGA maintains higher six-task average success rates across all three training runs in both architectures and settings, providing additional evidence that the observed advantage persists under training randomness.
For real-world evaluation, we verified that SF and VEGA used matched initial conditions for each of the 20 trials per task. We report per-task success counts and rates below. We estimate uncertainty in the four-task mean difference using a task-stratified paired percentile bootstrap with 100,000 resamples, preserving trial pairs within each task and weighting the four tasks equally.
Each task entry reports successes out of 20 trials, with the success rate in parentheses. Mean SR is the unweighted four-task average. Differences compare VEGA with SF within the same architecture; confidence intervals quantify evaluation uncertainty for the evaluated checkpoints.
These matched evaluations show an observed average improvement of 5.0 percentage points on both architectures, complementing the simulation results.
[W2.] Evidence for the Alignment Location
We thank the reviewer for highlighting how information can propagate across tokens. Figure 2 motivates testing encoder-output alignment. In the controlled comparison in Table 2, with DINOv2-FiT3D held fixed on OpenVLA-OFT, aligning at the visual encoder output improves Easy and Hard success rates by 8.3 and 10.7 percentage points, respectively, over alignment at LLM layer 24. VGGT favors layer 24.
On π₀.₅, we compare three teachers at the encoder output and LLM layer 12, the site used by Spatial Forcing, under matched protocols. The same pattern holds: SigLIP-FiT3D favors the encoder output by 7.0/4.2 percentage points on Easy/Hard, and VGGT favors layer 12 by 3.2/3.0 points.
Mean success rates across six tasks on π₀.₅.