- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces GeoAct, which jointly predicts future RGB, geometric latent states, and robot actions to provide geometric supervision for manipulation policies. GeoAE compresses multi-layer VGGT-Omega features into spatiotemporal latents while preserving geometry through feature, depth, and camera reconstruction. Following multi-domain video pretraining and adaptation to robot demonstrations, deployment generates actions from current RGB, language, and proprioception without online geometry encoding or future video generation. Evaluations cover LIBERO-Plus, RoboTwin 2.0, and three real-world bimanual tasks, with ablations examining geometric supervision, prediction targets, and compression design.
Critical strengths
- Geometric supervision is supported by substantial ablations. Matched comparisons demonstrate its benefit for policy learning, while comparisons of prediction targets and compression methods further support GeoAE’s design.
- Geometric supervision preserves efficient deployment. Geometry prediction shapes representations during training, while deployment generates actions directly, avoiding online geometry encoding and future video generation. The inspected implementation is consistent with this design.
Critical weaknesses
- Robustness to background perturbations remains limited. GeoAct trails some baselines under background perturbations, and joint supervision does not consistently outperform RGB-only.
- Future geometry prediction is validated mainly qualitatively. Quantitative geometric metrics primarily assess GeoAE’s reconstruction of teacher geometry, while future forecasts are mainly visualized on training clips. Direct evidence of future geometric prediction accuracy on unseen clips therefore remains limited.
Questions for authors
- Based on existing failure cases, what primarily causes the weaker performance under background perturbations? What evidence supports the proposed explanation?
- Could the authors quantitatively evaluate future geometry prediction on unseen clips using the existing models, clarifying whether the reference is teacher predictions or simulator ground truth?
4 comments
Future Geometry Accuracy on Held-Out Clips
Corresponding to: W2, Question 2
Quantitative geometric metrics primarily evaluate GeoAE's reconstruction of teacher geometry, while future predictions are mainly visualized on training clips. Can you quantitatively evaluate future geometry on unseen clips using the existing model, and clarify whether the reference is teacher predictions or simulator ground truth?
Following your suggestion, we quantitatively evaluate future geometry prediction on held-out validation clips across all five domains, using both teacher estimates and simulator ground truth as references. Conditioned solely on the current observation and language instruction, GeoAct decodes predicted geometric latents into depth maps, which are compared against VGGT-Ω teacher estimates computed on the actual future frames; for LIBERO and RoboTwin 2.0, we additionally compare against simulator ground-truth depth.
Each entry below reports AbsRel (↓) / δ₁ (%, ↑); "—" denotes datasets without simulator ground-truth depth:
| Dataset | GeoAE Recon. vs. VGGT-Ω | Future Pred. vs. VGGT-Ω | Future Pred. vs. Sim. GT | VGGT-Ω vs. Sim. GT |
|---|---|---|---|---|
| LIBERO | 0.042 / 96.5 | 0.066 / 94.6 | 0.177 / 75.0 | 0.139 / 70.8 |
| RoboTwin 2.0 | 0.056 / 95.9 | 0.118 / 84.4 | 0.156 / 75.4 | 0.107 / 81.8 |
| WorldArena 2.0 | 0.070 / 97.0 | 0.126 / 87.3 | — | — |
| BridgeData V2 | 0.071 / 96.6 | 0.330 / 62.2 | — | — |
| SSv2 | 0.021 / 99.1 | 0.107 / 87.6 | — | — |
Against simulator ground truth, GeoAct's predictions reach accuracy on par with the teacher's direct estimates: on LIBERO, its δ₁ exceeds the teacher's (75.0% vs. 70.8%), while on RoboTwin 2.0 both metrics are slightly below the teacher's (see table above). This confirms that on unseen clips, GeoAct predicts 3D scene evolution comparable to teacher estimates computed directly from future frames. This evaluation has been added to Table 7 in Section 4.5 of the revised manuscript.
We hope these responses fully address your questions and welcome any further discussion.
Thank you for addressing both of my questions. The ablation results help explain the trade-off between retaining appearance cues for target identification and improving robustness to background changes. The new evaluation on held-out clips, including comparisons against simulator ground-truth depth where available, provides direct quantitative evidence for future geometry prediction. I appreciate these additions and maintain my rating of 3 (Weak acceptance).
Thank you very much for your follow-up and for confirming that both of your questions have been addressed. We are glad that the ablation analysis on the appearance-robustness trade-off and the quantitative evaluations on held-out clips have resolved your initial concerns regarding the soundness and completeness of our empirical validation. We truly appreciate your constructive engagement throughout the review process, and we hope these additions prove helpful in your final overall assessment of our work.
How Future RGB Prediction Affects Background Robustness
We thank the reviewer for carefully examining our implementation and recognizing the efficiency of our closed-loop RGB deployment. In response to your two questions, we have added an analysis of performance under background perturbations and a quantitative evaluation of future geometry prediction on unseen clips, as detailed below. All revisions in the updated manuscript are highlighted in blue.
Corresponding to: W1, Question 1
We attribute this to the dual role of future RGB prediction: identifying target objects from language instructions requires the appearance cues provided by RGB supervision, yet reconstructing full-frame future RGB latents also requires the representation to retain background textures. Removing the RGB branch (Geometry-only) renders the policy invariant to background texture replacement—achieving 87.73% under background perturbations (Table 3)—but deprives it of color and appearance cues, causing its performance under language perturbations to drop to 35.91% and yielding the lowest overall success rate (65.88% vs. 78.27% for joint prediction). Consequently, to ensure accurate instruction-conditioned target identification, GeoAct retains future RGB prediction, and the Action expert attends to the current RGB representation. Because reconstructing future video frames requires this representation to preserve background details, unseen background textures at test time propagate into the Action expert's input.
Video-prediction baselines in Table 1 exhibit the exact same pattern: although both lighting and background perturbations alter only visual appearance, LingBot-VA, Fast-WAM, and 4D-WAM achieve 78.2%–94.3% under lighting changes but drop to 53.1%–57.7% under background replacement, which introduces unseen textures. In contrast, VLA baselines that do not predict future video vary by no more than 4 percentage points between the two perturbations. The ablation variants in Table 3 are trained only on unaugmented LIBERO demonstrations, so unseen background textures at test time directly perturb their RGB representation. π₀.₅ mitigates this through large-scale multi-scene pretraining, whereas our main GeoAct model expands appearance coverage with image augmentation (Table A4) and reaches 75.2% under background perturbations (Table 1). We have incorporated this analysis into Section 4.4.1 of the revised manuscript.