- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents GEM3D, a two-stage framework for learning amodal 3D representations for view-generalizable robotic manipulation from single-view RGB-D observations. During pretraining, the model combines coarse-to-fine point-cloud reconstruction with Gaussian-splatting-based novel-view rendering under multi-view supervision to learn scene geometry and appearance beyond the input view. During policy learning, the pretrained encoder is frozen as a teacher, and a multi-step distillation objective aligns the policy’s current and predicted next-state representations with corresponding teacher features. The resulting policy operates on calibrated single-view RGB-D inputs without requiring the teacher or explicit reconstruction at inference. Experiments cover 12 RLBench tasks and three real-world tasks, with additional viewpoint-generalization evaluations on eight RLBench tasks. The reported results show improvements over the evaluated baselines in both manipulation success and robustness to camera viewpoint changes.
Critical strengths
-
The paper addresses a practically relevant and well-motivated problem: leveraging multi-view supervision during training while requiring only a single RGB-D view at deployment. An important observation is that world-frame alignment alone does not ensure viewpoint generalization, as camera changes alter surface visibility and occlusion patterns even in a shared coordinate system. This motivates learning representations that account for partial observability rather than relying solely on coordinate normalization.
-
The framework coherently integrates representation learning, knowledge transfer, and policy learning. Point-cloud reconstruction provides geometric supervision, Gaussian rendering introduces appearance and cross-view constraints, and distillation from a frozen teacher aims to preserve pretrained geometric knowledge during policy training. Importantly, deployment requires neither the teacher nor explicit reconstruction or Gaussian rendering, allowing the policy to benefit from these training objectives without retaining the reconstruction pipeline at inference.
-
The reported results demonstrate substantial gains in both simulation and real-world manipulation. GEM3D achieves a 44.2% average success rate across 12 RLBench tasks, exceeding the strongest baseline by 9.1 percentage points, and reaches 41.0% and 29.5% under moderate and large viewpoint shifts across eight tasks. Particularly compelling is the transfer from RH20T pretraining to three real-world tasks with different equipment and scenes, where full-task success rates reach 88.3%, 78.3%, and 63.3% at the training viewpoint and under moderate and large shifts, respectively. These Stage II results provide encouraging evidence of the framework’s practical utility.
Critical weaknesses
-
The evidence for amodal geometric inference could be more targeted. Multi-view reconstruction supervision, overall Chamfer distance, and viewpoint-generalization results provide support for the proposed representation. However, the evaluations do not distinguish between visible and occluded regions, leaving unclear how much of the reconstruction gain reflects improved recovery of occluded, task-relevant geometry. Reporting reconstruction accuracy separately for these regions would strengthen the central claim.
-
The ablation does not cleanly isolate the contribution of Gaussian rendering. In the single-view pretraining setting, the authors both reduce the number of viewpoints and disable the rendering objective. The resulting performance drop therefore conflates reduced geometric supervision with the removal of rendering, and does not by itself establish the importance of Gaussian Splatting. A controlled ablation that disables rendering while keeping the multi-view supervision unchanged would clarify its contribution.
-
The average rank reported in Table D.4 appears incorrect. GEM3D is assigned an average rank of 1.00, although DP3 outperforms it on Meat off Grill, Place Wine, and Stack Blocks. Based on the seven methods listed, GEM3D ranks first on nine tasks and second on three, yielding an average rank of 1.25. This entry should be corrected.
Questions for authors
Please refer to the "Critical Weaknesses" for specific issues. If these can be resolved, I will raise the score.
4 comments
Response to Reviewer sBYP (2/2): Controlled Rendering Ablation and Table D.4
Q2: Ablate the rendering loss while keeping multi-view supervision fixed.
We thank the reviewer for pointing out this important issue. The original single-view setting indeed confounds two factors, and we should have controlled for them more carefully. Following the reviewer’s suggestion, we have extended Table 2(a) into a fully controlled ablation (see Table 2(a) of the revised PDF (in red)). For each number of pretraining views (8, 4, 2, 1), we now compare pretraining with and without Gaussian rendering. Without rendering, pretraining is still supervised by the complete point cloud fused from multiple views, which guides the recovery of occluded geometry. The success rates (%) are:
| Views | Training (w/ GS) | Moderate (w/ GS) | Large (w/ GS) | Training (w/o GS) | Moderate (w/o GS) | Large (w/o GS) |
|---|---|---|---|---|---|---|
| 8 | 58.8 | 45.9 | 39.6 | 52.6 | 39.0 | 34.8 |
| 4 | 54.6 | 42.4 | 37.3 | 49.0 | 36.2 | 33.3 |
| 2 | 54.9 | 40.3 | 33.6 | 49.4 | 34.1 | 32.5 |
| 1 | 48.1 | 35.6 | 30.4 | 47.2 | 33.4 | 30.8 |
With two or more pretraining views, disabling Gaussian rendering consistently degrades performance: by 5–7 points at the training and moderate settings, and by 1–5 points under large shifts (e.g., 39.6 → 34.8 with 8 views); notably, 2 views with rendering perform on par with 8 views without it. With a single view, rendering brings no clear gain, as no held-out view is available for novel-view supervision. This shows that rendering contributes beyond the amount of multi-view geometric supervision.
We attribute this to the additional pixel-aligned cross-view supervision from rendering: since the Gaussians are centered at the predicted points, the photometric loss constrains both appearance and point positions, complementing the set-level Chamfer supervision on the fused point cloud. We have updated Table 2(a) and the corresponding text in Sec. 4.3 of the revised PDF (in red).
Q3: Correct the average rank in Table D.4 (GEM3D: 1.00 → 1.25).
We thank the reviewer for catching this, and we apologize for the error. DP3 and Spatial Forcing were added to the table later, and we forgot to recompute the ranks at that point. Only the Avg. Rank column was wrong; all success rates in the table are correct. We have now recomputed the ranks over all seven methods; the ranks of the other methods were also outdated, and the corrected values are listed below. We have also corrected the second-best marker in the Avg. SR column (now Spatial Forcing, 35.1%).
| Avg. Rank ↓ | PerAct | GNFactor | ManiGaussian | ManiGaussian w/ Distill | DP3 | Spatial Forcing | GEM3D |
|---|---|---|---|---|---|---|---|
| Original | 4.58 | 3.33 | 3.08 | 3.00 | 4.17 | 3.17 | 1.00 |
| Corrected | 6.50 | 4.75 | 4.33 | 4.25 | 3.42 | 3.50 | 1.25 |
We have updated Table D.4 in the revised PDF (in red). We thank the reviewer again for helping us improve the accuracy of the paper.
Thank authors for the rebuttal.
The author's reply addressed some of my concerns. So I decided to increase my rating.
We sincerely thank the reviewer for the very prompt response and for raising the rating! We truly appreciate the time and effort the reviewer has devoted to our paper, and the suggestions on occluded-region evaluation and the controlled rendering ablation have substantially strengthened our work.
If any concerns remain, we would be grateful if the reviewer could let us know, and we will do our best to address them during the remaining discussion period.
Response to Reviewer sBYP (1/2): Occluded-Region Reconstruction
We are grateful to the reviewer for the detailed and constructive feedback, which has helped us clarify the central evidence of our paper. Please find our point-by-point responses below.
Q1: Report reconstruction accuracy separately for visible vs. occluded (task-relevant) regions.
We thank the reviewer for this suggestion. Fig. 8(a) was in fact designed to examine exactly this question, but we realize that our original description did not make it clear. To clarify: the policy is trained from view c and tested from views a, b, d and e.
Views a, b and c all capture the cabinet from behind, so the task-relevant drawer front is fully occluded in these inputs. Pred. a–c therefore show exactly the occluded-region reconstruction the reviewer asked about, for both GEM3D and the baseline: GEM3D recovers the cabinet and drawer structure, while ManiGaussian w/ distill produces no meaningful structure there. This difference is reflected in policy success: GEM3D outperforms the baseline under every occluded view (8% / 23% / 62% vs. 0% / 9% / 51% for views a / b / c).
We believe the confusion stems from our insufficiently clear description of this figure. In the revised PDF (in red), we have rewritten the corresponding paragraph in Sec. 5 and the Fig. 8 caption. We have also added the Chamfer distance on the task-relevant region from each input view (Table 3) for a quantitative comparison: under the occluded views a / b / c, GEM3D achieves 12.77 / 17.87 / 9.19 mm vs. 45.69 / 46.80 / 33.14 mm for ManiGaussian w/ distill. The full per-view results are:
We thank the reviewer again for helping us strengthen the presentation of our central claim!