acceptodds
Under review as a conference paper at ICLR 2027

Learning Geometrically-Grounded Amodal 3D Representations for View-Generalizable Robotic Manipulation

25 views

Abstract

Real-world robotic manipulation requires visuomotor policies capable of robust 3D scene reasoning under varying camera viewpoints. While recent 3D-aware manipulation policies have shown promise, they still face several limitations: (i) reliance on multi-view observations during inference, which is impractical in camera-constrained deployments; (ii) insufficient geometric fidelity in learned representations, limiting precise and viewpoint-robust control; and (iii) the lack of effective mechanisms for transferring pretrained 3D representations to downstream visuomotor policies. To address these challenges, we present GEM3D (Geometrically-Grounded 3D Manipulation), a unified 3D representation and policy learning framework for view-generalizable robotic manipulation. GEM3D learns amodal 3D representations from single-view RGB-D observations by jointly enforcing point-cloud reconstruction and Gaussian-splatting-based novel-view consistency during pretraining. A multi-step distillation strategy then transfers the learned geometric understanding into a deployable single-view visuomotor policy for downstream manipulation control. We evaluate 3D scene reasoning through zero-shot viewpoint generalization under unseen camera poses. Extensive experiments across 12 RLBench tasks and three real-world robotic manipulation tasks demonstrate that GEM3D achieves strong viewpoint robustness and significantly outperforms existing state-of-the-art methods.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot learning, cv for robotics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sBYPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents GEM3D, a two-stage framework for learning amodal 3D representations for view-generalizable robotic manipulation from single-view RGB-D observations. During pretraining, the model combines coarse-to-fine point-cloud reconstruction with Gaussian-splatting-based novel-view rendering under multi-view supervision to learn scene geometry and appearance beyond the input view. During policy learning, the pretrained encoder is frozen as a teacher, and a multi-step distillation objective aligns the policy’s current and predicted next-state representations with corresponding teacher features. The resulting policy operates on calibrated single-view RGB-D inputs without requiring the teacher or explicit reconstruction at inference. Experiments cover 12 RLBench tasks and three real-world tasks, with additional viewpoint-generalization evaluations on eight RLBench tasks. The reported results show improvements over the evaluated baselines in both manipulation success and robustness to camera viewpoint changes.

Critical strengths
  1. The paper addresses a practically relevant and well-motivated problem: leveraging multi-view supervision during training while requiring only a single RGB-D view at deployment. An important observation is that world-frame alignment alone does not ensure viewpoint generalization, as camera changes alter surface visibility and occlusion patterns even in a shared coordinate system. This motivates learning representations that account for partial observability rather than relying solely on coordinate normalization.

  2. The framework coherently integrates representation learning, knowledge transfer, and policy learning. Point-cloud reconstruction provides geometric supervision, Gaussian rendering introduces appearance and cross-view constraints, and distillation from a frozen teacher aims to preserve pretrained geometric knowledge during policy training. Importantly, deployment requires neither the teacher nor explicit reconstruction or Gaussian rendering, allowing the policy to benefit from these training objectives without retaining the reconstruction pipeline at inference.

  3. The reported results demonstrate substantial gains in both simulation and real-world manipulation. GEM3D achieves a 44.2% average success rate across 12 RLBench tasks, exceeding the strongest baseline by 9.1 percentage points, and reaches 41.0% and 29.5% under moderate and large viewpoint shifts across eight tasks. Particularly compelling is the transfer from RH20T pretraining to three real-world tasks with different equipment and scenes, where full-task success rates reach 88.3%, 78.3%, and 63.3% at the training viewpoint and under moderate and large shifts, respectively. These Stage II results provide encouraging evidence of the framework’s practical utility.

Critical weaknesses
  1. The evidence for amodal geometric inference could be more targeted. Multi-view reconstruction supervision, overall Chamfer distance, and viewpoint-generalization results provide support for the proposed representation. However, the evaluations do not distinguish between visible and occluded regions, leaving unclear how much of the reconstruction gain reflects improved recovery of occluded, task-relevant geometry. Reporting reconstruction accuracy separately for these regions would strengthen the central claim.

  2. The ablation does not cleanly isolate the contribution of Gaussian rendering. In the single-view pretraining setting, the authors both reduce the number of viewpoints and disable the rendering objective. The resulting performance drop therefore conflates reduced geometric supervision with the removal of rendering, and does not by itself establish the importance of Gaussian Splatting. A controlled ablation that disables rendering while keeping the multi-view supervision unchanged would clarify its contribution.

  3. The average rank reported in Table D.4 appears incorrect. GEM3D is assigned an average rank of 1.00, although DP3 outperforms it on Meat off Grill, Place Wine, and Stack Blocks. Based on the seven methods listed, GEM3D ranks first on nine tasks and second on three, yielding an average rank of 1.25. This entry should be corrected.

Questions for authors

Please refer to the "Critical Weaknesses" for specific issues. If these can be resolved, I will raise the score.

4 comments
AuthorsOpenReview ↗

Response to Reviewer sBYP (1/2): Occluded-Region Reconstruction

We are grateful to the reviewer for the detailed and constructive feedback, which has helped us clarify the central evidence of our paper. Please find our point-by-point responses below.


Q1: Report reconstruction accuracy separately for visible vs. occluded (task-relevant) regions.

We thank the reviewer for this suggestion. Fig. 8(a) was in fact designed to examine exactly this question, but we realize that our original description did not make it clear. To clarify: the policy is trained from view c and tested from views a, b, d and e.

Views a, b and c all capture the cabinet from behind, so the task-relevant drawer front is fully occluded in these inputs. Pred. a–c therefore show exactly the occluded-region reconstruction the reviewer asked about, for both GEM3D and the baseline: GEM3D recovers the cabinet and drawer structure, while ManiGaussian w/ distill produces no meaningful structure there. This difference is reflected in policy success: GEM3D outperforms the baseline under every occluded view (8% / 23% / 62% vs. 0% / 9% / 51% for views a / b / c).

We believe the confusion stems from our insufficiently clear description of this figure. In the revised PDF (in red), we have rewritten the corresponding paragraph in Sec. 5 and the Fig. 8 caption. We have also added the Chamfer distance on the task-relevant region from each input view (Table 3) for a quantitative comparison: under the occluded views a / b / c, GEM3D achieves 12.77 / 17.87 / 9.19 mm vs. 45.69 / 46.80 / 33.14 mm for ManiGaussian w/ distill. The full per-view results are:

Input viewabcde
Drawer front occluded✓✓✓✗✗
Chamfer L1 (mm) ↓, ManiGaussian w/ distill45.6946.8033.1437.1942.44
Chamfer L1 (mm) ↓, GEM3D12.7717.879.191.601.70
Success rate (%) ↑, ManiGaussian w/ distill0951343
Success rate (%) ↑, GEM3D823624639

We thank the reviewer again for helping us strengthen the presentation of our central claim!

AuthorsOpenReview ↗

Response to Reviewer sBYP (2/2): Controlled Rendering Ablation and Table D.4


Q2: Ablate the rendering loss while keeping multi-view supervision fixed.

We thank the reviewer for pointing out this important issue. The original single-view setting indeed confounds two factors, and we should have controlled for them more carefully. Following the reviewer’s suggestion, we have extended Table 2(a) into a fully controlled ablation (see Table 2(a) of the revised PDF (in red)). For each number of pretraining views (8, 4, 2, 1), we now compare pretraining with and without Gaussian rendering. Without rendering, pretraining is still supervised by the complete point cloud fused from multiple views, which guides the recovery of occluded geometry. The success rates (%) are:

ViewsTraining (w/ GS)Moderate (w/ GS)Large (w/ GS)Training (w/o GS)Moderate (w/o GS)Large (w/o GS)
858.845.939.652.639.034.8
454.642.437.349.036.233.3
254.940.333.649.434.132.5
148.135.630.447.233.430.8

With two or more pretraining views, disabling Gaussian rendering consistently degrades performance: by 5–7 points at the training and moderate settings, and by 1–5 points under large shifts (e.g., 39.6 → 34.8 with 8 views); notably, 2 views with rendering perform on par with 8 views without it. With a single view, rendering brings no clear gain, as no held-out view is available for novel-view supervision. This shows that rendering contributes beyond the amount of multi-view geometric supervision.

We attribute this to the additional pixel-aligned cross-view supervision from rendering: since the Gaussians are centered at the predicted points, the photometric loss constrains both appearance and point positions, complementing the set-level Chamfer supervision on the fused point cloud. We have updated Table 2(a) and the corresponding text in Sec. 4.3 of the revised PDF (in red).


Q3: Correct the average rank in Table D.4 (GEM3D: 1.00 → 1.25).

We thank the reviewer for catching this, and we apologize for the error. DP3 and Spatial Forcing were added to the table later, and we forgot to recompute the ranks at that point. Only the Avg. Rank column was wrong; all success rates in the table are correct. We have now recomputed the ranks over all seven methods; the ranks of the other methods were also outdated, and the corrected values are listed below. We have also corrected the second-best marker in the Avg. SR column (now Spatial Forcing, 35.1%).

Avg. Rank ↓PerActGNFactorManiGaussianManiGaussian w/ DistillDP3Spatial ForcingGEM3D
Original4.583.333.083.004.173.171.00
Corrected6.504.754.334.253.423.501.25

We have updated Table D.4 in the revised PDF (in red). We thank the reviewer again for helping us improve the accuracy of the paper.

Reviewer_sBYPOpenReview ↗

Thank authors for the rebuttal.

The author's reply addressed some of my concerns. So I decided to increase my rating.

We sincerely thank the reviewer for the very prompt response and for raising the rating! We truly appreciate the time and effort the reviewer has devoted to our paper, and the suggestions on occluded-region evaluation and the controlled rendering ablation have substantially strengthened our work.

If any concerns remain, we would be grateful if the reviewer could let us know, and we will do our best to address them during the remaining discussion period.

Reviewer_zQPfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces GEM3D, a unified 3D representation and policy learning framework for view-generalizable robotic manipulation, that jointly enforces pointcloud reconstruction and novel view consistency during pretraining. A multi-step distillation strategy then produces a deployable single-view visuomotor policy for downstream manipulation control. Results show better generalization to unseen camera poses and substantial gains over existing methods.

Critical strengths

Robustness to camera viewpoints and partially observed scenes is (in my opinion) a key gap in current robotics, and the development of solutions capable of improvements in this setting is a highly relevant step towards real-world deployment with practical usefulness. Amodal leaning is an interesting approach within this setting, and distilling from a multi-view pre-trained model to a lightweight single-view visuomotor policy seems like a promising way to preserve information while achieving faster inference.

Critical weaknesses

To enable amodal learning during the pre-training stage, the proposed method requires multi-view information, which is a much smaller subset of available robotics datasets. So a key question is how much scalable this pre-training approach would be, relative to pre-training on much larger amounts of single-view / uncalibrated multi-view robotics datasets, and learn viewpoint robustness via such diversity. An interesting study here would be to evaluate sensitivity to camera miscalibrations during pre-training, and to which extent off-the-shelf foundation models could be used to label available datasets and provide the required information. And, by extension, to which extent such models could be used as pre-training representations themselves, for distillation onto the visuomotor policy.

Another concern I have, that extends to many multi-view robustness papers, is the role of wrist cameras. They have been shown to consistently improve performance and also (and perhaps the reason why that's the case) to introduce some amount of viewpoint robustness. A very important question here then becomes: to which extent this method is still useful when wrist cameras are also being used? That would shed some light into the role that amodal pre-training can have in robotics when used in combination with other techniques also capable of achieving multi-view robustness.

Questions for authors

See my weakness section above:

  • How sensitive the proposed method is to camera miscalibration, and to which extent off-the-shelf foundation models can be used to provide such information for pre-training?
  • How scalable would this method be, compared to single-view or uncalibrated multi-view pre-training?
  • How much these results hold when wrist cameras are also taken into consideration?
4 comments
AuthorsOpenReview ↗

Response to Reviewer zQPf (1/2): Camera Calibration and Scalability

We thank the reviewer for the positive assessment and the insightful questions. We respond to each of them below.


Q1: How sensitive the proposed method is to camera miscalibration, and to which extent off-the-shelf foundation models can be used to provide such information for pre-training?

We thank the reviewer for raising this practical question. GEM3D is fundamentally a 3D policy learning method. Like most work in this area, including DP3 and ManiGaussian, it relies on reasonably accurate camera calibration, and our work is built on this assumption to some extent (see also the limitations in Sec. 6).

At the same time, we fully agree with the value of this question: most existing robot data come without depth, especially monocular data. During the rebuttal, we tested the off-the-shelf 3D foundation model VGGT-Ω for point-cloud estimation from uncalibrated observations (Fig. 9 and Sec. 5 of the revised PDF (in red)). Even this recent model produces noisy point clouds that deviate substantially from the ground truth, with an average Chamfer L1 of 0.349 m and a rotation error of 3.23°, together with clear scale bias and blurred object contours. We also tried training policies with such estimated point clouds, but they did not achieve any success on the downstream tasks.

We believe that further calibrating the estimated geometry may resolve this issue, but at present it remains difficult to rely on off-the-shelf foundation models to provide this information for pre-training. We consider this a promising direction for future work.


Q2: How scalable would this method be, compared to single-view or uncalibrated multi-view pre-training?

We thank the reviewer for this insightful question. We understand it as comparing our calibrated multi-view 3D pre-training with large-scale 2D pre-training on single-view or uncalibrated multi-view data, and we discuss it from three perspectives.

  • Explicit vs. implicit 3D supervision. Large-scale 2D pre-training can, in principle, learn spatial structure only implicitly: images are not related through a shared 3D frame, and regions occluded in the input view are never supervised. Our pre-training explicitly supervises the complete 3D scene, including the regions the input view cannot observe, which is what view generalization requires. Moreover, although single-view datasets are abundant, 2D datasets that capture the same scene from rich and diverse viewpoints are also scarce. Viewpoint diversity is therefore a bottleneck for both paradigms, rather than a limitation specific to ours.

  • Spatial Forcing as a direct comparison. Spatial Forcing takes the same single-view input and policy backbone as GEM3D, but distills representations from VGGT-Ω, a 3D foundation model pretrained at scale on uncalibrated multi-view images. In the real world, it improves training-view success over PerAct (41.7% → 45.0%), but falls below PerAct under viewpoint shifts (30.0% / 6.7% vs. 36.7% / 15.0% for moderate / large shifts), whereas GEM3D achieves 78.3% / 63.3% (Fig. 4; see the table below). Its representation is also less view-consistent than ours (MDS similarity 0.688 vs. 0.881, Fig. 8(b)). Scale alone therefore does not yield amodal, view-consistent representations.

Method (real world)TrainingModerateLarge
PerAct41.736.715.0
Spatial Forcing45.030.06.7
GEM3D88.378.363.3
  • Growing availability of 3D robot data. Open-source datasets with calibrated multi-view or RGB-D observations are growing rapidly, e.g., RH20T (110K+ episodes with calibrated multi-view RGB-D cameras; our pre-training uses 48 of its tasks), DROID (76K trajectories, 350 hours, with calibrated stereo cameras), and AgiBot World (1M+ trajectories, ~3,000 hours, with RGB-D cameras). We believe that how to make effective use of such 3D data will become an increasingly important question, and we hope GEM3D is a step in this direction.
AuthorsOpenReview ↗

Response to Reviewer zQPf (2/2): Wrist Cameras


Q3: How much do these results hold when wrist cameras are also taken into consideration?

We thank the reviewer for this valuable question. In the paper, we use a single third-person camera to simulate resource-constrained real-world deployment. Our method itself does not depend on the number of cameras: all observations are fused into a point cloud in a unified calibrated frame. Additional views, including a wrist camera, therefore make the observation more complete and are expected to benefit every method.

We have added this experiment on Open Drawer, Meat off Grill and Turn Tap (Table 4 and Sec. 5 of the revised PDF (in red), reproduced below), and the results confirm this expectation. With a wrist camera, PerAct improves substantially, and its success under moderate shifts nearly matches that at the training view (52.0% vs. 53.7%). GEM3D also benefits, reaching 69.0% / 63.0% / 59.0% at the training / moderate / large settings.

MethodWrist cameraTrainingModerateLarge
PerAct✗36.917.89.0
PerAct✓53.752.028.5
GEM3D✗54.741.127.4
GEM3D✓69.063.059.0

However, no matter how many cameras are added, incomplete observation cannot be fully avoided. For example, once the gripper grasps an object, the wrist camera is largely occluded by it, and eliminating all blind spots would require densely mounted cameras, which is impractical in real deployment. This is reflected under large shifts: PerAct with a wrist camera still drops to 28.5%, whereas GEM3D retains 59.0%; even without a wrist camera, GEM3D (27.4%) is on par with PerAct with one.

Moreover, GEM3D outperforms PerAct even at the training view (69.0% vs. 53.7% with a wrist camera, and 54.7% vs. 36.9% without). This is because our goal is not only view generalization, but learning amodal 3D representations that are better suited to 3D manipulation. We therefore believe that amodal pre-training remains complementary to additional cameras.

We would greatly appreciate it if you could take a look at our first-round revisions. Please let us know if you have any further comments or suggestions. We would be happy to make additional improvements and engage in further discussion.

AuthorsOpenReview ↗

Gentle Reminder: Discussion Period Ends on Nov 18 (AoE)

Thank you again for your positive assessment and insightful questions. As the author-reviewer discussion period ends on Nov 18 (AoE), we would like to kindly follow up. For your convenience, we briefly summarize our responses to your three questions:

  • Calibration and foundation models: we evaluated VGGT-Ω on uncalibrated observations (Fig. 9); its estimates (Chamfer L1 of 0.349 m) remain too coarse for reliable pretraining supervision.
  • Scalability: Spatial Forcing, which distills a 3D foundation model pretrained at scale on uncalibrated images, falls below PerAct under real-world viewpoint shifts (30.0% / 6.7%), whereas GEM3D achieves 78.3% / 63.3%, suggesting that scale alone does not yield amodal, view-consistent representations.
  • Wrist cameras: with a wrist camera, GEM3D still clearly outperforms PerAct, especially under large shifts (59.0% vs. 28.5%), and also at the training view (69.0% vs. 53.7%) (Table 4).

We would be grateful to hear whether these responses address your questions. If so, we would sincerely appreciate it if you would consider reflecting this in your final assessment. Thank you again for your time and effort.

Reviewer_K1bcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GEM3D, a 3D representation learning framework for cross-view generalization in robot manipulation. The method learns modality-agnostic 3D representations through complete point cloud reconstruction and Gaussian Splatting-based novel-view supervision, and then transfers pretrained geometric knowledge to a single-view manipulation policy via multi-step distillation. The authors evaluate in-domain performance and assess view generalization on a set of RLBench tasks, and further validate the method on real-world tasks. The results outperform several baselines, but the methodological novelty and the independent contribution of each component to the performance improvement are not sufficiently justified.

Critical strengths
  1. This paper proposes a relatively complete framework for 3D representation learning and policy distillation, which integrates single-view RGB-D input with robot manipulation and has certain practical application value.
  2. The authors conduct experiments in both simulation and real-robot environments, and evaluate generalization under unseen viewpoints. The experimental results demonstrate the effectiveness of the proposed method.
Critical weaknesses
  1. Limited novelty. This paper mainly combines existing techniques, including point cloud reconstruction, Snowflake point cloud upsampling, Gaussian Splatting, PerceiverIO, and feature distillation. The main novelty appears to lie in integrating these components into a unified framework, but the paper has not sufficiently clarified the substantive methodological innovation beyond existing methods. In particular, multi-step distillation is implemented only through feature alignment between the current timestep and the next timestep, and its independent contribution and necessity remain insufficiently justified.
  2. Insufficient empirical validation of the core contributions. Although the overall performance improves, the current experiments do not adequately disentangle whether the gains come from the modality-agnostic geometric representation, multi-view pretraining supervision, specific network design, or the distillation strategy. In particular, the ablation of the pretraining component mainly reports changes in reconstruction quality, without corresponding results on downstream manipulation performance and view generalization. Therefore, the paper has not yet convincingly demonstrated that its specific design choices are the key drivers of the performance improvement, rather than the combined effect of existing techniques.
Questions for authors
  1. Compared with existing methods, what is the core methodological innovation of GEM3D? In particular, what independent and irreplaceable contribution does multi-step distillation provide over single-step distillation?
  2. Could the authors conduct controlled experiments that isolate individual factors to disentangle the respective contributions of the modality-agnostic geometric representation, multi-view supervision, network design, and distillation strategy to manipulation success rate and view generalization?
4 comments
AuthorsOpenReview ↗

Response to Reviewer K1bc (1/2): Clarification of “Amodal” and Success-Rate Ablations

We sincerely thank the reviewer for the careful review. Before answering the questions, we would like to clarify one term. We noticed that the review describes our representation as “modality-agnostic”, and we may not have made our terminology clear enough. Amodal refers to inferring the complete scene structure, including occluded parts, from a partial observation. It does not mean modality-agnostic (i.e., independent of the input modality): GEM3D is designed for RGB-D inputs, and its core challenge is recovering unobserved 3D structure from a single view. We have made this definition more prominent in the revised PDF (in red).

We address each of the reviewer’s comments as follows:


Q1: Success-rate ablations of pretraining components.

We thank the reviewer for pointing this out. We indeed did not evaluate these components with task success. Following the setting of the paper, we pretrained each variant in Fig. 7 and trained the policy on three tasks (Open Drawer, Meat off Grill, Turn Tap). We additionally ablated Gaussian rendering under each number of pretraining views (8, 4, 2, 1). The results are reported in Table 2(c) and Table 2(a) of the revised PDF (in red); the success rates (%) of the pretraining variants are summarized below:

VariantTrainingModerateLarge
w/o DCA40.029.012.0
w/o Snowflake41.032.715.3
w/o Focal Loss48.031.717.7
GEM3D54.741.127.4

Removing any component lowers success under all three settings, and the relative drop is largest under large shifts (27.4% → 17.7% / 15.3% / 12.0% without focal loss / Snowflake / DCA). The ranking is consistent with the reconstruction analysis in Fig. 7, and specifically with geometric fidelity. Removing DCA or Snowflake, which recover local geometric detail and dense structure, damages the reconstructed point cloud most and causes the largest drops (15.4 and 12.1 points under large shifts). This indicates that better amodal geometry reconstruction translates into better policy learning, supporting that the pretraining learns task-useful representations. Likewise, disabling Gaussian rendering consistently lowers success with two or more pretraining views (e.g., 58.8 / 45.9 / 39.6 → 52.6 / 39.0 / 34.8 with 8 views), showing that rendering contributes beyond multi-view geometric supervision.

AuthorsOpenReview ↗

Response to Reviewer K1bc (2/2): Core Contribution and Multi-Step Distillation


Q2: What is the core methodological innovation of GEM3D?

We respectfully clarify that our contribution is not any individual module but a learning formulation: learning amodal 3D representations from single-view RGB-D, i.e., inferring the complete scene including regions the input view cannot observe.

In our experiments, GEM3D and the PerAct-based baselines share exactly the same policy network; they differ only in how auxiliary representation learning shapes the visual representation. ManiGaussian adds Gaussian rendering to PerAct, Spatial Forcing adds VGGT feature distillation to the same backbone, and GEM3D distills an amodal 3D representation, pretrained through reconstruction, into the policy. What distinguishes these methods is what their components are trained to learn, rather than any unique new component. Under large viewpoint shifts (Table D.6), neither ManiGaussian nor Spatial Forcing reaches GEM3D, nor does ManiGaussian trained with our distillation pipeline (14.5% / 13.4% / 20.1% vs. 29.5%), suggesting that the gain comes from the amodal learning target rather than from the modules themselves.

Each component serves this target, and removing any of them hurts policy success (Table 2(a)–(c) of the revised PDF (in red)):

  • Multi-view point cloud reconstruction supervises occluded geometry. With single-view supervision only, success under large shifts drops from 39.6 to 30.4.

  • Snowflake and DCA recover dense structure and local geometric detail from a compact set of 7³ seed queries; removing them causes the largest drops under large shifts (27.4% → 15.3% without Snowflake and 12.0% without DCA).

  • Gaussian rendering adds pixel-aligned cross-view supervision: since the Gaussians are centered at the predicted points, the photometric loss constrains both appearance and point positions, complementing the set-level Chamfer supervision. With two or more pretraining views, removing it consistently lowers success (e.g., 58.8 / 45.9 / 39.6 → 52.6 / 39.0 / 34.8 with 8 views).

  • Representation distillation transfers the amodal representation to the policy. Directly fine-tuning the pretrained encoder suffers from the representation gap between pretraining and downstream tasks, and success drops sharply (58.8 / 45.9 / 39.6 → 32.2 / 21.1 / 15.1, Table 2(b)). Distillation instead provides a soft, adaptive transfer, so that the learned amodal representation brings effective gains to the policy.

We hope this clarifies that the components are not stacked independently but jointly instantiate the amodal formulation, which we consider the main contribution of this work.


Q3: What independent and irreplaceable contribution does multi-step distillation provide over single-step distillation?

Our policy is built on a PerceiverIO backbone, which maps the input voxel tokens to output voxel tokens, and the action is decoded from the output tokens. Single-step distillation supervises only the input tokens. Multi-step distillation additionally aligns the output tokens with the teacher’s next-state features, so the tokens used for action prediction are also geometry-grounded. This adds no parameters and no inference cost. We have clarified this in Sec. 4.3 of the revised PDF (in red).

We agree that the three-task average gain shown in Table 2(b) is moderate. We therefore list the full per-task success rates at the training view here, hoping to show this more clearly:

TaskSingle-stepMulti-stepΔ
Phone on Base55.0 ± 5.667.3 ± 8.1+12.3 (+22.4%)
Turn Tap52.3 ± 4.062.0 ± 6.9+9.7 (+18.5%)
Meat on Grill56.3 ± 13.147.0 ± 3.0−9.3 (−16.5%)
Average54.658.8+4.2 (+7.7%)

**The moderate average gain is mainly due to a single task, Meat on Grill: multi-step distillation improves the other two tasks by about 10 points each (+11.0 on average). The drop on Meat on Grill is consistent with its trajectory being unseen during pretraining (Appendix C), which makes the teacher’s next-state targets less reliable. This task also requires high manipulation precision, which leads to high variance (±13.1) even for single-step distillation. Overall, in our tests, multi-step distillation brings clear gains on most tasks, so we retain this design.

We do not claim multi-step distillation is irreplaceable; it is a low-cost refinement on top of distillation.

We would greatly appreciate it if you could take a look at our first-round revisions. Please let us know if you have any further comments or suggestions. We would be happy to make additional improvements and engage in further discussion.

AuthorsOpenReview ↗

Gentle Reminder: Discussion Period Ends on Nov 18 (AoE)

As the author-reviewer discussion period ends on Nov 18 (AoE), we would like to kindly follow up on our responses. For your convenience, we briefly summarize how we addressed your main concerns:

  • Contribution of each component: we pretrained each variant and trained policies on three tasks; removing DCA, Snowflake, or focal loss lowers success under all settings, with up to a 56% relative drop under large shifts (Table 2(c)). Gaussian rendering also improves success under a controlled ablation with fixed multi-view supervision (Table 2(a)).
  • Novelty: all PerAct-based methods, including GEM3D, share the same policy network; under large viewpoint shifts, ManiGaussian, Spatial Forcing, and ManiGaussian with our distillation reach 14.5% / 13.4% / 20.1% vs. 29.5% for GEM3D, indicating that the gain comes from the amodal learning target rather than the modules themselves.
  • Multi-step distillation: it improves two of three tasks by about 10 points each, at no extra parameters or inference cost.

We would be very grateful if you could let us know whether these responses address your concerns, and we would sincerely appreciate it if you would consider them in your final assessment. Thank you again for your time and effort in reviewing our paper.

Comments on the paper

1 comment

Dear Reviewers,

Thanks for your time and efforts. Please have a look at the authors’ response and let the authors know whether it has addressed the concerns, as the rebuttal deadline is approaching.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.