acceptodds
Under review as a conference paper at ICLR 2027

LEAP: Making Privileged Geometry Supervision Effective for Visuomotor Learning

Abstract

Privileged 3D supervision uses additional geometric information during training to guide RGB-based visuomotor policy learning, without requiring geometric inputs at deployment. However, low reconstruction error does not ensure that visual representations capture geometry useful for control. We identify three limitations that weaken this supervision: proprioceptive shortcuts, dominant-view reliance, and reconstruction objectives dominated by task-irrelevant geometry. To address these limitations, we propose Latent Encoding with Aligned Privileged Geometry (LEAP). Our framework uses an auxiliary decoder to reconstruct point clouds within the manipulation workspace from visual features alone, while retaining proprioception for action prediction. Alongside full reconstruction, we introduce wrist-view dropout and partial reconstruction targets matched to the retained wrist, encouraging the encoder to capture complementary local geometry. The auxiliary decoder is removed at inference, leaving only RGB observations and robot state as policy inputs. Extensive experiments on RoboTwin, ManiSkill, and real-world tasks demonstrate consistent and substantial improvements over Diffusion Policy and ACT, with only a small increase in parameter count and no reduction in inference speed.

Primary area:
applications to robotics, autonomy, planning
Keywords:
3D Robotic Manipulation, Imitation Learning, Geometry Supervision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7vhROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces LEAP. Basically, it aims to improve RGB-based robotic visuomotor policies by using privileged 3D geometric supervision during training. The authors point out three potential issues with existing geometric supervision methods: proprioceptive shortcuts, reliance on dominant viewpoints, and task-irrelevant geometric information dominating the reconstruction objective. To tackle these problems, LEAP uses an auxiliary geometric decoder that only takes in visual features. It combines full point cloud reconstruction, wrist-view dropping, visibility-aligned partial point cloud reconstruction, and geometric supervision within workspace constraints. During deployment, the auxiliary decoder is removed, so the policy only relies on RGB observations and the robot's state. The authors evaluated it on various robot manipulation tasks. The results show that LEAP achieves a significant success rate boost compared to the baselines, and ablation studies confirm that the different design components contribute to the performance. While the paper proposes a geometric supervision framework with practical value, its novelty and the validation of its core mechanisms are still worth a closer look.

Critical strengths

1.The authors tested their method on a bunch of simulation benchmarks and real-world robot tasks, and even applied it to two different policy architectures. This shows the approach actually works in practice, it's not just some idea on paper. 2. With LEAP, they use point cloud reconstruction as a side goal during training, but they totally ditch the geometric decoder when it's time for inference, so there's no need for extra point cloud input. This is a smart move because it means you don't have to rely on explicit 3D data when deploying the robot, but you still get the benefits of using geometry to boost visual learning. On top of that, their ablation studies break down how things like view dropping and target alignment contribute to the results, which really backs up why they designed the method this way.

Critical weaknesses
  1. Limited Novelty. LEAP combines several reasonable components, including visual-only geometric reconstruction, view dropout, visibility-aligned partial reconstruction, and workspace cropping. However, the paper does not clearly establish what substantial new capabilities this combination offers beyond existing geometric supervision and multi-view learning methods.

  2. Insufficient Support for the Core Claims. The theoretical propositions establish potential limitations of reconstruction objectives but do not demonstrate that the claimed shortcuts actually arise during training or that LEAP effectively resolves them. The empirical diagnostics are also insufficient to establish the proposed mechanisms: input perturbations do not conclusively demonstrate proprioceptive shortcuts, and frozen-feature object-position prediction does not directly verify complementary geometric learning across views. Moreover, the task-relevance analysis relies on pixel-wise averaging, whereas the proposed method uses point-cloud Chamfer Distance and workspace cropping. The paper does not adequately explain how these designs address insufficient supervision of task-relevant local geometry.

Questions for authors
  1. Regarding the method's novelty. Compared to existing methods for multi-view reconstruction, view dropping, and auxiliary geometric supervision, what exactly is the core innovation of LEAP? Can the paper more clearly point out what kind of synergy exists among full reconstruction, visibility-aligned partial reconstruction, and workspace constraints, something that can't be directly achieved by existing methods? This point will directly affect how I judge the paper's novelty and overall contribution.
  2. Regarding the empirical validation of the core mechanisms. The current experiments mainly analyze proprioceptive shortcuts and view dominance through reconstruction error and object position prediction with frozen visual features. Do these results directly prove that LEAP has learned more useful visual-geometric representations, and that the wrist view truly provides complementary information to the policy? Specifically, can the authors provide more direct evidence from their existing experiments to distinguish whether the performance boost comes from the specific mechanisms proposed in the paper, or simply from an overall improvement in visual representation quality? This issue will affect my evaluation of how well the method is technically justified.
3 comments
AuthorsOpenReview ↗

Rebuttal Part (1/3)

Thank you for your careful review and constructive feedback. We address your concerns and questions point by point below and have revised the manuscript accordingly, marked in blue.

W1 & Q1: Limited novelty and unclear added capability.

Thank you for highlighting the need to clarify both the novelty and the added capability of LEAP. LEAP's contribution lies in analyzing how privileged geometric supervision can fail to train the intended visual representations, and developing a supervision formulation guided by this analysis.

These choices serve complementary roles. Visual-only reconstruction excludes direct proprioceptive access, but can still permit head-view dominance. View dropout changes the available observations, but does not itself specify the local geometry to emphasize. LEAP therefore couples pre-fusion masking with a partial target approximately visible from the retained wrist. Full reconstruction preserves overall workspace context, while the changing partial target supplies local geometric tasks. Workspace constraints give both objectives a common spatial focus. All objectives train the visual encoder shared with action prediction.

The added functionality is geometric supervision that follows the available camera evidence while jointly preserving workspace context and emphasizing local geometry. This relationship is not supplied by fixed-target reconstruction or generic camera dropout alone. We regard workspace cropping as a supporting operation, rather than a standalone algorithmic invention.

The experimental evidence supports this formulation:

  • Workspace constraints: With the same 1,024-point target budget, the new cropping ablation reduces mean common-workspace coverage RMSE from 26.821 to 11.873 mm, a 55.7% reduction.
  • Visual geometric learning: In the new frozen-encoder comparison, visual-only training reduces Z-only probe Chamfer error by 30.9% relative to state-conditioned training, improving on all five tasks despite the state-conditioned decoder having lower average native reconstruction error.
  • Benefit beyond ordinary reconstruction: Reaggregating the existing controls on the same five tasks gives 21.2% → 55.2% → 76.6% success for no geometry supervision, full-only visual reconstruction, and LEAP. Thus, LEAP adds 21.4% beyond ordinary reconstruction.
  • Importance of target matching: Table 3 in the main text further shows that retaining dropout but removing target alignment reduces success from 76.6% to 62.0% for DP and 49.0% to 44.4% for ACT.

Together, they support the complementary roles and practical value of the joint formulation. We make this contribution clearer in Section 1, 4, 5 of the revised manuscript.

W2 & Q2: Insufficient evidence for the proposed mechanisms.

Thank you for asking us to connect the proposed mechanisms more directly to learned representations and policy behavior. We have added reconstruction diagnostics, frozen visual-feature probes, and policy-level view interventions to examine the proposed mechanisms more directly.

(1) The propositions establish limitations of the objective, not inevitable training outcomes.

The propositions show that low reconstruction error alone cannot establish that visual features encode useful geometry or that multiple views contribute. They do not imply that every training run develops these shortcuts. Similarly, excluding proprioception from the auxiliary decoder blocks its direct state-based reconstruction route, but better visual representations and control performance must be established experimentally.

AuthorsOpenReview ↗

Rebuttal Part (2/3)

(2) The proprioceptive-shortcut interpretation is supported by more than input perturbations.

State-conditioned reconstruction can be more accurate while producing weaker visual geometric representations. On all five tasks, perturbing state causes greater task-object coverage degradation than perturbing visual features, even after matching perturbation severity.

InterventionMatched severityState perturbation: RMSE increaseVisual perturbation: RMSE increase
Interpolation toward training mean0.838.984 mm12.373 mm
Cross-trajectory shuffling1.048.949 mm17.786 mm

These tests use independent held-out trajectories and match dimension-normalized, per-coordinate standardized displacement. Shuffling results average 20 permutations. Matching displacement does not guarantee equal semantic information removal, so these measurements indicate decoder reliance rather than proving a harmful shortcut on their own.

The independent visual probes provide the second piece of evidence. Visual-only auxiliary training reduces mean Z-only probe Chamfer error from 12.422 to 8.587, in units of 10−4 m210^{-4}\,\mathrm{m}^{2}, a 30.9% reduction, with improvements on all five tasks. For this comparison, we freeze the encoders and train identical fresh probes using ZZ alone, with independent validation and test trajectories.

The state-conditioned model nevertheless has lower native reconstruction error on average. It therefore reconstructs more accurately using both inputs while encoding less recoverable geometry in its visual features. This is a trained-model example of why low state-conditioned reconstruction error does not establish effective visual supervision.

The paired policy ablation in Table 3 in the main text also shows higher success when state is excluded from reconstruction: 58.2% → 76.6% for DP and 36.8% → 49.0% for ACT. Both conditions retain proprioception for action prediction.

(3) The policy evaluations test whether wrist observations benefit control.

Wrist observations provide incremental control benefits, and LEAP’s gains extend beyond ordinary geometric supervision. Across all ten task–backbone combinations, LEAP performs better when either wrist is added to the head view, and improves further when both wrists are available.

BackboneMethodHead onlyHead + both wristsIncremental wrist benefit
DPFull-only49.0%55.2%+6.2 pp
DPLEAP56.6%76.6%+20.0 pp
ACTFull-only33.6%38.6%+5.0 pp
ACTLEAP38.4%49.0%+10.6 pp

These controlled view evaluations provide policy-level evidence that the additional wrist observations are useful. Frozen-feature object-position prediction alone could not establish this.

The existing controls also distinguish LEAP’s gains from the general benefit of geometric supervision. Reaggregating results on the same five tasks gives 21.2% → 55.2% → 76.6% success for DP without geometry supervision, with ordinary visual-only full reconstruction, and with LEAP. Ordinary reconstruction helps, and LEAP adds a further 21.4 percentage points.

Target alignment makes an additional difference. In Table 3 in the main text, retaining dropout but removing alignment reduces success from 76.6% to 62.0% for DP and 49.0% to 44.4% for ACT. Visual-only supervision and dropout remain present in these controls.

LEAP also improves head-only performance, so general representation improvements can coexist with better use of wrist observations. The incremental wrist gains are consistent with complementary information useful for control.

AuthorsOpenReview ↗

Rebuttal Part (3/3)

(4) Task relevance is connected directly to the Chamfer objective.

Workspace cropping concentrates supervision on relevant geometry and improves its reconstruction under a fixed point budget. Cropping increases the mean fraction of annotated task-object points in the full target from 5.7% to 55.1%. Mean common-workspace coverage RMSE falls from 26.821 to 11.873 mm, a 55.7% reduction, and task-object coverage improves in all task/target comparisons.

MethodTask-object points (%) ↑Workspace coverage RMSE (mm) ↓Task-object coverage RMSE (mm) ↓Success (%) ↑
DP-LEAP w/o workspace crop5.7426.8224.2565.2
DP-LEAP with workspace crop55.0611.8710.9276.6

These comparisons use the same 1,024-point budget, matched training settings, and common reference clouds. They therefore measure accuracy on the same geometry rather than comparing native losses computed against different targets.

The connection to the actual reconstruction objective can be stated explicitly. Specifically, we partition the target cloud into task-relevant and remaining points. If α\alpha is the task-point fraction, the Chamfer objective decomposes as LCD=Lprediction→target+αLtask+(1−α)Lrest.L_{\mathrm{CD}} = L_{\mathrm{prediction}\rightarrow\mathrm{target}} +\alpha L_{\mathrm{task}} +(1-\alpha)L_{\mathrm{rest}}. where α=∣Grel∣/∣G∣\alpha=|G_{\mathrm{rel}}|/|G|, up to an overall normalization factor. Since all terms are nonnegative, LCD≤ϵL_{\mathrm{CD}}\leq\epsilon guarantees only Ltask≤ϵ/αL_{\mathrm{task}}\leq\epsilon/\alpha. Thus, when task points occupy a small fraction of the target, low global reconstruction loss can coexist with substantial task-region error. Cropping increases the task-point share under a fixed point budget, giving relevant geometry greater weight in supervision. The detailed derivation is provided in revised Section 4.1.

We added all the additional experiments and the discussions above in the draft.

Reviewer_2JNZOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The basic idea of the paper is a robotics policy trained in the following way: you have a model that takes images X and robot state S as input. It produces a combined visual feature Z. The visual feature has heads that predict both the next action (either VAE chunk or diffusion policy; they try both) and a point cloud of the scene. The point cloud is supervised via an auxiliary loss since this information is known at train time. Then at test time the model can operate without point clouds but has presumably learned a strong geometric prior such that the performance is improved.

Critical strengths

The idea is sound and benchmarks (RoboTwin, ManiSkill, their own real-world benchmark) are impressive. The discussions of how to ensure the model actually learns geometric features (as I understand it buy dropping out views) are appreciated. Another strength is that the DP3 baseline has access to the privileged point cloud information at test time but performs comparable to LEAP.

Critical weaknesses

I feel the system is somewhat contrived. Given how good multiview reconstruction is getting, it probably makes more sense to simply have a modular pipeline that (1) deploys the best available multiview 3D reconstruction algorithm, ie DUSTR/MAST3R or follow-on work and (2) deploys the best available 3D-conditioned robot policy such as DP3.

It would be great if the authors could clarify what they perceive to be the key advantages of unifying everything like this versus a more modular approach.

Questions for authors

How many demonstrations per task for ManiSkill? The paper seems inconsistent on this point.

1 comment

Thank you for your careful review and constructive feedback. We address your concerns and questions point by point below and have revised the manuscript accordingly, marked in blue.

W1: Insufficient justification for the integrated approach.

We agree that multi-view reconstruction followed by a 3D-conditioned policy is a valid alternative. However, this introduces a considerably heavier deployment stack: a multi-view reconstruction network must operate alongside the point-cloud encoder and action policy. Beyond the reconstruction model’s parameters, multi-view attention and dense geometric predictions require additional activation memory and intermediate storage. Point-cloud post-processing, such as filtering and downsampling where needed, adds further computational cost and latency before action prediction.

LEAP retains the benefits of multi-view observations while avoiding these reconstruction-related deployment costs. Geometry supervises the shared visual encoder during training, and the reconstruction decoder is discarded afterwards. At deployment, the policy directly consumes multi-view RGB observations and robot state, without generating, processing and re-encoding an explicit point cloud. This transfers geometric knowledge into the policy’s visual representation while keeping the deployment pipeline lightweight. This distinction is particularly relevant when GPU memory and closed-loop inference time are constrained, which is the usual case during the deployment of robotic policy. LEAP integrates geometric supervision during training, rather than requiring a separate reconstruction system to remain in the control loop. We modified the draft for this discussion in Section 1.

Q1: ManiSkill demonstration count

Thank you for pointing out the inconsistency. We use 100 expert demonstrations per task for ManiSkill. We have corrected this reporting error in the revised manuscript.

Reviewer_QWbqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the failure of privileged geometry supervision in RGB-based visuomotor learning: low reconstruction error cannot guarantee that visual representations truly encode the geometry required for control tasks. The authors summarize three core limitations — proprioceptive shortcuts, dominant-view reliance, and reconstruction objectives dominated by task-irrelevant geometry — and formally prove the non-identifiability of the first two via two propositions.

To address these limitations, the authors propose LEAP (Latent Encoding with Aligned Privileged Geometry), a training framework with three corresponding designs: visual-only geometric reconstruction, wrist-view dropout with visibility-aligned partial reconstruction, and workspace-constrained geometric supervision. 3D information is used to supervise the visual encoder during training, while the auxiliary reconstruction branch is removed at inference, leaving only RGB images and robot state as inputs. Extensive experiments on RoboTwin, ManiSkill benchmarks and real-world tasks demonstrate consistent performance improvements across both Diffusion Policy and ACT backbones.

Critical strengths

Clear cognitive reference value. This work is the first to systematically decompose three failure modes of privileged geometry supervision, correcting the default cognitive bias that "low reconstruction loss equals high-quality geometric representations" in the field. It provides a new evaluation perspective and problem awareness for subsequent method design in this direction, and positively promotes the privileged supervision subfield.

Solid experimental validation with strong practicality. The experiments cover simulation and real-world scenarios, single-arm and bimanual tasks, and two mainstream policy backbones, with large scale and highly consistent performance improvements. The method itself is lightweight, brings no inference overhead, can be plugged into existing visuomotor learning frameworks, and has clear engineering application value.

Critical weaknesses

Insufficient core technical originality with limited academic increment. None of the three core designs are originally proposed in this paper: visual-only reconstruction is a classic paradigm in 3D vision, view dropout has long been used to improve multi-view robustness, and workspace cropping is a common engineering preprocessing step in robotic point cloud processing. This work essentially reuses existing technologies to address specific failure modes of privileged supervision, which belongs to combinatorial innovation.

Incomplete argumentation for core claims with rigor gaps. "Task-irrelevant geometry diluting supervision signals" is listed as one of the three core limitations, but it has neither corresponding formal theoretical support nor ablation verification of "removing workspace cropping". It is impossible to prove that this problem truly exists under the experimental settings, nor to quantify the independent performance contribution of the corresponding design. One third of the core arguments lack direct empirical support, which directly affects the argumentation rigor and contribution credibility of the work.

Questions for authors

Please clarify which of the three core designs are original technologies first proposed in this paper, and which are targeted reuse of existing technologies? How to define the core innovation increment of this work compared with existing visual-only geometric supervision methods?

Why is the ablation experiment of "removing workspace cropping and using full-scene point clouds" not provided? Is there experimental data to verify the core hypothesis that "task-irrelevant geometry dilutes supervision signals", and to quantify the independent performance contribution of workspace cropping?

The diagnostic experiment in Figure 2(a) assumes equivalent perturbation intensity for proprioceptive state SS and visual features ZZ, but does not verify the consistency of their distribution dispersion. The perturbation severity is not comparable between low-dimensional state vectors and high-dimensional visual features. Could you supplement the statistical characteristics of SS and ZZ, or provide comparison results under equivalent perturbation intensity to support the inference of this experiment?

This paper only mentions GAP and Robo3R as representative works of proprioceptive shortcuts. Why are works such as GeoProp and Vysics that also use proprioceptive state to assist geometric reconstruction not discussed? Do such methods also suffer from the supervision failure problem identified in this paper?

Why is there no horizontal performance comparison with more representative visual-only geometric supervision methods? How to prove that the performance advantage of LEAP comes from its unique designs rather than the general benefit of visual-only geometric supervision?

Will you replace figures and charts with vector format in the revised version to improve the presentation quality for publication?

3 comments
AuthorsOpenReview ↗

Rebuttal Part (1/3)

Thank you for your careful review and constructive feedback. We address your concerns and questions point by point below and have revised the manuscript accordingly, marked in blue.

W1 & Q1: Limited technical originality.

Thank you for highlighting the need to clarify our contribution. We acknowledge that visual-only reconstruction, view dropout, and workspace cropping are established techniques. LEAP’s contribution lies in analyzing how privileged geometric supervision can fail to train the intended visual representations, and developing a supervision formulation guided by this analysis.

First, regarding visual-only reconstruction, we examine the distinction between accurate geometric prediction and effective geometric supervision of a policy’s visual encoder. Proposition 4.1 formalizes how state-conditioned reconstruction can achieve low error even when visual features contain insufficient geometric information. This motivates separating the two pathways: proprioception remains available for action prediction, while the geometry decoder receives only visual features. Our contribution here is the analysis and its implication for reconstruction-based policy learning.

Second, for the view-dropout design, our central contribution is the coupling of view dropout with visibility-aligned partial reconstruction. Masking views alone does not specify which local geometry the retained wrist should encode. LEAP masks features before cross-view fusion and constructs a partial target approximately visible from the retained wrist, using camera projection and depth filtering. Changing the retained wrist therefore changes both the available visual evidence and its corresponding geometric target. Together with full reconstruction, this provides supervision for overall scene structure and local wrist-visible geometry.

Third, workspace cropping supports this target formulation. Explicit point clouds in a common world frame allow us to select the manipulation workspace and construct view-aligned subsets of the same geometric target. We regard cropping as standard preprocessing supporting this formulation, rather than an independently novel algorithm.

Importantly, Table 3 in the main text directly evaluates the contribution beyond dropout itself: retaining wrist dropout while removing target alignment reduces average success from 76.6% to 62.0% for DP and 49.0% to 44.4% for ACT. These results support the practical value of matching geometric targets to the retained observations. We revised the contribution statement to distinguish this methodological contribution and our analytical findings from the established implementation components.

W2 & Q2: Insufficient support for the task-irrelevant geometry claim

MethodTask-object points (%) ↑Workspace coverage RMSE (mm) ↓Task-object coverage RMSE (mm) ↓Success (%) ↑
DP-LEAP w/o workspace crop5.7426.8224.2565.2
DP-LEAP with workspace crop55.0611.8710.9276.6

We agree that the original manuscript lacked a direct cropping ablation. We now remove cropping from both reconstruction targets while keeping all other components fixed.

Specifically, we partition the target cloud into task-relevant and remaining points. If α\alpha is the task-point fraction, the Chamfer objective decomposes as LCD=Lprediction→target+αLtask+(1−α)Lrest.L_{\mathrm{CD}} = L_{\mathrm{prediction}\rightarrow\mathrm{target}} +\alpha L_{\mathrm{task}} +(1-\alpha)L_{\mathrm{rest}}. Since all terms are nonnegative, LCD≤ϵL_{\mathrm{CD}}\leq\epsilon guarantees only Ltask≤ϵ/αL_{\mathrm{task}}\leq\epsilon/\alpha. Thus, when task points occupy a small fraction of the target, low global reconstruction loss can coexist with substantial task-region error. Cropping increases the task-point share under a fixed point budget, giving relevant geometry greater weight in supervision. The detailed derivation is provided in revised Section 4.1.

In the five-task ablation, cropping increases the task-object point fraction from 5.74% to 55.06%, reduces task-object coverage RMSE from 24.25 to 10.92 mm (55.0%), and improves policy success from 65.2% to 76.6% (+11.4 percentage points). These results support the supervision-allocation motivation and quantify cropping’s incremental contribution within LEAP. Full settings and results are provided in revised Table 3, Table 7 and Appendix D.2.

AuthorsOpenReview ↗

Rebuttal Part (2/3)

Q3: Validity of Figure 2(a)

We agree that applying the same intervention to SS and ZZ does not guarantee comparable perturbation severity. We therefore added a severity-matched diagnostic on independent held-out trajectories across five tasks.

At the decoder interfaces, SS and ZZ contain 42 and 1,536 coordinates, respectively. We measure perturbation severity using dimension-normalized, per-coordinate standardized displacement: ρX=1NdX∑i,j(X~ij−XijσX,j)2,\rho_X= \sqrt{\frac{1}{Nd_X}\sum_{i,j} \left(\frac{\widetilde X_{ij}-X_{ij}}{\sigma_{X,j}}\right)^2}, where σX,j\sigma_{X,j} is estimated from training inputs only. Mean replacement and cross-trajectory shuffling are interpolated to match ρ\rho between modalities. We perturb only the selected decoder input, keeping the other modality, reconstruction target, and camera geometry unchanged.

InterventionMatched ρ\rhoObject coverage RMSE increase: perturb SS (mm)Perturb ZZ (mm)
Mean replacement0.838.98412.373
Shuffling1.048.94917.786

Values are five-task macro-averages; shuffling averages 20 permutations. State interventions cause greater degradation on all five tasks under both intervention types. Thus, the observed asymmetry persists after matching standardized displacement. This normalization does not imply equal semantic information removal or eliminate joint-distribution shifts.

We additionally assess whether decoder dependence accompanies weaker visual geometry learning. After freezing each visual encoder, we also train identical probes using ZZ alone:

Auxiliary decoder inputNative CD ↓Z-only probe CD ↓Paired policy success ↑ (%)
Z,SZ,S — state-conditioned3.07812.42258.2
ZZ — visual-only3.4348.58776.6

CD values are in 10−4m210^{-4} \mathrm{m}^2; native reconstruction and probes use 1,024- and 256-point targets, respectively.

Although state conditioning yields lower native reconstruction error on average, excluding state reduces the visual-only probe CD by 30.9%, with improvements on all five tasks, and improves paired policy success by 18.4 percentage points. Together, these results support our interpretation that low state-conditioned reconstruction error can coexist with weaker visual-geometric representations. We updated it in the Section 4.1 and Appendix D.3 of the revised manuscript.

Q4: Related-work coverage and applicability

Thank you for pointing out these relevant works. Our original discussion focused on state-conditioned geometric reconstruction, but we agree that the broader discussion of vision–proprioception integration should include GeoProp and Vysics.

GeoProp aligns proprioception with vision for action prediction, without auxiliary geometric reconstruction. LEAP’s reconstruction-supervision failure therefore does not directly apply, although broader proprioceptive over-reliance remains possible.

Vysics supplements RGB-D observations with contact-dynamics constraints derived from robot proprioception and observed object motion to recover occluded geometry. This differs from LEAP’s visual-only auxiliary reconstruction setting, whose purpose is to supervise the policy’s visual encoder. Therefore, using this additional physical information does not itself indicate the supervision failure studied in LEAP.

Our concern is specifically whether auxiliary geometric prediction provides effective supervision for the policy’s visual representation. LEAP excludes state from this reconstruction pathway while retaining it for action prediction. Our analysis identifies a possible failure mode, rather than implying that every method using proprioception suffers from it. Furthermore, our diagnostic uses a controlled state-conditioned DP baseline; it does not establish that the published implementations of GAP, Robo3R, GeoProp or Vysics exhibit the same behavior. We made this scope explicit in Section 2 of the draft.

AuthorsOpenReview ↗

Rebuttal Part (3/3)

Q5: Comparison with visual-only supervision methods

Thank you for raising this concern. Our experiments address it through both targeted ablations and comparisons with recent geometric representation learning methods.

Internal ablations. Table 3 in the main text compares LEAP with variants that retain visual-only geometric supervision while removing specific designs. Removing wrist dropout reduces average success from 76.6% to 55.2% for DP and from 49.0% to 38.6% for ACT. Retaining dropout but removing target alignment yields 62.0% and 44.4%, respectively. Since both variants still use visual-only geometric supervision, these comparisons directly support the additional benefits of wrist dropout and visibility-aligned targets, beyond simply introducing geometric supervision.

External comparisons. Table 1 in the main text includes two recent representative approaches: Spatial Forcing, which supervises visual representations through alignment with VGGT features during policy training, and eVGGT, which transfers geometrically distilled visual representations into the policy. These cover complementary approaches to learning geometry-aware visual features. Across the 50 RoboTwin tasks, DP-LEAP achieves 56.5% average success, compared with 42.8% for DP-SF and 33.0% for DP-eVGGT.

The ablations isolate the added value of specific LEAP designs, while the external comparisons establish its effectiveness relative to existing approaches. We clarified these complementary roles in the revised manuscript.

Q6: Presentation quality

Thank you for the suggestion. We updated the figures and charts to vector format in the revised manuscript to improve clarity and readability.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Summary

Dear Area Chair and Reviewers,

Thank you for the constructive feedback. During the rebuttal period, we addressed all reviewer concerns by clarifying LEAP’s contributions in the manuscript and adding controlled experiments to evaluate how its design choices affect visual geometry learning and policy performance. As the rebuttal comes to the end, we want to give a summary:

  1. Contribution beyond visual-only reconstruction. We analyze how low reconstruction error can coexist with weak visual geometry learning. LEAP couples wrist-view dropout before feature fusion with a partial target matched to the retained wrist’s visibility, while full reconstruction preserves workspace context. Across the same five tasks, Diffusion Policy achieves 21.2%, 55.2%, and 76.6% success with no geometric supervision, full-only visual reconstruction, and LEAP, respectively. With dropout held fixed, target alignment improves success from 62.0% to 76.6% for Diffusion Policy and 44.4% to 49.0% for ACT (Table 3 in the draft). Table 1 provides complementary RoboTwin-50 comparisons with Spatial Forcing and eVGGT.

  2. Proprioceptive reliance and visual geometry. We added two perturbation tests on held-out trajectories, matching the standardized displacement of state and visual features. Perturbing state degraded reconstruction more on all five tasks under both interventions. In a separate experiment, identical probes trained on frozen visual features achieved 30.9% lower Chamfer error with visual-only training, even though the state-conditioned decoder had lower reconstruction error with its original inputs (Section 4.1 and Appendix D.3).

  3. Wrist-view contribution to control. We added policy evaluations with the head view alone, the head view plus either wrist, and all three views. Adding both wrists to the head view improved success by 20.0 vs. 6.2 percentage points for Diffusion Policy and 10.6 vs. 5.0 points for ACT, comparing LEAP with full-only reconstruction. Either wrist alone also improved LEAP over head-only input in all ten task–backbone combinations.

  4. Task-relevant geometry under the Chamfer objective. The revised analysis shows how the fraction of task-relevant points determines their weight in the target-to-prediction term. In the new ablation with a fixed 1,024-point budget, workspace cropping increased the task-object share from 5.74% to 55.06%, reduced task-object coverage RMSE from 24.25 to 10.92 mm, and raised Diffusion Policy success from 65.2% to 76.6% (Section 4.1, Tables 3 and 7, and Appendix D.2).

We also clarified the deployment trade-off with a modular reconstruction-plus-policy pipeline. LEAP removes the reconstruction decoder after training and runs directly on RGB and robot state, avoiding online reconstruction and point-cloud processing. We expanded the discussion of GeoProp and Vysics and corrected the reported ManiSkill setting to 100 expert demonstrations per task.

Thank you again for your suggestions on our work.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.