- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces LEAP. Basically, it aims to improve RGB-based robotic visuomotor policies by using privileged 3D geometric supervision during training. The authors point out three potential issues with existing geometric supervision methods: proprioceptive shortcuts, reliance on dominant viewpoints, and task-irrelevant geometric information dominating the reconstruction objective. To tackle these problems, LEAP uses an auxiliary geometric decoder that only takes in visual features. It combines full point cloud reconstruction, wrist-view dropping, visibility-aligned partial point cloud reconstruction, and geometric supervision within workspace constraints. During deployment, the auxiliary decoder is removed, so the policy only relies on RGB observations and the robot's state. The authors evaluated it on various robot manipulation tasks. The results show that LEAP achieves a significant success rate boost compared to the baselines, and ablation studies confirm that the different design components contribute to the performance. While the paper proposes a geometric supervision framework with practical value, its novelty and the validation of its core mechanisms are still worth a closer look.
Critical strengths
1.The authors tested their method on a bunch of simulation benchmarks and real-world robot tasks, and even applied it to two different policy architectures. This shows the approach actually works in practice, it's not just some idea on paper. 2. With LEAP, they use point cloud reconstruction as a side goal during training, but they totally ditch the geometric decoder when it's time for inference, so there's no need for extra point cloud input. This is a smart move because it means you don't have to rely on explicit 3D data when deploying the robot, but you still get the benefits of using geometry to boost visual learning. On top of that, their ablation studies break down how things like view dropping and target alignment contribute to the results, which really backs up why they designed the method this way.
Critical weaknesses
-
Limited Novelty. LEAP combines several reasonable components, including visual-only geometric reconstruction, view dropout, visibility-aligned partial reconstruction, and workspace cropping. However, the paper does not clearly establish what substantial new capabilities this combination offers beyond existing geometric supervision and multi-view learning methods.
-
Insufficient Support for the Core Claims. The theoretical propositions establish potential limitations of reconstruction objectives but do not demonstrate that the claimed shortcuts actually arise during training or that LEAP effectively resolves them. The empirical diagnostics are also insufficient to establish the proposed mechanisms: input perturbations do not conclusively demonstrate proprioceptive shortcuts, and frozen-feature object-position prediction does not directly verify complementary geometric learning across views. Moreover, the task-relevance analysis relies on pixel-wise averaging, whereas the proposed method uses point-cloud Chamfer Distance and workspace cropping. The paper does not adequately explain how these designs address insufficient supervision of task-relevant local geometry.
Questions for authors
- Regarding the method's novelty. Compared to existing methods for multi-view reconstruction, view dropping, and auxiliary geometric supervision, what exactly is the core innovation of LEAP? Can the paper more clearly point out what kind of synergy exists among full reconstruction, visibility-aligned partial reconstruction, and workspace constraints, something that can't be directly achieved by existing methods? This point will directly affect how I judge the paper's novelty and overall contribution.
- Regarding the empirical validation of the core mechanisms. The current experiments mainly analyze proprioceptive shortcuts and view dominance through reconstruction error and object position prediction with frozen visual features. Do these results directly prove that LEAP has learned more useful visual-geometric representations, and that the wrist view truly provides complementary information to the policy? Specifically, can the authors provide more direct evidence from their existing experiments to distinguish whether the performance boost comes from the specific mechanisms proposed in the paper, or simply from an overall improvement in visual representation quality? This issue will affect my evaluation of how well the method is technically justified.
3 comments
Rebuttal Part (2/3)
(2) The proprioceptive-shortcut interpretation is supported by more than input perturbations.
State-conditioned reconstruction can be more accurate while producing weaker visual geometric representations. On all five tasks, perturbing state causes greater task-object coverage degradation than perturbing visual features, even after matching perturbation severity.
| Intervention | Matched severity | State perturbation: RMSE increase | Visual perturbation: RMSE increase |
|---|---|---|---|
| Interpolation toward training mean | 0.8 | 38.984 mm | 12.373 mm |
| Cross-trajectory shuffling | 1.0 | 48.949 mm | 17.786 mm |
These tests use independent held-out trajectories and match dimension-normalized, per-coordinate standardized displacement. Shuffling results average 20 permutations. Matching displacement does not guarantee equal semantic information removal, so these measurements indicate decoder reliance rather than proving a harmful shortcut on their own.
The independent visual probes provide the second piece of evidence. Visual-only auxiliary training reduces mean Z-only probe Chamfer error from 12.422 to 8.587, in units of , a 30.9% reduction, with improvements on all five tasks. For this comparison, we freeze the encoders and train identical fresh probes using alone, with independent validation and test trajectories.
The state-conditioned model nevertheless has lower native reconstruction error on average. It therefore reconstructs more accurately using both inputs while encoding less recoverable geometry in its visual features. This is a trained-model example of why low state-conditioned reconstruction error does not establish effective visual supervision.
The paired policy ablation in Table 3 in the main text also shows higher success when state is excluded from reconstruction: 58.2% → 76.6% for DP and 36.8% → 49.0% for ACT. Both conditions retain proprioception for action prediction.
(3) The policy evaluations test whether wrist observations benefit control.
Wrist observations provide incremental control benefits, and LEAP’s gains extend beyond ordinary geometric supervision. Across all ten task–backbone combinations, LEAP performs better when either wrist is added to the head view, and improves further when both wrists are available.
| Backbone | Method | Head only | Head + both wrists | Incremental wrist benefit |
|---|---|---|---|---|
| DP | Full-only | 49.0% | 55.2% | +6.2 pp |
| DP | LEAP | 56.6% | 76.6% | +20.0 pp |
| ACT | Full-only | 33.6% | 38.6% | +5.0 pp |
| ACT | LEAP | 38.4% | 49.0% | +10.6 pp |
These controlled view evaluations provide policy-level evidence that the additional wrist observations are useful. Frozen-feature object-position prediction alone could not establish this.
The existing controls also distinguish LEAP’s gains from the general benefit of geometric supervision. Reaggregating results on the same five tasks gives 21.2% → 55.2% → 76.6% success for DP without geometry supervision, with ordinary visual-only full reconstruction, and with LEAP. Ordinary reconstruction helps, and LEAP adds a further 21.4 percentage points.
Target alignment makes an additional difference. In Table 3 in the main text, retaining dropout but removing alignment reduces success from 76.6% to 62.0% for DP and 49.0% to 44.4% for ACT. Visual-only supervision and dropout remain present in these controls.
LEAP also improves head-only performance, so general representation improvements can coexist with better use of wrist observations. The incremental wrist gains are consistent with complementary information useful for control.
Rebuttal Part (3/3)
(4) Task relevance is connected directly to the Chamfer objective.
Workspace cropping concentrates supervision on relevant geometry and improves its reconstruction under a fixed point budget. Cropping increases the mean fraction of annotated task-object points in the full target from 5.7% to 55.1%. Mean common-workspace coverage RMSE falls from 26.821 to 11.873 mm, a 55.7% reduction, and task-object coverage improves in all task/target comparisons.
| Method | Task-object points (%) ↑ | Workspace coverage RMSE (mm) ↓ | Task-object coverage RMSE (mm) ↓ | Success (%) ↑ |
|---|---|---|---|---|
| DP-LEAP w/o workspace crop | 5.74 | 26.82 | 24.25 | 65.2 |
| DP-LEAP with workspace crop | 55.06 | 11.87 | 10.92 | 76.6 |
These comparisons use the same 1,024-point budget, matched training settings, and common reference clouds. They therefore measure accuracy on the same geometry rather than comparing native losses computed against different targets.
The connection to the actual reconstruction objective can be stated explicitly. Specifically, we partition the target cloud into task-relevant and remaining points. If is the task-point fraction, the Chamfer objective decomposes as where , up to an overall normalization factor. Since all terms are nonnegative, guarantees only . Thus, when task points occupy a small fraction of the target, low global reconstruction loss can coexist with substantial task-region error. Cropping increases the task-point share under a fixed point budget, giving relevant geometry greater weight in supervision. The detailed derivation is provided in revised Section 4.1.
We added all the additional experiments and the discussions above in the draft.
Rebuttal Part (1/3)
Thank you for your careful review and constructive feedback. We address your concerns and questions point by point below and have revised the manuscript accordingly, marked in blue.
W1 & Q1: Limited novelty and unclear added capability.
Thank you for highlighting the need to clarify both the novelty and the added capability of LEAP. LEAP's contribution lies in analyzing how privileged geometric supervision can fail to train the intended visual representations, and developing a supervision formulation guided by this analysis.
These choices serve complementary roles. Visual-only reconstruction excludes direct proprioceptive access, but can still permit head-view dominance. View dropout changes the available observations, but does not itself specify the local geometry to emphasize. LEAP therefore couples pre-fusion masking with a partial target approximately visible from the retained wrist. Full reconstruction preserves overall workspace context, while the changing partial target supplies local geometric tasks. Workspace constraints give both objectives a common spatial focus. All objectives train the visual encoder shared with action prediction.
The added functionality is geometric supervision that follows the available camera evidence while jointly preserving workspace context and emphasizing local geometry. This relationship is not supplied by fixed-target reconstruction or generic camera dropout alone. We regard workspace cropping as a supporting operation, rather than a standalone algorithmic invention.
The experimental evidence supports this formulation:
Together, they support the complementary roles and practical value of the joint formulation. We make this contribution clearer in Section 1, 4, 5 of the revised manuscript.
W2 & Q2: Insufficient evidence for the proposed mechanisms.
Thank you for asking us to connect the proposed mechanisms more directly to learned representations and policy behavior. We have added reconstruction diagnostics, frozen visual-feature probes, and policy-level view interventions to examine the proposed mechanisms more directly.
(1) The propositions establish limitations of the objective, not inevitable training outcomes.
The propositions show that low reconstruction error alone cannot establish that visual features encode useful geometry or that multiple views contribute. They do not imply that every training run develops these shortcuts. Similarly, excluding proprioception from the auxiliary decoder blocks its direct state-based reconstruction route, but better visual representations and control performance must be established experimentally.