- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Action Map Policy (AMP), which casts closed-loop 3D manipulation as dense pixel classification. AMP predicts multi-view heatmaps of future end-effector keypoints and recovers action chunks through triangulation and geometric pose reconstruction. The method is evaluated on six simulated and five real-world manipulation tasks.
Critical strengths
1.Clear formulation with strong spatial grounding. AMP provides an intuitive geometric connection between visual observations and action predictions. Its spatially aligned representation preserves fine-grained visual cues and enables joint image–label augmentation. The laser-pointer experiments effectively demonstrate its responsiveness to small, task-relevant signals.
2.Strong empirical performance and efficient inference. AMP outperforms the evaluated baselines on five of six simulated tasks and achieves 80–90% success on three long-horizon real-world tasks. Its single-pass prediction achieves a 13.80 ms end-to-end latency, including geometric action recovery—approximately 6.8× faster than the evaluated 16-step Diffusion Policy. Supporting ablations and analyses further strengthen the empirical evaluation.
Critical weaknesses
AMP depends on at least two calibrated cameras and sufficient shared workspace coverage, increasing deployment requirements and restricting its operating range. Although the paper acknowledges this limitation and Appendix A.9 analyzes triangulation errors under calibration noise, the impact on closed-loop task success remains unclear. This limits confidence in its practical robustness when camera calibration or placement changes.
Questions for authors
-
Attribution of improvements. What evidence distinguishes the contribution of the cross-entropy objective from that of the heatmap representation and spatial architecture? A comparison between cross-entropy and heatmap MSE under otherwise identical settings would help clarify this distinction.
-
Scope of multimodality. Does the multimodality claim refer to individual keypoint distributions or the joint distribution over complete action chunks? While Appendix A.9 provides useful consistency evidence, please clarify how separate argmax decoding maintains a consistent trajectory mode when multiple valid modes coexist.
-
Robustness to Calibration Drift and Camera Placement Changes. How sensitive is closed-loop task success to calibration drift? If camera placement changes, is recalibration sufficient to maintain performance, or is additional policy training required? A focused experiment on one representative precision task, reporting success rates under controlled calibration perturbations with the policy fixed, would help quantify the practical tolerance of the method.
4 comments
[Part 2/2]
Q3. Robustness to camera placement changes and calibration drift.
- Camera placement changes. We thank the reviewer for this suggestion. We conducted a new experiment: we shifted one external camera (the front view) by around 15 cm and only recalibrated it, without retraining the policy. On the precision-demanding real-world Toast task, AMP achieves 85% (17/20), close to its original 90% (18/20). In the failed trials, the policy completed most of the task and only failed to fully press the toaster button. In contrast, Diffusion Policy and ACT implicitly absorb the camera parameters into their network weights, so the effect of a camera change cannot be separated from the learned policy. Under the same shift, Diffusion Policy drops from 40% to 10% and ACT from 35% to 15%. This shows that the potential for novel-view generalization induced by our formulation, together with explicit calibration, gives AMP the advantage of adapting to a new camera setup quickly and easily.
- Calibration drift. As stated in Sect. 4.4, AMP performs well in the real world despite imperfect off-the-shelf camera calibration and extrinsic perturbations caused by table vibration, suggesting its practical robustness to mild drift. Appendix A.9 further quantifies how calibration noise affects triangulation errors. Accurate calibration is a shared requirement of many prior robot learning policies (e.g., Motion Tracks, PerAct, RVT-2, 3D Diffuser Actor), and large calibration errors are a sensor-maintenance issue rather than a property of the learned policy. When large drift occurs, the experiment above shows that a quick recalibration restores performance without retraining.
W1. Calibrated multi-view requirement and operating range. We hope the following points help alleviate this concern.
- Prior work makes the same assumption. Motion Tracks, one of our baselines, requires two cameras with known extrinsics. Multiple calibrated cameras have long been widely used in prior policy learning (e.g., Transporter Networks, PerAct, RVT-2, Act3D, 3D Diffuser Actor). Besides, DROID, a commonly used large-scale real-world dataset, is collected with two calibrated external cameras plus a wrist camera. This calibration requirement is stated in in Sect. 5, and the calibration itself is a one-time procedure shared across tasks.
- Side views are our experimental choice, not a requirement. AMP is a new policy learning framework: its classification formulation, keypoint heatmap representation, and fast single-pass inference do not depend on where the cameras are placed. It extends easily to other camera setups, e.g., two in-hand cameras, whose poses follow from forward kinematics and hand-eye calibration. Since in-hand cameras move with the end-effector, their shared coverage follows the robot rather than a fixed workspace. This flexibility gives AMP a path to plug-and-play deployment without external cameras, rather than limiting it to a particular hardware setup.
- Explicit calibration gives AMP the advantage of adapting to a new camera setup quickly and easily, as shown in our response to Q3 above.
We hope these responses address the reviewer's concerns, and we would be grateful if the reviewer could consider updating the score accordingly.
Thank you for the thoughtful responses and additional experiments. I continue to appreciate the clear geometric formulation, strong spatial grounding, and efficient single-pass inference of AMP. The new cross-entropy versus heatmap-MSE comparison strengthens the evidence for the training objective. The camera-relocation experiment is also valuable: maintaining 85% success after moving one camera and recalibrating, without policy retraining, provides encouraging evidence of adaptability under the tested change.
My remaining reservation concerns the practical requirements of the camera setup. AMP still requires at least two calibrated cameras with sufficient overlapping coverage of the predicted action keypoints. This constrains camera placement and the usable workspace, as also reflected by the handling of out-of-view action targets in the simulation experiments. Although similar assumptions are used by prior methods, they remain a meaningful deployment limitation.
The new experiment partially addresses this concern, but evaluates one camera displacement followed by recalibration. It does not directly quantify closed-loop robustness to uncorrected calibration drift or establish performance across substantially different camera configurations. The proposed extension to dual in-hand cameras is promising, but remains unvalidated in the reported experiments. I would therefore encourage the authors to distinguish the demonstrated flexibility from these potential extensions.
Overall, I view AMP as a valuable contribution with convincing results within its evaluated setup. The rebuttal strengthens my positive assessment, while the remaining hardware and workspace constraints limit my confidence in broader deployment. I therefore maintain my original rating.
We sincerely thank the reviewer for the thoughtful follow-up and the positive assessment, and we will clearly distinguish in the paper the demonstrated flexibility from potential extensions, as suggested.
[Part 1/2]
We thank the reviewer for the constructive feedback. Please see our responses below.
Q1. Attribution of the cross-entropy loss. We thank the reviewer for this suggestion. We add a new ablation that replaces the cross-entropy objective with an MSE loss on the heatmaps, under otherwise identical settings. For a fair comparison, we rescale the soft Gaussian target so that its peak is 1. With the original sum-to-one target, the peak is only about 1/(2πσ2)≈0.04, so the target is almost entirely zero and the network easily collapses to predicting all zeros under MSE.
Replacing cross-entropy with MSE causes an average drop of 21.3 points. We attribute this gap to how cross-entropy treats the heatmap. Cross-entropy, which equals the KL divergence between the target and predicted distributions over pixels up to a constant, has a gradient of p−q on each logit. It heavily penalizes missing any target mode, and its gradient is near zero on background pixels, so training is not dominated by them. MSE regresses each pixel independently, so its loss could be dominated by background pixels and only weakly penalizes missing a mode, which could lead to blurry peaks and less precise localization.
Q2. Scope of multimodality. We thank the reviewer for requesting this clarification. Our multimodality claim refers to the action distribution of human demonstrations, which is multimodal. For example, for each of the Toast, Egg, and Coffee tasks, the policy needs to learn the complex distribution of 70 human demonstrations. AMP represents this distribution with individual keypoint heatmaps rather than an explicit joint distribution over action chunks, but all heatmaps are predicted jointly and decoded consistently, as follows.