acceptodds
Under review as a conference paper at ICLR 2027

Action Map Policy: Learning 3D Closed-loop Manipulation via Pixel Classification

Abstract

The action space poses a major challenge in robot learning, since it is often high-dimensional, can span long time horizons, and frequently admits multi-modal optimal solutions. A good choice of action representation and loss function can help to address these concerns, but there are often trade-offs. We propose Action Map Policy (AMP), which casts 3D closed-loop manipulation policy learning as a classification problem in image space. While classification has been an effective formulation in generative language models, applying it to robot action learning is difficult because naively discretizing high-dimensional continuous actions explodes the token vocabulary. Our key idea is to project 3D actions onto the camera image planes and treat each pixel location as a discrete class, thus controlling dimensionality while retaining multi-modality. This method supports millimeter-level precision for high-dimensional actions without requiring a prohibitively large vocabulary, while preserving fine-grained pixel-wise visual signals. Furthermore, it can predict the entire action chunk in a single forward pass, avoiding complex noise scheduling and iterative denoising while achieving substantially faster inference. Experiments on six simulated and five real-world manipulation tasks show that AMP outperforms strong baselines, improving the completion rate by 50%–70% on long-horizon real-world tasks, while running faster than Diffusion Policy and exhibiting stronger spatial reasoning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Manipulation Policy Learning, Action Representation, Imitation Learning for Robotics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wJxcOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Action Map Policy (AMP), which casts closed-loop 3D manipulation as dense pixel classification. AMP predicts multi-view heatmaps of future end-effector keypoints and recovers action chunks through triangulation and geometric pose reconstruction. The method is evaluated on six simulated and five real-world manipulation tasks.

Critical strengths

1.Clear formulation with strong spatial grounding. AMP provides an intuitive geometric connection between visual observations and action predictions. Its spatially aligned representation preserves fine-grained visual cues and enables joint image–label augmentation. The laser-pointer experiments effectively demonstrate its responsiveness to small, task-relevant signals.

2.Strong empirical performance and efficient inference. AMP outperforms the evaluated baselines on five of six simulated tasks and achieves 80–90% success on three long-horizon real-world tasks. Its single-pass prediction achieves a 13.80 ms end-to-end latency, including geometric action recovery—approximately 6.8× faster than the evaluated 16-step Diffusion Policy. Supporting ablations and analyses further strengthen the empirical evaluation.

Critical weaknesses

AMP depends on at least two calibrated cameras and sufficient shared workspace coverage, increasing deployment requirements and restricting its operating range. Although the paper acknowledges this limitation and Appendix A.9 analyzes triangulation errors under calibration noise, the impact on closed-loop task success remains unclear. This limits confidence in its practical robustness when camera calibration or placement changes.

Questions for authors
  1. Attribution of improvements. What evidence distinguishes the contribution of the cross-entropy objective from that of the heatmap representation and spatial architecture? A comparison between cross-entropy and heatmap MSE under otherwise identical settings would help clarify this distinction.

  2. Scope of multimodality. Does the multimodality claim refer to individual keypoint distributions or the joint distribution over complete action chunks? While Appendix A.9 provides useful consistency evidence, please clarify how separate argmax decoding maintains a consistent trajectory mode when multiple valid modes coexist.

  3. Robustness to Calibration Drift and Camera Placement Changes. How sensitive is closed-loop task success to calibration drift? If camera placement changes, is recalibration sufficient to maintain performance, or is additional policy training required? A focused experiment on one representative precision task, reporting success rates under controlled calibration perturbations with the policy fixed, would help quantify the practical tolerance of the method.

4 comments

[Part 1/2]

We thank the reviewer for the constructive feedback. Please see our responses below.

Q1. Attribution of the cross-entropy loss. We thank the reviewer for this suggestion. We add a new ablation that replaces the cross-entropy objective with an MSE loss on the heatmaps, under otherwise identical settings. For a fair comparison, we rescale the soft Gaussian target so that its peak is 1. With the original sum-to-one target, the peak is only about 1/(2πσ2)≈0.041/(2\pi\sigma^2) \approx 0.04, so the target is almost entirely zero and the network easily collapses to predicting all zeros under MSE.

MethodStack Three D1Hammer Cleanup D1Coffee D2
AMP (cross-entropy)908878
MSE instead of cross-entropy (new)72 (↓18)64 (↓24)56 (↓22)

Replacing cross-entropy with MSE causes an average drop of 21.3 points. We attribute this gap to how cross-entropy treats the heatmap. Cross-entropy, which equals the KL divergence between the target and predicted distributions over pixels up to a constant, has a gradient of p−qp - q on each logit. It heavily penalizes missing any target mode, and its gradient is near zero on background pixels, so training is not dominated by them. MSE regresses each pixel independently, so its loss could be dominated by background pixels and only weakly penalizes missing a mode, which could lead to blurry peaks and less precise localization.

Q2. Scope of multimodality. We thank the reviewer for requesting this clarification. Our multimodality claim refers to the action distribution of human demonstrations, which is multimodal. For example, for each of the Toast, Egg, and Coffee tasks, the policy needs to learn the complex distribution of 70 human demonstrations. AMP represents this distribution with individual keypoint heatmaps rather than an explicit joint distribution over action chunks, but all heatmaps are predicted jointly and decoded consistently, as follows.

  1. The heatmaps of all keypoints at all timesteps are decoded from the shared multi-view features of X-Net. The decoder further mixes channels across all keypoints and timesteps: the K × T heatmaps (K keypoints, T timesteps) are stacked as channels and processed jointly by convolutional layers, so that each heatmap is predicted with awareness of the others.
  2. AMP models multimodal actions with multi-peak heatmaps. Each peak corresponds to one mode in the demonstrations, and its height reflects the weight of that mode. Since all heatmaps share the same mode weights, the dominant mode yields the highest peak in every heatmap, so the per-heatmap argmax is aligned. The visualized trajectories in Figure 11 (Appendix A.6) illustrate this. In our real-world experiments, near-ties appear to be rare, since the observation and robot state usually reveal which mode to follow. Even when a near-tie occurs, the new observation could break the tie once the robot moves toward one mode. This is supported by the evidence below.
  3. As the reviewer noted, Appendix A.9 provides consistency evidence: on the validation sets of all five real-world tasks, the MSE between the triangulated and ground-truth action chunks (horizon of 12 steps) is only 1×10−41\times10^{-4}–2×10−42\times10^{-4}, which would be much larger if the per-view predictions selected different modes. In addition, the real-world rollout videos in the supplementary material provide direct qualitative evidence, since inconsistent predictions would yield erroneous triangulated actions. In our videos, the rollouts remain smooth, and we observe no inconsistent motion. We also tried an explicit consistency loss (Jensen–Shannon divergence between the predictions of two consecutive steps), which brought no further improvement (Appendix A.9), suggesting that consistency is learned implicitly.

[Part 2/2]

Q3. Robustness to camera placement changes and calibration drift.

  1. Camera placement changes. We thank the reviewer for this suggestion. We conducted a new experiment: we shifted one external camera (the front view) by around 15 cm and only recalibrated it, without retraining the policy. On the precision-demanding real-world Toast task, AMP achieves 85% (17/20), close to its original 90% (18/20). In the failed trials, the policy completed most of the task and only failed to fully press the toaster button. In contrast, Diffusion Policy and ACT implicitly absorb the camera parameters into their network weights, so the effect of a camera change cannot be separated from the learned policy. Under the same shift, Diffusion Policy drops from 40% to 10% and ACT from 35% to 15%. This shows that the potential for novel-view generalization induced by our formulation, together with explicit calibration, gives AMP the advantage of adapting to a new camera setup quickly and easily.
  2. Calibration drift. As stated in Sect. 4.4, AMP performs well in the real world despite imperfect off-the-shelf camera calibration and extrinsic perturbations caused by table vibration, suggesting its practical robustness to mild drift. Appendix A.9 further quantifies how calibration noise affects triangulation errors. Accurate calibration is a shared requirement of many prior robot learning policies (e.g., Motion Tracks, PerAct, RVT-2, 3D Diffuser Actor), and large calibration errors are a sensor-maintenance issue rather than a property of the learned policy. When large drift occurs, the experiment above shows that a quick recalibration restores performance without retraining.

W1. Calibrated multi-view requirement and operating range. We hope the following points help alleviate this concern.

  1. Prior work makes the same assumption. Motion Tracks, one of our baselines, requires two cameras with known extrinsics. Multiple calibrated cameras have long been widely used in prior policy learning (e.g., Transporter Networks, PerAct, RVT-2, Act3D, 3D Diffuser Actor). Besides, DROID, a commonly used large-scale real-world dataset, is collected with two calibrated external cameras plus a wrist camera. This calibration requirement is stated in in Sect. 5, and the calibration itself is a one-time procedure shared across tasks.
  2. Side views are our experimental choice, not a requirement. AMP is a new policy learning framework: its classification formulation, keypoint heatmap representation, and fast single-pass inference do not depend on where the cameras are placed. It extends easily to other camera setups, e.g., two in-hand cameras, whose poses follow from forward kinematics and hand-eye calibration. Since in-hand cameras move with the end-effector, their shared coverage follows the robot rather than a fixed workspace. This flexibility gives AMP a path to plug-and-play deployment without external cameras, rather than limiting it to a particular hardware setup.
  3. Explicit calibration gives AMP the advantage of adapting to a new camera setup quickly and easily, as shown in our response to Q3 above.

We hope these responses address the reviewer's concerns, and we would be grateful if the reviewer could consider updating the score accordingly.

Thank you for the thoughtful responses and additional experiments. I continue to appreciate the clear geometric formulation, strong spatial grounding, and efficient single-pass inference of AMP. The new cross-entropy versus heatmap-MSE comparison strengthens the evidence for the training objective. The camera-relocation experiment is also valuable: maintaining 85% success after moving one camera and recalibrating, without policy retraining, provides encouraging evidence of adaptability under the tested change.

My remaining reservation concerns the practical requirements of the camera setup. AMP still requires at least two calibrated cameras with sufficient overlapping coverage of the predicted action keypoints. This constrains camera placement and the usable workspace, as also reflected by the handling of out-of-view action targets in the simulation experiments. Although similar assumptions are used by prior methods, they remain a meaningful deployment limitation.

The new experiment partially addresses this concern, but evaluates one camera displacement followed by recalibration. It does not directly quantify closed-loop robustness to uncorrected calibration drift or establish performance across substantially different camera configurations. The proposed extension to dual in-hand cameras is promising, but remains unvalidated in the reported experiments. I would therefore encourage the authors to distinguish the demonstrated flexibility from these potential extensions.

Overall, I view AMP as a valuable contribution with convincing results within its evaluated setup. The rebuttal strengthens my positive assessment, while the remaining hardware and workspace constraints limit my confidence in broader deployment. I therefore maintain my original rating.

We sincerely thank the reviewer for the thoughtful follow-up and the positive assessment, and we will clearly distinguish in the paper the demonstrated flexibility from potential extensions, as suggested.

Reviewer_MaiGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AMP, which transforms 3D closed-loop robot manipulation policy learning into a pixel classification problem in image space. The method represents the end-effector as a set of keypoints, predicts keypoint heatmaps from multi-view images, and recovers 3D actions via triangulation. The model is capable of predicting complete action sequences in a single forward pass, thereby avoiding the iterative denoising process required by diffusion policies. Evaluated on six simulation tasks and five real-world tasks, the results demonstrate that the proposed method is highly competitive in terms of task success rate, inference speed, and spatial reasoning capabilities.

Critical strengths
  1. Strong performance in real-world experiments. AMP achieves impressive success rates across various real-world tasks, clearly outperforming the baselines reported in the paper. Additionally, its inference latency is significantly lower than that of Diffusion Policy, demonstrating strong potential for practical, real-time deployment.
  2. Practical value in action representation. By representing actions as keypoint heatmaps aligned with the image space, AMP effectively preserves fine-grained spatial information and leverages multi-view geometry to recover 3D actions. The laser-pointing experiment also nicely demonstrates its ability to perform spatial reasoning using subtle visual cues.
Critical weaknesses
  1. The core novelty could be more distinct. Concepts like image-space keypoint prediction, heatmap representation, and multi-view geometry recovery have been explored in prior robot manipulation research. AMP’s main contribution lies in combining these ideas with pixel classification objectives, soft-label supervision, and single-pass action sequence prediction. However, the paper does not fully clarify what essential algorithmic innovations this combination brings compared to existing methods, leaving its originality somewhat open to discussion.
  2. Concerns regarding multimodal modeling and generalization. While pixel classification can represent multimodal keypoint distributions, predicting heatmaps for each keypoint and timestep separately does not necessarily guarantee the joint consistency of the full action trajectory. Furthermore, the method relies on a calibrated multi-camera system, and action prediction is constrained by the camera's field of view. Although the current results are convincing, there is insufficient validation regarding its generalization to complex occlusions, out-of-view actions, and a broader range of tasks.
Questions for authors
  1. Compared with existing image-space keypoint prediction and heatmap-based action representations, what is the core algorithmic novelty of AMP? Does the performance gain mainly come from the pixel classification objective, cross-view joint modeling, or equivariant data augmentation?
  2. How do the authors ensure that heatmap predictions across different keypoints and timesteps form geometrically consistent and temporally coherent action trajectories? Is there a direct experimental evaluation of the model’s ability to capture the joint distribution of multimodal actions?
  3. Could checkpoint selection and random seed choice affect the reported success rates? Can the authors provide statistics from multiple independent runs to further validate the stability of the observed performance improvements?
3 comments

[Part 1/2]

We thank the reviewer for the constructive feedback. Please see our responses below.

W1/Q1-a. Core algorithmic novelty. We appreciate the efforts of prior and concurrent work on image-space keypoint actions and heatmap-based action learning. In Sect. 2 (Related Work), we discuss the differences between our method and related methods in detail. To better clarify our novelty, we summarize the key differences here. Image-space keypoint actions have been explored, but not with a pixel-classification objective (e.g., Motion Tracks denoises 2D keypoint tracks with diffusion and also lacks cross-view consistency). Heatmap-based methods (e.g., PerAct, RVT-2) use heatmaps only to localize the translation, while rotation and gripper state require separate prediction heads. Moreover, these heatmaps are defined either in voxel space (PerAct) or in re-rendered orthographic views (RVT-2), both of which require depth cameras, whereas AMP predicts heatmaps directly on raw RGB images. Finally, these methods only predict sparse keyposes, which are reached by a motion planner rather than through high-frequency closed-loop control.

To the best of our knowledge, AMP is the first closed-loop imitation learning method that learns keypoint distributions to represent the full action, i.e., the 6-DoF end-effector pose and gripper width. AMP stands out among popular closed-loop policy learning frameworks by combining strong performance with fast single-pass inference. To realize this, we design X-Net for cross-view consistency and an equivariant data augmentation for n-equivariance across cameras. We also adopt the standard soft-label trick for the classification objective. Overall, we regard AMP as a new imitation learning framework with original contributions, novel extensions of prior ideas, and principled combinations of existing techniques.

Q1-b. Source of the performance gains. We thank the reviewer for the suggestion to identify the source of the gains. Table 3 in our submission shows that removing the equivariant data augmentation leads to a large drop. We additionally run two new ablations:

  1. Pixel classification objective. We replace the cross-entropy objective with an MSE loss on the heatmaps. For a fair comparison, we rescale the soft Gaussian target so that its peak is 1. With the original sum-to-one target, the peak is only about 1/(2πσ2)≈0.041/(2\pi\sigma^2) \approx 0.04, so the target is almost entirely zero and the network easily collapses to predicting all zeros under MSE.
  2. Per-view attention instead of cross-view attention. We replace the cross-view attention layers with in-image attention and supervise each view independently, which completely removes information sharing across cameras.

All variants use the same training and evaluation protocol.

MethodStack Three D1Hammer Cleanup D1Coffee D2
AMP908878
w/o equivariant data aug. (Table 3)58 (↓32)62 (↓26)66 (↓12)
w/o pixel classification (MSE loss, new)72 (↓18)64 (↓24)56 (↓22)
Per-view attention (new)64 (↓26)56 (↓32)56 (↓22)

On average, removing the equivariant augmentation causes a drop of 23.3 points, and removing cross-view attention causes a drop of 26.7 points. Besides, the pixel classification objective brings a 21.3-point improvement over MSE. These differences are nearly an order of magnitude larger than the seed-to-seed standard deviation (at most 2.8 points; see our response to Q3). These results further validate our three designs: the pixel classification objective, cross-view attention, and equivariant data augmentation. Other ablations, on the in-hand camera and soft-label classification, are reported in Table 3 in our submission.

[Part 2/2]

Q2. Consistency under multimodality and its evaluation. We thank the reviewer for this insightful question.

  1. The heatmaps of all keypoints at all timesteps are decoded from the shared multi-view features of X-Net. The decoder further mixes channels across all keypoints and timesteps: the K × T heatmaps (K keypoints, T timesteps) are stacked as channels and processed jointly by convolutional layers, so that each heatmap is predicted with awareness of the others.
  2. AMP models multimodal actions with multi-peak heatmaps. Each peak corresponds to one mode in the demonstrations, and its height reflects the weight of that mode. Since all heatmaps share the same mode weights, the dominant mode yields the highest peak in every heatmap, so the per-heatmap argmax is aligned. The visualized trajectories in Figure 11 (Appendix A.6) illustrate this. In our real-world experiments, near-ties appear to be rare, since the observation and robot state usually reveal which mode to follow. Even when a near-tie occurs, the new observation could break the tie once the robot moves toward one mode. This is supported by the evidence below.
  3. Sect. 4.4 and Appendix A.9 evaluate the resulting consistency. (1) The real-world rollout videos in the supplementary material provide direct qualitative evidence, since inconsistent predictions would yield erroneous triangulated actions. In our videos, the rollouts remain smooth, and we observe no inconsistent motion. (2) We quantitatively measure the consistency on the validation sets of all five real-world tasks. The MSE between the triangulated and ground-truth action chunks (horizon of 12 steps; 6D rotation, translation in meters, and gripper width) is only 1×10−41\times10^{-4}–2×10−42\times10^{-4}. Since inconsistent per-view predictions would lead to much larger triangulation errors, this indicates that the maxima of the learned multimodal distributions are geometrically consistent in our experiments. (3) We also tried an explicit consistency loss (Jensen–Shannon divergence between the predictions of two consecutive steps), which brought no further improvement, suggesting that temporal consistency is learned implicitly.

Q3. Checkpoint selection and random seeds. We thank the reviewer for this suggestion. To clarify, the simulation experiments include two independent training runs with different random seeds, with each checkpoint evaluated on 50 unseen episodes. We additionally provide the mean and sample standard deviation across the top-3 evaluations of each run (2 seeds × 3 checkpoints, 6 evaluations in total):

Stack Three D1Hammer Cleanup D1Mug Cleanup D1Coffee D2Square D2Threading D2
AMP87.7±2.387.7 \pm 2.385.7±2.085.7 \pm 2.051.0±1.751.0 \pm 1.774.7±2.174.7 \pm 2.147.0±2.847.0 \pm 2.829.0±1.129.0 \pm 1.1

The results show that the std is at most 2.8 points on every task, well below AMP's gap to the strongest baseline. Moreover, the top-3 mean is within 3.3 points of the best-checkpoint results (reported in the paper for all methods), so checkpoint selection has a limited effect.

We hope these responses address the reviewer's concerns, and we would be grateful if the reviewer could consider updating the score accordingly.

Thank you again for your constructive review. As the discussion period is coming to an end, we would like to briefly check whether our responses have addressed your concerns. In summary, we:

(1) added two new ablations that isolate the contributions of the pixel classification objective and cross-view attention, complementing the existing ablation of the equivariant data augmentation in Table 3;

(2) reported the variance across seeds and checkpoints, showing that checkpoint selection has a limited effect;

(3) clarified how AMP keeps predictions consistent across keypoints and timesteps under multimodal demonstrations, also supported by our existing consistency evaluation results;

(4) explained the core novelty of AMP relative to prior image-space keypoint actions and heatmap-based methods: AMP learns keypoint distributions with a pixel-classification objective to represent the full action chunk directly on RGB images for high-frequency closed-loop control (fast single-pass inference).

We would be happy to address any remaining questions.

Reviewer_TgBrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary
  1. The action representation is well motivated. Casting action chunks as dense keypoint heatmaps turns policy learning into a classification problem with a stable cross-entropy objective, while still recovering full 3D end-effector poses via triangulation.

  2. The single-forward-pass inference is substantially faster than Diffusion Policy and close to ACT, and the precision analysis gives a concrete argument for why heatmap classification can achieve fine spatial resolution without an enormous discrete action vocabulary.

  3. The empirical results show large gains on spatially demanding tasks, including fine-grained visual cue use. AMP reports strong improvements on MimicGen insertion and articulated-object tasks and on real-world long-horizon tasks.

Critical strengths

(1) The action representation is well motivated. Casting action chunks as dense keypoint heatmaps turns policy learning into a classification problem with a stable cross-entropy objective, while still recovering full 3D end-effector poses via triangulation.

(2) The empirical results show large gains on spatially demanding tasks, including fine-grained visual cue use. AMP reports strong improvements on MimicGen insertion and articulated-object tasks and on real-world long-horizon tasks.

Critical weaknesses
  1. The method’s practical applicability is limited by its dependence on calibrated multi-view coverage, and the experimental protocol may favor AMP in ways that complicate fair comparison. AMP requires at least two calibrated side-view cameras, predicts heatmaps only for the side views, and truncates training actions when keypoints leave the camera view; baselines are trained on the original, untruncated actions.

(2) The empirical evidence is relatively narrow and lacks statistical rigor, especially for the real-world claims. Real-world evaluation uses only 20 runs per task, no variance or confidence intervals are reported, and the real-world baselines are limited to Diffusion Policy and ACT. The simulated baselines are trained for 800 epochs while AMP is trained for 200 epochs, and several tables in the provided manuscript are malformed or difficult to verify.

Questions for authors

Please refer to paper weaknes

4 comments

[Part 1/2]

We thank the reviewer for the constructive feedback. Please see our responses below.

W1-a. Side-view requirement and camera calibration.

  1. Side views are our experimental choice, not a hard requirement. AMP is a new policy learning framework: its classification formulation, keypoint heatmap representation, and fast single-pass inference do not depend on where the cameras are placed. It extends easily to other camera setups, e.g., two in-hand cameras, whose poses follow from forward kinematics and hand-eye calibration. This flexibility gives AMP a path to plug-and-play deployment without external cameras, rather than limiting it to a particular hardware setup.

  2. Prior work makes the same assumption. Motion Tracks, one of our baselines, requires two cameras with known extrinsics. Multiple calibrated cameras have long been widely used in prior policy learning (e.g., Transporter Networks, PerAct, RVT-2, Act3D, 3D Diffuser Actor). Besides, DROID, a commonly used large-scale real-world dataset, is collected with two calibrated external cameras plus a wrist camera. We explicitly state this calibration requirement in Sect. 5, and the calibration itself is a one-time procedure shared across tasks. We therefore believe that calibrated multi-view sensing is a common setup in policy learning, rather than a limitation specific to AMP.

  3. Explicit calibration gives AMP the advantage of adapting to a new camera setup quickly and easily. We conducted a new experiment: we shifted one external camera (the front view) by 15 cm and only recalibrated it, without retraining the policy. On the precision-demanding real-world Toast task, AMP achieves 85% (17/20), close to its original 90% (18/20). In the failed trials, the policy completed most of the task and only failed to fully press the toaster button. In contrast, Diffusion Policy and ACT implicitly absorb the camera parameters into their network weights, so the effect of a camera change cannot be separated from the learned policy. Under the same shift, Diffusion Policy drops from 40% to 10% and ACT from 35% to 15%.

W1-b. Action truncation and fair comparison. As explained in Sect. 4.2, truncation avoids out-of-view keypoints. It only removes supervision from AMP: the baselines are trained on the full, untruncated actions from the same demonstrations, a superset of AMP's supervision. Truncation could lower AMP's performance, but it leaves the baselines unaffected.

[Part 2/2]

W2-a. Variance of simulation results. We thank the reviewer for this suggestion. To clarify, the simulation experiments include two independent training runs with different random seeds, with each checkpoint evaluated on 50 unseen episodes. We additionally provide the mean and sample standard deviation across the top-3 evaluations of each run (2 seeds ×\times 3 checkpoints, 6 evaluations in total):

Stack Three D1Hammer Cleanup D1Mug Cleanup D1Coffee D2Square D2Threading D2
AMP87.7±2.387.7 \pm 2.385.7±2.085.7 \pm 2.051.0±1.751.0 \pm 1.774.7±2.174.7 \pm 2.147.0±2.847.0 \pm 2.829.0±1.129.0 \pm 1.1

The results show that the std is at most 2.8 points on every task, well below AMP's gap to the strongest baseline. Moreover, the top-3 mean is within 3.3 points of the best-checkpoint results (reported in the paper for all methods).

W2-b. Real-world statistical rigor. Given the cost of physical evaluation, we use 20 trials per task, a common protocol in robot learning; all methods are tested under the same randomized initialization protocol. To better quantify uncertainty, we report 95% Wilson score intervals and two-sided Fisher's exact tests (success rate in %):

MethodCoffeeToastEggOverall (n=60n = 60)
DiffPo (DDIM)25 [11, 47]40 [22, 61]25 [11, 47]30 [20, 43]
ACT15 [5, 36]35 [18, 57]15 [5, 36]22 [13, 34]
AMP (ours)80 [58, 92]90 [70, 97]85 [64, 95]85 [74, 92]
pp vs. DiffPo1.2×10−31.2 \times 10^{-3}2.2×10−32.2 \times 10^{-3}3.3×10−43.3 \times 10^{-4}–
pp vs. ACT8.8×10−58.8 \times 10^{-5}7.7×10−47.7 \times 10^{-4}1.9×10−51.9 \times 10^{-5}–

On all three tasks, AMP's interval does not overlap with either baseline's, and all six comparisons are significant (p≤0.0022p \le 0.0022), also after Bonferroni correction (α=0.05/6\alpha = 0.05/6). Intuitively, the Wilson interval is the range of success rates consistent with what we observed in 20 trials. Even on Toast, where the methods are closest, the lower end of AMP's interval (70%) is still above the upper end of the best baseline's (61%). The p-value is the chance of seeing a gap this large if the two methods were actually equally good, and here it is at most 0.22% (about 1 in 450). So 20 trials are enough to show that AMP's improvement is very unlikely to come from chance.

W2-c. Real-world baselines. We chose Diffusion Policy and ACT because they are the strongest baselines in our simulation experiments. They are also directly comparable to our method (all three predict action chunks from RGB images), the most widely adopted visuomotor policies for real-world manipulation, and trained from scratch, like ours. Appendix A.7 additionally reports real-world results for Motion Tracks, a closely related prior method, using the authors' official implementation. It achieves only a 5% completion rate, since its independently denoised per-view keypoint trajectories are often inconsistent across views and yield erroneous triangulated actions.

W2-d. Training epochs. Baselines are trained for 800 epochs and AMP for 200. This setup allows all methods to converge. As described in Sect. 4.2, the baselines are evaluated 20 times on 50 unseen test episodes, i.e., every 40 epochs. Since AMP converges faster, it is trained for fewer epochs but evaluated the same number of times (20, i.e., every 10 epochs). As a result, we consider this a fair comparison.

W2-e. Table formatting. We re-checked the submitted PDF in multiple viewers, and all tables render correctly. We would be grateful if the reviewer could point to the specific table(s). In light of this, we kindly ask the reviewer to reconsider the presentation score.

We hope these responses address the reviewer's concerns and would appreciate it if the reviewer would reconsider the rating.

Thank the authors for the rebuttal. They have addressed my concerns. I have raised the score accordingly.

We thank the reviewer for the thoughtful review and for raising the score. We will incorporate the new results and clarifications into the paper.

Comments on the paper

1 comment

Dear reviewers,

As the reminded earlier, please check the author's response and provide your feedback before the end of the discuss phase.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.