acceptodds
Under review as a conference paper at ICLR 2027

Learning from Last-Mile Failures in Fine-Grained Robotic Manipulation

Abstract

Fine-grained robotic manipulation operates at tight tolerances; therefore, real-robot data collection inevitably yields many failed trials. Standard practice discards these failures or compresses each into a scalar reward, preference, or advantage. We take a global view: all trials form one record of how a task is performed, and success is a label set by the task tolerance. This record reveals a sharp regularity: most failures are **last-mile**, reproducing the successful motion and departing only where contact decides the outcome. We show that this structure turns a trajectory-level outcome label into a precise correction signal. (i) Every two-weight guidance rule over success-, failure-, and null-conditioned predictions, including existing attraction, repulsion, and two-branch rules, acts along one contrast direction and differs only in its gain. (ii) This direction aligns with the task-critical correction whenever a measurable scaffold ratio is below one, as for last-mile failures, so one label per trajectory localizes the error without step annotations. On this basis, we propose **CALM** (**C**ontrast **A**t the **L**ast **M**ile), which trains a world action model with outcome tokens and applies contrastive denoising guidance that attracts actions toward success and repels them from failure. On six real-robot tasks, CALM improves average success by 12.8 points over the standard baseline, with gains on every task, and loses this gain once off-scaffold failures raise the scaffold ratio above one, as the validity condition predicts.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotic Manipulation, Fine-grained, Imitation Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CFp1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to learn from “last-mile” failures in fine-grained robotic manipulation, where failed trajectories share most of their motion with successful demonstrations but deviate near task-critical contact. It proposes CALM, a world action model trained with success, failure, and null outcome tokens using one binary label per trajectory. At inference, contrastive denoising guidance steers action generation toward success and away from failure; the default configuration combines only the success and failure predictions because the null term cancels. The paper also presents an empirical audit of failure trajectories and a theoretical analysis explaining when the success–failure contrast aligns with task-critical corrections. On six real-robot tasks, CALM achieves 77.2% average success, outperforming the strongest standard baseline by 12.8 percentage points. Ablations examine guidance components, label granularity, and failure quality, showing that the benefit diminishes when failures deviate substantially from successful motion.

Critical strengths
  1. The paper identifies last-mile failures as a substantial source of supervision: 96.2% of procedure-complete failures in its main audit share most of the successful motion, with supporting evidence across operators and a public dataset. This grounds the method in an observed property of robot data and motivates retaining demonstrations that would otherwise be discarded.

  2. The theoretical analysis unifies outcome-guidance rules as gains along one success-failure contrast direction and gives a sufficient alignment condition under a local residual model. This provides a testable explanation of when failure guidance should be useful, beyond introducing another guidance formula.

Critical weaknesses

None

Questions for authors
  1. Have you compared CALM with complete training implementations of closely related outcome-guidance methods, rather than only applying their guidance rules to CALM checkpoints? Such results would help distinguish the contribution of CALM’s training procedure from its inference weights.

  2. How much manual effort and task-specific knowledge are needed to audit a new dataset? What evidence shows that this audit can predict, before training CALM, whether a failure pool will provide useful guidance?

Reviewer_j41cOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper examines when failed demonstrations help fine-grained manipulation. Many failures preserve most of the successful motion and diverge near the outcome-critical contact phase. CALM uses this structure through outcome-conditioned training and success–failure guidance in Cosmos Policy. The main contribution is the characterization of useful failures and the analysis of when they help.

Critical strengths
  1. The last-mile observation is useful and supported by data audits across operators, tasks, and failure sources. It offers a concrete explanation for why some failed demonstrations are valuable while others are not.
  2. The real-robot results are convincing. Longer-horizon and simulation experiments provide additional support.
  3. The contamination experiment strengthens the explanation. Replacing half the training failures with off-scaffold failures reduces success from 77.2% to 62.8%, while filtering recovers much of the loss. This connects the proposed failure structure to actual policy performance.
Critical weaknesses
  1. Applicability and predictive value of the failure audit. The main experiments exclude 20 of 152 raw failures using procedure-completion rules. The audit also relies on task events and tolerances, leaving its applicability to tasks without a fixed procedure unclear. It measures scaffold mismatch, but the theoretical alignment condition also depends on residual–scaffold coupling that is diagnosed only after training. Although the audit separates natural last-mile failures from constructed off-scaffold failures, the paper does not establish whether it can reliably predict before training when CALM will help on a new task or failure source.
  2. The real-robot statistics are careful, but training variability remains unclear. The reported statistics characterize uncertainty across evaluation rollouts, but it is unclear whether the results include independently trained policies. With fixed trajectory lists and only ten failures per task, robustness to training randomness and failure selection remains unestablished.
Questions for authors

See Weakness, and one more question: Contrast-energy measurements and trajectory examples support the last-mile interpretation, but do not directly show how guidance changes actions throughout execution. A comparison over task progress would make this explanation more concrete.

Reviewer_5E6EOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines when failed demonstrations can teach corrections through a single outcome label. Its audit finds that many procedure-complete teleoperation failures share successful motion until a late interaction error. CALM learns success- and failure-conditioned denoising in a world action model, then samples with 2ε_success − ε_failure. A local residual model gives a condition under which the contrast aligns with a task-critical correction. CALM achieves 77.2% success on six real-robot tasks, compared with 64.4% for π0.5 + DPO. Additional experiments examine guidance rules, observation prediction, failure-pool contamination, and longer tasks.

Critical strengths
  1. The audit makes the failure-structure claim testable. Procedure screening is independent of the audit score, tolerances are fixed before inspecting failures, and sensitivity checks leave the last-mile count largely unchanged. The 127/132 result is explicitly conditional on procedure-complete failures; including excluded attempts gives 127/152. Public-data tests include held-out successes and wrong-target controls, while the contrast between REASSEMBLE and Can-MG shows that the finding depends on the failure source (Section 3.1; Appendix B). This is useful evidence about the data conditions under which outcome contrast may help.

  2. At the same total trajectory budget, CALM improves over π0.5 + DPO on all six tasks. The aggregate gain is 12.8 percentage points, with a 95% difference interval of [3.4, 21.9] and worst-case paired McNemar p ≤ 0.032 (Table 1). The controls go beyond success-only and mixed imitation: they include a classifier trained on the same failures and a full DIPOLE-style training reproduction, which reaches 66.1% (Tables 2 and 16). Removing future-observation prediction lowers CALM from 77.2% to 68.3% (Table 19). The empirical contribution includes both the failure-data analysis and the way the contrast is learned in the world action model.

Critical weaknesses
  1. The claim that gains disappear when the scaffold ratio exceeds one goes beyond Proposition 2. Under the local model, ρ < 1 is sufficient for an angle below 45 degrees between the contrast and the task-critical correction. The proposition does not give a converse for ρ ≥ 1 or a guarantee about closed-loop success. The contamination experiment also changes more than the quality of repulsion: Tables 3–4 compare contaminated CALM with the clean checkpoint's 63.3% attraction-only result, and Appendix A.4 notes that contamination changes the attraction field too. Thus 62.8% success at ρ̂ = 1.08 does not show that repulsion stops helping relative to attraction on the same model. The measured alignment and gate recovery support the mechanism, but the proposed performance boundary remains unestablished.

  2. Each task uses a fixed set of ten training failures, with no reported variation across training seeds or alternative failure subsets. The rollout intervals and matched evaluation starting configurations do not address this source of uncertainty. The stability of the classifier comparison, observation-target ablation, and contamination trend across training runs is therefore unclear, even though the primary result is consistent across six tasks.

Questions for authors
  1. What does attraction-only (2,0) achieve on each contamination level's own checkpoint? Does the additional benefit of two-branch guidance disappear near ρ̂ = 1, or do both rules deteriorate together?

  2. How many independent training runs and failure-subset selections were used? Please report any between-run results for the main comparison and mechanism ablations. If these are single-run results, a repeat of the primary comparison and one mechanism ablation would help assess their stability.

2 comments

Thank you for the detailed response, which addresses most of my previous concerns. I maintain my score.

Reply to R1 5E6E

We sincerely thank the reviewer for the reply and for the continued support of our paper. Following the reviewer's suggestions, the revised paper will include the converse of Prop. 2 for the local alignment, the same-model controls for closed-loop success, and the multi-run results.

If any specific concern remains, we would warmly welcome the chance to address it before the discussion closes. We thank the reviewer again for the rigorous and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.