acceptodds
Withdrawn from ICLR 2027

Beyond Context Sensitivity: Cross-Embodiment In-Context Policy Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Adapting robot policies to new tasks through post-training incurs demonstrationcollection and optimization costs. In-context learning (ICL) avoids test-time weight updates, but robot-reference methods still require demonstrations from the same embodiment. We ask whether human videos can provide references without robot teleoperation. Human-motion retargeting often relies on hand-designed kinematic correspondences, tying motion transfer to a robot-specific mapping. We introduce MIRAL (Mixed-embodiment In-context Reference Action Learning), a cross-embodiment ICL pretraining framework with a shared human–robot action interface. Disjoint action coordinates and modality-specific reference-action projectors support joint learning with human and robot references and queries. To discourage reference neglect, Context Error Margin (CEM) training fits query actions under matched references and raises mismatched prediction error. On The Imitator Game, we achieve 35% zero-shot success, 22 percentage points above the strongest listed comparator. Real-robot experiments on soybean pouring and tableware arrangement demonstrate effective human-reference-conditioned control.

Primary area:
applications to robotics, autonomy, planning
Keywords:
In context learning, VLA, Embodied AI

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Esf7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes MIRAL, an in-context learning framework built on top of π0.5. The core idea is to train a single policy on aligned human and robot data so that it can be prompted with human demonstrations and execute robot actions. To do this, the policy predicts a single action vector whose coordinates are split between robot controls and human hand motion. Inputs to the model are the current image, state, and instruction of the query, plus a set of references. References are short demo clips, each with an image and its actions, from either a human or a robot. The model outputs robot actions if the query is a robot and hand motion if the query is a human. It is trained on every human/robot combination and deployed with human demos as references and the robot as the query. To prevent the policy from ignoring the references, the authors introduce the Context Error Margin (CEM) loss, which raises the prediction error under mismatched references relative to matched ones. On The Imitator Game the paper reports 84% seen and 35% zero-shot success, and a real-world deployment on two tasks is shown.

Critical strengths
  1. The problem is relevant. Learning from human and robot data jointly, and then using a human video as context to drive a robot without further training, would remove the need for robot teleoperation when adapting to new tasks. The proposed action interface is simple and, if the claims hold, the approach would be a meaningful contribution.
  2. The paper includes real-world rollouts on a bimanual platform, including an out-of-distribution object, and supplies video. This shows the method runs on hardware and is not only a simulation result.
Critical weaknesses
  1. The headline zero-shot result is not credible as reported. The benchmark holds out five of its 50 simulation tasks from all pretraining. MIRAL trains on 9,994 of the 10,000 simulation episodes and states that "the simulator covers 50 tasks" (L306 to L309), with no mention of a holdout. Unless the five unseen tasks were removed, the 35% zero-shot number (L022, L080, L335) is on tasks the model was trained on. The ablations add to the doubt: with reference actions removed, zero-shot is still 0.22 (L411), above every baseline, so most of the gain over the benchmark does not come from the context mechanism the paper is about. Since the zero-shot gain is the main claim, this determines my recommendation.
  2. The comparisons do not isolate the method. None of the baselines were retrained. The benchmark numbers come from recipes that freeze the backbone and train only the action head , whereas MIRAL trains the full backbone for 100k steps at batch 128. The Scratch column is misleading, since it is defined as training from scratch while MIRAL starts from π0.5 weights. The real-robot evaluation has the same problem: MIRAL is the only policy that receives references , and there is no MIRAL-without-context row in Table 2 so the hardware gains cannot be attributed to reference use rather than to different pretraining. The ablations in Table 4 (L409 to L413) are single runs without error bars, and Seen, Scratch, and P+FT move by only 1 to 6 points over 200 trials per regime. Taken together, no experiment in the paper cleanly attributes the reported gains to the proposed mechanism.
Questions for authors
  1. Were the five benchmark zero-shot tasks excluded from the 9,994-episode training catalog? If so, which episodes were dropped, and how does 9,994 relate to the benchmark's 10,000 simulation episodes?
  2. Can you report simulation results per level L0 to L3? The benchmark is built around these levels, and the paper only reports aggregates.
  3. Can you add a MIRAL-without-context and, if safe, a MIRAL-with-random-context row to the real-robot evaluation?
  4. The real-robot setup uses three cameras including two wrist views, while pretraining uses one main view. Were the wrist views used at post-training and test time, and how does this interact with the single-view references?
  5. How many trials and seeds underlie each entry in Tables 3 and 4? Most Table 4 differences outside the zero-shot column are within 1 to 6 points, and without error bars those takeaways cannot be interpreted.
  6. The separate-projector ablation changes results by at most 1 point. Since human and robot actions occupy disjoint zero-padded coordinates, a single linear map is already equivalent to two separate maps. Should this be dropped from the stated contributions?
Reviewer_ErEjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MIRAL trains a reference-conditioned policy from human and robot demonstrations using π0.5. A contrastive objective encourages the policy to use matched references and distinguish them from mismatched ones. The method is evaluated on The Imitator Game and two real-robot tasks with local post-training.

Critical strengths

In-context adaptation from human demonstrations is an interesting direction for robot learning. The paper is clearly written and easy to follow.

Critical weaknesses

The real-robot experiments use task-specific robot demonstrations to post-train on both evaluated tasks. They do not test whether the policy can acquire a new behavior from a human reference or perform an unseen task without further training. The unseen-bowl experiment changes the object within a trained task. The reference ablations show that context is useful, but do not establish whether the policy follows demonstrated behavior beyond identifying the task or goal. Only two physical tasks are evaluated, with ten trials per method per condition and no reported uncertainty, making the reliability of the hardware results difficult to assess. The cross-embodiment evaluation also covers few robot platforms and does not test transfer to a held-out robot embodiment, limiting the generality of the claim.

In Table 1, MIRAL also performs best on seen tasks. It is unclear how much of this gain comes from the proposed cross-embodiment learning rather than the training recipe.

Questions for authors

See weaknesses

Reviewer_U4WeOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes MIRAL, a cross-embodiment in-context learning framework that enables robot policies to leverage human and robot demonstrations without additional parameter updates at test time. The method introduces a unified human–robot action interface with modality-specific reference-action projectors and a Context Error Margin (CEM) objective to encourage effective reference utilization. Experiments on The Imitator Game demonstrate substantial improvements in zero-shot performance on unseen tasks, achieving a 35% success rate compared to 13% for the strongest listed baseline. However, insufficient justification of baseline selection, ambiguity in the evaluation protocol, and limited real-robot experiments weaken the empirical evidence supporting the claimed generalizability. Furthermore, missing supplementary material and substantial manuscript inconsistencies raise concerns about the completeness and reproducibility of the submission.

Critical strengths

1. Substantial improvement in zero-shot generalization to unseen tasks.

MIRAL achieves a 35% success rate on unseen tasks without additional parameter updates, outperforming the strongest listed baseline by 22 percentage points. This result demonstrates the potential of leveraging human demonstrations to adapt robot policies to new tasks without collecting task-specific robot demonstrations.

2. A unified reference-conditioned framework for human and robot demonstrations.

The paper proposes a unified framework that accommodates both human and robot references through modality-specific action projectors and a shared action interface. The proposed Context Error Margin (CEM) objective further encourages the policy to utilize relevant references, offering a promising approach to cross-embodiment in-context learning.

Critical weaknesses

1. Insufficient justification and evaluation of the selected baselines.

The baseline selection in Tables 1 and 2 is insufficiently justified, as most compared methods are conventional VLA or imitation-learning models rather than approaches explicitly designed for in-context learning. Consequently, the reported results do not adequately establish the effectiveness of MIRAL relative to existing reference-conditioned or in-context learning approaches.

2. Serious manuscript inconsistencies and missing supplementary material.

The submission contains several significant issues, including inconsistent titles between the PDF and OpenReview, missing supplementary material explicitly referenced in Section 3.1, and numerous incomplete or incorrectly formatted references. These issues raise concerns about the manuscript's completeness and reliability, particularly because the missing supplementary material contains essential methodological details required for reproducibility.

Questions for authors

Q1. Could the authors clarify whether the zero-shot evaluation tasks are entirely excluded from training across both human and robot modalities, given that the episode-level split described in Section 3.2 does not necessarily guarantee task-disjoint evaluation?

Q2. Could the authors justify the baseline selection in Tables 1 and 2 and explain how comparisons primarily against conventional VLA and imitation-learning models adequately establish the effectiveness of MIRAL's in-context learning framework?

Q3. Could the authors clarify how the real-robot evaluation, limited to three tasks, supports the claimed generalizability, including the number of independent trials, diversity of task configurations, and overlap with training tasks?

Q4. Could the authors explain why the supplementary material explicitly referenced in Section 3.1 is missing, given that it contains essential definitions and ordering information for the human and robot action descriptors required to understand and reproduce the proposed method?

Q5. Could the authors explain the inconsistencies between the PDF and OpenReview titles, as well as the numerous incomplete or incorrectly formatted references, and clarify whether the submitted manuscript has undergone thorough verification?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.