- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes MIRAL, an in-context learning framework built on top of π0.5. The core idea is to train a single policy on aligned human and robot data so that it can be prompted with human demonstrations and execute robot actions. To do this, the policy predicts a single action vector whose coordinates are split between robot controls and human hand motion. Inputs to the model are the current image, state, and instruction of the query, plus a set of references. References are short demo clips, each with an image and its actions, from either a human or a robot. The model outputs robot actions if the query is a robot and hand motion if the query is a human. It is trained on every human/robot combination and deployed with human demos as references and the robot as the query. To prevent the policy from ignoring the references, the authors introduce the Context Error Margin (CEM) loss, which raises the prediction error under mismatched references relative to matched ones. On The Imitator Game the paper reports 84% seen and 35% zero-shot success, and a real-world deployment on two tasks is shown.
Critical strengths
- The problem is relevant. Learning from human and robot data jointly, and then using a human video as context to drive a robot without further training, would remove the need for robot teleoperation when adapting to new tasks. The proposed action interface is simple and, if the claims hold, the approach would be a meaningful contribution.
- The paper includes real-world rollouts on a bimanual platform, including an out-of-distribution object, and supplies video. This shows the method runs on hardware and is not only a simulation result.
Critical weaknesses
- The headline zero-shot result is not credible as reported. The benchmark holds out five of its 50 simulation tasks from all pretraining. MIRAL trains on 9,994 of the 10,000 simulation episodes and states that "the simulator covers 50 tasks" (L306 to L309), with no mention of a holdout. Unless the five unseen tasks were removed, the 35% zero-shot number (L022, L080, L335) is on tasks the model was trained on. The ablations add to the doubt: with reference actions removed, zero-shot is still 0.22 (L411), above every baseline, so most of the gain over the benchmark does not come from the context mechanism the paper is about. Since the zero-shot gain is the main claim, this determines my recommendation.
- The comparisons do not isolate the method. None of the baselines were retrained. The benchmark numbers come from recipes that freeze the backbone and train only the action head , whereas MIRAL trains the full backbone for 100k steps at batch 128. The Scratch column is misleading, since it is defined as training from scratch while MIRAL starts from π0.5 weights. The real-robot evaluation has the same problem: MIRAL is the only policy that receives references , and there is no MIRAL-without-context row in Table 2 so the hardware gains cannot be attributed to reference use rather than to different pretraining. The ablations in Table 4 (L409 to L413) are single runs without error bars, and Seen, Scratch, and P+FT move by only 1 to 6 points over 200 trials per regime. Taken together, no experiment in the paper cleanly attributes the reported gains to the proposed mechanism.
Questions for authors
- Were the five benchmark zero-shot tasks excluded from the 9,994-episode training catalog? If so, which episodes were dropped, and how does 9,994 relate to the benchmark's 10,000 simulation episodes?
- Can you report simulation results per level L0 to L3? The benchmark is built around these levels, and the paper only reports aggregates.
- Can you add a MIRAL-without-context and, if safe, a MIRAL-with-random-context row to the real-robot evaluation?
- The real-robot setup uses three cameras including two wrist views, while pretraining uses one main view. Were the wrist views used at post-training and test time, and how does this interact with the single-view references?
- How many trials and seeds underlie each entry in Tables 3 and 4? Most Table 4 differences outside the zero-shot column are within 1 to 6 points, and without error bars those takeaways cannot be interpreted.
- The separate-projector ablation changes results by at most 1 point. Since human and robot actions occupy disjoint zero-padded coordinates, a single linear map is already equivalent to two separate maps. Should this be dropped from the stated contributions?