- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper looks at in-context learning for robots: the model gets a demonstration (e.g. a human doing the task) as part of its input and uses it to act or predict, without retraining. The question is which form of demonstration is more useful when the demonstrator and the robot have different bodies: video, action trajectories (hand motion or commands), or both. The intuition is that video shows how the world responded (a box sliding far on ice), while actions only show what the demonstrator did. On the theory side, they extend the mean-field linear-attention model of Kim & Suzuki (2024) to different demonstrator and learner feature maps (f and g). Proposition 1 says, roughly, that a bigger model can't recover information the demonstration doesn't contain, and that the more the demonstrator and learner differ, the wider the model has to be.
The experiments have two parts. The first is policy execution in 2D and MuJoCo tasks (plus curling), comparing "fixed-probe" demonstrations (commands that ignore the hidden physics) with "adaptive" ones (commands chosen using the hidden physics); roughly, pushing with the same force on every surface vs adjusting the force to hit a target. The second is future video and state prediction with a ~0.6B flow-matching model on 1316 paired human-robot episodes. Here the paper compares separately trained None / Action /Video / Full models and then runs fixed weight interventions, i.e. changing only the demonstration given to the trained model (removing it, swapping in one from another task, cutting it short, or giving it late).
The claimed findings are: (i) video works well, and actions catch up when the demonstrator adapts to the hidden conditions, (ii) video and hand trajectories are complementary for human robot prediction, (iii) wrong demonstrations hurt more than none, and a late demonstration can correct drifting predictions.
Critical strengths
-
The fixed-probe vs. adaptive comparison (Table 2, Table 13). With fixed probes, Action does no better than having no demonstration (throwing 29.65 vs. 29.85), but with adaptive demonstrations it goes up to 99.89%. So whether "action" is informative depends on how the demonstration was produced. Papers comparing video and action conditioned robot policies usually don't separate these two things, and I think this is the most useful takeaway of the paper.
-
The intervention experiments. The authors keep the weights, queries and noise fixed and compare each condition against its own no-context control, which is more careful than what I usually see. The effects are large: wrong-physics context drops curling success from about 71% to 0% (Table 11), error goes down steadily as more of the demonstration video is shown (Table 10), and a late demonstration brings the long-horizon error from 81.44 down to 17.73 mm (Table 8). This is much better evidence that the models use the demonstration than comparing separately trained models.
Critical weaknesses
-
Finding (i) mostly follows from the setup, and the theory doesn't connect to the experiments. The fixed/adaptive idea is useful as an illustration (see strengths), but as evidence it is mostly built in. Fixed probes ignore the hidden physics by definition, so Action isn't expected to beat no demonstration. Adaptive commands are chosen using the true physics, and in simple deterministic simulators one (state, command) pair nearly reveals the hidden parameter, so Action doing well is expected; partially or noisily adaptive demonstrators would be more convincing. The theory doesn't model video vs. action: a fixed probe is just f = 0 (the demonstration doesn't depend on the task), where no learner can succeed. The width threshold, the non-trivial part, is never tested, and the analysis (infinitely many i.i.d. demonstrations, linear attention) is far from the single-demonstration, nonlinear models in the experiments. It also doesn't explain why Video+Action is worse than Video in fixed throwing (91.83 vs. 96.55), or why Video is worse than no demonstration in adaptive aerial gate (45.70 vs. 49.32; the authors note the gate is omitted from these videos, but the theory doesn't account for it).
-
The human robot results may come from time aligned retargeting rather than In Context Learning. From App. A.1 and Eq. 6, the human video and hand trajectory for steps 1-32 are given as context for predicting the robot over the same steps 1-32, and the paper doesn't say how the (unnamed) dataset was collected, e.g. whether the robot was teleoperated from the human hand. If they move in lockstep, the human recording works more like an answer key than an example to learn from. That alone could explain the large None vs. Full gap, the gap growing with horizon (Fig. 3), the large wrong-task error (128 vs. 36 mm, Table 9) and the late arrival "rescue". Please correct me if I've misread the setup.
Other issues: Finding (iii) likely depends on training. The Full model never saw mismatched demonstrations, and Table 6 shows the reverse effect (a model trained without context gets worse with the correct one, 43.07 vs. 26.50 mm). Complementarity is weakly supported: the Full vs. Video gap is 1.3 mm from one training seed, with no confidence interval in Table 3 (and the bands in Fig. 3 overlap). Most of Table 2(c,d) is single-seed, the OOD shifts are very small, and there are no external baselines or real-robot experiments.
Questions for authors
-
Where does the human-robot dataset come from, and how were the human and robot episodes paired? Was the robot teleoperated from the human hand, or recorded separately?
-
Could you rerun Table 3 and Fig. 5 with the demonstration taken from a different episode of the same instruction, so it isn't time-aligned with the target? To me this is the most important missing experiment. The current "Incorrect" condition changes both the task and the time alignment, so it can't tell whether the model is inferring the task or just following the human's motion.
-
Does the finding that incorrect demonstrations hurt more than no demonstration still hold if some mismatched demonstrations are included during training? Table 6 makes me think this may depend on the training mix.
-
Could you report multiple training seeds with confidence intervals for Table 3 and the single-seed parts of Table 2(c,d), including the Full vs. Video comparison? Suggestion (not affecting my rating): a closed-loop real-robot experiment conditioned on human demonstrations would make the paper much stronger.