acceptodds
Under review as a conference paper at ICLR 2027

Video or Action? Demonstration Modality Matters in Embodied In-Context Learning

Abstract

Demonstrations can guide an agent’s predict proper actions through In-Context Learning (ICL) without updating model parameters. Yet identical demonstration can be represented in different modalities (video or action) while providing different task information (e.g., in videos, movements can accompany environmental responses). In this work, we study the effectiveness and complementarity of embodied demonstration modalities. Theoretically, we assume that demonstrator and learner responses depend on a shared, unobserved task condition through distinct feature maps. Then, we derive necessary and sufficient conditions for exact learner prediction in the infinite-context limit. These conditions distinguish insufficient task information in the demostration from insufficient representation capacity: increasing representation width cannot recover task-relevant information absent from the demonstration. Practically, we evaluate policy execution in various 2D and 3D environments as well as future video and action prediction on aligned human–robot recordings. Independently trained modality variants measure the benefit of each input configuration, while fixed-weight interventions test reliance on demonstration content. Our experiments yield three key findings. (i) Video alone supports effective execution, and action trajectories can achieve comparable performance when the demonstrator adapts its actions to environmental conditions. (ii) Video and measured action trajectories provide complementary cues for human–robot prediction: together they yield lower state and visual errors than either alone. Gains from combining modalities in policy execution vary across tasks. (iii) Incorrect demonstrations are more harmful than absent ones, but correct demonstrations can rescue drifting predictions even when supplied after prediction begins, without restarting the rollout or updating model weights. Code is available at anonymous.4open.science/r/anonymous-core-code-CBD2.

Primary area:
applications to robotics, autonomy, planning
Keywords:
In-Context-Learning, Robotics Policy, Human Demonstration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VrvWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at in-context learning for robots: the model gets a demonstration (e.g. a human doing the task) as part of its input and uses it to act or predict, without retraining. The question is which form of demonstration is more useful when the demonstrator and the robot have different bodies: video, action trajectories (hand motion or commands), or both. The intuition is that video shows how the world responded (a box sliding far on ice), while actions only show what the demonstrator did. On the theory side, they extend the mean-field linear-attention model of Kim & Suzuki (2024) to different demonstrator and learner feature maps (f and g). Proposition 1 says, roughly, that a bigger model can't recover information the demonstration doesn't contain, and that the more the demonstrator and learner differ, the wider the model has to be.

The experiments have two parts. The first is policy execution in 2D and MuJoCo tasks (plus curling), comparing "fixed-probe" demonstrations (commands that ignore the hidden physics) with "adaptive" ones (commands chosen using the hidden physics); roughly, pushing with the same force on every surface vs adjusting the force to hit a target. The second is future video and state prediction with a ~0.6B flow-matching model on 1316 paired human-robot episodes. Here the paper compares separately trained None / Action /Video / Full models and then runs fixed weight interventions, i.e. changing only the demonstration given to the trained model (removing it, swapping in one from another task, cutting it short, or giving it late).

The claimed findings are: (i) video works well, and actions catch up when the demonstrator adapts to the hidden conditions, (ii) video and hand trajectories are complementary for human robot prediction, (iii) wrong demonstrations hurt more than none, and a late demonstration can correct drifting predictions.

Critical strengths
  1. The fixed-probe vs. adaptive comparison (Table 2, Table 13). With fixed probes, Action does no better than having no demonstration (throwing 29.65 vs. 29.85), but with adaptive demonstrations it goes up to 99.89%. So whether "action" is informative depends on how the demonstration was produced. Papers comparing video and action conditioned robot policies usually don't separate these two things, and I think this is the most useful takeaway of the paper.

  2. The intervention experiments. The authors keep the weights, queries and noise fixed and compare each condition against its own no-context control, which is more careful than what I usually see. The effects are large: wrong-physics context drops curling success from about 71% to 0% (Table 11), error goes down steadily as more of the demonstration video is shown (Table 10), and a late demonstration brings the long-horizon error from 81.44 down to 17.73 mm (Table 8). This is much better evidence that the models use the demonstration than comparing separately trained models.

Critical weaknesses
  1. Finding (i) mostly follows from the setup, and the theory doesn't connect to the experiments. The fixed/adaptive idea is useful as an illustration (see strengths), but as evidence it is mostly built in. Fixed probes ignore the hidden physics by definition, so Action isn't expected to beat no demonstration. Adaptive commands are chosen using the true physics, and in simple deterministic simulators one (state, command) pair nearly reveals the hidden parameter, so Action doing well is expected; partially or noisily adaptive demonstrators would be more convincing. The theory doesn't model video vs. action: a fixed probe is just f = 0 (the demonstration doesn't depend on the task), where no learner can succeed. The width threshold, the non-trivial part, is never tested, and the analysis (infinitely many i.i.d. demonstrations, linear attention) is far from the single-demonstration, nonlinear models in the experiments. It also doesn't explain why Video+Action is worse than Video in fixed throwing (91.83 vs. 96.55), or why Video is worse than no demonstration in adaptive aerial gate (45.70 vs. 49.32; the authors note the gate is omitted from these videos, but the theory doesn't account for it).

  2. The human robot results may come from time aligned retargeting rather than In Context Learning. From App. A.1 and Eq. 6, the human video and hand trajectory for steps 1-32 are given as context for predicting the robot over the same steps 1-32, and the paper doesn't say how the (unnamed) dataset was collected, e.g. whether the robot was teleoperated from the human hand. If they move in lockstep, the human recording works more like an answer key than an example to learn from. That alone could explain the large None vs. Full gap, the gap growing with horizon (Fig. 3), the large wrong-task error (128 vs. 36 mm, Table 9) and the late arrival "rescue". Please correct me if I've misread the setup.

Other issues: Finding (iii) likely depends on training. The Full model never saw mismatched demonstrations, and Table 6 shows the reverse effect (a model trained without context gets worse with the correct one, 43.07 vs. 26.50 mm). Complementarity is weakly supported: the Full vs. Video gap is 1.3 mm from one training seed, with no confidence interval in Table 3 (and the bands in Fig. 3 overlap). Most of Table 2(c,d) is single-seed, the OOD shifts are very small, and there are no external baselines or real-robot experiments.

Questions for authors
  1. Where does the human-robot dataset come from, and how were the human and robot episodes paired? Was the robot teleoperated from the human hand, or recorded separately?

  2. Could you rerun Table 3 and Fig. 5 with the demonstration taken from a different episode of the same instruction, so it isn't time-aligned with the target? To me this is the most important missing experiment. The current "Incorrect" condition changes both the task and the time alignment, so it can't tell whether the model is inferring the task or just following the human's motion.

  3. Does the finding that incorrect demonstrations hurt more than no demonstration still hold if some mismatched demonstrations are included during training? Table 6 makes me think this may depend on the training mix.

  4. Could you report multiple training seeds with confidence intervals for Table 3 and the single-seed parts of Table 2(c,d), including the Full vs. Video comparison? Suggestion (not affecting my rating): a closed-loop real-robot experiment conditioned on human demonstrations would make the paper much stronger.

Reviewer_tPXFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the effective of video, action, and combined demonstrations for embodied in-context learning (ICL). Empirical experiments evaluates sim policy execution and future prediction on aligned human-robot recordings, including interventions that remove, replace or delay context. A theoretical analysis characterizes exact prediction in a simplified model.

Critical strengths
  • The experiments cover a variety of tasks, and report a nuanced result that combining modalities does not always help; task-relevant information matters.
Critical weaknesses
  • The scientific takeaway is unclear. In general, the main claim seems to be that extra task-relevant information helps with the tasks. For example, in the fixed/adaptive comparison, fixed commands don't inform hidden physics, while adaptive commands are explicitly correlated with the latent physics, and we observe an improvement in the latter case, almost by construction.
  • The theory is weakly connected to the experiments, and not empirically tested.
  • Temporally aligned human trajectories might be a limitation for generalization.
Questions for authors
  • Could you clarify what the main contribution / takeaway is here?
  • How does performance change when the human demonstration and robot query come from different episodes with different initial conditions?
  • What falsifiable predictions follows from the theory, and can they be empirically tested? How do they connect to the experiments?
Reviewer_3YUCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies an important and popular problem in robot learning that is in-context learning (zero-shot prompting). The paper uses Mujoco experiments to try to answer the important question of how to make the context most effective for robot ICL.

Critical strengths

The paper is well presented and studies an important problem.

Critical weaknesses

My biggest concern is that answering the question of what modalities and how in composition make them effective as context for robot ICL, requires large-scale study or at least training from real-world robotics dataset or using pretrained robot policy (many open-sourced). Mujoco experiments at the scale do not necessarily transfer to real-world / large-scale setup.

Questions for authors

Can the authors design and run experiments on real-world data and robot?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.