acceptodds
Withdrawn from ICLR 2027

In-context Robot Learning Made Simple: A Democratized Recipe for Manipulation Tasks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We study robotic in-context learning (ICL), an emerging paradigm that enables robots to infer and execute tasks from visual demonstrations. Despite its growing promise, the problem itself remains under-defined: a visual demonstration simultaneously conveys action trajectories, object semantics, manipulation affordances, spatial relations, and task goals, making it unclear what information the robot is actually expected to follow. In this work, we first provide a clear problem definition of robot ICL that explicitly defines its learning target and resolves this fundamental prompt ambiguity. Building on this definition, we develop a minimalist and reproducible ICL framework (SimpleICL) with a universal visual prompt encoder and a standardized low-cost data collection protocol. Without massive pre-training or specialized data infrastructure, our framework achieves strong performance in both simulation and real-world environments. Extensive experiments further reveal several key properties of robot ICL, including action, semantic, composition, and affordance discrimination. We will fully open-source our data and training pipeline to facilitate systematic and reproducible research on robot ICL.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Intelligence, In-context Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6AwgOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a general method for In-Context Learning in robotics, which is not bound by model type or tasks. It proposes an overview of what a model should learn in ICL and what it should not learn. It further defines how data should be collected and what sort of data should be used using four different discriminative definitions. To enable models to actually work with this data the paper introduces a video conditioning module which gets a video as input and outputs tokens which the model receives as input. The proposed method performs significantly better in simulation compared to the baseline and better on real robot experiments when looking at hard perturbations.

Critical strengths
  • The topic and question from the paper matters, as ICL is an important part for future robot learning
  • The real robot evaluation with 3 different difficulty parts helps to distinguish if the model is actually following the conditioning
  • Cross-group pairing and affordance findings are interesting
Critical weaknesses

Weaknesses

  • The assumption in the second paragraph of the Introduction that humans only observe and learn from that is not supported. References would help here.
  • The approach is only tested using a WAM. There is no VLA adaption, which makes the claims from line 80 and 81 questionable.
  • Future prediction seems to be very important for the performance of the model. It is only mentioned on page 6 and is not defined earlier or mentioned later. Explaining this would be very important to understand why the model drops so significantly in performance and it should be isolated through experiments so that FP is not the thing that pushes performance but the proposed method is.
  • The paper does not define how many rollouts were done for the simulated and real robot experiments. It does not clarify if models with different seeds were trained and evaluated or only one model.
  • In general there are many things missing to actually reproduce the experiments, like losses, model types, how were the baseline trained, etc.
  • In 4.3.1 it is not clear how the findings correspond to the actual experiments. How was measured that the ICL model accurately follows human actions? How was it better in identifying the target object or disentangling scene observations from conditioned task intents? Also did the other models also receive the human demonstration as conditioning input?
  • The paper does not address any limitations of the method.
  • The paper does not compare any ICL baselines (like Zero-WAM or BC-Z) or defines zero or one-shot methods in the related work section.
  • The simulation benchmark uses robot videos for conditioning, which does not test the claims of the paper.
  • The dataset size is not defined
  • Section 3.3 and 4.1 are contradictory, as one talks about ViT or frozen backbones and the other mentions the VAE latent.
  • The contributions of the paper are not clear/novel. Being the first to expose and examine the issue of prompt ambiguity is not supported as not related work of zero-shot or few-shot papers is cited or compared with. Further, the prompt/video encoder is a standard learned-query cross-attention resampler and should be named as such. It is also not clear how these tokens then learn to ignore background noise or distractor objects.

Minor Weaknesses

  • First two paragraphs in the introduction to not have any references supporting the claims made.
  • Figure 1 has a small font size which is hard to read if not zoomed in. Removing the benchmark results and compressing this figure more would make it easier to read and more understandable
  • Line 92, the term “without bells and whistles” is colloquial
  • Method section (3) and Experiments (4) is empty
  • Line 134, In-Context Robot Learning should be ICRL and not ICL
  • Line 184 uses a : instead of .
  • Figure 2 and Figure 3 have very short captions, which do not explain the figure.
  • 3.3 can be one paragraph
Questions for authors
  • How do the four definitions what a ICL model should focus on differ from standard Imitation Learning? Should not all four semantic dimensions also be applied onto the standard IL model?
  • How does the perturbation-based augmentation across groups work? Does it mean it changes objects and we just execute already collected demonstrations on these new objects or how is the collection cost almost zero?
  • Line 244 - 247 describes what the context tokens learn, who is this supervised or learned?
  • How does your method improve the performance of VLA models?
  • On how many demonstrations are the models trained? 18,000 + 27,500?
  • What is AgileX? It is also not cited (Line 306/307)
  • How were the baselines fine-tuned?
  • What are the language instructions at medium and hard?
  • Did the baselines also receive the human demonstration conditioning?

Please also refer to all weaknesses as potential questions.

Reviewer_xwE2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SimpleICL, which uses human demonstration videos to guide robots in performing new tasks. It organizes task-relevant information into four semantic dimensions—action, object, composition, and affordance—and develops corresponding data collection strategies and a video encoding module. Experiments cover both simulation and real-world robots, with substantial improvements in challenging settings. However, the problem formulation, fairness of the comparisons, and experimental details require further clarification.

Critical strengths
  1. The paper addresses intent ambiguity in video demonstrations, a practically relevant problem. Designing training data around the four semantic dimensions establishes a clear connection between the method and the evaluation.

  2. The experiments cover unseen tasks, robustness to disturbances, and ablations of key components. In the real-world Hard setting, SimpleICL achieves a success rate of 68%, compared with 42% for FAST-WAM, demonstrating its potential to identify task intent in complex scenes.

Critical weaknesses
  1. The claim of “resolving prompt ambiguity” is too strong. The paper primarily reduces ambiguity by specifying which information should be learned or ignored and by constraining data collection. It does not explain how the model determines intent when a video admits multiple reasonable interpretations. The relationship between the object uniqueness assumption and the introduction of similar objects in the Hard setting also needs clarification.

  2. The fairness of the comparisons needs further justification. The approach changes both the training data and the video conditioning, whereas the baselines primarily use language conditioning. The current results do not clearly distinguish gains from data design, additional video information, and the proposed encoding module. I suggest adding a simple video encoding baseline using the same training data and video inputs, and specifying the data and training budgets for each method in the real-world experiments.

  3. The claims of simplicity, low cost, and reproducibility are insufficiently supported. The method still uses Wan2.2-5B, an approximately 1B-parameter action model, and a 0.6B-parameter query module. The main text does not adequately report the total amount of real-world data, training compute and time, inference latency, or data collection labor. Cross-group pairing increases the number of training pairs, which should be clearly distinguished from the number of independently collected trajectories.

  4. Some experimental conclusions should be more qualified. The real-world experiments do not clearly report the number of trials per setting or uncertainty estimates. The difference between 81% and 80% in the Easy setting is insufficient to establish a clear advantage. Similarly, the conclusion that compositional capabilities can emerge without explicitly collecting composition-discriminative data is based on a limited set of tasks and should be restricted to the evaluated setting.

Questions for authors

If the authors can address the following concerns convincingly, I would consider raising my score:

  1. How does SimpleICL handle demonstrations with multiple plausible interpretations? How is the object uniqueness assumption reconciled with the similar distractors in the Hard setting?

  2. Can you provide a simple video encoding baseline using the same training data and video inputs to isolate the contribution of the proposed module?

  3. Can you report the real-world dataset size, training budgets, inference latency, and data collection costs, distinguishing independently collected trajectories from augmented pairs?

  4. Can you clarify the number of evaluation trials and provide uncertainty estimates? Does the conclusion about learning compositional semantics hold across a broader range of tasks?

Reviewer_PeHqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies ambiguous visual prompting by first defining what is visual prompting and what information can be conveyed from a visual prompt. To study this, the authors collect data in a specific procedure to isolate what the visual prompt provides: 1) action semantics 2) compositional semantics 3) object semantics and 4) affordance semantics. The paper contributions include the data-collection scheme and a minimal training recipe for ICL policies. The paper has simulation experiments with RoboTwin comparing against Fast-WAM and pi0.5 and real-world experiments with custom tasks with a single robotic arm.

Critical strengths

The evaluation protocol with three different tiers indicating in the scene how many unique tasks can be done, only the easiest tier is what is currently tested in literature. This evaluation protocol should be adopted by works testing ICL.

Critical weaknesses

I would argue that this paper actually has a focus on visual prompting and not on ICL as the classification (e.g. action semantics, compositional semantics, object semantics and affordance semantics) aims to resolve ambiguity in visual human prompts. For a paper focused on ICL, there would be more experiments that would involve the robot recognizing patterns within the prompt and then reasoning how to do a similar task or different task.

Questions for authors

Why is this work written from the perspective of ICL? Why does the ICL model learn to disentangle scene content from composition patterns?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.