- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes a general method for In-Context Learning in robotics, which is not bound by model type or tasks. It proposes an overview of what a model should learn in ICL and what it should not learn. It further defines how data should be collected and what sort of data should be used using four different discriminative definitions. To enable models to actually work with this data the paper introduces a video conditioning module which gets a video as input and outputs tokens which the model receives as input. The proposed method performs significantly better in simulation compared to the baseline and better on real robot experiments when looking at hard perturbations.
Critical strengths
- The topic and question from the paper matters, as ICL is an important part for future robot learning
- The real robot evaluation with 3 different difficulty parts helps to distinguish if the model is actually following the conditioning
- Cross-group pairing and affordance findings are interesting
Critical weaknesses
Weaknesses
- The assumption in the second paragraph of the Introduction that humans only observe and learn from that is not supported. References would help here.
- The approach is only tested using a WAM. There is no VLA adaption, which makes the claims from line 80 and 81 questionable.
- Future prediction seems to be very important for the performance of the model. It is only mentioned on page 6 and is not defined earlier or mentioned later. Explaining this would be very important to understand why the model drops so significantly in performance and it should be isolated through experiments so that FP is not the thing that pushes performance but the proposed method is.
- The paper does not define how many rollouts were done for the simulated and real robot experiments. It does not clarify if models with different seeds were trained and evaluated or only one model.
- In general there are many things missing to actually reproduce the experiments, like losses, model types, how were the baseline trained, etc.
- In 4.3.1 it is not clear how the findings correspond to the actual experiments. How was measured that the ICL model accurately follows human actions? How was it better in identifying the target object or disentangling scene observations from conditioned task intents? Also did the other models also receive the human demonstration as conditioning input?
- The paper does not address any limitations of the method.
- The paper does not compare any ICL baselines (like Zero-WAM or BC-Z) or defines zero or one-shot methods in the related work section.
- The simulation benchmark uses robot videos for conditioning, which does not test the claims of the paper.
- The dataset size is not defined
- Section 3.3 and 4.1 are contradictory, as one talks about ViT or frozen backbones and the other mentions the VAE latent.
- The contributions of the paper are not clear/novel. Being the first to expose and examine the issue of prompt ambiguity is not supported as not related work of zero-shot or few-shot papers is cited or compared with. Further, the prompt/video encoder is a standard learned-query cross-attention resampler and should be named as such. It is also not clear how these tokens then learn to ignore background noise or distractor objects.
Minor Weaknesses
- First two paragraphs in the introduction to not have any references supporting the claims made.
- Figure 1 has a small font size which is hard to read if not zoomed in. Removing the benchmark results and compressing this figure more would make it easier to read and more understandable
- Line 92, the term “without bells and whistles” is colloquial
- Method section (3) and Experiments (4) is empty
- Line 134, In-Context Robot Learning should be ICRL and not ICL
- Line 184 uses a : instead of .
- Figure 2 and Figure 3 have very short captions, which do not explain the figure.
- 3.3 can be one paragraph
Questions for authors
- How do the four definitions what a ICL model should focus on differ from standard Imitation Learning? Should not all four semantic dimensions also be applied onto the standard IL model?
- How does the perturbation-based augmentation across groups work? Does it mean it changes objects and we just execute already collected demonstrations on these new objects or how is the collection cost almost zero?
- Line 244 - 247 describes what the context tokens learn, who is this supervised or learned?
- How does your method improve the performance of VLA models?
- On how many demonstrations are the models trained? 18,000 + 27,500?
- What is AgileX? It is also not cited (Line 306/307)
- How were the baselines fine-tuned?
- What are the language instructions at medium and hard?
- Did the baselines also receive the human demonstration conditioning?
Please also refer to all weaknesses as potential questions.