- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper is an empirical study of "behavior prompting": a policy is conditioned in context on a single raw sensorimotor demonstration so it can perform new tasks at test time. The authors ask when this works, how the policy uses the prompt, and why it helps. To support the study they contribute two procedurally generated benchmarks: DrawAnything, with up to 2000 drawing tasks in sim and real, and LIBERO-Gen, with unseen combinations and chains of tabletop skills. They also contribute a reference architecture, BPP (a cross-attention prompt encoder plus a diffusion action decoder), and an iPhone-based handheld gripper, iPhUMI, for collecting prompts at test time.
Critical strengths
- The paper moves behavior prompting from "it can work" (ICRT) to a controlled study of when, how and why it works. The procedural benchmarks make systematic data and prompt ablations affordable without industrial-scale compute, and the promised open release of benchmarks, code and hardware adds lasting value to the community.
- All task descriptors (behavior prompt, language, goal image) share the same diffusion action decoder, so the comparisons are clean. The results that task diversity beats per-task quantity under a fixed demo budget, and that dense observations plus actions are needed, give practitioners concrete guidance on data collection.
- The authors report where prompting does not help (low-diversity laundry folding), attribute most of BPP's gain over ICRT to the diffusion head, and avoid an efficiency claim. Real-robot drawing and bimanual folding experiments with iPhUMI show the paradigm works beyond simulation.
Critical weaknesses
- Most of the ablations (data diversity, prompt modalities, prompt frequency, prompt robustness) are run only on DrawAnything-Sim, a 2D drawing domain where step-by-step trajectory tracking is naturally favored. The unseen tasks in LIBERO-Gen are only recombinations of seen primitives in one fixed scene, and prompts and rollouts never differ in scene or object identity. The low-diversity conclusion also rests on a single 3-task, single-seed laundry experiment, where the authors themselves suspect other confounds (e.g., spurious background cues). It is therefore unclear how far the findings transfer to more realistic manipulation settings.
- The claim that "the policy uses the prompt as dense sub-goals" relies mainly on attention visualizations. There are no interventional tests, such as shuffling or reversing prompt chunks, truncating the prompt, or giving a mismatched prompt, to show the policy causally depends on that lookup. In addition, the headline task-diversity result is consistent with prior findings on in-context learning and imitation-learning data scaling and with concurrent industrial reports, so the contribution lies more in controlled confirmation than in new insight.
Questions for authors
Please see the weaknesses.
5 comments
Response to Reviewer NzVH (1/2)
Most of the ablations (data diversity, prompt modalities, prompt frequency, prompt robustness) are run only on DrawAnything-Sim, a 2D drawing domain where step-by-step trajectory tracking is naturally favored.
Thank you for this suggestion. We now run all of these on LIBERO-Gen Chain.
- Task diversity finding transfers. On LIBERO-Gen Chain, spreading a fixed budget of chained demonstrations over more tasks raises unseen success (Fig. 5d). We also ran the other two data ablations there: success grows with the number of chained training tasks (Fig. 5e), and training on single-step tasks alone gives almost no success on unseen chains (Fig. 5f).
- Prompt content matters less on LIBERO-Gen, and this is informative. Which prompt contents are necessary depends on the task distribution: dense observations and actions for drawing, only the next manipulation step for LIBERO-Gen Chain. Please refer to our response to Reviewer 6f5H for further explanation (Fig. 8d-f, Section 4.3).
- Prompt robustness. On LIBERO-Gen Chain, the policy is also robust to slowed or paused prompts, as it was on DrawAnything-Sim (Appendix F, Table 2). We also added further interventional tests on both benchmarks; please see our response below.
The unseen tasks in LIBERO-Gen are only recombinations of seen primitives in one fixed scene, and prompts and rollouts never differ in scene or object identity.
We agree, and we state this scope upfront in the benchmark descriptions in Section 4, as well as in Appendix D.4 and the Limitations. It is also a deliberate design choice: holding the scene and the primitives fixed isolates whether a policy can follow a prompt for an unseen combination of familiar skills, and lets us vary task diversity in a controlled way, which is the focus of our study. Recombining seen skills is still challenging: with the same action decoder, the Goal-Image and Language baselines adapt to these unseen tasks worse than BPP (Fig. 4c-e). Evaluating prompts across scenes, under larger object variation, and for adaptation to entirely new action primitives is important future work, which we note in Section 4.3 and the Limitations.
The low-diversity conclusion also rests on a single 3-task, single-seed laundry experiment, where the authors themselves suspect other confounds (e.g., spurious background cues).
Weaker task conditioning at low task diversity is the central finding of the laundry experiment, where the policy sometimes executes the wrong fold and language conditioning does better (Fig. 6). Our simulated benchmarks support this finding as well. DrawAnything-Sim shows the same failure in a controlled setting: with few training tasks, the policy's ability to follow the prompt substantially degrades (see qualitative examples in Fig. 5b). More broadly, controlled experiments in both simulated benchmarks show that behavior prompting fails under low task diversity: on DrawAnything-Sim with fewer training tasks (Fig. 5b), and on LIBERO-Gen Chain with few or no chained training tasks (Fig. 5e,f). We present laundry as a real-world illustration consistent with these results, not as standalone evidence.
It is therefore unclear how far the findings transfer to more realistic manipulation settings.
We agree this is the main open question, and the Limitations state it: we do not train a general-purpose behavior prompting policy on large amounts of diverse real-world manipulation data. The purpose of our work is to introduce controlled benchmarks as a necessary first step toward such a model, since they let us vary one factor at a time to gain insight without industrial-scale data collection. Their findings directly help guide how that real-world data should be collected:
- Under a fixed budget, favor more tasks over more demonstrations per task (Fig. 5a,d).
- Include complex training tasks, not only simple ones, as both benchmarks show (Fig. 5c,f).
- Record observations and actions densely enough for fine-grained tasks (Fig. 8a-c).
- Accept imperfect human demonstrations, since prompts tolerate slowed or paused ones (Appendix F).
- Vary object placement and ordering (as mentioned regarding LIBERO-Gen Chain, Section 4.3).
Response to Reviewer NzVH (2/2)
The claim that "the policy uses the prompt as dense sub-goals" relies mainly on attention visualizations. There are no interventional tests, such as shuffling or reversing prompt chunks, truncating the prompt, or giving a mismatched prompt, to show the policy causally depends on that lookup.
Thank you for this valuable suggestion. We added interventional tests that edit the prompt at test time, on unseen tasks, on DrawAnything-Sim and LIBERO-Gen Chain (Appendix F and summarized in Section 4.2 to support the attention visualizations).
Table 2 in the revised PDF, partially included below:
| Intervention | DrawAnything-Sim error (px) | LIBERO-Gen Chain success (%) |
|---|---|---|
| Clean prompt | 3.47 | 85.4 |
| Remove middle 25% | 17.99 | 72.0 |
| Remove middle 50% | 37.06 | 34.2 |
| Remove middle 75% | 85.21 | 1.8 |
| Shuffle all chunks | 6.91 | 62.0 |
| Reverse chunks | 5.54 | 25.2 |
| Truncate, keep 75%† | 2.78 | 19.4 |
| Truncate, keep 50%† | 2.22 | 8.4 |
| Truncate, keep 25%† | 2.49 | 0.0 |
†Drawing error against the partial drawing at the end of the kept prompt.
These results support the lookup mechanism suggested by the attention maps (Section 4.2). Removing more of the prompt's middle steadily degrades both benchmarks, so the policy depends on every part of the prompt, and a truncated prompt is reproduced only up to where it ends. Strikingly, on DrawAnything-Sim the policy still reproduces the drawing when the prompt's chunks are played in reverse order (each chunk still running forward), with only a moderate rise in error: through the lookup seen in the attention maps, it attends to the prompt chunk that best matches its current observation, wherever that chunk sits (Fig. 7a,c). On LIBERO-Gen Chain, reordering hurts far more, because the order of the chunks sets which manipulation step comes next (Fig. 7b,d).
We did not test mismatched prompts, such as a LIBERO-Gen prompt from a different scene or for a task the current scene does not support; this relates to the limitation we note above, that prompts and rollouts share the same scene.
In addition, the headline task-diversity result is consistent with prior findings on in-context learning and imitation-learning data scaling and with concurrent industrial reports, so the contribution lies more in controlled confirmation than in new insight.
We agree the result is consistent with this prior work in other settings (in-context learning in sequence models, data scaling in imitation learning, and concurrent industrial reports on in-context demonstration prompting), which we state upfront in Section 4.1. To our knowledge, however, it has not been shown for in-context behavior prompting on robot tasks in a controlled, openly reproducible setting that varies task diversity and per-task quantity, so we believe this confirmation is valuable in its own right.
We appreciate the thoughtful review, hope these additions address your concerns, and we are open to additional discussion.
Dear authors,
I have carefully read the comments from other reviewers, and I remain positive about this work at this stage. Looking forward to your response to the concerns I raised.
Best