acceptodds
Under review as a conference paper at ICLR 2027

What Enables In-Context Behavior Prompting for Manipulation?

Abstract

Behavior prompting is a paradigm in which a sensorimotor robot demonstration, called a behavior prompt, serves as an in-context prompt for performing new tasks at test time. While prior work has shown that this capability is possible, the conditions that enable it remain poorly understood. We present an empirical study of when, how, and why behavior prompting works. To support this study, we introduce DrawAnything and LIBERO-Gen, benchmarks with up to 2000 procedurally generated tasks that evaluate test-time adaptation to unseen drawing and tabletop manipulation tasks. We also present Behavior Prompting Policy (BPP), an in-context visuomotor architecture, and iPhUMI, a handheld interface to demonstrate behavior prompts at test time. Our main finding is that task diversity, rather than demonstrations per task, is a key driver of prompting capability. Given sufficient diversity, a behavior prompt improves adaptation to unseen tasks, reducing drawing error by 80.7% over goal-image conditioning and improving success on chained manipulation tasks by up to 20.8% over language conditioning. Given insufficient diversity in a real-world laundry experiment, behavior prompting has weaker task conditioning than a language baseline. An attention analysis shows how the prompt is used: the policy follows it step by step as a source of dense sub-goals. Prompt ablations show that dense sensorimotor detail matters: removing actions or downsampling the prompt hurts fine-grained action adaptation. We will open-source all components to enable reproducible research on behavior prompting without needing industrial-scale data collection or compute.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot manipulation, imitation learning, in-context learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NzVHOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper is an empirical study of "behavior prompting": a policy is conditioned in context on a single raw sensorimotor demonstration so it can perform new tasks at test time. The authors ask when this works, how the policy uses the prompt, and why it helps. To support the study they contribute two procedurally generated benchmarks: DrawAnything, with up to 2000 drawing tasks in sim and real, and LIBERO-Gen, with unseen combinations and chains of tabletop skills. They also contribute a reference architecture, BPP (a cross-attention prompt encoder plus a diffusion action decoder), and an iPhone-based handheld gripper, iPhUMI, for collecting prompts at test time.

Critical strengths
  1. The paper moves behavior prompting from "it can work" (ICRT) to a controlled study of when, how and why it works. The procedural benchmarks make systematic data and prompt ablations affordable without industrial-scale compute, and the promised open release of benchmarks, code and hardware adds lasting value to the community.
  2. All task descriptors (behavior prompt, language, goal image) share the same diffusion action decoder, so the comparisons are clean. The results that task diversity beats per-task quantity under a fixed demo budget, and that dense observations plus actions are needed, give practitioners concrete guidance on data collection.
  3. The authors report where prompting does not help (low-diversity laundry folding), attribute most of BPP's gain over ICRT to the diffusion head, and avoid an efficiency claim. Real-robot drawing and bimanual folding experiments with iPhUMI show the paradigm works beyond simulation.
Critical weaknesses
  1. Most of the ablations (data diversity, prompt modalities, prompt frequency, prompt robustness) are run only on DrawAnything-Sim, a 2D drawing domain where step-by-step trajectory tracking is naturally favored. The unseen tasks in LIBERO-Gen are only recombinations of seen primitives in one fixed scene, and prompts and rollouts never differ in scene or object identity. The low-diversity conclusion also rests on a single 3-task, single-seed laundry experiment, where the authors themselves suspect other confounds (e.g., spurious background cues). It is therefore unclear how far the findings transfer to more realistic manipulation settings.
  2. The claim that "the policy uses the prompt as dense sub-goals" relies mainly on attention visualizations. There are no interventional tests, such as shuffling or reversing prompt chunks, truncating the prompt, or giving a mismatched prompt, to show the policy causally depends on that lookup. In addition, the headline task-diversity result is consistent with prior findings on in-context learning and imitation-learning data scaling and with concurrent industrial reports, so the contribution lies more in controlled confirmation than in new insight.
Questions for authors

Please see the weaknesses.

5 comments

Dear authors,

I have carefully read the comments from other reviewers, and I remain positive about this work at this stage. Looking forward to your response to the concerns I raised.

Best

AuthorsOpenReview ↗

Response to Reviewer NzVH (1/2)

Most of the ablations (data diversity, prompt modalities, prompt frequency, prompt robustness) are run only on DrawAnything-Sim, a 2D drawing domain where step-by-step trajectory tracking is naturally favored.

Thank you for this suggestion. We now run all of these on LIBERO-Gen Chain.

  • Task diversity finding transfers. On LIBERO-Gen Chain, spreading a fixed budget of chained demonstrations over more tasks raises unseen success (Fig. 5d). We also ran the other two data ablations there: success grows with the number of chained training tasks (Fig. 5e), and training on single-step tasks alone gives almost no success on unseen chains (Fig. 5f).
  • Prompt content matters less on LIBERO-Gen, and this is informative. Which prompt contents are necessary depends on the task distribution: dense observations and actions for drawing, only the next manipulation step for LIBERO-Gen Chain. Please refer to our response to Reviewer 6f5H for further explanation (Fig. 8d-f, Section 4.3).
  • Prompt robustness. On LIBERO-Gen Chain, the policy is also robust to slowed or paused prompts, as it was on DrawAnything-Sim (Appendix F, Table 2). We also added further interventional tests on both benchmarks; please see our response below.

The unseen tasks in LIBERO-Gen are only recombinations of seen primitives in one fixed scene, and prompts and rollouts never differ in scene or object identity.

We agree, and we state this scope upfront in the benchmark descriptions in Section 4, as well as in Appendix D.4 and the Limitations. It is also a deliberate design choice: holding the scene and the primitives fixed isolates whether a policy can follow a prompt for an unseen combination of familiar skills, and lets us vary task diversity in a controlled way, which is the focus of our study. Recombining seen skills is still challenging: with the same action decoder, the Goal-Image and Language baselines adapt to these unseen tasks worse than BPP (Fig. 4c-e). Evaluating prompts across scenes, under larger object variation, and for adaptation to entirely new action primitives is important future work, which we note in Section 4.3 and the Limitations.

The low-diversity conclusion also rests on a single 3-task, single-seed laundry experiment, where the authors themselves suspect other confounds (e.g., spurious background cues).

Weaker task conditioning at low task diversity is the central finding of the laundry experiment, where the policy sometimes executes the wrong fold and language conditioning does better (Fig. 6). Our simulated benchmarks support this finding as well. DrawAnything-Sim shows the same failure in a controlled setting: with few training tasks, the policy's ability to follow the prompt substantially degrades (see qualitative examples in Fig. 5b). More broadly, controlled experiments in both simulated benchmarks show that behavior prompting fails under low task diversity: on DrawAnything-Sim with fewer training tasks (Fig. 5b), and on LIBERO-Gen Chain with few or no chained training tasks (Fig. 5e,f). We present laundry as a real-world illustration consistent with these results, not as standalone evidence.

It is therefore unclear how far the findings transfer to more realistic manipulation settings.

We agree this is the main open question, and the Limitations state it: we do not train a general-purpose behavior prompting policy on large amounts of diverse real-world manipulation data. The purpose of our work is to introduce controlled benchmarks as a necessary first step toward such a model, since they let us vary one factor at a time to gain insight without industrial-scale data collection. Their findings directly help guide how that real-world data should be collected:

  • Under a fixed budget, favor more tasks over more demonstrations per task (Fig. 5a,d).
  • Include complex training tasks, not only simple ones, as both benchmarks show (Fig. 5c,f).
  • Record observations and actions densely enough for fine-grained tasks (Fig. 8a-c).
  • Accept imperfect human demonstrations, since prompts tolerate slowed or paused ones (Appendix F).
  • Vary object placement and ordering (as mentioned regarding LIBERO-Gen Chain, Section 4.3).
AuthorsOpenReview ↗

Response to Reviewer NzVH (2/2)

The claim that "the policy uses the prompt as dense sub-goals" relies mainly on attention visualizations. There are no interventional tests, such as shuffling or reversing prompt chunks, truncating the prompt, or giving a mismatched prompt, to show the policy causally depends on that lookup.

Thank you for this valuable suggestion. We added interventional tests that edit the prompt at test time, on unseen tasks, on DrawAnything-Sim and LIBERO-Gen Chain (Appendix F and summarized in Section 4.2 to support the attention visualizations).

Table 2 in the revised PDF, partially included below:

InterventionDrawAnything-Sim error (px)LIBERO-Gen Chain success (%)
Clean prompt3.4785.4
Remove middle 25%17.9972.0
Remove middle 50%37.0634.2
Remove middle 75%85.211.8
Shuffle all chunks6.9162.0
Reverse chunks5.5425.2
Truncate, keep 75%†2.7819.4
Truncate, keep 50%†2.228.4
Truncate, keep 25%†2.490.0

†Drawing error against the partial drawing at the end of the kept prompt.

These results support the lookup mechanism suggested by the attention maps (Section 4.2). Removing more of the prompt's middle steadily degrades both benchmarks, so the policy depends on every part of the prompt, and a truncated prompt is reproduced only up to where it ends. Strikingly, on DrawAnything-Sim the policy still reproduces the drawing when the prompt's chunks are played in reverse order (each chunk still running forward), with only a moderate rise in error: through the lookup seen in the attention maps, it attends to the prompt chunk that best matches its current observation, wherever that chunk sits (Fig. 7a,c). On LIBERO-Gen Chain, reordering hurts far more, because the order of the chunks sets which manipulation step comes next (Fig. 7b,d).

We did not test mismatched prompts, such as a LIBERO-Gen prompt from a different scene or for a task the current scene does not support; this relates to the limitation we note above, that prompts and rollouts share the same scene.

In addition, the headline task-diversity result is consistent with prior findings on in-context learning and imitation-learning data scaling and with concurrent industrial reports, so the contribution lies more in controlled confirmation than in new insight.

We agree the result is consistent with this prior work in other settings (in-context learning in sequence models, data scaling in imitation learning, and concurrent industrial reports on in-context demonstration prompting), which we state upfront in Section 4.1. To our knowledge, however, it has not been shown for in-context behavior prompting on robot tasks in a controlled, openly reproducible setting that varies task diversity and per-task quantity, so we believe this confirmation is valuable in its own right.

We appreciate the thoughtful review, hope these additions address your concerns, and we are open to additional discussion.

Reviewer_6f5HOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors investigate under what conditions a robot can adapt to an unseen task from a single demonstration at test time, without updating the policy’s parameters. They introduce Behavior Prompting Policy (BPP) and develop two benchmarks, DrawAnything and LIBERO-Gen, to study when behavior prompting works, how the policy uses the demonstration, and which information in the prompt matters. Their drawing experiments show that, for a fixed demonstration budget, increasing the diversity of training tasks improves adaptation more than collecting additional demonstrations of the same tasks. Attention analysis suggests that BPP follows the demonstration as a sequence of subgoals. Further ablations on drawing tasks show that visual observations help the policy locate the relevant part of the prompt, while action information and sufficiently frequent observations support fine-grained adaptation.

Critical strengths
  1. The authors design a useful experimental setting for studying adaptation to unseen tasks through behavior prompting. Drawing is particularly well chosen because it requires fine-grained action adaptation and allows task diversity and demonstration quantity to be varied systematically.
  2. The paper provides a balanced evaluation of behavior prompting by examining both settings where it improves adaptation and settings where language conditioning performs similarly or better. This helps clarify when providing a full demonstration may be useful.
  3. The experimental study is broad and well structured. Training-data ablations, prompt-content ablations, attention analysis, and simulation and real-world evaluations provide useful empirical support for the main findings within the tested settings.
  4. The finding about training-task diversity is practically useful. Under a fixed demonstration budget, the drawing experiments show that collecting demonstrations across more tasks improves adaptation more than collecting additional demonstrations of the same tasks.
Critical weaknesses
  1. The conditions under which full behavior prompts provide substantial benefits over simpler task descriptions remain incompletely characterized. Although BPP improves adaptation on LIBERO-Gen, the prompt-content ablations are limited to drawing tasks. It remains unclear whether the manipulation gains require action information or could also be obtained from observation-only demonstrations or a sequence of visual subgoals. Extending these ablations to manipulation would better clarify the value of the additional sensorimotor information.

  2. The generality of the findings across model capacity and pretraining regimes remains unclear. Although the paper includes a comparison with fine-tuned π0.5, this does not isolate how capacity or pretraining affects the relative benefit of behavior prompting. A modest capacity ablation or replication of a central finding with another architecture would help clarify whether the conclusions extend beyond the evaluated configuration.

Questions for authors

See Weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 6f5H (1/2)

The conditions under which full behavior prompts provide substantial benefits over simpler task descriptions remain incompletely characterized. Although BPP improves adaptation on LIBERO-Gen, the prompt-content ablations are limited to drawing tasks. It remains unclear whether the manipulation gains require action information or could also be obtained from observation-only demonstrations or a sequence of visual subgoals. Extending these ablations to manipulation would better clarify the value of the additional sensorimotor information.

Thank you for this suggestion, which led to a new finding that we now state in Section 4.3: which prompt contents are necessary depends on the task distribution.

On DrawAnything-Sim, dense sensorimotor detail matters (Fig. 8a-c). Observations are needed to anchor the policy's lookup of where it is in the prompt, actions further improve adaptation, and sampling observations below 1 Hz hurts. Drawing requires following the demonstrated motion closely, stroke by stroke.

We repeated these ablations on LIBERO-Gen Chain (3 seeds, Fig. 8d-f). In contrast, prompt contents matter much less there: most ablated prompts stay within a few points of the full prompt.

Fig. 8d in the revised PDF, as a table:

PromptSuccess on unseen chains (%, mean ± std over 3 seeds)
Full prompt88.9 ± 0.8
Observations + actions86.8 ± 0.7
Observations + proprioception (no actions)86.4 ± 1.9
Proprioception only85.9 ± 1.0
Actions + proprioception85.8 ± 1.1
Observations only82.6 ± 2.1
Actions only79.0 ± 0.3

Removing actions costs only 2.5 points (86.4%), and observation-only prompts, the closest we tested to a sequence of visual subgoals, still reach 82.6%; downsampling observations from 1 Hz to 0.2 Hz gives 84.5% (Fig. 8e).

We attribute the difference from DrawAnything to two properties of the LIBERO-Gen task distribution. First, as in LIBERO, object positions vary only slightly between episodes, so the arm's path alone (e.g., a prompt with just proprioception or just actions) reveals which objects are manipulated and how. Second, the tasks are built from discrete steps, and the policy uses the prompt mainly to identify the next step (Fig. 7b,d), which a sparser or partial prompt still conveys.

This is also why even proprioception-only prompts (85.9%) outperform language conditioning (80.3%, Fig. 4d): the arm's motion provides guidance to the robot in the robot's sensorimotor space, whereas a language-conditioned policy must first map the semantics of an unseen task's instruction to motor control, a less direct route to the same task. For an empirical study of behavior prompting, this contrast is what makes LIBERO-Gen useful alongside DrawAnything: together they show when dense prompt contents matter and when they do not. Studying prompt contents on a manipulation benchmark where objects vary more in placement, or swap places, is important future work, which we note in Section 4.3 and the Limitations.

AuthorsOpenReview ↗

Response to Reviewer 6f5H (2/2)

The generality of the findings across model capacity and pretraining regimes remains unclear. Although the paper includes a comparison with fine-tuned π0.5, this does not isolate how capacity or pretraining affects the relative benefit of behavior prompting. A modest capacity ablation or replication of a central finding with another architecture would help clarify whether the conclusions extend beyond the evaluated configuration.

We ran both checks you suggest on DrawAnything-Sim (new Appendices M and N), with three seeds each.

Model capacity. We varied the depth of the BPP prompt encoder. Table 12 in the revised PDF and included below:

Prompt encoder layersPrompt encoder parametersDrawing error on unseen drawings (px)
3 (half)28.4M3.72 ± 0.05
6 (default)56.7M3.41 ± 0.13
12 (twice)113.4M3.46 ± 0.08

Twice the prompt encoder depth matches the default, and half the depth raises error only slightly, so the default prompt encoder is not a capacity bottleneck.

Second architecture. We repeated the fixed-budget task diversity experiment (Fig. 5a) with ICRT. Fig. 22 in the revised PDF and included below:

Fixed budgetICRT (px)BPP (px)
500 tasks x 20 demos9.44 ± 0.659.50 ± 0.27
1000 tasks x 10 demos6.59 ± 0.455.08 ± 0.13
2000 tasks x 5 demos5.07 ± 0.223.41 ± 0.13

Spreading a fixed demonstration budget across more tasks lowers error for ICRT as well as BPP, so the central task diversity finding holds for a second architecture, not only for BPP.

Beyond this small study of prompt encoder size, we do not study how pretraining data or much larger models affect the benefit of behavior prompting, and we now state this in the Limitations section.

Thank you for the thoughtful review. We hope these additions address your concerns, and we are open to discussing further.

Thanks for your response. I really like this work, and I think it should be a poster/spotlight at ICLR2027. Some of the concerns and weaknesses are not that easy to address (I know the author tries to figure out the pretraining parts, but it doesn't really show the final picture of why we need behavior prompting in pretraining/post-training). Behavior prompting is a good idea in research but seems not to be that useful when the task becomes much harder (which is the really important thing might be solved in the future as the LLM comes better and better). Congrats on your work! I keep the score because it seems it doesn't influence the acceptance. I think the work still needs some polish for an oral presentation. Anyway, thanks for your response!

Thank you for the continued support of this work! Studying how behavior prompting fits into the pretraining and post-training recipes of large policies to solve complex, real-world tasks is an important direction for future work, and we hope the findings and insights in our work serve as a foundation for such studies.

Reviewer_ieW8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors present this paper as an empirical study into behavior prompting. The work also presents two benchmarks (DrawAnything and LIBERO-GEN), and also evaluates on Laundry folding task, to evaluate performance on unseen drawing and table top manipulation tasks. The authors also propose Behavior Prompting policy (BPP) and iPhUMI (a handheld interface which lets humans specify behavior prompts at test time without a robot). The main claims include that task diversity matters more than demonstrations for each task and reduces error compared to goal-image conditioning and language conditioning.

Critical strengths

This paper is well motivated: is like a more mechanistic approach with examining behavior prompting and looks into details surrounding when it works, how the prompt is used, and why the prompt helps. The paper clearly defines what a behavior prompt consists of (a signal demonstration that consists of sequence of observations, proprioception, and actions in the root’s sensorimotor execution space). The authors present two benchmarks that are procedurally generated, which allows for various tasks (reconstruct an unseen drawing given demo in simulation and on a real robot, focuses on test-time instruction following capability on unseen tasks, and sequentially executing multiple tasks). The work compares against several baselines (goal-image, language, ICRT, and pi_0.5), and generally remarks that behavior prompting improves test-time adaptation for unseen tasks, task diversity is more important than the quantity per task (otherwise task conditioning weakens), and BPP relies on dense sub-goal conditioning, and dense sensorimotor detail in the prompt is helpful, which are nice high level insights for this very broad topic of behavior prompting.

Critical weaknesses

VLAs/policies are known to be sensitive to wording/phrasings for prompts (ie: for the Language baseline comparisons). It would be interesting to do another version for the Language baselines (and Pi0.5) that compared against different rephrasing/wordings for tasks and see if the gap is as big for performance compared to BPP. Is it also perhaps why the Language baseline does extremely well on laundry folding due to the three fold left arm, fold right arm, and fold bottom up are distinct and short language commands that don’t need any disambiguation.

The abstract mentions 80.7% reduction of error for goal-image conditioning: but this is only in simulation, real was 47.9%. Should make this clear in the abstract, otherwise it seems a bit misleading to report just one out of the two.

The paper repeatedly mentions task diversity as an important factor, however there is no clean measure/or number of how diverse apart the tasks are (especially within a benchmark), and it seems to be based on number and design of the individual benchmarks, and it is hard to compare what that means nicely across domains as well.

The improvement test-time adaptation to unseen tasks paragraph in 4.1 can be made more readable if it contains less number comparison within the main text (include within the tables/graphs or highlight in the captions instead).

Questions for authors

See weaknesses for more details but a summary of questions: Do the authors anticipate differences in performances of baselines depending on prompt wording/representation? Would be nice to compare against a rephrasing/wording adjustment version of the Language baseline and Pi0.5 to cleanly isolate whether it’s task diversity or wording sensitivity. Is there a measure of task diversity within a benchmark, and would be nice to have some examples as well? (ie: similarity between task embeddings or actionable objects or parameters in space?)

4 comments
AuthorsOpenReview ↗

Response to Reviewer ieW8 (1/2)

VLAs/policies are known to be sensitive to wording/phrasings for prompts (ie: for the Language baseline comparisons). It would be interesting to do another version for the Language baselines (and Pi0.5) that compared against different rephrasing/wordings for tasks and see if the gap is as big for performance compared to BPP.

Thank you for this suggestion. We ran a new experiment (Appendix L) that evaluates the Language baseline and π0.5 on the unseen LIBERO-Gen Chain tasks with the original instructions and four rewordings per task. We keep the trained models fixed (trained on original instruction wording) and reword the instructions at test time, which tests whether our wording of the unseen tasks puts the language-conditioned models at a disadvantage.

Table 11 in the revised PDF and included below.

WordingLanguage baseline success (%)π0.5 success (%)
Original76.679.4
Synonyms58.878.6
Reordered clauses39.419.0
Terse32.644.0
Two sentences63.667.2

Both the Language baseline and π0.5 do best with the original wording used in our main results, so the gap to BPP does not come from our choice of instruction wording for the unseen tasks. BPP, which does not use language, reaches 88.9% on these tasks.

Is it also perhaps why the Language baseline does extremely well on laundry folding due to the three fold left arm, fold right arm, and fold bottom up are distinct and short language commands that don't need any disambiguation.

We agree, and this is consistent with our interpretation in Section 4.1 (Fig. 6). With only three tasks, three short, distinct commands are an effective task descriptor, while a behavior prompt's spatial and temporal detail does not further disambiguate the task and instead adds variation to the task descriptor (varying prompt lengths and content), which we hypothesize complicates task identification. This is what we expect at low task diversity. A behavior prompt offers the most when it provides additional visual or temporal information that informs task execution or simplifies policy reasoning.

The abstract mentions 80.7% reduction of error for goal-image conditioning: but this is only in simulation, real was 47.9%. Should make this clear in the abstract, otherwise it seems a bit misleading to report just one out of the two.

We have fixed this. The abstract now reports 80.7% in simulation and 47.9% on a real robot.

AuthorsOpenReview ↗

Response to Reviewer ieW8 (2/2)

The paper repeatedly mentions task diversity as an important factor, however there is no clean measure/or number of how diverse apart the tasks are (especially within a benchmark), and it seems to be based on number and design of the individual benchmarks, and it is hard to compare what that means nicely across domains as well.

We added a task diversity analysis in new Appendix K. For each benchmark it reports spread (how different the training tasks are from each other) and coverage (how far each unseen task is from its nearest training task), and shows the most similar training tasks for every unseen task:

  • DrawAnything-Sim: distances between CLIP embeddings of the finished drawings (Table 8, Fig. 20), and every unseen drawing next to its nearest training drawings (Fig. 21).
  • LIBERO-Gen: distances in the task parameters themselves, the (pick location, place location) of a Combination task and the (primitive, object, target) of each step of a Chain task (Table 9, Table 10).

What these results show is summarized at the top of the Appendix K section of the linked page. Diversity measures differ by domain, so we compare them within a benchmark, not across domains; we explore multiple domains in this work to provide empirical understanding of behavior prompting in multiple settings.

The improvement test-time adaptation to unseen tasks paragraph in 4.1 can be made more readable if it contains less number comparison within the main text (include within the tables/graphs or highlight in the captions instead).

We made this paragraph (Section 4.1) more readable by reporting relative changes (percent reductions and gains) instead of raw error values.

Do the authors anticipate differences in performances of baselines depending on prompt wording/representation? Would be nice to compare against a rephrasing/wording adjustment version of the Language baseline and Pi0.5 to cleanly isolate whether it's task diversity or wording sensitivity.

Please see our first response above.

Is there a measure of task diversity within a benchmark, and would be nice to have some examples as well? (ie: similarity between task embeddings or actionable objects or parameters in space?)

Yes, please see the related comment above and new Appendix K, which follows both of your suggestions: task-embedding similarity for DrawAnything-Sim, and actionable objects and parameters for LIBERO-Gen. Fig. 21 shows every unseen drawing next to its nearest training drawings, and Appendix K.2 lists every unseen LIBERO-Gen task with its most similar training tasks (also listed on the rebuttal website).

We appreciate the thoughtful review, hope these changes address your concerns, and we would be happy to answer further questions. Please let us know if there is anything else that would help in reassessing the paper.

Thank you for your rebuttal, many of my concerns have been addressed. I will raise my score.

Thank you again for your detailed review, and we are glad that our rebuttal has addressed many of your concerns.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to reviews

We thank the reviewers for their constructive feedback. We have uploaded a revised PDF in which changes made in response to the reviews are highlighted in blue. All section and figure references in our responses refer to the revised PDF. For convenience, this page collects every new or updated figure and table, grouped by section. The revised PDF describes these results in more detail than the page or our responses, so we ask the reviewers to refer to it as well for additional details.

Main additions in the revision:

  • LIBERO-Gen Chain versions of our DrawAnything-Sim ablations, for training data (Fig. 5d-f) and prompt contents (Fig. 8d-f), with a new finding in Section 4.3: which prompt contents are necessary depends on the task distribution.
  • Prompt interventions that remove, truncate, shuffle, and reverse the prompt at test time on both benchmarks (Appendix F, summarized in Section 4.2) to support the findings from the attention analysis.
  • Task diversity analysis with spread and coverage measures and the most similar training tasks for every unseen task (Appendix K).
  • Language rewording test of the Language baseline and π0.5 (Appendix L).
  • Prompt encoder capacity sweep (Appendix M) and a replication of the task diversity finding with ICRT (Appendix N), both with three seeds.

Dear Reviewers,

A quick reminder that the discussion period ends on November 18 (AoE). If you haven't already, please review the authors' rebuttal and provide a brief follow-up indicating whether your concerns have been adequately addressed or if further clarification is needed.

Your timely feedback would be greatly appreciated and will help facilitate the final decision.

Thank you for your time and contributions!

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.