acceptodds
Under review as a conference paper at ICLR 2027

Building Exploratory Robot Policies via Adversarial Mid Training

Abstract

Robot policies today are predominantly trained on human-collected data, which naturally concentrates on competent, successful behavior. As a result, such data provides limited coverage of off-distribution states and diverse low-level behaviors needed to solve complex tasks over long horizons or improve through their own practice. In this paper, we study how to explicitly build this coverage into general-purpose robot policies. Specifically, we study a distinct collection protocol, adversarial data collection: while a human attempts a task, an adversary reversibly perturbs the environment or the task, forcing the operator to encounter unfamiliar states and adapt via low-level behaviors. We collect such rollouts, relabel segments with natural-language annotations identifying the local outcome pursued in each segment, and use them to midtrain a history-conditioned pretrained robot policy. The resulting policy retries after failures, makes sustained progress over longer horizons, and internalizes diverse ways of solving a given task. Across several long-horizon tasks on three real-robot platforms and in simulation, compared to training with standard demonstration data, our approach yields better scaling of zero-shot performance, greater robustness and generalization to novel configurations, gains from additional test-time compute (i.e., longer rollouts), and faster improvement during online finetuning. Project page and robot videos are available at https://explr-beta.pages.dev/https://explr-beta.pages.dev/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, Mid Training, Imitation Learning, Online Adaptation, Vision-Language-Action Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HWDLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies on how to construct a dataset for behavioural cloning such that it covers a large variety of states that can be found at deployment time. This makes sure that the learned policies result more robust, for example to sim-to-real gaps.

The crucial idea is to perturb the environment while the learning process is being carried out. That is while the robot is trying to solve a manipulation task one of the objects can be rotated or shifted by the adversary. When the agent manages to solve the task despite the perturbations this trajectory is added to a dataset which is then used for behavioural cloning.

Experiments carried out in a simulated task (64 grocery packing tasks) and 3 bimanual real world tasks.

For additional efficiency, the authors choose to (D1) consider a success the accomplishment of each subtask under adversarial disturbances rather than considering as success only the solution of the full task. A second decision (D2) is to consider some degree of non-Markovianity in which the learner policy is not conditioned only on the current state but on a short portion of the history (i.e. 2 seconds window of recent history)

The experiments show that β\beta-DATA achieve higher success rate compared to the expert policy on both simulations and real robot tasks (see Table 1) and the performance keeps increasing with larger amount of training data and test-time compute (see Figure 5) . D2, i.e. the decision to add non Markovianity, helps especially in Long OOD tasks but less than considering subtasks as individual successes which seems to be the crucial ingredient for OOD generalization.

Finally, section 7 shows that β\beta-DATA also offers a better initial checkpoint for online finetuning compared to standard imitation learning.

Critical strengths
  1. The method is simple and well explained.

  2. I really enjoyed the design of the experiments which really aim at understanding which are the desirable characteristics of β\beta-DATA.

Critical weaknesses
  1. why in Table 1 the success of Both the expert and β\beta-DATA is better Out Of Distribution compared to In Distribution? I would have expected the opposite

  2. Concerning the tuna can example in Figure 4, can you clarify if β\beta-DATA ever saw an expert demonstration in which the tuna can was grasped successfully. I find it surprising if β\beta-DATA learned to do something that the expert systematically fails to do. Please elaborate on this point.

  3. No baselines have been used for comparison. Why only standard BC is used as comparison and not recent variants that improves the coverage of BC such as PostBC ([1]).

  4. I think that the need for introducing disturbances during the data collection time can make it way more complex than the one for standard behavioural cloning. The authors should be transparent about this limitation.

  5. I think that there is an important missing citation ([2]) concerning the role of action chuncking and noise injection at data collection time.

I think that one weakness on which the authors should work on is that some plots are difficult to read, see for example my comment on Figure 6.

[1] Posterior behavioral cloning: Pretraining bc policies for efficient rl finetuning Wagenmaker et al. 2025

[2] Action Chunking and Exploratory Data Collection Yield Exponential Improvements in Behavior Cloning for Continuous Control, Zhang et al. 2025

Questions for authors

Q1) Does standard in Figure 5 refers to standard behavioural cloning?

Q2) I do not understand the legend of Figure 6a, it seems that the gray boxes in the legend (No history and 120 step history) are not plotted. If I understand correctly you mean that darker orange or darker blue means with history but this is just a guess and I think that the legend should do a better job at making this clear.

Q3) In Table 2, why does success decreases for expert data and progress decrease for both approaches? Moreover what is the meaning of the numbers in brackets (e.g. [48.1, 85.5]).

5 comments
AuthorsOpenReview ↗

Response to Reviewer HWDL (1/3)

Thank you for your feedback. To address your concerns we add the requested comparison with PostBC, explain why OOD success exceeds ID success in Table 1, clarify the tuna can example in Figure 4, and provide an additional comparison from the same initial state. We also explain the bracket values in Table 2 and correct the legend in Figure 6(a). Our new results show that β\beta-DATA achieves higher success rates than PostBC, whose performance remains close to that of BC.

"Why in Table 1 the success of both the expert and β\beta-DATA is better Out Of Distribution compared to In Distribution?" (W1)

Thanks for the question, we indeed agree this is confusing. The two splits contain different tasks, and OOD tasks are not necessarily harder than the ID tasks in terms of raw success rates. They are just different configurations from what was used for training of the policy. The OOD split has three Short, four Medium and five Long tasks. The ID split has three Medium and five Long tasks and no Short tasks (Appendix A). We also evaluated both policies on four additional short ID tasks with two subtasks each, and both succeed in all 200 trials (Appendix A). As such, higher success rate on the OOD split is true for both the β\beta-DATA and standard demonstration trained policies, and not an artifact of any of the methods.

Zero-shot success (Table 1)IDOOD
Medium tasks, β\beta-DATA91.3%91.5%
Medium tasks, standard demonstrations80.7%71.0%
Long tasks, β\beta-DATA34.0%57.6%
Long tasks, standard demonstrations20.0%39.2%

That said, on the Medium horizon tasks the order that you expect holds for standard demonstrations, which indicates that OOD success rates being higher is not a systematic pattern. The ID Long tasks are also much longer, with up to ten subtasks against at most five for the OOD Long tasks (Appendix A, Table 3).

"Can you clarify if β\beta-DATA ever saw an expert demonstration in which the tuna can was grasped successfully?" (W2)

Figure 4 illustrates that broader low-level behavioral coverage matters even when the subtask is already specified. Both policies receive the same subtask instruction and know what to do, and they differ in whether their low-level execution completes the subtask (Section 6). To answer your question, yes. Both datasets are collected by human operators, and both contain demonstrations that grasp and pack the tuna can.

The paper does not claim that the policy trained on standard demonstrations systematically fails on the tuna can. Figure 4 shows one selected rollout of each policy from the same state on OOD Task 9 (Appendix A, Table 3), as an example where the policy trained on standard demonstrations fails and the β\beta-DATA policy succeeds.

This difference does not rest on one rollout. In the new Figure 20 in Appendix M, from the same initial state in simulation, the β\beta-DATA policy places a sugar box, and when the box slides over the tray rim, it regrasps the box, sets it flat in the tray, and packs all five objects. The policy trained on standard demonstrations holds the sugar box beside the tray until 100 s. In Figure 21 in Appendix M, on the real robot, both policies are instructed to pack a jar that lies on its side. The policy trained on standard demonstrations repeats the same grasp below the lid, and the jar stays on the table. The policy trained on β\beta-DATA lifts the jar, and when the jar falls, it grasps the jar across its body and packs it.

AuthorsOpenReview ↗

Response to Reviewer HWDL (2/3)

"Why only standard BC is used as comparison and not recent variants that improves the coverage of BC such as PostBC" (W3)

Thanks for the great suggestion. We now have run PostBC, and Section 6 now reports the comparison. We midtrain a PostBC policy on the same standard demonstrations, starting from the same pretrained π0.5\pi_{0.5} checkpoint. The policy uses an ensemble of 20 members trained on frozen policy features and a noise scale of α = 1.

Zero-shot successPostBCStandard demonstrationsβ\beta-DATA
Twelve OOD tasks64.0%62.7%79.2%
Eight ID tasks43.0%42.8%55.5%

PostBC performs comparably to midtraining on standard demonstrations, and it underperforms β\beta-DATA on both splits. Upon visually inspecting rollouts, we do actually see that PostBC broadens the action distribution, but it does not add the transitions between abstract states that β\beta-DATA contains to improve recovery and correction behaviors upon deployment.

"the need for introducing disturbances during the data collection time can make it way more complex than the one for standard behavioural cloning" (W4)

Real-world collection of β\beta-DATA does in fact need a second person: two humans jointly collect each rollout, an operator and an adversary (Section 5). Each hour of real-world β\beta-DATA therefore takes two person hours.

To address this concern is precisely why we ran the data scaling experiments in Figure 5. This experiment showcases that training on half as much data can result in β\beta-DATA matching standard demonstrations: 2.6 hours of β\beta-DATA reach 60.3% success comparable to 58.7% with 5.2 hours of standard demonstrations (Section 6, Result 2). This means that at this budget, for the same total human time, we are able to get a matching policy. But in addition, we also find that continuing to scale β\beta-DATA further keeps on providing improvements, and β\beta-DATA stays ahead of standard demonstrations at every data budget from 1.3 hours onward (Figure 5). β\beta-DATA also provides for a better initialization and more effective online finetuning as shown by the new scaling law results in our response to Reviewer 8Dfn (Appendix L, Figure 19), which further justifies why β\beta-DATA is preferable despite needing two humans for collecting one datapoint. It is more data efficient, results in better initializations for online policy performance, and leads to improvements in zero shot performance.

"there is an important missing citation ([2]) concerning the role of action chunking and noise injection at data collection time" (W5)

Thank you for these references. The revised paper now cites Zhang et al. (2026) in Section 1 and in Section 4. Section 4 now states that injecting noise into the actions of the expert during data collection, or predicting action chunks, avoids the exponential compounding of imitation errors. Section 1 states that noise injection into the policy or the teleoperation interface broadens experience by perturbing actions, but offers limited control over semantically useful challenging states. Our protocol changes the environment or the task and records how the operator continues, and it does not perturb the actions of the operator. The result of Zhang et al. is a worst case bound, and we do not claim that errors from training on standard demonstrations compound exponentially. Our claim is that standard demonstrations lack the low-level behaviors needed to sustain long rollouts and to connect challenging states to states from which further progress is possible.

Our comparison with PostBC gives a similar picture. PostBC tries to make the action chunks at each state more robust for downstream adaptation, but it broadens the action distribution without improving connected behavioral coverage meaningfully (Section 4). These connecting behaviors matter for test-time scaling, zero-shot success and online improvement.

"some plots are difficult to read" and "I do not understand the legend of Figure 6a" (W6, Q2)

Thank you for pointing this out. We have fixed the legend of Figure 6(a) in the revised paper, and we hope it is clearer now.

AuthorsOpenReview ↗

Response to Reviewer HWDL (3/3)

"Does standard in Figure 5 refers to standard behavioural cloning?" (Q1)

Yes. "Standard" denotes the same pretrained π0.5\pi_{0.5} policy midtrained on standard demonstrations, with the same objective, relabeling and training settings as the β\beta-DATA policy (Section 5, Appendix E).

"In Table 2, why does success decreases for expert data and progress decrease for both approaches? Moreover what is the meaning of the numbers in brackets (e.g. [48.1, 85.5])." (Q3)

The brackets are 95% intervals for the values after online finetuning. The interval for success is a Wilson interval, and the interval for task progress is a percentile bootstrap interval. The revised caption defines them now. For example, [48.1, 85.5] is the interval for the 70% success of the β\beta-DATA policy.

Real-world packing (Table 2)Before online finetuningAfter online finetuning
β\beta-DATA, full-task success35%70%, interval [48.1, 85.5]
β\beta-DATA, task progress91.7%90.6%, interval [82.2, 97.2]
Standard demonstrations, full-task success20%15%, interval [5.2, 36.0]
Standard demonstrations, task progress66%53.3%, interval [41.7, 65.6]

The policy trained on standard demonstrations relies on spurious cues in the scene, such as packing an item that the instruction does not name. Its attention does not fall on the instructed item, while the β\beta-DATA policy attends to it (Appendix I, Figure 14). Online finetuning on its own successful rollouts reinforces this reliance, which is why both its success and its task progress doesn't improve. With β\beta-DATA, we observe that the policy acts on the instructed object more reliably and recovers from failed grasps more often (Appendix I). Its success rises from 35% to 70%, and its task progress stays within its interval (91.7% to 90.6%, interval [82.2, 97.2]). Real-robot videos of both policies before and after online finetuning are on our anonymous project website (https://explr-beta.pages.dev/).

If our revisions address your concerns, we would be grateful if you could consider raising your score.

References

  • Huang et al. (2025). Adversarial data collection: Human-collaborative perturbations for efficient and robust robotic imitation learning. arXiv:2503.11646.
  • Wagenmaker et al. (2025). Posterior behavioral cloning: Pretraining BC policies for efficient RL finetuning. arXiv:2512.16911.
  • Zhang et al. (2026). Action chunking and data augmentation yield exponential improvements in behavior cloning for continuous spaces. ICLR 2026 (arXiv:2507.09061).
Reviewer_GEuKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper focuses on data generation technique that enhances the coverage for training robot policies. The idea uses an adversarial framework where there is an operator collecting teleoperation data while there is an adversary perturbing the trajectories. The paper shows that using this data collection technique improves the OOD generalization and requires less train data and test-time scaling.

Critical strengths

None.

Critical weaknesses
  1. In terms of contribution, I fail to see what this paper builds on top of adversarial data collection (Huang et al., 2025) in terms of methodology.
  2. On lines 250-256, the paper claims the state-and-action diversity is insufficient, but there is no evidence in this paper nor references supporting these claims.
  3. The data-collection protocol may not be reproducible: there is no consistent rubric or metric for human adversary. For example, a human adversary can make the task impossible the solve, rather than undo'ing the operator's current trajectory with the potential to recover. Furthermore, it is difficult to say whether the "OOD tasks" are still OOD once the adversary "perturbs" by making the training data to be similar to the OOD tasks.
Questions for authors
  1. On lines 389-392, the paper claims that this method is important on long-horizon tasks. I believe this claim is well supported in the paper, but are there any results showing that the agent is not trying the same behaviour (as many LLM papers show that the model usually does repeat [1]).

References

  • [1]: Pipis, Charilaos, et al. "Wait, Wait, Wait... Why Do Reasoning Models Loop?." arXiv preprint arXiv:2512.12895 (2025).
5 comments
AuthorsOpenReview ↗

Response to Reviewer GEuK (1/3)

Thank you for your constructive feedback. To address your concerns, we define and measure the behavioral coverage targeted by our collection protocol, clarify the constraints on the adversary and the OOD split and add a qualitative comparison from the same initial state together with the rollout time results that address repeated behavior.

"I fail to see what this paper builds on top of adversarial data collection (Huang et al., 2025) in terms of methodology" (W1)

The collection protocol is that of Huang et al. (2025), and we do not claim it as new. Section 5 attributes the protocol to them, and Section 4 states that our goal is not to propose a new protocol. The paper asks a different question: whether data with high connected behavioral coverage (CBC) is better midtraining data for a pretrained robot policy, both for zero-shot performance and for online finetuning. We do find that to get high CBC, adversarial data collection (ADC) is a valid approach, but the contribution of the paper is a clear study of data scaling behavior, test-time compute scaling behavior, and online improvement behavior when training with data of different CBC values.

While our original submission did not include a measure of connected behavioral coverage, we are now adding a new quantitative metric for measuring CBC to make our claims precise. Section 5 in the revised PDF defines CBC as the number of distinct directed edges between abstract states in the annotated demonstrations collected from any data collection strategy (Eq. 1), and Figures 17 and 18 in Appendix K measure it. At matched data budgets β\beta-DATA contains 85 edges versus 54 for standard demonstrations in simulated packing, 125 versus 72 in real-world packing, and 19 versus 15 in LEGO disassembly. Moreover, as we scale the total number of hours of each type of data, the number of edges in β\beta-DATA grows faster than that in standard demonstration data, further illustrating the point that collecting data with an adversarial protocol improves CBC.

Finally, our new scaling laws for online finetuning (Appendix L, Figure 19), and our data scaling result in Figure 5 already establish the benefits from scaling up β\beta-DATA, and address questions around its interplay with the extra human effort that collection demands.

ADC motivates its protocol by broader state and action coverage. CBC counts the connections between states that a dataset demonstrates, and our work additionally studies how these connections affect a pretrained robot policy. We find that data with high CBC scales more favorably with both training data and test-time compute (Figure 5), and that it makes online experience more useful: policies trained on β\beta-DATA start from a higher success rate and improve more from the same number of successful rollouts (Appendix L, Figure 19). These studies of data scaling, test-time compute scaling and online improvement are new to this work, and we believe they should be valuable to any robotic practitioner looking to scale data for training robot policies. We therefore believe our work should be of value and interest to the robot learning community.

In addition to our broad contribution listed above, methodological decisions that we make are also novel in the context of training on our data and interesting for downstream practitioners. We relabel each trajectory segment with its local outcome (D1), and we condition the policy on recent action history (D2). ADC also uses subtask-level labels, but it does not evaluate this choice separately, and it targets "memory-less" policies, whereas we condition on action history. Both choices raise the success of β\beta-DATA on OOD Long tasks (Figure 6), and the collection protocol does not imply this result.

β\beta-DATA policyWithoutWith
120-step action history (D2)41.6%57.2%
Subtask annotations (D1)20.0%57.6%
AuthorsOpenReview ↗

Response to Reviewer GEuK (2/3)

"The paper claims the state-and-action diversity is insufficient, but there is no evidence in this paper nor references supporting these claims" (W2)

This is indeed a valid question and to address this, we have now quantified our claim precisely and added a new measure of diversity that precisely points out the notion of coverage/diversity we are referring to.

We define connected behavioral coverage (CBC) in Section 5. A dataset of rollouts can be represented as a graph, where nodes correspond to abstract states of the environment and robot, and directed edges represent transitions demonstrated in the dataset. Each annotated trajectory segment manipulates one object, taking it from a starting abstract state to an ending abstract state. When the two states differ, the segment demonstrates a transition between them, which we count as one directed edge from the first to the second. An abstract state records the tracked object and its location. The CBC of a dataset is the number of distinct directed edges in its demonstrations at a given data budget (Eq. 1). Each distinct directed edge counts once, regardless of how often it appears or which low-level action sequences produce it.

The revision also adds two results that support this and at the same time it words the statement more precisely. Sections 1 and 5 now say that diversity over states or actions alone does not ensure coverage of the low-level action sequences needed to recover and resume progress from challenging states.

State diversity. The CBC measurement compares the two datasets on the same abstract states. For instance, in the simulated grocery packing task, both datasets contain a cracker box on the table and to be packed into each of the three trays. β\beta-DATA connects these four states with seven distinct directed edges, while standard demonstrations connect them with only three edges (Figure 17). Thus, β\beta-DATA adds new transitions between the same abstract states, and not merely new states.

Action diversity. Posterior behavioral cloning (PostBC, Wagenmaker et al., 2025) attempts to broaden pretrained robot-policy behavior by modeling uncertainty in demonstrator actions. It therefore tests whether more diverse actions, on the same data, are enough. We midtrain a PostBC policy on the same standard demonstrations, starting from the same pretrained π0.5\pi_{0.5} checkpoint (Section 6).

Zero-shot successPostBCStandard demonstrationsβ\beta-DATA
Twelve OOD tasks64.0%62.7%79.2%
Eight ID tasks43.0%42.8%55.5%

PostBC performs comparably to midtraining on standard demonstrations, and it underperforms β\beta-DATA. Broadening the action distribution of a policy trained on standard demonstrations therefore does not reach the success of midtraining on β\beta-DATA.

"there is no consistent rubric or metric for human adversary" (W3, first part)

We do not give the human adversary a fixed rubric, because the protocol relies on the adversary choosing where to challenge the operator. Instead, every intervention must leave the task feasible (meaning actions of the adversary can be undone if needed by the operator), with the relevant objects within reach of the robot. The adversary changes the scene or the goal, and never touches the robot. For example, the adversary does not pull an object out of a gripper. The adversary also gives the operator time to respond before the next intervention. The objective of the operator remains task completion, and the operator is never instructed to behave diversely or to produce a particular recovery strategy. After each intervention the operator continues from the perturbed state without restarting (Section 5).

The revision also adds a metric for the collected data, the CBC defined above (Section 5, Eq. 1). This metric measures the data and not the adversary, so it checks the outcome of the protocol. It can be computed for any annotated dataset collected with this protocol, and Figure 18 reports it as a function of the data budget.

The benefit does not depend on one particular set of demonstrations. Policies trained on 1.3, 2.6 and 5.2 hours of β\beta-DATA each succeed more often than policies trained on the same number of hours of standard demonstrations (Appendix F, Table 6).

Hours of dataβ\beta-DATAStandard
1.343.2%26.8%
2.660.3%46.5%
5.269.3%58.7%
AuthorsOpenReview ↗

Response to Reviewer GEuK (3/3)

"a human adversary can make the task impossible the solve" (W3, second part)

To clarify, we make sure that the perturbations leave the task solvable. Figures 3, 11 and 15 show examples. The adversary returns a packed item to the table, adds objects to the goal, joins two bricks, or moves a placed brick to another tray. In each example the operator undoes the perturbation or completes the changed task. We also label each trajectory segment with the local outcome that the operator achieved (D1, Section 5). No segment is therefore trained with an instruction that the operator did not accomplish.

"it is difficult to say whether the "OOD tasks" are still OOD once the adversary "perturbs" by making the training data to be similar to the OOD tasks" (W3, third part)

We would like to clarify that the split between in-distribution and out-of-distribution is defined by the task, and perturbations do not change which tasks are held out. They only change the scene or the goal during an episode of a training task. The twelve OOD simulation tasks have held-out goals and configurations, and they are separate from the training tasks (Section 6 and Appendix A, Table 3). The comparison also holds on tasks that are drawn from the training set. On the eight ID tasks, zero-shot success is 55.5% for β\beta-DATA versus 42.8% for standard demonstrations (Appendix D, Table 4).

In the real-world packing evaluation, the task involves packing pink gum that does not appear in either midtraining dataset, so no perturbation during collection can have covered it. Across the twenty zero-shot trials of each policy, we observe that it ends in the box in all 20 trials for the β\beta-DATA policy and in 15 for the policy trained on standard demonstrations.

The gains therefore do not come from just covering the OOD tasks during collection. For the final version, we will run a quantitative analysis of how close the perturbed training data comes to the OOD tasks.

"are there any results showing that the agent is not trying the same behaviour" (Q1)

Yes. The revision adds a qualitative comparison from the same initial state, and the rollout time results give the rates.

The new Figure 20 in Appendix M shows both policies on an OOD simulation task, where both are instructed to pack the sugar box into the medium tray. The policy trained on β\beta-DATA places the sugar box. When the box slides over the tray rim, this policy regrasps the box, sets it flat in the tray, and packs all five objects. The policy trained on standard demonstrations lifts the sugar box and holds it beside the tray until 100 s.

Additional rollout time produces completed tasks, and repeating a failing behavior would not. The table gives success on the OOD simulation tasks (Section 6, Result 2, and Appendix F, Table 6).

Time limit per rolloutβ\beta-DATAStandard
Two minutes55.5%47.8%
Four minutes65.2%56.7%
Six minutes69.3%58.7%
Rollouts unfinished at four minutes that succeed by six minutes12.0%4.6%

The repetition that you describe appears in the policy trained on standard demonstrations. In Figure 4 the β\beta-DATA policy packs the tuna can, while from the same initial state the policy trained on standard demonstrations repeatedly fails to grasp and lift the tuna can, till failure. In Figure 16, on the real robot, the β\beta-DATA policy separates the bricks of a LEGO structure, while the policy trained on standard demonstrations repeatedly regrasps the structure and leaves its bricks connected. In Figure 21 in Appendix M, on the real robot, both policies are instructed to pack a jar that lies on its side. The policy trained on standard demonstrations repeats the same grasp below the lid, and the jar stays on the table. The policy trained on β\beta-DATA lifts the jar, and when the jar falls, it grasps the jar across its body and packs it. Furthermore, Figure 1 shows three rollouts of the β\beta-DATA policy that start from the same initial LEGO block assembly and task goal. Slight deviations in grasps, orientations and contact interactions lead to diverse sequences of outcome states: the top and middle rollouts each detach a different red brick, and the bottom rollout separates all the bricks.

If our revisions address your concerns, we would be grateful if you could consider raising your score.

References

  • Huang et al. (2025). Adversarial data collection: Human-collaborative perturbations for efficient and robust robotic imitation learning. arXiv:2503.11646.
  • Wagenmaker et al. (2025). Posterior behavioral cloning: Pretraining BC policies for efficient RL finetuning. arXiv:2512.16911.
Reviewer_8DfnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies adversarially perturbed human demonstrations as midtraining data for a pretrained robot policy. An operator continues toward task completion while a second person changes the scene or goal; the resulting trajectories are annotated with local outcomes and used to train a policy conditioned on recent actions. Experiments compare this corpus with standard demonstrations in simulated bimanual packing and three real-robot settings, examine data and rollout-duration scaling, and evaluate subsequent filtered behavioral cloning from successful autonomous rollouts.

Critical strengths
  1. The empirical benefit extends beyond selected recovery videos. With approximately matched recording duration and common training settings, simulation success increases from 547/1,000 to 697/1,000 trials (Tables 1 and 4), including an 18.4-point improvement on long OOD tasks. The real LEGO interlock result, 5/20 versus 19/20, is also substantial. Applying outcome relabeling to both corpora and comparing both with and without action history makes the main comparison more informative than an unmatched comparison of two complete architectures.

  2. The rollout-duration analysis gives concrete evidence of sustained progress. Table 6 evaluates the same trained policies at different time limits and reports late completion conditional on still being unfinished at four minutes. At the largest data budget, approximately 12% of the remaining β-DATA rollouts succeed by six minutes, versus 4.6% for standard data. This is more informative about productive continuation than the pass@k figures, which Appendix D correctly identifies as separate attempts with resets. Together with the action-history ablation, it supports the practical value of training on interrupted execution.

Critical weaknesses
  1. The proposed explanation in terms of “connected behavioral coverage” is not separated from broader state and task coverage. Adversarial collection changes object arrangements, task goals, and the frequency of repeated manipulations, as well as introducing connected recovery trajectories. The comparisons with ordinary demonstrations establish that the resulting corpus is useful, but do not identify which of these changes causes the gain. Figure 6 tests history and instruction granularity, not the necessity of collecting difficult states inside uninterrupted trajectories rather than through suitably varied initial states. This matters because the claim that state diversity alone is insufficient is a central distinction drawn in Sections 2 and 5. The present evidence warrants a narrower mechanism claim, or a targeted comparison matching difficult states and subgoals while varying how recovery transitions are collected.

  2. Matching the number of successful online trajectories does not isolate their usefulness. Section 7 interprets Table 2 as evidence that β-DATA produces more useful online supervision. However, the two learners start from different checkpoints and Appendix I supplies 192/256 training examples from each policy’s own offline corpus. Thus both initialization and most replay data remain different. The observation that 95% of the expert policy’s successes come from one arrangement is suggestive, but the collection distribution and acceptance procedure need clarification: the text contrasts those successes with re-randomized configurations for β-DATA. The result establishes a benefit of the complete training pipeline, not a causal effect of the online trajectories alone. A shared initialization/replay control or a crossed comparison of the collected success buffers would distinguish these explanations.

Questions for authors
  1. Were the two collection protocols matched in task/subgoal frequencies and exposure to difficult configurations? Is there evidence separating the benefit of connected recovery transitions from simply presenting more varied states and goals?
  2. In the real-world online experiment, were initial arrangements drawn from the same distribution with identical success-admission rules for both policies? What were the total collection attempts? Were the successful buffers ever compared while holding initialization and offline replay fixed?
  3. What is the total human effort, including the second operator and annotation, for the recording budgets in Figure 5? The approximately twofold gain in recorded-data efficiency is useful, but it need not imply a twofold saving in collection labor.
7 comments
AuthorsOpenReview ↗

Response to Reviewer 8Dfn (1/4)

Thank you for the detailed and constructive review. To address your concerns, we define a concrete metric of connected behavioral coverage (CBC) and measure it explicitly, precisely quantify the mechanism claim that you suggest for online finetuning and report the human effort behind each dataset in person hours. Overall, we find that β\beta-DATA achieves higher connected behavioral coverage and is more effective even at total matched human time. We also present new results showing empirical scaling law for online finetuning with standard demonstrations and β\beta-DATA. These results show that both policy initialization and the quality of online experience contribute to faster and greater improvement during online finetuning.

"The proposed explanation in terms of connected behavioral coverage is not separated from broader state and task coverage" (W1, and the second question of Q1)

Thank you for the concern. We agree, and therefore, we have now added a new metric to quantify connected behavioral coverage that compares datasets concretely as well.

We define connected behavioral coverage (CBC) in Section 5. We begin by noting that a dataset of rollouts can be represented as a graph, where nodes correspond to abstract states of the environment and robot, and directed edges represent transitions demonstrated in the dataset. Each annotated trajectory segment ii manipulates one object, taking it from a starting abstract state uiu_i to an ending abstract state viv_i. When the two states differ, the segment demonstrates a transition between them, which we count as one directed edge from uiu_i to viv_i. For a dataset D\mathcal{D} at a data budget BB, measured in recorded hours, CBC(D;B)=∣{(ui,vi):i∈I(DB),ui≠vi}∣\mathrm{CBC}(\mathcal{D}; B) = \lvert \lbrace (u_i, v_i) : i \in \mathcal{I}(\mathcal{D}_B), u_i \neq v_i \rbrace \rvert, the number of distinct directed edges in its annotated segments (Eq. 1), where I(DB)\mathcal{I}(\mathcal{D}_B) indexes the annotated segments of the demonstrations selected at budget BB. Each distinct directed edge counts once, regardless of how often it appears or which low-level action sequences produce it. We have now updated Section 5 of the paper with this definition.

Domain (matched data budget)Edges, β\beta-DATAEdges, standard demonstrations
Simulated packing8554
Real-world packing12572
LEGO disassembly1915

Measurement. We measure CBC on the datasets collected in all three domains and present the results in the table above. Additionally, to control the confounding variable suggested in your review of state coverage being different, i.e. the two data collection protocols may also create datasets that differ in their respective state coverage, we additionally analyzed CBC under matching state coverage settings as well and present our results below.

β\beta-DATA has 36 distinct starting states versus 27 in simulated packing, 65 versus 36 in real-world packing, and 13 versus 9 in LEGO disassembly (Table 7). This showcases that β\beta-DATA clearly contains more states than standard demonstrations. To address the state coverage confound, we therefore restrict both real-world packing graphs to the 82 object-location states that both datasets contain. On these shared states β\beta-DATA still has 121 distinct edges versus 72 for standard demonstrations, a 68% increase. The same holds even for a single object. For instance, in simulated packing both datasets contain the cracker box on the table and in each of the three trays. β\beta-DATA connects these four states with seven distinct directed edges, and standard demonstrations connect them with three (Figure 17). Thus, β\beta-DATA adds new transitions between the same abstract states, and not merely new states. Hence, we believe that policies trained on β\beta-DATA work well not just due to state coverage but due to connections between states that enable them to generalize better in in-distribution and out-of-distribution settings at test time.

We also find that the edge count of β\beta-DATA grows faster with the data budget. On log-log axes the slopes are 0.30 versus 0.22 in simulated packing and 0.37 versus 0.15 in real-world packing (see Figure 18).

AuthorsOpenReview ↗

Response to Reviewer 8Dfn (2/4)

"Matching the number of successful online trajectories does not isolate their usefulness" (W2, Q2)

Thanks for the question. We would first like to clarify that Section 7 gives two reasons why midtraining on β\beta-DATA could help online finetuning. First, the policy generates successful experiences more easily and efficiently. Second, its successful rollouts could provide richer supervision as they exhibit more interesting behaviors that, upon reinforcement, can improve downstream successes in the environment. The simulation results support both reasons: β\beta-DATA gives a better initialization, and it also learns more efficiently from the successful trajectories that online finetuning collects. In this rebuttal, we provide a new result and scaling law showing that for the same number of retained successes, β\beta-DATA policies also improve more (see below).

To disentangle and identify the relative contribution of the various factors, we conduct an empirical scaling law analysis on online finetuning in our simulation tasks. Concretely, we aim to fit a law to the success rate of the online finetuned policy as a function of the initial success rate of the base policy and the online interaction budget, specified as the number of successful rollouts collected during online finetuning. By empirically fitting such a law independently for both standard demonstrations and β\beta-DATA we are able to assess the importance of the initialization checkpoint as well as the efficacy of learning from successful online rollouts collected by each policy. We hypothesize the functional form of the law as:

ln⁡pM1−pM=a+bln⁡p01−p0+dln⁡M\ln\frac{p_M}{1-p_M} = a + b \ln\frac{p_0}{1-p_0} + d \ln M

Here, the constant aa defines the offset, p0p_0 denotes the success rate of the initial base model obtained by training on β\beta-DATA or standard demonstrations, and MM denotes the number of successful rollouts collected during finetuning, with dd as its coefficient. We note that MM does not denote the full interaction budget, but only the number of successful rollouts collected by each policy. We expect β\beta-DATA policies to produce more successful rollouts during finetuning because of their higher success rates, but when we fit this law with only the successful rollouts, the coefficient dd helps us understand the effect of the “quality” of the successful experience from each method. Alternatively, we can say the coefficient bb gives the effect of the initial success rate on the final success rate, and dd gives the effect of the quality of the successful online experience collected by each policy.

We fit this law on the held-out long-horizon simulation tasks. β\beta-DATA policies start from a higher success rate (57.6% against 39.2%). The fit gives d=0.43d = 0.43 for β\beta-DATA and d=0.20d = 0.20 for standard demonstrations. Appendix L and Figure 19 show the fit.

We find that, in addition to starting from a higher success rate, β\beta-DATA attains a higher value of dd. The constant dd is meant to isolate how useful the successful rollouts are, both from the base policy and from the intermediate policies during online finetuning, and a higher value of dd means that online experience is more effective at improving the success rate of the policy. Therefore, successful rollouts from policies trained on β\beta-DATA are more effective at improving performance. This showcases that the gains from β\beta-DATA do not just come from a better policy initialization, but also from the higher quality of the successful rollouts that it collects during finetuning.

AuthorsOpenReview ↗

Response to Reviewer 8Dfn (3/4)

"The observation that 95% of the expert policy’s successes come from one arrangement is suggestive, but the collection distribution and acceptance procedure need clarification" (W2, second part)

We agree that the sentence on arrangements in Section 7 of the submission deserves more clarity. Both policies collect their successes in the same arrangements, and the revised paper corrects this sentence, which is now in Appendix I. The difference lies in what the collected successes reinforce, which we study qualitatively in our real-world experiments (Appendix I, Figure 13). Filtered behavioral cloning on successful rollouts can reinforce spurious correlations, such as picking up the item nearest to the gripper rather than the item that the instruction names. Before online finetuning, the β\beta-DATA policy packs the chip bag while it is instructed to pack the mustard, and after online finetuning it packs the mustard (Figure 13). After online finetuning, it also acts on the instructed object more reliably than the policy trained on standard demonstrations (Appendix I, Figure 14 shows the attention of both policies). The initial success rate alone does not capture this difference. Real-robot videos of these policies before and after online finetuning are on our anonymous project website as well (https://explr-beta.pages.dev/).

"In the real-world online experiment, were initial arrangements drawn from the same distribution with identical success-admission rules for both policies? What were the total collection attempts? Were the successful buffers ever compared while holding initialization and offline replay fixed?" (Q2)

Both policies use the same arrangements in online collection and in evaluation, and they use the same success rule: a trajectory is accepted only when all nine items are packed, each under its own instruction (Appendix C). Both runs also share the batch mix, the number of accepted successes and the number of gradient updates. (Appendix I).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.