acceptodds
Under review as a conference paper at ICLR 2027

Data Collection with Adaptive Perturbations Enables Effective Policy Learning

Abstract

Modern robot learning systems feed on high-quality demonstration data, and their performance hinges on coverage. Without good coverage, small errors compound and the policy can quickly go into out-of-distribution states from which it cannot recover. Yet there is no principled way of collecting high-coverage data, and data collection often relies on heuristic instructions to human operators. Purely online reinforcement learning (RL) explores at the cost of safety and sample efficiency, while DAgger-style interventions require the policy to fail by chance before a human can correct it. In this paper, we ask instead whether we can modify how the robot behaves during data collection to collect more useful data. We propose Data Inoculation (DI), a shared-control framework in which a human provide closed-loop teleoperation commands while a residual policy perturbs these commands. The residual policy is trained online, on the growing dataset, with the same reward-maximizing objective later used for policy extraction, so its perturbations steer data collection toward the deviations a policy trained on the data would make. These perturbations expose the robot to a wider range of states, while the human’s continuous corrections provide recovery demonstrations, "inoculating" the dataset against the learner's future mistakes while the human remains in control. On three challenging real-world robotic tasks, DI consistently improves or matches the success rate of policies trained with both behavioral cloning and offline RL, showing DI as an effective data collection strategy for improving downstream policy performance.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robotics, reinforcement learning, data collection, shared-control

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ce2UOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces Data Inoculation (DI), teleoperation data collection method. A residual policy adds a bounded perturbation to the teleoperator's command, and the policy is learned via off-policy RL on the growing dataset. The authors argue that the resulting perturbations mimic the deviations a downstream policy would make, so that teleoperation corrections produce targeted recovery data. On three real robot tasks, DI improves or matches downstream BC and offline RL success rates relative to clean teleoperation and uniform random perturbations.

Critical strengths
  1. The paper is easy to follow. The topic is well motivated, and the paper does a good job outlining and connecting to relevant literature.
  2. The physical robot experiment setup is good; 3 tasks + multiple seeds and sufficiently challenging
Critical weaknesses

My primary concern is that the empirical setup only compares DI with random action perturbation (a naive baseline) and no perturbation. Why not compare to DART and Dagger? Both target the same failure mode DI addresses (DART fits its noise to the learner’s errors, similar to DI). If either is incompatible with this setting I think ADC (Huang et. al 2025) is another relevant baseline but perhaps does not fit the specific RL setting this paper considers. In such case, the paper should describe why relevant/related baselines are excluded if there's a good reason. While I would agree that current experiments do answers the questions Q1-Q3 posed in the paper, missing baselines makes the utility of the method difficult to assess.

Experiments only consider 3 real world robotic tasks. This is a good physical evaluation, but giving that error bars are pretty close or overlapping in some setups (especially keycap), additional simulated experiments (e.g. similar to those in the DART paper) would strengthen the paper. The lack of improvement over clean in keycap means that we only have 2 tasks that show clear improvement. It may be difficult to run additional physical experiments, but simulated experiments can still provide useful signal and as easier to run.

Questions for authors

Why does the evaluation exclude DART and DAgger-style interventions?

Reviewer_51oKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how to improve demonstration data collection by modifying robot behavior during human teleoperation. The proposed Data Inoculation (DI) framework combines continuous human control with bounded perturbations generated by an online-trained residual policy. The residual policy optimizes task reward, while the human responds to its deviations and provides corrective actions. Experiments on three real-world manipulation tasks show that policies trained on DI data through behavior cloning or offline RL generally outperform or match those trained on clean demonstrations and random-perturbation data.

Critical strengths
  • The paper focuses on improving the data collection process rather than only modifying downstream learning algorithms. Addressing the limited coverage of conventional demonstrations is relevant to the robustness and data efficiency of robot learning.

  • Combining bounded learned perturbations with continuous human feedback provides a concrete mechanism for collecting corrective demonstrations throughout a trajectory. Optimizing the perturbation policy cooperatively through task reward is an interesting alternative to purely random or explicitly adversarial disturbances.

  • The evaluation includes three manipulation tasks and multiple data-collection runs involving different operators. Improvements are demonstrated with both behavior cloning and offline RL, particularly on cable insertion and cube balancing, suggesting that the collected data have value beyond a single downstream learner.

Critical weaknesses
  • The effects of asynchronous learning and human-action approximation are insufficiently characterized. The system trains the residual policy, critic, and BC human-action model concurrently, while periodically synchronizing weights with the execution process. However, the training throughput, synchronization interval, and resulting policy staleness are not reported. It therefore remains unclear whether delayed updates affect the perturbation policy’s ability to adapt to the evolving data distribution. Moreover, the BC model approximates the next human command when constructing the critic’s bootstrap target, but its prediction error and downstream effects are not quantified.

  • Figure 4 provides useful evidence that DI elicits human corrections during task-critical phases. However, this does not directly establish that downstream policies trained on the collected data acquire stronger recovery capabilities. More evaluations from perturbed or near-failure states, measuring recovery success, recovery time, and robustness to previously unseen deviations, would help determine whether DI improves error recovery rather than primarily improving nominal task execution.

  • The sources of the downstream performance improvements remain insufficiently disentangled. The reported success-rate gains establish the usefulness of DI in the evaluated settings, but do not fully explain why the collected data are more effective. Possible mechanisms include broader coverage of proprioceptive or task-relevant states, increased visual diversity, more informative recovery demonstrations, or more consistent and successful trajectories produced through learned assistance. The current phase-level correction analysis does not isolate these alternatives.

Questions for authors

see weakness

Reviewer_uYMDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces data inoculation (DI), which is a shared-control framework for teleoperated data collection designed to increase data coverage to promote downstream policy performance. DI trains a residual policy to perturb a teleoperator's commands in order to maximize the task rewards. The operator is responsible for responding to the perturbations in real time, with the goal of collecting recovery data that "innoculates" the downstream policies against their own mistakes. The collected datasets are used to train offline BC and RL policies with higher task success rates than clean datasets, or datasets collected with random noise injections.

Critical strengths
  1. The tasks are challenging and precise, which is the task family where policy learning methods that rely solely on offline data often struggle. This makes the task suite a good testbed for this type of data inoculation contribution.
  2. Treating teleoperators as "seeds" is a good practice for data collection papers; I wish more data collection contributions would adopt it! Since there are only three operators, it would be an interesting supplementary result to see per-operator breakdowns.
  3. Showing that the benefits hold for two policy learning algorithms, one BC and one offline RL, strengthens the conclusions.
Critical weaknesses
  1. The baselines are somewhat weak relative to prior work; only one type of perturbation is ablated, and it is a fixed, uniform noise scale. How was the noise scale for this baseline selected? Was it tuned? Perhaps a better baseline would be to fit the noise covariance to the learner's error (this is what DART does). I also think some comparisons with DAgger-style approaches are warranted, since several variants have been proposed to avoid the "intervention discontinuities" issue (*, **, as some examples).
  2. It is somewhat unclear what the improvement mechanism actually is - is it eliciting recovery behavior? Or is it assisting the operator? The paper's own text is somewhat unclear; the main claim is that the residual policy perturbs the teleoperator's commands to elicit recovery behavior the mirrors downstream policy errors, but it also says at points that the residual can "assist the human demonstrator". I think this distinction is especially important to pin down, since the residual is trained by maximizing task reward, which makes me think its role would be more similar to an assistive co-pilot. I think it is plausible that this "eliciting recovery behavior" mechanism could emerge, but that needs to be demonstrated more decisively.
  3. Related to point 2, the paper treats human demonstrators as a noise-less source of ground truth, but I think that is not a good approximation, particularly as tasks become very precise. In fact, there have been several works (*** as a recent example) that take the exact opposite approach as this paper - making human demonstrations less error-filled in order to promote task success. Therefore, measuring the deviation from the human teleoperator command and the policy perturbation is not necessarily measuring a task-irrelevant noise injection; the policy could be nudging the teleoperator in the direction necessary to solve the task. I think the authors should come up with some experiments to disambiguate these potential attributions.

*Xu, Xiaomeng, et al. "Compliant residual dagger: Improving real-world contact-rich manipulation with human corrections." Advances in Neural Information Processing Systems 38 (2026): 139559-139581. **Fang, Junjie, et al. "Robopocket: Improve robot policies instantly with your phone." arXiv preprint arXiv:2603.05504 (2026). ***Song, Yuchen, Aditya Mittal, and Unnat Jain. "Learning Beyond What Humans Can Demonstrate." arXiv preprint arXiv:2609.24996 (2026).

Questions for authors
  1. How were the noise scales for the random-noise baseline chosen?
  2. What evidence shows the residual is injecting perturbations that induce recovery behavior rather than providing boosts that guide the operator towards completing the task?
  3. Are the downstream BC/RL policies trained on a or ~a?
  4. Can the operators perceive the deviations? Did they judge them as helping or hurting? Does the noise injection change the cognitive load of teleoperation?
  5. I like the idea of measuring the strength of corrective demonstrations, but why are the aligned changes clipped to zero?

What would most change my assessment: the empirical gains are convincing, and I think that evidence disambiguating whether the residual elicits recovery or assists the operator (I think either mechanism is interesting) would make me most open to changing my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.