acceptodds
Under review as a conference paper at ICLR 2027

Making Every Interaction Count: Efficient Real-World RL for Robot Policies

Abstract

Real-world reinforcement learning (RL) is crucial for adapting robotic control policies to physical environments. Yet, it faces several challenges: costly environment interactions and human supervision, sparse rewards in long-horizon tasks, and underutilized suboptimal experience from autonomous policy rollouts. To address this, we introduce **D**etect, **E**valuate, and **C**ompose (DEC), an RL framework that minimizes human intervention while enabling efficient autonomous policy improvement. DEC first trains a dynamics-aware discriminator to detect failures and determine when to trigger human takeover and resume autonomous execution. It then evaluates a robust value function to tackle the sparse reward challenge by adopting trajectory-held-out bootstrapping and discounted partial returns. For policy optimization, DEC learns complementary positive and negative sub-policies to exploit both successful and suboptimal experience, and compose them at inference time to enable controllable action generation. Across simulated manipulation environment and long-horizon, dexterous bimanual tasks in the real world, DEC consistently outperforms existing RL and policy fine-tuning methods with limited human intervention, demonstrating that effective real-world RL can make every interaction count. More results can be seen at https://iclr27-4081.github.io/

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotics, Reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_iiZeOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents DEC, a framework for sample-efficient real-world reinforcement learning with limited human intervention. DEC uses a dynamics-aware discriminator to decide when human takeover is needed, a robust value-learning scheme for long-horizon sparse-reward tasks, and positive/negative policy branches that utilize both successful and suboptimal experience. The method is evaluated on ROBOSUITE tasks and four real-world bimanual manipulation tasks. The results show substantial policy improvement with relatively limited interaction and human supervision, including challenging insertion and long-horizon assembly tasks.

Critical strengths
  1. The paper addresses an important practical problem in real-world robot learning. Reducing both robot interaction and human supervision is highly relevant for real-world RL, and the DEC formulation provides a coherent way to make better use of the collected experience.

  2. The real-world validation is strong. The evaluation includes challenging bimanual tasks such as pen assembly, rather than simple short-horizon simulation tasks. The improvements over the base policy and competing methods are substantial, which makes the practical contribution convincing.

Critical weaknesses

A critical weakness I observe is that discriminator and value models are trained from task-specific experience, and their ability to transfer across tasks or environments is not verified. As a core component, this may limit how easily DEC can be reused when adapting a pretrained policy to many different tasks. Retrain a specific discriminator or value model for each new task may bring higher cost to reducing human intervene time.

Questions for authors
  1. Were the real-world training experiments repeated with multiple independent runs? If not, how sensitive did you observe DEC to different rollout data or human intervention trajectories?
  2. Is there a ablation study that directly compares removing each major component from the full DEC system to see the contribution of each component to the final performance?
  3. The core problem the paper tries to solve is reduce the time of human intervene in RL training. Based on the results show in Fig.7, the time of human intervene reduce about 20%-50% on real world task. However, I assume the fact that during real-world RL training, human need to stay by the side all the time to monitor, reset, and intervene sometimes. In this case, does the reducation of intervene time real matter as it only takes a smaller part of the whole training time. Answer this question will greatly help me understand how this method really contributes to the efficiency of real world RL.
Reviewer_Tb39OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a human-in-the-loop framework named DEC (Detect, Evaluate, and Compose) for real-world RL fine-tuning of robot policies. DEC combines a dynamics-aware failure discriminator for control handoffs, trajectory-held-out bootstrapping and learned partial returns for value estimation, and positive-negative policy composition for exploiting mixed-quality experience.

Critical strengths
  1. Real-world gains on genuinely hard tasks with a large pretrained VLA. DEC significantly improves π0.5 on several real-world tasks. At every reported stage it also uses less intervention time than the human-in-the-loop baselines.
  2. The frame-annotated evaluation over nine tasks (Tables 5–6) shows clearly higher failure recall than OOD-style detectors at the calibrated thresholds.
Critical weaknesses
  1. Eq. 8 adds a positive per-step bonus that is not potential-based. Over most of a trajectory, the critic therefore mainly learns discounted discriminator scores. This may undercut the long-horizon sparse-reward claims. The handoff uses only ten successful trajectories, a fixed 98th percentile threshold and a fixed positive prior. The paper reports no sensitivity analysis for these parameters.
  2. The experimental evidence for the efficiency claims is not rigorous enough. Trial counts, training seeds and error bars are not reported. Real-world results appear to be single runs.
  3. Reported numbers disagree across tables and figures: DEC on PickPlaceCereal is 84.7/86.7/88.7 in Table 2 but 86.7/87.5/89.3 in Fig. 6.
Questions for authors
  1. Can you provide downstream success-rate or learning-curve ablations that separately remove trajectory-held-out bootstrapping, learned partial returns, and discriminator-based rewards?
  2. Evaluation protocol, how many evaluation trials and independent training runs underlie each number. Please reconcile Table 2 with Figs. 6 - 7, and explain the inconsistent trial counts noted above. Which checkpoint does Fig. 8 use? Is it a post-hoc sweep over w or separate training runs?
  3. For each task and baseline, can you specify the initial demonstration, warmup rollouts, calibration trajectories, online interaction budget, and cumulative human intervention time?
  4. What are DAgger's results on PenAssembly and USBInsertion? How were the baseline hyperparameters tuned, and why does RLT fall to 0% on PenAssembly?
  5. Please define the sparse task reward function renv exactly and report λdisc for each task.
  6. How sensitive are success and intervention time to η, δ and the number of calibration trajectories?
  7. Are the detection baselines in Tables 1, 5 and 6 thresholded with the same calibration protocol?
Reviewer_7gE8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Detect, Evaluate, and Compose (DEC) that combines selective human intervention with reinforcement learning (RL) of generative robot policies. A dynamics-pretrained discriminator uses positive-unlabeled learning to decide when a human should take over and when autonomous execution can resume. A critic combines trajectory held out bootstrapping, learned discounted partial returns, and discriminator-derived reward. Positive and negative policies then learn from complementary advantage weights and are combined through classifier-free guidance. The composition rule follows Dichotomous Diffusion Policy Optimization. The contribution of DEC is the integrated detection and value-learning system around it. Experiments cover two robosuite policy-learning tasks and four physical bimanual tasks, with π0.5\pi0.5 used for the physical experiments. Separate detector experiments, representation comparisons, and negative-branch, failed-data, and intervention ablations provide useful supporting evidence.

Critical strengths

The paper’s physical evaluation gives the paper practical usefulness. It tackles complete bimanual tasks from natural initial states, including pose adjustment followed by insertion and multi-component assembly….

Additionally, the treatment of human effort is useful. For the two hardest tasks, DEC reports approximately 24% less active correction time than RLT.

Critical weaknesses

Sec 3.3 clearly credits DIPOLE for the positive and negative decomposition and guidance rule. Building a useful robot-learning system around that construction can be a contribution. The missing comparison is a DIPOLE-style learner using the same pretrained policy and corrective data, with a standard critic and intervention mechanism….. ConRFT already demonstrates online VLA improvement with human corrections [R1], and ARMADA studies selective intervention through failure detection [R2]. These precedents make the added effect of DEC’s detector and critic central to the assessment..

[R1] Chen, Y., et al. ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy. RSS 2025. DOI: 10.15607/RSS.2025.XXI.019

[R2] Yu, W., et al. ARMADA: Autonomous Online Failure Detection and Human Shared Control Empower Scalable Real-world Deployment and Adaptation. 2025 arXiv:2510.02298.

The main curves emphasize 20-60 rollout episodes, but the draft does not give the initialization rollout counts, evaluation denominators, independent training-run counts, or the meaning of the error bars. Algo. 1 reserves successful trajectories for calibration, and Tab. 3 specifies ten. So, it is unclear how they enter the displayed interaction budget. Appendix D.2 describes a separate detector dataset of 500 simulated and 400 real trajectories. The paper should state whether any of those data or fitted representations also initialize the policy-earning experiments. I would not assume they do. The dense failure labels in that appendix are explicitly for evaluation.

Appendix C.4 states that DSRL shares the initial rollout episodes, which is a useful matching condition, but the missing counts still matter. The non-pretrained HIL-SERL implementation is openly described and serves as context; its poor result cannot isolate DEC’s advantage over another pretrained learner. The reported gains are promising, but complete initialization, interaction, and active-intervention budgets, together with independent autonomous evaluation, are needed to establish sample and supervision efficiency. Appendix A acknowledges the computational overhead…. end-to-end timing would help assess the wider practical tradeoff.

The successful and failed trajectory plots are helpful illustrations, yet smoothness does not establish accurate continuation values or correct action ranking. Some correctly estimated values should decrease when additional work becomes necessary.

I would ask for held-out continuation-return measurements, advantage ranking accuracy and a computation-matched conventional five-critic ensemble. The learned partial-return model and compositional loss should be removed separately.

The trajectory exclusion also needs a precise interpretation. A critic does not directly train on its held-out split, but other critics exchange bootstrap targets, and shared models use data across splits. This may still reduce overfitting…. it does not establish complete statistical independence from the excluded trajectory.

Questions for authors

With the same base policy and corrective data, how does a DIPOLE-style learner with a standard critic compare?.. What changes in autonomous success and intervention time when the detector, trajectory holdout, learned partial returns, or discriminator reward are removed individually?

How many initialization, calibration, and online trajectories are used per task, and which are included in the plotted budgets? Does the separate detector benchmark contribute data or fitted models to these runs? Please report evaluation episode counts, independent training seeds, error-bar definitions, and whether all test successes are measured without human assistance.

How are discriminator threshold and guidance weight selected without using final evaluation outcomes... At a matched false-alarm or intervention budget, do the detector improvements reduce human effort?

1 comment

Please take a moment to respond to the rebuttal. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.