acceptodds
Under review as a conference paper at ICLR 2027

Efficient-RLT: Sample-efficient Real-robot RL via Optimizing Human-intervention Strategy with Dense Process Rewards

Abstract

Real-world online reinforcement learning (RL) is a promising approach for post-training vision-language-action policies, particularly for precision-sensitive, long-horizon manipulation where supervised fine-tuning alone rarely achieves reliable execution. Existing human-in-the-loop methods intervene only after the policy has deviated, so the critical decision state never receives both a policy action and a corrective one. The critic therefore either extrapolates over unexecuted proposals or propagates the return of a successful recovery back through the very actions that caused the failure. We introduce Efficient RLT, a framework built on RL Token (RLT) that learns through action contrast from *rewind interventions*. When an operator flags an erroneous action chunk, the robot is physically rewound to the pre-error decision state by reverse-replaying its waypoints, and the operator issues a corrective action from there, yielding a state-aligned pair of a rejected action and its corrective replacement. Efficient RLT trains the critic and actor with a *pairwise preference learning* on these pairs, shaping a local gradient field over the action space that flows from rejected toward corrective actions, while branch-consistent credit assignment keeps the corrective return out of the rejected branch. We further train a progress estimator that supplies *dense progress rewards* under sparse task feedback, and study how takeover timing and horizon affect convergence. On Franka, Cobot, and X-Square platforms across diverse and challenging manipulation tasks, Efficient RLT improves sample efficiency, convergence speed, and final success rate.

Primary area:
reinforcement learning
Keywords:
Robotic Manipulation, Online Reinforcement Learning, Human-intervention, Preference Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_sPR6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Efficient-RLT, a real-robot online RL post-training framework built on RL Token. Its central idea is to physically rewind the robot to a satisfactory pre-error state before human correction, so that the rejected policy action and the corrective action form a state-aligned preference pair. These pairs are used to train the critic and actor, while branch-consistent credit assignment prevents the corrective return from propagating through the rejected branch. A learned progress estimator provides additional dense rewards. The main contribution is the use of rewind interventions as action-preference supervision for residual actor-critic learning, rather than treating human corrections as ordinary replay data. The authors evaluate the method on a range of real-robot platforms and manipulation tasks, with component ablations and a cross-backbone check.

Critical strengths
  1. S1. Rewind provides well-aligned supervision for intervention RL. Returning to the same pre-error state pairs the rejected and corrective actions under comparable conditions, making the preference signal meaningful. Branch-consistent credit assignment further prevents the corrective return from being attributed to the rejected branch.

  2. S2. The real-robot evaluation is a notable strength. Testing across multiple platforms and checking one additional frozen base provide preliminary evidence that the approach is not tied to a single hardware setup or policy architecture.

Critical weaknesses
  1. W1. The sample-efficiency claim is not directly supported. The paper lacks experimental data and convergence analysis for its claims regarding sample efficiency, convergence speed, takeover timing, and intervention horizon.

  2. W2. Lacking important baselines. The paper does not discuss GCENT, so its contribution remains ambiguous. The absence of this baseline also obscures whether the gains arise from rewind-and-refine data collection or action-preference learning. It also inherits EXPO-FT's multi-candidate design without evaluating EXPO-FT as a baseline.

Questions for authors
  1. Efficiency claims and takeover analysis. Please provide training curves or interaction-budget comparisons supporting the efficiency and convergence claims, and clarify the claimed takeover-timing and horizon analysis.
  2. Baseline comparisons and component attribution. Could you compare against GCENT and EXPO-FT under compatible protocols, and add a rewind-only experiment without preference-learning objectives to clarify the contribution of action preference learning?
Reviewer_osoMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Efficient-RLT, a framework for online reinforcement learning of a residual policy on top of a frozen VLA model and its RL-token representation. Following an erroneous action, an operator rewinds the robot to a pre-error state and provides a corrective continuation. The resulting rejected and corrective actions supply preference supervision, while branch-consistent replay prevents successful recovery returns from propagating through the rejected branch. A progress head supplies dense rewards from predicted progress differences. Evaluation covers 12 manipulation tasks on four platforms using pi0.5, with an additional humanoid experiment using a different frozen base. The main comparison reports mean success rates of 71.1% for Efficient-RLT and 63.9% for RL Token.

Critical strengths
  1. The main contribution is a useful perspective on human intervention: its learning value depends on how a correction is acquired and related to the rejected action. Rewinding to a common decision state turns a corrective demonstration into an action comparison, supplying information about what to prefer and what to avoid. This links the intervention protocol directly to critic learning and provides a concrete direction for improving human-assisted robot RL through more informative supervision.

  2. The treatment of preference supervision and replay gives the intervention idea a concrete algorithmic form. Separate bootstrap paths distinguish the rejected branch from its successful correction, while an action-distance-scaled margin avoids requiring the same value gap for both tiny and large corrections. A progress head on the existing RL-token representation adds a complementary task-level signal. Together, these choices offer a useful design for combining local corrective guidance with longer-horizon learning, and the reported multi-platform robot results illustrate its practical relevance.

Critical weaknesses
  1. The distinctive contribution depends on obtaining a meaningful common-state action comparison, but the method does not fully specify how that correspondence is established. Returning the robot along its previous waypoints need not restore object configuration, contact conditions, or irreversible task changes. A correction from the restored configuration may consequently address a different decision problem from the rejected action. The authors acknowledge this limitation, but an operational rule for accepting, rejecting, or weakening such pairs is needed to turn the intervention idea into a reusable learning method. This is a central methodological boundary because common-state preference supervision is what distinguishes the approach from ordinary corrective demonstrations.

  2. The relationship between local corrective preferences and the dense progress signal needs clearer explanation. The progress head uses normalized time in successful trajectories, which is an approximation to physical task progress and may depend on execution speed or pauses. A useful correction may also temporarily move away from the state that the progress head associates with later completion. The framework would be easier to generalize with an explicit account of how these signals are balanced when recovery and predicted progress disagree. The main innovation remains the intervention-and-preference design; the progress head is a supporting component whose assumptions should be stated accordingly.

Questions for authors
  1. What degree of physical state correspondence is needed for the preference pair to retain its intended meaning? Please clarify which state variables must be restored and how the method should handle corrections when only approximate restoration is possible.

  2. How are preference supervision and dense progress rewards reconciled when an operator's corrective action temporarily reduces predicted progress? Please explain the intended role of normalized-time supervision in recovery states.

  3. Which aspects of the method depend specifically on physical rewind, and which principles, such as branch-consistent credit assignment, remain applicable to conventional human takeovers? Clarifying this distinction would make the contribution more useful to the broader human-assisted RL community.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.