- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Efficient-RLT, a real-robot online RL post-training framework built on RL Token. Its central idea is to physically rewind the robot to a satisfactory pre-error state before human correction, so that the rejected policy action and the corrective action form a state-aligned preference pair. These pairs are used to train the critic and actor, while branch-consistent credit assignment prevents the corrective return from propagating through the rejected branch. A learned progress estimator provides additional dense rewards. The main contribution is the use of rewind interventions as action-preference supervision for residual actor-critic learning, rather than treating human corrections as ordinary replay data. The authors evaluate the method on a range of real-robot platforms and manipulation tasks, with component ablations and a cross-backbone check.
Critical strengths
-
S1. Rewind provides well-aligned supervision for intervention RL. Returning to the same pre-error state pairs the rejected and corrective actions under comparable conditions, making the preference signal meaningful. Branch-consistent credit assignment further prevents the corrective return from being attributed to the rejected branch.
-
S2. The real-robot evaluation is a notable strength. Testing across multiple platforms and checking one additional frozen base provide preliminary evidence that the approach is not tied to a single hardware setup or policy architecture.
Critical weaknesses
-
W1. The sample-efficiency claim is not directly supported. The paper lacks experimental data and convergence analysis for its claims regarding sample efficiency, convergence speed, takeover timing, and intervention horizon.
-
W2. Lacking important baselines. The paper does not discuss GCENT, so its contribution remains ambiguous. The absence of this baseline also obscures whether the gains arise from rewind-and-refine data collection or action-preference learning. It also inherits EXPO-FT's multi-candidate design without evaluating EXPO-FT as a baseline.
Questions for authors
- Efficiency claims and takeover analysis. Please provide training curves or interaction-budget comparisons supporting the efficiency and convergence claims, and clarify the claimed takeover-timing and horizon analysis.
- Baseline comparisons and component attribution. Could you compare against GCENT and EXPO-FT under compatible protocols, and add a rewind-only experiment without preference-learning objectives to clarify the contribution of action preference learning?