- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on practical matters: When a robot makes a mistake and is corrected, it not only repeats the person's actions, but also compares "the corrected actions by the person" with "the original actions intended by the robot", learning which action is better and further adjusting its action generation process. The overall thinking of the article is smooth, and the methods are also relatively complete. The author conducted experiments on six real robot tasks, and the success rate and the number of manual interventions have significantly improved, indicating that the method does have certain effects. However, I think the innovation at the method level and the setting of some experiments are not clear and rigorous enough.
Critical strengths
1.This paper proposes a more comprehensive approach than simply imitating the corrective actions, which involves simultaneously utilizing the results of the corrective actions and the relative preferences of the rejected actions.
2.The experimental results are quite comprehensive, not only providing the final success rate, but also examining the critic ranking, gradient validity, reference strategy refreshing, and computational cost.
Critical weaknesses
The innovation points of this paper are insufficient, and it is somewhat similar to the methods used in previous papers.
Questions for authors
1.The paper directly regards human intervention as a reliable preference label, but it fails to observe the actual results of the rejected actions, thus unable to confirm that the corrective actions do indeed have higher returns. This design primarily learns about human intervention preferences rather than the actual ranking of action values. Especially when human operations are conservative, there are delays, or there are judgment errors, this supervision may introduce systematic biases.
2.I still have doubts about the novelty of the method. CVL is highly correlated with existing local preference value learning based on human intervention, such as VLA-Precision [1], PACT [2], and Inverse Preference Learning [3]. For APO, previous QAM [4] and Q-VGM [5] have already solved almost the same core problem: the flow/diffusion strategy generates actions through multiple steps, and directly backpropagating the Q gradient is unstable. Therefore, the current paper is more like an effective integration of existing methods in real VLA online learning rather than two independent new algorithms. The authors need to explain more clearly the technical contributions that cannot be directly obtained from existing methods.
3.The results of the paper indicate the errors, but the performance of these errors is not yet clear: it is not known how many independent training sessions were conducted in total; whether the "±" represents standard deviation, standard error or confidence interval; whether the calculation was done between different random seeds or among the six tasks; and whether data was re-collected and the model re-trained for each experiment.
4.The current tasks of the experiment are mostly single-objective. We don't know yet how long it takes to complete each task. Additionally, could it be expanded to multi-objective long-term sequence tasks?
[1] VLA-Precision: Asymmetric Co-Bootstrapping for Efficient Real-World Online RL of Vision-Language-Action Models
[2] Preference-Calibrated Human-in-the-Loop Reinforcement Learning for Robotic Manipulation
[3]Inverse Preference Learning: Preference-based RL without a Reward Function
[4]Q-learning with Adjoint Matching
[5] Q-VGM: Q-Guided Value-Gradient Matching for Offline-to-Online RL of Flow-Matching VLA Policies
8 comments
Response to Reviewer #AxeN (2/4)
Response to Reviewer #AxeN (2/4)
(5) Relation to the cited works, and why CAPO's design is advantageous (L120–133 and L1644–1647). VLA-Precision, PACT, and Q-VGM are concurrent works available only on arXiv. Under the ICLR 2027 Reviewer Guidelines, a comparison with arXiv-only papers is not required, but we discuss them in Section 2 and explain below why the design choices of CAPO are advantageous.
- VLA-Precision also ranks corrective actions above overwritten ones, but turns the calibrated values into relative action advantages for policy improvement. A scalar advantage can only reweight actions that the policy already generates. CAPO instead uses the critic's action gradient, which points toward better actions that may never be sampled, and Proposition 3 shows that the ranking loss makes this gradient point toward the human correction. This matters for the fine-grained corrections required in insertion and alignment. Under the same CVL critic, advantage-weighted updates (AWM) reach 61.7%, whereas APO reaches 93.3% (Figure 6b).
- PACT pairs human actions with resampled policy actions and penalizes Bellman targets of progress-identified segments. A resampled action may differ from the action that actually failed, and penalized targets propagate errors of the progress model to other states through bootstrapping. CVL compares a correction only with the proposal that actually triggered the intervention, so each pair is grounded in an observed failure, and it leaves TD targets unbiased (Lemma 1). The bounded hinge limits the effect of a wrong pair to a local margin with a compatible fixed point (Proposition 2), and CAPO still outperforms TD+APO with 20% flipped labels (85.0% versus 81.7%, Figure 8b).
- IPL learns values from offline preferences without rewards, so it discards the task reward and cannot correct its values with new on-policy experience. In real-world HiL-RL, success rewards are available and informative. Compared with TD-only critic learning, CVL reduces normalized Monte Carlo MAE from 0.190 to 0.140 (Figure 5b). Adding corrective ranking also improves policy success both with and without corrective BC (Figure 5c).
- Q-VGM approximates velocity corrections locally at late denoising steps and uses a critic unaware of interventions. A local approximation ignores how a velocity change propagates through the remaining flow to the final action. APO transports the gradient through the reference-field adjoint, so its targets are exact within the matched interval and stay close to the exact feedback gradient (cosine similarity 0.89–0.94, Table 10). Our cutoff sweep also shows that supervising only the latest steps loses useful signal, since success drops from 93.3% at to 80.0% at (Figure 7a).
- QAM matches a stochastic memoryless process that differs from the deterministic sampler used at deployment, and regularizes toward a behavior policy learned from replay that mixes demonstrations, corrections, and earlier policies. APO optimizes the trajectories the VLA actually executes and regularizes toward an evolving snapshot of the current policy. Under the same critic and BC, APO reaches 93.3% versus 83.3% for QAM at the same cutoff (Figure 6b), and a fixed reference limits success to 81.7% versus 93.3% for the phase snapshot (Figure 7b).
(continued in the next comment)
Response to Reviewer #AxeN (3/4)
Response to Reviewer #AxeN (3/4)
Q1: Does CVL learn human preferences rather than actual action values?
CVL does not regress values to human preferences. Its value targets come from TD learning on executed transitions with real rewards, including the outcomes of the corrections themselves, and the bootstrap uses fresh policy actions, so values of human actions do not back up to earlier states (Appendix A.2, L826–833). The ranking term only constrains relative values at intervention states and vanishes once the margin is met. When Eq. (14) holds, both objectives share a fixed point at the true value (Proposition 2).
Regarding conservative, delayed, or mistaken operators, we follow the intervention protocol of representative HiL-RL works [1–4]: interventions occur mainly when the policy is about to enter an unrecoverable or unsafe state or stalls without progress (Appendix A.2, L819–824 and Appendix C.1, L1128–1135). In these states the overwritten proposal is clearly failing and indeed needs correction. We also observe that ranking helps the policy learn corrective behavior faster, reducing the intervention ratio from 21.9% to 14.9% without corrective BC and from 12.9% to 6.9% with it (Figure 5c, L419–420).
To verify the assumption directly, we added a return-based validation (Appendix D.3, L1378–1409). We reset the scene to 30 held-out intervention states per task and executed the correction, the rejected proposal, or a chunk from the final CAPO policy, 3 times each, followed by the same frozen VLA without further assistance.
Table 7 (L1391). Return-based validation of the corrective preference assumption.
| Task | Pairs | Succ. | Succ. | Succ. | Better/Tie/Worse | Agr. TD | Agr. CVL | |
|---|---|---|---|---|---|---|---|---|
| USB Insertion | 30 | 81.1 | 14.4 | 85.6 | 83.3/6.7/10.0 | 80.0 | 75.0 | 89.3 |
| Screw Tightening | 30 | 74.4 | 18.9 | 77.8 | 76.7/10.0/13.3 | 73.3 | 70.4 | 85.2 |
| Block Stacking | 30 | 68.9 | 21.1 | 73.3 | 73.3/13.3/13.3 | 70.0 | 69.2 | 80.8 |
| Overall | 90 | 74.8 | 18.1 | 78.9 | 77.8/10.0/12.2 | 74.4 | 71.6 | 85.2 |
The correction yields a higher return than the overwritten proposal in 77.8% of the pairs and a lower return in 12.2%. The CVL critic agrees with the sign of the real return gap more often than the TD-only critic (85.2% versus 71.6%), so ranking supervision moves the critic toward the true return ordering. The final policy succeeds from these states in 78.9% of the attempts, compared with 74.8% for the human correction, so CAPO is not capped at the operator's level.
Together with the label-flipping study (Figure 8b), these results indicate that CVL learns actual action values and tolerates realistic violations (L1538–1542).
[1] Kelly et al. HG-DAgger: Interactive imitation learning with human experts. ICRA, 2019.
[2] Mandlekar et al. Human-in-the-loop imitation learning using remote teleoperation. arXiv:2012.06733, 2020.
[3] Luo et al. Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning. Science Robotics, 2025.
[4] Chen et al. ConRFT: A reinforced fine-tuning method for VLA models via consistency policy. RSS, 2025.
(continued in the next comment)
Response to Reviewer #AxeN (1/4)
Response to Reviewer #AxeN (1/4)
We thank the reviewer for the thoughtful comments and for recognizing that CAPO goes beyond imitating corrective actions. Line numbers refer to the revised PDF, where changes for this review are shown in purple and changes shared with other reviewers in blue.
Weakness & Q2: Novelty relative to existing methods
CAPO is not an integration of existing components. Its distinctive feature is that it turns human interventions into action-gradient supervision for the native flow policy of a VLA. Existing methods either use interventions to calibrate values or advantages, or update flow policies from critics that are unaware of interventions. CAPO connects the two through the following components, each supported by controlled experiments (L252–254 and Appendix F, L1557–1647).
(1) Interventions shape the critic's action gradient (CVL). Proposition 3 proves that the corrective ranking loss tilts the critic's action gradient at a rejected proposal toward the paired correction, which is exactly the terminal signal APO consumes. This provides a principled link from human corrections to the update direction of a generative policy, beyond using corrections as imitation targets or value labels. Empirically, gradient-directed perturbations of the CVL critic improve return over random perturbations by 0.037, compared with 0.008 for TD-only (Figure 5b), and ranking improves success and reduces interventions both with and without corrective BC (Figure 5c).
(2) The gradient drives the native, deployment-preserving VLA policy (APO). APO transports ∇AQ backward along the VLA's own deterministic ODE through the reference-field Jacobians, so every intermediate velocity target reflects how that step affects the final chunk. Training uses the same trajectories as the deployment sampler, and deployment needs no auxiliary actor or critic query. With full matching at a freshly copied reference, the matching gradient is proportional to the exact gradient of the regularized objective (Appendix B.4), and the lean adjoint keeps a cosine similarity of 0.89–0.94 with the exact feedback gradient (Table 10). Under the same CVL critic, APO reaches 93.3%, compared with 83.3% for stochastic adjoint matching (QAM) at the same cutoff and 61.7% for scalar weighting (AWM) (Figure 6b), and the auxiliary-actor method RLToken reaches 71.7% (Figure 4).
(3) An evolving reference for online real-world adaptation. Instead of a behavior reference learned from replay, APO regularizes toward a snapshot of the current VLA held fixed within each actor phase of 500 actor updates. The phase snapshot reaches 93.3%, compared with 81.7% for a fixed initial reference, 88.3% for an EMA, and 86.7% for a per-step snapshot (Figure 7b, Table 11).
(4) Practicality on a billion-parameter VLA. Terminal-interval matching keeps exact targets inside [τc,1] while retaining full forward generation, reducing actor-update cost by 32% with a 1.7-point change in success (Figure 7a), and prefix KV caching avoids re-encoding the vision-language context (Appendix B.5).
Table 12 (L1627). Properties of policy-update routes for flow-based policies.
(continued in the next comment)