acceptodds
Under review as a conference paper at ICLR 2027

CAPO: Real-World Reinforcement Learning for VLA via Corrective Adjoint Policy Optimization

Abstract

Human-in-the-Loop Reinforcement Learning (HiL-RL) offers a practical route for adapting Vision-Language-Action (VLA) models through real-world interaction guided by human interventions. Yet effectively leveraging these interventions remains challenging, their corrective signals must first support informative value learning and then be translated into effective updates of the VLA's multi-step flow policy. To address these challenges, we propose **C**orrective **A**djoint **P**olicy **O**ptimization (**CAPO**), a real-world RL framework that couples Corrective Value Learning (CVL) with Adjoint-based Policy Optimization (APO). Specifically, CVL exploits human corrections both to learn action values from their observed outcomes and to provide comparative supervision against rejected policy proposals. Guided by the learned critic, APO propagates action-value gradients along the VLA's native flow trajectories and converts them into local policy updates, while preserving the original deployment sampler. Across six real-world manipulation tasks, CAPO achieves the highest mean autonomous success of 93.3% and the lowest intervention rate among the evaluated methods.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Reinforcement Learning, Vision Language Action Model, Human-in-the-Loop, Adjoint Matching

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_AxeNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on practical matters: When a robot makes a mistake and is corrected, it not only repeats the person's actions, but also compares "the corrected actions by the person" with "the original actions intended by the robot", learning which action is better and further adjusting its action generation process. The overall thinking of the article is smooth, and the methods are also relatively complete. The author conducted experiments on six real robot tasks, and the success rate and the number of manual interventions have significantly improved, indicating that the method does have certain effects. However, I think the innovation at the method level and the setting of some experiments are not clear and rigorous enough.

Critical strengths

1.This paper proposes a more comprehensive approach than simply imitating the corrective actions, which involves simultaneously utilizing the results of the corrective actions and the relative preferences of the rejected actions.

2.The experimental results are quite comprehensive, not only providing the final success rate, but also examining the critic ranking, gradient validity, reference strategy refreshing, and computational cost.

Critical weaknesses

The innovation points of this paper are insufficient, and it is somewhat similar to the methods used in previous papers.

Questions for authors

1.The paper directly regards human intervention as a reliable preference label, but it fails to observe the actual results of the rejected actions, thus unable to confirm that the corrective actions do indeed have higher returns. This design primarily learns about human intervention preferences rather than the actual ranking of action values. Especially when human operations are conservative, there are delays, or there are judgment errors, this supervision may introduce systematic biases.

2.I still have doubts about the novelty of the method. CVL is highly correlated with existing local preference value learning based on human intervention, such as VLA-Precision [1], PACT [2], and Inverse Preference Learning [3]. For APO, previous QAM [4] and Q-VGM [5] have already solved almost the same core problem: the flow/diffusion strategy generates actions through multiple steps, and directly backpropagating the Q gradient is unstable. Therefore, the current paper is more like an effective integration of existing methods in real VLA online learning rather than two independent new algorithms. The authors need to explain more clearly the technical contributions that cannot be directly obtained from existing methods.

3.The results of the paper indicate the errors, but the performance of these errors is not yet clear: it is not known how many independent training sessions were conducted in total; whether the "±" represents standard deviation, standard error or confidence interval; whether the calculation was done between different random seeds or among the six tasks; and whether data was re-collected and the model re-trained for each experiment.

4.The current tasks of the experiment are mostly single-objective. We don't know yet how long it takes to complete each task. Additionally, could it be expanded to multi-objective long-term sequence tasks?

[1] VLA-Precision: Asymmetric Co-Bootstrapping for Efficient Real-World Online RL of Vision-Language-Action Models

[2] Preference-Calibrated Human-in-the-Loop Reinforcement Learning for Robotic Manipulation

[3]Inverse Preference Learning: Preference-based RL without a Reward Function

[4]Q-learning with Adjoint Matching

[5] Q-VGM: Q-Guided Value-Gradient Matching for Offline-to-Online RL of Flow-Matching VLA Policies

8 comments
AuthorsOpenReview ↗

Response to Reviewer #AxeN (1/4)

Response to Reviewer #AxeN (1/4)

We thank the reviewer for the thoughtful comments and for recognizing that CAPO goes beyond imitating corrective actions. Line numbers refer to the revised PDF, where changes for this review are shown in purple and changes shared with other reviewers in blue.

Weakness & Q2: Novelty relative to existing methods

CAPO is not an integration of existing components. Its distinctive feature is that it turns human interventions into action-gradient supervision for the native flow policy of a VLA. Existing methods either use interventions to calibrate values or advantages, or update flow policies from critics that are unaware of interventions. CAPO connects the two through the following components, each supported by controlled experiments (L252–254 and Appendix F, L1557–1647).

(1) Interventions shape the critic's action gradient (CVL). Proposition 3 proves that the corrective ranking loss tilts the critic's action gradient at a rejected proposal toward the paired correction, which is exactly the terminal signal APO consumes. This provides a principled link from human corrections to the update direction of a generative policy, beyond using corrections as imitation targets or value labels. Empirically, gradient-directed perturbations of the CVL critic improve return over random perturbations by 0.037, compared with 0.008 for TD-only (Figure 5b), and ranking improves success and reduces interventions both with and without corrective BC (Figure 5c).

(2) The gradient drives the native, deployment-preserving VLA policy (APO). APO transports ∇AQ\nabla_A Q backward along the VLA's own deterministic ODE through the reference-field Jacobians, so every intermediate velocity target reflects how that step affects the final chunk. Training uses the same trajectories as the deployment sampler, and deployment needs no auxiliary actor or critic query. With full matching at a freshly copied reference, the matching gradient is proportional to the exact gradient of the regularized objective (Appendix B.4), and the lean adjoint keeps a cosine similarity of 0.89–0.94 with the exact feedback gradient (Table 10). Under the same CVL critic, APO reaches 93.3%, compared with 83.3% for stochastic adjoint matching (QAM) at the same cutoff and 61.7% for scalar weighting (AWM) (Figure 6b), and the auxiliary-actor method RLToken reaches 71.7% (Figure 4).

(3) An evolving reference for online real-world adaptation. Instead of a behavior reference learned from replay, APO regularizes toward a snapshot of the current VLA held fixed within each actor phase of 500 actor updates. The phase snapshot reaches 93.3%, compared with 81.7% for a fixed initial reference, 88.3% for an EMA, and 86.7% for a per-step snapshot (Figure 7b, Table 11).

(4) Practicality on a billion-parameter VLA. Terminal-interval matching keeps exact targets inside [τc,1][\tau_c,1] while retaining full forward generation, reducing actor-update cost by 32% with a 1.7-point change in success (Figure 7a), and prefix KV caching avoids re-encoding the vision-language context (Appendix B.5).

Table 12 (L1627). Properties of policy-update routes for flow-based policies.

Route (examples)Uses ∇AQ\nabla_A QUpdates native VLANative deterministic samplerNo sampler backpropAdjoint transportIntervention-shaped gradient
Scalar weighting (AWM, RECAP, ALOE)✗✓–✓✗✗
Auxiliary actor (RLToken, FQL)✓✗–✓✗✗
Full-sampler backpropagation✓✓✓✗✗✗
Stochastic adjoint matching (QAM)✓✓✗✓✓✗
Local late-step matching (Q-VGM)✓✓✓✓✗✗
CAPO✓✓✓✓✓✓

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer #AxeN (2/4)

Response to Reviewer #AxeN (2/4)

(5) Relation to the cited works, and why CAPO's design is advantageous (L120–133 and L1644–1647). VLA-Precision, PACT, and Q-VGM are concurrent works available only on arXiv. Under the ICLR 2027 Reviewer Guidelines, a comparison with arXiv-only papers is not required, but we discuss them in Section 2 and explain below why the design choices of CAPO are advantageous.

  • VLA-Precision also ranks corrective actions above overwritten ones, but turns the calibrated values into relative action advantages for policy improvement. A scalar advantage can only reweight actions that the policy already generates. CAPO instead uses the critic's action gradient, which points toward better actions that may never be sampled, and Proposition 3 shows that the ranking loss makes this gradient point toward the human correction. This matters for the fine-grained corrections required in insertion and alignment. Under the same CVL critic, advantage-weighted updates (AWM) reach 61.7%, whereas APO reaches 93.3% (Figure 6b).
  • PACT pairs human actions with resampled policy actions and penalizes Bellman targets of progress-identified segments. A resampled action may differ from the action that actually failed, and penalized targets propagate errors of the progress model to other states through bootstrapping. CVL compares a correction only with the proposal that actually triggered the intervention, so each pair is grounded in an observed failure, and it leaves TD targets unbiased (Lemma 1). The bounded hinge limits the effect of a wrong pair to a local margin with a compatible fixed point (Proposition 2), and CAPO still outperforms TD+APO with 20% flipped labels (85.0% versus 81.7%, Figure 8b).
  • IPL learns values from offline preferences without rewards, so it discards the task reward and cannot correct its values with new on-policy experience. In real-world HiL-RL, success rewards are available and informative. Compared with TD-only critic learning, CVL reduces normalized Monte Carlo MAE from 0.190 to 0.140 (Figure 5b). Adding corrective ranking also improves policy success both with and without corrective BC (Figure 5c).
  • Q-VGM approximates velocity corrections locally at late denoising steps and uses a critic unaware of interventions. A local approximation ignores how a velocity change propagates through the remaining flow to the final action. APO transports the gradient through the reference-field adjoint, so its targets are exact within the matched interval and stay close to the exact feedback gradient (cosine similarity 0.89–0.94, Table 10). Our cutoff sweep also shows that supervising only the latest steps loses useful signal, since success drops from 93.3% at τc=0.5\tau_c=0.5 to 80.0% at τc=0.9\tau_c=0.9 (Figure 7a).
  • QAM matches a stochastic memoryless process that differs from the deterministic sampler used at deployment, and regularizes toward a behavior policy learned from replay that mixes demonstrations, corrections, and earlier policies. APO optimizes the trajectories the VLA actually executes and regularizes toward an evolving snapshot of the current policy. Under the same critic and BC, APO reaches 93.3% versus 83.3% for QAM at the same cutoff (Figure 6b), and a fixed reference limits success to 81.7% versus 93.3% for the phase snapshot (Figure 7b).

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer #AxeN (3/4)

Response to Reviewer #AxeN (3/4)

Q1: Does CVL learn human preferences rather than actual action values?

CVL does not regress values to human preferences. Its value targets come from TD learning on executed transitions with real rewards, including the outcomes of the corrections themselves, and the bootstrap uses fresh policy actions, so values of human actions do not back up to earlier states (Appendix A.2, L826–833). The ranking term only constrains relative values at intervention states and vanishes once the margin is met. When Eq. (14) holds, both objectives share a fixed point at the true value QπQ^\pi (Proposition 2).

Regarding conservative, delayed, or mistaken operators, we follow the intervention protocol of representative HiL-RL works [1–4]: interventions occur mainly when the policy is about to enter an unrecoverable or unsafe state or stalls without progress (Appendix A.2, L819–824 and Appendix C.1, L1128–1135). In these states the overwritten proposal is clearly failing and indeed needs correction. We also observe that ranking helps the policy learn corrective behavior faster, reducing the intervention ratio from 21.9% to 14.9% without corrective BC and from 12.9% to 6.9% with it (Figure 5c, L419–420).

To verify the assumption directly, we added a return-based validation (Appendix D.3, L1378–1409). We reset the scene to 30 held-out intervention states per task and executed the correction, the rejected proposal, or a chunk from the final CAPO policy, 3 times each, followed by the same frozen VLA without further assistance.

Table 7 (L1391). Return-based validation of the corrective preference assumption.

TaskPairsSucc. AHA^HSucc. AπA^\piSucc. AfinalA^{\rm final}Better/Tie/WorseΔG≥m\Delta G\ge mAgr. TDAgr. CVL
USB Insertion3081.114.485.683.3/6.7/10.080.075.089.3
Screw Tightening3074.418.977.876.7/10.0/13.373.370.485.2
Block Stacking3068.921.173.373.3/13.3/13.370.069.280.8
Overall9074.818.178.977.8/10.0/12.274.471.685.2

The correction yields a higher return than the overwritten proposal in 77.8% of the pairs and a lower return in 12.2%. The CVL critic agrees with the sign of the real return gap more often than the TD-only critic (85.2% versus 71.6%), so ranking supervision moves the critic toward the true return ordering. The final policy succeeds from these states in 78.9% of the attempts, compared with 74.8% for the human correction, so CAPO is not capped at the operator's level.

Together with the label-flipping study (Figure 8b), these results indicate that CVL learns actual action values and tolerates realistic violations (L1538–1542).

[1] Kelly et al. HG-DAgger: Interactive imitation learning with human experts. ICRA, 2019.
[2] Mandlekar et al. Human-in-the-loop imitation learning using remote teleoperation. arXiv:2012.06733, 2020.
[3] Luo et al. Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning. Science Robotics, 2025.
[4] Chen et al. ConRFT: A reinforced fine-tuning method for VLA models via consistency policy. RSS, 2025.

(continued in the next comment)

Reviewer_vDAaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CAPO adapts a flow-based vision-language-action model using human-assisted real-world reinforcement learning. Corrective Value Learning combines TD learning on genuinely executed action chunks with within-state margin ranking between accepted human corrections and overwritten policy proposals. Adjoint-based Policy Optimization converts terminal critic action gradients into detached velocity targets along the native flow trajectory, using a reference snapshot held fixed within each actor phase. Matching only the terminal part of the trajectory reduces actor-update computation while retaining full forward generation and the original deployment sampler. Experiments on six manipulation tasks report 93.3% mean autonomous success and 6.9% final-window intervention. The paper also reports corrective-BC/ranking ablations, critic-gradient diagnostics, actor compute comparisons, reference/cutoff studies, and robustness experiments.

Critical strengths

The paper tests whether corrective preferences add useful information beyond imitation. Figure 5(c) and Appendix C.3 cross ranking with corrective BC while keeping demonstration BC and APO active. Ranking improves reported success both without corrective BC (73.3% to 82.8%) and with it (81.7% to 93.3%). The separate Monte Carlo value-error and matched-norm action-gradient diagnostics (Figure 5(b); Appendix C.3) examine properties directly relevant to the actor. These are consequential controls for the claim that interventions help through value learning rather than only through copying accepted actions.

The actor design offers a concrete computation/performance trade-off without changing deployment. Figure 6 includes full and half-interval QAM controls and a separate fixed-replay, equal-compute comparison, rather than attributing all savings uniquely to deterministic matching. Half-interval APO reports 0.68 relative actor-update cost versus 1.00 for full APO, with success changing from 95.0% to 93.3%. Appendix A.4 and B.3 acknowledge that lean-adjoint matching is a surrogate away from self-consistency and give the discrete target indexing. These qualifications make the proposed mechanism easier to assess.

Critical weaknesses

The actor comparison may be confounded by unequal corrective imitation, and the manuscript does not resolve the discrepancy. Section 5.1 describes CVL+AWM and CVL+QAM as using "demo-only BC." CAPO's Eq. (11) and Appendix B.1 instead include demonstrations and accepted online corrections in the BC buffer. Figure 6's caption says all actors share CVL and BC, without resolving whether the BC examples and mixtures are identical. If the baselines actually exclude corrections, this does not isolate APO: Figure 5(c) shows a 10.5-point benefit from enabling corrective BC with ranking active, which is larger than the headline 8.3-point gap over CVL+QAM. This is not proof that BC explains the gap, but it makes a matched-supervision comparison essential. If all actors already received identical corrective BC, the issue is a consequential reporting inconsistency that should be corrected explicitly.

The reported performance and efficiency advantage lacks enough protocol detail to judge its reliability. Section 5.1 bases learning-curve success on the ten most recent completed attempts per task/run, while Appendix C.2 defines autonomy by absence of within-attempt correction. The number of independent runs, aggregation underlying the reported means, meaning of the error bars, and whether final results use separate frozen-policy trials are not specified. This matters for an 8.3-point six-task mean improvement and for a metric potentially affected by intervention decisions. Moreover, Appendix B.1 says Table 1 specifies BC mixture, critic/APO update ratios, and reference-refresh frequency, but Table 1 does not provide those entries; its 40/40/20 replay mixture is unlabeled. Actual execution horizon, reward construction, and critic architecture are also unspecified. These details affect both the learned values and the claimed update cost, so the current evidence cannot establish how robustly the advantage would reproduce.

Questions for authors

Did every actor in Figure 6 receive exactly the same demonstration and accepted-correction BC examples, sampling proportions, and BC weight? Please reconcile Section 5.1's "demo-only BC" with Eq. (11), Appendix B.1, and the figure caption. If supervision differed, what is the result with it matched?

How many independent training runs and final autonomous trials support each main result, what do the error bars represent, and are final scores from separate frozen-policy evaluations or the rolling collection window? Please provide per-task/run counts and the intervention policy, together with the missing execution horizon, reward, critic architecture, replay-mixture labels, and update/refresh schedule.

Appendix D.2/Table 5 evaluates learned-critic agreement with intervention labels and interprets this as validation of the true autonomous-continuation ordering in Eq. (14). Is there independent return-based evidence for the sign and margin of the correction-versus-proposal gap under the same frozen autonomous continuation? Learned-critic agreement alone does not determine whether the human preference assumption is correct. This concerns support for the present mechanism, rather than a request for unrelated future tasks.

4 comments
AuthorsOpenReview ↗

Response to Reviewer #vDAa (1/4)

Response to Reviewer #vDAa (1/4)

We thank the reviewer for the careful reading and the precise, constructive comments. Line numbers refer to the revised PDF, where changes for this review are shown in green and changes shared with other reviewers in blue. References [1]–[5] are listed at the end of this comment.

W1 & Q1: Is the actor comparison confounded by unequal corrective imitation?

All actors receive the same type and source of BC supervision, and the comparison is fair. In Section 5.1, "demo-only BC" meant BC on the buffer Ddemo\mathcal D_{\rm demo} only. As defined in Figure 2, Section 4.1, Algorithm 1 Step 2, and Appendix B.1, this buffer contains both demonstrations and online human corrections, following representative HiL-RL works [1–5]. Hence corrective BC is enabled for all actors, not only for CAPO. CVL+AWM, CVL+QAM, and CAPO use the same BC objective and BC weight, sample BC batches uniformly from Ddemo\mathcal D_{\rm demo}, imitate the same offline demonstrations, and imitate the corrections collected during their own online training, provided by the same operator under the same intervention protocol [1–4].

Why corrective BC cannot explain the gap. Figure 5(c) is an ablation inside CAPO: removing corrective BC from CAPO lowers its success by 10.5 points. This does not give CAPO an advantage over the baselines, because CVL+AWM and CVL+QAM also imitate their own corrections and benefit from corrective BC in the same way. The 8.3-point gap between CAPO (93.3%) and CVL+QAM (85.0%) is therefore measured with corrective BC enabled for both methods, and it reflects the actor update. Moreover, CVL+QAM needs more interventions than CAPO (20.6% versus 14.5% on Block Stacking and 16.1% versus 8.3% on Screw Tightening), so it receives more corrections to imitate. If more corrective data were the source of the gain, it would favor CVL+QAM, yet CAPO still performs better.

We agree that "demo-only" invites the reading "demonstrations only", and we have revised the text in Figure 2 (L177), Section 4.1 (L189–191), Section 5.1 (L357–362), and Appendix B.1 (L918–926), for example:

CVL+AWM and CVL+QAM share our critic and the same BC objective on Ddemo\mathcal D_{\rm demo}, which contains the shared demonstrations and the corrections collected during each method's own training under the same operator and intervention protocol. (L357–362)

[1] Kelly et al. HG-DAgger: Interactive imitation learning with human experts. ICRA, 2019.
[2] Mandlekar et al. Human-in-the-loop imitation learning using remote teleoperation. arXiv:2012.06733, 2020.
[3] Luo et al. Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning. Science Robotics, 2025.
[4] Chen et al. ConRFT: A reinforced fine-tuning method for VLA models via consistency policy. RSS, 2025.
[5] Yang et al. ALOE: Action-Level Off-Policy Evaluation for Vision-Language-Action Model Post-Training. arXiv:2602.12691, 2026.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer #vDAa (2/4)

Response to Reviewer #vDAa (2/4)

W2 & Q2: Evaluation protocol and implementation details

(1) Runs, error bars, and attempts (L345–348 and Appendix C.3, L1147–1159). In Figure 4, CAPO, CVL+QAM, and CVL+AWM were each trained with 3 independent runs per task, and HIL-SERL, EXPO-FT, and RLToken with 1 run per task. All settings in Figures 5–7 use 3 runs. Each run starts from the same SFT checkpoint, collects its own data, and retrains the model. Each final score uses the 10 most recent completed attempts per task and run, that is, 30 attempts per task for methods with 3 runs. In the original manuscript, Figure 5 reported the standard error over runs, whereas Figures 6–7 averaged per-task standard deviations. In the updated manuscript, all three figures report the standard deviation across the 3 runs of the six-task mean, which is exactly the quantity being compared and captures both seed and data-collection variability.

(2) Why a rolling window, and a new frozen-policy evaluation. During training, object poses are randomized within a 50×50 cm region instead of fixed positions (L1124–1126), so the rolling-window attempts cover diverse initial configurations. An attempt counts as a success only without any within-attempt correction, timeout, or safety abort, so interventions can only lower the reported success. To remove any dependence on intervention decisions, we additionally evaluated the frozen final checkpoints of all methods without any human intervention, from the same 10 fixed initial positions per task.

Table 4 (L1166). Frozen-policy evaluation without human intervention (success %, number of runs in parentheses).

MethodUSBScrewCupBlockPegCardAvg.
SFT20.040.070.030.020.050.038.3
HIL-SERL (1)30.00.060.00.050.050.031.7
EXPO-FT (1)60.040.070.020.060.070.053.3
RLToken (1)70.060.090.060.080.090.075.0
CVL+AWM (3)63.353.383.343.373.370.064.4
CVL+QAM (3)83.383.3100.073.393.390.087.2
CAPO (3)93.393.3100.086.7100.0100.095.6

CAPO achieves the highest frozen-policy success on average (95.6%), followed by CVL+QAM (87.2%), and the ranking of all methods matches the rolling-window results in Figure 4. The six-task means of this evaluation also form the Orig. column of Figure 8a, so the environmental perturbations are compared under the same frozen-policy protocol. (L1177–1181 and L511–512)

HIL-SERL trains its own non-VLA policy from scratch rather than from the SFT checkpoint. The appendix diagnostics (Tables 5 and 11) use frozen policies from 20 fixed initial settings per task, which differs from the rolling window (L1182–1183 and L1334–1335).

(3) Intervention protocol (L353–354 and Appendix C.1, L1128–1135). We follow the protocol of representative HiL-RL works [1–4], listed in Part 1/4:

The operator takes over with the SpaceMouse when the policy is about to enter an unrecoverable or unsafe state, or when it stalls without progress, issues a short and specific correction, and then returns control to the policy. The operator lets the policy explore on its own otherwise and avoids long interventions that complete the task. The same operator and rules are used for all methods.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer #vDAa (3/4)

Response to Reviewer #vDAa (3/4)

(4) Implementation details. Table 1 now contains every entry referenced in Appendix B.1 (L986–999), Appendix B.1 describes the update schedule (L958–962), and the new Appendix B.3 describes the critic and the reward classifier (L969–1016).

Item (added to Table 1)Setting
TD replay mixture (demo buffer / executed buffer / intervention buffer)40 / 40 / 20
BC samplingUniform over Ddemo\mathcal D_{\rm demo}
Generated / executed chunk length50 / 10 primitive steps at 30 Hz
Update schedule200 critic-only warm-up updates, then alternating 1 critic-only update and 1 critic + actor update
Reference refresh500 actor updates
Reward−0.05 per step, +10 at success

The schedule is defined by update counts rather than by a fixed number of collected episodes per round. After the 200 warm-up updates, the critic is updated twice per actor update, and after every 500 actor updates the reference is replaced by a snapshot of the current actor and the counter restarts.

  • Critic. A frozen ImageNet-pretrained ResNet-10 (from HIL-SERL) encodes each 224×224 camera image, a per-camera spatial embedding pools it into 256 dimensions, and proprioception is projected to 64 dimensions. These features, the executed 10×710\times7 action chunk, and its execution mask feed K=2K=2 independent Q heads, each an MLP with two hidden layers of width 256 with LayerNorm and tanh.
  • Reward classifier. A binary classifier on the base and wrist cameras (frozen ResNet-10 and an MLP with width 256) detects success with 97.6% held-out accuracy. Predicted successes above probability 0.9 are confirmed by the operator, and rejected predictions and missed successes are added to the training set as hard examples.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer #vDAa (4/4)

Response to Reviewer #vDAa (4/4)

Q3: Return-based evidence for the corrective preference assumption

We agree with the reviewer, and this is not a misunderstanding of the original manuscript. Table 5 (now Table 6) measures agreement between the learned critic and the intervention labels. Because CVL is trained with such labels, this agreement cannot by itself validate Eq. (14), and our original interpretation overstated it. We have retitled Appendix D.2 as a critic-label agreement analysis and revised its interpretation (L1355–1375). The TD-only critic never observes preference labels, so its 76% agreement (Figure 5a) is label-independent, although still model-based.

To provide the requested independent evidence, we added a return-based validation (Appendix D.3, L1378–1409). On USB Insertion, Screw Tightening, and Block Stacking, we reset the scene to 30 held-out intervention states per task and executed the human correction AHA^H, the rejected proposal AπA^\pi, or a chunk AfinalA^{\rm final} from the final CAPO policy, 3 times each, followed by the same frozen VLA snapshot that was deployed when the intervention occurred, with no further human assistance.

Table 7 (L1391). Return-based validation. Better/Tie/Worse compares G(AH)G(A^H) with G(Aπ)G(A^\pi), and the last two columns give the sign agreement of each critic with the real return gap on non-tied pairs.

TaskPairsSucc. AHA^HSucc. AπA^\piSucc. AfinalA^{\rm final}Better/Tie/WorseΔG≥m\Delta G\ge mAgr. TDAgr. CVL
USB Insertion3081.114.485.683.3/6.7/10.080.075.089.3
Screw Tightening3074.418.977.876.7/10.0/13.373.370.485.2
Block Stacking3068.921.173.373.3/13.3/13.370.069.280.8
Overall9074.818.178.977.8/10.0/12.274.471.685.2

The correction yields a higher return than the overwritten proposal in 77.8% of the pairs and a lower return in 12.2%, and the gap reaches the training margin in 74.4% of the pairs. The CVL critic agrees with the sign of the real return gap more often than the TD-only critic (85.2% versus 71.6%), so ranking supervision moves the critic toward the true return ordering rather than merely reproducing the labels. Because interventions are triggered by clear failures, the overwritten proposals rarely lead to success (18.1%), which explains why the assumption holds for most real pairs.

We thank the reviewer again for these comments, which have substantially improved the rigor and reproducibility of the paper. We hope that the revisions and new experimental results have addressed the reviewer's concerns, and we would sincerely appreciate it if the rating could be updated accordingly. If any concerns remain, please let us know, and we will respond promptly.

Reviewer_7u1EOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CAPO, a real-world reinforcement learning framework for flow-based VLA models. It combines Corrective Value Learning (CVL), which leverages human corrections for value estimation, with Adjoint-based Policy Optimization (APO), which uses value gradients to optimize flow-based policies. Experiments on six real-world manipulation tasks using π0.5 demonstrate improved autonomous success rates and reduced human intervention compared to existing baselines.

Critical strengths

1. Well-motivated problem setting. The paper addresses the practical challenge of effectively leveraging human interventions to improve Flow-based VLA models in real-world robotic deployment. In particular, the use of human corrections for both value learning and policy optimization provides a well-justified motivation for the proposed approach.

2. Efficient policy optimization. APO converts value gradients with respect to final actions into Flow Velocity Supervision, avoiding the need to backpropagate through the entire sampling process. Furthermore, Terminal-interval Matching reduces the computational cost of actor updates by approximately 32% while largely maintaining task success rates, without introducing additional modules during inference.

3. Comprehensive real-world evaluation. The authors conduct experiments on six real-world robotic manipulation tasks and compare their approach against multiple RL methods. Additionally, the paper includes component ablations, computational efficiency analyses, robustness evaluations under environmental variations, and experiments on human preference noise, providing comprehensive validation of the proposed framework.

4. Clear presentation and high-quality figures. The paper is well-organized and clearly written, with visually appealing and informative figures and tables. The overall presentation makes the methodology and experimental results easy to follow and understand.

Critical weaknesses

1. Missing AI Use Statement. The paper appears to omit the mandatory AI Use Statement required by the ICLR 2027 Author Guidelines. The authors should clarify this omission and provide the required disclosure.

2. Strong assumption regarding human corrections. CVL assumes that human corrective actions are superior to the rejected policy actions. However, this assumption may not always hold in practice, particularly when human interventions are noisy, inconsistent, or suboptimal.

3. Limited generalization evaluation. Although the paper demonstrates promising results across six real-world robotic manipulation tasks, all experiments are conducted using a specific π0.5 backbone and within relatively controlled manipulation scenarios. This limits the evidence for the generalizability of the proposed approach to other VLA architectures and more diverse real-world environments.

Questions for authors

1. AI Use Statement. The paper appears to omit the mandatory AI Use Statement required by the ICLR 2027 Author Guidelines. Since this section appears to be a submission requirement, I am concerned that its omission may pose a potential desk-rejection risk.

2. Generalization Beyond the Evaluated Setting. All experiments are conducted using the π0.5 backbone in relatively controlled manipulation environments. What evidence supports the generalizability of APO and CVL to other VLA architectures or more diverse real-world environments?

8 comments
AuthorsOpenReview ↗

Response to Reviewer #7u1E (1/3)

Response to Reviewer #7u1E (1/3)

We thank the reviewer for the positive assessment and the constructive comments. Since W1/Q1 and W3/Q2 raise the same issues, we answer each pair together. Line numbers refer to the revised PDF, where changes for this review are shown in orange and changes shared with other reviewers in blue.

W1 & Q1: Missing AI Use Statement

We thank the reviewer for pointing out this omission and apologize for it. Following the ICLR 2027 AI Policy for Authors, the revised manuscript now includes an AI use statement (L524–532):

In this work, we used generative AI tools mainly to write and debug code, and additionally to assist with literature search and to polish the manuscript for clarity and readability. Generative AI tools were not used to propose research ideas, generate data, or analyze experimental results. All AI-assisted work was reviewed by the authors. In particular, AI-assisted code was manually inspected and tested, and AI-polished text was checked for technical accuracy and consistency with the original sources. We take full responsibility for the final content of this work, including the text, claims, code, and any artifacts produced with the aid of generative AI.

We have also added a Reproducibility Statement (L534–539) that points to the implementation details and to our anonymous code repository.

W2: Assumption regarding human corrections

We agree that human interventions are not always optimal, and we address this concern in four ways.

(1) Interventions occur at clear failures, where the policy action indeed needs correction. We follow the intervention protocol of representative HiL-RL works [1–4] (Appendix C.1, L1128–1135): the operator intervenes mainly when the policy is about to enter an unrecoverable or unsafe state, or when it stalls and cannot make further progress. In these states the overwritten proposal is clearly failing, so preferring the correction over that specific proposal is a mild requirement (L241–242 and Appendix A.2, L819–824).

(2) CVL relies on a local assumption, not on human optimality. Eq. (14) only requires an accepted correction to outperform the proposal it overwrote at the same state, under the same autonomous continuation. It does not assume that the human is optimal or better than the policy elsewhere.

(3) CVL limits the influence of an imperfect correction by design (Appendix A.2, L826–833). An executed correction is regressed to its observed outcome, so a poor correction receives a low value. The TD bootstrap uses fresh policy actions, so values of human actions do not back up to earlier states. The squared hinge vanishes once the margin is met, so a correction acts as a bounded local constraint rather than an imitation target.

(4) CVL helps the policy learn corrective behavior faster (L419–420). In Figure 5(c), ranking reduces the final-window intervention ratio from 21.9% to 14.9% without corrective BC and from 12.9% to 6.9% with it, while raising success in both settings. CAPO also retains 91.7% and 85.0% success with 10% and 20% flipped preference labels, both above TD+APO (81.7%, Figure 8b).

[1] Kelly et al. HG-DAgger: Interactive imitation learning with human experts. ICRA, 2019.
[2] Mandlekar et al. Human-in-the-loop imitation learning using remote teleoperation. arXiv:2012.06733, 2020.
[3] Luo et al. Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning. Science Robotics, 2025.
[4] Chen et al. ConRFT: A reinforced fine-tuning method for VLA models via consistency policy. RSS, 2025.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer #7u1E (2/3)

Response to Reviewer #7u1E (2/3)

New: return-based validation on real interventions (Appendix D.3, L1378–1409). To test the assumption directly and without any learned critic, we reset the scene to 30 held-out intervention states per task on USB Insertion, Screw Tightening, and Block Stacking. From each state, we executed the human correction AHA^H, the rejected proposal AπA^\pi, or a chunk AfinalA^{\rm final} from the final CAPO policy, 3 times each, followed by the frozen VLA that was deployed when the intervention occurred, without further human assistance.

Table 7 (L1391). Return-based validation of the corrective preference assumption.

TaskPairsSucc. AHA^HSucc. AπA^\piSucc. AfinalA^{\rm final}Better/Tie/WorseΔG≥m\Delta G\ge mAgr. TDAgr. CVL
USB Insertion3081.114.485.683.3/6.7/10.080.075.089.3
Screw Tightening3074.418.977.876.7/10.0/13.373.370.485.2
Block Stacking3068.921.173.373.3/13.3/13.370.069.280.8
Overall9074.818.178.977.8/10.0/12.274.471.685.2

The correction yields a higher return than the overwritten proposal in 77.8% of the pairs and a lower return in 12.2%. This violation rate lies within the 10–20% label-noise range under which CAPO still outperforms TD+APO. The CVL critic agrees with the sign of the real return gap more often than the TD-only critic (85.2% versus 71.6%). Because interventions are triggered by clear failures, the overwritten proposals in these states rarely lead to success (18.1%), which explains why the assumption holds for most real pairs. The final policy succeeds from the same states in 78.9% of the attempts, compared with 74.8% for the human correction, so CAPO is not capped at the operator's level.

The last observation is important for the reviewer's concern. CVL only requires a correction to beat the failing proposal it overwrote, not the best possible action, and APO can follow the critic gradient beyond the correction. We have also expanded the Limitations section accordingly (L1538–1542).

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer #7u1E (3/3)

Response to Reviewer #7u1E (3/3)

W3 & Q2: Generalization to other backbones and more diverse environments

(1) CAPO applies to mainstream diffusion- and flow-based VLAs. A key contribution of CAPO is a general recipe for stable real-world RL on this model family. CVL trains a separate critic with its own encoder, so it is independent of the VLA backbone. APO only requires the action expert's velocity field vθv_\theta, vector–Jacobian products of the reference field, and the native sampler, all of which are available in mainstream diffusion- and flow-based VLAs such as π0, π0.5, and GR00T. APO keeps updates stable by matching detached local targets instead of backpropagating through the sampler, and by regularizing toward a phase-wise snapshot of the current policy (L1415).

(2) New: transfer to GR00T (L494 and Appendix D.4, L1411–1437). Following the reviewer's suggestion, we applied CAPO without modification to GR00T, whose action head differs in architecture from that of π0.5. All methods start from the same task-specific SFT checkpoint, use the same demonstrations and budgets as in the main experiments, and are evaluated with the frozen-policy protocol of Table 4.

Table 8 (L1424). Frozen-policy success rate (%) with two VLA backbones, using the protocol of Table 4. The π0.5 rows are taken from Table 4, and GR00T uses one run per task.

BackboneMethodUSB InsertionCup HangingPeg-in-HoleAvg. Succ.
π0.5SFT20.070.020.036.7
π0.5CVL+QAM83.3100.093.392.2
π0.5CAPO93.3100.0100.097.8
GR00TSFT10.050.020.026.7
GR00TCVL+QAM70.090.080.080.0
GR00TCAPO80.0100.090.090.0

With GR00T, CAPO improves average success over SFT by 63.3 points, compared with 61.1 points with π0.5, and outperforms CVL+QAM by 10.0 points, compared with 5.6 points with π0.5. The ordering of the methods is consistent across both backbones. (L1434–1437)

(3) Environmental diversity is evaluated in Figure 8a. Beyond the six tasks on two robot platforms (Franka FR3 and Rizon 4), Section 5.5 evaluates frozen checkpoints under environmental changes, without adaptation. We now describe these changes in the main text (L489–493) and in Appendix C.4 (L1185–1293), and we add example images of each change in Figure 9:

  • Layout. The tablecloth is replaced with one of a different color, and clutter objects unseen during training are added to the workspace.
  • Lighting. Non-uniform illumination is added, creating uneven brightness and shadows on the task objects.
  • Distractors. The target position is perturbed while the policy is executing, so the policy must react to a moving goal in closed loop.

Under these changes, CAPO achieves 83.3%, 86.7%, and 88.3%, compared with 70.0%, 76.7%, and 78.3% for the strongest baseline, CVL+QAM. The Orig. column of Figure 8a now reports the frozen-policy evaluation in the original setting, so all conditions share the same protocol (L511–512). During training, object poses are also randomized within a 50×50 cm region rather than fixed positions (L344–345).

We thank the reviewer again for the insightful comments. We hope that our responses have resolved the reviewer's concerns, and we would be very grateful for a corresponding update of the rating. If any concerns remain, please let us know, and we will respond promptly.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers and the AC

General Response to All Reviewers and the AC

We sincerely thank the Area Chair and all three reviewers (#7u1E, #vDAa, #AxeN) for the time and effort they devoted to reviewing our manuscript, and for their constructive and insightful feedback. We are encouraged that Reviewer #7u1E finds our motivation "well-justified", Reviewer #vDAa considers our ablations "consequential controls", and Reviewer #AxeN recognizes our method "comprehensive".

We have carefully considered every comment, actively incorporated the suggestions, and revised the manuscript accordingly. In the revised PDF, new or modified text is color-coded: blue marks changes that address several reviewers, orange marks changes for Reviewer #7u1E, green for Reviewer #vDAa, and purple for Reviewer #AxeN. All line (L) numbers in our responses refer to the revised PDF.

The main revisions are:

  1. AI use and reproducibility statements (L524–539).
  2. Behavior-cloning supervision. Following prior Human-in-the-Loop Reinforcement Learning (HiL-RL) works, Ddemo\mathcal D_{\rm demo} contains demonstrations and online human corrections. All actor variants apply the same BC objective to the shared demonstrations and to their own online corrections, collected by the same operator under the same intervention protocol (L177 and L189–191, L358–363, L918–926).
  3. Evaluation protocol, error bars, and implementation details. We report the number of runs and the error bars, describe the workspace randomization and the intervention protocol, add a frozen-policy evaluation without intervention (Table 4), and specify the execution horizon, reward classifier, critic architecture, and replay and update schedules (L345–355, Appendices B.1, B.3, C.1, and C.3, Table 1).
  4. Human corrections. We explain why the corrective preference assumption is mild and how CVL limits the influence of imperfect corrections (L241–242 and Appendix A.2), show that CVL helps the policy learn corrective behavior faster (L419–420), reframe the critic-label agreement analysis (Appendix D.2), and add a return-based validation (Appendix D.3, Table 7).
  5. Generalization. We describe the environmental variations (L489–494, Appendix C.4) and add a transfer experiment to GR00T, showing that CAPO applies to mainstream diffusion- and flow-based VLAs (Appendix D.4, Table 8).
  6. Novelty. We clarify what makes CAPO distinct from existing policy-update routes and from concurrent intervention-based critics, although these concurrent works are available only as preprints and have not yet been peer-reviewed (L120–133, L252–254, Appendix F, Table 12).
  7. Long-horizon tasks. We add two four-stage, long-horizon tasks with an illustration of both workflows (Appendix D.5, Figure 10, Table 9).

To support reproducibility and further research by the community, we have released the full implementation of CAPO at an anonymous repository: https://anonymous.4open.science/r/ICLR-Submission-26906/.

We hope that our responses and revisions address the reviewers' concerns. If any concerns remain, please do not hesitate to let us know, and we will respond promptly.

AuthorsOpenReview ↗

Rebuttal Summary and Main Revisions

Rebuttal Summary and Main Revisions

Dear PC, SAC, AC, and Reviewers,

We sincerely thank you for your time, thoughtful suggestions, and engagement throughout the review process. As the discussion period draws to a close, we summarize the reviewer feedback and the main revisions made in response.

Reviewer #vDAa confirmed that the concerns raised in the review had been resolved and increased the rating. Reviewer #AxeN indicated that most questions had been answered and also raised the rating. Reviewer #7u1E acknowledged our efforts and maintained the original rating. All three reviewers currently rate the paper as 3 (Weak Acceptance). We appreciate the reviewers’ careful consideration of our responses and additional experiments.

The main revisions are as follows:

1. Clarified the technical contribution. We expanded the explanation of how CVL shapes the critic’s action gradients using human corrections and how APO translates these gradients into updates of the VLA’s native flow policy while preserving its deployment sampler. We also clarified the distinctions from related methods and explained how the component ablations support these design choices (Section 2, Appendix F, Table 12).

2. Clarified comparison fairness and strengthened evaluation. We clarified that all compared actor variants use the same BC objective and weight, share the offline demonstrations, and incorporate corrections collected during each variant’s own online training under the same intervention protocol. We also specified run counts, reported error-bars, and added frozen-policy evaluation without human intervention, confirming that CAPO maintains its performance gains over the evaluated baselines (Section 5.1, Appendix C.3, Table 4).

3. Validated the reliability of corrective preferences. We added return-based evaluation comparing human corrections with rejected policy proposals under the same frozen-policy continuation. The results support the reliability of corrective preferences and show that CVL better captures the observed return ordering than the TD-only critic. We also clarified the distinction between critic-label agreement and direct return-based validation (Appendices D.2–D.3, Table 7).

4. Broadened the empirical evidence. We added experiments on the GR00T backbone and multi-stage, long-horizon manipulation tasks. CAPO consistently outperforms the compared baselines in these settings, supporting its applicability beyond the original backbone and to longer tasks (Appendices D.4–D.5, Tables 8–9).

5. Improved reproducibility and reporting. We added a reproducibility statement, expanded the implementation and evaluation details, and clarified the environmental perturbation settings. The full implementation is available through our anonymous code repository: https://anonymous.4open.science/r/ICLR-Submission-26906/.

We sincerely appreciate the reviewers’ suggestions, which have helped improve the clarity and rigor of our paper. We hope this summary is helpful for the final assessment.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.