acceptodds
Under review as a conference paper at ICLR 2027

CPO-RL: COT-POLICY CO-OPTIMIZATION FOR EMBODIED REINFORCEMENT LEARNING

Abstract

Reinforcement learning (RL) can adapt Vision-Language-Action (VLA) models through interaction, but reward feedback alone does not explicitly identify the contact regions needed for precise manipulation. We propose CPO-RL, a CoT–Policy Co-Optimization framework that jointly trains a contact-grounded intermediate representation and a continuous control policy. CPO-RL uses dynamic 2D contact masks as supervision for an embodied Chain-of-Thought (CoT) latent. An input-dependent contact token conditions action prediction, while an auxiliary decoder combines this latent with RGB features to predict the contact mask. Contact-SFT first initializes the policy and the contact representation from demonstrations. During online learning, PPO uses all collected trajectories, while valid contact targets derived from successful simulated interactions supervise the auxiliary branch and its shared policy representation. This couples policy optimization with refinement of contact localization without requiring predicted masks or the auxiliary decoder at deployment. On four ManiSkill manipulation tasks, CPO-RL achieves 94.2% average success, improving over the official RLinf baseline by 19.3 percentage points and reducing mean episode length from 68.5 to 52.7 steps. A controlled component ablation shows that contact supervision contributes beyond adding the latent token alone. After sim-real co-training, deployment on a physical Franka robot yields 66.9% average success across original and distractor settings, compared with 29.4% for SFT. These results support contact-grounded representation learning as a practical complement to reward-based VLA post-training.

Primary area:
applications to robotics, autonomy, planning
Keywords:
embodied reinforcement learning, sample-efficient reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Ms8fOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CPO-RL, which jointly optimizes a contact-grounded latent representation and a VLA policy by combining PPO with contact-mask supervision from successful trajectories. It offers a practical approach for continuously refining an action-conditioned spatial representation during online RL, achieving strong performance in both simulation and real-world manipulation.

Critical strengths
  1. It introduces a simple and practical way to refine a contact-aware representation together with the policy during online RL.
  2. The experimental results are clear. It demonstrates consistent improvements across different simulation tasks and real-world experiments, and the ablation study is also clear.
Critical weaknesses
  1. Novelty is kind of limited. The naming of 'CoT–Policy Co-Optimization' sounds big, but the actual method is more like adding contact supervision and doing it in an online manner, which sounds pretty natural.
  2. Contact supervision depends on privileged, task-specific heuristics, which may make it hard to scale up.
Questions for authors

Could the authors compare CPO-RL with a simpler baseline that applies the contact mask loss directly to the policy representation, without introducing a dedicated CoT token? This would clarify whether the gain comes from the proposed latent architecture or simply from additional contact supervision.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Ms8f (1/2): Contribution and Privileged Heuristics (W1, W2)

Thank you for the positive assessment and for noting that the method is simple and practical. We address the two weaknesses and the question below.

W1. Novelty.

Our contribution goes beyond adding contact supervision online: CPO-RL is a framework in which a contact-grounded representation and the policy improve together during online RL. We highlight three aspects.

  1. Co-optimization accelerates RL training. Contact targets are regenerated from the policy's own successful rollouts, so the supervision keeps improving as the policy improves. With the same initialization and interaction budget, CPO-RL reaches 89.0% training success on Peg Insertion by iteration 400, already above the final value of w/o CoT-Opt at iteration 600 (81.9%; Table 7), and reaches 90% evaluation success at iteration 450, a level w/o CoT-Opt does not reach within 600 iterations.

  2. Each design component contributes. Under identical settings on Peg Insertion: routing the supervision through a dedicated latent token is essential, since applying the same loss directly to action features reaches only 72.2% versus 95.6% (Q1); contact masks outperform object-segmentation and gripper-trajectory targets (91.7% vs. 82.2% and 85.2%); and offline Contact-SFT and the online contact loss are complementary (84.4% and 91.7% alone, 95.6% together; Table 2).

  3. Transfer to the real robot without simulator ground truth. With targets generated from SAM segmentations of real rollouts, online contact supervision raises real-world Peg Insertion success from 75% to 85% (W2).

These gains require no additional sensors and add no inference cost at deployment. Following your suggestion, we will narrow the naming to "contact-grounded latent CoT" so that it describes the method precisely.

W2. Dependence on privileged, task-specific heuristics.

To quantify whether removing privileged information from mask construction affects our method, we ran online RL directly on the real robot for Peg Insertion using the RLinf-USER infrastructure [1], with auxiliary masks constructed automatically from SAM segmentations of real-world rollouts; these masks reach 82.1% mean IoU against manually checked annotations. All methods use the same base policy, demonstration data, real-world interaction budget, and evaluation protocol (20 trials); w/o CoT-Opt shares the Contact-SFT initialization with CPO-RL and only disables online contact supervision. CPO-RL achieves 85% (17/20), versus 75% (15/20) for w/o CoT-Opt and 70% (14/20) for RLinf, so online contact supervision remains effective without privileged information.

The heuristics share common interaction templates across tasks. Targets are built from two categories of rules, robot–object contact and object–object contact, both of which use only object masks and the gripper state. These two categories cover all four tasks.

AuthorsOpenReview ↗

Response to Reviewer Ms8f (2/2): Direct Contact Supervision Baseline (Q1)

Q1. Direct contact supervision without the CoT token.

We implemented this baseline on Peg Insertion. The CoT token is removed; the action-token hidden states from the final action-expert layer (before the action output projection) are mean-pooled and passed to the same adapter and mask decoder, so the contact loss directly updates the action expert. During training, these hidden states are taken at the flow-matching timestep sampled for each example; as in CPO-RL, the decoder is not used at inference. Targets, loss weights, Contact-SFT, and budget are identical to CPO-RL.

MethodRLinfDirect contact supervisionCPO-RL
Peg Insertion SR (%)59.472.295.6

With identical targets, loss, Contact-SFT, and budget, applying the supervision directly to action features reaches only 72.2%, barely above the token-only variant without any contact supervision (68.6%, Table 2), whereas applying it through the latent token reaches 95.6%. We attribute this to the latent token decoupling contact supervision from action generation. Action-token hidden states are computed from noisy actions at a randomly sampled noise level and directly produce the action output; a mask loss on them asks these noise-dependent features to also predict a noise-independent spatial target, which competes with the action objective. In CPO-RL, the causal mask prevents the latent token from attending to action tokens, so the token is independent of action noise. The contact loss shapes this separate interface, and the action tokens read from it without their own features being overwritten. We will add this baseline to Sec. 4.3.

Thank you again for the positive assessment. Please let us know if you have any remaining concerns.

References

[1] Zang, H., Yu, S., Lin, H., Zhou, T., Huang, Z., Guo, Z., Xu, X., Zhou, J., Sheng, Y., Zhang, S., Gao, F., Tang, W., Yue, Y., Zhang, Q., Chen, X., Yu, C., & Wang, Y. (2026). RLinf-USER: A unified and extensible system for real-world online policy learning in embodied AI (arXiv:2602.07837). arXiv. https://doi.org/10.48550/arXiv.2602.07837

Thanks the author for the response. My W2 is largely resolved, though I still feel like the novelty is slightly overclaimed. Overall, this is a good paper. I hope it can be accepted, and I'll increase my confidence for my current score.

Thank you for the positive assessment. Following your comment, we have narrowed “CoT” to contact-grounded latent CoT throughout the paper. We have also uploaded the revised PDF, with all changes marked in blue, for your reference. Please let us know if you have any further questions.

Reviewer_qrPPOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CPO-Policy, which jointly learns contact latents and the policy for a VLA. Contact latents are obtained using 2D masks from SAM and used to condition the policy. The SAM features and the contact latent are used via a learned adapter to form a learned query that is used together with the context, i.e., the robot state and text instruction to predict an action chunk. The method does not use the mask directly, but only for training via an auxiliary contact loss.

Critical strengths
  • Improves upon the baseline by using contact information
  • Experiments with a real robot show method transfers to real robots
Critical weaknesses
  • Limited to ManiSkill environments proposed by the authors (tabletop manipulation) and no comparison against stronger RLHF/VLA baselines, e.g. methods that predict contact such as ContactFlow (https://arxiv.org/abs/2607.26579), or use proprioceptive information, e.g. ForceVLA (https://arxiv.org/abs/2505.22159) or TactileVLA (https://arxiv.org/pdf/2507.09160) or object-centric representations such as SlotVLA (https://arxiv.org/abs/2511.06754). Could the authors at least position their work with respect to related works and clarify potential benefits of their approach?
  • Requires ground-truth contact targets from the simulator and successful trajectories/expert demonstrations, which may not always be available, thus limiting using for example unlabeled video data for training.
  • Presentation and soundness of the paper:
    • In general, hard to follow because of the (dense) writing style
    • Would be helpful if references are next to the methods, e.g. MolmoAct (Lee et al., 2025)
    • Contact-SFT is used in the main part without any explanation, which is confusing. The only explanation is in Appendix A.2, but I am not sure about the differences and whether this is used as a basis. Could you please clarify this?
    • Acronyms and notation are often not explained directly, e.g. masks MtM_t in line 214/215 (which should be probably also M^t\hat{M}_t or not?) or NCD in Fig. 5
    • Some terms that are used are confusing and not standard, e.g. "real-world sim-real co-training" in line 670 (does this refer to sim2real?) and for some sentences like "Curve endpoints therefore need not equal table entries" (line 363/264) it is hard to understand their meaning
    • Formatting of references should be improved and some references such as SegmentAnything be replaced by the published conference paper (CVPR in this case)
  • I am not sure if I would be able to reproduce the method from the explanations in the paper because it is not clear to me how exactly the mask heuristics work? Appendix A.4 seems quite superficial and lacks details (see also my question below).
Questions for authors
  • What is the difference between CPO and w/o CoT-Opt? Is this ablation to check whether the performance improvement comes from the additional capacity through the learned latent rather then from the contact information itself?
  • How does the contact heuristic for the targets exactly work? Please also elaborate what "For grasping and object-object interactions, we intersect relevant 2D masks rendered separately at different time steps within the same episode. Peg Insertion additionally uses dilation to estimate the slot’s central opening for the insertion target" exactly mean.
  • How are successful trajectories defined? Overall success or for example only grasp? If only overall success, why not use contact heuristics/masks if grasp is successful?
  • Why is there no number of steps for the baseline OpenPI? As I understand the caption of Tab. 1, does this mean that it does not finish the task within the maximum number of steps/episode horizon? If so, where do the success rates come from?
  • At inference time SAM is needed to get the features for the learned query - how big is the additional computational effort (inference time compared to baselines)?
  • What makes the 2D masks "dynamic"?
  • How were loss coefficients determined and how robust is the method to their choice?
6 comments
AuthorsOpenReview ↗

Response to Reviewer qrPP (1/3): Clarifications, Related Methods, and Presentation (W1–W3)

Thank you for the careful reading. We first clarify two points that may have caused confusion: SAM is not used at inference (the auxiliary branch is training-only, L217–219), and the four tasks are digital twins of our real-robot setup rather than a benchmark designed for our method. Detailed responses follow.

W1. Environments and comparison with related methods.

ManiSkill is used only as the simulation platform. The four tasks are digital twins of our physical Franka setup, built following the RLinf digital-twin construction (Appendix A.5, C), so that the same tasks support sim-real co-training and real-robot evaluation (Table 3).

The suggested methods mainly contribute new model designs or input modalities for manipulation: contact representations (ContactFlow), force and tactile sensing (ForceVLA, TactileVLA), and object-centric representations (SlotVLA). CPO-RL is complementary: it keeps a standard RGB VLA unchanged at deployment and instead uses contact as a training signal, generated online from the policy's own successful rollouts during RL. It requires no additional sensors and adds no inference cost. We will add these works to the related work.

W2. Dependence on simulator targets and successful trajectories.

CPO-RL is designed for the RL post-training stage, where interaction and rewards are available by construction. It is therefore complementary to pretraining on unlabeled video rather than a replacement for it, and it builds on a pretrained VLA (π0.5).

Our real-robot experiment shows that neither simulator information nor additional demonstrations are required. We ran online RL directly on the real robot for Peg Insertion, where contact targets are generated automatically from SAM segmentations of the policy's own successful rollouts. CPO-RL reaches 85% success, versus 75% without online contact supervision and 70% for RLinf. Demonstrations are used only for the initial SFT.

W3. Presentation.

Writing and references. Thank you for these suggestions. We will revise the dense passages, place citations next to method names (e.g., MolmoAct (Lee et al., 2025)), and update references to their published versions (e.g., Segment Anything, ICCV 2023).

Contact-SFT. Contact-SFT is the offline initialization stage of CPO-RL (Sec. 3.4, Stage 1). It differs from standard SFT in one respect: in addition to the flow-matching action loss, it trains the CoT token, adapter, and mask decoder with the contact-mask loss on demonstrations (LSFT=10Lflow+2LCoT\mathcal{L}_{SFT} = 10\mathcal{L}_{flow} + 2\mathcal{L}_{CoT}). Both CPO-RL and CPO-RL w/o CoT-Opt start from the same Contact-SFT checkpoint, while OpenPI (SFT) and RLinf use standard SFT. We will state this explicitly in Sec. 4.1.

Notation. MtM_t in L214–215 is correct: it denotes the predicted mask, while M^t\hat{M}_t denotes the heuristic target; both appear in Eq. 3. NCD is the normalized centroid distance (Sec. 4.3, L424–426), i.e., the distance between predicted and reference centroids divided by the image diagonal. We will define both at first use, add NCD to the caption of Fig. 5, and check all acronyms.

Terminology. Sim-real co-training follows RL-Co (Shi et al., 2026): the policy is trained jointly on simulation interaction and real-world demonstrations, rather than trained purely in simulation and transferred zero-shot (sim2real). "Real-world sim-real co-training" in L670 refers to the real-world data used in this joint training. L363–364 means that Fig. 3 plots the success of training rollouts, which include exploration noise, whereas Table 1 reports a separate final evaluation (5 evaluation seeds × 128 episodes), so the curve endpoints differ from the table values. We will rephrase both passages.

AuthorsOpenReview ↗

Response to Reviewer qrPP (2/3): Mask Construction and Ablation Details (W4, Q1–Q4)

W4/Q2. How are the contact masks constructed?

We generate contact masks from successful simulated rollouts. Gripper closure separates the grasp phase from the placement or insertion phase. The table lists the target region for each task and phase.

TaskGrasp targetPlacement or insertion target
Multi-object Pick and PlaceGripper–object contact region for the object named in the instructionContact region between that object and the plate
Stack CubeContact region between the gripper and the movable cubeContact region between the two cubes
Place SphereContact region between the gripper and the sphereContact region between the sphere and the bin
Peg InsertionContact region between the gripper and the pegCentral opening in the slot board

For the contact regions in the table, "temporal intersection" means taking the pixelwise intersection of two object masks from different frames of the same rollout. For example, for the peg grasp, we intersect the peg mask from the initial frame with the gripper mask at grasp completion. Since the peg remains in place until grasped, the overlapping pixels identify the part of the peg covered by the gripper.

The insertion target uses a separate rule. Sec. 3.3 and Appendix A.4 refer to this step as "dilation", which is imprecise. Concretely, in the initial-frame segmentation, we identify connected regions of background pixels and retain those that do not touch the image border. The region enclosed by the slot board gives the insertion target.

We resize each target to 224 × 224 pixels, use it only for frames in the corresponding phase, and skip all-zero masks. We will replace this term with the precise description, add these task-specific rules to Appendix A.4, and release the mask-generation code.

Q1. Difference between CPO-RL and w/o CoT-Opt.

The two variants have the same architecture, including the latent token, and start from the same Contact-SFT checkpoint; the only difference is whether the contact-mask loss is applied during online RL. This ablation therefore isolates online contact supervision (86.0% → 94.2% on average).

Capacity is tested separately in Table 2. Adding the latent token to RLinf without contact supervision raises Peg Insertion success only from 59.4% to 68.6%, whereas the same token trained with the online contact loss reaches 91.7%. The improvement therefore comes from contact supervision of the latent rather than from added capacity.

Q3. Definition of successful trajectories.

A trajectory is successful when it satisfies the overall task success criterion (Appendix C), not only a successful grasp. We do not use trajectories that grasp successfully but fail later, because a grasp can succeed while still being unsuitable for the rest of the task; for example, a peg grasped at a poor position may not be insertable. Using such grasps as targets would reinforce contacts that do not lead to task completion. Failed trajectories are still used in the PPO update and are excluded only from the contact loss.

Q4. Missing step count for OpenPI (SFT).

"-/H" does not mean that the SFT policy fails to finish within the horizon. All methods, including SFT, use the same evaluation protocol: an episode counts as successful if the task is completed within the maximum horizon H, which is the step limit shown in the table. For SFT we report the success rate as the imitation-only reference, and we report episode length for the RL variants because more efficient task completion is one goal of RL post-training. We will clarify this in the caption of Table 1.

AuthorsOpenReview ↗

Response to Reviewer qrPP (3/3): Inference Cost and Loss Coefficients (Q5–Q7)

Q5. Inference cost.

SAM is not used at inference. The latent token is formed from the policy's own context (observation, language, and robot state), not from SAM features; the SAM encoder and mask decoder are used only to compute the training loss and are removed at deployment (L217–219, Appendix B.1). Relative to the baselines, the only inference overhead is one additional token in the policy's input sequence, which is negligible. We will make this clearer in Fig. 2.

Q6. Why the masks are "dynamic".

The target changes with the manipulation phase within an episode (L237–238): during grasping it marks the gripper–object contact region, and afterwards the placement or insertion region. The targets are also regenerated from new successful rollouts throughout online training.

Q7. Loss coefficients.

The coefficients are chosen to keep the loss terms at comparable magnitudes. In online RL, the PPO and contact losses are summed without weighting (Eq. 4); Contact-SFT uses 10Lflow+2LCoT10\mathcal{L}_{flow} + 2\mathcal{L}_{CoT} (Eq. 5) for the same reason. The same coefficients are used for all four tasks without per-task tuning. To test robustness, we varied the online contact weight λ\lambda on Peg Insertion, evaluating every 50 PPO iterations:

λ\lambda0 (w/o CoT-Opt)0.51 (default)
SR at iteration 600 (%)84.493.395.6
Iterations to reach 90%not reached500450

Halving the weight barely changes final success (2.3 points) and mainly affects convergence speed (500 vs. 450 iterations to reach 90%). Both settings remain far above training without contact supervision, which does not reach 90% within 600 iterations.

Thank you again for the careful reading. Several concerns stemmed from points we did not state clearly enough, notably that SAM is used only during training and that the four tasks are digital twins of our real setup; we will make these explicit in the revision. We hope these clarifications and the new experiments address your concerns, and we would be grateful if you would re-evaluate the paper and consider raising your rating.

Reviewer_6qAuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CPO-RL, an RL post-training framework for VLA models that jointly optimizes a contact-grounded latent and the action policy. A learnable query becomes an input-dependent "CoT" latent conditioning action-chunk prediction; during training, an auxiliary SAM-based decoder predicts 2D contact masks (BCE+Dice) from targets heuristically built out of simulator segmentation and contact signals on successful trajectories only. The branch is discarded at deployment. On four ManiSkill tasks it reports 94.2% average success vs. 74.9% for RLinf, and 66.9% vs. 29.4% (SFT) on a real Franka after sim-real co-training.

Critical strengths
  1. Well-motivated mechanism: reward does not localize contact regions, and phase-varying 2D contact masks give the policy a spatial training signal that is absent from scalar rewards, with clear gains on precision tasks (Peg Insertion 59.4% -> 95.6%).
  2. Practical asymmetric design: the auxiliary SAM branch is training-only, so deployment adds no inference cost and needs no privileged information.
  3. More honest ablation than typical: Table 2 separates token architecture (+9.2), Contact-SFT (+15.8), and online contact loss (+23.1) under matched budget, directly addressing the "just an extra token" objection.
  4. Real-robot evaluation with distractor settings and matched interaction budgets across all RL variants.
Critical weaknesses
  1. The decisive control experiment is missing. The paper shows contact-mask supervision helps, but never tests whether the gain comes from contact semantics specifically versus any dense spatial auxiliary task (object segmentation, depth, keypoints) under the same budget. The core claim — "contact-grounded" as the causal ingredient — is asserted, not isolated.
  2. The mechanism is an instance of an established paradigm the paper does not position itself against: auxiliary spatially-grounded supervision shaping an internal representation during training, discarded at inference. This spans segmentation/affordance auxiliary losses in imitation learning and region-focused alignment objectives in video world models (e.g., PhysisForcing, arXiv:2606.28128). The distinctive part here — targets self-generated from successful online rollouts — is real but narrower than the framing suggests. Relatedly, the "CoT" label oversells a single supervised latent token.
  3. The mask-localization analysis is circular: IoU/NCD are measured against the same heuristic pipeline (temporal intersection, gripper-state phase boundaries, insertion-hole dilation) that produces the training targets. Better agreement with one's own supervision signal is nearly definitional and does not evidence correct contact semantics.
  4. No training-seed variance anywhere: Table 2 ablations use a single seed, and the five "evaluation seeds" vary test conditions, not training initialization. On the real robot there is no RLinf or w/o-CoT-Opt arm, so the contribution of contact co-optimization versus sim-real co-training itself cannot be separated — and the online contact mechanism is inherently sim-bound since it needs simulator segmentation/contact ground truth.
Questions for authors
  1. Does the gain depend specifically on contact masks? What happens under identical budgets with alternative spatial auxiliary targets (object segmentation, depth, keypoints)?
  2. How stable are the key deltas (e.g., the 8.2-point gap vs. w/o CoT-Opt) across multiple training seeds?
  3. How sensitive are results to the mask-construction heuristics (phase boundaries, temporal intersection, hole dilation), and can comparable targets be produced without simulator privileges for real-world rollouts?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 6qAu (1/3): Contact-Specific Gains and Positioning (W1, Q1, W2)

Thank you for the detailed and constructive review, and for recognizing the motivation, the training-only design, and our ablations. We address each point below.

W1 / Q1. Does the gain depend specifically on contact masks?

To isolate the role of contact semantics, we added a control experiment on Peg Insertion in which only the auxiliary target changes. All variants follow the "RLinf + CoT Token + Contact Loss" setting in Table 2 (no Contact-SFT), with the same architecture, mask decoder, BCE+Dice loss, interaction budget, training seed, and evaluation protocol. We compare four settings:

  • Token only: the CoT token is trained by PPO only, without any spatial supervision (the "RLinf + CoT Token" row of Table 2).
  • Object segmentation: full masks of the task-relevant objects, switched at gripper closure: the whole peg during grasping and the whole slot board during insertion. This target indicates which objects matter, but not where on them contact occurs.
  • Gripper trajectory: the path the gripper is about to follow. For each frame, we project the gripper-center positions of the next action chunk into the image and mark a 5-pixel-radius disk around each point. This target is aligned with the action, but carries no information about which object regions are contacted.
  • Contact (ours): the part of the peg covered by the gripper during grasping, and the hole opening during insertion.
SettingToken onlyObject seg.Gripper traj.Contact (ours)
SR (%)68.682.285.291.7

Dense spatial supervision helps in general: object segmentation and gripper trajectory improve over the token-only variant by 13.6 and 16.6 points. Contact masks add a further 9.5 and 6.5 points over these two targets, even though the trajectory target is directly aligned with the action. The gain therefore depends on contact semantics, not only on having a dense spatial loss. We did not include depth: it is a continuous regression target that requires a different loss and output head, so its effect could not be separated from these architectural changes. We will add this experiment to Sec. 4.3.

W2. Positioning, scope of novelty, and the "CoT" label.

We agree that CPO-RL builds on training-only auxiliary supervision and on CoT-style VLAs (e.g., CoT-VLA, LaST0), and we will position it against these works in the related work, including segmentation/affordance auxiliary losses and PhysisForcing. What distinguishes CPO-RL is that the auxiliary representation is co-optimized with the policy during online RL: targets are regenerated from the policy's own successful rollouts, so the supervision and the policy improve together. This yields three properties that a fixed auxiliary loss does not provide:

  1. Faster RL training. With the same initialization and budget, CPO-RL reaches 89.0% training success on Peg Insertion by iteration 400, already above the final value of w/o CoT-Opt (81.9%; Table 7), and the latent itself is refined during training (W3).
  2. Each component contributes. Contact targets outperform object-segmentation and gripper-trajectory targets (91.7% vs. 82.2% and 85.2%; W1); a dedicated latent token is essential, since applying the same loss directly to action features reaches only 72.2% versus 95.6%; and online updates raise average success from 86.0% to 94.2% (Table 1).
  3. Real-robot transfer without simulator ground truth (W4 (b)).

On the label, we follow VLA work that conditions actions on an intermediate representation formed beforehand (e.g., CoT-VLA, LaST0, LaRA-VLA). To avoid ambiguity, we will restrict the term to contact-grounded latent CoT throughout the paper.

AuthorsOpenReview ↗

Response to Reviewer 6qAu (2/3): Independent Localization Evaluation and Training Seeds (W3, W4, Q2)

W3. Circularity of the mask-localization analysis.

IoU/NCD against heuristic targets only measure agreement with our own supervision. We therefore added an evaluation that checks whether contact localization improves during online training, using references independent of our heuristics:

  • Reference (from physics, not heuristics). For each held-out trajectory, we take one keyframe at grasp completion and one at placement/insertion. The references are 3D points from simulator geometry: gripper–object and object–object contact points reported by the physics engine, and, for insertion, points sampled on the known hole-opening disk. They are projected into the image with the camera parameters.
  • Prediction. For each checkpoint, the auxiliary decoder outputs a contact probability map for the same keyframe; we keep its top 2% of pixels.
  • Metric. Coverage@2% is the fraction of reference points falling inside this region (a random 2% region would cover ~2%), averaged over the two keyframes and then over trajectories. All checkpoints use the same keyframes and references.
TaskPPO iterationsCoverage@2% (%)
Peg Insertion0 / 200 / 400 / 60048.5 / 57.3 / 55.8 / 69.2
Stack Cube0 / 50 / 100 / 22052.1 / 56.4 / 61.7 / 66.3

Coverage rises by 20.7 and 14.2 points over online training, showing that online co-optimization refines the latent toward physically correct interaction regions. We will add this evaluation to Sec. 4.3.

W4 (part 1) / Q2. Training-seed variance.

The 8.2-point gain in Table 1 is the average across four tasks under the original training seed. To test whether the gain on Peg Insertion is reproducible, we trained both CPO-RL and w/o CoT-Opt with two additional training seeds. Each trained policy was evaluated over 640 episodes.

Training seedw/o CoT-Opt SR (%)CPO-RL SR (%)Paired gain (pp)
Original84.495.6+11.2
Additional seed 186.494.7+8.3
Additional seed 283.194.4+11.3
Mean ± SD84.6 ± 1.794.9 ± 0.6+10.3 ± 1.7

Online contact supervision improved Peg Insertion success under all three training seeds, with a mean paired gain of 10.3 percentage points (95% CI: [6.0, 14.5]). The smallest gain (+8.3) is still several times the seed-to-seed SD of either method (≤ 1.7 points), and the component gaps in Table 2 (+9.2 to +23.1 points) are larger still. CPO-RL also varied less across seeds than w/o CoT-Opt (SD 0.6 vs. 1.7).

AuthorsOpenReview ↗

Response to Reviewer 6qAu (3/3): Real-Robot Attribution and Heuristic Sensitivity (W4, Q3)

W4 (part 2). Real-robot attribution and dependence on simulation.

Evaluation protocol. All our real-robot evaluations use the same protocol. For each task and setting, we randomly selected 20 initial object placements once and evaluated every policy on this same set, so all methods face identical starting conditions. Each policy is tested once per placement, and the objects are reset to the designated placement before every trial. A trial is counted as successful if the robot completes the task. We report the number of successful trials out of 20 together with the success rate.

(a) Attribution under sim-real co-training. We added CPO-RL w/o CoT-Opt on real-world Peg Insertion (original setting). Both variants share the Contact-SFT initialization, real-world demonstrations, and simulation interaction budget; they differ only in whether online contact supervision is enabled.

Sim-real co-trainingPeg Insertion SR (%)
Sim-Real SFT25 (5/20)
CPO-RL w/o CoT-Opt60 (12/20)
CPO-RL75 (15/20)

RL co-training without online contact supervision raises success from 25% to 60%, and online contact supervision adds a further 15 points.

(b) Without simulator information. To test whether the online mechanism is inherently sim-bound, we also ran online RL directly on the real robot using the RLinf-USER infrastructure [1]. Contact targets are generated automatically from SAM segmentations of the policy's own successful real rollouts and the gripper state, with no simulator access. These targets reach 82.1% mean IoU against manually checked annotations. All methods use the same base policy, demonstration data, and real-world interaction budget; w/o CoT-Opt shares the Contact-SFT initialization with CPO-RL and only disables online contact supervision.

Real-world online RLPeg Insertion SR (%)
RLinf70 (14/20)
CPO-RL w/o CoT-Opt75 (15/20)
CPO-RL85 (17/20)

Under both settings, adding online contact supervision to an otherwise identical model improves real-world success (+15 and +10 points). In (b), this holds with targets built entirely from real rollouts.

Q3. Sensitivity to the heuristics and targets without simulator privileges.

No tuned parameters. None of the three steps has a threshold or hyperparameter to tune. Phase boundaries are the binary gripper open/close transitions. Temporal intersection is a bitwise AND of masks from different time steps. The insertion target is the hole opening, extracted as the background region enclosed by the board.

Target quality. We checked how often the heuristics produce degenerate targets. Over successful trajectories from ten rollout batches (128 environments each), we count a non-empty target as abnormally small if its area is below 0.1× the median for the same task phase (mean ± std over batches):

TaskSuccessful trajectoriesLow-coverage rate (%)
Peg Insertion4977.75 ± 6.26
Stack Cube6035.50 ± 2.93
Place Sphere7980.83 ± 2.64
PnP8124.32 ± 1.57

Abnormally small targets stay below 8% on all tasks. They are most frequent on Peg Insertion, where the insertion region is smallest, yet this task shows the largest gain from contact supervision. The method therefore tolerates this level of target noise.

Without simulator privileges. The heuristics require only object masks and the gripper state, both of which are available on a real robot. In real-world online RL (W4 (b)), targets generated from SAM segmentations of real rollouts reach 82.1% mean IoU against manually checked annotations. Even with these noisier targets, online contact supervision raises Peg Insertion success from 75% to 85%.

Thank you again for the detailed review, which led to several new experiments on contact specificity (W1), independent localization evaluation (W3), seed robustness, and real-robot attribution (W4). If these address your concerns, we would be grateful if you would consider raising your rating.

References

[1] Zang, H., Yu, S., Lin, H., Zhou, T., Huang, Z., Guo, Z., Xu, X., Zhou, J., Sheng, Y., Zhang, S., Gao, F., Tang, W., Yue, Y., Zhang, Q., Chen, X., Yu, C., & Wang, Y. (2026). RLinf-USER: A unified and extensible system for real-world online policy learning in embodied AI (arXiv:2602.07837). arXiv. https://doi.org/10.48550/arXiv.2602.07837

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response: Summary of Shared Concerns and Revisions

We thank all reviewers for their constructive feedback. CPO-RL co-optimizes a contact-grounded latent with the VLA policy during online RL, using contact targets generated from the policy's own successful rollouts; the supervision is used only in training and adds no cost at deployment. We summarize our responses to the concerns shared across reviewers:

  1. Contact targets and dependence on simulator information (6qAu, qrPP, Ms8f). The target heuristics use only object masks and the gripper state. On the real robot, online contact supervision raises Peg Insertion success from 60% to 75% under sim-real co-training, and from 75% to 85% in real-world online RL with targets generated from SAM segmentations of real rollouts, without any simulator information (RLinf: 70%). We will give a step-by-step, task-by-task description of the mask construction in Appendix A.4 and release the mask-generation code.

  2. Contribution and terminology (6qAu, Ms8f, qrPP). CPO-RL is a framework in which a contact-grounded latent and the policy are co-optimized during online RL, with targets regenerated from the policy's own successful rollouts. This accelerates RL training while refining the representation, and our ablations show that the contact target, the dedicated latent token, and online updates each contribute (see individual responses). We will narrow the "CoT" terminology to a contact-grounded latent CoT and extend the related work with the methods suggested by the reviewers.

Additional experiments requested by individual reviewers are reported in the corresponding responses.

AuthorsOpenReview ↗

Summary of Revisions in the Uploaded PDF

We thank all reviewers for their feedback. We have uploaded a revised PDF; all changes made during the discussion are marked in blue. The main revisions are:

  • Claims and positioning (6qAu, Ms8f, qrPP): "CoT" is narrowed to contact-grounded latent CoT throughout; the abstract and contributions are rewritten to match the ablation evidence; a new Sec. 2.3 positions CPO-RL against contact-aware and training-only auxiliary-supervision methods.
  • New experiments: alternative auxiliary targets and direct contact supervision without the latent token (Table 2b–c); localization against physics-derived references (Table 3); three training seeds (App. B.6); contact-loss weight sensitivity (App. B.7); real-robot attribution, including real-world online RL without simulator information (Sec. 4.4, Table 5).
  • Clarity and reproducibility (qrPP): precise task-by-task mask construction and target-quality statistics (Sec. 3.3, App. A.4); definitions of Contact-SFT, MtM_t/M^t\hat{M}_t, NCD, and sim-real co-training; SFT episode length added to Table 1; the main text stays within the 10-page limit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.