- Rating
- 3
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CPO-RL, which jointly optimizes a contact-grounded latent representation and a VLA policy by combining PPO with contact-mask supervision from successful trajectories. It offers a practical approach for continuously refining an action-conditioned spatial representation during online RL, achieving strong performance in both simulation and real-world manipulation.
Critical strengths
- It introduces a simple and practical way to refine a contact-aware representation together with the policy during online RL.
- The experimental results are clear. It demonstrates consistent improvements across different simulation tasks and real-world experiments, and the ablation study is also clear.
Critical weaknesses
- Novelty is kind of limited. The naming of 'CoT–Policy Co-Optimization' sounds big, but the actual method is more like adding contact supervision and doing it in an online manner, which sounds pretty natural.
- Contact supervision depends on privileged, task-specific heuristics, which may make it hard to scale up.
Questions for authors
Could the authors compare CPO-RL with a simpler baseline that applies the contact mask loss directly to the policy representation, without introducing a dedicated CoT token? This would clarify whether the gain comes from the proposed latent architecture or simply from additional contact supervision.
4 comments
Response to Reviewer Ms8f (2/2): Direct Contact Supervision Baseline (Q1)
Q1. Direct contact supervision without the CoT token.
We implemented this baseline on Peg Insertion. The CoT token is removed; the action-token hidden states from the final action-expert layer (before the action output projection) are mean-pooled and passed to the same adapter and mask decoder, so the contact loss directly updates the action expert. During training, these hidden states are taken at the flow-matching timestep sampled for each example; as in CPO-RL, the decoder is not used at inference. Targets, loss weights, Contact-SFT, and budget are identical to CPO-RL.
| Method | RLinf | Direct contact supervision | CPO-RL |
|---|---|---|---|
| Peg Insertion SR (%) | 59.4 | 72.2 | 95.6 |
With identical targets, loss, Contact-SFT, and budget, applying the supervision directly to action features reaches only 72.2%, barely above the token-only variant without any contact supervision (68.6%, Table 2), whereas applying it through the latent token reaches 95.6%. We attribute this to the latent token decoupling contact supervision from action generation. Action-token hidden states are computed from noisy actions at a randomly sampled noise level and directly produce the action output; a mask loss on them asks these noise-dependent features to also predict a noise-independent spatial target, which competes with the action objective. In CPO-RL, the causal mask prevents the latent token from attending to action tokens, so the token is independent of action noise. The contact loss shapes this separate interface, and the action tokens read from it without their own features being overwritten. We will add this baseline to Sec. 4.3.
Thank you again for the positive assessment. Please let us know if you have any remaining concerns.
References
[1] Zang, H., Yu, S., Lin, H., Zhou, T., Huang, Z., Guo, Z., Xu, X., Zhou, J., Sheng, Y., Zhang, S., Gao, F., Tang, W., Yue, Y., Zhang, Q., Chen, X., Yu, C., & Wang, Y. (2026). RLinf-USER: A unified and extensible system for real-world online policy learning in embodied AI (arXiv:2602.07837). arXiv. https://doi.org/10.48550/arXiv.2602.07837
Thanks the author for the response. My W2 is largely resolved, though I still feel like the novelty is slightly overclaimed. Overall, this is a good paper. I hope it can be accepted, and I'll increase my confidence for my current score.
Thank you for the positive assessment. Following your comment, we have narrowed “CoT” to contact-grounded latent CoT throughout the paper. We have also uploaded the revised PDF, with all changes marked in blue, for your reference. Please let us know if you have any further questions.
Response to Reviewer Ms8f (1/2): Contribution and Privileged Heuristics (W1, W2)
Thank you for the positive assessment and for noting that the method is simple and practical. We address the two weaknesses and the question below.
W1. Novelty.
Our contribution goes beyond adding contact supervision online: CPO-RL is a framework in which a contact-grounded representation and the policy improve together during online RL. We highlight three aspects.
Co-optimization accelerates RL training. Contact targets are regenerated from the policy's own successful rollouts, so the supervision keeps improving as the policy improves. With the same initialization and interaction budget, CPO-RL reaches 89.0% training success on Peg Insertion by iteration 400, already above the final value of w/o CoT-Opt at iteration 600 (81.9%; Table 7), and reaches 90% evaluation success at iteration 450, a level w/o CoT-Opt does not reach within 600 iterations.
Each design component contributes. Under identical settings on Peg Insertion: routing the supervision through a dedicated latent token is essential, since applying the same loss directly to action features reaches only 72.2% versus 95.6% (Q1); contact masks outperform object-segmentation and gripper-trajectory targets (91.7% vs. 82.2% and 85.2%); and offline Contact-SFT and the online contact loss are complementary (84.4% and 91.7% alone, 95.6% together; Table 2).
Transfer to the real robot without simulator ground truth. With targets generated from SAM segmentations of real rollouts, online contact supervision raises real-world Peg Insertion success from 75% to 85% (W2).
These gains require no additional sensors and add no inference cost at deployment. Following your suggestion, we will narrow the naming to "contact-grounded latent CoT" so that it describes the method precisely.
W2. Dependence on privileged, task-specific heuristics.
To quantify whether removing privileged information from mask construction affects our method, we ran online RL directly on the real robot for Peg Insertion using the RLinf-USER infrastructure [1], with auxiliary masks constructed automatically from SAM segmentations of real-world rollouts; these masks reach 82.1% mean IoU against manually checked annotations. All methods use the same base policy, demonstration data, real-world interaction budget, and evaluation protocol (20 trials); w/o CoT-Opt shares the Contact-SFT initialization with CPO-RL and only disables online contact supervision. CPO-RL achieves 85% (17/20), versus 75% (15/20) for w/o CoT-Opt and 70% (14/20) for RLinf, so online contact supervision remains effective without privileged information.
The heuristics share common interaction templates across tasks. Targets are built from two categories of rules, robot–object contact and object–object contact, both of which use only object masks and the gripper state. These two categories cover all four tasks.