- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Decomposed Contrastive Critics (DCC) for sparse-reward continual goal-conditioned RL. The method combines a shared critic representation with a task-specific branch, a success replay buffer for behavior cloning, and an action-sensitive gate to control the strength of imitation. The authors evaluate DCC on ten sequential Meta-World manipulation tasks and report substantial improvements over contrastive RL and sparse SAC baselines.
Critical strengths
- The paper addresses an interesting problem, and the empirical improvements are substantial. The experiments include a fairly comprehensive set of contrastive transfer baselines and component ablations.
- I find the analysis of action-insensitive contrastive critics interesting. In particular, the observation that a critic can learn to predict future states without providing useful action rankings is insightful, and the diagnostic experiments provide some evidence for this failure mode.
Critical weaknesses
- The main experimental results do not sufficiently isolate the contribution of critic decomposition and continual transfer.
- This is my primary concern. From Table 1, a large portion of the improvement therefore appears to come from success-based imitation rather than critic decomposition itself.
- More importantly, the paper does not compare against standard CRL equipped with the same success replay and gated BC. This is a crucial missing baseline, as it is unclear whether the proposed critic decomposition is actually necessary to obtain these gains.
- The results on Task 0 further highlight this concern. Since no transfer has occurred at this point, the substantial improvement from success replay suggests that the method is also addressing a single-task learning problem. I would like to see how much of the improvement remains when DCC is trained from scratch on every task, with all other components unchanged.
- The interpretation of shared and task-specific representations is not well supported.
- The paper motivates critic decomposition as a way to preserve transferable knowledge while maintaining task-specific plasticity. However, both branches are trained using the same contrastive objective, without any explicit mechanism encouraging them to capture different types of information. It is therefore unclear whether the shared branch actually learns reusable goal-reaching knowledge.
- The action-informativeness analysis is interesting, yet the proposed explanation is not fully established.
- The paper argues that certain tasks are difficult because action-informative states are rarely encountered. However, the theoretical discussion concerns the true action-conditioned future-state distribution, whereas the empirical measurements primarily examine the action sensitivity of an already trained critic.
- Also, a critic being insensitive to actions could itself be a consequence of insufficient training or representation learning, rather than evidence that the underlying task contains few action-informative states.
- The critic gate is also somewhat heuristic.
Questions for authors
- How does standard CRL perform when equipped with the same success replay and critic-gated BC, without critic decomposition?
- How much of DCC's improvement remains if the shared critic representation is reinitialized at every task boundary?
- Can the authors report final-checkpoint success rates in addition to the best success rates, to better characterize the stability of learning?
9 comments
Takeaways
-
We agree with your observation in Weakness 1 that, on individual early tasks (T0–T4), equipping a freshly initialized critic with our perturbation gate and success BC can already achieve strong performance.
-
However, the difference becomes particularly pronounced on later tasks, especially T8 and T9. Task 5 and Task 8 share similar physical characteristics: both involve brief-contact Sawyer manipulation, where the robotic arm approaches an articulated mechanism and pushes it toward a target configuration. When is transferred across tasks (Full DCC), the accumulated representation from earlier tasks, including Task 5 (AUC 0.698), can facilitate learning on Task 8, which achieves an AUC of 0.926 ± 0.090 and final success of 88.0 ± 25.7%.
-
In contrast, when is reinitialized at every task boundary (Capacity-Matched CRL), despite using the identical critic gate and success BC, the agent fails to stabilize its performance on Task 8. AUC drops from 0.926 to 0.333 ± 0.194, while final success drops from 88.0% to 16.0 ± 35.0%. Similarly, on Task 9, transferring improves AUC from 0.314 to 0.468 and best success from 72.0% to 89.0%, increasing the overall ten-task final success rate from 72.3% to 80.0%.
-
We would also like to clarify that critic decomposition alone already improves performance over vanilla CRL, even without success replay or critic-gated BC. As shown in Table 1 and Figure 2, Critic-only DCC, which uses neither success replay nor critic-gated BC, improves the ten-task average best success from 78.7% (CRL P/P) and 84.0% (CRL R/R) to 90.4%. The final success results in Appendix Table 7 further show improvements from 43.2% (CRL P/P) and 47.5% (CRL R/R) to 52.3%. We hope these comparisons help clarify the contribution of critic decomposition independently of the additional actor regularization.
Q3
Can the authors report final-checkpoint success rates in addition to the best success rates, to better characterize the stability of learning?
Yes! In addition to the AUC metric, we compiled the Final-Checkpoint Success Rate vs. Best Success Rate across all baselines and ablations.
| Task | CRL R/R (Final / Best %) | CRL P/P (Final / Best %) | Critic-only DCC (Final / Best %) | Ungated DCC (Final / Best %) | Capacity-Matched CRL (Final / Best %) | Full DCC (Final / Best %) |
|---|---|---|---|---|---|---|
| T0 Hammer | 96.7±5.8 / 100±0 | 93.3±11.5 / 100±0 | 96.0±9.7 / 100±0 | 100±0 / 100±0 | 100±0 / 100±0 | 100±0 / 100±0 |
| T1 Push wall | 62.0±27.8 / 98.0±4.5 | 86.0±16.7 / 100±0 | 49.0±38.1 / 97.0±6.7 | 83.0±31.3 / 97.0±6.7 | 89.0±19.1 / 98.0±6.3 | 86.0±30.6 / 100±0 |
| T2 Faucet close | 34.0±20.7 / 88.0±8.4 | 54.0±27.0 / 90.0±10.0 | 42.0±19.3 / 94.0±7.0 | 66.0±21.7 / 97.0±4.8 | 54.0±15.8 / 95.0±5.3 | 55.0±16.5 / 100±0 |
| T3 Push back | 70.0±36.7 / 96.0±8.9 | 38.0±47.1 / 78.0±26.8 | 89.0±19.1 / 100±0 | 95.0±15.8 / 100±0 | 99.0±3.2 / 100±0 | 98.0±4.2 / 100±0 |
| T4 Stick pull | 64.0±35.1 / 98.0±4.5 | 76.0±33.6 / 100±0 | 40.0±38.9 / 99.0±3.2 | 87.0±31.3 / 97.0±6.7 | 88.0±31.6 / 91.0±28.5 | 82.0±33.6 / 90.0±31.6 |
| T5 Handle press | 0.0±0.0 / 24.0±8.9 | 0.0±0.0 / 20.0±12.2 | 24.0±23.2 / 64.0±19.0 | 78.0±29.4 / 99.0±3.2 | 41.0±50.9 / 100±0 | 42.0±44.4 / 100±0 |
| T6 Push | 80.0±20.0 / 96.7±5.8 | 56.7±37.9 / 96.7±5.8 | 68.0±22.0 / 97.0±4.8 | 90.0±17.0 / 99.0±3.2 | 95.0±10.8 / 100±0 | 96.0±5.2 / 100±0 |
| T7 Shelf place | 42.0±32.7 / 90.0±10.0 | 12.0±16.4 / 92.0±8.4 | 61.0±24.7 / 97.0±4.8 | 23.0±20.6 / 43.0±20.0 | 93.0±18.9 / 99.0±3.2 | 98.0±4.2 / 100±0 |
| T8 Window close | 3.3±5.8 / 36.7±5.8 | 0.0±0.0 / 36.7±15.3 | 18.0±11.4 / 75.0±17.8 | 89.0±31.4 / 100±0 | 16.0±35.0 / 100±0 | 88.0±25.7 / 100±0 |
| T9 Peg unplug | 23.3±5.8 / 76.7±11.5 | 20.0±10.0 / 63.3±5.8 | 36.0±18.4 / 73.0±12.5 | 32.0±22.0 / 54.0±35.7 | 48.0±10.3 / 72.0±11.4 | 55.0±20.1 / 89.0±8.7 |
| Average (%) | 47.5±1.9 / 84.0 | 43.2±1.5 / 78.7 | 52.3±2.5 / 90.4 | 74.3±1.8 / 89.6 | 72.3±5.3 / 95.5 | 80.0±3.1 / 97.9 |
You raised a very fair point in your comment! Could low critic action sensitivity simply reflect insufficient training rather than a property of the task?
Looking at Critic-only DCC on T5 (64.0% best to 24.0% final) and T8 (75.0% best to 18.0% final), notice that the agent already learns to solve the task mid-training (reaching 64%–75% success!), and its critic action sensitivity collapses after successful episodes fill the replay buffer. This happens because of a structural property of Sawyer T5 (handle-press-side) and T8 (window-close): once the handle is pressed down or the window is pushed shut, the target object sits at its mechanical stop for the rest of the episode while the arm continues moving freely. Thus, post-contact states in the buffer satisfy for all actions , driving the true action advantage to 0.
We have also added this full Final-Checkpoint table (Table 7 in Appendix) to the revised manuscript.
We appreciate the reviewer's thoughtful questions and constructive suggestions! We hope the newly incorporated experiments meet your expectations! If so, we would be grateful for a re-evaluation of your score.
We sincerely thank Reviewer Zxy5 for your thorough review and insightful experimental questions! We also greatly appreciate your positive remarks on the interesting theoretical results, empirical improvements, and our analysis of action sensitivity in contrastive critics.
Your suggestions motivated us to conduct additional controlled experiments and examine final-checkpoint performance to better isolate the contributions of critic decomposition and cross-task knowledge transfer.
Q1 & Q2
We address both questions through the following controlled experiment. Specifically, we evaluate Capacity-Matched CRL, which reinitializes ϕshared at every task boundary while retaining the exact same two-branch architecture, parameter count (ϕshared of width 1024 and ϕtask of width 256), critic perturbation gate, and success BC as Full DCC. The only difference is that no critic knowledge is transferred across tasks. Thus, Capacity-Matched CRL serves as a capacity-matched control for CRL + Gate + BC without continual critic transfer.
The following table reports the results across all ten tasks and ten random seeds.
Table A. Capacity-Matched CRL vs. Full DCC (10 Seeds, Mean ± SD)