acceptodds
Under review as a conference paper at ICLR 2027

Decomposed Contrastive Critics for Continual Reinforcement Learning

Abstract

Continual reinforcement learning is commonly studied with dense, task-specific rewards, although many control problems are more naturally specified by goals and sparse feedback on task completion. We study this setting on a sequence of goal-conditioned manipulation tasks and ask how contrastive reinforcement learning can transfer knowledge across task boundaries. Because contrastive critics must capture both reusable goal-reaching structure and task-specific action preferences, transfer requires the former to persist across tasks while the latter remains adaptable. We introduce Decomposed Contrastive Critics (DCC), an algorithm that decomposes the critic into a persistent shared state-action representation and a task-specific representation re-initialized at each task boundary, coupled with a success replay buffer and an action-sensitive critic gate that deploys targeted policy supervision when the critic cannot differentiate actions. On the ten-task Continual World V2 Sawyer sequence, we compare against nine contrastive transfer baselines and two sparse goal-conditioned SAC baselines. DCC improves both continual adaptation and task success by a substantial margin, successfully solving all ten manipulation tasks in the curriculum.

Primary area:
reinforcement learning
Keywords:
continual reinforcement learning, contrastive reinforcement learning, goal-conditioned reinforcement learning, sparse rewards, transfer learning, representation learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_Zxy5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Decomposed Contrastive Critics (DCC) for sparse-reward continual goal-conditioned RL. The method combines a shared critic representation with a task-specific branch, a success replay buffer for behavior cloning, and an action-sensitive gate to control the strength of imitation. The authors evaluate DCC on ten sequential Meta-World manipulation tasks and report substantial improvements over contrastive RL and sparse SAC baselines.

Critical strengths
  • The paper addresses an interesting problem, and the empirical improvements are substantial. The experiments include a fairly comprehensive set of contrastive transfer baselines and component ablations.
  • I find the analysis of action-insensitive contrastive critics interesting. In particular, the observation that a critic can learn to predict future states without providing useful action rankings is insightful, and the diagnostic experiments provide some evidence for this failure mode.
Critical weaknesses
  1. The main experimental results do not sufficiently isolate the contribution of critic decomposition and continual transfer.
    • This is my primary concern. From Table 1, a large portion of the improvement therefore appears to come from success-based imitation rather than critic decomposition itself.
    • More importantly, the paper does not compare against standard CRL equipped with the same success replay and gated BC. This is a crucial missing baseline, as it is unclear whether the proposed critic decomposition is actually necessary to obtain these gains.
    • The results on Task 0 further highlight this concern. Since no transfer has occurred at this point, the substantial improvement from success replay suggests that the method is also addressing a single-task learning problem. I would like to see how much of the improvement remains when DCC is trained from scratch on every task, with all other components unchanged.
  2. The interpretation of shared and task-specific representations is not well supported.
    • The paper motivates critic decomposition as a way to preserve transferable knowledge while maintaining task-specific plasticity. However, both branches are trained using the same contrastive objective, without any explicit mechanism encouraging them to capture different types of information. It is therefore unclear whether the shared branch actually learns reusable goal-reaching knowledge.
  3. The action-informativeness analysis is interesting, yet the proposed explanation is not fully established.
    • The paper argues that certain tasks are difficult because action-informative states are rarely encountered. However, the theoretical discussion concerns the true action-conditioned future-state distribution, whereas the empirical measurements primarily examine the action sensitivity of an already trained critic.
    • Also, a critic being insensitive to actions could itself be a consequence of insufficient training or representation learning, rather than evidence that the underlying task contains few action-informative states.
    • The critic gate is also somewhat heuristic.
Questions for authors
  1. How does standard CRL perform when equipped with the same success replay and critic-gated BC, without critic decomposition?
  2. How much of DCC's improvement remains if the shared critic representation is reinitialized at every task boundary?
  3. Can the authors report final-checkpoint success rates in addition to the best success rates, to better characterize the stability of learning?
9 comments

We sincerely thank Reviewer Zxy5 for your thorough review and insightful experimental questions! We also greatly appreciate your positive remarks on the interesting theoretical results, empirical improvements, and our analysis of action sensitivity in contrastive critics.

Your suggestions motivated us to conduct additional controlled experiments and examine final-checkpoint performance to better isolate the contributions of critic decomposition and cross-task knowledge transfer.

Q1 & Q2

  1. How does standard CRL perform when equipped with the same success replay and critic-gated BC, without critic decomposition?
  2. How much of DCC's improvement remains if the shared critic representation is reinitialized at every task boundary?

We address both questions through the following controlled experiment. Specifically, we evaluate Capacity-Matched CRL, which reinitializes ϕshared\phi_{\text{shared}} at every task boundary while retaining the exact same two-branch architecture, parameter count (ϕshared\phi_{\text{shared}} of width 1024 and ϕtask\phi_{\text{task}} of width 256), critic perturbation gate, and success BC as Full DCC. The only difference is that no critic knowledge is transferred across tasks. Thus, Capacity-Matched CRL serves as a capacity-matched control for CRL + Gate + BC without continual critic transfer.

The following table reports the results across all ten tasks and ten random seeds.

Table A. Capacity-Matched CRL vs. Full DCC (10 Seeds, Mean ± SD)

TaskCapacity-Matched CRL AUCFull DCC AUCCapacity-Matched CRL BestFull DCC BestCapacity-Matched CRL FinalFull DCC Final
T0 Hammer0.981 ± 0.0320.994 ± 0.0121.000 ± 0.0001.000 ± 0.0001.000 ± 0.0001.000 ± 0.000
T1 Push wall0.740 ± 0.1930.712 ± 0.0820.980 ± 0.0631.000 ± 0.0000.890 ± 0.1910.860 ± 0.306
T2 Faucet close0.645 ± 0.0350.648 ± 0.0480.950 ± 0.0531.000 ± 0.0000.540 ± 0.1580.550 ± 0.165
T3 Push back0.919 ± 0.0410.874 ± 0.0701.000 ± 0.0001.000 ± 0.0000.990 ± 0.0320.980 ± 0.042
T4 Stick pull0.719 ± 0.2690.616 ± 0.2540.910 ± 0.2850.900 ± 0.3160.880 ± 0.3160.820 ± 0.336
T5 Handle press0.601 ± 0.2910.698 ± 0.1621.000 ± 0.0001.000 ± 0.0000.410 ± 0.5090.420 ± 0.444
T6 Push0.864 ± 0.1040.671 ± 0.2701.000 ± 0.0001.000 ± 0.0000.950 ± 0.1080.960 ± 0.052
T7 Shelf place0.818 ± 0.1480.608 ± 0.1390.990 ± 0.0321.000 ± 0.0000.930 ± 0.1890.980 ± 0.042
T8 Window close0.333 ± 0.1940.926 ± 0.0901.000 ± 0.0001.000 ± 0.0000.160 ± 0.3500.880 ± 0.257
T9 Peg unplug0.314 ± 0.0610.468 ± 0.0680.720 ± 0.1140.890 ± 0.0870.480 ± 0.1030.550 ± 0.201
Average0.693 ± 0.0450.722 ± 0.0710.955 ± 0.0300.979 ± 0.0450.723 ± 0.0530.800 ± 0.031

Takeaways

  • We agree with your observation in Weakness 1 that, on individual early tasks (T0–T4), equipping a freshly initialized critic with our perturbation gate and success BC can already achieve strong performance.

  • However, the difference becomes particularly pronounced on later tasks, especially T8 and T9. Task 5 and Task 8 share similar physical characteristics: both involve brief-contact Sawyer manipulation, where the robotic arm approaches an articulated mechanism and pushes it toward a target configuration. When ϕshared\phi_{\text{shared}} is transferred across tasks (Full DCC), the accumulated representation from earlier tasks, including Task 5 (AUC 0.698), can facilitate learning on Task 8, which achieves an AUC of 0.926 ± 0.090 and final success of 88.0 ± 25.7%.

  • In contrast, when ϕshared\phi_{\text{shared}} is reinitialized at every task boundary (Capacity-Matched CRL), despite using the identical critic gate and success BC, the agent fails to stabilize its performance on Task 8. AUC drops from 0.926 to 0.333 ± 0.194, while final success drops from 88.0% to 16.0 ± 35.0%. Similarly, on Task 9, transferring ϕshared\phi_{\text{shared}} improves AUC from 0.314 to 0.468 and best success from 72.0% to 89.0%, increasing the overall ten-task final success rate from 72.3% to 80.0%.

  • We would also like to clarify that critic decomposition alone already improves performance over vanilla CRL, even without success replay or critic-gated BC. As shown in Table 1 and Figure 2, Critic-only DCC, which uses neither success replay nor critic-gated BC, improves the ten-task average best success from 78.7% (CRL P/P) and 84.0% (CRL R/R) to 90.4%. The final success results in Appendix Table 7 further show improvements from 43.2% (CRL P/P) and 47.5% (CRL R/R) to 52.3%. We hope these comparisons help clarify the contribution of critic decomposition independently of the additional actor regularization.

Q3

Can the authors report final-checkpoint success rates in addition to the best success rates, to better characterize the stability of learning?

Yes! In addition to the AUC metric, we compiled the Final-Checkpoint Success Rate vs. Best Success Rate across all baselines and ablations.

TaskCRL R/R (Final / Best %)CRL P/P (Final / Best %)Critic-only DCC (Final / Best %)Ungated DCC (Final / Best %)Capacity-Matched CRL (Final / Best %)Full DCC (Final / Best %)
T0 Hammer96.7±5.8 / 100±093.3±11.5 / 100±096.0±9.7 / 100±0100±0 / 100±0100±0 / 100±0100±0 / 100±0
T1 Push wall62.0±27.8 / 98.0±4.586.0±16.7 / 100±049.0±38.1 / 97.0±6.783.0±31.3 / 97.0±6.789.0±19.1 / 98.0±6.386.0±30.6 / 100±0
T2 Faucet close34.0±20.7 / 88.0±8.454.0±27.0 / 90.0±10.042.0±19.3 / 94.0±7.066.0±21.7 / 97.0±4.854.0±15.8 / 95.0±5.355.0±16.5 / 100±0
T3 Push back70.0±36.7 / 96.0±8.938.0±47.1 / 78.0±26.889.0±19.1 / 100±095.0±15.8 / 100±099.0±3.2 / 100±098.0±4.2 / 100±0
T4 Stick pull64.0±35.1 / 98.0±4.576.0±33.6 / 100±040.0±38.9 / 99.0±3.287.0±31.3 / 97.0±6.788.0±31.6 / 91.0±28.582.0±33.6 / 90.0±31.6
T5 Handle press0.0±0.0 / 24.0±8.90.0±0.0 / 20.0±12.224.0±23.2 / 64.0±19.078.0±29.4 / 99.0±3.241.0±50.9 / 100±042.0±44.4 / 100±0
T6 Push80.0±20.0 / 96.7±5.856.7±37.9 / 96.7±5.868.0±22.0 / 97.0±4.890.0±17.0 / 99.0±3.295.0±10.8 / 100±096.0±5.2 / 100±0
T7 Shelf place42.0±32.7 / 90.0±10.012.0±16.4 / 92.0±8.461.0±24.7 / 97.0±4.823.0±20.6 / 43.0±20.093.0±18.9 / 99.0±3.298.0±4.2 / 100±0
T8 Window close3.3±5.8 / 36.7±5.80.0±0.0 / 36.7±15.318.0±11.4 / 75.0±17.889.0±31.4 / 100±016.0±35.0 / 100±088.0±25.7 / 100±0
T9 Peg unplug23.3±5.8 / 76.7±11.520.0±10.0 / 63.3±5.836.0±18.4 / 73.0±12.532.0±22.0 / 54.0±35.748.0±10.3 / 72.0±11.455.0±20.1 / 89.0±8.7
Average (%)47.5±1.9 / 84.043.2±1.5 / 78.752.3±2.5 / 90.474.3±1.8 / 89.672.3±5.3 / 95.580.0±3.1 / 97.9

You raised a very fair point in your comment! Could low critic action sensitivity Δa(s,g)\Delta_a(s,g) simply reflect insufficient training rather than a property of the task?

Looking at Critic-only DCC on T5 (64.0% best to 24.0% final) and T8 (75.0% best to 18.0% final), notice that the agent already learns to solve the task mid-training (reaching 64%–75% success!), and its critic action sensitivity collapses after successful episodes fill the replay buffer. This happens because of a structural property of Sawyer T5 (handle-press-side) and T8 (window-close): once the handle is pressed down or the window is pushed shut, the target object sits at its mechanical stop for the rest of the episode while the arm continues moving freely. Thus, post-contact states in the buffer satisfy p(g∣s,a)=p(g∣s)p(g \mid s,a) = p(g \mid s) for all actions aa, driving the true action advantage log⁡p(g∣s,a)p(g∣s)\log \frac{p(g\mid s,a)}{p(g\mid s)} to 0.

We have also added this full Final-Checkpoint table (Table 7 in Appendix) to the revised manuscript.


We appreciate the reviewer's thoughtful questions and constructive suggestions! We hope the newly incorporated experiments meet your expectations! If so, we would be grateful for a re-evaluation of your score.

Reviewer_phSCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the problem of Continual Reinforcement Learning (CRL) in a goal-conditioned setting. The authors propose decomposing the representation learned by a goal-conditioned contrastive learning critic into two components: a global representation shared across tasks and a task-specific residual. In addition, the paper introduces a regularization strategy for the actor based on behavioral cloning using a separate dataset containing only successful trajectories. This success dataset is reset whenever the task changes.

Critical strengths
  • The paper presents a simple modification to CRL algorithms that improves their performance in the continuous reinforcement learning setting considered.

  • The empirical evaluation is extensive, and the reported results demonstrate a clear improvement over both the CRL and SAC baselines.

Critical weaknesses
  • I think the authors should include a dedicated limitations section discussing in more detail how the success of solving individual tasks affects their approach. In particular, if a task remains difficult and the algorithm is unable to successfully solve it, the success dataset will never be populated, which could significantly limit the effectiveness of the proposed actor regularization. The authors should also discuss the limitations of their approach for estimating the gating mechanism. As currently presented, the method appears to rely on state-based observations where the state can be explicitly perturbed. It is unclear whether the same approach would extend to high-dimensional observation spaces, such as pixel-based observations, and I would encourage the authors to discuss this limitation explicitly.

  • I think the paper would benefit from a more formal characterization of what each component of the critic, the global and task-specific components, is expected to capture or converge to. Providing such a formal definition would help clarify the motivation for the proposed decomposition and make the underlying assumptions more explicit. In particular, it would be useful to establish why this decomposition is desirable and under what conditions the global component should capture task-invariant information while the residual captures task-specific information.

  • The empirical evaluation seems to focus on a relatively simplified setting in which the robot/arm dynamics remain fixed, while changes in the transition dynamics arise primarily from changes in the objects involved in each task, potentially through differences in contact and friction dynamics. This raises questions about how broadly the proposed decomposition applies to more substantial changes in the underlying dynamics. Related to the previous point, I think the paper would benefit from explicitly stating what type of decomposition the authors expect between the global and task-specific components, as well as identifying the settings in which this decomposition is expected to be useful and those in which it may fail. Additional experiments involving more diverse or substantially different task dynamics would help substantiate these claims.

Questions for authors
  • Can you discuss the limitations of the perturbation-based gating mechanism, particularly how it could be extended to high-dimensional or pixel-based observations?

  • Can you clarify how task difficulty and success rate affect the behavioral cloning regularization, especially when few or no successful trajectories are collected?

  • Can you formally characterize what the global and task-specific representations are expected to capture or converge to?

  • Can you formally characterize how transition probabilities are expected to change across tasks in the experiments and what types of transition probability changes we expect the proposed method to support in relation to the decomposition between global and task critic?

5 comments

We sincerely thank Reviewer phSc for your thoughtful and constructive review, and especially appreciate your helpful suggestion to include a dedicated limitations section!

Your insightful questions have greatly helped us sharpen both our theoretical exposition and our discussion of practical limitations. Below, we provide answers to your questions.

Q1.

Can you discuss the limitations of the perturbation-based gating mechanism, particularly how it could be extended to high-dimensional or pixel-based observations?

This is a sharp observation! We completely agree with this limitation and have added a Limitation section in the revised paper discussing it explicitly.

  • Our critic gate relies on raw state-space perturbations to estimate local sensitivity and is thus best suited to low-dimensional observations. In pixel-based or high-dimensional settings, adding isotropic Gaussian noise directly to image pixels produces out-of-distribution visual artifacts rather than meaningful local state transitions.

  • To extend the critic gate (Eq. 7) to visual GCRL, the state encoder can first map image observations oo into a compact latent representation hs=fenc(o)h_s = f_{\text{enc}}(o) before fusing with actions aa. Local state sensitivity σstate(s,g)\sigma_{\text{state}}(s,g) can then be estimated by perturbing hsh_s within the normalized latent manifold (hs(j)=hs+εσ^h⊙ξ(j)h_s^{(j)} = h_s + \varepsilon \hat{\sigma}_h \odot \xi^{(j)}), or by measuring critic variation across temporally adjacent states (st−1,st,st+1)(s_{t-1}, s_t, s_{t+1}) along the trajectory in Dsucc\mathcal{D}_{\text{succ}}.

Q2.

Can you clarify how task difficulty and success rate affect the behavioral cloning regularization, especially when few or no successful trajectories are collected?

This is a very valid concern. Thank you for raising this point!

  • Indeed, before the first success is collected, Dsucc\mathcal{D}_{\mathrm{succ}} remains empty and the regularizer contributes no update, so the actor is trained exclusively with the standard contrastive objective. If the policy fails to discover any successful trajectory throughout training, even a suboptimal one, the agent may ultimately fail to solve the task. We acknowledge this limitation of the proposed approach and discuss it in the revised manuscript.

  • Once successful trajectories become available, the regularizer is activated. Its contribution is controlled by λsucc=0.1\lambda_{\mathrm{succ}}=0.1 and the critic gate w(s,g)w(s,g), rather than directly by the task success rate. Even with abundant successful trajectories, the regularizer can remain weak when the critic exhibits strong relative action discrimination. Conversely, when only a few successful trajectories are available and the critic provides limited action discrimination, BC may dominate the actor updates despite the small regularization coefficient.

  • Importantly, task difficulty in contrastive RL is not necessarily determined by the complexity of the required behavior. As discussed in Section 5, tasks involving sustained interaction may provide abundant action-informative states, whereas seemingly simple tasks requiring a brief decisive action can be more difficult for the contrastive critic. Success replay is particularly useful in the latter setting, where the critic may struggle to distinguish the action(s) responsible for task completion.

  • There's an interesting interplay between actor and critics. CRL has been shown to exhibit emergent exploration even before discovering any successful trajectories [1]. Its contrastive objective learns from achieved future states through HER, allowing the agent to continue exploring without relying on task success rewards. Consequently, DCC can leverage CRL's exploratory behavior to discover initial successes.

  • Once CRL discovers its first successful trajectory through exploration, this trajectory may still contain suboptimal actions. Nevertheless, it provides an example of behavior that achieves the commanded goal. The BC regularizer encourages the actor to reproduce this behavior, shifting its policy toward successful trajectories. As the updated policy generates new rollouts, it also changes the replay distribution used for contrastive critic training.

[1] M. Bastankhah, G. Liu, D. Arumugam, T. L. Griffiths, and B. Eysenbach. Demystifying the Mechanisms Behind Emergent Exploration in Goal-Conditioned RL. ICLR, 2026.

Q3. W2.

Can you formally characterize what the global and task-specific representations are expected to capture or converge to?

Thank you for raising this precious question, which helped us clarify our motivation and greatly improve the manuscript (see the revised Section 3.1 and Appendix G)!

In a continual learning setting where tasks and replay distributions shift sequentially, the representations are continually adapted rather than converging to a single static fixed point; however, we can formally characterize what the two branches are expected to capture from two complementary perspectives:

  • State-to-Future Reachability vs. Action-Conditional Advantage (Section 3.1, Eq. 7 & Appendix G.1):

    At population optimality, the InfoNCE critic approximates the log density ratio of future state occupancy (up to a goal-independent constant). Through the mutual-information chain rule I((S,A);F)=I(S;F)+I(A;F∣S)I((S,A); F) = I(S; F) + I(A; F \mid S), this ideal critic decomposes into two terms:

    f∗(s,a,g)≈log⁡p(k)(g∣s)p(k)(g)+log⁡p(k)(g∣s,a)p(k)(g∣s)f^*(s,a,g) \approx \log \frac{p^{(k)}(g \mid s)}{p^{(k)}(g)} + \log \frac{p^{(k)}(g \mid s,a)}{p^{(k)}(g \mid s)}

    The first term captures state-to-future reachability (I(S;F)I(S; F)), while the second captures the action-conditional advantage (I(A;F∣S)I(A; F \mid S)). In our continual setting, all tasks share the same robot kinematics and underlying simulator physics. Even though different tasks may attend to different subsets of features, evaluating state-to-future reachability relies on common physical features that can be reused and accumulated across the task chain, which is what the persistent shared branch (ϕshared\phi_{\mathrm{shared}}) is expected to capture. By contrast, the action-conditional advantage depends on the specific action and policy tailored to the current object and goal, which is what the re-initialized task-specific branch (ϕtask(k)\phi_{\mathrm{task}}^{(k)}) captures.

  • Shared Predictive Compression vs. Task-Specific Reweighting (Appendix G.2):

    A second way to characterize the two branches at any learning step is through how they form predictions over candidate future goals. As the shared branch ϕshared\phi_{\mathrm{shared}} is carried across tasks, it continually compresses past experience to provide a baseline prediction Pshared(g∣s,a)∝exp⁡(S(s,a,g))P_{\mathrm{shared}}(g \mid s,a) \propto \exp(S(s,a,g)) for the next task. Because this retained prediction will not perfectly match the new task's target distribution Pk∗(g∣s,a)P_k^*(g \mid s,a), adding the task-specific score Rk(s,a,g)=ϕtask(k)(s,a)⊤ψ(g)/τR_k(s,a,g) = \phi_{\mathrm{task}}^{(k)}(s,a)^\top \psi(g)/\tau multiplicatively reweights the shared prediction:

    Pk(g∣s,a)∝Pshared(g∣s,a),exp⁡!(Rk(s,a,g))P_k(g \mid s,a) \propto P_{\mathrm{shared}}(g \mid s,a)\\,\exp\\!\bigl(R_k(s,a,g)\bigr)

    Matching the new task's ideal contrastive distribution Pk∗(g∣s,a)P_k^*(g \mid s,a) therefore requires the task branch to capture the log-ratio residual Rk∗(s,a,g)=log⁡Pk∗(g∣s,a)Pshared(g∣s,a)R_k^*(s,a,g) = \log \frac{P_k^*(g \mid s,a)}{P_{\mathrm{shared}}(g \mid s,a)} (up to a goal-independent offset, Eq. 28 in Appendix G.2), raising the scores of underpredicted futures and revising action preferences for the current task.

  • On convergence. Since InfoNCE optimizes the combined critic score, the two branches are encouraged to learn their intended representational roles without being constrained to converge to uniquely defined functions. Moreover, state-to-future reachability itself evolves across tasks as the agent encounters different objects, goals, and state distributions, even when the underlying physics remains shared. We therefore do not expect the shared branch to converge to a fixed reachability function. Rather, we expect it to continually accumulate and refine features that remain useful for predicting future-state reachability across tasks, while the task-specific branch adapts to the action-dependent distinctions required by the current task.

Reviewer_DtLbOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the "Decomposed Contrastive Critics" (DCC) framework, designed to address the challenge of continual reinforcement learning under sparse rewards and to enhance knowledge transfer and adaptation across a sequential series of goal-conditioned tasks. DCC comprises three core components: (1) a decomposed contrastive critic that integrates persistent, shared representations with re-initialized, task-specific representations; (2) a success-based experience replay buffer that facilitates the imitation of successful trajectories; and (3) an action-sensitive critic gating mechanism, which selectively incorporates behavioral cloning when the critic fails to effectively discriminate between actions. Furthermore, the paper provides an information-theoretic analysis, highlighting the critical importance of "state representations that incorporate action information" for contrastive reinforcement learning. Experimental results across ten Sawyer robot manipulation tasks demonstrate that DCC significantly outperforms existing baselines, achieving a normalized AUC of 0.722 and an average peak success rate of 97.9%, thereby confirming its effectiveness in continual learning scenarios characterized by sparse rewards.

Critical strengths
  1. The paper addresses a meaningful limitation of conventional continual RL, which has predominantly been studied with dense task-specific rewards.

  2. This provides an insightful explanation of contrastive RL failure modes and highlights an important distinction between predictive representation quality and usefulness for policy improvement.

  3. The improvements of the proposed DCC method are substantial and consistent across the overall curriculum, making the empirical contribution convincing within the evaluated setting.

Critical weaknesses
  1. The novelty of this paper is limited. The proposed method is almost like a combination of several known techniques. First, shared and task-specific representations have been extensively explored in previous papers on meta and continual RL, e.g. [1-3].

[1] Meta reinforcement learning with task embedding and shared policy. IJCAI 2019 [2] Learning Shared Representations for Value Functions in Multi-task Reinforcement Learning. arXiv 2016 [3] Multi-Task Reinforcement Learning with Shared-Unique Features and Task-Aware Prioritized Experience Replay. 2024

Second, successful trajectory replay and behavior cloning are also used in previous methods, such as self-imitation learning [4, 5].

[4] Self-imitation learning. ICML 2018 [5] Phasic self-imitative reduction for sparsereward goal-conditioned reinforcement learning. ICML 2022

Third, selective behavior cloning has previously been investigated by previous work [6], using critic-based advantage filtering.

[6] Overcoming exploration in reinforcement learning with demonstrations. ICRA 2026

The primary methodological distinction is the action-sensitive critic gate. However, its novelty is not enough for a paper published in major conference like ICLR.

  1. The writing of this paper needs to be improved. It is not pedagogical engough. Authors should re-organize the presentation and add more introductory and preliminary materials. For example, Section 3.1 should not be in the section of methodology. More introduction on the GCRL should be added before Section 3.

  2. Authors directly add the outputs of shared and task networks together, which is heuristic. However, the reason of using addition, not other operations, is not demonstrated enough. Authors should discuss the reason of selecting addition in this part.

  3. Although the paper is framed as continual RL, its experiments primarily measure how efficiently the agent learns each new task. Actually, authors do not sufficiently evaluate whether the agent can still solve earlier tasks after learning subsequent ones.

  4. All experiments use ten simulated Sawyer manipulation tasks from a single domain, i.e. Continual World V2. These environments share relatively similar observation and action spaces. The proposed DCC method is not evaluated on more diverse continual RL problems, such as navigation, locomotion, or visual control.

Questions for authors
  1. Why not to use other operations, such as a neural network, to integrate the outputs of shared and task networks in (4)? Can authors demonstrate it empirically?

  2. How effectively does the shared representation preserve earlier knowledge, and does DCC still suffer from catastrophic forgetting? Can authors provide more empirical evidence for that?

  3. How much of DCC's improvement comes from separating shared and task-specific representations rather than simply increasing network capacity or resetting part of the critic? Can the authors provide comparisons with larger conventional critics which have similar amount of parameters, non or partial-reset methods, or some other alternative architectures?

12 comments

We sincerely thank Reviewer DtLb for your constructive critiques and detailed suggestions on both the methodology and presentation. Regardless of the outcome of the rebuttal process, your suggestions have already helped us improve the paper considerably!

We have added a preliminaries section, revised the methodology section with an additional figure (Figure 1) in an attempt to improve the presentation, substantially reorganized the experimental sections, and explicitly discussed the limitations of our paper. We have also included all the relevant references in the related work section. Below, we address the questions you raised.

Why not to use other operations, such as a neural network, to integrate the outputs of shared and task networks in (4)? Can authors demonstrate it empirically?

Thank you for raising this question! Given the short rebuttal window and the approximately 48 GPU hours required per seed, we have not yet completed this comparison. We would like to explain the motivation for our additive architecture and provide an interpretation of this choice.

1. Why not use a neural network?

Both ϕshared(s,a)\phi_{\text{shared}}(s,a) and ϕtask(k)(s,a)\phi_{\text{task}}^{(k)}(s,a) are already 4-layer nonlinear neural networks with linear output layers. If we concatenate their outputs and apply a linear projection W=[Ws,Wt]W = [W_s, W_t], we obtain:

W[ϕshared(s,a)ϕtask(k)(s,a)]=Wsϕshared(s,a)+Wtϕtask(k)(s,a).\displaystyle W\begin{bmatrix}\phi_{\text{shared}}(s,a) \\\\ \phi_{\text{task}}^{(k)}(s,a)\end{bmatrix} = W_s\phi_{\text{shared}}(s,a) + W_t\phi_{\text{task}}^{(k)}(s,a).

Because WsW_s and WtW_t can be absorbed directly into the final linear layers of ϕshared\phi_{\text{shared}} and ϕtask(k)\phi_{\text{task}}^{(k)}, concatenation followed by a linear fusion layer has the exact same representational capacity as addition. Furthermore, if one instead placed a nonlinear MLP on top of [ϕshared;ϕtask(k)][\phi_{\text{shared}};\phi_{\text{task}}^{(k)}], randomly reinitializing ϕtask(k)\phi_{\text{task}}^{(k)} at each new task boundary would corrupt the nonlinear hidden activations of the fusion network, destroying the transferred signal from ϕshared\phi_{\text{shared}} at step 0 of every new task.

2. Variational Interpretation of Additive Scores

The inner-product critic f(k)(s,a,g)=bshared(s,a,g)+rtask(k)(s,a,g)f^{(k)}(s,a,g) = b_{\text{shared}}(s,a,g) + r_{\text{task}}^{(k)}(s,a,g) defines an unnormalized log-probability over candidate goals gg. Adding the shared and task scores yields a product of experts:

p(k)(g∣s,a)∝exp⁡(bshared(s,a,g)+rtask(k)(s,a,g))\displaystyle p^{(k)}(g \mid s,a) \propto \exp\left(b_{\text{shared}}(s,a,g) + r_{\text{task}}^{(k)}(s,a,g)\right)

∝pshared(g∣s,a),exp⁡(rtask(k)(s,a,g)).\displaystyle \propto p_{\text{shared}}(g \mid s,a)\\,\exp\left(r_{\text{task}}^{(k)}(s,a,g)\right).

This has a clean variational interpretation as the exact solution to:

\displaystyle p^{(k)}(\cdot \mid s,a) = \arg\max_q \left\\{ \mathbb{E}_q[r_{\text{task}}^{(k)}(s,a,g)] - \mathrm{KL}\left(q(\cdot \mid s,a) \Vert p_{\text{shared}}(\cdot \mid s,a)\right) \right\\}.

In intuitive terms, addition allows the task branch to learn a residual correction that tilts the shared reference distribution toward the current task's goals, with deviation naturally penalized through KL divergence. We have also provided additional motivation for the decomposition in Appendix G.

Q2

How effectively does the shared representation preserve earlier knowledge, and does DCC still suffer from catastrophic forgetting? Can authors provide more empirical evidence for that?

Thank you for raising this point!

  • As noted in our Introduction, this paper focuses primarily on forward transfer and plasticity under sparse rewards. Because DCC deliberately resets the actor π(k)\pi^{(k)} and the task-specific critic branch ϕtask(k)\phi_{\text{task}}^{(k)} at each task boundary (which our experiments in Table 1 show is essential to avoid negative actor interference), DCC does not retain executable policies for past tasks without replay or distillation, and therefore still suffers from policy-level forgetting on previous tasks if evaluated zero-shot after task KK. We have made this scope boundary explicit in the revised Conclusion section, and we acknowledge that this could be seen as one of the limitations of our paper.

  • Empirical Evidence That ϕshared\phi_{\text{shared}} Preserves Earlier Knowledge for Forward Transfer: While we do not claim backward retention, our new controlled experiment in Q3 below (Table A) provides direct empirical proof that ϕshared\phi_{\text{shared}} preserves earlier physical knowledge to accelerate later tasks: when ϕshared\phi_{\text{shared}} is carried forward from earlier tasks (especially from Task 5 handle-press-side), the agent learns and stabilizes Task 8 (window-close) with 0.926 AUC and 88.0% final success, whereas resetting ϕshared\phi_{\text{shared}} at every task boundary causes Task 8 to collapse to 0.333 AUC and 16.0% final success.

Q3

How much of DCC's improvement comes from separating shared and task-specific representations rather than simply increasing network capacity or resetting part of the critic? Can the authors provide comparisons with larger conventional critics which have similar amount of parameters, non or partial-reset methods, or some other alternative architectures?

Thank you for raising this point!

We attempt to answer both parts of your question with existing results in the paper and a new 10-seed capacity-matched experiment:

1. Non-Reset, Full-Reset, and Partial-Reset Alternatives

In Table 1, we already compare Critic-only DCC (P+R / R, 90.4% best success, 52.3% final) against full-reset CRL R/R (84.0% best, 47.5% final), non-reset CRL P/P (78.7% best, 43.2% final), and partial-reset CRL R/P (persisting the critic while resetting the actor: 77.3% best). Furthermore, in Tables 5, 6, and 7 of the Appendix, we compare against CKA-RL (46.4%), which uses a wider critic (width 1280 vs. DCC's 1024 + 256 = 1280). Thus, simply using a wider critic or resetting only the actor does not yield DCC's gains. In total, we have constructed 2 non-reset, 2 full-reset, and 7 partial-reset alternatives for the baselines in this paper. We have reported their results in Tables 5, 6, and 7 of the Appendix.

2. New Capacity-Matched Control (Capacity-Matched CRL vs. Full DCC, 10 Seeds)

To isolate the exact benefit of carrying ϕshared\phi_{\text{shared}} across tasks from network capacity, critic resetting, and gated BC, we evaluated Capacity-Matched CRL. It uses the exact same 2-branch architecture (ϕ1\phi_1 width 1024 + ϕ2\phi_2 width 256), identical parameter count, perturbation gate, and success BC as Full DCC, but reinitializes both ϕ1\phi_1 and ϕ2\phi_2 at every task boundary:

Table A. Capacity-Matched CRL vs. Full DCC (10 Seeds, Mean ± SD)

TaskCapacity-Matched CRL AUCFull DCC AUCCapacity-Matched CRL BestFull DCC BestCapacity-Matched CRL FinalFull DCC Final
T0 Hammer0.981 ± 0.0320.994 ± 0.0121.000 ± 0.0001.000 ± 0.0001.000 ± 0.0001.000 ± 0.000
T1–T4 Avg0.7560.7130.9600.9750.8250.803
T5 Handle press0.601 ± 0.2910.698 ± 0.1621.000 ± 0.0001.000 ± 0.0000.410 ± 0.5090.420 ± 0.444
T6 Push0.864 ± 0.1040.671 ± 0.2701.000 ± 0.0001.000 ± 0.0000.950 ± 0.1080.960 ± 0.052
T7 Shelf place0.818 ± 0.1480.608 ± 0.1390.990 ± 0.0321.000 ± 0.0000.930 ± 0.1890.980 ± 0.042
T8 Window close0.333 ± 0.1940.926 ± 0.0901.000 ± 0.0001.000 ± 0.0000.160 ± 0.3500.880 ± 0.257
T9 Peg unplug0.314 ± 0.0610.468 ± 0.0680.720 ± 0.1140.890 ± 0.0870.480 ± 0.1030.550 ± 0.201
10-Task Average0.693 ± 0.0450.722 ± 0.0710.955 ± 0.0300.979 ± 0.0450.723 ± 0.0530.800 ± 0.031

Takeaway: With identical capacity and identical gated BC, resetting ϕshared\phi_{\text{shared}} (Capacity-Matched CRL) loses the physical knowledge learned on earlier tasks (T5 Handle press), causing Task 8 (window-close) AUC to drop from 0.926 to 0.333 (88.0% vs. 16.0% final success) and Task 9 (peg-unplug) AUC to drop from 0.468 to 0.314 (89.0% vs. 72.0% best success). This confirms that separating persistent shared representations from task-specific representations, rather than mere parameter count, is essential for forward transfer on later tasks.

The task-specific branch in DCC in the paper uses a neural network with hidden width 256, adding only approximately 6% more parameters compared with vanilla CRL. We hope this further addresses your concern regarding network capacity.


We are deeply thankful for the reviewer's thorough technical assessment and thoughtful advice, both of which have made our manuscript much stronger. We hope the newly incorporated experiments effectively answer your questions. If so, we would be very grateful if you would consider updating your score!

We're happy to address any remaining questions you might have.

Reviewer_yBX9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies continual reinforcement learning with sparse feedback on task completion and investigates how contrastive reinforcement learning can effectively transfer knowledge across task boundaries. The authors propose Decomposed Contrastive Critics (DCC), which decomposes the critic into a persistent shared state-action representation and a task-specific representation that is re-initialized at each task boundary, together with a success replay buffer and an action-sensitive critic gate. Experimental results on ten manipulation tasks show that DCC substantially outperforms both goal-conditioned SAC and a range of contrastive RL transfer baselines.

Critical strengths
  • The authors formulate a meaningful and practically relevant problem of continual reinforcement learning under sparse rewards.
  • The analysis of how action-informative states govern task difficulty provides useful insight into the failure modes of contrastive reinforcement learning.
  • The experimental study is well designed. This is a highlight of the paper. The authors evaluate DCC against a comprehensive set of sparse SAC and contrastive RL baselines. The authors also include careful ablation studies. Furthermore, the authors provide diagnostic analysis to shed light on failure modes of contrastive RL and how the proposed components address them.
Critical weaknesses

-The individual building blocks of DCC (representation decomposition, success replay/self-imitation, and selective policy supervision) have substantial precedent in the literature. As such, the paper is mostly integration work.

  • Evaluations are conducted on the same 10-task Continual World V2 Sawyer manipulation sequence (the tasks differ, but same robot embodiment, state/action structure, simulator, and benchmark family). It is unclear if DCC generalizes well to other continual-RL domains such as navigation, locomotion, different robots, or substantially different task distributions.
Questions for authors

To what extent do the authors expect DCC to generalize beyond the 10-task Continual World V2 Sawyer manipulation benchmark?

1 comment

We are deeply grateful for reviewer yBX9 kind and encouraging review, as well as for your recognition of our problem formulation, experimental design, and analysis of action-informative states.

Q1

To what extent do the authors expect DCC to generalize beyond the 10-task Continual World V2 Sawyer manipulation benchmark?

We expect critic decomposition to be particularly beneficial when tasks share reusable predictive structure but require substantially different action-dependent adaptations. When both predictive structure and action preferences remain similar across tasks, retaining the entire critic may already provide effective transfer, leaving less room for improvement through decomposition.

In settings with high-dimensional observations or changing embodiments, DCC may generalize with modifications. Suppose a visual encoder maps an observation oo to a state representation h=E(o)h = E(o). The critic then becomes f(h,a,g)f(h, a, g), and the action-sensitivity calculation remains largely unchanged. State sensitivity requires meaningful nearby representations h(j)h^{(j)}. The simplest option is h(j)=h+εξ(j)h^{(j)} = h + \varepsilon \xi^{(j)}, but arbitrary perturbations can push embeddings off the learned state manifold. A better approach is to estimate state sensitivity from neighboring states in real trajectories: σstate=Std⁡jf(E(o(j)),a~,g)\sigma_{\mathrm{state}} = \operatorname{Std}_j f(E(o^{(j)}), \tilde a, g), where the observations o(j)o^{(j)} are temporally adjacent observations or nearest neighbors in the learned representation space.

Generalization remains uncertain when tasks share little reusable structure. For example, transferring between robots with different action spaces may require an aligned representation before the critic branches can be reused. One possible extension is to give tasks a common representation space in which each task occupies only the subspace it needs, rather than forcing them to learn the same representation. Suppose there is a large shared latent space RD\mathbb{R}^D, and each task kk learns an encoder Ek:Sk→RdkE_k: \mathcal{S}_k \rightarrow \mathbb{R}^{d_k} with dk≪Dd_k \ll D. A mapping TkT_k then embeds the task-specific representation into the common space: zk(s)=Tk(Ek(s))∈RDz_k(s) = T_k(E_k(s)) \in \mathbb{R}^D. A natural parameterization is Tk(h)=UkhT_k(h) = U_k h with Uk∈RD×dkU_k \in \mathbb{R}^{D \times d_k}, where the columns of UkU_k span the task’s subspace. Different tasks may occupy different, partially overlapping subspaces, and transfer occurs through the structure those subspaces share. To encourage sparse usage, we can add an L1 penalty: L=Lpredict+λ∣αk∣1\mathcal{L} = \mathcal{L}_{\mathrm{predict}} + \lambda \\|\alpha_k\\|_1, where αk\alpha_k are task kk’s coefficients over the shared feature dictionary. Here Lpredict\mathcal{L}_{\mathrm{predict}} could be a contrastive future-goal prediction loss, provided goals can be meaningfully represented in the common space. A generalist critic could then maintain a growing dictionary of predictive features, with each new task learning a sparse combination of existing features plus whatever new ones it needs.

We agree that self-imitation is already used in the existing literature, as you noted in your review. In our experiments, we adopted Success Replay to handle the sparse action-informative states in Tasks 5 and 8. While the technique itself should transfer to other sparse-reward tasks, we acknowledge there may be better ways to shape CRL’s energy landscape. Both the critic gate and Success Replay are motivated by the broader action-information problem, namely the distinction between future-state prediction and action discrimination, which is independent of any specific benchmark. Critic-gated Success Replay is one possible intervention among several. Although our solution may not be optimal, we believe the motivation behind the gate generalizes more broadly than its current implementation.

We thank the reviewer for raising this important question.

Comments on the paper

1 comment

Dear reviewers,

Thank you so much for taking the time and giving your attention to reading our paper and providing such in-depth feedback. Time and attention are the most valuable things anyone has to give, and it means a great deal to us that you gave them to our work.

Following your constructive and valuable feedback, we have made substantial revisions to the manuscript: (i) added an additional ablation variant, an additional metric, and a new Table 2 in the main text (plus Table 8 in Appendix F.2); (ii) added a dedicated Section 2 (Problem Setup and Preliminaries) on GCRL and Contrastive RL and a dedicated Section 6 (Limitations); (iii) formalized what the shared and task-specific critic branches capture in a new Appendix G; and (iv) substantially reorganized and revised the writing in the Methodology, Experiments, and Action-Informative States sections (Sections 3–5), added a new architecture overview (Figure 1), and redesigned Figures 2–4 for clearer presentation.

We appreciate your perspective on the scope and current limitations of our work, and we kindly invite you to consider the broader contributions our study brings to the field:

  • We formalize sparse-reward, goal-conditioned continual RL, an important and underexplored setting in which agents must sequentially acquire new skills without hand-engineered dense rewards.
  • We construct 11 continual baselines spanning two algorithm classes, and use them to systematically study three transfer strategies, giving an in-depth look at representation transfer in this setting.
  • We propose DCC (Decomposed Critic Transfer with success replay and action-sensitive critic gating), which improves AUC and success rate on 9 of the 10 tasks.
  • We design four ablation variants that isolate why contrastive critics predict future goals accurately yet fail to discriminate between actions in brief-contact manipulation tasks.

Thank you again for helping us make this a much better paper, and we hope our revised manuscript earns your support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.