acceptodds
Under review as a conference paper at ICLR 2027

Scaling Contrastive Reinforcement Learning to Open-Ended Environments

Abstract

Developing generally capable agents requires scalable methods for learning a wide range of goal-directed behaviours. Contrastive reinforcement learning (CRL) offers a promising approach by reframing goal-conditioned value learning as self-supervised classification of an agent's own future states. We extend CRL to open-ended environments, where agents pursue semantic goals that represent desired outcomes rather than particular target states. We formalize these goals as predicates over the state space, which makes CRL's training labels computable in closed form, and derive a Rao-Blackwellized estimator of the resulting discounted goal-occupancy measure. We further find that this estimator is biased when an agent's actions can terminate the episode, as it implicitly conditions on the episode continuing and fails to penalize actions that cause termination. We address this bias with an absorbing class and termination-discounted targets that account for the probability mass beyond the episode end. The resulting critic serves as an effective teacher for a policy-gradient student. On Craftax, our method achieves comparable success to an agent trained with a per-goal Q-learning teacher across thousands of semantic goals while using up to less training compute. These results demonstrate that CRL can scale to large semantic goal spaces while accounting for irreversible failures in open-ended environments.

Primary area:
reinforcement learning
Keywords:
contrastive reinforcement learning, goal-conditioned reinforcement learning, open-ended environments, semantic goals, termination bias

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vRpHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper adapts contrastive RL, a goal-conditioned RL algorithm, to Craftax, an open-ended Minecraft-like benchmark with hundreds to thousands of goals. The authors make the following changes to the algorithm: (i) goals become semantic predicates over states, and InfoNCE is replaced by a full softmax cross-entropy over the finite goal set; (ii) Rao-Blackwellized targets, i.e., an exact average over every future step in the window; (iii) an absorbing class that receives the probability mass beyond the episode’s end; (iv) termination-discounted targets, which extend that penalty to the steps before a death; and (v) following the previous SOTA recipe of Matthews et al. (2026), the CRL critic as a teacher distilled into PPO. They “scale” CRL to far more goals than it has previously been tested on, up to 9,637. Empirically, its success rate is comparable to the SOTA algorithm with far fewer parameters and less training compute.

Critical strengths
  1. The paper brings several interesting ideas to CRL, such as the natural semantic-goal formulation and the identification of the termination bias.
  2. Scaling the number of goals rather than parameters is also an interesting direction.
  3. To the authors' knowledge, this is the first work to apply CRL to open-ended environments, which makes the study interesting in its own right.
Critical weaknesses
  1. The paper makes many modifications to CRL, yet the performance gains are less impressive. Most of Dual-CRL-PPO's advantage appears to come from the teacher-student paradigm of Dual-LEO-PPO rather than from the proposed changes.
  2. The computational savings are claimed as a contribution, but they follow from original CRL's design itself. CRL scores state-action embeddings against a goal embedding instead of keeping a separate head per goal, and none of the proposed modifications is responsible for this efficiency.
  3. There is no ablation within Dual-CRL-PPO, so the headline results cannot be attributed to the proposed components.
  4. Goals are binary masks read directly off the symbolic observation, so labels are free and exact. Evaluation is also limited to one benchmark family.
Questions for authors
  1. Which CRL variant serves as the teacher in Dual-CRL-PPO in Table 2? The captions of Tables 5 and 6 state that the absorbing class and Rao-Blackwellized targets are used without termination-discounted targets. Please report Dual-CRL-PPO with a plain CRL teacher and with each component added in turn. This would show whether contributions 1 and 2 matter for the result the paper leads with.
  2. The parameter counts in Tables 4 and 6 imply roughly 64 parameters per goal for CRL (10.64M to 11.22M) and roughly 44k for LEO (30.35M to 432.53M), which matches a 1024 × 43 head per goal. This gap comes from the factorized goal embedding described in the paper lines 314–315. Could the authors compare against LEO with a factorized goal head, so that CRL's contribution is separated from the factorization? This bears directly on contribution 3.
  3. The paper states that unseen goals can be scored with a forward pass through the goal encoder. The parameter growth suggests the goal encoder is effectively a per-goal lookup table. If so, how would an unseen goal receive a meaningful embedding?
  4. Lines 200–202 state that K_t is limited by the window length W. Table 8 lists a rollout length of 64 with γ = 0.995, so about 73% of the discount mass lies beyond the window. Does the window span rollout boundaries? If not, could this truncation explain CRL's difficulty with long-horizon goals such as coal and iron (Figure 5)?
  5. Lines 272–275 state that γ _term = γ recovers the goal-occupancy measure exactly. Yet success declines as γ_term approaches γ (Table 3), from 0.586 at 0.8 to 0.559 at 0.995. Why does the exact correction hurt goal success? Does this suggest the termination-discounted target is a tuning heuristic rather than a principled fix?
  6. The state-action embedding does not appear to be conditioned on the commanded goal. With Monte Carlo targets from behaviour data, the critic would then estimate goal reachability under average behaviour rather than under a policy pursuing that goal. LEO's Q-learning instead learns per-goal optimal values. Could this explain why the CRL teacher trails the LEO teacher?
  7. Does the "CRL" baseline use the original InfoNCE loss with sampled negatives (Eq. 2) or the full softmax (Eq. 3)? A comparison against original CRL would clarify how much the semantic-goal formulation itself contributes.
5 comments
AuthorsOpenReview ↗

Author Response (1/4)

We thank the reviewer for their constructive feedback and for recognising the contributions of our semantic-goal formulation, the analysis of termination bias, and the scalability of CRL to larger goal spaces. We have revised the manuscript to address their comments and questions and incorporate the additional experimental results. Changes relevant to this review are highlighted in green\textcolor{green}{\text{green}}.

W1 / W2: Performance gains and computational efficiency

The proposed modifications address different limitations of CRL, and their effects depend on the environment. Our experiments show that they successfully address the failure modes they were designed for. The absorbing class is effective when termination results from a single fatal action. In Gridworld Lava, it increases success from 0.635 to 0.935 (Table 1). Similarly, on Craftax Classic, lava deaths are eliminated when the absorbing class is included (Figure 3). Termination-discounted targets, on the other hand, address deaths that result from damage accumulated over multiple timesteps. They reduce combat deaths from 44% to 18% of episodes and increase average episode length to 247 steps. RB targets also reduce the number of environment steps needed to reach a given success threshold by 10-20% (Figures 7-8).

We agree that the teacher-student approach builds on Matthews et al. (2026) [1] and do not claim this mechanism as a new contribution. However, our new ablations show that the proposed CRL modifications also improve the final Dual-CRL-PPO policy. A plain CRL teacher achieves 0.792 on Craftax Classic and 0.240 on Full Craftax. Adding the co-satisfaction labels for semantic goals, absorbing class and RB targets improves success to 0.821 and 0.255, respectively (see Q1). With symmetric InfoNCE, Dual-CRL-PPO matches Dual-LEO-PPO on Classic (0.841 vs. 0.838; see general response).

We also agree that CRL's computational efficiency is a consequence of its underlying factorised architecture. Our contribution is demonstrating that CRL can be extended to open-ended environments with diverse semantic goals, that its limitations in episodic settings can be addressed, and that it can scale to thousands of goals while maintaining competitive performance. At 9,637 goals, Dual-CRL-PPO achieves comparable performance to Dual-LEO-PPO with 19$\times$ fewer parameters and 38$\times$ less network-training compute. Our factorised LEO ablation also shows that CRL's efficiency-performance advantage depends on both its factorised architecture and the contrastive learning objective (see Q2).

W4: Label availability and benchmark scope

Having access to exact goal-satisfaction labels is a useful property of Craftax, allowing us to check achievements directly from the symbolic state without human annotation or a learned reward model. This assumption is also common in goal-conditioned RL. For example, hindsight experience replay requires a function to determine whether a goal has been achieved (Andrychowicz et al., 2017) [2]. All baselines in our comparison, including LEO, use the same goal-satisfaction labels. Settings without directly checkable goals would require learned goal classifiers, which we leave for future work.

Our main evaluation is limited to one benchmark family, but covers two Craftax environments with substantially different difficulty and goal spaces ranging from 136 to 9,637 goals. We also use a separate Gridworld environment to isolate the termination bias (Figure 2). While many previous CRL benchmarks focus on individual task families such as navigation, manipulation, or locomotion (Bortkiewicz et al., 2025) [3], Craftax combines navigation, resource acquisition, crafting, and combat within the same environment. We chose Craftax because it also provides a strong, well-tuned baseline from Matthews et al. (2026) [1], enabling a direct comparison. Extending the evaluation to other open-ended environments remains future work.

AuthorsOpenReview ↗

Author Response (2/4)

W3 / Q1: Teacher variant and component ablation

The teacher in Table 2 uses co-satisfaction targets, the absorbing class, and Rao-Blackwellized (RB) targets, but no termination-discounted targets, as in Tables 5 and 6 (Tables 6 and 7 in the revised manuscript). We have clarified this in the caption of Table 2.

We ran the requested ablation (see general response for the full results). A plain CRL teacher achieves 0.792 on Craftax Classic and 0.240 on Full Craftax. Adding co-satisfaction targets improves success to 0.816 and 0.247. Adding the absorbing class maintains similar success rates (0.814 and 0.245) while reducing the student's lava deaths on Craftax Classic from 1.2% to 0.7%. Adding RB targets further improves success to 0.821 and 0.255, matching the configuration used in Table 2. Termination-discounted targets (γterm=0.8\gamma_{\mathrm{term}} = 0.8) reduce combat deaths on Craftax Classic from 30.3% to 23.3% at comparable success (0.814 vs. 0.821), while improving success on Full Craftax to 0.260.

These results show that each component improves either goal success or survival, and that the benefits extend to the final Dual-CRL-PPO policy rather than only the CRL critic.

Q2: LEO with a factorised goal head

The reviewer's interpretation of the parameter counts is correct. We ran the requested comparison using LEO's TD objective with CRL's factorised critic (FacLEO; see general response for the full ablation across embedding dimensions).

At D=64D=64, FacLEO matches CRL's critic architecture and parameter count, but substantially underperforms dense LEO on both Craftax Classic (0.576 vs. 0.650) and Full Craftax (0.122 vs. 0.163). With the same architecture, CRL achieves similar performance to FacLEO on Classic (0.564 vs. 0.576) and better performance on Full Craftax (0.158 vs. 0.122). Increasing FacLEO's embedding dimension to match dense LEO's parameter count does not close the gap, with success rates of 0.598 and 0.141, respectively.

We also tested FacLEO as a teacher for PPO. On Craftax Classic, replacing the dense critic in Dual-LEO-PPO with FacLEO reduces success from 0.838 to 0.804, below Dual-CRL-PPO (0.821). The efficiency-performance trade-off therefore depends on both the factorised architecture and the learning objective, rather than factorisation alone.

Q3: Goal encoder and unseen goals

The reviewer is correct. In the reported configuration, the critic uses a per-goal embedding table, which cannot produce embeddings for unseen goals. Our statement in Section 4.2 referred to CRL's architecture, which supports arbitrary goal encoders, whereas LEO's per-goal output heads cannot score unseen goals by construction (and rely on these heads for their performance, see Q2). We used an embedding table for the critic to minimise architectural differences in our comparison with LEO. The PPO baseline and the students of both dual methods already encode goals directly from their raw templates. We have clarified this distinction in the revised manuscript.

We also ran additional experiments replacing the critic's embedding table with an MLP that encodes either raw goal templates or parsed semantic features (category, entity, direction, count). Within Dual-CRL-PPO, both variants match or improve on the embedding table. On Craftax Classic, Dual-CRL-PPO achieves 0.821 and 0.828, respectively, compared with 0.821 using the embedding table. On Full Craftax, success increases to 0.264 and 0.268, respectively, compared with 0.255. The semantic-feature encoder thus matches Dual-LEO-PPO on Full Craftax (0.268). Standalone CRL performs similarly across all three encoders.

Unlike the embedding table, these encoders can produce embeddings for unseen goals from their features. We have added the encoder ablation to the revised manuscript and plan to evaluate generalisation to held-out goals in future work.

AuthorsOpenReview ↗

Author Response (3/4)

Q4: Window truncation

The window does not span rollout boundaries. Future offsets are sampled within the same 64-step rollout, so KtK_t is limited by the remaining steps in the rollout or episode. Standalone CRL in Table 2 uses no additional offset cap, while the Dual-CRL-PPO teacher and observation-level configurations use kmax⁡=32k_{\max}=32. The reviewer is correct that approximately 73% of the discount mass lies beyond the 64-step window. However, the targets are renormalised over the observed offsets through the 1−γKt1-\gamma^{K_t} term in Equation 6. The critic therefore learns a truncated-horizon goal occupancy distribution rather than simply underestimating goal occupancy due to missing future steps.

To test whether truncation limits performance, we varied the offset cap for standalone CRL on Craftax Classic. Success increases from 0.500 at kmax⁡=8k_{\max}=8 to 0.569 at 16 and 0.591 at 32. Increasing the cap further does not help, with success reaching 0.571 without a cap. Truncation therefore seems unlikely to explain the difficulty with coal and iron. These goals may instead receive little positive target mass because they are rarely achieved under the behaviour policy. This explanation is also supported by Figure 5: the standalone CRL variants perform similarly poorly on coal, iron, and diamond, whereas Dual-CRL-PPO achieves much higher success on these goals. We have added the offset-cap ablation to the appendix.

Q5: Termination discount

Success declines as γterm\gamma_{\mathrm{term}} approaches γ\gamma because the corrected goal occupancy measure also accounts for survival. At γterm=γ\gamma_{\mathrm{term}} = \gamma, the probability mass beyond episode termination is assigned to the absorbing class. This lowers the scores of actions associated with early death and makes the greedy policy more cautious. However, the success metric still counts goals achieved immediately before death. This explains why combat deaths in Figure 3 decrease from 44% to 18% of episodes and average episode length increases to 247 steps, even as goal success declines.

Our formulation can be viewed as an instance of transition-based discounting (White, 2017) [4], which allows a separate discount for terminating transitions. As γterm→0\gamma_{\mathrm{term}} \to 0, only the fatal action is penalised, recovering the absorbing-class correction. At γterm=γ\gamma_{\mathrm{term}} = \gamma, the targets recover the exact occupancy measure for anchors whose termination is observed within the window (see Q4 for the truncation effect). The termination discount γterm\gamma_{\mathrm{term}} can therefore be treated as a hyperparameter controlling how strongly the agent is penalised for actions leading to episode termination.

Q6: Monte Carlo occupancy vs. optimal goal values

The reviewer is correct that our Monte Carlo CRL critic estimates discounted goal occupancy under the behaviour policy, whereas LEO uses TD learning to approximate optimal goal-conditioned action values. Both learn from behaviour-policy data, but LEO bootstraps towards the highest estimated action value for each goal, which may help with rarely achieved goals. This dependence on the behaviour policy is known from prior work, and off-policy alternatives exist [5, 6].

Our FacLEO ablation shows that switching to TD learning with the same critic architecture does not consistently improve performance. CRL performs similarly to FacLEO on Craftax Classic (0.564 vs. 0.576) and better on Full Craftax (0.158 vs. 0.122; see Q2). As a teacher, Dual-FacLEO-PPO also performs worse than Dual-CRL-PPO on Classic (0.804 vs. 0.821). With uniform symmetric InfoNCE, Dual-CRL-PPO further matches Dual-LEO-PPO (0.841 vs. 0.838; see general response). The dependence on the behaviour policy may therefore explain part of the performance difference, but not all of it.

Reviewer_PSntOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This work proposed a new contrastive reinforcement learning (CRL) method adapted to open-ended environments. The authors identify and address two limitations of previous CRL approaches, namely the sensitivity to termination states and the difficulty to manage goal predicates. The results show that the method is slightly more efficient than the main baseline.

Critical strengths
  • In general, the paper is easy to follow.
  • The experimental protocol is neat. Considered environments and experiments are appropriate.
Critical weaknesses
  • The proposed Dual-CRL-PPO method is incremental, and is mostly an adaptation of previous CRL approaches to address specific limitations.
  • Dual-CRL-PPO is outperformed by Dual-LEO-PPO by raw performance. The teaching method produces improvements in computational efficiency, but it is unclear whether a simple goal-conditioned critic similar to what is used in the present paper could be used alongside Dual-LEO-PPO.
  • The motivation and idea of using a teacher are unclear (Section 3.3). Furthermore, in the ablation study, the authors did not test CRL + teacher. Therefore it is unclear whether other technical improvements matter when combining CRL and Teacher.
Questions for authors
  • Could the authors clarify the motivation and idea of using a teacher ?
  • Could they provide the missing experiments, as discussed above ?
  • What is the fundamental insight of this paper, beyond making few technical improvements to CRL for a specific use-case ? What do we fundamentally learn from this 38x less training compute ?
3 comments
AuthorsOpenReview ↗

Author Response (1/2)

We thank the reviewer for their constructive feedback and for recognising the clarity of the paper and the quality of our experimental protocol. We have revised the manuscript to address their comments and questions and incorporate the additional experimental results. Changes relevant to this review are highlighted in magenta\textcolor{magenta}{\text{magenta}}.

W1 / Q3: Novelty and fundamental insights

What is the fundamental insight of this paper, beyond making few technical improvements to CRL for a specific use-case ? What do we fundamentally learn from this 38x less training compute ?

Our main contribution is extending CRL beyond state-based goals and demonstrating its effectiveness in large, diverse goal spaces. Our formulation supports arbitrary state-based goal predicates, including goals that can be co-satisfied. The standard state-based goal setting can be recovered as a special case of our formulation for finite state spaces.

The proposed modifications address limitations of CRL that become particularly relevant in more complex environments. On Craftax, the absorbing class addresses termination caused by a single fatal action, while termination-discounted targets account for actions leading up to it. Our Gridworld Lava results show that the termination bias also affects standard CRL with state-based goals, and that the absorbing class corrects it (Table 1). Our Rao-Blackwellized targets reduce estimator variance and require 10-20% fewer environment steps to reach a given success threshold, without increasing the parameter count (Figures 7-8). The effects on downstream goal success depend on the environment, but our new teacher ablations show that the absorbing class and Rao-Blackwellized targets also improve the final Dual-CRL-PPO policy (see general response).

Like LEO, CRL learns about all goals from each trajectory, but scales much more efficiently as the goal space grows. Our results demonstrate that the CRL critic can serve as an effective teacher for PPO, achieving comparable final performance to Dual-LEO-PPO with substantially less network-training compute. Our factorised LEO ablation further shows that the advantage is not purely architectural. With the same critic architecture, CRL outperforms factorised LEO on Full Craftax (0.158 vs. 0.122; see general response). These results show that CRL can scale to thousands of semantic goals while addressing limitations of the original method and achieving strong performance in Craftax.

W2 / Q2: Comparison with factorised LEO

The small performance gap reported in the paper is not fixed. With a symmetric InfoNCE objective and uniform weighting, Dual-CRL-PPO matches Dual-LEO-PPO on Craftax Classic at the original training budget (0.841 vs. 0.838; see general response). Our additional scaling experiments also show that the gap closes with a larger training budget on the observation-level goal set (0.861 vs. 0.858 at 2×1092 \times 10^9 steps; see our response to Reviewer EjFj\textcolor{red}{\text{EjFj}}, Q2).

We interpret the reviewer's suggestion of a simpler goal-conditioned critic as replacing LEO's dense per-goal output head with the factorised architecture used by CRL, while retaining LEO's TD objective (FacLEO). We tested this and report the full ablation across embedding dimensions and both environments in our general response. On Craftax Classic, factorising LEO reduces standalone success from 0.650 to 0.576 at D=64D=64. Even with an embedding dimension that matches dense LEO's parameter count, success reaches only 0.598. On Full Craftax, FacLEO also underperforms dense LEO (0.122 vs. 0.163) and CRL (0.158) at D=64D=64.

We then tested using FacLEO as a teacher for PPO. Replacing the dense critic in Dual-LEO-PPO with FacLEO reduces success from 0.838 to 0.804 on Craftax Classic, below Dual-CRL-PPO (0.821). These results show that LEO does not retain its performance with the tested factorised architecture, either as a standalone critic or as a teacher for PPO. CRL's efficiency-performance advantage therefore depends on both its factorised architecture and the contrastive learning objective, rather than factorisation alone.

AuthorsOpenReview ↗

Author Response (2/2)

W3 / Q1 / Q2: Teacher motivation and ablations

Could the authors clarify the motivation and idea of using a teacher ?

The teacher-student framework combines the strengths of CRL and PPO. The CRL critic learns goal-conditioned state-action scores across the entire goal space from shared experience, but does not directly optimise a policy. PPO, by contrast, directly optimises the commanded-goal reward without receiving explicit supervision for other goals. Dual-CRL-PPO combines both by distilling the critic's greedy actions into the PPO student through behavioural cloning (Equation 9), while the student continues to optimise the commanded-goal reward. We have made this motivation more explicit in Section 3.3.

To evaluate this approach, we compared three ways of obtaining a policy from CRL, using the same experimental setup as in the paper. Acting greedily on the critic achieves success rates of 0.591 on Craftax Classic and 0.167 on Full Craftax. Training a separate actor to maximise the critic's scores, as commonly done in continuous-action CRL, performs worse (0.493 and 0.132). Using the critic as a teacher for PPO performs substantially better (0.821 and 0.255). These results show that combining CRL supervision with PPO is more effective than either greedy action selection or training an actor solely against the critic.

We have also run the missing teacher ablations within Dual-CRL-PPO (see general response for the full results). With a standard CRL teacher, success reaches 0.792 on Craftax Classic and 0.240 on Full Craftax. Adding the co-satisfaction targets for semantic goals and the absorbing class improves success to 0.814 and 0.245, while adding Rao-Blackwellized targets further improves it to 0.821 and 0.255. The termination discount additionally reduces the student's combat deaths and improves success on Full Craftax to 0.260. The proposed CRL modifications therefore also improve the final Dual-CRL-PPO policy.

Conclusion

We thank the reviewer again for their constructive feedback and hope that our additional experiments and clarifications have addressed their questions. We welcome any further suggestions for improving the paper. If their concerns have been addressed, we would appreciate it if they considered increasing their support for the paper.

I thank the authors for the response.

I will increase my score to weak accept, as most of my concerns are addressed. The contribution incrementally builds on CRL, which is why I do not further increase my score.

Reviewer_EjFjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an extension of contrastive reinforcement learning (CRL) to open-ended environments with “semantic goals,” where a single state may satisfy many different goals (and a goal may be satisfied by many different states). To take advantage of the discrete, limited number of goals, the authors introduce a closed-form objective that estimates discounted goal-occupancy measure with lower variance. They observe a bias in CRL for environments with actions that can cause early termination, and introduce a correction method for their closed for estimator. Finally, they demonstrate that a CRL critic can be used to train a strong PPO policy on the Craftax environment.

Critical strengths
  • Provides a simple, closed form objective for CRL when “semantic goals” are required.
  • Gives interesting insight into the termination bias of CRL, and introduces a simple fix to the problem for their objective. Provides a clear ablation in Table 1 showing that their proposed change significantly improves performance when many terminating actions are available.
  • Demonstrates scalability compared to LEO as the number of goals increase (Figure 4), since LEO’s parameter count scales with the number of goals. Compared to LEO, they achieve comparable performance in significantly less compute.
Critical weaknesses
  • The proposed solution to the CRL termination bias is tailored for the discrete semantic goal setting. However, the termination bias is an issue that applies in general for CRL, and no general solution is proposed.

  • In general, the empirical results are not super strong. For instance, the termination bias correction does not significantly improve success rates (and can decrease it depending on strength), despite reducing combat/lava death rates. The closed-form Rao-Blackwellized estimator also does not appear to provide very large speedups or performance gains. Furthermore, CRL still lags behind LEO/PPO, although the gap closes after PPO+CRL distillation.

Questions for authors
  • Where does the increase wall-clock time come from when scaling to full Craftax for CRL (I see that it increases from 3.9 to 32.9, despite having similar parameter count)?

  • Why does CRL end up having a similar performance ceiling to e.g. LEO, despite being more compute efficient (i.e. why can't additional flops be transferred into better performance)?

  • Is it possible to generalize the solution you have proposed to the termination bias issue for the normal CRL objective?

2 comments
AuthorsOpenReview ↗

Author Response (1/2)

We thank the reviewer for their constructive feedback and for recognising our contributions to the closed-form CRL objective for semantic goals, the analysis and correction of termination bias, and scalability compared to LEO.

We have revised the manuscript to address the reviewer's comments and incorporate the additional experimental results. Changes relevant to this review are highlighted in red\textcolor{red}{\text{red}}.

W1 / Q3: Generality of the termination-bias solution

Is it possible to generalize the solution you have proposed to the termination bias issue for the normal CRL objective?

The termination bias and the underlying idea of our corrections are not specific to semantic goals. The bias arises because renormalising the sampling distribution over observed future states redistributes probability mass that would otherwise lie beyond episode termination (Section 3.2). Our Gridworld Lava experiment (Figure 2, Table 1) demonstrates this bias and the effectiveness of the absorbing-class correction in an environment without any semantic goals. Here, goals correspond to individual target cells, and each state satisfies at most one goal. This is the standard state-based goal setting with a finite discrete goal set.

Both proposed corrections, the absorbing class and termination-discounted targets, modify the target distribution and can in principle be extended to sampled goals. The absorbing class has a fixed zero logit and can be added to the softmax regardless of whether the other candidates are enumerated or sampled (Equation 7). Termination-discounted targets assign probability mass γtermΔt\gamma_{\mathrm{term}}^{\Delta_t} to the absorbing class, with the remaining mass distributed over observed future goals (Equation 8). Neither modification requires enumerating the full goal space.

However, with sampled negatives, the softmax no longer normalises over the full goal space, so the probability assigned to the absorbing class may not be calibrated to the termination mass. We identify this as an open problem in Section 6 and leave its resolution to future work.

W2: Experimental results

The proposed modifications address different limitations of CRL, and their effects on success rates depend on the characteristics of the environment.

The two termination corrections address different failure modes. The absorbing class accounts for actions that immediately terminate an episode. In Gridworld Lava, where stepping into lava ends the episode, it increases success from 0.635 to 0.935 (Table 1). On Craftax Classic, adding the absorbing class helps eliminate lava deaths and increase average episode length from 182 to 199 steps (Figure 3). Our new teacher ablations also show that adding the co-satisfaction targets and the absorbing class improves Dual-CRL-PPO success from 0.792 to 0.814 on Classic and from 0.240 to 0.245 on Full Craftax (see general response).

Termination-discounted targets address deaths that occur after multiple timesteps. On Craftax Classic, they reduce combat deaths from 44% to 18% of episodes and further increase average episode length to 247 steps (Figure 3). At γterm=0.8\gamma_{\mathrm{term}} = 0.8, standalone CRL success also improves to 0.586.

The Rao-Blackwellized estimator preserves the original estimator's bias while reducing its variance. In our experiments, this translates into modest but consistent performance gains. The RB estimator requires 10-20% fewer environment steps to reach a given success threshold and yields a higher final mean in every configuration without adding parameters (Figures 7-8).

Finally, standalone CRL with the absorbing class already outperforms PPO, PQN, and PQN+HER on Full Craftax (0.168 vs. 0.123, 0.098, and 0.125, respectively). More importantly, our proposed agent is Dual-CRL-PPO, which uses the critic as a teacher rather than acting greedily with respect to it. As shown in our general response, adding symmetric InfoNCE with uniform weighting allows Dual-CRL-PPO to close the performance gap to Dual-LEO-PPO on Craftax Classic (0.841 vs. 0.838). CRL achieves this performance while scaling much more efficiently to larger goal spaces than dense LEO (Figure 4).

AuthorsOpenReview ↗

Author Response (2/2)

Q1: Wall-clock time scaling with goal-set size

Where does the increase wall-clock time come from when scaling to full Craftax for CRL (I see that it increases from 3.9 to 32.9, despite having similar parameter count)?

The increase comes from scaling the goal set from 512 semantic goals to 9,637 observation-level goals. Two parts of our implementation scale with the number of goals ∣G∣\lvert\mathcal{G}\rvert. First, the critic computes scores and normalises over all goals at every gradient step. Second, constructing the training targets requires checking which goals are satisfied at each stored state and accumulating these labels over the rollout window. The parameter count changes little because each goal only adds a 64-dimensional embedding, which is small relative to the state-action encoder. Wall-clock time increases by 8.4$\times$ while the goal-set size increases by 18.8$\times$, since other costs such as the environment simulation and the state-action encoder do not scale with ∣G∣\lvert\mathcal{G}\rvert.

Q2: Performance ceiling vs. compute efficiency

Why does CRL end up having a similar performance ceiling to e.g. LEO, despite being more compute efficient (i.e. why can't additional flops be transferred into better performance)?

To answer this question, we extended the scaling experiments in Section 4.3 with additional runs at 2×1092 \times 10^9 and 4×1094 \times 10^9 environment steps on Craftax Classic with observation-level goals. We used three seeds per configuration, with each run trained on a single NVIDIA H100 GPU, as in the original experiments.

Dual-CRL-PPODual-LEO-PPO
Env. stepsSuccess rateTime (h)Params (M)Success rateTime (h)Params (M)
1×1091 \times 10^90.820 ±\pm 0.0084.915.370.840 ±\pm 0.0046.137.85
2×1092 \times 10^90.861 ±\pm 0.0078.815.370.858 ±\pm 0.00611.737.85
4×1094 \times 10^90.868 ±\pm 0.00516.315.370.873 ±\pm 0.00522.937.85

Additional training does improve performance. Dual-CRL-PPO increases from 0.820 to 0.868, and the gap to Dual-LEO-PPO at 10910^9 steps closes at 2×1092 \times 10^9 steps (0.861 vs. 0.858). At higher budgets, the differences remain within one standard deviation across seeds, with neither method consistently outperforming the other. The gap observed at 10910^9 steps therefore does not establish a lower performance ceiling for CRL.

CRL also retains its compute advantage across all tested budgets. When trained for the same number of environment steps, Dual-CRL-PPO requires 20-29% less wall-clock time, and the advantage increases at larger budgets. It also uses less than half as many parameters (15.37M vs. 37.85M, Table 5; Table 6 in the revised manuscript). This allows Dual-CRL-PPO to train for more environment steps within the same wall-clock budget, and our scaling experiments show that additional steps continue to improve performance.

Conclusion

We thank the reviewer again for their constructive feedback and hope that our additional experiments and clarifications have addressed their questions. We welcome any further suggestions for improving the paper. If their concerns have been addressed, we would appreciate it if they considered increasing their support for the paper.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response (1/3)

We thank all reviewers for their thorough reviews and constructive feedback. We appreciate their positive assessments of our contributions and methodology:

vRpH\textcolor{green}{\text{vRpH}}: “The paper brings several interesting ideas to CRL, such as the natural semantic-goal formulation”

EjFj\textcolor{red}{\text{EjFj}}: “Gives interesting insight into the termination bias of CRL, and introduces a simple fix to the problem” and “[d]emonstrates scalability compared to LEO as the number of goals increase”

PSnt\textcolor{magenta}{\text{PSnt}}: “The experimental protocol is neat”

We also appreciate the reviewers' requests for clarification and further experiments. We have conducted extensive new experiments to address their questions. In this general response, we summarise the results relevant to multiple reviewers and address the remaining points individually under each review. Unless stated otherwise, all new experiments follow the same experimental setup as in the paper, with success rates measured in separate greedy evaluation rollouts, as in Table 2.

We have also uploaded a revised manuscript incorporating the requested clarifications and new experimental results. Changes to existing text are colour-coded by reviewer, and new sections and appendices are marked with the coloured IDs of the reviewers who requested them.

Teacher ablation within Dual-CRL-PPO

Reviewers vRpH\textcolor{green}{\text{vRpH}} and PSnt\textcolor{magenta}{\text{PSnt}} asked whether the proposed CRL modifications improve performance when the critic is used as a teacher for PPO. We therefore ran additional Dual-CRL-PPO ablations for the different components. Starting from a standard CRL teacher, we add the co-satisfaction targets for the semantic goals, the absorbing class, the Rao-Blackwellized (RB) targets, and the termination discount in turn. The fourth configuration is the teacher used throughout the paper (Table 2). All variants use the same PPO student and training setup and differ only in the CRL teacher objective. The absorbing class and the termination discount target early termination. We therefore also report the student's training episode statistics (episode length, lava deaths, and combat deaths) on Craftax Classic.

Craftax ClassicFull Craftax
TeacherSuccess rateEpisode lengthLava deaths (%)Combat deaths (%)Success rate
standard CRL teacher0.792 ±\pm 0.019227 ±\pm 32.2 ±\pm 0.934.8 ±\pm 2.00.240 ±\pm 0.010
+ co-satisfaction targets0.816 ±\pm 0.015233 ±\pm 61.2 ±\pm 0.831.8 ±\pm 2.90.247 ±\pm 0.008
+ absorbing class0.814 ±\pm 0.004237 ±\pm 100.7 ±\pm 0.631.3 ±\pm 6.30.245 ±\pm 0.010
+ RB targets0.821 ±\pm 0.004236 ±\pm 30.6 ±\pm 0.230.3 ±\pm 1.30.255 ±\pm 0.009
+ termination discount (γterm=0.8\gamma_{\mathrm{term}} = 0.8)0.814 ±\pm 0.028251 ±\pm 70.8 ±\pm 0.423.3 ±\pm 4.30.260 ±\pm 0.006

Co-satisfaction targets improve success on both Craftax Classic (0.792 to 0.816) and Full Craftax (0.240 to 0.247). The absorbing class achieves similar success rates while reducing lava deaths caused by single fatal actions. Adding Rao-Blackwellized targets further improves success to 0.821 and 0.255, respectively. Termination-discounted targets penalise actions leading up to death, reducing combat deaths from 30.3% to 23.3% of episodes and extending average episode length to 251 steps. On Full Craftax, they also improve success to 0.260. These results show that the CRL modifications also improve the final Dual-CRL-PPO policy's success rates and survival, not just the standalone critic.

AuthorsOpenReview ↗

General Response (2/3)

Factorized LEO

Reviewers vRpH\textcolor{green}{\text{vRpH}} and PSnt\textcolor{magenta}{\text{PSnt}} asked whether LEO could achieve similar computational savings by replacing its dense goal head with a factorized architecture. We therefore implemented a factorised LEO critic using a dot product between state-action and goal embeddings, while keeping LEO's original learning objective. We evaluated different embedding dimensions to measure the trade-off between parameter count and performance.

Craftax ClassicFull Craftax
MethodSuccess rateParams (M)Time (h)Success rateParams (M)Time (h)
LEO (dense)0.650 ±\pm 0.0117.782.10.163 ±\pm 0.01230.355.7
FacLEO, D=64D = 64 (as CRL)0.576 ±\pm 0.0366.532.40.122 ±\pm 0.00610.645.3
FacLEO, D=∣G∣D = \lvert\mathcal{G}\rvert (as dense LEO)0.598 ±\pm 0.008 (D=136)(D = 136)7.802.30.141 ±\pm 0.026 (D=512)(D = 512)30.616.1
FacLEO, D≈2∣G∣D \approx 2\lvert\mathcal{G}\rvert0.608 ±\pm 0.054 (D=256)(D = 256)9.912.40.122 ±\pm 0.002 (D=1024)(D = 1024)53.447.2

Factorising LEO's goal head reduces performance across all tested configurations. At D=64D=64, factorised LEO uses the same critic architecture and parameter count as CRL, but performs worse than dense LEO on both Craftax Classic (0.576 vs. 0.650) and Full Craftax (0.122 vs. 0.163). At D=136D=136 on Craftax Classic and D=512D=512 on Full Craftax, the embedding dimension equals the goal-set size and the parameter count is nearly identical to dense LEO, but the performance gap remains. Increasing the embedding dimension further improves performance on Classic, but still falls short of dense LEO while having the same or higher parameter count. On Full Craftax, doubling the embedding dimension to D=1024D=1024 does not improve performance (0.122).

Comparing CRL with factorised LEO at D=64D=64 helps isolate the effect of the learning objective. While performance is similar on Craftax Classic (0.564 vs. 0.576), CRL performs substantially better on Full Craftax (0.158 vs. 0.122). This suggests that CRL's better efficiency-performance trade-off on Full Craftax comes from both its factorised architecture and its learning objective.

MethodSuccess rate
Dual-LEO-PPO0.838 ±\pm 0.007
Dual-FacLEO-PPO0.804 ±\pm 0.011
Dual-CRL-PPO0.821 ±\pm 0.004

To test the factorised critic within the dual system, we replaced the dense critic in Dual-LEO-PPO with its factorised counterpart at D=64D=64, while leaving the PPO student unchanged (Dual-FacLEO-PPO). On Craftax Classic it reaches 0.804, below both Dual-LEO-PPO (0.838) and Dual-CRL-PPO (0.821). The performance loss from factorising LEO therefore persists in the teacher-student framework.

AuthorsOpenReview ↗

General Response (3/3)

Backward / symmetric InfoNCE loss

The reviewers also noted that Dual-CRL-PPO still falls slightly short of Dual-LEO-PPO in performance. Our paper uses only the forward InfoNCE objective, which predicts future goals from a state-action pair. This objective assigns more positive target mass to goals that are frequently achieved under the behaviour policy, which may limit learning for rarely achieved goals. We therefore investigate the backward InfoNCE objective, which reverses the classification task and predicts which state-action pairs lead to a given goal, as well as a symmetric objective combining both directions, following Bortkiewicz et al. (2025) [1].

For the backward objective, we consider two weighting schemes: mass-weighted, where each goal is weighted by its positive target mass, and uniform, where all goals receive equal weight. Uniform weighting places more emphasis on rarely achieved goals. All experiments use γterm=0.8\gamma_{\mathrm{term}} = 0.8 on Classic and 0.90.9 on Full Craftax.

ObjectiveCraftax ClassicFull Craftax
forward0.586 ±\pm 0.0260.164 ±\pm 0.005
symmetric mass0.571 ±\pm 0.0370.160 ±\pm 0.002
symmetric uniform0.620 ±\pm 0.0040.174 ±\pm 0.011
backward mass0.554 ±\pm 0.0110.165 ±\pm 0.006
backward uniform0.630 ±\pm 0.0380.150 ±\pm 0.011

Symmetric InfoNCE with uniform weighting improves over the forward-only objective on both Craftax Classic (0.620 vs. 0.586) and Full Craftax (0.174 vs. 0.164).

ObjectiveCraftax ClassicFull Craftax
forward0.814 ±\pm 0.0280.254 ±\pm 0.009
symmetric mass0.799 ±\pm 0.0190.251 ±\pm 0.007
symmetric uniform0.841 ±\pm 0.0110.249 ±\pm 0.008

With the uniform symmetric objective, Dual-CRL-PPO reaches 0.841 ±\pm 0.011 on Craftax Classic, matching Dual-LEO-PPO (0.838 ±\pm 0.007) without increasing the parameter count. On Full Craftax, this improvement does not transfer to Dual-CRL-PPO, which remains slightly below Dual-LEO-PPO (0.249 vs. 0.268). Mass-weighting does not improve on the forward-only objective in either environment, which is consistent with both placing more weight on frequently achieved goals.

References

[1] Michał Bortkiewicz, Władysław Pałucki, Vivek Myers, Tadeusz Dziarmaga, Tomasz Arczewski, Łukasz Kuciński, and Benjamin Eysenbach. Accelerating Goal-Conditioned Reinforcement Learning Algorithms and Research. International Conference on Learning Representations (ICLR), 2025. URL: https://arxiv.org/abs/2408.11052.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.