- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper adapts contrastive RL, a goal-conditioned RL algorithm, to Craftax, an open-ended Minecraft-like benchmark with hundreds to thousands of goals. The authors make the following changes to the algorithm: (i) goals become semantic predicates over states, and InfoNCE is replaced by a full softmax cross-entropy over the finite goal set; (ii) Rao-Blackwellized targets, i.e., an exact average over every future step in the window; (iii) an absorbing class that receives the probability mass beyond the episode’s end; (iv) termination-discounted targets, which extend that penalty to the steps before a death; and (v) following the previous SOTA recipe of Matthews et al. (2026), the CRL critic as a teacher distilled into PPO. They “scale” CRL to far more goals than it has previously been tested on, up to 9,637. Empirically, its success rate is comparable to the SOTA algorithm with far fewer parameters and less training compute.
Critical strengths
- The paper brings several interesting ideas to CRL, such as the natural semantic-goal formulation and the identification of the termination bias.
- Scaling the number of goals rather than parameters is also an interesting direction.
- To the authors' knowledge, this is the first work to apply CRL to open-ended environments, which makes the study interesting in its own right.
Critical weaknesses
- The paper makes many modifications to CRL, yet the performance gains are less impressive. Most of Dual-CRL-PPO's advantage appears to come from the teacher-student paradigm of Dual-LEO-PPO rather than from the proposed changes.
- The computational savings are claimed as a contribution, but they follow from original CRL's design itself. CRL scores state-action embeddings against a goal embedding instead of keeping a separate head per goal, and none of the proposed modifications is responsible for this efficiency.
- There is no ablation within Dual-CRL-PPO, so the headline results cannot be attributed to the proposed components.
- Goals are binary masks read directly off the symbolic observation, so labels are free and exact. Evaluation is also limited to one benchmark family.
Questions for authors
- Which CRL variant serves as the teacher in Dual-CRL-PPO in Table 2? The captions of Tables 5 and 6 state that the absorbing class and Rao-Blackwellized targets are used without termination-discounted targets. Please report Dual-CRL-PPO with a plain CRL teacher and with each component added in turn. This would show whether contributions 1 and 2 matter for the result the paper leads with.
- The parameter counts in Tables 4 and 6 imply roughly 64 parameters per goal for CRL (10.64M to 11.22M) and roughly 44k for LEO (30.35M to 432.53M), which matches a 1024 × 43 head per goal. This gap comes from the factorized goal embedding described in the paper lines 314–315. Could the authors compare against LEO with a factorized goal head, so that CRL's contribution is separated from the factorization? This bears directly on contribution 3.
- The paper states that unseen goals can be scored with a forward pass through the goal encoder. The parameter growth suggests the goal encoder is effectively a per-goal lookup table. If so, how would an unseen goal receive a meaningful embedding?
- Lines 200–202 state that K_t is limited by the window length W. Table 8 lists a rollout length of 64 with γ = 0.995, so about 73% of the discount mass lies beyond the window. Does the window span rollout boundaries? If not, could this truncation explain CRL's difficulty with long-horizon goals such as coal and iron (Figure 5)?
- Lines 272–275 state that γ _term = γ recovers the goal-occupancy measure exactly. Yet success declines as γ_term approaches γ (Table 3), from 0.586 at 0.8 to 0.559 at 0.995. Why does the exact correction hurt goal success? Does this suggest the termination-discounted target is a tuning heuristic rather than a principled fix?
- The state-action embedding does not appear to be conditioned on the commanded goal. With Monte Carlo targets from behaviour data, the critic would then estimate goal reachability under average behaviour rather than under a policy pursuing that goal. LEO's Q-learning instead learns per-goal optimal values. Could this explain why the CRL teacher trails the LEO teacher?
- Does the "CRL" baseline use the original InfoNCE loss with sampled negatives (Eq. 2) or the full softmax (Eq. 3)? A comparison against original CRL would clarify how much the semantic-goal formulation itself contributes.
5 comments
Author Response (2/4)
W3 / Q1: Teacher variant and component ablation
The teacher in Table 2 uses co-satisfaction targets, the absorbing class, and Rao-Blackwellized (RB) targets, but no termination-discounted targets, as in Tables 5 and 6 (Tables 6 and 7 in the revised manuscript). We have clarified this in the caption of Table 2.
We ran the requested ablation (see general response for the full results). A plain CRL teacher achieves 0.792 on Craftax Classic and 0.240 on Full Craftax. Adding co-satisfaction targets improves success to 0.816 and 0.247. Adding the absorbing class maintains similar success rates (0.814 and 0.245) while reducing the student's lava deaths on Craftax Classic from 1.2% to 0.7%. Adding RB targets further improves success to 0.821 and 0.255, matching the configuration used in Table 2. Termination-discounted targets () reduce combat deaths on Craftax Classic from 30.3% to 23.3% at comparable success (0.814 vs. 0.821), while improving success on Full Craftax to 0.260.
These results show that each component improves either goal success or survival, and that the benefits extend to the final Dual-CRL-PPO policy rather than only the CRL critic.
Q2: LEO with a factorised goal head
The reviewer's interpretation of the parameter counts is correct. We ran the requested comparison using LEO's TD objective with CRL's factorised critic (FacLEO; see general response for the full ablation across embedding dimensions).
At , FacLEO matches CRL's critic architecture and parameter count, but substantially underperforms dense LEO on both Craftax Classic (0.576 vs. 0.650) and Full Craftax (0.122 vs. 0.163). With the same architecture, CRL achieves similar performance to FacLEO on Classic (0.564 vs. 0.576) and better performance on Full Craftax (0.158 vs. 0.122). Increasing FacLEO's embedding dimension to match dense LEO's parameter count does not close the gap, with success rates of 0.598 and 0.141, respectively.
We also tested FacLEO as a teacher for PPO. On Craftax Classic, replacing the dense critic in Dual-LEO-PPO with FacLEO reduces success from 0.838 to 0.804, below Dual-CRL-PPO (0.821). The efficiency-performance trade-off therefore depends on both the factorised architecture and the learning objective, rather than factorisation alone.
Q3: Goal encoder and unseen goals
The reviewer is correct. In the reported configuration, the critic uses a per-goal embedding table, which cannot produce embeddings for unseen goals. Our statement in Section 4.2 referred to CRL's architecture, which supports arbitrary goal encoders, whereas LEO's per-goal output heads cannot score unseen goals by construction (and rely on these heads for their performance, see Q2). We used an embedding table for the critic to minimise architectural differences in our comparison with LEO. The PPO baseline and the students of both dual methods already encode goals directly from their raw templates. We have clarified this distinction in the revised manuscript.
We also ran additional experiments replacing the critic's embedding table with an MLP that encodes either raw goal templates or parsed semantic features (category, entity, direction, count). Within Dual-CRL-PPO, both variants match or improve on the embedding table. On Craftax Classic, Dual-CRL-PPO achieves 0.821 and 0.828, respectively, compared with 0.821 using the embedding table. On Full Craftax, success increases to 0.264 and 0.268, respectively, compared with 0.255. The semantic-feature encoder thus matches Dual-LEO-PPO on Full Craftax (0.268). Standalone CRL performs similarly across all three encoders.
Unlike the embedding table, these encoders can produce embeddings for unseen goals from their features. We have added the encoder ablation to the revised manuscript and plan to evaluate generalisation to held-out goals in future work.
Author Response (3/4)
Q4: Window truncation
The window does not span rollout boundaries. Future offsets are sampled within the same 64-step rollout, so is limited by the remaining steps in the rollout or episode. Standalone CRL in Table 2 uses no additional offset cap, while the Dual-CRL-PPO teacher and observation-level configurations use . The reviewer is correct that approximately 73% of the discount mass lies beyond the 64-step window. However, the targets are renormalised over the observed offsets through the term in Equation 6. The critic therefore learns a truncated-horizon goal occupancy distribution rather than simply underestimating goal occupancy due to missing future steps.
To test whether truncation limits performance, we varied the offset cap for standalone CRL on Craftax Classic. Success increases from 0.500 at to 0.569 at 16 and 0.591 at 32. Increasing the cap further does not help, with success reaching 0.571 without a cap. Truncation therefore seems unlikely to explain the difficulty with coal and iron. These goals may instead receive little positive target mass because they are rarely achieved under the behaviour policy. This explanation is also supported by Figure 5: the standalone CRL variants perform similarly poorly on coal, iron, and diamond, whereas Dual-CRL-PPO achieves much higher success on these goals. We have added the offset-cap ablation to the appendix.
Q5: Termination discount
Success declines as approaches because the corrected goal occupancy measure also accounts for survival. At , the probability mass beyond episode termination is assigned to the absorbing class. This lowers the scores of actions associated with early death and makes the greedy policy more cautious. However, the success metric still counts goals achieved immediately before death. This explains why combat deaths in Figure 3 decrease from 44% to 18% of episodes and average episode length increases to 247 steps, even as goal success declines.
Our formulation can be viewed as an instance of transition-based discounting (White, 2017) [4], which allows a separate discount for terminating transitions. As , only the fatal action is penalised, recovering the absorbing-class correction. At , the targets recover the exact occupancy measure for anchors whose termination is observed within the window (see Q4 for the truncation effect). The termination discount can therefore be treated as a hyperparameter controlling how strongly the agent is penalised for actions leading to episode termination.
Q6: Monte Carlo occupancy vs. optimal goal values
The reviewer is correct that our Monte Carlo CRL critic estimates discounted goal occupancy under the behaviour policy, whereas LEO uses TD learning to approximate optimal goal-conditioned action values. Both learn from behaviour-policy data, but LEO bootstraps towards the highest estimated action value for each goal, which may help with rarely achieved goals. This dependence on the behaviour policy is known from prior work, and off-policy alternatives exist [5, 6].
Our FacLEO ablation shows that switching to TD learning with the same critic architecture does not consistently improve performance. CRL performs similarly to FacLEO on Craftax Classic (0.564 vs. 0.576) and better on Full Craftax (0.158 vs. 0.122; see Q2). As a teacher, Dual-FacLEO-PPO also performs worse than Dual-CRL-PPO on Classic (0.804 vs. 0.821). With uniform symmetric InfoNCE, Dual-CRL-PPO further matches Dual-LEO-PPO (0.841 vs. 0.838; see general response). The dependence on the behaviour policy may therefore explain part of the performance difference, but not all of it.
Author Response (1/4)
We thank the reviewer for their constructive feedback and for recognising the contributions of our semantic-goal formulation, the analysis of termination bias, and the scalability of CRL to larger goal spaces. We have revised the manuscript to address their comments and questions and incorporate the additional experimental results. Changes relevant to this review are highlighted in green.
W1 / W2: Performance gains and computational efficiency
The proposed modifications address different limitations of CRL, and their effects depend on the environment. Our experiments show that they successfully address the failure modes they were designed for. The absorbing class is effective when termination results from a single fatal action. In Gridworld Lava, it increases success from 0.635 to 0.935 (Table 1). Similarly, on Craftax Classic, lava deaths are eliminated when the absorbing class is included (Figure 3). Termination-discounted targets, on the other hand, address deaths that result from damage accumulated over multiple timesteps. They reduce combat deaths from 44% to 18% of episodes and increase average episode length to 247 steps. RB targets also reduce the number of environment steps needed to reach a given success threshold by 10-20% (Figures 7-8).
We agree that the teacher-student approach builds on Matthews et al. (2026) [1] and do not claim this mechanism as a new contribution. However, our new ablations show that the proposed CRL modifications also improve the final Dual-CRL-PPO policy. A plain CRL teacher achieves 0.792 on Craftax Classic and 0.240 on Full Craftax. Adding the co-satisfaction labels for semantic goals, absorbing class and RB targets improves success to 0.821 and 0.255, respectively (see Q1). With symmetric InfoNCE, Dual-CRL-PPO matches Dual-LEO-PPO on Classic (0.841 vs. 0.838; see general response).
We also agree that CRL's computational efficiency is a consequence of its underlying factorised architecture. Our contribution is demonstrating that CRL can be extended to open-ended environments with diverse semantic goals, that its limitations in episodic settings can be addressed, and that it can scale to thousands of goals while maintaining competitive performance. At 9,637 goals, Dual-CRL-PPO achieves comparable performance to Dual-LEO-PPO with 19$\times$ fewer parameters and 38$\times$ less network-training compute. Our factorised LEO ablation also shows that CRL's efficiency-performance advantage depends on both its factorised architecture and the contrastive learning objective (see Q2).
W4: Label availability and benchmark scope
Having access to exact goal-satisfaction labels is a useful property of Craftax, allowing us to check achievements directly from the symbolic state without human annotation or a learned reward model. This assumption is also common in goal-conditioned RL. For example, hindsight experience replay requires a function to determine whether a goal has been achieved (Andrychowicz et al., 2017) [2]. All baselines in our comparison, including LEO, use the same goal-satisfaction labels. Settings without directly checkable goals would require learned goal classifiers, which we leave for future work.
Our main evaluation is limited to one benchmark family, but covers two Craftax environments with substantially different difficulty and goal spaces ranging from 136 to 9,637 goals. We also use a separate Gridworld environment to isolate the termination bias (Figure 2). While many previous CRL benchmarks focus on individual task families such as navigation, manipulation, or locomotion (Bortkiewicz et al., 2025) [3], Craftax combines navigation, resource acquisition, crafting, and combat within the same environment. We chose Craftax because it also provides a strong, well-tuned baseline from Matthews et al. (2026) [1], enabling a direct comparison. Extending the evaluation to other open-ended environments remains future work.