acceptodds
Under review as a conference paper at ICLR 2027

Reward-Free Exploration through Calibrated Ordering

Abstract

Reinforcement learning agents are often guided towards exploratory behavior with the help of intrinsic rewards. Maximizing the entropy of the state visitation distribution is a principled choice for such a reward. The objective pays the agent , where is the visitation density. However, density estimation in high-dimensional spaces is unreliable. Furthermore, the shape of the latent space from which distances are measured is crucial, as it determines the semantic meaning of proximity, yet there is little methodology for selecting or validating such spaces, and we are left relying on assumptions about their semantic structure. We introduce CALOR (CALibrated ORdering), which extracts from the latent space only a pairwise ordering among states, thereby relying less on the exact geometry of that space. A predictor network is trained to reproduce that ordering with a pairwise ranking loss, and at the same time to place its own outputs on a calibrated scale, which restores the reward magnitude that an ordering alone leaves undetermined. The construction accepts any source of pairwise rarity comparisons, which we demonstrate by adding a second ordering derived from how long the agent has stayed alive. On reward-free Craftax-Classic at one billion interactions, CALOR reaches a Crafter score of against for the strongest baseline, and unlocks every one of the achievements in every seed at least once. On three continuous-control exploration tasks it matches or exceeds every baseline, and on the humanoid maze it visits five times as many distinct states as the strongest of them. Ablations show that the ordinal signal and its calibration each contribute, and that neither on its own accounts for the result.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, exploration, intrinsic motivation, reward-free reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MJ9gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies intrinsic motivated exploration in reinforcement learning. The paper proposed Calibrated ordering, which build rarity ordering based on encoded latent space's nearest neighbor distance, and train the ordering relationship into a predictor h. Then combine rarity reward and rec reward for exploration motivation. They have conducted experiments in 2 environments, discrete action space environment CRAFT, and continuous action space mujoco based environment.

Critical strengths
  1. This experiment results are impressive, where the proposed method achieved significantly higher score in crafter.
  2. Detailed ablation, the author clearly reported the ablation study, where we can see the necessity of each components.
Critical weaknesses
  1. Rarity predictor h may fail under OOD observations. As a neural network based predictor, h's prediction seems only hold on trained data. If the deployment, for example enter a new map or a new cave, h's prediction maybe pure random, which may fail. This is important because exploration task particularly focus on motivating agent to achieve novel states, and this predictor h may frequently fail as the exploration goes into more novel states. So essentially means more success exploration -> more novel states -> more h OOD failure.

  2. The idea of providing intrinsic reward based on visitation counts is not novel, where a lot of count based method has used, and some of them also have adaption of neural network. For example EME (Rethinking exploration in reinforcement learning with effective metric-based exploration bonus.), and EDT (Episodic novelty through temporal distance.).

  3. Robustness to noise. As a visitation count based intrinsic motivation method, I am worried how this method may react to environment stochasticity. It is likely method will fail under any kind of aleatoric uncertainty.

  4. I first appreciate the paper use a recent baseline of C-tec. However all other baselines are too old. There are a number of intrinsic motivation methods recently.

Questions for authors

Please try to address my weakness.

3 comments
AuthorsOpenReview ↗

Addressing weaknesses

We would like to address the weaknesses you described, as requested.

  1. We agree, and it is why we wrote in the paper that we argue for combining curiosity-driven with rarity-based methods. Quoting the paper "curiosity offers a focused signal towards what is currently surprising, while rarity encourages the agent to spread out across known states". These OOD observations could be detected by a curiosity-driven mechanism which we do not claim to propose any novelty on in this paper. We use reconstruction loss as a very simple placeholder for the concept of "curiosity" which can be improved through other methods. Our ablations show that reconstruction indeed assists rarity, however rarity also succeeds to climb achievements on its own in this environment. Since CALOR attempts to normalize the visitation of all known states, this will often cause OOD states to return. As the craftax-classic results suggest, new discoveries (achievements) become rewarded by CALOR eventually even without reconstruction. Hopefully, a good curiosity method can assist whilst these new states are OOD. But novel states are learned over time, so pure curiosity methods may have diminishing returns (as the reconstruction-only ablation suggests). By combining both approaches we can gain the benefits from both.

  2. We did not intend to claim that visitation counts is a novel concept and we cite several existing count-based approaches in the paper as related work. However, CALOR proposes a novel approach, the ordering taken from the latent, but then the reward is calibrated to a fixed distribution. Regarding the methods you mention (EME and ETD) they could potentially compose well with CALOR. As mentioned in our Future Work section "Alternative sources of ordering" incorporating these alternative notions of distance as CALOR's source of ordering would be very interesting to investigate. So we now cite them also in related work, and ETD when discussing future work.

  3. To be precise, craftax-classic has aleatoric uncertainty (procedurally generated world different every episode, random animal movement), but the chosen latent (observation autoencoder) does not in these environments, by design. We chose to design in this way not because we saw any evidence it would be harmful for CALOR but because of the noisy-tv problem which is known to be problematic for curiosity methods. By removing the noise we can assume to have a simple but effective curiosity signal as our placeholder, removing the potential confound of noise from the comparisons. Furthermore, creating a noise-robust encoding is a separate problem that can be solved through other methods, so we did not feel the need to explore that question in depth. If the rarity signal is also sensitive to aleatoric noise then a noise-robust encoding should be preferred. Please note that craftax-classic has a high dimensional observation (1345), and latent size (512) which creates epistemic noise from the limitations of our models (see Appendix A.1). The sensitivity to noise remain a question that can be explored in future work, but it is not a question that needs to be resolved for CALOR to be a useful method.

  4. In the paper we opted to follow C-TeC's baseline set as the most recent published reward-free result on this benchmark. Given the large gap in score we did not expand or update this baseline set. Following your suggestion we have ported ETD (the more recent method of the two you mentioned) from the author's implementation into our codebase, and added it to our comparison for craftax-classic. It reaches a Crafter score of 1.7 ± 0.1 against 40.7 ± 2.7 for CALOR (Section 4, Tables 2-4), and the minimal adaptations needed for the port are listed in the Limitations and Appendix A.1.

I appreciate the author for their response. However I am still concerned that this paper as proposing an exploration method which was intended for acquiring new states, need an external curiosity method to handle OOD states. Hence I don't buy the idea and I am keeping my score.

We agree that h has no guarantee of producing an accurate rarity estimate on the first encounter with an OOD observation and explicitly present curiosity as a complementary signal for that reason. However, we respectfully disagree that CALOR therefore requires curiosity to acquire novel behavior. In the ablation that removes the reconstruction reward and episode-age ordering, leaving only calibrated latent-distance rarity, CALOR achieves a Crafter score of 24.5 ± 1.5, compared with 3.3 for the strongest existing baseline. Since the Crafter score is a geometric aggregation across the achievement hierarchy, this improvement requires broad exploration well beyond states reached by prior methods. Moreover, h is updated online, so newly encountered observations enter its training distribution and can subsequently be promoted by the rarity objective. The full score of 40.7 ± 2.7 shows that curiosity is beneficial and complementary, but the rarity-only result directly contradicts the claim that curiosity is necessary for CALOR to explore effectively.

Reviewer_ydcEOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

CALOR defines an intrinsic reward algorithm and showcases in a setting without any extrinsic rewards against strong baselines (C-TeC, ICM, RND, E3B/APT) in both discrete-action Craftax-classic with PPO and continuous action Ant Hardest Maze, Humanoid U Maze, and Arm Binpick Hard with SAC. In particular in the Craft-ax setting, CALOR shows order's of magnitude improvement against strong and recent baselines. The intrinsic reward is the sum of two terms: 1) an observation reconstruction error, 2) a rarity score trying to approximate rrar≈−log⁡(p(z))r^{rar}\approx -\log(p(z)) where p(z)p(z) is the probability density of seeing latent zz in the current experience sampling regime. The main contribution of the paper is approximating p(z)p(z) using pairwise ordering of the density by equating density ordering with inverse nearest neighbour distance (in latent space) ordering, and then a calibration loss fitting any sampling of densities p(z1)=p1≥p2≥…≥pn=p(zn)p(z_1) = p_1 \geq p_2 \geq \ldots \geq p_n=p(z_n) (where the indices of the zz's are chosen, without loss of generality, to be such that densities are decreasing) to a power law: pk/p1=k−(α+1)p_k/p_1 = k^{-(\alpha+1)}.

Critical strengths
  • The paper shows empirically that their method performs orders of magnitude better than legitimate strong baselines on the hard exploration task of Craftax.
  • The paper finds a creative and ingenious way to find a proxy for the hard estimation task of p(z)p(z).
Critical weaknesses
  1. Section 3, the methods section, seriously lacks in precision and clarity. Thankfully the method is also given in Algorithm 1. Many things are ill-defined such as the rarity score and the calibration loss. That is even more true when one considers the added modifications of Section 6. What is the calibration loss when there are 2 orderings (i.e. when the age-ordering is added)? This lack of precision puts reproducibility in doubt.

  2. The metric for the continuous control tasks could be better explained. As it is, it is quiet unclear what this metric means and whether it can be trusted to actually represent good exploration.

Questions for authors

Clarification requests:

  1. In the introduction, the paper rightly states that "A latent space is free to stretch one region and compress another, and its decoder can adapt at no cost, so the resulting density may not reflect the true rarity of states." Implicitely suggesting that CALOR will not be suffer from that. However: "Similar to prior work (Liu & Abbeel, 2021; Seo et al., 2021), CALOR obtains rarity comparisons from nearest-neighbor distances in a batch of on-policy observations", this is vulnerable to random scaling of latent dimensions too. So something is needed to make the narrative coherent.
  2. It would be really helpful to put the Burges et al. 2005 ranking loss directly in the paper, it takes only 1 line and will make the paper much more smoothly readable.
  3. The paragraph of lines 165-168 is unclear. What Pareto rarity scale? Please define mathematically, this is a core part of the algorithm and is critical for reproducibility. The calibration is not well defined.
  4. At line 177 in rrarr^{rar} the exponent −1/α-1/\alpha is presumably at the wrong place (missing brackets).
  5. In the case where episode age is used, there are two distinct orderings, so which one is used for calibration? This is not clearly stated.
  6. Algorithm 1: λcal\lambda_{cal} and λord\lambda_{ord} at line 18 are not defined. The meaning of ii not defined in lines 4-7. The rarity-ordering proxy step at lines 13-14 should be explained more clearly.
  7. Method, last paragraph (Section 3.3): the word “codec” might be ambiguous and confusing to some readers as in many domains where the word has a different meaning. A short definition would help since it is used several times later in the paper. Also, this paragraph explains components that are not in Algorithm 1 which can be confusing.
  8. Consider giving a version of the algorithm (maybe in appendix) with all that is actually included to make it work.
  9. The metric for the continuous control tasks could be better explained. As it is, it is quiet unclear what this metric means and whether it can be trusted to actually represent good exploration.

I am more than open to increasing my score if the above points are satisfactorily addressed. This paper can be a fantastic and innovative paper.

Suggestions:

  1. λage−ord=10λord\lambda_{age-ord} = 10 \lambda_{ord} is used for Craftax, strongly suggesting that age-ordering is actually doing most of the work in the case of Craftax. In order to show whether this is or is not the case, an ablation using age-ord but not ord should be performed.
6 comments
AuthorsOpenReview ↗

Clarifications and suggested update

We will address your questions below. One small clarification: CALOR does not estimate the density p(z) (or -log p(z)) so it does not assume a power law on them, instead it calibrates the rewards to follow an Exponential distribution of mean 1 by calibrating P to Pareto and using rarity reward α * log(P). We hope the new presentation makes it clearer.

Q1. We now state in the Introduction how CALOR reduces but does not remove this dependence: only ranks are used, which are invariant to monotone transformations of distances, and the reward magnitude comes from the calibration. CALOR is also insensitive to scaling of individual latent dimensions, because each latent feature is standardized before distances are computed (Section 3.3).

Q2. Added as Eq. 2.

Q3. Added in Section 3.2 (Eqs. 3-4) and Section 3.3.

Q4. Correct, we added the brackets.

Q5. Neither. The calibration loss sorts the predictor's own outputs and uses no external ordering. This is now explicitly explained in Section 6.

Q6. Algorithm 1 now defines i, links lambda_cal and lambda_ord to Eq. 4, and comments lines 13-14 (small distance = dense, permutation from densest to rarest) for clarity.

Q7. Now we define codec explicitly in Section 3.2. The end of Section 3.3 names what Algorithm 1 omits and where it is specified.

Q8. We added the full algorithm with everything included as Appendix B.

Q9. True, we added a paragraph "Coverage metric" to Section 5 and include more details in Appendix A.2. It is C-TeC's own metric and code and we now explain it. Simplifying, it discretizes space into cells of side 0.5, then counts how many were reached during training. This metric cannot be gamed as it only considers physically reached positions by the agent, and is identically computed for all methods. For Arm Binpick Hard it's the cube's position that counts, not the arm.

Suggestion. Thank you, this was a valid concern, and the ablation now appears in Section 7, Figure 7 and Table 6. With only episode age the Crafter score is 15.1 ± 10.5, about the same as with no ordering at all (14.9 ± 4.8). Only nearest-neighbor distance gives 27.6 ± 1.0 and both give 40.7 ± 2.7, so episode age is not doing most of the work. The weight of 10 was chosen only because 1 made no visible difference while 10 did. We did not try other values.

Reviewer_ydcEOpenReview ↗

Response to answers to questions

Q1

CALOR is also insensitive to scaling of individual latent dimensions, because each latent feature is standardized before distances are computed.

W1: A sentence like this in the paper would make the whole thing clearer. Please add that sentence in 3.3.

Since insensitive to scaling of latent is a key claim of the method, the part of the method which makes that happen should be highlighted clear and centre, hence W1 above. Furthermore it also needs to be in the Algorithm 1, otherwise the algorithm does not posses this claimed property.

W2: Add standardization of z to Algorithm 1.

Q2 ✅ Q3 ✅ Q4 ✅ Q5 ✅ Q6 ✅ Q7 ✅ Q8 ✅ Q9 ✅

If the above W1 and W2 above are implemented, I will increase my score to 4.

W1. Section 3.3 already described the standardization: "Before computing nearest-neighbor distances or applying the rarity predictor, each latent feature is standardized by its running mean and standard deviation, while the decoder continues to receive the unstandardized code." To highlight this, we have now extended that sentence stating that this makes CALOR insensitive to the scale of individual latent dimensions.

W2. Agreed, we have added the standardization to Algorithm 1 (statistics μ and v). To keep the notation of Algorithms 1 and 2 consistent, Algorithm 2 now uses the same constants βz and ϵv as Algorithm 1. They are defined in the hyperparameter table in the appendix and referenced in Section 3.1. We also added a comment to Algorithm 1 "decoder uses unstandardized zi" which also highlights this difference between the decoder and the predictor.

Reviewer_6thWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces CALOR, a reward-free exploration method that learns to rank states by their relative rarity rather than directly estimating their visitation density. The authors obtain pairwise rarity comparisons from nearest-neighbor distances in a learned latent space and train a predictor to reproduce this ordering. A calibration objective then maps the predicted rarities onto a fixed reward distribution, providing a stable intrinsic reward signal. This is combined with a reconstruction-based curiosity reward to encourage exploration. The method is evaluated on Craftax-Classic and three continuous-control environments, showing substantial improvements over existing exploration methods, particularly in Craftax-Classic and Humanoid U Maze. Extensive ablations further demonstrate the importance of the ranking and calibration objectives.

Critical strengths
  • The core idea of learning a relative ordering of states rather than relying directly on density estimates is interesting and well motivated. The combination with reward calibration provides a relatively simple and principled exploration objective.

  • The experimental results are impressive, particularly on Craftax-Classic, where CALOR substantially outperforms the baselines and consistently discovers all 22 achievements across seeds.

  • The evaluation covers both discrete and continuous-control environments, using different reinforcement learning algorithms. This provides useful evidence that the method is not limited to a single exploration setting.

  • The authors provide extensive ablation studies that help explain the contributions of the individual components. In particular, the importance of both ranking and calibration is demonstrated convincingly.

  • The paper is generally well structured, and the authors are transparent about implementation differences and potential limitations in their experimental comparisons.

Critical weaknesses
  • Although the method is motivated by the difficulties of density estimation in high-dimensional latent spaces, the experiments provide limited evidence that CALOR can effectively address these difficulties. The continuous-control environments use structured physical-state observations rather than high-dimensional visual inputs.

  • The approach still relies on nearest-neighbor distances in the learned latent space to establish its ordering. While ranking may be less sensitive to the precise magnitude of these distances, it remains dependent on the quality of the learned representation. In high-dimensional latent spaces, where fine-grained distance comparisons can become particularly unreliable, it is unclear whether the resulting ordering would remain meaningful.

  • The continuous-control experiments measure coverage through low-dimensional position coordinates. While these experiments demonstrate successful spatial exploration, they provide limited insight into how well the method explores more complex aspects of the underlying state space.

Questions for authors
  1. How well does CALOR scale to higher-dimensional observation spaces, particularly those involving visual observations or irrelevant features? Given that the method still relies on nearest-neighbor distances to construct its ordering, it would be interesting to understand how robust that ordering is as the dimensionality and complexity of the learned representation increase.

  2. Have the authors considered directly evaluating the robustness of the learned ordering to distortions of the latent space? For example, varying the latent dimensionality or introducing irrelevant observation features could help establish whether ranking offers a meaningful advantage over direct distance-based intrinsic rewards.

2 comments
AuthorsOpenReview ↗

Response to questions

Below we address your two questions, which also cover the weaknesses you raised.

  1. We agree this concern applies to the continuous-control experiments, whose states are low-dimensional. Craftax-Classic, however, has a 1345-dimensional symbolic observation. We encode it into a 512-dimensional latent (32 for continuous control). Our encoder is a plain MLP with no convolutional prior, so it does not learn to encode rare tiles equally well at every map position, and the latent therefore contains some epistemic noise. The rarity ordering is measured from this imperfect codec, which is evidence that the ordering stays useful in a moderately high-dimensional, noisy latent space. During development we also briefly tried pixel observations with a convolutional encoder and saw comparable performance. These were exploratory runs that we did not keep. We stayed with the plain MLP on symbolic observations to remain close to the setup of prior benchmarks, and because we see the resulting representation noise as a realistic part of the test rather than something to engineer away.

  2. We did not modify the environment with artificial distractor features. During development we tried latent sizes of 256, 512 and 1024 without a noticeable difference, so we did not include this as a formal ablation. CALOR also works across a much larger change in latent size between our two settings: 512 on Craftax-Classic and 32 in continuous control. We have also added a paragraph to the Introduction on how CALOR reduces, but does not remove, its dependence on the latent geometry.

Regarding comparison with direct distance-based rewards, in continuous control APT is the closest such method, a nearest-neighbor state-entropy reward computed directly from latent distances. On Craftax-Classic, E3B derives its bonus directly from the geometry of a learned embedding. In addition, the ablations in Section 7 show that removing either the ordering loss or the calibration loss substantially lowers performance.

On the position-based coverage metric in continuous control, we agree that it says little about other aspects of the state. We now describe exactly what it measures in a new "Coverage metric" paragraph in Section 5 and in Appendix A.2. This is also one reason the main emphasis of the paper is on Craftax-Classic, which includes non-spatial features such as crafting and inventory.

Following your comments, we have stated the observation dimensionality explicitly in Appendix A.1.

Dear Authors,

thank you for your clarifications. My main concern regarding the dimensionality is not whether CALOR can actually handle high dimensional vectors themselves, but rather the observation dynamics of the environment itself. For instance, a 3D egocentric view produces a completely different state space where similar states can be hard to distinguish and are not as "clear cut" as in Craftax.

Considering that the method depends on reliable ordering, this would be a crucial experiment for me to assess the generality of CALOR. Nonetheless I think the method as it currently stands is solid and interesting. I would have no problems with it being accepted.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of main paper changes after revision

We thank all reviewers for their time and feedback. The main changes in the revised PDF are:

  • ETD added as a Craftax-Classic baseline (Section 4, Tables 2-4), and EME cited in Related Work.
  • A new ablation separating the two ordering sources (Section 7, Figure 7, Table 6).
  • All losses written out explicitly (Eqs. 1-6) and the full algorithm in a new Appendix B.
  • A description of the continuous-control coverage metric (Section 5, Appendix A.2).

Details are in the individual responses to each reviewer.

Dear Reviewers,

We're the half-way through the discussion phase. The authors have provided their response to your reviews. Please engage with the authors as soon as possible, we need to keep the conversation going!

Many thanks,

AC

yes! done.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.