- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies intrinsic motivated exploration in reinforcement learning. The paper proposed Calibrated ordering, which build rarity ordering based on encoded latent space's nearest neighbor distance, and train the ordering relationship into a predictor h. Then combine rarity reward and rec reward for exploration motivation. They have conducted experiments in 2 environments, discrete action space environment CRAFT, and continuous action space mujoco based environment.
Critical strengths
- This experiment results are impressive, where the proposed method achieved significantly higher score in crafter.
- Detailed ablation, the author clearly reported the ablation study, where we can see the necessity of each components.
Critical weaknesses
-
Rarity predictor h may fail under OOD observations. As a neural network based predictor, h's prediction seems only hold on trained data. If the deployment, for example enter a new map or a new cave, h's prediction maybe pure random, which may fail. This is important because exploration task particularly focus on motivating agent to achieve novel states, and this predictor h may frequently fail as the exploration goes into more novel states. So essentially means more success exploration -> more novel states -> more h OOD failure.
-
The idea of providing intrinsic reward based on visitation counts is not novel, where a lot of count based method has used, and some of them also have adaption of neural network. For example EME (Rethinking exploration in reinforcement learning with effective metric-based exploration bonus.), and EDT (Episodic novelty through temporal distance.).
-
Robustness to noise. As a visitation count based intrinsic motivation method, I am worried how this method may react to environment stochasticity. It is likely method will fail under any kind of aleatoric uncertainty.
-
I first appreciate the paper use a recent baseline of C-tec. However all other baselines are too old. There are a number of intrinsic motivation methods recently.
Questions for authors
Please try to address my weakness.
3 comments
I appreciate the author for their response. However I am still concerned that this paper as proposing an exploration method which was intended for acquiring new states, need an external curiosity method to handle OOD states. Hence I don't buy the idea and I am keeping my score.
We agree that h has no guarantee of producing an accurate rarity estimate on the first encounter with an OOD observation and explicitly present curiosity as a complementary signal for that reason. However, we respectfully disagree that CALOR therefore requires curiosity to acquire novel behavior. In the ablation that removes the reconstruction reward and episode-age ordering, leaving only calibrated latent-distance rarity, CALOR achieves a Crafter score of 24.5 ± 1.5, compared with 3.3 for the strongest existing baseline. Since the Crafter score is a geometric aggregation across the achievement hierarchy, this improvement requires broad exploration well beyond states reached by prior methods. Moreover, h is updated online, so newly encountered observations enter its training distribution and can subsequently be promoted by the rarity objective. The full score of 40.7 ± 2.7 shows that curiosity is beneficial and complementary, but the rarity-only result directly contradicts the claim that curiosity is necessary for CALOR to explore effectively.
Addressing weaknesses
We would like to address the weaknesses you described, as requested.
We agree, and it is why we wrote in the paper that we argue for combining curiosity-driven with rarity-based methods. Quoting the paper "curiosity offers a focused signal towards what is currently surprising, while rarity encourages the agent to spread out across known states". These OOD observations could be detected by a curiosity-driven mechanism which we do not claim to propose any novelty on in this paper. We use reconstruction loss as a very simple placeholder for the concept of "curiosity" which can be improved through other methods. Our ablations show that reconstruction indeed assists rarity, however rarity also succeeds to climb achievements on its own in this environment. Since CALOR attempts to normalize the visitation of all known states, this will often cause OOD states to return. As the craftax-classic results suggest, new discoveries (achievements) become rewarded by CALOR eventually even without reconstruction. Hopefully, a good curiosity method can assist whilst these new states are OOD. But novel states are learned over time, so pure curiosity methods may have diminishing returns (as the reconstruction-only ablation suggests). By combining both approaches we can gain the benefits from both.
We did not intend to claim that visitation counts is a novel concept and we cite several existing count-based approaches in the paper as related work. However, CALOR proposes a novel approach, the ordering taken from the latent, but then the reward is calibrated to a fixed distribution. Regarding the methods you mention (EME and ETD) they could potentially compose well with CALOR. As mentioned in our Future Work section "Alternative sources of ordering" incorporating these alternative notions of distance as CALOR's source of ordering would be very interesting to investigate. So we now cite them also in related work, and ETD when discussing future work.
To be precise, craftax-classic has aleatoric uncertainty (procedurally generated world different every episode, random animal movement), but the chosen latent (observation autoencoder) does not in these environments, by design. We chose to design in this way not because we saw any evidence it would be harmful for CALOR but because of the noisy-tv problem which is known to be problematic for curiosity methods. By removing the noise we can assume to have a simple but effective curiosity signal as our placeholder, removing the potential confound of noise from the comparisons. Furthermore, creating a noise-robust encoding is a separate problem that can be solved through other methods, so we did not feel the need to explore that question in depth. If the rarity signal is also sensitive to aleatoric noise then a noise-robust encoding should be preferred. Please note that craftax-classic has a high dimensional observation (1345), and latent size (512) which creates epistemic noise from the limitations of our models (see Appendix A.1). The sensitivity to noise remain a question that can be explored in future work, but it is not a question that needs to be resolved for CALOR to be a useful method.
In the paper we opted to follow C-TeC's baseline set as the most recent published reward-free result on this benchmark. Given the large gap in score we did not expand or update this baseline set. Following your suggestion we have ported ETD (the more recent method of the two you mentioned) from the author's implementation into our codebase, and added it to our comparison for craftax-classic. It reaches a Crafter score of 1.7 ± 0.1 against 40.7 ± 2.7 for CALOR (Section 4, Tables 2-4), and the minimal adaptations needed for the port are listed in the Limitations and Appendix A.1.