- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper argues that exiting intrinsic motivation based method define novelty over states, which can be problematic as novel maps, change of pixels may trigger the intrinsic motivation. Hence the paper defined so called capability novelty, where intrinsic motivation rewards skills over novel observations. The paper used a simple count based method and test it in crafter environment, with additional evaluation on franka kitchen, meta world. Yet as a paper studies intrinsic reward, I don't see any comparison against existing methods.
Critical strengths
The paper's research question focus on examines whether novelty representations align with task progression. Reducing exploration dimensions for task irrelevant variation can be a meaningful direction.
Critical weaknesses
- The paper failed to define a systematic methodology. Instead it manually selects environment specific variables for all environments. Hence the results can either be interpreted as a task specific reward engineering, or in a way as an encoded expert knowledge in reward form.
- The word for capability is too strong. According to the definition of c, it is only representing different combination of inventories, which does not represent agent's capability or skills.
- No intrinsic motivation baselines. As there are abundant number of intrinsic motivation methods, including RND, ICM, NovelID, and more recent episodic bonus like E3B, EME [1], EDT[5] lack experiment against.
- Furthermore, I wonder how the author position their insight of 'capability novelty' against some unrelated intrinsic reward filtering methods like: Information gain based method that avoid reward states that contain no new information ([1], [2]), and epistemic uncertainty estimation method [3][6] that remove 'unpredictable' states that can be potentially harmful. Especially the case you have raised in figure 1. Situation A does not mean this scene does not worth explore at all (you consider it meaningless because you as an expert player, whereas the agent may have no knowledge of this scene) and Situation B with the axe, the agent will be capable of do a lot of new actions, which an intrinsic motivation method may provide continuous intrinsic reward)
ref: [1] Rethinking exploration in reinforcement learning with effective metric-based exploration bonus. [2] Information-theoretic safe exploration with gaussian processes. [3] Beyond Noisy-TVs: Noise-Robust Exploration Via Learning Progress Monitoring. [4] How to Stay Curious while Avoiding Noisy TVs using Aleatoric Uncertainty Estimation [5] Episodic novelty through temporal distance. [6] Self-Supervised Exploration via Disagreement
Questions for authors
Please try to address my listed weakness.
2 comments
Thank you for clarifying the role of manual inputs. I do acknowledge the empirical improvement on the environments, especially Crafter. However my central concern remains unresolved.
The paper's proposed design principles in section 3 (low cardinality, capability-gated, and persistent), but there is no methodology that translate these into a sufficiently concrete and reusable framework across environments. Applying this approach still require substantial domain specific design decisions. For example, applying your method in Kitchen, still require domain specific representation design.
Given the representation design cannot be generalized, the paper also lack ablation for their reward construction (change the mathematical equation for the reward design while keeping the representation).
Overall I find the design principles hard to be generalized, and the reward construction intuitive. Therefore I am keeping my score.
Thank you for your comments on baseline coverage and the role of domain knowledge. We would like to clarify that Section 6 and Table 7 of the submission report RND and three SimHash variants under the same base-agent configuration, 500k replay and 10M budget:
The submission therefore includes both count-based and prediction-error intrinsic baselines. We clarify their matching conditions and limitations below.
The collection-time latent comparator matches CFE's count formula, coefficient, cap, registration at collection, current-count lookup and nominal 14-bit capacity. Its three seeds have zero diamond success, versus 4.30%, 4.69% and 7.81% for CFE. The compared intervention is the key-construction pipeline; domain information and stationarity differ. The capacity match applies to this latent comparator, not the 22-bit pixel arm.
A completed comparison also gives both CFE and RND 2M replay, with the same base-agent architecture, achievement inputs/head, crafting prior and 10M budget. Scores are 67.67 ± 2.04 (n=3) and 46.94 ± 5.11 (n=5); diamond means are 21.88% and 0.55%. Thus equal replay capacity does not reproduce the full recipe's outcome for the tested RND implementation. RND coefficients 30/100/300 were explored only in short pilots, not a full-budget multiseed sweep, and the bonus distributions differ. The matched-count comparison provides separate evidence that does not depend on RND tuning.
The representation is manually specified. Given those variables, the algorithm is explicit: threshold inventory slots at “holds at least one,” count the joint binary configuration once per collected transition, and add 0.5 × min(1/sqrt(N(c)+1), 0.5) using the current lifetime count. The bonus adds no item-specific weights, recipe graph or target-achievement ordering. Variable selection supplies domain knowledge; the experiments test a particular way of turning it into an exploration reward.
Additional controls make this distinction concrete. Inventory-configuration, per-item and first-acquisition rewards read identical inventory variables. In the five-seed sample, two per-item policies learn iron swords at about 60% success while almost never making iron pickaxes; configuration counting more often reaches the mining branch. These exploratory outcomes do not resolve the ranking or isolate aggregation from reward timing and magnitude, but they characterize what different uses of the supplied knowledge learn.
“Capability configuration” denotes a resource proxy for feasible interactions; policy competence is evaluated separately through achievements. We will make this operational definition explicit. The contribution is the demonstrated final mining-chain behaviour and the controlled evidence for this resource-count construction.
A new scene can be informative, and acquiring a tool can create transitions rewarded by other intrinsic methods. Figure 1 will illustrate how the counting spaces partition observations, rather than label visual information worthless. Our conclusions concern the tested pixel/latent counting and RND implementations. Information-gain, learning-progress and uncertainty-based methods address related objectives; E3B, EME, EDT, ICM, NovelD and Disagreement remain additional comparisons, rather than methods shown to fail by our experiments.
Manual inputs and scaffold dependence limit transfer, which remains unresolved. We will also distinguish the observed benefit of replay expansion from the hypothesis that it prevents forgetting. The completed controls support the resource-count intervention within this stated setting, and the revision will make its information requirements and scope explicit.