acceptodds
Under review as a conference paper at ICLR 2027

Counting Capabilities, Not States: Capability Representations for Exploration in Open-Ended Worlds

Abstract

Intrinsic exploration bonuses reward novelty, and novelty is nearly always measured over states. In open-ended environments with deep achievement hierarchies, we find that the choice of novelty space is the dominant effect. We hold the standard count bonus fixed and vary only the space it counts. Every state-derived space (pixels, hashed features, prediction error) stays near the no-bonus floor, because appearance novelty is abundant yet orthogonal to progress: an agent can sightsee forever without gaining a capability. Counting the agent's capability configuration instead, the binary vector of which items it holds, read from the environment's inventory for counting only, turns the same bonus into a climbing signal: it is worth +15.7 points on a DreamerV3-class agent that already observes its achievement state, and +0.5 on the same agent stripped of that input and its crafting-action prior. With that scaffold, the agent sets a new state of the art among agents trained from scratch on Crafter (67.7 vs. the prior best 58.1 at 10M steps) and raises collection of the deepest milestone, the diamond, from 0.5% to 22% of episodes. A ladder of single-flag ablations attributes every point of the improvement and uncovers a second phenomenon: without extended replay retention the agent forgets its own breakthroughs: a rare deep-tier trajectory leaves the default buffer within steps, and holding it four times longer halves the relapses and adds 6.9 points. The same bonus lifts success on sparse-reward Meta-World pick-place from 0.22 to 0.83, and it is inert on dense-reward Meta-World and on Craftax, where the reward is already dense enough to learn from. Curiosity, in progression-structured worlds, should be about what you can do, not what you see.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, exploration, intrinsic motivation, count-based exploration, world models, open-ended environments, Crafter

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QrJaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that exiting intrinsic motivation based method define novelty over states, which can be problematic as novel maps, change of pixels may trigger the intrinsic motivation. Hence the paper defined so called capability novelty, where intrinsic motivation rewards skills over novel observations. The paper used a simple count based method and test it in crafter environment, with additional evaluation on franka kitchen, meta world. Yet as a paper studies intrinsic reward, I don't see any comparison against existing methods.

Critical strengths

The paper's research question focus on examines whether novelty representations align with task progression. Reducing exploration dimensions for task irrelevant variation can be a meaningful direction.

Critical weaknesses
  1. The paper failed to define a systematic methodology. Instead it manually selects environment specific variables for all environments. Hence the results can either be interpreted as a task specific reward engineering, or in a way as an encoded expert knowledge in reward form.
  2. The word for capability is too strong. According to the definition of c, it is only representing different combination of inventories, which does not represent agent's capability or skills.
  3. No intrinsic motivation baselines. As there are abundant number of intrinsic motivation methods, including RND, ICM, NovelID, and more recent episodic bonus like E3B, EME [1], EDT[5] lack experiment against.
  4. Furthermore, I wonder how the author position their insight of 'capability novelty' against some unrelated intrinsic reward filtering methods like: Information gain based method that avoid reward states that contain no new information ([1], [2]), and epistemic uncertainty estimation method [3][6] that remove 'unpredictable' states that can be potentially harmful. Especially the case you have raised in figure 1. Situation A does not mean this scene does not worth explore at all (you consider it meaningless because you as an expert player, whereas the agent may have no knowledge of this scene) and Situation B with the axe, the agent will be capable of do a lot of new actions, which an intrinsic motivation method may provide continuous intrinsic reward)

ref: [1] Rethinking exploration in reinforcement learning with effective metric-based exploration bonus. [2] Information-theoretic safe exploration with gaussian processes. [3] Beyond Noisy-TVs: Noise-Robust Exploration Via Learning Progress Monitoring. [4] How to Stay Curious while Avoiding Noisy TVs using Aleatoric Uncertainty Estimation [5] Episodic novelty through temporal distance. [6] Self-Supervised Exploration via Disagreement

Questions for authors

Please try to address my listed weakness.

2 comments

Thank you for your comments on baseline coverage and the role of domain knowledge. We would like to clarify that Section 6 and Table 7 of the submission report RND and three SimHash variants under the same base-agent configuration, 500k replay and 10M budget:

MethodScore (mean ± seed SD)Training seeds
RND43.4 ± 4.05
Pixel SimHash43.3 ± 4.55
Latent SimHash41.8 ± 1.05
Collection-time latent SimHash39.9 ± 1.83
CFE, inventory + workshop60.8 ± 0.63

The submission therefore includes both count-based and prediction-error intrinsic baselines. We clarify their matching conditions and limitations below.

  1. What the controls establish.

The collection-time latent comparator matches CFE's count formula, coefficient, cap, registration at collection, current-count lookup and nominal 14-bit capacity. Its three seeds have zero diamond success, versus 4.30%, 4.69% and 7.81% for CFE. The compared intervention is the key-construction pipeline; domain information and stationarity differ. The capacity match applies to this latent comparator, not the 22-bit pixel arm.

A completed comparison also gives both CFE and RND 2M replay, with the same base-agent architecture, achievement inputs/head, crafting prior and 10M budget. Scores are 67.67 ± 2.04 (n=3) and 46.94 ± 5.11 (n=5); diamond means are 21.88% and 0.55%. Thus equal replay capacity does not reproduce the full recipe's outcome for the tested RND implementation. RND coefficients 30/100/300 were explored only in short pilots, not a full-budget multiseed sweep, and the bonus distributions differ. The matched-count comparison provides separate evidence that does not depend on RND tuning.

  1. Domain knowledge, methodology and “capability.”

The representation is manually specified. Given those variables, the algorithm is explicit: threshold inventory slots at “holds at least one,” count the joint binary configuration once per collected transition, and add 0.5 × min(1/sqrt(N(c)+1), 0.5) using the current lifetime count. The bonus adds no item-specific weights, recipe graph or target-achievement ordering. Variable selection supplies domain knowledge; the experiments test a particular way of turning it into an exploration reward.

Additional controls make this distinction concrete. Inventory-configuration, per-item and first-acquisition rewards read identical inventory variables. In the five-seed sample, two per-item policies learn iron swords at about 60% success while almost never making iron pickaxes; configuration counting more often reaches the mining branch. These exploratory outcomes do not resolve the ranking or isolate aggregation from reward timing and magnitude, but they characterize what different uses of the supplied knowledge learn.

“Capability configuration” denotes a resource proxy for feasible interactions; policy competence is evaluated separately through achievements. We will make this operational definition explicit. The contribution is the demonstrated final mining-chain behaviour and the controlled evidence for this resource-count construction.

  1. Visual novelty and other intrinsic objectives.

A new scene can be informative, and acquiring a tool can create transitions rewarded by other intrinsic methods. Figure 1 will illustrate how the counting spaces partition observations, rather than label visual information worthless. Our conclusions concern the tested pixel/latent counting and RND implementations. Information-gain, learning-progress and uncertainty-based methods address related objectives; E3B, EME, EDT, ICM, NovelD and Disagreement remain additional comparisons, rather than methods shown to fail by our experiments.

Manual inputs and scaffold dependence limit transfer, which remains unresolved. We will also distinguish the observed benefit of replay expansion from the hypothesis that it prevents forgetting. The completed controls support the resource-count intervention within this stated setting, and the revision will make its information requirements and scope explicit.

Thank you for clarifying the role of manual inputs. I do acknowledge the empirical improvement on the environments, especially Crafter. However my central concern remains unresolved.

The paper's proposed design principles in section 3 (low cardinality, capability-gated, and persistent), but there is no methodology that translate these into a sufficiently concrete and reusable framework across environments. Applying this approach still require substantial domain specific design decisions. For example, applying your method in Kitchen, still require domain specific representation design.

Given the representation design cannot be generalized, the paper also lack ablation for their reward construction (change the mathematical equation for the reward design while keeping the representation).

Overall I find the design principles hard to be generalized, and the reward construction intuitive. Therefore I am keeping my score.

Reviewer_NubtOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a method that replaces conventional state-based novelty with novelty measured over discrete capability configurations. In Crafter, these configurations are represented by binary inventory vectors, encouraging exploration of previously uncommon item combinations rather than visually novel states. Using a standard count-based exploration bonus, the method improves performance over conventional novelty baselines and achieves a new state-of-the-art Crafter score when combined with additional environment information and extended replay retention. The authors also evaluate the approach in Meta-World, Franka Kitchen, and Craftax. While the experiments demonstrate the benefits of using task-relevant representations for exploration, the method relies on manually defined capability abstractions.

Critical strengths
  • The paper presents a simple and intuitive approach to exploration, demonstrating that the choice of representation can be more important than the novelty estimation mechanism itself.

  • The improvements on Crafter are substantial, particularly for difficult achievements such as collecting diamonds, where the method considerably outperforms previous approaches.

  • The authors provide extensive ablations that help isolate the contributions of inventory-based novelty, additional environment information, and replay-buffer retention.

Critical weaknesses
  • The methodological novelty is limited. Using task-relevant state abstractions, inventory information, or object-centric representations to guide exploration has already been investigated in prior work, including Go-Explore, Kanitscheider et al. (2021), and MEAD for instance.

  • The comparisons with conventional exploration methods are not entirely convincing. CFE benefits from a manually selected, task-relevant representation, whereas the baselines measure novelty over generic states or learned features. The experiments therefore do not establish whether capability representations are uniquely useful compared with other appropriately chosen abstractions.

  • The approach depends heavily on the structure of Crafter, where inventory configurations naturally correlate with progression through a technology tree. It is unclear whether these representations would be similarly useful in environments where collecting items or changing discrete variables does not correspond to meaningful progress.

  • The strongest improvements depend on additional scaffolding, including achievement-state observations and a crafting-action exploration prior. Without these components, the inventory-based bonus provides almost no improvement over the corresponding baseline.

  • The generalization experiments provide limited support for the broader claims. Meta-World uses manually selected task-progress indicators, Franka Kitchen shows no statistically resolved improvement in task performance, and Craftax shows no meaningful benefit. Furthermore, the authors' attempts to learn capability representations fail to reproduce the benefits of the manually defined representations.

Questions for authors
  1. What distinguishes capability-based novelty from novelty measured over any other task-relevant state abstraction? For example, how would counting achievement configurations or other low-dimensional representations compare with counting inventory configurations?

  2. How does the proposed method differ fundamentally from existing approaches that use inventory information or object-centric abstractions to guide exploration? In particular, what is the conceptual advance over Kanitscheider et al. (2021) and MEAD, beyond the application of a standard count bonus to a different representation?

  3. How well does the method generalize to environments where capability configurations are not directly correlated with task progress? For instance, would it still be effective when many collectible items are irrelevant to completing the task, or when the same inventory configuration permits substantially different actions depending on the environment state?

1 comment

Thank you for your questions about prior work, alternative abstractions and applicability. We clarify the additional findings and their scope below. Our contribution is an experimentally established use of resource information that produces substantial final success on Crafter's mining chain, with controls explaining what comparison supports that result. It gives subsequent exploration methods a concrete resource-informed reference under a specified learner, interface and budget.

  1. What is added beyond prior abstraction-based exploration?

We build on Tang's work on counting representations, Kanitscheider's item rewards and curricula, and MEAD's structured object abstractions. MEAD also includes Dreamer/PPO count-reward baselines over its abstract observation/action interface. The additional result here is the measured effectiveness, progression profile and learner dependence of a resource key used in the existing agent's reward calculation, retaining Crafter's native actions and the stated achievement scaffold.

The comparisons address several possible explanations for the outcome. Before workshop bits or enlarged replay, inventory-only counting raises final iron-pickaxe success from 0% to 41% (three seeds per arm). With workshop context and 500k replay, the collection-time latent comparator matches the count formula, scale, cap, registration/lookup schedule and nominal 14-bit capacity: CFE scores 60.77 ± 0.63 versus 39.89 ± 1.76 (n=3 each). This comparison tests the specified key-construction pipelines, including their different domain information and representation properties.

The complete recipe achieves 21.9% diamond success, with all three final policies between 7.4% and 33.2% at 10M steps. Its significance is completion of the long mining chain at substantial measured frequencies. Researchers gain an implemented training alternative and a concrete target for evaluating learned abstractions or new novelty objectives under the same information conditions. The controlled outcomes make this more informative than the general observation that task-relevant features can help.

  1. Achievement configurations and other informed rewards.

The requested comparison uses five seeds per arm, 10M steps, 500k replay and no workshop bits:

Count keyScore (mean ± seed SD)Iron pickaxeDiamond
Inventory configuration52.93 ± 8.9236.48%2.34%
Achievement configuration43.86 ± 6.3621.72%1.72%

Achievement counting also reaches the mining branch. The inventory-minus-achievement score interval is [-2.44, 20.58], and both arms have heterogeneous seed outcomes; the comparison does not establish inventory superiority. Current inventory and achievement history encode different information—consumption changes held resources without reversing achievements—so this is an informative alternative representation, not an identical-information control.

Our additional per-item and first-acquisition arms do read the same inventory variables. Their exploratory profiles show why the reward construction matters to the evaluation: two per-item seeds achieve about 60% iron-sword success but only 0.39% pickaxe success and no diamonds. The configuration arm more often reaches the mining branch in the observed sample. Ranking remains uncertain, and aggregation, event timing and bonus magnitude differ. These analyses characterize alternatives; the primary effectiveness evidence comes from the separate bonus-off and matched-count comparisons.

  1. Applicability.

The method uses manual inventory/workshop information, and its large benefit depends on the learner configuration: the minimal-stack gain is +0.5 and second-agent transfer remains unresolved. Workshop bits address one contextual ambiguity; irrelevant collectible items and arbitrary omitted context remain untested. Meta-World provides another positive manually represented case, while Kitchen and Craftax show no resolved task-score benefit. Automatic discovery of a useful key remains open.

Together, these results establish a useful resource-count approach for learning the mining chain and clarify its scope. We will revise the manuscript to make this contribution, the alternative-abstraction results, and the observed seed variability explicit.

Prior work: Tang — https://proceedings.neurips.cc/paper_files/paper/2017/file/3a20f62a0af1aa152670bab3c602feed-Paper.pdf Kanitscheider — https://arxiv.org/html/2106.14876v1 MEAD — https://arxiv.org/html/2408.11816v3

Reviewer_K1hrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how intrinsic exploration rewards should measure novelty in open-ended environments. The authors keep the standard count-based bonus but count binary combinations of the items the agent currently holds. In some settings, they also include information about nearby crafting tables. Combined with a world-model agent and a larger replay buffer, the method improves both the overall Crafter score and the diamond collection rate. The paper also examines the role of auxiliary components in the base agent and explores the method’s benefits and limitations through experiments on robotic tasks and Craftax, as well as tests with learned representations.

Critical strengths

Under the same agent configuration, inventory counting raises the score from 41.0 to 56.7 and substantially improves iron-pickaxe success. With the additional components, diamond collection also becomes more reliable, showing progress on difficult skills. The ablations are useful, and the limitations are clearly reported. The authors examine the contribution of individual components and report inconclusive results in other environments, as well as failures with learned representations. This helps clarify when the method is useful

Critical weaknesses

The gains depend on specific auxiliary components and access to structured environment information. The improvement is small when these components are removed, and a clear gain is not reproduced on the second agent. It therefore remains unclear how well the method transfers to other agents or environments. The results show that inventory-configuration counting outperforms the tested pixel and feature baselines, but do not establish whether counting complete item combinations is necessary. Do Per-item counting or first-acquisition rewards provide similar benefits?

Questions for authors

How does complete inventory-configuration counting compare with per-item counting and first-acquisition rewards under the same agent and training budget? The learned-representation experiments introduce additional reward terms and change the count-update rules. Do the learned representations still perform poorly when they use exactly the same bonus and counting rules as the explicit inventory representation?

1 comment

Thank you for suggesting these additional controls. We have completed the requested per-item and first-acquisition comparisons. At the tested budget, configuration counting has higher observed iron-pickaxe and diamond success, while per-item counting can learn advanced sword crafting. The results distinguish these outcomes, with the uncertainty reported below.

  1. Per-item and first-acquisition controls.

All arms use five training seeds, 10M steps, 256 final evaluation episodes, the same scaffold and 500k replay, without workshop bits. The three objectives read the same twelve thresholded inventory variables.

RuleScore (mean ± seed SD)Iron pickaxeIron swordDiamond
Inventory configuration52.93 ± 8.9236.48%20.94%2.34%
Per-item45.26 ± 4.430.39%24.69%0.00%
First acquisition43.06 ± 1.840.31%1.33%0.00%

The inventory-minus-comparator score differences have 95% Welch intervals [-3.30, 18.63] and [-1.10, 20.84], respectively. Five seeds do not resolve the ranking. Inventory has three seeds with 55–69% pickaxe success and two near zero; every per-item/first-acquisition seed is at or below 1.18%. The pickaxe comparisons also remain statistically inconclusive. Shared coefficients and caps do not equalize realized reward magnitude or timing.

Two per-item seeds achieve 59.77% and 61.72% iron-sword success, yet each has only 0.39% pickaxe success and no diamonds. Thus the exploratory behavioural distinction is specifically progression along the mining branch, rather than an inability to craft advanced items. We will report both cohorts and their seed-level variation separately, rather than generalize from the original three-seed SD.

  1. Exact-rule learned keys.

The historical learned-key experiments change reward terms and count updates, so they do not answer the exact-rule question. A completed strict comparison is not available. We will restrict the manuscript's conclusion to the learned implementations actually evaluated, rather than infer that learned representations fail under the explicit key's rule.

  1. Scaffold dependence and the supported contribution.

The +0.5 minimal-stack result and unresolved second-agent result limit portability; the noisy individual-removal effects do not identify the interaction mechanism. Within the shared full scaffold, however, inventory-only counting raises score from 41.0 ± 0.3 to 56.7 ± 1.2 (n=3 each) and final pickaxe success from 0% to 41%, before workshop context or enlarged replay.

This establishes a useful resource-count intervention in the stated learner. The requested controls add evidence about what alternative uses of the same information learn, while leaving their ranking open.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.