acceptodds
Under review as a conference paper at ICLR 2027

Scaling Empowered Agents

Abstract

Empowerment measures the capacity of an agent to actively control its environment. In this work, we explain the connection between skill learning and empowerment. While skill learning is not empowerment, we introduce a simple scalable recipe for empowerment maximization: empowerment corresponds to the number of executable skills from a state, so maximizing empowerment corresponds to reaching states that are good starting points for many skills. Our recipe applies to arbitrary underlying skill learning methods. Whereas most prior work on empowerment looks at tabular settings, this recipe enables us to study empowerment at a scale where we can realize theoretically-predicted but empirically-elusive properties of empowerment (e.g., information gathering, survival and homeostasis). Experiments show that empowerment maximization unlocks interesting emergent properties, such as building a house in Crafter. Code and videos at https://anonymous. 4open.science/w/scaling-empowerment-iclr-1056/.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, exploration, empowerment, intrinsic motivation, unsupervised

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_4YXkOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies empowerment. The empowerment itself and the connection between skill learning and empowerment are interesting. The paper proposes a new method for empowerment maximization. The paper explains the empowerment to reaching states that are good starting points for many skills. The paper connects the unsupervised skill discovery with the empowerment. The paper uses scalable skill learning methods to estimate the empowerment of a particular state. The paper also provides interesting experiments.

Critical strengths

The motivation that reaching an empowered state is clear.

The definition is formalised well and clear.

The paper uses multiple environments to support the proposed method.

Critical weaknesses

To reach states that have more options may bring more unuseful behaviors. The paper may need to provide explanation.

It would be better to explain more about the connection between unsupervised skill learning and the proposed method. For example, there is a relevant work missing: TASK ADAPTATION FROM SKILLS: INFORMATION GEOMETRY, DISENTANGLEMENT, AND NEW OBJECTIVES FOR UNSUPERVISED REINFORCEMENT LEARNING. ICLR 2024

Questions for authors

What is the main difference between the proposed method and previous unsupervised skill learning methods?

Why not include some skill learning baselines?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 4YXk (1/2)

Thank you for the thoughtful review. Below we address delayed adaptation (new section §5.4, Table 6), task-adaptation related work (new in §2), and 2B-step Craftax baselines. Please let us know if you have any more questions or concerns, and we would be happy to address them.

Do states with more options bring more unuseful behaviors?

Thank you for the suggestion! This led to some new, interesting results.

Consider the "delayed adaptation" setting when an agent must prepare to zero-shot adapt to a random task revealed later in a rollout (Section 5.4, Fig. 6). We show that SEE's potential empowerment policy finds states that are good for future adaptation, while SEE's effective empowerment policy can do zero-shot nonparametric adaptation through skill selection (Table 6). Our added experiments show that when SEE executes the potential empowerment policy for 100 steps without knowing the task, and then we reveal one of the 12 inventory-based tasks, the effective empowerment policy πeff\pi_{\text{eff} } can adapt to the revealed task 56.2±1.456.2 \pm 1.4% of the time. Furthermore, even for alternative baseline skill learning methods (CSF and METRA), running SEE's potential policy for 100 steps before the task is revealed improves their zero-shot adaptation success rates, when compared with a no-op initialization or a random action initialization. Thus, SEE's potential empowerment policy positions the agent to be able to do many possible tasks in the future, without knowing what will be asked.

The full preparation-by-skill comparison is given in our third response below (Table 6).

Connection to unsupervised skill learning & Yang et al. (ICLR 2024)

We thank the reviewer for referring us to Yang et al. [1], which we now discuss in Related Work (§2) under "Adaptation to Downstream Tasks". This work is particularly relevant to our new adaptation results, which the authors link to mutual information skill learning. While we do not implement LSEPIN, WSEP, and PWSEP as baselines (for WSEP and PWSEP, due to practical difficulties of scalably estimating Wasserstein distances), we believe that alternative formulations of skill learning like those proposed in their paper are necessary to develop stronger adaptation abilities.

Main difference from prior unsupervised skill learning methods

Prior skill methods (DIAYN [2], DADS [3], METRA [4], and CSF [5]) optimize skill-diversity objectives related to effective empowerment, i.e., the amount of influence the agent exerts on the environment. SEE learns to maximize potential empowerment [6], [7], finding states where agents accumulate power without "spending" it.

These two forms of empowerment can be very different: in a casino, effective empowerment learns to pull all the slot machines at random until it goes bankrupt (maximum diversity of behavior); potential empowerment finds a game that you can win and makes money.

The problem is that to learn to maximize potential empowerment, you need to try a bunch of games. Our insight is that learning both policies and switching from potential to effective empowerment maximization lets us learn the potential-maximizing behavior.

As Table 6 shows, this state-preparation behavior is not explainable by unsupervised skill learning alone and improves the zero-shot adaptation under uncertainty of past skill-learning methods like CSF and METRA, which optimize skill-diversity objectives rather than a separate preparation policy.

Success rates (%) after 100 task-blind preparation actions followed by up to 900 actions of a frozen, task-selected skill (mean ±\pm seed standard error across three training seeds and 128 worlds per task per seed, averaged over 12 inventory tasks):

Skill policy100 SEE πpot\pi_{\text{pot} } actions100 no-op actions100 random actions
SEE πeff\pi_{\text{eff} }56.2±1.456.2 \pm 1.4——
CSF65.4±3.465.4 \pm 3.443.8±2.543.8 \pm 2.539.6±1.339.6 \pm 1.3
METRA63.8±3.863.8 \pm 3.843.3±4.243.3 \pm 4.239.6±2.739.6 \pm 2.7

Why not include some skill learning baselines?

In matched 2B-step Craftax-Classic PQN runs (Fig. 2, Tables 5 and 8, §C.3), SEE reaches 7.78±0.347.78 \pm 0.34 achievements (35.435.4%), beating CSF (6.05±0.336.05 \pm 0.33), METRA (6.02±0.146.02 \pm 0.14), Skills-only SEE (4.17±0.324.17 \pm 0.32), and RND [8] (3.20±0.053.20 \pm 0.05).

Dear Reviewer 4YXk,

Thank you for your thoughtful feedback. We have worked hard to incorporate your feedback with new adaptation experiments, new baselines, and edits to our figures and main text.

Have our revisions and response sufficiently resolved your questions and concerns? We appreciate your engagement as the end of the discussion phase nears. Thank you.

Thank you for the response. I updated score. The new version seems to be good to me.

Reviewer_GW2TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SEE to maximize empowerment by exploiting the connection between skill learning and empowerment. A skill policy estimates a lower bound on state-wise potential empowerment and a potential empowerment policy moves the agent toward states from which it can exert control (maximizing e_pot). Experiments are run on BuilderBench cubes, Ant locomotion, Ant Soccer, and Crafter.

Critical strengths

Timely and well motivated question. Empowerment has been frequently invoked, but most often in toy settings, approaches that scale to high dimensional state spaces like Crafter are always needed. The authors clearly describe the link between skill learning and empowerment.

The algorithm works in Ant continuous control settings, the potential policy navigates the ant to the center or the ball depending on the task.

Full hyperparameter tables are included in appendix to aid in reproducibility

Critical weaknesses

Craftax-Classic is the largest and most open-ended task of the ones presented in the paper and it is the environment used when citing the emergent behavior claim (house building as mentioned in the abstract). However, no external baselines are run in this environment (they cite a number 5.5 achievements from a different paper). The emergent behaviors are only supported by a few frames in Figure 5. Figure 6 shows when achievements occur on a single seed in a single episode. Figure 1 should also include intrinsic motivation and skill learning baselines (RND, CSF, METRA, etc). Can you also report the achievement success rate table?

In Craftax there is also no skills only control. Since the potential policy is run for 100 steps and a sampled skill for 900, how do we know the emergent behaviors are the result of empowerment maximization rather than the CSF-style skills alone? The no switching ablation is not sufficient to answer this, it still trains and runs the potential policy (Figure 10b " Achievements per two-phase rollout" in caption which implies the 2-phase rollout is still used). Moreover, in Figure 10b it looks like the no-switching variant matches SEE's achievement count for the first 1B steps. Please report achievements split by phase, averaged over seeds and episodes. Also, the proper comparison should be launching the same skills with randomly sampled z from 1) pi_pot's switch state 2) the state after 100 random action steps 3) the reset state and then comparing achievement counts.

Questions for authors

What is the intended use case for SEE’s outputs? Can you show its outputs improve downstream performance in some way? SEE’s skills condition on the switch state, do these make the skills more general and thereby useful?

How important is the choice of variables over which empowerment is defined in influencing agent behavior? In Appendix B it was reported that the difference between Ant Soccer and Ant Locomotion is that in Ant Soccer only the Ball obs is used as the state for empowerment. How much of Craftax's behavior (and number of achievements unlocked) is dictated by the choice of full inventory as a state?

Two of the emergent behaviors cited in the abstract "survival and homeostasis" (line 18) seem like behaviors any trained policy (with non negative per step reward) would learn to do well in crafter, it is not specific to the empowerment objective. Can you quantify these against policy learning (non empowerment) baselines? How was homeostasis measured?

4 comments
AuthorsOpenReview ↗

Response to Reviewer GW2T (1/2)

Thank you for your thoughtful feedback. We have added baselines, evaluations, and experiments and edited the manuscript based on your suggestions. Please let us know if you have any remaining concerns. We look forward to continuing the discussion.

Craftax-Classic baselines (RND, CSF, METRA) & multi-seed achievement rates

We implemented RND [1], CSF [2], and METRA [3] on SEE's base recurrent architecture [4] at a matched 2B-step budget (Fig. 2; Appendix Tables 5 & 8–10, §B.2.2, §C.3, §D). Across 5 training seeds for SEE, CSF, and METRA and 3 seeds for RND (128 fresh worlds/seed; Table 5), SEE reaches 7.78±0.347.78 \pm 0.34 achievements (35.4±1.535.4 \pm 1.5% of 22), beating CSF (6.05±0.336.05 \pm 0.33, 27.5±1.527.5 \pm 1.5%), METRA (6.02±0.146.02 \pm 0.14, 27.4±0.627.4 \pm 0.6%), and RND (3.20±0.053.20 \pm 0.05, 14.5±0.214.5 \pm 0.2%).

Single-rollout visualizations (Fig. 6)

Yes, we clarified in the captions that Figures 4 and 7 (submitted Fig. 6) show branches from one seed, and Figure 5b shows a single rollout (with the other four seeds in Figure 11).

Pure skills-only control without potential policy in Craftax

Thank you for the suggestion. We have replaced the ablation without switching in Figure 10 with the skills-only ablation at 2 billion steps. The skill method gets to 4.17 while SEE gets to 7.78, showing that the initial potential empowerment phase is important for final performance.

Achievements from potential switch state vs. 100 random steps vs. reset

Thank you for noticing this. Table 10 shows SEE's frozen skills both before and after timestep 100:

PreparationPrep. phaseSkill phase (new)Total
SEE potential (100 steps)4.67±0.324.67 \pm 0.323.11±0.233.11 \pm 0.237.78±0.347.78 \pm 0.34
Random actions (100 steps)1.88±0.041.88 \pm 0.040.16±0.040.16 \pm 0.042.04±0.052.04 \pm 0.05
Reset (0 steps)0.00±0.000.00 \pm 0.001.16±0.241.16 \pm 0.241.16±0.241.16 \pm 0.24

Both phases contribute substantially to SEE's total, and SEE's skills do significantly better when rolled out from the potential policy state than from random actions or reset.

Downstream task utility of SEE outputs

Indeed, SEE's outputs support downstream adaptation under uncertainty (§5.4, Fig. 6, Table 6). Please refer to our new Section 5.4 in the main text for more details (revisions in green).

Consider the "delayed adaptation" setting when an agent must prepare to zero-shot adapt to a random task revealed later in a rollout (Section 5.4, Fig. 6). We show that SEE's potential empowerment policy finds states that are good for future adaptation, while SEE's effective empowerment policy can do zero-shot nonparametric adaptation through skill selection (Table 6). Our added experiments show that when SEE executes the potential empowerment policy for 100 steps without knowing the task, and then we reveal one of the 12 inventory-based tasks, the effective empowerment policy πeff\pi_{\text{eff} } can adapt to the revealed task 56.2±1.456.2 \pm 1.4% of the time. Furthermore, even for alternative baseline skill learning methods (CSF and METRA), running SEE's potential policy for 100 steps before the task is revealed improves their zero-shot adaptation success rates, when compared with a no-op initialization or a random action initialization. Thus, SEE's potential empowerment policy positions the agent to be able to do many possible tasks in the future, without knowing what will be asked.

Success (%, mean ±\pm SE; Table 6):

PreparationSEECSFMETRA
SEE56.2±1.456.2 \pm 1.465.4±3.465.4 \pm 3.463.8±3.863.8 \pm 3.8
Random—39.6±1.339.6 \pm 1.339.6±2.739.6 \pm 2.7
No-op—43.8±2.543.8 \pm 2.543.3±4.243.3 \pm 4.2

Sensitivity to empowerment state representation (12D inventory vs. full state)

At 2B steps in Craftax (Table 9, Section D.4), replacing the 12D inventory encoder (7.78±0.347.78 \pm 0.34, 5 seeds) with the 1,345D full symbolic state drops SEE to 2.56±0.342.56 \pm 0.34 (3 seeds), with full-state policies in both. We have also added discussion and citations on why this result is to be expected, given prior work in ML and psychology [5], [6] showing that empowerment and empowering behavior are highly dependent on an agent's underlying perception and representation of the world.

We also clarified p. 6 to match Tables 1–2 (Ant: 42D; AntSoccer: 2D ball XY).

Survival & homeostasis claims vs. reward baselines (L18)

We ran an experiment giving the agent a reward of 1 for each step it stays alive: as you predicted, it learns survival skills without SEE's deep crafting behaviors. Survival and homeostasis were used interchangeably in our submitted draft, though as you note, homeostasis implies physiological stability beyond staying alive (e.g., hunger/thirst); we therefore removed homeostasis claims from the paper (using the space for our downstream adaptation study).

Dear Reviewer GW2T,

Thank you for your thoughtful feedback. We have worked hard to incorporate your feedback with new adaptation experiments, new evaluations, new baselines, and edits to our figures and main text.

Have our revisions and response sufficiently resolved your questions and concerns? We appreciate your engagement as the end of the discussion phase nears. Thank you.

I would believe that homeostasis would emerge in a long-term process as a proxy for potential empowerment (i.e. hunger indicates an erosion of the energy resources required) - but invoking an explicit empowerment computation just for the energy balance tracking would be quite burdensome, so "stable" proxies could increasingly replace empowerment values for specific resources. Of course, this is beyond the scope of the present paper, and the author's call to replace it by the downstream adaptation was probably a wise one. On the long run, however, such proxy features might be acquired in a similar way as the skills are and allow further scaling.

Reviewer_sMaSOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper uses skills to enhance the ability of empowerment driven agents to handle larger tasks. Importantly the author show that they can scale the effectiveness of the method to substantially large problems. It combines different types of empowerment methodologies together with general skilled learning to achieve this.

Critical strengths

Empowerment used to suffer by the difficulty to scale it up to larger problems. The paper shows a nice combination of methods, including different types of empowerment and empowerment, like quantities and skill learning to achieve just this.

It might open a way to use these types of intrinsic motivations for much larger and more serious applications than hitherto achieved. It is nicely demonstrated in the Craftax environment.

Critical weaknesses

I didn’t see where else to put my comments about the paper. I didn’t identify any substantial weakness, only minor points, which should be fixed.

There are a few aspects about the methodology which are not entirely clear. I’m going to detail these below. In particular, the relation between how the effective empowerment and the potential empowerment are used and how are they linked to the skills might be a bit better described.

Details:

Line 028: acquiring resources and maintaining homeostasis appear to be coming from the same paper so it might make sense to combine both citations.

Line 063: “while skill learning methods do not directly learn policy that maximize empowerment…” – I did not really understand what this sentence is saying

Line 133, equation 1: it appears that empowerment is used as a reward. Strictly spoken, empowerment is more like a value function, that is a cumulated reward. It is true that it is sometimes treated as a reward, but it’s not entirely clear whether the claimed properties and motivations for empowerment make so much sense if it’s treated as a reward. That being said, if it works, it works.

Line 139 following: as far as I understand all the papers cited here using empowerment as a "value function" to decide where the next action should go to. Empowerment itself is computed from an action sequence distribution (like in your potential empowerment), or in the case of process empowerment, as a policy sequence. The decision where to go is the state of highest empowerment. So the sentence is slightly confusing to an uninitiated reader. I do not think Salge 2014b is different in this respect. So the computation goes away multiple steps and the action taken is the one step that leads to the highest empowered state.

Line 145: what is the difference of s_0 to s?

Line 148: is a skill basically the same as an option?

Line 166: are you basically treating the skills as Meta actions which are selected according to the probing distribution?

Line 170: “that the skill vector“ – insert "choice of" before “skill vector“

Line 202: “a classifier that identifies continuous skills from one step transitions” - this part seems crucial in the whole argument, but it is not entirely clear. Do you mean that you construct your classifier from observed sequences? How are these sequences generated in the first place?

Line 223: equation 8, is this inequality obvious? I would have thought that if you stratify by Z the expectation value might actually be higher on the left side. So maybe this deserves explanation

Line 228 equation 9: I think you might want to write an approximate equal between the hat and the precise expectation

Line 233: “classifier identify skills from both s_0 and s_+" – I do not understand the sentence

Line 242: “the classify may receive different degrees of freedom from the full observation policy critic“ – is that indifferent situations or how do these differences come about?

Line 248: how is the skill sampled? Does it come from \pi_pot or something else?

Lines 251 following: method summary: I’m not entirely sure what the box is saying. Once you evaluated empowerment in different states even an approximation what is the second potential empowerment policy actually doing? Where does it come from? This is not clear.

Line 267: “we defined the empowerment degree of freedom to be over the agent inventory” – what do you mean by inventory? The actuators and sensors that the agent has? The object that the agent controls? Something else? Please explain.

Line 274: what do you mean by “strictly the ball position“ – what else is there?

Lines 288 following: it is not entirely clear how you combine the skills with empowerment. Are the skills given ahead of time and empowerment simply allowed to use them or what happens here? Just for your own information you might want to have a look at the concept of “impoverished empowerment“ (Anthony et al. 2014) for empowerment-based generation of skills, which has a bit of effective empowerment-like way of creating the skill candidates (no need to cite here).

Lines 292 following: I do not understand exactly what you are ablating. How is the skill switching actually implemented?

Line 296: when you say MI is an important signal for training. What exactly are you using the signal for? Do you talk about empowerment in its variance or something else and what is it actually training?

What is method-to-does?

Lines 302 following: you mentioned that you measure the distance of the end to the center, to the soccer ball and the distance of the gripper to the cubes. I assume these are purely performance measures and are not used to train the agent. Can you please confirm this? Because if they would be used in any form to train the behaviors, this obviates the use of an intrinsic motivation.

Line 313: “per rollout from" – do you mean “in“ instead of “from“?

Line 344: caption of figure 2 – I did not understand the colorful lines and blocks

In point (B) you distinguish the first version without interacting with a ball and the potential empowerment policy – I’m not sure I understand the difference, maybe rephrase

Line 355: why do the agents erroneously kick the ball to the very corner? What causes this?

Line 361: “in contrast the baseline methods perturb the blocks” – is the baseline method empowerment or something else?

Line 368: how does the switch of SEE from potential to effective empowerment get triggered? That is not clear. It is also not clear what the ablation is over.

Also not in lines 408 and 409.

Line 424: I like the picture of the policies “spending“ empowerment, but I’m not sure how that is actually realized in the concrete case. What does the spending look like? I assume it has to do with a switch between potential and effective empowerment but as I mentioned, it is not entirely clear how it is algorithmically implemented.

Lines 430 following: “the agent executes diverse skills, such as building houses…“ – how does it decide to execute a specific skill?

Line 445: figure 4– I don’t understand in which direction the history goes. Which way to follow the trajectories? I like the statement “potential empowerment policies accrue empowerment while skills spend it“. Very nice picture.

Line 468: when you say one seed per example is that multiple runs with that example or what does it mean?

Line 667: “train persistent skill behavior that spread over late in space”– replace spread by spreads

Line 679: is that really a plus before the partition function? shouldn’t that be a minus, since you normalize by it?

Line 699: “beyond this noise scale” – strictly spoken, when you apply noise, you essentially decide on a sensor with resolution to observe the effect. It is your particular choice of sensor and it can of course be replaced by any sensor you consider important to the task.

Line 746: is 1 the maximum value?

Line 755: what is b?

Line 1 003: you say “unlike our method,… consider open loop definitions” – are you using closed loop empowerment? How so? Where do you say that and can you explain in which way it is used?

Line 1021: “where h_\phi is normalized“ – why do you have to normalize it? Isn’t this the job of "\psi_\phi(s_0)"?

Line 1060: remove the question marks

Line 1157 following: you say that CMI leads to hurt training– what do you use instead? You then say that in Craftax Classic you get collapsed, empowerment value values, even achievement is maintained. This is what happens when you use the joint mutual information. Can you be a bit more detailed about this? Which of these matches the empowerment definition from literature?

Line 1256: it is not entirely clear to me what you do to create the skills. You seem to use mutual information maximization but not in the same way as empowerment does it – is that correct? Can you make a little bit more clear how the different parts of the method separate since they use different aspects of the information theoretic measures?

Line 1266: Leibler, not Liebler

Questions for authors

The most important thing is:

  • how do you separate the transition from potential empowerment to effective empowerment?
  • how do you train the skills, whether they are part of this previous optimization or by a separate method,
  • and if they are also driven by some mutual information maximization
  • and, if so, how that is different from effective empowerment maximization?
5 comments
AuthorsOpenReview ↗

Response to Reviewer sMaS (1/2)

Thank you for your extensive feedback. We address your main questions and some of the line-by-line comments below and have uploaded a revised pdf, with revisions in green. Please let us know if you have any concerns, and we will be happy to continue the discussion.

L63, L296 (effective/potential & MI)

We clarified that we are using a variational InfoNCE bound on empowerment.

Concretely, πpot\pi_{\text{pot} } uses rpot(s)=Ez[Veff(s;s,z)]r_{\text{pot} }(s) = \mathbb{E}_ z[V_{\text{eff} }(s; s, z)] as its reward while πeff\pi_{\text{eff} } uses the classifier reward (annealing the log-partition weight λ\lambda from 00 to 11 so the final annealed objective matches reff(s,a,s′;s0,z)=log⁡q(z∣s0,s′)p(z)r_{\text{eff} }(s, a, s'; s_0, z) = \log\frac{q(z \mid s_0, s')}{p(z)}, App. A.3.1), following MISL [3], [4] (see §1, §4.1, and App. A.4 (Alg. 1)).

L133, L1003 (reward & closed-loop)

Yes, πpot\pi_{\text{pot} } uses E(s)\mathcal{E}(s) as a state reward (Eq. 1). Prior work has noted cumulative discounted empowerment works well empirically despite being value-like [1].

Unlike open-loop empowerment, our skill policy πeff(a∣s,z)\pi_{\text{eff} }(a \mid s, z) is closed-loop and conditions on the current state ss at each step (§2, §3; cf. [2]).

L145, L248 (s0s_0 & switch step)

As described in §3, §4.2, and App. A.4 (Alg. 1), we switch at a fixed rollout step t=Mt=M (sampling uniform zz at t=Mt=M to index πeff,fϕ\pi_{\text{eff} }, f_\phi and holding s0=sMs_0=s_M fixed as sts_t evolves), and leave the problem of learning this time step to future work.

L223–251 (Eqs. 8–9 & πpot\pi_{\text{pot}})

Thanks for noting these issues. We clarified that the classifier predicts skills zz from switch and future states (s0,s+)(s_0, s_+), along with the approximations used, and that the inequality follows from the Barber–Agakov bound. The classifiers get just the task-relevant DoFs, while policies get the full state features. In §A.3, we also clarify how evaluating rpotr_{\text{pot} } separates from training πpot\pi_{\text{pot} } with RL.

L288, L302 (eval metrics & online skills)

Correct, these are purely performance measures. We do not use these metrics at all to train the agent. Similarly, the Craftax achievement metrics are also purely performance measures and do not show up in the optimization.

The skills are not given ahead of time. SEE simultaneously learns skills (effective empowerment policies) using MISL and potential empowerment policies that navigate to regions of high empowerment. We learn both of these policies purely online and do not use any pretrained skills for empowerment estimation. We have now clearly highlighted this problem setting within the Methods box.

L424 ("spending" empowerment)

In the Craftax example, expending items influences the world, exercising effective empowerment. But this is at the cost of losing the ability to choose how to use the items (potential empowerment). Consuming an item directly converts potential capacity to realized influence. More generally, environments with a clear potential-effective split include settings where capital or other resources can be accumulated and expended for influence, or idealized physical settings where potential empowerment can literally correspond to potential energy. For instance, a skier ascending to the top of a mountain by chairlift can then easily ski down to any position on the mountain. At the top of the mountain, they have more potential energy (or potential empowerment), and descending down a path of their choosing converts that potential into a realized choice.

L430, L679, L699, L746, L1021

Thank you for the suggestions. We clarified that the skills are sampled uniformly and discussed the connection with sensor resolution. We also clarified that f∗f^* learns an unnormalized logit and the normalization (annealed from weight 00 to 11 over training, §A.3.1) handles the partition function, and we added the corresponding VIM normalization details in §C.

L1157, L1256 (potential vs. effective & MI)

In addition to doing effective empowerment maximization, SEE scalably learns a policy that takes the potential empowerment estimate from the effective empowerment value function as a reward (without learning skills everywhere over state space). Simply doing effective empowerment maximization would only learn distinguishable options for some starting state distribution, while potential empowerment maximization learns a policy that navigates to high-empowerment regions. Our additional downstream adaptation experiments, which we detail in our responses to Reviewers 4YXk and GW2T and §5.4 (Table 6), help conceptually and practically clarify the difference between potential empowerment maximization and effective empowerment maximization.

Empirically, we found the joint MI was more stable in continuous settings (§5.2, §D.2).

The authors have addressed my comments and resolved many of my questions.

I had one minor question left:

  • line 211: you say "separated by geometric offsets" - do you mean you run it until the end? Not clear how that s+ sampling works

Some of the details of the work might profit from slightly more clarity, especially how the transition from potential to effective strategy is achieved is not entirely clear - what makes the tipping point? I reiterate that I find the picture of "spending" potential empowerment in the "effective" phase very expressive. This lucid picture could profit from a clearer explanation how this "spending" is triggered.

Other than that, the paper has improved and is quite readable and interesting, original and effective. I stick with my original evaluation and recommend it should be accepted.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Summary of Manuscript Revisions and Rebuttal Experiments

We thank the reviewers for their thoughtful feedback. We have revised the manuscript in green based on the reviewers' suggestions. Most notably:

  • Ablations & Evaluations (GW2T, 4YXk), changed ablations in Section 5, added baselines in Fig. 2, added evaluations in Section 5.2, with tables in appendix: We have implemented RND, METRA, CSF, and skill-only ablation of SEE in Craftax-Classic, finding that SEE beats all of the baselines in achievement count and percentage with the same training budget and similar architectures. Furthermore, both the potential and skill phases significantly contribute to SEE's achievement count. Additionally, the learned SEE skills acquire more achievements/have richer behaviors at the navigate empowered state, versus random action initialization or reset.

  • Downstream usefulness of SEE, both potential and effective empowerment policies (GW2T, 4YXk), new Section 5.4, new "Adaptation to Downstream Tasks" in Related Works, B.2.3 for new results:
    Without any additional training, preparing to adapt using SEE’s potential policy increases the zero-shot task adaptation success of CSF and METRA over random initialization or no-op initialization, a significant improvement which cannot be explained solely by CSF-style skill learning alone. In the paper, we have now highlighted the ability of the potential policy to prepare agents for adaptation under uncertainty as a key emergent behavior.

  • Clarifications in the paper about the difference between potential and effective empowerment policies (sMaS, 4YXk), new green text in Section 4 in particular: We have several edits in the revised text to clarify the difference between potential and effective empowerment policies, as well as our specific online training protocols to learn both methods at once.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.