I didn’t see where else to put my comments about the paper. I didn’t identify any substantial weakness, only minor points, which should be fixed.
There are a few aspects about the methodology which are not entirely clear. I’m going to detail these below. In particular, the relation between how the effective empowerment and the potential empowerment are used and how are they linked to the skills might be a bit better described.
Details:
Line 028: acquiring resources and maintaining homeostasis appear to be coming from the same paper so it might make sense to combine both citations.
Line 063: “while skill learning methods do not directly learn policy that maximize empowerment…” – I did not really understand what this sentence is saying
Line 133, equation 1: it appears that empowerment is used as a reward. Strictly spoken, empowerment is more like a value function, that is a cumulated reward. It is true that it is sometimes treated as a reward, but it’s not entirely clear whether the claimed properties and motivations for empowerment make so much sense if it’s treated as a reward. That being said, if it works, it works.
Line 139 following: as far as I understand all the papers cited here using empowerment as a "value function" to decide where the next action should go to. Empowerment itself is computed from an action sequence distribution (like in your potential empowerment), or in the case of process empowerment, as a policy sequence. The decision where to go is the state of highest empowerment. So the sentence is slightly confusing to an uninitiated reader. I do not think Salge 2014b is different in this respect. So the computation goes away multiple steps and the action taken is the one step that leads to the highest empowered state.
Line 145: what is the difference of s_0 to s?
Line 148: is a skill basically the same as an option?
Line 166: are you basically treating the skills as Meta actions which are selected according to the probing distribution?
Line 170: “that the skill vector“ – insert "choice of" before “skill vector“
Line 202: “a classifier that identifies continuous skills from one step transitions” - this part seems crucial in the whole argument, but it is not entirely clear. Do you mean that you construct your classifier from observed sequences? How are these sequences generated in the first place?
Line 223: equation 8, is this inequality obvious? I would have thought that if you stratify by Z the expectation value might actually be higher on the left side. So maybe this deserves explanation
Line 228 equation 9: I think you might want to write an approximate equal between the hat and the precise expectation
Line 233: “classifier identify skills from both s_0 and s_+" – I do not understand the sentence
Line 242: “the classify may receive different degrees of freedom from the full observation policy critic“ – is that indifferent situations or how do these differences come about?
Line 248: how is the skill sampled? Does it come from \pi_pot or something else?
Lines 251 following: method summary: I’m not entirely sure what the box is saying. Once you evaluated empowerment in different states even an approximation what is the second potential empowerment policy actually doing? Where does it come from? This is not clear.
Line 267: “we defined the empowerment degree of freedom to be over the agent inventory” – what do you mean by inventory? The actuators and sensors that the agent has? The object that the agent controls? Something else? Please explain.
Line 274: what do you mean by “strictly the ball position“ – what else is there?
Lines 288 following: it is not entirely clear how you combine the skills with empowerment. Are the skills given ahead of time and empowerment simply allowed to use them or what happens here? Just for your own information you might want to have a look at the concept of “impoverished empowerment“ (Anthony et al. 2014) for empowerment-based generation of skills, which has a bit of effective empowerment-like way of creating the skill candidates (no need to cite here).
Lines 292 following: I do not understand exactly what you are ablating. How is the skill switching actually implemented?
Line 296: when you say MI is an important signal for training. What exactly are you using the signal for? Do you talk about empowerment in its variance or something else and what is it actually training?
What is method-to-does?
Lines 302 following: you mentioned that you measure the distance of the end to the center, to the soccer ball and the distance of the gripper to the cubes. I assume these are purely performance measures and are not used to train the agent. Can you please confirm this? Because if they would be used in any form to train the behaviors, this obviates the use of an intrinsic motivation.
Line 313: “per rollout from" – do you mean “in“ instead of “from“?
Line 344: caption of figure 2 – I did not understand the colorful lines and blocks
In point (B) you distinguish the first version without interacting with a ball and the potential empowerment policy – I’m not sure I understand the difference, maybe rephrase
Line 355: why do the agents erroneously kick the ball to the very corner? What causes this?
Line 361: “in contrast the baseline methods perturb the blocks” – is the baseline method empowerment or something else?
Line 368: how does the switch of SEE from potential to effective empowerment get triggered? That is not clear. It is also not clear what the ablation is over.
Also not in lines 408 and 409.
Line 424: I like the picture of the policies “spending“ empowerment, but I’m not sure how that is actually realized in the concrete case. What does the spending look like? I assume it has to do with a switch between potential and effective empowerment but as I mentioned, it is not entirely clear how it is algorithmically implemented.
Lines 430 following: “the agent executes diverse skills, such as building houses…“ – how does it decide to execute a specific skill?
Line 445: figure 4– I don’t understand in which direction the history goes. Which way to follow the trajectories? I like the statement “potential empowerment policies accrue empowerment while skills spend it“. Very nice picture.
Line 468: when you say one seed per example is that multiple runs with that example or what does it mean?
Line 667: “train persistent skill behavior that spread over late in space”– replace spread by spreads
Line 679: is that really a plus before the partition function? shouldn’t that be a minus, since you normalize by it?
Line 699: “beyond this noise scale” – strictly spoken, when you apply noise, you essentially decide on a sensor with resolution to observe the effect. It is your particular choice of sensor and it can of course be replaced by any sensor you consider important to the task.
Line 746: is 1 the maximum value?
Line 755: what is b?
Line 1 003: you say “unlike our method,… consider open loop definitions” – are you using closed loop empowerment? How so? Where do you say that and can you explain in which way it is used?
Line 1021: “where h_\phi is normalized“ – why do you have to normalize it? Isn’t this the job of "\psi_\phi(s_0)"?
Line 1060: remove the question marks
Line 1157 following: you say that CMI leads to hurt training– what do you use instead? You then say that in Craftax Classic you get collapsed, empowerment value values, even achievement is maintained. This is what happens when you use the joint mutual information. Can you be a bit more detailed about this? Which of these matches the empowerment definition from literature?
Line 1256: it is not entirely clear to me what you do to create the skills. You seem to use mutual information maximization but not in the same way as empowerment does it – is that correct? Can you make a little bit more clear how the different parts of the method separate since they use different aspects of the information theoretic measures?
Line 1266: Leibler, not Liebler
Response to Reviewer 4YXk (1/2)
Thank you for the thoughtful review. Below we address delayed adaptation (new section §5.4, Table 6), task-adaptation related work (new in §2), and 2B-step Craftax baselines. Please let us know if you have any more questions or concerns, and we would be happy to address them.
Thank you for the suggestion! This led to some new, interesting results.
Consider the "delayed adaptation" setting when an agent must prepare to zero-shot adapt to a random task revealed later in a rollout (Section 5.4, Fig. 6). We show that SEE's potential empowerment policy finds states that are good for future adaptation, while SEE's effective empowerment policy can do zero-shot nonparametric adaptation through skill selection (Table 6). Our added experiments show that when SEE executes the potential empowerment policy for 100 steps without knowing the task, and then we reveal one of the 12 inventory-based tasks, the effective empowerment policy πeff can adapt to the revealed task 56.2±1.4% of the time. Furthermore, even for alternative baseline skill learning methods (CSF and METRA), running SEE's potential policy for 100 steps before the task is revealed improves their zero-shot adaptation success rates, when compared with a no-op initialization or a random action initialization. Thus, SEE's potential empowerment policy positions the agent to be able to do many possible tasks in the future, without knowing what will be asked.
The full preparation-by-skill comparison is given in our third response below (Table 6).
We thank the reviewer for referring us to Yang et al. [1], which we now discuss in Related Work (§2) under "Adaptation to Downstream Tasks". This work is particularly relevant to our new adaptation results, which the authors link to mutual information skill learning. While we do not implement LSEPIN, WSEP, and PWSEP as baselines (for WSEP and PWSEP, due to practical difficulties of scalably estimating Wasserstein distances), we believe that alternative formulations of skill learning like those proposed in their paper are necessary to develop stronger adaptation abilities.
Prior skill methods (DIAYN [2], DADS [3], METRA [4], and CSF [5]) optimize skill-diversity objectives related to effective empowerment, i.e., the amount of influence the agent exerts on the environment. SEE learns to maximize potential empowerment [6], [7], finding states where agents accumulate power without "spending" it.
These two forms of empowerment can be very different: in a casino, effective empowerment learns to pull all the slot machines at random until it goes bankrupt (maximum diversity of behavior); potential empowerment finds a game that you can win and makes money.
The problem is that to learn to maximize potential empowerment, you need to try a bunch of games. Our insight is that learning both policies and switching from potential to effective empowerment maximization lets us learn the potential-maximizing behavior.
As Table 6 shows, this state-preparation behavior is not explainable by unsupervised skill learning alone and improves the zero-shot adaptation under uncertainty of past skill-learning methods like CSF and METRA, which optimize skill-diversity objectives rather than a separate preparation policy.
Success rates (%) after 100 task-blind preparation actions followed by up to 900 actions of a frozen, task-selected skill (mean ± seed standard error across three training seeds and 128 worlds per task per seed, averaged over 12 inventory tasks):
In matched 2B-step Craftax-Classic PQN runs (Fig. 2, Tables 5 and 8, §C.3), SEE reaches 7.78±0.34 achievements (35.4%), beating CSF (6.05±0.33), METRA (6.02±0.14), Skills-only SEE (4.17±0.32), and RND [8] (3.20±0.05).