- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
AEGIS couples reward-free exploration and representation learning through an approximate Stackelberg formulation, combining local episodic diversity with global model-error novelty. Experiments on MiniGrid and Procgen show faster learning than several intrinsic-motivation baselines.
Critical strengths
The paper studies an important interaction between exploration and representation learning, and the component-wise degeneracy analysis gives some theoretical justification beyond empirical ablations.
Critical weaknesses
The theoretical claims are limited: the practical algorithm is only alternating optimization, with no convergence guarantee for the claimed Stackelberg game. The empirical scope is narrow, with only a few MiniGrid/Procgen tasks and three Procgen seeds. The method also heavily combines ideas already present in DEIR, NovelD, and NGU, so the incremental novelty is not fully convincing.
Questions for authors
Can the authors justify why the alternating updates meaningfully approximate the claimed Stackelberg solution? Can the authors provide broader experiments or stronger comparisons against matched combinations of DEIR, NovelD, and NGU?
2 comments
Dear Reviewer GRTz,
The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?
Thank you for serving as a reviewer for ICLR.
Best,
AC
We thank the reviewer for carefully reading our work and recognising the importance of the interaction between exploration and representation learning, as well as the value of the component-wise degeneracy analysis. We address each concern below. Corrections found while preparing the rebuttal will be provided as a general official comment.
On the Stackelberg game definition. Eq. (10) states the problem formulation, and this isn’t limited to the algorithm design. The LMDP (leader) defines the reward landscape the policy optimises, and the policy (follower) determines the data the LMDP is fitted on: the leader-follower structure of model-as-leader MBRL (Rajeswaran et al., 2020). As there, we use first-order updates and do not differentiate through the follower’s response.
Why doesn’t our theory address convergence guarantees? No exploration method that trains its bonus model alongside PPO (DEIR, NovelD, NGU, RND), and no model-as-leader MBRL method, has a convergence guarantee. Deep PPO has none even on a fixed MDP. Our theory, therefore, addresses a different question: which solutions the objective admits. If the updates settle, both players’ first-order conditions hold, and Theorem 1 rules out the degenerate solutions each component alone allows. The formulation earns its place through what it predicts: because the policy’s reward contains an episodic term the leader does not minimise, the game is general-sum. When that term is removed, and the policy only seeks the model’s error, learning fails, as Lemma 1 predicts and Fig. 8 shows. We will update our text to explain this more clearly.
“The incremental novelty is not fully convincing.” AEGIS takes one idea from NGU, a reward with an episodic and a lifelong component, and realises it differently. The local term follows DEIR's episodic reward. The global term applies NovelD's novelty-difference formula to the forward-prediction error of the latent dynamics model rather than to an RND error, so AEGIS trains no separate RND network: the model that supplies the global reward is the one that learns the representation. That model is trained with forward, inverse, and discriminative losses, where DEIR uses the discriminator alone, and our analysis shows each term excludes a degenerate solution the others allow, the part you note as a strength. The objective is where the gain comes from: on DoorKey-16x16 HPT, with the reward held fixed, removing the forward loss from the objective returns performance to DEIR's level (Fig. 7).
AEGIS is primarily a reward-free pretraining method, and its design targets that setting, so it should be judged by what pretraining achieves rather than by its components alone. This is also why its advantage is stronger with pretraining and smaller when training from scratch.
On the reviewer’s request for broader experiments. Our scope follows DEIR, our closest baseline: the same two environment families, and the same Procgen protocol of three seeds per game. A single AEGIS Procgen run takes about 40 GPU-hours, which is why the paper calls those results indicative. The revision also adds three more MiniGrid tasks (Appendix E4: DoorKey-8x8, FourRooms and KeyCorridorS4R3), where AEGIS is ahead of DEIR or level with it under pretraining, and a loss-weight sensitivity study (Appendix E1).
We read the second part of the request (matched combinations) as hybrids built only from existing components: e.g., DEIR's episodic reward multiplied by NovelD's RND novelty difference, under the same settings. That isolates what our global term and three-loss model add. We added a few such variants (DEIR×NovelD, DEIR×NGU, DEIR×ICM), and none of them matches AEGIS performance. Reviewer E8Cg asked for AEGIS-RND, which is comparable to AEGIS; this actually adds to its robustness since the global novelty doesn’t have to rely on the forward-model error, and in fact other novelty estimators with similar properties may also work. Removing the global reward (Fig. 6) or the forward loss (Fig. 7) returns performance to DEIR's level. The general comment also adds six further baselines the reviewers mention (ETD, RIDE, E3B, APT, C-TeC, TNCOA) run under the same protocol with and without pretraining; with pretraining, none of them matches AEGIS (Appendix E5). Other environment families will be explored in a journal extension.
We hope these clarifications address your concerns about the formulation and novelty of AEGIS. We would be glad to answer any further questions during the discussion, and we would appreciate it if you would reconsider your assessment in light of them.