acceptodds
Under review as a conference paper at ICLR 2027

AEGIS: Adversarial exploration for generalized improved state representations

Abstract

Reward-free pretraining (RFPT) seeks transferable state representations and exploration behaviors before task-specific rewards are available, but existing approaches treat exploration and representation learning as independent processes. AEGIS combines them as an approximate bilevel Stackelberg problem between (i) a latent dynamics model that minimizes the prediction error and (ii) an exploration policy that uncovers states that expose its weaknesses. The policy is driven by an intrinsic reward combining intra-episode diversity with novelty toward states the dynamics model has not yet fitted. We prove that each component of the representation model and the intrinsic reward exclude a degenerate solution the others allow, so only their combination is non-degenerate. AEGIS outperforms state-of-the-art methods with faster convergence and less redundant exploration on MiniGrid and Procgen.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Reward-free Pretraining, Representation Learning, Intrinsic Motivation, Exploration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GRTzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AEGIS couples reward-free exploration and representation learning through an approximate Stackelberg formulation, combining local episodic diversity with global model-error novelty. Experiments on MiniGrid and Procgen show faster learning than several intrinsic-motivation baselines.

Critical strengths

The paper studies an important interaction between exploration and representation learning, and the component-wise degeneracy analysis gives some theoretical justification beyond empirical ablations.

Critical weaknesses

The theoretical claims are limited: the practical algorithm is only alternating optimization, with no convergence guarantee for the claimed Stackelberg game. The empirical scope is narrow, with only a few MiniGrid/Procgen tasks and three Procgen seeds. The method also heavily combines ideas already present in DEIR, NovelD, and NGU, so the incremental novelty is not fully convincing.

Questions for authors

Can the authors justify why the alternating updates meaningfully approximate the claimed Stackelberg solution? Can the authors provide broader experiments or stronger comparisons against matched combinations of DEIR, NovelD, and NGU?

2 comments

We thank the reviewer for carefully reading our work and recognising the importance of the interaction between exploration and representation learning, as well as the value of the component-wise degeneracy analysis. We address each concern below. Corrections found while preparing the rebuttal will be provided as a general official comment.

On the Stackelberg game definition. Eq. (10) states the problem formulation, and this isn’t limited to the algorithm design. The LMDP (leader) defines the reward landscape the policy optimises, and the policy (follower) determines the data the LMDP is fitted on: the leader-follower structure of model-as-leader MBRL (Rajeswaran et al., 2020). As there, we use first-order updates and do not differentiate through the follower’s response.

Why doesn’t our theory address convergence guarantees? No exploration method that trains its bonus model alongside PPO (DEIR, NovelD, NGU, RND), and no model-as-leader MBRL method, has a convergence guarantee. Deep PPO has none even on a fixed MDP. Our theory, therefore, addresses a different question: which solutions the objective admits. If the updates settle, both players’ first-order conditions hold, and Theorem 1 rules out the degenerate solutions each component alone allows. The formulation earns its place through what it predicts: because the policy’s reward contains an episodic term the leader does not minimise, the game is general-sum. When that term is removed, and the policy only seeks the model’s error, learning fails, as Lemma 1 predicts and Fig. 8 shows. We will update our text to explain this more clearly.

“The incremental novelty is not fully convincing.” AEGIS takes one idea from NGU, a reward with an episodic and a lifelong component, and realises it differently. The local term follows DEIR's episodic reward. The global term applies NovelD's novelty-difference formula to the forward-prediction error of the latent dynamics model rather than to an RND error, so AEGIS trains no separate RND network: the model that supplies the global reward is the one that learns the representation. That model is trained with forward, inverse, and discriminative losses, where DEIR uses the discriminator alone, and our analysis shows each term excludes a degenerate solution the others allow, the part you note as a strength. The objective is where the gain comes from: on DoorKey-16x16 HPT, with the reward held fixed, removing the forward loss from the objective returns performance to DEIR's level (Fig. 7).

AEGIS is primarily a reward-free pretraining method, and its design targets that setting, so it should be judged by what pretraining achieves rather than by its components alone. This is also why its advantage is stronger with pretraining and smaller when training from scratch.

On the reviewer’s request for broader experiments. Our scope follows DEIR, our closest baseline: the same two environment families, and the same Procgen protocol of three seeds per game. A single AEGIS Procgen run takes about 40 GPU-hours, which is why the paper calls those results indicative. The revision also adds three more MiniGrid tasks (Appendix E4: DoorKey-8x8, FourRooms and KeyCorridorS4R3), where AEGIS is ahead of DEIR or level with it under pretraining, and a loss-weight sensitivity study (Appendix E1).

We read the second part of the request (matched combinations) as hybrids built only from existing components: e.g., DEIR's episodic reward multiplied by NovelD's RND novelty difference, under the same settings. That isolates what our global term and three-loss model add. We added a few such variants (DEIR×NovelD, DEIR×NGU, DEIR×ICM), and none of them matches AEGIS performance. Reviewer E8Cg asked for AEGIS-RND, which is comparable to AEGIS; this actually adds to its robustness since the global novelty doesn’t have to rely on the forward-model error, and in fact other novelty estimators with similar properties may also work. Removing the global reward (Fig. 6) or the forward loss (Fig. 7) returns performance to DEIR's level. The general comment also adds six further baselines the reviewers mention (ETD, RIDE, E3B, APT, C-TeC, TNCOA) run under the same protocol with and without pretraining; with pretraining, none of them matches AEGIS (Appendix E5). Other environment families will be explored in a journal extension.

We hope these clarifications address your concerns about the formulation and novelty of AEGIS. We would be glad to answer any further questions during the discussion, and we would appreciate it if you would reconsider your assessment in light of them.

Dear Reviewer GRTz,

The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?

Thank you for serving as a reviewer for ICLR.

Best,

AC

Reviewer_E8CgOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AEGIS trains an exploration policy alongside a separate latent dynamics model (LMDP). Its intrinsic reward multiplies DEIR-style episodic diversity by a forward-error novelty-difference term, and the representation objective combines forward, inverse, and discriminative losses. The paper describes the interaction as approximate Stackelberg optimization and evaluates reward-free pretraining followed by task training on MiniGrid and Procgen. The experiments support the combination as an exploration method. They provide less support for the claimed representation improvement and the contribution attributed to adversarial coupling. The geometry and injectivity claims also exceed what the argument establishes. Relative to established and recent work, the defensible novelty lies in the specific integration and its conditional failure-mode analysis; broader framework-level novelty and state-of-the-art superiority remain insufficiently established.

Critical strengths
  1. With half-budget pretraining (HPT), AEGIS performs particularly well on BlockedUnlockPickup and reaches high returns earlier on DoorKey-16x16. Table 1 reports approximately 27.7% and 15.0% fewer steps to T90 than DEIR on DoorKey and KeyCorridor, respectively. These are gains in interaction-step efficiency .

  2. The comparisons use common PPO optimization hyperparameters. AEGIS's configuration is fixed after tuning on DoorKey-8x8, and MiniGrid experiments use ten seeds. Pretraining-budget and reward/loss ablations, episode-length and coverage analyses, and per-seed T90 crossings help interpret the results.

  3. Theorem 1's lower-bound comparison excludes constant-collapse global minimizers on a fixed data distribution, provided Assumption 1's common parameter point exists and the loss weights are positive . The authors also report that the full objective does not improve decodability over DSCM-only and qualify the conclusions drawn from three Procgen seeds.

Critical weaknesses

W1. state-of-the-art discussion? The abstract and Related Work overstate the separation between exploration and representation learning in prior work. BYOL-Explore (Guo et al., NeurIPS 2022), Hong et al. (CoRL 2019), CRL (Du et al., ICCV 2021), and EUCLID (Yuan et al., ICLR 2023) all couple exploration with representation or dynamics learning. Henaff et al. (ICML 2023) further study combinations of global and episodic exploration bonuses, including additive and multiplicative variants. ICM and Plan2Explore also contain feedback between exploration and model learning. Thus, neither coupling these processes nor combining global and episodic novelty signals is itself new.

More recent work further narrows the claimed gap. METRA (ICLR 2024), EME (NeurIPS 2024), BCD (AAAI 2024), WAKER (ICLR 2024), DVFB (ICLR 2025), and ExDM (ICLR 2026) explore closely related ideas involving jointly learned representations, behavioral metrics, model-error-driven exploration, data diversity, and learned density signals. Their objectives differ, but collectively they make a broad claim that prior exploration and representation learning were largely independent difficult to sustain.

W2. experiments The experiments improve over the reported baselines but omit several directly relevant exploration methods. ETD (Jiang et al., ICLR 2025) is the highest-priority missing comparator because it also uses learned temporal distance, PPO, and overlapping MiniGrid tasks. TNCOA, E3B, EME, and BCD are also relevant. Broader superiority claims therefore require matched comparisons under comparable interaction, tuning, and pretraining budgets.

The interpretation of the global signal should also be narrowed. LPM measures prediction improvement across model updates, whereas AEGIS compares prediction errors on adjacent transitions, clips the difference, and adds a positive floor. This does not directly measure learning progress on a fixed transition. The noise experiments show robustness, but do not establish that the signal separates learnable error from irreducible noise.

Questions for authors

method discussion

  1. How does AEGIS differ from BYOL-Explore's joint representation/dynamics/exploration learning, Hong et al.'s adversarial inverse-model learning, and the global-episodic bonus combinations studied by Henaff et al.? Please distinguish the representation objective, reward composition, optimization updates, and guarantees, also accounting for metric-based methods such as METRA and EME.

  2. Could the authors prioritize ETD and consider another relevant MiniGrid comparator, such as EME or TNCOA, under matched observations, PPO architecture, interaction and tuning budgets, reporting T90, final returns, and uncertainty across seeds?

some writing clarification

  1. Which representation improves, fθf_\theta or fϕf_\phi, and what evidence separates its benefit from policy initialization and the continued intrinsic bonus? Please support that attribution or scope the claim to the exploration and continuation results demonstrated.

  2. Please specify each Figure 8 comparator's rewards, losses, and updates, including their relationship to Eq. (10). What evidence separates the game formulation's effect on collapse avoidance from the simultaneous component changes, or how should the causal attribution be narrowed?

  3. Which of constant-collapse exclusion, metric sensitivity, geometry identification, and injectivity follow from the objective, and which are assumptions? How do the geometry claims account for the affine-copy examples and the partial collisions allowed by Definition 1?

  4. Could the authors hold the representation losses and update schedule fixed while comparing FDM-based and RND-based novelty differences with the same episodic term, composition rule, and normalization?

3 comments

We thank the reviewer for the detailed review and for noting the efficiency gains under pretraining, the shared PPO settings and fixed configuration with ten seeds and supporting analyses and the collapse-exclusion result of Theorem 1. Corrections we found and the revisions we have made are noted in our general comment.

Q1. Objective: AEGIS trains forward, inverse and discriminative terms together; BYOL-Explore uses one latent prediction loss for both representation and reward. Reward: an episodic term multiplied by a forward-error novelty difference on a recurrent latent, from the same encoder. It follows NGU's two-level structure, with DEIR's episodic term and NovelD's novelty difference applied to the model's forward error rather than an RND error, so it trains no separate RND network. Henaff et al. find that multiplying episodic and global bonuses works best; AEGIS uses such a product, but its global factor comes from the same model whose objective shapes the encoder the episodic term reads. Updates: PPO and LMDP steps interleave per batch, as in Hong et al. The real difference from Hong et al. and CRL is that the reward contains an episodic term the model does not minimise, so the game is not zero-sum. Guarantees: only conditional exclusions of degenerate solutions, where METRA and EME make positive metric statements. In our revised draft, Appendix E6 compares all these methods with AEGIS on the four axes you list, in two tables: the methods we do not run, with the reason, and the baselines we run.

Q2. We agree ETD is the most relevant comparator, with one correction: AEGIS learns no temporal distance, and its episodic term follows the DEIR formula. We ran ETD at its published hyperparameters on DoorKey-16x16 and BlockedUnlockPickup with pretraining, 10 seeds, under matched settings, with E3B, RIDE, APT, C-TeC, and TNCOA. With pretraining, AEGIS is ahead of ETD on both tasks: ETD does not solve BlockedUnlockPickup (0/10 seeds) and on DoorKey-16x16 it needs 1.14M steps to 90% of the best final return against 0.84M for AEGIS (p < .001). Appendix E5 reports T90, its interquartile range across seeds, per-seed tests, and final returns, as you asked. TNCOA has no public code, so we implemented it from the paper; it solves neither task within the budget. We omit EME and BCD: both build part of their bonus from the task reward (a reward-model ensemble and successor features of the reward, respectively), so that part gives no signal during reward-free pretraining. BCD, whose losses are the closest to ours, is discussed in Appendix E6, which gives the reason for each method we do not run.

Q3. On which encoder improves, and whether policy initialisation or the continued bonus explains the gain: it is fθf_\theta, which both reward terms read; the policy acts on its own encoder fϕf_\phi, which receives no LMDP gradient (Section 3, l. 201-204 in the revised) and we do not claim that fθf_\theta transfers as policy features. Of the three possible sources of the gain, policy initialisation and the continued bonus are matched across methods; what differs is the intrinsic reward, both its formula and the encoder it is computed on, and fθf_\theta matters on its own: (i) Representation: With the reward formula fixed, changing only fθf_\theta's objective changes the outcome: without the forward loss, return falls to DEIR's level (Fig. 7) and under noise the same episodic formula separates first visits from revisits better on AEGIS's encoder than on DEIR's (AUROC .84 vs .79, p < .001; Appendix E2). (ii) Policy initialisation: Every method pretrains the same PPO policy for the same budget, so initialisation is matched; what differs is the reward it is trained on. (iii) Continued bonus: Every method also keeps its bonus after the boundary and AEGIS reaches 90% of the best final return before any task reward. The gain therefore comes from the intrinsic reward computed on fθf_\theta, through the exploratory policy it trains.

Q4. On separating the game formulation from the component changes: Fig. 8 now also holds the loss fixed. With the full three-loss objective held fixed, a policy rewarded only with the model’s error (a zero-sum game) ends at return 0, while the product reward, whose episodic term the model does not minimise, ends at .98. ICM’s own model repeats this: ICM alone does not solve DoorKey-16x16 (final return .009), whereas DEIR×ICM does (Appendix E3). For the forward-model variant, the probes of App. B.5 show the collapse Lemma 1 predicts (distance targets stay at their floor); the inverse-model variants, which Lemma 1 does not let collapse, fail because the zero-sum reward drives novelty down (App. B.2). The revision specifies each Fig. 8 variant's reward, losses, and updates, and its relation to Eq. (10).

We hope these answers address your questions. We would be glad to discuss any further points during the discussion and we would appreciate it if you would reconsider your assessment in light of them.

Follow-up on the remaining questions and weaknesses.

Q5. On what follows from the objective and what is assumed: constant-collapse exclusion is derived under Assumption 1 (Theorem 1), and the metric sensitivity of the forward loss, which the inverse and discriminative losses lack, holds without it (Lemma 2). Assumed: injectivity of fθf_\theta (Notation, condition (iii), Appendix A). Not established by the objective: geometry identification; the paper states that Theorem 1 does not establish that the geometry approximates reachability (Section 7, Remark 2) and that affine copies of a zero-loss solution tie (Remark 1), and the revision aligns Remark 1 with this. A uniform rescaling leaves the ordering of distances and of rlocalr^{local} unchanged, and the per-rollout normalisation (Remark 6) removes its scale. Partial collisions are bounded by the discriminative loss (new proposition in App. A): among contrasted pairs with different observations, the share with identical discriminator inputs is at most LDSCM/ln⁡2\mathcal{L}_{DSCM}/\ln 2, about 1.85% in our runs. Q6. On holding the representation fixed and comparing FDM- and RND-based novelty: yes, we ran this on DoorKey-16x16 with pretraining. AEGIS-RND keeps the LMDP, its losses, the update schedule, the episodic term, the composition, and the normalisation, and changes only the novelty vtv_t from the forward error to an RND error. It is comparable to AEGIS (steps to 90% of the best final return: 0.95M vs 0.84M, p = .29), with more spread across seeds. The same swap under DEIR’s objective (DEIR×NovelD) needs 1.26M (p < .001). This supports the design: the gain comes from the three-loss objective together with a global novelty term, and AEGIS does not depend on one particular novelty estimator. We keep the forward error because it comes from the LMDP at no extra cost, whereas RND trains a separate network (Appendix E3).

W1. On novelty relative to prior work: we agree that the abstract's sentence, that existing approaches treat exploration and representation learning as independent processes, is too broad, and the revision narrows it. The methods you list that couple the two do so in one of two ways. Hong et al., CRL, ICM, and BYOL-Explore reward the policy with the loss the model itself minimises, which makes the game zero-sum on that loss; Fig. 8 shows this structure failing under our protocol. Plan2Explore and EUCLID let the policy plan or train in a learned world model, so they are model-based. AEGIS is model-free: its policy acts on its own encoder and learns only from the intrinsic reward, whose episodic term the model does not minimise, so the coupling is general-sum, and its representation combines three losses, each excluding a degenerate solution the others allow (Theorem 1). Henaff et al. study how to combine global and episodic bonuses, which the paper does not claim as new; BCD and EME need the task reward; METRA, DVFB, ExDM and WAKER change the agent or the environment rather than the reward. Appendix E6 compares all of them on the four axes of Q1.

W2 (tuning). Tuning does not favour AEGIS. The shared PPO hyperparameters are the same as DEIR’s: Wan et al. searched them (up to six candidate values each) together with each baseline’s intrinsic reward parameters, on DoorKey-8x8 and then on DoorKey-16x16 and KeyCorridorS6R3, two of our tasks (their App. A.4). AEGIS reuses their best configuration unchanged, so the PPO settings remain as tuned for DEIR, and only AEGIS’s own parameters were selected, once, on DoorKey-8x8. Four loss-weight settings on DoorKey-16x16 all reach the T90 threshold before DEIR, by 8% to 29% (p < .001 each; Appendix E1). We hope this settles the question of fairness in the comparison.

W2 (global signal). "Not yet fitted" refers to the whole LMDP: all three losses train on the collected data and shape the encoder both reward terms read, so revisited transitions earn less as the model fits them. Eq. (9) takes the forward error relative to the preceding transition, as NovelD does. Learning Progress Monitoring (LPM) instead rewards the drop in a dynamics model's error on the same transition across model updates, and it predicts raw observations with no representation objective. Its aim is to separate learnable from unlearnable error under noise; AEGIS couples representation learning with exploration, and its global factor is error-based novelty, not learning progress.

W2 (noise). To test separation directly, we measured it under noise with the true grid state logged. Under noise, the global factor is higher for never-visited states than for familiar ones (AUROC .59) and is negatively correlated with the lifelong visit count, so it still tracks the true state rather than the noisy observation. The local reward ranks first visits in an episode above revisits (AUROC .84 vs .79 for DEIR, p < .001; Appendix E2).

We hope these answers address your remaining questions and concerns. We would be glad to discuss any further points.

Dear Reviewer E8Cg,

The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?

Thank you for serving as a reviewer for ICLR.

Best,

AC

Reviewer_RchSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a global intrinsic reward which is combined with a local intrinsic reward to improve exploration efficiency and convergence for reward-free pretraining and policy learning. The authors argue that representation learning and exploration should be done jointly and that they justify their decision theoretically in contrast to prior works. The paper also provides empirical results for Minigrid and Procgen.

Critical strengths
  • Exploration and reward-free pretraining are important and interesting topics in RL
  • The introduction is well-written and clearly motivates the work
Critical weaknesses
  • The paper is in my opinion technically not sound because its framing/scope, the claims and the main paper do not align. The local intrinsic reward appears to be the same as the one introduced in DEIR, the global component seems similar to NovelD, and the central focus is that this work theoretically justifies the design decisions in contrast to previous works. However, the main paper describes mainly the methodology and provides the empirical results, but the theory is only in the appendix. Motivating the methodology with the theory would in my opinion better match the framing of the paper and at least theorem 1 should be in the main text. The empirical results match or show only small improvement compared to DEIR, suggesting that the local reward component drives most of the improvement for Procgen. While the authors provide a possible explanation, this also limits the applicability of the method.
  • The theory in Appendix A lacks structure, is hard to follow along and raises some questions about the assumptions and soundness. Assumption 1 seems arbitrary, particularly that the discriminator loss is smaller than ln(2), and is only supported empirically. The limitations section itself limits the scope and explanatory power of the theory so much that the generality and contribution appears to be small.
  • The task diversity is limited, especially since the only performance benefits compared to DEIR appear on some Minigrid environments but not on Procgen, and there is a lack of comparison to stronger baselines such as APT (https://arxiv.org/pdf/2603.02008v1), RIDE (https://arxiv.org/pdf/2002.12292), CIC (https://arxiv.org/pdf/2202.00161) or any of the other RFPT methods mentioned in the related work section. This may also help to back the claim that representation learning and exploration should be done jointly.
  • Some parts of the paper should be refined, e.g. I'd suggest to either move the related work section before the results or cite the papers for the baselines in the evaluation section, the representation learning model should be simply called Latent MDP model since it also contains the dynamics model. I also don't think deterministic emissions is a commonly used term in RL.
  • The paper argues states that representation learning and exploration should be done jointly, but this claim is only backed weakly by the paper and has already been shown by previous works such as DEIR.
  • The paper states that extending the approach to continuous actions would require using an off-policy algorithm, but PPO can be also used for continuous actions and whether an algorithm is on-policy or off-policy has nothing to do with the actions being continuous or concrete.

Overall, the paper needs significant refinement to clarify the actual contribution and scope of the work before it can be accepted. The proposed method itself appears to be a combination of previous works, which itself is fine as long as it contributes for example by proving new insights. However, the central claims, e.g. that the method is theoretically justified in contrast to previous works, is in my opinion not sufficiently backed by the paper. Thus, I'd recommend to reject the paper in its current form. Nevertheless, I encourage the authors to clarify the actual contributions, position their work more clearly with respect to previous works and revise the structure of the paper to adequately match the claims.

Questions for authors
  • The proposed method appears like a combination of the local intrinsic reward proposed in DEIR and the global component seems similar to NovelD. If this is not the case, could you please clarify the central differences and position the work more clearly towards previous works?
  • The global reward component is defined as the novelty difference (which for example NovelD uses) in line 178 while eq. 8 is rather only the squared distance between latent state. Could you please clarify whether the novelty difference is used or not?
  • Can you please confirm which of your results backs the claim in the conclusion that representation learning and exploration should be done jointly to improve performance and generalization?
5 comments

We thank the reviewer for the careful reading and for finding the topic important and the introduction clear. Corrections we found ourselves are in our general comment.

Q2. Yes, novelty difference is used. Eq. (8) defines the novelty vtv_t, the forward-prediction error of the LMDP model, and Eq. (9) applies NovelD's clipped difference to it: max(vt−0.5vt−1,0)+1max(v_t - 0.5 v_{t-1}, 0) + 1. The formula is similar to NovelD's (we add the +1 to not cancel out the local reward term), but the novelty is not calculated from RND errors as in NovelD: it comes from the forward model, which is also one of the three losses training the encoder that the local reward reads.

The inverse and discriminative losses are blind to the latent metric, while the forward loss is not (Lemma 2), and the local reward is a Euclidean distance in that space (Eq. 7), so the forward loss shapes what the local reward measures. Also, under noise, AEGIS's local reward separates first visits from revisits better than DEIR's (added as Appendix E2). The two terms therefore do not compete: each needs the other (Q1), and the model behind the global term also improves the local one.

Q1 (relation to DEIR and NovelD). Your reading of the formulas is partly right: the local term is DEIR’s episodic reward (Eq. 7), and the global term uses NovelD’s formula, but with the forward-model error in place of an RND error (Q2), and the two are combined multiplicatively. The central difference is the model both terms read. DEIR trains its encoder with a discriminator alone, and NovelD trains a separate RND network; AEGIS trains one latent MDP model with forward, inverse, and discriminative losses, and both rewards come from it. Holding the reward formula fixed isolates this: on DoorKey-16x16 HPT, AEGIS’s own reward, global term included, falls to DEIR's level when the forward loss is removed from the objective or the forward and inverse losses are (Fig. 7). The global term cannot produce the gain without the forward loss, and removing the global term also returns performance to DEIR's level (Fig. 6): the gain comes from the objective and the two terms together, not from the reward formula alone. Combining the two existing rewards directly, DEIR's episodic reward multiplied by NovelD's RND novelty difference, is no faster than NovelD (T90 1.26M vs 0.84M for AEGIS, p < .001; Appendix E3).

Q3 (joint learning). By "jointly" we mean that the representation is trained by the same model that produces the exploration rewards: the data the rewards collect trains the encoder, and the encoder defines the rewards. The results backing this are those in Q1 and Q2. With the reward fixed, changing only the objective that trains the shared encoder moves AEGIS between DEIR's level and its full performance (Figs. 6, 7), and the forward loss makes the local reward separate first visits from revisits better than DEIR's (Appendix E2). DEIR showed that the objective training its episodic encoder affects exploration; it did not tie a dynamics model to both the encoder and a global reward.

By "generalization", the conclusion means two things, both shown. First, the method is general: one configuration, tuned once on DoorKey-8x8, performs well with and without reward-free pretraining (Fig. 3, Table 1; Appendix E4), and runs unchanged on pixel-based Procgen (Fig. 5). Second, what it learns generalizes across layouts: MiniGrid and Procgen generate a new layout every episode, so behaviour learned without any reward cannot be memorised, yet AEGIS solves the task soon after the reward appears (Fig. 3).

We hope these answers address your main concerns; we will follow up on your remaining points in a second comment, and would appreciate it if you would reconsider your assessment.

Follow-up on the remaining points.

Theory. We agree. The final version will use its additional page to move the formal statements into Section 3, where they motivate the method. The proofs and the supporting corollaries will stay in Appendix A. The revision already opens Appendix A with a map of the theoretical results: for each one, the conditions it needs, what it states, and the ablation that tests it.

On Assumption 1. It is checked empirically, but it is not arbitrary. Its bounds are chance levels: ln 2 is the loss of a discriminator that cannot tell the true next observation from a negative, and H(a∣t)H(a|t) is that of an inverse model predicting the action from the time step alone. Lemma 1 shows that a collapsed encoder attains exactly these. The assumption therefore asks only that some encoder beats chance, which any learned discriminator, DEIR's included, relies on. It is also the weakest condition the result can use: an encoder with total loss below wIH(a∣t)+wDln2w_I H(a|t) + w_D ln 2 is necessary and sufficient for a collapsed encoder not to be a global minimiser at fixed data. As an existence statement, it cannot be proven for a network class in general, but a trained encoder settles it for a task, so we checked: on DoorKey-16x16 (Table 3), each clears its own bound by a wide margin: the discriminative loss is .0128 against ln 2 = .693, and the inverse loss .0384 against H(a|t) = .953.

Tasks and comparators. We agree the task set is narrow. Three more MiniGrid tasks already have results for all seven methods, and the revision adds them as Appendix E4. With pretraining, AEGIS is ahead of DEIR on DoorKey-8x8 and FourRooms and level on KeyCorridorS4R3. A new table (Appendix E1) also shows that on DoorKey-16x16 every loss-weight setting we ran beats DEIR, so the result does not rest on one tuned setting. We limit the claimed benefit to the tasks where it appears.

Baselines. The APT link in your review (arXiv 2603.02008) points to C-TeC; APT is 2103.04551. The submission cites APT and RIDE; the revision adds CIC and C-TeC. Our comparison holds the PPO agent and policy fixed and changes only the intrinsic reward. APT and C-TeC fit that protocol, so we ran both, together with RIDE, E3B, ETD, and TNCOA, under the same pretraining setting; the results are in Appendix E5. AEGIS is the only method that solves BlockedUnlockPickup, and the fastest on DoorKey-16x16 (p < .001 against each method that solves it); APT and C-TeC solve neither task within the budget. CIC does not fit: it learns a skill-conditioned policy with DDPG and is evaluated by fine-tuning a chosen skill, so it changes the policy and its training, not just the reward.

Smaller points: Section 3 already introduces the representation learning model as a Latent MDP (LMDP) model and calls it that throughout. “Emission” is the standard term for the observation distribution q(o|s) in the POMDP and Block MDP literature (Du et al., 2019)(https://arxiv.org/abs/1901.09018), and Section 2 defines it; we will add “observation function” at its first mention. The revision cites the baselines in the evaluation section instead of the related work, which comes later. On continuous actions, you are right that PPO handles them, and our sentence gave the wrong impression. The limitation remains for other reasons: the LMDP's inverse model is a classifier over a discrete action set; Assumption 1 and the analysis in Appendix A assume a finite action set; and the codebase and several baselines are built for discrete actions, so a fair comparison would require adapting each of them, and those adaptations would themselves be open to question. We will rewrite the sentence to state these reasons more clearly.

We thank the reviewer again for the detailed review.

Reviewer_RchSOpenReview ↗

Rebuttal acknowledgement

Thank you very much for your answers and your effort! After carefully reading the updated version of the paper, your answers, the other reviews and your answers to them, some of my concerns (clarifications, baselines, PPO) are addressed. However, my main concerns about the paper's soundness, its claims and scope remain. Please see my remaining concerns below.

My main concern is still that the paper is incoherent and the theory lacks structure, which makes it really hard to follow and estimate its actual contribution. The paper relies mostly on reward definitions proposed by DEIR and NovelD, and uses a multiplicative combination, which was studied in https://proceedings.mlr.press/v202/henaff23a/henaff23a.pdf. The paper claims that the main focus or contribution compared to previous work is a combination of these global and local intrinsic rewards that is theoretically justified. While the authors have now clarified during the rebuttal that AEGIS uses the forward-model error in place of an RND error compared to NovelD, the scope of the paper is quite narrow. The theoretical justification is in my opinion incoherent and hard-to-follow. Some assumptions, e.g. assumption 1 are only empirically checked, and the scope/generalization are limited. Thus, the central contribution of the paper is not sufficiently backed up. Additionally, please refer to my other concerns and follow-up questions below.

The conclusion states: "In this work, we identify a critical gap in model-free RFPT for RL: existing methods either learn the representation independently of exploration...", is inaccurate since previous works such as CIC or DEIR do learn both jointly.

Table 8:

  • Global results barely better than random
  • Why are there no global results for DEIR? They can be computed even if DEIR does not use it

Figure 23: Why was only E3B used and not E3B x NovelD or E3B x RND, which scored higher in the paper and which would be the more direct competitors?

Comments on the paper

1 comment

We thank all reviewers and the AC for their time and their careful and constructive reviews. This comment lists the content the revised draft adds, the reviewer who asked for it, and the updates that came from re-checking our results while answering. New content is marked in red in the revision; new experiments and comparisons are collected in Appendix E, and the final version will move them to their appropriate places. The results are discussed in the revised draft and in our reply to each reviewer.

Added in the revision:

  1. Loss-weight sensitivity on DoorKey-16x16 (Appendix E1), on the tuning budget raised by E8Cg.
  2. First visits against revisits under observation noise, for the local and global rewards (Appendix E2), requested by E8Cg.
  3. Matched combinations of existing components (DEIR×NovelD, DEIR×NGU, DEIR×ICM), requested by GRTz, and AEGIS-RND, requested by E8Cg (Appendix E3).
  4. Three more MiniGrid tasks: DoorKey-8x8, FourRooms, and KeyCorridorS4R3 (Appendix E4), on the task scope raised by RchS and GRTz.
  5. Extra baselines: ETD, RIDE, E3B, APT, C-TeC, and TNCOA, under the paper's protocol with reward-free pretraining on BlockedUnlockPickup and DoorKey-16x16, 10 seeds (Appendix E5). Requested by E8Cg (ETD, E3B, TNCOA) and RchS (APT, C-TeC, RIDE).
  6. A map of the theoretical results at the start of Appendix A (RchS), a comparison table with related methods (Appendix E6; E8Cg), a specification of the Fig. 8 variants and a min-max variant with the full LMDP loss (E8Cg), citations of the baselines in the evaluation section (RchS).

Updates from re-checking our results:

  1. Wording. Remark 1 now matches what Lemma 2 shows, and a bound on partial collisions is added (Appendix A). The abstract, related work, and conclusion now state the gap to prior work more narrowly, and the works E8Cg lists are cited in Appendix E6 for now. The hyperparameter tables now include Procgen, where all methods use the settings of Wan et al., and ICM’s MiniGrid coefficient typo was corrected to the value used in the runs.
  2. Fig. 8. An argument-parsing bug made the FDM+IDM curve duplicate the FDM run; it is re-run and the figure updated.
  3. Fig. 9. The noise option of the public DEIR code we built on never reached the environment. We re-ran the experiment with noise; the text now says AEGIS degrades less than DEIR, not that it is completely unaffected.

Planned for the final version:

  1. The extra baselines on KeyCorridorS6R3.
  2. Appendix E collects the additions of this revision in one place for review; its parts will move to where they belong, including the comparison with related methods (Appendix E6) into Section 6.

Based on the reviewers' comments, the main text changes in about 20 lines; the rest of the revision is in the appendix, most of it the new experiments in Appendix E.

We thank the reviewers and the AC for their time and careful reading, and we would be glad to discuss any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.