acceptodds
Under review as a conference paper at ICLR 2027

Exploration by Maximizing the Entropy of Random-Network Residuals

Abstract

State-entropy maximization is a standard objective for reward-free exploration: it rewards states that are rare among those the current policy visits, so collecting diverse data is the objective itself. But the objective equates explored with visited. A rarely visited state stays rewarding after the agent's models have fit it, and a frequently visited one loses its reward while they still fail there. We propose to maximize the entropy of prediction residuals instead of states. We use the residual vector of random network distillation (RND), the difference between the outputs of a fixed random network and of a predictor trained to match it on the agent's experience, and apply to it the nearest-neighbour entropy estimator of state-entropy methods. The residuals of states the predictor has fit are small and lie close together, so these states earn little reward. The new objective still rewards diversity, now measured in what the predictor has yet to fit. We show that the resulting reward keeps RND's preference for large residuals and adds a preference for residuals that are rare among those the current policy produces. We implement the method in a world-model agent and evaluate it in the reward-free setting on a range of continuous-control tasks. It is competitive with the state of the art and nearly always significantly improves over state-entropy maximization.

Primary area:
reinforcement learning
Keywords:
exploration, random network distillation, RND, reward-free, entropy-maximization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iMxUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Random-Network Residual Entropy (RRE), an exploration objective for reward-free reinforcement learning. Instead of maximizing the entropy of states directly, RRE maximizes the entropy of prediction residuals from a Random Network Distillation (RND) model using a nearest-neighbor estimator. This approach bridges state-entropy maximization and prediction-error exploration: it maintains the wide coverage of entropy-based methods, while states that are already well-predicted earn little reward. Consequently, high rewards are concentrated specifically on states that are both unlearned and rarely visited. Experiments are conducted within a unified DreamerV2 framework across four pixel-based OGBench environments and three Noisy-TV variants. RRE roughly doubles the aggregate state coverage of both APT and RND, and is competitive with the state-of-the-art ensemble method CIG while training only a single predictor. The ablations confirm that the full vector geometry of residuals is essential (scalar prediction error falls short), cross-rollout pooling is critical to prevent collapse, and splitting the sample pool reveals a trade-off between computational efficiency and estimation accuracy.

Critical strengths
  1. Simple and intuitive idea The RRE resolves both the epistemic blindness of state-entropy methods and the frequency blindness of prediction-error methods. It achieves both effects from a single quantity (e.g. the k-NN density of residual vectors) without adding any extra neural networks. In addition, the scale-invariance property (Sec. 3.2) is a practical bonus that removes RND's dependence on running normalization statistics.
  2. Principled Theoretical Formulation and Insightful Two-Space Analysis (Sec. 3.1-3.3, Eqs. 4-8) The theoretical derivation in Section 3 is well-structured and provides insight into why residual entropy works. Rather than compressing prediction error into a scalar norm, retaining the full dimensional residual vector (Eq. 4) preserves directional error structure. The authors frame exploration as maximizing the entropy of the residual distribution (Eq. 5), which is computed via a nearest-neighbor estimator (Eq. 6). A key advantage of this k-NN reward is its scale-invariance. The analysis in Section 3.3 clearly shows the mechanism from both state space and residual space.
  3. Fair experiments. (Sec. 4, Figs. 2-4, Tables 1-4) The experimental setup is fair and thoroughly analyzed, evaluating all methods on a shared DreamerV2 backbone with bit-identical resets, and measuring state coverage across diverse tasks using standard evaluation metrics.
  4. Insightful ablations. (Sec. 4.3-4.4, Fig. 4) The APT x RND ablation (Sec. 4.3) effectively confirms that residual vector geometry is essential. In addition, the own-rollout vs. other-rollouts ablation (Sec. 4.4) clearly shows the importance of cross-trajectory occupancy estimation over within-trajectory diversity.
Critical weaknesses
  1. No downstream task evaluation (Sec. 4) Exploration is evaluated solely through state coverage without testing downstream task adaptation. Although state coverage is the standard metric in OGBench, showing that RRE actually accelerates fine-tuning on 1–2 downstream tasks would significantly strengthen the empirical claims.
  2. Quadratic computational complexity (Sec. 3.2, 4.4, Table 3) Computing k-NN rewards over a pool of N = 37,500 residual vectors (L = 512) incurs an O(N^2 L) pairwise distance cost per update. Sec. 4.4 shows that partitioning the pool to save computation degrades coverage, confirming dependence on the full pool. Without wall-clock times or FPS comparisons against ensemble baselines (CIG, P2E), it remains unclear whether relying on a single predictor actually makes the method lighter or simpler, as the computational burden shifts to the k-NN distance calculation.
  3. Unverified theoretical assumptions in MLP residual mapping (Sec. 3.3) The derivations in Eqs. 7-8 assume an injective residual map with a full-rank Jacobian. However, the 4-layer ELU predictor and target MLPs (Table 3) may inherently violate injectivity as learned states collapse to near-zero residuals, turning these identities into upper bounds (Footnote 2). Without empirical inspection of Jacobian rank or bound tightness, this theoretical equivalence remains unverified during training.
  4. Ablations are limited to a single environment (Sec. 4.3-4.4, Fig. 4) Both the residual geometry ablation (APT x RND vs. RRE) and the pool composition ablation are conducted only on Puzzle 3x3. Because Puzzle 3x3 is a discrete task, it remains unclear whether the conclusions generalize to continuous locomotion domains like AntMaze, where exploration dynamics may differ. Testing these ablations on at least one continuous environment would strengthen the claims.
  5. Clarification on reward normalization (Sec. 3.2, Table 4) Section 3.2 emphasizes that RRE is scale-invariant and avoids running normalization, yet Appendix Table 4 shows that both EMA z-score and Welford running normalizers are applied to all methods in practice. The authors should clarify whether this normalization is simply a shared DreamerV2 default or if RRE actually relies on it.
Questions for authors
  1. The ablation as shown in Sec. 4.4 (Fig. 4) demonstrates monotonic coverage degradation as K increases. Is there an empirical or theoretical guideline for choosing pool size N against the available computational budget when adapting RRE to new environments?
  2. How sensitive is RRE to the residual dimension L (e.g. 512 in Table 3 in Appendix)? Could reducing L (e.g., to 64 or 128) alleviate the O(N^2 L) computational burden without sacrificing coverage quality?
  3. On Cube-Triple-NT as shown in Sec. 4.2( Figures 2–3), the visual noisy-TV increases APT's state coverage from 0.26 to 0.86. Do the authors have a deeper intuitive explanation for why state-entropy exploration benefits so dramatically from visual noise in this specific environment?
2 comments

We thank the reviewer for the careful reading and the constructive questions. We answer each one below.


1. Is there a guideline for choosing the pool size NN against the computational budget?

Our recommendation is simple: use the largest pool that the available memory and runtime allow. The pool size does not need to be tuned per environment. A larger pool lets each residual be compared with more of the occupancy, so the only reason to limit it is the cost of the pairwise distances, which grows quadratically in NN.

The Sub-Pools ablation supports this: within the range we tested, coverage improves steadily with the pool size, and we see no point beyond which a larger pool stops helping. The ablation also shows that the choice is forgiving. Coverage declines gradually as the pool shrinks, so a pool that is smaller than ideal costs some coverage but does not cause exploration to fail.

For a new environment, we therefore suggest starting with the largest pool that fits on the GPU and, if the reward computation becomes a bottleneck, splitting the pool into as few sub-pools as needed.

2. How sensitive is RRE to the residual dimension L (e.g. 512 in Table 3 in Appendix)? Could reducing L (e.g., to 64 or 128) alleviate the O(N^2 L) computational burden without sacrificing coverage quality?

Generally, reducing L does not necessarily alleviate the computational burden without sacrificing coverage quality.

Sensitivity to LL. We ran an ablation over L∈{32,128,512,2048}L \in \lbrace 32, 128, 512, 2048 \rbrace on the three OGBench environments (AntMaze, Cube-Triple, Puzzle-3x3), with five seeds per setting. The table reports final coverage after 2.5M environment steps (IQM across seeds with 95% bootstrap confidence intervals):

EnvironmentL=32L=32L=128L=128L=512L=512 (default)L=2048L=2048
AntMaze0.97 [0.94, 1.00]1.00 [1.00, 1.00]1.00 [0.97, 1.00]1.00 [0.96, 1.00]
Cube-Triple0.65 [0.59, 0.70]0.69 [0.63, 0.76]0.70 [0.64, 0.76]0.75 [0.69, 0.81]
Puzzle-3x30.59 [0.55, 0.64]0.81 [0.75, 0.86]0.96 [0.94, 0.98]0.97 [0.94, 0.99]

RRE is moderately sensitive to LL and degrades gracefully. Reducing LL to 128 leaves final coverage unchanged on AntMaze, where every L≥128L \geq 128 explores the whole maze, and nearly unchanged on Cube-Triple (0.69 vs. 0.70). It costs 0.15 final coverage on Puzzle-3x3, where coverage increases monotonically with LL. L=32L=32 is the worst setting on every environment. We therefore keep L=512L=512 as a default that works across domains instead of tuning LL per environment.

Computational cost. Only the pairwise-distance computation scales as O(N2L)O(N^2 L). Storing the N×NN \times N distance matrix (N=34,300N = 34{,}300 imagined states per update) and selecting the kk nearest neighbours from it cost O(N2)O(N^2) regardless of LL, and at our scale these terms dominate. We measured the per-update cost of computing the intrinsic reward on a single GPU:

LLTime per updatePeak GPU memory
3291 ms5.7 GiB
6489 ms5.7 GiB
12890 ms5.7 GiB
512 (default)113 ms5.9 GiB
2048220 ms7.1 GiB

Reducing LL from 512 to 128 saves about 20% of the reward computation (6.3 instead of 7.8 GPU-hours per 2.5M-step run) and less than 0.2 GiB of memory. Going below L=128L=128 saves nothing further, because the LL-independent O(N2)O(N^2) part sets the floor: all N2N^2 entries must be written to GPU memory and then scanned to find the nearest neighbours, and for small LL this memory traffic takes longer than the O(L)O(L) arithmetic needed to compute each distance. The quadratic cost is in NN, not LL, so the effective lever is the size of the nearest-neighbour pool (e.g., approximate nearest-neighbour search), which we leave for future work.

3. On Cube-Triple-NT as shown in Sec. 4.2( Figures 2–3), the visual noisy-TV increases APT's state coverage from 0.26 to 0.86. Do the authors have a deeper intuitive explanation for why state-entropy exploration benefits so dramatically from visual noise in this specific environment?

We do not currently have an explanation that we can support with evidence. The gain is specific to the two methods that maximize a kk-NN entropy estimate (APT and RRE), whereas RND and P2E are unaffected and CIG degrades, and it does not appear for APT on AntMaze-NT. This points to an interaction between entropy-based objectives and the distractor in this particular environment.


We hope these answers resolve the reviewer's questions. If these answers address the reviewer's concerns, we would be grateful if they would consider updating their score, and we are happy to answer any remaining questions during the discussion period.

Thank you for their detailed response, clarifications and additional experiments. However, several issues still remain. Therefore, I summarize unresolved points as below.

(Unresolved Points)

  1. Efficiency argument vs ensemble baselines : The paper motivates RRE to be competitive with SOTA methods while training only a single predictor. This assumption is not supported by the results in additional experiments (the authors’ response to Reviewer UEFc question 2). RRE is the slowest and most memory-heavy method (214 ms/update, 10,6GB peak memory and 14.8 GPU-hours of training) compared with four other methods (RND, P2E, CIG, APT). Especially, CIG already matches or exceeds RRE’ coverage on most tasks (3 of 4 non-distractor tasks) at lower runtime (155 ms) and memory cost (7.4 GB). These indicate that RRE has a higher computational and memory overhead despite using a simple predictor.
  2. Incomplete ablation : the APT x RND and pool-composition ablation experiments have not been conducted beyond Puzzle-3x3.
  3. Missing downstream validation . None of the rebuttal responses to this. 1-2 downstream tasks should be evaluated to show whether higher cumulative coverage during pre-training turns into better task performance when extrinsic rewards are applied.
  4. Theoretical reframing of Section 3.3 : Neither Jacobian rank nor bound tightness is measured. Moreover, in the reply to Reviewer Hzfo (Question 3), the author concedes that the residual map is non-injective and Eq (7)-(8) only hold as a special case. Under the replacement proposed in the response to Reviewer Hzfo, the claim in Section 5 that RRE is a form of State Marginal Matching (SMM) over state space is not theoretically supported.
  5. Reward normalization : it is unclear whether RRE depends on the EMA z-score and Welford running normalizers in Table 4 despite Section 3.2 emphasis on scale-invariance.
  6. Cube-Triple-NT anomaly: the authors honestly admit that they have no evidence-backed explanation for why the Noisy-TV distractor boosts APT (0.26 to 0.86) above RRE. This could leave it unclear whether the NT settings reflect distractor robustness or an environment-specific interaction with the world model's latent representation.

(Conclusion) After the authors’ response, I maintain Weak Rejection. The idea of maximizing residual entropy is clear. However, my main concerns are practical efficiency and theoretical contribution. First, the additional experiments on runtime and memory usage show that RRE is slower and memory-heavier than the compared methods. This weakens RRE's positioning against SOTA ensemble baselines like CIG and P2E.

Reviewer_UEFcOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Random-Network Residual Entropy (RRE), an intrinsic motivation objective for reward-free reinforcement learning. Rather than maximizing state entropy or prediction error magnitude, RRE maximizes the entropy of residual vectors produced by a random network distillation (RND) predictor. The proposed reward is estimated using k-nearest-neighbor distances in residual space, encouraging exploration of states whose prediction residuals are underrepresented in the current policy's occupancy.

The authors provide a theoretical interpretation connecting residual entropy maximization with state entropy and prediction-error-based exploration. Experiments on seven pixel-based continuous-control settings demonstrate substantial improvements over APT and RND, with competitive performance against ensemble-based approaches.

Critical strengths

The paper introduces a conceptually appealing exploration objective that combines predictive residuals with distributional diversity. Unlike conventional RND, the proposed method considers the geometry and density of prediction errors rather than their magnitude alone.

In terms of the experiments, the proposed RRE substantially improves exploration coverage over APT and RND in several environments, particularly AntMaze. Comparisons with Plan2Explore and CIG further demonstrate the potential of the approach.The ablation studies on residual geometry and pool composition also provide useful evidence for the proposed design.

Critical weaknesses

a. Unclear applicability of the theoretical analysis. The analysis in Section 3.3 relies on full-rank and injectivity assumptions, while the implementation maps a higher-dimensional latent representation into a 512-dimensional residual space. The authors should clarify whether these assumptions hold for the actual implementation and to what extent Equations (7) and (8) remain applicable.

b. Insufficient validation of the central mechanism. The paper attributes its improvements to the geometric properties of prediction residuals, arguing that well-predicted states occupy smaller residual-space volumes. However, the experiments primarily demonstrate exploration coverage improvements without directly validating this mechanism. An analysis of residual distributions and their evolution during training would provide stronger evidence for the claimed contribution.

c. Unclear computational efficiency and scalability. The proposed k-NN reward incurs quadratic computational complexity with respect to the pool size. Despite avoiding ensemble models, the paper provides insufficient runtime and memory comparisons to establish whether RRE offers a practical efficiency advantage over existing exploration methods.

Questions for authors
  1. Can the authors provide direct empirical evidence that well-predicted regions contract in residual space while poorly predicted regions retain larger residual-space volume?

  2. How does RRE compare with RND, APT, and ensemble-based methods in terms of wall-clock time, peak GPU memory, and total training cost?

  3. Could the authors extend the residual-geometry and pool-composition ablations to at least one navigation environment and one additional manipulation environment?

1 comment

We thank the reviewer for the careful review and the constructive questions.


Question 1:

We ran some additional experiments to verify this. Each time RRE computes its reward, it does so on a batch of imagined next states. We sort the states of each batch by their prediction error ∥e(s)∥\Vert e(s)\Vert into five equal-size groups, from the best-predicted to the worst-predicted fifth. For each group we record how crowded its states are, in two spaces:

  1. Residual space: the mean distance to the kk nearest neighbours among the residuals e(s)e(s). This is exactly the quantity the RRE reward is built from.
  2. State space: the same statistic among the world model's latent states, on which APT computes its reward.

If the residual map squeezes well-predicted states together, their neighbours should be close in residual space. Their neighbours in state space need not be any closer. Comparing the two separates contraction by the map from the agent simply visiting some states more often.

Result. AntMaze, RRE, 5 seeds, averaged over 0.45–0.55M training steps (so right in the middle of exploration). Each entry is the change relative to the best-predicted group; the last column is mean ± std over seeds.

Prediction-error group1 (best)2345 (worst)
Prediction error ∥e∥\Vert e\Vert–+8%+14%+20%+30%
Neighbour distance, residual space–+4.5%+7.4%+10.3%+15.1% ± 0.4
Neighbour distance, state space–+1.8%+1.8%+1.3%−0.6% ± 1.5
  • Residual space: crowding decreases steadily from best- to worst-predicted states, in every seed. The worst-predicted fifth's neighbours are 15% farther away than the best-predicted fifth's.
  • State space: all five groups are equally crowded (differences under 2%), so the agent does not visit well-predicted states more often.
  • Interpretation: well-predicted states are packed together by the residual map itself, while poorly predicted states keep more residual volume.

Control: is this just a side effect of grouping by norm? We ran the same agent with the kk-NN computed on the frozen target output g(s)g(s) instead of the residual. The predictor is still trained but does not enter the reward. We grouped states by ∥g(s)∥\Vert g(s)\Vert, which has a similar spread across groups. The trend disappears: −1.1% ± 0.6 between the outer groups, and no seed shows a steady increase. The contraction therefore comes from the predictor's fit, not from the embedding or the grouping.

Over training. The contraction also shows as the predictor learns. From the first 50k steps to 0.5M, typical neighbour distances in residual space shrink by about 40%, while those in state space grow by about 80% as the agent spreads out. In the frozen-target control, neighbour distances instead grow by about 60%, following the states.

Finally, this volume difference follows directly from how RND is trained. The predictor is fit on replay states, that is, on states the agent has already explored, so their residuals shrink, while states it has rarely or never reached stay unfitted. We also expect a much sharper contrast for states directly at the exploration frontier.

We will add this analysis to the appendix.

Question 2:

We measured the training cost of each method on one RTX 5070 Ti (16 GB) with the paper's configuration (compiled, mixed precision, 50×50 replay batch, NN = 34,300 imagined states per update), excluding environment interaction, which is identical across methods.

RNDP2ECIGAPTRRE
Time per update (ms)116140155197214
Peak GPU memory (GB)7.47.67.410.210.6
Training GPU-hours per 2.5M steps8.09.710.713.714.8

All methods share world-model and actor-critic training (≈100 ms per update). RRE's additional cost comes from the kk-nearest-neighbour entropy estimate it shares with APT, so RRE costs 9% more time and 0.4 GB more memory than APT. RRE's peak memory is dominated by the pairwise distance matrix over imagined states and fits on a 16 GB consumer GPU. RRE's higher per-update cost is partly offset by faster exploration: averaged over the six environments, RRE reaches RND's final coverage with 34% of the training compute of a full RND run, while RND does not reach RRE's final coverage in any of the six environments. When interaction with the environment is costly, the comparison favours RRE further, since it reaches RND's final coverage in far fewer environment steps; in this setting RRE is preferable to RND.

Question 3:

For the residual-geometry ablations across an additional manipulation and navigation environment, we kindly refer you to our response to Reviewer Hzfo. For the pool-composition ablations, the experiments are currently running; we will share those results here, provided the runs conclude in time.


We hope our answers resolve your concerns. If so, we would be grateful if you would consider re-evaluating your score.

Reviewer_HzfoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies reward-free exploration, the contribution is RRE, which maximizes the entropy of RND residuals instead of states, using the APT k-NN estimator on residuals, with a change-of-variables reading of it as state entropy plus a log-stretch term set by the predictor.

The validation is on DreamerV2 with 4 OGBench pixel envs and 3 noisy-TV variants (5 seeds, IQM with bootstrap CI) against APT, RND, Plan2Explore and CIG.

Critical strengths
  1. The idea is simple and well motivated: one objective that combines RND's "what is not learned yet" with the dispersion of state entropy. The change-of-variables reading (Eq 7-8) explains why well-fitted regions get little reward..

  2. The reported gains over the two methods it combines are large and the limitations section is honest about noisy TV, cost and the single agent.

Critical weaknesses
  1. The gain is not isolated to the residual. APT vs RRE changes the input (GRU state vs full latent, Table 3) and in the code also the reward (APT log(1+d) in apt.py, RRE log d in apt_deviation.py; Eq 2 has no +1). No k-NN on g(s) or the same latent is tested, and the only control (APT x RND) is on Puzzle 3x3.
  2. The evaluation does not support the abstract's "nearly always significantly improves". No test is named and CIs overlap on three of seven envs. The code runs 2.5M steps but Table 1 reports 2.0M / 1.0M / 2.0M, seeds come from time(), Fig 3 leaves out P2E and CIG, and only one agent and one benchmark are used.
Questions for authors
  1. Can you provide k-NN entropy on g(s) alone and on the same latent, with the same pool, k, reward form (log d) and agent, on all envs? And how does APT do with log d instead of log(1+d)? (suggestion)
  2. Why are the reported budgets (Table 1) shorter than the 2.5M steps in the configs, especially Puzzle 3x3 at 1.0M? Does the ranking hold to the end of the runs? (clarification)
  3. Eq 7 assumes a fixed predictor, L >= n and an injective map (footnote 2), but L=512 while the predictor input is 1,624-dim (computed from Table 3/4), and the predictor keeps training. Does the analysis still describe the trained agent? (clarification)
1 comment

Question 1:

Thank you for the suggestion. In AntMaze, Puzzle-3x3 and Cube-Triple, we ran kk-NN entropy on g(s)g(s) with reward log⁡d\log d, and APT (kk-NN on the latent ss) with log⁡d\log d instead of log⁡(1+d)\log(1+d), both with RRE's pool, kk and agent. The original APT implementation (Liu & Abbeel, 2021) uses log⁡(1+d)\log(1+d), which our APT baseline adopts.

AntMazePuzzle-3x3Cube-Triple
RRE1.0000.9620.695
kk-NN on g(s)g(s), log⁡d\log d0.1670.9540.566
APT (log⁡d\log d)0.1670.9890.289
APT (log⁡(1+d)\log (1+d))0.1920.9350.319
RND0.2820.5380.501

IQM coverage over 5 seeds at 2.5M steps.

In AntMaze and Cube-Triple, both controls remain below RRE, in AntMaze by a wide margin. This suggests that the predictor's residual, rather than the random projection or the reward form alone, accounts for much of RRE's gain in these environments. In Puzzle-3x3, log⁡d\log d improves APT, which then slightly exceeds RRE; the reward form thus appears to matter where latent kk-NN already explores well. We will add the noisy TV variants to the revision once they finish.

Question 2:

In Puzzle-3x3, the leading methods approach the ceiling of the metric early, which compresses later differences. Among the methods of Table 1, RRE's rank is unchanged for 2.5M in Puzzle-3x3 (first) and Cube-Triple (second, behind CIG); in AntMaze, RRE and CIG both reach full coverage. RND, which rewards the squared prediction error, ends well below RRE in every environment. We plan to report both budgets, CIs and full learning curves in the revision.

Question 3:

We thank the reviewer for this question. The mechanism described by the analysis also holds for the trained agent, where L=512L = 512, the predictor input is 1,624-dimensional and the predictor keeps training. The conditions of footnote 2 are needed only for the closed forms (7)–(8).

RRE rewards a state when few of the current residuals lie near its own. Two states thus count as different only insofar as the predictor's error tells them apart. States the predictor has fit all have residuals near the origin, so they crowd one another however far apart they are as states, and are rewarded like a single, densely visited state. States it has not fit have residuals that vary from state to state, so there the policy is paid for spreading out, as in state-entropy maximization. In short, RRE disperses the policy wherever the predictor is still wrong.

Formally, the standard entropy decomposition applies to the residuals:

\mathcal{H}(e_\\# d^\pi) = \log \mathrm{vol}(e(\mathcal{S}_R)) - D_{\mathrm{KL}}(e_\\# d^\pi \Vert U_{e(\mathcal{S}_R)})

RRE thus pulls the residuals toward the uniform distribution on the reachable residuals, under which a region of states receives at most the share of residual volume it covers, however large it is in state space. For a tolerance ε>0\varepsilon > 0, consider all states fit to within ε\varepsilon, i.e., with ∥e(s)∥≤ε\lVert e(s) \rVert \le \varepsilon. Their residuals lie in the ball of radius ε\varepsilon around the origin, so together they receive at most that ball's share of the residual volume, which vanishes as ε→0\varepsilon \to 0 for any ee.

Merging states is how the filter operates: an exact fit maps a whole region to one point and is itself non-injective. Under the conditions, states with nearby residuals are nearby states, and the residual surprisal becomes "state surprisal plus log-stretch". Here, the relevant nn is the effective dimension of the reachable states, which may lie on a low-dimensional surface in the 1,624-dimensional input space. Without the conditions, RRE disperses the policy across the states that the residual separates.

Each reward is computed with the current predictor, so the fixed-ee analysis describes the reward the agent receives, just as the state density is held fixed in state-entropy maximization. Training moves the target: fitting the visited states draws their residuals toward the origin, they cover less residual volume, and the policy is directed elsewhere. The analysis concerns this per-step, non-stationary objective; the limit of the coupled dynamics is beyond its scope.

Revision. We will present the analysis in this general form, without conditions on LL, nn or injectivity, keep (7)–(8) as the special case where the residual separates all reachable states, and report the predictor's input dimension.


We thank the reviewer again for the constructive feedback. If the additional results and clarifications resolve the reviewer's concerns, we would kindly ask the reviewer to consider updating their score accordingly; we are happy to clarify any remaining points during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.