- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Random-Network Residual Entropy (RRE), an exploration objective for reward-free reinforcement learning. Instead of maximizing the entropy of states directly, RRE maximizes the entropy of prediction residuals from a Random Network Distillation (RND) model using a nearest-neighbor estimator. This approach bridges state-entropy maximization and prediction-error exploration: it maintains the wide coverage of entropy-based methods, while states that are already well-predicted earn little reward. Consequently, high rewards are concentrated specifically on states that are both unlearned and rarely visited. Experiments are conducted within a unified DreamerV2 framework across four pixel-based OGBench environments and three Noisy-TV variants. RRE roughly doubles the aggregate state coverage of both APT and RND, and is competitive with the state-of-the-art ensemble method CIG while training only a single predictor. The ablations confirm that the full vector geometry of residuals is essential (scalar prediction error falls short), cross-rollout pooling is critical to prevent collapse, and splitting the sample pool reveals a trade-off between computational efficiency and estimation accuracy.
Critical strengths
- Simple and intuitive idea The RRE resolves both the epistemic blindness of state-entropy methods and the frequency blindness of prediction-error methods. It achieves both effects from a single quantity (e.g. the k-NN density of residual vectors) without adding any extra neural networks. In addition, the scale-invariance property (Sec. 3.2) is a practical bonus that removes RND's dependence on running normalization statistics.
- Principled Theoretical Formulation and Insightful Two-Space Analysis (Sec. 3.1-3.3, Eqs. 4-8) The theoretical derivation in Section 3 is well-structured and provides insight into why residual entropy works. Rather than compressing prediction error into a scalar norm, retaining the full dimensional residual vector (Eq. 4) preserves directional error structure. The authors frame exploration as maximizing the entropy of the residual distribution (Eq. 5), which is computed via a nearest-neighbor estimator (Eq. 6). A key advantage of this k-NN reward is its scale-invariance. The analysis in Section 3.3 clearly shows the mechanism from both state space and residual space.
- Fair experiments. (Sec. 4, Figs. 2-4, Tables 1-4) The experimental setup is fair and thoroughly analyzed, evaluating all methods on a shared DreamerV2 backbone with bit-identical resets, and measuring state coverage across diverse tasks using standard evaluation metrics.
- Insightful ablations. (Sec. 4.3-4.4, Fig. 4) The APT x RND ablation (Sec. 4.3) effectively confirms that residual vector geometry is essential. In addition, the own-rollout vs. other-rollouts ablation (Sec. 4.4) clearly shows the importance of cross-trajectory occupancy estimation over within-trajectory diversity.
Critical weaknesses
- No downstream task evaluation (Sec. 4) Exploration is evaluated solely through state coverage without testing downstream task adaptation. Although state coverage is the standard metric in OGBench, showing that RRE actually accelerates fine-tuning on 1–2 downstream tasks would significantly strengthen the empirical claims.
- Quadratic computational complexity (Sec. 3.2, 4.4, Table 3) Computing k-NN rewards over a pool of N = 37,500 residual vectors (L = 512) incurs an O(N^2 L) pairwise distance cost per update. Sec. 4.4 shows that partitioning the pool to save computation degrades coverage, confirming dependence on the full pool. Without wall-clock times or FPS comparisons against ensemble baselines (CIG, P2E), it remains unclear whether relying on a single predictor actually makes the method lighter or simpler, as the computational burden shifts to the k-NN distance calculation.
- Unverified theoretical assumptions in MLP residual mapping (Sec. 3.3) The derivations in Eqs. 7-8 assume an injective residual map with a full-rank Jacobian. However, the 4-layer ELU predictor and target MLPs (Table 3) may inherently violate injectivity as learned states collapse to near-zero residuals, turning these identities into upper bounds (Footnote 2). Without empirical inspection of Jacobian rank or bound tightness, this theoretical equivalence remains unverified during training.
- Ablations are limited to a single environment (Sec. 4.3-4.4, Fig. 4) Both the residual geometry ablation (APT x RND vs. RRE) and the pool composition ablation are conducted only on Puzzle 3x3. Because Puzzle 3x3 is a discrete task, it remains unclear whether the conclusions generalize to continuous locomotion domains like AntMaze, where exploration dynamics may differ. Testing these ablations on at least one continuous environment would strengthen the claims.
- Clarification on reward normalization (Sec. 3.2, Table 4) Section 3.2 emphasizes that RRE is scale-invariant and avoids running normalization, yet Appendix Table 4 shows that both EMA z-score and Welford running normalizers are applied to all methods in practice. The authors should clarify whether this normalization is simply a shared DreamerV2 default or if RRE actually relies on it.
Questions for authors
- The ablation as shown in Sec. 4.4 (Fig. 4) demonstrates monotonic coverage degradation as K increases. Is there an empirical or theoretical guideline for choosing pool size N against the available computational budget when adapting RRE to new environments?
- How sensitive is RRE to the residual dimension L (e.g. 512 in Table 3 in Appendix)? Could reducing L (e.g., to 64 or 128) alleviate the O(N^2 L) computational burden without sacrificing coverage quality?
- On Cube-Triple-NT as shown in Sec. 4.2( Figures 2–3), the visual noisy-TV increases APT's state coverage from 0.26 to 0.86. Do the authors have a deeper intuitive explanation for why state-entropy exploration benefits so dramatically from visual noise in this specific environment?
2 comments
Thank you for their detailed response, clarifications and additional experiments. However, several issues still remain. Therefore, I summarize unresolved points as below.
(Unresolved Points)
- Efficiency argument vs ensemble baselines : The paper motivates RRE to be competitive with SOTA methods while training only a single predictor. This assumption is not supported by the results in additional experiments (the authors’ response to Reviewer UEFc question 2). RRE is the slowest and most memory-heavy method (214 ms/update, 10,6GB peak memory and 14.8 GPU-hours of training) compared with four other methods (RND, P2E, CIG, APT). Especially, CIG already matches or exceeds RRE’ coverage on most tasks (3 of 4 non-distractor tasks) at lower runtime (155 ms) and memory cost (7.4 GB). These indicate that RRE has a higher computational and memory overhead despite using a simple predictor.
- Incomplete ablation : the APT x RND and pool-composition ablation experiments have not been conducted beyond Puzzle-3x3.
- Missing downstream validation . None of the rebuttal responses to this. 1-2 downstream tasks should be evaluated to show whether higher cumulative coverage during pre-training turns into better task performance when extrinsic rewards are applied.
- Theoretical reframing of Section 3.3 : Neither Jacobian rank nor bound tightness is measured. Moreover, in the reply to Reviewer Hzfo (Question 3), the author concedes that the residual map is non-injective and Eq (7)-(8) only hold as a special case. Under the replacement proposed in the response to Reviewer Hzfo, the claim in Section 5 that RRE is a form of State Marginal Matching (SMM) over state space is not theoretically supported.
- Reward normalization : it is unclear whether RRE depends on the EMA z-score and Welford running normalizers in Table 4 despite Section 3.2 emphasis on scale-invariance.
- Cube-Triple-NT anomaly: the authors honestly admit that they have no evidence-backed explanation for why the Noisy-TV distractor boosts APT (0.26 to 0.86) above RRE. This could leave it unclear whether the NT settings reflect distractor robustness or an environment-specific interaction with the world model's latent representation.
(Conclusion) After the authors’ response, I maintain Weak Rejection. The idea of maximizing residual entropy is clear. However, my main concerns are practical efficiency and theoretical contribution. First, the additional experiments on runtime and memory usage show that RRE is slower and memory-heavier than the compared methods. This weakens RRE's positioning against SOTA ensemble baselines like CIG and P2E.
We thank the reviewer for the careful reading and the constructive questions. We answer each one below.
1. Is there a guideline for choosing the pool size N against the computational budget?
Our recommendation is simple: use the largest pool that the available memory and runtime allow. The pool size does not need to be tuned per environment. A larger pool lets each residual be compared with more of the occupancy, so the only reason to limit it is the cost of the pairwise distances, which grows quadratically in N.
The Sub-Pools ablation supports this: within the range we tested, coverage improves steadily with the pool size, and we see no point beyond which a larger pool stops helping. The ablation also shows that the choice is forgiving. Coverage declines gradually as the pool shrinks, so a pool that is smaller than ideal costs some coverage but does not cause exploration to fail.
For a new environment, we therefore suggest starting with the largest pool that fits on the GPU and, if the reward computation becomes a bottleneck, splitting the pool into as few sub-pools as needed.
2. How sensitive is RRE to the residual dimension L (e.g. 512 in Table 3 in Appendix)? Could reducing L (e.g., to 64 or 128) alleviate the O(N^2 L) computational burden without sacrificing coverage quality?
Generally, reducing L does not necessarily alleviate the computational burden without sacrificing coverage quality.
Sensitivity to L. We ran an ablation over L∈{32,128,512,2048} on the three OGBench environments (AntMaze, Cube-Triple, Puzzle-3x3), with five seeds per setting. The table reports final coverage after 2.5M environment steps (IQM across seeds with 95% bootstrap confidence intervals):
RRE is moderately sensitive to L and degrades gracefully. Reducing L to 128 leaves final coverage unchanged on AntMaze, where every L≥128 explores the whole maze, and nearly unchanged on Cube-Triple (0.69 vs. 0.70). It costs 0.15 final coverage on Puzzle-3x3, where coverage increases monotonically with L. L=32 is the worst setting on every environment. We therefore keep L=512 as a default that works across domains instead of tuning L per environment.
Computational cost. Only the pairwise-distance computation scales as O(N2L). Storing the N×N distance matrix (N=34,300 imagined states per update) and selecting the k nearest neighbours from it cost O(N2) regardless of L, and at our scale these terms dominate. We measured the per-update cost of computing the intrinsic reward on a single GPU:
Reducing L from 512 to 128 saves about 20% of the reward computation (6.3 instead of 7.8 GPU-hours per 2.5M-step run) and less than 0.2 GiB of memory. Going below L=128 saves nothing further, because the L-independent O(N2) part sets the floor: all N2 entries must be written to GPU memory and then scanned to find the nearest neighbours, and for small L this memory traffic takes longer than the O(L) arithmetic needed to compute each distance. The quadratic cost is in N, not L, so the effective lever is the size of the nearest-neighbour pool (e.g., approximate nearest-neighbour search), which we leave for future work.
3. On Cube-Triple-NT as shown in Sec. 4.2( Figures 2–3), the visual noisy-TV increases APT's state coverage from 0.26 to 0.86. Do the authors have a deeper intuitive explanation for why state-entropy exploration benefits so dramatically from visual noise in this specific environment?
We do not currently have an explanation that we can support with evidence. The gain is specific to the two methods that maximize a k-NN entropy estimate (APT and RRE), whereas RND and P2E are unaffected and CIG degrades, and it does not appear for APT on AntMaze-NT. This points to an interaction between entropy-based objectives and the distractor in this particular environment.
We hope these answers resolve the reviewer's questions. If these answers address the reviewer's concerns, we would be grateful if they would consider updating their score, and we are happy to answer any remaining questions during the discussion period.