- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes training RL agents to maximize return minus a penalty on the entropy of their noisy history representations, pooled over the episode. The authors motivate this with a finite-sample argument: when the visited representations have low entropy, small policy updates can be evaluated from fewer episodes, and they give a construction where a high-entropy optimal policy needs exponentially many episodes while a simpler near-optimal one does not. They also derive the policy gradient of the objective, which splits into a surprise-like reward that shapes behavior and a direct penalty on the encoder. They argue the first term makes agents act to keep their situations predictable (“entropy resistance”). The resulting algorithm, BRPO, estimates the penalty with a diffusion model and plugs it into PPO. Experiments on several POMDP benchmarks (RockSample, NoisyTV, LaserTag, LightDark) report higher returns than recurrent PPO and planning baselines, with the gap growing with task size, along with a double-pendulum experiment on entropy resistance and ablations of the two penalty terms.
Critical strengths
-
Statistical account of why predictability helps learning. Section 5 gives an entropy-dependent bound on the sample cost of evaluating local policy updates, together with a matching construction (Thm 5.2) showing that this exponential dependence on representation entropy can actually occur. The upper/lower-bound pair makes the argument more than a heuristic: low-entropy representations make greedy local improvement statistically easier.
-
Two mechanisms, cleanly separated and tested. The policy-gradient result (Thm 6.1) splits the penalty into a reward-side term that shapes behavior and a direct encoder term that compresses representations. The λ1/λ2 ablations then show complementary roles. The reward-side term changes which situations the agent visits, while the encoder term helps retain a task-relevant cue amid distractors, a somewhat counterintuitive effect of penalizing entropy.
-
Sound and transparent. The paper is mostly sound, and the proofs are correct as far as I checked. It is candid about the limits of its theory: Appendix B.3 and the Limitations section state which conditions are assumed and what remains open. Environments, hyperparameters, and evaluation protocols are documented in detail.
Critical weaknesses
-
The paper presents its objective (return minus the entropy of history representations) as the main contribution, but this objective is not new. The closest work, Robust Predictable Control (Eysenbach et al., NeurIPS 2021), is not cited. RPC jointly trains encoder and policy to minimize representation information and observes the same behavioral effect: the policy learns to keep its representations predictable. SMiRL (Berseth et al., 2021) is cited, but its closeness is understated: it already minimizes state-marginal entropy, including on learned latents, combined with task reward. IC2 (Rhinehart et al., 2021) and information-bottleneck RL (e.g., Igl et al., 2019) are also closely related. Introducing the objective is therefore not a contribution. The paper could instead be positioned as explaining why such objectives are beneficial and proposing a method (BRPO, with its diffusion-ELBO estimator) for optimizing them. Similarly, BRPO is essentially PPO with a SMiRL-style surprise reward on the recurrent representations plus a direct encoder penalty, with the occupancy density estimated by a diffusion model instead of a simpler density model (e.g., the Gaussian used in SMiRL). The paper does not ablate this choice, so it is unclear whether the diffusion estimator matters.
-
Regarding the theoretical contribution: Section 5 bounds the sample cost of evaluating local updates around a fixed policy, i.e., a possible benefit during learning. It gives no guarantee on the return of the policy that learning produces. The motivation as "following the principle of SRM" is therefore misleading: SRM's justification is precisely such a guarantee, bounding the selected predictor's true risk by empirical risk plus complexity.
-
Regarding the empirical contribution, the claimed advantages of BRPO are under-supported. BRPO is not compared against the most relevant baselines: a SMiRL-style surprise reward on the recurrent representations, RPC, and an information-bottleneck encoder (e.g., VIB/IBAC). Clear gains appear on only two of four tasks (RockSample, NoisyTV); LaserTag and LightDark are within noise. All results use three seeds without significance tests, and the tuning budgets for BRPO versus the baselines (including planner simulation budgets) are not reported.
Questions for authors
-
Given the closeness to RPC, SMiRL, and IC2, could the authors state precisely which contributions they consider novel relative to this prior work? In particular, what does the pooled occupancy entropy capture that RPC's information penalty or SMiRL's latent surprise does not?
-
Could the authors compare BRPO against the most relevant baselines: PPO-RNN with a SMiRL-style surprise reward on the recurrent representations, RPC, and an information-bottleneck encoder (e.g., VIB/IBAC)?
-
Could the authors ablate the density model, replacing the diffusion estimator with a simple Gaussian (as in SMiRL)? This would show whether the diffusion-ELBO estimator matters in practice.
-
Could the authors report results over more seeds with significance tests or confidence intervals, and state the hyperparameter tuning budgets for BRPO and each baseline?