acceptodds
Under review as a conference paper at ICLR 2027

Rational Agents Resist Entropy: Bounded Rationality as the Principled Objective for RL under Finite Samples

Abstract

Reinforcement learning optimises expected return without explicitly constraining representation complexity, which can make learning statistically difficult under finite samples. We propose bounded-rational RL: augment the return with a penalty on the differential entropy of the agent's noisy representations to control statistical complexity. We justify this objective by proving, under explicit conditions, an entropy-dependent bound on the samples needed to evaluate nearby policy changes. Analysing the policy gradient of this objective provides a behavioral characterisation: the complexity penalty supplies an intrinsic reward for low-surprisal representations and can drive the agent to maintain recurring, predictable situations—entropy resistance. This behavioral characterisation distinguishes RL from passive supervised learning and connects bounded-rational RL to active inference and related frameworks that also produce entropy-resisting behavior from different theoretical foundations. Experiments across partially observable control tasks show both: bounded rationality improves mean returns over unconstrained learning, and agents resist entropy, with larger gains in the more demanding task variants studied.

Primary area:
reinforcement learning
Keywords:
bounded rationality, reinforcement learning, partial observability, representation learning, structural risk minimization, entropy resistance

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Z5DSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes training RL agents to maximize return minus a penalty on the entropy of their noisy history representations, pooled over the episode. The authors motivate this with a finite-sample argument: when the visited representations have low entropy, small policy updates can be evaluated from fewer episodes, and they give a construction where a high-entropy optimal policy needs exponentially many episodes while a simpler near-optimal one does not. They also derive the policy gradient of the objective, which splits into a surprise-like reward that shapes behavior and a direct penalty on the encoder. They argue the first term makes agents act to keep their situations predictable (“entropy resistance”). The resulting algorithm, BRPO, estimates the penalty with a diffusion model and plugs it into PPO. Experiments on several POMDP benchmarks (RockSample, NoisyTV, LaserTag, LightDark) report higher returns than recurrent PPO and planning baselines, with the gap growing with task size, along with a double-pendulum experiment on entropy resistance and ablations of the two penalty terms.

Critical strengths
  1. Statistical account of why predictability helps learning. Section 5 gives an entropy-dependent bound on the sample cost of evaluating local policy updates, together with a matching construction (Thm 5.2) showing that this exponential dependence on representation entropy can actually occur. The upper/lower-bound pair makes the argument more than a heuristic: low-entropy representations make greedy local improvement statistically easier.

  2. Two mechanisms, cleanly separated and tested. The policy-gradient result (Thm 6.1) splits the penalty into a reward-side term that shapes behavior and a direct encoder term that compresses representations. The λ1/λ2 ablations then show complementary roles. The reward-side term changes which situations the agent visits, while the encoder term helps retain a task-relevant cue amid distractors, a somewhat counterintuitive effect of penalizing entropy.

  3. Sound and transparent. The paper is mostly sound, and the proofs are correct as far as I checked. It is candid about the limits of its theory: Appendix B.3 and the Limitations section state which conditions are assumed and what remains open. Environments, hyperparameters, and evaluation protocols are documented in detail.

Critical weaknesses
  1. The paper presents its objective (return minus the entropy of history representations) as the main contribution, but this objective is not new. The closest work, Robust Predictable Control (Eysenbach et al., NeurIPS 2021), is not cited. RPC jointly trains encoder and policy to minimize representation information and observes the same behavioral effect: the policy learns to keep its representations predictable. SMiRL (Berseth et al., 2021) is cited, but its closeness is understated: it already minimizes state-marginal entropy, including on learned latents, combined with task reward. IC2 (Rhinehart et al., 2021) and information-bottleneck RL (e.g., Igl et al., 2019) are also closely related. Introducing the objective is therefore not a contribution. The paper could instead be positioned as explaining why such objectives are beneficial and proposing a method (BRPO, with its diffusion-ELBO estimator) for optimizing them. Similarly, BRPO is essentially PPO with a SMiRL-style surprise reward on the recurrent representations plus a direct encoder penalty, with the occupancy density estimated by a diffusion model instead of a simpler density model (e.g., the Gaussian used in SMiRL). The paper does not ablate this choice, so it is unclear whether the diffusion estimator matters.

  2. Regarding the theoretical contribution: Section 5 bounds the sample cost of evaluating local updates around a fixed policy, i.e., a possible benefit during learning. It gives no guarantee on the return of the policy that learning produces. The motivation as "following the principle of SRM" is therefore misleading: SRM's justification is precisely such a guarantee, bounding the selected predictor's true risk by empirical risk plus complexity.

  3. Regarding the empirical contribution, the claimed advantages of BRPO are under-supported. BRPO is not compared against the most relevant baselines: a SMiRL-style surprise reward on the recurrent representations, RPC, and an information-bottleneck encoder (e.g., VIB/IBAC). Clear gains appear on only two of four tasks (RockSample, NoisyTV); LaserTag and LightDark are within noise. All results use three seeds without significance tests, and the tuning budgets for BRPO versus the baselines (including planner simulation budgets) are not reported.

Questions for authors
  1. Given the closeness to RPC, SMiRL, and IC2, could the authors state precisely which contributions they consider novel relative to this prior work? In particular, what does the pooled occupancy entropy capture that RPC's information penalty or SMiRL's latent surprise does not?

  2. Could the authors compare BRPO against the most relevant baselines: PPO-RNN with a SMiRL-style surprise reward on the recurrent representations, RPC, and an information-bottleneck encoder (e.g., VIB/IBAC)?

  3. Could the authors ablate the density model, replacing the diffusion estimator with a simple Gaussian (as in SMiRL)? This would show whether the diffusion-ELBO estimator matters in practice.

  4. Could the authors report results over more seeds with significance tests or confidence intervals, and state the hyperparameter tuning budgets for BRPO and each baseline?

Reviewer_q1CCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work proposes a representation complexity regularization for RL that jointly encourages entropy resistance in actions and learning lower complexity representations. The method proposes to optimize a modified RL objective that penalizes the representation entropy across a trajectory, similar to the SMiRL [1] objective but with a jointly learned encoder. The policy gradient for this objective results in two terms, a reward penalty term that penalizes actions that result in states with high entropy representations, as in [1], and a penalty on the encoder complexity. The authors motivate this approach theoretically by showing that under restrictive assumptions the number of trajectories required to make good updates to the policy is related to the representation complexity, so that reducing representation complexity could improve sample efficiency. The practical instantiation of the algorithm requires a density estimate of the representation distribution, for which the authors propose a diffusion model and demonstrate that the denoising cost upper bounds the representation entropy. They test their proposed method in four environments, training with a small amount of experience, and demonstrate their method outperforms baselines. Further, the ablate the impact of the reward and encoder penalities and find both to be important.

[1] Berseth, Glen, et al. "Smirl: Surprise minimizing reinforcement learning in unstable environments." arXiv preprint arXiv:1912.05510 (2019).

Critical strengths

(1) The diffusion model for estimating the representation entropy is novel and well-motivated. Proposition 7.1 showing that the denoising cost upper bounds the representation entropy is a nice result that directly relates to the proposed method.

(2) Experimental results are good - The proposed method shows improved performance across a number of tasks with limited experience. The ablations of the two components, the reward penalty and encoder penalty are pertinent and demonstrate that both components are important in different settings. The authors test multiple alternative penalty terms (in a single environment) which demonstrates the value of their proposed representation entropy objective.

Critical weaknesses

(1) Significant clarity issues: My primary concern with this paper is the significant clarity issues. The central point of the paper is difficult to extract, theorems in the main text are stated very informally, rigorous results in the Appendix are poorly organized and results are stated with dense newly introduced notation, links between variables are not made explictly and left to the reader to decipher. Several theorems seem to add very little to the contribution and, most importantly, theorems are presented without reference to existing literature and known results, making it difficult to understand what is novel about the contribution. Here are a few (non-exhaustive) examples:

  • Eq. 1 is closely related to the SMiRL objective, which uses the entropy of the state distribution rather than the entropy of the representation distribution, though SMiRL also contains a variant SMiRL-VAE over representations. The key difference between the proposed method and SMiRL-VAE seems to be that in the proposed method the representation is shaped via the policy gradient. It would be helpful to the reader to make these relationships to prior work more explicit.
  • Prop 1 is true for any non-negative penalty function, it is not specific to this method. Stating it as a proposition adds noise for the reader.
  • Theorem 5.1 as stated doesn't make sense - first the gaurantee is stated and then the condition on the samples without a connection between them. It is also state very informally with assumptions relegated to the Appendix and finding it in the Appendix was itself challenging as its formal statement is not directly mentioned in the main text. Additionally, Theorem 5.1's bound seems closely related to Kearns et al [2], which does not depend on representation complexity - it appears to me in the proof of Theorem B.1 that the dependence on complexity enters entirely through the authors assumption at line 750, but the importance of this assumption, its validity and when it holds are not discussed.
  • Theorem 5.2 - It is not clear to me what the utility of this theorem as it concerns a representation with 0 complexity, i.e. one which maps all states to the same representation, which is practically irrelevant.
  • Theorem 6.1. This is stated as a theorem on entropy resistance, however the policy gradient doesnt provide any more intuition about entropy resistance than the original objective itself. Further, the entropy resistance mechanism is the central claim of prior work SMiRL [1] which is referenced seemingly to contrast to the proposed method despite sharing the same reward penalty.
  • Theorem 7.1 seems to build on the standard TRPO bounds [3] without any reference.

[1] Berseth, Glen, et al. "Smirl: Surprise minimizing reinforcement learning in unstable environments." arXiv preprint arXiv:1912.05510 (2019). [2]Kearns, Michael, Yishay Mansour, and Andrew Ng. "Approximate planning in large POMDPs via reusable trajectories." Advances in Neural Information Processing Systems 12 (1999). [3] Schulman, John, et al. "Trust region policy optimization." International conference on machine learning. Pmlr, 2015.

Questions for authors

(1) What is the purpose of Theorem 5.2?

(2) Can you justify the tail-bound assumption in Theorem 5.1 that introduces representation complexity into the bound?

(3) How does your method contrast with [4]?

(4) What is greedy return?

(5) Do the alternative penalty ablations include the reward penalty term or not? Since the reward penalty/encoder penalty are tested in different environments, it is not clear if the difference in performance is due to the absence of the reward penalty (SMiRL) or the encoder penalty

(6) Why are ablations only conducted in some environments?

[4] Igl, Maximilian, et al. "Generalization in reinforcement learning with selective noise injection and information bottleneck." Advances in neural information processing systems 32 (2019).

Reviewer_YYBYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper augments return with a penalty on the entropy of learned representations and argues that lower representation complexity can reduce the finite-sample difficulty of evaluating nearby policy updates. The main theoretical result is conditioned on already being near a policy with sufficiently simple representations. The method derived from the theory estimates representation entropy with an auxiliary diffusion model and incorporates this penalty into PPO, but the experiments are limited to very small problems. The main question is, for such problems, whether the proposed machinery is necessary: if the benefit comes from restricting effective policy complexity under limited data, simpler alternatives such as directly using smaller recurrent states, finite-state controllers, or shallow policies should all be considered as baselines.

Critical strengths
  • Interesting connection between representation entropy and finite-sample RL.
  • Provides a theoretical justification for preferring low-complexity representations.
  • BRPO shows consistent gains over PPO on several partially observed tasks.
Critical weaknesses
  • The theory is local and assumes a good low-complexity policy is already available or nearby.
  • The sample bound is exponential when representation complexity grows with horizon.
  • Experiments are too small to demonstrate promise, and at such scales a lot of much simpler methods become competitive but are not compared against.
Questions for authors
  • How does BRPO compare with directly restricting the controller class or latent dimension under the same interaction budget?
  • Can the method scale to substantially higher-dimensional or longer-horizon tasks where representation entropy itself is difficult to estimate?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.