acceptodds
Withdrawn from ICLR 2027

On Distributional Reinforcement Learning in Chaotic Dynamical Systems

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Chaotic dynamical systems pose a fundamental challenge for Reinforcement Learning (RL): exponential sensitivity to initial conditions induces high-variance bootstrap targets and poorly conditioned gradient updates. Chaotic dynamics arise across scientific and engineering domains, from fluid flows and climate systems to multi-agent systems, where reliable learning is highly desirable. Standard RL methods optimise expected returns through scalar value functions, implicitly averaging over diverging trajectories and entangling trajectory level instability with the learning objective. We show that under mild statistical stability assumptions, the return distribution evolves more regularly than individual trajectories when measured under the 1-Wasserstein metric, yielding a smoother distributional Bellman objective. By aligning optimisation with this measure level structure, distributional RL provides better conditioned learning. We offer a principled explanation for the advantages of distributional methods in chaotic systems and the geometries of RL objectives under chaos.

Primary area:
reinforcement learning
Keywords:
Distributional Reinforcement Learning, Stability, Chaotic Systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KBFgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper considers reinforcement learning (RL) for chaotic dynamical systems. The key claim is that while the expected return is, or at least can be, an unstable optimization objective in chaotic systems, the distribution is often much smoother. Thus, distributional RL may be better suited for chaotic systems. The authors provide a theoretical analysis of the smoothness of both objectives and evaluate RL and distributional RL in canonical chaotic environments.

Critical strengths

The problem seems relatively rarely considered within the RL community, but chaotic systems are an important class of systems that RL algorithms should be able to handle. The paper is well written and understandable, even for people who may not be very familiar with chaotic systems. The theoretical analysis is insightful, and the empirical evaluation confirms the key claims. Thus, this is an important contribution to ICLR.

Critical weaknesses
  1. The authors claim that strange attractors have statistically stable invariant probability measures, which motivates the smoothness assumption. It would be nice to support this with a reference. The distribution-level assumption is intuitively hard to verify. Therefore, it would be important to provide some evidence.

  2. The final results show that PPO outperforms the distributional RL approach. Therefore, one may conclude from the paper that PPO is sufficient and that distributional RL is unnecessary. This should be discussed further. Why is this a valuable contribution, despite the existence of PPO and its performance? The related work section is also quite thin, even though there are various applications of RL to chaotic systems in different disciplines, e.g., Bucci et al., "Control of chaotic systems by deep reinforcement learning," Proc. R. Soc. A, 2019; Weissenbacher et al., "Reinforcement learning of chaotic ssytems control in partially observable environments," Flow, Turbulence and Combustion, 2025; Cheng et al., "A deep reinforcement learning method to control chaos synchronization between two identical chaotic systems," Chaos, Solitons & Fractals, 2023. From that perspective, it is also helpful to motivate why the distributional approach offers advantages.

Questions for authors
  1. Can you motivate the proposed approach further in light of PPO's performance?

  2. Can you provide some further evidence for chaotic dynamical systems being typically relatively well-behaved at the distribution-level?

Reviewer_x6j1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the influence of chaotic dynamics in RL, and provides a systematic understanding on how distributional RL methods mitigate it.

Critical strengths

Overall it is a clear, rigorous mathematical paper focusing an interesting problem, supported by some empirical results. The idea that the distributional RL objective has better conditioning is especially interesting.

Critical weaknesses

This paper would be strengthened if more complex experiments such as MuJoCo Hopper and Walker2d that are standard RL benchmarks with chaotic environments. I would be happy to increase the rating if the authors can address it during the rebuttal.

Minor: Line 28: "oft" should be "often";

Questions for authors

Please see Weaknesses

Reviewer_2kP9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates whether distributional RL improves learning in chaotic dynamical systems. It proposes a theoretical explanation based on Wasserstein regularity of return distributions and compares QR-DQN, DQN, and PPO on four chaotic control benchmarks. Experiments across six seeds include return curves and loss/gradient diagnostics, showing benefits over DQN in some environments, while PPO generally learns faster.

Critical strengths
  1. The paper addresses an interesting question: how sensitivity to initial conditions affects RL optimization, and whether distributional representations help.
  2. The four benchmarks and optimization diagnostics provide useful empirical observations beyond final performance, including clear acknowledgment of mixed results and PPO’s advantages.
Critical weaknesses
  1. The central theoretical claims are not established. In Appendix C, Eq. (21) applies a fixed-action Lipschitz bound to returns with potentially different subsequent actions, omitting policy dependence. A counterexample to Theorem 4.6 is S′=sS'=s, R(s,a)=aR(s,a)=a, a∈{0,1}a\in\{0,1\}, and π(1∣s)=s\pi(1\mid s)=s. Here KR=0K_R=0, KP=1K_P=1, and γKP<1\gamma K_P<1, yet, fixing the initial action to zero, returns from states 0 and 1 differ by γ/(1−γ)\gamma/(1-\gamma), contradicting the claimed zero bound. Additionally, Theorem 4.5 establishes an exponentially growing upper bound, not necessarily exponential growth of the actual Lipschitz constant.

  2. The proposed optimization advantage is not demonstrated by the theory or isolated experimentally. Wasserstein Lipschitz continuity also implies Lipschitz continuity of the expected return and does not establish better conditioning with respect to network parameters. Comparisons of TD and quantile Huber losses and unnormalized gradient norms do not isolate distributional effects from loss shape, output dimensionality, or scaling. These gaps undermine the paper’s main explanatory contribution.

Questions for authors
  1. How can Theorem 4.6 be corrected to address the counterexample and account for subsequent policy-dependent actions?
  2. Which experimental environments satisfy γKP<1\gamma K_P<1, and what transition mechanism makes this condition hold?
  3. How are the plotted losses defined and normalized? Do the reported advantages persist with a matched scalar Huber baseline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.