- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper considers reinforcement learning (RL) for chaotic dynamical systems. The key claim is that while the expected return is, or at least can be, an unstable optimization objective in chaotic systems, the distribution is often much smoother. Thus, distributional RL may be better suited for chaotic systems. The authors provide a theoretical analysis of the smoothness of both objectives and evaluate RL and distributional RL in canonical chaotic environments.
Critical strengths
The problem seems relatively rarely considered within the RL community, but chaotic systems are an important class of systems that RL algorithms should be able to handle. The paper is well written and understandable, even for people who may not be very familiar with chaotic systems. The theoretical analysis is insightful, and the empirical evaluation confirms the key claims. Thus, this is an important contribution to ICLR.
Critical weaknesses
-
The authors claim that strange attractors have statistically stable invariant probability measures, which motivates the smoothness assumption. It would be nice to support this with a reference. The distribution-level assumption is intuitively hard to verify. Therefore, it would be important to provide some evidence.
-
The final results show that PPO outperforms the distributional RL approach. Therefore, one may conclude from the paper that PPO is sufficient and that distributional RL is unnecessary. This should be discussed further. Why is this a valuable contribution, despite the existence of PPO and its performance? The related work section is also quite thin, even though there are various applications of RL to chaotic systems in different disciplines, e.g., Bucci et al., "Control of chaotic systems by deep reinforcement learning," Proc. R. Soc. A, 2019; Weissenbacher et al., "Reinforcement learning of chaotic ssytems control in partially observable environments," Flow, Turbulence and Combustion, 2025; Cheng et al., "A deep reinforcement learning method to control chaos synchronization between two identical chaotic systems," Chaos, Solitons & Fractals, 2023. From that perspective, it is also helpful to motivate why the distributional approach offers advantages.
Questions for authors
-
Can you motivate the proposed approach further in light of PPO's performance?
-
Can you provide some further evidence for chaotic dynamical systems being typically relatively well-behaved at the distribution-level?