- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the training dynamics of neural networks in high-dimensional observations’ setting in the context of reinforcement learning (RL). The paper studies Stiefel manifold learning (optimization) for deep linear networks and argues that, under some assumptions, by enforcing orthogonality of the weight matrices (constraining the parameters to the Stiefel manifold) the learning dynamics of high-dimensional optimization matches that of a low-dimensional latent state problem. This means that the RL agent’s learning in this setting can be described by a fewer number of variables and parameters. This results in acceleration and stability in the learning due to the introduction of the Stiefel manifold.
The paper validates their framework in vision-based Mujoco control tasks (control directly from pixels), as well as on large-scale LLM finetuning using RL. The paper also presents additional results on the recovery on the low dimensional latent representation of an inverted pendulum from high-dimensional, pixel-based, encoders when constrained to the Stiefel manifold.
Critical strengths
The strengths of the paper can be summarized as follows:
- The paper tackles an interesting and important problem of studying the learning dynamics of RL agents.
- It provides a good perspective on the mapping between high-dimensional (upstairs) and low dimensional (downstairs) learning dynamics in the setting of deep linear networks and Stiefel manifold in RL context.
- The linear probes experiment on the inverted pendulum is a good proof of concept.
- The paper mentions the computational cost of the retraction part to the Stiefel manifold and highlights that, although it’s not critical ~30% decrease in speed, this is an active area of research.
Critical weaknesses
The weaknesses of the paper can be summarized as follows:
- In high-dimensional setting (upstairs), the Stiefel manifold is enforced at the initialization level and then using the retraction mechanism. I don’t see why the low-dimensional counterpart (downstairs) it also follows the Stiefel manifold and especially at the initialization of the network? How and why does the matrix for example follow the manifold both at the initialization and at subsequent updates level? Is there a proof or a mathematical justification for this?
- It seems that the “implicit acceleration” happens due to increasing the number of layers of the neural network? For example, in page 6 line 289, the policy acceleration happens due to multiplying the policy gradient by a factor of (L – 1) where L is the number of layers in the policy? Doesn’t that mean that the network just takes a large gradient step at the current weights? If so, how is this different from increasing the learning rate? Furthermore, wouldn’t that potentially make the learning unstable? the opposite of the stability benefit the paper is trying to introduce.
- The paper claims that the manifold constraint helps the learning process become more stable. It is unclear how/where in the paper was this theoratically established or empirically validated?
Minor:
- The presentation could be made clearer. Specifically, the paper could benefit from a clearer contribution description. For example, Stiefel manifold has been studied before, but as far as I know, not in the RL setting the paper is tackling. A clarification on the exact contributions of the paper would make the reader focus more on the novelty of the proposed approach.
- From visual inspection different seeds in evaluation environments have larger standard deviation of returns when compared to vanilla RL for example. This is especially visible in the LLM setting as well as some Mujoco such as Ant Goal. Could the authors provide numbers as well as justify why this might be the case?
Questions for authors
- Regarding the implicit acceleration, can multiplying the neural network by a smaller learning rate to counter the large multiplicative factor due to the number of layers (e.g.: the L-1 term in the policy gradient formula)
2 comments
Response to Reviewer Continued
Minor 1. [...] the paper could benefit from a clearer contribution description.
We have added a contributions paragraph at the end of the introduction. Optimization on the Stiefel manifold is well studied; our contributions are (i) a theoretically tractable model of RL with high-dimensional observations, namely continuous-time LQR with linear observations and deep linear actor-critic networks, (ii) the theorem that, under Stiefel manifold optimization, the learning dynamics of the parameters converge to an ODE in variables, and (iii) the application of manifold Muon to PPO from pixels and to distillation fine-tuning of an LLM.
We hope this addresses your concerns.
Response to Reviewer
Thank you for your detailed review of our work. We address the issues below.
It does not in general, and we have revised the paper on this point. A summary matrix such as U=M⊤UeffM is a ds×ds matrix obtained from the orthogonal d×d matrix Ueff. Therefore ∥U∥op≤1, at initialization and after every gradient step, but U is not orthogonal in general. What holds is the following.
We have revised Theorem 1, the text after it and the caption of Figure 2 to state this.
For the policy summary matrix alone, the factor L−1 has the same effect as the learning rate (L−1)η, and we now say this in the text. It differs from increasing the learning rate in two ways.
On stability: the theorem is for fixed L and η=O(1/d)→0, so (L−1)η also goes to zero. For finite η a larger L does have the same effect as a larger learning rate for Wˉ, and we do not claim that depth improves stability. What the Stiefel constraint guarantees is that, for every η and L, the trainable matrices remain on the Stiefel manifold, so their products do not expand Euclidean norms and ∥Wˉ∥op≤1.
We agree that the paper claimed more than it shows, and we have revised the claim. What is established is the following.
We have replaced "stability" in the Discussion by the statement that the parameter matrices and their summary matrices remain bounded, and the sentence on low-variance returns in Section 5.3 by the comparison with vanilla Muon.