acceptodds
Under review as a conference paper at ICLR 2027

Stiefel Manifold Optimization Accelerates Reinforcement Learning

Abstract

Reinforcement learning with high-dimensional observations and overparameterized neural networks is common in practice, yet its learning dynamics remain poorly understood. To develop a theoretically grounded approach to improving learning in this setting, we study learning a feedback controller for the linear-quadratic regulator problem with high-dimensional observations generated from low-dimensional latent states. We use deep linear networks to model overparameterized policies and value functions, with weight matrices constrained to have orthonormal columns, i.e., to lie on the Stiefel manifold. Within this framework, we show that the learning dynamics of the high-dimensional optimization problem match those of the underlying low-dimensional latent-state problem, allowing actor-critic learning to exploit the hidden low-dimensional structure. Motivated by this theory, we improve the performance of proximal policy optimization by incorporating Stiefel-manifold optimization in MuJoCo environments with pixel observations, trained end-to-end from reward and without auxiliary representation-learning losses. We further show that the same optimization principle improves large language model fine-tuning using distillation for the Qwen3-4B model.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, control, optimization, deep linear networks, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jrrZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the training dynamics of neural networks in high-dimensional observations’ setting in the context of reinforcement learning (RL). The paper studies Stiefel manifold learning (optimization) for deep linear networks and argues that, under some assumptions, by enforcing orthogonality of the weight matrices (constraining the parameters to the Stiefel manifold) the learning dynamics of high-dimensional optimization matches that of a low-dimensional latent state problem. This means that the RL agent’s learning in this setting can be described by a fewer number of variables and parameters. This results in acceleration and stability in the learning due to the introduction of the Stiefel manifold.

The paper validates their framework in vision-based Mujoco control tasks (control directly from pixels), as well as on large-scale LLM finetuning using RL. The paper also presents additional results on the recovery on the low dimensional latent representation of an inverted pendulum from high-dimensional, pixel-based, encoders when constrained to the Stiefel manifold.

Critical strengths

The strengths of the paper can be summarized as follows:

  1. The paper tackles an interesting and important problem of studying the learning dynamics of RL agents.
  2. It provides a good perspective on the mapping between high-dimensional (upstairs) and low dimensional (downstairs) learning dynamics in the setting of deep linear networks and Stiefel manifold in RL context.
  3. The linear probes experiment on the inverted pendulum is a good proof of concept.
  4. The paper mentions the computational cost of the retraction part to the Stiefel manifold and highlights that, although it’s not critical ~30% decrease in speed, this is an active area of research.
Critical weaknesses

The weaknesses of the paper can be summarized as follows:

  1. In high-dimensional setting (upstairs), the Stiefel manifold is enforced at the initialization level and then using the retraction mechanism. I don’t see why the low-dimensional counterpart (downstairs) it also follows the Stiefel manifold and especially at the initialization of the network? How and why does the U^\hat{U} matrix for example follow the manifold both at the initialization and at subsequent updates level? Is there a proof or a mathematical justification for this?
  2. It seems that the “implicit acceleration” happens due to increasing the number of layers of the neural network? For example, in page 6 line 289, the policy acceleration happens due to multiplying the policy gradient by a factor of (L – 1) where L is the number of layers in the policy? Doesn’t that mean that the network just takes a large gradient step at the current weights? If so, how is this different from increasing the learning rate? Furthermore, wouldn’t that potentially make the learning unstable? the opposite of the stability benefit the paper is trying to introduce.
  3. The paper claims that the manifold constraint helps the learning process become more stable. It is unclear how/where in the paper was this theoratically established or empirically validated?

Minor:

  1. The presentation could be made clearer. Specifically, the paper could benefit from a clearer contribution description. For example, Stiefel manifold has been studied before, but as far as I know, not in the RL setting the paper is tackling. A clarification on the exact contributions of the paper would make the reader focus more on the novelty of the proposed approach.
  2. From visual inspection different seeds in evaluation environments have larger standard deviation of returns when compared to vanilla RL for example. This is especially visible in the LLM setting as well as some Mujoco such as Ant Goal. Could the authors provide numbers as well as justify why this might be the case?
Questions for authors
  1. Regarding the implicit acceleration, can multiplying the neural network by a smaller learning rate to counter the large multiplicative factor due to the number of layers (e.g.: the L-1 term in the policy gradient formula)
2 comments
AuthorsOpenReview ↗

Response to Reviewer

Thank you for your detailed review of our work. We address the issues below.

W1. How and why does the matrix [...] follow the manifold both at the initialization and at subsequent updates level? Is there a proof or a mathematical justification for this?

It does not in general, and we have revised the paper on this point. A summary matrix such as U^=M⊤UeffM\widehat U = M^\top U_{\mathrm{eff}} M is a ds×dsd_s \times d_s matrix obtained from the orthogonal d×dd \times d matrix UeffU_{\mathrm{eff}}. Therefore ∥U^∥op≤1\Vert \widehat U \Vert_{\mathrm{op}} \le 1, at initialization and after every gradient step, but U^\widehat U is not orthogonal in general. What holds is the following.

  • The summary matrices are in the set of matrices with operator norm at most one, and they remain in it. The revised Theorem 1 states this.
  • The theorem does not need the summary matrices to be on the Stiefel manifold. Its proof uses only that the upstairs matrices are on the Stiefel manifold and Assumption 2, M⊤M=IM^\top M = I (Lemmas 4 to 8).
  • If a summary matrix XX is initialized on the Stiefel manifold, it remains on it: for X⊤X=IX^\top X = I the update direction D=G−XG⊤XD = G - XG^\top X, where GG is the Euclidean gradient direction, satisfies X⊤D+D⊤X=0X^\top D + D^\top X = 0, meaning it is tangent to the Stiefel manifold. This is the case in Figure 3, where the summary matrices of the downstairs agent are initialized on the Stiefel manifold.

We have revised Theorem 1, the text after it and the caption of Figure 2 to state this.

W2. Doesn't that mean that the network just takes a large gradient step at the current weights? If so, how is this different from increasing the learning rate? Furthermore, wouldn't that potentially make the learning unstable?

For the policy summary matrix alone, the factor L−1L-1 has the same effect as the learning rate (L−1)η(L-1)\eta, and we now say this in the text. It differs from increasing the learning rate in two ways.

  • No trainable policy matrix takes a larger step: each of the L−1L-1 trainable policy matrices is updated with the learning rate η\eta. The Stiefel constraints ensure that the first-order changes induced by these matrices contribute the same downstairs update direction, and these contributions add, multiplying the rate of change of the policy summary matrix Wˉ\bar W by L−1L-1 (Lemma 8). Without the constraint, each contribution is multiplied by powers of the end-to-end matrix (Arora et al., 2018), so the factor depends on the current weights and is not a constant.
  • The factors differ between the summary matrices: LL for U^\widehat U, L−1L-1 for u^\hat u, Z^\widehat Z and Wˉ\bar W, and L−2L-2 for z^\hat z. Meaning that depth changes the relative rates of the critic and the actor, which a single learning rate does not.

On stability: the theorem is for fixed LL and η=O(1/d)→0\eta = O(1/\sqrt d) \to 0, so (L−1)η(L-1)\eta also goes to zero. For finite η\eta a larger LL does have the same effect as a larger learning rate for Wˉ\bar W, and we do not claim that depth improves stability. What the Stiefel constraint guarantees is that, for every η\eta and LL, the trainable matrices remain on the Stiefel manifold, so their products do not expand Euclidean norms and ∥Wˉ∥op≤1\Vert \bar W \Vert_{\mathrm{op}} \le 1.

W3. The paper claims that the manifold constraint helps the learning process become more stable. It is unclear how/where in the paper was this [...] established or empirically validated?

We agree that the paper claimed more than it shows, and we have revised the claim. What is established is the following.

  • Theory: the retraction keeps every trainable matrix on the Stiefel manifold throughout the training process, so the matrices, their products and the summary matrices have operator norm at most one, and the moments of the update directions are bounded uniformly in dd (Lemma 13). These are bounds on the matrices and on the update directions; we do not prove convergence or a bound on the variance of the returns.
  • Experiments: vanilla Muon orthonormalizes the gradient update but does not restrict the weights to lie on the Stiefel manifold. With identical global hyperparameters it prevents learning in almost all environments. We could tune a learning rate for individual environments, but we did not find a learning rate that performs well across all environments, whereas manifold Muon uses the same hyperparameters in all environments (Appendices I and J).

We have replaced "stability" in the Discussion by the statement that the parameter matrices and their summary matrices remain bounded, and the sentence on low-variance returns in Section 5.3 by the comparison with vanilla Muon.

AuthorsOpenReview ↗

Response to Reviewer Continued

Minor 1. [...] the paper could benefit from a clearer contribution description.

We have added a contributions paragraph at the end of the introduction. Optimization on the Stiefel manifold is well studied; our contributions are (i) a theoretically tractable model of RL with high-dimensional observations, namely continuous-time LQR with linear observations and deep linear actor-critic networks, (ii) the theorem that, under Stiefel manifold optimization, the learning dynamics of the O(Ld2)O(Ld^2) parameters converge to an ODE in O(ds2+dsda)O(d_s^2 + d_s d_a) variables, and (iii) the application of manifold Muon to PPO from pixels and to distillation fine-tuning of an LLM.

We hope this addresses your concerns.

Reviewer_8fKxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies continuous-time LQR with high-dimensional observations and deep linear actor–critic networks constrained to the Stiefel manifold. Its main theoretical claim is that compressed parameter summaries follow closed, low-dimensional limiting dynamics, with depth-dependent factors interpreted as implicit acceleration. Motivated by this analysis, the authors combine manifold Muon with ISTA-style representations for PPO from pixels and introduce a factorized manifold optimizer for pretrained language-model fine-tuning. Validation includes synthetic LQR, inverted-pendulum representation studies, several pixel-control tasks, and Qwen3-4B distillation on three tasks. The reported experiments show improved sample efficiency in several configurations, alongside increased computational cost per step.

Critical strengths
  1. Practical evidence. Figure 6 compares both optimizers within both CNN and ISTA architectures, showing useful gains from manifold Muon, particularly with ISTA on Humanoid and goal-reaching tasks. The main pixel-control experiments learn from reward without auxiliary state supervision, making these improvements practically relevant.

  2. Interesting perspective. Describing high-dimensional actor–critic learning through a small collection of latent-state summaries is a worthwhile approach. The orthogonal-product structure provides a plausible foundation for a corrected theory connecting parameterization, optimization, and latent structure.

Critical weaknesses
  1. The central theorem has substantive gaps. Equation (42) contains ϵ2ξ⊤Ueffξ\epsilon^2\xi^\top U_{\mathrm{eff}}\xi. Even under an ordinary Gaussian interpretation of the noise, the permitted choice Ueff=IdU_{\mathrm{eff}}=I_d and ϵ=d−1/2\epsilon=d^{-1/2} gives expectation one, rather than a vanishing remainder. The boundedness assumption introduced in Lemma 9's proof is not established uniformly in dimension. Independently, the discounted stability assumption does not justify Lemma 12's gradient-moment bounds, as critic gradients involve quartic state terms. Furthermore, M⊤UeffMM^\top U_{\mathrm{eff}}M need not be a Stiefel matrix, so the claimed downstairs-manifold interpretation requires revision.

  2. The empirical evidence does not establish the proposed geometric mechanism. Table 7 reports substantial off-diagonal cosines and omits one state coordinate, while Equation (52) describes a state-prediction readout. The ISTA architecture also changes projection capacity and policy-distribution constraints. For LLMs, factorization changes the parameterization alongside the optimizer, without a matched-factorization baseline or sufficient rank specifications.

Questions for authors
  1. What dimension-uniform condition controls ϵ2tr⁡(Ueff)\epsilon^2\operatorname{tr}(U_{\mathrm{eff}}) in Lemma 9, and how does the theorem handle Ueff=IdU_{\mathrm{eff}}=I_d with ϵ=d−1/2\epsilon=d^{-1/2}? Is centering or a redefined critic intercept intended?

  2. What stronger integrability assumption justifies Lemma 12's infinite-horizon gradient moments? If finite rollout horizons are essential, how does the theorem account for truncation?

  3. What is the correct domain of the compressed summaries, given that compression does not preserve orthonormality? What exact downstairs update, including depth scaling, is used in Figure 3?

  4. Which encoder output is passed to PPO in Figure 4? Please reconcile Section 5.2 with Equation (52), and explain how Table 7 supports the required geometry for all four state coordinates.

  5. Do the gains persist with matched projection capacity and policy-distribution constraints? For LLMs, what ranks and layers are factorized, and does the advantage persist against a baseline using the same factorization and initialization?

3 comments
AuthorsOpenReview ↗

Response to Reviewer

Thank you for your deep engagement with the work. We address your concerns below.

The central theorem has substantive gaps.

We agree with the four points below and have revised the statement and the proof. The form of the result is unchanged: as d→∞d \to \infty, the parameters Θτ\Theta_\tau are governed by the ODE Θ˙τ=F(Θτ)\dot\Theta_\tau = F(\Theta_\tau). The changes are: the intercept c~\tilde c replaces cc in Θ\Theta, Assumption 1 has the bound β/4\beta/4 in place of β/2\beta/2, Assumption 3 on the initialization of the value estimate is new, and the summary matrices are in the set of matrices with operator norm at most one. All changes are marked in blue.

Equation (42) contains ϵ2ξ⊤Ueffξ\epsilon^2 \xi^\top U_{\mathrm{eff}} \xi. Even under an ordinary Gaussian interpretation of the noise, the permitted choice Ueff=IdU_{\mathrm{eff}} = I_d and ϵ=d−1/2\epsilon = d^{-1/2} gives expectation one, rather than a vanishing remainder.

This is correct. E[ϵ2ξ⊤Ueffξ]=ϵ2tr⁡(Ueff)\mathbb{E}[\epsilon^2 \xi^\top U_{\mathrm{eff}} \xi] = \epsilon^2 \operatorname{tr}(U_{\mathrm{eff}}), which is O(1)O(1) and not O(ϵ)O(\epsilon). We now separate the mean of this term from its fluctuation. We move the mean into the intercept, c~≜c+ϵ2tr⁡(Ueff)\tilde c \triangleq c + \epsilon^2 \operatorname{tr}(U_{\mathrm{eff}}), and c~\tilde c replaces cc in Θ\Theta (Lemma 9, Theorem 1). The fluctuation ξ⊤Ueffξ−tr⁡(Ueff)\xi^\top U_{\mathrm{eff}} \xi - \operatorname{tr}(U_{\mathrm{eff}}) has variance at most 2d2d, so ϵ2\epsilon^2 times it is O(ϵ)O(\epsilon) for ϵ=O(1/d)\epsilon = O(1/\sqrt d). Further, the new Lemma 10 shows Δc~=Δc+O(ηϵ2)\Delta \tilde c = \Delta c + O(\eta\epsilon^2), meaning that c~\tilde c follows the same one-step dynamic as cc. By Assumption 3, ϵ2tr⁡(Ueff)→b0\epsilon^2 \operatorname{tr}(U_{\mathrm{eff}}) \to b_0 in probability at initialization, and therefore c~0=c0+b0\tilde c_0 = c_0 + b_0. In the reviewer's example b0=1b_0 = 1, and the remainder ϵ2(∥ξ∥2−d)\epsilon^2(\|\xi\|^2 - d) has standard deviation 2/d\sqrt{2/d}.

The boundedness assumption introduced in Lemma 9's proof is not established uniformly in dimension.

We no longer assume that the noise projections are bounded; we bound each one of them uniformly in dd. The observation noise ξt∼N(0,Id)\xi_t \sim \mathcal N(0, I_d) enters the TD-error through four projections, which are Gaussian in the fixed dimensions ds,1,da,dad_s, 1, d_a, d_a, and through the fluctuation above. The covariances of the projections are ⪯4Ids\preceq 4I_{d_s}, 11, IdaI_{d_a}, IdaI_{d_a}, which do not depend on dd. For the fluctuation XX, the hypercontractivity inequality gives (E∣X∣p)1/p≤(p−1)(EX2)1/2≤(p−1)2d(\mathbb{E}|X|^p)^{1/p} \le (p-1)(\mathbb{E}X^2)^{1/2} \le (p-1)\sqrt{2d} (Nourdin and Peccati, 2009, Eq. 2.12), so ϵ\epsilon times it is bounded for ϵ=O(1/d)\epsilon = O(1/\sqrt d). The state is governed by an SDE in the fixed dimension dsd_s, which depends on dd only through ϵ\epsilon in its covariance. Therefore, Lemma 9 now states that the upstairs TD-error is δt=δˉt+ϵRt\delta_t = \bar\delta_t + \epsilon R_t, where δˉt\bar\delta_t is the downstairs TD-error with the intercept c~\tilde c, and every moment of RtR_t is bounded uniformly in dd and in the trainable matrices. Further, the terms O(η2)O(\eta^2) and O(ηϵ)O(\eta\epsilon) of the one-step dynamic, denoted by qiηq_i^\eta, satisfy (E∥qiη∥4)1/4≤CK(η2+ηϵ)(\mathbb{E}\|q_i^\eta\|^4)^{1/4} \le C_K(\eta^2 + \eta\epsilon) with CKC_K independent of dd.

Independently, the discounted stability assumption does not justify Lemma 12's gradient-moment bounds, as critic gradients involve quartic state terms.

This is correct. The bound β/2\beta/2 is sufficient for the objective, which is of degree two in the state, and not for the critic update, which is of degree four. Assumption 1 is now strict discounted stability with the bound β/4\beta/4: there exists γ>0\gamma > 0 such that α(Acl(Wˉ))≤β/4−γ\alpha(A_{\mathrm{cl}}(\bar W)) \le \beta/4 - \gamma for every ∥Wˉ∥op≤1\Vert \bar W \Vert_{\mathrm{op}} \le 1. Under it, Lemma 11 gives (E[sup⁡u∈[t,t+Δ]∥su∥p])1/p≤Cpeμt(\mathbb{E}[\sup_{u \in [t,t+\Delta]} \Vert s_u \Vert^p])^{1/p} \le C_p e^{\mu t} with μ<β/4\mu < \beta/4 and CpC_p independent of Wˉ\bar W and dd. In Lemma 13 (Lemma 12 in the submission) the integrand is of degree four in the state and the action, and Minkowski's integral inequality bounds the fourth moment of the update direction by (CK∫0∞e−(β−4μ)tdt)4=(CK/(β−4μ))4(C_K \int_0^\infty e^{-(\beta - 4\mu)t} \mathrm{d}t)^4 = (C_K/(\beta - 4\mu))^4. The assumption holds, for example, when λmax⁡(12(G+G⊤))+∥H∥op<β/4\lambda_{\max}(\tfrac12(G + G^\top)) + \Vert H \Vert_{\mathrm{op}} < \beta/4.

AuthorsOpenReview ↗

Response to Reviewer Continued

The empirical evidence does not establish the proposed geometric mechanism.

Thank you for pointing out these issues. We respond to each of them below.

Table 7 reports substantial off-diagonal cosines and omits one state coordinate

The table reports three of the four state coordinates because two of them are correlated on-policy. The cart velocity and the pole angular velocity are correlated at ≈−0.99\approx -0.99, so the rows of the ridge-regression matrix WW associated with them are not separately identifiable, and we report one of the two. An effective state dimension of three is also consistent with Tiwari et al. (2025), whose Theorem 1 bounds the dimension of the attained states by 2da+12d_a + 1, which is 33 here.

The off-diagonal entries measure the approximation error with respect to Assumption 2, and we do not expect them to vanish. Assumption 2 holds exactly in the LQR setting. In the environments from pixels the map from the state to the features is learned from the RL objective and is non-linear, and neither the Stiefel constraint on the actor and critic heads nor the ISTA layer enforces M⊤M=IM^\top M = I on it. The learned directions are therefore separated but not orthogonal (angles of 58∘58^\circ, 89∘89^\circ and 111∘111^\circ), and we have revised the text to say "approximately orthogonal".

The ISTA architecture also changes projection capacity and policy-distribution constraints.

This is correct, and we now list the differences in Appendix I.3. The ISTA-CNN has two convolutional layers instead of three, so its 512-dimensional projection acts on a 9734497344-dimensional input instead of 31363136, and its policy has a tanh-bounded mean and log⁡σ∈[−5,−1.4]\log\sigma \in [-5, -1.4].

These differences do not affect the comparison between optimizers. Adam and manifold Muon are compared on the same network: CNN-Adam against CNN-manifold Muon, and Adam against manifold Muon with the ISTA-CNN (Figure 6 and Appendix J). Within each pair the projection and the policy distribution are identical, and only the optimizer of the actor and critic parameter matrices changes. Further, vanilla Muon on the same heads, which orthonormalizes the gradient update but does not restrict the weights to lie on the Stiefel manifold, prevents learning in almost all environments with identical global hyperparameters (Appendix J). Therefore, the improvement within an architecture is due to the Stiefel constraint.

We agree that the comparison between the CNN and the ISTA-CNN does not isolate the ISTA layer. We have revised the text to attribute that improvement to the ISTA-CNN architecture, and not to the ISTA layer alone.

For LLMs, factorization changes the parameterization alongside the optimizer, without a matched-factorization baseline or sufficient rank specifications.

The factorization and the optimizer are not two independent changes. The manifold Muon optimizer only works if the matrices are already on the Stiefel manifold, which is not true for pretrained weights. The fixed-rank factorization W≈X′Γ′Y′⊤W \approx X'\Gamma'Y'^\top, with X′∈St(n,r)X' \in \mathrm{St}(n,r) and Y′∈St(k,r)Y' \in \mathrm{St}(k,r), is what gives matrices on the Stiefel manifold for a pretrained model, so it is part of factored manifold Muon.

The baseline that keeps the Muon update and removes the Stiefel manifold is vanilla Muon (Jordan et al., 2024), which orthonormalizes the gradient update of the pretrained weights WW but does not restrict them to lie on the Stiefel manifold. We report it in Appendix L: for DFT on Tool Use, none of the five learning-rate combinations yields any improvement on the loss, while factored manifold Muon outperforms AdamW on the same task. Therefore, the improvement does not come from the Muon update itself, and it appears only when X′X' and Y′Y' are constrained to the Stiefel manifold.

We now state in Appendix L the rank r=256r = 256 and that the factorization is applied to all the layers, and we have moved the vanilla Muon comparison into the main text.

AuthorsOpenReview ↗

Response to Reviewer Continued

Which encoder output is passed to PPO in Figure 4? Please reconcile Section 5.2 with Equation (52), and explain how Table 7 supports the required geometry for all four state coordinates.

Figure 4. The actor and critic heads receive the output ϕ(I)=Mh(I)∈R4\phi(I) = M h(I) \in \mathbb{R}^4 of the frozen encoder in Equation (52). The blue curve is PPO using the low-dimensional states.

Section 5.2 and Equation (52). Section 5.2 was imprecise, and it used MM for a different matrix than Equation (52). In Equation (52), M∈R4×256M \in \mathbb{R}^{4 \times 256} has orthonormal rows, MM⊤=I4MM^\top = I_4, and ϕ\phi is regressed onto the standardised raw state zz, so ϕ(I)≈z\phi(I) \approx z. The matrix of Assumption 2 is its transpose: M⊤∈R256×4M^\top \in \mathbb{R}^{256 \times 4} has orthonormal columns, and Mh(I)≈ zM h(I) \approx\ z means h(I)≈M⊤z+h⊥h(I) \approx M^\top z + h_\perp with Mh⊥=0M h_\perp = 0. Meaning that in the output h(I)h(I) of the trunk, of width 256256, each state coordinate lies along a unique direction and these directions are orthonormal, as in Assumption 2 with d=256d = 256, and multiplying by MM removes the remaining component h⊥h_\perp. We have rewritten Section 5.2 to state this. Figure 4 therefore shows that such an encoder can be learned from pixels, and that the learning dynamics of PPO on its output match those of PPO using the low-dimensional states for the same learning rate. The match between the upstairs and downstairs learning dynamics of Theorem 1 is validated in Figure 3.

Table 7. Table 7 does not concern the encoder of Figure 4. It measures the ISTA-CNN of Section 5.3, which is trained end-to-end from reward without the underlying low-dimensional states, and we have made this explicit. For the frozen encoder of Figure 4 the orthonormality holds for all four state coordinates by construction: the four directions are the rows of MM, and MM⊤=I4MM^\top = I_4 is maintained by a QR retraction applied after every optimizer step (∥MM⊤−I∥≈3×10−7\|MM^\top - I\| \approx 3 \times 10^{-7}). For the ISTA-CNN, Table 7 shows approximate orthogonality for three coordinates, and the fourth is not separately identifiable because the two velocity coordinates are correlated at ≈−0.99\approx -0.99 on-policy (see our response on Table 7 above).

What is the correct domain of the compressed summaries, given that compression does not preserve orthonormality? What exact downstairs update, including depth scaling, is used in Figure 3?

Domain. The summary matrices are products of matrices with operator norm one, e.g. U^=M⊤UeffM\widehat U = M^\top U_{\mathrm{eff}} M, so their domain is the set X\mathcal X of parameters whose matrices have operator norm at most one. The ODE of Theorem 1 is defined on X\mathcal X, and its solutions remain in X\mathcal X. The Stiefel manifold is the special case: if a summary matrix is initialized on the Stiefel manifold, it remains on it. We have revised Theorem 1, the text after it and Figure 2 accordingly (see also our response on M⊤UeffMM^\top U_{\mathrm{eff}} M above).

Figure 3. The downstairs agent is in this special case. Its summary matrices are initialized on the Stiefel manifold, and each summary matrix XX is updated by the Cayley retraction of Equation (1) with its downstairs update direction DXD_X: X+=(I+kXη2ΩX)−1(I−kXη2ΩX)X,ΩX=DXX⊤−XDX⊤.X^+ = \bigl(I + \tfrac{k_X \eta}{2}\Omega_X\bigr)^{-1}\bigl(I - \tfrac{k_X \eta}{2}\Omega_X\bigr)X, \qquad \Omega_X = D_X X^\top - X D_X^\top . The Cayley retraction keeps the summary matrices on the Stiefel manifold throughout training. There X+=X−kXη (DX−XDX⊤X)+O(η2)X^+ = X - k_X\eta\,(D_X - X D_X^\top X) + O(\eta^2), which is the matrix update operator of Theorem 1 with the factor kXk_X. For the summary matrices with more columns than rows (u^\hat u, z^\hat z, Wˉ\bar W) the retraction is applied to X⊤X^\top. The policy summary matrix is updated by ascent, with −DX-D_X, and the intercept is updated by descent with learning rate η\eta, without retraction.

Depth scaling. The factors kXk_X are those of Theorem 1: LL for U^\widehat U, L−1L-1 for u^\hat u, Z^\widehat Z and Wˉ\bar W, and L−2L-2 for z^\hat z. With L=3L = 3 in Figure 3 the learning rates kXηk_X\eta are 3η3\eta, 2η2\eta, 2η2\eta, 2η2\eta and η\eta. We have added this to Section 5.1.

We hope these changes and fixes address your concerns.

Reviewer_ybYqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies Stiefel manifold optimization in the context of reinforcement learning (RL), where the available observation is high-dimensional while the underlying state is low-dimensional. Through the LQR analysis, the paper argues that, with an orthonormal and linear observation map assumption and Stiefel constrained deep linear networks, the limiting upstairs (observation space) learning dynamics match the downstairs (state space) learning dynamics (Theorem 1). The authors report several empirical results including the pixel control environments, where they show the performance gains of ISTA manifold Muon. They also provide empirical results in Qwen3 distillation fine-tuning tasks with the proposed factored manifold Muon method.

Critical strengths

(S1) I find the idea of using low-dimensional control to study RL from high-dimensional observations interesting and appealing. This study could provide insight into how the observation map and network parameterization affect the learning dynamics, which may lead to further research in this direction.

(S2) The paper proposes factored manifold Muon for fine-tuning pretrained weights without requiring the full weight matrices to be orthogonal. I find this an interesting side contribution. With this method, the authors show improvements over AdamW on three Qwen3 distillation fine-tuning tasks.

Critical weaknesses

(W1) After reading the paper, I feel like I am missing the main connection between the role of the Stiefel constraints and their practical benefit in the context of RL. Preserving norms simplifies the update equations, but I do not see why this should help an agent learn control in the lower-level structure. Since the analysis starts with that structure already present (o=Mso = Ms), it leaves much of the difficulty motivating the pixel-control experiments outside the theoretical argument. Without clarification of the main arguments and how they are supported, I lean toward recommending weak rejection.

Questions for authors

(Q1) Section 5.2 connects encoder features to the linear observation map. How realistic is it to expect learned features to approximate MsMs, with orthonormality, in tasks beyond inverted pendulum?

(Q2) Have you assessed the loss of expressivity caused by the Stiefel constraints [1]?

(Q3) What is the comparison in Figure 3 intended to establish? If I understand correctly, one agent uses observation (MsMs) as input with deep linear network (i.e., learning action a=WL..W1Msa = W_L .. W_1 M s), while the other agent uses state (ss) as input with a single linear layer (i.e., learning action a=Wsa = W s). I do not see how this experiment conveys a nontrivial message.

(Q4) What explains the higher returns of manifold Muon combined with ISTA-CNN over Adam on Mujoco tasks in Figure 6? Are these gains due to learning features approximately of the form MsMs, with M⊤M=IdsM^\top M=I_{d_s} as assumed in Assumption (2), or due to the better optimizer?

[1] Lei Huang, Li Liu, Fan Zhu, Diwen Wan, Zehuan Yuan, Bo Li, and Ling Shao. Controllable Orthogonalization in Training DNNs. CVPR, 2020

2 comments
AuthorsOpenReview ↗

Response to Review

Thank you for your careful reading of our work. We respond to all the issues raised by you below:

W1 Since the analysis starts with that structure already present

Our analysis assumes that the linear mapping MM from the low-dimensional state to the high-dimensional observation has orthonormal columns (Assumption 2). Under this assumption, Theorem 1 shows that with Stiefel manifold optimization the learning dynamics in the upstairs formulation match those in the downstairs formulation, which need not be true for vanilla stochastic gradient descent (Section 4). We connect this to control from pixels through optimizer-architecture co-design. The assumption o=Mso = Ms, where MM has orthonormal columns, suggests that when the mapping from state to observation is complex and non-linear, an agent with an orthonormal encoder can use it in conjunction with the optimizer to obtain the upstairs/downstairs match. We show this for the inverted pendulum from pixels in Section 5.2, and we use the ISTA layer to learn such a representation in Section 5.3. As with deep linear networks in supervised learning, we study a theoretically tractable model. Its strict assumptions are the main limitation of our analysis, and our empirical results show that the motivated optimizer-architecture co-design is useful in practice even when these assumptions do not hold (Section 6).

Q1 How realistic is it to expect learned features to approximate MsMs, with orthonormality, in tasks beyond inverted pendulum?

We present the case study only for the inverted pendulum because it is LQR-stabilizable (Section 5.2 and Appendix K). It connects the LQR setting of our theory to complex control environments from pixels, and it is one step in that direction. The ISTA layer does not learn an exactly orthonormal representation. It maps each underlying latent state along a unique direction via sparsification (Section 5.3), and for the inverted pendulum the learned representation is approximately orthogonal (Table 7). Beyond the inverted pendulum we do not have this measurement (see our response to Q4).

Q2 Have you assessed the loss of expressivity caused by the Stiefel constraints [1]?

Thank you for pointing us to this work. Our work is focused on the learning dynamics. The Stiefel constraints restrict the representable functions, and our analysis concerns this particular policy and critic class (Appendix B). Despite this, in the LQR experiment the upstairs agent converges close to the optimal policy (Figure 3). In the environments from pixels, where the actor and critic parameter matrices are optimized with manifold Muon and the encoder with Adam, the returns are higher than with Adam (Figure 6).

Q3 What is the comparison in Figure 3 intended to establish? [...] I do not see how this experiment conveys a nontrivial message.

Figure 3 validates Theorem 1: the learning dynamics of the upstairs agent match those of the downstairs agent. The upstairs agent does not observe MsMs. It observes ot=Mst+ϵξto_t = Ms_t + \epsilon\xi_t in d=128d = 128 dimensions, with observation noise ϵ=0.05\epsilon = 0.05. Further, its function approximators (policy, value estimate and advantage rate estimate) are depth-3 deep linear networks with d×dd \times d matrices, whereas the downstairs agent uses the summary matrices with the state of dimension ds=8d_s = 8. The two agents are initialized identically and are then trained separately, with the same martingale loss and gradient updates with a Cayley retraction. The match is not automatic: it holds for Stiefel manifold optimization under Assumption 2, and it need not be true for vanilla stochastic gradient descent or when Assumption 2 is not true (Section 4). Therefore, Figure 3 shows that the policy learning dynamics with O(Ld2)O(Ld^2) parameters and high-dimensional observations are nearly identical to those with O(ds2+dsda)O(d_s^2 + d_s d_a) parameters and the low-dimensional state.

Q4 What explains the higher returns of manifold Muon combined with ISTA-CNN over Adam on Mujoco tasks in Figure 6?

Both contribute, and Figure 6 and the additional experiments in Appendix J separate them. With the architecture fixed, CNN-manifold Muon generally matches or improves on CNN-Adam, which is the effect of the optimizer. With the optimizer fixed, the ISTA-CNN improves both Adam and manifold Muon, which is the effect of the architecture. Manifold Muon combined with the ISTA-CNN is the best on almost all tasks.

The optimizer effect is specific to the Stiefel constraint. In Appendix J, vanilla Muon orthonormalizes the gradient update but does not restrict the weights to lie on the Stiefel manifold, and with identical global hyperparameters it prevents learning in almost all environments.

AuthorsOpenReview ↗

Response Continued

On whether the learned features have the form MsMs with M⊤M=IM^\top M = I: we have direct evidence only for the inverted pendulum, where the downstairs state is recoverable by a linear probe with fewer samples under manifold Muon (Figure 5), and the learned representation is approximately orthogonal (Table 7). We do not claim that Assumption 2 holds for the other MuJoCo environments. There the theory motivates the optimizer-architecture co-design, and the experiments show that it improves performance.

We hope this addresses the concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.