acceptodds
Under review as a conference paper at ICLR 2027

Implicit Low-Rank Training Dynamics in Directly Parameterized Lipschitz-Bounded deep Networks

Abstract

Recent work has dedicated efforts to developing deep networks with prescribed Lipschitz bounds, in order to limit sensitivity to noisy or adversarial input perturbations. These networks enforce prescribed Lipschitz bounds through built-in structural constraints, typically derived from linear matrix inequalities (LMIs)/ semidefinite programs (SDPs), or imposed through orthogonality constraints on network layers. While these constraints provide certification, they can also make training less efficient. Empirical evidence shows that the training dynamics of large-scale networks often remain in low-dimensional subspaces and that certain initializations can lead to more effective training. Inspired by recent work on emergent low-rank training dynamics of multi-layer perceptrons (MLPs) under gradient descent (GD), we analyze the training dynamics of an existing 1-Lipschitz layer derived from an SDP certification, known as the “Sandwich layer", and show that the training dynamics of its effective weight concentrate within a low-dimensional subspace under GD. We first theoretically characterize this low-dimensional subspace in a shallow network with smooth, slope-restricted nonlinear activations, and empirically demonstrate that similar low-dimensional structure persists in deep networks under more general settings. Building on these observations, we develop a low-rank Sandwich layer that preserves its Lipschitz guarantees. This low-rank parameterization matches the performance of the fully parameterized Sandwich layer across multiple image classification tasks.

Primary area:
optimization
Keywords:
Lipschitz Network, Gradient Descent, Low-Dimensional Subspaces, Low-rank training dynamics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_R5kEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates low-rank training dynamics in sandwich layers, a layer type that structurally imposes Lipschitz bounds for the network. By considering a matrix decomposition that is in part independent of training time, it is possible to characterize low-rank training dynamics for one of the trainable matrices. By calculating said decomposition at train time, it is possible to train directly within this lower dimensional space. Various numerical experiments help support the mathematical analysis.

POST REBUTTAL: Raised my score to weak acceptance

Critical strengths
  • The idea of low-rank training is very useful and the demonstration that how one determines or parametrizes the low-rank representation can greatly affect performance is insightful towards general training dynamics of networks.
  • The paper provides nice numerical examples throughout the paper to illustrate theoretical concepts and results.
Critical weaknesses
  • It is not clear how one can conclude low-rank training dynamics from Theorem 3.6. Bounds are provided for the evolution of several sub-matrices but how does one know that the low-rank matrix Y~1,1\tilde Y_{1,1} will not evolve with even slower dynamics?
  • The explanation in Section 4 of how one should initialize a low-rank model based on Theorem 3.6 should be better explained. What is the motivation for (26) - why do we want to couple weights between layers?
  • The proofs in the appendix can be hard to follow, with inconsistent notation and non-trivial steps. Some points are mentioned in author questions.
Questions for authors

Queries:

  • What are the mathematical challenges for considering Sandwich layers with Ψk≠I\Psi_k\neq I? In terms of the performance, what are the benefits for considering a non-identity Ψk\Psi_k?
  • The product of Lipschitz constants for each layer is only an upper bound for the Lipschitz constant of the network. Does this analysis provide any insight into this issue?
  • The paper assumes that matrices whose Frobenius norm changes the most are the most important. What is the justification for this as opposed to, say, those whose span or maximal principal angle changes the most?
  • What is the main bottleneck to extending the theorem to allowing all parameters to be trainable?
  • A benefit of low-rank representations is to address memory restrictions. However, to determine L and R you need a dense representation to calculate a gradient. Is it possible to circumvent this issue?
  • Is it possible for a low-rank model to outperform the dense model?
  • How do sandwich layers compare numerically to other ways of enforcing a prescribed Lipschitz constant?

Clarifications:

  • Why do you assume the XiX_i are symmetric in (8)?
  • The spectral norm is a lower bound for the Frobenius norm, however you seem to use it as an upper bound. e.g. (59), (64).
  • Why does slope restrictedness allow one to conclude (60)?
  • Equation (38) should appear before P. In P, what does D_{-1} correspond to? Why do you choose Ψk≠I\Psi_k\neq I?
  • Shouldn't both A1A_1 and B1B_1 in (57) be transposed?

Requests:

  • The paper is a almost direct retelling of (Xiu et al. 2026) but in the setting of Sandwich layers instead of MLPs. While it is acknowledged that the paper is inspired by said work, this similarity should be more explicitly stated.
  • The empirical verification of the assumptions and low-rank training dynamics are only for a toy Gaussian mixture problem. Is it possible to verify these on actual datasets?
  • The metric (13) should be better motivated. For example, it is connected to the Frobenius norm of the difference of the projection matrices onto U1U_1 and U2U_2.
  • Can you provide plots for the upper bound (16) to see how it compares in practice?
  • Please include the derivations of (24) and (25), as they are not straightforward.
  • The grammar, punctuation, and notational clarity can be improved throughout the paper, e.g.: equations ending in a full-stop despite not being the end of the sentence; using d for both the input dimension and differential
6 comments
AuthorsOpenReview ↗

Response to Reviewer R5kE - CW1-CW3

Thank you for reviewing our manuscript. We appreciate the opportunity to discuss your questions. In the following response, we use “revised” to indicate equation numbers in the revised manuscript and “original” to indicate those in the original submission for clarity.

Response to Critical Weaknesses 1:

Theorem 3.6 bounds the components of each parameter update that lie outside the leading low-dimensional subspace. The connection between Theorem 3.6 and low-rank concentration should be made more explicit. Here, we briefly summarize the important steps. The details have been added to Appendix A.9.6 of the revised manuscript. For equation references, we use “revised” to indicate equation numbers in the revised manuscript and “original” to indicate those in the original submission.

Let p:=d−2Kp:=d-2K, ΔY1(t):=Y1(t+1)−Y1(t)\Delta Y_1(t):=Y_1(t+1)-Y_1(t), and ΔY~1,j(t):=Y~1,j(t+1)−Y~1,j(t)\Delta\widetilde Y_{1,j}(t):=\widetilde Y_{1,j}(t+1)-\widetilde Y_{1,j}(t). Squaring equation (17, revised; 16, original) and summing over the update components outside the leading low-dimensional subspace gives

{\lVert\Delta Y_1(t)\rVert_F^2} \le \mathcal{D}(t):= \frac{3p\\,e(t)^2}{\lVert G_1(t)\rVert_F^2}.$$ By (18, revised), $e(0)=O(\rho)$. Therefore, with small initialization, $\mathcal D(0)=O(\rho^2)$. For $t>0$, we obtain the following bounds: $$e(t) \lesssim r_{\rho} + \left(\frac{c}{\sqrt{p}} + L_G c_1 \lVert G_1(0) \rVert_F\right)\eta t,$$ $$\mathcal{D}(t) \le \frac{12p e(t)^2}{\lVert G_1(0) \rVert_F^2}.$$ Here, $c$ and $c_1$ are constants, $L_G$ is defined in (84, revised), and $r_\rho=O(\rho)$ bounds the initial gradient approximation error, as established in Lemma A.6. Therefore, when the initialization scale $\rho$ and the learning rate $\eta$ are sufficiently small, $e(t)$ is small and thus $\mathcal D(t)$ is small. This implies that the updates to components outside the leading low-dimensional subspace are small during the early training phase. Thus, the parameter updates are predominantly concentrated in the leading low-dimensional subspace during the early training phase. We also ran additional simulations on Fashion-MNIST beyond the synthetic dataset and showed that updates to components outside the leading low-dimensional subspace are small (see Figure 19 in Appendix A.10 of the revised manuscript).   **Response to Critical Weaknesses 2:** This coupling design is intended to handle the “interlacing” coupling induced by the Cayley structure. Recall that each hidden layer is given by $$h_k = \sqrt{2}(2C_k-I)\sigma\left(-2\sqrt{2}g_k C_k^\top Y_k^\top h_{k-1}\right).$$ By Assumption 3.3, $\sigma(0)=0$ and $a=\sigma'(0)>0$. With small initialization, $C_k=(I+S_k+Y_k^\top Y_k)^{-1}\approx I$. Therefore, each hidden layer can be approximated by $$h_k \approx \sqrt{2}I a\left(-2\sqrt{2}g_k IY_k^\top h_{k-1}\right) = -4ag_k Y_k^\top h_{k-1}.$$ Substituting the low-rank parameterization $Y_k=L_kM_kR_k^\top$, we see that each hidden layer acts approximately as $$h_k \approx -4ag_k R_k M_k^\top L_k^\top h_{k-1}.$$ Adjacent layers interact through $L_{k+1}^\top R_k$. Choosing $R_k=L_{k+1}=F_k$ with $F_k^\top F_k=I_r$ is one way to align the internal factors to avoid additional attenuation caused by internal subspace misalignment. During training, freezing these shared internal factors preserves this alignment while reducing the number of trainable parameters. We have added Figure 4 to illustrate the overall architecture and revised the initialization section to improve clarity.   **Response to Critical Weaknesses 3:** We have clarified the notation and added more details. Please see our detailed response to the related questions.
AuthorsOpenReview ↗

Response to Reviewer R5kE - Queries1-4

Response to Queries 1: Non-Identity Ψk\Psi_k Challenges and Benefits

We set Ψk=I\Psi_k=I to isolate the training dynamics of the Cayley parameters and simplify the analysis. For a fixed positive diagonal Ψk\Psi_k, its effect can be absorbed into the activations. The leading-order linearization at zero remains unchanged, but we need to account for additional details, such as Taylor expansion remainders and related bounds. Thus, for non-identity Ψk\Psi_k, we expect the same proof strategy to apply with minor changes.

In terms of performance, Ψk\Psi_k parameterizes the positive diagonal multiplier Λk=12Ψk2\Lambda_k=\tfrac12\Psi_k^2 used in the Lipschitz-certification LMI in (40, revised). Choosing a fixed non-identity Ψk\Psi_k changes the multiplier and the associated unit-specific activation scales. Relaxing Ψk\Psi_k to be non-identity may be beneficial only when the chosen scales suit the task. Allowing it trainable would make the gradient analysis more challenging. Therefore, for clarity, we set it to the identity matrix.

 

Response to Queries 2: Insights of Lipschitz upper bound

Our analysis focuses on low-rank training dynamics in Sandwich networks and does not theoretically characterize the tightness of the Lipschitz bound. Empirically, Figure 6 shows that our low-rank parameterization achieves tightness comparable to that of the dense Sandwich network, which retains a tight certificate. A general theoretical analysis of upper bound tightness is beyond the scope of this work but could be an interesting direction for future research.

 

Response to Queries 3: Metric choice

We use the Frobenius metric to quantify the concentration of the updates, which captures changes in both magnitude and orientation. Our goal is to identify low-dimensional subspaces that capture most of the parameter update during training. Since our block decomposition uses fixed orthogonal bases, the squared Frobenius displacements of the blocks sum to the total squared displacement, so the metric directly captures each block's contribution to the overall parameter update. Principal angles provide information about the subspace orientation but do not reflect changes in magnitude. Therefore, principal angles alone cannot quantify how much of the parameter update a subspace captures, and Frobenius metric is a better choice for our measurement.

 

Response to Queries 4: Bottleneck to make all parameters trainable

The main bottleneck to extending the theorem to allow all parameters to be trainable is that we need to control the gradient coupling and dominant-subspace rotation. The current theory focuses only on the Y1Y_1 variable in the two-layer case, and its analytical gradients already couple to many terms. When these additional parameters are trainable, their dynamics also affect the gradient of Y1Y_1 through the readout, the ``interlacing'' coupling induced by the Cayley structure. Extending the theorem would require substantial additional analysis to obtain joint trajectory bounds and control the spectral separation. It could be quite challenging.

AuthorsOpenReview ↗

Response to Reviewer R5kE - Queries 5

Response to Queries 5: Initialization without of a Full-Rank Network

We had thought about this before and spent some time addressing it. Now we propose an initialization that can potentially avoid it. The intuition and motivation are given in Appendix A.11 of the revised manuscript. Below, we summarize this new initialization, which we call the low-rank initialization and which does not use the full-rank network.

  1. We directly construct a low-rank deep network with randomly initialized endpoint factors L^1\widehat{L}_1 and R^L−1\widehat{R}_{L-1}. The cores, internal factors, and readout layer are initialized at small scales or set to zero, as before.

  2. We compute the gradients with respect to L^1\widehat{L}_1 and R^L−1\widehat{R}_{L-1}, denoted by JLJ_L and JRJ_R, respectively.

  3. We perform an SVD of JLJ_L and JRJ_R and take their top-qq left and right singular vectors, denoted by (UL,q,VL,q)(U_{L,q},V_{L,q}) and (UR,q,VR,q)(U_{R,q},V_{R,q}), respectively, where q:=r/2q:=r/2 and rr is the preselected rank:

    JL=ULΣLVL⊤,JR=URΣRVR⊤.J_L=U_L\Sigma_LV_L^\top, \qquad J_R=U_R\Sigma_RV_R^\top.

  4. We initialize the endpoint factors as

    L1(0)=orth⁡[UL,q,L^1VL,q],L_1(0)=\operatorname{orth}\left[U_{L,q}, \widehat{L}_1 V_{L,q}\right],

    RL−1(0)=orth⁡[UR,q,R^L−1VR,q].R_{L-1}(0)=\operatorname{orth}\left[U_{R,q}, \widehat{R}_{L-1} V_{R,q}\right].

We then use those values for the endpoint factors, reset the other components, and start training.

The intuition is that at small initialization, both networks are approximately linear. Under suitable conditions, the low-rank endpoint gradients can identify the same informative subspaces as the dense-network gradient with respect to Y1Y_1 and YL−1Y_{L-1}. See Appendix A.11 of the revised manuscript for details.

We also validated this low-rank initialization on CIFAR-10. Its performance matches that of the initialization that relies on the full-rank network.

Comparison of low-rank and full-rank initialization

Clean and robust classification accuracy (%) at different ranks. AA denotes AutoAttack, and CRA denotes certified robust accuracy. The fractions in the AA and CRA headers indicate the perturbation budget ϵ\epsilon. Results are reported as mean ± standard deviation.

RankInitializationClean Acc.AA 36/255AA 72/255AA 108/255CRA 36/255CRA 72/255CRA 108/255
10Low-rank60.8±0.352.5±0.343.8±0.236.5±1.042.6±0.425.4±0.712.6±0.6
10Full-rank60.7±0.153.4±0.544.1±0.635.4±1.141.8±0.624.0±0.811.2±0.8
20Low-rank64.1±0.055.1±0.346.2±0.137.2±0.345.8±0.328.7±0.215.5±0.1
20Full-rank64.2±0.255.3±0.445.9±0.337.3±0.346.2±0.029.0±0.115.6±0.1
40Low-rank64.7±0.155.4±0.746.5±0.338.0±0.446.7±0.129.6±0.215.6±0.2
40Full-rank64.6±0.155.9±0.146.6±0.337.2±0.346.8±0.229.7±0.115.8±0.1
80Low-rank65.2±0.156.6±0.647.5±0.238.4±0.247.5±0.129.9±0.215.6±0.0
80Full-rank65.2±0.156.3±0.747.7±0.238.5±0.247.3±0.130.0±0.115.7±0.2
160Low-rank65.6±0.157.2±0.347.4±0.338.6±0.247.5±0.130.3±0.115.7±0.1
160Full-rank65.6±0.256.8±0.447.6±0.338.2±0.747.5±0.230.0±0.215.6±0.2
Dense—66.3±0.157.8±0.347.9±0.438.7±0.348.2±0.130.3±0.115.3±0.0

 

Response to Queries 6: Can low-rank models outperform dense models

In principle, it is possible. Low-rank parameterization provides a form of structural regularization that may reduce overfitting compared with the dense network. However, although the dominant subspaces are designed to capture most of the useful signal, they might also exclude useful gradient components. From this perspective, we speculate that some tasks could favor the low-rank model and lead to better performance. That said, we have not observed a consistent advantage over dense models in our experiments.

 

Response to Queries 7: Numerical Comparison with Other Lipschitz-Constrained Methods

In terms of Lipschitz-bound tightness, test accuracy, and certified robust accuracy, Sandwich networks are competitive with methods that require strict norm constraints or orthogonality constraints on weights. However, some methods have achieved impressive performance, such as LipNext (Hu, 2026), which scales the network to billions of parameters and leverages additional diffusion-generated training data. Our primary focus in this work is understanding low-rank training dynamics, not tuning the hyperparameters to achieve the best possible performance.

Reviewer_uUCfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the training dynamics of the “Sandwich layer”, which stands for an existing 1-Lipschitz layer derived from an SDP certification using the Cayley transformation. In a two-layer network (the second layer is frozen), the authors prove that gradient-descent updates of Y1Y_1 are concentrated in a fixed subspace, using a small orthogonal initialization. The analysis builds on recent work for standard MLPs (Xu et al., 2026) and adapts it to the Sandwich structure. In the experiments, deep networks are tested. The proposed low-rank Sandwich networks can achieve nearly the same test accuracy and test loss, compared to the full-rank Sandwich network, using fewer parameters while preserving the Lipschitz certification. With the proposed initialization, the low-rank model nearly matches the full model’s test accuracy and test loss on Fashion-MNIST, and its clean accuracy and AutoAttack robustness on CIFAR-10 and CIFAR-100. When the random initialization is used, the low-rank Sandwich classifier performs much worse.

Critical strengths

Strength 1:

This paper provides an interesting and motivated idea that connects two research directions. It links certified Lipschitz networks with the growing literature on low-dimensional training dynamics. The authors extend existing observations to the Sandwich architecture and investigate the phenomenon in both shallow and deep networks. There is a detailed mathematical analysis in the paper. The Sandwich step using the Cayley structure under specific assumptions is clearly explained and discussed.

Strength 2:

The experimental design with different initializations is nice and it gives many interesting results. It also shows how the choice of initialization affects the performance of low-rank models.

Critical weaknesses

Weakness 1:

The presented theory covers only a two-layer network with frozen second layer, whitened input data and K<d/2K < d/2 (Assumption 3.1), and a training trajectory assumption (Assumption 3.4), which is validated only empirically

Questions for authors
  1. Eq. (27) specifies the initialization for r=2Kr=2K, but the experiments use other ranks, including many cases where r≠2Kr\neq 2K. Could you clarify how they are initialized in these experiments?

  2. In L. 1054, you wrote 3 seeds, but why does Table 1 not report the respective standard deviations?

  3. One important advantage of using a Lipschitz network is that it gives a certified robustness. Table 1 reports AutoAttack results, but could you report the certified robust accuracy for all models in Table 1?

  4. Is the CIFAR-100 random initialization r=20r = 20 (as written in the legend of Figure 12) or r=200r = 200 (Table 1)?

3 comments
AuthorsOpenReview ↗

Response to Reviewer uUCf - CW1 and Q1

Thank you for reviewing our work, and we are happy to discuss your questions. In the following response, we use “revised” to indicate equation numbers in the revised manuscript and “original” to indicate those in the original submission for clarity.

Response to Critical Weaknesses 1

In the theoretical analysis, we freeze the second layer to simplify the analysis. This is a common assumption in related work (Xu et al., 2026; Oymak & Soltanolkotabi, 2019; 2020; Frei et al., 2022; Kou et al., 2023; Gopalani & Mukherjee, 2025). For the rank assumption, KK denotes the output dimension and bounds the rank of the leading-order initial gradient. The condition K<d/2K<d/2 ensures that the rank-2K2K description is nontrivial relative to the input dimension in our analysis. Whitening is a technical assumption that simplifies the analysis. Our existing experiments in Figure 8 (Figure 9 in the revised manuscript) in Appendix A.4 also examine deep Sandwich networks with unwhitened inputs and exhibit similar low-rank training dynamics. This empirically supports our assumption.

We have also added experiments in Appendix A.10 of the revised manuscript to empirically evaluate Assumption 3.4 on CIFAR-10 and inequality (17, revised; 16, original) of Theorem 3.6 both under our theoretical setting and on Fashion-MNIST. These results provide empirical support for our assumptions and theoretical results on real-world datasets.

Response to Question 1 - Rank r≠2Kr \neq 2K Initialization

Thank you for pointing this out. Equation (27, original) presented the special case r=2Kr=2K, following our theoretical analysis. For a general even rank rr, we first construct a full-rank network, compute its initial gradient G1(0)G_1(0), and take its SVD, G1(0)=UΣV⊤G_1(0)=U\Sigma V^\top. We then set q:=r/2q:=r/2 and let L1,1(0)∈Rd×qL_{1,1}(0)\in\mathbb{R}^{d\times q} and R1,1(0)∈Rm×qR_{1,1}(0)\in\mathbb{R}^{m\times q} denote the top-qq left and right singular vectors of G1(0)G_1(0):

L1,1(0)=U[:,1:q],R1,1(0)=V[:,1:q].L_{1,1}(0)=U_{[:,1:q]}, \quad R_{1,1}(0)=V_{[:,1:q]}.

Then we initialize the input endpoint factor L1(0)L_1(0) as:

L1(0)=orth[L1,1(0),Q1R1,1(0)]∈Rd×r.L_1(0)=\mathrm{orth}\left[L_{1,1}(0), Q_1 R_{1,1}(0)\right] \in \mathbb{R}^{d\times r}.

where Q1Q_1 is a random semi-orthogonal matrix. We then propagate this factor through the full-rank hidden layers to initialize the other endpoint factor RL−1(0)R_{L-1}(0), as described in (31, revised)--(32c, revised). The remaining low-rank factors and cores follow the same initialization method described in Section 4, with their dimensions adjusted to rr. The dense readout initialization is independent of rr. Thus, the key change when varying the rank rr is the number q=r/2q=r/2 of leading left and right singular vectors used to initialize the endpoint factors.

AuthorsOpenReview ↗

Response to Reviewer uUCf - Q2-Q4

Response to Question 2 - Report Table 1 standard deviation

Due to page limits, we did not report all standard deviations in the old submission. We have now included them in the revised manuscript and provide them below for convenience.

CIFAR-10 — SMALL, γ=1\gamma=1

RankClean Acc.AA 36/255AA 72/255AA 108/255CRA 36/255CRA 72/255CRA 108/255Parameters
1060.7±0.153.4±0.644.1±0.835.4±1.341.8±0.724.0±1.011.2±0.952K
2064.2±0.255.3±0.545.9±0.437.3±0.346.2±0.129.0±0.215.6±0.199K
4064.6±0.155.9±0.146.6±0.437.2±0.446.8±0.229.7±0.115.8±0.1194K
8065.2±0.156.3±0.847.7±0.238.5±0.247.3±0.230.0±0.115.7±0.2393K
20*16.0±1.98.2±1.64.5±0.92.1±0.30.00.00.099K
Dense66.3±0.257.8±0.447.9±0.538.7±0.348.2±0.230.3±0.115.3±0.12.63M

CIFAR-100 — LARGE, γ=2\gamma=2

RankClean Acc.AA 36/255AA 72/255AA 108/255CRA 36/255CRA 72/255CRA 108/255Parameters
20052.8±0.143.9±0.435.7±0.528.7±1.321.2±0.17.0±0.12.0±0.04.02M
40054.2±0.345.3±0.236.7±0.630.8±0.622.5±0.27.7±0.12.3±0.08.06M
80054.8±0.545.8±0.137.9±0.730.6±0.422.9±0.17.9±0.12.4±0.116.87M
200*5.6±0.32.9±1.01.3±0.50.8±0.20.00.00.04.02M
Dense55.4±0.146.1±1.137.4±0.630.8±0.524.0±0.28.7±0.12.7±0.142.15M

CIFAR-100 — SMALL-LLN, γ=1\gamma=1

RankClean Acc.AA 36/255AA 72/255AA 108/255CRA 36/255CRA 72/255CRA 108/255Parameters
20040.6±0.534.5±0.228.7±0.424.0±0.429.1±0.120.8±0.215.0±0.11.31M
40041.0±0.234.5±0.128.6±0.524.0±0.429.6±0.121.1±0.115.4±0.12.87M
80041.3±0.134.9±0.429.0±0.424.1±0.229.8±0.121.0±0.115.3±0.17.08M
200*1.0±0.01.1±0.41.1±0.41.1±0.40.00.00.01.31M
Dense41.9±0.234.9±0.729.3±0.324.7±0.330.2±0.221.8±0.116.0±0.119.50M

 

Response to Question 3 - Report certified robust accuracy (CRA)

We now report CRA scores as mean ±\pm standard deviation for the previously evaluated models and datasets. We also include an additional Sandwich variant that achieves higher CRA. The general trends still hold for CRA: as the rank increases, the low-rank model approaches the performance of the dense model, while the randomly initialized model remains stuck at a low performance level. The full table is included in our answer to the previous question.

 

Response to Question 4 - Typo in the legend

Thank you for catching this typo. It has now been corrected in the revised manuscript.

Reviewer_uUCfOpenReview ↗

Response to authors' rebuttal

Thank you for the detailed response and the revised manuscript. My questions regarding the rank initialization, standard deviations, certified robust accuracy, and the typo have been addressed. I also appreciate the additional experiments in Appendix A.10, which provide useful empirical support.

Regarding Weakness 1, I acknowledge that freezing the second layer is a standard simplification in theoretical analyses of two-layer networks, including Xu et al. (2026). My concern is with the combined restrictions of the theoretical setting. The additional experiments provide evidence that the observed dynamics extend beyond these assumptions, but a theoretical guarantee for fully trainable deep Sandwich networks remains open. I therefore maintain my overall positive rating of 3.

Reviewer_GMSbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper first identifies the low-rank training dynamics in Lipschitz-bounded neural networks (specifically the Sandwich architecture), then studies the dynamics from both theoretical empirical perspective.

In particular, the authors theoretically characterize gradient concentration in a shallow (2-layer, with only one layer trainable parameter) network under restrictive assumptions and empirically observe similar behavior in deeper networks. Motivated by these results, they propose a new Sandwich parameterization that preserves Lipschitz guarantees while reducing trainable parameters. The authors then conduct experiments in real data to demonstrate the performance.

Critical strengths
  1. I think this paper has a meaningful motivation. And it seems novel to me to study the low-rank training dynamics for the Lipschitz-bounded networks.

    Furthermore, following the line "motivation for studying low-rank learning dynamics --> theoretical investigation of the low-rank learning dynamics on toy models --> practical inspiration for new parameterization" provides a complete, compelling story.

  2. Extending low-rank dynamics analysis to the nonlinear Cayley parameterization requires additional technical work.

Critical weaknesses
  1. I have some concerns on the novelty and significance.

    While the authors acknowledge the inspiration from Xu et al. (2026), the central theoretical result still closely resembles Xu et al. (2026), which already establishes low-rank gradient concentration and motivates parameter-efficient training. And the current paper extends the same phenomenon and proof strategy to the Sandwich architecture.

    Several questions remain unanswered. For example, how does Lipschitz constraints fundamentally alter learning dynamics? what new mechanism specific to the constrained networks do you observe? I thus think the contribution appears to be an extension, rather than a new theoretical insight.

  2. There are some unclear theoretical results/arguments.

    • Theorem 3.6 does not establish strong low-rank concentration. The theorem bounds ∥Y~1,j(t+1)−Y~1,j(t)∥Fd−2K≤ηe(t)\frac{\|\widetilde Y_{1,j}(t + 1) - \widetilde Y_{1,j}(t)\|_F}{\sqrt{d-2K}} \leq \eta e(t), but it provides no guarantee that these updates are small relative to the total parameter update. As a result, the theorem does not generally establish that most training occurs within the claimed low-dimensional subspace.

    • The theoretical setup is too limited. For example, the network only has a frozen readout, and Assumption 3.4 is too strong.

Questions for authors
  1. What fundamentally new insight about learning dynamics arises from the Lipschitz-bounded setting beyond the results of Xu et al. (2026)?

  2. Please see my concerns in the second point of Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer GMSb

Thank you for reviewing our work. We appreciate your effort and the opportunity to discuss your questions.

Response to Question 1 and Critical Weakness 1

We extend the low-rank training dynamics analysis to the Sandwich layer, which has a nontrivial nonlinear ``interlacing'' coupling used to enforce Lipschitz certification. This coupling complicates the analysis but provides a Lipschitz upper-bound guarantee.

This coupling structure contributes a gradient to Y1Y_1 through C1C_1. An interesting finding is that, in our theoretical setting (small initialization in particular), the contribution to the gradient with respect to Y1Y_1 via coupling term C1C_1 is small. The full gradient with respect to Y1Y_1 has a leading rank-KK term with a small remainder. This motivated us to further investigate Sandwich networks, and our analysis may also shed light on the training dynamics of other architectures with similar nonlinear and inter-layer parameter coupling.

Our low-rank parameterization also differs from that of previous work. The ``interlacing'' coupling in the Cayley structure requires additional technical work to expose the internal factor involving only r×rr\times r matrix inversions while preserving the built-in Lipschitz guarantee.

 

Response to Question 2 and Critical Weakness 2

  • Connection between Theorem 3.6 and low-rank concentration

    Theorem 3.6 bounds the components of each parameter update that lie outside the leading low-dimensional subspace. The connection between Theorem 3.6 and low-rank concentration should be made more explicit. Here, we briefly summarize the important steps. The details have been added to Appendix A.9.6 of the revised manuscript. For equation references, we use “revised” to indicate equation numbers in the revised manuscript and “original” to indicate those in the original submission.

    Let p:=d−2Kp:=d-2K, ΔY1(t):=Y1(t+1)−Y1(t)\Delta Y_1(t):=Y_1(t+1)-Y_1(t), and ΔY~1,j(t):=Y~1,j(t+1)−Y~1,j(t)\Delta\widetilde Y_{1,j}(t):=\widetilde Y_{1,j}(t+1)-\widetilde Y_{1,j}(t). Squaring equation (17, revised; 16, original) and summing over the update components outside the leading low-dimensional subspace gives

    {\lVert\Delta Y_1(t)\rVert_F^2} \le \mathcal{D}(t) := \frac{3p\\,e(t)^2}{\lVert G_1(t)\rVert_F^2}.$$ By (18, revised), $e(0)=O(\rho)$. Therefore, with small initialization, $\mathcal D(0)=O(\rho^2)$. For $t>0$, we obtain the following bounds: $$e(t) \lesssim r_{\rho} + \left(\frac{c}{\sqrt{p}} + L_G c_1 \lVert G_1(0) \rVert_F\right)\eta t,$$ $$\mathcal{D}(t) \le \frac{12p e(t)^2}{\lVert G_1(0) \rVert_F^2}.$$ Here, $c$ and $c_1$ are constants, $L_G$ is defined in (84, revised), and $r_\rho=O(\rho)$ bounds the initial gradient approximation error, as established in Lemma A.6. &nbsp; Therefore, when the initialization scale $\rho$ and the learning rate $\eta$ are sufficiently small, $e(t)$ is small and thus $\mathcal D(t)$ is small. This implies that the updates to components outside the leading low-dimensional subspace are small during the early training phase. Thus, the parameter updates are predominantly concentrated in the leading low-dimensional subspace during the early training phase. We also ran additional simulations on Fashion-MNIST beyond the synthetic dataset and showed that updates to components outside the leading low-dimensional subspace are small (see Figure 19 in Appendix A.10 of the revised manuscript).

 

  • Theoretical setup and additional Assumption validation

    In terms of the theoretical setup, freezing the second layer to simplify the analysis is a common assumption in related work (Xu et al., 2026; Oymak & Soltanolkotabi, 2019; 2020; Frei et al., 2022; Kou et al., 2023; Gopalani & Mukherjee, 2025).

    We have also added experiments in Appendix A.10 of the revised manuscript to empirically evaluate Assumption 3.4 on CIFAR-10 and inequality (17, revised; 16, original) of Theorem 3.6 both under our theoretical setting and on Fashion-MNIST. These results provide additional support for our assumptions and theoretical results on real-world datasets.

Thank you for the detailed response and revisions.

The added analysis addresses my concern about the connection between Theorem 3.6 and low-rank concentration during early training.

My concerns about novelty and theoretical scope remain. While handling the Cayley coupling requires additional technical work, I still view the theoretical contribution primarily as an extension of Xu et al. (2026), with limited new insight specific to Lipschitz-constrained learning dynamics. The theory also remains restricted to a shallow network with a frozen readout.

Taking the improvements and remaining limitations together, I thus maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.