- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper investigates low-rank training dynamics in sandwich layers, a layer type that structurally imposes Lipschitz bounds for the network. By considering a matrix decomposition that is in part independent of training time, it is possible to characterize low-rank training dynamics for one of the trainable matrices. By calculating said decomposition at train time, it is possible to train directly within this lower dimensional space. Various numerical experiments help support the mathematical analysis.
POST REBUTTAL: Raised my score to weak acceptance
Critical strengths
- The idea of low-rank training is very useful and the demonstration that how one determines or parametrizes the low-rank representation can greatly affect performance is insightful towards general training dynamics of networks.
- The paper provides nice numerical examples throughout the paper to illustrate theoretical concepts and results.
Critical weaknesses
- It is not clear how one can conclude low-rank training dynamics from Theorem 3.6. Bounds are provided for the evolution of several sub-matrices but how does one know that the low-rank matrix will not evolve with even slower dynamics?
- The explanation in Section 4 of how one should initialize a low-rank model based on Theorem 3.6 should be better explained. What is the motivation for (26) - why do we want to couple weights between layers?
- The proofs in the appendix can be hard to follow, with inconsistent notation and non-trivial steps. Some points are mentioned in author questions.
Questions for authors
Queries:
- What are the mathematical challenges for considering Sandwich layers with ? In terms of the performance, what are the benefits for considering a non-identity ?
- The product of Lipschitz constants for each layer is only an upper bound for the Lipschitz constant of the network. Does this analysis provide any insight into this issue?
- The paper assumes that matrices whose Frobenius norm changes the most are the most important. What is the justification for this as opposed to, say, those whose span or maximal principal angle changes the most?
- What is the main bottleneck to extending the theorem to allowing all parameters to be trainable?
- A benefit of low-rank representations is to address memory restrictions. However, to determine L and R you need a dense representation to calculate a gradient. Is it possible to circumvent this issue?
- Is it possible for a low-rank model to outperform the dense model?
- How do sandwich layers compare numerically to other ways of enforcing a prescribed Lipschitz constant?
Clarifications:
- Why do you assume the are symmetric in (8)?
- The spectral norm is a lower bound for the Frobenius norm, however you seem to use it as an upper bound. e.g. (59), (64).
- Why does slope restrictedness allow one to conclude (60)?
- Equation (38) should appear before P. In P, what does D_{-1} correspond to? Why do you choose ?
- Shouldn't both and in (57) be transposed?
Requests:
- The paper is a almost direct retelling of (Xiu et al. 2026) but in the setting of Sandwich layers instead of MLPs. While it is acknowledged that the paper is inspired by said work, this similarity should be more explicitly stated.
- The empirical verification of the assumptions and low-rank training dynamics are only for a toy Gaussian mixture problem. Is it possible to verify these on actual datasets?
- The metric (13) should be better motivated. For example, it is connected to the Frobenius norm of the difference of the projection matrices onto and .
- Can you provide plots for the upper bound (16) to see how it compares in practice?
- Please include the derivations of (24) and (25), as they are not straightforward.
- The grammar, punctuation, and notational clarity can be improved throughout the paper, e.g.: equations ending in a full-stop despite not being the end of the sentence; using d for both the input dimension and differential
6 comments
Response to Reviewer R5kE - Queries1-4
Response to Queries 1: Non-Identity Challenges and Benefits
We set to isolate the training dynamics of the Cayley parameters and simplify the analysis. For a fixed positive diagonal , its effect can be absorbed into the activations. The leading-order linearization at zero remains unchanged, but we need to account for additional details, such as Taylor expansion remainders and related bounds. Thus, for non-identity , we expect the same proof strategy to apply with minor changes.
In terms of performance, parameterizes the positive diagonal multiplier used in the Lipschitz-certification LMI in (40, revised). Choosing a fixed non-identity changes the multiplier and the associated unit-specific activation scales. Relaxing to be non-identity may be beneficial only when the chosen scales suit the task. Allowing it trainable would make the gradient analysis more challenging. Therefore, for clarity, we set it to the identity matrix.
Response to Queries 2: Insights of Lipschitz upper bound
Our analysis focuses on low-rank training dynamics in Sandwich networks and does not theoretically characterize the tightness of the Lipschitz bound. Empirically, Figure 6 shows that our low-rank parameterization achieves tightness comparable to that of the dense Sandwich network, which retains a tight certificate. A general theoretical analysis of upper bound tightness is beyond the scope of this work but could be an interesting direction for future research.
Response to Queries 3: Metric choice
We use the Frobenius metric to quantify the concentration of the updates, which captures changes in both magnitude and orientation. Our goal is to identify low-dimensional subspaces that capture most of the parameter update during training. Since our block decomposition uses fixed orthogonal bases, the squared Frobenius displacements of the blocks sum to the total squared displacement, so the metric directly captures each block's contribution to the overall parameter update. Principal angles provide information about the subspace orientation but do not reflect changes in magnitude. Therefore, principal angles alone cannot quantify how much of the parameter update a subspace captures, and Frobenius metric is a better choice for our measurement.
Response to Queries 4: Bottleneck to make all parameters trainable
The main bottleneck to extending the theorem to allow all parameters to be trainable is that we need to control the gradient coupling and dominant-subspace rotation. The current theory focuses only on the variable in the two-layer case, and its analytical gradients already couple to many terms. When these additional parameters are trainable, their dynamics also affect the gradient of through the readout, the ``interlacing'' coupling induced by the Cayley structure. Extending the theorem would require substantial additional analysis to obtain joint trajectory bounds and control the spectral separation. It could be quite challenging.
Response to Reviewer R5kE - Queries 5
Response to Queries 5: Initialization without of a Full-Rank Network
We had thought about this before and spent some time addressing it. Now we propose an initialization that can potentially avoid it. The intuition and motivation are given in Appendix A.11 of the revised manuscript. Below, we summarize this new initialization, which we call the low-rank initialization and which does not use the full-rank network.
-
We directly construct a low-rank deep network with randomly initialized endpoint factors and . The cores, internal factors, and readout layer are initialized at small scales or set to zero, as before.
-
We compute the gradients with respect to and , denoted by and , respectively.
-
We perform an SVD of and and take their top- left and right singular vectors, denoted by and , respectively, where and is the preselected rank:
-
We initialize the endpoint factors as
We then use those values for the endpoint factors, reset the other components, and start training.
The intuition is that at small initialization, both networks are approximately linear. Under suitable conditions, the low-rank endpoint gradients can identify the same informative subspaces as the dense-network gradient with respect to and . See Appendix A.11 of the revised manuscript for details.
We also validated this low-rank initialization on CIFAR-10. Its performance matches that of the initialization that relies on the full-rank network.
Comparison of low-rank and full-rank initialization
Clean and robust classification accuracy (%) at different ranks. AA denotes AutoAttack, and CRA denotes certified robust accuracy. The fractions in the AA and CRA headers indicate the perturbation budget . Results are reported as mean ± standard deviation.
| Rank | Initialization | Clean Acc. | AA 36/255 | AA 72/255 | AA 108/255 | CRA 36/255 | CRA 72/255 | CRA 108/255 |
|---|---|---|---|---|---|---|---|---|
| 10 | Low-rank | 60.8±0.3 | 52.5±0.3 | 43.8±0.2 | 36.5±1.0 | 42.6±0.4 | 25.4±0.7 | 12.6±0.6 |
| 10 | Full-rank | 60.7±0.1 | 53.4±0.5 | 44.1±0.6 | 35.4±1.1 | 41.8±0.6 | 24.0±0.8 | 11.2±0.8 |
| 20 | Low-rank | 64.1±0.0 | 55.1±0.3 | 46.2±0.1 | 37.2±0.3 | 45.8±0.3 | 28.7±0.2 | 15.5±0.1 |
| 20 | Full-rank | 64.2±0.2 | 55.3±0.4 | 45.9±0.3 | 37.3±0.3 | 46.2±0.0 | 29.0±0.1 | 15.6±0.1 |
| 40 | Low-rank | 64.7±0.1 | 55.4±0.7 | 46.5±0.3 | 38.0±0.4 | 46.7±0.1 | 29.6±0.2 | 15.6±0.2 |
| 40 | Full-rank | 64.6±0.1 | 55.9±0.1 | 46.6±0.3 | 37.2±0.3 | 46.8±0.2 | 29.7±0.1 | 15.8±0.1 |
| 80 | Low-rank | 65.2±0.1 | 56.6±0.6 | 47.5±0.2 | 38.4±0.2 | 47.5±0.1 | 29.9±0.2 | 15.6±0.0 |
| 80 | Full-rank | 65.2±0.1 | 56.3±0.7 | 47.7±0.2 | 38.5±0.2 | 47.3±0.1 | 30.0±0.1 | 15.7±0.2 |
| 160 | Low-rank | 65.6±0.1 | 57.2±0.3 | 47.4±0.3 | 38.6±0.2 | 47.5±0.1 | 30.3±0.1 | 15.7±0.1 |
| 160 | Full-rank | 65.6±0.2 | 56.8±0.4 | 47.6±0.3 | 38.2±0.7 | 47.5±0.2 | 30.0±0.2 | 15.6±0.2 |
| Dense | — | 66.3±0.1 | 57.8±0.3 | 47.9±0.4 | 38.7±0.3 | 48.2±0.1 | 30.3±0.1 | 15.3±0.0 |
Response to Queries 6: Can low-rank models outperform dense models
In principle, it is possible. Low-rank parameterization provides a form of structural regularization that may reduce overfitting compared with the dense network. However, although the dominant subspaces are designed to capture most of the useful signal, they might also exclude useful gradient components. From this perspective, we speculate that some tasks could favor the low-rank model and lead to better performance. That said, we have not observed a consistent advantage over dense models in our experiments.
Response to Queries 7: Numerical Comparison with Other Lipschitz-Constrained Methods
In terms of Lipschitz-bound tightness, test accuracy, and certified robust accuracy, Sandwich networks are competitive with methods that require strict norm constraints or orthogonality constraints on weights. However, some methods have achieved impressive performance, such as LipNext (Hu, 2026), which scales the network to billions of parameters and leverages additional diffusion-generated training data. Our primary focus in this work is understanding low-rank training dynamics, not tuning the hyperparameters to achieve the best possible performance.
Response to Reviewer R5kE - CW1-CW3
Thank you for reviewing our manuscript. We appreciate the opportunity to discuss your questions. In the following response, we use “revised” to indicate equation numbers in the revised manuscript and “original” to indicate those in the original submission for clarity.
Response to Critical Weaknesses 1:
Theorem 3.6 bounds the components of each parameter update that lie outside the leading low-dimensional subspace. The connection between Theorem 3.6 and low-rank concentration should be made more explicit. Here, we briefly summarize the important steps. The details have been added to Appendix A.9.6 of the revised manuscript. For equation references, we use “revised” to indicate equation numbers in the revised manuscript and “original” to indicate those in the original submission.
Let p:=d−2K, ΔY1(t):=Y1(t+1)−Y1(t), and ΔY1,j(t):=Y1,j(t+1)−Y1,j(t). Squaring equation (17, revised; 16, original) and summing over the update components outside the leading low-dimensional subspace gives
{\lVert\Delta Y_1(t)\rVert_F^2} \le \mathcal{D}(t):= \frac{3p\\,e(t)^2}{\lVert G_1(t)\rVert_F^2}.$$ By (18, revised), $e(0)=O(\rho)$. Therefore, with small initialization, $\mathcal D(0)=O(\rho^2)$. For $t>0$, we obtain the following bounds: $$e(t) \lesssim r_{\rho} + \left(\frac{c}{\sqrt{p}} + L_G c_1 \lVert G_1(0) \rVert_F\right)\eta t,$$ $$\mathcal{D}(t) \le \frac{12p e(t)^2}{\lVert G_1(0) \rVert_F^2}.$$ Here, $c$ and $c_1$ are constants, $L_G$ is defined in (84, revised), and $r_\rho=O(\rho)$ bounds the initial gradient approximation error, as established in Lemma A.6. Therefore, when the initialization scale $\rho$ and the learning rate $\eta$ are sufficiently small, $e(t)$ is small and thus $\mathcal D(t)$ is small. This implies that the updates to components outside the leading low-dimensional subspace are small during the early training phase. Thus, the parameter updates are predominantly concentrated in the leading low-dimensional subspace during the early training phase. We also ran additional simulations on Fashion-MNIST beyond the synthetic dataset and showed that updates to components outside the leading low-dimensional subspace are small (see Figure 19 in Appendix A.10 of the revised manuscript). **Response to Critical Weaknesses 2:** This coupling design is intended to handle the “interlacing” coupling induced by the Cayley structure. Recall that each hidden layer is given by $$h_k = \sqrt{2}(2C_k-I)\sigma\left(-2\sqrt{2}g_k C_k^\top Y_k^\top h_{k-1}\right).$$ By Assumption 3.3, $\sigma(0)=0$ and $a=\sigma'(0)>0$. With small initialization, $C_k=(I+S_k+Y_k^\top Y_k)^{-1}\approx I$. Therefore, each hidden layer can be approximated by $$h_k \approx \sqrt{2}I a\left(-2\sqrt{2}g_k IY_k^\top h_{k-1}\right) = -4ag_k Y_k^\top h_{k-1}.$$ Substituting the low-rank parameterization $Y_k=L_kM_kR_k^\top$, we see that each hidden layer acts approximately as $$h_k \approx -4ag_k R_k M_k^\top L_k^\top h_{k-1}.$$ Adjacent layers interact through $L_{k+1}^\top R_k$. Choosing $R_k=L_{k+1}=F_k$ with $F_k^\top F_k=I_r$ is one way to align the internal factors to avoid additional attenuation caused by internal subspace misalignment. During training, freezing these shared internal factors preserves this alignment while reducing the number of trainable parameters. We have added Figure 4 to illustrate the overall architecture and revised the initialization section to improve clarity. **Response to Critical Weaknesses 3:** We have clarified the notation and added more details. Please see our detailed response to the related questions.