acceptodds
Under review as a conference paper at ICLR 2027

Beyond Orthogonality: Rotation Parameterization Shapes Sustained Learning

Abstract

Continual learning requires networks to keep learning as tasks change, yet prolonged training can erode this ability. Spectral constraints such as orthogonality can support adaptation, but leave open how the constrained weights should be represented and optimized. Motivated by rotation constructions shared by classical and quantum learning models, we examine this question while maintaining hidden-weight orthogonality throughout training. Across 1,000-task permutation streams on MNIST, CIFAR-10, CIFAR-100, and Tiny ImageNet, increasing rotation repetition from one to nineteen improves mean continual accuracy by 3.58–14.42 percentage points and increases the benefit of training history relative to networks trained anew on each task. Most of this improvement is reached by eight repetitions and persists under learning-rate selection and matched initialization; further repetition yields diminishing returns, while comparisons at a fixed parameter count reveal a complementary benefit from broader coordinate interactions. Beyond the number and arrangement of rotations, constraining their optimization coordinates also changes adaptation within the same nominal representable family. More broadly, these findings show that favorable weight spectra alone do not determine sustained learning: the structure and optimization of the underlying parameterization remain consequential.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual learning, loss of plasticity, orthogonal parameterizations, structured rotations, optimization geometry

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dawyOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates the optimization of hidden layer weights matrices constrained to SO(n)SO(n) in continual learning settings. Numerous experiments and benchmarks are run to confirm that stratification in performance exists among parameterizations despite the methods all preserving orthogonality (and therefore singular value spectrum). In particular, increased butterfly iterations (structured product of Given rotations) is shown to consistently improve both absolute and relative performance in the continual training setting. Similarly, constrained representations in the exponential space of rotations mostly perform better over their unconstrained counterpart, subject to learning rate.

Critical strengths

The experiments conducted are quite extensive. They are documented well and control for several confounding variables. The results are broadly consistent with the central narrative that orthogonal parameterizations affect continual adaptation.

Critical weaknesses

The empirical results are unsurprising and seemingly yield little insight. Most of the main paper body focuses primarily on butterfly iterations, but section 5 avoids promoting it as a superior method, leaving the paper without a clear takeaway. Although appendix E presents extensive general baselines for plasticity, the paper fails to compare its rotation parameterization against many other known parameterizations of SO(n)SO(n) in learning aside from exponential map (e.g. Cayley transform, projection to manifold, etc.).

Many of the experiments are ablations of low-level parameterization/optimization choices (e.g. learning rate, iteration count) with limited comparisons to materially different architecture alternatives. The results of the two best performing design features (increased butterfly iterations and bounded exponential coordinates) do not reveal an effect beyond the well-known benefits of greater model capacity and constrained optimization geometry in structured learning.

Questions for authors
  1. Are there additional contributions from the paper outside of the observation that changes in orthogonal parameterization affect optimization? E.g. explanatory mechanisms, recommended methods/settings for training, etc.
  2. Section 2.2 defines Bθ\mathcal{B}_{\theta}, but it is never mentioned in the analysis. Does this matrix (or any of its quantities like rank) relate to model performance?
  3. How do butterfly iterations scale with hidden layer size? Presumably 19 iterations was chosen as it is the first variant whose parameter count exceeds the dimensionality of SO(128)SO(128), so would same math be applied to larger SO(n)SO(n)? This may be a practical limitation of method depending on its runtime.
4 comments

We thank the reviewer for the precise objections. Terms (R$k$, reach, full reach, BθB_\theta, Polar, Cayley, block-Givens, layer names) are defined in our general response.

The paper's claim. Orthogonality alone does not determine whether a layer keeps learning; the parameterization does, through its reach and the conditioning of BθB_\theta. We do not propose a new method and do not claim that rotation products are a better orthogonal layer; the contribution is explanatory. Butterfly was chosen as the instrument because its repetition count is a direct dial on reach while orthogonality stays exact.

Capacity does not explain the results; reach does (CIFAR-100, Adam 10−310^{-3}, 10 seeds, continual accuracy):

modelanglesrank(BθB_\theta)accuracy
block-Givens b=8b{=}8, 133 rounds8,51244812.61%
Butterfly R41,7921,79211.84%
Butterfly R83,5843,58414.15%
Butterfly R198,5128,12816.94%

At equal parameter count, reach 8,128 beats reach 448 by +4.34+4.34 pp in 10/10 seeds; with 2.4×2.4\times fewer parameters and 8×8\times the reach, R8 beats the 133-round block model. Holding reach fixed instead (rank 448 for every b=8b{=}8 model with ≥14\ge 14 rounds) and initializing all depths to the same weight matrix, deeper products still learn better: 9.74% / 10.95% / 11.87% / 12.61% at 14 / 28 / 56 / 133 rounds (+2.86+2.86 pp, 10/10 seeds), while the condition number of BθB_\theta at initialization falls from 214 to 1.9. The best of three learning rates for the shallow model (11.24%) does not close the gap. "Constrained optimization geometry" names this effect; the fixed-reach control shows it is separate from capacity.

BθB_\theta is now used. Rank equals the angle count up to full reach. Along training (R19, 4 seeds), effective rank goes 639 → 4,156 → 5,145 and condition number 1.5×1061.5{\times}10^6 → 212 → 89 at tasks 0 / 200 / 1,000: full reach is brought into use within ~200 tasks. The conditioning claim is scoped to the fixed-reach ladder; across families the initial condition number does not rank performance (R19 10610^6: 16.94%; R32 7×1047{\times}10^4: 17.95%; b=128b{=}128 520: 17.26%).

Cayley and projection baselines. MLP (10 seeds, Adam 10−310^{-3}), CIFAR-10 / CIFAR-100: dense 34.1% / 11.2%, Cayley 37.0% / 14.3%, Polar 37.6% / 14.2%, exponential 38.4% / 14.9%, Butterfly R19 40.4% / 16.9%. Over five learning rates, R19's history gap is always positive; Cayley's turns negative at 5×10−35{\times}10^{-3} and 10−210^{-2} (CIFAR-10) and 10−210^{-2} (CIFAR-100), Polar's at 10−210^{-2} on CIFAR-10. CNN (Dohare et al. 2024 Continual ImageNet, protocol unchanged, 5 seeds, tasks 951–1,000): dense 82.42%; conv2/conv3/fc1/fc2 orthogonal at R1 81.35%, R8 85.53%, full reach 87.13%, Polar 87.92%, Cayley 87.82%; fc2 only at R1/R8/R19 84.05% / 86.99% / 87.71%. Shrink-and-Perturb (Dohare et al.'s setting) 88.76% beats every orthogonal arm by ~1 pp at 6×6\times lower cost than R8.

Scaling and runtime. Full reach for an n×nn{\times}n layer needs R⋆=⌈(n−1)/log⁡2n⌉R^\star=\lceil (n-1)/\log_2 n\rceil: 19, 37, 73 at n=128,256,512n=128, 256, 512, i.e. O(n2)O(n^2) parameters like Cayley but O(n)O(n) sequential stages; a fixed RR costs O(nlog⁡n)O(n\log n) parameters. Width 256 (CIFAR-100, 4 seeds): dense 12.90%, R1 10.43%, R8 16.49%, R32 20.59%, same ordering as at 128. Seconds per task on one H100 (CNN): dense 6 s, Polar 8 s, conv R8 43 s, Cayley 70 s, full-reach conv ~75 s; trainable parameters 177k dense/Polar, 696k Cayley, 58k conv R8, 152k full-reach conv.

Recommendations. Three actionable points.

  1. Orthogonality is not enough; reach is the prerequisite. An orthogonal layer with too few free directions is worse than a dense one (conv R1: −1.1-1.1 pp below dense). Orthogonality constrains the weights; the parameterization decides whether the layer can keep moving within it.
  2. Budget reach; do not default to full reach. The gain rises steeply and then flattens: R8 delivers 86% of the full-reach gain in fc2 and 66% in the conv set with 38% of the trainable parameters. Any full-reach parameterization costs O(n2)O(n^2) parameters per layer, a fixed repetition count O(nlog⁡n)O(n\log n), so the saving grows with width. For large models, choose the reach that captures the gain, whether with rotation products or any other efficient orthogonal parameterization.
  3. BθB_\theta at initialization is an early indicator of capability.

Scope. The mechanism is an association; a test we have not run is a low-rank skew matrix in Cayley or exp as an alternative reach dial.

Does the fixed-reach control address the concern of the reviewer?

Thank you for your response and updated results. The claim is now more narrow and clear, and I believe discussing the rank and conditioning of BθB_{\theta} will lead to a much better framing of the results and analysis. The scaling and runtime results are also informative.

However, I believe my two primary concerns still stand:

  1. The central claim of the paper yields little new insight. Improved performance from increased effective dimensionality (rank of BθB_{\theta}) is an expected result, and there is insufficient evidence for a conclusion on the effect of conditioning.

Recommendation: The tradeoff analysis between parameter count, runtime, and learning capability through "dialing" up or down the representation ability is a more interesting direction, but promoting budget reach as the recommended design variable requires more experimental evidence and comparison with other rank-tunable alternatives.

  1. Although new experimental results have been presented in the responses, the bulk of the existing paper's experiments and analysis is insufficient in comparing meaningful design choices.

Recommendation: Focus the analysis on representation differences (beyond dimensionality), performance comparison on different public benchmarks, and scaling behaviors.

For both points, applying the recommendations would constitute a significant rewrite of the existing paper, so I am therefore inclined to maintain my score.

We thank the reviewer for the follow-up. Our previous reply named one test we had not run: a low-rank skew matrix in Cayley as a second reach dial. We have now run it, together with a direct intervention on the conditioning of BθB_\theta. Terms are as in our general response.

1. A second rank-tunable family: low-rank Cayley. W=R0 (I+S)−1(I−S)W=R_0\,(I+S)^{-1}(I-S) with S=UV⊤−VU⊤S=UV^\top-VU^\top and U,V∈Rn×rU,V\in\mathbb{R}^{n\times r}, so the rank rr dials the reach, 2rn−r(2r+1)2rn-r(2r+1). Every rank starts from the initial weight matrix of the full Cayley arm. CIFAR-100 MLP, Adam 10−310^{-3}, 10 seeds, continual accuracy:

Reach (low-rank Cayley / Butterfly)502 / 448988 / 8961,912 / 1,7923,568 / 3,5847,182 / 7,1688,128 / 8,128
Low-rank Cayley, r=2,4,8,16,42,64r=2,4,8,16,42,648.17%9.08%10.30%11.90%14.94%15.27%
Butterfly R1, R2, R4, R8, R16, R198.74%10.20%11.84%14.15%16.52%16.94%

The second family reproduces both parts of the paper's claim, that sustained learning is determined by the parameterization through its reach and the conditioning of BθB_\theta, not by orthogonality alone.

(i) Reach. Accuracy rises monotonically with reach in both families, and at about 44% of full reach each has recovered more than half of its gain from lowest to full reach (53% and 66%).

(ii) The parameterization at a given reach. At matched reach the two families differ in 10/10 seeds at every level (0.58 to 2.25 pp). The full-reach arms already in the paper show the same at reach 8,128 (exponential 14.94%, R19 16.94%, block-Givens b=128b{=}128 17.26%), as do Cayley (14.33%) and low-rank Cayley (15.27%). Reach alone predicts none of these differences, nor that an orthogonal layer of low reach is worse than a dense one (conv R1: −1.1-1.1 pp).

2. The conditioning of BθB_\theta, tested by intervention. Our earlier evidence was an association; we now set the condition number ourselves and hold it for the whole run.

What is changed. Only the coordinates in which the optimizer takes its steps. The fixed-reach block-Givens layer (reach 448) and the loss are untouched. We write the angles as θ=θ0+Mϕ\theta=\theta_0+M\phi and train ϕ\phi, so the optimizer sees the Jacobian Bϕ=BθMB_\phi=B_\theta M in place of BθB_\theta.

How MM is built. From the SVD Bθ=U diag(s) V⊤B_\theta=U\,\mathrm{diag}(s)\,V^\top at the current angles we set M=I+V diag(d−1) V⊤M=I+V\,\mathrm{diag}(d-1)\,V^\top, which multiplies each singular value sis_i by did_i and leaves the singular vectors unchanged. We choose sidi=g (si/g)ps_id_i=g\,(s_i/g)^p, with gg the geometric mean of ss and p=ln⁡κ/ln⁡(smax⁡/smin⁡)p=\ln\kappa/\ln(s_{\max}/s_{\min}). BϕB_\phi then has condition number exactly κ\kappa and the geometric mean gg of BθB_\theta.

How it is held. After every optimizer step we set θ0←θ\theta_0\leftarrow\theta, ϕ←0\phi\leftarrow 0 and recompute the SVD and MM at the new angles (BθB_\theta is block-diagonal, so this is 16 eigenproblems of size 28 per layer). κ\kappa therefore holds at every step, not only at initialization; checkpoints (4 seeds) confirm it. Reach, depth, parameter count and initial weights equal those of the unmodified model.

Protocol. SGD with momentum 0.9, because Adam rescales every coordinate and so alters the conditioning the optimizer sees. The step size is chosen per depth on a 200-task search and kept for the held arms. 10 seeds:

Model (reach 448)Condition number of BϕB_\phiContinual accuracyvs unmodifiedSeeds lower
133 roundsunmodified (settles near 1.6)11.32%––
133 roundsheld at 910.50%−0.82-0.82 pp10/10
133 roundsheld at 1406.45%−4.87-4.87 pp10/10
14 roundsunmodified (settles between 7 and 9)11.13%––
14 roundsheld at 14010.72%−0.41-0.41 pp10/10

The held values are those of the 14-round model during training (9) and at initialization (about 140, geometric mean over seeds). Worse conditioning lowers sustained learning in every seed, in both models, and more so the worse it is. Within a model only the spectrum seen by the optimizer differs, so under this protocol poor conditioning causes a loss in sustained learning at fixed reach.

3. Public benchmark and scaling. On Continual ImageNet (Dohare et al., 2024, protocol unchanged) the reach ordering holds in both layer sets (R1 < R8 < full reach), and R8 recovers 86% and 66% of the full-reach gain. Width 256 reproduces the ordering (4 seeds: R1 10.43%, R8 16.49%, R32 20.59%).

Revised draft. The claim and the existing experiments stay. All new results, from this reply and from our earlier replies, will be included in the revised draft, in the main paper and the appendix. On reach as the design variable, the requested comparison now exists: two unrelated rank-tunable families agree that most of the gain arrives before full reach.

Do the second reach dial and the conditioning intervention address the two concerns?

We have uploaded the revised paper, which contains the results from our replies:

  • Reach and conditioning of the parameterization, including the fixed-reach ladder, the intervention on the condition number, and the low-rank Cayley, Cayley and Polar comparisons: Section 4.4 and Appendix F.
  • Continual ImageNet: Section 5.1 and Appendix G.
  • Width 256, trainable parameters and runtime: Appendix H.

Two values differ from our earlier reply. Full reach needs 19, 32 and 57 repetitions at widths 128, 256 and 512, not 19, 37 and 73 as we wrote. Runtimes are now measured on a single RTX 4090 and replace the H100 values; the ordering of the models is the same.

We hope this addresses your concerns and are happy to answer any further questions.

Reviewer_CBRdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study how applying repeated rotations enables continual learning in two-hidden-layer ReLU networks. They show that applying repeated butterfly transformations enables these networks to learn continuously across tasks constructing by permuting pixels, and that this holds for several different datasets. With enough repeated butterfly transformations, the networks exhibit continued positive transfer across tasks, and that this holds even when the number of angles are fixed using Block-Givens. They tie this to maintaining higher rank on the NTK and Jacobians.

Critical strengths
  • Large number of controls and follow-up studies: The paper features a large number of carefully-conducted experiments.
Critical weaknesses
  • Small networks and constrained problem settings: The networks used are two-layer ReLU, and the tasks are all constructed using pixel permutations of established datasets. It would be interesting to see if similar results transfer to more natural continual learning settings like Continual ImageNet (Dohare et al), which also feature thousands of tasks.

  • Description of the training was unclear: Even reading the appendix, I had trouble following details of the implementation: The notation R1-R19, for instance, is used initially without explanation. It would be nice to add a clear description of how the repetitions affects the construction of the matrices and of how the network is trained in Section 2.2.

Questions for authors

Both clarification and future work: Would you expect these training methods to work on other kinds of tasks?

3 comments

We thank the reviewer; we followed both suggestions. Terms (R$k$, reach, full reach, BθB_\theta, Polar, Cayley, block-Givens, layer names) are defined in our general response.

Continual ImageNet (Dohare et al., 2024). We ran their CNN and protocol unchanged: three convolutional layers (conv1–3, filters 5/3/35/3/3, channels 32/64/12832/64/128) and two fully connected layers (fc1: 512 ⁣→ ⁣128512\!\to\!128, fc2: 128 ⁣→ ⁣128128\!\to\!128) before a two-way output head; SGD momentum 0.9, lr 0.01, 250 epochs per task, 1,000 binary tasks, each a new pair of ImageNet classes, head re-zeroed per task, 5 seeds. We replace either fc2 alone, or conv2, conv3, fc1 and fc2 together, by orthogonal layers (conv1 and the head stay dense); all arms share seeds and task order. Mean test accuracy over tasks 951–1,000 (our rotation arms in bold):

Orthogonal layersR1R8full reachPolarCayley
fc2 only84.05%86.99%87.71% (R19)––
conv2, conv3, fc1, fc281.35%85.53%87.13%87.92%87.82%
same, with 4×44{\times}4 filters79.80%83.35%85.10%––

Dense baselines: 82.42% (5/3/35/3/3) and 81.99% (4/4/44/4/4); both peak near 87% around task 120 and then decline, reproducing the loss of plasticity of Dohare et al. Shrink-and-Perturb with their hyperparameters reaches 88.76% and remains the strongest. Relative to training from scratch on each task, dense ends 5.8 pp below and full-reach orthogonal layers 0.9–1.8 pp below.

The ordering from the paper transfers to natural tasks and a CNN: few rotations give no benefit or a loss relative to dense; the benefit rises steeply with the rotation budget and flattens before full reach (R8 gives 86% of the full-reach gain in fc2 with 42% of the angles, and 66% in the conv set with 38% of the trainable parameters; in the 4×44{\times}4 network full reach adds 1.8 pp over R8). Full-reach rotations (87.13%), Polar (87.92%) and Cayley (87.82%) end within 0.8 pp of each other, consistent with the paper's claim that the parameterization's reach, not orthogonality itself, is what matters. As in the paper, full reach is not always required, and stopping short of it is where the parameter saving lies: 58k trainable parameters at R8 versus 152k at full reach (177k dense, 696k Cayley), a saving that grows with width (O(nlog⁡n)O(n\log n) per layer for a fixed repetition count against O(n2)O(n^2) for any full-reach parameterization). We expect this to matter most at large scale, with rotation products or other efficient orthogonal parameterizations. Butterfly was chosen because its repetition count dials reach directly; we are not proposing it as a new or better orthogonal method.

Section 2.2 and the notation. We will rewrite Section 2.2 as follows. A Butterfly block on nn coordinates consists of log⁡2n\log_2 n stages; stage ss pairs the coordinates whose indices differ in bit ss and applies an independent plane rotation to each pair (n/2n/2 angles per stage). R$k$ denotes kk repetitions of the block with independent angles: klog⁡2nk\log_2 n stages and k (n/2)log⁡2nk\,(n/2)\log_2 n angles (7k7k stages, 448k448k angles at n=128n=128). The hidden weight is Wℓ=Q0 G(θℓ)W_\ell = Q_0\,G(\theta_\ell), with Q0Q_0 a fixed orthogonal matrix and GG the product of the stages, so Wℓ∈SO(n)W_\ell\in SO(n) for every θ\theta. Training is ordinary Adam (MLP) or SGD (CNN) on θ\theta and the biases; nothing is projected. Full reach is the smallest kk whose angle count reaches the dimension of the set of matrices the layer can express, nm−m(m+1)/2nm-m(m+1)/2 for an n×mn\times m layer with orthonormal columns: k=19k=19 for 128×128128\times128 (dim⁡SO(128)=8,128\dim SO(128)=8{,}128) and 13/23/25/1913/23/25/19 for conv2/conv3/fc1/fc2 of the CNN above (conv2 and conv3 have fan-ins 288 and 576, not powers of two, so their stages use fixed random matchings, as in Section 3.2). We will add this table, a stage diagram and the training loop in pseudocode.

Other kinds of tasks. Continual ImageNet is class-incremental with a reset binary head, so it tests input distribution shift and new labels rather than input permutation, and the results hold there. We expect the reach effect wherever a layer must keep changing its rotation across tasks, which is what low reach prevents; regression, RL and sequence models remain open and we will say so. Appendix E already has a cumulative class-incremental CIFAR-100 setting and a random-label relearning setting, which we will cite more clearly.

We will add the Continual ImageNet experiment, the per-layer full-reach table, a runtime/parameter table (seconds per task on one H100: dense 6 s, Polar 8 s, conv R8 43 s, Cayley 70 s, full-reach conv ~75 s) and the rewritten Section 2.2 to the revision.

Does this address the two weaknesses?

I thank the authors for their responses to my questions. The extension to Continual ImageNet is useful, and I thank the authors for pointing me to the experiments that were already done in the appendix. However, despite the additional clarifications, I still have some trouble following the methods. This may be due to my lack of knowledge, but for me, a bit more intuition and background into the given terms would be useful. I will therefore maintain my score for now, awaiting discussions with the other reviewers and AC.

Thank you for the reply, and for telling us where the paper remains hard to follow. We have uploaded a revised version in which we tried to make the method readable without prior knowledge of rotation parameterizations. The intuition in brief:

  • Rotation parameterization. We do not train the entries of a hidden weight matrix. The matrix is built as a product of many two-dimensional rotations, each mixing two coordinates by one trainable angle. Any such product is orthogonal, so training the angles keeps the weights orthogonal without a projection step.
  • Butterfly and repetitions (R1 … R19). One Butterfly block is a fixed pattern of such rotations in which every coordinate is paired with a different partner at each stage. R is the number of times the block is repeated with new angles; a larger R means more angles.
  • Reach. The number of independent directions in which the weight matrix can move when the angles change. With few angles the matrix can move in only a few directions; at width 128, nineteen repetitions make all 8,128 directions of a rotation available, which we call full reach.
  • Conditioning. How unevenly these directions respond to a step in the angles.

In these terms, the result is that two networks with equally orthogonal weights can differ strongly in sustained learning, depending on how many directions their parameterization can reach and how evenly it reaches them.

In the revised paper, Section 2.2 now builds the Butterfly layer step by step and defines reach and full reach; Appendix A.2 adds a drawing of the layer (Figure 7) and the training loop (Algorithm 1), and Table 1 lists full reach for every layer we use. The Continual ImageNet experiment is in Section 5.1 and Appendix G, and Section 5 now points directly to the appendix experiments you asked about. Runtimes (Appendix H) are now measured on a single RTX 4090 and replace the H100 values of our earlier reply; the ordering of the models is the same.

If a particular term or step is still unclear, we would be glad to explain it here.

Reviewer_cALPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies loss of plasticity, where a network trained on many tasks in sequence gradually gets worse at learning new ones, in networks whose hidden weight matrices are constrained to be exactly orthogonal. The model is an MLP with two orthogonal hidden layers, each built as a product of two-dimensional rotations with trainable angles. Since all orthogonal variants share the same weight spectra, the question is whether the parameterization itself affects plasticity. The authors vary the number of rotations, their pairing pattern, and how broadly they couple coordinates, and also compare exponential-map parameterizations with and without a norm constraint. On 1,000-task pixel-permutation streams of four image datasets, they compare a continual learner with a fresh reference reinitialized on every task. Increasing the number of rotations turns the accuracy gap between the two from negative or near zero to clearly positive, with diminishing returns, and this persists under several ablations. The authors conclude that how the orthogonal matrices are built and trained matters for plasticity, but do not identify why.

Critical strengths
  1. Loss of plasticity is an active research topic in continual learning, and prior work links it to the spectral properties of the weights. Fixing the weight spectrum exactly, rather than regularizing toward orthogonality, cleanly isolates the question of whether the parameterization of the weights matters beyond their spectrum.
  2. The experimental methodology appears sound, and the ablations target the natural alternative explanations for the findings. Results cover long task streams on four datasets with multiple seeds, which makes the core observations convincing within the scope of the experiments.
  3. The authors are upfront about the limits of their results. They report absolute accuracies alongside gaps, include unfavorable results, and avoid overclaiming.
Critical weaknesses

My main concerns are the limited scope of the experiments and the limited connection to closely related literature.

  1. The models are very small, with at most a few tens of thousands of trained parameters, and their absolute accuracies are low (e.g., about 17% on CIFAR-100 and 7% on Tiny ImageNet). It is unclear whether the observed effects carry over to the architectures and settings where loss of plasticity is typically studied. Even a brief experiment with a larger architecture, such as a CNN, would help a lot.

  2. The paper engages only briefly with the literature on overparameterization. The benefits of overparameterization for optimization are well studied, including for parameterized quantum circuits, which motivate this work. Most closely related are deep linear networks. A product of rotations is a deep linear factorization of a single matrix with constrained factors, and adding rotations is analogous to adding depth. Work on deep linear networks studies how such factorizations affect optimization and generalization, e.g., through the implicit acceleration and implicit bias induced by depth, yet the paper cites it only to motivate orthogonal initialization. Engaging with this line of work could also suggest a mechanism for the observed effects, which the paper currently leaves open.

Questions for authors
  1. Adding rotations both enlarges the set of reachable matrices and changes the optimization geometry. Do the authors attribute their results mainly to the added expressiveness or to improved optimization? Additional evidence separating the two would clarify what drives the observed effects.
  2. What are the practical takeaways of this work? Can the proposed parameterizations be used to mitigate loss of plasticity in practical settings?
3 comments

We thank the reviewer for the two questions. Terms (R$k$, reach, full reach, BθB_\theta, Polar, Cayley, block-Givens, layer names) are defined in our general response.

New experiment: Continual ImageNet CNN. We ran the network and protocol of Dohare et al. (2024) unchanged (1,000 binary tasks with new class pairs, 250 epochs per task, 5 seeds). We replace either fc2 alone (128×128128{\times}128) or conv2, conv3, fc1 and fc2 ("conv2–fc2") by orthogonal layers; conv1 and the head stay dense. All arms share seeds and task order. Tasks 951–1,000, paired difference to dense on the same seeds, trainable parameters (ours in bold):

MethodOrthogonal layersTasks 951–1,000vs denseParams
Dense (Dohare et al.)–82.42%–177k
Shrink-and-Perturb–88.76%+6.33+6.33 pp177k
Rotations R1fc284.05%+1.62+1.62 pp161k
Rotations R8fc286.99%+4.56+4.56 pp164k
Rotations R19 (full reach)fc287.71%+5.29+5.29 pp169k
Rotations R1conv2–fc281.35%−1.07-1.07 pp10k
Rotations R8conv2–fc285.53%+3.10+3.10 pp58k
Rotations, full reachconv2–fc287.13%+4.70+4.70 pp152k
Polarconv2–fc287.92%+5.49+5.49 pp177k
Cayleyconv2–fc287.82%+5.40+5.40 pp696k

Dense peaks at 87.0% near task 120 and then declines; relative to training from scratch on each task it ends 5.8 pp below, full-reach orthogonal layers 0.9–1.8 pp below. The paper's findings carry over: a few rotations do not help (conv set) or help little (fc2); the benefit rises steeply with the rotation budget and then flattens.

Q1: expressiveness or optimization? Both; the new experiments separate them.

Expressiveness is reach. Adding repetitions enlarges the expressible set, and accuracy follows: R1 < R8 < full reach in both layer sets above. Reach is not parameter count: on CIFAR-100 (MLP, 10 seeds, Adam 10−310^{-3}), block-Givens with 133 rounds has 8,512 angles but reach 448 and gets 12.61%; Butterfly R8 has 3,584 angles and reach 3,584, 14.15%; Butterfly R19 has the same 8,512 angles but reach 8,128, 16.94%, +4.34+4.34 pp over the block model in 10/10 seeds.

Optimization is conditioning. To test whether the parameterization matters beyond what it expresses, we hold reach fixed using block-Givens layers: after 14 rounds their reachable set stops growing (rank(BθB_\theta) stays at 448), so further rounds change only how that fixed set is parameterized. Every depth is initialized to the same weight matrix, so the models differ only in parameterization. Accuracy nevertheless rises with depth: 9.74% / 10.95% / 11.87% / 12.61% at 14 / 28 / 56 / 133 rounds (+2.86+2.86 pp, positive in 10/10 seeds), and the shallowest model does not catch up at its best learning rate (11.24%). What changes alongside is the conditioning of BθB_\theta: its condition number at initialization falls from 214 to 1.9 across the ladder, so a step in angle space moves the weights more evenly across the available directions. We call this an association, not a mechanism: across families the initial condition number does not rank performance (R19: 10610^6, 16.94%; R32: 7×1047{\times}10^4, 17.95%; b=128b{=}128: 520, 17.26%). Along training, R19's BθB_\theta goes from effective rank 639 and condition number ∼ ⁣106\sim\!10^6 to 4,156 and 212 by task 200: the full reach is brought into use early.

Deep linear networks. We agree and will add a paragraph: the fixed-reach ladder is overparameterization at a fixed function class and initial function, the setting of implicit acceleration (Arora et al., 2018), and our BθB_\theta measurements show the accompanying change in conditioning; we will present it as the natural reading, not a mechanism.

Q2: practical takeaways. (1) Orthogonality alone is not enough: with few rotations, orthogonal conv layers are worse than dense (−1.1-1.1 pp). (2) Full reach is not always required, and this is where the practical value lies: a moderate budget captures most of the gain with a fraction of the parameters (R8: 86% of the full-reach gain in fc2 with 42% of the angles, 66% in the conv set with 38% of the parameters). Any full-reach parameterization costs O(n2)O(n^2) parameters per layer, a fixed repetition count O(nlog⁡n)O(n\log n), so the saving grows with width; at scale, reach is the quantity to budget, with rotation products or any other efficient orthogonal parameterization. (3) Among full-reach methods the trade-off is runtime versus parameters (per task on one H100: dense 6 s, Polar 8 s, conv R8 43 s, Cayley 70 s, full-reach conv ~75 s); rotation products are exactly orthogonal without projection or solve, at the cost of depth. (4) BθB_\theta at initialization is an early indicator of capability.

Does this address the concerns on scope and mechanism?

Thank you for the new experiments. The CNN results address my concern about scope, and the fixed-reach ladder answers my question about expressiveness versus optimization. Assuming these results are incorporated in the revised paper, I will raise my score.

We thank the reviewer for the follow-up and are glad the new experiments address both concerns. We confirm that the following will be included in the revised draft, in the main paper and the appendix:

  1. Continual ImageNet CNN (Dohare et al., 2024): the full table from our reply (dense and Shrink-and-Perturb baselines; R1, R8 and full reach in both layer sets; Polar and Cayley), the learning curves over the 1,000 tasks, and the 4×44{\times}4-filter variant.
  2. Fixed-reach ladder (block-Givens, 14 / 28 / 56 / 133 rounds at rank(BθB_\theta) = 448, same initial weight matrix) with the BθB_\theta figures: rank and condition number at initialization and along training.
  3. Conditioning intervention (new; details in our reply to Reviewer dawy): at fixed reach, we hold the condition number seen by the optimizer at a prescribed value along training. Worse conditioning lowers sustained learning in every seed (−0.82-0.82 pp at 9 and −4.87-4.87 pp at 140 in the 133-round model), so poor conditioning of BθB_\theta is a cause of the loss, not only an association.
  4. Second reach dial (new; details in our reply to Reviewer dawy): low-rank Cayley, where the rank of the skew matrix sets the reach, reproduces the reach ordering in a second family (8.17% at reach 502 to 15.27% at full reach).
  5. Expressiveness versus optimization (Q1): the equal-parameter comparison of reach 448 against reach 8,128, and the fixed-reach ladder, as the two controls that separate them.
  6. Deep linear networks: a related-work paragraph relating the ladder to overparameterization at a fixed function class (Arora et al., 2018).
  7. Practical takeaways (Q2): a recommendations and limitations section, with a table of trainable parameters and seconds per task for every arm.

We will also rewrite Section 2.2 and add the per-layer full-reach table.

Comments on the paper

1 comment

We thank all three reviewers. The shared concerns were: small MLPs on permuted-input tasks (cALP, CBRd); no Cayley or projection baselines (dawy); results reducible to capacity plus constrained optimization, with no mechanism or takeaway (dawy, cALP); notation (CBRd). Two new experiments test the paper's claim in the requested settings.

Terms used throughout. Dense: an ordinary unconstrained layer. R$k$: a rotation layer built from kk repetitions of the Butterfly block; each repetition adds (n/2)log⁡2n(n/2)\log_2 n trainable rotation angles and enlarges the set of orthogonal matrices the layer can express. Reach: the dimension of that set, measured as rank(BθB_\theta) with Bθ=∂ vecW/∂θB_\theta=\partial\,\mathrm{vec}W/\partial\theta (angle changes → weight changes); full reach: the smallest kk whose angle count attains it (k=19k{=}19 at n=128n{=}128). Polar: unconstrained weights projected back to orthonormal columns after every step. Cayley: W=Q0(I+S)−1(I−S)W=Q_0(I+S)^{-1}(I-S) with trainable skew-symmetric SS. Block-Givens: rotations confined to 8×88{\times}8 blocks (see Experiment B). In the CNN, conv1–3 are the convolutional and fc1–2 the fully connected layers before the two-way output head.

The paper's claim, restated. Orthogonality alone does not determine sustained learning; the parameterization of the orthogonal weights does, through its reach and the conditioning of BθB_\theta. Most of the benefit arrives well before full reach (the paper's eight-repetition result), so full reach is not always required. Butterfly rotation products are our instrument, not our proposal: we chose them because the repetition count is a simple dial on reach while the layer stays exactly orthogonal; we are not proposing them as a new or better orthogonal method.

Experiment A: Continual ImageNet CNN (Dohare et al., 2024; network and protocol unchanged; 1,000 binary tasks on new class pairs, 250 epochs each; 5 seeds shared across arms). We replace fc2 alone or conv2, conv3, fc1, fc2 by orthogonal layers; test accuracy over tasks 951–1,000 (our arms in bold):

Orthogonal layersR1R8full reachPolarCayley
fc2 only84.05%86.99%87.71% (R19)––
conv2, conv3, fc1, fc281.35%85.53%87.13%87.92%87.82%

Dense: 82.42% (peaks at 87.0% near task 120, then declines, as in Dohare et al.). Shrink-and-Perturb (Dohare et al.'s setting): 88.76%. Relative to training from scratch on each task, dense ends 5.8 pp below, full-reach orthogonal layers 0.9–1.8 pp below. Few rotations are no better than dense (conv set: −1.1-1.1 pp); the benefit rises steeply with the rotation budget and then flattens; all full-reach parameterizations end within 0.8 pp of each other.

Experiment B: does conditioning matter once reach is held fixed? (MLP, CIFAR-100, Adam 10−310^{-3}, 10 seeds.) In a Butterfly layer every added repetition both enlarges the reachable set and deepens the product. To separate the two effects we use block-Givens layers, whose rotations act only inside fixed 8×88{\times}8 blocks. Their reachable set stops growing at 14 rounds (rank(BθB_\theta) stays at 448 from then on), so further rounds change only how that fixed set is parameterized. We also initialize every depth to the same weight matrix. Any difference between depths is then due to the parameterization alone. Continual accuracy still rises with depth: 9.74% / 10.95% / 11.87% / 12.61% at 14 / 28 / 56 / 133 rounds (+2.86+2.86 pp, positive in 10/10 seeds), and the shallowest model does not catch up at its best learning rate (11.24%). What changes alongside is the conditioning of BθB_\theta: its condition number at initialization falls from 214 to 1.9 across the ladder, so steps in angle space translate more evenly into weight changes. Reach, in turn, matters independently of parameter count: at the same 8,512 angles, Butterfly R19 (reach 8,128) beats the 133-round block model (reach 448) by +4.34+4.34 pp in 10/10 seeds. We present conditioning as an association within this fixed-reach family, not a demonstrated mechanism.

Full reach is not always required, and that is where the parameter saving lies. R8 recovers 86% of the full-reach gain in fc2 with 42% of the angles and 66% in the conv set with 38% of the trainable parameters (58k vs 152k; dense 177k, Cayley 696k). Any full-reach parameterization costs O(n2)O(n^2) parameters per layer; a fixed repetition count costs O(nlog⁡n)O(n\log n), so the saving grows with width. We think this is the result most useful at scale: reach is the quantity to budget, with rotation products or any other efficient orthogonal parameterization.

What we do not claim. Rotation products are not the best plasticity method here: Shrink-and-Perturb is ~1 pp better at dense cost; our orthogonal arms were untuned.

Revision. We will add both experiments, the BθB_\theta figures, a runtime/parameter table, a rewritten Section 2.2 and related work on deep linear networks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.