- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper investigates the optimization of hidden layer weights matrices constrained to in continual learning settings. Numerous experiments and benchmarks are run to confirm that stratification in performance exists among parameterizations despite the methods all preserving orthogonality (and therefore singular value spectrum). In particular, increased butterfly iterations (structured product of Given rotations) is shown to consistently improve both absolute and relative performance in the continual training setting. Similarly, constrained representations in the exponential space of rotations mostly perform better over their unconstrained counterpart, subject to learning rate.
Critical strengths
The experiments conducted are quite extensive. They are documented well and control for several confounding variables. The results are broadly consistent with the central narrative that orthogonal parameterizations affect continual adaptation.
Critical weaknesses
The empirical results are unsurprising and seemingly yield little insight. Most of the main paper body focuses primarily on butterfly iterations, but section 5 avoids promoting it as a superior method, leaving the paper without a clear takeaway. Although appendix E presents extensive general baselines for plasticity, the paper fails to compare its rotation parameterization against many other known parameterizations of in learning aside from exponential map (e.g. Cayley transform, projection to manifold, etc.).
Many of the experiments are ablations of low-level parameterization/optimization choices (e.g. learning rate, iteration count) with limited comparisons to materially different architecture alternatives. The results of the two best performing design features (increased butterfly iterations and bounded exponential coordinates) do not reveal an effect beyond the well-known benefits of greater model capacity and constrained optimization geometry in structured learning.
Questions for authors
- Are there additional contributions from the paper outside of the observation that changes in orthogonal parameterization affect optimization? E.g. explanatory mechanisms, recommended methods/settings for training, etc.
- Section 2.2 defines , but it is never mentioned in the analysis. Does this matrix (or any of its quantities like rank) relate to model performance?
- How do butterfly iterations scale with hidden layer size? Presumably 19 iterations was chosen as it is the first variant whose parameter count exceeds the dimensionality of , so would same math be applied to larger ? This may be a practical limitation of method depending on its runtime.
4 comments
Thank you for your response and updated results. The claim is now more narrow and clear, and I believe discussing the rank and conditioning of will lead to a much better framing of the results and analysis. The scaling and runtime results are also informative.
However, I believe my two primary concerns still stand:
- The central claim of the paper yields little new insight. Improved performance from increased effective dimensionality (rank of ) is an expected result, and there is insufficient evidence for a conclusion on the effect of conditioning.
Recommendation: The tradeoff analysis between parameter count, runtime, and learning capability through "dialing" up or down the representation ability is a more interesting direction, but promoting budget reach as the recommended design variable requires more experimental evidence and comparison with other rank-tunable alternatives.
- Although new experimental results have been presented in the responses, the bulk of the existing paper's experiments and analysis is insufficient in comparing meaningful design choices.
Recommendation: Focus the analysis on representation differences (beyond dimensionality), performance comparison on different public benchmarks, and scaling behaviors.
For both points, applying the recommendations would constitute a significant rewrite of the existing paper, so I am therefore inclined to maintain my score.
We thank the reviewer for the follow-up. Our previous reply named one test we had not run: a low-rank skew matrix in Cayley as a second reach dial. We have now run it, together with a direct intervention on the conditioning of . Terms are as in our general response.
1. A second rank-tunable family: low-rank Cayley. with and , so the rank dials the reach, . Every rank starts from the initial weight matrix of the full Cayley arm. CIFAR-100 MLP, Adam , 10 seeds, continual accuracy:
| Reach (low-rank Cayley / Butterfly) | 502 / 448 | 988 / 896 | 1,912 / 1,792 | 3,568 / 3,584 | 7,182 / 7,168 | 8,128 / 8,128 |
|---|---|---|---|---|---|---|
| Low-rank Cayley, | 8.17% | 9.08% | 10.30% | 11.90% | 14.94% | 15.27% |
| Butterfly R1, R2, R4, R8, R16, R19 | 8.74% | 10.20% | 11.84% | 14.15% | 16.52% | 16.94% |
The second family reproduces both parts of the paper's claim, that sustained learning is determined by the parameterization through its reach and the conditioning of , not by orthogonality alone.
(i) Reach. Accuracy rises monotonically with reach in both families, and at about 44% of full reach each has recovered more than half of its gain from lowest to full reach (53% and 66%).
(ii) The parameterization at a given reach. At matched reach the two families differ in 10/10 seeds at every level (0.58 to 2.25 pp). The full-reach arms already in the paper show the same at reach 8,128 (exponential 14.94%, R19 16.94%, block-Givens 17.26%), as do Cayley (14.33%) and low-rank Cayley (15.27%). Reach alone predicts none of these differences, nor that an orthogonal layer of low reach is worse than a dense one (conv R1: pp).
2. The conditioning of , tested by intervention. Our earlier evidence was an association; we now set the condition number ourselves and hold it for the whole run.
What is changed. Only the coordinates in which the optimizer takes its steps. The fixed-reach block-Givens layer (reach 448) and the loss are untouched. We write the angles as and train , so the optimizer sees the Jacobian in place of .
How is built. From the SVD at the current angles we set , which multiplies each singular value by and leaves the singular vectors unchanged. We choose , with the geometric mean of and . then has condition number exactly and the geometric mean of .
How it is held. After every optimizer step we set , and recompute the SVD and at the new angles ( is block-diagonal, so this is 16 eigenproblems of size 28 per layer). therefore holds at every step, not only at initialization; checkpoints (4 seeds) confirm it. Reach, depth, parameter count and initial weights equal those of the unmodified model.
Protocol. SGD with momentum 0.9, because Adam rescales every coordinate and so alters the conditioning the optimizer sees. The step size is chosen per depth on a 200-task search and kept for the held arms. 10 seeds:
| Model (reach 448) | Condition number of | Continual accuracy | vs unmodified | Seeds lower |
|---|---|---|---|---|
| 133 rounds | unmodified (settles near 1.6) | 11.32% | – | – |
| 133 rounds | held at 9 | 10.50% | pp | 10/10 |
| 133 rounds | held at 140 | 6.45% | pp | 10/10 |
| 14 rounds | unmodified (settles between 7 and 9) | 11.13% | – | – |
| 14 rounds | held at 140 | 10.72% | pp | 10/10 |
The held values are those of the 14-round model during training (9) and at initialization (about 140, geometric mean over seeds). Worse conditioning lowers sustained learning in every seed, in both models, and more so the worse it is. Within a model only the spectrum seen by the optimizer differs, so under this protocol poor conditioning causes a loss in sustained learning at fixed reach.
3. Public benchmark and scaling. On Continual ImageNet (Dohare et al., 2024, protocol unchanged) the reach ordering holds in both layer sets (R1 < R8 < full reach), and R8 recovers 86% and 66% of the full-reach gain. Width 256 reproduces the ordering (4 seeds: R1 10.43%, R8 16.49%, R32 20.59%).
Revised draft. The claim and the existing experiments stay. All new results, from this reply and from our earlier replies, will be included in the revised draft, in the main paper and the appendix. On reach as the design variable, the requested comparison now exists: two unrelated rank-tunable families agree that most of the gain arrives before full reach.
Do the second reach dial and the conditioning intervention address the two concerns?
We have uploaded the revised paper, which contains the results from our replies:
- Reach and conditioning of the parameterization, including the fixed-reach ladder, the intervention on the condition number, and the low-rank Cayley, Cayley and Polar comparisons: Section 4.4 and Appendix F.
- Continual ImageNet: Section 5.1 and Appendix G.
- Width 256, trainable parameters and runtime: Appendix H.
Two values differ from our earlier reply. Full reach needs 19, 32 and 57 repetitions at widths 128, 256 and 512, not 19, 37 and 73 as we wrote. Runtimes are now measured on a single RTX 4090 and replace the H100 values; the ordering of the models is the same.
We hope this addresses your concerns and are happy to answer any further questions.
We thank the reviewer for the precise objections. Terms (R$k$, reach, full reach, Bθ, Polar, Cayley, block-Givens, layer names) are defined in our general response.
The paper's claim. Orthogonality alone does not determine whether a layer keeps learning; the parameterization does, through its reach and the conditioning of Bθ. We do not propose a new method and do not claim that rotation products are a better orthogonal layer; the contribution is explanatory. Butterfly was chosen as the instrument because its repetition count is a direct dial on reach while orthogonality stays exact.
Capacity does not explain the results; reach does (CIFAR-100, Adam 10−3, 10 seeds, continual accuracy):
At equal parameter count, reach 8,128 beats reach 448 by +4.34 pp in 10/10 seeds; with 2.4× fewer parameters and 8× the reach, R8 beats the 133-round block model. Holding reach fixed instead (rank 448 for every b=8 model with ≥14 rounds) and initializing all depths to the same weight matrix, deeper products still learn better: 9.74% / 10.95% / 11.87% / 12.61% at 14 / 28 / 56 / 133 rounds (+2.86 pp, 10/10 seeds), while the condition number of Bθ at initialization falls from 214 to 1.9. The best of three learning rates for the shallow model (11.24%) does not close the gap. "Constrained optimization geometry" names this effect; the fixed-reach control shows it is separate from capacity.
Bθ is now used. Rank equals the angle count up to full reach. Along training (R19, 4 seeds), effective rank goes 639 → 4,156 → 5,145 and condition number 1.5×106 → 212 → 89 at tasks 0 / 200 / 1,000: full reach is brought into use within ~200 tasks. The conditioning claim is scoped to the fixed-reach ladder; across families the initial condition number does not rank performance (R19 106: 16.94%; R32 7×104: 17.95%; b=128 520: 17.26%).
Cayley and projection baselines. MLP (10 seeds, Adam 10−3), CIFAR-10 / CIFAR-100: dense 34.1% / 11.2%, Cayley 37.0% / 14.3%, Polar 37.6% / 14.2%, exponential 38.4% / 14.9%, Butterfly R19 40.4% / 16.9%. Over five learning rates, R19's history gap is always positive; Cayley's turns negative at 5×10−3 and 10−2 (CIFAR-10) and 10−2 (CIFAR-100), Polar's at 10−2 on CIFAR-10. CNN (Dohare et al. 2024 Continual ImageNet, protocol unchanged, 5 seeds, tasks 951–1,000): dense 82.42%; conv2/conv3/fc1/fc2 orthogonal at R1 81.35%, R8 85.53%, full reach 87.13%, Polar 87.92%, Cayley 87.82%; fc2 only at R1/R8/R19 84.05% / 86.99% / 87.71%. Shrink-and-Perturb (Dohare et al.'s setting) 88.76% beats every orthogonal arm by ~1 pp at 6× lower cost than R8.
Scaling and runtime. Full reach for an n×n layer needs R⋆=⌈(n−1)/log2n⌉: 19, 37, 73 at n=128,256,512, i.e. O(n2) parameters like Cayley but O(n) sequential stages; a fixed R costs O(nlogn) parameters. Width 256 (CIFAR-100, 4 seeds): dense 12.90%, R1 10.43%, R8 16.49%, R32 20.59%, same ordering as at 128. Seconds per task on one H100 (CNN): dense 6 s, Polar 8 s, conv R8 43 s, Cayley 70 s, full-reach conv ~75 s; trainable parameters 177k dense/Polar, 696k Cayley, 58k conv R8, 152k full-reach conv.
Recommendations. Three actionable points.
Scope. The mechanism is an association; a test we have not run is a low-rank skew matrix in Cayley or exp as an alternative reach dial.
Does the fixed-reach control address the concern of the reviewer?