acceptodds
Under review as a conference paper at ICLR 2027

Factorization Depth as a Regularization Dimension for LoRA

Abstract

Existing deep low-rank formulations largely treat depth as a fixed design choice; we investigate whether factorization depth can instead serve as a regularization dimension for Low-Rank Adaptation (LoRA), beyond trainable parameter count and structural-rank capacity. To characterize this role, we analyze a mergeable -factor parameterization, with controlling factorization depth, through its representation cost, induced optimization geometry, and idealized singular-mode dynamics. These analyses motivate a finite-horizon mechanism hypothesis: deeper factorization may selectively delay weak singular modes, promoting spectral concentration at moderate depths but hindering task fitting when taken too far. We test this hypothesis through four complementary approaches: capacity controls, depth ablations, spectral diagnostics, and synthetic experiments. Across these experiments, the resulting accuracy–rank–loss patterns support the hypothesized trade-off between spectral concentration and task fit, rather than a monotonic benefit from increasing depth. Finally, to probe the mechanism behind this effect, we show that a nuclear-norm intervention recovers only part of the performance gain observed at moderate depths, suggesting that the depth effect is not fully explained by explicit nuclear-norm regularization in the tested settings. Overall, our results support viewing factorization depth as a promising, non-monotonic regularization dimension for LoRA, while motivating further study of whether this phenomenon generalizes across optimizers, backbone models, and tasks.

Primary area:
optimization
Keywords:
parameter-efficient fine-tuning, low-rank adaptation, factorization depth, implicit regularization, optimization geometry, spectral dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UCDTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates whether factorization depth can serve as another regularization dimension for LoRA, alongside trainable parameter count and structural rank. Analyses of a mergeable LL-factor parameterization motivate a finite-horizon hypothesis that deeper factorization may selectively delay weak singular modes, promoting spectral concentration at moderate depths but hindering task fitting at excessive depths. Experiments on LLaMA-2-7B, including capacity controls, depth ablations, spectral diagnostics, and synthetic experiments, support this trade-off rather than a monotonic benefit from increasing depth. A nuclear-norm intervention recovers only part of the gain, suggesting that the depth effect is not fully explained by explicit nuclear-norm regularization.

Critical strengths

The main comparison controls capacity in two complementary ways. LoRA r8 matches the trainable parameter count of the default DR-LoRA, and LoRA r4 matches its structural-rank bound. DR-LoRA achieves higher task performance than both. The complementary ablation experiments (e.g., the structural-rank ablation) further enhance the value of the work. Additionally, the depth sweep reports the trade-off rather than only the best configuration (Table 3, Figure 3). As depth increases, effective rank falls monotonically, while accuracy peaks at a moderate depth and training loss rises sharply beyond it.

Critical weaknesses

The evidence that the gain is robust to the optimization setting is limited. The main six-task comparison uses a single learning rate shared by all methods (Section 4). Although a learning-rate check is provided, it covers only CommonsenseQA, and LoRA performs best at the lowest tested rate, so its optimal setting is not reached (Appendix B.3, Table 11).

The default configuration is not compared with the close prior method. Deep LoRA [1] already uses a three-factor adapter and reports benefits in reducing overfitting in limited-data fine-tuning. It would be a natural baseline and would strengthen this work.

[1] Can Yaras, Peng Wang, Laura Balzano, and Qing Qu. Compressible dynamics in deep overparameterized low-rank learning and adaptation. In ICML, 2024.

Questions for authors
  1. Adapters are inserted only into q_proj and v_proj (Section 4). Does the depth effect persist when adapters are applied to all linear layers, including the MLP projections?

  2. The intermediate dimensions r1,…,rL−1r_1,\dots,r_{L-1} used for L=4L=4 and L=5L=5 in the depth sweep are not reported. Could the authors provide them, together with the resulting structural-rank bound and parameter count?

  3. Figure 2(b) reports the pre-clip gradient norm. Since Section 4 states that gradient clipping is not used, could the authors clarify whether any clipping was applied during training?

  4. All methods are trained for a fixed 3 epochs without mentioned early stopping (Section 4). As LoRA already reaches near-zero training loss (Table 3), the gain may depend on this training length. Could the authors clarify how the training length and the default configuration were chosen, and report results with early stopping or different training lengths?

My score could be raised if the concerns in the Questions and Weaknesses sections are properly addressed.

3 comments
AuthorsOpenReview ↗

New experiments on learning rate, training length, and all-linear adapters, plus a Deep LoRA baseline, show that the gains hold; the clipping statement is corrected.

Thank you for the detailed reading. Below are new LLaMA-2-7B experiments and a Deep LoRA baseline; the revision adds Sec. 7.3 and 8.2 and fixes the clipping statement in Sec. 4.

Protocol. 10% of the CSQA training set is held out as a tune split for selecting learning rates and early-stopping epochs; validation is used only for reporting. All methods use s = 2, AdamW, cosine schedule, clipping 1.0, effective batch 16, and q/v adapters unless noted.

W1: Robustness to the optimization setting

Table R1. CSQA, 3 epochs; tune/report accuracy per learning rate (selected rate in bold).

  • LoRA r8: 1e-5 0.706/0.727; 2e-5 0.709/0.745; 5e-5 0.734/0.776; 2e-4 0.738/0.769; 8e-4 –/0.725
  • DR-LoRA L=3: 5e-5 0.760/0.800; 2e-4 0.778/0.816; 8e-4 0.782/0.809; 1.6e-3 diverged

LoRA's optimum is bracketed, and its tune-selected rate is the shared rate of Table 1. DR-LoRA leads by 4.0 points at the selected rates (0.809† vs. 0.769) and at the best report rates (0.816 vs. 0.776), and is less rate-sensitive (0.800–0.816 vs. 0.725–0.776).

Table R2. Six tasks at the selected rates (LoRA's rate equals the shared one, so its row is from Table 1).

MethodMRPCCoLARTESciQACSQAOBQAMacro
LoRA r8 (2e-4)0.8650.8500.8360.8190.7340.7470.809
DR-LoRA L=3 (8e-4)0.8850.8630.8880.8260.8130.8180.849

DR-LoRA leads on all six tasks; the macro gain (+4.0) is no smaller than with the shared rate (+3.4, Table 1) and far above the seed std (0.3–0.4 points).

W2: Deep LoRA

Deep LoRA (official code, tune-selected 8e-4, seed 42) reaches a six-task macro-average of 0.844, vs. 0.849 for DR-LoRA and 0.809 for LoRA r8, and is within 0.8 points of DR-LoRA on every task. The two deep methods are comparable, so the gain comes from the added factor itself. Our contribution is depth as a tunable regularization dimension, not another three-factor adapter: Deep LoRA fixes L = 3, whereas our sweep (L = 2–5) shows falling effective rank, rising training loss (0.001 to 0.205), and accuracy that first rises then falls (Table 3), with the optimum shifting with the learning rate (Table 11).

Q1: All linear layers (Sec. 8.2, Table 7)

MethodParamsReporteRank attn / MLP
LoRA r8 (2e-4)20.0M0.7737.42 / 7.61 (bound 8)
DR-LoRA L=3 (8e-4)20.9M0.8042.85 / 3.01 (bound 4)

With adapters on all seven projections and the q/v learning rates, DR-LoRA leads by 3.1 points and stays spectrally concentrated in attention and MLP layers, while LoRA stays near its bound.

Q2: Depth-sweep dimensions

Each added factor is a 4×4 matrix at the input-side bottleneck, so the rank bound stays at 4 and each matrix gains 16 parameters: L = 3/4/5 are 4096→4→12→4096, 4096→4→4→12→4096, and 4096→4→4→4→12→4096 (65,584/65,600/65,616 params per matrix), vs. 65,536 for LoRA r8 (bound 8) and 32,768 for LoRA r4 (bound 4). In Eq. (1), r₁ = 12 (output side) and r₂ = 4 (input side). The rank-matched LoRA r4 (0.776) is also below DR-LoRA (0.816). The final version will add this table and a LoRA r4 row to Table 3.

Q3: Gradient clipping

Section 4 had a writing error: the code calls clip_grad_norm_(max_norm=1.0) before every update, so all runs of all methods clip the global adapter-gradient norm to 1.0 (corrected in the revision). Figure 2(b) and the peaks in Table 2 are the pre-clip norms returned by this function; they stay above 1.0, so almost every update is clipped.

Q4: Training length, early stopping, and defaults

The defaults follow from capacity constraints, not tuning: width 4 matches LoRA r4's rank bound, width 12 matches LoRA r8's parameter count, α gives s = 2, and L = 3 is the smallest depth beyond LoRA. Like Yaras et al. (2024), we evaluate the final checkpoint; to check whether this disadvantages LoRA, we trained for 5 epochs with and without early stopping on the tune split.

Table R6. CSQA report accuracy.

Method3 ep5 ep, last5 ep, early stop
LoRA r8 (2e-4)0.7690.7530.753 (ep 5)
DR-LoRA L=3 (8e-4)0.8090.8030.803 (ep 5)
DR-LoRA L=4 (8e-4)0.8160.7990.803 (ep 4)

Longer training does not close the gap (+5.0 at 5 epochs, p < 1e-5), and early stopping picks the last or second-to-last epoch for every method. At 8e-4, L = 4 matches or beats L = 3 with lower effective rank (2.32 vs. 2.89), whereas at 2e-4 it underfits within 3 epochs (Table 3): the optimal depth depends on the effective training horizon.

I want to thank the authors for the detailed clarifications, which addresses most of my concerns. I also encourage the authors to incorporate these clarifications into the revised version of the paper. I will consider raising my score from 2 to 3.

Thank you for the careful follow-up and for considering a higher score. As suggested, we have incorporated the key clarifications into the revised version of the paper. We are grateful for your constructive comments, which have substantially strengthened our work.

Reviewer_qszFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies how the number of factors in LoRA affects training and performance. The authors explain how depth changes training and observe that deeper factorizations lead to updates with lower effective rank. However, too many factors can increase training loss and hurt performance. They also test whether nuclear-norm regularization can reproduce the lower effective rank observed with deeper factorizations, but find that it does not fully reproduce the performance improvements.

Critical strengths

The paper provides a reasonably thorough experimental analysis of factorization depth. In particular, the authors show, under restrictive assumptions, that increasing depth can slow the learning of weaker singular directions, which may explain the lower effective rank observed in their experiments. However, I am not convinced that the findings offer much practical value.

Critical weaknesses
  • Clarity and presentation. The abstract is unnecessarily difficult to read. It uses convoluted expressions such as “finite-horizon mechanism hypothesis” and “accuracy–rank–loss patterns” to describe relatively simple ideas. Although these expressions have a meaning in context, they make the main contribution harder to understand than necessary. Similar wording appears elsewhere in the paper, particularly in the introduction and theoretical discussion. The experimental sections are generally clearer, although some passages remain difficult to follow.

  • Limited novelty and unclear contribution. I struggle to identify the main contribution beyond existing work. As the authors acknowledge, Yaras et al. (ICML 2024) have already studied deeper factorizations for LoRA. The authors argue that factorization depth should be treated as a regularization hyperparameter rather than a fixed design choice. However, depth is already an available design choice in deep-factorization methods, and I do not find that studying its effect across different values constitutes a sufficiently substantial contribution on its own. The theoretical analysis characterizes how depth affects optimization dynamics, while the experiments show that increasing depth is not always beneficial and that the tested nuclear-norm regularization does not fully reproduce its effects. Nevertheless, the analysis does not provide a way to select an appropriate depth for a given task or layer. In practice, depth must still be selected through ordinary hyperparameter tuning. Overall, I find the contribution too incremental relative to the existing literature.

Questions for authors
  • It is possible that I have missed the main point of the paper. Could the authors clarify what they consider their core contribution and explain its concrete practical implications?

  • Figure 1 could be improved, particularly by increasing the font size to make the text easier to read.

9 comments
AuthorsOpenReview ↗

We simplified the abstract wording and show that depth acts as a regularization dimension: fixed-depth designs miss the non-monotonic effect, and per-layer depth allocation improves accuracy from 0.800 to 0.820.

Thank you for your comments. All changes have been incorporated in the revised version.

Abstract, introduction, and Figure 1. We rewrote the two expressions you pointed out. "Finite-horizon mechanism hypothesis" now reads: "within a fixed training budget, each additional factor slows the learning of weak singular directions: moderate depth then concentrates the learned spectrum, whereas excessive depth also slows directions needed to fit the task." "Accuracy–rank–loss patterns" now reads: "Depth ablations, spectral diagnostics, and synthetic experiments support this prediction: as depth grows, the effective rank of the learned update falls and the training loss rises, while accuracy first rises and then falls." The revised abstract also states the main capacity-controlled comparison and the per-layer allocation result. The corresponding "finite-horizon hypothesis" sentence in the introduction is rewritten in the same plain terms. Figure 1 has been redrawn with a larger font.

Our core claims have two parts.

  1. Depth is a regularization dimension: increasing L strengthens regularization. On CommonsenseQA, as L grows from 2 to 5, the effective rank drops from 7.61 to 2.10 and the training loss rises from 0.001 to 0.205, while accuracy first rises and then falls (Table 3). Capacity or an explicit spectral penalty cannot explain this: one added factor beats both the parameter-count-matched and the rank-bound-matched LoRA (by 3.4 and 2.2 points, Table 1), while the best nuclear-norm penalty reaches only 0.786, below 0.816 with depth (Table 4).

  2. The optimal depth is not fixed. Like other regularization strengths, it changes with the training setup: L = 3 is best at learning rates 5e-5 and 2e-4, whereas L = 4 is best at 8e-4 and ties the best L = 3 result (0.8161 vs. 0.8160, Table 11). As our analysis predicts, a longer effective training horizon calls for more depth. This is the key difference from Deep LoRA (Yaras et al., 2024), which fixes the depth at 3.

On "depth is already an available design choice." We agree that depth can be set, but prior work does not say what value it should take or why. Treating depth as a tunable quantity reveals three regularities:

  • Effective rank falls monotonically with depth while accuracy first rises then falls, and training loss rises sharply at large depth (underfitting): depth acts like a regularization strength with a too-weak/too-strong trade-off (Table 3).
  • The optimal depth shifts with the learning rate (Table 11).
  • A nuclear-norm penalty cannot replace depth: the best penalty lowers the effective rank only from 7.61 to 7.38 and recovers only part of the gain (Table 4).

None of these is visible when depth is fixed. The theory explains why the trade-off arises (an added factor slows weak directions), beyond recording that depth is not always beneficial.

Practical value. Our analysis does not give a closed-form optimal depth. But treating depth as a regularization dimension implies that, like weight decay or rank, depth need not be one constant across the model: the optimal depth depends on the effective training horizon, which can differ across layers, tasks, and backbones. Depth can thus be allocated per layer, as AdaLoRA (Zhang et al., 2023) allocates rank. We added this experiment (Section 7.3 and Table 5 of the revision).

Per-layer depth allocation vs. uniform L = 3 (CSQA)

Uniform L = 3Per-layer (depths 2/3/4)
Projections per depthall 64 at depth 322 / 20 / 22 at depths 2 / 3 / 4
Mean depth3.03.0
Trainable parameters4,197,3764,196,672 (−0.02%)
Tune accuracy0.7820.789
Report accuracy0.8000.820
Report difference–+2.0 points, 95% CI [+0.1, +3.9], McNemar p = 0.053

Everything else is identical (q/v adapters, 3 epochs, learning rate 8e-4, same recipe and data split).

Allocation rule: the 64 q/v projections are sorted by the effective rank of their update under uniform L = 3; the 22 most concentrated use depth 4, the middle 20 depth 3, and the 22 most spread depth 2. This rule is one of 4 rules fixed before training and was selected by tune accuracy only. The other 3 rules (depth increasing with layer index, decreasing with layer index, and larger depth for the most spread spectra) all have lower tune accuracy.

Reviewer_qszFOpenReview ↗

Answer to author

Hello, thank you for your rebuttal.

Regarding the "available design choice," if I understand correctly, the main novelty compared to existing work is to interpret factorization depth as a regularization dimension and to propose allocating different depths across layers.

While additional interpretations and insights are always welcome, I am not fully convinced by their significance here. Most of the findings seem rather expected for a hyperparameter, such as the existence of a trade-off or the fact that the optimal value depends on other hyperparameters (e.g., the learning rate).

Apart from the distinction with nuclear-norm regularization, I don't see a particularly surprising or non-trivial finding. I am also not convinced that the per-layer depth allocation adds much novelty, given that similar ideas have already been explored for rank allocation.

I think what is still missing for me is a clearer explanation of what we actually learn from this work that we could not have anticipated before. For example, are there concrete practical implications or guidelines that follow from your analysis, beyond simply considering depth as another hyperparameter to tune?

At this stage, I am therefore leaning toward maintaining my original score.

AuthorsOpenReview ↗

A nuclear-norm penalty recovers only one-third of depth's gain, even at matched rank; excessive depth fails by underfitting; and the optimal depth tracks the training horizon as the theory predicts.

Thank you for engaging with our rebuttal — your summary of the intended novelty is accurate.

On "what do we learn that we could not have anticipated": we should have made this sharper. Three things, in decreasing order of surprise.

  1. The default norm-based account fails, against our own initial expectation. Proposition 1 implies that the representation cost of an L-factor product is a Schatten-2/L penalty, so the natural expectation — shared by the implicit-regularization literature — is that depth's effect should be reproducible by an explicit spectral penalty. It is not: at its best setting, the nuclear-norm penalty recovers roughly one-third of depth's accuracy gain (0.786 vs. 0.816 over 0.769) and leaves effective rank at 7.38 where depth reaches 2.47. A negative result against the explanation suggested by the parameterization itself is, to us, the main new information in this work. It also has practical bite: depth's gains cannot be obtained by adding the natural explicit alternative to LoRA. We additionally applied the penalty to the rank-matched LoRA r4 control (seed 42, λ ∈ {5e-4, 1e-3}, which includes the best setting of Table 4). The best setting raises accuracy from 0.760 to 0.777, again only about one-third of the gap to depth (0.816). The penalty shrinks the top singular value (13.5 → 7.5) but leaves effective rank at 3.88 (bound 4), whereas depth reaches 2.47.

  2. The failure mode at excessive depth is underfitting, and it is sharp. Prior deep-LoRA work reports depth as beneficial in limited-data fine-tuning; the intuition "deeper = stronger spectral bias" predicts saturation at worst. Instead, at a constant rank bound and near-constant parameter count (+32 parameters from L=3 to L=5), training loss rises from 0.003 to 0.205 (≈70×) while accuracy falls only 1.4 points — the regularizer turns into a bottleneck, and the useful window is narrow (L=3 among the tested 2–5).

  3. The mechanism predicts a directional coupling that a generic hyperparameter view does not. If depth were just another knob, its optimum would shift with the learning rate arbitrarily. The finite-horizon account predicts a specific direction — longer effective horizons require more depth — which is what we observe: L=3 is best at 5e-5 and 2e-4, while L=4 is best at 8e-4 and ties the best L=3 result (Table 11). It is this theory-linked directionality, not the mere existence of a trade-off, that we consider non-trivial.

On concrete practical guidelines, three follow directly from the analysis; we have added a short paragraph summarizing them at the end of Section 7 of the revision:

  • Start at L=3 and use training loss as the early-warning signal: it rises ≈25× at L=4 before accuracy visibly degrades, so loss flags the excessive-depth regime earlier than the benchmark metric.
  • When the training horizon grows (larger LR, more epochs), shift depth up rather than re-tuning from scratch; when it shrinks, stay shallow.
  • If per-layer gains are wanted, one uniform-depth run suffices: sort projections by the effective rank of their update and allocate depths 2/3/4 by concentration (+2.0 points at matched mean depth and parameter count).

On per-layer allocation: the analogy with rank allocation is intentional, and we agree the mechanism (allocating a capacity knob per layer) is not new. What is new is the empirical fact that depth has per-layer structure predictable from a cheap, measurable statistic (update effective rank) — something rank-allocation work did not establish for depth.

We recognize that each individual pattern, taken alone, resembles familiar hyperparameter behavior. Our claim concerns the package: a mechanism that predicts the shape of the trade-off, its direction of coupling to the optimization horizon, its non-reducibility to the explicit penalty its own cost function suggests, and an allocation rule derived from that mechanism. We hope this answers what is learned beyond treating depth as another hyperparameter, and we are happy to discuss further.

Reviewer_h27oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes DR-LoRA to insert extra mergeable factors into LoRA, which treats factorization depth as an implicit spectral regularizer. Experiments are mainly fine-tune the Q/V projections of LLaMA-2-7B on GLUE/QA and CommonsenseQA tasks.

Critical strengths

The paper's insight on factorization depth as a non-monotonic regularization dimension for LoRA is indeed interesting.

Critical weaknesses

The paper's visual presentation in Figure 1 is problematic. It is barely readable, with tiny fonts and dense equations crammed into panels. The submission needs to put in extra effort to present the framework suitably.

Questions for authors

The paper assumes that aligned factors under squared-loss gradient flow can describe AdamW training with unbalanced (and zero-initialized) factors. I look forward to the authors justifying this assumption.

3 comments
AuthorsOpenReview ↗

We clarify that the theory is a conditional qualitative prediction, and add a synthetic experiment that removes each idealized assumption step by step.

Thank you for the careful reading, and for finding the non-monotonic role of depth interesting.

Figure 1. Following your comment, we redrew Figure 1: the font is larger, each panel keeps a single equation, and each panel states its conclusion in one sentence.

Q: Can gradient flow with aligned, balanced factors and squared loss describe AdamW training with unbalanced factors, one of which is zero-initialized?

Not exactly, and we do not claim that it does. The specific learning rates in Eq. (8) hold only under aligned, balanced factors and gradient flow; under AdamW and unbalanced initialization, the learning speed of each direction no longer follows this exact form. We use the idealized analysis only for a qualitative prediction: an additional factor slows the learning of weak directions, so within a fixed training budget deeper adapters have lower effective rank, and excessive depth underfits. The paper states this scope in the last paragraph of Section 3.3: "Our AdamW initialization is initially unbalanced; the analysis is therefore a conditional mechanism hypothesis rather than a Transformer training law." However, this statement appears only at the end of Section 3, and the main text of Section 3 together with the original Figure 1 could suggest that the theory directly describes AdamW training. We apologize for this misunderstanding. The caption of the new Figure 1 states that panel (d) holds only under aligned, balanced gradient flow.

To test the prediction under realistic training conditions, we ran a synthetic experiment that removes the idealized assumptions one at a time.

Setup. We fit a rank-8 target matrix (256 × 256) with L = 2, 3, and 4 factors over 3 random seeds in three settings:

  • Theory setting: aligned, balanced factors with small initialization, trained by gradient descent (GD). The simulation matches Eq. (8) to within 1%.
  • Our initialization (one zero factor, Kaiming-uniform for the others), trained by GD.
  • Our initialization, trained by AdamW with cosine decay, as in the LLaMA experiments.

All depths use the same budget: the number of steps LoRA (L = 2) needs to fit the target, matching LoRA's near-zero training loss in Table 3.

Results (L = 2 / 3 / 4; the effective rank of the target is 6.28):

SettingFraction of the weakest direction learnedEffective rank
Theory setting58% / 1% / 1%6.09 / 3.99 / 2.30
Our init. + GD76% / 0.5% / 0.5%6.23 / 5.40 / 4.23
Our init. + AdamW98% / 1% / 9%6.28 / 5.65 / 5.00

The prediction holds in all three settings: for L ≥ 3, the fraction of the weakest direction learned within the budget is far below that for L = 2, and the effective rank decreases with depth. Under realistic conditions the effect is more selective. In the theory setting depth delays all directions, whereas with our initialization and AdamW the strongest direction is learned equally fast at every depth (about 40 steps) and only weak directions are delayed. The factors stay unbalanced throughout training, so strict balance is not required. With a 4× longer budget, all depths fully recover the target: depth delays weak directions without changing the final solution. The results are unchanged with learning rates 3e-4 or 3e-3 and with weight decay 0.01. The effect disappears only when the nonzero factors are initialized at 3× the usual scale, consistent with the small-initialization assumption.

In short, the exact equations hold only under their assumptions, but the qualitative prediction holds in our actual training setup.

Thank you again for your constructive feedback. We wanted to follow up on our rebuttal and would appreciate hearing whether our response addresses your main concerns. If any points remain unclear, we would be happy to clarify them during the discussion period.

If the clarifications resolve your concerns, we would be grateful if you would consider revisiting your rating in light of our response. Thank you for taking the time to review and discuss our work.

Since the discussion is closing, we would like to briefly summarize the two responses prepared for your review, in case they are helpful for your final assessment.

  1. Figure 1 has been redrawn with larger fonts, one equation per panel, and a one-sentence conclusion in each panel.

  2. On the gap between the aligned, balanced gradient-flow analysis and AdamW training with unbalanced, zero-initialized factors: we agree the analysis does not directly describe AdamW training and use it only as a qualitative prediction. To test this prediction under realistic conditions, we added a synthetic experiment that removes the idealized assumptions one at a time — our initialization with GD, and our initialization with AdamW — and finds the same qualitative effect (weak-direction delay, lower effective rank with depth) in all settings, robust to learning rate and weight decay.

We would be glad to answer any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.