- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies Global Factorized Adaptation (GFA), which represents the updates to all target modules through a single low-rank factorization of a reshaped global matrix. Individual module updates are reconstructed by slicing and reshaping this matrix. The authors identify GFA as a single-group, matrix-reshape configuration of SuperLoRA and contribute rank analysis, an efficient reconstruction implementation, and experiments on language models. Evaluations on two 3B models across six tasks show improvements over LoRA primarily at small parameter budgets, including budgets below that of uniform rank-one LoRA.
Critical strengths
-
GFA shows improvements in the low-budget regime. It outperforms LoRA in 36 model–task–budget combinations at the three smallest standard budget anchors. The experiments in the uniform rank1 LoRA budget also demonstrate that GFA can adapt all target modules with very few trainable parameters while outperforming partial-module LoRA at comparable budgets. Reporting results over three seeds strengthens this empirical evidence.
-
The parameterization is simple and retains convenient deployment through weight merging. The block-and-crop implementation avoids materializing the complete global update during training, and the paper provides measurements of reconstruction cost and end-to-end training overhead.
Critical weaknesses
-
The evaluation leaves the method's broader applicability unclear. Experiments are limited to Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct, six multiple-choice or binary classification tasks, and adaptation of query/value projections only. More recent and larger models, generation tasks, and broader target-module coverage would provide stronger evidence of generality. The parameter budgets also stop at the equivalent of LoRA rank 16, while the advantage diminishes as the budget increases. On Llama, the average differences from LoRA are only +0.08 and −0.11 percentage points at the rank-8 and rank-16 budget anchors, respectively.
-
Important design choices lack empirical ablation. Since GFA is explicitly a configuration within the existing SuperLoRA design space, it is particularly important to establish what is gained by the specific choices of a single global group and a near-square reshape. The paper does not compare their downstream performance against alternative grouping or reshape configurations under matched parameter budgets. The rank analysis characterizes the representable updates, but does not by itself establish which structural properties account for the observed accuracy gains. Such ablations would strengthen the empirical insight beyond demonstrating that this particular configuration works.
Questions for authors
-
How does GFA compare with LoRA at larger matched budgets, such as those corresponding to LoRA ranks 32, 64 and 128? Can a representative experiment on a newer or larger model, and on generation tasks, help establish whether the observed advantages extend beyond the current setting?
-
At a fixed trainable-parameter budget, how sensitive is performance to the number of groups, reshape dimensions, and module ordering? In particular, how does a single global group compare with multiple groups?
-
How was the shared learning rate selected? Are the relative gains robust to a small, equally budgeted hyperparameter search, and to matching dropout between GFA and LoRA?
6 comments
Response to Reviewer 4xeQ (2/3)
W2 and Q2: Matched-budget ablations of groups, reshape dimensions, and module ordering.
Response to W2 and Q2: Full-rank global layouts win all 14 one-column and no-reshape comparisons: gains are 0.14–3.79 points from the rank cap alone and up to 15.15 without reshaping. Group count and module order change accuracy by less than one point.
The submitted Theorem 4.1 states that a module is generically full rank at any global rank if , where is the reshape width. The submission also measured reconstruction costs. We now add the following ablations. All tables report accuracy (%), mean ± standard deviation over 3 seeds; Q/L denote Qwen2.5-3B/Llama-3.2-3B.
(a) Number of groups (Q2). Each group has its own near-square factorization; ranks are split to match GFA's LoRA-rank-4 budget (shared rate 1e-4):
| Model | Groups | ARC-E | PIQA | WinoG | Mean |
|---|---|---|---|---|---|
| Q | 1 (GFA) | 94.53 ± 0.29 | 86.65 ± 0.19 | 84.29 ± 0.36 | 88.49 |
| Q | 4 | 94.56 ± 0.14 | 86.63 ± 0.53 | 84.35 ± 1.11 | 88.51 |
| Q | Per layer | 94.42 ± 0.24 | 86.80 ± 0.60 | 83.82 ± 0.41 | 88.35 |
| Q | Per module, capped* | 94.02 ± 0.37 | 86.05 ± 0.69 | 82.93 ± 0.60 | 87.67 |
| Q | Per module, full rank | 94.36 ± 0.13 | 86.85 ± 0.64 | 83.56 ± 0.72 | 88.26 |
| L | 1 (GFA) | 91.18 ± 0.09 | 85.62 ± 0.19 | 85.08 ± 1.19 | 87.29 |
| L | 4 | 90.87 ± 0.28 | 85.55 ± 0.27 | 84.74 ± 0.20 | 87.05 |
| L | Per layer | 90.90 ± 0.11 | 85.44 ± 0.35 | 84.64 ± 0.36 | 86.99 |
| L | Per module, full rank | 90.82 ± 0.11 | 85.22 ± 0.70 | 84.93 ± 0.52 | 86.99 |
*At default width, each square query module forms one block, capping its rank at the group rank. Moving the width by one column restores full rank.
On Qwen, 1, 4, and per-layer groups are within 0.2 points; per-module full-rank factors perform similarly. We retain one group to minimize parameters per rank and avoid a grouping hyperparameter.
(b) Reshape dimensions (Q2). Without reshaping ( equals module width, SuperLoRA's unreshaped single group), module ranks are at most , where is global rank, and all updates share 's row space. This test changes both properties; (c) isolates the rank cap. At , budgets differ by at most 1.1%. Full-rank widths are 82,025 (Q) and 114,001 (L); each arm uses its own best rate:
| Model | Task | No reshape | Full-rank layout | Gain |
|---|---|---|---|---|
| Q | ARC-E | 93.03 ± 0.61 (3e-4) | 94.70 ± 0.07 (3e-4) | +1.67 |
| Q | OBQA | 85.87 ± 0.42 (1e-3) | 89.00 ± 0.60 (1e-4) | +3.13 |
| Q | WinoG | 73.64 ± 1.31 (3e-4) | 84.29 ± 0.55 (3e-4) | +10.65 |
| L | ARC-E | 89.31 ± 0.07 (1e-4) | 90.87 ± 0.37 (3e-4) | +1.56 |
| L | OBQA | 79.53 ± 0.50 (3e-4) | 84.80 ± 1.04 (1e-4) | +5.27 |
| L | WinoG | 68.72 ± 1.87 (1e-4) | 83.87 ± 0.18 (1e-4) | +15.15 |
Among full-rank layouts, wider performs better: 83.71 at , 85.74 at near-square width , and 86.24 at (Qwen, WinoGrande, rank-16 budget). We retain near-square width to maximize global rank per budget.
(c) One-column layout test (Q2). At Qwen attention scope, square width gives and rank at most . Width 18,431 permits full rank at the same global rank and budget. Each arm uses its own best rate:
| Task | Square (rank ) | Width 18,431 (full rank) | Gain | |
|---|---|---|---|---|
| 1 | ARC-C | 82.74 ± 0.94 | 83.59 ± 0.85 | +0.85 |
| 1 | ARC-E | 93.64 ± 0.15 | 94.25 ± 0.21 | +0.61 |
| 1 | OBQA | 86.20 ± 0.40 | 88.07 ± 0.42 | +1.87 |
| 1 | WinoG | 75.53 ± 1.81 | 79.32 ± 0.41 | +3.79 |
| 6 | ARC-C | 83.56 ± 0.13 | 83.87 ± 0.73 | +0.31 |
| 6 | ARC-E | 94.22 ± 0.39 | 94.36 ± 0.08 | +0.14 |
| 6 | OBQA | 88.67 ± 0.99 | 89.40 ± 1.06 | +0.73 |
| 6 | WinoG | 81.14 ± 0.75 | 83.16 ± 1.15 | +2.02 |
Selected rates are 3e-4 except WinoG and full-rank OBQA (1e-4).
Across six / rank caps at equal budget, full rank beats the lowest cap on both tasks. OBQA gains mostly between caps 9 and 38 (+1.27 of +1.87); ARC-E improves gradually up to 1,250 (+0.61 from cap 9 to full rank). The curves have dips within seed variation, consistent with the condition.
Across six models, three scopes, and two width rules, 7/36 near-square layouts violate the condition (none at query/value scope); fixes all seven.
(d) Module ordering (Q2). Same budget and rate, Qwen WinoGrande:
| Module order | Accuracy | vs default |
|---|---|---|
| Lexicographic (default) | 85.74 ± 0.40 | default |
| Fixed random permutation | 85.11 ± 0.68 | -0.63 |
| Random permutation per seed | 85.16 ± 0.72 | -0.58 |
| Numeric layer order (q, v within layer) | 85.45 ± 0.87 | -0.29 |
The condition holds for every order: it depends only on and module widths.
Location: Appendix G.1, Tables 20–24, contains all four ablations and the layout survey.
Response to Reviewer 4xeQ (3/3)
Q3: Selection of the shared learning rate, an equally budgeted search, and matched dropout.
Response to Q3: With both methods tuned on the same learning-rate grid, GFA is on par with LoRA: GFA minus LoRA is -0.08 [-0.30, +0.11] (17/12/21, 50 cells). At the rank-one global budget GFA minus partial-module LoRA has a median of +0.52 on Qwen and +0.83 on Llama (ranges +0.02 to +4.66 and +0.06 to +13.42), while tuned GPart and Uni-LoRA lead GFA in all 6 cells there (by 0.07 to 1.40 and 0.31 to 1.73 points).
The submitted paper used one learning rate, 1e-4, for every method, and dropout 0.05 for LoRA and Tied-LoRA and 0 for the other methods. The revision adds the missing details, point by point:
(a) Selection (Q3). The shared rate 1e-4 was inherited from the LoRA configuration at the start of the project and was not tuned for any method; Section 5 now states this. Every tuned rate is chosen on the reported split, for all methods alike.
(b) Equally budgeted search (Q3). GFA and LoRA were re-run on the same grid, {1e-4, 3e-4, 1e-3} at the LoRA-rank-1 and rank-2 budgets and {1e-4, 3e-4} above, on ARC-C, ARC-E, OBQA, BoolQ, and PIQA (WinoGrande, the most expensive task, was not re-run). The first row gives the shared rate on the same cells (mean [95% interval], wins/ties/losses; 3 seeds per rate):
| Comparison | GFA minus LoRA [95% CI] (win/tie/loss) |
|---|---|
| Same cells, one shared rate 1e-4 (as submitted) | +0.68 [+0.41, +0.98] (43/2/5, 50 cells) |
| Same cells, each method at its own best rate on the same grid | -0.08 [-0.30, +0.11] (17/12/21, 50 cells) |
| LoRA-rank-1 budget | -0.05 [-0.27, +0.17] (4/2/4, 10 cells) |
| LoRA-rank-2 budget | +0.16 [-0.19, +0.49] (7/0/3, 10 cells) |
| LoRA-rank-4 budget | -0.04 [-0.26, +0.16] (3/4/3, 10 cells) |
| LoRA-rank-8 budget | -0.28 [-0.49, -0.09] (1/4/5, 10 cells) |
| LoRA-rank-16 budget | -0.20 [-0.46, +0.03] (2/2/6, 10 cells) |
In 4 cells of the grid (all Qwen PIQA), a GFA seed diverged to chance accuracy at the highest rate; that rate is not used for GFA there (LoRA: 0 cells).
Dropout (Q3). LoRA with dropout 0 instead of 0.05, at the LoRA-rank-1, 4, and 16 budgets on ARC-C, ARC-E, and OBQA for both models, changes its accuracy by 0.01 points on average and by at most 0.27 (18 cells, 3 seeds).
(d) Smallest budgets, both tuned (Q3). Below the rank-one LoRA budget, the LoRA that fits is partial-module LoRA (rank one on a subset of modules). Both are tuned on the grid {1e-4, 3e-4, 1e-3}, and on ARC-C, ARC-E, and OBQA at global rank 1 each grid is extended one step past its edge (3e-3 for partial LoRA, 3e-5 for GFA on Llama). Per budget (3 seeds):
| Model | Budget | GFA minus partial LoRA, shared rate | Both tuned |
|---|---|---|---|
| Qwen2.5-3B | rank-one global (26K / 37.5K) | +6.22 [+3.54, +9.42] (6/0/0, 6 cells) | +1.17 [+0.19, +2.60] (4/2/0, 6 cells); median +0.52 |
| Qwen2.5-3B | about 84K / 104K | +4.55 [+2.50, +6.72] (6/0/0, 6 cells) | +0.65 [+0.22, +0.98] (5/0/1, 6 cells); median +0.88 |
| Llama-3.2-3B | rank-one global (26K / 37.5K) | +5.79 [+2.45, +10.85] (6/0/0, 6 cells) | +2.89 [+0.36, +7.16] (5/1/0, 6 cells); median +0.83 |
| Llama-3.2-3B | about 84K / 104K | +2.22 [+1.10, +3.84] (6/0/0, 6 cells) | +0.65 [-0.13, +1.62] (2/1/3, 6 cells); median -0.04 |
Chosen rates (24 cells): partial-module LoRA 1 at 3e-4, 19 at 1e-3, 4 at 3e-3; GFA 7 at 1e-4, 16 at 3e-4, 1 at 1e-3. Where partial-module LoRA's best is the top of its grid (17 cells), GFA's margin is an upper bound. Rates at which a seed diverged are excluded (GFA: 2 cells; partial-module LoRA: 2 cells, at 1e-2).
In the revised paper, Section 5 states how the shared rate was chosen, Section 6 opens with the equal grid (Table 3), and Appendix D gives the dropout control and the tuned smallest budgets (Tables 10 and 11).
We thank the reviewer again. The design ablation the reviewer asked for now carries the paper's account of GFA's accuracy: it shows that full-rank module updates, the property our layout condition isolates, win every cell of the one-column and no-reshape tests at the same budget, and the one-column test confirms the condition's prediction in all 8 cells.
In the reviewer's order, we added larger budgets, GSM8K, MBPP, and broader target scopes (W1 and Q1); added matched-budget ablations of groups, reshape width, no reshape, and module order (W2 and Q2); and stated the learning-rate choice and added an equal grid, a dropout-matched LoRA, and tuned the smallest budgets (Q3).
If these results resolve the reviewer's concerns, we would be grateful if the reviewer would reconsider their evaluation. Please let us know which point is most important to you, and we will prioritize it; we also welcome any further questions.
Response to Reviewer 4xeQ (1/3)
We thank the reviewer for recognizing that "the block-and-crop implementation avoids materializing the complete global update during training". We added:
Changes are marked in blue.
Response to W1 and Q1: With both methods tuned, GFA is on par with LoRA across rank-1 to rank-16 budgets on the evaluated classification tasks. Below rank one, GFA's median gains over partial-module LoRA are +0.52 on Qwen and +0.83 on Llama (ranges +0.02 to +4.66 and +0.06 to +13.42; Q3(d)). These margins may shrink with a wider search: partial-module LoRA selects its highest tested rate in 17/24 cells. Tuned GPart and Uni-LoRA lead GFA in all six comparisons, by 0.07–1.40 and 0.31–1.73 points. LoRA leads above rank 16, on GSM8K, and in all four MBPP comparisons.
The submission evaluated Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct on six classification tasks with query/value adaptation. “LoRA-rank-r budget” denotes LoRA's trainable-parameter count at rank r. The revision adds:
(a) Larger matched budgets (W1 and Q1). GFA's global rank rg, the rank of its shared matrix, ranges from 244 to 1,131, matching LoRA's parameter count within 0.3%. On ARC-C, ARC-E, and OBQA, each method selects its best three-seed mean from {1e-4, 3e-4}. Results are mean differences [95% intervals], with wins/ties/losses; ties are differences below 0.1 points:
At 3e-4, a GFA seed diverged to chance in 5/18 cells: Qwen ARC-C at ranks 32/64/128, Qwen ARC-E at rank 128, and Llama ARC-C at rank 128. Those GFA results use 1e-4; no LoRA seed diverged.
(b) A larger model (Q1). We add Qwen2.5-7B-Instruct at rank-1/rank-16 budgets: parameter counts (GFA/LoRA) are 283,843/315,392 and 5,028,076/5,046,272, respectively. Each method selects its best rate from {1e-4, 3e-4}. Its query/value layout permits generically full-rank module updates. A 13B model's FP16 weights alone (26 GB) exceed our 24 GB GPU.
Results are mixed at rank one; LoRA leads at rank 16. Accuracy (%), mean ± standard deviation, 3 seeds:
(c) Generation tasks (Q1). GSM8K evaluates exact match of the generated final number. At shared rate 1e-4, GFA trails LoRA by 0.77 points on average (Qwen: 0.45; Llama: 1.09), despite lower training loss in all ten cells. Only 2.7% of supervised tokens are the graded answer; lower reasoning-text loss need not improve answer accuracy. Both tuned at the rank-one budget (3 seeds):
MBPP pass@1 requires generated Python code to pass all unit tests. Each method selects its best rate from {1e-4, 3e-4} (3 seeds):
(d) Broader target modules (W1). On Llama-3.2-3B at query/value budgets (shared rate 1e-4, 3 seeds), GFA leads in 1/3 attention-scope comparisons (q,k,v,o) and trails in all four all-linear comparisons (attention and MLP), by 0.63–1.72 points. At the all-linear rank-4 budget, LoRA's minimum rank of one gives it 43% more parameters than GFA.
Locations: Appendix F, Tables 16–17 (scopes and larger budgets); Appendix E, Tables 13–15 (GSM8K and MBPP).