acceptodds
Under review as a conference paper at ICLR 2027

Global Factorized Adaptation: Efficient Parameter Sharing for Fine-Tuning Language Models

Abstract

Low-rank adaptation splits the trainable budget across modules and constrains the rank of every module update, and existing sharing methods typically keep this module-wise structure or rely on fixed expansion maps. We study Global Factorized Adaptation (GFA), the simplest configuration of the SuperLoRA design space (one group, a matrix reshape, and no projection), which instead imposes low-rank structure on the model-wide update: GFA arranges the coordinates of all target updates into a near-square matrix, learns a single low-rank factorization of it, and reconstructs each module update by slicing and reshaping. This layout minimizes the parameter count for a fixed global rank and makes the budget about times finer than that of uniform rank-one LoRA over equally shaped modules. We give an arithmetic layout condition, independent of the module order and satisfied by our layouts, under which the module updates are generically full rank even at global rank one, and we show that on both evaluated models, GFA and LoRA are far from containing each other at matched budgets. Against six baselines on two language models, six tasks, and five matched budgets, GFA improves over LoRA by percentage points on average, with the largest gains at small budgets. With to fewer parameters than rank-one LoRA, it outperforms partial-module LoRA of the same size by to percentage points on both models, and its training cost stays close to that of LoRA.

Primary area:
optimization
Keywords:
Parameter-efficient fine-tuning, low-rank adaptation, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4xeQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies Global Factorized Adaptation (GFA), which represents the updates to all target modules through a single low-rank factorization of a reshaped global matrix. Individual module updates are reconstructed by slicing and reshaping this matrix. The authors identify GFA as a single-group, matrix-reshape configuration of SuperLoRA and contribute rank analysis, an efficient reconstruction implementation, and experiments on language models. Evaluations on two 3B models across six tasks show improvements over LoRA primarily at small parameter budgets, including budgets below that of uniform rank-one LoRA.

Critical strengths
  1. GFA shows improvements in the low-budget regime. It outperforms LoRA in 36 model–task–budget combinations at the three smallest standard budget anchors. The experiments in the uniform rank1 LoRA budget also demonstrate that GFA can adapt all target modules with very few trainable parameters while outperforming partial-module LoRA at comparable budgets. Reporting results over three seeds strengthens this empirical evidence.

  2. The parameterization is simple and retains convenient deployment through weight merging. The block-and-crop implementation avoids materializing the complete global update during training, and the paper provides measurements of reconstruction cost and end-to-end training overhead.

Critical weaknesses
  1. The evaluation leaves the method's broader applicability unclear. Experiments are limited to Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct, six multiple-choice or binary classification tasks, and adaptation of query/value projections only. More recent and larger models, generation tasks, and broader target-module coverage would provide stronger evidence of generality. The parameter budgets also stop at the equivalent of LoRA rank 16, while the advantage diminishes as the budget increases. On Llama, the average differences from LoRA are only +0.08 and −0.11 percentage points at the rank-8 and rank-16 budget anchors, respectively.

  2. Important design choices lack empirical ablation. Since GFA is explicitly a configuration within the existing SuperLoRA design space, it is particularly important to establish what is gained by the specific choices of a single global group and a near-square reshape. The paper does not compare their downstream performance against alternative grouping or reshape configurations under matched parameter budgets. The rank analysis characterizes the representable updates, but does not by itself establish which structural properties account for the observed accuracy gains. Such ablations would strengthen the empirical insight beyond demonstrating that this particular configuration works.

Questions for authors
  1. How does GFA compare with LoRA at larger matched budgets, such as those corresponding to LoRA ranks 32, 64 and 128? Can a representative experiment on a newer or larger model, and on generation tasks, help establish whether the observed advantages extend beyond the current setting?

  2. At a fixed trainable-parameter budget, how sensitive is performance to the number of groups, reshape dimensions, and module ordering? In particular, how does a single global group compare with multiple groups?

  3. How was the shared learning rate selected? Are the relative gains robust to a small, equally budgeted hyperparameter search, and to matching dropout between GFA and LoRA?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 4xeQ (1/3)

We thank the reviewer for recognizing that "the block-and-crop implementation avoids materializing the complete global update during training". We added:

  • larger budgets, GSM8K and MBPP generation, and broader target scopes (W1, Q1);
  • matched-budget ablations of grouping, reshape, and module order, plus a theory-predicted one-column layout test (W2, Q2);
  • equal-grid tuning for GFA and LoRA, dropout-matched LoRA, and tuned comparisons below rank-one budgets (Q3).

Changes are marked in blue.


W1 and Q1: Larger budgets, a larger model, generation tasks, and broader target modules.

Response to W1 and Q1: With both methods tuned, GFA is on par with LoRA across rank-1 to rank-16 budgets on the evaluated classification tasks. Below rank one, GFA's median gains over partial-module LoRA are +0.52 on Qwen and +0.83 on Llama (ranges +0.02 to +4.66 and +0.06 to +13.42; Q3(d)). These margins may shrink with a wider search: partial-module LoRA selects its highest tested rate in 17/24 cells. Tuned GPart and Uni-LoRA lead GFA in all six comparisons, by 0.07–1.40 and 0.31–1.73 points. LoRA leads above rank 16, on GSM8K, and in all four MBPP comparisons.

The submission evaluated Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct on six classification tasks with query/value adaptation. “LoRA-rank-rr budget” denotes LoRA's trainable-parameter count at rank rr. The revision adds:

(a) Larger matched budgets (W1 and Q1). GFA's global rank rgr_g, the rank of its shared matrix, ranges from 244 to 1,131, matching LoRA's parameter count within 0.3%. On ARC-C, ARC-E, and OBQA, each method selects its best three-seed mean from {1e-4, 3e-4}. Results are mean differences [95% intervals], with wins/ties/losses; ties are differences below 0.1 points:

LoRA rank (budget)GFA minus LoRA, both tuned
32-0.40 [-0.75, -0.07] (1/2/3, 6 cells)
64-1.27 [-1.64, -0.90] (0/0/6, 6 cells)
128-2.13 [-2.64, -1.72] (0/0/6, 6 cells)

At 3e-4, a GFA seed diverged to chance in 5/18 cells: Qwen ARC-C at ranks 32/64/128, Qwen ARC-E at rank 128, and Llama ARC-C at rank 128. Those GFA results use 1e-4; no LoRA seed diverged.

(b) A larger model (Q1). We add Qwen2.5-7B-Instruct at rank-1/rank-16 budgets: parameter counts (GFA/LoRA) are 283,843/315,392 and 5,028,076/5,046,272, respectively. Each method selects its best rate from {1e-4, 3e-4}. Its query/value layout permits generically full-rank module updates. A 13B model's FP16 weights alone (26 GB) exceed our 24 GB GPU.

Results are mixed at rank one; LoRA leads at rank 16. Accuracy (%), mean ± standard deviation, 3 seeds:

LoRA-rank budgetTaskGFA (rate)LoRA (rate)GFA minus LoRA
1ARC-C90.19 ± 0.17 (1e-4)89.99 ± 0.18 (1e-4)+0.20
1OBQA92.80 ± 0.72 (1e-4)93.60 ± 0.00 (3e-4)-0.80
1ARC-E96.58 ± 0.35 (3e-4)96.18 ± 0.14 (3e-4)+0.40
16ARC-C90.27 ± 0.76 (1e-4)90.39 ± 0.21 (3e-4)-0.12
16OBQA92.87 ± 0.46 (1e-4)93.73 ± 0.61 (1e-4)-0.86
16ARC-E96.63 ± 0.55 (1e-4)97.05 ± 0.00 (3e-4)-0.42

(c) Generation tasks (Q1). GSM8K evaluates exact match of the generated final number. At shared rate 1e-4, GFA trails LoRA by 0.77 points on average (Qwen: 0.45; Llama: 1.09), despite lower training loss in all ten cells. Only 2.7% of supervised tokens are the graded answer; lower reasoning-text loss need not improve answer accuracy. Both tuned at the rank-one budget (3 seeds):

ModelGFA (rate)LoRA (rate)GFA minus LoRA
Qwen2.5-3B65.78 ± 0.53 (1e-4)66.24 ± 0.43 (1e-4)-0.46
Llama-3.2-3B66.19 ± 1.32 (1e-4)67.63 ± 0.50 (1e-4)-1.44

MBPP pass@1 requires generated Python code to pass all unit tests. Each method selects its best rate from {1e-4, 3e-4} (3 seeds):

ModelLoRA-rank budgetGFA pass@1 (rate)LoRA pass@1 (rate)GFA minus LoRA
Qwen2.5-3B152.47 ± 0.42 (3e-4)53.07 ± 2.20 (3e-4)-0.60
Qwen2.5-3B1652.73 ± 1.22 (1e-4)53.60 ± 1.20 (3e-4)-0.87
Llama-3.2-3B147.93 ± 1.55 (1e-4)49.27 ± 0.50 (3e-4)-1.34
Llama-3.2-3B1645.07 ± 0.61 (3e-4)49.40 ± 0.40 (1e-4)-4.33

(d) Broader target modules (W1). On Llama-3.2-3B at query/value budgets (shared rate 1e-4, 3 seeds), GFA leads in 1/3 attention-scope comparisons (q,k,v,oq,k,v,o) and trails in all four all-linear comparisons (attention and MLP), by 0.63–1.72 points. At the all-linear rank-4 budget, LoRA's minimum rank of one gives it 43% more parameters than GFA.

Locations: Appendix F, Tables 16–17 (scopes and larger budgets); Appendix E, Tables 13–15 (GSM8K and MBPP).

AuthorsOpenReview ↗

Response to Reviewer 4xeQ (2/3)

W2 and Q2: Matched-budget ablations of groups, reshape dimensions, and module ordering.

Response to W2 and Q2: Full-rank global layouts win all 14 one-column and no-reshape comparisons: gains are 0.14–3.79 points from the rank cap alone and up to 15.15 without reshaping. Group count and module order change accuracy by less than one point.

The submitted Theorem 4.1 states that a pi×qip_i\times q_i module is generically full rank at any global rank if hi=k/gcd⁡(k,qi)≥min⁡(pi,qi)h_i=k/\gcd(k,q_i)\ge\min(p_i,q_i), where kk is the reshape width. The submission also measured reconstruction costs. We now add the following ablations. All tables report accuracy (%), mean ± standard deviation over 3 seeds; Q/L denote Qwen2.5-3B/Llama-3.2-3B.

(a) Number of groups (Q2). Each group has its own near-square factorization; ranks are split to match GFA's LoRA-rank-4 budget (shared rate 1e-4):

ModelGroupsARC-EPIQAWinoGMean
Q1 (GFA)94.53 ± 0.2986.65 ± 0.1984.29 ± 0.3688.49
Q494.56 ± 0.1486.63 ± 0.5384.35 ± 1.1188.51
QPer layer94.42 ± 0.2486.80 ± 0.6083.82 ± 0.4188.35
QPer module, capped*94.02 ± 0.3786.05 ± 0.6982.93 ± 0.6087.67
QPer module, full rank94.36 ± 0.1386.85 ± 0.6483.56 ± 0.7288.26
L1 (GFA)91.18 ± 0.0985.62 ± 0.1985.08 ± 1.1987.29
L490.87 ± 0.2885.55 ± 0.2784.74 ± 0.2087.05
LPer layer90.90 ± 0.1185.44 ± 0.3584.64 ± 0.3686.99
LPer module, full rank90.82 ± 0.1185.22 ± 0.7084.93 ± 0.5286.99

*At default width, each square query module forms one block, capping its rank at the group rank. Moving the width by one column restores full rank.

On Qwen, 1, 4, and per-layer groups are within 0.2 points; per-module full-rank factors perform similarly. We retain one group to minimize parameters per rank and avoid a grouping hyperparameter.

(b) Reshape dimensions (Q2). Without reshaping (kk equals module width, SuperLoRA's unreshaped single group), module ranks are at most 2rg2r_g, where rgr_g is global rank, and all updates share AA's row space. This test changes both properties; (c) isolates the rank cap. At rg=2r_g=2, budgets differ by at most 1.1%. Full-rank widths are 82,025 (Q) and 114,001 (L); each arm uses its own best rate:

ModelTaskNo reshapeFull-rank layoutGain
QARC-E93.03 ± 0.61 (3e-4)94.70 ± 0.07 (3e-4)+1.67
QOBQA85.87 ± 0.42 (1e-3)89.00 ± 0.60 (1e-4)+3.13
QWinoG73.64 ± 1.31 (3e-4)84.29 ± 0.55 (3e-4)+10.65
LARC-E89.31 ± 0.07 (1e-4)90.87 ± 0.37 (3e-4)+1.56
LOBQA79.53 ± 0.50 (3e-4)84.80 ± 1.04 (1e-4)+5.27
LWinoG68.72 ± 1.87 (1e-4)83.87 ± 0.18 (1e-4)+15.15

Among full-rank layouts, wider performs better: 83.71 at k/4k/4, 85.74 at near-square width kk, and 86.24 at 4k4k (Qwen, WinoGrande, rank-16 budget). We retain near-square width to maximize global rank per budget.

(c) One-column layout test (Q2). At Qwen attention scope, square width k=18,432=9×2,048k=18{,}432=9\times2{,}048 gives hi=9h_i=9 and rank at most 9rg9r_g. Width 18,431 permits full rank at the same global rank and budget. Each arm uses its own best rate:

rgr_gTaskSquare (rank 9rg9r_g)Width 18,431 (full rank)Gain
1ARC-C82.74 ± 0.9483.59 ± 0.85+0.85
1ARC-E93.64 ± 0.1594.25 ± 0.21+0.61
1OBQA86.20 ± 0.4088.07 ± 0.42+1.87
1WinoG75.53 ± 1.8179.32 ± 0.41+3.79
6ARC-C83.56 ± 0.1383.87 ± 0.73+0.31
6ARC-E94.22 ± 0.3994.36 ± 0.08+0.14
6OBQA88.67 ± 0.9989.40 ± 1.06+0.73
6WinoG81.14 ± 0.7583.16 ± 1.15+2.02

Selected rates are 3e-4 except WinoG and full-rank rg=6r_g=6 OBQA (1e-4).

Across six qq/oo rank caps at equal budget, full rank beats the lowest cap on both tasks. OBQA gains mostly between caps 9 and 38 (+1.27 of +1.87); ARC-E improves gradually up to 1,250 (+0.61 from cap 9 to full rank). The curves have dips within seed variation, consistent with the condition.

Across six models, three scopes, and two width rules, 7/36 near-square layouts violate the condition (none at query/value scope); k−1k-1 fixes all seven.

(d) Module ordering (Q2). Same budget and rate, Qwen WinoGrande:

Module orderAccuracyvs default
Lexicographic (default)85.74 ± 0.40default
Fixed random permutation85.11 ± 0.68-0.63
Random permutation per seed85.16 ± 0.72-0.58
Numeric layer order (q, v within layer)85.45 ± 0.87-0.29

The condition holds for every order: it depends only on kk and module widths.

Location: Appendix G.1, Tables 20–24, contains all four ablations and the layout survey.

AuthorsOpenReview ↗

Response to Reviewer 4xeQ (3/3)

Q3: Selection of the shared learning rate, an equally budgeted search, and matched dropout.

Response to Q3: With both methods tuned on the same learning-rate grid, GFA is on par with LoRA: GFA minus LoRA is -0.08 [-0.30, +0.11] (17/12/21, 50 cells). At the rank-one global budget GFA minus partial-module LoRA has a median of +0.52 on Qwen and +0.83 on Llama (ranges +0.02 to +4.66 and +0.06 to +13.42), while tuned GPart and Uni-LoRA lead GFA in all 6 cells there (by 0.07 to 1.40 and 0.31 to 1.73 points).

The submitted paper used one learning rate, 1e-4, for every method, and dropout 0.05 for LoRA and Tied-LoRA and 0 for the other methods. The revision adds the missing details, point by point:

(a) Selection (Q3). The shared rate 1e-4 was inherited from the LoRA configuration at the start of the project and was not tuned for any method; Section 5 now states this. Every tuned rate is chosen on the reported split, for all methods alike.

(b) Equally budgeted search (Q3). GFA and LoRA were re-run on the same grid, {1e-4, 3e-4, 1e-3} at the LoRA-rank-1 and rank-2 budgets and {1e-4, 3e-4} above, on ARC-C, ARC-E, OBQA, BoolQ, and PIQA (WinoGrande, the most expensive task, was not re-run). The first row gives the shared rate on the same cells (mean [95% interval], wins/ties/losses; 3 seeds per rate):

ComparisonGFA minus LoRA [95% CI] (win/tie/loss)
Same cells, one shared rate 1e-4 (as submitted)+0.68 [+0.41, +0.98] (43/2/5, 50 cells)
Same cells, each method at its own best rate on the same grid-0.08 [-0.30, +0.11] (17/12/21, 50 cells)
LoRA-rank-1 budget-0.05 [-0.27, +0.17] (4/2/4, 10 cells)
LoRA-rank-2 budget+0.16 [-0.19, +0.49] (7/0/3, 10 cells)
LoRA-rank-4 budget-0.04 [-0.26, +0.16] (3/4/3, 10 cells)
LoRA-rank-8 budget-0.28 [-0.49, -0.09] (1/4/5, 10 cells)
LoRA-rank-16 budget-0.20 [-0.46, +0.03] (2/2/6, 10 cells)

In 4 cells of the grid (all Qwen PIQA), a GFA seed diverged to chance accuracy at the highest rate; that rate is not used for GFA there (LoRA: 0 cells).

cc Dropout (Q3). LoRA with dropout 0 instead of 0.05, at the LoRA-rank-1, 4, and 16 budgets on ARC-C, ARC-E, and OBQA for both models, changes its accuracy by 0.01 points on average and by at most 0.27 (18 cells, 3 seeds).

(d) Smallest budgets, both tuned (Q3). Below the rank-one LoRA budget, the LoRA that fits is partial-module LoRA (rank one on a subset of modules). Both are tuned on the grid {1e-4, 3e-4, 1e-3}, and on ARC-C, ARC-E, and OBQA at global rank 1 each grid is extended one step past its edge (3e-3 for partial LoRA, 3e-5 for GFA on Llama). Per budget (3 seeds):

ModelBudgetGFA minus partial LoRA, shared rateBoth tuned
Qwen2.5-3Brank-one global (26K / 37.5K)+6.22 [+3.54, +9.42] (6/0/0, 6 cells)+1.17 [+0.19, +2.60] (4/2/0, 6 cells); median +0.52
Qwen2.5-3Babout 84K / 104K+4.55 [+2.50, +6.72] (6/0/0, 6 cells)+0.65 [+0.22, +0.98] (5/0/1, 6 cells); median +0.88
Llama-3.2-3Brank-one global (26K / 37.5K)+5.79 [+2.45, +10.85] (6/0/0, 6 cells)+2.89 [+0.36, +7.16] (5/1/0, 6 cells); median +0.83
Llama-3.2-3Babout 84K / 104K+2.22 [+1.10, +3.84] (6/0/0, 6 cells)+0.65 [-0.13, +1.62] (2/1/3, 6 cells); median -0.04

Chosen rates (24 cells): partial-module LoRA 1 at 3e-4, 19 at 1e-3, 4 at 3e-3; GFA 7 at 1e-4, 16 at 3e-4, 1 at 1e-3. Where partial-module LoRA's best is the top of its grid (17 cells), GFA's margin is an upper bound. Rates at which a seed diverged are excluded (GFA: 2 cells; partial-module LoRA: 2 cells, at 1e-2).

In the revised paper, Section 5 states how the shared rate was chosen, Section 6 opens with the equal grid (Table 3), and Appendix D gives the dropout control and the tuned smallest budgets (Tables 10 and 11).


We thank the reviewer again. The design ablation the reviewer asked for now carries the paper's account of GFA's accuracy: it shows that full-rank module updates, the property our layout condition isolates, win every cell of the one-column and no-reshape tests at the same budget, and the one-column test confirms the condition's prediction in all 8 cells.

In the reviewer's order, we added larger budgets, GSM8K, MBPP, and broader target scopes (W1 and Q1); added matched-budget ablations of groups, reshape width, no reshape, and module order (W2 and Q2); and stated the learning-rate choice and added an equal grid, a dropout-matched LoRA, and tuned the smallest budgets (Q3).

If these results resolve the reviewer's concerns, we would be grateful if the reviewer would reconsider their evaluation. Please let us know which point is most important to you, and we will prioritize it; we also welcome any further questions.

Reviewer_cedgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies Global Factorized Adaptation (GFA), a single-group, matrix-reshape configuration of SuperLoRA without projection. It factorizes a global matrix and reconstructs module updates through slicing and reshaping. The paper provides rank and representational-separation analyses and evaluates Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct on six tasks across five parameter budgets.

Critical strengths
  1. The method is simple and supports merging updates into the original weights.

  2. The paper includes rank analysis and evaluates multiple parameter budgets with three seeds.

Critical weaknesses

W1. Limited novelty. GFA corresponds to SuperLoRA's single-group, two-dimensional reshape configuration without projection, as acknowledged by the authors. SuperLoRA already evaluated this configuration. The contribution beyond applying it to language models and adding structural analysis is limited.

W2. Missing design ablations. The paper does not compare single versus multiple groups, alternative reshape layouts, or projection choices at matched budgets. The theory supports parameter counting and rank properties, but does not explain the accuracy gains. Appendix C compares reconstruction implementations rather than the principal design choices.

W3. Limited model coverage. Experiments use only two approximately 3B instruction-tuned models, and all six tasks are English multiple-choice or binary classification. Results on larger models and generative tasks are missing. Related work such as Uni-LoRA and VB-LoRA includes 7B/13B models and broader task coverage.

W4. Inconsistent parameter counts and notation. Table 3 reports 84,096 and 104,268 GFA parameters for Qwen and Llama. Under the formula PGFA=rg(m+k)P_{\mathrm{GFA}}=r_g(m+k) (p. 4, l. 213) and the layouts in Table 10, these imply non-integer ranks of approximately 3.226 and 2.777. The configuration used for these results is therefore unclear. Figure 1(a) also labels the global matrix as N×N\sqrt N\times\sqrt N and the vector as θN\theta_N, although NN denotes the module count and DD denotes the coordinate count. Figure 2 uses lowercase nn in the final slice interval while the text uses uppercase NN. These inconsistencies should be corrected.

Questions for authors
  1. What exact layouts, target scopes, and integer global ranks produced the GFA parameter counts in Table 3?

  2. What matched-budget ablations support the single-group layout and absence of projection, and which design choice accounts for the observed gains?

4 comments
AuthorsOpenReview ↗

Response to Reviewer cedg (1/3)

We thank the reviewer for the careful assessment and for recognizing that the paper includes rank analysis and evaluates multiple parameter budgets with three seeds. We answer every question below. In particular, we added:

  • a test of a square-reshape failure (SuperLoRA's, also our ⌊D⌋\lfloor\sqrt{D}\rfloor default) that our layout condition predicts, and a survey over six open models (W1);
  • matched-budget ablations of the number of groups, the reshape layout, the module order, and a fixed random projection, on two models (W2 and Q2);
  • GSM8K math reasoning, MBPP code generation, and Qwen2.5-7B-Instruct (W3);
  • the exact layouts and integer global ranks behind the Table 3 counts, and unified notation in Figures 1 to 4 (W4 and Q1).

W1: Novelty relative to SuperLoRA.

Response to W1: As the submitted paper stated, GFA's parameterization is SuperLoRA's single-group, matrix-reshape, no-projection configuration. SuperLoRA evaluated it, mainly on vision models, and found that reshaping helps, but it does not say when the reshape keeps a module update full rank. Our contribution is that rule (Theorem 4.1). The rule can be checked before training, and it predicts that the square reshape (SuperLoRA's, and also our default width) caps every module at rank 9rg9r_g at attention scope on Qwen2.5-3B. A one-column change restores full rank and wins all 8 tested cells, by +0.14 to +3.79 points. It also bears on SuperLoRA's reshape advantage: removing the reshape caps module rank and costs 1.56 to 15.15 points, though it also puts every module update in one row space (W2).

The submitted paper gave this rule with its proof. The revision adds the missing details, point by point:

(a) When the reshape preserves module rank. GFA learns one global matrix M=BAM=BA of size m×km\times k and global rank rgr_g, reads MM row by row, and gives each module a contiguous slice reshaped to its pi×qip_i\times q_i shape. A module's rows then start at hi=k/gcd⁡(k,qi)h_i=k/\gcd(k,q_i) distinct positions within the rows of MM. If hi≥min⁡(pi,qi)h_i\ge\min(p_i,q_i), every module update is generically full rank, for any module order; without a reshape (k=qik=q_i) it has rank at most 2rg2r_g. To our knowledge, this condition has not been stated before; we relate it to Kronecker adapters (KronA, LoKr) and the Van Loan rearrangement.


(b) A failure of the square reshape that the condition predicts. SuperLoRA's square-like reshape is full rank at qq/vv scope, the scope it evaluated. At attention scope (q,k,v,oq,k,v,o) on Qwen2.5-3B, D=18,4322D=18{,}432^2 adapted coordinates, so the square width, which our own default k=⌊D⌋k=\lfloor\sqrt{D}\rfloor also gives, is k=18,432k=18{,}432, giving hi=9h_i=9: every module update has rank exactly 9rg9r_g. The width k=18,431k=18{,}431 is full rank at the same global rank and one extra parameter per unit of rgr_g; the revision defines the width as the largest such k≤⌊D⌋k\le\lfloor\sqrt{D}\rfloor (equal to ⌊D⌋\lfloor\sqrt{D}\rfloor for every qq/vv layout we train). Each arm at its own best of 1e-4 and 3e-4; WinoGrande at 1e-4 only (3 seeds):

rgr_gTaskSquare, k=18,432k=18{,}432 (rank 9rg9r_g)k=18,431k=18{,}431 (full rank)Gain
1ARC-C82.74 ± 0.94 (3e-4)83.59 ± 0.85 (3e-4)+0.85
1ARC-E93.64 ± 0.15 (3e-4)94.25 ± 0.21 (3e-4)+0.61
1OBQA86.20 ± 0.40 (3e-4)88.07 ± 0.42 (3e-4)+1.87
1WinoG75.53 ± 1.81 (1e-4)79.32 ± 0.41 (1e-4)+3.79
6ARC-C83.56 ± 0.13 (3e-4)83.87 ± 0.73 (3e-4)+0.31
6ARC-E94.22 ± 0.39 (3e-4)94.36 ± 0.08 (3e-4)+0.14
6OBQA88.67 ± 0.99 (3e-4)89.40 ± 1.06 (1e-4)+0.73
6WinoG81.14 ± 0.75 (1e-4)83.16 ± 1.15 (1e-4)+2.02

cc How common the failure is. Over six open models (Qwen2.5-3B/7B, Llama-3.2-3B, Llama-3.1-8B, Mistral-7B, Gemma-2-2B), three scopes, and both square-width rules, 7 of 36 layouts violate the condition. The severe cases are the attention scopes of Qwen2.5-3B and Qwen2.5-7B (rank 9rg9r_g and 8rg8r_g). No qq/vv layout violates it, and in every violating case the width k−1k-1 satisfies it.


(d) Matched-budget comparison with sharing methods on decoder LLMs. The paper compares GFA with LoRA and five sharing baselines (Tied-LoRA, SingLoRA, VB-LoRA, a permutation-based Uni-LoRA variant, and GPart) at five matched budgets, and the faithful Uni-LoRA at the rank-one budget (Table 7, shared rate); SuperLoRA compares against LoRA and other non-sharing adapters. With each method at its own best rate at the rank-one global budget (ARC-C, ARC-E, and OBQA, chosen rates in Table 7), GFA minus GPart is -1.40 to -0.07, and GFA minus Uni-LoRA is -1.73 to -0.31.

In the revised paper, Sec. 2 states the relation to SuperLoRA, Appendix L relates the condition to prior rank results, and Appendix G.1 gives the layout survey and the attention-scope test (Table 23).

AuthorsOpenReview ↗

Response to Reviewer cedg (2/3)

W2 and Q2: Matched-budget ablations of groups, reshape, and projection, and which design choice accounts for the gains.

Response to W2 and Q2: In the matched-budget tests, among global layouts at the same budget, full-rank layouts win all 14 cells of the one-column and no-reshape tests: the one-column test, which changes rank alone, gains 0.14 to 3.79 points, and removing the reshape, which also puts every module update in one shared row space, costs 1.56 to 15.15 points. Among full-rank layouts, the width moves WinoGrande by up to 2.5 points, the number of groups and the module order move average accuracy by less than one point, and with a fixed random projection in place of the learned factor (GPart), GFA minus GPart is -0.48 to +0.34 on Qwen at the about-84K and LoRA-rank-4 budgets and -0.57 to -0.46 on Llama at the LoRA-rank-4 budget, each method at its best rate.

The submitted paper compared reconstruction implementations (in Appendix C) and gave the rank theory. The revision adds the missing ablations, point by point, all at matched budgets:

(a) Reshape layout (W2). Without a reshape (k=qik=q_i, SuperLoRA's unreshaped single group), module rank is capped at 2rg2r_g. Each arm at its own best rate (3 seeds):

ModelTaskNo reshape (rank capped)Full-rank layoutGap
Qwen2.5-3BARC-E93.03 ± 0.61 (3e-4)94.70 ± 0.07 (3e-4)+1.67
Qwen2.5-3BOBQA85.87 ± 0.42 (1e-3)89.00 ± 0.60 (1e-4)+3.13
Qwen2.5-3BWinoG73.64 ± 1.31 (3e-4)84.29 ± 0.55 (3e-4)+10.65
Llama-3.2-3BARC-E89.31 ± 0.07 (1e-4)90.87 ± 0.37 (3e-4)+1.56
Llama-3.2-3BOBQA79.53 ± 0.50 (3e-4)84.80 ± 1.04 (1e-4)+5.27
Llama-3.2-3BWinoG68.72 ± 1.87 (1e-4)83.87 ± 0.18 (1e-4)+15.15

Over six qq/oo rank caps at the same budget, full rank beats the lowest cap on both tasks, on OBQA mostly between caps 9 and 38 (+1.27 of +1.87) and on ARC-E gradually up to 1,250 (+0.61 from cap 9 to full rank); the curve is not monotone (dips within seed variation), consistent with the condition.

Among full-rank layouts, accuracy rises with width: 83.71 at k/4k/4, 85.74 near-square, and 86.24 at 4k4k (Qwen, WinoGrande, LoRA-rank-16 budget, 3 seeds). The near-square layout is used because it gives the largest global rank at a given budget.


(b) Single versus multiple groups (W2 and Q2). One global group, 4 groups, one group per layer, and one group per module perform alike once no group's rank is capped (LoRA-rank-4 budget, shared rate 1e-4, 3 seeds):

ModelGroupsARC-EPIQAWinoGMean
Qwen2.5-3B1 (GFA)94.53 ± 0.2986.65 ± 0.1984.29 ± 0.3688.49
Qwen2.5-3B494.56 ± 0.1486.63 ± 0.5384.35 ± 1.1188.51
Qwen2.5-3Bone per layer94.42 ± 0.2486.80 ± 0.6083.82 ± 0.4188.35
Qwen2.5-3Bone per module, rank-capped*94.02 ± 0.3786.05 ± 0.6982.93 ± 0.6087.67
Qwen2.5-3Bone per module, full rank94.36 ± 0.1386.85 ± 0.6483.56 ± 0.7288.26
Llama-3.2-3B1 (GFA)91.18 ± 0.0985.62 ± 0.1985.08 ± 1.1987.29
Llama-3.2-3B490.87 ± 0.2885.55 ± 0.2784.74 ± 0.2087.05
Llama-3.2-3Bone per layer90.90 ± 0.1185.44 ± 0.3584.64 ± 0.3686.99
Llama-3.2-3Bone per module, full rank90.82 ± 0.1185.22 ± 0.7084.93 ± 0.5286.99

*At the default width, a square qq module is its own block and is capped at the group rank; the full-rank arm moves the width by one column to separate "no sharing" from "rank cap".


cc Projection and module order (W2 and Q2). GPart spends the same budget on a trainable vector mapped into the same update space by a fixed random projection. At each method's best rate on Qwen, GFA minus GPart is -0.48 to +0.34 points at the about-84K and LoRA-rank-4 budgets, and +0.53 to +1.72 at the LoRA-rank-1 budget, where GPart's search stopped at 1e-3, so those gaps are upper bounds (3 seeds). On Llama at the LoRA-rank-4 budget (GFA over 3e-5, 1e-4, and 3e-4, GPart over 1e-4, 1e-3, and 3e-3, its OBQA best at the top; 3 seeds):

Task (Llama, rank-4 budget)GFA (rate)GPart (rate)GFA minus GPart
ARC-C78.21 ± 0.26 (1e-4)78.78 ± 0.34 (1e-3)-0.57
ARC-E90.94 ± 0.05 (1e-4)91.40 ± 0.27 (1e-3)-0.46
OBQA84.93 ± 0.64 (1e-4)85.47 ± 0.92 (3e-3)-0.53

A fixed random module permutation gives 85.11 ± 0.68 versus 85.74 ± 0.40 for the default order (WinoGrande, 3 seeds); within-seed variation and the layout condition hold for every order.

The single group and the near-square layout are the cheapest full-rank choice: fewest parameters per unit of rank and no grouping hyperparameter. In the revised paper, Appendix G.1 gives these ablations (Tables 21, 22, and 24) and the projection comparison (Tables 25 and 26).

AuthorsOpenReview ↗

Response to Reviewer cedg (3/3)

W3: Larger models and generative tasks.

Response to W3: We added both. On generation, LoRA is ahead: GFA trails LoRA on GSM8K on both models at the LoRA-rank-1 budget, each tuned, and in all four MBPP cells. On Qwen2.5-7B, GFA minus LoRA is -0.80 to +0.40 points at the LoRA-rank-1 budget and -0.86 to -0.12 at the LoRA-rank-16 budget (3 seeds).

The submitted paper evaluated two 3B instruction-tuned models on six multiple-choice or binary tasks. The revision adds, point by point:

(a) Generation (W3). GSM8K (exact match of the final answer) at the LoRA-rank-1 budget and MBPP (pass@1 in %, by running each generated program against its unit tests), each method at its own best rate (3 seeds):

ModelLoRA-rank budgetGFA (rate)LoRA (rate)GFA minus LoRA
Qwen2.5-3B165.78 ± 0.53 (1e-4)66.24 ± 0.43 (1e-4)-0.46
Llama-3.2-3B166.19 ± 1.32 (1e-4)67.63 ± 0.50 (1e-4)-1.44
ModelLoRA-rank budgetGFA pass@1 (rate)LoRA pass@1 (rate)GFA minus LoRA
Qwen2.5-3B152.47 ± 0.42 (3e-4)53.07 ± 2.20 (3e-4)-0.60
Qwen2.5-3B1652.73 ± 1.22 (1e-4)53.60 ± 1.20 (3e-4)-0.87
Llama-3.2-3B147.93 ± 1.55 (1e-4)49.27 ± 0.50 (3e-4)-1.34
Llama-3.2-3B1645.07 ± 0.61 (3e-4)49.40 ± 0.40 (1e-4)-4.33

At the shared rate of 1e-4, GFA minus LoRA (Qwen, Llama) is +4.66 and +0.40 on MBPP at the LoRA-rank-1 budget, -0.54 and -4.33 on MBPP at the LoRA-rank-16 budget, and +0.15 and -0.73 on GSM8K at the LoRA-rank-16 budget (3 seeds).


(b) A larger model (W3). Qwen2.5-7B-Instruct at matched budgets (3 seeds). A 13B model's FP16 weights alone (26 GB) exceed our 24 GB GPUs.

LoRA-rank budgetTaskGFA (rate)LoRA (rate)GFA minus LoRA
1ARC-C90.19 ± 0.17 (1e-4)89.99 ± 0.18 (1e-4)+0.20
1OBQA92.80 ± 0.72 (1e-4)93.60 ± 0.00 (3e-4)-0.80
1ARC-E96.58 ± 0.35 (3e-4)96.18 ± 0.14 (3e-4)+0.40
16ARC-C90.27 ± 0.76 (1e-4)90.39 ± 0.21 (3e-4)-0.12
16OBQA92.87 ± 0.46 (1e-4)93.73 ± 0.61 (1e-4)-0.86
16ARC-E96.63 ± 0.55 (1e-4)97.05 ± 0.00 (3e-4)-0.42

In the revised paper, Appendix E gives the generation results (Tables 13, 14, and 15), and Appendix F gives the 7B results (Table 19).


W4 and Q1: Layouts and integer global ranks behind the Table 3 counts; notation.

Response to W4 and Q1: The runs of the submitted Table 3 (now Table 6) use integer rg=1r_g=1 on the qq/vv scope with a wide layout, and P=rg(m+k)P=r_g(m+k) gives the printed counts exactly.

The submitted paper gave the near-square layouts used at the five budget anchors (Table 10). The revision adds the layout of the sub-unit runs, point by point:

(a) Exact layouts (Q1). The widths were set to match comparator counts (VeRA on Qwen, the GPart control on Llama):

ModelModulesm×km\times krgr_grg(m+k)r_g(m+k)
Qwen2.5-3B722,071 × 82,025184,096
Llama-3.2-3B563,497 × 100,7711104,268

The values 3.226 and 2.777 arise from dividing by the near-square unit m+km+k (26,067 and 37,541). Both wide layouts satisfy the layout condition (gcd⁡(k,qi)=1\gcd(k,q_i)=1), so every module update is generically full rank.

(b) Notation (W4). Figure 1 uses θD\theta_D for the vector of all DD update coordinates, Mm×kM_{m\times k} for the global matrix, and φ\varphi for the concatenated LoRA factors; NN counts modules. Figure 2 writes the intervals as [c0,c1),…,[cN−1,cN)[c_0,c_1),\dots,[c_{N-1},c_N). Rank is rℓr_\ell for LoRA and rgr_g for GFA throughout.

In the revised paper, Table 30 (formerly Table 10) states which runs use its layouts, and Appendix C gives the wide layouts next to Table 6 (formerly Table 3).


We thank the reviewer again. The matched-budget ablations the reviewer asked for are now in Appendix G.1, and they show that, among global layouts at the same budget, full-rank layouts win all 14 cells of the one-column and no-reshape tests, while changing the number of groups and module order changes accuracy by less than one point.

We clarified the contribution beyond SuperLoRA and tested its prediction (W1), added the matched-budget ablations (W2 and Q2), added GSM8K, MBPP, and a 7B model (W3), and stated the exact layouts and unified the notation (W4 and Q1).

If the exact parameter counts (W4) and the matched-budget ablations (W2) now address those points, we would be grateful if the reviewer would reconsider their evaluation.

AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer cedg,

Thank you for your time and careful assessment of our work. As the discussion period ends soon, we would like to check whether our responses have addressed your concerns. We would be happy to clarify anything further before the discussion closes.

Paper19882 Authors

Reviewer_nywxOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Global Factorized Adaptation (GFA), a parameter-efficient fine-tuning method that replaces independent module-wise LoRA factors with a single pair of globally shared low-rank factors. The resulting global matrix is partitioned and reshaped to reconstruct individual module updates. The authors provide theoretical analysis of the representational properties of GFA and evaluate it on two 3B language models across six downstream tasks. The results show that GFA generally outperforms LoRA under matched parameter budgets, particularly in low-budget settings.

Critical strengths
  1. Parameter efficiency. GFA enables fine-grained control over the trainable parameter budget through global factor sharing and demonstrates competitive performance even at extremely small parameter budgets.
  2. Meaningful theoretical insights. The paper provides a theoretical analysis showing that globally low-rank factors can produce full-rank module updates after slicing and reshaping, offering useful insights into the representational properties of global parameter sharing.
Critical weaknesses
  1. Limited novelty. The core idea of GFA is relatively simple, relying on a single pair of globally shared low-rank factors to generate updates for all target modules, with limited additional architectural innovation. Moreover, sharing the same factors across different layers and modules may introduce gradient conflicts and optimization interference, which are not sufficiently investigated.
  2. Marginal performance gains and limited experimental validation. Under comparable parameter budgets, GFA improves average performance on Llama-3.2-3B by only 0.47 percentage points over LoRA, without clear advantages in training efficiency. Furthermore, the experiments are limited to two 3B models and six multiple-choice or binary classification tasks, lacking evaluations on more challenging tasks such as code generation and mathematical reasoning. This limits the evidence for the method's broader effectiveness.
Questions for authors
  1. Optimization interference: Does sharing global factors across all target modules introduce gradient conflicts or optimization interference between different layers? Could the authors provide further analysis or ablation studies?
  2. Practical efficiency: GFA incurs higher training time than LoRA in some configurations, while exhibiting similar peak memory usage. What are its practical efficiency advantages over standard LoRA?
  3. Generalizability: Could the authors provide additional evaluations on code generation, mathematical reasoning, or larger language models to demonstrate the generalizability of GFA?
6 comments
AuthorsOpenReview ↗

Response to Reviewer nywx (1/3)

We thank the reviewer for recognizing the meaningful theoretical insights into the representational properties of global parameter sharing. The revision adds:

  • a layout-condition test: the full-rank layout wins all 8 comparisons (W1);
  • gradient measurements, a coupling lemma, and a no-sharing intervention (W1, Q1);
  • matched learning-rate searches and tuned small-budget comparisons (W2);
  • memory, time, storage, and adapter-switching comparisons (W2, Q2);
  • GSM8K, MBPP, and Qwen2.5-7B-Instruct (W2, Q3).

Changes are marked in blue.


W1 (first part): Novelty beyond a simple shared factorization.

Response to W1 (first part): GFA is simple by design. Its parameterization is SuperLoRA's single-group, matrix-reshape, no-projection configuration, as acknowledged in the submission. Our contribution is a condition, checkable before training, explaining when it combines fine-grained budgets with full-rank module updates.

For a global matrix M=BAM=BA of width kk and rank rgr_g, a contiguous slice reshaped into a pi×qip_i\times q_i module encounters hi=k/gcd⁡(k,qi)h_i=k/\gcd(k,q_i) row offsets. It is generically full rank if hi≥min⁡(pi,qi)h_i\ge\min(p_i,q_i) (Theorem 4.1). To our knowledge, prior work does not state this condition.

(a) Relation to prior results. KronA, LoKr, and the Van Loan rearrangement also obtain full-rank updates from low-rank objects, but impose a per-module structure. SuperLoRA observed benefits from reshaping without characterizing the rank retained by each module.


(b) A tested prediction. For Qwen2.5-3B attention (q,k,v,oq,k,v,o), D=18,4322D=18{,}432^2. Square width k=18,432k=18{,}432 gives hi=9h_i=9 and module rank 9rg9r_g, verified numerically. Width 18,43118{,}431 permits full rank at the same global rank. Across six models, three scopes, and two width rules, 7/36 layouts violate the condition; k−1k-1 fixes all seven. No qq/vv layout violates it.

Accuracy (%), mean ± standard deviation over 3 seeds; each arm uses its best rate from {1e-4, 3e-4}:

rgr_gTaskSquare (rank 9rg9r_g)One column narrower (full rank)Gain
1ARC-C82.74 ± 0.9483.59 ± 0.85+0.85
1ARC-E93.64 ± 0.1594.25 ± 0.21+0.61
1OBQA86.20 ± 0.4088.07 ± 0.42+1.87
1WinoG75.53 ± 1.8179.32 ± 0.41+3.79
6ARC-C83.56 ± 0.1383.87 ± 0.73+0.31
6ARC-E94.22 ± 0.3994.36 ± 0.08+0.14
6OBQA88.67 ± 0.9989.40 ± 1.06+0.73
6WinoG81.14 ± 0.7583.16 ± 1.15+2.02

Selected rates are 3e-4 except WinoG and the full-rank rg=6r_g=6 OBQA arm (1e-4).

Locations: Appendix G.1, Table 23 (survey and test); Appendix L (prior results).

W1 (second part) and Q1: Gradient conflicts and optimization interference from sharing.

Response to W1 (second part) and Q1: Removing sharing does not improve accuracy in the tested setting, and measured module gradients show no substantial cancellation. This does not rule out interference generally.

(a) Intervention. At the LoRA-rank-4 budget and shared rate 1e-4, separate full-rank factorizations per module do not outperform GFA (%, 3 seeds):

ModelFactorizationARC-EPIQAWinoGMean
Qwen2.5-3BGlobal94.53 ± 0.2986.65 ± 0.1984.29 ± 0.3688.49
Qwen2.5-3BPer module94.36 ± 0.1386.85 ± 0.6483.56 ± 0.7288.26
Llama-3.2-3BGlobal91.18 ± 0.0985.62 ± 0.1985.08 ± 1.1987.29
Llama-3.2-3BPer module90.82 ± 0.1185.22 ± 0.7084.93 ± 0.5286.99

Four and 36 groups likewise give Qwen means of 88.51 and 88.35 versus GFA's 88.49.


(b) Measurement during training. Only 71/13,034 rows of BB cross module boundaries on Qwen, so interaction is mainly through AA. We decompose ∇AL=∑igi\nabla_A\mathcal L=\sum_i g_i exactly, matching autograd to machine precision, and measure pairwise cosines and ∥∑igi∥/∑i∥gi∥\|\sum_i g_i\|/\sum_i\|g_i\| every 10 steps.

Across ARC-C/OBQA, budgets 1/4/16, and 3 seeds, the final 20% of shared-rate training gives median cosine approximately zero, spread 1.01–1.05 times random directions, and no pair below −0.1. The norm ratio matches its orthogonal reference: 0.181–0.182 on Llama and 0.139–0.140 on Qwen. Near-zero cosines are expected in high dimensions, so we also use random references and the intervention.


(c) Theory. Under Lemma O.1's initialization assumptions, module-gradient pairs have zero expected inner product and normalized inner-product RMS at most 1/rg1/\sqrt{r_g}.


(d) Stability. GFA can collapse at its highest searched rates; partial-module LoRA collapses at 1e-2. At 3e-4 on Qwen PIQA, rank-4 budget, GFA collapses in 1/3 seeds versus 0/3 without sharing. Three seeds do not establish a difference, but GFA's rate search should be centered lower.

Locations: Appendix O, Table 38 and Lemma O.1; Appendix G.1, Table 21 (intervention).

AuthorsOpenReview ↗

Response to Reviewer nywx (2/3)

W2 (first part): Size of the gains over LoRA.

Response to W2 (first part): Below the rank-one LoRA budget, with both methods tuned, GFA minus partial-module LoRA has a median of +0.52 on Qwen and +0.83 on Llama (ranges +0.02 to +4.66 and +0.06 to +13.42), while tuned GPart and Uni-LoRA lead GFA in all 6 cells there (by 0.07 to 1.40 and 0.31 to 1.73 points). At budgets, full LoRA can reach; with both methods tuned on the same learning-rate grid, GFA is on par with LoRA: GFA minus LoRA is -0.08 [-0.30, +0.11] (17/12/21, 50 cells) (points, 95% interval, wins/ties/losses with ties below 0.1).

The submitted paper reported, at one learning rate of 1e-4 shared by every method, GFA ahead of LoRA on both models; with one bootstrap and tie rule used throughout the revision, GFA minus LoRA is +1.25 [+0.71, +1.94] points on Qwen and +0.47 [+0.23, +0.76] on Llama, ahead in 35 of the 36 cells at the three smallest budgets and tied in the other.

(a) Equal learning-rate search. The rate 1e-4 was inherited from the LoRA configuration (now stated in Sec. 5). We re-ran GFA and LoRA on the same grid ({1e-4, 3e-4, 1e-3} at the LoRA-rank-1 and rank-2 budgets, {1e-4, 3e-4} above), each at its own best rate (best three-seed mean), on ARC-C, ARC-E, OBQA, BoolQ, and PIQA (WinoGrande was not re-run). GFA minus LoRA (3 seeds per arm):

ComparisonGFA minus LoRA [95% CI] (win/tie/loss)
Same cells, one shared rate 1e-4 (as submitted)+0.68 [+0.41, +0.98] (43/2/5, 50 cells)
Same cells, each method at its own best rate on the same grid-0.08 [-0.30, +0.11] (17/12/21, 50 cells)
of which Qwen2.5-3B-0.11 [-0.49, +0.17] (8/9/8, 25 cells)
of which Llama-3.2-3B-0.05 [-0.28, +0.20] (9/3/13, 25 cells)

Per budget: our response to Reviewer 4xeQ (Q3 (b)).

In 4 cells of the grid (all Qwen PIQA), a GFA seed diverged to chance accuracy at the highest rate; we do not use that rate for GFA there (LoRA: 0 cells).


(b) Smallest budgets. One unit of GFA global rank costs 26,067 parameters on Qwen (37,541 on Llama), while rank-one LoRA on the same modules costs 230,400 (286,720). Below that, the only LoRA that fits is partial-module LoRA, which adapts a subset of the modules. GFA minus partial-module LoRA, at the shared rate and with both tuned (3 seeds):

ModelBudgetGFA minus partial LoRA, shared rateBoth tuned
Qwen2.5-3Brank-one global (26K / 37.5K)+6.22 [+3.54, +9.42] (6/0/0, 6 cells)+1.17 [+0.19, +2.60] (4/2/0, 6 cells); median +0.52
Qwen2.5-3Babout 84K / 104K+4.55 [+2.50, +6.72] (6/0/0, 6 cells)+0.65 [+0.22, +0.98] (5/0/1, 6 cells); median +0.88
Llama-3.2-3Brank-one global (26K / 37.5K)+5.79 [+2.45, +10.85] (6/0/0, 6 cells)+2.89 [+0.36, +7.16] (5/1/0, 6 cells); median +0.83
Llama-3.2-3Babout 84K / 104K+2.22 [+1.10, +3.84] (6/0/0, 6 cells)+0.65 [-0.13, +1.62] (2/1/3, 6 cells); median -0.04

Chosen rates (24 cells): partial-module LoRA 1 at 3e-4, 19 at 1e-3, 4 at 3e-3; GFA 7 at 1e-4, 16 at 3e-4, 1 at 1e-3. Where partial-module LoRA's best is the top of its grid (17 cells), GFA's margin is an upper bound.

Rates at which a seed diverged are excluded (GFA: 2 cells; partial-module LoRA: 2 cells, at 1e-2).

In the revised paper, Section 6 opens with the equal-grid comparison (Table 3) before the shared-rate results, and Appendix D gives the smallest budgets (Table 11).


W2 (second part) and Q2: Practical efficiency advantages.

Response to W2 (second part) and Q2: GFA's practical advantages over LoRA are the fine-grained budget control the reviewer highlights and adapter size; its training cost is close to LoRA's, with equal peak memory.

The submitted paper reported, over 179 paired GFA and LoRA runs, a median peak-memory ratio to LoRA of 1.000, median wall-clock 1.03 times LoRA's on Qwen and 1.09 times on Llama, and no inference cost after merging.

(a) Source of the overhead. The median peak-memory ratio is 1.000 on both models. The extra time comes from one extra dense matrix product per adapted module, so it grows with scope: at all-linear scope on Llama, the median training time is 1.5 to 2.0 times LoRA's (Appendix F).

(b) Finer budgets with every module adapted. GFA's budget moves in steps of 26,067 parameters on Qwen, and every module stays adapted at every step; the smallest full LoRA adapter has 230,400. At these budgets, the comparison is with partial-module LoRA (table above).

(c) Smaller adapters. At rg=1r_g=1 a GFA adapter is 52 KB in FP16 on Qwen (75 KB on Llama), against 461 KB (573 KB) for rank-one LoRA, which matters when many task adapters are stored or transferred; switching merged adapters stays cheap (one global product, 1.54 ms).

In the revised paper, Section 6 and Appendix H give these numbers (Tables 27, 28, and 29).

AuthorsOpenReview ↗

Response to Reviewer nywx (3/3)

W2 (third part) and Q3: Mathematical reasoning, code generation, and larger models.

Response to W2 (third part) and Q3: We added all three: GSM8K for mathematical reasoning, MBPP for code generation, and Qwen2.5-7B-Instruct as a larger model. On GSM8K and MBPP, LoRA is ahead of GFA; the revised abstract, introduction, and conclusion state the GSM8K result.

The submitted paper evaluated two 3B instruction-tuned models on six multiple-choice or binary tasks. The revision adds the missing details, point by point:

(a) Mathematical reasoning (GSM8K). Exact match of the final number in the generated solution, on the 1,319 test problems. At the shared rate, over the five budgets from LoRA rank 1 to 16, GFA trails LoRA in 9 of 10 cells, by 0.77 points on average [-1.21, -0.35]. With both tuned over {1e-4, 3e-4} at the LoRA-rank-1 budget, accuracy in % (3 seeds):

ModelLoRA-rank budgetGFA (rate)LoRA (rate)GFA minus LoRA
Qwen2.5-3B165.78 ± 0.53 (1e-4)66.24 ± 0.43 (1e-4)-0.46
Llama-3.2-3B166.19 ± 1.32 (1e-4)67.63 ± 0.50 (1e-4)-1.44

GFA reaches a lower training loss than LoRA in all ten shared-rate cells, so it fits the data; only about 2.7% of the supervised tokens are the graded final answer.


(b) Code generation (MBPP). Trained on the 374 training problems; pass@1 in % on the 500 test problems, where a solution counts only if it passes all of the problem's unit tests when executed; each method at its own best of {1e-4, 3e-4} (3 seeds):

ModelLoRA-rank budgetGFA pass@1 (rate)LoRA pass@1 (rate)GFA minus LoRA
Qwen2.5-3B152.47 ± 0.42 (3e-4)53.07 ± 2.20 (3e-4)-0.60
Qwen2.5-3B1652.73 ± 1.22 (1e-4)53.60 ± 1.20 (3e-4)-0.87
Llama-3.2-3B147.93 ± 1.55 (1e-4)49.27 ± 0.50 (3e-4)-1.34
Llama-3.2-3B1645.07 ± 0.61 (3e-4)49.40 ± 0.40 (1e-4)-4.33

On Qwen both gaps are within one seed standard deviation of LoRA; on Llama, LoRA is clearly ahead.


(c) A larger model (Qwen2.5-7B-Instruct). Matched budgets of 283,843 (GFA) against 315,392 (LoRA) and 5,028,076 against 5,046,272 parameters, each method at its own best of two rates, accuracy in % (3 seeds). Its qq/vv layout satisfies the layout condition (hi=10,137≥3,584h_i=10{,}137\ge3{,}584).

At the LoRA-rank-1 budget there is no consistent difference between GFA and LoRA (GFA minus LoRA -0.80 to +0.40, 3 cells). At the LoRA-rank-16 budget LoRA is ahead of GFA (GFA minus LoRA -0.86 to -0.12, 3 cells). Accuracy in % (3 seeds):

LoRA-rank budgetTaskGFA (rate)LoRA (rate)GFA minus LoRA
1ARC-C90.19 ± 0.17 (1e-4)89.99 ± 0.18 (1e-4)+0.20
1OBQA92.80 ± 0.72 (1e-4)93.60 ± 0.00 (3e-4)-0.80
1ARC-E96.58 ± 0.35 (3e-4)96.18 ± 0.14 (3e-4)+0.40
16ARC-C90.27 ± 0.76 (1e-4)90.39 ± 0.21 (3e-4)-0.12
16OBQA92.87 ± 0.46 (1e-4)93.73 ± 0.61 (1e-4)-0.86
16ARC-E96.63 ± 0.55 (1e-4)97.05 ± 0.00 (3e-4)-0.42

In the revised paper, Appendix E gives GSM8K and MBPP (Tables 13, 14, and 15), and Appendix F gives the 7B results (Table 19).


We thank the reviewer again. The interference question the reviewer raised is now a dedicated appendix with a gradient measurement, a lemma, and an intervention, which together show no gradient conflict from sharing, and three seeds of the stability control do not settle whether sharing lowers the stable learning rate.

In the reviewer's order, we tested a prediction of the layout condition (W1), added the interference analysis (W1 and Q1), an equal learning-rate search and tuned smallest-budget comparisons (W2), a memory, time, storage, and switching comparison (Q2), and GSM8K, MBPP code generation, and Qwen2.5-7B-Instruct (W2 and Q3).

If these results resolve the reviewer's concerns, we would be grateful if the reviewer would reconsider their evaluation. Please let us know which point is most important to you, and we will prioritize it; we also welcome any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all three reviewers. The revision separates the benefits of the global layout from those of learning-rate tuning. GFA offers finer budget control, while its accuracy depends on both the layout and the tuning protocol. Changes are marked in blue.

1. Contribution and layout analysis. We explicitly credit SuperLoRA for the parameterization. Our contribution is analyzing when a global layout permits full-rank module updates, when it restricts them, and how it performs at matched budgets on language models.

Theorem 4.1 gives a condition that can be checked before training. Seven of 36 layouts across six models violate it. For Qwen attention, the default near-square layout restricts module rank to nine times the global factor rank. Reducing its width by one column removes this restriction and improves all eight comparisons by 0.14–3.79 points. The same adjustment fixes every violating layout in the survey (Table 23).

2. Equal tuning changes the accuracy conclusion. With GFA and LoRA independently tuned on the same grid, GFA minus LoRA is −0.08 points, with a 95% interval of [−0.30, +0.11], across 50 comparisons on five tasks at LoRA-rank-1 through rank-16 budgets: 17 wins, 12 ties and 21 losses. The +0.68-point advantage at a shared rate therefore does not persist under equal tuning (Table 3, App. D).

The ranking against Uni-LoRA and GPart also reverses. Across the same six rank-one-budget comparisons, GFA's margins against Uni-LoRA change from +0.48 to +12.20 at the shared rate to −1.73 to −0.31 with both tuned. Against GPart, they change from +0.68 to +11.67 to −1.40 to −0.07. Four comparisons exclude a GFA seed that collapsed at the highest rate; no LoRA seed collapsed. Setting LoRA dropout to zero changes accuracy by only 0.01 points on average (Table 10).

3. Layout matters more than grouping. At matched budgets, full-rank layouts win all 14 comparisons in the one-column and no-reshape tests. Removing the reshape restricts module rank and forces updates into a shared row space, costing 1.56–15.15 points with each arm separately tuned. Group count and module order change accuracy by less than one point. In the Qwen shared-rate test, 1, 4, and 36 groups are within 0.2 points, and separate full-rank factorizations per module perform similarly (App. G.1). Module gradients on the shared factor are approximately as uncorrelated as random directions under the shared-rate protocol (App. O, Lemma O.1).

4. Small-budget results require a tuning caveat. With both methods tuned, GFA exceeds partial-module LoRA in every tested comparison at 26,067/37,541 parameters. Median gains are +0.52 points on Qwen and +0.83 on Llama, with ranges of +0.02 to +4.66 and +0.06 to +13.42 (Table 11). However, partial-module LoRA selects the highest tested rate in 17 of 24 comparisons, so these margins may shrink with a wider search.

5. The expanded evaluation identifies clear limits. LoRA leads at larger budgets: GFA's deficit grows from 0.40 points at rank 32 to 2.13 at rank 128. LoRA also leads on GSM8K (0.46/1.44 points on Qwen/Llama), all four MBPP comparisons (0.60–4.33), all three Qwen2.5-7B tasks at rank 16 (0.12–0.86), and all four Llama all-linear comparisons (0.63–1.72 at the shared rate). The all-linear rank-4 anchor gives LoRA 43% more parameters. Qwen2.5-7B results at rank-one budgets are mixed. A 13B model's FP16 weights exceed our single 24 GB GPU.

6. Efficiency means budget and storage flexibility. GFA adapts every module in budget increments as small as 26,067 parameters on Qwen, with 52 KB adapters versus 461 KB for rank-one LoRA. Median training-memory ratio is 1.000; median time ratios are 1.03 on Qwen and 1.09 on Llama. The benchmark uses randomly initialized FP16 models without gradient checkpointing (Sec. 6, App. H).

Protocol and manuscript changes. New experiments use three seeds (42–44) and report mean ± standard deviation; submitted tables use standard errors, as clarified in the captions. Budgets are matched unless stated otherwise. Each method selects its best rate from the grid specified for that comparison, using the reported evaluation split, unless labeled “shared rate.” Intervals are 95% model-task-cluster bootstrap intervals; ties are differences below 0.1 points. Applying this procedure consistently corrects the submitted counts from 52/1/7 to 51/2/7 and from 45/1/14 to 40/11/9.

In our revision, the abstract, Section 6, and conclusion lead with equal-tuning results; shared-rate results are secondary. We clarify layouts, module ordering, dropout, serving costs, and notation. Related work adds SuperLoRA's BMVC version, distinguishes the two Uni-LoRA methods, and connects the analysis to Kronecker adapters and matrix rearrangement. Detailed sub-unit tables move to the appendix to retain the 10-page main text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.