acceptodds
Under review as a conference paper at ICLR 2027

GPart: End-to-End Isometric Fine-Tuning via Global Parameter Partitioning

Abstract

Low-rank adaptation (LoRA) has become a dominant paradigm for parameter-efficient fine-tuning (PEFT) of large-scale deep learning models. However, its bilinear parameterization induces a parameter-dependent geometry: the mapping from trainable parameters to weight updates is not generally distance-preserving. Related methods that project a low-dimensional vector into LoRA's parameter space, such as Uni-LoRA, improve parameter efficiency, but the subsequent bilinear map breaks end-to-end isometry. We propose GPart (Global Partition fine-tuning), a highly parameter-efficient fine-tuning method that maps a -dimensional trainable vector directly into the full weight space through a sparse, isometric partition matrix. GPart retains a fixed global parameter-sharing prior while removing the additional low-rank reconstruction used by LoRA-based methods. This yields a simple parameterization with a single main hyperparameter (), exact end-to-end isometry, and a minimal checkpoint representation consisting of the trainable vector and a random seed. GPart builds on the premise of effective fine-tuning within random low-dimensional subspaces of the full weight space without requiring a low-rank matrix factorization. Across natural language understanding, computer vision, and mathematical reasoning benchmarks, GPart matches or improves over existing PEFT methods at ultra-low parameter budgets. Beyond offering mathematical tractability and memory efficiency, the direct linear parameterization of GPart streamlines model selection and paves the way for compact adapter composition. Overall, GPart provides an elegant and competitive alternative for fine-tuning under small parameter budgets, with a fixed and predictable geometry between trainable coordinates and weight-space updates.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
fine-tuning, LLM, PEFT, LoRA

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9DRzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the parameter-dependent geometry of LoRA's bilinear parameterization, where distances between trainable parameters are not generally preserved in the weight-update space. To address this limitation, the authors propose GPart, a highly parameter-efficient fine-tuning method that directly maps a low-dimensional trainable vector into the full weight space through a sparse, isometric partition matrix. GPart ensures exact end-to-end isometry with a single hyperparameter and compact checkpoint storage. Experiments across diverse tasks demonstrate competitive performance under ultra-low parameter budgets, while maintaining a fixed and predictable geometry between trainable parameters and weight updates.

Critical strengths
  1. The paper proposes an interesting isometric fine-tuning approach that achieves competitive performance under ultra-low parameter budgets.
  2. The paper is well structured and clearly presented. The proposed method is easy to follow, supported by theoretical analysis of its isometric properties and extensive experiments.
  3. The experimental evaluation covers diverse models and tasks. The results demonstrate the effectiveness of GPart under ultra-low parameter budgets, supporting its claims of parameter efficiency.
Critical weaknesses
  1. The motivation for isometry is insufficiently justified. Although the authors establish that LoRA's bilinear parameterization is not isometric, they do not clearly explain why preserving Euclidean geometry is beneficial for fine-tuning. The theoretical analysis establishes isometry but does not demonstrate its practical advantages for optimization or generalization.
  2. The proposed method imposes a strong inductive bias that may limit expressiveness. GPart randomly groups parameters across different layers and forces them to share updates. This substantially reduces trainable parameters, but it also restricts the update space. The paper provides limited discussion of how these constraints affect adaptation capability.
  3. Parameter efficiency comes at the expense of computational efficiency. GPart introduces non-negligible runtime overhead on larger models. Given the growing importance of computational and memory efficiency alongside parameter efficiency, this trade-off may limit its practical applicability.
Questions for authors
  1. What practical benefits does isometry provide for fine-tuning?
  2. How sensitive is GPart to random seeds? Furthermore, how does GPart ensure reproducibility across different devices and software environments? Since checkpoints store only the trainable vector and random seed, reconstructing the exact same partition matrix is essential for reproducing the finetuned model.
  3. How does GPart scale with different trainable parameter budgets? Could the authors provide a more systematic analysis of performance across different values of (d).
4 comments

We thank the reviewer for the feedback and questions.

W1/Q1: Practical benefits of isometry

By preserving distances, a change in GPart's trainable parameters produces an equally large change in model weights (Section 4.1, Proposition 1, Eq. 9). This holds throughout the trainable subspace, not just locally. An L2L_2 penalty on the trainable parameters also equals the same penalty on the weight updates (Appendix A.3, Eq. 14). In the paper we test whether this matters in practice. Keeping the groups and parameter budget fixed, removing isometry lowers average GLUE from 84.1 to 80.7 on RoBERTa-base and 86.3 to 84.8 on RoBERTa-large (Table 5). Figure 4 (Section 5.4) also shows a smoother loss landscape for GPart than UniLoRA. These results support the practical value of isometry, although they do not prove that isometry alone causes better optimization.

GPart also supports merging directly in its compact parameter space (Appendix H). For adapters sharing the same projection PP, merging their vectors exactly reproduces linear merging of their weight updates: P(θ1+θ2)=Pθ1+Pθ2P(\theta_1 + \theta_2) = P\theta_1 + P\theta_2. LoRA factor merging and UniLoRA vector merging generally introduce additional cross terms. Table 14 provides empirical evidence: across all task pairs, GPart has the lowest relative merging degradation among adapter-space methods for both linear and TIES merging, on both RoBERTa-base and large. Direct full-weight LoRA merging performs better still; GPart's advantage is retaining a compact representation and merging without materializing full weight updates.

Across GLUE tasks, we evaluated GPart and Uni-LoRA at aggressive learning rates. Raising the learning rate from 5e-3 to 5e-2 reduced CoLA scores by 13.03 points for GPart versus 16.12 for Uni-LoRA, and MRPC accuracy by 3.51 versus 11.03 points, respectively. Both methods retained strong performance on SST-2. Together with our broader experimental experience, these results suggest that GPart can be more forgiving of learning-rate choices and easier to tune. We consider this an important practical benefit.

W2/Q3: Expressiveness and parameter budget

Increasing dd gives GPart more independent trainable coordinates. We evaluated eight budgets on CoLA, MRPC, and SST-2, with three seeds each. Performance generally improves with the budget, although gains diminish and MRPC peaks before the largest budget. GPart and Uni-LoRA show similar overall trends, with small differences on SST-2 and MRPC and larger differences on CoLA. We will expand Figure 3 to show this comparison across all three tasks.

W3: Training and inference cost

GPart's current implementation has higher training and unmerged inference costs. However, training is a one-time cost per adapter, while compact adapter checkpoints offer a practical advantage for storage and deployment on edge devices. After merging the adapter into the base weights, deployment uses the original model architecture. On Qwen2.5-0.5B and 3B, merged GPart and LoRA differ by less than 1% in forward latency and 0.5% in fixed-length generation latency, with essentially identical memory use. The one-time merge takes less than 1 second on the tested models.

Q2: Projection-seed sensitivity and reproducibility

Table 4 isolates partition sensitivity by fixing the training seed and varying the projection seed. Variation is small on most GLUE tasks, with greater sensitivity on RTE. We also successfully reproduced seed-based partition reconstruction on two different machines. With the same environment, base model, configuration, and parameter ordering, reconstruction has the same reproducibility requirements as other methods that regenerate random projections from a seed (e.g., Uni-LoRA and VeRA).

I thank the authors for the additional empirical evidence, which has partially addressed my concerns about the motivation for isometry and reproducibility. However, I still have concerns about the trade-off between parameter efficiency and expressiveness, as well as the substantial computational overhead on larger models.

Although the authors have provided additional experiments showing how performance varies with different values of dd, my concerns about the expressiveness of GPart remain. I would appreciate further discussion of how the GPart balances parameter efficiency and expressiveness, particularly given the constraints imposed by fixed random grouping. Additionally, I could not find the expanded Figure 3 mentioned in the response.

I agree that the inference overhead is relatively small. However, the substantial training overhead observed on larger models remains a concern, as it may limit the practical applicability of GPart for large-scale fine-tuning.

Thank you for recognizing the additional evidence. We apologize for the delay and have now updated the paper PDF with the expanded Figure 3 and additional experimental results.

Motivation and benefits of isometry

Our motivation is to give the trainable parameters a direct, predictable relationship with weight updates: ∥Pθ1−Pθ2∥2=∥θ1−θ2∥2\|P\theta_1-P\theta_2\|_2=\|\theta_1-\theta_2\|_2. Consequently, an L2L_2 penalty on the trainable vector also equals the same penalty on its weight update (Section 4.1 and Appendix A.3).

We assess the practical value of this normalization through an ablation: removing isometry while retaining the groups and budget lowers average GLUE performance from 84.1 to 80.7 on RoBERTa-base and from 86.3 to 84.8 on RoBERTa-large (Table 5). The loss-landscape comparison in Figure 4 provides complementary qualitative evidence. We believe this evidence addresses W1/Q1. If a specific concern remains, we would appreciate clarification so that we can address it directly.

Expressiveness

GPart achieves parameter efficiency through fixed grouping, an alternative structural constraint to LoRA's low-rank factorization. Weights within each group share a normalized update, but the resulting update matrices are not constrained to a fixed low rank. In practice, we trade the low-rank constraint for grouped updates. Increasing dd provides more independent groups; at d=Nd=N, every targeted weight can change independently.

To address Q3, the updated Figure 3 compares both methods across eight matched budgets on CoLA, MRPC, and SST-2, with three seeds per configuration. The following summarizes the median scores at the smallest and largest budgets, d=768d=768 and d=98,304d=98{,}304:

MethodCoLAMRPCSST-2
GPart49.45→61.0749.45 \rightarrow 61.0778.43→87.2578.43 \rightarrow 87.2590.60→93.9290.60 \rightarrow 93.92
Uni-LoRA47.40→55.0647.40 \rightarrow 55.0679.90→88.2479.90 \rightarrow 88.2491.40→93.8191.40 \rightarrow 93.81

The full curves show that increasing dd provides more independent trainable coordinates and greater expressiveness, generally improving performance, with gains diminishing at task-dependent budgets. Table 4 also shows that global sharing outperforms per-layer sharing at the same total budget, achieving 84.1 versus 82.9 average GLUE score. Together, these findings suggest that grouped updates retain useful adaptation capacity in the evaluated settings. If your concern targets an aspect of expressiveness beyond this budget–performance trade-off, we would appreciate a more specific question.

Training overhead

Training efficiency depends on the implementation backend, as clarified in the expanded Appendix F. Materialized GPart stores the assignment mappings in VRAM, avoiding repeated reconstruction. Its training throughput ranges from 0.8×0.8\times to 1.3×1.3\times that of LoRA, at the expense of higher VRAM usage. Stateless GPart regenerates assignments during computation, reducing memory use but incurring the larger training overhead reported for 7B/8B models. Users can therefore choose the backend according to available VRAM and their memory–speed priorities.

We agree that improving large-model training efficiency remains an important implementation direction.

I thank the authors for their additional clarifications. However, I still have reservations about the practical advantages of GPart.

Although the expanded Figure 3 provides a more systematic evaluation across different parameter budgets, GPart does not demonstrate a consistent advantage over Uni-LoRA. The results on GSM8K also show mixed performance. Therefore, I remain unconvinced that the current evidence establishes a clear and consistent budget-performance advantage over Uni-LoRA.

Furthermore, the substantial training slowdown of the stateless implementation on 7B/8B models remains a concern. Although the materialized implementation provides an alternative, the resulting memory-speed trade-off raises questions about the practicality of GPart for large-scale fine-tuning.

Reviewer_BGWxOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose GPart (Global Partition fine-tuning), a highly parameter-efficient fine-tuning method that directly maps a dd-dimensional trainable vector into the full weight space through a sparse, isometric partition matrix. My understanding of the difference between GPart and Uni-LoRA is that Uni-LoRA imposes a low-rank factorization constraint through ABAB, whereas GPart removes this constraint. The paper claims that the bilinear mapping breaks end-to-end isometry.

Critical strengths

The experimental results seem to demonstrate that the low-rank factorization constraint ABAB in Uni-LoRA is not important for fine-tuning.

Critical weaknesses
  1. This work is based on the assumption that the vectorized weights of different layers lie in the same DD-dimensional subspace. I suggest adding a simulation experiment to illustrate and validate this assumption.

  2. I do not understand why Eq. (4) is introduced in Section 3.1. What is the relationship between Eqs. (2) and (4)?

  3. The authors’ discussion of isometry in GPart does not convince me at all. Proposition 1 is about local isometry. The conclusion in Eq. (9) is too obvious, and I do not consider it a theoretical contribution. Moreover, isometry itself does not imply better optimization or generalization, and the experiments do not demonstrate such benefits either.

  4. Furthermore, the motivation behind the construction of PP is unclear. If the goal is to ensure isometry, any column-orthogonal matrix would suffice.

  5. The experimental comparisons include too few baseline methods. Compared with Uni-LoRA, there is little improvement in either parameter count or downstream task accuracy.

  6. The paper is poorly written, particularly Sections 3 and 4.

  7. Does the randomness in your random subspace come from gg?

Questions for authors

see above

1 comment

We appreciate the reviewer's questions and the opportunity to clarify our method.

W1: Shared-subspace assumption

GPart does not assume that the vectorized weights of different layers lie in the same low-dimensional subspace. It parameterizes the concatenated weight update as Δw=Pθ\Delta w=P\theta, so optimization takes place in a dd-dimensional subspace of the full model parameter space, anchored at the pretrained weights. Each layer receives the corresponding block of this global update; this does not require the layer weights themselves to share a subspace. This is motivated by the low intrinsic dimensionality observed by Aghajanyan et al. (ACL 2021), and our experiments support the novel partition-based construction.

W2: Weight space and factor space

Eqs. (3) and (4) distinguish the spaces in which the methods operate. Eq. (3) vectorizes the pretrained weight matrices; GPart adds PθP\theta directly in this space, as described in Eq. (2). Eq. (4) instead vectorizes LoRA’s factors: Uni-LoRA reconstructs these factors from its shared vector and then forms the weight updates through BABA. We agree that this connection could be clearer and will explicitly show both mapping paths in Section 3.1.

W3: Isometry and learning-rate robustness

Proposition 1 characterizes the geometry of our parameterization; the contribution lies in the construction and its practical consequences.

GPart preserves distances throughout the whole trainable subspace, not just locally: a change in its trainable parameters produces an equally large change in model weights (Section 4.1, Proposition 1, Eq. (9)). An L2L_2 penalty on the trainable parameters also equals the same penalty on the weight updates (Appendix A.3, Eq. (14)). We tested whether this matters in practice. Keeping the groups and parameter budget fixed, removing isometry lowers GLUE avg. accuracy from 84.1 to 80.7 on RoBERTa-base and 86.3 to 84.8 on RoBERTa-large (Table 5). Figure 4 (Section 5.4) also shows a smoother loss landscape for GPart than UniLoRA. These results support the practical value of isometry.

Across GLUE tasks, we evaluated GPart and Uni-LoRA at aggressive learning rates. Raising the learning rate from 5e-3 to 5e-2 reduced MRPC accuracy by 3.51 for GPart versus 11.03 points for Uni-LoRA, and CoLA scores by 13.03 points versus 16.12 point, respectively. Both methods retained strong performance on SST-2. Together with our broader experimental experience, these results suggest that GPart can be more forgiving of learning-rate choices and easier to tune. We consider this an important practical benefit.

W4: Partition construction

You are correct that many matrices could satisfy the isometry condition. We choose our particular implementation, because it offers three practical advantages:

  • Exact isometry for every draw: PTP=IdP^T P = I_d, while normalized Fastfood is isotropic, i.e. isometric only in expectation (Appendix A.4).
  • Simple projection: one nonzero per row allows O(N)O(N) gather/scatter operations without explicitly constructing PP (Sections 3.2–3.3).
  • Compact storage: checkpoints contain only the trainable vector and seed (Section 3.4), and stateless assignment avoids storing PP or an O(N)O(N) index buffer (Appendix F).

GPart's memory benefit comes from its simple, seed-defined assignment.

W5: Baselines and budget comparison

We added SHiRA, FourierFT, and RandLoRA on the six GLUE tasks, using RoBERTa-base and three seeds. We follow published learning rates where available and transfer SHiRA’s language-task learning rate.

MethodTrainable params.GLUE avg.
GPart23,04084.10
FourierFT23,04082.82
RandLoRA36,86482.32
SHiRA36,86478.78

GPart has the highest average, matching FourierFT’s adapter budget and using fewer parameters than SHiRA and RandLoRA.

We also compared GPart and Uni-LoRA across eight matched parameter budgets on RoBERTa-base for CoLA, MRPC, and SST-2, using three seeds. The methods behave similarly overall, with task-dependent differences and no consistent winner. GPart achieves this with a direct linear parameterization and also shows lower degradation when merging adapters in their compact parameter space (Appendix H, Table 14).

W6: Presentation clarity

We strive to revise the paper for clarity. Would you provide specific examples in Sections 3 and 4 that we should focus on revising?

W7: Partition randomness

Yes. The seed-defined random assignment gg fixes the groups and thus PP and its normalization. The partition stays fixed during training and is implemented through gg and group sizes, without materializing PP.

Reviewer_zVc1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper takes the insight and appraoch of DID - Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning (ACL 2021) and creates a LORA/PEFT variant based on it. The proposed method, called GPart, defines a shared trainable vector that is used to "fine-tune" original trainable parameters of the model by associating / expanding the low-dimensional parameterization via a random partition matrix.

Critical strengths

The paper presents a well-written motivation with clear connections to DID and Uni-LORA. Overall, the proposed method is a sensible approach.

The experimental results are promising, especially in certain cases such as FGVC dataset results with small models (Table 3).

Critical weaknesses

Having good results comparable to those of state-of-the-art techniques is not surprising as the paper is mostly a combination of formulations that are known to work well. In fact, the gains are not uniformly impressive. While the some improvements are reported on average in LLM and vision tasks, individual results are rather mixed: e.g. in Table 2 MATH results, Uni-LORA outperforms in 3 out of 5 cases. In GSM8K, LORA wins 2 out of 5 cases, and Uni-LORA wins in another of them. Table 1 GLUE results are similarly mixed. Overall, these results make the impression that given this technique that shares many elements similar to prior work with random transformations/projections (Uni-LORA, VERA, etc) and results similar to them, the significance of the contribution may rather be limited. Rather, it feels like a sensible variation of existing solutions.

The paper could have benefited from a complete technical summary that defines all random projection/transformation based PEFT methods using a unified notation. Some of the methods that could be included: RLoRA, VERA, UNI-LoRA, SuperLoRA, SHiRA, etc. (I believe that are more works to be found).

Some of the techniques I've listed above are missing in the discussions and comparisons (RLoRA, SHiRA, SuperLoRA, as far as I remember). Some of them attack the problem w/o a low-rank decomposition, similar to this paper, unlike LoRA or Uni-LORA. Therefore, I think a conceptual + experimental comparison is missing.

Currently the results are reported at discrete opereating points. However, most methods , especially those that can operate at sub-rank-1 (in LORA terms) provide the ability to work at various operating points. The experimental focus could have been much more inclusive in studying these methods in the complexity spectrum, via scatter plots, principled model selection techniques, and so.

The experimental analysis could have been strong with results containing instruction following LLM tasks, and image generation tasks.

Questions for authors

Is "isometric partition matrix" an existing term in the math literature? Even the partition matrix sounds both familiar and strange, as it is more commonly knonwn "assignment matrix" (plus scaling) ? This is a minor issue but I'd appreciate a clarification.

3 comments

We thank the reviewer for their feedback and suggestions.

Significance

GPart builds on random-subspace fine-tuning and Uni-LoRA, but changes how the shared trainable vector produces weight updates. Uni-LoRA first reconstructs LoRA’s factors AA and BB, then multiplies them to obtain BABA. GPart instead maps the vector directly into weight updates through Δw=Pθ\Delta w=P\theta. This removes the intermediate factorization and the need to choose a rank, while ensuring that distances between trainable vectors exactly match distances between their weight updates. Our contribution is this simpler, globally shared, isometric parameterization. Tables 1, 2, and 3 show that GPart retains broadly comparable performance to LoRA and Uni-LoRA, while using substantially fewer trainable parameters than LoRA.

Unified comparison of PEFT methods

Following your suggestion, we will add this comparison to Section 3. Trainable parameters are factors A,BA,B, scaling vectors u,vu,v, or vectors θ\theta. D(v)=diag⁡(v)D(v)=\operatorname{diag}(v); RR and SS are fixed random and selection maps. Scalars and some layer indices are omitted.

MethodTrainable parametersUpdate parameterizationFixed structure
LoRALayer-specific A,BA,BΔW=BA\Delta W=BANone.
Uni-LoRAGlobally shared θ\thetaΔWℓ=Bℓ(θ)Aℓ(θ)\Delta W_\ell=B_\ell(\theta)A_\ell(\theta)Assignments from θ\theta into LoRA factors.
VeRA / RLoRALayer-specific u,vu,vΔW=D(u)RBD(v)RA\Delta W=D(u)R_BD(v)R_ARandom factors / randomized Walsh–Hadamard factors.
RandLoRALayer- and basis-specific uj,vju_j,v_jΔW=∑jRB,jD(uj)RA,jD(vj)\Delta W=\sum_j R_{B,j}D(u_j)R_{A,j}D(v_j)Multiple low-rank random bases.
FourierFTLayer-specific θ\thetaΔW=Re⁡F−1(Sθ)\Delta W=\operatorname{Re}\mathcal{F}^{-1}(S\theta)Selected frequencies and Fourier transform.
SHiRALayer-specific θ\thetaΔw=Sθ\Delta w=S\thetaSparse coordinate selection SS.
SuperLoRAGroup-specific θg\theta_gΔwg=Fg(Tg(θg))\Delta w_g=\mathcal{F}_g(T_g(\theta_g))Grouping and projection Fg\mathcal{F}_g; TgT_g builds the chosen dense/factorized update.
Intrinsic-dimension FT / FastfoodGlobally shared θ\thetaΔw=Rθ\Delta w=R\thetaStructured random transform RR.
GPartGlobally shared θ\thetaΔw=Pθ\Delta w=P\theta, P⊤P=IdP^\top P=I_dNormalized partition PP, one nonzero per row.

GPart preserves additive merges in compact space when adapters share PP. Appendix H, Table 14 reports lower adapter-space merging degradation than LoRA and Uni-LoRA.

Additional baselines

We added SHiRA, FourierFT, and RandLoRA on the six GLUE tasks, using RoBERTa-base and three seeds. We follow published learning rates where available and transfer SHiRA’s language-task learning rate.

MethodAdapter params.GLUE avg.
GPart (paper result)23,04084.10
FourierFT23,04082.82
RandLoRA36,86482.32
SHiRA36,86478.78

GPart has the highest average, matching FourierFT’s adapter budget and using fewer parameters than SHiRA and RandLoRA.

Performance across parameter budgets

We compared GPart and Uni-LoRA across eight matched budgets on CoLA, MRPC, and SST-2, with three seeds per configuration. Overall, they show very similar budget–performance trends. Differences are small on SST-2 and MRPC, with no consistent winner. On CoLA, Uni-LoRA performs better at intermediate budgets, while GPart achieves a lower performance drop at the largest budgets. We will update Figure 3 with a more comprehensive plot covering both methods and all three tasks.

Instruction following

Following your suggestion, we expanded the evaluation with multi-task common-sense reasoning on eight benchmarks: BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-Easy, ARC-Challenge, and OpenBookQA. We evaluated five language models from the Qwen2.5, Gemma, and Llama families, comparing GPart, LoRA, and Uni-LoRA at matched adapter budgets. GPart performs competitively, with an average accuracy across models of 76.88%, compared with 76.73% for LoRA and 76.72% for Uni-LoRA. We will add the full results and experimental details to the appendix.

Image generation would require different model architectures and separate training and evaluation pipelines, making these experiments infeasible within the rebuttal period.

Isometric Partition Matrix term

We use "isometric partition matrix" descriptively, rather than as a standard mathematical term. Your description is accurate, and will be clarified in our revision: PP is a column-normalized assignment matrix. "Partition" emphasizes the grouping of weight coordinates, and "isometric" describes this normalization property. Section 3.2, Eqs. (5)–(6), defines the construction and shows that P⊤P=IdP^\top P = I_d.

I'd like to thank the authors for their reply & added experiments. Here is my summary of what I've understood so far from the rebuttal:

  • I think the proposed "Unified comparison of PEFT methods" will improve the presentation when added to the paper.
  • I had the concern that the results / the gains achieved are mixed & had written "Overall, these results make the impression that given this technique that shares many elements similar to prior work with random transformations/projections (Uni-LORA, VERA, etc)". When I check the results in the revised pdf, my concerns still appear to be valid: -- Figure 3 GPart vs UniLoRA winner fluctuates across the complexity for COLA and MRPC GPart is consistently not-worse on SST-2, although performance matches at 3 operating points + confidence intervals do not appear to be small at all. This Figure was added to in response to my "Performance across parameter budgets" concern. -- The added discussion of new common-sense-reasoning Table 15 says "is competitive, but its advantage over Uni-LoRA is not consistent across models". -- Table 1: Roberta-BASE results are strong, yet the asked baselines do not appear in Boberta-LARGE, where LoRA is already the winner.
  • Image generation weren't added, but common-sense-reasoning were added (with mixed results)
  • "isometric partition matrix" term -- clarified that it is not a standard term. Its use is OK.

I will keep these in mind in the final discussions (Reviewer- A C Discussion). I don't have additional questions at this point but I wanted to read them in detail & summarize, feel free to correct me if I've missed anything major.

Thank you for taking the time to read the revision and summarize the discussion. We agree that the relative gains vary across tasks and budgets. We were able to run the additional RoBERTa-large baselines in time to share the results below.

MethodAdapter params.GLUE avg.
LoRA (paper result)786,43286.70
GPart (paper result)23,04086.30
FourierFT23,04086.26
RandLoRA98,30486.18
SHiRA98,30481.58

At the same adapter budget, GPart and FourierFT perform similarly. LoRA retains the highest average in this comparison and uses about 34 times as many adapter parameters as GPart.

Comments on the paper

1 comment

Dear Reviewers and Area Chair,

We have uploaded an updated version of the paper, with all revised and newly added content in red. The updates include:

  • Additional baseline results in Table 1.
  • An expanded Figure 3 and updated discussion in Section 5.4.
  • An extended Appendix F: Runtime and Memory Analysis.
  • A new Appendix J: Additional Experiments on Common-Sense Reasoning.

Given the limited rebuttal period, the remaining additions and revisions discussed in our responses will be incorporated into the camera-ready version if the paper is accepted.

Thank you for your feedback and suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.