- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the parameter-dependent geometry of LoRA's bilinear parameterization, where distances between trainable parameters are not generally preserved in the weight-update space. To address this limitation, the authors propose GPart, a highly parameter-efficient fine-tuning method that directly maps a low-dimensional trainable vector into the full weight space through a sparse, isometric partition matrix. GPart ensures exact end-to-end isometry with a single hyperparameter and compact checkpoint storage. Experiments across diverse tasks demonstrate competitive performance under ultra-low parameter budgets, while maintaining a fixed and predictable geometry between trainable parameters and weight updates.
Critical strengths
- The paper proposes an interesting isometric fine-tuning approach that achieves competitive performance under ultra-low parameter budgets.
- The paper is well structured and clearly presented. The proposed method is easy to follow, supported by theoretical analysis of its isometric properties and extensive experiments.
- The experimental evaluation covers diverse models and tasks. The results demonstrate the effectiveness of GPart under ultra-low parameter budgets, supporting its claims of parameter efficiency.
Critical weaknesses
- The motivation for isometry is insufficiently justified. Although the authors establish that LoRA's bilinear parameterization is not isometric, they do not clearly explain why preserving Euclidean geometry is beneficial for fine-tuning. The theoretical analysis establishes isometry but does not demonstrate its practical advantages for optimization or generalization.
- The proposed method imposes a strong inductive bias that may limit expressiveness. GPart randomly groups parameters across different layers and forces them to share updates. This substantially reduces trainable parameters, but it also restricts the update space. The paper provides limited discussion of how these constraints affect adaptation capability.
- Parameter efficiency comes at the expense of computational efficiency. GPart introduces non-negligible runtime overhead on larger models. Given the growing importance of computational and memory efficiency alongside parameter efficiency, this trade-off may limit its practical applicability.
Questions for authors
- What practical benefits does isometry provide for fine-tuning?
- How sensitive is GPart to random seeds? Furthermore, how does GPart ensure reproducibility across different devices and software environments? Since checkpoints store only the trainable vector and random seed, reconstructing the exact same partition matrix is essential for reproducing the finetuned model.
- How does GPart scale with different trainable parameter budgets? Could the authors provide a more systematic analysis of performance across different values of (d).
4 comments
I thank the authors for the additional empirical evidence, which has partially addressed my concerns about the motivation for isometry and reproducibility. However, I still have concerns about the trade-off between parameter efficiency and expressiveness, as well as the substantial computational overhead on larger models.
Although the authors have provided additional experiments showing how performance varies with different values of , my concerns about the expressiveness of GPart remain. I would appreciate further discussion of how the GPart balances parameter efficiency and expressiveness, particularly given the constraints imposed by fixed random grouping. Additionally, I could not find the expanded Figure 3 mentioned in the response.
I agree that the inference overhead is relatively small. However, the substantial training overhead observed on larger models remains a concern, as it may limit the practical applicability of GPart for large-scale fine-tuning.
Thank you for recognizing the additional evidence. We apologize for the delay and have now updated the paper PDF with the expanded Figure 3 and additional experimental results.
Motivation and benefits of isometry
Our motivation is to give the trainable parameters a direct, predictable relationship with weight updates: . Consequently, an penalty on the trainable vector also equals the same penalty on its weight update (Section 4.1 and Appendix A.3).
We assess the practical value of this normalization through an ablation: removing isometry while retaining the groups and budget lowers average GLUE performance from 84.1 to 80.7 on RoBERTa-base and from 86.3 to 84.8 on RoBERTa-large (Table 5). The loss-landscape comparison in Figure 4 provides complementary qualitative evidence. We believe this evidence addresses W1/Q1. If a specific concern remains, we would appreciate clarification so that we can address it directly.
Expressiveness
GPart achieves parameter efficiency through fixed grouping, an alternative structural constraint to LoRA's low-rank factorization. Weights within each group share a normalized update, but the resulting update matrices are not constrained to a fixed low rank. In practice, we trade the low-rank constraint for grouped updates. Increasing provides more independent groups; at , every targeted weight can change independently.
To address Q3, the updated Figure 3 compares both methods across eight matched budgets on CoLA, MRPC, and SST-2, with three seeds per configuration. The following summarizes the median scores at the smallest and largest budgets, and :
| Method | CoLA | MRPC | SST-2 |
|---|---|---|---|
| GPart | |||
| Uni-LoRA |
The full curves show that increasing provides more independent trainable coordinates and greater expressiveness, generally improving performance, with gains diminishing at task-dependent budgets. Table 4 also shows that global sharing outperforms per-layer sharing at the same total budget, achieving 84.1 versus 82.9 average GLUE score. Together, these findings suggest that grouped updates retain useful adaptation capacity in the evaluated settings. If your concern targets an aspect of expressiveness beyond this budget–performance trade-off, we would appreciate a more specific question.
Training overhead
Training efficiency depends on the implementation backend, as clarified in the expanded Appendix F. Materialized GPart stores the assignment mappings in VRAM, avoiding repeated reconstruction. Its training throughput ranges from to that of LoRA, at the expense of higher VRAM usage. Stateless GPart regenerates assignments during computation, reducing memory use but incurring the larger training overhead reported for 7B/8B models. Users can therefore choose the backend according to available VRAM and their memory–speed priorities.
We agree that improving large-model training efficiency remains an important implementation direction.
I thank the authors for their additional clarifications. However, I still have reservations about the practical advantages of GPart.
Although the expanded Figure 3 provides a more systematic evaluation across different parameter budgets, GPart does not demonstrate a consistent advantage over Uni-LoRA. The results on GSM8K also show mixed performance. Therefore, I remain unconvinced that the current evidence establishes a clear and consistent budget-performance advantage over Uni-LoRA.
Furthermore, the substantial training slowdown of the stateless implementation on 7B/8B models remains a concern. Although the materialized implementation provides an alternative, the resulting memory-speed trade-off raises questions about the practicality of GPart for large-scale fine-tuning.
We thank the reviewer for the feedback and questions.
W1/Q1: Practical benefits of isometry
By preserving distances, a change in GPart's trainable parameters produces an equally large change in model weights (Section 4.1, Proposition 1, Eq. 9). This holds throughout the trainable subspace, not just locally. An L2 penalty on the trainable parameters also equals the same penalty on the weight updates (Appendix A.3, Eq. 14). In the paper we test whether this matters in practice. Keeping the groups and parameter budget fixed, removing isometry lowers average GLUE from 84.1 to 80.7 on RoBERTa-base and 86.3 to 84.8 on RoBERTa-large (Table 5). Figure 4 (Section 5.4) also shows a smoother loss landscape for GPart than UniLoRA. These results support the practical value of isometry, although they do not prove that isometry alone causes better optimization.
GPart also supports merging directly in its compact parameter space (Appendix H). For adapters sharing the same projection P, merging their vectors exactly reproduces linear merging of their weight updates: P(θ1+θ2)=Pθ1+Pθ2. LoRA factor merging and UniLoRA vector merging generally introduce additional cross terms. Table 14 provides empirical evidence: across all task pairs, GPart has the lowest relative merging degradation among adapter-space methods for both linear and TIES merging, on both RoBERTa-base and large. Direct full-weight LoRA merging performs better still; GPart's advantage is retaining a compact representation and merging without materializing full weight updates.
Across GLUE tasks, we evaluated GPart and Uni-LoRA at aggressive learning rates. Raising the learning rate from 5e-3 to 5e-2 reduced CoLA scores by 13.03 points for GPart versus 16.12 for Uni-LoRA, and MRPC accuracy by 3.51 versus 11.03 points, respectively. Both methods retained strong performance on SST-2. Together with our broader experimental experience, these results suggest that GPart can be more forgiving of learning-rate choices and easier to tune. We consider this an important practical benefit.
W2/Q3: Expressiveness and parameter budget
Increasing d gives GPart more independent trainable coordinates. We evaluated eight budgets on CoLA, MRPC, and SST-2, with three seeds each. Performance generally improves with the budget, although gains diminish and MRPC peaks before the largest budget. GPart and Uni-LoRA show similar overall trends, with small differences on SST-2 and MRPC and larger differences on CoLA. We will expand Figure 3 to show this comparison across all three tasks.
W3: Training and inference cost
GPart's current implementation has higher training and unmerged inference costs. However, training is a one-time cost per adapter, while compact adapter checkpoints offer a practical advantage for storage and deployment on edge devices. After merging the adapter into the base weights, deployment uses the original model architecture. On Qwen2.5-0.5B and 3B, merged GPart and LoRA differ by less than 1% in forward latency and 0.5% in fixed-length generation latency, with essentially identical memory use. The one-time merge takes less than 1 second on the tested models.
Q2: Projection-seed sensitivity and reproducibility
Table 4 isolates partition sensitivity by fixing the training seed and varying the projection seed. Variation is small on most GLUE tasks, with greater sensitivity on RTE. We also successfully reproduced seed-based partition reconstruction on two different machines. With the same environment, base model, configuration, and parameter ordering, reconstruction has the same reproducibility requirements as other methods that regenerate random projections from a seed (e.g., Uni-LoRA and VeRA).