acceptodds
Under review as a conference paper at ICLR 2027

Parameter-Space Symmetry in Fine-Tuning

Abstract

Parameter-efficient fine-tuning introduces structured trainable parameters into a frozen model, but distinct parameter settings can represent exactly the same function. We characterize the parameter-space symmetries of LoRA, prefix tuning, and prompt tuning in multihead attention, making explicit the redundancies in their parameterizations. These results provide a foundation for studying linear mode connectivity and model merging, where accounting for symmetries helps distinguish functional differences from differences in parameter representation.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
finetuning, parameter space symmetry, mode connectivity

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EA56OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PEFT introduces structured trainable parameters into frozen models, but different parameter configurations can represent the same function. This paper characterizes the parameter-space symmetries of LoRA, prefix tuning, and prompt tuning in multihead attention, revealing redundancies in their parameterizations. This paper provides a theoretical foundation for studying linear mode connectivity and model merging by distinguishing functional differences from those arising solely from parameter representations.

Critical strengths

1.The paper investigates how different PEFT parameter configurations can represent the same function and how such redundancies affect parameter alignment, LMC, and model merging. Extending parameter-space symmetry analysis from conventional neural networks to multiple PEFT methods provides an interesting perspective and meaningful theoretical contribution.

2.The paper characterizes the parameter-space symmetries of LoRA, prefix tuning, and prompt tuning in multihead attention, considering both standard attention and RoPE. Beyond identifying function-preserving transformations, the paper establishes the maximality of the corresponding symmetry groups under specified assumptions.

3.The paper applies its theoretical findings to parameter alignment, LMC, and model merging. Table 1 demonstrates substantial reductions in interpolation loss barriers after alignment, while the model-merging experiments on BERT and Llama show competitive performance. The evaluation across vision and language tasks provides empirical support for the practical relevance of the theoretical results.

Critical weaknesses

1.The abstract and Introduction require substantial improvement in organization and writing, as they currently read more like an early draft than a carefully structured and polished manuscript. The Introduction resembles a survey of related concepts instead of a coherent motivation for the proposed work. In particular, the research gap and contributions relative to existing symmetry studies are not clearly articulated, making it difficult for readers to understand the paper's positioning and significance.

2.The theoretical results rely on specific PEFT configurations and assumptions. For example, the LoRA analysis considers only query and value adaptation with frozen key and output projections, while prefix tuning focuses on directly parameterized key–value prefixes.

3.The proposed matching algorithms do not fully exploit the identified function-preserving transformations. Although the LoRA symmetry group is characterized as GL(r)GL(r), the practical algorithm considers only its orthogonal subgroup. While this restriction simplifies computation, the paper does not sufficiently investigate whether it limits alignment quality or whether exploiting a broader set of symmetries could further reduce interpolation loss barriers.

4.Although the experiments cover both vision and language tasks, they rely largely on traditional benchmarks and relatively small models, leaving the applicability of the findings to larger, modern LLMs insufficiently explored. Furthermore, many recent PEFT variants and more general adaptation settings are not considered.

Questions for authors

1.The LoRA matching algorithm considers only the orthogonal subgroup of the theoretically identified (GL(r)) symmetry group. Could the authors clarify the practical implications of this restriction in Weakness 3? Would incorporating a broader set of function-preserving transformations further improve linear mode connectivity?

2.The proposed greedy merging method combines symmetry-based parameter alignment with validation-based coefficient selection. Could the authors provide an ablation comparing merging with and without alignment under the same coefficient-selection strategy?

3 comments

We will incorporate all revisions and additional tables into the revised manuscript. Please see our global comment for details.

W1. We thank the reviewer for the suggestion. However, we respectfully prefer to retain the current writing format. Some introductions present the material as a continuous narrative, while others organize it into several thematic parts. We follow the latter approach.

We have improved the writing within each part and added transitions to help readers understand our contributions. In summary:

While parameter-space symmetries have been studied for feedforward neural networks, multihead attention, and mixture-of-experts architectures, the symmetry structure of parameter-efficient fine-tuning methods remains less well understood. Freezing the backbone and restricting the trainable parameters change which function-preserving transformations are available. We characterize these transformations for LoRA, prefix tuning, and prompt tuning in multihead attention, making their parameter redundancies explicit and providing a foundation for studying linear mode connectivity and model merging in PEFT.

W2. The rank and invertibility assumptions establish completeness and uniqueness of the characterization, rather than the validity of the symmetry transformations: θ\theta and gθg\theta represent the same function even when these assumptions fail. However, additional functional equivalences may arise, so f(⋅;θ)=f(⋅;θˉ)f(\cdot;\theta)=f(\cdot;\bar\theta) need not imply θˉ=gθ\bar\theta=g\theta for any gg in the stated group. For example, in standard attention, an admissible query perturbation satisfying ΔQiKi⊤=0\Delta Q_iK_i^\top=0 leaves the attention scores unchanged. Thus, the maximality conclusion cannot generally be extended beyond its stated assumptions.

Additional parameter assumptions also appear in previous symmetry and identifiability results, including those for attention and feedforward networks [1-4]. In our results, the excluded degeneracies, such as rank deficiency and coincident prefix keys, are contained in proper real algebraic subsets of the relevant parameter spaces and therefore have Lebesgue measure zero. This establishes generic completeness.

For prefix tuning, our formulation follows the standard key–value representation of prefix tuning introduced by [8]. For LoRA, we have extended the LoRA analysis to joint Q/K, V/O, and Q/K/V/O adaptation. Under explicit genericity and rank conditions, the maximal symmetry group remains a product of GL(rT)\mathrm{GL}(r_T), one per adapted projection, both with and without RoPE. The alignment transformations remain exactly function preserving even when these conditions fail. Please see Section 3 and Appendix A.1.1 of the revised manuscript.

W3+Q1. Our choice of orthogonal matching is motivated by empirical evidence and computational efficiency. [6] reports small additional gains from rescaling beyond rotations, while [5] shows that orthogonal alignment captures most barrier reduction before refinement. [1] demonstrates low- and zero-barrier connections between independently trained Transformers using orthogonal residual-stream alignment with feedforward and attention-head permutations. Our matching admits a closed-form SVD solution and exactly preserves each LoRA update. We acknowledge that broader GL(r)GL(r) matching may further improve alignment and reduce interpolation barriers.

W4. Our experiments cover ViT and BERT, alongside Qwen2.5 for language modeling and Llama-3.2-1B for model merging. Standard benchmarks enable controlled comparisons of alignment and connectivity. We focus on LoRA, prefix tuning, and prompt tuning as distinct parameterizations with different symmetry structures. Our theoretical results apply across attention-module dimensions under the stated assumptions, while validation at substantially larger scales remains necessary. Additional PEFT variants and broader adaptation settings may require dedicated symmetry analyses, which we regard as important future work.

Q2. On both merging benchmarks, we removed only alignment, retaining the paper’s checkpoints, task orders, coefficient grid, and selection data. Full results appear in Tables 17 and 18: Accuracy of LMC-LoRA greedy with and without the alignment step (Appendix E.3) of the revised manuscript.

Alignment improves average accuracy and reduces variability across merge orders in both domains. The moderate gap suggests that greedy search partly compensates for missing alignment. LoRA factor interpolation introduces cross-adapter terms (B_iA_j), which may disrupt merging when adapter bases are misaligned. Search can mitigate this by choosing coefficients near 0 or 1, limiting adapter mixing. Alignment can make intermediate coefficients more effective, allowing adapters to combine more successfully.

References. Please see our response to Reviewer a3sk for the complete reference list.

I thank the authors for their clarifications and additional experiments, which have addressed my main concerns regarding W2 and W3, as well as my questions Q1 and Q2. I also understand the limitations discussed in response to W4.

However, my concern regarding W1 has not been fully resolved. I respect the authors' choice of writing style, but regardless of the structure adopted, the introduction should establish a clear logical connection between the research motivation, research gap, and contributions.

Although the authors have added transitions between sections, the introduction still reads largely as a sequence of descriptions of related work. The introduction still lacks a sufficiently coherent argument connecting the relevant literature to the paper's central research motivation. As a result, readers must navigate several relatively independent background paragraphs before fully understanding the paper's positioning and significance. I would encourage the authors to strengthen the logical progression and synthesis of related work to improve readability and reduce the cognitive burden on readers.

The abstract would also benefit from further refinement. While it identifies the methods studied, it does not clearly articulate the research gap relative to existing symmetry studies or highlight the specific theoretical findings. Consequently, readers understand what the paper investigates but gain limited insight into what new understanding it provides and why these results are significant. A more explicit summary of the key findings would help communicate the paper's contributions more effectively.

Nevertheless, considering the authors' clarifications and additional experimental evidence, which have addressed my main concerns, I have decided to raise my score to 3 (Weak Accept).

We thank the reviewer for their endorsement and constructive feedback. We acknowledge the remaining concerns regarding the clarity and coherence of the introduction and abstract. In the final revision, we will strengthen the logical connection between the research motivation, gap, and contributions in the introduction, and refine the abstract to more explicitly highlight the research gap, our key theoretical findings, and their significance.

Authors

Reviewer_a3skOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies parameter-space symmetries in PEFT, where different parameter configurations can represent the same function. It theoretically characterizes these symmetries for LoRA, prefix tuning, and prompt tuning in multihead attention, and develops symmetry-based parameter alignment for model merging. Experiments evaluate linear mode connectivity across vision and language tasks, robustness under distribution shifts, and LoRA merging across multiple downstream tasks. Results show that alignment substantially reduces interpolation loss barriers for LoRA and prefix tuning, while improving LoRA merging performance, but does not consistently establish linear mode connectivity for prompt tuning.

Critical strengths
  1. The paper provides rigorous symmetry characterizations with maximality results under explicit assumptions, extending the theoretical understanding of parameter redundancy in PEFT.
  2. The analysis covers LoRA, prefix tuning, and prompt tuning, including the effects of rotary positional embeddings on their symmetry structures.
  3. The proposed alignment method demonstrates substantial reductions in interpolation barriers and competitive model merging results across vision and language tasks.
Critical weaknesses
  1. The theoretical guarantees rely on specific rank and invertibility assumptions in a single multihead attention module, limiting their applicability to practical architectures with more general parameterizations.
  2. The empirical evaluation of symmetry-based alignment is limited, particularly for prompt tuning and prefix tuning in model merging, leaving its broader practical benefits unclear.
  3. The model-merging experiments rely on a small number of task collections, and hyperparameter selection on validation data raises concerns about the generalizability of the reported gains.
Questions for authors
  1. How robust are the symmetry characterizations and alignment results when the assumptions on frozen attention projections are violated or only approximately satisfied?
  2. Why does symmetry-based alignment fail to achieve comparable linear mode connectivity for prompt tuning, and can this failure be distinguished from limitations of the proposed matching algorithm?
  3. Can the authors demonstrate that the merging improvements generalize across more diverse task combinations and independent evaluation data, given that method selection and coefficient tuning use validation sets?
4 comments

We will incorporate all revisions and additional tables into the revised manuscript. Please see our global comment for details.

W1+Q1. The rank and invertibility assumptions establish completeness and uniqueness of the characterization, rather than the validity of the symmetry transformations: θ\theta and gθg\theta represent the same function even when these assumptions fail. However, additional functional equivalences may arise, so f(⋅;θ)=f(⋅;θˉ)f(\cdot;\theta)=f(\cdot;\bar\theta) need not imply θˉ=gθ\bar\theta=g\theta for any gg in the stated group. For example, in standard attention, an admissible query perturbation satisfying ΔQiKi⊤=0\Delta Q_iK_i^\top=0 leaves the attention scores unchanged. Thus, the maximality conclusion cannot generally be extended beyond its stated assumptions.

Similar parameter assumptions appear in symmetry and identifiability results for attention and feedforward networks [1–4]. Our excluded degeneracies, including rank deficiency and coincident prefix keys, lie in proper real algebraic subsets and thus have Lebesgue measure zero, establishing generic completeness.

Layerwise symmetries are inherited by models formed through composition. For example, if f1(⋅;θ1)f_1(\cdot;\theta_1) and f2(⋅;θ2)f_2(\cdot;\theta_2) have maximal parameter symmetry groups G1G_1 and G2G_2, respectively, then G1×G2G_1\times G_2 acts as a symmetry group of f1∘f2f_1\circ f_2. However, this product group need not be maximal, since coupled parameter transformations may preserve the overall function without preserving individual layer functions. Characterizing a multilayer model’s full symmetry group is therefore substantially harder than analyzing its constituent components.

W2. Alignment for prefix and prompt tuning is evaluated on ViT and Qwen2.5 in the connectivity experiments (Section 7.1, Figure 1). Our merging study (Section 7) focuses on LoRA because of its widespread deployment and the availability of large collections of independently trained task adapters, a practical setting for model merging. The revised manuscript adds experiments that further confirm that our alignment method is effective: Table 6: LoRA alignment methods for linear mode connectivity on vision and language benchmarks (Appendix C) compares it with Git Re-Basin and OT on linear mode connectivity, Tables 15 and 16: Accuracy of both LMC-LoRA variants when the alignment step is replaced by Git Re-Basin or OT fusion (Appendix E.3) compare it with them in merging, and Tables 17 and 18: Accuracy of LMC-LoRA greedy with and without the alignment step (Appendix E.3) compare merging with and without alignment.

W3+Q3. Task coverage and generalization. Our merging study covers two collections with different backbones and task types, comprising five NLI tasks on BERT-base and seven GLUE tasks on Llama-3.2-1B. For the language model, SNLI and WNLI are held out from both training and selection. Greedy achieves the best average on these unseen tasks (Table 3). Its advantage also holds when performance is averaged over merge orders across both collections (Tables 11–12).

Use of validation data. Validation-based hyperparameter tuning is common in model merging because training sets are often unavailable at merge time. Greedy follows the same protocol as baselines with scale parameters, using no more validation data. On NLI, it uses only 2% of the validation split, while baselines use the full split. We acknowledge that dependence on labelled data limits greedy selection. Our dynamic variant requires no data at merge time and clearly outperforms Simple Average, although it does not yet match greedy.

Q2. Prompt tuning permits fewer alignments because permutations must be shared across heads, and RoPE can leave only the identity. We tested several strategies that reduced barriers without recovering comparable LMC. Our observations suggest continuous alignment is easier to refine numerically, while data-free permutation matching optimizes surrogate objectives that need not minimize interpolation barriers. For nontrivial groups, we cannot conclusively distinguish matching limitations from the absence of LMC within admissible transformations. For trivial groups, only the identity is available, so search failure cannot explain the remaining barrier.

[1] Tran et al. (2026) Functional Equivalence in Attention: A Comprehensive Study with Applications to Linear Mode Connectivity

[2] Chen et al. (1993) On the Geometry of Feedforward Neural Network Error Surfaces

[3] Fefferman and Markel (1993) Recovering a Feed-Forward Net From Its Output

[4] Phuong and Lampert (2020) Functional vs. Parametric Equivalence of ReLU Networks

[5] Theus et al. (2025) Generalized Linear Mode Connectivity for Transformers

[6] Zhang et al. (2025) Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion

[7] Singh et al. (2020) Model fusion via optimal transport

[8] Li and Liang (2021) Prefix-Tuning: Optimizing Continuous Prompts for Generation

AuthorsOpenReview ↗

Any Questions from Reviewer a3sk on Our Rebuttal?

We would like to thank the reviewer again for your thoughtful review and valuable feedback.

We would appreciate it if you could let us know if our responses have addressed your concerns and whether you still have any other questions about our rebuttal.

We would be happy to do any follow-up discussion or address any additional comments.

AuthorsOpenReview ↗

Another Call for Your Response

Dear Reviewer a3sk,

We sincerely appreciate the time you have taken to provide feedback on our work, which has helped us greatly improve its clarity, among other attributes. This is a gentle reminder that the Paper Discussion Phase will end in less than 2 days from this comment, i.e., 11:59 pm AoE on November 18. We are happy to answer any remaining questions you may have or provide further clarification before then, but we will no longer be able to respond once the discussion phase has concluded.

If you agree that our responses to your reviews have addressed the concerns you listed, we kindly ask that you consider whether raising your score would more accurately reflect your updated evaluation of our paper. Thank you again for your time and thoughtful comments!

Sincerely,

Authors

Thank you for the detailed rebuttal and clarifications. The authors' responses have largely addressed my concerns. I will therefore maintain my current score of weak accept.

Reviewer_b4zTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper theoretically characterizes the parameter-space symmetries of LoRA, prefix tuning, and prompt tuning in multi-head attention, formally proving which distinct parameter configurations yield identical functions. Based on these findings, the authors propose efficient, data-free alignment algorithms to resolve parameterization redundancies before interpolating or merging independently fine-tuned models. Experiments demonstrate that applying these symmetry-aware alignments nearly eliminates loss barriers for linear mode connectivity in LoRA and prefix tuning, while enabling new merging methods that achieve state-of-the-art performance in multi-task model fusion.

Critical strengths

1.The paper presents detailed theoretical derivations and formally characterizes the parameter-space symmetries of LoRA, prefix tuning, and prompt tuning using group theory, proving their maximality while accounting for modern architectural details such as RoPE. 2.The paper demonstrates across vision and language tasks that symmetry-aware alignment nearly eliminates loss barriers for LMC, enabling LMC-LoRA merging methods to significantly outperform strong baselines such as Task Arithmetic and Ties-Merging. 3.The proposed method is simple and easy to reproduce.

Critical weaknesses

1.The LoRA analysis is restricted to Query/Value projections rather than all matrices (Q, K, V, O), and the maximality proofs rely on generic conditions that may not hold during practical optimization trajectories. 2. While the theoretical symmetry group for LoRA is the general linear group GL(r)GL(r), the proposed matching algorithm restricts optimization to the orthogonal subgroup O(r)O(r) for efficiency, potentially missing non-orthogonal transformations that could further reduce loss barriers. 3. Alignment fails to eliminate interpolation barriers for prompt tuning; although theoretically explained by its highly restricted symmetry group under RoPE, the paper offers no practical mitigation strategy. 4. The paper lacks direct benchmarking against established weight-matching techniques (e.g., Git Re-basin, Optimal Transport) adapted for PEFT scenarios.

Questions for authors

Could the authors clarify how the proposed symmetry-aware alignment compares empirically to established methods such as Git Re-basin and Optimal Transport? Specifically, it would be helpful to see a comparison of their effectiveness in terms of reducing loss barriers for LMC and improving model merging performance

3 comments

We will incorporate all revisions and additional tables into the revised manuscript. Please see our global comment for details.

W1. The rank and invertibility assumptions establish completeness and uniqueness of the characterization, rather than the validity of the symmetry transformations: θ\theta and gθg\theta represent the same function even when these assumptions fail. However, additional functional equivalences may arise, so f(⋅;θ)=f(⋅;θˉ)f(\cdot;\theta)=f(\cdot;\bar\theta) need not imply θˉ=gθ\bar\theta=g\theta for any gg in the stated group. For example, in standard attention, an admissible query perturbation satisfying ΔQiKi⊤=0\Delta Q_iK_i^\top=0 leaves the attention scores unchanged. Thus, the maximality conclusion cannot generally be extended beyond its stated assumptions.

Additional parameter assumptions also appear in previous symmetry and identifiability results, including those for attention and feedforward networks [1-4]. In our results, the excluded degeneracies, such as rank deficiency and coincident prefix keys, are contained in proper real algebraic subsets of the relevant parameter spaces and therefore have Lebesgue measure zero. This establishes generic completeness.

For LoRA with Value-Key matrices, we have extended the LoRA analysis to joint Q/K, V/O, and Q/K/V/O adaptation. Under explicit genericity and rank conditions, the maximal symmetry group remains a product of GL(rT)\mathrm{GL}(r_T), one per adapted projection, both with and without RoPE. The alignment transformations remain exactly function preserving even when these conditions fail. Please see Section 3 and Appendix A.1.1 of the revised manuscript.

W2. We thank the reviewer for this observation. Our choice of O(r)O(r) is motivated by empirical evidence and computational efficiency. [6] report small additional gains from rescaling beyond rotation-based matching, while [5] show that orthogonal alignment captures most of the barrier reduction before further refinement. [1] also demonstrate low- and zero-barrier connections between independently trained Transformers using orthogonal residual-stream alignment alongside feedforward and attention-head permutations. These findings motivate our use of orthogonal matching, which admits a closed-form SVD solution and exactly preserves each LoRA update. We acknowledge that full GL(r)GL(r) matching may provide further improvements.

W3. We explored several alignment strategies for prompt tuning. Although most reduced the interpolation loss barriers, a residual gap remained, and we could not recover LMC in these settings. Based on our observations, alignment over continuous symmetry groups tends to be more tractable, as the transformation matrices can be refined through numerical optimization. Permutation-based symmetries pose a combinatorial search challenge. Existing data-free matching methods typically optimize surrogate objectives, such as parameter distance, whose optima need not minimize interpolation loss barriers. In our experiments, these methods reduced barriers but did not consistently recover LMC. The restricted prompt symmetries under RoPE further limit the admissible transformations. Our strategies therefore provide partial mitigation, while achieving LMC in these configurations remains an open challenge.

W4+Q1. We thank the reviewer for this suggestion. We added data-free PEFT adaptations of Git Re-Basin and weight-based OT [7], keeping the backbone fixed. Git Re-Basin matches rank permutations using both LoRA factors. Exact balanced OT matches incoming weights and propagates the permutation to the outgoing factor. Our method uses orthogonal alignment.

Table 6: LoRA alignment methods for linear mode connectivity on vision and language benchmarks (Appendix C) reports LMC barriers and untuned equal-weight merging (λ=0.5)(\lambda=0.5) for ViT-LoRA on CIFAR-10/100 and ImageNet, and Qwen2.5-0.5B LoRA on WikiText-103, LM1B, and CC-News. It shows clear improvements over both baselines in the evaluated LoRA settings.

Direct-prefix objectives coincide under exact assignment. For projected-prefix MLPs, Git Re-Basin beats our row-only alignment on WikiText-103/LM1B, while ours beats both baselines on CC-News. The consistent advantage is therefore specific to the tested LoRA settings.

Model merging. We also replace our alignment step with Git Re-Basin and with OT inside both merging methods. Git Re-Basin is adapted as in Table 6, and OT follows [7] with the default settings of its public code. Only the alignment step changes; checkpoints, merge orders, coefficient grid, selection data and the λi=1/i\lambda_i = 1/i schedule are the same as in the paper. The full results are in Tables 15 and 16: Accuracy of both LMC-LoRA variants when the alignment step is replaced by Git Re-Basin or OT fusion (Appendix E.3) of the revised manuscript. Our alignment gives the highest average in every case.

References. Please see our response to Reviewer a3sk for the complete reference list.

AuthorsOpenReview ↗

Any Questions from Reviewer b4zT on Our Rebuttal?

We would like to thank the reviewer again for your thoughtful review and valuable feedback.

We would appreciate it if you could let us know if our responses have addressed your concerns and whether you still have any other questions about our rebuttal.

We would be happy to do any follow-up discussion or address any additional comments.

AuthorsOpenReview ↗

Another Call for Your Response

Dear Reviewer b4zT,

We sincerely appreciate the time you have taken to provide feedback on our work, which has helped us greatly improve its clarity, among other attributes. This is a gentle reminder that the Paper Discussion Phase will end in less than 2 days from this comment, i.e., 11:59 pm AoE on November 18. We are happy to answer any remaining questions you may have or provide further clarification before then, but we will no longer be able to respond once the discussion phase has concluded.

If you agree that our responses to your reviews have addressed the concerns you listed, we kindly ask that you consider whether raising your score would more accurately reflect your updated evaluation of our paper. Thank you again for your time and thoughtful comments!

Sincerely,

Authors

Comments on the paper

1 comment

Dear Chairs and Reviewers,

We sincerely thank you for your time and constructive feedback. We have made several revisions to improve the manuscript’s presentation and clarify our results. Changes relative to the previous version will be highlighted in blue.

In particular, we will include five additional tables addressing comparisons with established matching methods and the contribution of alignment to model merging. Inserting these tables will also change the numbering of some existing tables. All table numbers below refer to the revised manuscript.

  • Table 6. Comparison of LoRA alignment methods for linear mode connectivity and model merging on vision and language benchmarks. Values are mean ± sample standard deviation over three overlapping checkpoint pairs. Best results per metric are in bold.
  • Table 15. NLI, per-task validation accuracy (%) of both LMC-LoRA variants when the alignment step is replaced by Git Re-Basin or OT fusion. Every other setting is the one used in Table 2. The last column is the mean ± standard deviation of Avg over the merge orders of Table 11.
  • Table 16. Language model, per-task accuracy (%) of both LMC-LoRA variants when the alignment step is replaced by Git Re-Basin or OT fusion. Every other setting is the one used in Table 3. The last two columns are the mean ± standard deviation of Avg IID and Avg OOD over the merge orders of Table 12.
  • Table 17. NLI, per-task validation accuracy (%) of LMC-LoRA greedy with and without the alignment step, under the same coefficient search. Every other setting is the one used in Table 2. The last column is the mean ± standard deviation of Avg over the merge orders of Table 11.
  • Table 18. Language model, per-task accuracy (%) of LMC-LoRA greedy with and without the alignment step, under the same coefficient search. Every other setting is the one used in Table 3. The last two columns are the mean ± standard deviation of Avg IID and Avg OOD over the merge orders of Table 12.

We would be grateful if the reviewers could consider these additional results alongside our individual responses. Thank you again for your careful review and helpful suggestions.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.