- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PEFT introduces structured trainable parameters into frozen models, but different parameter configurations can represent the same function. This paper characterizes the parameter-space symmetries of LoRA, prefix tuning, and prompt tuning in multihead attention, revealing redundancies in their parameterizations. This paper provides a theoretical foundation for studying linear mode connectivity and model merging by distinguishing functional differences from those arising solely from parameter representations.
Critical strengths
1.The paper investigates how different PEFT parameter configurations can represent the same function and how such redundancies affect parameter alignment, LMC, and model merging. Extending parameter-space symmetry analysis from conventional neural networks to multiple PEFT methods provides an interesting perspective and meaningful theoretical contribution.
2.The paper characterizes the parameter-space symmetries of LoRA, prefix tuning, and prompt tuning in multihead attention, considering both standard attention and RoPE. Beyond identifying function-preserving transformations, the paper establishes the maximality of the corresponding symmetry groups under specified assumptions.
3.The paper applies its theoretical findings to parameter alignment, LMC, and model merging. Table 1 demonstrates substantial reductions in interpolation loss barriers after alignment, while the model-merging experiments on BERT and Llama show competitive performance. The evaluation across vision and language tasks provides empirical support for the practical relevance of the theoretical results.
Critical weaknesses
1.The abstract and Introduction require substantial improvement in organization and writing, as they currently read more like an early draft than a carefully structured and polished manuscript. The Introduction resembles a survey of related concepts instead of a coherent motivation for the proposed work. In particular, the research gap and contributions relative to existing symmetry studies are not clearly articulated, making it difficult for readers to understand the paper's positioning and significance.
2.The theoretical results rely on specific PEFT configurations and assumptions. For example, the LoRA analysis considers only query and value adaptation with frozen key and output projections, while prefix tuning focuses on directly parameterized key–value prefixes.
3.The proposed matching algorithms do not fully exploit the identified function-preserving transformations. Although the LoRA symmetry group is characterized as , the practical algorithm considers only its orthogonal subgroup. While this restriction simplifies computation, the paper does not sufficiently investigate whether it limits alignment quality or whether exploiting a broader set of symmetries could further reduce interpolation loss barriers.
4.Although the experiments cover both vision and language tasks, they rely largely on traditional benchmarks and relatively small models, leaving the applicability of the findings to larger, modern LLMs insufficiently explored. Furthermore, many recent PEFT variants and more general adaptation settings are not considered.
Questions for authors
1.The LoRA matching algorithm considers only the orthogonal subgroup of the theoretically identified (GL(r)) symmetry group. Could the authors clarify the practical implications of this restriction in Weakness 3? Would incorporating a broader set of function-preserving transformations further improve linear mode connectivity?
2.The proposed greedy merging method combines symmetry-based parameter alignment with validation-based coefficient selection. Could the authors provide an ablation comparing merging with and without alignment under the same coefficient-selection strategy?
3 comments
I thank the authors for their clarifications and additional experiments, which have addressed my main concerns regarding W2 and W3, as well as my questions Q1 and Q2. I also understand the limitations discussed in response to W4.
However, my concern regarding W1 has not been fully resolved. I respect the authors' choice of writing style, but regardless of the structure adopted, the introduction should establish a clear logical connection between the research motivation, research gap, and contributions.
Although the authors have added transitions between sections, the introduction still reads largely as a sequence of descriptions of related work. The introduction still lacks a sufficiently coherent argument connecting the relevant literature to the paper's central research motivation. As a result, readers must navigate several relatively independent background paragraphs before fully understanding the paper's positioning and significance. I would encourage the authors to strengthen the logical progression and synthesis of related work to improve readability and reduce the cognitive burden on readers.
The abstract would also benefit from further refinement. While it identifies the methods studied, it does not clearly articulate the research gap relative to existing symmetry studies or highlight the specific theoretical findings. Consequently, readers understand what the paper investigates but gain limited insight into what new understanding it provides and why these results are significant. A more explicit summary of the key findings would help communicate the paper's contributions more effectively.
Nevertheless, considering the authors' clarifications and additional experimental evidence, which have addressed my main concerns, I have decided to raise my score to 3 (Weak Accept).
We thank the reviewer for their endorsement and constructive feedback. We acknowledge the remaining concerns regarding the clarity and coherence of the introduction and abstract. In the final revision, we will strengthen the logical connection between the research motivation, gap, and contributions in the introduction, and refine the abstract to more explicitly highlight the research gap, our key theoretical findings, and their significance.
Authors
We will incorporate all revisions and additional tables into the revised manuscript. Please see our global comment for details.
W1. We thank the reviewer for the suggestion. However, we respectfully prefer to retain the current writing format. Some introductions present the material as a continuous narrative, while others organize it into several thematic parts. We follow the latter approach.
We have improved the writing within each part and added transitions to help readers understand our contributions. In summary:
While parameter-space symmetries have been studied for feedforward neural networks, multihead attention, and mixture-of-experts architectures, the symmetry structure of parameter-efficient fine-tuning methods remains less well understood. Freezing the backbone and restricting the trainable parameters change which function-preserving transformations are available. We characterize these transformations for LoRA, prefix tuning, and prompt tuning in multihead attention, making their parameter redundancies explicit and providing a foundation for studying linear mode connectivity and model merging in PEFT.
W2. The rank and invertibility assumptions establish completeness and uniqueness of the characterization, rather than the validity of the symmetry transformations: θ and gθ represent the same function even when these assumptions fail. However, additional functional equivalences may arise, so f(⋅;θ)=f(⋅;θˉ) need not imply θˉ=gθ for any g in the stated group. For example, in standard attention, an admissible query perturbation satisfying ΔQiKi⊤=0 leaves the attention scores unchanged. Thus, the maximality conclusion cannot generally be extended beyond its stated assumptions.
Additional parameter assumptions also appear in previous symmetry and identifiability results, including those for attention and feedforward networks [1-4]. In our results, the excluded degeneracies, such as rank deficiency and coincident prefix keys, are contained in proper real algebraic subsets of the relevant parameter spaces and therefore have Lebesgue measure zero. This establishes generic completeness.
For prefix tuning, our formulation follows the standard key–value representation of prefix tuning introduced by [8]. For LoRA, we have extended the LoRA analysis to joint Q/K, V/O, and Q/K/V/O adaptation. Under explicit genericity and rank conditions, the maximal symmetry group remains a product of GL(rT), one per adapted projection, both with and without RoPE. The alignment transformations remain exactly function preserving even when these conditions fail. Please see Section 3 and Appendix A.1.1 of the revised manuscript.
W3+Q1. Our choice of orthogonal matching is motivated by empirical evidence and computational efficiency. [6] reports small additional gains from rescaling beyond rotations, while [5] shows that orthogonal alignment captures most barrier reduction before refinement. [1] demonstrates low- and zero-barrier connections between independently trained Transformers using orthogonal residual-stream alignment with feedforward and attention-head permutations. Our matching admits a closed-form SVD solution and exactly preserves each LoRA update. We acknowledge that broader GL(r) matching may further improve alignment and reduce interpolation barriers.
W4. Our experiments cover ViT and BERT, alongside Qwen2.5 for language modeling and Llama-3.2-1B for model merging. Standard benchmarks enable controlled comparisons of alignment and connectivity. We focus on LoRA, prefix tuning, and prompt tuning as distinct parameterizations with different symmetry structures. Our theoretical results apply across attention-module dimensions under the stated assumptions, while validation at substantially larger scales remains necessary. Additional PEFT variants and broader adaptation settings may require dedicated symmetry analyses, which we regard as important future work.
Q2. On both merging benchmarks, we removed only alignment, retaining the paper’s checkpoints, task orders, coefficient grid, and selection data. Full results appear in Tables 17 and 18: Accuracy of LMC-LoRA greedy with and without the alignment step (Appendix E.3) of the revised manuscript.
Alignment improves average accuracy and reduces variability across merge orders in both domains. The moderate gap suggests that greedy search partly compensates for missing alignment. LoRA factor interpolation introduces cross-adapter terms (B_iA_j), which may disrupt merging when adapter bases are misaligned. Search can mitigate this by choosing coefficients near 0 or 1, limiting adapter mixing. Alignment can make intermediate coefficients more effective, allowing adapters to combine more successfully.
References. Please see our response to Reviewer a3sk for the complete reference list.