acceptodds
Under review as a conference paper at ICLR 2027

Gauge Symmetries Impose Limits on Global Parameter Recovery for LoRA and Attention

Abstract

Low-Rank Adaptation (LoRA) updates and Multi-Head Attention (MHA) heads have inherent symmetries, so many weight arrays describe the same update or attention computation. In this work, we ask how many deterministic continuous candidate rules are needed to recover parameters up to these symmetries. Each rule takes a population distribution as input and returns a point in the (compact) quotient space and we require that, for every target, at least one rule returns a nearby point. We show that for even embedding dimension and even rank , sufficiently accurate LoRA recovery requires at least candidates in the low rank-regime. So for example, at and , this gives that continuous candidate rules are required for recovery. For even and attention heads with equal even key/value dimension , we show a corresponding lower bound of . Thus stable, uniform, population-level proper recovery requires lists of size for LoRA and for MHA. Our approach for the proof is to turn the success domains of the candidate rules into a topologically simple cover and then argue about its size through nonzero products in cohomology, which is measured by the cup-length. We develop a cup-length addition principle for fiber bundles admitting a global section under a freeness condition, and show that stronger hypotheses yield an exact formula. We do an explicit computation using tools from algebraic topology to identify the surviving classes for the quotient parameter space of LoRA, and symmetrized products handle head permutations for MHA. Together, these necessary list sizes establish an architecture-level lower bound to stable parameter reconstruction for both LoRA and MHA.

Primary area:
learning theory
Keywords:
LoRA, multi-head attention, gauge symmetries, parameter recovery, list recovery, topological lower bounds, Lusternik–Schnirelmann category, cup-length

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HvgkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies global parameter recovery for LoRA and multi-head attention after quotienting out specified architectural symmetries. Each candidate recovery rule must be continuous on the entire space of probability distributions and must always return a point in the target quotient space. The recovery guarantee requires that, for every target, at least one candidate be sufficiently accurate. By showing that the candidates’ success domains form a categorical cover, the paper relates list size to Lusternik–Schnirelmann category and derives explicit lower bounds through cohomological calculations.

Critical strengths
  1. A clear theoretical framework. Theorem 3.6 connects recovery to LS category. Proposition C.1 strengthens this connection by establishing that, under injectivity of the observation map and additional stated conditions, the minimum list size equals this topological invariant.

  2. Several limitations are acknowledged. The authors distinguish list size from runtime and sample complexity and explain that local recovery or a larger output space changes the problem. The explicit Frobenius error threshold for LoRA is also useful.

Critical weaknesses
  1. The importance of the recovery objective is not sufficiently established. The paper requires recovery of the generating parameters up to specified architectural symmetries. However, prediction, distillation, and many deployment tasks require only behavioral agreement. Even when a locally executable adapter is desired, recovering the original generating parameter class may be unnecessary if a behaviorally equivalent replacement suffices. The paper mentions teacher–student identification, adapter reconstruction, and parameter comparison, but does not establish a concrete setting in which these weaker objectives are inadequate. This affects the significance of the research question. A specific task should clarify the available observations, the recovery target, and why recovering the generating parameter class is necessary.

  2. The lower bound critically depends on global continuous proper extension, whose necessity remains unclear. Definition 3.1 requires each candidate to be continuous on all of P(Z)\mathcal P(Z) and to return a point in the specified parameter space for every distribution. This is substantially stronger than stable recovery on the realizable observation set G(X)G(X), or on a neighborhood of that set.

  3. LoRA and MHA share a geometric structure, but their recovery problems are not equally well unified. There is a legitimate mathematical reason to study these architectures together: LoRA involves a low-rank product, one attention head involves query–key and value–output products, and multiple heads introduce a permutation quotient. Nevertheless, shared geometry does not imply shared identifiability or recovery requirements.

  4. The implications for practical methods remain indirect. The paper does not demonstrate a concrete recovery method that satisfies all the assumptions and is consequently constrained by the theorem, or show how relaxing an assumption preserves a task’s essential guarantees while avoiding the obstruction. I do not expect a theoretical paper to include conventional empirical benchmarks. Nevertheless, a worked example showing how the theorem changes our assessment of a concrete recovery problem would strengthen the contribution.

Questions for authors

See Weaknesses.

My concerns include the importance of the chosen recovery objective, the necessity of the strong global assumptions, and the explanatory value of the results for concrete learning problems. The calculations are technically substantive, but the current manuscript does not sufficiently establish that they characterize an important machine learning bottleneck. I therefore lean toward rejection. A compelling task-level example and a clearer separation of global extension obstructions, and practical recovery requirements could change my assessment.

Reviewer_r135OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies parameter recovery for Low Rank Adaptation (LoRA) and Multi Head Attention (MHA) after quotienting out architectural symmeties. It considers a finite family of deterministic and continuous recovery rules, each defined on the entire space of population distributions and returning a point in a compact quotient parameter space. The central argumen conencts this recovery problem to Listernik-Schnirelmann category. Since the space of probability distributions is contractible, each recovery rule induces a null-homotopic map on the target space. The authors also proves an abstract matching characterization in terms of category when the observation map is injective.

Critical strengths

The reduction in Theorem 3.6 is easy to interpret once the recovery model is specified. It separates a topological obstruction from sampling limitations and applies beyond the two architectures considered. Also, the LoRA analysis explicitly constructs surviving fiber classes and estabilishes their independence over the base cohomology. The attention analysis additionally handles head permutations through invariant products, including the singular strata where heads coincide.

Critical weaknesses

None

Questions for authors
  1. Can the authors provide a concrete LoRA or attention recovery example whose candidate maps are continuous on the full distribution space?

  2. In the statement and proof of Theorem 6.8. are you using Remark 6.7 or 3.7?

Reviewer_YDMDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies parameter recovery using a fixed list of deterministic, globally continuous maps from population distributions into compact quotient parameter spaces. It proves that sufficiently accurate uniform proper recovery requires at least as many maps as the Lusternik–Schnirelmann category of the target space. Under injectivity and metrizability assumptions, this also characterizes the minimum list size. The paper then derives explicit lower bounds for exact-rank LoRA cores and multi-head attention quotients. In the low-rank regime, these bounds grow at least proportionally to rank times ambient dimension for LoRA, and to head count times rank times ambient dimension for attention. Its contributions include a cup-length addition result, a real cohomology calculation for the LoRA quotient, and permutation-invariant products for attention. Validation is theoretical, through detailed proofs and explicit calculations.

Critical strengths
  1. A precise connection between recovery and a topological invariant. The reduction to LS category provides a coherent explanation of why multiple continuous recovery rules can be necessary. Proposition C.1 strengthens this connection by establishing equality with the minimum list size under the stated injectivity and metrizability assumptions. This characterization gives the framework value beyond the individual numerical lower bounds.

  2. Substantive calculations for the LoRA and attention quotient spaces. The LoRA argument establishes independence of surviving fiber classes over base cohomology and handles their descent through orientation reversal. The attention argument constructs nonzero permutation-invariant products that descend to the quotient, including configurations with nontrivial stabilizers. These calculations support explicit architecture-dependent bounds and constitute meaningful technical work.

Critical weaknesses
  1. The applicability of global proper recovery remains limited. Every candidate must continuously return a point in the exact-rank target family on all population distributions, including unrealizable ones. Appendix B.1 appropriately acknowledges that neighborhood decoding and allowing rank-deficient outputs change the problem. However, the consequences for the motivating recovery, alignment, and merging tasks remain conditional on their pipelines satisfying these strong requirements. A concrete explanation of when this restriction is essential would strengthen the significance of the results. This limits the reach of the conclusions without invalidating the stated theorems.

  2. The relationship to the closest literature needs sharper discussion. Relevant comparisons include de Haan and Falorsi’s Topological Constraints on Homeomorphic Auto-Encoding (2018), Dym et al.’s Equivariant Frames and the Impossibility of Continuous Canonicalization (2024), and Lőrincz and Raicu’s Borel–Moore Homology of Determinantal Varieties (2023), particularly its cohomology calculations for complex fixed-rank matrix spaces. These works address different formulations or spaces and do not establish the submission’s real quotient-recovery results. Nevertheless, a direct comparison would clarify the novelty of the recovery framework and the additional difficulties resolved by the real cohomology calculation.

Questions for authors
  1. Clarification of applicability: Can you give a concrete instance of a motivating recovery task in which continuous outputs must remain in the exact-rank family for every population distribution? In that instance, why would neighborhood decoding or allowing rank-deficient outputs be unsuitable?

  2. Clarification of contribution: How should the framework and real quotient-space calculation be distinguished from the continuous encoding/canonicalization obstructions and complex fixed-rank cohomology results cited in Weakness 2? Which technical steps or conclusions do those existing results not provide?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.