- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies global parameter recovery for LoRA and multi-head attention after quotienting out specified architectural symmetries. Each candidate recovery rule must be continuous on the entire space of probability distributions and must always return a point in the target quotient space. The recovery guarantee requires that, for every target, at least one candidate be sufficiently accurate. By showing that the candidates’ success domains form a categorical cover, the paper relates list size to Lusternik–Schnirelmann category and derives explicit lower bounds through cohomological calculations.
Critical strengths
-
A clear theoretical framework. Theorem 3.6 connects recovery to LS category. Proposition C.1 strengthens this connection by establishing that, under injectivity of the observation map and additional stated conditions, the minimum list size equals this topological invariant.
-
Several limitations are acknowledged. The authors distinguish list size from runtime and sample complexity and explain that local recovery or a larger output space changes the problem. The explicit Frobenius error threshold for LoRA is also useful.
Critical weaknesses
-
The importance of the recovery objective is not sufficiently established. The paper requires recovery of the generating parameters up to specified architectural symmetries. However, prediction, distillation, and many deployment tasks require only behavioral agreement. Even when a locally executable adapter is desired, recovering the original generating parameter class may be unnecessary if a behaviorally equivalent replacement suffices. The paper mentions teacher–student identification, adapter reconstruction, and parameter comparison, but does not establish a concrete setting in which these weaker objectives are inadequate. This affects the significance of the research question. A specific task should clarify the available observations, the recovery target, and why recovering the generating parameter class is necessary.
-
The lower bound critically depends on global continuous proper extension, whose necessity remains unclear. Definition 3.1 requires each candidate to be continuous on all of and to return a point in the specified parameter space for every distribution. This is substantially stronger than stable recovery on the realizable observation set , or on a neighborhood of that set.
-
LoRA and MHA share a geometric structure, but their recovery problems are not equally well unified. There is a legitimate mathematical reason to study these architectures together: LoRA involves a low-rank product, one attention head involves query–key and value–output products, and multiple heads introduce a permutation quotient. Nevertheless, shared geometry does not imply shared identifiability or recovery requirements.
-
The implications for practical methods remain indirect. The paper does not demonstrate a concrete recovery method that satisfies all the assumptions and is consequently constrained by the theorem, or show how relaxing an assumption preserves a task’s essential guarantees while avoiding the obstruction. I do not expect a theoretical paper to include conventional empirical benchmarks. Nevertheless, a worked example showing how the theorem changes our assessment of a concrete recovery problem would strengthen the contribution.
Questions for authors
See Weaknesses.
My concerns include the importance of the chosen recovery objective, the necessity of the strong global assumptions, and the explanatory value of the results for concrete learning problems. The calculations are technically substantive, but the current manuscript does not sufficiently establish that they characterize an important machine learning bottleneck. I therefore lean toward rejection. A compelling task-level example and a clearer separation of global extension obstructions, and practical recovery requirements could change my assessment.