acceptodds
Under review as a conference paper at ICLR 2027

Same Predictor, Different Learners: How LoRA Factorization Changes Active Learning

Abstract

Two low-rank adaptation (LoRA) learners can make identical predictions and equivalent updates on common data, yet choose different examples to label next. In rank-one Gaussian task families, exact refactorization under the same posterior-variance rule produces a label-cost ratio for the same prediction risk that grows without bound with pool size. The gap persists with arbitrarily accurate conditional covariance and nearly optimal information about the teacher: learning private coefficients improves few predictions, whereas learning a shared coefficient improves many. We formulate a factorization-independent acquisition Gram from effective updates and effective gradients, retaining the geometry of balanced factor gradients. With matched selection inputs and randomness, it preserves ordered purchases across equivalent factorizations; with corresponding common-data updates, it also preserves learning paths. Controlled multi-round experiments with Qwen3 classifiers isolate acquisition as the source of divergence and recover learner correspondence through common purchases or the invariant Gram. A retrospective analysis of repeated training continuations on MultiNLI finds repeatable per-example correctness changes despite similar mean accuracy. Equivalence on common training data therefore does not extend automatically to active learning: the acquisition geometry must also be specified.

Primary area:
learning theory
Keywords:
LoRA, active learning, parameter-efficient fine-tuning, data acquisition, reparameterization invariance, gradient geometry, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jdJPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows that two LoRA learners with the same effective update make identical predictions but can select different examples when active learning uses factor gradients. In rank-one Gaussian constructions, the authors prove that this difference can make the label cost of the same acquisition rule grow with pool size across equivalent factorizations. They propose a factorization-independent acquisition Gram based on the inverse-Bures geometry of Balanced LoRA. Controlled experiments with Qwen3-1.7B on AG News show that raw acquisition separates equivalent learners, while forced-common purchases and the invariant Gram restore exact correspondence. A retrospective MultiNLI analysis shows repeatable per-example correctness changes with similar mean accuracy.

Critical strengths
  1. Precise problem formulation and carefully controlled experiments.
  • The paper clearly defines factorization independence of acquisition and isolates acquisition from the update rule with matched optimizers and randomness.
  • Forced-common purchases and the invariant Gram exactly recover learner correspondence in all 24 fit–round endpoints, and additional replays on AG News and TREC confirm the invariance.
  1. Careful statistical reporting and strong reproducibility.
  • Claims are stated with clear scope, and non-significant results are reported honestly (e.g., intervals including zero, an unmet prospective criterion).
  • Proofs, numerical-rank audits, and code are provided in detail.
Critical weaknesses
  1. The practical impact of factorization dependence is not yet clear.
  • Scope of the theory: Theorems 1 and 2 are existence results on constructed tasks, where one representation’s Gram matches the true covariance. They show how large the effect can be, but not how large it is in realistic LoRA settings.
  • Neural outcomes: Raw acquisition changes 17–36 of 128 first-round purchases. However, the main signed NLL contrast (−0.032) has an interval including zero, so no consistent directional effect on learning performance is established.
  • MultiNLI analysis: The analysis is retrospective, and the prospective criterion was unmet. Also, per-example correctness changes may be expected for any two different purchase sets, so the result may not be specific to factorization.
  • Practical settings and guidance: It remains unclear how often equivalent LoRA factorizations arise in practical gradient-based data-selection pipelines, beyond the controlled settings and adapter-composition example. In addition, the paper discusses other invariant Grams (e.g., PRISM’s tangent-Frobenius Gram), but there is no empirical comparison to guide which geometry to use.
  1. The presentation makes the contributions difficult to evaluate.
  • The text is very compressed and uses many self-defined terms ("purchase", "arm", "admission", "forced-common").
  • Much of the evidence is deferred to a lengthy appendix, making the main claims difficult to assess from the main text alone.
Questions for authors

Clarification Questions

  1. Comparison of invariant Grams: How do inverse-Bures acquisition, the ambient Gram of effective gradients, and PRISM’s tangent-Frobenius Gram compare in label efficiency on AG News?
  2. Practical frequency: In which practical settings do equivalent factorizations appear, besides adapter composition in PEFT?
  3. Specificity of MultiNLI result: How large is the per-example response between two random purchase sets of the same size, compared with the response between the two refactorizations?
  4. Magnitude in neural settings: Across NLL targets not already reached by the old-only baseline, how frequently do the two raw recipes differ in label cost, and what is the typical magnitude of this difference?

Suggestions

  1. Please distinguish more clearly in the introduction between factorization-independent acquisition and improved label efficiency, which depends on the acquisition geometry and task.
  2. If possible, evaluate the invariant Gram in a real gradient-based data-selection pipeline (e.g., LESS or ActiveDPO) at a larger scale.
  3. Please simplify the terminology and move key evidence from the appendix to the main text.
Reviewer_qLGmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper observes that LoRA factor gradients transform as convectors under the exact refactorization (B,A)→(BQ,Q−1A)(B,A) \rightarrow (BQ, Q^{-1}A), so any acquisition rule built on a Euclidean Gram of concatenated factor gradients shall select different examples from two checkpoints that define the identical predictor. It constructs rank-one Gaussian tasks in which a scalar rescaling c=2c=2 flips the variance rule from buying "shared" to "private" coefficients, giving label-cost rations of Ω(N)\Omega(\sqrt{N}) (Thm.1) and >N16>\frac{N}{16} even with uniformly δ−\delta-accurate conditional covariance and near-optimal teacher information (Thm.2). It also proposes KbalK_bal, the Gram of balanced-factor gradients, which has the factor-free form <Gi,PLGj+GjPR><G_i, P_L G_j + G_j P_R> with PLP_L and PRP_R is invariant across the  ΔW~_\Delta W class including rank-deficient cases, and preserves the entire ordered selection path when other selector inputs are matched.

Critical strengths
  1. The core observation is correct and well-stated. Fig.1 and Prop.1 make the failure mode obvious in two lines, and the paper connects it to real pipelines where LoRA gradient features drive data selection.

  2. K_bal is a sensible invariant with a tidy factor-free expression. The thin-QR + r×rr \times r SVD computation, the rank-loss recovery via two small least-squares solves and the column-streaming selector are all worked out with care.

Critical weaknesses
  1. This paper establishes that two factorizations select differently, but not that why it matters. On AG News, random acquisition is competitive with every gradient-based arm. The pre-specified primary contrast between raw c=1c=1 and c=2c=2 has a bootstrap range spanning zero. The fixed-purchase replications (table 15) leave all six conditional means unresolved. The raw-vs-raw trajectory separation is the same order as the same-purchase training-stream variation in Table 9. The only significant number, the complete-consumption contrast of 0.0044 NLL, is tiny and 76% from round one. So the acquisition rule under study is not shown to beat random, and the divergence it indices is not shown to exceed seed noise. Here, this method is more like a non-variance of a method that has no demonstrated benefit.

  2. The theory makes the balanced factorization optimal by construction. In both theorem the checkpoint is chosen so that the balanced Gram equals the true prior covariance. The c=2c=2 arm is then compared to an oracle. The symmetric construction (c=2c=2 calibrated, c=1c=1 miscalibrated) would give the the opposite ordering. The theorems therefore show that a scalar covariance mis-scaling can cost Ω(N)\Omega(\sqrt{N}) or Ω(N)\Omega(N) labels under a greedy variance rule, a valuable but different claim from the one suggested by the framing, and one that is close to the Chen et al. pivoting counterexample the authors cite. This paper only acknowledges half of this as "its balanced Gram matches the true covariance here" but the abstract does not.

Questions for authors

See above weaknesses.

Reviewer_siavOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how equivalent LoRA update (ΔW = BA = (BQ)(Q⁻¹A)) affect gradient-based active learning. This work shows that equivalent LoRA predictors can select different query batches when acquisition relies on gradient. The key idea is to compute gradient-based acquisition scores in SVD coordinate system rather than from raw (B, A) factor gradients, ensuring that equivalent LoRA factorizations always select the same examples.

Critical strengths
  • The paper points out an interesting issue of combining factorization-invariant optimizers with gradient-based active learning does not preserve equivalence across the full active learning loop

  • Tractable invariance fix: The proposed formulation (via thin QR and small r × r SVD) computes the invariant Gram matrix without materializing the dense ΔW matrix, making it practical.

Critical weaknesses
  • Presentation needs a lot of improvement by having undefined/non-standard terminology: The paper is not easy to follow, particularly in the abstract and introduction. It has undefined, or non-standard terms introduced before formal setup (e.g. Sec 2). From the very first sentence ("Two low-rank adaptation (LoRA) learners... choose different examples to label next"), the paper jumps into mechanics without properly framing the active learning setting. Some terms, such as "purchases" (used throughout instead of standard terms like queries or acquisitions), "private vs. shared coefficients," "cost gap," and "conditional covariance" are not commonly used in active learning community. For being self contained, I'd also suggest the authors to even define the problem setting of active learning even very concisely in few sentences.

  • Overclaimed scope relative to a narrow acquisition setting: The title and framing suggest a broad finding for "Active Learning" with LoRA. However, the theoretical and empirical scope is strictly limited to factor-gradient Gram/inner-product acquisition rules. The title and claims should be narrowed to reflect that this only affects gradient-based acquisition.

  • Trivial baseline mechanism (Proposition 1) and the theoretical constructions (Theorems 1–2): Proposition 1 is elementary linear algebra: multiplying by Q and Q⁻¹ preserves BA, and taking non-linear operations will break it. To show sample complexity, Thm 1/2 rely some Gaussian construction, which is standard for many analysis, probably more analyzable, but the criticism is always how practical it is.

  • Related the above, the experiment doesn't show significant difference in overall test NLL or accuracy between the different factorizations compared with its theory argument.

  • The experimental design choices can be better clarified. The paper evaluates a 1.7B LLM on simple AG News text classification, which seems too easy for modern LLM, and not commonly used for evaluating LLM performance.

Questions for authors
  • Clarifying terminology and scope

  • Practical implications beyond Proposition 1

  • Justification of experiment setting

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.