- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper shows that two LoRA learners with the same effective update make identical predictions but can select different examples when active learning uses factor gradients. In rank-one Gaussian constructions, the authors prove that this difference can make the label cost of the same acquisition rule grow with pool size across equivalent factorizations. They propose a factorization-independent acquisition Gram based on the inverse-Bures geometry of Balanced LoRA. Controlled experiments with Qwen3-1.7B on AG News show that raw acquisition separates equivalent learners, while forced-common purchases and the invariant Gram restore exact correspondence. A retrospective MultiNLI analysis shows repeatable per-example correctness changes with similar mean accuracy.
Critical strengths
- Precise problem formulation and carefully controlled experiments.
- The paper clearly defines factorization independence of acquisition and isolates acquisition from the update rule with matched optimizers and randomness.
- Forced-common purchases and the invariant Gram exactly recover learner correspondence in all 24 fit–round endpoints, and additional replays on AG News and TREC confirm the invariance.
- Careful statistical reporting and strong reproducibility.
- Claims are stated with clear scope, and non-significant results are reported honestly (e.g., intervals including zero, an unmet prospective criterion).
- Proofs, numerical-rank audits, and code are provided in detail.
Critical weaknesses
- The practical impact of factorization dependence is not yet clear.
- Scope of the theory: Theorems 1 and 2 are existence results on constructed tasks, where one representation’s Gram matches the true covariance. They show how large the effect can be, but not how large it is in realistic LoRA settings.
- Neural outcomes: Raw acquisition changes 17–36 of 128 first-round purchases. However, the main signed NLL contrast (−0.032) has an interval including zero, so no consistent directional effect on learning performance is established.
- MultiNLI analysis: The analysis is retrospective, and the prospective criterion was unmet. Also, per-example correctness changes may be expected for any two different purchase sets, so the result may not be specific to factorization.
- Practical settings and guidance: It remains unclear how often equivalent LoRA factorizations arise in practical gradient-based data-selection pipelines, beyond the controlled settings and adapter-composition example. In addition, the paper discusses other invariant Grams (e.g., PRISM’s tangent-Frobenius Gram), but there is no empirical comparison to guide which geometry to use.
- The presentation makes the contributions difficult to evaluate.
- The text is very compressed and uses many self-defined terms ("purchase", "arm", "admission", "forced-common").
- Much of the evidence is deferred to a lengthy appendix, making the main claims difficult to assess from the main text alone.
Questions for authors
Clarification Questions
- Comparison of invariant Grams: How do inverse-Bures acquisition, the ambient Gram of effective gradients, and PRISM’s tangent-Frobenius Gram compare in label efficiency on AG News?
- Practical frequency: In which practical settings do equivalent factorizations appear, besides adapter composition in PEFT?
- Specificity of MultiNLI result: How large is the per-example response between two random purchase sets of the same size, compared with the response between the two refactorizations?
- Magnitude in neural settings: Across NLL targets not already reached by the old-only baseline, how frequently do the two raw recipes differ in label cost, and what is the typical magnitude of this difference?
Suggestions
- Please distinguish more clearly in the introduction between factorization-independent acquisition and improved label efficiency, which depends on the acquisition geometry and task.
- If possible, evaluate the invariant Gram in a real gradient-based data-selection pipeline (e.g., LESS or ActiveDPO) at a larger scale.
- Please simplify the terminology and move key evidence from the appendix to the main text.