- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper focuses on training LoRA adapters for multi-objective fine-tuning. It identifies an issue arising from the fact that the LoRA adapters are non-identifiable as infinitely many factor pairs correspond to the same adapter. Such non-identifiable factorization affects the gradients of each objective and thus the multi-objective optimization procedure as a whole. Focusing on the multiple-gradient descent algorithm (MGDA) for multi-objective optimization, two simple examples show that different factorizations result in either different objective weights or identical objective weights but different gradient updates. The paper then introduces two update rules for the LoRA factors ensuring identical optimization trajectories of distinct but equivalent parametrizations. This is achieved by re-casting optimization domain as the manifold of matrices with a fixed rank instead of the two LoRA factors, performing a gradient descent step on the tangent space of this manifold computed at the current LoRA adapter, and projecting the result back to the manifold via Frobenius norm minimization. Experiments show that the proposed methods are robust to equivalent perturbations of LoRA factors, in the sense that the perturbed factors do not result in contradictory gradients, as well as that the effective perturbed and unperturbed LoRA adapters remain very close during optimization when the proposed algorithms are used. Finally, a real-world example shows that DPO finetuning while balancing helpfulness and safety show that the methods achieves higher scores on these two metrics while improving hypervolume of the pareto frontier, HarmBench and XSTest scores.
Critical strengths
The problem is well motivated with clear examples, and the proposed solution solution elegantly extend the insight of BaLoRA, to perform optimization on the set of low-rank matrices, to multi-objective optimization on a manifold. The paper also extends the methods to stateful optimizers.
Critical weaknesses
The key motivating issue raised by the paper, namely "if two LoRA factorizations represent exactly the same effective model, they should induce the same multi-objective optimization behavior" (L040) is not solved by the proposed algorithms. In a narrow sense, each of the proposed algorithms satisfies this requirement in isolation (Theorems 5 and 6), however, "QPA and SPA can nevertheless select different Pareto weights, showing that representation correctness does not uniquely determine the optimization geometry" (L314) and "Representation correctness does not require QPA and SPA to choose the same optimization geometry." (L1259). This directly contradicts the "same multi-objective optimization behavior" desiderata, and since different algorithms result in different optimization trajectories, it is not clear why representation correctness is a desirable property at all.
The usefulness of such property is also questionable in light of the fact that "the simplex set producing an exactly zero combined factor gradient is identical across all equivalent full-rank factorizations" (L173) and "exact first-order Pareto stationarity is representation-invariant" (L176). Adapter re-parameterization thus only changes convergence properties of the optimization algorithm, and it is not clear why this is valuable from the paper. In fact, BaLoRA already showed that the convergence speed depends on adapter parameterization, but the experiments in the paper used the same number of steps and step size for all methods without showing convergence; it is thus unclear whether performance gain comes from faster convergence speed. If this is not the case, since "exact first-order Pareto stationarity is representation-invariant", the demonstrated performance gains are rather surprising and need to be explained.
The exposition is very hard to follow and exhibits typical LLM-isms such as name-dropping and talking about things without introducing them first. A few rounds of editing and proof-reading are necessary to clarify key aspects of the paper. Critically, the motivation for SPA, namely "SPA removes the non-orthogonal factor ambiguity through product-preserving Stiefelization" (L259) is highly unintelligible: what is the non-orthogonal factor, why is it ambiguous, why is it a problem, and why/how does "Stiefelization" solve this problem? It sounds like this ambiguity is a flaw of QPA that stands to be corrected, but experiments show these algorithms to be qualitatively different rather than SPA being the superior adaptation. What is then the need for SPA? The notation (L264) is ambiguous: does the big-oh notation refer to a matrix norm (which one?) or to individual entries of the matrix O? Another example is "a strict rank-r spectral cutoff gap for every accepted trial" (L252): what are accepted trials? What is the acceptance criterion? What matrix needs to have its spectrum altered, and why?
Questions for authors
- Please clarify why representation correctness for an algorithm is a desirable property, even when (1) different algorithms can still exhibit different optimization trajectories and (2) stationary points in the loss landsape are not changed by different parameterizations.
- If the performance gains in Table 2 are due to improved optimization speed, provide evidence in form of loss curves and hyper-parameter optimization; using the same hyper-parameters does not automatically establish superiority of the proposed algorithms.
- If the performance gains in Table 2 are not due to improved optimization speed, please provide evidence that all models converged and that the best hyper-parameters were used for each of them, and explain why improved performance of converged models is to be expected even though stationary points are representation-independent.