- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper looks at the possibility of resuming LoRA training after function-preserving reparameterisations of the adapter factors. They note that preserving the product of the LoRA factors does not in fact guarantee that the subsequent optimisation trajectories will be preserved and propose LoRA-GACT which transforms the optimiser state and update hyperparameters to maintain equivalence with the reference run that would have been achieved had the training been uninterrupted. I have skimmed the proofs but have not checked the derivations step-by-step.
Critical strengths
- The paper deals with a very well-defined problem and the proposed transformations are quite sensible from a mathematical point of view.
- The authors carry out a wide range of experiments across optimisers, models and checkpoint exchange implementation, showing that their method achieves its aim.
Critical weaknesses
- Very low originality and no real motivation.
The fact that different parameterisations of the same function can lead to different optimisation trajectories is a well-known observation. The proposed solution of transforming the optimiser state and update rules to the same invertible transformation also seems fairly straightforward. A lot of the theoretical work is just applying change-of-coordinates identities to specific optimiser recurrences. The authors should be clear what they would regard as a genuinely new theoretical contribution that provides insight to the reader.
- Excessively verbose and full of metaphor.
The quality of the scientific writing is frankly very poor. The manuscript includes many instances where it replaces a clear explanation with lofty language and unnecessary jargon. "The silence is not hypothetical" (line 50), "the missing second standard" (line 80) and the incessant use of "transport boundaries" and "continuation objects" are all examples.
- Poor figure design.
Figs 1 and 2 are over the top in terms of clutter with too many arrows, too many decorative elements and too many small annotations. Fig 3 has 14 individual panels which makes it very difficult to look at the results at normal reading size. All of these figures would be vastly improved by a significant degree of simplification. The figures should also be vector graphics instead of raster.
- The experiments establish algebraic consistency rather than practical usefulness.
Much of the evaluation shows that applying the derived transforms leads to a preserved reference trajectory whereas not applying them does not. That's of course as we would expect given the theoretical basis. At no point does the evaluation establish any practical utility of the method.
Questions for authors
- What meaningful and novel insights can we draw from your work? I mean insights that would at least surprise, if not alter, the academic community's beliefs in some way.
- How often does exact trajectory preservation after checkpoint conversion actually matter in a real training pipeline?
- Why isn't inverse gauge restoration included as the baseline?