acceptodds
Under review as a conference paper at ICLR 2027

LoRA-GACT: Gauge-Aware Continuation Transport for Optimizer-Preserving Resume

Abstract

LoRA checkpoint exchange has become routine infrastructure: adapters are merged, exported, compressed, and reloaded inside a single training process, and each of these handoffs is validated by one standard—function equivalence at the exchange point. This standard is silent about everything after the exchange. Real PEFT-SVD merge–export–reload–resume runs expose the gap: function-matched checkpoints follow different function-space trajectories, because the handoff changes factor coordinates and optimizer history while the training process continues. We define the missing second standard, optimizer-preserving resume correctness: agreement between the resumed continuation and the uninterrupted reference under a specified continuation system. Gauge-Aware Continuation Transport (LoRA-GACT) realizes this standard, lifting checkpoint conversion into continuation transport by jointly mapping factor-wise update rules, optimizer state, decay, and numerical metadata while retaining the original optimizer. Its theory derives gradient covariance, the global-learning-rate obstruction, and an update-commutation condition that yields multi-step restoration. Across tasks, optimizers, models, ranking evaluation, and real checkpoint exchange, handoffs that pass today's standard still produce trajectory and decision drift, whereas LoRA-GACT restores the specified continuation to numerical precision with zero candidate search.

Primary area:
optimization
Keywords:
LoRA, parameter-efficient fine-tuning, optimizer state transport, checkpoint continuation, gauge transformation, optimization dynamics, function-preserving handoff

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xqsuOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper looks at the possibility of resuming LoRA training after function-preserving reparameterisations of the adapter factors. They note that preserving the product of the LoRA factors does not in fact guarantee that the subsequent optimisation trajectories will be preserved and propose LoRA-GACT which transforms the optimiser state and update hyperparameters to maintain equivalence with the reference run that would have been achieved had the training been uninterrupted. I have skimmed the proofs but have not checked the derivations step-by-step.

Critical strengths
  • The paper deals with a very well-defined problem and the proposed transformations are quite sensible from a mathematical point of view.
  • The authors carry out a wide range of experiments across optimisers, models and checkpoint exchange implementation, showing that their method achieves its aim.
Critical weaknesses
  1. Very low originality and no real motivation.

The fact that different parameterisations of the same function can lead to different optimisation trajectories is a well-known observation. The proposed solution of transforming the optimiser state and update rules to the same invertible transformation also seems fairly straightforward. A lot of the theoretical work is just applying change-of-coordinates identities to specific optimiser recurrences. The authors should be clear what they would regard as a genuinely new theoretical contribution that provides insight to the reader.

  1. Excessively verbose and full of metaphor.

The quality of the scientific writing is frankly very poor. The manuscript includes many instances where it replaces a clear explanation with lofty language and unnecessary jargon. "The silence is not hypothetical" (line 50), "the missing second standard" (line 80) and the incessant use of "transport boundaries" and "continuation objects" are all examples.

  1. Poor figure design.

Figs 1 and 2 are over the top in terms of clutter with too many arrows, too many decorative elements and too many small annotations. Fig 3 has 14 individual panels which makes it very difficult to look at the results at normal reading size. All of these figures would be vastly improved by a significant degree of simplification. The figures should also be vector graphics instead of raster.

  1. The experiments establish algebraic consistency rather than practical usefulness.

Much of the evaluation shows that applying the derived transforms leads to a preserved reference trajectory whereas not applying them does not. That's of course as we would expect given the theoretical basis. At no point does the evaluation establish any practical utility of the method.

Questions for authors
  1. What meaningful and novel insights can we draw from your work? I mean insights that would at least surprise, if not alter, the academic community's beliefs in some way.
  2. How often does exact trajectory preservation after checkpoint conversion actually matter in a real training pipeline?
  3. Why isn't inverse gauge restoration included as the baseline?
Reviewer_RZTfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows that reparameterizing LoRA factors can change subsequent training trajectories even when the effective weights remain unchanged. It proposes LoRA-GACT, which jointly transforms factor-wise learning rates, optimizer states, weight decay, and numerical parameters to preserve the original training dynamics. Theoretical analysis and experiments demonstrate restoration of the uninterrupted training trajectory to numerical precision under supported settings.

Critical strengths
  • (1) The paper focuses on an important problem about LoRA checkpoint conversion. The paper is very easy to understand and follow.

  • (2) The paper illustrates that tuning s global learning may not good enough and proposes an interesting conversion to recover the optimization trajectory of multiple training steps

  • (3) The paper tries to adjust the update rules, optimizer state, and the corresponding settings like weight decay, which can let the training resume from the previous state and the same training trajectory.

  • (4) The experiments cover several optimizers, model architectures, and downstream tasks. They also include an actual PEFT-SVD checkpoint conversion with training resumed in a fresh process, which supports the method’s practical value.

Critical weaknesses
  • (1) The recovery guarantee for AdamW only covers scalar and positive diagonal transformations. The full derivation for general non-diagonal transformations is mainly for SGD and Momentum, so the combination of a real SVD reparameterization with AdamW is not well covered yet.

  • (2) The largest evaluated model is Qwen3-1.7B. The main experiments focus on a small dataset and short continuation windows. I‘m not sure whether the method is scalable. The paper does not provide experiments on a larger model and a longer training task.

  • (3) When the model adapter is released publicly, it is not very easy to reproduce the experiments and continue fine-tuning with the proposed method. Because the coordinate transformation, optimization state, and training configuration are still needed.

  • (4) Most experiments rely on artificially chosen scaling transformations. The paper does include a real PEFT-SVD experiment, but it should show more clearly how often this kind of trajectory drift happens in common training pipelines, and how much final performance it typically costs.

Questions for authors

N/A

Reviewer_oV8aOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper is about resuming LoRA training after the factors are reparameterized (e.g., A→cAA \to cA, B→B/cB \to B/c). The authors observe that keeping BABA and the model output unchanged is not enough. After resuming, the training trajectory can still drift away from the uninterrupted run. They call the missing property optimizer-preserving resume correctness and propose LoRA-GACT. Basically, it rescales the per-factor learning rates, the optimizer states, weight decay and epsilon together with the factors, so that training after the conversion follows the same path as training without it.

Critical strengths
  1. The research problem is clear. Keeping the training process consistent across a checkpoint transformation is relevant to reproducibility and experimental validation.

  2. The theoretical derivations are coherent and consistent. The gradient covariance relation, the factor-wise learning-rate transport for SGD, and the argument from one-step commutation to multi-step restoration follow a coherent line.

Critical weaknesses
  1. The figure quality is poor. Figures 1 and 2 are low-resolution images, and the x-axis tick labels overlap in Figure 4.

  2. LoRA-GACT requires the gauge map GG to be known (Sec. 4.5; App. F.6). Given GG, the simplest alternative is to map the factors back, A=G−1A′A = G^{-1}A' and B=B′GB = B'G, and resume with the stored optimizer state unchanged. The paper does not report this baseline, and it is unclear what advantage LoRA-GACT offers over it.

  3. For SST-2, the reference model only gets 51.38% accuracy, about the same as random guessing. For Beauty, NDCG@10 drops by only 0.000339, and the paper does not report its variance. The training runs are also short. So I am not convinced these results show that the drift would hurt a model with good task performance.

Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.