acceptodds
Under review as a conference paper at ICLR 2027

LoRA-GACT: Gauge-Aware Continuation Transport for Optimizer-Preserving Resume

Abstract

LoRA checkpoint exchange has become routine infrastructure: adapters are merged, exported, compressed, and reloaded inside a single training process, and each of these handoffs is validated by one standard—function equivalence at the exchange point. This standard is silent about everything after the exchange. Real PEFT-SVD merge–export–reload–resume runs expose the gap: function-matched checkpoints follow different function-space trajectories, because the handoff changes factor coordinates and optimizer history while the training process continues. We define the missing second standard, optimizer-preserving resume correctness: agreement between the resumed continuation and the uninterrupted reference under a specified continuation system. Gauge-Aware Continuation Transport (LoRA-GACT) realizes this standard, lifting checkpoint conversion into continuation transport by jointly mapping factor-wise update rules, optimizer state, decay, and numerical metadata while retaining the original optimizer. Its theory derives gradient covariance, the global-learning-rate obstruction, and an update-commutation condition that yields multi-step restoration. Across tasks, optimizers, models, ranking evaluation, and real checkpoint exchange, handoffs that pass today's standard still produce trajectory and decision drift, whereas LoRA-GACT restores the specified continuation to numerical precision with zero candidate search.

Then back it, or bet against it.

Related papers

Open the market on this paper to see 7 more related papers.