Back to the Weights: Training-Free Model Enhancement via Training Trajectory Extrapolation

Fine-tuning large language models with cosine learning-rate decay drives the step size to near zero long before the model reaches a true optimum. The training direction $\Delta ={\theta }_{\mathrm{end}}-{\theta }_{\mathrm{start}}$—an aggregate of thousands of gradient steps—still points toward lower loss, yet the vanishing learning rate prevents the model from reaching it. This gap is not an artifact of a particular run but a structural consequence of the schedule, leaving exploitable structure in the weight space beyond the training endpoint. We present TrajectoryForge, a training-free framework that closes this gap: given only two checkpoints (${\theta }_{\mathrm{start}}$ and ${\theta }_{\mathrm{end}}$), it extrapolates along $\Delta$ with per-matrix stochastic scaling factors, producing an improved single model without any gradient computation or inference overhead. Across 7 model pairs (1.5B–14B parameters), directed perturbation matches or exceeds the fine-tuning endpoint for 6 of 7 models, with gains up to +7.1% on HumanEval. Hessian spectral analysis reveals that the training direction occupies a geometrically privileged “sharp valley,” enabling $2\times$ the success rate of isotropic random search. Cross-task evaluation further confirms a direction-task alignment principle: improvement on aligned tasks with no degradation on orthogonal ones, and large-scale held-out validation (529 candidates on MBPP) confirms that models selected without target benchmark information still exceed the fine-tuning endpoint.