- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The work shows that by forecasting future weights of a model in a learned latent space, it is possible to accelerate neural network training. The authors train an autoencoder to encode the model checkpoints and adjust this encoding so that linear extrapolation in latent space predicts future weight changes accurately after decoding. Once this representation gets fixed, they train latent forecaster, an MLP or transformer to predict future encoded points. Experiments on several models such as CNN, Resnet, small GPT-2 models show improvements over other weight space forecasting baseline.
Critical strengths
- The latent forecasters outperform the baselines WNN and Nino across the three architectures.
- The learned latent representations can make training trajectories easier to forecast.
Critical weaknesses
- The primary weakness is the total cost of training the encoder, decoder, linear predictor and latent predictor all of them. It seems just simple training is cheaper and there is really no requirement for using a fancy compression based forecasting method.
- The method requires a separate autoencoder and forecaster for each architecture family, limiting its reuse when the network architecture changes.
- Compression introduces reconstruction error. The paper should show how much model performance changes when weights are encoded and decoded without any forecasting.
- The font of the paper do not match standard iclr font.
- The paper does not discuss Bach’s Regularized Nonlinear Acceleration (2016), or its neural-network extension by Scieur et al. (2018). These methods also accelerate optimization by extrapolating past parameter iterates.
Questions for authors
-
Can the authors report the total cost for training all the other models separately and how much improvement does it give over simply letting the original model run its training course.
-
If I switch to a different architecture family, how much of the LFNT pipeline must be trained again, and would the savings still justify that effort?
5 comments
Response to Reviewer 3Uxz (2/3)
Performance without any forecasting
To evalaute the performance of the models without any forecasting, we performed an additional ablation. We encode and decode models with no forecasting and resume training from these reconstructed weights, using the same evaluation protocol as LFNT.
ResNet (CIFAR-100): optimizer-step speedup (%)
Width multipliers are relative to the original smaller ResNet18 configuration used in the paper.
| Architecture | AE-only | Latent MLP | Latent Transf. |
|---|---|---|---|
| R14 | 2.2 | 34.4 | 36.3 |
| R26 | 5.3 | 27.3 | 32.4 |
| R34 | -2.9 | 22.9 | 30.1 |
| Width 2x | -5.3 | 22.9 | 24.5 |
| Width 4x | -11.7 | 14.1 | 15.2 |
GPT-2: optimizer-step speedup (%)
Forecasting the token embedding table for the new widths
| Architecture | AE-only | Latent Transf. |
|---|---|---|
| Depth 2-24 | 20.5 | 55.0 |
| Depth 4-24 | 21.2 | 57.8 |
| Depth 6-24 | 19.7 | 58.1 |
| Width 3-32 | 15.0 | 43.9 |
| Width 3-48 | 10.3 | 30.1 |
| Width 3-64 | 7.0 | 18.5 |
The results show that autoencoder reconstruction alone cannot explain the observed speedups, particularly for architectures unseen during autoencoder training. For example, on ResNet-34, reconstruction yields -2.9% speedup, compared with 30.1% for LFNT. The positive reconstruction-only speedups for GPT-2 may come from projecting checkpoints toward the learned weight distribution. However, this effect diminishes for wider models, whilst latent forecasting continues to provide substantial speedup.
Reconstruction performance. We agree that the compression introduces reconstruction error which can degrade the performance of the model right after reconstruction, even if the MSE is low as some parameters can be particularly sensitive. However, we do not aim to create a model that performs well right after decoding but rather a model that serves as a better starting point for subsequent optimization, quantified by the speedup. Nevertheless we quantify this by evaluating the reconstructed models right after decoding as well as after 100 steps of continued optimization, we evaluate both the reconstruction from Stage I as well as Stage II. For comparison we have also included the result for training a model from scratch, to visualize the recovery of the decoded model.
Vision: validation accuracy (%), higher is better.
| Architecture (original performance) | Method | Step 0 | Step 100 |
|---|---|---|---|
| ResNet18 (57.8) | Stage I | 24.5 | 53.7 |
| Stage II | 1.0 | 45.2 | |
| Training from scratch | 1.0 | 9.5 | |
| EfficientNet-B0 (50.7) | Stage I | 1.0 | 28.4 |
| Stage II | 0.9 | 14.7 | |
| Training from scratch | 0.8 | 2.1 | |
| MobileNetV2 (46.3) | Stage I | 1.0 | 30.9 |
| Stage II | 0.9 | 20.2 | |
| Training from scratch | 0.9 | 2.3 |
GPT-2 (LM1B): validation perplexity, lower is better.
Original checkpoint perplexity: 403
| Method | Step 0 | Step 100 |
|---|---|---|
| Stage I | 1488 | 452 |
| Stage II | 1630 | 466 |
| Training from scratch | 50455 | 25172 |
The results show that even though reconstruction does degrade the immediate performance it recovers very quickly once training resumes. Stage II reconstructs worse than Stage I which is understandable given that the latent space is optimized for forecasting in the second stage. We want to highlight that LFNT achieves substantial speedups despite these reconstruction errors, which reiterates why the direct reconstruction performance does not reflect the performance of a forecasting approach.
Response to Reviewer 3Uxz (3/3)
New baseline - ”Regularized Nonlinear Acceleration”
We thank the reviewer for pointing out this line of work, we agree that it is related to our setting and will include and discuss this in the related-works section.
However an important difference is that RNA performs extrapolation using recent optimization iterates, making its offline setting closer to checkpoint averaging methods such as Stochastic Weight Averaging (SWA) than to *trajectory forecasting. Offline RNA constructs an improved checkpoint without modifying the underlying training trajectory, whereas LFNT learns to forecast future weights from which optimization can resume. RNA can also be applied in an online setting, although this was shown in the paper to be less effective for neural networks.
We evaluated both online and offline RNA, as well as their combination with LFNT, using the same training-speedup metric. Online RNA combines the five context checkpoints that LFNT receives as input into a single extrapolated model, then continues training of that model to the target. Offline RNA never alters the training run. Training proceeds normally, and at every checkpoint (every 200 steps) we apply RNA to a sliding window of the most recent checkpoints (the last five). We evaluate the combined model and record the first step at which it reaches the target accuracy. The combined model is used only for evaluation and is never fed back into training. LFNT + RNA applies the same offline procedure to the training run that continues after the LFNT jump, using only post-jump checkpoints.
Optimizer-step speedup (%) compared with RNA, averaged over 3 trajectory runs
| Dataset | Online RNA | Offline RNA | LFNT | LFNT+RNA |
|---|---|---|---|---|
| ResNet CIFAR-10 | -0.2 | 44.4 | 23.4 | 51.0 |
| ResNet CIFAR-100 | -2.2 | 31.5 | 36.7 | 56.8 |
| ResNet TinyImageNet | -2.7 | 24.5 | 25.1 | 44.7 |
| ResNet Food-101 | -0.7 | 21.2 | 36.4 | 49.2 |
| ResNet Mean | -1.5 | 30.4 | 30.4 | 50.4 |
| GPT-2 (LM1B) | -1.4 | -3.9 | 59.7 | 57.1 |
For ResNet, online RNA does not improve training speed, whereas offline RNA achieves an average speedup of 30.4%, matching LFNT. Combining RNA with LFNT further increases this to 50.4%, showing that they can be complementary. For GPT-2, neither RNA variant provides positive speedups, whilst LFNT achieves 59.7%. These results establish offline RNA as a competitive but complementary checkpoint-improvement baseline rather than a replacement for forecasting, which is the main focus of this work.
Font
We thank the reviewer for bringing this discrepancy to our attention. The paper was compiled with LuaLaTeX instead of pdfLaTeX by accident, which resulted in a different interpretation of the official fonts. This was actually to our disadvantage as the resulting font takes up more space. In the revision we will amend this and use pdfLaTeX.
Response to Reviewer 3Uxz (1/3)
We thank Reviewer 3Uxz for their review as we respond to the reviewer’s questions and concerns and provide additional experiments.
Architectural generalization and computational overhead
We address the reviewer's concerns about computational overhead and architectural generalization through additional experiments and a cost analysis.
Architectural generalization. Beyond the transfer to unseen model depths shown in Table 4, we performed additional experiments evaluating speedup for wider ResNets and GPT-2 models. In addition, we test three different vision architectures, MobileNetV2, EfficientNet-B0, and DenseNet-121, scaled down to similar param counts as our resnets. These experiments reuse the trained autoencoder and forecaster.
ResNet (CIFAR-100): optimizer-step speedup (%) across widths
Width multipliers are relative to the original smaller ResNet18 configuration used in the paper.
Cross-architecture generalization (OOD datasets): optimizer-step speedup (%)
GPT-2: optimizer-step speedup (%) across widths
Forecasting the token embedding table.
These results show that LFNT generalizes to unseen architectures and achieves higher speedups than WNN and NiNo under the evaluated architectural changes. Furthermore, recent works such as [A] and [B] have shown that shared weight representations can be learned across heterogeneous architectures. Using such general-purpose weight encoders in LFNT could extend latent forecasting to a broader range of architectures, but is left for future work.
Computational overhead. We measured the computational cost of the LFNT pipeline. Training the autoencoder (Stages I and II) and latent forecaster requires ~9 GPU-hours for ResNet and ~6 GPU-hours for GPT-2, using the H100 gpu. The main upfront cost is model-zoo construction, requiring ~80 GPU-hours for ResNet and ~200 GPU-hours for GPT-2 using V100s. However this trajectory collection cost is also incurred by other learned weight-space forecasting methods such as WNN and NiNo, and to reduce this cost for future research we will make our model zoos public.
Regarding the amortization question, we want to highlight that in several cases our LFNT needs to be trained once and reused for many training runs, including runs on novel tasks and deeper/wider architectures. We have estimated that, excluding the model-zoo construction, the ResNet pipeline breaks even after ~ 600 accelerated runs in the original evaluation setup, decreasing to ~100 runs for the wider models. For GPT-2, amortization is more favorable, requiring ~50 runs when training until the convergence target. We acknowledge that including model-zoo construction substantially increases the break-even point. However that comparison would be unfair to our method since it ignores the dataset and architecture generalization properties it demonstrates. In particular, one trained instance of our method can be used across different datasets and architecture configurations. Further, recent works [A, B] have shown that weight representations can be learned from publicly available models, and our pretrained WeightCLIP experiments (Table 9) show that LFNT can reuse existing autoencoders. Although our method relies on model-zoos including training trajectories, these arguments hint that future work could aim to train the weight encoder on publicly available data making its amortization easier.
[A] Joëlle Hanna, Damian Falk, Stella X. Yu, and Damian Borth. Geosane: Learning geospatial representations from models, not data. CVPR 2026
[B] Damian Falk, Konstantin Schürholt, Konstantinos Tzevelekakis, Léo Meynent, and Damian Borth. Learning Model Representations Using Publicly Available Model Hubs. arXiv preprint 2025.