acceptodds
Under review as a conference paper at ICLR 2027

LFNT : Forecasting training trajectories in the latent space of weights

Abstract

Forecasting neural network weights can accelerate training by skipping portions of the optimization trajectory, but existing approaches operate directly in weight space. Motivated by the success of learned latent representations in other domains, we propose forecasting optimization trajectories in a learned latent space of weights. We introduce **L**atent **F**orecasting of **N**eural **T**rajectories (**LFNT**, pronounced “elephant”), a three-stage approach that (i) learns a latent representation of model weights, (ii) aligns these representations using forecasting supervision, and (iii) trains a forecaster on the resulting latent trajectories. Across vision and language tasks, latent-space forecasting provides greater training speedups than weight space forecasting baselines, with speedups of up to 59%. Overall, the results show that the representation of optimization trajectories substantially affects forecasting performance, and that a learned latent space can provide a better alternative to forecasting directly in weight space.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Weight Forecasting, Weight Nowcasting, Training Dynamics, Representation Learning, Predictive Learning, Latent Space Forecasting, Weight Space Learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_3UxzOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The work shows that by forecasting future weights of a model in a learned latent space, it is possible to accelerate neural network training. The authors train an autoencoder to encode the model checkpoints and adjust this encoding so that linear extrapolation in latent space predicts future weight changes accurately after decoding. Once this representation gets fixed, they train latent forecaster, an MLP or transformer to predict future encoded points. Experiments on several models such as CNN, Resnet, small GPT-2 models show improvements over other weight space forecasting baseline.

Critical strengths
  1. The latent forecasters outperform the baselines WNN and Nino across the three architectures.
  2. The learned latent representations can make training trajectories easier to forecast.
Critical weaknesses
  1. The primary weakness is the total cost of training the encoder, decoder, linear predictor and latent predictor all of them. It seems just simple training is cheaper and there is really no requirement for using a fancy compression based forecasting method.
  2. The method requires a separate autoencoder and forecaster for each architecture family, limiting its reuse when the network architecture changes.
  3. Compression introduces reconstruction error. The paper should show how much model performance changes when weights are encoded and decoded without any forecasting.
  4. The font of the paper do not match standard iclr font.
  5. The paper does not discuss Bach’s Regularized Nonlinear Acceleration (2016), or its neural-network extension by Scieur et al. (2018). These methods also accelerate optimization by extrapolating past parameter iterates.
Questions for authors
  1. Can the authors report the total cost for training all the other models separately and how much improvement does it give over simply letting the original model run its training course.

  2. If I switch to a different architecture family, how much of the LFNT pipeline must be trained again, and would the savings still justify that effort?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3Uxz (1/3)

We thank Reviewer 3Uxz for their review as we respond to the reviewer’s questions and concerns and provide additional experiments.


Architectural generalization and computational overhead

We address the reviewer's concerns about computational overhead and architectural generalization through additional experiments and a cost analysis.

Architectural generalization. Beyond the transfer to unseen model depths shown in Table 4, we performed additional experiments evaluating speedup for wider ResNets and GPT-2 models. In addition, we test three different vision architectures, MobileNetV2, EfficientNet-B0, and DenseNet-121, scaled down to similar param counts as our resnets. These experiments reuse the trained autoencoder and forecaster.

ResNet (CIFAR-100): optimizer-step speedup (%) across widths

Width multipliers are relative to the original smaller ResNet18 configuration used in the paper.

WidthWNNNiNoLatent MLPLatent Transf.
1x (ID)-5.64.734.938.9
2x (OOD)1.64.022.924.5
4x (OOD)–-5.514.115.2

Cross-architecture generalization (OOD datasets): optimizer-step speedup (%)

ArchitectureNiNoLatent Transf.
ResNet (reference)17.030.0
MobileNetV23.323.8
EfficientNet-B0-6.224.6
DenseNet-121-4.313.3
Mean (new architectures)-2.420.6

GPT-2: optimizer-step speedup (%) across widths

Forecasting the token embedding table.

ArchitectureLatent Transf.
3-3243.9
3-4830.1
3-6418.5

These results show that LFNT generalizes to unseen architectures and achieves higher speedups than WNN and NiNo under the evaluated architectural changes. Furthermore, recent works such as [A] and [B] have shown that shared weight representations can be learned across heterogeneous architectures. Using such general-purpose weight encoders in LFNT could extend latent forecasting to a broader range of architectures, but is left for future work.


Computational overhead. We measured the computational cost of the LFNT pipeline. Training the autoencoder (Stages I and II) and latent forecaster requires ~9 GPU-hours for ResNet and ~6 GPU-hours for GPT-2, using the H100 gpu. The main upfront cost is model-zoo construction, requiring ~80 GPU-hours for ResNet and ~200 GPU-hours for GPT-2 using V100s. However this trajectory collection cost is also incurred by other learned weight-space forecasting methods such as WNN and NiNo, and to reduce this cost for future research we will make our model zoos public.

Regarding the amortization question, we want to highlight that in several cases our LFNT needs to be trained once and reused for many training runs, including runs on novel tasks and deeper/wider architectures. We have estimated that, excluding the model-zoo construction, the ResNet pipeline breaks even after ~ 600 accelerated runs in the original evaluation setup, decreasing to ~100 runs for the wider models. For GPT-2, amortization is more favorable, requiring ~50 runs when training until the convergence target. We acknowledge that including model-zoo construction substantially increases the break-even point. However that comparison would be unfair to our method since it ignores the dataset and architecture generalization properties it demonstrates. In particular, one trained instance of our method can be used across different datasets and architecture configurations. Further, recent works [A, B] have shown that weight representations can be learned from publicly available models, and our pretrained WeightCLIP experiments (Table 9) show that LFNT can reuse existing autoencoders. Although our method relies on model-zoos including training trajectories, these arguments hint that future work could aim to train the weight encoder on publicly available data making its amortization easier.

[A] Joëlle Hanna, Damian Falk, Stella X. Yu, and Damian Borth. Geosane: Learning geospatial representations from models, not data. CVPR 2026

[B] Damian Falk, Konstantin Schürholt, Konstantinos Tzevelekakis, Léo Meynent, and Damian Borth. Learning Model Representations Using Publicly Available Model Hubs. arXiv preprint 2025.

AuthorsOpenReview ↗

Response to Reviewer 3Uxz (2/3)

Performance without any forecasting

To evalaute the performance of the models without any forecasting, we performed an additional ablation. We encode and decode models with no forecasting and resume training from these reconstructed weights, using the same evaluation protocol as LFNT.

ResNet (CIFAR-100): optimizer-step speedup (%)

Width multipliers are relative to the original smaller ResNet18 configuration used in the paper.

ArchitectureAE-onlyLatent MLPLatent Transf.
R142.234.436.3
R265.327.332.4
R34-2.922.930.1
Width 2x-5.322.924.5
Width 4x-11.714.115.2

GPT-2: optimizer-step speedup (%)

Forecasting the token embedding table for the new widths

ArchitectureAE-onlyLatent Transf.
Depth 2-2420.555.0
Depth 4-2421.257.8
Depth 6-2419.758.1
Width 3-3215.043.9
Width 3-4810.330.1
Width 3-647.018.5

The results show that autoencoder reconstruction alone cannot explain the observed speedups, particularly for architectures unseen during autoencoder training. For example, on ResNet-34, reconstruction yields -2.9% speedup, compared with 30.1% for LFNT. The positive reconstruction-only speedups for GPT-2 may come from projecting checkpoints toward the learned weight distribution. However, this effect diminishes for wider models, whilst latent forecasting continues to provide substantial speedup.


Reconstruction performance. We agree that the compression introduces reconstruction error which can degrade the performance of the model right after reconstruction, even if the MSE is low as some parameters can be particularly sensitive. However, we do not aim to create a model that performs well right after decoding but rather a model that serves as a better starting point for subsequent optimization, quantified by the speedup. Nevertheless we quantify this by evaluating the reconstructed models right after decoding as well as after 100 steps of continued optimization, we evaluate both the reconstruction from Stage I as well as Stage II. For comparison we have also included the result for training a model from scratch, to visualize the recovery of the decoded model.

Vision: validation accuracy (%), higher is better.

Architecture (original performance)MethodStep 0Step 100
ResNet18 (57.8)Stage I24.553.7
Stage II1.045.2
Training from scratch1.09.5
EfficientNet-B0 (50.7)Stage I1.028.4
Stage II0.914.7
Training from scratch0.82.1
MobileNetV2 (46.3)Stage I1.030.9
Stage II0.920.2
Training from scratch0.92.3

GPT-2 (LM1B): validation perplexity, lower is better.

Original checkpoint perplexity: 403

MethodStep 0Step 100
Stage I1488452
Stage II1630466
Training from scratch5045525172

The results show that even though reconstruction does degrade the immediate performance it recovers very quickly once training resumes. Stage II reconstructs worse than Stage I which is understandable given that the latent space is optimized for forecasting in the second stage. We want to highlight that LFNT achieves substantial speedups despite these reconstruction errors, which reiterates why the direct reconstruction performance does not reflect the performance of a forecasting approach.

AuthorsOpenReview ↗

Response to Reviewer 3Uxz (3/3)

New baseline - ”Regularized Nonlinear Acceleration”

We thank the reviewer for pointing out this line of work, we agree that it is related to our setting and will include and discuss this in the related-works section.

However an important difference is that RNA performs extrapolation using recent optimization iterates, making its offline setting closer to checkpoint averaging methods such as Stochastic Weight Averaging (SWA) than to *trajectory forecasting. Offline RNA constructs an improved checkpoint without modifying the underlying training trajectory, whereas LFNT learns to forecast future weights from which optimization can resume. RNA can also be applied in an online setting, although this was shown in the paper to be less effective for neural networks.

We evaluated both online and offline RNA, as well as their combination with LFNT, using the same training-speedup metric. Online RNA combines the five context checkpoints that LFNT receives as input into a single extrapolated model, then continues training of that model to the target. Offline RNA never alters the training run. Training proceeds normally, and at every checkpoint (every 200 steps) we apply RNA to a sliding window of the most recent checkpoints (the last five). We evaluate the combined model and record the first step at which it reaches the target accuracy. The combined model is used only for evaluation and is never fed back into training. LFNT + RNA applies the same offline procedure to the training run that continues after the LFNT jump, using only post-jump checkpoints.

Optimizer-step speedup (%) compared with RNA, averaged over 3 trajectory runs

DatasetOnline RNAOffline RNALFNTLFNT+RNA
ResNet CIFAR-10-0.244.423.451.0
ResNet CIFAR-100-2.231.536.756.8
ResNet TinyImageNet-2.724.525.144.7
ResNet Food-101-0.721.236.449.2
ResNet Mean-1.530.430.450.4
GPT-2 (LM1B)-1.4-3.959.757.1

For ResNet, online RNA does not improve training speed, whereas offline RNA achieves an average speedup of 30.4%, matching LFNT. Combining RNA with LFNT further increases this to 50.4%, showing that they can be complementary. For GPT-2, neither RNA variant provides positive speedups, whilst LFNT achieves 59.7%. These results establish offline RNA as a competitive but complementary checkpoint-improvement baseline rather than a replacement for forecasting, which is the main focus of this work.




Font

We thank the reviewer for bringing this discrepancy to our attention. The paper was compiled with LuaLaTeX instead of pdfLaTeX by accident, which resulted in a different interpretation of the official fonts. This was actually to our disadvantage as the resulting font takes up more space. In the revision we will amend this and use pdfLaTeX.

Reviewer_KPj3OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LFNT, which forecasts neural network training trajectories in a learned latent space of weights instead of directly in weight space. In Stage I, a weight autoencoder is trained on checkpoints from model zoos. In Stage II, its space is shaped using a closed-form linear extrapolator, the decoded weight displacement is matched to the true displacement. In Stage III, the autoencoder is frozen, and an MLP or Transformer forecaster is trained on the latent trajectories. At inference, a forecasted latent state is decoded and standard training resumes from it. In the main comparisons, the proposed method LFNT gives larger step and wall-clock savings than WNN and NiNo, with about 59% optimizer-step savings on LM1B. These gains also extend to unseen datasets and model depths. The authors also show greater directional consistency and robustness to perturbations in the learned latent space.

Critical strengths
  • The comparison with WNN is reasonably controlled. Latent MLP uses the same forecaster architecture and training recipe, and this makes table 1 a useful comparison of the representation spaces. The λ\lambda ablation in table 8 also shows a clear improvement from Stage II forecast shaping over the Stage I representation which supports the main claim, and I think whether the forecasting space matters is a useful question for weight space learning.
  • The gains hold across several settings. For example, latent forecasting outperforms both baselines in the aggregate ID results in table 1, on unseen datasets in table 2, as well as on unseen depths in table 4. The wall-clock results also account for encoding, forecasting and decoding costs. The hybrid result in Table 3 is interesting, which suggests that latent and weight-space forecasting can complement each other instead of competing.
Critical weaknesses
  • I am not fully convinced of the practical significance. The evaluation is limited in scale since the models are very small, the main GPT-2 model has 3 blocks with hidden size 24, and most parameters are in embedding. In table 3, the single-jump gain with h=4kh=4k falls from 58.0% to 45.0% at stricter targets. Table 7 shows speedup on AdamW trajectories falling from 59.2% at 2×10−42\times10^{-4}, the main zoo learning rate, to 7.0% at 6×10−36\times10^{-3}. The target perplexity also changes across learning rates, so the benefit over a tuned baseline at a common target is unclear. Also, the zoo cost is not counted in the speedup, it would be hard to tell how much this helps when it comes to a more realistic setting.
  • It is unclear how much of the gain comes from forecasting itself. The decoder may partly act as a projection toward typical trained weights, which could help independently of forecasting accuracy. The current analysis does not clearly separate these effects.
Questions for authors
  • What speedup do you get by using the Stage II linear extrapolator at inference? And how much does validation performance change when a checkpoint is only encoded and decoded without forecasting?
  • In the robustness experiment, how is the perturbation direction chosen? Is random noise rescaled to a fraction ff of the TstartT_{\text{start}}-to-TrefT_{\text{ref}} displacement norm, computed separately in each space?
  • Could you also report the std over the 20 runs in table 1 and 2?
4 comments
AuthorsOpenReview ↗

Response to Reviewer KPj3 (1/4)

We thank Reviewer KPj3 for their review as we respond to the reviewer’s questions and concerns and provide additional experiments.


Computational cost and generalization

We address the reviewer's concerns about computational overhead and generalization through additional experiments and a cost analysis.


Architectural generalization. Beyond the transfer to unseen model depths shown in Table 4, we performed additional experiments evaluating speedup for wider ResNets and GPT-2 models. In addition, we test three different vision architectures, MobileNetV2, EfficientNet-B0, and DenseNet-121, scaled down to similar param counts as our resnets. These experiments reuse the trained autoencoder and forecaster.

ResNet (CIFAR-100): optimizer-step speedup (%) across widths

Width multipliers are relative to the original smaller ResNet18 configuration used in the paper.

WidthWNNNiNoLatent MLPLatent Transf.
1x (ID)-5.64.734.938.9
2x (OOD)1.64.022.924.5
4x (OOD)–-5.514.115.2

Cross-architecture generalization (OOD datasets): optimizer-step speedup (%)

ArchitectureNiNoLatent Transf.
ResNet (reference)17.030.0
MobileNetV23.323.8
EfficientNet-B0-6.224.6
DenseNet-121-4.313.3
Mean (new architectures)-2.420.6

GPT-2: optimizer-step speedup (%) across widths

Forecasting the token embedding table.

ArchitectureLatent Transf.
3-3243.9
3-4830.1
3-6418.5

These results show that LFNT generalizes to unseen architectures and achieves higher speedups than WNN and NiNo under the evaluated architectural changes. Furthermore, recent works such as [A] and [B] have shown that shared weight representations can be learned across heterogeneous architectures. Using such general-purpose weight encoders in LFNT could extend latent forecasting to a broader range of architectures, but is left for future work.


Generalization across optimization settings. Our original Appendix B.2 (Table 7) already evaluates different learning rates and optimizers, including AdamW and Muon, which shows the performance for more well-tuned baselines. To further address the reviewer's question regarding well-tuned baselines, we performed additional experiments with a linear warmup followed by cosine learning-rate decay for the training schedule. Importantly, unlike the experiments in Table 7, these new comparisons use a fixed target validation performance across all learning rates. 57.8% for the ResNet and 352 for the language models.

ResNet (CIFAR-100): optimizer-step speedup (%) with 500-step warmup and cosine decay

Peak LRSpeedup (%)
6 × 10⁻⁴18.9
1.2 × 10⁻³16.1
1.8 × 10⁻³5.7

GPT-2: optimizer-step speedup (%) with 250-step warmup and cosine decay

Peak LRFixed contextLoss-matched context
4 × 10⁻⁴52.653.6
6 × 10⁻⁴34.737.2
8 × 10⁻⁴27.936.2
2 × 10⁻³7.824.1

LFNT maintains positive speedups under warmup and cosine schedules for both ResNet and GPT-2. As expected, speedup generally decreases with more aggressive learning rates, but remains substantial across several settings.

We additionally evaluated them with loss-matched context selection, which selects the context checkpoints according to their validation perplexity rather than just taking the last 5 checkpoints with 200 steps spacing, as in our normal fixed context setting from the paper. This accounts for the differences in training speed when increasing the learning rates. We provide an explanation of this strategy in our separate response to the reviewer BUHv’s under Choosing jump step and horizon.

AuthorsOpenReview ↗

Response to Reviewer KPj3 (2/4)


Computational overhead. We measured the computational cost of the LFNT pipeline. Training the autoencoder (Stages I and II) and latent forecaster requires ~9 GPU-hours for ResNet and ~6 GPU-hours for GPT-2, using the H100 gpu. The main upfront cost is model-zoo construction, requiring ~80 GPU-hours for ResNet and ~200 GPU-hours for GPT-2 using V100s. However this trajectory collection cost is also incurred by other learned weight-space forecasting methods such as WNN and NiNo, and to reduce this cost for future research we will make our model zoos public.

Regarding the amortization question, we want to highlight that in several cases our LFNT needs to be trained once and reused for many training runs, including runs on novel tasks and deeper/wider architectures. We have estimated that, excluding the model-zoo construction, the ResNet pipeline breaks even after ~ 600 accelerated runs in the original evaluation setup, decreasing to ~100 runs for the wider models. For GPT-2, amortization is more favorable, requiring ~50 runs when training until the convergence target. We acknowledge that including model-zoo construction substantially increases the break-even point. However that comparison would be unfair to our method since it ignores the dataset and architecture generalization properties it demonstrates. In particular, one trained instance of our method can be used across different datasets and architecture configurations. Further, recent works [A, B] have shown that weight representations can be learned from publicly available models, and our pretrained WeightCLIP experiments (Table 9) show that LFNT can reuse existing autoencoders. Although our method relies on model-zoos including training trajectories, these arguments hint that future work could aim to train the weight encoder on publicly available data making its amortization easier.

[A] Joëlle Hanna, Damian Falk, Stella X. Yu, and Damian Borth. Geosane: Learning geospatial representations from models, not data. CVPR 2026

[B] Damian Falk, Konstantin Schürholt, Konstantinos Tzevelekakis, Léo Meynent, and Damian Borth. Learning Model Representations Using Publicly Available Model Hubs. arXiv preprint 2025.

AuthorsOpenReview ↗

Response to Reviewer KPj3 (3/4)

Are the gains from forecasting or from other effects of the autoencoder.

To understand whether the speedups are from forecasting or from other effects by the autoencoder, we performed an additional ablation. We encode and decode models with no forecasting and resume training from these reconstructed weights, using the same evaluation protocol as LFNT.

ResNet (CIFAR-100): optimizer-step speedup (%)

Width multipliers are relative to the original smaller ResNet18 configuration used in the paper.

ArchitectureAE-onlyLatent MLPLatent Transf.
R142.234.436.3
R265.327.332.4
R34-2.922.930.1
Width 2x-5.322.924.5
Width 4x-11.714.115.2

GPT-2: optimizer-step speedup (%)

Forecasting the token embedding table for the new widths

ArchitectureAE-onlyLatent Transf.
Depth 2-2420.555.0
Depth 4-2421.257.8
Depth 6-2419.758.1
Width 3-3215.043.9
Width 3-4810.330.1
Width 3-647.018.5

The results show that autoencoder reconstruction alone cannot explain the observed speedups, particularly for architectures unseen during autoencoder training. For example, on ResNet-34, reconstruction yields -2.9% speedup, compared with 30.1% for LFNT. The positive reconstruction-only speedups for GPT-2 may come from projecting checkpoints toward the learned weight distribution. However, this effect diminishes for wider models, whilst latent forecasting continues to provide substantial speedup.


Reconstruction performance. Compression introduces reconstruction error which can degrade the performance of the model right after reconstruction, even if the MSE is low as some parameters can be particularly sensitive. However, we do not aim to create a model that performs well right after decoding but rather a model that serves as a better starting point for subsequent optimization, quantified by the speedup. Nevertheless we quantify this by evaluating the reconstructed models right after decoding as well as after 100 steps of continued optimization, we evaluate both the reconstruction from Stage I as well as Stage II. For comparison we have also included the result for training a model from scratch, to visualize the recovery of the decoded model.

Vision: validation accuracy (%), higher is better.

Architecture (original performance)MethodStep 0Step 100
ResNet18 (57.8)Stage I24.553.7
Stage II1.045.2
Training from scratch1.09.5
EfficientNet-B0 (50.7)Stage I1.028.4
Stage II0.914.7
Training from scratch0.82.1
MobileNetV2 (46.3)Stage I1.030.9
Stage II0.920.2
Training from scratch0.92.3

GPT-2 (LM1B): validation perplexity, lower is better.

Original checkpoint perplexity: 403

MethodStep 0Step 100
Stage I1488452
Stage II1630466
Training from scratch5045525172

The results show that even though reconstruction does degrade the immediate performance it recovers very quickly once training resumes. Stage II reconstructs worse than Stage I which is understandable given that the latent space is optimized for forecasting in the second stage. We want to highlight that LFNT achieves substantial speedups despite these reconstruction errors, which reiterates why the direct reconstruction performance does not reflect the performance of a forecasting approach.




Perturbation direction

Yes, we sample a random perturbation direction and normalize it to have magnitude (f) times the displacement between TstartT_{\mathrm{start}} and TrefT_{\mathrm{ref}}. This displacement norm is computed separately in weight space and latent space. We will clarify this in the paper.

AuthorsOpenReview ↗

Response to Reviewer KPj3 (4/4)

Stage II linear extrapolator at inference

To isolate the contribution of the latent forecaster, we additionally evaluated the linear extrapolator used during Stage II directly at inference, replacing the Latent MLP or Transformer.

GPT-2 (LM1B): optimizer-step speedup (%)

MethodSpeedup (%)
Linear (weights)-81.9
Linear (Stage I latent)-53.4
Linear (Stage II latent)2.2
Latent Transformer59.2

ResNet: optimizer-step speedup (%)

DatasetLinear (weights)Linear (Stage II)Latent Transf.
CIFAR-10-92.6-8.315.6
CIFAR-100-144.31.638.9
TinyImageNet-100.9-12.124.9
Food-101-107.97.036.2
Mean-111.4-3.028.9

The results show that directly extrapolating weights destroys training performance. Extrapolation in the Stage II latent space is considerably more stable, achieving 2.2% speedup on GPT-2 and -3.0% on average for ResNet. For GPT-2, this also represents a substantial improvement over the Stage I latent representation, where linear extrapolation results in a -53.4% speedup. This improvement is expected, as Stage II explicitly enforces the latent to support linear extrapolation of trajectories.

However linear latent extrapolation alone provides little or no speedup. Replacing it with the learned Latent Transformer increases speedup to 59.2% on GPT-2 and 28.9% on ResNet. These findings show that while Stage II improves the suitability of the representation for extrapolation the learned Stage III forecaster is still essential for achieving meaningful training speedups.




Table 1 and 2

We will include the updated tables in the revised paper, reporting the mean speedups ± SEM to quantify the uncertainty.

ResNet: Optimizer-step speedup (%), mean ± SEM

Dataset / ArchitectureWNNNiNoLatent MLPLatent Transf.
CIFAR-103.3 ± 1.613.6 ± 1.914.6 ± 1.415.6 ± 2.0
CIFAR-100-5.6 ± 1.94.7 ± 1.534.9 ± 1.038.9 ± 1.1
Food-101-1.5 ± 1.15.7 ± 1.034.9 ± 0.836.2 ± 0.6
Tiny-ImageNet-0.1 ± 1.34.8 ± 1.121.6 ± 1.124.9 ± 1.1
ID mean-0.9 ± 0.87.2 ± 0.726.5 ± 0.528.9 ± 0.6
BreakHis11.1 ± 4.618.5 ± 4.726.6 ± 4.333.3 ± 3.4
LEGO7.5 ± 3.315.8 ± 3.617.6 ± 4.618.2 ± 3.2
Simpsons5.8 ± 4.416.5 ± 4.437.9 ± 2.938.6 ± 3.7
OOD mean8.2 ± 2.417.0 ± 2.527.4 ± 2.330.0 ± 2.0
R14-0.7 ± 1.75.6 ± 1.334.3 ± 0.936.3 ± 0.7
R26-5.1 ± 2.11.0 ± 1.430.4 ± 1.532.4 ± 1.5
R34-4.6 ± 2.10.4 ± 1.927.9 ± 1.330.1 ± 1.0
Depth mean-3.5 ± 1.12.3 ± 0.930.8 ± 0.733.0 ± 0.6

GPT-2: Optimizer-step speedup (%), mean ± SEM

ArchitectureWNNNiNoLatent MLPLatent Transf.
LM1B 3-24 (ID)-1.8 ± 1.715.7 ± 0.741.9 ± 0.659.2 ± 0.2
LM1B 2-24-4.2 ± 2.613.6 ± 0.536.1 ± 0.655.0 ± 0.3
LM1B 4-240.6 ± 2.316.8 ± 0.943.5 ± 0.357.8 ± 0.3
LM1B 6-242.9 ± 1.115.7 ± 0.546.2 ± 0.658.1 ± 0.2
Depth mean-0.2 ± 1.215.4 ± 0.441.9 ± 0.357.0 ± 0.1
Reviewer_BUHvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LFNT, a three-stage framework that accelerates NN training by predicting future weights within a learned latent space. The authors train an autoencoder conditioned with a forecasting objective to structure the latent representation, subsequently using a latent-space forecaster to predict future checkpoints and resume optimization. Empirical evaluations demonstrate consistent wall-clock and step speedups over prior weight-space forecasters across varying datasets, architectures, and optimization setups.

Critical strengths
  1. The paper makes an interesting connection between weight-space representation learning and weight-forecasting-based training acceleration.

  2. Rather than simply applying an existing autoencoder to weight forecasting, the paper introduces a forecasting objective that explicitly shapes the latent space to support predictable optimization trajectories.

  3. Although the evaluated models are relatively small, the paper includes CNNs, ResNets, and GPT-2-style Transformers.

Critical weaknesses
<Method>
  1. The positioning relative to prior work (SANE/WeightCLIP vs. Introspection vs. WNN/NiNo) needs clarification. While conceptually bridging weight autoencoders and far-future forecasting (like Introspection), comparisons are drawn against WNN/NiNo with some modification. The performance gains could stem either from forecasting or simply from generating favorable initializations (analogous to SANE/WeightCLIP). Distinguishing whether LFNT accelerates training via accurate extrapolation versus better weight-state re-initialization would significantly clarify its core mechanism and contribution.

  2. LFNT requires training architecture-specific autoencoders and predictor, unlike prior methods (WNN, NiNo, Introspection) that exhibit broader cross-architecture transferability in their original papers. Applying LFNT to a new architecture family demands constructing substantial trajectory datasets, it is necessary to provide these overhead.

  3. It remains unclear whether LFNT is applicable for transfer learning or fine-tuning. Collecting a model zoo of diverse fine-tuning trajectories is practically burdensome, and a single large forecast jump risks severe instability along sensitive fine-tuning trajectories.

<Experiments>
  1. In Section 4.1 (lines 303–306), the authors state that the main experiments evaluate all methods using a single forecast. However, WNN and NiNo were originally developed for periodic weight forecasting. While this controls for representation effects, it may underestimate baseline utility in their intended regimes. Although periodic NiNo is tested in Table 3, the authors do not report subsequent training curves after LFNT forecasting. Full post-forecasting trajectories under periodic protocols are necessary for a faithful comparison.

  2. In the main experiments, the evaluation point exactly coincides with the starting step plus the forecast horizon (e.g., 1k + 9k = 10k target). This setup creates an unrealistic scenario that artificially favors LFNT, as practitioners rarely know the target step ahead of time. Moreover, the paper offers no principled criteria for choosing the intervention step or jump length, leaving both as heuristic choices.

  3. The evaluation focuses on steps to intermediate targets rather than overall time to final convergence. Crucially, the target performance in Figure 1 appears unconverged, leaving it unclear whether LFNT maintains asymptotic accuracy after a single large jump. The authors should report complete learning curves through full convergence.

  4. It is unclear whether the gains come from accurate trajectory forecasting or simply finding a favorable warm-start initialization (similar to SANE/WeightCLIP). Comparing against standard initialization baselines and evaluating LFNT purely as an initialization generator would clarify whether modeling trajectory dynamics is truly necessary.

  5. Experiments incorporating common learning-rate schedules, such as warmup and cosine decay, and larger Transformer architectures would provide stronger evidence of practical applicability.

<Minor>
  1. In Figure 2, the Stage II diagram appears to contain a red spell-check underline.

  2. The main tables report average speedups over 20 evaluation runs, but do not provide standard deviations.

Questions for authors
  1. How does LFNT conceptually differ from SANE/WeightCLIP and Introspection and WNN/NiNo, and can the authors verify that speedups stem from accurate trajectory forecasting rather than favorable warm-start initialization?

  2. What is the total overhead for model-zoo construction and meta-training, how many runs are needed to amortize it, and can LFNT transfer across architectures without retraining?

  3. Can the authors provide full training curves through final convergence to show whether the single-jump speedup persists over follow-up steps and realistic training schedules?

  4. How does LFNT compare against WNN and NiNo when they are evaluated under their intended periodic schedules with longer training steps?

  5. What principled guidelines should practitioners follow to select the jump step and horizon?

8 comments
AuthorsOpenReview ↗

Response to Reviewer BUHv (1/5)

We thank Reviewer BUHv for their review as we respond to the reviewer’s questions and concerns and provide additional experiments.


Distinction from prior work and forecasting versus favorable re-initialization

Positioning relative to prior work. SANE and WeightCLIP learn representations of neural network weights, but do not explicitly model or forecast optimization trajectories. Introspection, WNN, and NiNo learn to predict future weights directly in weight space. LFNT combines these directions by learning weight representations explicitly aligned with training dynamics, enabling forecasting in the resulting latent space rather than in the weight space.

Forecasting versus warm-start initialization. To understand whether the observed speedups are from forecasting or simply from favorable warm-start initialization induced by the autoencoder, we performed an additional ablation. We encode and decode models without any forecasting and resume training from these reconstructed weights, using the same evaluation protocol as LFNT.

ResNet (CIFAR-100): optimizer-step speedup (%)

Width multipliers are relative to the original smaller ResNet18 configuration used in the paper.

ArchitectureAE-onlyLatent MLPLatent Transf.
R142.234.436.3
R265.327.332.4
R34-2.922.930.1
Width 2x-5.322.924.5
Width 4x-11.714.115.2

GPT-2: optimizer-step speedup (%)

Forecasting the token embedding table for the new widths

ArchitectureAE-onlyLatent Transf.
Depth 2-2420.555.0
Depth 4-2421.257.8
Depth 6-2419.758.1
Width 3-3215.043.9
Width 3-4810.330.1
Width 3-647.018.5

The results show that autoencoder reconstruction alone cannot explain the observed speedups, particularly for architectures unseen during autoencoder training. For example, on ResNet-34, reconstruction yields -2.9% speedup, compared with 30.1% for LFNT. The positive reconstruction-only speedups for GPT-2 may come from projecting checkpoints toward the learned weight distribution. However, this effect diminishes for wider models, whilst latent forecasting continues to provide substantial speedup.

AuthorsOpenReview ↗

Response to Reviewer BUHv (2/5)

Architectural generalization and computational overhead

We address the reviewer's concerns about computational overhead and architectural generalization through additional experiments and a cost analysis.


Architectural generalization. Beyond the transfer to unseen model depths shown in Table 4, we performed additional experiments evaluating speedup for wider ResNets and GPT-2 models. In addition, we test three different vision architectures, MobileNetV2, EfficientNet-B0, and DenseNet-121, scaled down to similar param counts as our resnets. These experiments reuse the trained autoencoder and forecaster.

ResNet (CIFAR-100): optimizer-step speedup (%) across widths

Width multipliers are relative to the original smaller ResNet18 configuration used in the paper.

WidthWNNNiNoLatent MLPLatent Transf.
1x (ID)-5.64.734.938.9
2x (OOD)1.64.022.924.5
4x (OOD)–-5.514.115.2

Cross-architecture generalization (OOD datasets): optimizer-step speedup (%)

ArchitectureNiNoLatent Transf.
ResNet (reference)17.030.0
MobileNetV23.323.8
EfficientNet-B0-6.224.6
DenseNet-121-4.313.3
Mean (new architectures)-2.420.6

GPT-2: optimizer-step speedup (%) across widths

Forecasting the token embedding table.

ArchitectureLatent Transf.
3-3243.9
3-4830.1
3-6418.5

These results show that LFNT generalizes to unseen architectures and achieves higher speedups than WNN and NiNo under the evaluated architectural changes. Furthermore, recent works such as [A] and [B] have shown that shared weight representations can be learned across heterogeneous architectures. Using such general-purpose weight encoders in LFNT could extend latent forecasting to a broader range of architectures, but is left for future work.


Computational overhead. We measured the computational cost of the LFNT pipeline. Training the autoencoder (Stages I and II) and latent forecaster requires ~9 GPU-hours for ResNet and ~6 GPU-hours for GPT-2, using the H100 gpu. The main upfront cost is model-zoo construction, requiring ~80 GPU-hours for ResNet and ~200 GPU-hours for GPT-2 using V100s. However this trajectory collection cost is also incurred by other learned weight-space forecasting methods such as WNN and NiNo, and to reduce this cost for future research we will make our model zoos public.

Regarding the amortization question, we want to highlight that in several cases our LFNT needs to be trained once and reused for many training runs, including runs on novel tasks and deeper/wider architectures. We have estimated that, excluding the model-zoo construction, the ResNet pipeline breaks even after ~ 600 accelerated runs in the original evaluation setup, decreasing to ~100 runs for the wider models. For GPT-2, amortization is more favorable, requiring ~50 runs when training until the convergence target. We acknowledge that including model-zoo construction substantially increases the break-even point. However that comparison would be unfair to our method since it ignores the dataset and architecture generalization properties it demonstrates. In particular, one trained instance of our method can be used across different datasets and architecture configurations. Further, recent works [A, B] have shown that weight representations can be learned from publicly available models, and our pretrained WeightCLIP experiments (Table 9) show that LFNT can reuse existing autoencoders. Although our method relies on model-zoos including training trajectories, these arguments hint that future work could aim to train the weight encoder on publicly available data making its amortization easier.

[A] Joëlle Hanna, Damian Falk, Stella X. Yu, and Damian Borth. Geosane: Learning geospatial representations from models, not data. CVPR 2026

[B] Damian Falk, Konstantin Schürholt, Konstantinos Tzevelekakis, Léo Meynent, and Damian Borth. Learning Model Representations Using Publicly Available Model Hubs. arXiv preprint 2025.

AuthorsOpenReview ↗

Response to Reviewer BUHv (3/5)

Convergence and Generalization Across Optimization Settings

Training until convergence. To investigate whether LFNT's speedups persist beyond the targets used in our main experiments, we extended evaluation to convergence and measured speedups at the corresponding later performance targets.

ResNet: mean validation accuracy (%) and optimizer-step speedup (%) at 10k and 25k targets

DatasetTarget accuracy (10k)Target accuracy (25k)Speedup (10k)Speedup (25k)
CIFAR-1087.889.315.625.5
CIFAR-10057.859.738.951.8
TinyImageNet38.542.324.941.3
Food-10139.642.236.239.9
Mean speedup––28.939.6

The results show that the original targets already correspond to converged training performance, with small improvements for the remaining 15k steps. LFNT's speedups persist and even increase at the later targets, from 28.9% to 39.6% on average. This demonstrates that the reported speedup is not restricted to the original evaluation targets.

For GPT-2, we similarly evaluate a converged target perplexity of 352.

GPT-2: optimizer-step speedup (%) at target perplexity 352

MethodSpeedup (%)
LFNT (h=4k)39.6
Periodic NiNo44.1
LFNT (h=16k)45.6
LFNT (2 jumps, h=16k)50.2
LFNT + NiNo62.7

LFNT maintains substantial speedup at this later target. Increasing the forecast horizon or applying an additional jump further improves the speedup, while combining LFNT with periodic NiNo achieves 62.7%, reiterating that latent-space forecasting and periodic weight-space forecasting are complementary. These findings extend the results already reported in Table 3 of the main paper.



Generalization across optimization settings. Our original Appendix B.2 (Table 7) already evaluates different learning rates and optimizers, including AdamW and Muon, which shows the performance for more well-tuned baselines. To further address the reviewer's question regarding well-tuned baselines, we performed additional experiments with a linear warmup followed by cosine learning-rate decay for the training schedule.

ResNet (CIFAR-100): optimizer-step speedup (%) with 500-step warmup and cosine decay

Peak LRSpeedup (%)
6 × 10⁻⁴18.9
1.2 × 10⁻³16.1
1.8 × 10⁻³5.7

GPT-2: optimizer-step speedup (%) with 250-step warmup and cosine decay

Peak LRFixed contextLoss-matched context
4 × 10⁻⁴52.653.6
6 × 10⁻⁴34.737.2
8 × 10⁻⁴27.936.2
2 × 10⁻³7.824.1

LFNT maintains positive speedups under warmup and cosine schedules for both ResNet and GPT-2. As expected, speedup generally decreases with more aggressive learning rates, but remains substantial across several settings.

We additionally evaluated them with loss-matched context selection, which selects the context checkpoints according to their validation perplexity rather than just taking the last 5 checkpoints with 200 steps spacing, as in our normal fixed context setting from the paper. This accounts for the differences in training speed when increasing the learning rates. We provide an explanation of this strategy in our separate response to the reviewer BUHv’s under Choosing jump step and horizon.

Reviewer_GKK7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript introduces LFNT, a three-stage framework that accelerates neural network training by forecasting future checkpoints in a learned latent space of model weights. Unlike WNN and NiNo, which predict future weights directly in parameter space, LFNT learns representations tailored to trajectory forecasting in the latent space. Experiments across multiple architectures and tasks support the effectiveness of these representations and demonstrate improved training efficiency relative to existing weight-space forecasting methods.

Critical strengths

1.This paper offers an alternative to training acceleration by learning the weight representation via the trajectory forecasting in the latent space. 2.The evaluation covers multiple architectures and both vision and language tasks. Transfer experiments on unseen datasets and model depths, conducted without additional training costs, demonstrate the generalization ability of the proposed framework.

Critical weaknesses

1.It remains unclear how much training time LFNT saves compared with well-tuned baselines that achieve the same performance. The comparison should also include the time needed to train the autoencoder and forecaster. 2.Although the ablations show that forecasting supervision improves performance, they do not fully explain why it helps. Stronger evidence is needed to determine whether the gains come from making latent trajectories easier to predict or from other effects of the learned encoder and decoder.

Questions for authors

1.The authors should clarify whether Stage II training improves forecasting performance with the predictor architecture held fixed, and whether the gains arise from forecast alignment or the additional capacity introduced by the encoder and decoder. Matching the final MLP architecture alone is insufficient to demonstrate the performance difference between LFNT and WNN. 2.For the vision experiments, Tables 1, 2, 5, and 6 report training speedups without the corresponding absolute classification accuracies. Since speedup is defined by the steps or time required to reach a reference performance, the target validation accuracy for each dataset should be stated explicitly. Although Fig.3 reports accuracy degradation under perturbations, this does not present classification performance after forecasting and subsequent training.

4 comments
AuthorsOpenReview ↗

Response to Reviewer GKK7 (1/3)

We thank Reviewer GKK7 for their review as we respond to the reviewer’s questions and concerns and provide additional experiments.


Computational overhead and generalization across optimization settings

Computational overhead. We measured the computational cost of the LFNT pipeline. Training the autoencoder (Stages I and II) and latent forecaster requires ~9 GPU-hours for ResNet and ~6 GPU-hours for GPT-2, using the H100 gpu. The main upfront cost is model-zoo construction, requiring ~80 GPU-hours for ResNet and ~200 GPU-hours for GPT-2 using V100s. However this trajectory collection cost is also incurred by other learned weight-space forecasting methods such as WNN and NiNo, and to reduce this cost for future research we will make our model zoos public.

Regarding the amortization question, we want to highlight that in several cases our LFNT needs to be trained once and reused for many training runs, including runs on novel tasks and deeper/wider architectures. We have estimated that, excluding the model-zoo construction, the ResNet pipeline breaks even after ~ 600 accelerated runs in the original evaluation setup, decreasing to ~100 runs for the wider models. For GPT-2, amortization is more favorable, requiring ~50 runs when training until the convergence target. We acknowledge that including model-zoo construction substantially increases the break-even point. However that comparison would be unfair to our method since it ignores the dataset and architecture generalization properties it demonstrates. In particular, one trained instance of our method can be used across different datasets and architecture configurations. Further, recent works [A, B] have shown that weight representations can be learned from publicly available models, and our pretrained WeightCLIP experiments (Table 9) show that LFNT can reuse existing autoencoders. Although our method relies on model-zoos including training trajectories, these arguments hint that future work could aim to train the weight encoder on publicly available data making its amortization easier.

[A] Joëlle Hanna, Damian Falk, Stella X. Yu, and Damian Borth. Geosane: Learning geospatial representations from models, not data. CVPR 2026

[B] Damian Falk, Konstantin Schürholt, Konstantinos Tzevelekakis, Léo Meynent, and Damian Borth. Learning Model Representations Using Publicly Available Model Hubs. arXiv preprint 2025.

Generalization across optimization settings. Our original Appendix B.2 (Table 7) already evaluates different learning rates and optimizers, including AdamW and Muon, which shows the performance for more well-tuned baselines. To further address the reviewer's question regarding well-tuned baselines, we performed additional experiments with a linear warmup followed by cosine learning-rate decay for the training schedule.

ResNet (CIFAR-100): optimizer-step speedup (%) with 500-step warmup and cosine decay

Peak LRSpeedup (%)
6 × 10⁻⁴18.9
1.2 × 10⁻³16.1
1.8 × 10⁻³5.7

GPT-2: optimizer-step speedup (%) with 250-step warmup and cosine decay

Peak LRFixed contextLoss-matched context
4 × 10⁻⁴52.653.6
6 × 10⁻⁴34.737.2
8 × 10⁻⁴27.936.2
2 × 10⁻³7.824.1

LFNT maintains positive speedups under warmup and cosine schedules for both ResNet and GPT-2. As expected, speedup generally decreases with more aggressive learning rates, but remains substantial across several settings.

We additionally evaluated them with loss-matched context selection, which selects the context checkpoints according to their validation perplexity rather than just taking the last 5 checkpoints with 200 steps spacing, as in our normal fixed context setting from the paper. This accounts for the differences in training speed when increasing the learning rates. We provide an explanation of this strategy in our separate response to the reviewer BUHv’s under Choosing jump step and horizon.

AuthorsOpenReview ↗

Response to Reviewer GKK7 (2/3)

Forecasting versus other effects of the autoencoder

To understand whether the speedups are from forecasting or simply from other effects of the autoencoder, we performed an additional ablation. We encode and decode models without any forecasting and resume training from these reconstructed weights, using the same evaluation protocol as LFNT.

ResNet (CIFAR-100): optimizer-step speedup (%)

Width multipliers are relative to the original smaller ResNet18 configuration used in the paper.

ArchitectureAE-onlyLatent MLPLatent Transf.
R142.234.436.3
R265.327.332.4
R34-2.922.930.1
Width 2x-5.322.924.5
Width 4x-11.714.115.2

GPT-2: optimizer-step speedup (%)

Forecasting the token embedding table for the new widths

ArchitectureAE-onlyLatent Transf.
Depth 2-2420.555.0
Depth 4-2421.257.8
Depth 6-2419.758.1
Width 3-3215.043.9
Width 3-4810.330.1
Width 3-647.018.5

The results show that autoencoder reconstruction alone cannot explain the observed speedups, particularly for architectures unseen during autoencoder training. For example, on ResNet-34, reconstruction yields -2.9% speedup, compared with 30.1% for LFNT. The positive reconstruction-only speedups for GPT-2 may come from projecting checkpoints toward the learned weight distribution. However, this effect diminishes for wider models, whilst latent forecasting continues to provide substantial speedup.

Together with the Stage II ablation in Table 8 and the analysis in Figure 3, we think these results provide more insight into why forecasting in the learned latent space works better.

AuthorsOpenReview ↗

Response to Reviewer GKK7 (3/3)

Validation accuracies

Below, we provide the target validation accuracies used in Tables 1, 2, 5, and 6, reported as mean ± standard deviation over the baseline Adam trajectories.

Table 5: CNN (ID), target validation accuracy (%)

DatasetTarget accuracy (%)
CIFAR-1065.3 ± 0.6
SVHN86.8 ± 0.3
notMNIST87.9 ± 0.2
ASL Alphabet99.3 ± 0.8
EuroSAT81.9 ± 1.3
NumtaDB94.7 ± 0.4
Mean86.0

Table 6: ResNet (ID), target validation accuracy (%)

DatasetTarget accuracy (%)
CIFAR-1087.8 ± 0.5
CIFAR-10057.8 ± 0.6
Food-10139.6 ± 0.9
Tiny-ImageNet38.5 ± 0.4
Mean55.9

Table 2: OOD vision, target validation accuracy (%)

DatasetCNNResNet
BreakHis61.7 ± 2.085.3 ± 1.4
LEGO80.5 ± 1.694.7 ± 1.5
Simpsons71.2 ± 1.391.7 ± 1.0
Mean71.190.6

We will include these target accuracies in the revised paper to make the evaluation protocol and reported speedups clearer.

Many thanks to the answers! I will keep my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their thoughtful and constructive feedback! We appreciate that they recognized the importance of studying the forecasting representation (KPj3), our improvements over existing weight-space forecasting methods (3Uxz, GKK7) and generalization across different architectures and tasks (KPj3, GKK7).

Below we summarize the main concerns raised, particularly by Reviewers 3Uxz and BUHv, and how we addressed them.


Generalization (BUHv, 3Uxz)

We expanded and clarified our generalization experiments to show that a single ResNet autoencoder and forecaster generalize across multiple different axes:

  • New datasets, Table 2.
  • New depths and widths, Table 4 and rebuttal.
  • New architectures, MobileNetV2, EfficientNet-B0 and DenseNet-121 (results).
  • Fine-tuning trajectories, (results).
  • Optimization settings, (results).

We additionally demonstrate similar transfer capabilities for the GPT-2 autoencoder and forecaster. These experiments directly address concerns that LFNT requires retraining for each new application.


Computational overhead (BUHv, 3Uxz)

We measured the complete computational costs, separating model training ($\sim$9/$\sim$6 GPU-hours) from zoo construction ($\sim$80/$\sim$200 GPU-hours) for ResNet/GPT-2. We also provide break-even estimates and explicitly discuss the initial costs. Importantly, our generalization results show that these costs can be amortized across many different datasets, architectures and training configurations (analysis).


Forecasting versus other effects (BUHv, 3Uxz)

To understand whether the gains come from forecasting or other effects of the autoencoder, we introduced a reconstruction-only baseline (results). The results show that reconstruction alone cannot explain the observed speedups, particularly for architectures unseen during autoencoder training.


New baseline requested (3Uxz)

We evaluated both online and offline Regularized Nonlinear Acceleration (RNA). Online RNA fails to provide any speedup, whilst offline RNA underperforms our approach (results).


Fairness of our comparisons (BUHv)

We clarify that our original single-jump setting was designed to isolate the effect of the representation space, particularly through the matched WNN/Latent MLP architectures, which is required for the main claim of the paper. To additionally evaluate the baselines under their intended periodic settings, we performed new experiments in addition to Table 3 in the original paper. They show that on unseen ResNet datasets, LFNT achieves 30.0% versus 23.8% for periodic NiNo. For GPT-2, periodic NiNo achieves 44.1%, while LFNT achieves 50.2% (results).


Longer training and realistic optimization (BUHv)

We extended both ResNet and GPT2 targets to 25k steps, to be sure to reach the converged performance. We provide post-jump training curves and new experiments with warmup and cosine learning-rate schedules, demonstrating positive speedups under more realistic optimization settings (results).



We believe these additional experiments strengthen the evidence for our central contribution: forecasting training trajectories in a learned latent space is better than forecasting directly in weight space, whilst addressing the main concerns raised during review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.