acceptodds
Under review as a conference paper at ICLR 2027

Learning Dynamics-Informed Learning Curve Extrapolation

Abstract

As the computational demands of machine learning training keep growing, how to obtain more high-quality models under limited resources has attracted increasing attention. Recently, Computational Resource Efficient Learning (CoRE-Learning) has been proposed to address this problem. A key component of CoRE-Learning is learning curve extrapolation (LCE), which predicts future model performance from partially observed training trajectories to guide resource allocation. Existing LCE methods largely rely on curve families or empirical descriptors, leaving the underlying learning process implicit. Nonetheless, we observe that similar early curves can yield different future improvements under the same learning-rate schedule, while a shared checkpoint can evolve differently under different schedules. These observations motivate us to construct learning dynamics-informed generative priors for LCE. We propose Learning Dynamics-Informed PFN (LD-PFN), which builds a generative prior from a reduced spectral model of learning dynamics. A spectral surrogate represents the remaining loss as components with different decay rates, and learning-rate schedules map resource usage to effective optimization progress. We train a PFN on trajectories sampled from this prior for probabilistic extrapolation of validation cross-entropy, without task-specific updates or access to model internals at prediction time. Experiments across diverse fine-tuning scenarios and evaluation settings show that LD-PFN improves predictive accuracy and probabilistic quality over existing baselines.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
learning curve extrapolation, learning dynamics, prior-data fitted networks, fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oQPBOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a new prior based on theory of learning dynamics. The prior is used to develop a new PFN for learning curve extrapolation, which was ablated and evaluated for a new applications: extrapolating finetuning runs and OpenAssistant training runs, and are also evaluated in a resource allocation framework.

Critical strengths

The paper introduces a prior that is well-supported by learning theory. The prior can integrate different learning rate schedules, so that the extrapolation can take the learning rate status and schedule into account

Critical weaknesses

Major issues

  1. There is no statistical analysis. In particular, the application in section 4.4 is not very convincing. In particular, it is strongly suspiciously cherry picked: "We use a fixed 20-task scenario with 10 trajectories from LoRA-Train and 10 from OA, randomly selected using a fixed seed of 42." This does not make a lot of sense, since there are many more curves. Why would only 10 be selected? The mentioning of "seed 42" is suspicious and misleading: it is not meaningful to report at all. It does not improve reproducibility at all, since it is not clear what was used to randomize the tasks (e.g. seed 42 is something different in Python, matlab, etc...). As such, at a minimum, I expect that ALL tasks are included in this application, and all results should be mentioned. Furthermore, the authors do not seem to include the code of the paper, which further raises serious reproducibility questions. Other results also suffer from missing error bars and indications of statistical significance.

  2. Table 2 lists the prior hyperparameters. It is not clear how these were obtained. For reproducibility's sake, this should be clearly reported. In particular, the values seem eerily precise, raising the suspicion that they were tuned on the benchmark (in a big gridsearch with training multiple PFNs). How did you make sure that the performance of your approach does not appear inflated because of this tuning? If they were not tuned using gridsearch, how were the values obtained? Please describe your procedure carefully and in a reproducible way. Reproducibility means that I should be able to give this paper to a coworker, and they should more or less get the same result. This paper does not come close to that standard.

  3. In Figure 5 it remains unclear and unexplained why the proposed PFN has less prediction calls and lower total time.

  4. While the paper introduces new and interesting benchmark problems, it is also in some way unconvincing. Is the method generalizeable to other, more well-established benchmarks? In particular, the chosen values for the prior further emphasize this even more. It is highly suggested to show the method's performance on the benchmarks that also appear in the LC-PFN: LCBench, NAS-Bench, Taskset and PD1. I also suggest that the early stopping application will be evaluated against LC-PFN (see 4.3 in the LCPFN paper). This application is much simpler, and would further highlight the generalizeability of this approach.

  5. Like any good paper, this paper should discuss its weaknesses. The weaknesses are not discussed in any serious matter.

Minor

  1. It would be nice if the paper includes figures to explain and illustrate the newly proposed prior, e.g. like the Figure 1 in LC-PFN. Now, it is rather hard to intuitively grasp how the pretraining data looks like.

  2. It is not explained what is DKF

Questions for authors
  1. Why are the 10 trajectories chosen? What are the results on all trajectories?

  2. How were the prior hyperparameters chosen?

  3. Why does the proposed PFN have lower prediction calls and lower total time?

  4. What is the performance of the approach on LCBench, NAS-Bench, Taskset and PD1 ? How does it perform in the LC-PFN's early stopping application?

  5. What is DKF?

Reviewer_XYyEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LD-PFN, a prior-data fitted network for probabilistic extrapolation of validation cross-entropy curves. Its main contribution is a synthetic curve generator motivated by a local spectral model of learning dynamics. Remaining loss is represented as a mixture of decaying modes, with power-law assumptions on their rates and energies. Learning-rate schedules determine the progression along these curves and are also supplied to the predictor through a context token. The method is evaluated on 108 LoRA fine-tuning runs and 117 OpenAssistant validation trajectories from 16 runs. It substantially improves extrapolation on LoRA-Train, while improvements on OpenAssistant are smaller and metric-dependent. Ablations examine the spectral prior and schedule conditioning. A fixed 20-task resource-allocation replay provides preliminary evidence of downstream usefulness.

Critical strengths
  1. The prior construction is a useful methodological contribution. The spectral representation provides a coherent way to combine multiple learning timescales with known schedule information. Importantly, the resulting predictor only needs the observed curve and training configuration at inference time. This makes the approach practically relevant without requiring Hessian estimation or access to model internals.

  2. The improvements on LoRA-Train are substantial and consistent. Table 1 reports RMSLE of 0.00304 versus 0.00597 for the strongest competing point predictor, and CRPS of 0.00159 versus 0.00461 for the strongest competing probabilistic predictor. Figure 2 shows improvements across all evaluated cutoffs on this dataset. Evaluating the complete unseen suffix, together with both log-likelihood and CRPS, makes these results more informative than an endpoint-only accuracy comparison.

Critical weaknesses
  1. The experiments do not yet isolate the benefit of the proposed spectral structure convincingly. LD-PFN receives schedule information that the main baselines do not receive. The internal ablations help address this, but the crucial “no spectral” variant is described only as sampling rates and energies independently; its sampling distributions are not specified. Consequently, it is unclear whether this comparison isolates the relationships in Eq. 7 or also changes the range of timescales and curve shapes substantially. Table 2 also gives specific endpoint, floor, and exponent distributions without explaining how these choices were selected or whether evaluation curves informed them. Since the contribution centers on prior design, these choices are consequential. A controlled comparison with the same schedule inputs, PFN training setup, and comparable prior marginals would make the attribution much stronger.

  2. The evidence for generalization beyond LoRA-Train remains limited. On OpenAssistant, RMSLE improves from 0.01439 to 0.01362 relative to NSL-PFN, while CRPS is slightly worse than LC-PFN. These 117 trajectories come from only 16 training runs, so curves from the same run should not be treated as independent evidence when assessing uncertainty. No run-level uncertainty estimates or variation across PFN training seeds are reported. Moreover, the procedure for selecting the pre-overfitting region is not sufficiently explicit to establish whether evaluation horizons were chosen independently of future losses, as required by Section 3.1. The single scheduling replay, with 12 successful tasks versus 10 for the competing PFNs, is encouraging but does not resolve these robustness concerns.

Questions for authors
  1. Prior construction and experimental controls: How were the distributions in Table 2 selected, and were either evaluation collection’s curves or results used in that process? What are the exact sampling distributions for “no spectral,” and which other settings are held fixed?

  2. Schedule modeling: In Eqs. 10–11, fixing the sampled latent quantities and endpoint forces different schedules to reach the same noiseless endpoint; schedules change how progress is distributed along the horizon. How should this construction be reconciled with the schedule-dependent final improvements motivating Figure 1(b)? Can you show LD-PFN predictions for that example? This distinction matters because prior work explicitly models loss reductions associated with annealing beyond cumulative learning-rate exposure, including Luo et al. (2025).

  3. Evaluation robustness: What exact rule identifies the pre-overfitting region, and does it inspect observations beyond the eventual context? Do the OpenAssistant improvements persist with equal weighting of training runs and paired uncertainty estimates that account for curves sharing a run?

  4. Reproducibility: Please specify the PFN architecture, predictive-density parameterization, attention masking, batch size, number of training updates, and optimization settings. Were these identical across the ablations? These details are necessary to assess whether the reported differences can be attributed to prior construction.

Reviewer_ypadOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper examines the problem of learning curve extrapolation (LCE) - predicting the validation loss dynamics of a learning process given its observed dynamics. The main contribution is the introduction of LD-PFN (Learning Dynamics-Informed Prior-Data Fitted Network). The authors motivate it by observing that the existing LCE approaches contain existing design flaws that lead to their poor performance. For example, by relying on curve families they will not work in a case where the early learning curve trajectories are similar but lead to different learning dynamics in the later stages of the learning process, or in situations where the future learning from the same model checkpoints utilizes different learning rate schedules. To address this, the authors generate synthetic training curves using a generative model based on generalized Gauss-Newton (GGN) geometry. The generative model represents the remaining validation loss dynamics as a combination of various components decaying at different rates based on the learning rate schedule. Given such generated samples, a PFN is trained on them so that, during inference, it can be used to predict future validation loss based on the available learning curve at any time and a variable that represents the planned learning rate schedule.

The authors further validate the effectiveness of the proposed approach on a set of experiments across two LLM fine-tuning tasks: LoRA fine-tuning of three base models (Gemma2-2B, Llama-3.2-3B, and Qwen2.5-3B) on three instruction corpora, and fine-tuning logs from OpenAssistant. The authors consider learning curves of these tasks, setting observation cutoffs from 20% to 80% of the validation loss curve available to the predictor, at 10% steps. In majority of cases LD-PFN outperforms deterministic curve fits and other PFN-based baselines. Further ablation studies demonstrate that all parts of the LD-PFN design contribute to the framework's achieved gains. Finally, the authors examine the use of LD-PFN in a CoRE-Learning resource allocation setting, on a look-ahead feasibility check benchmark, where different tasks are evaluated on if they can reach their target loss before a set deadline. LD-PFN is utilized as a predictor in that experiment and demonstrates performance improvement over the compared baselines.

Critical strengths

The two critical strengths that support acceptance are as follows:

  • Novel theory-informed and issue-accommodating design of LD-PFN. Instead of relying on heuristic generative approaches for generating the training samples for the PFN, the authors utilize the Generalized Gauss-Newton (GGN) geometry and power-law decay. Since PFNs mostly rely on pretraining on synthetic data, the quality of that data significantly affects their final predictive performance. As such, a better quality, theory-based generative model allows a PFN to capture realistic loss decay curves, potentially helping it distinguish between curves that start out similar but diverge during later stages of the training process. Furthermore, LD-PFN design incorporates a very practical issue of different learning schedules by including the planned LR schedule as an input variable to the PFN, making it more adaptive and realistic to modern training frameworks.
  • Well designed experiments with compelling experimental results. The experiments provided in the paper are informative and provide useful insights. LD-PFN is tested across two different LLM fine-tuning tasks, (one with LoRA fine-tuning runs in controlled environment and one with real-world learning curve data), and a CoRE-learning task, with the LD-PFN compared against a set of PFN-based and deterministic LCE approaches as baselines. In most cases LD-PFN demonstrates improvement over the next best baseline (or closely matches the best-case performance), achieving up to 65% improvement, highlighting the usefulness and novelty of the proposed approach. Furthermore, the ablation studies clearly verify that each component of LD-PFN's design contributes to its final improved performance.
Critical weaknesses

One critical weakness is:

  • Limited experimental scope in different areas. While the overall design of the experiments is sound, the scope is mostly restricted to fine-tuning trajectories of LLMs around the 2B-3B parameter sizes (on the LoRA-Train experiments specifically). It would be interesting to see if LD-PFN generalizes to larger model sizes, non-LLM LCE benchmarks, or pretraining curves. These settings might introduce unexpected behaviors of the learning dynamics, which the GGN-based training data generator might not have been able to capture, reducing effectiveness of LD-PFN. Furthermore, LD-PFN performance benefit on the OpenAssistant learning curves over the other PFN-based baselines is modest, potentially demonstrating the before-mentioned issue.
Questions for authors
  • Is there a particular reason you decided to focus on the LLM fine-tuning benchmarks for the experimental section? In particular, why not include other modalities?
  • What potential factors could cause LD-PFN to perform worse? Are there situations where LD-PFN could actually perform worse that existing baselines?
  • The paper acknowledges various restrictions on the modeling and evaluation frameworks. Along with acknowledging the above points, could you please add a limitation section that would combine all these discussions and highlight the practical scope of LD-PFN? I believe the overall structure of the paper is sound an the proposed framework is novel, but I would like to see some discussion on situations where the suggested approach might not be beneficial or may even become detrimental, as well as what could limit its effectiveness or applicability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.