- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a new prior based on theory of learning dynamics. The prior is used to develop a new PFN for learning curve extrapolation, which was ablated and evaluated for a new applications: extrapolating finetuning runs and OpenAssistant training runs, and are also evaluated in a resource allocation framework.
Critical strengths
The paper introduces a prior that is well-supported by learning theory. The prior can integrate different learning rate schedules, so that the extrapolation can take the learning rate status and schedule into account
Critical weaknesses
Major issues
-
There is no statistical analysis. In particular, the application in section 4.4 is not very convincing. In particular, it is strongly suspiciously cherry picked: "We use a fixed 20-task scenario with 10 trajectories from LoRA-Train and 10 from OA, randomly selected using a fixed seed of 42." This does not make a lot of sense, since there are many more curves. Why would only 10 be selected? The mentioning of "seed 42" is suspicious and misleading: it is not meaningful to report at all. It does not improve reproducibility at all, since it is not clear what was used to randomize the tasks (e.g. seed 42 is something different in Python, matlab, etc...). As such, at a minimum, I expect that ALL tasks are included in this application, and all results should be mentioned. Furthermore, the authors do not seem to include the code of the paper, which further raises serious reproducibility questions. Other results also suffer from missing error bars and indications of statistical significance.
-
Table 2 lists the prior hyperparameters. It is not clear how these were obtained. For reproducibility's sake, this should be clearly reported. In particular, the values seem eerily precise, raising the suspicion that they were tuned on the benchmark (in a big gridsearch with training multiple PFNs). How did you make sure that the performance of your approach does not appear inflated because of this tuning? If they were not tuned using gridsearch, how were the values obtained? Please describe your procedure carefully and in a reproducible way. Reproducibility means that I should be able to give this paper to a coworker, and they should more or less get the same result. This paper does not come close to that standard.
-
In Figure 5 it remains unclear and unexplained why the proposed PFN has less prediction calls and lower total time.
-
While the paper introduces new and interesting benchmark problems, it is also in some way unconvincing. Is the method generalizeable to other, more well-established benchmarks? In particular, the chosen values for the prior further emphasize this even more. It is highly suggested to show the method's performance on the benchmarks that also appear in the LC-PFN: LCBench, NAS-Bench, Taskset and PD1. I also suggest that the early stopping application will be evaluated against LC-PFN (see 4.3 in the LCPFN paper). This application is much simpler, and would further highlight the generalizeability of this approach.
-
Like any good paper, this paper should discuss its weaknesses. The weaknesses are not discussed in any serious matter.
Minor
-
It would be nice if the paper includes figures to explain and illustrate the newly proposed prior, e.g. like the Figure 1 in LC-PFN. Now, it is rather hard to intuitively grasp how the pretraining data looks like.
-
It is not explained what is DKF
Questions for authors
-
Why are the 10 trajectories chosen? What are the results on all trajectories?
-
How were the prior hyperparameters chosen?
-
Why does the proposed PFN have lower prediction calls and lower total time?
-
What is the performance of the approach on LCBench, NAS-Bench, Taskset and PD1 ? How does it perform in the LC-PFN's early stopping application?
-
What is DKF?