- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents an empirical study demonstrating that an existing tabular foundation model (TabPFN) can serve as an effective surrogate for predicting language-model training outcomes. The insight that a foundation model is performing well on this task as such is useful, but the main methodological components are established techniques and TabPFN has been demonstrated as good surrogate model on similar tasks. The experimental evaluation is generally good, including results on the Marin Hero Run to rule out selection bias, but also quite easy to do. Overall, I'd consider the paper to have the level of a properly done Master thesis.
Critical strengths
The fact that TabPFN is useful for predicting the outcome of training runs is quite valuable for everybody who can benefit from such predictions. Although alternative methods, also alternative PFN approaches, can also provide such information, and there is no conclusive evidence in the present paper, if TabPFN is necessarily the best, TabPFN has the advantage over these methods that it can run without training/adaptation of the surrogate model: it just requires providing training run data as context. This makes it convenient to use, although this is a property inherited directly from TabPFN itself.
Critical weaknesses
The most critical weakness in my view is the limited originality. Applying a foundation model like TabPFN on X is a common procedure and quite obvious. The finding that it works well for predicting the outcome of training runs is interesting, but can be figured out quickly by everybody who needs such prediction. This is especially, as is argued by the paper, because TabPFN can be used easily and without retraining.
There is no comparison to existing specialized PFN-based methods, particularly FT-PFN and Adam-PFN. I understand that these approaches are not as readily usable as TabPFN, which gives TabPFN a structural advantage. Nonetheless, it would be interesting to see, if there is also a difference in performance, and in favor of which one. However, I consider this point minor compared to the previous one on the originality.
The additional KAN-based features provide an improvement, but I would not see them as a significant novel algorithmic contribution either.
Testing on the Marin Hero Run is good to avoid selection bias. However, there are two minor issues: (1) TabPFN receives the first 4% of the target run, whereas the official scaling-law baseline does not. (2) The reported evaluation covers checkpoints through only some part of the training, not the completion of the run.
Questions for authors
No questions.
5 comments
Reviewer 6fas [2/3]
W2. "There is no comparison to existing specialized PFN-based methods, particularly FT-PFN and Adam-PFN. I understand that these approaches are not as readily usable as TabPFN, which gives TabPFN a structural advantage. Nonetheless, it would be interesting to see, if there is also a difference in performance, and in favor of which one. However, I consider this point minor compared to the previous one on the originality."
FT-PFN / Adam-PFN (as you requested). We added LC-PFN and FT-PFN; Adam-PFN has no public code or weights. The methods are complementary. Curve PFNs are better from a single short prefix; TabPFN is clearly better once historical runs are available.
| MAE | LC-PFN | FT-PFN | TabPFN |
|---|---|---|---|
| Delphi, prefix only, 20% / 50% / 80% | 0.077 / 0.095 / 0.164 | 0.088 / 0.105 / 0.164 | 0.166 / 0.267 / 0.172 |
| Delphi, history only | — | 0.190 / 0.192 / 0.225 | 0.058 / 0.054 / 0.063 |
| Delphi, history + prefix | — | 0.057 / 0.080 / 0.127 | 0.025 / 0.031 / 0.039 |
Thank you for the clear response. I appreciate the clarification w.r.t. the comparison to alternative approaches. I still think that applying TabPFN to X is quite obvious and straightforward for the X here (I do not doubt the relevance), which is why I keep my rating.
Reviewer 6fas [1/3]
Thank you for noting that the result is valuable for everybody who needs such predictions. Below, we address your concerns in detail.
LLM pretraining is not an arbitrary X. Forecasting LLM pretraining outcomes is among the costliest decisions in machine learning: a frontier run cannot be repeated, so its configuration, data mixture and architecture must be chosen from smaller runs beforehand. The task is also demanding for a surrogate: in our evaluations it must extrapolate beyond the observed scales, distinguish small loss differences, and handle heterogeneous configurations (optimizers, experts, data recipes). This is why practitioners build scaling ladders, why dedicated laws have been fitted for loss and for hyperparameters [1, 2], and why a fine-tuned LLM was recently proposed for exactly this task (NCPL). To our knowledge, no prior work evaluates a tabular foundation model as a meta-model of LLM pretraining; related uses of TabPFN act as black-box surrogates for Bayesian optimization on standard HPO benchmarks [3, 4].
[1] Hoffmann et al. (2022). Training Compute-Optimal Large Language Models.
[2] Li et al. (2025). Predictable Scale: Part I — Optimal Hyperparameter Scaling Law in Large Language Model Pretraining.
[3] Rogers & Ponnada (2026). Zero-shot Bayesian optimization with TabPFN: Competitive with state-of-the-art without per-task training. AutoML Conference 2026.
[4] Yu et al. (2026). GIT-BO: High-Dimensional Bayesian Optimization with Tabular Foundation Models. ICLR 2026.
Related work of this kind. Studying what a frozen foundation model can do on a new problem, used in context without weight updates, is an established line of work, including two widely cited papers: Delétang et al. [5] show that pretrained LLMs compress ImageNet patches and LibriSpeech samples better than PNG and FLAC (excluding model parameter storage), alongside a prediction–compression view; Kojima et al. [6] show that off-the-shelf LLMs reason zero-shot with one shared prompt (ask LLM to think step by step), and argue for exploring what pretrained models already know before crafting fine-tuning data.
[5] Delétang et al. (2024). Language Modeling Is Compression. ICLR 2024.
[6] Kojima et al. (2022). Large Language Models are Zero-Shot Reasoners. NeurIPS 2022.
Our paper makes this kind of contribution for LLM pretraining:
An example: the Marin Hero run. Hero is a frontier-scale MoE, the largest fully open pretraining run to date, trained on trillions of tokens with more than two orders of magnitude more compute than any of its ladder runs. Within its first week, after seeing only the first few percent of training, we froze a TabPFN forecast of the entire run and archived it alongside Marin's official scaling law, which is fitted to a dedicated ladder at every stage of training. The pretraining run has just finished. Scored against the finished run, our forecast, made by forward passes with no weights updated, predicted the final loss more closely than the official law and was second to it along the curve; XGBoost with the same inputs is far less accurate, and NCPL fine-tuned on the ladder alone extrapolates poorly: