acceptodds
Under review as a conference paper at ICLR 2027

Meta Models: Models that Model Learning

Abstract

Every neural-network training run produces not only a model, but also evidence about how models learn. Can these accumulated experiments enable us to predict and compare new training runs before executing them? We study meta-models: models that predict the learning behavior of other models. From this perspective, final-performance prediction, learning-curve prediction, and hyperparameter selection are different queries to the same training response function. We take a first, humble empirical step toward this goal: treating real language-model training records as native numeric data, we find that an off-the-shelf tabular in-context learner, TabPFN, can match or outperform a fine-tuned LLM meta-model in multiple evaluated settings. Whereas recent work fine-tunes LLMs on tokenized configurations at a cost of tens to hundreds of GPU-hours, TabPFN requires seconds of fit-and-predict time without task-specific weight updates. We argue that meta-modeling is better treated as its own numeric modality rather than as a text task for LLMs. Going one step further, augmenting TabPFN with Kolmogorov–Arnold Networks (KANs) further improves accuracy on selected prediction tasks. Finally, we pre-registered a forecast of the Marin Hero Run, a 535B-parameter public training run, whose ground truth is revealed point by point as training progresses, putting meta-model predictions to the test against training outcomes not yet observed at the time of forecasting.

Primary area:
foundation or frontier models, including LLMs
Keywords:
meta-modeling, training dynamics, tabular foundation models, in-context learning, scaling laws, hyperparameter optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6fasOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents an empirical study demonstrating that an existing tabular foundation model (TabPFN) can serve as an effective surrogate for predicting language-model training outcomes. The insight that a foundation model is performing well on this task as such is useful, but the main methodological components are established techniques and TabPFN has been demonstrated as good surrogate model on similar tasks. The experimental evaluation is generally good, including results on the Marin Hero Run to rule out selection bias, but also quite easy to do. Overall, I'd consider the paper to have the level of a properly done Master thesis.

Critical strengths

The fact that TabPFN is useful for predicting the outcome of training runs is quite valuable for everybody who can benefit from such predictions. Although alternative methods, also alternative PFN approaches, can also provide such information, and there is no conclusive evidence in the present paper, if TabPFN is necessarily the best, TabPFN has the advantage over these methods that it can run without training/adaptation of the surrogate model: it just requires providing training run data as context. This makes it convenient to use, although this is a property inherited directly from TabPFN itself.

Critical weaknesses

The most critical weakness in my view is the limited originality. Applying a foundation model like TabPFN on X is a common procedure and quite obvious. The finding that it works well for predicting the outcome of training runs is interesting, but can be figured out quickly by everybody who needs such prediction. This is especially, as is argued by the paper, because TabPFN can be used easily and without retraining.

There is no comparison to existing specialized PFN-based methods, particularly FT-PFN and Adam-PFN. I understand that these approaches are not as readily usable as TabPFN, which gives TabPFN a structural advantage. Nonetheless, it would be interesting to see, if there is also a difference in performance, and in favor of which one. However, I consider this point minor compared to the previous one on the originality.

The additional KAN-based features provide an improvement, but I would not see them as a significant novel algorithmic contribution either.

Testing on the Marin Hero Run is good to avoid selection bias. However, there are two minor issues: (1) TabPFN receives the first 4% of the target run, whereas the official scaling-law baseline does not. (2) The reported evaluation covers checkpoints through only some part of the training, not the completion of the run.

Questions for authors

No questions.

5 comments

Reviewer 6fas [1/3]

Thank you for noting that the result is valuable for everybody who needs such predictions. Below, we address your concerns in detail.

W1. "The most critical weakness in my view is the limited originality. Applying a foundation model like TabPFN on X is a common procedure and quite obvious. The finding that it works well for predicting the outcome of training runs is interesting, but can be figured out quickly by everybody who needs such prediction."

LLM pretraining is not an arbitrary X. Forecasting LLM pretraining outcomes is among the costliest decisions in machine learning: a frontier run cannot be repeated, so its configuration, data mixture and architecture must be chosen from smaller runs beforehand. The task is also demanding for a surrogate: in our evaluations it must extrapolate beyond the observed scales, distinguish small loss differences, and handle heterogeneous configurations (optimizers, experts, data recipes). This is why practitioners build scaling ladders, why dedicated laws have been fitted for loss and for hyperparameters [1, 2], and why a fine-tuned LLM was recently proposed for exactly this task (NCPL). To our knowledge, no prior work evaluates a tabular foundation model as a meta-model of LLM pretraining; related uses of TabPFN act as black-box surrogates for Bayesian optimization on standard HPO benchmarks [3, 4].

[1] Hoffmann et al. (2022). Training Compute-Optimal Large Language Models.

[2] Li et al. (2025). Predictable Scale: Part I — Optimal Hyperparameter Scaling Law in Large Language Model Pretraining.

[3] Rogers & Ponnada (2026). Zero-shot Bayesian optimization with TabPFN: Competitive with state-of-the-art without per-task training. AutoML Conference 2026.

[4] Yu et al. (2026). GIT-BO: High-Dimensional Bayesian Optimization with Tabular Foundation Models. ICLR 2026.

Related work of this kind. Studying what a frozen foundation model can do on a new problem, used in context without weight updates, is an established line of work, including two widely cited papers: Delétang et al. [5] show that pretrained LLMs compress ImageNet patches and LibriSpeech samples better than PNG and FLAC (excluding model parameter storage), alongside a prediction–compression view; Kojima et al. [6] show that off-the-shelf LLMs reason zero-shot with one shared prompt (ask LLM to think step by step), and argue for exploring what pretrained models already know before crafting fine-tuning data.

[5] Delétang et al. (2024). Language Modeling Is Compression. ICLR 2024.

[6] Kojima et al. (2022). Large Language Models are Zero-Shot Reasoners. NeurIPS 2022.

Our paper makes this kind of contribution for LLM pretraining:

  • It challenges current practice. Commonly used approaches design and fit a dedicated law for each question (loss against scale, optimal learning rate and batch size, per-progress curves). TabPFN (a general prior), given the training records and configurations, answers all of these questions and is competitive with these approaches, and more accurate in several settings.
  • It gives a general and reusable recipe. Training records become tabular data: runs as rows with the target run as a query.
  • It applies broadly. One frozen backbone covers final loss, loss curves, internal observables, downstream accuracy and configuration choice, across dense and MoE models, optimizers and data recipes, from small proxy models to a frontier-scale run forecast prospectively.
  • It sets a baseline. Its accuracy without task-specific weight updates suggests that a frozen tabular prior should be a standard baseline for pretraining loss prediction, before a dedicated law or model is built.

An example: the Marin Hero run. Hero is a frontier-scale MoE, the largest fully open pretraining run to date, trained on trillions of tokens with more than two orders of magnitude more compute than any of its ladder runs. Within its first week, after seeing only the first few percent of training, we froze a TabPFN forecast of the entire run and archived it alongside Marin's official scaling law, which is fitted to a dedicated ladder at every stage of training. The pretraining run has just finished. Scored against the finished run, our forecast, made by forward passes with no weights updated, predicted the final loss more closely than the official law and was second to it along the curve; XGBoost with the same inputs is far less accurate, and NCPL fine-tuned on the ladder alone extrapolates poorly:

Hero (535B-A23B MoE)InputsCurve MAEEndpoint error (pred. − true)
Official scaling lawL0.0202+0.057
TabPFNL + P0.0373+0.049
XGBoost, retrospectiveL + P0.1704+0.356
NCPL fine-tuned on the ladder, retrospectiveL0.8398+0.775

Reviewer 6fas [2/3]

W2. "There is no comparison to existing specialized PFN-based methods, particularly FT-PFN and Adam-PFN. I understand that these approaches are not as readily usable as TabPFN, which gives TabPFN a structural advantage. Nonetheless, it would be interesting to see, if there is also a difference in performance, and in favor of which one. However, I consider this point minor compared to the previous one on the originality."

FT-PFN / Adam-PFN (as you requested). We added LC-PFN and FT-PFN; Adam-PFN has no public code or weights. The methods are complementary. Curve PFNs are better from a single short prefix; TabPFN is clearly better once historical runs are available.

MAELC-PFNFT-PFNTabPFN
Delphi, prefix only, 20% / 50% / 80%0.077 / 0.095 / 0.1640.088 / 0.105 / 0.1640.166 / 0.267 / 0.172
Delphi, history only—0.190 / 0.192 / 0.2250.058 / 0.054 / 0.063
Delphi, history + prefix—0.057 / 0.080 / 0.1270.025 / 0.031 / 0.039

Thank you for the clear response. I appreciate the clarification w.r.t. the comparison to alternative approaches. I still think that applying TabPFN to X is quite obvious and straightforward for the X here (I do not doubt the relevance), which is why I keep my rating.

Reviewer_Lxs8OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies meta-models, models that can predict the training behaviour of other models. They find that a frozen TabPFN backbone can already outperform a fine-tuned LLM meta-model at predicting final loss and training trajectories. They show that such a meta-model can be leveraged for various tasks such as predicting downstream performance or r selecting training configurations before launching a run. They further augment TabPFN with KANs for modest gains on selected tasks, and pre-register a forecast of a large public training run (the Marin Hero Run) to test predictions prospectively.

Critical strengths

None

Critical weaknesses
  • Major presentation issues: the paper presents various experiments with different baselines, metrics and protocols sometimes without matched inputs. The connections between the experiments and their relevance to the paper's core claimed contributions are not always clear and make the paper hard to follow. The related work section is also too thin and makes it harder to assess the significance of the contributions.
  • Lack of novelty: the paper applies a frozen TabPFN and compares it to other predictors under different meta-learning settings. The empirical gains don’t isolate the backbone’s contribution, so their significance is hard to assess. The performance gap with the selected baselines does not give us any insight into the practical utility of such a meta-model. The use of KANs to complement TabPFN could be interesting but looks insufficiently explored.
Questions for authors
  • Beyond the given comparisons, is there evidence that the meta-model leads to better practical decisions? While you do compare decisions with baselines, it’s not clear whether acting on the predictions saves compute or beats a standard scaling law under an equalized budget comparison.
  • How much of TabPFN’s advantage comes from the backbone itself versus the fitted scaling-law offsets and the extra input fields it has access to? Across the comparisons, it’s hard to tell whether gains merely reflect a more favorable setup than the baselines get.
6 comments

Reviewer Lxs8 [1/4]

Thank you for your review. We take your concerns seriously and try our best to address each of them. We would also kindly ask you to consider the paper with an open mind about the value it may hold for the ICLR community. Our aim is not a state-of-the-art number on one benchmark, but to show that a frozen tabular foundation model can serve as a meta-model of training, which we believe is new and useful knowledge for practitioners.

Why the problem matters. Large pretraining runs are too expensive to repeat, so configurations, schedules and which runs to continue must be decided before the compute is committed. A meta-model that turns past training records into reliable forecasts can save much of that compute. NCPL established this problem by fine-tuning an LLM for it; we show that a frozen tabular foundation model does the same job without any weight update, and that its forecasts improve real training decisions.

W1. "The connections between the experiments and their relevance to the paper's core claimed contributions are not always clear and make the paper hard to follow."

  • How the experiments connect. We first define what Meta Model is and then show a frozen tabular foundation model can serve as a meta-model in six steps that build on each other. Steps 1–3 widen what it predicts: final loss across configurations, then the whole loss curve, then quantities beyond loss. Step 4 turns these predictions into choices, tuning several hyperparameters at once. Step 5 tests this in replays of real decisions and in a forecast frozen before a frontier-scale run finished. Step 6 improves it through better input coordinates, without changing its weights.
ClaimExperiments in the paper
1. A frozen TabPFN learns how configurations (hyperparameters, data recipes, MoE experts) affect final lossTable 1: final loss on Marin, StepLaw, Marin-MoE and DataDecide against NCPL, XGBoost and scaling laws. Fig. 2: ranking optimizer settings at a held-out width–batch–budget combination (Delphi) and predicting data-mixture losses
2. It extends to full loss curvesTable 1 (curve columns). Fig. 3: learning-rate sweeps (StepLaw), optimizers (Marin), expert configurations (MoE), data recipes at held-out scales (DataDecide)
3. It extends to observables beyond lossFig. 4: parameter and gradient norms (Marin); downstream benchmark accuracy at held-out scales (DataDecide)
4. It supports joint tuning of several hyperparametersFig. 5b–c: picking learning rate and batch size (StepLaw) and seven optimizer hyperparameters (Delphi). Fig. 5d: selection with updated context (StepLaw)
5. It works in replay and in real forecastingFig. 5a and Table 7: updating forecasts with new observations. §4, Fig. 7: a frozen forecast of the 535B Marin Hero run
6. It can be improved§3, Fig. 6: KAN reference coordinates for StepLaw

W1. "Major presentation issues: the paper presents various experiments with different baselines, metrics and protocols sometimes without matched inputs. "

  • Matched inputs. We aligned the inputs in the key comparisons. For example, NCPL fine-tuned with the same fields as TabPFN:
Marin-MoEFinal (ID)Final (OOD)Curve (ID)Curve (OOD)
NCPL fine-tuned, inputs aligned0.02200.07110.04000.0524
XGBoost, same inputs0.05670.06280.05870.1154
Frozen TabPFN0.01600.01870.02590.0326
DataDecideFinal (ID)Final (OOD)Curve (ID)Curve (OOD)
NCPL fine-tuned + recipe, inputs aligned0.01050.01620.01320.0332
XGBoost, same inputs0.01160.01750.01420.0243
Frozen TabPFN0.01050.01780.01250.0133

W1. "The related work section is also too thin and makes it harder to assess the significance of the contributions." The section now covers PFN surrogates, and we now compare directly against LC-PFN and FT-PFN. For example, on the finished Marin Hero run,

Hero, scheduled horizon after the freeze (4K-context evaluation)InputsCurve MAEEndpoint error (prediction − true final loss)
Official scaling law (frozen)L0.0202+0.057
TabPFN (frozen)L + P0.0373+0.049
Prefix power law, retrospectiveP0.0998+0.280
TabPFN, prefix only, retrospectiveP0.1196+0.325
LC-PFN, prefix only, retrospectiveP0.1220+0.302
FT-PFN, prefix only, retrospectiveP0.1305+0.341
XGBoost, prefix only, retrospectiveP0.1709+0.359

Reviewer Lxs8 [2/4]

Q1. "Beyond the given comparisons, is there evidence that the meta-model leads to better practical decisions? While you do compare decisions with baselines, it’s not clear whether acting on the predictions saves compute or beats a standard scaling law under an equalized budget comparison."

Yes, in three settings, and the pretrained TabPFN needs no task-specific training (inference only).

We first clarify how we position the approaches. Scaling laws, NCPL and TabPFN are not competing alternatives; they look at the same problem along different, complementary dimensions.

On compute, TabPFN costs very little. It requires no training: inference costs orders of magnitude less than large-scale pretraining runs, and also less than NCPL, which needs to be fine-tuned. Moreover, TabPFN uses as context the pretraining logs that are routinely collected during LLM development, so no dedicated experiments are needed to build its inputs.

End-to-end compute savings are hard to estimate precisely, because they depend on the size of the search space, how runs are scheduled and how much existing logs can be reused. Prior work on loss scaling laws [1, 2] and hyperparameter scaling laws [3] likewise does not provide a directly comparable end-to-end cost analysis that accounts for these factors.

[1] Hoffmann et al. (2022). Training Compute-Optimal Large Language Models.

[2] Kaplan et al. (2020). Scaling Laws for Neural Language Models.

[3] Li et al. (2025). Predictable Scale: Part I — Optimal Hyperparameter Scaling Law in Large Language Model Pretraining.

Nevertheless, the following three experiments all show the advantage of TabPFN:

  1. Forecasting a frontier-scale run, without fitting a law. Marin's official Hero forecast fits a dedicated scaling law to smaller ladder runs. TabPFN read the same ladder runs, plus the first few Hero checkpoints that the official law does not use, as context and produced the full-curve forecast in forward passes, frozen early in training. Now that the run has finished, it is closer at the final loss, the official law is more accurate along the curve, and both over-predict the final loss; trees and NCPL fine-tuned on the ladder are far less accurate. This is forecast evidence rather than a decision replay. | Hero (535B-A23B MoE), frozen forecasts, 4K-context evaluation | Inputs | Curve MAE | Final-loss error (prediction − truth) | |---|---|---|---| | Official scaling law, fitted on the ladder | L | 0.0202 | +0.057 | | TabPFN, in context (no training or fitting) | L + P | 0.0373 | +0.049 | | XGBoost, retrospective | L + P | 0.1704 | +0.356 | | NCPL fine-tuned on the ladder runs, retrospective | L | 0.8398 | +0.775 | L = ladder runs; P = the first Hero checkpoints.

  2. Hyperparameter selection with almost no training. A target-scale run is too expensive to repeat, so the configuration must be chosen before launching it. With TabPFN, we can enumerate a discrete grid of candidate configurations and select the configuration that minimizes TabPFN's estimated final loss. Its single pick has low mean regret and is close to the optimum. | Single pick, loss above optimum, % (StepLaw OOD) | TabPFN + KAN | Scaling law | TabPFN | |---|---|---|---| | 537M, 2B | +0.136 | +0.519 | +0.137 | | 537M, 4B | +0.137 | +0.525 | +0.245 | | 537M, 8B | +0.009 | −0.007 | −0.035 | | 537M, 10B | −0.060 | −0.015 | −0.064 | | 537M, 20B | −0.076 | −0.130 | −0.123 | | 537M, 28B | +0.076 | +0.008 | +0.020 | | 537M, 50B | +0.132 | +0.074 | +0.083 | | 1074M, 2B | +0.254 | +1.414 | +1.396 | | 1074M, 4B | +0.139 | +0.445 | +0.895 | | 1074M, 8B | −0.013 | +0.076 | +0.520 | | 1074M, 20B | +0.080 | +0.161 | +0.358 | | 1074M, 57B | +0.044 | +0.056 | +0.092 | | OOD mean | +0.071 | +0.261 | +0.294 | | ID mean (5 sweeps) | +0.075 | +0.106 | +0.045 |

  3. Deciding which runs to continue. At the same incremental training budget, the best run kept by acting on TabPFN's forecasts has lower mean regret than with the standard current-loss rule or with trees given the same history. | Early promotion (20 Marin sweeps), regret % | Keep 1: 10 / 20 / 30% | Keep 3: 10 / 20 / 30% | |---|---|---| | Current loss (standard practice) | 0.360 / 0.612 / 0.676 | 0.136 / 0.178 / 0.171 | | XGBoost, same history | 0.147 / 0.172 / 0.376 | 0.078 / 0.049 / 0.169 | | TabPFN | 0.083 / 0.073 / 0.079 | 0.040 / 0.027 / 0.036 |

Reviewer Lxs8 [3/4]

W2. "Lack of novelty: the paper applies a frozen TabPFN and compares it to other predictors under different meta-learning settings."

Related work of this kind. Studying what a frozen foundation model can do on a new problem, used in context without weight updates, is an established line of work, including two widely cited papers: Delétang et al. [1] show that pretrained LLMs compress ImageNet patches and LibriSpeech samples better than PNG and FLAC (excluding model parameter storage), alongside a prediction–compression view; Kojima et al. [2] show that off-the-shelf LLMs reason zero-shot with one shared prompt (ask LLM to think step by step), and argue for exploring what pretrained models already know before crafting fine-tuning data.

[1] Delétang et al. (2024). Language Modeling Is Compression. ICLR 2024.

[2] Kojima et al. (2022). Large Language Models are Zero-Shot Reasoners. NeurIPS 2022.

Our paper makes this kind of contribution for LLM pretraining:

  • It challenges current practice. Commonly used approaches design and fit a dedicated law for each question (loss against scale, optimal learning rate and batch size, per-progress curves). TabPFN (a general prior), given the training records and configurations, answers all of these questions and is competitive with these approaches, and more accurate in several settings.
  • It gives a general and reusable recipe. Training records become tabular data: runs as rows with the target run as a query.
  • It applies broadly. One frozen backbone covers final loss, loss curves, internal observables, downstream accuracy and configuration choice, across dense and MoE models, optimizers and data recipes, from small proxy models to a frontier-scale run forecast prospectively.
  • It sets a baseline. Its accuracy without task-specific weight updates suggests that a frozen tabular prior should be a standard baseline for pretraining loss prediction, before a dedicated law or model is built.

W2. "The empirical gains don’t isolate the backbone’s contribution, so their significance is hard to assess."

Q2. "How much of TabPFN’s advantage comes from the backbone itself versus the fitted scaling-law offsets and the extra input fields it has access to?"

The backbone contributes on its own. We separated the three sources you name; in the offset and backbone comparisons, every learner gets the same context, fields and target:

  • Offset. Removing the scaling-law offset hurts all three learners, but the offset is not where TabPFN's advantage comes from: without it, NCPL and XGBoost over-predict the larger held-out models, while TabPFN loses less on Marin, MoE and DataDecide. | OOD MAE, with offset → without offset | TabPFN | XGBoost | NCPL (fine-tuned) | |---|---|---|---| | Marin final | 0.0173 → 0.0414 | 0.0356 → 0.1576 | 0.0163 → 0.0664 | | StepLaw final | 0.0227 → 0.0974 | 0.0221 → 0.1253 | 0.0196 → 0.0648 | | Marin curves | 0.0457 → 0.0708 | 0.0423 → 0.1521 | 0.0355 → 0.0746 | | StepLaw curves | 0.0367 → 0.0904 | 0.0390 → 0.1346 | 0.0416 → 0.0783 | | MoE final | 0.0187 → 0.0321 | 0.0628 → 0.2229 | 0.0711 → 0.1889 | | MoE curves | 0.0326 → 0.0347 | 0.1154 → 0.2380 | 0.0524 → 0.2332 | | DataDecide final | 0.0178 → 0.0578 | 0.0175 → 0.1390 | 0.0162 → 0.4094 | | DataDecide curves | 0.0133 → 0.0480 | 0.0243 → 0.1697 | 0.0332 → 0.1417 |

  • Fields. The fields are not a private advantage of TabPFN: every learner gets them, and the recipe is essential on DataDecide for all three. What differs is how the backbone uses them: expert fields help TabPFN on MoE, while adding them worsens NCPL's final loss. | OOD MAE, all fields → without the extra fields | TabPFN | XGBoost | NCPL | |---|---|---|---| | MoE final / curves (expert fields removed) | 0.0187 → 0.0242 / 0.0326 → 0.0381 | 0.0628 → 0.0672 / 0.1154 → 0.0816 | 0.0711 → 0.0390 / 0.0524 → 0.0691 | | DataDecide final / curves (recipe removed) | 0.0178 → 0.1117 / 0.0133 → 0.1027 | 0.0175 → 0.1113 / 0.0243 → 0.1064 | 0.0162 → 0.1100 / 0.0332 → 0.1067 |

Reviewer_NSg2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the prediction of language-model training outcomes from historical training records, treating final performance, learning curves, and configuration selection as queries to a training response function. The authors show that a frozen TabPFN can compete with fine-tuned LLM predictors across several tasks, with additional improvements from KAN-derived features and early training observations. They also present a real-world training forecast frozen before subsequent outcomes became available.

Critical strengths
  • Practically valuable empirical question. The paper demonstrates that treating training records as numerical data yields competitive predictors, establishing an important alternative baseline to approaches requiring LLM fine-tuning.
  • Broad experimental coverage. Evaluations extend beyond final loss to trajectories, hyperparameter ranking, data recipes, internal observables, downstream accuracy, and configuration selection, with matched-context controls and grouped bootstrap analyses.
  • Careful disclosure of limitations. The authors distinguish frozen weights from the absence of labeled context, retrospective evaluation from prospective forecasting, and explicitly report extrapolation failures and differences in information available to competing predictors.
Critical weaknesses
  • Several important comparisons do not match input information. The MoE NCPL baseline lacks expert fields, the DataDecide NCPL baseline lacks recipe fields, and the prospective TabPFN forecast receives target-run observations unavailable to the official scaling law. These comparisons therefore do not isolate the advantage of the numerical prediction backbone.
  • Evidence for adaptation and extrapolation remains limited. In Appendix Table 7, XGBoost using the same early observations achieves lower average errors at all three cutoffs, while the 20% prefix improves only four of seven runs. The prospective large-model forecast is evaluated only through 14.61% of training, leaving long-horizon reliability untested.
  • The methodological contribution largely consists of applying existing models and constructing features. Matched-context controls support the KAN gains, but comparisons with simpler reference-coordinate models are missing. Direct comparisons with specialized learning-curve predictors and hyperparameter optimization methods are also limited.
Questions for authors
  • Could NCPL receive the same expert/recipe information? For the prospective case, could the authors add prefix-only predictions and curve-fitting or XGBoost baselines receiving the same target-run prefix?
  • How should Table 7’s consistently lower average errors for prefix-conditioned XGBoost be interpreted? Could per-run results and uncertainty estimates clarify when TabPFN is preferable for updating forecasts?
  • How do KAN-derived coordinates compare with log-linear, additive, or simple spline-based alternatives? Do the advantages persist against specialized learning-curve prediction or Bayesian optimization methods under matched information and evaluation budgets?
5 comments

Reviewer NSg2 [1/3]

Thank you for recognizing the practical value of the question, the breadth of the evaluation and our careful disclosure of limitations, and for the precise questions. Below, we address your concerns in detail.

Q1a. "Could NCPL receive the same expert/recipe information?"

W1. "Several important comparisons do not match input information. The MoE NCPL baseline lacks expert fields, the DataDecide NCPL baseline lacks recipe fields, ..."

Yes, and TabPFN's advantage remains: fine-tuned NCPL is still clearly less accurate than frozen TabPFN on held-out (OOD) MoE final loss and curves and on DataDecide curves. We fine-tuned NCPL from its released checkpoint with inputs aligned to TabPFN's: the expert fields on MoE, the recipe on DataDecide. MAE:

Marin-MoEFinal (ID)Final (OOD)Curve (ID)Curve (OOD)
NCPL fine-tuned, inputs aligned0.02200.07110.04000.0524
XGBoost, same inputs0.05670.06280.05870.1154
Frozen TabPFN0.01600.01870.02590.0326
DataDecideFinal (ID)Final (OOD)Curve (ID)Curve (OOD)
NCPL fine-tuned, inputs aligned0.01050.01620.01320.0332
XGBoost, same inputs0.01160.01750.01420.0243
Frozen TabPFN0.01050.01780.01250.0133

What differs is how each model uses new information, and at what cost: NCPL needs a new prompt field and a new GPU fine-tuning run for every corpus, while TabPFN only adds a column to its context and predicts in a single forward pass, with no training cost. We see the two as different dimensions rather than rivals.

Q1b. "For the prospective case, could the authors add prefix-only predictions and curve-fitting or XGBoost baselines receiving the same target-run prefix?"

W2. "Evidence for adaptation and extrapolation remains limited. The prospective large-model forecast is evaluated only through 14.61% of training, leaving long-horizon reliability untested."

The Marin Hero run (Marin 535B-A23B MoE) has just finished pretraining, so we report the final forecast results directly. Over the scheduled horizon after the freeze, the official law is the most accurate on the curve and our frozen forecast is second, while ours is closer at the final loss (both over-predict it). Prefix-only extrapolation and XGBoost are far less accurate.

Marin Hero Run, scheduled horizon after the freezeInputsCurve MAEEndpoint error (prediction − true final loss)
Official scaling law (frozen)L0.0202+0.057
TabPFN (frozen)L + P0.0373+0.049
Prefix power law, retrospectiveP0.0998+0.280
TabPFN, prefix only, retrospectiveP0.1196+0.325
LC-PFN, prefix only, retrospectiveP0.1220+0.302
FT-PFN, prefix only, retrospectiveP0.1305+0.341
XGBoost, prefix only, retrospectiveP0.1709+0.359
NCPL fine-tuned on the ladder runs, retrospectiveL0.8398+0.775

Reviewer NSg2 [2/3]

Q2. "How should Table 7’s consistently lower average errors for prefix-conditioned XGBoost be interpreted? Could per-run results and uncertainty estimates clarify when TabPFN is preferable for updating forecasts?"

  • Interpretation. Table 7 was meant to show that TabPFN can update its forecasts in context, by appending the target run's new observations without changing any weights, not that it updates better than trees. This holds: adding the prefix lowers TabPFN's mean error at each cutoff relative to its own history-only forecast, although the gains vary across runs. XGBoost has lower mean error once the prefix is long, but among the three corpora we tested only on Delphi: when we repeated the test at finer cutoffs on Marin-MoE and DataDecide, TabPFN has lower error at every tested cutoff.

  • When TabPFN is preferable. In most settings we tested. At the earliest new cutoff TabPFN has lower error than XGBoost on every Delphi run; on Marin-MoE and DataDecide it has lower error at every tested cutoff. | Remaining-curve MAE, TabPFN / XGBoost (same prefix) | 10% | 20% | 50% | 80% | 90% | |---|---|---|---|---|---| | Delphi | 0.0269 / 0.0534 | 0.0254 / 0.0237 | 0.0308 / 0.0195 | 0.0392 / 0.0175 | 0.0223 / 0.0207 | | Marin-MoE | 0.0168 / 0.0443 | 0.0094 / 0.0430 | 0.0042 / 0.0168 | 0.0027 / 0.0141 | 0.0033 / 0.0137 | | DataDecide | 0.0127 / 0.0169 | 0.0102 / 0.0138 | 0.0071 / 0.0134 | 0.0047 / 0.0077 | 0.0035 / 0.0064 |

  • At Table 7's cutoffs, per run, with descriptive paired bootstrap intervals: at the shortest prefix the difference is not resolved; at the middle prefix XGBoost is ahead on average but not resolved; only at the longest prefix is XGBoost better on every run. | Delphi, MAE over 7 runs (descriptive paired bootstrap) | TabPFN with − without prefix | TabPFN − XGBoost | Runs TabPFN wins | |---|---|---|---| | 20% prefix | −0.0325 [−0.072, −0.005] | +0.0017 [−0.005, +0.009] | 4/7 | | 50% prefix | −0.0235 [−0.055, +0.002] | +0.0113 [−0.001, +0.026] | 2/7 | | 80% prefix | −0.0242 [−0.050, −0.006] | +0.0217 [+0.008, +0.041] | 0/7 |

Reviewer NSg2 [3/3]

W3. "The methodological contribution largely consists of applying existing models and constructing features."

Thank you for this point.

Related work of this kind. Studying what a frozen foundation model can do on a new problem, used in context without weight updates, is an established line of work, including two widely cited papers: Delétang et al. [1] show that pretrained LLMs compress ImageNet patches and LibriSpeech samples better than PNG and FLAC (excluding model parameter storage), alongside a prediction–compression view; Kojima et al. [2] show that off-the-shelf LLMs reason zero-shot with one shared prompt (ask LLM to think step by step), and argue for exploring what pretrained models already know before crafting fine-tuning data.

[1] Delétang et al. (2024). Language Modeling Is Compression. ICLR 2024.

[2] Kojima et al. (2022). Large Language Models are Zero-Shot Reasoners. NeurIPS 2022.

Our paper makes this kind of contribution for LLM pretraining:

  • It challenges current practice. Commonly used approaches design and fit a dedicated law for each question (loss against scale, optimal learning rate and batch size, per-progress curves). TabPFN (a general prior), given the training records and configurations, answers all of these questions and is competitive with these approaches, and more accurate in several settings.
  • It gives a general and reusable recipe. Training records become tabular data: runs as rows with the target run as a query.
  • It applies broadly. One frozen backbone covers final loss, loss curves, internal observables, downstream accuracy and configuration choice, across dense and MoE models, optimizers and data recipes, from small proxy models to a frontier-scale run forecast prospectively.
  • It sets a baseline. Its accuracy without task-specific weight updates suggests that a frozen tabular prior should be a standard baseline for pretraining loss prediction, before a dedicated law or model is built.

Q3a. "How do KAN-derived coordinates compare with log-linear, additive, or simple spline-based alternatives?"

They perform alike. What helps is a reference coordinate with a global power law in N; whether it comes from KAN, ordinary least squares or an additive model does not matter, while a spline in N is worse. Our aim in §3 is to show that the meta-model can be improved without changing its weights, not to argue that KAN is the best way to build the coordinates. That a simple log-linear reference suffices makes the recipe easier to reuse; KAN remains useful as an inspectable form of the same reference.

StepLaw OODFinal MAECurve MAESelection regret %
Table 10.02270.03670.427
OLS log-linear0.01640.03020.089
Additive model, linear in N0.01620.03030.086
Additive model, spline in N0.02620.05130.116
KAN0.01610.03090.084

Q3b. "Do the advantages persist against specialized learning-curve prediction or Bayesian optimization methods under matched information and evaluation budgets?"

Yes for both scenarios Given access to the same history and target-run prefix, TabPFN is more accurate than FT-PFN on every corpus and cutoff we tested. TabPFN supports substantially longer contexts than FT-PFN; but its advantage does not rely on this: given only the subset FT-PFN reads, it is still more accurate at the cutoffs shown.

History + prefix, remaining-curve MAE, TabPFN / FT-PFN20%50%80%
Delphi0.025 / 0.0570.031 / 0.0800.039 / 0.127
Marin-MoE0.009 / 0.0490.004 / 0.0500.003 / 0.050
DataDecide0.010 / 0.0610.007 / 0.0480.005 / 0.013

For BO, we gave transfer-BO methods the same history, initial points and budget: TabPFN keeps the lowest mean regret.

StepLaw OOD, same history and budgetRegret % after 3 / 5 evaluations
TabPFN0.089 / 0.014
Multi-task GP0.129 / 0.052
RGPE0.166 / 0.036
Contextual GP0.143 / 0.044
GP-EI without history0.629 / 0.400

Overall, we hope our response has addressed your concerns. If so, we would greatly appreciate it if you could reconsider your rating. We would also be happy to address any further questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.