acceptodds
Under review as a conference paper at ICLR 2027

Separating Supervision Randomness in Prior-Data Fitted Networks

20 views

Abstract

Prior-data fitted networks (PFNs) learn predictive distributions from simulated tasks. At a fixed training budget, removing supervision randomness can improve source-prior fidelity while worsening prediction under prior shift, despite preserving the expected training objective. A four-arm intervention in a finite structural equation model separates additive component-target variation from within-component label randomness. Across five paired training seeds within one source prior, replacing posterior-resampled labels with component targets reduces source predictive KL by 0.00975 nats per query but increases coefficient-shift KL by 0.00413 at . A signed loss estimator retains within-component label randomness while removing additive component-target variation and reproduces the reversal. At the same interpolation level, including the residual instead increases predictive KL under covariance-family shift. A separate six-prior, two-arm study shows that the pooled supervision ranking changes with shift magnitude and construction: full-predictive supervision wins at the smaller tested shift and sampled supervision at the larger one in both families. Source diagnostics show that residuals dominate native target variance. Source-selected output softening remains competitive in aggregate risk while moving predictions farther from the residual-trained predictor on source assessment inputs. These findings establish supervision-estimator choice as a consequential variable in finite-budget PFN training, requiring joint evaluation of source fidelity, shifted risk and predictive behavior.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Prior-Data Fitted Networks, Tabular Foundation Models, Posterior Predictive Approximation, In-Context Learning, Amortized Inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eF2kOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper describes new intervention methods (basically, overwriting in the synthetic data generation process) to improve training of prior-data fitted networks (PFN).

PFNs are the state of the art in tabular data modeling and improvements are highly relevant.

While the results are probably very good and highly important, the paper is not accessible to the readers in its present form. It reads like an internal note accessible only to those very closely working in the area. As a reader with only general knowledge on PFNs I had to query term by term all notions in the Abstract and the Introduction, and I would still not be able to reproduce the algorithm of Section 3. I will expand in the Questions.

The reproducibility statement and Appendix H refers to a reviewer source code package but I find neither supplementary material in the submission nor a link to an anonymous repository in the paper.

Overall, I find the results probably very valuable, but the paper not yet ready for publication.

Critical strengths

PFNs are the state-of-the-art in tabular data modeling, outperforming all competitors in almost all aspects. Hence improved PFN training is a very important contribution.

Critical weaknesses

The paper reads like a highly technical internal note to a developer team. As a reviewer with general knowledge only, I had to search term by term and still cannot reproduce the intervention algorithm for myself. In its current form, the paper is not accessible for the conference audience.

Questions for authors

Please include definitions of

  • Structural equation models - just a sentence that this is the process of generating the synthetic training data
  • What is an intervention, how does it modify training the foundation model
  • Instead of four-arm, more accessible to say four ways of intervening/overwirting(?)
  • Say at least once KL is Kullback-Leibler (I hope so)
  • Signed, Fresh, without explanation are just arbitrary names of Q+R and Q+A+R. And in the Introduction, we don't yet know what is the definition of q_Z. The meaning of the sentence "Signed supplies an experimental counterfactual" still remains unclear to me.

Please revise Section 3, clearly listing the notation and the algorithm. When reading H=(D,x) starting with "20 complete 3-variable ..." is it a general definition or an illustration with parameters 20 and 3? Is it a 2-dimensional synthetic data plus a label? Why 2-dimensional and why 20 observations? Later in my understanding, q, Q are used as general definitions.

Please also give a clear single-chart summary on what is the baseline training procedure, I assume using synthetic data generated by broadly described in Table 4 but I would like to see a more precise statement.

Reviewer_yNRQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how label noise in the training targets affects PFNs at a fixed training budget, both in distribution and under prior shift. It claims that the choice between sampled hard labels and soft targets from the exact posterior predictive matters even though both give the same expected loss, that soft targets fit the training prior better but can predict worse under prior shift, and that this trade-off is driven by observation noise rather than by uncertainty about which task generated the data. To test this, the authors use a prior of 48 linear structural equation models, where the posterior predictive can be computed exactly, and build four training targets with the same expected loss that include or exclude each of the two noise sources. They then train a small transformer on each. Removing observation noise improves in-distribution KL, worsens KL when the structural coefficients shift, and improves it when the covariances shift. A study across six priors shows that the preferred target switches from soft to sampled as the shift grows. Temperature-softening and early-stopping controls and a small real-data study complete the evaluation, and the real-data study shows no significant effect.

Critical strengths

The experimental design is sound. All four proposed targets have the same expected loss, so any difference between arms has to necessarily come from training noise, not from different objectives. The signed target Q+RQ + R removes one noise source while keeping the other, something which a simple sampled versus soft label comparison cannot separate. The derivations are correct. Therefore, this design is reusable, and future work can run it in a prior where the two noise sources are comparable in size.

The within-prior results seem reliable and are honestly reported. Seeds are paired across arms. Evaluations use withheld blocks, which are separate from the blocks used during discovery. Also, failed control matches are kept, not dropped, and per-seed tables are reported.

Critical weaknesses

-- The headline contribution of the paper is to identify which component of randomness drives the trade-off. In the toy setting, Table 3 gives the trace of BB as 0.0021 and the trace of WW as 0.911, so the component variation AA is only 0.23% of the target variance. Therefore, the finding that RR drives the trade-off is nearly forced. The question therefore is whether the amount of noise or its kind matters, and the authors rightly concede that the design cannot separate the two (L317–318). The scaled targets that would separate them are defined in Appendix A but never run (L565), which leaves the main result of the paper hanging.

-- The paper's controls show that the shifted benefit does not need supervision randomness. Softening the noise-free qZq_Z model with a single temperature recovers all of Fresh's coefficient-shift advantage in every seed, even with slightly lower source KL than Fresh, and early stopping recovers most of it in the seeds that pass source matching (Tables 7, 12 and 13). The authors' response is that the softened predictions differ from Fresh's, but different predictions do not show that the benefit requires supervision randomness. This undercuts the claim that the estimator choice is consequential.

-- The roughly 3% change in shifted KL cannot be interpreted without a reference, namely the exact source predictor's shifted risk, which the paper never reports (L716–720). Since the objective is identical across arms, the effect is purely an optimization effect, yet only one training recipe is tested, so it is unknown whether the effect survives a change of budget, batch size or learning rate. The real-data studies show no significant effect (Table 18), so nothing in the paper supports the claim of practical consequence (L028–029).

-- The paper is not presented well, to the point where it obscures what it establishes. It does not build an argument from its results. The results are reported as facts, immediately qualified with what they do not show rather than what they do show, and left for the reader to interpret. When the reader does interpret them, as with the softening control above, they cut against the paper's own claims. As minor points, numbers are reported to machine precision, the appendix reads like an audit log of file names and commit hashes, Figure 2 repeats Table 6, and Figure 4 shows no uncertainty even though none of its differences are significant (Table 2).

Questions for authors
  1. Can you report a scaled-residual arm, for example T0,βT_{0,\beta} with β=0.0477\beta = 0.0477 from Appendix A, or vary the context size so that tr B\mathrm{tr}\,B and tr W\mathrm{tr}\,W are comparable? This would separate the amount of noise from its kind.

  2. Does tempering the exact source predictor QsQ_s itself, with no training, reproduce Fresh's coefficient-shift advantage? If it does, the benefit comes from predictive sharpness rather than from supervision randomness.

  3. What is E DKL(Qt∥Qs)\mathbb{E}\,D_{\mathrm{KL}}(Q_t\Vert Q_s) on each shifted panel and at each λ\lambda, and does any trained arm achieve lower coefficient-shift KL than the exact source predictor? Relatedly, do the source gap and the coefficient-shift advantage persist with 2–4× more updates, a larger batch, or a lower peak learning rate?

Reviewer_cmfHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on the training-targets for prior-fitted networks. PFNs are trained on a large amount of synthetic tasks and often one has complete control over the data generation process. Therefore, for a given query, the paper studies in detail about what should be the training target of that query. There are two main choices, one being a realization (noisy) from the generation process like in conventional neural network training with static datasets and the second one is providing the entire bayes-optimal target as the full posterior could be calculated. While in expectation they will be same, since the training happens in finite-sample regime, the paper posits that each choice will have non-trivial effects. To that end, the paper instead breaks this down into a four-way factorial and study the effect of each choice. Namely, they consider the targets as Q (full posterior i.e,, averaged across unknown including the component), qz ( the target obtained once one knows the true component that generated the target), Q + R where R is the residual label and fresh which is given by Q+A+R, where A= qz-Q, i,e., the response that cames purely from not knowing the true component.

They train a 2-layer transformer with a 3-variable structural equation model and discretizes the continuous target into 100 bins and has the network predict a probability over those bins. The authors make a series of interesting findings and in my reading, paints a mixed picture. For example in the the six-prior study. Pooled across all six priors, Fresh beats full-predictive supervision under coefficient shift at the primary interpolation level but underperforms under covariance-family shift at the same level. Across shift magnitude as well, the order changes within each shift type i.e, full-predictive supervision wins at the smallest tested interpolation level, while Fresh wins at the largest.

Critical strengths

The paper does a good job in the exposition of the core contributions, it is principled and the decomposition is principled. Writing a sampled label as L = Q + A + R, provided the pathway to answer the question of the better training targets. The signed arm is well motivated and its rationale and constructing it by hand was well explained.

The paper spends considerable energy in running sufficient statistical analyses- with paired bootstrap intervals,, two different numerical integration grids, and a reconstructed CRPS score. The temperature scaling experiment is also well-thought out.

Critical weaknesses

Section 4.6 is poorly written. It is of highest practical value as it deals with real data but only one paragraph is devoted to it. It is hard to parse. The results are negative, which the reviewer appreciates disclosing but that also limits the utilty of this analysis, coupled with the mixed results. It is unclear to the practitioner, the right targets. While the answer to that question, could be dependant on the regime the practitioner will oprtate the system in- (e.g. coefficient or covariance-shift), the issue is exacerbated because almost all of the findings are done on a two-layer transformer architecture. Would the same findings hold one transformer with different number of layers i.e, with scale?.

The discussion about seeds is also rather quite odd. "Seed 71 contributes an unusually large +0.006147 difference; the other four range from +0.000404 to +0.001159. Only two seeds pass both source- matching checks; the pooled comparison includes all five. Appendix E reports each selection and effect; its descriptive 95% interval is not the primary result." The experimental section in general needs careful rewriting.

Questions for authors

Please see the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.