acceptodds
Under review as a conference paper at ICLR 2027

GIST: Generator-state-Informed Supervision for Tabular Foundation Models

Abstract

Tabular foundation models (TFMs) based on prior-data fitted networks typically use sampled hard query labels, leaving avoidable sampling variability in pretraining. A key observation is that intermediate generator states can reveal label uncertainty beyond a sampled outcome. We propose **G**enerator-state-**I**nformed **S**upervision for **T**FMs (**GIST**), which uses these states to average query losses over conditional label distributions. GIST couples this supervision with entropy-calibrated noise in selected feature-generation steps, making the required distributions tractable and consistent with the model input. We show that this averaging preserves the population objective and yields no larger gradient covariance at fixed parameters within each generating distribution. Under matched pretraining schedules, GIST continuously improves mean rank across three architectures on both TabArena and TALENT without changing architecture or inference. Across these benchmarks and architectures, GIST wins 62.1% of paired dataset-level comparisons against the corresponding hard-label controls. Code is available at https://anonymous.4open.science/r/GIST-anonymous-B54D.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Deep Learning, Tabular Foundation Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2s9FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GIST changes how PFN style tabular foundation models are pretrained. On branches with tractable conditional targets, GIST replaces sampled hard labels with conditional label distributions, preserving the expected loss and not increasing gradient covariance at fixed parameters within the same generating distribution. Supported terminal branches use their sampling probabilities directly, supported label-dependent branches use an entropy-calibrated Gaussian channel, and unsupported branches retain hard labels. Small re-implementations of TabICLv2, TabSwift and TabPFN-2.5 trained this way improve mean rank on TabArena and TALENT over hard label training

Critical strengths

S1: The idea is simple and well motivated. The generator knows more about the label than the single sample it outputs, and using this information requires no change to the model or to inference. The link to Rao-Blackwellization and to ALICE is appropriate.

S2: The comparison is controlled in several ways. Three architectures are trained with the same schedules, Hard and GIST are compared in pairs, both IID and OOD benchmarks are used, and cases where GIST is worse are marked in Table 1.

S3: The gradient variance analysis in Appendix D.4 and D.5 is thorough, with paired bootstrap intervals, a decomposition by component and a comparison between the full mixture and the tractable subset in Table 8. The finding that 72% to 91.5% of the reduction comes from components outside the prediction head is interesting

Critical weaknesses

W1: GIST changes the generator itself, and the paper states that the Gaussian encoding "changes the affected branch's feature-label distribution". The downstream comparison, however, is between "the original generator and hard query labels" and the new generator with conditional targets. The theory compares losses within the same generating distribution, and the gradient analysis uses a shared calibrated prior, but the downstream results do not. The key missing control is the calibrated prior trained with hard labels, and ideally also the original prior with conditional targets only on the terminal branches. Without these, the downstream gains cannot be attributed to the new supervision rather than to a different and noisier prior. This is the main reason for my rating.

W2: The gains are small and their benefit is not well established. First, on TabArena the AUC changes by only 0.0005 to 0.0034, and some metrics get worse. For example, the log loss of TabICLv2 on TabArena goes from 0.2541 to 0.2549, and that of TabPFN-2.5 on TALENT from 0.3909 to 0.3915. Each setting is a single pretraining run with one model seed per architecture (Appendix D.1). Under a simple sign test several paired comparisons are not significant, for example 21 wins and 17 losses for TabICLv2 on TabArena and 101 wins and 85 losses for TabPFN-2.5 on TALENT, although the TALENT results for TabICLv2 and TabSwift are significant. On the 12 common BeyondArena datasets, the mean rank gets worse for TabICLv2 (7.21 to 7.26) and TabPFN-2.5 (4.19 to 4.68) in Table 7, which the OOD paragraph in Section 4.2 does not mention. Second, the variance reduction reaches only part of the data. In the full mixture only about 26% to 31% of query rows get conditional targets (Table 8), and the reduction there is 1.37% to 7.33%, with -0.002% for TabSwift at length 8192. The main text reports the 6.8% to 26.04% measured on the tractable subset, and the full mixture numbers appear only in Table 8 of the appendix. There are no learning curves and no comparison of the steps needed to reach a given performance, so faster or more stable training is not shown. Finally, the models are small re-implementations trained for 36K updates, so it is unclear whether the gains hold at the scale of the released models.

W3: The theory is standard and the paper is hard to read. Proposition 1 follows directly from the laws of total expectation and total covariance, so the contribution lies mainly in the construction that makes some branches tractable, which makes clear writing important. The abstract and Section 3 use many new terms, such as terminal draw, supported label-dependent branches, input-consistent targets, binary-cap budget and all-row averaging, without a simple example. Figures 1 and 2 are dense schematics with text too small to read when printed, the stacked bars in Figure 3 are separated only by color, and Table 1 marks better and worse results only with blue and green.

Questions for authors

Q1 (see W1): Please add the calibrated prior with hard labels under the same schedule, at least for TabSwift, and if possible the original prior with conditional targets only on terminal branches. If the calibrated prior with hard labels gives most of the gain, the main claim of the paper would need to be reframed.

Q2 (see W2 ): Could the authors repeat Hard and GIST pretraining with two or three more seeds for TabSwift, the cheapest model, and report confidence intervals and paired tests on TabArena and TALENT? Please also discuss the OOD rank results in Table 7. This could change my rating.

Q3 (see W2 ): Do learning curves show that GIST reaches the final Hard performance earlier, for example downstream metrics at intermediate checkpoints or loss on a fixed held out synthetic stream? Since conditional targets cover only about 26% to 31% of query rows, can this coverage be increased, and does the gain grow with it?

Q4 (see W3 ): How does GIST compare with simpler soft targets, such as label smoothing with matched entropy? I also suggest adding a small worked example with three classes in the main text, simplifying and enlarging Figures 1 and 2, and using hatching or a colorblind friendly palette in Figure 3 and Table 1, so that the results can be read without relying on color.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 2s9F

We thank the reviewer for the detailed and careful review and for the time and effort spent examining the method, experiments, and presentation. We appreciate the recognition of the motivation behind generator-informed supervision, the controlled experimental design, and the gradient-variance analysis.

1. W1, Q1: Calibrated prior with hard labels

We trained TabSwift with GIST's calibrated generator and sampled hard labels under the paper's full 30K + 6K schedule, with the same seeds and effective batch sizes as the paper's TabSwift Hard and GIST runs.

MethodGeneratorQuery targetsTabArena AUC ↑TabArena log loss ↓TALENT AUC ↑TALENT log loss ↓
HardOriginalSampled0.8420150.2848040.8712640.426727
Calibrated-hardCalibratedSampled0.8423110.2819560.8722200.424278
GISTCalibratedConditional0.8424690.2811340.8726050.424334
Comparison (first vs second)TabArena W / L / TTALENT W / L / T
Calibrated-hard vs Hard21 / 17 / 0112 / 75 / 1
GIST vs Calibrated-hard21 / 17 / 0105 / 82 / 1
GIST vs Hard25 / 13 / 0126 / 61 / 1

AUC uses binary datasets (30 on TabArena, 120 on TALENT) and log loss uses multiclass datasets (8 and 68).

Each component of GIST improves prediction, and the complete design is strongest. The calibrated generator improves all four aggregates over Hard. With that generator fixed, conditional targets further raise AUC on both benchmarks, lower TabArena log loss, and win on a majority of datasets on both benchmarks, with TALENT log loss within 0.00006 of calibrated-hard. The complete design has the best value in three of the four aggregates and the strongest paired result against Hard: 126 TALENT wins versus 112 for calibrated-hard, and 25 TabArena wins versus 21.

These results show that both the calibrated task generation and the conditional query supervision contribute to downstream performance, rather than the gains coming from either component alone. The two components are coupled by construction: on label-dependent branches, exact conditional targets exist only once the class reaches downstream features through the calibrated channel (Appendix B.2), and the channel's scale is chosen so that these targets retain residual uncertainty to average over (Appendix C). In a future version of the paper, we will include this three-arm ablation alongside fuller component analyses and clarify how both components jointly account for the empirical gains.

Given the time and compute available during the rebuttal, we have not yet trained the original-generator control with conditional targets only on terminal branches; the selective-supervision identity predicts its outcome. With support restricted to terminal branches, the removed covariance E[A Γθ(Z)]\mathbb E[A\,\Gamma_\theta(Z)] covers only those rows, and terminal targets account for about 4% of query rows in the synthetic task mixture, compared with 26% to 31% for all conditional targets (Table 8). This control therefore changes only a small part of the supervision, and we expect it to perform close to Hard.

2. W2, Q2: Additional seeds, confidence intervals, and paired tests

Within the rebuttal period, we pretrained TabSwift Hard/GIST pairs from Stage 1 with three different seeds, the paper's pair and two new pairs, under the paper's 30K + 6K schedule. To test whether the gain holds with longer training, we then extended Stage 2 of every pair from 6K to 20K updates, more than three times its original budget, with a second warmup-stable-decay cycle. Within each pair, both arms share the schedule, effective batch size, and data. TALENT results, mean ± standard deviation across the three pairs:

Stage 2 updatesMetricHardGISTPairs improvedDataset-bootstrap 95% CI of the gain
6KAUC ↑0.871352 ± 0.0002140.872435 ± 0.0005423 / 3[+0.00033, +0.00208]
6KLog loss ↓0.425658 ± 0.0009350.424200 ± 0.0003163 / 3[-0.00261, -0.00039]
20KAUC ↑0.872219 ± 0.0004580.873556 ± 0.0006493 / 3[+0.00052, +0.00239]
20KLog loss ↓0.423665 ± 0.0004880.421914 ± 0.0002253 / 3[-0.00317, -0.00068]

GIST improves both metrics in every pair at both endpoints, all four intervals exclude zero, and the mean gain exceeds the across-seed standard deviation of either arm. After averaging the runs within each dataset, GIST wins on 112 of 188 TALENT datasets at 6K and on 123 at 20K. On TabArena, the paper's pair improves AUC and log loss at both endpoints and wins on 25 of 38 datasets, with log-loss intervals above zero. Intervals resample datasets 20,000 times after averaging the runs; the standard deviations describe run-to-run variation.

3. W2, Q2: BeyondArena results

The appendix reports 12 BeyondArena datasets because that is the set all three architectures can complete. TabPFN-2.5 exceeds the inference time limit on many of the large OOD classification datasets, TabSwift supports at most 10 classes, and TabICLv2 runs on every dataset in BeyondArena's non-IID classification collection. Appendix Table 7 places the three architectures and the external baselines on this common set to give additional information. The most appropriate comparison for GIST is the paired comparison within each architecture, and more datasets make that comparison more reliable, so the main text reports each architecture's paired win rates on as many BeyondArena datasets as it can complete (Table 2). In a future version of the paper, the OOD paragraph of Section 4.2 will state this design and discuss the common-set results, including mean rank.

4. W2, Q3: Learning curves and coverage

The table summarizes the validation loss of Stage 1 for the TabICLv2, TabPFN-2.5 and TabSwift runs. Each threshold is Hard's CE at exactly 30K, and each crossing is an observed validation point.

ArchitectureHard CE, 30KGIST CE, 30KFirst GIST observation at or below Hard 30K CEUpdates saved
TabICLv20.4671720.46394727.5K (CE 0.465616)2.5K (8.3%)
TabPFN-2.50.4469250.44408527K (CE 0.446875)3K (10.0%)
TabSwift0.5164930.51338027.5K (CE 0.515248)2.5K (8.3%)

GIST reaches Hard's final Stage 1 CE with 8.3% to 10.0% fewer updates under the same learning-rate schedule, and every later observation through 30K stays at or below that value. At 30K, GIST has the lower CE in each case.

On coverage: raising it requires valid conditional targets on further generator branches, which also changes the generator and hence the mix of training tasks, so its downstream effect is difficult to isolate. The selective-supervision identity shows that every additional supported branch removes further label-sampling variance, which should make training more stable. We will add experiments that vary coverage in a future version of the paper.

5. W3, Q4: Entropy-matched label smoothing

During the rebuttal, due to time and compute constraints, we trained a new TabSwift arm that replaces GIST's conditional targets with uniform smoothing. It starts from the same Stage 1 endpoint (the 30K GIST checkpoint) as the paper's GIST run and trains Stage 2 for 6K updates on the same calibrated generator and schedule, so the two arms differ only in the Stage 2 query targets. For each query row with sampled class yy, the smoothing target q=(1−ϵ)ey+ϵ1/Kq=(1-\epsilon)e_y+\epsilon\mathbf 1/K uses the ϵ\epsilon for which H(q)H(q) equals the entropy of the GIST target, solved by bisection.

MethodStage 1Stage 2 query targetsTALENT AUC ↑TALENT log loss ↓TabArena AUC ↑TabArena log loss ↓
Entropy-matched smoothingGISTUniform smoothing, matched entropy0.8723200.4259360.8422090.280381
GISTGISTConditional0.8726050.4243340.8424690.281134
ComparisonTALENT W / L / TTabArena W / L / T
GIST vs Smoothing123 / 62 / 324 / 14 / 0

At matched target entropy, GIST outperforms smoothing on three of the four aggregates and on a majority of datasets on both benchmarks. On TALENT, GIST raises AUC by 0.00029 and lowers log loss by 0.00160 relative to smoothing, winning on 123 datasets and losing on 62 (3 ties). Smoothing has the lower TabArena mean log loss, while GIST raises TabArena mean AUC (0.842469 vs. 0.842209) and wins on 5 of those 8 multiclass datasets (24 of 38 overall). The target identities explain why: with ϵ\epsilon set from the generator state, E[q∣Z]=(1−ϵ)pZ+ϵ1/K\mathbb E[q\mid Z]=(1-\epsilon)p_Z+\epsilon\mathbf 1/K and Cov⁡(q∣Z)=(1−ϵ)2[Diag⁡(pZ)−pZpZ⊤]\operatorname{Cov}(q\mid Z)=(1-\epsilon)^2[\operatorname{Diag}(p_Z)-p_Zp_Z^\top], so smoothing retains part of the label-sampling covariance and shifts the expected target toward uniform, whereas GIST removes that covariance and keeps E[pZ∣S]=P(Y∣S)\mathbb E[p_Z\mid S]=P(Y\mid S).

6. W3: Presentation and theory

We apologize for the reading effort that the current presentation required, and we thank the reviewer again for the patient reading. Proposition 1 is the classical Rao-Blackwell step, which we credit as the basis. The contribution is what makes it hold inside TFM generators: an exactness condition after label-dependent feature generation and task filtering (Appendix B), a closed-form Gaussian-channel posterior that keeps label uncertainty to average (Appendix B.2), a certified entropy bound guaranteeing at least (1−β)H‾(1-\beta)\overline H nats of retained label information (Appendix C), covariance identities for mixed and multi-query tables (Appendix A.4 and A.5), and the transfer to the asymptotic parameter covariance of PFN estimation (Appendix A.6).

In a future version of the paper, Section 3 will open with a three-class worked example covering a terminal draw, the Gaussian channel, and its calibration, and will define each new term where it first appears; the abstract will use plain language; Figures 1 and 2 will be split into larger, simpler panels; Figure 3 will add hatching; and Table 1 will mark changes with ▲ and ▼ in addition to color. We thank the reviewer for these thoughtful and constructive suggestions, which will make the paper considerably easier to read.

Thanks for running the extra experiments. They clear up most of my concerns.

Still, in the three-arm ablation, calibrated-hard already gets 65–78% of the improvement over Hard (and all of the TALENT log-loss gain), and GIST vs. calibrated-hard isn't significant under a sign test. As I said in Q1, the main story seems to be the calibrated generator and conditional supervision working together rather than the conditional supervision alone, so I'm keeping my score.

Reviewer_mPQyOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces GIST, which is short for Generator-state-informed supervision for tabular foundation models. The main idea is to replace the hard labels in typical pfn-style training with soft labels, which are devised from the uncertainty of the states during data generation. The novel generator jointly builds the tasks as well as the query supervision using entropy-calibrated encodings. The authors show that this procedure preserves in expectation the population objective and yields no larger gradient. Empirically, the method improves the mean rank and wins 62.1% of paired dataset-level comparisons against the corresponding hard-label controls.

Critical strengths
  • The idea to use the generator state to obtain soft-labels and more complete supervision information is very interesting.
Critical weaknesses
  • The improvements are very minor and also within the noise range of a different seed and other noises in training. Hence, it is not possible to evaluate if this is an actual improvement. The authors would need to run several runs with different seeds to establish the effect of the method.
  • The covariance reduction would work for any method that introduces soft labels due to the structure of the Jacobian and how it is backpropagated to earlier layers. Furthermore, it is not clear why having a lower covariance is an advantage. This needs to be explained to make this a valid contribution of your proposed method.
Questions for authors

see Weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer mPQy

We thank the reviewer for the careful reading and for the time and effort devoted to the review. We appreciate your interest in using information retained by synthetic generators to improve pretraining supervision.

1. W1: Size and stability of the gains

Within the rebuttal period, we pretrained TabSwift Hard/GIST pairs from Stage 1 with three different seeds, the paper's pair and two new pairs, under the paper's 30K + 6K schedule. To test whether the gain holds with longer training, we then extended Stage 2 of every pair from 6K to 20K updates, more than three times its original budget, with a second warmup-stable-decay cycle. Within each pair, both arms share the schedule, effective batch size, and data. TALENT results, mean ± standard deviation across the three pairs:

Stage 2 updatesMetricHardGISTPairs improved
6KAUC ↑0.871352 ± 0.0002140.872435 ± 0.0005423 / 3
6KLog loss ↓0.425658 ± 0.0009350.424200 ± 0.0003163 / 3
20KAUC ↑0.872219 ± 0.0004580.873556 ± 0.0006493 / 3
20KLog loss ↓0.423665 ± 0.0004880.421914 ± 0.0002253 / 3

GIST improves both metrics in every pair at both endpoints; the smallest per-pair gain is +0.00063 AUC and -0.00084 log loss. The gain persists after Stage 2 is extended to 20K updates, where the mean gain is 2.1 to 7.8 times the across-seed standard deviation of either arm and GIST wins on 123 of 188 TALENT datasets after averaging the runs within each dataset. On TabArena, the paper's pair improves both aggregate metrics and wins on 25 of 38 datasets at both endpoints.

The gain also holds across architectures. We chose three representative TFM architectures that cover the main attention designs: TabSwift uses row-wise attention, TabICLv2 column-then-row encoding followed by in-context attention, and TabPFN-2.5 alternating within-row and within-column attention. In the paper's comparisons, GIST wins on a majority of datasets for all three architectures on both benchmarks:

ArchitectureAttention organizationGIST win rate on TabArenaGIST win rate on TALENT
TabICLv2Column-then-row encoding, followed by ICL attention55.3% (21/38)64.4% (121/188)
TabPFN-2.5Alternating within-row and within-column attention71.1% (27/38)53.7% (101/188)
TabSwiftRow-wise attention65.8% (25/38)67.0% (126/188)

2. W2: Why lower covariance helps, and what distinguishes GIST's soft targets

Why lower gradient covariance helps. PFN pretraining minimizes a Monte Carlo estimate of the expected cross-entropy over mm synthetic datasets. Nagler (2023), Statistical Foundations of Prior-Data Fitted Networks shows in Appendix B (Theorems B.1 and B.2) that the learned parameters then satisfy m(θ^−θ∗)→N(0,I−1VI−1)\sqrt m(\hat\theta-\theta^\ast)\to\mathcal N(0,I^{-1}VI^{-1}), where VV is the covariance of the per-task gradient, and that the variance of the resulting predictor scales with this covariance divided by mm. For a fixed number of synthetic tasks, lowering VV while keeping the objective, and therefore θ∗\theta^\ast and II, unchanged directly lowers this estimation error. Equation (15) gives the per-step counterpart: at fixed parameters and under the same generator, Hard and GIST estimate the same expected gradient, and

=\mathbb E[\operatorname{tr}\Gamma_\theta(Z)]\ge0.$$ This matters for TFMs because each synthetic task is costly to generate and process at long table lengths, and a larger effective batch requires proportionally more tasks. **Why generic soft targets differ.** We agree that softer targets generally reduce label-sampling variance, since every target $t$ enters the softmax gradient as $J_\theta^\top(q_\theta-t)$. The criterion above also requires the objective to stay unchanged. GIST's target $p_Z=P(Y\mid Z)$, with $Z\supseteq S$ the retained generator state, satisfies $\mathbb E[p_Z\mid S]=P(Y\mid S)$ and $G_{\mathrm{GIST}}=\mathbb E[G_{\mathrm{Hard}}\mid Z]$, so it meets this requirement and removes the entire label-sampling term. Label smoothing $q=(1-\epsilon)e_Y+\epsilon\,\mathbf 1/K$ with $\epsilon$ set from the generator state gives instead $$\mathbb E[q\mid Z]=(1-\epsilon)\,p_Z+\epsilon\,\mathbf 1/K, \qquad \operatorname{Cov}(q\mid Z)=(1-\epsilon)^2\left[\operatorname{Diag}(p_Z)-p_Zp_Z^\top\right],$$ so it retains part of the label-sampling covariance and shifts the expected target, and with it the minimizer, toward uniform. Furthermore, we add additional experimental evidence here. **Learning curves.** The table summarizes Stage 1 for the TabICLv2, TabPFN-2.5 and TabSwift runs. | Architecture | Hard CE, 30K | GIST CE, 30K | First GIST observation at or below Hard 30K CE | Updates saved | | ------------ | -----------: | -----------: | ---------------------------------------------: | ------------: | | TabICLv2 | 0.467172 | **0.463947** | 27.5K | 2.5K (8.3%) | | TabPFN-2.5 | 0.446925 | **0.444085** | 27K | 3K (10.0%) | | TabSwift | 0.516493 | **0.513380** | 27.5K | 2.5K (8.3%) | **GIST reaches Hard's final Stage 1 CE with 8.3% to 10.0% fewer updates** under the same learning-rate schedule, and every later observation through 30K stays at or below that value. **Entropy-matched label smoothing.** During the rebuttal, we trained a new TabSwift arm that replaces GIST's conditional targets with uniform smoothing whose $\epsilon$ is solved per row to match the entropy of the GIST target. It starts from the same Stage 1 checkpoint as the paper's GIST run and trains Stage 2 for 6K updates on the same calibrated generator and schedule, so the two arms differ only in the Stage 2 query targets. **GIST beats entropy-matched smoothing on 123 of 188 TALENT datasets (62 losses, 3 ties)**, raising AUC from 0.872320 to 0.872605 and lowering log loss from 0.425936 to 0.424334. At equal entropy, allocating probability according to the generator's conditional distribution outperforms spreading it uniformly, providing direct evidence that GIST's unbiased conditional supervision improves downstream prediction beyond generic label smoothing.

Dear authors, thank you for the elaborate rebuttal. I acknowledge that  gain persists over three runs, yet it is still miniscuel and therefore could still be due to randomness. On the other hand, the gain in training speedup is very interesting, and I would kindly suggest making this the main advantage of  the method. Given this result, I raise my score to 2.

Reviewer_zoCdOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes to use information retained by a synthetic data generator to construct conditional query-label targets for pretraining tabular foundation models. For supported label-dependent branches, a calibrated Gaussian encoding makes these targets tractable. The authors establish objective preservation and reduced raw-gradient covariance within a fixed generating distribution, and evaluate the complete generator-supervision design across three architectures.

Critical strengths
  1. The method uses generator information only during pretraining and does not require changing the predictor or its inference procedure.
  2. The distinction between terminal label draws and label-dependent feature generation is useful. The paper also explicitly recognizes that the Gaussian encoding changes the affected generating distribution.
Critical weaknesses
  1. Separate the effects of generator modification and conditional supervision. The main Hard control uses the original generator, whereas GIST changes both the generator and the query targets. It is recommended to include downstream training with the calibrated generator and hard labels under the same schedule. The shared-prior gradient diagnostics are useful, but do not replace this predictive-performance ablation. This comparison is important for understanding how much of the gain actually comes from conditional averaging.

  2. The paper should be better motivated and the novelty should be stated more precisely. Proposition 1 follows from conditional expectation and the law of total covariance, and the paper discusses related conditional-probability supervision in ALICE. The authors should distinguish the established statistical principle from the new target construction, the handling of filtering, and the entropy-calibrated encoding. A clearer explanation of why these ingredients are necessary in this setting would improve the paper.

Questions for authors
  1. How does the calibrated-generator hard-label control perform after the full pretraining schedule?
  2. Are the downstream gains stable across independent pretraining seeds?
  3. What is the end-to-end cost of calibration and conditional-target construction?
3 comments
AuthorsOpenReview ↗

Response to Reviewer zoCd

We thank the reviewer for the careful review and for the time and effort invested in evaluating our work.

1. W1, Q1: Calibrated generator with hard labels

We trained TabSwift with GIST's calibrated generator and sampled hard labels under the paper's full 30K + 6K schedule, with the same seeds and effective batch sizes as the paper's TabSwift Hard and GIST runs. The three arms separate GIST's two components.

MethodGeneratorQuery targetsTabArena AUC ↑TabArena log loss ↓TALENT AUC ↑TALENT log loss ↓
HardOriginalSampled0.8420150.2848040.8712640.426727
Calibrated-hardCalibratedSampled0.8423110.2819560.8722200.424278
GISTCalibratedConditional0.8424690.2811340.8726050.424334
Comparison (first vs second)TabArena W / L / TTALENT W / L / T
Calibrated-hard vs Hard21 / 17 / 0112 / 75 / 1
GIST vs Calibrated-hard21 / 17 / 0105 / 82 / 1
GIST vs Hard25 / 13 / 0126 / 61 / 1

AUC uses binary datasets (30 on TabArena, 120 on TALENT) and log loss uses multiclass datasets (8 and 68), averaged over splits within each dataset and then equally across datasets.

Each component of GIST improves prediction, and the complete design is strongest. The calibrated generator improves all four aggregates over Hard. With that generator fixed, conditional targets further raise AUC on both benchmarks, lower TabArena log loss, and win on a majority of datasets on both benchmarks, with TALENT log loss within 0.00006 of calibrated-hard. The complete design has the best value in three of the four aggregates and the strongest paired result against Hard: 126 TALENT wins versus 112 for calibrated-hard, and 25 TabArena wins versus 21. These results show that both the calibrated task generation and the conditional query supervision contribute to downstream performance, rather than the gains coming from either component alone. The two are coupled by design, as exact conditional targets on label-dependent branches are defined through the calibrated channel (Section 2 below). In a future version of the paper, we will include this three-arm ablation alongside fuller component analyses.

2. W2: Motivation, contribution, and relation to ALICE

Motivation. PFN pretraining fits the posterior predictive distribution by minimizing a Monte Carlo estimate of the expected cross-entropy over mm synthetic datasets. Nagler (2023), Statistical Foundations of Prior-Data Fitted Networks, shows in Appendix B (Theorems B.1 and B.2) that the learned parameters then satisfy m(θ^−θ∗)→N(0,I−1VI−1)\sqrt m(\hat\theta-\theta^\ast)\to\mathcal N(0,I^{-1}VI^{-1}), where VV is the covariance of the per-task gradient, and that the variance of the resulting predictor scales with this covariance divided by mm; more complex models therefore need more synthetic datasets to keep the variance small. For a fixed budget of mm tasks, the remaining lever is VV.

Core idea. A synthetic generator knows more about each query label than the single class it samples: it holds the class probabilities and the state from which the features were produced. GIST uses this retained information to supervise each query with the class distribution conditional on the model input and the generator state, in place of the sampled label. This is Rao-Blackwellization (Rao, 1945; Blackwell, 1947), the established principle on which we build (Proposition 1). It leaves the objective, and therefore θ∗\theta^\ast and II, unchanged and removes the label-sampling component of VV.

From principle to construction. The step requires the class distribution conditional on everything the model sees. A natural candidate is the probability vector from which the generator samples the class, and it is valid only when the sampled class leaves the model input unchanged. When the class also shapes the features, it becomes the wrong target. For example, if YY is uniform on {0,1}\{0,1\} and a feature copies it, X=YX=Y, the pre-draw target is always (1/2,1/2)(1/2,1/2) and teaches the model to ignore a perfectly predictive feature, whereas the correct target is a point mass at the observed class (Appendix B.1). Synthetic task generators create this situation throughout: sampled classes feed later features through random MLP and tree nodes, and generated tables then pass acceptance checks. The correct target must therefore condition on the final input and on acceptance, and the original generator provides no tractable likelihood for it. GIST builds a controlled noise channel into the generator so that this conditional distribution has a closed form:

  1. Gaussian channel (Section 3.2, Appendix B.2). The class reaches downstream features only through a retained noisy code W=bY+σεW=b_Y+\sigma\varepsilon, so the downstream likelihood is common to all candidate classes and cancels, leaving the closed-form local posterior in Equation (5).
  2. Acceptance condition (Appendix B.1 to B.3). When acceptance depends on query labels only through retained variables, the same posterior remains exact on accepted tasks.
  3. Entropy calibration (Section 3.3, Appendix C). The noise is the largest certified by the budget βmin⁡{H‾,log⁡2}\beta\min\{\overline H,\log 2\}: enough to leave label uncertainty to average, while the encoding provably retains at least (1−β)H‾(1-\beta)\overline H nats of label information on average.

Relation to ALICE. ALICE (Stoye et al., 2018) applies the same conditional-target principle to a different task, likelihood-ratio estimation for simulator parameters. Its label records which parameter value generated an observation, and the simulator's latent path supplies a tractable joint likelihood ratio, so the target follows with the simulator used as given. In TFM pretraining, the label is a class drawn inside the generator, and the model observes a whole table, including features produced downstream of that class. The label's likelihood given what the model sees is therefore unavailable from the generator as given; GIST constructs it through the channel, acceptance condition, and calibration above.

3. Q2: Stability across pretraining seeds

Within the rebuttal period, we pretrained TabSwift Hard/GIST pairs from Stage 1 with three different seeds, the paper's pair and two new pairs, under the paper's 30K + 6K schedule. To test whether the gain holds with longer training, we then extended Stage 2 of every pair from 6K to 20K updates, more than three times its original budget, with a second warmup-stable-decay cycle. Within each pair, both arms share the schedule, effective batch size, and data. TALENT results, mean ± standard deviation across the three pairs:

Stage 2 updatesMetricHardGISTPairs improved
6KAUC ↑0.871352 ± 0.0002140.872435 ± 0.0005423 / 3
6KLog loss ↓0.425658 ± 0.0009350.424200 ± 0.0003163 / 3
20KAUC ↑0.872219 ± 0.0004580.873556 ± 0.0006493 / 3
20KLog loss ↓0.423665 ± 0.0004880.421914 ± 0.0002253 / 3

GIST improves both metrics in every pair at both endpoints, including after the extended Stage 2, and the mean gain exceeds the across-seed standard deviation of either arm in all four cases. After averaging the runs within each dataset, GIST wins on 112 and 123 of the 188 TALENT datasets at 6K and 20K.

4. Q3: Cost of calibration and target construction

GIST adds CPU work to task generation and no model computation. Calibration evaluates the entropy bound in O(LK2)O(LK^2) time within a 24-step bisection per task, and each query target is a KK-way softmax. End-to-end training time for the paper's 30K + 6K schedule on the paper's training hardware:

ArchitectureHardGISTChange
TabICLv217.79 h17.98 h+1.1%
TabPFN-2.570.42 h70.32 h-0.1%
TabSwift13.52 h15.97 h+18.1%

For TabICLv2 and TabPFN-2.5, the overhead is negligible: their accelerator step is long enough for prefetching to hide the extra generation work. TabSwift, the smallest and fastest of the three models, is limited by synthetic-task generation even with hard labels, so the added CPU work appears as an 18.1% increase; TabSwift's own training pipeline likewise generates synthetic data in batches before training. Because GIST's conditional targets depend only on the generator state and not on model parameters, they can be generated offline together with the tasks, which moves this cost out of the training loop. The architecture and inference procedure are unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.