- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
GIST changes how PFN style tabular foundation models are pretrained. On branches with tractable conditional targets, GIST replaces sampled hard labels with conditional label distributions, preserving the expected loss and not increasing gradient covariance at fixed parameters within the same generating distribution. Supported terminal branches use their sampling probabilities directly, supported label-dependent branches use an entropy-calibrated Gaussian channel, and unsupported branches retain hard labels. Small re-implementations of TabICLv2, TabSwift and TabPFN-2.5 trained this way improve mean rank on TabArena and TALENT over hard label training
Critical strengths
S1: The idea is simple and well motivated. The generator knows more about the label than the single sample it outputs, and using this information requires no change to the model or to inference. The link to Rao-Blackwellization and to ALICE is appropriate.
S2: The comparison is controlled in several ways. Three architectures are trained with the same schedules, Hard and GIST are compared in pairs, both IID and OOD benchmarks are used, and cases where GIST is worse are marked in Table 1.
S3: The gradient variance analysis in Appendix D.4 and D.5 is thorough, with paired bootstrap intervals, a decomposition by component and a comparison between the full mixture and the tractable subset in Table 8. The finding that 72% to 91.5% of the reduction comes from components outside the prediction head is interesting
Critical weaknesses
W1: GIST changes the generator itself, and the paper states that the Gaussian encoding "changes the affected branch's feature-label distribution". The downstream comparison, however, is between "the original generator and hard query labels" and the new generator with conditional targets. The theory compares losses within the same generating distribution, and the gradient analysis uses a shared calibrated prior, but the downstream results do not. The key missing control is the calibrated prior trained with hard labels, and ideally also the original prior with conditional targets only on the terminal branches. Without these, the downstream gains cannot be attributed to the new supervision rather than to a different and noisier prior. This is the main reason for my rating.
W2: The gains are small and their benefit is not well established. First, on TabArena the AUC changes by only 0.0005 to 0.0034, and some metrics get worse. For example, the log loss of TabICLv2 on TabArena goes from 0.2541 to 0.2549, and that of TabPFN-2.5 on TALENT from 0.3909 to 0.3915. Each setting is a single pretraining run with one model seed per architecture (Appendix D.1). Under a simple sign test several paired comparisons are not significant, for example 21 wins and 17 losses for TabICLv2 on TabArena and 101 wins and 85 losses for TabPFN-2.5 on TALENT, although the TALENT results for TabICLv2 and TabSwift are significant. On the 12 common BeyondArena datasets, the mean rank gets worse for TabICLv2 (7.21 to 7.26) and TabPFN-2.5 (4.19 to 4.68) in Table 7, which the OOD paragraph in Section 4.2 does not mention. Second, the variance reduction reaches only part of the data. In the full mixture only about 26% to 31% of query rows get conditional targets (Table 8), and the reduction there is 1.37% to 7.33%, with -0.002% for TabSwift at length 8192. The main text reports the 6.8% to 26.04% measured on the tractable subset, and the full mixture numbers appear only in Table 8 of the appendix. There are no learning curves and no comparison of the steps needed to reach a given performance, so faster or more stable training is not shown. Finally, the models are small re-implementations trained for 36K updates, so it is unclear whether the gains hold at the scale of the released models.
W3: The theory is standard and the paper is hard to read. Proposition 1 follows directly from the laws of total expectation and total covariance, so the contribution lies mainly in the construction that makes some branches tractable, which makes clear writing important. The abstract and Section 3 use many new terms, such as terminal draw, supported label-dependent branches, input-consistent targets, binary-cap budget and all-row averaging, without a simple example. Figures 1 and 2 are dense schematics with text too small to read when printed, the stacked bars in Figure 3 are separated only by color, and Table 1 marks better and worse results only with blue and green.
Questions for authors
Q1 (see W1): Please add the calibrated prior with hard labels under the same schedule, at least for TabSwift, and if possible the original prior with conditional targets only on terminal branches. If the calibrated prior with hard labels gives most of the gain, the main claim of the paper would need to be reframed.
Q2 (see W2 ): Could the authors repeat Hard and GIST pretraining with two or three more seeds for TabSwift, the cheapest model, and report confidence intervals and paired tests on TabArena and TALENT? Please also discuss the OOD rank results in Table 7. This could change my rating.
Q3 (see W2 ): Do learning curves show that GIST reaches the final Hard performance earlier, for example downstream metrics at intermediate checkpoints or loss on a fixed held out synthetic stream? Since conditional targets cover only about 26% to 31% of query rows, can this coverage be increased, and does the gain grow with it?
Q4 (see W3 ): How does GIST compare with simpler soft targets, such as label smoothing with matched entropy? I also suggest adding a small worked example with three classes in the main text, simplifying and enlarging Figures 1 and 2, and using hatching or a colorblind friendly palette in Figure 3 and Table 1, so that the results can be read without relying on color.
4 comments
2. W2, Q2: Additional seeds, confidence intervals, and paired tests
Within the rebuttal period, we pretrained TabSwift Hard/GIST pairs from Stage 1 with three different seeds, the paper's pair and two new pairs, under the paper's 30K + 6K schedule. To test whether the gain holds with longer training, we then extended Stage 2 of every pair from 6K to 20K updates, more than three times its original budget, with a second warmup-stable-decay cycle. Within each pair, both arms share the schedule, effective batch size, and data. TALENT results, mean ± standard deviation across the three pairs:
| Stage 2 updates | Metric | Hard | GIST | Pairs improved | Dataset-bootstrap 95% CI of the gain |
|---|---|---|---|---|---|
| 6K | AUC ↑ | 0.871352 ± 0.000214 | 0.872435 ± 0.000542 | 3 / 3 | [+0.00033, +0.00208] |
| 6K | Log loss ↓ | 0.425658 ± 0.000935 | 0.424200 ± 0.000316 | 3 / 3 | [-0.00261, -0.00039] |
| 20K | AUC ↑ | 0.872219 ± 0.000458 | 0.873556 ± 0.000649 | 3 / 3 | [+0.00052, +0.00239] |
| 20K | Log loss ↓ | 0.423665 ± 0.000488 | 0.421914 ± 0.000225 | 3 / 3 | [-0.00317, -0.00068] |
GIST improves both metrics in every pair at both endpoints, all four intervals exclude zero, and the mean gain exceeds the across-seed standard deviation of either arm. After averaging the runs within each dataset, GIST wins on 112 of 188 TALENT datasets at 6K and on 123 at 20K. On TabArena, the paper's pair improves AUC and log loss at both endpoints and wins on 25 of 38 datasets, with log-loss intervals above zero. Intervals resample datasets 20,000 times after averaging the runs; the standard deviations describe run-to-run variation.
3. W2, Q2: BeyondArena results
The appendix reports 12 BeyondArena datasets because that is the set all three architectures can complete. TabPFN-2.5 exceeds the inference time limit on many of the large OOD classification datasets, TabSwift supports at most 10 classes, and TabICLv2 runs on every dataset in BeyondArena's non-IID classification collection. Appendix Table 7 places the three architectures and the external baselines on this common set to give additional information. The most appropriate comparison for GIST is the paired comparison within each architecture, and more datasets make that comparison more reliable, so the main text reports each architecture's paired win rates on as many BeyondArena datasets as it can complete (Table 2). In a future version of the paper, the OOD paragraph of Section 4.2 will state this design and discuss the common-set results, including mean rank.
4. W2, Q3: Learning curves and coverage
The table summarizes the validation loss of Stage 1 for the TabICLv2, TabPFN-2.5 and TabSwift runs. Each threshold is Hard's CE at exactly 30K, and each crossing is an observed validation point.
| Architecture | Hard CE, 30K | GIST CE, 30K | First GIST observation at or below Hard 30K CE | Updates saved |
|---|---|---|---|---|
| TabICLv2 | 0.467172 | 0.463947 | 27.5K (CE 0.465616) | 2.5K (8.3%) |
| TabPFN-2.5 | 0.446925 | 0.444085 | 27K (CE 0.446875) | 3K (10.0%) |
| TabSwift | 0.516493 | 0.513380 | 27.5K (CE 0.515248) | 2.5K (8.3%) |
GIST reaches Hard's final Stage 1 CE with 8.3% to 10.0% fewer updates under the same learning-rate schedule, and every later observation through 30K stays at or below that value. At 30K, GIST has the lower CE in each case.
On coverage: raising it requires valid conditional targets on further generator branches, which also changes the generator and hence the mix of training tasks, so its downstream effect is difficult to isolate. The selective-supervision identity shows that every additional supported branch removes further label-sampling variance, which should make training more stable. We will add experiments that vary coverage in a future version of the paper.
5. W3, Q4: Entropy-matched label smoothing
During the rebuttal, due to time and compute constraints, we trained a new TabSwift arm that replaces GIST's conditional targets with uniform smoothing. It starts from the same Stage 1 endpoint (the 30K GIST checkpoint) as the paper's GIST run and trains Stage 2 for 6K updates on the same calibrated generator and schedule, so the two arms differ only in the Stage 2 query targets. For each query row with sampled class , the smoothing target uses the for which equals the entropy of the GIST target, solved by bisection.
| Method | Stage 1 | Stage 2 query targets | TALENT AUC ↑ | TALENT log loss ↓ | TabArena AUC ↑ | TabArena log loss ↓ |
|---|---|---|---|---|---|---|
| Entropy-matched smoothing | GIST | Uniform smoothing, matched entropy | 0.872320 | 0.425936 | 0.842209 | 0.280381 |
| GIST | GIST | Conditional | 0.872605 | 0.424334 | 0.842469 | 0.281134 |
| Comparison | TALENT W / L / T | TabArena W / L / T |
|---|---|---|
| GIST vs Smoothing | 123 / 62 / 3 | 24 / 14 / 0 |
At matched target entropy, GIST outperforms smoothing on three of the four aggregates and on a majority of datasets on both benchmarks. On TALENT, GIST raises AUC by 0.00029 and lowers log loss by 0.00160 relative to smoothing, winning on 123 datasets and losing on 62 (3 ties). Smoothing has the lower TabArena mean log loss, while GIST raises TabArena mean AUC (0.842469 vs. 0.842209) and wins on 5 of those 8 multiclass datasets (24 of 38 overall). The target identities explain why: with set from the generator state, and , so smoothing retains part of the label-sampling covariance and shifts the expected target toward uniform, whereas GIST removes that covariance and keeps .
6. W3: Presentation and theory
We apologize for the reading effort that the current presentation required, and we thank the reviewer again for the patient reading. Proposition 1 is the classical Rao-Blackwell step, which we credit as the basis. The contribution is what makes it hold inside TFM generators: an exactness condition after label-dependent feature generation and task filtering (Appendix B), a closed-form Gaussian-channel posterior that keeps label uncertainty to average (Appendix B.2), a certified entropy bound guaranteeing at least nats of retained label information (Appendix C), covariance identities for mixed and multi-query tables (Appendix A.4 and A.5), and the transfer to the asymptotic parameter covariance of PFN estimation (Appendix A.6).
In a future version of the paper, Section 3 will open with a three-class worked example covering a terminal draw, the Gaussian channel, and its calibration, and will define each new term where it first appears; the abstract will use plain language; Figures 1 and 2 will be split into larger, simpler panels; Figure 3 will add hatching; and Table 1 will mark changes with ▲ and ▼ in addition to color. We thank the reviewer for these thoughtful and constructive suggestions, which will make the paper considerably easier to read.
Thanks for running the extra experiments. They clear up most of my concerns.
Still, in the three-arm ablation, calibrated-hard already gets 65–78% of the improvement over Hard (and all of the TALENT log-loss gain), and GIST vs. calibrated-hard isn't significant under a sign test. As I said in Q1, the main story seems to be the calibrated generator and conditional supervision working together rather than the conditional supervision alone, so I'm keeping my score.
Response to Reviewer 2s9F
We thank the reviewer for the detailed and careful review and for the time and effort spent examining the method, experiments, and presentation. We appreciate the recognition of the motivation behind generator-informed supervision, the controlled experimental design, and the gradient-variance analysis.
1. W1, Q1: Calibrated prior with hard labels
We trained TabSwift with GIST's calibrated generator and sampled hard labels under the paper's full 30K + 6K schedule, with the same seeds and effective batch sizes as the paper's TabSwift Hard and GIST runs.
AUC uses binary datasets (30 on TabArena, 120 on TALENT) and log loss uses multiclass datasets (8 and 68).
Each component of GIST improves prediction, and the complete design is strongest. The calibrated generator improves all four aggregates over Hard. With that generator fixed, conditional targets further raise AUC on both benchmarks, lower TabArena log loss, and win on a majority of datasets on both benchmarks, with TALENT log loss within 0.00006 of calibrated-hard. The complete design has the best value in three of the four aggregates and the strongest paired result against Hard: 126 TALENT wins versus 112 for calibrated-hard, and 25 TabArena wins versus 21.
These results show that both the calibrated task generation and the conditional query supervision contribute to downstream performance, rather than the gains coming from either component alone. The two components are coupled by construction: on label-dependent branches, exact conditional targets exist only once the class reaches downstream features through the calibrated channel (Appendix B.2), and the channel's scale is chosen so that these targets retain residual uncertainty to average over (Appendix C). In a future version of the paper, we will include this three-arm ablation alongside fuller component analyses and clarify how both components jointly account for the empirical gains.
Given the time and compute available during the rebuttal, we have not yet trained the original-generator control with conditional targets only on terminal branches; the selective-supervision identity predicts its outcome. With support restricted to terminal branches, the removed covariance E[AΓθ(Z)] covers only those rows, and terminal targets account for about 4% of query rows in the synthetic task mixture, compared with 26% to 31% for all conditional targets (Table 8). This control therefore changes only a small part of the supervision, and we expect it to perform close to Hard.