- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks if tabular foundation models use context labels only as values combined with fixed weights, like in linear regression or kernel smoothing. Or does one label change how much the others matter? The authors point out a trap. The packages standardize labels first, so plain second derivatives can look nonlinear even when every real prediction is fixed-weight. They propose two tests that only use predictions at standardized labels. One rules out fixed weights, the other also rules out per-label transforms. Across five public models and four regression datasets, both tests come out nonzero. The effect is much weaker in random copies, and most of the measured interaction goes through the attention scores.
Critical strengths
- The paper identifies a real and easily overlooked pitfall: because TFM packages standardize labels onto a sphere , naive Hessian and permutation tests report label adaptation even for maps that apply fixed weights on every admissible input.
- Claims are carefully scoped and backed by meaningful controls. Random copies and an entropy-matched control tie the interaction to training, and the authors state explicitly that the stop-gradient identifies derivative paths rather than necessity, and that the default ensemble is only shown to be non-affine.
Critical weaknesses
- The evidence is local and its practical size is unclear. The certificates are pointwise second-order quantities at the observed labels, and is a share of curvature, not of prediction change.
- The scope is narrow relative to the framing. Only regression is evaluated, the certificates apply to a single internal estimator rather than the deployed ensemble, and several analyses omit TabFM.
- Since the related work already notes that label-driven task inference changes every label’s weight and that the measurements agree with this picture, joint processing is arguably expected, and the novelty rests mainly on the methodology.
Questions for authors
- Can , or a four-point test calibrated against a separable null, be applied to the packages’ default ensemble predictions?
- How were the four queries per context selected, and do the certificates stay stable at label vectors other than the observed ones and with more contexts?
4 comments
[2/2]
Weakness 3: Since the related work already notes that label-driven task inference changes every label’s weight and that the measurements agree with this picture, joint processing is arguably expected, and the novelty rests mainly on the methodology.
Response: We agree that joint processing is expected and that the novelty lies in the methodology. Our contributions here are the two tests: the first detects non-affinity, which is a departure from fixed weights, and the second detects joint label processing. We identified the standardization trap, in which the tests in use report label adaptation even for a model with fixed weights, and we fixed it. We believe that a rigorous and solid test is a scientific contribution, even when its findings are expected, because an expectation cannot be verified without such a test.
The measurements also add three things that an expectation does not give.
- The amount of joint processing differs between models: Copies of TabICL-v2 and TabDPT with random weights, built with the reset procedure of Appendix C.3, have a joint share below 1% on average, against 40% to 71% for the trained models. Among the trained models, the ratio differs more than threefold (Table 17).
- The certificate measures how stable an explanation by examples is: The certificate predicts by how much the influence of the examples changes when other labels change. An explanation of a prediction by its most influential examples therefore holds only near the observed labels (Appendix C.10).
- Label-dependent weights can decide the accuracy: We added Appendix C.11 to show an example on synthetic tasks. When a table has many variables and only the labels reveal the few that matter, the five models are far more accurate than the best predictor with fixed weights. A predictor that transforms the labels one at a time does not reach the models either. We keep this example in the appendix because we want the paper to stay centered on the theory.
Question 1: Can , or a four-point test calibrated against a separable null, be applied to the packages’ default ensemble predictions?
Response: Yes. We ran the four-point test that you describe on the default predictions of the five packages, and we find the interaction in all of them.
In this test, we change the labels of two disjoint groups of rows, separately and together. For every model that treats labels one at a time, the two effects add up, so the contrast between the four predictions is zero. We use separable maps that we construct as the null, and a map with interacting labels as a positive control (Appendix C.7).
The contrast is nonzero and above the numerical noise in 192, 119, 144, 192 and 59 of 192 comparisons for TabICL-v2, TabPFN-3, TabPFN-2.6, TabFM and TabDPT (Table 14). We could not resolve the remaining comparisons numerically, and they do not show an absence of interaction.
These contrasts describe the complete package, with its label preprocessing and its averaging over estimators.
Question 2: How were the four queries per context selected, and do the certificates stay stable at label vectors other than the observed ones and with more contexts?
Response: We draw the queries at random. The detection is stable at other label vectors and with more contexts, and the magnitude changes with the label vector.
- Queries: We take a seeded random permutation of the rows, which gives the context rows and then the query rows. We select no query with a certificate value (Appendix B.2).
- Other label vectors: We recompute the certificates at two other label vectors, on two datasets: the observed labels in shuffled order, and labels in which only the residuals of a regression on the features are shuffled. We detect the interaction at 12 of 12 predictions for every model in both cases. The joint share moves by less than 9 points in median for four models. It is less stable for TabDPT, where it falls by 12 points with shuffled residuals and by 40 points with shuffled labels (Table 13).
- More contexts: We extended the experiment of Table 2 from three to six contexts per dataset and from four to eight datasets, and the conclusions stay the same (Tables 2 and 9). We also computed the certificates on 29 tables with one new context each (Table 17). With longer contexts of 1,024 rows, the joint share is lower than at 256 rows in 15 of 16 settings, by 0.085 on average, and it stays between 0.30 and 0.82 (Appendix C.7).
Thank you for your detailed responses and experiments. I appreciate the authors' efforts to address my concerns, particularly regarding the practical significance and experimental scope. However, I remain concerned about the methodological novelty of the proposed tests and the significance of the findings. While the additional experiments strengthen the paper, my main concerns have not been fully resolved. Therefore, I will maintain my original rating
Thank you for reviewing the additional experiments and acknowledging that they strengthen the paper.
We would appreciate clarification of your remaining concerns about methodological novelty and significance. In your original review, you described the standardization pitfall as "real and easily overlooked". Our methodological contribution addresses this pitfall through the two certificates in Lemmas 1–2.
Could you clarify which aspect of this contribution you consider insufficiently novel? If the concern is overlap with existing work, we would appreciate the relevant references so that we can explain the relationship precisely.
Regarding significance, the added analyses connect the certificates to changes in example sensitivities and examine predictive consequences. Could you clarify which limitations of these results remain decisive in your assessment?
Thank you again for your feedback.
[1/2] Thank you for the review. Its first weakness led us to the main addition of the revision.
Response: We agree with both points. In the revision, we give the certificates a practical size and we test it at finite label changes (Appendix C.10).
The certificates are second derivatives of the prediction, so they say how fast the sensitivities change with the labels. The sensitivity to a label is the weight of that example in the prediction. We make this reading exact in Proposition 1: the ratio Xsep/∥PTa∥ gives the relative change of the sensitivities of a group of rows when the labels of the other rows change.
We tested this prediction on 29 tables with finite label changes. When the labels of half of the rows move by 3% of a standard deviation, the measured change of the sensitivities of the other half is 0.94 to 1.08 times the predicted one on average, with no fitted constant (Table 18). At 10% the first-order prediction overestimates the change, with a ratio of 0.64 to 0.94, and the change remains large: 29% to 65%. The certificates therefore give the size of the change for label changes of a few percent, and its order of magnitude at 10%.
In a second test, we change the labels of two random groups of rows, separately and together. For a model that treats labels one at a time, the two effects add up (Section 4.4 and Appendix C.7). We measured the departure from this sum. It certifies that every such model misses the predictions of the tested model by at least 0.9% to 2.2% of its root mean squared error in the typical case, for label changes as large as that error. With the forward intervention we measure accuracy directly: the error is 2.1 times higher in median when the attention weights no longer follow the labels (Table 16). In this intervention we act on the attention weights and not on the certificate itself.
Response: We agree with the three points. We now state the scope for the first, and we added experiments for the other two.