- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors propose a new attention for tabular foundation models. The authors state that the key to long-context attention in TFMs is distinguishing between informative and uninformative samples. They then observe that QASSMax attention proposed in TabICLv2, which is based on the idea of per-query temperature scaling, tends to suppress both the uninformative samples and the lower-scoring informative samples together. Instead, they propose to learn the shift (masking) matrix, which for each query shifts the per-key attention scores. This new attention is only used in the ICL module. Empirically, compared to QASSMax and vanilla attention, the proposed attention mechanism is better on the subsample of 38 TabArena classification datasets.
Critical strengths
- Clear and novel problem formulation. The authors propose a novel formulation based on the idea of informative and uninformative samples. They define two quantities: - ratio of attention mass between informative and uninformative samples (tail-to-informative ratio); - ratio of the attention scores within informative samples (coverage). Intuitively, we want to be low, while the second ratio needs to stay within a constant range. In particular, using this formulation, the authors show that QASSMax is suboptimal as it suppresses both ratios at the same time.
- Proposed solution is intuitive. Instead of per-query scaling, the authors propose to introduce a per-query and per-key shifting (masking) term. The attention then has a clear mechanism to drive attention scores closer to 0 for uninformative samples, without affecting the spread (of attention scores) within the informative samples.
- Supported by empirical evidence. The authors re-train TabICLv2 from scratch and compare their proposed attention mechanism to the default QASSMax, SSMax, and vanilla attention. On a set of 38 classification datasets from TabArena, the inductive bias in the proposed attention proves beneficial and outperforms other attention versions. On a set of synthetic datasets, the authors also clearly demonstrate that the proposed attention achieves the desired goal: coverage stays high, while the tail-to-informative ratio is low.
Critical weaknesses
- Informative and uninformative samples. The authors build on the premise that for each test query the context (in each layer and head) can be naturally divided into informative (with min attention logit ) and uninformative (whose attention scores are ) samples such that . Is this assumption supported in practice? I would expect that per-query the distribution of per-key attention scores is smooth and does not have any jumps.
- QASSMax underperforms vanilla attention on the reported empirical set. In Table 1, QASSMax has 31% winrate against vanilla, which is below SSMax. Figure 11 in the original TabICLv2 paper, to the contrary, suggests that QASSMax is better than vanilla (albeit it reports log-loss during pre-training).
Questions for authors
See the weaknesses above. Also, I have the following questions:
- By default, does your algorithm prune rows? For example, in Figure 11, is the reported performance with or without pruning?
- In Figure 6, where is you default algorithm? I see the 3 ablated versions, but not the default (with confidence gates, with warm-up skips, and without compounding).
- Does Figure 5 suggest that there are no "uninformative" samples per se? If I am reading it correctly, my interpretation is that the first few layers don't suppress any rows (samples), i.e., the model considers all samples informative (and applies vanilla attention over them). Then, the model compresses the data into few informative tokens, and that is where the suppression starts to help. Is this interpretation correct?
3 comments
Thank you! My questions have been mostly answered. I am still a bit concerned about the QASSMax, but I suppose I need to play around with it myself.
On why QASSMax underperforms SSMax. We thank the reviewer for raising this point about QASSMax. Both SSMax and QASSMax multiply the standard attention logits by a factor of (with an additional learned scale, which is query-dependent in QASSMax). Because TabICLv2 does not use QK-normalization, the logits are themselves unbounded, so this rescaling acts on values with no fixed range.
This interpretation is consistent with findings in Kumo-Tabular [1] (Section 7), where the authors report that stable training with learned attention scales required both QK-normalization and weight decay on the scale parameters. We will add this discussion to the revised manuscript.
Reference:
[1] Qu, J., Hudovernik, V., Jurkovič, M., Lorenz, C., Holzmüller, D., Lopez, F., Gordeev, D., Nitta, A., Bendias, R., Wong, A., Jain, V., Kothapalli, V., Titericz Junior, G., Leskovec, J., & Fey, M. (2026). NVIDIA Kumo Tabular: Pushing the Accuracy$\unicode{8211}$Efficiency Frontier for Tabular Prediction. arXiv.
Response to reviewer 1muz
We thank the reviewer for the careful reading and the thoughtful questions.
W1 (Theory assumptions)
Real per-query logit profiles are indeed smooth: across all 38 TabArena datasets (3 splits), sorting each query's logits and looking for an unusually large gap among the top 64 finds one in only 2.7% of (layer, head, query) cases, and the largest gap is typically just ≈0.04 logit standard deviations.
The two-level setup (R at depth δ, T at depth Δ) is a deliberate simplification: it gives closed forms for ρR and MT/MR, and intuition for why a key-independent temperature trades coverage for suppression. We still do not claim that heads work always like this.
Our results do not need a gap: for any m, let R be the top-m rows. Then g=minRz−maxTz≥0 by construction, and the bounds hold with that g. In practice g≈0 (no jump), so the tail is suppressed by the learned F and not by a logit gap, which is what a scalar temperature cannot do.
On real data, R is not accessible, so we take either the top-m rows by logit or the m nearest neighbours in feature space (with m=32 or m=2% of n). T is penalised more than R in 85–91% of queries (with mean suppression score of 0.46 on T vs. 0.001–0.09 on R). "Informative" is therefore graded: a random nearest neighbour outscores a random other row 82% of the time (median AUC), but the two sets always overlap.
W2 (QASSMax baseline)
Our QASSMax baseline is a from-scratch reproduction of TabICLv2. All variants are pre-trained under TabICLv2's public prior with the official code, identical optimisation settings and curriculum, and FA-3 enabled in Stages 2–3 as in TabICLv2 (App. C.3–C.4), which is consistent with what was reported in the paper.
We observe that both SSMax and QASSMax degrade at later stages, where N grows to 60K. We hypothesise this is due to the logn term, which sharpens the logits beyond the range seen in earlier stages and becomes unstable at longer contexts. This is consistent with their lower win rates against vanilla (31.9% and 37.9%).
Q1 (Pruning not used as default)
No: by default, TSA keeps every training row in context, and all reported results are without row dropping.
Q2 (Fig 6)
Thank you for flagging this: Figure 6 will now include the default schedule (all components) as well.
Q3 (Fig 5)
We mostly agree, with a few caveats:
At initialisation, roughly half of all pairs are suppressed in every layer (W is zero-mean Gaussian). After training, the early-layer scores mostly fall just below zero, so F=0 and TSA reduces to vanilla attention in layers 0–3. The model therefore learns to switch suppression off early.
(i) We agree there is no globally uninformative row: informativeness is defined per head and test row. However, early layers do not treat all rows alike: on TabArena, their logits already rank a query's 32 nearest feature-space neighbours above the other rows (as mentioned in W1).
(ii) The context is not compressed into fewer tokens: every training row keeps its own token in every layer, and at most about 2% of rows are flagged per layer.
Please let us know if you have any remaining questions.