- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a query-agnostic method for context compression to scale tabular foundation models (TFMs) to large datasets. Specifically, the authors propose a two-step procedure: 1) select representative context points and 2) train a small model (ridge regression on the left-out context samples) to correct the internal representations. At inference time, only the selected context points and the small model are required to label query points.
The main contribution is the two-step procedure, which is evaluated on three TFMs across regression and classification datasets.
Critical strengths
The method to compress context points is principled, and the residual readout appears novel. Overall, the empirical results show that the proposed method maintains predictive power under context compression.
Critical weaknesses
-
The paper's presentation needs improvement / is hard for humans to follow. For example, Section 3 presents the technical details but doesn't intuitively explain why the method was designed that way or which components matter. Similarly, the experimental section just summarizes the experiments and tables without connecting the results to research questions or findings.
-
The experiments are insufficient and lack details (see question below). Furthermore, variances are not reported.
-
The method runs on the entire training set and makes several decisions on the training data. The risk of overfitting is not discussed. (see question below)
Questions for authors
Q1: Does your method use the entire training set (and if yes, why does it not overfit)?
[Longer version: The method finds an optimal context compression such that the prediction labels of the training set in eq(2) remain the same. Why not the predictions on a held-out validation set? Second, for training the ridge regression, the method uses the left-out data points, which may follow a different distribution than the full dataset. Is there something I overlook?]
Q2: Clarification on the experiments. a) Talent has 300 datasets; how did you select 100 datasets? b) Figure 2: What is RGCS loss? I don't see it in the Figure. c) Figure 3: TabICL seems to behave quite differently from the other models (and has almost no variance for large context). Why is that? d) Are the selected context points similar across models and across compression ratios (e.g., is ratio=32 a subset of ratio=64)
Q3: Are there any assumptions about the dataset distribution? From the method, I assume that the initial set of points is quite important, but I don't see what is needed for the context selection to work well (or not)
3 comments
W2 (variance). See GR3: we add dataset-bootstrap CIs and wins/ties/losses for all main contrasts; the main experiments use one seed per task, which we state as a limitation; timing noise is about 2%.
Q2a (dataset selection). By the rule in App. B.1: TALENT classification tasks with 2,000–50,000 official training rows and at most ten classes, excluding Volkert (out of memory). Of the 161 tasks with at most 50,000 training rows and ten classes, removing Volkert and those with fewer than 2,000 training rows leaves 100. Regression uses the PFN-REG, TabArena-REG and CTR23 lists distributed with LimiX (29 of 51 have at least 2,000 training rows). These rules move to Sec. 4.1.
Q2b ("RGCS loss" in Fig. 2). The hatched area marks where native RGCS is worse than native KMeans. This happens in 1 of 42 settings (TabICLv2 regression at 32×: nRMSE 60.9 vs. 57.1), so it is barely visible; we will mark it explicitly.
Q2c (TabICL in Fig. 3). Each point is one dataset, so the spread is across datasets, not run-to-run noise. (i) TabICLv2's lower speedups were mostly caused by the timing inefficiency in GR2; with the fix, its median single-query speedup rises from 2.6× to 11.0× (classification). (ii) The spread is small because RCC's per-query time is nearly constant, so speedups follow Full's latency, which for TabICLv2 depends almost only on the number of rows : on tables with at least 10k rows, a fit of log10 speedup on log10 leaves a residual standard deviation of 0.08 for TabICLv2, against 0.24 and 0.22 for TabPFNv2 and LimiX-2M. This matches TabICLv2 embedding each row before in-context learning over rows (Qu et al., 2026), whereas TabPFNv2 attends over both rows and columns (Hollmann et al., 2025).
Q2d (similarity of selected rows). Coverage rows are identical across backbones by design, since they use only features and labels; residual nominations overlap by a median 48–55% at 128× and 64–73% at 16×, about 126–140× and 21–24× the overlap expected by chance. Contexts are not nested across ratios, because coverage is recomputed for each budget: only 22–23% (classification) and 14–15% (regression) of the 64× rows lie in the 32× context.
Q3 (assumptions; the initial set). There is no parametric assumption. RCC relies on (a) exchangeable compilation and test rows (we claim no robustness to distribution shift), (b) locally structured errors of the compact context, which Fig. 4 tests directly, and (c) a remaining error that is predictable from the frozen representation, as the test-set CRR gains show. The initial set matters for native prediction: at 128×, giving the whole budget to RGCS's label-aware coverage step (a control from the same runs, not in the paper) reaches 67.7 / 66.7 / 69.2 macro-F1, versus 61.6 / 60.5 / 63.4 for a random context. After CRR, the context matters much less (Random+CRR 76.5 / 74.8 / 78.4 vs. RCC 77.2 / 75.5 / 78.8). Selection helps less in regression at high compression, where native RGCS is within about 2 nRMSE points of random at 128× (CIs spanning zero), and we expect smaller benefits under heavy label noise or for rare modes that class anchors miss. These conditions go into Sec. 3 and the limitations.
We hope this resolves your concerns.
Thank you for your effort.
- Q2 is resolved.
- Q1 remains open. I do not fully understand how many decisions are made on the training (context) data, or whether this introduces bias (which would not be a problem, but the paper should be transparent about it).
- Q3 is partially resolved (optimal and suboptimal conditions for the method to perform well remain open)
However, my biggest concern about the readability of a revised version remains open; thus, I stand by my original score.
Thank you for the detailed review. Shared points are in the general response (GR1–GR4); here we focus on overfitting, your main soundness concern.
A clarification. CRR does not modify internal representations. The backbone stays frozen; CRR is a ridge regression from the frozen query representation to a correction of the backbone's output (Sec. 3.3).
W3 / Q1 (whole training set; overfitting). Yes, RCC compiles from all labeled rows available at deployment, the same rows Full uses as context. Three things guard against overfitting.
Test-set evidence. With the context fixed, CRR improves the native test prediction on almost every dataset: macro-F1 rises by +4.00, +3.79 and +3.79 at 16× and by +8.10, +7.75 and +9.97 at 128× (all CIs exclude zero; better on 92–100 of 100 datasets), and test nRMSE drops on 27–29 of 29 datasets. A readout that overfit its fitting rows would not transfer like this. Conversely, selectors that chase the hardest training rows fail: filling the context with the rows the full-context prediction is least confident on (classification) or misses most (regression) gives 29.2–29.7 macro-F1 at 128×, versus 60.5–63.4 for a random context, and 200.7–281.6 nRMSE, versus 65.1–65.7. These selectors were run with the submission's experiments but not reported; we will add them to the appendix.