acceptodds
Withdrawn from ICLR 2027

Accelerating Tabular Foundation Models via Residual-Aware Context Compression

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Tabular foundation models (TFMs) enable in-context learning over tabular data by conditioning on a labeled training table without task-specific backbone updates. However, their inference cost grows with context size because each query attends to the training rows, leading to substantial latency and memory overhead on large datasets. In this paper, we study context compression for TFM inference and identify three fundamental issues when reducing the context: (1) insufficient preservation of predictive signals for the frozen TFM, (2) the overhead of incorporating supervision from omitted rows, and (3) the lack of a unified compression framework across frozen TFMs and downstream tasks. To address these issues, we propose Residual-Aware Context Compression (RCC), which compiles a large training table into a compact reusable artifact through a one-time offline process. RCC first selects an initial set of context rows, then uses prediction residuals under this context to identify additional rows with complementary predictive information. Afterwards, it fits a closed-form residual readout using the remaining rows to correct predictions under the compact context. Together, RCC preserves key information from the full context and generalizes across different TFMs and downstream tasks. We evaluate RCC on three frozen TFMs and 100 classification and 29 regression datasets. At compression, RCC retains 96.5%-97.5% of full-context classification macro-F1, with regression nRMSE increasing by 8.3-8.6 points, while achieving median single-query speedups of -. On 32 large-scale datasets with up to one million rows, our lightweight variant, RCC-lite, maintains comparable predictive performance at compression ratio, achieving single-query speedups of up to over the full-context baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
tabular foundation models, in-context learning, context compression, efficient inference, residual learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MKUiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a query-agnostic method for context compression to scale tabular foundation models (TFMs) to large datasets. Specifically, the authors propose a two-step procedure: 1) select representative context points and 2) train a small model (ridge regression on the left-out context samples) to correct the internal representations. At inference time, only the selected context points and the small model are required to label query points.

The main contribution is the two-step procedure, which is evaluated on three TFMs across regression and classification datasets.

Critical strengths

The method to compress context points is principled, and the residual readout appears novel. Overall, the empirical results show that the proposed method maintains predictive power under context compression.

Critical weaknesses
  1. The paper's presentation needs improvement / is hard for humans to follow. For example, Section 3 presents the technical details but doesn't intuitively explain why the method was designed that way or which components matter. Similarly, the experimental section just summarizes the experiments and tables without connecting the results to research questions or findings.

  2. The experiments are insufficient and lack details (see question below). Furthermore, variances are not reported.

  3. The method runs on the entire training set and makes several decisions on the training data. The risk of overfitting is not discussed. (see question below)

Questions for authors

Q1: Does your method use the entire training set (and if yes, why does it not overfit)?

[Longer version: The method finds an optimal context compression such that the prediction labels of the training set in eq(2) remain the same. Why not the predictions on a held-out validation set? Second, for training the ridge regression, the method uses the left-out data points, which may follow a different distribution than the full dataset. Is there something I overlook?]

Q2: Clarification on the experiments. a) Talent has 300 datasets; how did you select 100 datasets? b) Figure 2: What is RGCS loss? I don't see it in the Figure. c) Figure 3: TabICL seems to behave quite differently from the other models (and has almost no variance for large context). Why is that? d) Are the selected context points similar across models and across compression ratios (e.g., is ratio=32 a subset of ratio=64)

Q3: Are there any assumptions about the dataset distribution? From the method, I assume that the initial set of points is quite important, but I don't see what is needed for the context selection to work well (or not)

3 comments

Thank you for the detailed review. Shared points are in the general response (GR1–GR4); here we focus on overfitting, your main soundness concern.

A clarification. CRR does not modify internal representations. The backbone stays frozen; CRR is a ridge regression from the frozen query representation to a correction of the backbone's output (Sec. 3.3).

W3 / Q1 (whole training set; overfitting). Yes, RCC compiles from all labeled rows available at deployment, the same rows Full uses as context. Three things guard against overfitting.

  1. Protocol (App. B.1). Compilation uses the training and validation partitions; every reported number is measured on test rows that no compilation step touches.
  2. The readout is fit on rows outside the context. For each i∈Ifit=[n]∖I^i\in I_{\mathrm{fit}}=[n]\setminus\hat I, the inputs (Eq. 7) are computed with row ii outside the context, exactly like a test query. These rows differ from the full compilation table only by the context rows, so the total-variation distance between their empirical distributions is about 1/r1/r at compression ratio rr (6.25% at 16×, 0.78% at 128×).
  3. Eq. (2) is a target, not a constraint. tit_i defines the residuals that guide selection (Eq. 4) and the readout's regression target (Eq. 8); RCC never searches for a context that reproduces training predictions. Selection is a single pass, the only fitted parameters are closed-form ridge weights, and the constants (ridge c=10−2c=10^{-2}, RFF dimension, budget split) are global, with cc chosen once on validation rows. The in-sample term f[n](xi)f_{[n]}(x_i) only smooths the label and carries no test information; RCC-lite (ti=yit_i=y_i) drops it, and a cross-fitted version is a valid alternative.

Test-set evidence. With the context fixed, CRR improves the native test prediction on almost every dataset: macro-F1 rises by +4.00, +3.79 and +3.79 at 16× and by +8.10, +7.75 and +9.97 at 128× (all CIs exclude zero; better on 92–100 of 100 datasets), and test nRMSE drops on 27–29 of 29 datasets. A readout that overfit its fitting rows would not transfer like this. Conversely, selectors that chase the hardest training rows fail: filling the context with the rows the full-context prediction is least confident on (classification) or misses most (regression) gives 29.2–29.7 macro-F1 at 128×, versus 60.5–63.4 for a random context, and 200.7–281.6 nRMSE, versus 65.1–65.7. These selectors were run with the submission's experiments but not reported; we will add them to the appendix.

W2 (variance). See GR3: we add dataset-bootstrap CIs and wins/ties/losses for all main contrasts; the main experiments use one seed per task, which we state as a limitation; timing noise is about 2%.

Q2a (dataset selection). By the rule in App. B.1: TALENT classification tasks with 2,000–50,000 official training rows and at most ten classes, excluding Volkert (out of memory). Of the 161 tasks with at most 50,000 training rows and ten classes, removing Volkert and those with fewer than 2,000 training rows leaves 100. Regression uses the PFN-REG, TabArena-REG and CTR23 lists distributed with LimiX (29 of 51 have at least 2,000 training rows). These rules move to Sec. 4.1.

Q2b ("RGCS loss" in Fig. 2). The hatched area marks where native RGCS is worse than native KMeans. This happens in 1 of 42 settings (TabICLv2 regression at 32×: nRMSE 60.9 vs. 57.1), so it is barely visible; we will mark it explicitly.

Q2c (TabICL in Fig. 3). Each point is one dataset, so the spread is across datasets, not run-to-run noise. (i) TabICLv2's lower speedups were mostly caused by the timing inefficiency in GR2; with the fix, its median single-query speedup rises from 2.6× to 11.0× (classification). (ii) The spread is small because RCC's per-query time is nearly constant, so speedups follow Full's latency, which for TabICLv2 depends almost only on the number of rows nn: on tables with at least 10k rows, a fit of log10 speedup on log10 nn leaves a residual standard deviation of 0.08 for TabICLv2, against 0.24 and 0.22 for TabPFNv2 and LimiX-2M. This matches TabICLv2 embedding each row before in-context learning over rows (Qu et al., 2026), whereas TabPFNv2 attends over both rows and columns (Hollmann et al., 2025).

Q2d (similarity of selected rows). Coverage rows are identical across backbones by design, since they use only features and labels; residual nominations overlap by a median 48–55% at 128× and 64–73% at 16×, about 126–140× and 21–24× the overlap expected by chance. Contexts are not nested across ratios, because coverage is recomputed for each budget: only 22–23% (classification) and 14–15% (regression) of the 64× rows lie in the 32× context.

Q3 (assumptions; the initial set). There is no parametric assumption. RCC relies on (a) exchangeable compilation and test rows (we claim no robustness to distribution shift), (b) locally structured errors of the compact context, which Fig. 4 tests directly, and (c) a remaining error that is predictable from the frozen representation, as the test-set CRR gains show. The initial set matters for native prediction: at 128×, giving the whole budget to RGCS's label-aware coverage step (a control from the same runs, not in the paper) reaches 67.7 / 66.7 / 69.2 macro-F1, versus 61.6 / 60.5 / 63.4 for a random context. After CRR, the context matters much less (Random+CRR 76.5 / 74.8 / 78.4 vs. RCC 77.2 / 75.5 / 78.8). Selection helps less in regression at high compression, where native RGCS is within about 2 nRMSE points of random at 128× (CIs spanning zero), and we expect smaller benefits under heavy label noise or for rare modes that class anchors miss. These conditions go into Sec. 3 and the limitations.

We hope this resolves your concerns.

Thank you for your effort.

  • Q2 is resolved.
  • Q1 remains open. I do not fully understand how many decisions are made on the training (context) data, or whether this introduces bias (which would not be a problem, but the paper should be transparent about it).
  • Q3 is partially resolved (optimal and suboptimal conditions for the method to perform well remain open)

However, my biggest concern about the readability of a revised version remains open; thus, I stand by my original score.

Reviewer_2ZMROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces Residual-Aware Context Compression (RCC) a method to speed up Tabular foundation models (TFMs) by replacing the full training context by only a subset of it. This context subset is buit once offline, where initial rows are chosen via k-means on feature and labels, while the other half are chosen later using the prediction residuals under the initial subset selected, to get complementary information. Then a closed-form is fitted on the remaining rows to correct predictions. At inference time, only the small context and the linear mapping are used. The evaluation cover 3 TFM and both classification and regression datasets using TALENT and OpenML.

Critical strengths
  1. The method is simple: no gradient step, one-time fit and works with different architectures.
  2. Results show that for the classification task at 16x compression, RCC is within 0.2 macro-F1 points with respect to the full context model for TabPFN and LimiX. Also, in the same setting (at 128x) RCC outperform the TFM competitor (TACO).
Critical weaknesses
  1. I found the paper difficult to read and follow. The experiment section mostly reads out table numbers without takeaways or intuition, and two figures of the main text seems AI-generated and not refined.
  2. It is unclear how much the main novelty (the residual guided row selection) contributes to the final score. In table 4, with the readout both "Random+CRR" and "KMeans+CRR" perform very similar to "RGCS+CRR", especially in classification. While in Table 13, one of them is even better in regression when using LimiX and TabICL.
  3. It is not clear how much is actually saved:
  • Costs are only reported at 128x, where quality drops. At 16x (where classification quality is close to the full context) the savings are not reported.
  • KV caching is disabled for all methods. Since TabPFN and TabICL support KV caching, the savings should also be measured with caching enabled.
  • From my understanding, the offline fit must be repeated for each dataset, backbone and budget, and standard RCC needs full-context predictions on all training rows.
  1. Results are good only in specific settings and only 3 models are tested, not larger backbones:
  • Regression results are not as good as the classification ones.
  • Only three backbones are tested, and bigger models such as LimiX-16M or TabDPT are not tested. Also authors should expand related work section regarding TFM, that is very limited.
Questions for authors
  1. How does RCC behave with larger or different backbones (e.g., LimiX-16M, TabDPT)?
  2. What is the variance across seeds and splits? On the large-scale benchmark, why are published full-context scores used instead of local runs, so that both of them come from the same pipeline?
  3. Table 5 gives median compile times (offline step done once per dataset), while in appendix is reported the asymptotic cost. How do measured compile times scale with the table size, given that standard RCC needs full-context predictions on all training rows?
  4. How does "RGCS w/o residual" + CRR perform? While "Random+CRR" and "KMeans+CRR" are only reported at 128×, how do they compare with RCC at other compression ratios (e.g., 16x), and on how many datasets RCC perform better?
  5. Some of the figures seems AI-generated and not fixed (e.g., Figure 1, Figure 4, Figure 7), in particular Figure 1 becomes blurry when zoomed in, and there are some hallucinations in the sub plots (Diagnose and Select). Can the authors fix them?

Minor:

  • In figure 4, colours are not explained.
  • Remove abbreviations such as "w/o" in the text.
  • Which TFM representation is used for the readout is only specified in the appendix (Tab. 6).
2 comments

Thank you for the detailed review. Shared points are in the general response (GR1–GR4).

W2, Q4 (contribution of RGCS). You are right: once CRR is applied, selection adds little, and we will present the contributions this way (GR1). At 128×, RCC beats Random+CRR by +0.77, +0.79 and +0.43 macro-F1 (better on 65, 50 and 60 of 100 datasets) and KMeans+CRR by +1.00, +0.63 and +0.55 (61, 53 and 57). In regression, RCC's nRMSE reduction is +2.49, +1.75 and −0.32 points relative to Random+CRR and +1.90, −0.56 and +1.67 relative to KMeans+CRR (better on 10–18 of 29 datasets; five of these six CIs include zero). Without CRR, selection matters more: at 16×, native RGCS beats KMeans by +2.58, +2.59 and +2.09 macro-F1 and Random by +2.75, +2.82 and +1.94 (all CIs exclude zero). The submission did not run "RGCS w/o residual" + CRR or the +CRR baselines at other ratios, so we cannot report them.

W3, Q3 (actual savings).

  • KV caching. We timed Full with the official KV caches of TabPFNv2 and TabICLv2 (Full-KV; LimiX has none), in the same run as RCC with the GR2 fix and excluding cache construction. RCC's median speedup over Full-KV at 128× is 5.1× (single query) and 4.6× (batch of 128) for TabPFNv2 in classification, and 12.3× and 11.6× in regression. For TabICLv2 it is about 1.75× for single queries, but RCC is slower for batches of 128 (0.83× in classification, 0.73× in regression). Peak GPU memory stays 7–11× lower (TabICLv2 classification: 1,757 MB vs. 205 MB). We will add these rows to Table 5.
  • 16×. We did not time 16× queries. Compilation at 16× takes about as long as at 128× (medians over all 129 datasets: 38 vs. 38, 84 vs. 84 and 26 vs. 24 s), because generating the full-context targets of Eq. (2) dominates it.
  • Recompilation. An artifact is specific to a dataset, backbone and budget, but the full-context targets are computed once per dataset and backbone and shared across budgets, so each extra budget costs a median of 7–16 s. Target generation takes a median 59–88% of compilation time; RCC-lite removes it. Against uncached Full, RCC breaks even after a median of 36 / 120 / 70 single-query calls in classification and 42 / 124 / 65 in regression (Table 14, before the GR2 fix).
  • Scaling. Compilation time grows with the number of rows nn with a log–log slope of 1.3–1.9, driven by target generation (0.6–0.9 without it). In classification, the median rises from 7–15 s below 5k rows to 200–1,500 s for 20k–56k rows, consistent with App. D.

W4, Q1 (other backbones; regression). We report no results on LimiX-16M or TabDPT, for these reasons.

  • LimiX-16M failed to complete the full-context run on one dataset (QSAR-TID-11) on our 24 GB GPU during validation, before any test run. To keep one configuration for all datasets, we used the official LimiX-2M.
  • TabDPT is pretrained on real tables, and 35 of our 129 datasets (6 of the 32 large tables) appear in its pretraining corpus, so its results would confound compression with pretraining exposure.
  • We will state this scope in the limitations and expand the related work on TFMs.
  • Regression. Agreed. RCC trails Full by 1.6–4.3 nRMSE points at 16× and 8.3–8.6 at 128×, against 0.1–1.2 and 2.0–2.7 macro-F1 points in classification. CRR still lowers regression error on 28 of 29 datasets at 128× (GR1). We will discuss this gap in the limitations.

Q2 (variance; published Full scores).

  • Each dataset uses one split (TALENT's official partition; for regression, a fixed 80/20 split with seed 42) and one method seed, so the CIs in GR3 cover variability across datasets only; we state this as a limitation.
  • On BeyondArena, we computed Full's scores from the official TabICLv2 test predictions released with the benchmark, on the same split, test rows, labels and metric code as RCC-lite. We did not run Full locally: it exceeded memory on 11 of the 32 tables, and on the rest full test-set inference would take many hours per table. Preprocessing differs between the two pipelines, which we will state explicitly.

W1, Q5, minor (presentation). We agree (GR4). Fig. 1 will be redrawn as a vector figure whose Diagnose and Select panels match Sec. 3; Figs. 4 and 7 will be regenerated with explained colors and readable labels; "w/o" will be spelled out; the readout representation will be stated in Sec. 3.3; and Sec. 4 will state the takeaway of each experiment.

We hope these answers address your concerns.

Reviewer_siEkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on context compression in tabular foundation models. Specifically, it identifies three issues in context compression relating to preserving predictive information for the frozen TFM, supervision overheads for omitted rows, and misalignment with downstream tasks. The core contribution is a method Residual-Aware Context Compression (an offline compilation of a training table), which uses prediction residuals to correct predictions under the context compaction. Experiments are evaluated on three TFMs (TabPFNv2, LimiX-2M, and TabICLv2) on classification and regression datasets with up to 1 million rows, demonstrating large speedups over a full-context baseline and improved performance relative to alternative context compaction baselines.

Critical strengths

The major strength of the paper is its experimental results, which demonstrate late performance improvements at high compaction rates. The speed improvement of the RCC-lite variant between 119x and 1,514x on BeyondArena datasets is likely to be of interest in the TFM community, where improving the efficiency of TFMs on large datasets is an area of key concern. The experimentation on a wide array of appropriate tabular classification and regression tasks across multiple frozen TFM backbones provides confidence in the effectiveness of the approach.

Writing is overall clearly described, with related work appropriate and contemporary. The residual guided method is well motivated and likely to be of interest in the community. While some details in the methodology could be explained more clearly (e.g. including additional citation to motivate the approach, and less reliance on the Appendix), the method sections are broadly clear to read.

Critical weaknesses

The main weaknesses are clarity of some sections and reliance on the Appendix. Much of the appendix contains non-trivial development of theory (e.g. Proposition 1), which is presented without citation. This presents an issue for validating and comparison to prior literature (further citation, e.g. with standard textbooks would help address this issue).

Some of the writing contains unusual phrasing which can reduce clarity (e.g. L231 - "The selected context provides direct evidence to the backbone", is somewhat unclear). There are a few key terms that are not defined in the main paper (L215 "residual severity", which is undefined in the main paper except through the RankNorm (itself defined in the Appendix L1188, where it is not clear what the # symbol refers to).

Some details in the Appendix are difficult to read, with many extraneous details (e.g. specific seed numbers "We use seeds 0, 1, and 2" L932) and magic numbers (e.g. L1249 L=min{m(m-1)/2, 200000}), which reduces readability. Focusing on the key components, and simplifying the writing to the core essentials would improve clarity. The number of technical details included in the appendix makes it difficult to validate if the performance improvements arise from the proposed residual method or technical experimental modifications.

Certain phrases are difficult to parse (e.g. Figure 4 "Current-context alignment, Alternative-context alignment, Same-label control, Candidate severity, nearest in feature space) which are not defined in the main paper, and difficult to read in the appendix (e.g. the definitions for same-label control "Using the query's true label tests whether label agreement alone explains gains", current-context alignment "This tests our hypothesis that similar residual directions identify useful candidates", Alternative-context alignment "Replacing only the candidate's residual context tests whether its alignment with the query depends on using the same context I"). This figure is not discussed in the main paper, and it would be useful to understand the purpose of the ablation.

Questions for authors

It would be useful to provide context to the Figure 4 metrics and the purpose of the ablation.

It would be useful to understand which components in the Appendix the authors could provide additional motivation / context for in the main paper (e.g. severity weighted centroids).

2 comments

Thank you for the positive assessment and the concrete suggestions. Shared points are in the general response (GR1–GR4).

W1 (citations for the appendix theory). We will cite a standard source for each proof step: convexity of log-sum-exp (Boyd & Vandenberghe, 2004, Sec. 3.1.5), differentiation under the expectation by dominated convergence (Durrett, 2019, Thm. A.5.1), first-order optimality for convex functions (Boyd & Vandenberghe, 2004, Secs. 3.1.3 and 4.2.3) and the weighted ridge solution (Hoerl & Kennard, 1970). Proposition 1 is a population-level existence result, not a finite-sample guarantee; GR1 gives the test-set evidence.

W2, Q2 (undefined terms, motivation).

  • Residual severity uiu_i is the normalized rank of a row's residual norm aia_i under the coverage context: u_i=\big(\\#\\{j:a_j < a_i\\}+\tfrac12\\#\\{j:a_j=a_i\\}\big)/n, where \\# counts set elements. Ranks make severities comparable across datasets and keep a few extreme residuals from dominating. The definition will move next to Eq. (4).
  • Severity-weighted centroid. Eq. (6) picks the unselected row nearest the cell's uiu_i-weighted centroid (App. A.2). Because the selection representation joins position and residual direction, this row represents the cell's typical error rather than its single worst, possibly noisy, row.
  • L231 becomes: "The backbone sees the labels of the selected rows directly, as context. To also use the labels of the omitted rows, we fit a residual readout on them."
  • The coverage representation, class anchors, budget split and readout features will also be motivated in Sec. 3.

W3 (appendix constants; source of the gains). All implementation constants (seeds, numerical floors, the RFF bandwidth) move into one reproducibility table with a reason each; they are global and never tuned per dataset. LL in App. D is the number of row pairs used to estimate the bandwidth. Matched ablations attribute the gains to the residual components (GR1): at 128×, CRR adds +7.8 to +10.0 macro-F1 over the same context's native prediction, and removing only the residual signal from selection costs 0.9 to 1.2.

W4, Q1 (Fig. 4). Fig. 4 tests the premise of residual-guided nomination: a candidate whose residual direction matches a query's should help that query. On 37 multiclass datasets, we insert each candidate alone into a fixed 64-row context and measure the change in the query's true-label log-probability, without a readout. Six scores rank the candidates; a score's top-5% lift is the mean gain of its top 5% candidates minus that of all candidates. Current-context alignment (our hypothesis) has the largest lift on all three backbones (0.052 / 0.055 / 0.046 nats), ahead of same-label control (candidate shares the query's label; 0.030 / 0.035 / 0.024), candidate severity, nearest in feature space and random. It also beats alternative-context alignment (candidate residual under another context), even after adjusting for labels, distance and query and candidate effects. Sec. 4.4 will add this explanation with self-explanatory names.

As noted in GR2, the large-table speedups you highlight are conservative.

I thank the authors for their response. I recommend the authors to submit an updated version (marking the edits with red text will make it easier for the reviewers to evaluate the changes).

I will note that my core concern is on clarity, which has been noted by all reviewers. I encourage the authors to focus specifically on improving the clarity of writing if they submit an updated version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.