acceptodds
Under review as a conference paper at ICLR 2027

Support-Compiled Feature Folding: More Evidence at Lower Memory Across Tabular Foundation Models

Abstract

Wide tables offer tabular foundation models more evidence, but accessing it can exhaust their memory: full-width pairwise mixing grows quadratically with the number of columns, while feature selection makes inputs affordable by discarding evidence. We ask whether using more features requires interacting over all of them at once. We introduce Support-Compiled Feature Folding (SCFF), a training-free inference framework that encodes wide tables through bounded calls to a frozen backbone. SCFF organizes support-ranked features into a strong Core and a candidate Tail, folds them into narrow feature groups, and support-checks the Tail's added evidence before a single contextual prediction. This converts quadratic feature-interaction work into linear-in-width work with a bounded local working set, without ensembling predictions or training new parameters. On the exhaustive 18-dataset wide-table slice of fixed AMLB-29, TabZilla, and TabArena snapshots, SCFF improves dataset-macro accuracy and NLL on all six evaluated backbones. All four matched-width comparisons retain favorable 95% dataset-bootstrap intervals on locked folds, with relative error reductions up to 26.1%. Median paired GPU-memory savings are 2.09-2.36x, and the ratio of separately observed maximum peaks reaches 34.3x. Under a measured peak-memory ceiling, SCFF uses the saved budget to preserve more support-selected evidence, improving accuracy by 4.06 and 3.72 points over the widest feasible single leaf on predeclared wide-Core strata of TabICLv2 and TabPFN-3.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Tabular Foundation Models, In-Context Learning, Feature-Width Extrapolation, Training-Free Inference, Wide-Table Learning, Feature Selection, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_y1WSOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Support-Compiled Feature Folding (SCFF), a training-free inference framework designed to address the computational and memory challenges of high-dimensional tabular foundation models. The key idea is to replace full-width feature interactions with bounded feature-group computations while preserving a single contextual prediction.

Specifically, SCFF first partitions features into a Core and a candidate Tail using support-set ANOVA statistics and the Benjamini–Hochberg procedure. Features are then organized into bounded-width groups and independently processed by a shared, frozen feature encoder. The resulting representations are aggregated before being passed to the original context encoder. Additionally, the authors introduce a Tail Recovery mechanism that uses residual projection, class prototypes, Fisher-based optimization, and reciprocal support-subset validation to selectively incorporate complementary information beyond the Core.

The method is evaluated on 18 wide-table classification datasets from AMLB-29, TabZilla, and TabArena, covering six backbone models. The reported results show improvements in dataset-macro accuracy and NLL across all six backbones, with favorable confidence intervals for the four matched-width comparisons on held-out query folds. SCFF also substantially reduces peak GPU memory usage, with median paired savings of approximately 2.09–2.36× for the matched-width backbones. Under a measured memory constraint, the method retains more informative features and achieves accuracy improvements of 4.06 and 3.72 percentage points on the wide-Core subsets of TabICLv2 and TabPFN-3, respectively.

Critical strengths

1. A practical and training-free approach to memory-efficient inference for wide tabular data.

The paper addresses an important practical limitation of tabular foundation models: the substantial memory overhead associated with full-width feature interactions. By partitioning features into bounded groups and combining their intermediate representations before contextual prediction, SCFF reduces the feature-interaction complexity from quadratic to linear growth with respect to feature width, without modifying pretrained parameters or requiring prediction ensembling.

The approach is conceptually simple and demonstrates applicability across multiple frozen backbones. The reported median GPU memory savings of approximately 2.09–2.36× in matched-width comparisons, together with generally preserved or improved predictive performance, provide meaningful evidence of its practical value.

2. Comprehensive experimental evaluation with convincing evidence of improved feature capacity under memory constraints.

The authors evaluate SCFF across multiple backbones and 18 wide-table datasets, including controlled comparisons that distinguish feature selection, bounded folding, and Tail Recovery. The evaluation further includes held-out query-fold confirmation, dataset-bootstrap confidence intervals, and detailed resource measurements.

Particularly convincing are the memory-matched experiments in Table 6, where SCFF retains more informative features than the widest feasible single-leaf baseline under comparable memory constraints, achieving accuracy improvements of 4.06 and 3.72 percentage points on the wide-Core subsets of TabICLv2 and TabPFN-3, respectively. These results directly support the central motivation of SCFF: enabling frozen tabular foundation models to utilize more predictive evidence within limited GPU memory budgets.

Critical weaknesses

1. Insufficient analysis of information loss caused by cross-leaf partitioning and mean aggregation.

SCFF independently encodes feature groups and aggregates their representations using a simple arithmetic mean. While this substantially reduces memory requirements, it eliminates direct cross-leaf feature interactions within the feature encoder and may weaken or discard complementary information distributed across different groups.

Although the paper provides same-Core comparisons between full-width and bounded encoding, these experiments do not distinguish the effects of feature partitioning from those of mean aggregation. Moreover, the evaluation lacks targeted analyses of tasks where predictive information depends on interactions between features assigned to different leaves.

This limitation is important because preserving predictive information is a central requirement of memory-efficient feature folding. Without controlled comparisons using alternative aggregation strategies or interaction-dependent datasets, it remains unclear under which conditions the proposed approximation may introduce substantial predictive degradation.

2. The computational cost of Tail Recovery is not sufficiently justified by its incremental predictive benefits.

Tail Recovery introduces additional feature-encoder computations, residual and prototype construction, and support-based validation. However, Appendix Table 20 shows that its incremental improvements in Accuracy and NLL are relatively modest and inconsistent across backbones, with negative effects observed on RefineICL-24L.

Although the paper reports end-to-end resource measurements for the complete SCFF framework, it does not separately quantify the additional latency and memory consumption attributable to Tail Recovery. Consequently, the practical benefit of this relatively complex component remains difficult to assess.

Furthermore, multiclass tasks always fall back to the Bounded Core representation, even though Algorithm 1 specifies encoding all Tail features. This raises additional concerns about potentially unnecessary computations that do not contribute to the final prediction.

A controlled resource comparison between Bounded Core Only and complete SCFF, together with a breakdown of Tail encoding and validation costs, would be important for establishing whether the additional computational overhead is justified by the observed predictive improvements.

3. Presentation issues in the figures.

The main figures suffer from overlapping text labels, crowded annotations, and insufficient spacing, which reduce their readability.

Questions for authors

Q1. Information loss from cross-leaf interactions and mean aggregation.

SCFF independently encodes feature groups and combines their representations through simple mean aggregation. While this design substantially reduces memory consumption, it removes direct cross-leaf feature interactions within the feature encoder and may dilute or discard complementary information.

Although the paper includes same-Core comparisons between full-width and bounded encoding, these experiments do not isolate the effects of feature partitioning and mean aggregation.

Could the authors provide controlled comparisons with alternative training-free aggregation strategies, such as support-weighted averaging, while keeping the feature partitions and backbone fixed? Furthermore, could the authors evaluate interaction-dependent tasks in which informative feature pairs are deliberately placed within the same leaf or across different leaves?

Q2. Computational overhead and predictive benefits of Tail Recovery.

Tail Recovery introduces additional feature-encoder computations, residual and prototype construction, and support-based validation. However, Appendix Table 20 shows that its incremental predictive improvements are relatively modest and inconsistent across backbones, while the additional computational costs of these operations are not separately quantified.

Could the authors provide a controlled comparison between Bounded Core Only and the complete SCFF pipeline, reporting the additional inference latency, peak GPU memory consumption, and corresponding improvements in Accuracy and NLL? In particular, a breakdown of the computational costs associated with Tail encoding and subsequent validation would help assess whether the additional complexity is justified by the observed predictive benefits.

Furthermore, since multiclass tasks always fall back to the Bounded Core representation, could the authors clarify whether Tail encoding is performed in these cases and whether skipping these unnecessary computations could further improve inference efficiency?

Reviewer_SVidOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper tackles the memory cost of running tabular foundation models (TFMs) on wide tables (which usually grows quadratically with the number of columns). It proposes Support-Compiled Feature Folding (SCFF), a training-free inference framework that repeatedly calls a frozen backbone that can handle 128 features.

The baselines are weak: they're not the TFM defaults, but components for handling more features are stripped off. I would like to see the actual defaults.

While tackling a large number of features (like 20k in biomedical datasets) well in TFMs would be very impactful, I don't think the paper's method is going to be the best for this. The gains come mostly from classical feature selection. Also, feature grouping does not take relationships between the features into account and can thus have failure modes where the combination of two particular features matters.

LLM use in writing: The text reads heavily LLM-generated, to a point that makes reading it painful to me. Nomenclature/jargon is not defined (e.g., the "evidence" that's even in the title appears to just be the number of features kept). Figure 3 has text and images running into each other /overlapping, Figure 1 has labels not visually placed to be visible.

Critical strengths
  1. Handling a higher number of features effectively is very relevant. E.g., biomedical data regularly has 20k features. (I would've hoped for an evaluation on such benchmarks.)
Critical weaknesses
  1. Weak baselines: the paper does not compare to other methods for solving the same problem and disables mechanisms TFMs currently use to tackle large features. It is not unlikely that the method is rendered unnecessary when these mechanisms are used.

  2. The gains come mostly from classical feature selection. Also, feature grouping does not take relationships between the features into account and can thus have failure modes where the combination of two particular features matters.

Questions for authors

Can you please evaluate against the TFM defaults as well?

Reviewer_AkgPOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces Support-Compiled Feature Folding (SCFF), a training free inference method to reduce GPUs memory when using TFM with wide datasets. First, dataset training columns are ranked using ANOVA and then select a subset of it (the most informative). Then, the selected columns, are split into subgrups and each group is passed separately through the model feature encoder and resulting embeddings are averaged. Then the context encoder and the prediction head run once on the averaged embeddings. The method is evaluated on 18 classification datasets with 128 to 7,200 columns and 6 TFMs (some are the same model but different version i.e., TabICL1 and 2, TabPFN-v1 and v3), against running each model on all columns at once.

Critical strengths
  1. The paper tackles an important problem, reducing the GPU memory of TFMs on wide datasets, and the idea is simple.
  2. The method is evaluated on 6 TFMs from 4 different families (2 are variants of the same, i.e., TabPFN v1/3, TabICL v1/v2) as well as on multiple classification datasets.
Critical weaknesses
  1. The paper is really hard to read, entire sections read like unpolished LLM output and it is really hard to follow. For exmaple:
  • "Additional evidence must earn its place" (Sec. 3.3), "Memory savings buy more useful evidence" (Sec. 5.5), "Folding buys capacity with a measured latency cost" (Table 21).
  • The introduction says "Consider robert, with 7,200 features." (l. 43) without saying that robert is a dataset. This only becomes clear in Table 10 (p. 16).
  • Many terms are never defined, such as "native evaluation contract", "receipts", "width compiler", "latent-state backbones" and "Native@2,048". While, "Core" and "Tail" are used to identify the selected and non-selected features, "leaves" are groups of columns.

Additionally, figures are clearly AI-generated (e.g., Figure 1,2,3) and not polished. Text and subfigure are overlapped, text goes outside the figure boundaries and it is not clear. Most importantly, authors state that AI tools were used only to assist with code implementation, debugging, experiment orchestration, and manuscript editing and that the authors reviewed all generated content. But in my opinion entire sections have been drafted using AI and not edited using it. I do believe the paper doesn't meet the high quality standard of ICLR, and addressing all these problems would require a major rewriting, that is not feasible during the rebuttal period.

  1. For TabPFN-3 and TabICLv1, a large part of the quality gain comes from ANOVA/BH feature selection, a standard existing technique, as the authors acknowledge themselves.

  2. The native baseline seems weak, for example all backbones are run as a single member with no prediction averaging, and TabPFN-3 without its feature subsampling. This is not how these models are normally used in this setting. Also, the performance gains are concentrated on a few datasets and simpler alternatives such as averaging the predictions of the same groups of columns are not compared.

  3. The memory benefit is weaker than the abstract suggests, and the setting where it is really needed (i.e., the native model doesn't fit in memory) is not shown.

Questions for authors
  1. How does SCFF compare with each backbone's default inference, and does SCFF still help on top of them?
  2. How does SCFF compare with averaging the predictions of the "leaves" with the test-time divide-and-conquer approach for TabPFNv2 [1] and with random projection to the native width at the same memory?
  3. From my understanding SCFF doesn't discard the non-selected columns, but this step is only applied to binary task and improve performance by only 0.08-0.15 on 3 models, while reducing the performance of the other 2 (Table 20). Given that, is it worth to add this complexity? How much time and memory this add on top of the method?
  4. How can SCFF be adapted to extend to the regression task?

[1] Ye, Han-Jia, Si-Yang Liu, and Wei-Lun Harry Chao. "A closer look at TabPFN v2: Understanding its strengths and extending its capabilities." Advances in Neural Information Processing Systems 38 (2026): 135605-135637.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.