acceptodds
Withdrawn from ICLR 2027

Compiled Task Memory for Efficient Inference with Frozen Tabular Foundation Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Context optimization offers a way to reduce the inference cost of tabular foundation models by replacing large training contexts with compact learned representations. Building on this approach, we study compiled task memory: task-specific context vectors optimized offline and reused through a frozen predictor. Slots are initialized from stratified frozen row encodings and optimized using supervised loss and out-of-fold teacher guidance, with all backbone weights held fixed. Our study evaluates accuracy, retained state, and cached inference cost on TabDPT and TabICLv2 across 35 classification and 13 regression tasks, using complete training and test splits and three seeds. Compared with the same unoptimized 256-slot initializations, optimization improves mean classification accuracy by 2.14 and 2.55 percentage points and reduces geometric-mean RMSE by 19.2% and 17.6%, respectively, with essentially unchanged compact inference cost. On classification tasks, cached 256-slot inference achieves geometric-mean speedups of and over full cached context, with mean accuracy losses of 0.76 and 0.78 percentage points. One-time preparation costs are reported separately. Construction and teacher-weight ablations characterize how memory learning affects quality at fixed capacity. Across –1024, larger memories improve regression at higher serving and storage cost, while classification gains beyond 256 slots remain uncertain. These findings characterize the practical accuracy–cost tradeoffs of context optimization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tabular foundation models, context optimization, in-context learning, context compression, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_yBnWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper explores the problem of context optimization within tabular foundation models. The paper proposes the use of compiled task memory, which are task-specific context vectors optimized offline and reused through a frozen predictor. Experiments appear to show improvement over baseline methods evaluated across a range of tabular classification and regression tasks. The current submission has significant issues in clarity which presents issues in parsing the submission and evaluating the core scientific contribution.

Critical strengths

The core problem being examined by the paper (context optimization to improve the efficiency of tabular foundation models) is an important problem of interest to the research community. Experimental results show that the proposed method may lead to improvements relative to baseline methods. Evaluation has been performed on an appropriate array of classification and regression tasks.

Critical weaknesses

The critical weakness in this paper is clarity. Much of the paper employs LLM phrasing, which is unclear and contains unusual phrasing that is difficult to parse and contains extraneous information. Overall, this is the most significant issue with the submission.

The contributions details are not clear and largely relate to experimental results rather than proposing a conceptual contribution. This makes it difficult to evaluate the proposed scientific contribution. For example:

  • L050 "Optimized256 improves over its exact Unoptimized256 initialization on two backbones. That initialization is already a stratified frozen-encoding baseline. At the same K256 budget, cluster/PCA representatives and fixed-anchor Hopfield pooling provide additional construction controls on all 48 tasks; the K1024 construction follow-up is in Appendix B.6."
  • L059 "Slot counts of 64/256/512/1024 expose the accuracy, state and latency tradeoff. Larger memories improve regression but retain a mean gap to full context. Offline preparation is accounted for separately from repeated query serving."

This is an issue throughout the submission. For example in the related works:

  • "The TuneTables entries use adapted single prompts on TabPFN-v1, without its full search/ensembles; ICD is related work, not a reproduced baseline. The paired λ = 0 arm instead provides supervised context optimization on each evaluated backbone."

Overall the paper is written more as an experimental report rather than as a self-contained conference paper. The method section would be difficult to replicate based on the text contained within the submission. The Appendix contains references to the project README.md, requirements.txt, and python scripts in the supplementary materials. L615"reproduce.py verifies both input checksums and implements their aggregation, rank ties and the figure’s 10,000-draw bootstrap. ctm.py pins the TabDPT/TabICLv2 checkpoint filenames, download revisions; requirements.txt and requirements-model.txt pin the corresponding dependencies. README.md maps these files to Appendices A.1 and A.2."

Questions for authors

The critical issue with the current submission is clarity, which has made it difficult to evaluate the scientific contribution. While experimental results show that the implementation may show promise, it would be useful to understand the core contribution as the authors see it and how this relates to prior literature. Expressing this in the author's own words would be useful to understand the proposed contribution.

Reviewer_NEE2OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies a TuneTable-like optimization of the context to feed into a frozen tabular foundation model (TFM), motivated by reducing its inference time. For a new dataset, this context is optimized in 100 Adam steps. Compared to a standard KV cache, this yields faster predictions in evaluations on TabArena for TabDBT and TabICLv2, at the cost of worse predictions than with the full context. There are no results for larger datasets (as, e.g., in BeyondArena large), where the technique appears more relevant.

Some of the text reads LLM-generated. Nomenclature/jargon is often not introduced, or introduced in some later section but already used in earlier ones.

Critical strengths
  • Consistently reducing inference cost of TFMs would be helpful.
Critical weaknesses
  • Limited novelty compared to TuneTables. Also, when you have to pay compute for train per dataset that gives up a lot of the nice properties of TFMs, and feels more like standard ML.

  • Limited evidence of generalization. After reading the paper I am not convinced that this will be built into every TFM in the future. The evidence is limited: only TabDBT and TabICLv2, only TabArena. Also, in practice one would prefer to avoid a performance hit; is it possible to Pareto-dominate a modern TFM's performance both in terms of accuracy and inference cost, by paying a higher one-time cost? That would be very interesting!

Questions for authors
  • Can you clarify novelty compared to TuneTables?

  • Is it possible to Pareto-dominate a modern TFM's performance both in terms of accuracy and inference cost, by paying a higher one-time cost? That would be very interesting!

Reviewer_FhC6OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper replaces the full training context of a frozen tabular foundation model (TabDPT, TabICLv2) with K learned context vectors. These are optimized offline with a label-loss term plus distillation from the model’s own held-out predictions, and then cached for inference. Across classification and regression tasks, this beats unoptimized contexts of the same size, while performing below the full context baseline but with faster inference.

Critical strengths
  • Extends prior context-optimization approaches (ICD/TuneTables) to newer tabular foundation models (TabDPT, TabICLv2) and to regression, adding a distillation term toward the model’s own held-out predictions (KL for classification, MSE for regression).
Critical weaknesses
  • Presentation: It is often hard to follow both the explanation of the method, which in its core is simple, as well as the presentation of the experimental results. For example: the abstract contains many comparisons whose meaning is unclear before reading the rest of the paper, where exactly the K vectors are used is unclear from the main paper, different tables often report the same method with slightly different setups resulting in different numbers (Table 1 vs. Table 2).
  • Novelty: Although prior work is properly acknowledged, the difference from it is not large. Especially, the performance-cost trade-offs are hard to judge: some comparisons are missing (e.g., inference speed and memory of CatBoost-TD) and inference speedups are small on the on average stronger TFM (TabICLv2), where timings for e.g., classification are roughly 0.060 ms for different K slot versions compared to 0.067 ms for the full context version (Table 1).
Questions for authors

I do not have any concrete questions, but rather a constructive suggestion: I think the paper would highly benefit from substantial efforts in more extensive comparisons to additional baselines (for the inference speedup argument), broader evaluations (more splits as is common), and a much better presentation for figures, tables, and especially the text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.