acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Tabular Foundation Models On Data Streams

Abstract

Tabular foundation models (TFMs) outperform established machine learning models on tabular benchmarks through in-context learning. Building on this success, interest is growing in applying them to data streams, where data arrive continuously and evolve over time. On a stream, a TFM adapts by updating its context rather than its parameters, so its accuracy and cost depend on which examples it keeps and how often it rebuilds its context. We therefore present a systematic study of TFMs on data streams, covering memory management, computational cost, and stream-specific challenges such as concept drift and delayed labels. We find that TFMs achieve the highest predictive performance and that simply retaining the most recent examples is as effective as existing memory management techniques. They also recover faster than streaming learners after drift and keep the highest accuracy under label delay. This accuracy, however, comes at a high serving cost, since a nearly unchanged context is re-encoded at every prediction. These results point to architectural efficiency as the way forward for in-context stream learning.

Primary area:
datasets and benchmarks
Keywords:
Data stream learning, Concept drift, Tabular foundation models, In-context learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wKdBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a study of tabular foundation models on data streams. It compares them against streaming learners and streaming AutoML systems on 9 streams, consisting of 5 real-world and 4 synthetic streams. The authors find that tabular foundation models achieve the highest predictive performance and outperform specialised approaches developed for this streaming task. However, the authors find that computational cost is a significant issue in foundation models, mainly because a nearly unchanged context is re-encoded at every prediction. Beyond the main evaluation, the authors study the behaviour of the models under different types of drift, recovery after drift, label delay, memory size, context refresh rate and query batching.

Critical strengths
  • The paper presents a comparison of a broad range of existing methods to address data streams and picks a representative selection of tabular foundation models to compare against. All methods are evaluated under a shared and fair protocol. The insight that modern foundation models beat specialised models is both interesting and highly relevant.
  • The analysis of how models behave on different types of synthetic drift, how long they take to recover, and the effect of label delay is very interesting and relevant for both researchers in developing new approaches and practitioners that deploy such systems.
Critical weaknesses
  • The benchmark dataset size is small for a paper whose main contribution is a systematic comparison. The main benchmark uses 9 datasets, of which 5 are real-world and with one synthetic dataset for each drift type. Furthermore, most drift-specific conclusions are based on even fewer streams in the follow-up experiments. This makes it challenging to draw statistically significant conclusions from the results. Adding more synthetic and real-world data would substantially strengthen the evaluation. Additionally, including classical models such as CatBoost or XGBoost with the same FIFO window would be relevant baselines.
  • The refresh-rate study appears to show only minor runtime savings. Between context refreshes, the context should be fixed, so from my understanding, the KV cache could be reused. However, it appears the model reprocesses the full context on every call, negating most of the runtime advantage. This likely understates the actual runtime advantage of the context update parameter and weakens the conclusion that new architectures are required.
Questions for authors
  • In Appendix C.4, the text states that the two-sided Wilcoxon test with Holm correction does not separate the leading in-context methods. However, Figure 10 just below appears to show exactly this significance as no bar connects TabFM+FIFO to any other method. Could you please explain the difference between the text and the figure to clarify this?
  • Have you tried fitting larger context sizes with more efficient foundation models like TabFlex or adding a hybrid architecture with a drift detector that resets the memory on distribution shifts?
  • Why does the Majority-Class model grow by 4.48 KB per instance in Table 2, which is higher than most tabular foundation models? Shouldn’t it just require constant memory?
3 comments
AuthorsOpenReview ↗

Response to Reviewer wKdB

Thank you for the review. We added 3 real streams, GBDT baselines on the same FIFO window, the KV cache, larger contexts with a drift detector, and the two corrections you pointed out (C.4 text, memory column). All new results are shown below and will be included in the revised paper.

W1. More streams and GBDT baselines. We added 3 real streams from the USP repository: INSECTS (57,018 rows, 33 features, 6 classes), asfault (8,565 rows, 64 features, 5 classes) and keystroke (1,600 rows, 10 features, 4 classes). The benchmark now has 12 streams, and all methods of Table 2 run on all 12.

MethodMean acc. (12 streams)Wins vs SRP
Kumo Tabular large + FIFO (new).87012/12
TabFM + FIFO.86812/12
TabICL v2 + FIFO.86012/12
TabICL v2 + CURE.85812/12
LightGBM on FIFO window (best GBDT).8227/12
EvoAutoML (best AutoML).8176/12
SRP (best streaming ensemble).811–

We regenerated the critical-difference diagram (Fig. 10) on the 12 streams. Kumo large and TabFM form the top group (mean ranks 1.5 and 2.0), TabICL v2 + FIFO and CURE the second (3.3 and 3.5), and both groups are separated from all streaming learners, AutoML methods and GBDTs (mean ranks 7.2 and above).

GBDTs retrained every W instances on the same 1,000-row FIFO window:

W=1W=10W=100
LightGBM.821.816.785
XGBoost.815.810.779
CatBoost.787.784.763
TabICL v2 + FIFO.860.854.827

On all 12 streams, LightGBM (W=10) is below TabICL+FIFO on every stream. The refresh and memory-size studies below use the same 12 streams, and every table in the revised paper will state its number of streams.

W2. KV cache. TabICL v2 ships a KV cache. We re-ran the refresh study with it on the 12 streams.

Refresh every K1101001000
Accuracy.860.856.834.767
ms/inst., re-encode46.441.440.240.2
ms/inst., KV cache61.020.516.015.8

You are right, Table 4 understated the saving. The KV cache gives the same predictions as re-encoding, only faster. With it, K=10 is 2× faster for −.004 accuracy, and we now recommend it. But the saving stops near 16 ms/inst. (K=100, K=1000), and at K=1 the cache is slower than re-encoding (61.0 vs 46.4 ms), since every new row changes the whole cached context. So current TFMs can reuse context only in blocks. Updating at every instance needs causal attention over rows, which is an architecture change (Fig. 8, Table 22, App. D.6). This is the point of the paper. The focus should move to the architecture, not only to context management (which instances are in context) as in past work. We will revise Table 4 and the conclusion, and define both terms in Sec. 3 (context management is which rows are kept, context serving is how the context is encoded).

Q1. C.4 text vs. Figure 10. The text was written before we added TabFM and, more recently, Kumo Tabular, so the text and the figure used different sets of comparators. We regenerated the figure on 12 streams (groups given in W1) and will rewrite the C.4 text to match it.

Q2. Larger contexts and drift detector. We ran TabICL v2 with FIFO contexts of 2,000 and 5,000 rows, and with an ADWIN detector that resets the context on drift (up to 2,000 or 5,000 rows). We also implemented TabFlex, but its checkpoints cannot be downloaded (github.com/microsoft/ticl/issues/27).

TabICL v28 real streams (mean)
FIFO, 1,000 rows (paper).831
FIFO, 2,000 rows.833
FIFO, 5,000 rows.836
ADWIN reset, 2,000 rows.820
ADWIN reset, 5,000 rows.828

On the real streams, FIFO accuracy grows with the context size, and 5,000 FIFO rows give the best mean (+0.5 points over the paper's setting). The ADWIN reset is below the plain 1,000-row FIFO there (.820 and .828 vs .831), because the reset discards history that is still useful. On the synthetic streams with abrupt and recurring drift it is the opposite: a 5,000-row FIFO keeps outdated rows (agr_a .935 → .845, tree_r .767 → .604), and the ADWIN reset gives the best result (.959 and .824). Both results will be added next to the memory-size study (Table 3).

Q3. Majority-Class memory. The model needs constant memory (one counter per class). The column measured memory growth of the whole evaluation process, not of the model, and most of the 4.48 KB/inst. comes from one stream (meter, 30.3). In the revised paper we will remove this column from Table 2 and report peak GPU memory for the TFMs. The per-stream process trace (Table 14) stays in the appendix with this explanation.

Reviewer_wKdBOpenReview ↗

Response to authors

Thank you for the detailed response and for performing all my requested experiments. While I believe that the increased number of datasets and models strengthened the evaluation, I still think that 12 datasets is on the low end. Specifically, as these datasets are again split into subcategories of different types of drift. Regarding the KV caching, I thank the authors for correcting the table and for reframing their suggestions according to the updated evidence. Overall, I will raise my score to 3.

Thank you for the kind reply and for raising your score. We chose the streams to cover the main conditions found in data stream scenarios, and this is in line with the benchmark sizes used in earlier work on TFMs for streams. We will note the benchmark size in the revised paper.

Reviewer_SnCqOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how tabular foundation models (TFMs) behave when data arrive as a stream rather than as a fixed table. It assembles fourteen methods plus two reference baselines—five classical streaming learners, three streaming AutoML systems, and in-context TFMs (TabPFN v1, TabICL v2, TabFM, Drift-Resilient TabPFN) combined with FIFO, CURE, or DualFIFO memory—and evaluates them on nine classification streams (five real, four synthetic covering abrupt, gradual, incremental, and recurring drift) under a single-pass prequential protocol.

Critical strengths
  1. Prior TFM-for-streams papers each evaluate their own backbone on their own streams. A common harness that places streaming learners, streaming AutoML, and in-context TFMs on the same prequential protocol, with cost measured alongside accuracy, is genuinely useful and, to my knowledge, not available elsewhere.
  2. Reporting ms/instance and KB/instance next to accuracy changes the conclusion of the benchmark: the Pareto plot (Fig. 3) shows the accuracy gain is bought at 10–1000 times latency.
  3. The controlled memory study is well designed. Holding the backbone fixed and pairing FIFO against CURE/DualFIFO at B ∈ {100, 250, 500, 1000} (Table 3) isolates the policy from the backbone, which the main table cannot do.
Critical weaknesses
  1. The study covers classification only, streams with at most ten classes (a TabFM constraint), and memory budgets capped at 1,000. Regression—arguably the other half of tabular learning—is absent, and the paper itself notes that larger memories remain unexplored. The conclusion that "efficient streaming TFMs will require architectural changes" is plausible but is an extrapolation only from current settings.
  2. A windowed LightGBM/XGBoost retrained every W instances on the same memory budget is the baseline most practitioners would deploy? Given the paper's framing around deployment cost, this comparison is important: a periodic GBDT retrain may sit at a very different point on the cost–accuracy frontier.
  3. Cross-method wall-clock times compare CPU-hosted streaming learners with GPU-hosted TFMs.
  4. Can the ring-cache serving path be applied to a released TFM checkpoint without retraining?
  5. The paper leaves unlabelled instances unused during label delay. Would a transductive or semi-supervised use of the unlabelled buffer change the delay results, particularly on meter where CURE wins?
Questions for authors

See the weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer SnCq

Thank you for the review. We added GBDT baselines, contexts up to 5,000 rows, cost on each method's own hardware, the KV cache of a released TFM, and two ways to use the unlabelled buffer.

W1. Scope. The limitations (Sec. 5) already state that the study covers classification only and that larger memories were not explored. In the revised paper the title, abstract and conclusion will say stream classification, and the limitations will add the ten-class limit (from TabPFN v1 and TabFM). For memory size, we added 2,000 and 5,000 rows with TabICL v2 on the 12 streams. A 5,000-row FIFO helps on 6 streams (e.g. posture .629 → .670) and hurts on 6, mostly under drift (agr_a .935 → .845). In the conclusion, the cost argument for architectural change will be supported by the KV-cache result (W4).

W2. Windowed GBDT. We ran LightGBM, XGBoost and CatBoost retrained every W instances on the same 1,000-row FIFO window. They will be added to Table 2 and Fig. 3.

9 submission streamsW=1W=10W=100
LightGBM.821.816.785
XGBoost.815.810.779
CatBoost.787.784.763
TabICL v2 + FIFO.860.854.827

LightGBM W=10 costs 17 ms/inst. on CPU and is more accurate than SRP and ARF. It stays below TabICL+FIFO on all 12 streams (.822 vs .860 mean), and under label delay d=1000 (.715 vs .748, 4 streams).

W3. CPU vs. GPU time. The paper prices each method on its own hardware in Table 15 (Appendix C.7). We will move this table to the main text and label Fig. 3 as deployment cost. With the new GBDT rows:

$ per million instances
SRP (CPU)0.90
LightGBM W=10 (CPU)1.84
TabICL v2 + FIFO (GPU)9.08
LightGBM W=1 (CPU)17.57
TabFM + FIFO (GPU)26.0

All comparisons within a backbone (FIFO vs CURE, budget, refresh, batching, cache) run on the same GPU.

W4. Ring cache on a released checkpoint. Not without retraining. We tested the closest available option, the KV cache that TabICL v2 ships (12 streams). It gives identical predictions and halves latency when the context is refreshed every K ≥ 10 instances (20.5 vs 41.4 ms/inst. at K=10). When the context changes at every instance (K=1) it is slower than re-encoding (61.0 vs 46.4 ms), because each cached row depends on the whole context. Adding one row at a time needs causal attention over rows, so the model has to be trained that way, as in the cached-serving study of the paper (Fig. 8, Table 22).

W5. Unlabelled buffer under delay. We ran both variants. Pseudo-labelling: each pending row enters the context with its predicted label and gets the true label when it arrives. Transductive: pending rows are passed as extra test rows.

Accuracy, d=1000meternoaaagr_a
TabICL v2 + FIFO.597.804.885
+ pseudo-labels.649.797.859
+ transductive.594.806.878
TabFM + FIFO.613.806.889
+ pseudo-labels.703.799.856
CURE.638.803.886

It does change the result on meter: pseudo-labels add 5 to 9 points, and TabFM moves above CURE. On agr_a they cost about 3 points, because wrong pseudo-labels enter the context after the abrupt drift. At d=100 all variants are within .007 of FIFO. The transductive variant has no clear effect. These results will be added to the label-delay study (Fig. 6, Table 10).

Reviewer_XQWLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a systematic empirical study of Tabular Foundation Models (TFMs) for data stream learning. The authors compare recent TFM based streaming methods against conventional streaming learners and online AutoML methods across nine real-world and synthetic streams under a unified evaluation protocol. The study examines predictive performance, concept drift adaptation, memory management, delayed labels, and computational efficiency. The results show that TFMs generally achieve higher predictive accuracy and faster adaptation to concept drift, while simple FIFO memory management performs comparably to more sophisticated approaches. However, TFMs incur substantially higher inference costs due to repeated context encoding. The authors further demonstrate the potential of context caching using compact causal models, suggesting architectural efficiency as an important direction for future streaming TFMs.

Critical strengths

S1. Controlled comparisons of memory-management methods.

The paper compares CURE and DualFIFO with FIFO using the same backbones, allowing a more direct assessment of the benefits of memory management. In Table 2, FIFO achieves comparable or slightly better mean accuracy with lower serving latency (58.4 vs. 87.3 ms/instance for TabICL and 103 vs. 127 ms/instance for TabPFN). Table 3 further compares these methods under different memory budgets. These results suggest that more sophisticated memory-management methods do not necessarily outperform simple FIFO, raising an important question about whether their additional complexity is justified.

S2. Practical analysis of serving efficiency.

The paper examines several factors affecting serving efficiency, including context refresh frequency, query batching, and context caching. Table 4 shows that reducing context refresh frequency provides only limited latency improvements, while Table 5 shows much larger gains from query batching, albeit with some accuracy trade-offs. The compact-model experiments also demonstrate the potential benefits of reusing cached contexts. These analyses help identify where the main computational costs arise and which approaches are more effective at reducing them.

Critical weaknesses

W1. Limited analysis of how general tabular performance relates to streaming performance.

The paper identifies backbone choice as an important factor in streaming performance, but provides limited analysis of how this relates to performance on standard tabular benchmarks. Although the paper discusses architectural differences, model scale, and pretraining, it remains unclear whether stronger backbones on benchmarks such as TabArena also perform better in streaming settings, particularly under concept drift and delayed feedback.

This relationship is important for understanding whether advances in general-purpose TFMs translate to streaming applications. Comparing the streaming results with the backbones' performance on standard tabular benchmarks, while also considering inference costs, would help explain whether streaming performance is mainly determined by general predictive capability or requires additional stream-specific properties.

Questions for authors

Q1. Could the authors further discuss whether stronger performance on standard tabular benchmarks such as TabArena translates into better streaming performance, particularly in terms of drift recovery and robustness to delayed labels?

Q2. Could the authors extend the analysis to include more recent TFMs with stronger performance on standard tabular benchmarks to further examine whether these improvements also translate into better streaming performance?

3 comments
AuthorsOpenReview ↗

Response to Reviewer XQWL

Thank you for the review. We added a newer TFM and a comparison of backbones on streams, under drift and under label delay.

W1. General tabular strength vs. streaming. TabArena evaluates TFMs on static tables with i.i.d. train/test splits. Our study covers streams, where data arrives in order, concepts drift and labels can be late, so we compare the backbones in that setting.

Backbone + FIFO12 streamsINSECTS incrementalms/inst.
TabPFN v1.815.661116
TabICL v2.860.67857
TabFM (1.6B).868.686170
Kumo Tabular large (216M).870.68794

Newer backbones rank higher in each column. Drift recovery per backbone is shown in Fig. 4 of the paper. This table will be added to Sec. 4.

Q1. Does general tabular strength carry over to streams? Our results suggest it does. The main point of the paper is that recent TFMs perform well on streams with a plain FIFO context and no stream-specific changes to the model. From TabPFN v1 to TabICL v2 to TabFM and Kumo, the mean over 12 streams rises from .815 to .870 and accuracy on INSECTS incremental from .661 to .687. Under label delay d=1000 it rises from .696 (TabPFN v1) to .759 (TabFM). The general predictive ability of these models holds under drift and delayed labels, so we expect better backbones to be the main source of better streaming performance.

Q2. Newer TFMs. We added Kumo Tabular (NVIDIA, Sep. 2026). Kumo large has the highest mean accuracy among the FIFO backbones (.870), is higher than TabICL+FIFO on 11/12 streams, and is on par with TabFM (.868) with 7.6× fewer parameters. It will be added to Table 2, Fig. 3 and the critical-difference diagram (Fig. 10).

Thank you for the response. It alleviates my concerns, so I'll maintain my positive rating.

Thank you for taking the time to read our response. We are glad it addressed your concerns, and we appreciate your support.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of new results and questions to the reviewers

We thank the reviewers and the Area Chair for their time. Our responses of 14 Nov have not yet received a reply, so we summarise the new evidence here and ask each reviewer one short question.

What the new experiments show

  • The main finding holds on a larger benchmark. With 3 more real streams (12 in total), TFMs with a plain FIFO context remain the most accurate, with mean accuracy .860 to .870 against .811 for SRP, the best streaming ensemble (wKdB).
  • It holds against the baseline practitioners would deploy. LightGBM, XGBoost and CatBoost retrained on the same 1,000-row FIFO window stay below TabICL v2 + FIFO on all 12 streams, with LightGBM at .822 against .860 (SnCq, wKdB).
  • Stronger backbones give better streaming accuracy. From TabPFN v1 to the newly added Kumo Tabular, mean accuracy rises from .815 to .870 (XQWL).
  • The cost conclusion is now tested directly. The KV cache of TabICL v2 halves latency when the context is refreshed every 10 instances, but it is slower than re-encoding when the context changes at every instance. Per-instance updates therefore need an architectural change (wKdB, SnCq).
  • Contexts up to 5,000 rows, an ADWIN reset and pseudo-labels under label delay are also evaluated, and the C.4 text and the memory column of Table 2 are corrected (SnCq, wKdB).

All of these results will be included in the revised paper.

  • Reviewer wKdB, do the 12-stream benchmark, the GBDT baselines and the KV-cache study resolve your concerns on benchmark size and on the refresh study?
  • Reviewer SnCq, do the windowed GBDT results, the cost on each method's own hardware and the pseudo-label experiment answer W2 to W5?
  • Reviewer XQWL, does the backbone comparison with Kumo Tabular answer Q1 and Q2?

If any point remains open, we will gladly answer it before the discussion closes tomorrow. If your concerns are resolved, we would appreciate it if you could update your review to reflect this.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.