- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents a study of tabular foundation models on data streams. It compares them against streaming learners and streaming AutoML systems on 9 streams, consisting of 5 real-world and 4 synthetic streams. The authors find that tabular foundation models achieve the highest predictive performance and outperform specialised approaches developed for this streaming task. However, the authors find that computational cost is a significant issue in foundation models, mainly because a nearly unchanged context is re-encoded at every prediction. Beyond the main evaluation, the authors study the behaviour of the models under different types of drift, recovery after drift, label delay, memory size, context refresh rate and query batching.
Critical strengths
- The paper presents a comparison of a broad range of existing methods to address data streams and picks a representative selection of tabular foundation models to compare against. All methods are evaluated under a shared and fair protocol. The insight that modern foundation models beat specialised models is both interesting and highly relevant.
- The analysis of how models behave on different types of synthetic drift, how long they take to recover, and the effect of label delay is very interesting and relevant for both researchers in developing new approaches and practitioners that deploy such systems.
Critical weaknesses
- The benchmark dataset size is small for a paper whose main contribution is a systematic comparison. The main benchmark uses 9 datasets, of which 5 are real-world and with one synthetic dataset for each drift type. Furthermore, most drift-specific conclusions are based on even fewer streams in the follow-up experiments. This makes it challenging to draw statistically significant conclusions from the results. Adding more synthetic and real-world data would substantially strengthen the evaluation. Additionally, including classical models such as CatBoost or XGBoost with the same FIFO window would be relevant baselines.
- The refresh-rate study appears to show only minor runtime savings. Between context refreshes, the context should be fixed, so from my understanding, the KV cache could be reused. However, it appears the model reprocesses the full context on every call, negating most of the runtime advantage. This likely understates the actual runtime advantage of the context update parameter and weakens the conclusion that new architectures are required.
Questions for authors
- In Appendix C.4, the text states that the two-sided Wilcoxon test with Holm correction does not separate the leading in-context methods. However, Figure 10 just below appears to show exactly this significance as no bar connects TabFM+FIFO to any other method. Could you please explain the difference between the text and the figure to clarify this?
- Have you tried fitting larger context sizes with more efficient foundation models like TabFlex or adding a hybrid architecture with a drift detector that resets the memory on distribution shifts?
- Why does the Majority-Class model grow by 4.48 KB per instance in Table 2, which is higher than most tabular foundation models? Shouldn’t it just require constant memory?
3 comments
Response to authors
Thank you for the detailed response and for performing all my requested experiments. While I believe that the increased number of datasets and models strengthened the evaluation, I still think that 12 datasets is on the low end. Specifically, as these datasets are again split into subcategories of different types of drift. Regarding the KV caching, I thank the authors for correcting the table and for reframing their suggestions according to the updated evidence. Overall, I will raise my score to 3.
Thank you for the kind reply and for raising your score. We chose the streams to cover the main conditions found in data stream scenarios, and this is in line with the benchmark sizes used in earlier work on TFMs for streams. We will note the benchmark size in the revised paper.
Response to Reviewer wKdB
Thank you for the review. We added 3 real streams, GBDT baselines on the same FIFO window, the KV cache, larger contexts with a drift detector, and the two corrections you pointed out (C.4 text, memory column). All new results are shown below and will be included in the revised paper.
W1. More streams and GBDT baselines. We added 3 real streams from the USP repository: INSECTS (57,018 rows, 33 features, 6 classes), asfault (8,565 rows, 64 features, 5 classes) and keystroke (1,600 rows, 10 features, 4 classes). The benchmark now has 12 streams, and all methods of Table 2 run on all 12.
We regenerated the critical-difference diagram (Fig. 10) on the 12 streams. Kumo large and TabFM form the top group (mean ranks 1.5 and 2.0), TabICL v2 + FIFO and CURE the second (3.3 and 3.5), and both groups are separated from all streaming learners, AutoML methods and GBDTs (mean ranks 7.2 and above).
GBDTs retrained every W instances on the same 1,000-row FIFO window:
On all 12 streams, LightGBM (W=10) is below TabICL+FIFO on every stream. The refresh and memory-size studies below use the same 12 streams, and every table in the revised paper will state its number of streams.
W2. KV cache. TabICL v2 ships a KV cache. We re-ran the refresh study with it on the 12 streams.
You are right, Table 4 understated the saving. The KV cache gives the same predictions as re-encoding, only faster. With it, K=10 is 2× faster for −.004 accuracy, and we now recommend it. But the saving stops near 16 ms/inst. (K=100, K=1000), and at K=1 the cache is slower than re-encoding (61.0 vs 46.4 ms), since every new row changes the whole cached context. So current TFMs can reuse context only in blocks. Updating at every instance needs causal attention over rows, which is an architecture change (Fig. 8, Table 22, App. D.6). This is the point of the paper. The focus should move to the architecture, not only to context management (which instances are in context) as in past work. We will revise Table 4 and the conclusion, and define both terms in Sec. 3 (context management is which rows are kept, context serving is how the context is encoded).
Q1. C.4 text vs. Figure 10. The text was written before we added TabFM and, more recently, Kumo Tabular, so the text and the figure used different sets of comparators. We regenerated the figure on 12 streams (groups given in W1) and will rewrite the C.4 text to match it.
Q2. Larger contexts and drift detector. We ran TabICL v2 with FIFO contexts of 2,000 and 5,000 rows, and with an ADWIN detector that resets the context on drift (up to 2,000 or 5,000 rows). We also implemented TabFlex, but its checkpoints cannot be downloaded (github.com/microsoft/ticl/issues/27).
On the real streams, FIFO accuracy grows with the context size, and 5,000 FIFO rows give the best mean (+0.5 points over the paper's setting). The ADWIN reset is below the plain 1,000-row FIFO there (.820 and .828 vs .831), because the reset discards history that is still useful. On the synthetic streams with abrupt and recurring drift it is the opposite: a 5,000-row FIFO keeps outdated rows (agr_a .935 → .845, tree_r .767 → .604), and the ADWIN reset gives the best result (.959 and .824). Both results will be added next to the memory-size study (Table 3).
Q3. Majority-Class memory. The model needs constant memory (one counter per class). The column measured memory growth of the whole evaluation process, not of the model, and most of the 4.48 KB/inst. comes from one stream (meter, 30.3). In the revised paper we will remove this column from Table 2 and report peak GPU memory for the TFMs. The per-stream process trace (Table 14) stays in the appendix with this explanation.