acceptodds
Under review as a conference paper at ICLR 2027

Selective Context Retention for Tabular Foundation Models on Data Streams

Abstract

Tabular stream learning requires predictions on sequentially arriving examples under bounded memory and potential distribution shift. Standard stream learning methods operate by updating model state, such as split statistics or ensemble members. Recently, tabular foundation models (TFMs) that make predictions conditioned on a labeled context in an in-context manner have emerged as a natural alternative for stream learning. This turns stream adaptation via model updates into bounded context management, where the system must decide which examples to keep in the context. In this paper, we propose a future-information view that measures the usefulness of a context by the information it provides about near-future queries. This motivates preserving recent examples, retaining uncertain examples, and removing redundant examples. We instantiate these signals as RECAP (REtained Context for Adaptive Prediction), a simple context policy with entropy-gated admission and redundancy-aware eviction. RECAP achieves the highest prequential accuracy against six classical stream learners on seven streams. Under the same TFM backbone and context budget, it improves on DualFIFO, a prior context management approach. This advantage persists under noisy feedback, with gains of 0.27 to 1.95 percentage points over DualFIFO across three streams when 30% of feedback labels are corrupted. Code and datasets are available at https://anonymous.4open.science/r/RECAP-8D1B/.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Tabular Foundation Model, Stream Learning, In-context Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2a7LOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates bounded context management for frozen Tabular Foundation Models (TFMs, e.g., TabICL, TabPFN) in data stream classification. Instead of updating model parameters or decision tree splits, the model adapts via in-context learning by dynamically updating a fixed-capacity labeled context. The authors introduce a future-information view based on conditional mutual information to justify three context retention signals: recency, uncertainty, and redundancy. Based on this framework, the paper proposes RECAP (REtained Context for Adaptive Prediction). Experiments on seven streams compare RECAP against six classical stream learners (e.g., ARF, SRP, VFDT) and an existing dual-FIFO buffer policy (DualFIFO).

Critical strengths
  1. Interesting formulation. Reframing stream adaptation as bounded in-context memory management for frozen tabular foundation models is a timely and promising research direction.
  2. Clear writing. The paper is well-organized, with detailed appendices covering proofs, dataset statistics, and diagnostic analyses.
Critical weaknesses
  1. Marginal empirical improvements over DualFIFO. The primary contribution of RECAP is supposed to be its selective retention policy (entropy gating + redundancy eviction) over naive FIFO retention. However, when evaluated under the exact same TFM backbone and context budget (Table 2), the actual improvement over DualFIFO are marginal (90.60% VS 90.68%, 90.45% VS 90.52%).
  2. Table 3(a) examines RECAP across different context capacities (BB) and bank allocations (ρ\rho), exposing serious performance degradation. This demonstrates that RECAP is fragile. It only slightly outperforms FIFO when the context budget is large ( B≥1000B≥1000), but fails when memory is constrained. This directly undermines the motivation of bounded context management for resource-limited streams.
Questions for authors
  1. Why does RECAP degrade so significantly compared to DualFIFO when B=100B=100 (−1.80% on METER)? Does entropy gating cause severe class starvation when the buffer is small?
  2. In Table 14, the distance loss correlation is negative on three out of four streams (e.g., −0.126 on AGR(A)). If raw feature Euclidean distance does not predict information loss upon eviction, what is the actual mathematical and empirical mechanism preventing RECAP from discarding critical boundary samples?
2 comments

We thank the reviewer for the questions about the magnitude of the gains, performance at small budgets, and the eviction mechanism.


W1. Magnitude of the gains over DualFIFO.

The improvements over DualFIFO are modest in the main comparison, but they extend across streams and also appear under noisy feedback and concept drift. With the backbone and budget matched, Table 2 shows gains of 0.07 to 0.73 percentage points, averaging 0.31 across seven streams. All seven gains are positive at the fixed threshold (exact paired Wilcoxon p=0.015625p=0.015625), and six remain positive with prefix-only threshold selection.

In the additional analyses, the NOMAO accuracy gain reaches 2.42 points under 10% label noise (Table 11), while mean recovery time under gradual drift decreases from 1,073 to 988 steps (Table 4(b)). These benefits matter because a stream learner must continue predicting when feedback is noisy or predictive relationships change.


W2. Performance under limited memory.

The small-budget degradation occurs on METER, while RECAP improves over DualFIFO on NOMAO and AGR(A) at the same budgets. At B=100B=100, the gains on NOMAO and AGR(A) are 0.33 and 0.19 points, respectively (Table 3(a)). We examine the METER failure directly below.


Q1. Why does RECAP degrade at B=100B=100, and does entropy gating cause class starvation?

Our additional METER diagnostics at B=100B=100 do not support severe class starvation as the main explanation and suggest slow replacement of stored examples as a possible cause.

PolicyAccuracy at B=100 (%)
DualFIFO84.340
Entropy only83.885
Redundancy only82.318
RECAP82.541

To assess class starvation, we compared the class counts in the 25 long-bank slots, which are nearly the same across policies. Moreover, entropy-only loses 0.46 points versus DualFIFO, whereas redundancy-only loses 2.02 points, indicating that the degradation cannot be attributed to entropy gating alone.

The replacement diagnostics provide a possible explanation. Only 21.67% of admitted RECAP candidates remain after the same update, and mean long-bank age is 2,106 steps versus 89 for DualFIFO. This suggests that limited replacement within each class may contribute to the small-budget failure, although the observations do not establish causality.


Q2. What protects critical boundary examples during eviction?

RECAP retains a close same-class neighbor of the removed example, but this does not guarantee protection of critical boundary examples. Within the most represented class, it removes only one member of the closest pair. This preserves similar evidence when the two examples provide overlapping information.

Mathematically, Equation (10) bounds the information lost by deletion under the same-class redundancy assumption. This does not require actual loss to increase monotonically with distance. However, the correlations from -0.126 to +0.024 show that distance alone is a weak ranking signal.

To test the deletion rule empirically, we used 11 chronological METER snapshots per budget. Each rule removes one row from the same starting context and selected class. The resulting context is held fixed for the next 500 queries. Values below are mean NLL differences, with positive values favoring RECAP.

BFIFO NLL minus RECAPRandom NLL minus RECAP
100-0.00856-0.00331
500+0.00144+0.00161
1000+0.01378-0.00020

RECAP's deletion is worse at B=100B=100 and gives small gains at B=500B=500. At B=1000B=1000, the advantage over FIFO depends on one snapshot, with the difference becoming -0.00037 when it is omitted. These tests identify settings where the deletion rule helps, but do not establish general protection of boundary examples.

Thank you again for your feedback. To investigate your questions, we added METER ablations and diagnostics examining class starvation and context replacement, as well as direct deletion comparisons against FIFO and random removal. These results are included under Q1 and Q2 in our response above. We would appreciate it if you could take a look and share any remaining concerns.

Reviewer_ahzeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studied the problem of labeled context selection for tabular stream learning. The key idea is to keep one short bank and one long bank of the selected labeled contexts (stream samples). The selection strategy is the following: when the short bank overflows, its oldest sample becomes a candidate for the long bank; if a tabular foundation model has high prediction entropy (high uncertainty) on this sample, it is admitted to the long bank; when a long bank overflows, a sample far from the centroid (in terms of the sample representations) is evicted from the bank. The experiments on several tabular stream datasets demonstrate that the proposed greedy strategy outperforms existing context selection methods in terms of the final prediction accuracy of the TFM.

Critical strengths

S1. The problem of context sample selection for a tabular foundation model in a stream learning setting is important and practical.

S2. The idea of the proposed greedy sample selection method is interesting. Basically, two greedy selection strategies are provided to manage two context banks in a reasonable way.

S3. The experimental results are generally positive.

Critical weaknesses

W1. The theoretical analysis in Section 3 is too general, and the purpose of doing such analysis (with lots of assumptions, lemmas, etc.) is unclear.

W2. The theoretical analysis of the performance of the proposed context selection strategy is missing. It is unclear how the greedy strategy performs in worst cases, since no one knows the future information.

W3. Some claims in Section 3 are not easy to understand, e.g., what is "local feature region"?

W4. The choice of the datasets needs to be justified. Are the selected datasets challenging? Notably, the performance improvement of the proposed solution on these datasets is marginal;

W5. In a stream learning setting, it is important to ensure the selection method does not incur too much additional cost. What is the empirical time cost of the proposed sample selection method with a large budget on B (e.g., 1000), compared with baselines?

Questions for authors

Q1: Please clarify the purpose of the theoretical analysis in Section 3 and discuss how it guarantees the effectiveness of the proposed greedy strategy.

Q2: Are there any extreme cases (regarding the stream data characteristics) that the proposed greaddy context selection strategies fail?

2 comments

We thank the reviewer for recognizing the importance of context selection and for the questions about its theoretical justification, evaluation, and computational cost.


W. Theoretical analysis, dataset selection, and computational cost.

The purpose of Section 3 is to relate recency, uncertainty, and redundancy to the information retained labels provide about near-future labels. Under explicit assumptions, it motivates using recent observations to approximate that objective, considering label entropy for admission, and removing examples with close same-class neighbors.

The "local feature region" NiN_i is a fixed neighborhood around xix_i, such as a feature-space ball, and αt(i)\alpha_t(i) is its near-future probability mass. The theorem assumes local entropy consistency and label coherence within this region. These are analytical quantities, while the policy uses observed features and model scores.

Our dataset selection builds on benchmarks used in prior TFM stream-learning research [1], adding NOMAO and synthetic Agrawal streams. Table 7 covers six real-data benchmarks and synthetic drift, with 3 to 118 features and 2 to 10 classes.

As for the improvement magnitude, Table 2 shows a modest mean gain of 0.31 percentage points over DualFIFO with the backbone and budget matched. Although the mean gain is modest, the additional analyses show benefits when feedback is noisy or the predictive relationship changes. On NOMAO, the accuracy gain over DualFIFO reaches 2.42 points under 10% label noise (Table 11), while mean recovery time under gradual drift decreases from 1,073 to 988 steps (Table 4(b)). These benefits matter because a stream learner must continue making predictions while incorporating imperfect feedback and adapting to changes that make past observations less useful.

At B=1000B=1000, the policy update remains a small part of computation. The METER measurements below are mean ms per query under four concurrent workers per GPU.

BudgetRECAP updateDualFIFO updateRECAP model
1000.2010.15245.40
5000.3990.36952.39
10000.6290.62671.80

The model column covers context preparation and prediction. Separately, Appendix C.5 reports matched median end-to-end latencies of 31.48 and 31.43 ms for RECAP and DualFIFO. Direct nearest-pair search costs O(m2d)O(m^2d) per admitted overflow, where m≤BL+1m\leq B_L+1 is the selected class's candidate count and dd is the feature dimension.


Q1. Does the theoretical analysis guarantee the effectiveness of the greedy policy?

The analysis provides conditional information bounds, but not a worst-case performance guarantee for RECAP's greedy policy. Prediction-time entropy is a proxy for the conditional label entropy in the bound. Likewise, the redundancy assumption motivates close-pair removal but does not provide a performance guarantee for the class and endpoint selection rules. The centroid only selects which member of the closest pair in the most represented class is removed. We will clarify these distinctions in Section 3. Table 2 tests the resulting policy empirically.


Q2. Under what stream conditions can the policy fail?

The policy may fail when nearby same-class examples carry different predictive information, a shift makes retained examples less useful, or uncertainty mainly reflects irreducible noise.

In our experiments, an observed failure occurs on METER, where RECAP trails DualFIFO by 1.80 and 0.77 points at B=100B=100 and 500500. The additional ablation below examines B=100B=100 with the backbone, bank split, and stream order fixed.

PolicyAccuracy at B=100 (%)
DualFIFO84.340
Entropy only83.885
Redundancy only82.318
RECAP82.541

At this budget, ten classes share 25 long-bank slots, and the redundancy-only variant performs worse than entropy-only. Class counts remain similar across policies, but only 21.67% of admitted RECAP candidates remain after the same update, and mean long-bank age is 2,106 steps versus 89 for DualFIFO. These observations suggest limited replacement within classes as a possible explanation for the small-budget failure, although they do not establish causality.


Reference

[1] Lourenço, A., Gama, J., Xing, E. P., and Marreiros, G. In-context Learning of Evolving Data Streams with Tabular Foundational Models. KDD, 2026.

Thank you again for your feedback. In our response above, we clarified the scope of the theoretical analysis and provided runtime measurements across context budgets, along with additional METER ablations and diagnostics examining the small-budget failure. We would appreciate it if you could take a look and let us know whether these clarifications and results address your concerns, or if any points need further explanation.

Reviewer_xvpqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RECAP, a context management technique designed for applying tabular foundation models (TFMs) to data streams. This approach combines a buffer of recent examples with selective retention (based on predictive entropy) alongside an eviction mechanism that accounts for redundancy. The authors evaluate their approach by comparing it against both traditional stream-learning methods and existing context management policies.

Critical strengths

The paper demonstrates a good understanding of TFMs and their potential in the context of learning from data streams. The development around retaining recent examples and preserving potentially useful examples from further back in the stream is an old idea (in its general sense) in streams, but also a sensible and intuitive one that is good to adapt in the modern era.

The empirical evaluation is relatively extensive, including a number of comparisons and ablations and analyses.

Critical weaknesses

The problem statement is insufficiently motivated and inadequately specified. My main concern is not the proposed method but -- why this problem is important, and what specifically makes it challenging in a way that TFMs are necessary to resolve such challenges.

The introduction begins with "Tabular stream learning studies supervised prediction from examples generated and observed sequentially as a data stream". This is a strange and non-rigorous characterisation. Tabular stream learning is a learning setting or task, not a subject that 'studies' prediction. More importantly, the statement does little to establish the particular problem addressed in this paper.

The introduction continues "As examples arrive, predictions must be made before their labels become available, under bounded memory and real-time response requirements." Making predictions before labels are observed is a standard aspect -- why should we ever need to make predictions after labels are observed? Memory and response-time constraints are also generic considerations, unless a specific number of MB or seconds is precised.

The authors should immediately establish a rigorous problem definition in the paper, supported by motivations of either real-world applications or (if such a real-world setting does not exist) potential theoretical breakthroughs.

Questions for authors

How does 'on data streams' relate to 'stream learning', and what is a concrete scenario concerned by this work? In particularly, a streaming scenario in which existing context management approaches fail, and how the proposed method addresses this failure?

3 comments

We thank the reviewer for the feedback.


W. Problem definition, motivation, and resource constraints.

Regarding the problem definition, the task is already formally defined in Section 2.3. Equation (1) specifies online context updates for a frozen TFM using only currently available information, subject to a budget of BB labeled examples. The goal is to improve subsequent predictions through the retained context. We will bring this definition into the introduction to state the task directly, and clarify that prediction before label feedback describes the standard prequential protocol.

Regarding the motivation for studying TFMs on streams, they incorporate new labels through context updates without task-specific weight updates. The resulting challenge is deciding which past observations remain useful as the stream changes and the history exceeds context capacity. Lourenço et al. [1] showed that FIFO contexts can make TFMs competitive stream learners and identified diversification and redundancy removal as open directions. We investigate these directions without changing the pretrained model or increasing its context capacity.

For the memory and response-time constraints, our explicit context budget is B=1000B=1000 retained labeled rows by default. Appendix C.5 separately reports model and workspace memory, with matched median latencies of 31.48 ms for RECAP and 31.43 ms for DualFIFO. We will replace the generic reference to real-time requirements with the context budget and measured timing conditions.


Q. Relation to stream learning and a concrete streaming scenario.

Regarding terminology, "on data streams" refers to the online stream-learning protocol in Section 2.3. We will use "stream learning" consistently.

For a concrete streaming scenario, consider weather prediction, motivated by the NOAA task in our evaluation. Suppose a rain/no-rain classifier confidently predicts a common rain pattern but is uncertain about a less frequent one. Repeated observations of the common pattern can enter DualFIFO's long bank and displace older examples of the less frequent pattern. If that pattern becomes common later, useful evidence may already have been lost.

To address this loss of useful examples, RECAP uses uncertainty recorded at prediction time to admit examples of the less frequent pattern and filter confident repetitions once the long bank is full. Among retained rain examples, repeated observations of the common pattern may form the closest pair. When the rain class is selected for eviction, RECAP removes one of those repetitions, allowing the distinct older examples to remain. Admission thus limits unnecessary replacements, while eviction can preserve different patterns within the same class.


Reference

[1] Lourenço, A., Gama, J., Xing, E. P., and Marreiros, G. In-context Learning of Evolving Data Streams with Tabular Foundational Models. KDD, 2026.

Thank you to the authors for answering my question and providing further precision. I acknowledge that there is a clear problem statement, but I still do not see what is stream-specific (as opposed to simply 'future examples'). Streams have typically been understood as high-speed, but speed comparisons does not seem to be a major focus of the paper either; the number of instances treated seems very low to be considered as a natural streaming problem. Other methods in comparison have not received the same 'window' treatment as the proposed method, so it feels like an unfair comparison (and tables of results indicate this too; how else can a jump from e.g., to 99.60 from 84.05 be explained). There is some good work in this paper, but I think it needs a major reframing. I keep my current rating.

Thank you for taking the time to read our response and explain your remaining concerns.

What makes our setting stream-specific is the need to continually revise a limited context as the data distribution changes. Processing speed is important, but temporal dependence and concept drift are also central to stream learning [1]. Examples that previously supported accurate predictions may become less useful after a shift, so the learner must repeatedly decide which observations to retain without knowing what comes next. Our abrupt, gradual, and recurring drift experiments directly evaluate whether context selection helps a frozen TFM adapt to these changes (Table 4(b)).

To address the concern about data scale, our experiments process between 18,159 and 829,201 examples per stream (Table 7). Each stream is processed sequentially in its original order, while the retained context remains capped at 1,000 examples. The 1,000-example limit specifies how much evidence can be retained at once, rather than how many examples are evaluated. The task is therefore to maintain a useful context throughout a much longer sequence, which requires repeated decisions about retaining and replacing observations.

To examine your concern about window treatment, we additionally refit ARF on the exact context retained by DualFIFO. We chose DualFIFO because it uses the same window structure and context budget as RECAP, with a 750-example short bank and a 250-example long bank. This lets us examine ARF's performance when its training examples are restricted to that window, whereas the original comparison used its standard online updates.

The additional experiment covers the full POKER stream, retaining the original ARF hyperparameters and 100-example scoring warm-up. Before each query, we initialize a fresh ARF and train it on exactly the examples retained by DualFIFO.

MethodPOKER accuracy
ARF, standard online updates84.05%
ARF refit on the DualFIFO context77.68%
TFM + DualFIFO — Table 298.87%
RECAP — Table 299.60%

Applying this window treatment yielded lower ARF accuracy in the additional run than the original reported mean. Our evidence for the contribution of selective retention comes from the separate comparison in Table 2, where RECAP and DualFIFO share the same TFM backbone and bank capacities. Their 98.87%-to-99.60% comparison shows a gain of 0.73 percentage points while holding the predictor and window structure fixed.

Together with the drift experiments, this matched comparison supports our focus on bounded context management for frozen TFMs. These results show that choosing which observations to retain can improve stream prediction without changing the pretrained model or increasing its context budget.


Reference

[1] Read, J., and Žliobaitė, I. Learning from Data Streams: An Overview and Update.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.