- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates bounded context management for frozen Tabular Foundation Models (TFMs, e.g., TabICL, TabPFN) in data stream classification. Instead of updating model parameters or decision tree splits, the model adapts via in-context learning by dynamically updating a fixed-capacity labeled context. The authors introduce a future-information view based on conditional mutual information to justify three context retention signals: recency, uncertainty, and redundancy. Based on this framework, the paper proposes RECAP (REtained Context for Adaptive Prediction). Experiments on seven streams compare RECAP against six classical stream learners (e.g., ARF, SRP, VFDT) and an existing dual-FIFO buffer policy (DualFIFO).
Critical strengths
- Interesting formulation. Reframing stream adaptation as bounded in-context memory management for frozen tabular foundation models is a timely and promising research direction.
- Clear writing. The paper is well-organized, with detailed appendices covering proofs, dataset statistics, and diagnostic analyses.
Critical weaknesses
- Marginal empirical improvements over DualFIFO. The primary contribution of RECAP is supposed to be its selective retention policy (entropy gating + redundancy eviction) over naive FIFO retention. However, when evaluated under the exact same TFM backbone and context budget (Table 2), the actual improvement over DualFIFO are marginal (90.60% VS 90.68%, 90.45% VS 90.52%).
- Table 3(a) examines RECAP across different context capacities () and bank allocations (), exposing serious performance degradation. This demonstrates that RECAP is fragile. It only slightly outperforms FIFO when the context budget is large ( ), but fails when memory is constrained. This directly undermines the motivation of bounded context management for resource-limited streams.
Questions for authors
- Why does RECAP degrade so significantly compared to DualFIFO when (−1.80% on METER)? Does entropy gating cause severe class starvation when the buffer is small?
- In Table 14, the distance loss correlation is negative on three out of four streams (e.g., −0.126 on AGR(A)). If raw feature Euclidean distance does not predict information loss upon eviction, what is the actual mathematical and empirical mechanism preventing RECAP from discarding critical boundary samples?
2 comments
Thank you again for your feedback. To investigate your questions, we added METER ablations and diagnostics examining class starvation and context replacement, as well as direct deletion comparisons against FIFO and random removal. These results are included under Q1 and Q2 in our response above. We would appreciate it if you could take a look and share any remaining concerns.
We thank the reviewer for the questions about the magnitude of the gains, performance at small budgets, and the eviction mechanism.
W1. Magnitude of the gains over DualFIFO.
The improvements over DualFIFO are modest in the main comparison, but they extend across streams and also appear under noisy feedback and concept drift. With the backbone and budget matched, Table 2 shows gains of 0.07 to 0.73 percentage points, averaging 0.31 across seven streams. All seven gains are positive at the fixed threshold (exact paired Wilcoxon p=0.015625), and six remain positive with prefix-only threshold selection.
In the additional analyses, the NOMAO accuracy gain reaches 2.42 points under 10% label noise (Table 11), while mean recovery time under gradual drift decreases from 1,073 to 988 steps (Table 4(b)). These benefits matter because a stream learner must continue predicting when feedback is noisy or predictive relationships change.
W2. Performance under limited memory.
The small-budget degradation occurs on METER, while RECAP improves over DualFIFO on NOMAO and AGR(A) at the same budgets. At B=100, the gains on NOMAO and AGR(A) are 0.33 and 0.19 points, respectively (Table 3(a)). We examine the METER failure directly below.
Q1. Why does RECAP degrade at B=100, and does entropy gating cause class starvation?
Our additional METER diagnostics at B=100 do not support severe class starvation as the main explanation and suggest slow replacement of stored examples as a possible cause.
To assess class starvation, we compared the class counts in the 25 long-bank slots, which are nearly the same across policies. Moreover, entropy-only loses 0.46 points versus DualFIFO, whereas redundancy-only loses 2.02 points, indicating that the degradation cannot be attributed to entropy gating alone.
The replacement diagnostics provide a possible explanation. Only 21.67% of admitted RECAP candidates remain after the same update, and mean long-bank age is 2,106 steps versus 89 for DualFIFO. This suggests that limited replacement within each class may contribute to the small-budget failure, although the observations do not establish causality.
Q2. What protects critical boundary examples during eviction?
RECAP retains a close same-class neighbor of the removed example, but this does not guarantee protection of critical boundary examples. Within the most represented class, it removes only one member of the closest pair. This preserves similar evidence when the two examples provide overlapping information.
Mathematically, Equation (10) bounds the information lost by deletion under the same-class redundancy assumption. This does not require actual loss to increase monotonically with distance. However, the correlations from -0.126 to +0.024 show that distance alone is a weak ranking signal.
To test the deletion rule empirically, we used 11 chronological METER snapshots per budget. Each rule removes one row from the same starting context and selected class. The resulting context is held fixed for the next 500 queries. Values below are mean NLL differences, with positive values favoring RECAP.
RECAP's deletion is worse at B=100 and gives small gains at B=500. At B=1000, the advantage over FIFO depends on one snapshot, with the difference becoming -0.00037 when it is omitted. These tests identify settings where the deletion rule helps, but do not establish general protection of boundary examples.