- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose a new method for KV compression of tabular foundation models (TFMs). They first run joint K-Means to learn key and value prototypes. As a second step, they use ridge regression to calibrate value prototypes: full attention output is the target and initial (from K-Means) value vectors serve as anchors. Using TabICLv2 on a subset of CC-18 (classification) datasets, the authors show that their method outperforms other KV compression baselines, both in accuracy and compute time / cost.
Critical strengths
- Ridge regression correction. The second step of correcting value prototypes via ridge regression is interesting and appears novel. Empirically. the authors show its benefit by comparing to the naive weighted joint-KV clustering.
- Low serving cost. The authors show that their method scales better than competitors (CRUMB, MICP, KATE) on a given subset of datasets, both in terms of performance and time/cost.
Critical weaknesses
While the original idea is novel, I find the empirical evaluation to be lacking.
- No true long-context datasets. The largest dataset is <10,000 samples. The recent state-of-the-art models, e.g. TabPFNv3.5, Kumo, Causilo, Limix-2, have scaled to 1M rows.
- Evaluations are limited to TabICLv2 and classification. To claim generality, it would be great if the authors could additionally evaluate with more backbones. I also don't understand why the authors have chosen to limit the evaluation to classification.
- Dataset choice. The authors have evaluated on a subset of 64 out of 72 CC18 datasets. They provide no justification for this choice.
- Statistical significance. The authors don't report re-running on the same dataset with different seeds / splits. K-Means can be sensitive to initialization. Hence it's hard to judge the stability and significance of the obtained results.
Minor comments:
- Figure 3, panel C, the authors only show the scaling curves for KATE, CRUMB, and MICP. It would be great if they can additionally add LUCos, weighted joint-KV, and input K-Means to the plot.
Questions for authors
See above. Overall, my concerns are that the method has been evaluated on small datasets, where KV-compression is not even justified.