- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents SuffiKV - a per-task adaptation method that makes the inference of TFMs much faster while preserving or improving the prediction precision. SuffiKV builds a set of K and V and decouples the inference of a TFM into 2 steps: step 1: build K and V from the context, step 2: read K and V to make predictions. Unlike other methods that select a subset of the context, or build abstract rows, SuffiKV acts at the attention stage. The benefit is it decouples the routing and the information.
Critical strengths
The idea is interesting, the paper presents many experiments and ablation studies. The "claimed results" look good, especially the speed.
Critical weaknesses
The evidence does not support the claims.
- SuffiKV uses labels for training. It is unclear whether the gains in the performance come from the training or from the efficient compresion, or a mix of both. The paper's argument is that the memory can merge redundant rows, downweight noisy ones, reserve slots for rare classes and decision boundaries, and adjust calibration. That’s plausible, but it’s asserted rather than shown by ablation studies.
- It’s unclear the advantage of decoupling K and V.
- The results do not support the claims:
- The paper only reports mean performance. Do the gains come from just a few datasets or homogeneously? win/tie/loss counts should also be reported
- The +1.5 AUC number hides the spead: it’s an average of +2.7 (LimiX) and +0.2 (TabPFN-2.6), on a self-curated benchmark. Gains for TabPFN on regression is zero, and slower at prediction on TabArena-Reg.
- The results of TuneTables look broken: on TabArena-Reg, TuneTables performs far worse than TabPFN v2.6 and use more memory.
- Scaling with context size: The “TFMs saturate” story really holds for LimiX only, the performance TabPFN v2.6 keeps growing when the context grows
- The rank analysis: the rank result is uninformative. A Gaussian noise matrix can still have high rank.
- Linear probes: a linear classifier trained on the final embeddings scores higher on SuffiKV’s embeddings than on full context’s. That’s expected: SuffiKV’s state was trained on labels, so its final features align with them
Questions for authors
- Can you add an ablation study to Table 7: Task-rerm + Op-term (no Label refinement). The objective is to test whether the gain in the performance comes from finetuning for from compression. Additionally, if time permits, is it possible to add an additional baseline: full context plus the same label-trained memory, with no compression. If it matches SuffiKV’s accuracy, the gain is from training.
- Weakness 2: if time permits, is it possible to report an ablation study with a hidden-summary (Z) version at equal parameter count.
- Can you justify the claim that SuffiKV “can merge redundant examples into shared slots, reduce the influence of noisy or irrelevant rows”. Otherwise you should relax that claim.
- Can you justify the advantage of decoupling K and V?
- The paper only reports mean performance. Do the gains come from just a few datasets or homogeneously? win/tie/loss counts should also be reported
- Can you explain why SuffiKV is slower than TabPFN v2. on TabArena-Reg? is it because there are many small datasets?
- Can you explain why TuneTables performs that bad on TabArena-Reg? Is it because the pipeline was broken?
- Can you define a scenario where the speed up during inference worth the cost of the training of SuffiKV?
Thank you for recognizing our approach's appeal, extensive experiments and ablations and promising speedups. We appreciate your concerns about the evidence and the chance to clarify our rationale. We welcome further questions and will respond promptly.
Q1. Ablation.
Q2. Hidden summary and decoupled K/V.
Q3. Compression and denoising.
Q4. Win, tie and loss counts.
Q5. TabArena-Reg.
Q6. Applicable scenarios.
Q7. Rank analysis.
Q8. Context size and probes.