- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper investigates how to identify low-quality in-context demonstrations for tabular foundation models. It proposes TICDA, a simple method that extracts internal model representations, fits a ridge regression surrogate, and computes demonstration influence scores using closed-form expressions. The authors evaluate TICDA’s computational efficiency on 38 classification datasets. They also explore applications to context curation, cross-model transfer, and active learning.
Critical strengths
[S1] The paper addresses an important and challenging problem: identifying low-quality in-context demonstrations at low computational cost.
[S2] The method is simple and computationally inexpensive, requiring only a ridge regression fit on extracted TFM representations.
Critical weaknesses
[W1] As shown in Table 1, TICDA’s average detection performance across 38 datasets on TabDPT is significantly lower than that of DemoShapley (0.79 vs. 0.91).
[W2] The experiments cannot be reproduced from the information provided. The paper does not specify the names of the 38 datasets, their sizes, or how the in-context demonstrations are constructed.
[W3] TICDA uses an additional labeled validation set that is neither counted toward the labeling budget nor used by the baselines. Thus, the active learning comparisons are unfair.
Questions for authors
[Q1] TICDA relies on an attribution surrogate and the fixed-representation simplification in Equation (6). The authors should provide theoretical explatation or empirical experiments clarifying how these two approximations affect agreement with the TFM’s LOO influence in Equation (1), particularly its magnitude and sign.
[Q2] Please provide the per-dataset results underlying Table 1 to clarify whether the findings are consistent across datasets.
4 comments
Answer to reviewer f69q (2/2)
[...] The authors should provide theoretical explatation or empirical experiments clarifying how these two approximations affect agreement with the TFM's LOO influence [...]
In short, the two approximations largely preserve the ranking of the demonstrations, which is what Tables 1–2 measure, but they do not preserve the magnitude of the TFM's LOO effect, and the self-score carries no sign information. TICDA's scores are therefore ranking scores, not calibrated estimates of the LOO effect. For the applications of the paper, the rank agreement is what matters, and not the magnitude: label-error detection ranks the demonstrations by , and context curation removes the lowest-ranked demonstrations under a fixed budget (Section 3.1). We will state this limitation explicitly in Section 2.3 and in the conclusion, and the new Appendix F of the revised paper reports the experiment below.
Theory: Appendix B.2 shows that, inside the surrogate, the self-score is nonnegative and never exceeds the exact deletion effect, and that for another query the exact loss change can be positive when the first-order influence is negative, but never the reverse. These properties give no guarantee about the effect of the TFM, because the representations change when a demonstration leaves the context and the surrogate loss differs from the loss of the TFM (Appendix B.3).
Experiment: We therefore measured the magnitude and the sign against the native effect, that is, the TFM's own change of log loss when a demonstration is deleted, which is the reference of Tables 1–2. To compare magnitudes in one unit, we evaluate the ridge surrogate with a log loss. Table 8 of the revised paper (Appendix F.3) reports two estimates of this surrogate, one with fixed representations and one after the TFM has encoded the reduced context again. For each, it gives the error relative to the size of the native effect, so that predicting no change gives an error of 1, and the sign agreement with the native effect. As a reference for the sign, it also gives the agreement of a score that is positive for every deletion, the always-positive baseline.
Sign: Most native self-deletions raise the loss, so a score that is always positive already agrees with the TFM in 93–100% of the deletions. is such a score: it is nonnegative by construction, and it cannot reproduce the negative native effects, which make up the remaining 7% of the TabICL deletions and 1% of the TabDPT deletions at the submitted sizes. Neither of the two estimates exceeds this baseline in any row of Table 8.
Magnitude: The surrogate recovers little of the magnitude of the native effect: with fixed representations, its errors of 0.74–0.98 are only moderately below the error of 1 that predicting no change gives. Recomputing the representations lowers the error by 0.04–0.19 in every row but one, and the errors of 0.61–0.93 that remain are due to the surrogate, including the uncalibrated probability mapping.
Ranking: By contrast, the Spearman correlation of with the native effect is 0.64–0.79 for TabICL and 0.76–0.83 for TabDPT across the four context sizes (Table 7, last column).
[...] Please provide the per-dataset results underlying Table 1 to clarify whether the findings are consistent across datasets.
In the revised version, we added Appendix E.1 with a table that gives the per-dataset results underlying Table 1, for both TabDPT and TabICL. For each dataset, the table lists its OpenML ID, the labeling error detection AUC of all methods, and the faithfulness of each method.
The per-dataset results are consistent with the averages in Table 1:
TabDPT, detection.TICDAandDemoShapleyare within 0.05 AUC of each other on 31 of the 38 datasets.TabDPT, faithfulness.DemoShapleyhas higher faithfulness thanTICDAon 36 datasets. UnlikeTICDA, it re-evaluates the TFM on eight permuted contexts, which makes it two to three orders of magnitude slower (Figure 2).TICDAstill keeps a faithfulness above 0.7 on 32 datasets, and is aboveIGandDETAILon all 38.TabICL, detection.TICDAhas a higher AUC thanDemoShapley,IGandDETAILon all 38 datasets, and is aboveLOOon 31 of them.TabICL, faithfulness.TICDAhas the highest faithfulness on average (0.67 vs 0.65 forDemoShapley), and is higher thanDemoShapleyon 21 of the 38 datasets. This way, the findings are consistent across datasets.
References
[1] Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmüller, Prateek Mutalik Desai, David Salinas, and Frank Hutter. TabArena: A Living Benchmark for Machine Learning on Tabular Data. NeurIPS, 2025.
Thank you for your response. It addresses my concerns. I will raise my score to 3.
Dear reviewer, Thank you for your comment and updating your score. Kind regards,
Answer to reviewer f69q (1/2)
We thank reviewer f69q for their insighful comments. Please find our answer below:
Let us kindly clarifiy this result. The 0.79 vs 0.91 gap concerns faithfulness, not detection. On labeling error detection, which is the practical task,
TICDAandDemoShapleyare tied onTabDPT(0.88 vs 0.88). On faithfulness forTabDPT,DemoShapleydoes score higher thanTICDA, but, it is roughly 1000 times more expensive to compute thanTICDA, as shown in Figure 2.All 38 datasets are public. They are the 38 classification datasets of the
TabArenabenchmark [1], and each one can be downloaded from OpenML with its ID. We follow theTabArenapreprocessing and evaluation protocol and limit each context to at most 4,096 demonstrations.We thank the reviewer for this important and insightful comment. We reran the experiment on 36
TabArenadatasets. We also fixed the protocol so that our approach and its competitors have exactly the same labeling budget: we removed the access to validaiton set labels.TICDAnow scores each candidate from the self influence of its addition, based on theTFMpredicted label. The new results are below:TICDAvsentropyTICDAvsrandomWith the same labeling budget for every strategy,
TICDAachieves a higher area under the accuracy learning curve (AULC) than bothentropyandrandomsampling. This shows that data attribution can be used to build a competitive acquisition strategy for TFMs. We added these experiments in the updated version of our paper.