acceptodds
Under review as a conference paper at ICLR 2027

Interpretable by Design Tabular Prediction via In-Context Local Linear Learning

10 views

Abstract

In-context learning (ICL) based foundation models have shifted how we make predictions on tabular data: instead of training dataset-specific models, they make predictions by attending to labeled context samples. Pretrained on many tabular tasks, these models achieve strong accuracy, but their attention-based decisions are difficult to interpret in terms of feature contributions. Post-hoc methods can assign feature importance scores, yet those scores do not determine the prediction. We introduce (*abular In-Context ocal inear earning*), an interpretable-by-design predictor that uses a tabular foundation model to guide an explicit prediction rule. first learns a shared linear model from the foundation model's predictions. For each query, it retrieves similar training rows in the foundation model's representation space and fits a second linear model to the residuals of the shared model in that neighborhood. The two models together make the final prediction, and their combined coefficients determine each feature's contribution to its query-specific linear score. Across 386 classification benchmark–dataset evaluations, trails by 1.1 percentage points and beats CatBoost on 266 of them. It improves average support F1 over MAPLE by 4.7 points and leads both real-data selected-feature utility metrics.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tabular in-context learning, interpretable machine learning, tabular foundation models, local linear models, feature attributio

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_N7XFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TICL³ (Tabular In-Context Local Linear Learning), a query-adaptive predictor guided by a frozen TabICLv2 backbone. A shared ElasticNet is fitted to the backbone’s predictions, while a query-specific ElasticNet models the remaining residuals using neighbors retrieved in the backbone’s representation space. Their combined coefficients determine both the deployed linear class scores and the corresponding feature contributions. These contributions exactly decompose the query-specific scores, but not the normalized probabilities or the effects of representation-based retrieval and coefficient fitting.

Critical strengths
  1. A coherent integration of prediction and feature contributions.
  2. Broad predictive evaluation and competitive results against interpretable baselines.
  3. Promising results on selected explanation metrics.
Critical weaknesses

W1.The methodological contribution appears incremental, and the feature-selection results do not establish a consistently strong advantage.

W2.The emphasis on direct predictive performance is not matched by a sufficiently clear end-to-end value proposition. Table 3 shows that the matched TabICLv2 with one estimator achieves 0.8710 accuracy and 0.745 seconds of batch prediction time, whereas standard TICL³ achieves 0.8557 accuracy and 6.849 seconds. The fast and regional variants also remain less accurate and slower than this matched backbone. Thus, for prediction alone, the reported configurations provide neither an accuracy advantage nor a speed advantage.

W3.The real-data interpretability evaluation is too narrow to support broad conclusions. The encouraging Table 5 results are based on only 10 CC-18 datasets and 50 explained queries per dataset, using a fixed split and query sample. Although the authors state that this subset was selected before score-function tuning, the scope remains much narrower than that of the predictive evaluation. It is unclear whether the explanation-quality gains persist on TabArena-Lite, TALENT, or datasets with different dimensionality, categorical-feature composition, and task characteristics.

W4. The regression subsection is weakly connected to the central interpretability contribution.

Questions for authors

Q1. What deployment advantage should readers infer from the prediction and runtime comparisons? Given the matched one-estimator results in Table 3, could the authors clarify the intended use case in which TICL³ is preferable to TabICLv2? In particular, could they provide an end-to-end comparison against TabICLv2 paired with an efficient post-hoc explainer, jointly reporting predictive accuracy, explanation quality, one-off setup cost, and online prediction-plus-explanation latency? This would more directly justify the proposed trade-off than additional prediction-only comparisons.

Q2.Do the Table 5 conclusions generalize beyond the selected CC-18 subset? Could the authors extend the interpretability evaluation to TabArena-Lite and TALENT, or to representative, pre-specified subsets from these suites?

Q3. How does the regression extension strengthen the interpretability claim?

7 comments

We thank the reviewer for the careful review.

Compared with TabICLv2 [1] alone, TICL$^3$ trades a small accuracy loss for an exact account of each prediction’s score. We therefore compare complete prediction-and-explanation pipelines, including TabICLv2 paired with an explainer.

W1.1 — Methodological contribution

We design a simple predictor that is interpretable by design: a global linear model plus a local linear correction fitted on the query’s neighbors in a foundation model’s representation. The same coefficients predict and explain. Our main finding is that this simple rule retains near-foundation-model accuracy across diverse datasets.

It beats CatBoost [5] on 266/386 evaluations, trails TabICLv2 by 1.1 points, and beats MAPLE [3] in all four suites. Ablations support both design choices: representation neighbors improve accuracy by 2.30 points over input-space neighbors (95% CI [1.08, 3.49]); at K=30K=30, the global fit raises feature-contribution coverage from 46.9% to 100% and contribution correlation after 25% row removal from 0.36 to 0.66 (matched valid queries, nine datasets).

W1.2 — Feature selection

On real data, TICL$^3$’s selected features have the highest average utility (Table 5). With the predictor held fixed, we test how much useful information each explainer’s selected features retain. Gen-SVM measures how well an SVM using only those features reproduces the predictor’s decisions. Gen-kmeans measures how well k-means clusters using those features match the true classes. TICL$^3$ leads both, ahead of all nine post-hoc explainers. On 40 evaluations, it beats TabICLv2 + FastSHAP [6] on deletion and Gen-SVM (CIs exclude zero). In the leave-query-out comparison, LIME’s rules [2] are 19.5 and 11.7 points less accurate than TICL$^3$. LIME recovers more generating features on some synthetic tasks (Table 4). Prediction plus explanation takes 27 ms with TICL$^3$ versus 14.6 s with LIME (Table 5).

W2 + Q1 — Deployment advantage

FastSHAP estimates TabICLv2’s Shapley values; TICL$^3$ reports its own score’s terms. We compare 40 evaluations.

PipelineAcc. (%)Fit (s)Online, 50 queries (s)Fit + 50 queries (s)Deletion ↑Gen-SVM (%) ↑
TICL$^3$89.233.181.5894.770.62891.24
TabICLv2 + FastSHAP90.4718.520.08118.600.56988.35

Including fitting, TICL$^3$ is faster for 50 queries; FastSHAP is faster online. For prediction alone, one-estimator TabICLv2 is faster and more accurate (0.745 s, 87.10% vs. 6.849 s, 85.57%; Table 3).

On eight datasets with 5,000 training rows, fitting and explaining 50, 200, and 1,000 predictions takes 3.56, 5.55, and 16.23 s with TICL$^3$ versus 35.78, 35.93, and 36.80 s with TabICLv2 + FastSHAP. At 1,000 queries, accuracy is 87.33% versus 87.96%. TICL$^3$ suits frequent refits where inspecting the score calculation matters.

W3 + Q2 — Explanation quality beyond CC-18

We added ten evaluations each from TabArena-Lite, TALENT, and BeyondArena, with three seeds and 50 explained queries per run. Each pipeline explains its own predictor. Deletion checks how steadily class probability falls as features are removed.

PipelineDeletion ↑Gen-SVM (%) ↑Gen-kmeans (%) ↑Stability ↓
TICL$^3$0.62891.2468.780.0248
TabICLv2 + FastSHAP0.56988.3567.600.0165

Across 40 evaluations, paired 95% gain CIs are [0.006, 0.113] for deletion and [0.90, 5.08] points for Gen-SVM (34 valid pairs). On the 30 new evaluations, deletion is 0.662 vs. 0.595 (+0.067, CI [0.006, 0.131]); Gen-SVM is 92.25% vs. 89.00% (+3.25 points, CI [1.19, 5.61]; 24 valid pairs). FastSHAP is more stable; the clustering difference is uncertain.

Three new CC-18 splits favor TICL$^3$ on average, but CIs include zero. Deletion differences by feature count or categorical share are also uncertain. We will report these and add a class-count breakdown.

W4 + Q3 — Regression and interpretability

Regression extends the same interpretable construction to continuous outcomes: diabetes progression [7], gene expression [8], and drug properties [9]. Attribution also helps interpret climate predictions [10]. Each feature adds or subtracts an amount in the outcome’s units. After undoing target standardization, the baseline and signed contributions sum to the prediction.

Class-separating neighbors need not capture outcome magnitude; TabICLv2 conditions regression representations on continuous targets. On 119 TALENT evaluations, TICL$^3$ has 0.515 nRMSE versus 0.537 for CatBoost, 0.571 for MAPLE, 0.601 for EBM [4], and 0.482 for matched one-estimator TabICLv2 (Table 6). On ten regression datasets and three seeds, we compare feature contributions before and after changing 25% of training rows: correlation is 0.982 for removal and 0.980 for replacement. In both cases, contributions are measured relative to the original training-set mean.

[1] Qu et al. TabICLv2, 2026.

[2] Ribeiro et al. LIME, 2016.

[3] Plumb et al. MAPLE, 2018.

[4] Nori et al. InterpretML (EBM). 2019.

[5] Prokhorenkova et al. CatBoost, 2018.

[6] Jethani et al. FastSHAP, 2022.

[7] Efron et al. Least Angle Regression, 2004.

[8] Avsec et al. Enformer, 2021.

[9] Jiménez-Luna et al. Drug discovery with explainable AI, 2020.

[10] Toms et al. Interpretable neural networks for geosciences, 2020.

Thank you for the detailed response and the additional experimental results. The expanded CC-18 evaluation helps address my earlier concerns. I encourage the authors to incorporate these results into the main text, as I consider this part of the empirical evaluation to be the paper’s main contribution.

However, I do not find the response regarding the method’s weaker classification performance relative to the TabICL baseline sufficiently convincing. In particular, it remains unclear what makes the evaluation in Section 4.2 a significant contribution. Demonstrating stronger performance than tree-based models does not, on its own, establish a compelling contribution, especially when the method underperforms the relevant TabICL baseline. This is not to suggest that a contribution must necessarily achieve state-of-the-art performance, but rather that the distinct scientific insights or practical advantages demonstrated by this section need to be articulated more clearly. I would appreciate a more substantive explanation of its significance beyond the comparison with tree-based models.

Overall, while the additional CC-18 results strengthen the empirical evaluation, my concerns regarding the significance of Section 4.2 remain unresolved. I therefore maintain my current score at this stage.

Reviewer_7GbNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, authors propose a new modification for Tabular Prior Fitted Networks, specifically focusing on TabICLv2, although any in-context learning backbone would fit as well. The main advantage of the proposed model is that it is interpretable by design, since its prediction can be obtained as a linear combination of the input features. Below, I will briefly describe the method used.

  1. Authors perform cross-validation of TabICLv2 to obtain its prediction on the train examples in inference mode.
  2. Authors train an ElasticNet model to predict the predictions obtained on the previous step from the input features
  3. For each query, the algorithm finds a set of neighbours, and weights associated with each one of them.
  4. The algorithm then trains weighted ElasticNet on the residuals between TabICLv2's predictions and the prediction of the ElasticNet model from step 2.

The authors then evaluate their approach on 386 classification and 119 regression datasets. They demonstrate that their methods achieve the results close to TabICLv2's, while showing good interpretability in Table 5.

Critical strengths

I liked the experimental results related to the regional anchor variant. To me, they could lead to an interesting analysis of how TabICLv2's prediction really work, with clusterize->build linear model being shown to be a surprisingly close approximation. However, that would require the experiments to show results for different number of clusters and possibly improve the construction of the Linear Model. Additionally, if similarity in results with the full TabICLv2 was confirmed on larger datasets, that would increase the impact of the experiment.

Critical weaknesses
  1. The main weakness of this paper, is that it drives a separation between post-hoc methods and interpretable-by-design methods, such as the one proposed in this paper. However, TICL^3 fits a separate ElasticNet for each query, using similarity measures provided by TabICLv2 as weights. This makes its behaviour very similar to LIME, which the authors classify as post-hoc explanation method. In Table 4, it is clear that several post-hoc explainers, in particular LIME, achieve comparable or often better metrics describing feature selection.
  2. Weakness of evaluation: I) Using only 1600 training points severely limits the scope of evaluation, and especially favours prior-tuned networks. II) Additionally, no hyper-parameter tuning was performed for any of the baseline models, which makes it unclear how the model performs relatively to the baselines.
Questions for authors
  1. Why is it meaningful that the proposed method is interpretable-by-design as opposed to a post-hoc explainer? Is there a utility in using it instead of TabICLv2 in a practical set-up?
  2. Could you provide results with bigger datasets and tuned baselines?
4 comments

We thank the reviewer for the accurate summary and for highlighting the regional-anchor results.

TICL$^3$ is an interpretable predictor. We therefore compare it with methods that provide both a prediction and an explanation.

W1.1 — Comparison with LIME

Both methods fit local linear models, but for different purposes. LIME [2] fits a surrogate to perturbations of a query to explain TabICL [1]’s prediction. TICL$^3$ fits a correction to the global predictor’s residuals on nearby training rows. We add the correction to the global fit and use the result to predict. The coefficients that explain the score are the same ones we use to make the prediction.

We also tested LIME’s fitted rule as a predictor. With a narrow kernel, including the query can make the rule return the TabICL prediction it was given. We left the query out of the fit and kept TabICL’s predictions on the perturbed rows. We chose kernel width and regularization from 30 settings on validation data and used the same 50 test queries per dataset. Bold marks the best interpretable predictor; TabICLv2 is a prediction-only reference.

Predictor10 CC-18 (%)8 TabArena-Lite/TALENT (%)
LIME rule, default, query excluded67.2066.08
LIME rule, tuned, query excluded72.9375.08
TICL$^3$92.4086.75
TabICLv2, prediction only95.4087.75

With the query excluded, TICL$^3$ leads tuned LIME by 19.5 points on CC-18 (95% CI [8.5, 31.9]) and 11.7 on the eight additional datasets ([4.4, 19.7]). TICL$^3$ wins on 13 of 18 datasets, ties on four, and loses on one. Tuned LIME gives all-zero feature contributions on 30% of CC-18 queries and 13% on the other datasets. In the usual TabICL + LIME pipeline, TabICL still makes the prediction.

W1.2 — Feature selection

On real data (Table 5), all methods explain the same TICL$^3$ predictor. An SVM trained on our selected features agrees with the predictor on 88.6% of queries, compared with 81.3% for LIME. Gen-kmeans is 69.7% versus 60.1%, with stability of 0.032 for both.

We agree that LIME recovers the known generating features better on several synthetic tasks (support F1 in Table 4). The accuracy columns use different predictors: TICL$^3$ uses its own rule, and LIME’s selected features are passed to TabICL.

Q1 — Practical utility

If a user wants to see how a feature enters the score, they can inspect the term we used to compute it. On ten matched CC-18 datasets, predicting and explaining 50 queries takes 4.335 s with TICL$^3$ and 28.816 s with TabICLv2 + default LIME, excluding one-off setup. In a separate 18-dataset study, even at 50 perturbations, tuned LIME takes 188 ms per query versus 55 ms for TICL$^3$.

Including setup, TICL$^3$ takes 4.77 s to fit and explain 50 predictions versus 18.60 s for TabICLv2 + FastSHAP [6] (40 evaluations). FastSHAP is faster online (N7XF Q1).

W2 + Q2 — Larger training sets and tuned baselines

We evaluated eight TabArena-Lite/TALENT datasets across three seeds, using nested training sets of 400–5,000 rows and the same 400 test rows. TICL$^3$, CatBoost [5], MAPLE [3], and EBM [4] each received ten validation-selected configurations per size, followed by full-training-set refitting. We also include TabICLv2 with one estimator.

Method4008001,2001,6003,0005,000
TICL$^3$84.1885.0985.5985.9486.6887.48
CatBoost (tuned)83.4584.6985.1385.4886.0286.50
MAPLE (tuned)83.3684.4284.5685.1785.8186.38
EBM (tuned)83.5084.3484.8284.9985.2885.83
TabICLv2, prediction only84.8085.8485.9786.4487.1887.78

At 5,000 rows, TICL$^3$ reaches 87.48%, 0.30 points below TabICLv2 and 0.98 points above tuned CatBoost (95% CI [0.42, 1.63], seven wins in eight datasets). We also checked early stopping on validation data. CatBoost reaches 86.58%, and TICL$^3$ still leads by 0.90 points (95% CI [0.30, 1.58]), winning on seven of eight datasets. With the original tuning, balanced accuracy is 68.29% for TICL$^3$ and 68.40% for CatBoost. Moving from 1,600 to 5,000 rows improves TICL$^3$ by 1.54 points (95% CI [0.39, 2.88]).

Regional-rule variant

Thanks for suggesting the regional sweep. With 5,000 training rows, increasing the blended anchors from 8 to 128 raises accuracy from 84.01% to 87.36%, and nonzero contribution coverage falls from 48.6% to 20.1%. In a separate test, we used the nearest region’s residual rule on top of the global fit. With 16 regions, we reach 87.23% versus 87.27% for per-query TICL$^3$ on that panel. With 64 regions, agreement with TabICLv2 is 97.09%, compared with 87.96% for the global fit alone. We will report both sweeps.

References

[1] Qu et al. TabICLv2. ICML, 2026.

[2] Ribeiro et al. LIME. KDD, 2016.

[3] Plumb et al. MAPLE. NeurIPS, 2018.

[4] Nori et al. InterpretML (EBM). arXiv:1909.09223, 2019.

[5] Prokhorenkova et al. CatBoost. NeurIPS, 2018.

Thank you for the additional results. I am ready to raise my score from 2 to 3. However, I still don't see the utility of using TICL3TICL^3 in a practical setting; the reported improvements in speed don't seem large enough to justify performance decreases compared to the original TabICLv2 model.

Thank you for engaging in this discussion and for reconsidering the score.

TICL$^3$ can be very useful when understanding feature contributions for each prediction is part of the scientific goal. A few examples:

Drug sensitivity: Barretina et al. [7] used ElasticNet models (the same linear solver used in TICL$^3$) to identify genes associated with response to different drugs in cancer cell lines. They found that cell lines with higher expression of the gene NQO1 were more sensitive to a drug that this gene helps activate. This helped explain the biological mechanism of the drug.

Immunotherapy response: Chang et al. [8] developed a simple, explicit six-feature score combining tumor and routine blood test features. It finds additional patients likely to respond to immunotherapy, in particular patients with low tumor mutation counts.

Why per-patient rules matter. In both cases, a single model was trained across all samples. However, predictions from the same model can be made for very different reasons. For example, a cancer model may use both gene expression and blood test results, where gene expression is more important for some patients and blood test results for others. Ranking feature importance across the entire population does not explain why a prediction was made for a specific patient or subpopulation.

What TICL$^3$ provides. TICL$^3$ uses a linear rule that changes for each patient, and the signed feature contributions plus the baseline exactly add up to the class score (before clipping and normalization), so we know exactly how the score for each patient was determined.

Accuracy and speed. Our advance is that we find a simple linear rule while also providing much better accuracy (compared with a single ElasticNet or logistic regression model, or tree-based models). This can lead to more reliable scientific findings. TICL$^3$ is only 1.1 points less accurate than TabICLv2 on average across 386 classification evaluations, while MAPLE and EBM are 3.3 and 3.5 points less accurate. When LIME's local rules (fitted without the query) are used as predictors, they are 19.5 and 11.7 points less accurate than TICL$^3$ (on ten CC-18 and eight TabArena-Lite/TALENT datasets). Finally, TICL$^3$ is much faster for predicting and explaining (27 ms per query) than LIME explaining the same predictor (14.6 s per query, excluding setup, Table 5).

We have updated the Conclusion to clarify that this is one intended use of our method.

References

[7] Barretina et al. The Cancer Cell Line Encyclopedia enables predictive modelling of anticancer drug sensitivity. Nature, 2012.

[8] Chang et al. LORIS robustly predicts patient outcomes with immune checkpoint blockade therapy using common clinical, pathologic and genomic features. Nature Cancer, 2024.

Reviewer_HQTXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TICL³ (Tabular In-Context Local Linear Learning), which aims to combine the predictive capability of tabular foundation models with the interpretability of linear models by generating explicit feature contributions for each test sample. TICL³ first learns a shared linear model from the foundation model’s predictions. For each query, it retrieves similar training rows in the foundation model’s representation space and fits a second linear model to the residuals of the shared model in that neighborhood. The two models together make the final prediction, and their combined coefficients determine each feature’s contribution to its query-specific linear score. The interpretability of the method stems from the fact that the displayed feature contributions are computed from the linear coefficients actually used for prediction, enabling an exact decomposition of the current sample's prediction score. However, this explanation does not cover how context samples affect the representations, the neighbor selection, or the coefficient generation, nor does it directly decompose the final normalized probabilities. Therefore, it primarily provides sample-level feature contribution explanations rather than a complete explanation of the original ICL reasoning mechanism.

Critical strengths

The paper proposes a simple and effective method, which leverages the predictions and representations of an ICL foundation model to construct an explicitly decomposable local linear predictor. The proposed method offers clear interpretability, shows potential for extension to other ICL models, and achieves competitive performance while substantially reducing computational time.

Critical weaknesses
  1. The interpretability in this paper is mainly reflected in the contribution of the test sample's features to the current linear score, while it does not cover the role of context samples, the neighborhood selection, or the coefficient generation mechanism. Therefore, the contribution of this work is closer to ICL-guided local interpretable prediction rather than an explanation of the ICL reasoning mechanism itself. Specifically, the method selects K reference samples based on the cosine similarity of representations to fit the local residual, but the paper has not sufficiently validated why such a neighborhood is suitable for this task, nor has it established its connection to the context samples on which the original ICL prediction relies. Cosine nearest neighbors may differ from the samples that have a large influence on the original prediction. If the authors wish to emphasize the unique value of this design in the ICL context, it is recommended to add controlled ablations with different neighborhood definitions; if the authors wish to further interpret the neighbors as the decision basis of the original ICL, an influence analysis of context samples is also needed. These shortcomings limit its scope of interpretation and conclusions.

  2. This paper appears to present a simple and effective method (a global linear predictor plus a local residual corrector) in an unnecessarily complex manner. In the Method section, the paper lacks guidance that would allow readers to follow the exposition, which further amplifies the sense that the core method is merely a combination of components. This is not to say that combining methods is inherently undesirable, but rather each component should be justified by an observation. For example, why is a global linear predictor plus a local residual corrector sufficient to approximate the foundation model? (partially in Appendix D) The subsequent part of the Introduction is also very difficult to read. The layout and content of Figure 1 and Table 1 make it hard for readers to immediately understand the core contributions. The first paragraph directly presents results. In my opinion, for a paper intended to be reader-friendly, the goal should not be to showcase how impressive the results are, but to quickly let readers know what this approach can do. An example would be more appropriate here, rather than too many results, which can certainly be placed later. Moreover, the results presented in the Introduction are also a black box: the authors do not explain why other methods fail, but instead directly list a black-box summary, which is itself unexplainable.

Questions for authors
  1. Although TICL³ is trained to approximate the foundation model’s predictions, the experiments primarily report downstream predictive performance rather than faithfulness to the teacher. Similar accuracy does not necessarily imply similar predictions or probability distributions. It would strengthen the paper to report held-out probability MSE, KL divergence, and prediction agreement between TICL³ and the teacher, comparing the global-only, local-only, and global-plus-residual variants. Probability MSE is particularly relevant given the squared-loss objective and the theoretical approximation bound. These evaluations would help establish how faithfully the proposed decomposition approximates the foundation model and whether the global-plus-residual formulation offers an advantage over local fitting alone.

  2. I encourage the authors to further investigate the method’s sensitivity to context composition. With test queries held fixed, to what extent does removing or replacing context examples alter the teacher’s predictions, TICL³’s predictions, and its feature attributions? Such an analysis would help characterize the method’s dependence on context and provide a step toward understanding the utility of individual context examples in ICL.

3 comments

We thank the reviewer for noting the method’s simplicity and interpretability.

W1.1 — Scope of interpretability

We believe the main concerns come from a different reading of the paper’s goal. TICL$^3$ is a new predictor, not a new way to explain TabICL [1]. That is, our work targets attributional interpretability: we aim to build a strong, simple predictor that provides an exact explanation of how each feature contributes to its score. The role of the foundation model is to help build this predictor (providing targets for the global linear predictor and a way to measure similarity to select the neighbors). Once TICL$^3$ is fitted for a query, it provides its own prediction, and the coefficients explain the score behind that prediction before clipping and normalization.

Explaining how TabICL works internally (e.g., how it uses context samples to alter its representation of the query) is an important direction that falls under mechanistic interpretability. We describe the scope of our work in Sections 1 and 3, and have made it explicit in the revised abstract as well.

W1.2 — Neighborhood selection

Note that, for learning a strong and interpretable predictor, our method does not need to recover the same context rows that TabICL relies on. It only needs to find a neighborhood that it can fit a linear predictor to well locally. That said, following the reviewer’s suggestion, we ran a controlled ablation of how to select the neighbors. The two cosine methods differ only in the space in which we measure distance to the query. We held the backbone, reference rows, stored targets, global predictor, solver, and K=400K=400 fixed, and included random training rows as a baseline. The table reports TICL$^3$’s accuracy on the full test sets of ten CC-18 datasets.

RetrievalAccuracy (%)
TabICLv2 representation89.90
Cosine in standardized inputs88.47
Random training rows75.57

Representation neighbors beat random on all ten datasets and input neighbors by 1.43 points on average. With 100 irrelevant features, accuracy falls by 4.80 points for representation neighbors and 15.37 for input neighbors. With equal tuning across 40 evaluations, the accuracies are 89.12% and 86.82% (+2.30 points, 95% CI [1.08, 3.49]). The representation also leads on average under both distances (cosine and Euclidean) and both weighting schemes.

W2 — Presentation

We agree that the method is simpler than its presentation suggests. A global fit already agrees with TabICL on 77.5% of queries. Fitting its residuals locally raises this to 93.5%. Theorem 1 relates the approximation error to residual smoothness and neighborhood radius.

In the revised PDF, the Introduction walks through a worked prediction showing the global prediction, local correction, and resulting feature contributions. We have simplified Figure 1 and Table 1, moved the results to Section 4, and explained each component before introducing the equations. We also explain how the fits differ from LIME [2] (synthetic perturbations), MAPLE [3] (forest neighborhoods), and EBM [4] (shared feature functions).

Q1 — Agreement with TabICL

As requested, we compare the three fits on the full test sets of ten CC-18 datasets.

VariantAcc. (%)Prob. MSE ↓KL ↓Agreement (%) ↑
Global-only76.130.047640.76077.48
Local-only90.180.008360.24193.46
Global + residual89.900.009830.30093.47

Local-only has lower probability error and nearly identical agreement. Adding the global fit raises the share of queries with nonzero contributions from 70.2% to 100% (46.9% to 100% at K=30K=30, with almost identical accuracy). At K=400K=400, Gen-kmeans is 61.3% with the global fit and 55.0% without it on queries both variants explain. At K=30K=30, the global fit raises contribution correlation from 0.36 to 0.66 after 25% removal and from 0.34 to 0.63 after replacement (nine datasets).

Q2 — Sensitivity to context

We fixed up to 200 test queries per dataset on ten datasets (K=400K=400) and reserved 25% of training rows for replacements. We removed or replaced stratified subsets of 10% or 25% of the remaining context, with three draws per level, and refit both predictors each time.

InterventionTabICLv2 same class (%)TICL$^3$ same class (%)TabICLv2 prob. rTICL$^3$ prob. rAttribution rTop-10 Jaccard
Remove 10%98.1296.500.9940.9920.9480.82
Remove 25%96.9094.970.9890.9840.9040.77
Replace 10%97.6496.580.9930.9920.9470.81
Replace 25%96.7794.960.9860.9820.8940.75

After a 25% change, TICL$^3$ keeps its class on about 95% of queries, with attribution correlation at least 0.89. MNIST’s top-10 overlap is 0.24–0.29. Removing the 100 nearest rows changes TabICL’s probability for its original class by 6.6 points on average, vs. 4.2 for random rows with the same class mix. Neighbor rank is only weakly related to single-row removal effects.

[1] Qu et al. TabICLv2. ICML, 2026.

[2] Ribeiro et al. LIME. KDD, 2016.

[3] Plumb et al. MAPLE. NeurIPS, 2018.

[4] Nori et al. InterpretML (EBM). arXiv:1909.09223, 2019.

Dear Reviewer HQTX,

Thank you for the time and effort you have dedicated to reviewing our paper. As the discussion period is ending soon, we kindly ask you to look at the new results and clarifications in our rebuttal. We believe they address your main concerns, and we hope you will take them into account in your final assessment. If any concerns remain, we would be happy to provide further explanations.

Comments on the paper

1 comment

Dear Area Chair,

We propose TICL$^3$, an interpretable predictor. It fits a single linear model shared across queries and adjusts its prediction for each query using nearby training examples. The resulting coefficients are used to calculate both a prediction score and feature contributions. TabICL supplies the targets for the global fit as well as the representations used to find nearby examples.

Our goal is to explain our own prediction score by assigning contributions to the input features. Explaining how TabICL uses its context internally is a separate question of mechanistic interpretability.

TICL$^3$ outperforms CatBoost on 266 of 386 classification evaluations and trails TabICLv2 by 1.1 percentage points on average (Table 2). With the predictor fixed, the features selected by TICL$^3$ give higher Gen-SVM and Gen-kmeans scores than those selected by LIME or FastSHAP (Table 5). In that experiment, prediction and explanation take 27 ms per query with TICL$^3$ and 14.6 s when LIME explains the same predictor, both excluding setup.

Following the reviews, we tested the choice of neighborhood, the role of the global fit, and sensitivity to the training context. Across 40 evaluations, representation retrieval improves accuracy by 2.30 points over equally tuned input-space retrieval. With 5,000 training rows, TICL$^3$ is 0.30 points below TabICLv2 and beats tuned CatBoost on seven of eight datasets. Fitting and explaining 50 predictions takes 4.77 s with TICL$^3$ and 18.60 s with TabICLv2 + FastSHAP on the 40-evaluation panel. FastSHAP is faster online once fitted. The revised Introduction includes a worked prediction, and Figure 1 and Table 1 are simplified.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.