acceptodds
Withdrawn from ICLR 2027

Native Local Linear Explanations for Tabular In-Context Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Tabular in-context models adapt to unseen tasks without task-specific retraining, yet their cross-sample attention entangles feature representations into opaque predictions. We introduce TabLLE (Tabular Learning with Local Linear Explanations), an intrinsically interpretable in-context model that recasts support-conditioned prediction as dynamic, query-specific affine hyperplanes. Conditioned on query samples and column-level support sets via a Feature-Context Adapter (FCA), an adaptive hypernetwork generates instance-specific weights and biases in a single forward pass. For the parameters generated in the same forward pass, each class logit decomposes into an exact sum of coordinate-indexed local-linear terms and a sample-dependent additive offset, yielding native instance-level attributions without perturbation-based post-hoc evaluation. To prevent the hypernetwork from bypassing feature pathways via bias shortcuts, TabLLE is trained with a three-term objective that explicitly enforces standalone feature discriminability. Across 20 OpenML benchmarks, TabLLE is competitive with tree ensembles and leading in-context baselines, with 1.5–5.15 faster inference than TabICLv2 at matched retrieval sizes, while attaining the highest AUC and accuracy among intrinsically interpretable baselines. Finally, we formalize parameter drift under feature interventions and show that a behavioral head reliably accounts for cross-pass masking effects.

Primary area:
interpretability and explainable AI
Keywords:
Tabular In-Context Learning, Tabular Foundation Models, Intrinsic Interpretability, Local Linear Explanations, Feature Attribution, Hypernetworks, Meta-Learning, Tabular Classification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XndJOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces TabLLE, a tabular in-context learning model that generates a query-specific affine classifier conditioned on a labeled support set. Each class logit is decomposed exactly into feature contributions and a bias term. A three-term training objective combines full-prediction supervision, feature-only supervision, and bias regularization to discourage predictive shortcuts through the bias. The paper evaluates prediction on 20 OpenML datasets, studies the discrepancy between frozen-parameter contributions and changes under feature masking, and introduces an auxiliary behavioral head to approximate the latter. It also reports faster inference than TabICLv2 under matched local-support configurations. The intended contribution is to combine competitive tabular prediction with native, inexpensive local explanations.

Critical strengths
  1. A useful architectural direction with competitive predictive results. Generating support-conditioned affine outputs makes the final logit decomposition directly available during inference. The evaluation covers 20 classification datasets and a broad range of predictive baselines; the reported AUC IQM of 0.916 is competitive with the tree ensembles in Table 1. This makes the architecture worth investigating as a way to combine tabular in-context learning with explicit prediction decompositions.

  2. The paper exposes an important limitation of its own explanations. The parameter-drift analysis distinguishes frozen-parameter contributions from actual changes after re-inference. Reporting a median drift ratio of 1.42 and the weaker native attribution results is informative. This analysis identifies a substantive problem for query-dependent linear explanations, even though the proposed remedy does not yet establish the native explanations' faithfulness.

Critical weaknesses
  1. The native explanations perform substantially worse than LIME and SHAP on attribution faithfulness.

    • The attribution results show large gaps from standard explainers. Table 2 reports native Faithfulness of 0.263, compared with 0.516 for LIME, 0.678 for SHAP, and 0.687 for Kernel SHAP. Native Infidelity is also worse: 7.343 versus 4.525, 4.211, and 4.210, respectively. These metrics assess whether attributions reflect actual output changes under perturbation, a central requirement for explaining model behavior. Although the authors acknowledge parameter drift and limitations of masking-based evaluation, exact logit decomposition does not resolve the observed discrepancy. These gaps substantially weaken the paper's interpretability contribution.

    • The behavioral head only partially addresses this concern. Its Faithfulness improves to 0.599, exceeding LIME but remaining below SHAP and Kernel SHAP, while its Infidelity of 4.198 is comparable to both SHAP variants. These results concern a separate explainer trained on masking effects and do not validate the native coefficients used in prediction. The improved auxiliary output therefore does not establish the quality of the method's claimed native explanations.

  2. The paper does not establish an efficiency benefit over LIME that offsets the loss in native explanation quality.

    • The native explanations fall behind LIME on Faithfulness and Infidelity in Table 2, but the paper provides no runtime comparison with LIME. The 1.50x-5.15x speedups in Table 3 compare prediction against TabICLv2, not explanation against LIME. Although native attributions require no additional perturbation passes, the practical time savings are not quantified. The paper therefore does not demonstrate that the loss in attribution quality is offset by a meaningful computational advantage, limiting its contribution as an explanation tool.
Questions for authors
  1. How does TabLLE differ from GAMformer and LFR-IMN? Please compare TabLLE with GAMformer (Mueller et al., 2026) and LFR-IMN (Hammer et al., 2026) in architecture, training objective, and explanation definition. Please add a comparison of predictive performance and explanation quality with GAMformer, and a local-fidelity regularization control for the proposed objective.

  2. Why do the native attributions perform worse than LIME and SHAP? Please explain the large gaps between native TabLLE and LIME/SHAP in Table 2 and report uncertainty for these comparisons. Please provide evidence that the native attributions faithfully reflect model behavior, separately from the behavioral head's results.

  3. How were the standard deviations calculated, and were the low TabR accuracies checked? Appendix H describes standard deviations over 25 folds, but Table 10 reports Iris accuracy of 0.956 +/- 0.003. With 150 samples and five-fold testing, the minimum population standard deviation across 25 folds consistent with that mean is approximately 0.0155. Please provide the fold-level results and clarify whether the tables report standard deviations, standard errors, or variation across repeat-level averages. Please also explain TabR's accuracies of 0.339 on Iris and 0.323 on Wine, including whether training or implementation failures were checked.

  4. How does TabLLE's explanation efficiency compare with LIME? Please compare the explanation runtime of native TabLLE and the behavioral head with LIME under the evaluation protocol of Table 2. Report attribution quality alongside runtime, and specify the hardware and LIME sampling budget.

Reviewer_9emEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors introduce TabLLE to make the in-context model interpretable. Specifically, the proposed model consists of an encoder part similar to those in typical tabular foundation models, a feature-context adapter part that combines the column embeddings with the final representations provided by the encoder, and finally a weight/bias generator part that produces the final linear model. Experiments on real-world datasets demonstrate the effectiveness of the proposed method compared with non-TFM models. Further experiments show the interpretability and the role of each part of the proposed model.

Critical strengths
  1. The paper is generally well-written and easy to follow.
  2. The experiments are well-designed and extensive.
Critical weaknesses
  1. I am not convinced that the locally linear model is interpretable. The weights and biases of the final linear model are actually based on specific query samples. As such, I think all models, including TFMs, could generate a locally linear model for each query sample, for example, by using the gradient of the final prediction with respect to the input features. I think it would be better to show that the learned linear model is more consistent across samples, such that it could be used to explain the model's behavior for a group of samples. This could possibly be achieved by adding smoothness regularization to the weights and biases of the final linear model. I would also expect the authors to study whether the learned linear models provide this kind of global interpretation.

  2. The feature-context adapter part is not well-motivated. In my opinion, the final representation HH alone already contains enough information for the final linear model. I would expect the authors to provide some discussion of why the feature-context adapter part is needed and how it helps the final linear model. I think an ablation study on the feature-context adapter part would be helpful in demonstrating its effectiveness.

  3. In the experiments, the authors report a hierarchical explanation on a real-world benchmark. Is this result consistent with the explanations provided by the intrinsically interpretable models listed in the paper? I would expect the authors to provide some discussion of this.

Questions for authors

See the weakness part.

Reviewer_xXBSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, authors propose TabLLE, an intrinsically interpretable version of a Tabular Prior-Fitted Network. Given a set of query samples, a hypernetwork outputs sample-specific weights and a bias for linear prediction. To avoid uninformative predicted coefficients for the linear model and to prevent TabLLE from learning bias shortcuts, the authors train their model using three-term objective. The authors evaluate predictive performance on 20 OpenML datasets, which is used to demonstrate that the proposed method works almost as well as TabICLv2, while outperforming its alternatives. In addition to that, authors provide evaluation on XAI benchmark, to evaluate explanations provided by their model.

Critical strengths
  1. A broad analysis of the proposed algorithm is provided. Figures 2 and 3 offer an intuitive understanding of the way that the proposed linear explanations work in practice, while subsection 4.4 offers evidence that all three components of the objective, as well as the diversity of the synthetic prior are useful.
Critical weaknesses
  1. Evaluation is very weak. 20 OpenML datasets is not a lot, especially considering how close all the methods are to each other in terms of accuracy. Additionaly, strong performances of Random Forest and Logistic Regression seem to imply that the obtained benchmark is suboptimal for evaluating tabular ML models. On XAI datasets, both SHAP and KernelSHAP demonstrate stronger results on faithfulness and feature removal.
  2. It is unclear what role do the explanations play. After all, the weights change depending on the context, and a neighbourhood in which a query falls depends on several of its features. It is unclear which is the more important part of the prediction: finding the location where to build linear model weights, or the weights themselves?
Questions for authors
  1. Have you tuned the hyperparameters for the methods in Table 1?
  2. Why were these 20 datasets chosen? Could you provide the results on the BeyondArena, or at least TabArena benchmark?
  3. Could you provide some additional experiments demonstrating the usefulness of the learned weights?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.