acceptodds
Under review as a conference paper at ICLR 2027

Selecting Feature Interactions for Generalized Additive Models by Distilling Foundation Models

Abstract

Identifying meaningful feature interactions is a central challenge in building accurate and interpretable models for tabular data. Generalized additive models (GAMs) have shown great success at modeling tabular data, but often rely on heuristic procedures to select interactions, potentially missing higher-order or context-dependent effects. To meet this challenge, we propose TabDistill, a method that leverages tabular foundation models and post-hoc distillation methods. Our key intuition is that tabular foundation models implicitly learn rich, adaptive feature dependencies through large-scale representation learning. Given a dataset, TabDistill first fits a tabular foundation model (TFM) to the dataset, and then applies a post-hoc interaction attribution method to extract salient feature interactions from it. We evaluate these interactions by then using them as terms in a GAM. Across tasks, we find that interactions identified by TabDistill lead to consistent improvements in downstream GAMs' predictive performance. We additionally introduce TabDistill-Proxy, a more efficient version of TabDistill that pretrains a probe on TFM attention scores. It selects interactions that roughly match the accuracy of TabDistill at one tenth of the cost. Our results suggest that TFMs can serve as effective, data-driven guides for interaction discovery, bridging high-capacity models and interpretable additive frameworks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Generalized Additive Models, Interaction Detection, Tabular foundation models, Model Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cjYKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper addresses the interpretability-performance tradeoff between inherently interpretable models such as GAMs and black box models such as Tabular Foundation Models (TFMs). The paper proposes TabDistill, a sparse interaction selection procedure that is based on TFMs that allows GAMs to include a small number of interactions to improve their predictive performance compared to additive main effect GAMs. To select a sparse set of interactions, the authors suggest to fit a TFM (they choose TabPFN in their experiments) to the data. Then, they apply SPEX to each held-out observation to estimate (approximate) Faith-Banzhaf interactions up to an order k using a sparse Fourier representation. They rank and select interacting subsets for the final GAM based on their frequency of occurence. Further, they propose a more efficient variant, namely TabDistill-Proxy, which fits TabPFN to the residuals of an additive GAM and uses attention-based interaction detection instead of SPEX.

Critical strengths
  • The paper addresses a relevant problem: sparse selection of useful interactions to improve the predictive performance of interpretable models.
Critical weaknesses
  1. The proposed approach seems unnecessarily complicated and not intuitive. a) Important baselines that directly fit an interpretable model with sparse and/or low-order interactions are missing, e.g., GAMI-Net, SIAN, Random Planted Forests, SKIM-FA [1–4]. The paper does not provide any theoretical or empirical justification that fitting a foundation model and extracting its interactions is preferable to directly fit an interpretable model. b) I do not see why interactions found via SPEX for the TabPFN should generally be useful for adding to a GAM. The SPEX interactions rom TabPFN could also be model-specific or spurious. The paper does not provide a convincing theoretical or empirical justification for it.

  2. Chosen explanation method computationally expensive and experimental evaluation not supporting main claims. a) The goal is to select globally relevant interactions, but instead the authors apply SPEX on individual observations and aggregate the results afterward which seems computationally expensive. There are already methods for detecting or quantifying global interactions in black-box models, such as VIN, Friedman's H-statistic, ALE [5–7]. Since the suggested approach is agnostic to the explanation method, I would have expected at least a comparison with some of these approaches, especially regarding computational cost and the relevance of the selected interactions. I also would have expected a justification of why SPEX was chosen over other interaction detection methods (also missing in related work). Also, since Shapley interactions / SPEX are model-agnostic, it is not clear, why it should be applied to a foundation model rather than other models that work well on tabular data such as Random Forest or XGBoost. b) While the ground-truth interactions are known in the synthetic experiments, the paper only evaluates predictive performance. This was surprising to me, since the paper is mainly about interaction discovery. An evaluation of whether the correct interactions are actually recovered is missing. Also, the ML baselines are mostly evaluated with default hyperparameters, and the Fourier-based synthetic functions closely relate to the assumptions motivating SPEX, which feels a bit constructed in favor of their own method. Most of the empirical validation only considers datasets with fewer than 10 features, where exhaustive / greedy interaction selection is still feasible. Although they also report results on datasets with up to 100 features, they do not separate results by dimensionality. Thus, it is not clear if the approach is beneficial in higher-dimensional settings, where interaction selection becomes more challenging. Given the computational cost of the procedure, I would also expect an evaluation of how its runtime scales with the number of features.

[1] Yang, Z., Zhang, A., & Sudjianto, A. (2021). GAMI-Net: An Explainable Neural Network Based on Generalized Additive Models with Structured Interactions. Pattern Recognition.

[2] Enouen, J., & Liu, Y. (2022). Sparse Interaction Additive Networks via Feature Interaction Detection and Sparse Selection. Advances in Neural Information Processing Systems.

[3] Hiabu, M., Mammen, E., & Meyer, J. T. (2020). Random Planted Forest: A Directly Interpretable Tree Ensemble. arXiv:2012.14563.

[4] Agrawal, R., & Broderick, T. (2023). The SKIM-FA Kernel: High-Dimensional Variable Selection and Nonlinear Interaction Discovery in Linear Time. Journal of Machine Learning Research.

[5] Hooker, G. (2004). Discovering Additive Structure in Black Box Functions. Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.

[6] Friedman, J. H., & Popescu, B. E. (2008). Predictive Learning via Rule Ensembles. Annals of Applied Statistics.

[7] Apley, D. W., & Zhu, J. (2020). Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models. Journal of the Royal Statistical Society: Series B.

Questions for authors
  • Why are interactions aggregated based on how frequently they are identified across observations? This could potentially miss strong interactions that are relevant only in a small part of the feature space. Why not consider their magnitude or contribution to predictive performance?

  • How exactly are masked features treated when computing the interaction scores? In particular, how does the approach account for dependencies between features and potentially unrealistic feature combinations?

2 comments

Weakness 1. The motivation for TFM-guided interaction selection

Transferring pretrained knowledge into interpretable GAMs. Our goal is not to replace existing interpretable models, but to improve the interpretability of TFMs and to improve the interaction selection by leveraging knowledge acquired through TFM pretraining. Unlike models trained solely on the target dataset, TFMs can benefit from cross-task inductive biases, potentially identifying useful interactions in low-data settings.

Empirical justification and limitations. We also evaluate multiple interpretable backbones, including TabDistill with EBM, PyGAM, SNAM, and SIAN (Table A4). Our contribution is demonstrating their downstream predictive ability.

Weakness 2. Computational cost and experimental validation

Motivation for SPEX and TFMs. We choose SPEX for its model-agnostic higher-order interaction attribution and sparse Fourier recovery. TFMs provide pretrained knowledge beyond the target dataset. We acknowledge that comparisons with alternative global interaction measures (VIN, H-statistic, ALE) and conventional teachers (Random Forest, XGBoost) would further isolate these contributions.

Computational cost and scalability. The main computational bottleneck arises from repeated per-sample TFM queries rather than exhaustive enumeration over feature dimensions. Although our main experiments focus on smaller datasets to manage computational cost, Section 4.3 evaluates eight datasets with 10–100 features.

Synthetic evaluation and baseline configuration. We use mostly default hyperparameters to maintain a consistent and reproducible evaluation protocol across baselines. We acknowledge that this does not establish superiority over fully tuned baselines. Importantly, our Fourier-sparse experiments evaluate TabPFN's predictive ability, rather than SPEX's interaction recovery, so SPEX does not directly benefit from the Fourier-based construction in these comparisons. Furthermore, Appendix A.4 evaluates non-smooth tree-structured interactions, providing additional evidence beyond Fourier-sparse functions.

Q1. Frequency-based aggregation for global interaction selection

Population-level interaction relevance. Our intuition is that interactions with broad relevance across the data distribution should be consistently identified across observations. Frequency-based aggregation prioritizes such interactions while reducing sensitivity to unusually large attribution values from individual samples. This aligns with our goal of selecting a small set of globally useful interactions for interpretable GAMs.

Q2. Feature masking and dependencies

Reference-based feature masking. We replace masked numerical features with their means and categorical features with their modes. This provides a consistent reference for SPEX attribution. However, the masking procedure does not explicitly preserve dependencies between features and may produce unrealistic combinations. We acknowledge this limitation and will clarify that the resulting interaction attributions are conditional on the chosen masking scheme.

I thank the authors for their rebuttal, however, it does not address my concerns in the review appropriately, therefore I remain my score.

Reviewer_NArJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to select a small set of feature interactions for accurate and interpretable generalized additive models (GAMs), addressing the limitations of heuristics that can miss higher-order dependencies. It proposes TabDistill, which fits a tabular foundation model (TFM), uses SPEX with the Faith-Banzhaf Interaction Index to extract important interactions, and incorporates the selected interactions into a downstream GAM. TabDistill-Proxy uses TabPFN attention and a Global Learner trained on separate calibration datasets to select pairwise interactions at lower cost. Experiments report improved EBM prediction relative to FAST and RuleFit, with additional evaluations on larger datasets, another TFM, and synthetic tasks.

Critical strengths
  • TabDistill achieves better average MAE and F1 ranks than FAST and RuleFit across the regression and classification benchmarks, supporting its usefulness for GAM interaction selection.
  • TabDistill-Proxy retains much of TabDistill's predictive benefit while reducing average measured runtime on binary classification tasks, making interaction selection more practical.
Critical weaknesses
  • The paper does not compare SPEX/FBII selection from a TFM with the same procedure using a conventional teacher, leaving the specific benefit of foundation models unclear.
  • TabDistill and its proxy have much higher reported runtimes than FAST. Comparisons with additional higher-order selectors under matched runtime budgets are missing, leaving the practical value of their predictive gains unclear.
Questions for authors
  1. Can you compare downstream EBM performance using the same SPEX/FBII procedure with a TFM and a conventional teacher?
  2. How do TabDistill and its proxy compare with additional higher-order selectors, such as iRF [1] or NID [2], in downstream EBM performance under matched total runtime budgets?
  3. How does restricting TabDistill to pairwise interactions affect performance at the same interaction budget?
  4. The synthetic studies assess TabPFN's predictions. Can you directly evaluate TabDistill's recovery of the generating interactions to validate the claim of meaningful interaction discovery?

[1] Iterative random forests to discover predictive and stable high-order interactions. PNAS. 2018.

[2] Detecting Statistical Interactions from Neural Network Weights. ICLR. 2018.

2 comments

We thank the reviewer for the constructive feedback.

Re 1: Motivation for using pretrained TFMs. We agree that SPEX/FBII is model-agnostic and can also be applied to conventional teachers. Our motivation for using TFMs is their ability to leverage prior knowledge acquired through pretraining across diverse tasks, whereas conventional models primarily learn interaction structures from the target dataset alone. We hypothesize that such pretrained knowledge can provide useful guidance for interaction selection.

Re 2: Model-agnostic interaction discovery. Unlike iRF and NID, which rely on random forest structures and neural network weights, respectively, TabDistill uses model-agnostic interaction attribution that can be applied to different TFM architectures without requiring access to their internal parameters. Our evaluation therefore focuses on interaction selection methods within a common downstream EBM framework, using FAST and RuleFit as baselines.

Re 3: Performance under pairwise interactions restriction. Our existing experiments provide indirect evidence through TabDistill-Proxy, which selects only pairwise interactions, whereas TabDistill can select higher-order interactions. As shown in Figure 5, TabDistill-Proxy achieves comparable downstream F1 and accuracy to TabDistill.

Re 4: Interpretability and downstream utility. We agree that our synthetic experiments primarily evaluate TabPFN's ability to learn functions containing interaction structures. We also included a real-world Fiat 500 case study (Section 4.6) that provides complementary evidence of meaningful interaction discovery. TabDistill identifies a three-way interaction among longitude, latitude, and vehicle model type, revealing distinct geographical pricing patterns across vehicle types (Figure 7).

Thank you for the response. However, it does not provide additional evidence to address the concerns raised in my review. I will therefore keep my score.

Reviewer_Uur6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the problem of distilling higher-order interactions for interpretable GAM models in the modern era of tabular foundation models. The paper follows a typical procedure of fitting a blackbox model, in this case a tabular foundation model, followed by applying post-hoc attribution methods to distill the most important feature interactions. These interactions are then

Critical strengths

Strengths

  • The problem of extending higher-order selection to tabular foundation models is of great interest, not only for better understanding TFMs as a new blackbox learning algorithm, but also as a powerful tool for selecting the most relevant interactions for a given dataset.

  • Experiments are conducted across a large set of synthetic and real-world datasets to confirm generality of the code.

  • The TabDistill-Proxy method using attention scores to match TabDistill rankings in a fraction of the time is a major win, conditional on TabDistill being an effective selection mechanism.

  • Comparisons over a variety of interaction attribution methods are included for a top-8, n=500 setting and several methods FBII, FSII, STII, and Mobius are generally found to be better performing than alternatives.

Critical weaknesses

Major Weaknesses

  • The difficulty of higher-order selection is not discussed sufficiently. The choice of datasets with 10 or fewer features vastly simplifies the problem of third-order interactions, reducing it from cubic scaling to (103)=120\binom{10}{3} = 120. Additionally, assuming k=3k=3 was not mentioned until the experiments, and only mentioned for the synthetic data. Comparison with other higher-order selection procedures for GAMs [A,B,C] are not included in the paper.

  • It is a little confusing what the major contribution of the paper is purported to be. Because higher-order selection has been taken on by previous works, it is not novel here. Because performance comparisons with previous state-of-the-art interpretable models are not included, it is not claiming to be interpretable or GAM SOTA. Because TabPFN is shown to outperform TabDistill EBM on synthetic data (real data not included?), it is not claiming to be overall SOTA. Therefore, it is hard to understand what exactly this paper contributes. According to the introduction, it should be that TabDistill+SPEX improves over the 2013 method of FAST for EBM GAMs, and TabDistill-Proxy further speeds this up. Is this the major contribution?

  • Figure 7 needs to be improved significantly. As you are probably aware, the higher-order interaction is only interpretable if we can visualize the shape function. In the case of this figure, that requires looking at both figure 7a and 7b together. This is not possible currently. I highly suggest adding the other hexagons which correspond to landmass of Italy with a neutral color like gray. This will allow the reader to find the corresponding hexagons to compare. Further, there is also no discussion on the discretization procedure used to convert continuous coordinates into discrete hexagons. This is especially critical and concerning for how rough the shape function seems to be. If you find the plotting to be misleading, you might need to increase the density of hexagons, i.e. reduce the hexagon size.

Other Weaknesses

  • Figures 2,3,4 are relatively low value. The number of interactions selected are extremely low due to the dataset assumption that there are 10 or fewer features, making it unclear how good TabDistill is or is not in general. Furthermore, the varying number of features also makes these curves somewhat incomparable across datasets, and how much the variance of the method's ranks depends on the individual dataset performance vs. the distillation method is not discussed in the main paper.

  • Generally, I expect more focus on the shape functions for a GAM paper beyond a single real-world dataset.

[A] "Higher-order neural additive models: An interpretable machine learning model with feature interactions" Kim et al. ICDM 2025.

[B] "Sparse Interaction Additive Networks via Feature Interaction Detection and Sparse Selection" Enouen et al. NeurIPS 2022.

[C] "Tensor Product Neural Networks for Functional ANOVA Model" Park et al. ICML 2025.

Questions for authors

Major Questions

  • Algorithm 1 seems to imply that all of the typical difficulties of higher-order selection are handled by the SPEX algorithm. Can you give some more details about how the SPEX algorithm is called and automatically handles typical challenges like feature heredity and the exponential scaling of higher-order interactions?

  • What are the major contributions of this work?

Minor Questions

  • Would it not be more appropriate to have the final interactions be tested on a foundation model like [D]? Sometimes boosted forests and neural networks detect different interactions, and your abstract claims that tabular foundation models learn a rich set of interactions through representation learning which are not existing in those previous ML methods. Won't EBM models learn these distilled interactions incorrectly? Are there limitations to applying GamFormer with a fixed interaction set?

  • Why were the chosen datasets only having 10 features? Why was this not mentioned as a simplifying assumption?

  • Why were datasets averaged across different number of feature counts (p) and different number of samples (n)? Doesn't this invalidate any conclusions drawn from these curves?

  • Does the runtime vary across datasets? What are the main contributors to increased runtime of the method?

[D] "GAMformer: In-Context Learning for Generalized Additive Models" Mueller et al. 2026.

5 comments

We thank the reviewer for the constructive feedback.

MW 1: Interpretability-driven design. We agree that restricting the main experimental datasets to at most 10 features simplifies interaction selection. However, our goal is to select a small number of useful and interpretable interactions, rather than exhaustively explore high-dimensional interaction spaces. We intentionally restrict the maximum interaction order to k=3k=3, since higher-order shape functions become increasingly difficult to visualize and interpret. Even with a limited candidate space, identifying the most useful interactions under a constrained budget remains nontrivial. We acknowledge this dimensionality limitation. Unlike prior works [A–C], which primarily focus on higher-order additive modeling or interaction selection, our work investigates transferring TFM-derived interactions to a separately trained downstream GAM. We will include this discussion in the manuscript.

MW 2 & MaQ 2: Paper contribution. Our contribution is investigating whether interactions learned by pretrained TFMs can improve downstream interpretable GAMs. Interactions important to a teacher model are not necessarily useful to a separately trained GAM, making this transfer a distinct research question. TabDistill establishes this framework, evaluates its downstream utility against conventional interaction selection methods, and introduces a proxy-based implementation to reduce attribution costs.

MW 3: Figure 7 visualization. We agree that higher-order interactions require careful visualization. We will update Figure 7 to improve the presentation and interpretation of these interaction effects.

OW 1 & MiQ 3: Low number of interaction sets. A small interaction budget is intentional for interpretable GAMs, as including too many interaction terms would reduce interpretability. Furthermore, rankings are computed within each dataset before aggregation, which mitigates differences in absolute performance scales across datasets and allows meaningful average-rank comparisons.

OW 2: Shape-function analysis. We agree with the reviewer that our current work does not provide a systematic analysis of the learned shape functions. Our primary focus is on interaction selection and its downstream predictive utility. We will clarify this scope and discuss the limitations of our current shape-function analysis.

MaQ 1: Feature heredity and exponential scaling in SPEX. Regarding feature heredity, SPEX [D] does not require lower-order Fourier terms to be identified before recovering a higher-order term. Therefore, the recovery procedure does not impose a heredity constraint on the selected interactions.

Regarding scaling, under the sparse, low-degree setting analyzed by Kang et al. [D, Appendix A.5], SPEX achieves a computational complexity of O~(sdp)\widetilde{O}(sdp), where pp denotes the number of features, ss the Fourier sparsity, and dd the maximum Fourier degree. This avoids exhaustive enumeration of O(pd)O(p^d) candidate interactions.

MaQ 2: See MW 2 & MaQ 2.

MiQ 1: EBM instead of GAMformer. GAMformer's higher-order extension [E] models pairwise interactions by introducing feature products xixjx_i x_j and learning univariate shape functions over these products. In contrast, EBM can directly learn nonparametric interaction functions fij(xi,xj)f_{ij}(x_i,x_j) and fijk(xi,xj,xk)f_{ijk}(x_i,x_j,x_k). This provides greater flexibility in representing complex interaction patterns that cannot be expressed through functions of feature products.

MiQ 2: Low number of features in the datasets. The restriction to at most 10 features applies only to our main experiments, motivated by interpretability considerations and computational cost. Importantly, we already evaluate TabDistill on additional datasets with 10–100 features and up to 4,000 samples (Section 4.3, Figure 3). TabDistill continues to outperform baseline methods in these settings, demonstrating its applicability beyond the low-dimensional datasets used in the main experiments.

MiQ 3: See OW 1 & MiQ 3.

MiQ 4: Runtime variation across datasets. Yes, runtime varies across datasets, primarily due to differences in sample size, as SPEX requires per-sample queries to the TFM. We report runtime results from our main experiments with the interaction selection budget fixed at eight. Normalizing total runtime by dataset sample size, the estimated per-sample runtime ranges from 0.55 to 6.94 seconds for TabDistill and from 0.005 to 2.89 seconds for TabDistill-Proxy. We will include this discussion in the manuscript.

Due to the page limit, we have provided the references in a separate comment.


[A] Kim, M., Choi, H.-S., and Kim, J. "Higher-Order Neural Additive Models: An Interpretable Machine Learning Model with Feature Interactions." ICDM 2025.

[B] Enouen, J., and Liu, Y. "Sparse Interaction Additive Networks via Feature Interaction Detection and Sparse Selection." NeurIPS 2022.

[C] Park, S., Kong, I., Choi, Y., Park, C., and Kim, Y. "Tensor Product Neural Networks for Functional ANOVA Model." ICML 2025.

[D] Kang, J. S., et al. "SPEX: Scaling Feature Interaction Explanations for LLMs." ICML 2025.

[E] Mueller, A., et al. "GAMformer: Exploring In-Context Learning for Generalized Additive Models." NeurIPS 2024 Workshop.

I thank the authors for their careful response.

MW1 & MiQ2. Although the argument to restrict to k=3 or lower for interpretability reasons is standard, I do not think that the argument that "identifying the most useful interactions under a constrained budget remains nontrivial" holds up well for choosing amongst <45 interaction pairs and <120 interaction triples.

OW 1 & MiQ3 & MaQ1. Seeing that SPEX is using 'heredity-type' techniques to automatically handle the O(pd)O(p^d) scaling, it should mean that it is not difficult to scale to p>10p > 10.

MW2 & MaQ2 & MiQ1. I think it is hard to imagine that interactions learned by blackbox models would not be useful for downstream GAMs, and this pipeline has at least been demonstrated for multilayer perceptrons in [B]. I think the research question would be better qualified if it was checking whether the interactions important to the foundation teacher model are most useful to a foundation GAM (Gamformer [E]), neural GAM (NAM), or boosting GAM (EBM).

I am not sure if the authors are aware, but the PDF does not look updated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.