- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the General Additive Models (GAMs). They propose a new approach based on diffusion modeling, called Diffusion Additive Models (DAMs). Given the difficulties of learning feature contributions under the GAM setup (only the sum of contribution is observed), this paper introduces a framework to learn feature distributions. The authors discuss a 4 step process for posterior matching, and provide numerical experiments on regression datasets.
Critical strengths
The method proposed overall performs well numerically. The numerical experiments are overall expansive and cover several benchmark datasets which is welcome.
Critical weaknesses
-
The method is not well-motivated. There are several assumptions and choices made throughout the paper (the 4 step procedure, the data distribution assumptions, etc) that are not supported by evidence. The authors provide some limited ablation studies, but overall the algorithm design is not principled.
-
The numerical experiments expand several benchmarks, but do have some blind spots. For example, the interpretability results are not expanded upon, only several plots are presented in the appendix with no explanation. The authors can at least for one or two datasets discuss the interpretation behind why certain features have low or high contribution. The authors also do not compare with non-diffusion GAM methods, such as tree-based methods (see EBM). While a perfect comparison might not be possible between classical GAMs and DAM, some illustrative experiments will be appreciated. The paper also lack discussion on inference efficiency of DAM.
Questions for authors
Could the authors please expand upon the points raised above?
- More discussion on interpretability
- Results on inference efficiency
- Comparison with EBM
4 comments
The authors can at least for one or two datasets discuss the interpretation behind why certain features have low or high contribution.
We can unfortunately not do that. Additive models in general give associations, not causal relationships. Hence, we do not know why certain features have low or high contributions. We only know that they do. This again follows common practice in previous papers.
The authors also do not compare with non-diffusion GAM methods, such as tree-based methods (see EBM). While a perfect comparison might not be possible between classical GAMs and DAM, some illustrative experiments will be appreciated.
This is incorrect. Included non-diffusion additive model baselines are NAMLSS, gamboostLSS, GAMLSS and DAM_moments.
We do not think that adding EBM provides any additional value here. Please note that the official implementation (InterpretML) learns only the mean of the distribution, not the variance or any other distributional parameters. Hence, it cannot be directly compared to any of the included LSS (location, scale and shape) baselines. The included gamboostLSS already serves as a strong boosting baseline, albeit it does not use trees as a baselearner. The subject of our paper is about proposing a generative additive model, not a competition between neural-based versus tree-based models. After all, we could parameterize the drift of DAM with tree-based models as well [1], but this would just complicate the current setup and make the comparison to the included baselines more difficult. Lastly, the value of EBM mainly comes from automatically detecting feature interactions. However, this an orthogonal improvement to DAM and all included baselines. Ultimately, feature interactions could be added to any of these models to increase performance.
[1] Generating and Imputing Tabular Data via Diffusion and Flow-based Gradient-Boosted Trees. https://arxiv.org/abs/2309.09968
The paper also lack discussion on inference efficiency of DAM.
We added inference times for the DAM variants and CARD in a separate table and discuss them in a subsection in the results. DAM (ODE) needs on average 4.2 minutes of inference time versus CARD that needs 2.95 minutes. As we mention in the conclusion, this is a downside DAM inherents from the iterative generative process of diffusion models. Existing improvements that greatly accelerate diffusion model sampling, such as distillation or few-step sampling, are also applicable to DAM. We leave such applications for future work.
Thank you for your comments. I maintain my score.
Please let us know about any remaining concerns.
We thank you for your time investment and thoughtful remarks. Please find our answers and comments below.
We respectfully disagree. We mention in the introduction that existing additive models require a parametric assumption on p(y∣x) and do provide feature-wise uncertainty. This lack of ability motivates our model. We will clarify the motivation further in the paper.
The assumptions of additivity in the likelihood and conditional independence of the feature contributions follow the established additive model literature (see, e.g., [1]).
The four step procedure is an arrangement for expositional clarity not an algorithm in itself. Each step is principled. Step 1: Calibration based on a linear model is a principled mean and variance estimation. Step 2: The closed-form posterior is derived using principled Bayesian inference. Step 3: We minimize what the generative processes have to learn, which again relies on a principled mean and variance estimation. Step 4: We derive the feature-wise generative processes and corresponding losses using established and principled theory on diffusion and flow matching models.
We acknowledge that there is some confusion about the notion of interpretability (also raised by other reviewers). With interpretability we refer to the established inherent interpretability of additive models [2]. This stems from their additive decomposition of feature-wise functions fi(xi) that only act on a single feature xi. We essentially model y=α+∑ifi(xi)+ϵ, where α is an intercept and ϵ a feature-independent noise term. In this model, the total effect of xi can be summarized by plotting xi on the x-axis and fi(xi) on the y-axis for the entire support of xi. This is what our figures illustrate: For any given xi value, you directly see its contribution to the model prediction y. While existing models can provide a similar average total effect (plotted as the model-specific mean functions), DAM is the first to provide the full distribution of that effect.
This also makes clear that there is no obvious quantitative metric for additive interpretability. Instead, following the standard practice in previous papers (see, e.g., [1], [2], [3]) we show qualitatively in figures 2, 3 and 4 that DAM comes relatively close to the true unimodel feature contributions but does not recover multimodal ones (caused by the limits of the closed-form posterior).
Lastly, collaboration between feature-specific SDEs changes the model significantly to y=α+∑ifi(x1,x2,…)+ϵ, such that each feature contribution fi is now (implicitly) a function of all features. This leads to a complete collapse of the notion of interpretability explained above: We can no longer discern the total effect of feature xi and visualize it in a plot, as it is completely mixed in with the effects of other features. Hence, there is no quantitative evaluation necessary nor possible: Collaboration leads to a total loss of interpretability following the estbalished notion of interpretability in additive models.
[1] Neural Additive Models for Location Scale and Shape: A Framework for Interpretable Neural Regression Beyond the Mean. https://arxiv.org/abs/2301.11862
[2] Neural Additive Models: Interpretable Machine Learning with Neural Nets. https://arxiv.org/abs/2004.13912
[3] gamboostLSS: An R Package for Model Building and Variable Selection in the GAMLSS Framework. https://arxiv.org/abs/1407.1774