acceptodds
Under review as a conference paper at ICLR 2027

Diffusion Additive Models: Interpretable Generative Prediction through Feature-Wise Contributions

Abstract

Generalized additive models (GAMs) for location, scale and shape yield non-linear, interpretable feature-wise effects. However, they require parametric assumptions about the conditional outcome distribution and represent feature effects through deterministic functions. We introduce Diffusion Additive Models (DAMs), a generative framework for interpretable probabilistic regression and classification. Rather than specifying the shape of the conditional outcome distribution directly, DAMs treat individual feature contributions as latent variables and model their full distributions. Learning these contributions is challenging because only their sum is observed. To make this underdetermined composition learnable, we construct an encoder from a closed-form approximate posterior and calibrate its feature-wise distributions using aggregated posterior moments and homoskedastic variance routing. We then use SDE Matching to distill these distributions into feature-specific latent stochastic differential equations. DAMs capture flexible, potentially multimodal feature contributions and propagate their uncertainty to the outcome, while retaining the additive interpretability of GAMs. Across diverse regression and classification benchmarks, DAM consistently matches or outperforms existing additive models in predictive accuracy and calibration.

Primary area:
interpretability and explainable AI
Keywords:
generative model, diffusion, additive model, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_hVygOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the General Additive Models (GAMs). They propose a new approach based on diffusion modeling, called Diffusion Additive Models (DAMs). Given the difficulties of learning feature contributions under the GAM setup (only the sum of contribution is observed), this paper introduces a framework to learn feature distributions. The authors discuss a 4 step process for posterior matching, and provide numerical experiments on regression datasets.

Critical strengths

The method proposed overall performs well numerically. The numerical experiments are overall expansive and cover several benchmark datasets which is welcome.

Critical weaknesses
  • The method is not well-motivated. There are several assumptions and choices made throughout the paper (the 4 step procedure, the data distribution assumptions, etc) that are not supported by evidence. The authors provide some limited ablation studies, but overall the algorithm design is not principled.

  • The numerical experiments expand several benchmarks, but do have some blind spots. For example, the interpretability results are not expanded upon, only several plots are presented in the appendix with no explanation. The authors can at least for one or two datasets discuss the interpretation behind why certain features have low or high contribution. The authors also do not compare with non-diffusion GAM methods, such as tree-based methods (see EBM). While a perfect comparison might not be possible between classical GAMs and DAM, some illustrative experiments will be appreciated. The paper also lack discussion on inference efficiency of DAM.

Questions for authors

Could the authors please expand upon the points raised above?

  • More discussion on interpretability
  • Results on inference efficiency
  • Comparison with EBM
4 comments

We thank you for your time investment and thoughtful remarks. Please find our answers and comments below.

The method is not well-motivated. There are several assumptions and choices made throughout the paper (the 4 step procedure, the data distribution assumptions, etc) that are not supported by evidence. The authors provide some limited ablation studies, but overall the algorithm design is not principled.

We respectfully disagree. We mention in the introduction that existing additive models require a parametric assumption on p(y∣x)p(y | x) and do provide feature-wise uncertainty. This lack of ability motivates our model. We will clarify the motivation further in the paper.

The assumptions of additivity in the likelihood and conditional independence of the feature contributions follow the established additive model literature (see, e.g., [1]).

The four step procedure is an arrangement for expositional clarity not an algorithm in itself. Each step is principled. Step 1: Calibration based on a linear model is a principled mean and variance estimation. Step 2: The closed-form posterior is derived using principled Bayesian inference. Step 3: We minimize what the generative processes have to learn, which again relies on a principled mean and variance estimation. Step 4: We derive the feature-wise generative processes and corresponding losses using established and principled theory on diffusion and flow matching models.

The numerical experiments expand several benchmarks, but do have some blind spots. For example, the interpretability results are not expanded upon, only several plots are presented in the appendix with no explanation.

We acknowledge that there is some confusion about the notion of interpretability (also raised by other reviewers). With interpretability we refer to the established inherent interpretability of additive models [2]. This stems from their additive decomposition of feature-wise functions fi(xi)f_i(x_i) that only act on a single feature xix_i. We essentially model y=α+∑ifi(xi)+ϵy = \alpha + \sum_i f_i(x_i) + \epsilon, where α\alpha is an intercept and ϵ\epsilon a feature-independent noise term. In this model, the total effect of xix_i can be summarized by plotting xix_i on the x-axis and fi(xi)f_i(x_i) on the y-axis for the entire support of xix_i. This is what our figures illustrate: For any given xix_i value, you directly see its contribution to the model prediction yy. While existing models can provide a similar average total effect (plotted as the model-specific mean functions), DAM is the first to provide the full distribution of that effect.

This also makes clear that there is no obvious quantitative metric for additive interpretability. Instead, following the standard practice in previous papers (see, e.g., [1], [2], [3]) we show qualitatively in figures 2, 3 and 4 that DAM comes relatively close to the true unimodel feature contributions but does not recover multimodal ones (caused by the limits of the closed-form posterior).

Lastly, collaboration between feature-specific SDEs changes the model significantly to y=α+∑ifi(x1,x2,… )+ϵy = \alpha + \sum_i f_i(x_1, x_2, \dots) + \epsilon, such that each feature contribution fif_i is now (implicitly) a function of all features. This leads to a complete collapse of the notion of interpretability explained above: We can no longer discern the total effect of feature xix_i and visualize it in a plot, as it is completely mixed in with the effects of other features. Hence, there is no quantitative evaluation necessary nor possible: Collaboration leads to a total loss of interpretability following the estbalished notion of interpretability in additive models.

[1] Neural Additive Models for Location Scale and Shape: A Framework for Interpretable Neural Regression Beyond the Mean. https://arxiv.org/abs/2301.11862

[2] Neural Additive Models: Interpretable Machine Learning with Neural Nets. https://arxiv.org/abs/2004.13912

[3] gamboostLSS: An R Package for Model Building and Variable Selection in the GAMLSS Framework. https://arxiv.org/abs/1407.1774

The authors can at least for one or two datasets discuss the interpretation behind why certain features have low or high contribution.

We can unfortunately not do that. Additive models in general give associations, not causal relationships. Hence, we do not know why certain features have low or high contributions. We only know that they do. This again follows common practice in previous papers.

The authors also do not compare with non-diffusion GAM methods, such as tree-based methods (see EBM). While a perfect comparison might not be possible between classical GAMs and DAM, some illustrative experiments will be appreciated.

This is incorrect. Included non-diffusion additive model baselines are NAMLSS, gamboostLSS, GAMLSS and DAM_moments.

We do not think that adding EBM provides any additional value here. Please note that the official implementation (InterpretML) learns only the mean of the distribution, not the variance or any other distributional parameters. Hence, it cannot be directly compared to any of the included LSS (location, scale and shape) baselines. The included gamboostLSS already serves as a strong boosting baseline, albeit it does not use trees as a baselearner. The subject of our paper is about proposing a generative additive model, not a competition between neural-based versus tree-based models. After all, we could parameterize the drift of DAM with tree-based models as well [1], but this would just complicate the current setup and make the comparison to the included baselines more difficult. Lastly, the value of EBM mainly comes from automatically detecting feature interactions. However, this an orthogonal improvement to DAM and all included baselines. Ultimately, feature interactions could be added to any of these models to increase performance.

[1] Generating and Imputing Tabular Data via Diffusion and Flow-based Gradient-Boosted Trees. https://arxiv.org/abs/2309.09968

The paper also lack discussion on inference efficiency of DAM.

We added inference times for the DAM variants and CARD in a separate table and discuss them in a subsection in the results. DAM (ODE) needs on average 4.2 minutes of inference time versus CARD that needs 2.95 minutes. As we mention in the conclusion, this is a downside DAM inherents from the iterative generative process of diffusion models. Existing improvements that greatly accelerate diffusion model sampling, such as distillation or few-step sampling, are also applicable to DAM. We leave such applications for future work.

Thank you for your comments. I maintain my score.

Please let us know about any remaining concerns.

Reviewer_XnjmOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a variant of generative additive models that learns the distributions of individual feature-wise latent contributions using independent stochastic differential equations (SDEs). These contributions are then combined additively with Gaussian observation noise to model the conditional outcome distribution p(y∣x)p(y\mid x).

Critical strengths

The paper combines additive models with diffusion-based generative modelling to learn full distributions of feature-wise latent contributions. Unlike traditional additive models that provide deterministic feature contributions, DAM models full distributions of latent feature contributions, allowing uncertainty to be represented at the feature level.

Critical weaknesses

Baseless claims

  1. In many parts of the paper authors claim expressiveness of diffusion models ( I don't find a paper that claims this and I am not sure how do you measure "expressiveness" )
  2. Interpretability–performance tradeoff in the collaborative ablation ( throughout the paper ) particularly, In Section 5.3 (lines 465–466), the authors claim that allowing feature-wise prior SDEs to collaborate improves performance at the cost of interpretability. While Table 3 supports the improvement in average predictive rankings, the loss of interpretability is inferred solely from the removal of the independent feature-wise architecture rather than directly evaluated.

Strong writing and claims:

  1. L97- "additive models represent the gold standard of interpretability" - Completely false, completely unnecessary, I would advice authors to tone down the writing.
  2. L50 - "albeit challenging: Feature contributions are inherently latent constructs, requiring a diffusion model to be defined in the same latent space." There is a whole branch of latent diffusion, I am not sure why it is challenging.
  3. L195 - Bartosh et al. [3], improve writing.
  4. Throughout the paper, the citation format is a clear violation of ICLR policy. why don't authors stick to the citation format recommended by the ICLR.

Assumptions

  1. individual contributions are independent conditional on the features. This is a very strong assumption.

Unproven / not validated claim

L200 - "With access to y, the posterior probability mass collapses to a single z(i)z^{(i)} , setting all other contributions to zero."

Measure / Showcasing interpretability

This paper claims makes a claim about interpretability. Authors neither measure nor showcase, why such a method is more interpretable.

Additional works on Interpretability of Diffusion models

There are whole range of works that are on controllable latent space. One such work is [1], where with disentangling the latent space, authors achieve interpretability, where changing one latents leads to change in one feature in an image. This can be extended to tabular data.

[1] Hudson, D. A., et al. “SODA: Bottleneck Diffusion Models for Representation Learning.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 23115–23127.

Questions for authors

what diffusion contributes beyond a flexible one-dimensional conditional density estimator, because the structural interpretability claim does not depend on using an SDE.

6 comments

We thank you for your time and comments. Please find our response below.

In many parts of the paper authors claim expressiveness of diffusion models

The word ``expressiveness'' is mentioned exactly once, in line 49. This refers to the well-known ability of diffusion models to learn very complex continuous distributions. To provide some references, see, e.g., [1], which show that score-based generative models can efficiently sample from essentially any realistic data distribution, or [2], which shows that under certain conditions, distributions generated by diffusion models achieve the nearly minimax optimal estimation rates in the Wasserstein distance. If you are more interested in empirical results, please see [3], for an example that shows that diffusion models are more expressive than GANs for image generation.

[1] Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. https://arxiv.org/pdf/2209.11215

[2] Diffusion Models are Minimax Optimal Distribution Estimators. https://proceedings.mlr.press/v202/oko23a.html

[3] Diffusion Models Beat GANs on Image Synthesis. https://arxiv.org/abs/2105.05233


Interpretability–performance tradeoff in the collaborative ablation ( throughout the paper ) particularly, In Section 5.3 (lines 465–466), the authors claim that allowing feature-wise prior SDEs to collaborate improves performance at the cost of interpretability. While Table 3 supports the improvement in average predictive rankings, the loss of interpretability is inferred solely from the removal of the independent feature-wise architecture rather than directly evaluated.

We appreciate that you agree with the general idea of the trade-off and that this is illustrated in our results. We acknowledge that there is some confusion about the notion of interpretability (also raised by other reviewers). With interpretability we refer to the established inherent interpretability of additive models [1]. This stems from their additive decomposition of feature-wise functions fi(xi)f_i(x_i) that only act on a single feature xix_i. We essentially model y=α+∑ifi(xi)+ϵy = \alpha + \sum_i f_i(x_i) + \epsilon, where α\alpha is an intercept and ϵ\epsilon a feature-independent noise term. In this model, the total effect of xix_i can be summarized by plotting xix_i on the x-axis and fi(xi)f_i(x_i) on the y-axis for the entire support of xix_i. This is what our figures illustrate: For any given xix_i value, you directly see its contribution to the model prediction yy. While existing models can provide a similar average total effect (plotted as the model-specific mean functions), DAM is the first to provide the full distribution of that effect.

This also makes clear that there is no obvious quantitative metric for additive interpretability. Instead, following the standard practice in previous papers (see, e.g., [1], [2], [3]) we show qualitatively in figures 2, 3 and 4 that DAM comes relatively close to the true unimodel feature contributions but does not recover multimodal ones (caused by the limits of the closed-form posterior).

Lastly, collaboration between feature-specific SDEs changes the model significantly to y=α+∑ifi(x1,x2,… )+ϵy = \alpha + \sum_i f_i(x_1, x_2, \dots) + \epsilon, such that each feature contribution fif_i is now (implicitly) a function of all features. This leads to a complete collapse of the notion of interpretability explained above: We can no longer discern the total effect of feature xix_i and visualize it in a plot, as it is completely mixed in with the effects of other features. Hence, there is no quantitative evaluation necessary nor possible: Collaboration leads to a total loss of interpretability following the estbalished notion of interpretability in additive models.

[1] Neural Additive Models: Interpretable Machine Learning with Neural Nets. https://arxiv.org/abs/2004.13912

[2] Neural Additive Models for Location Scale and Shape: A Framework for Interpretable Neural Regression Beyond the Mean. https://arxiv.org/abs/2301.11862

[3] gamboostLSS: An R Package for Model Building and Variable Selection in the GAMLSS Framework. https://arxiv.org/abs/1407.1774

L97- "additive models represent the gold standard of interpretability" - Completely false, completely unnecessary, I would advice authors to tone down the writing.

We will tone done this claim. It should be interpreted within the context and narrower scope of the paper, since additive models are intrinsically interpretable so that we perfectly know each feature contribution fi(xi)f_i(x_i). We will mention in the paper that additive models are standard models to achieve feature-wise interpretability.

L50 - "albeit challenging: Feature contributions are inherently latent constructs, requiring a diffusion model to be defined in the same latent space." There is a whole branch of latent diffusion, I am not sure why it is challenging.

Latent diffusion works by learning a (V)AE to encode data into a latent space in which then a diffusion model is trained. In our setup, only yy and the xix_i are observed. A (V)AE would encode all xix_i into latents ziz_i, with goal of reconstructing the xix_i accurately. Our goal is different: (1) We want the latents ziz_i to combine additively into yy and (2) this requires the ziz_i to be predictive of yy, not xix_i. So the goal is not a reconstruction of the xix_i but the prediction of yy. Simply extending the (V)AE to condition on yy as well, i.e., with an encoder distribution q(z∣x,y)q(z | x, y), does not work either but instead leads to a posterior collapse. This relates to the modeling challenges we discus in Section 3.3. Hence, you cannot apply the typical latent diffusion model in this context. We will clarify this crucial difference in Section 2.2.

Throughout the paper, the citation format is a clear violation of ICLR policy. why don't authors stick to the citation format recommended by the ICLR.

Thank you for pointing this out. This is caused by an ambiguous phrase in the official tex file, which we apparently misinterpreted. The official ICLR 2027 tex file states:

As to the format of the references themselves, any style is acceptable as long as it is used consistently.

We thought this relates to the citation style. We have changed the citation style back to the ICLR style. We are sorry for the confusion.

individual contributions are independent conditional on the features. This is a very strong assumption.

Yes, we agree, this is a strong assumption. However, this is a fundamental assumption shared by all additive models.

L200 - "With access to yy, the posterior probability mass collapses to a single z(i)z^{(i)}, setting all other contributions to zero."

This is an empirical observation from our early experiments when starting this project. It is a form of posterior collapse similar to what can be encountered with VAEs. We will clarify this in the paper.

This paper claims makes a claim about interpretability. Authors neither measure nor showcase, why such a method is more interpretable.

Please see our explanation of interpretability in the context of additive models above. Similar to existing papers on additive models, we illustrate the interpretability using figures. With this we follow the convention of the field. There is not separate measure for interpretability: The whole function fi(xi)f_i(x_i), and therewith the entire impact of feature xix_i, can be illustrated in a figure. As you see in the figures, DAM is the only method able to provide feature-wise uncertainty. This adds to model interpretability: The practitioners now also get a decomposition of the uncertainty, not just the mean. Existing additive models cannot provide this information. In decision-making, a feature with a great contributions and low uncertainty might considered more important than a feature with the same mean contribution but large associated uncertainty. Lastly, we consider DAM to be more interpretable, since it provides the full distribution of each feature contribution. This is unlike existing additive models, which do not allow for a decomposition into feature-wise distributions. This is why we only plot the means of these models in our figures.

There are whole range of works that are on controllable latent space. One such work is [1], where with disentangling the latent space, authors achieve interpretability, where changing one latents leads to change in one feature in an image. This can be extended to tabular data.

Thank you for this reference. However, we emphasize that the proposed SODA model can not be extended to our problem. Our goal is not to interpret a diffusion model generally but the refinement of an additive model using diffusion principles. The SODA model would specify a diffusion model for yy and encode the observed xx into latents zz that condition the denoiser. Also, the SODA encoder produces latents that represent xx, but we specifically want the latents in DAM to (1) be predictive of yy and (2) decompose additively. These restrictions make the task non-trivial. For example, using a simple neural network to serve as the encoder and condition it on xx and yy leads to a posterior model collapse, as mentioned in the modeling challenges in section 3.3. Lastly, note that the disentangled space of SODA is not equivalent to an additive model: In their figure 4, it is clear that when change one latent feature, other characteristics of the image change as well. The interpretability of DAM is caused by the additive likelihood construction, closed-form posterior and feature-wise SDEs.

what diffusion contributes beyond a flexible one-dimensional conditional density estimator, because the structural interpretability claim does not depend on using an SDE.

This is correct, the structural interpretability does not depend on using an SDE. It does, however, depend on our closed-form posterior setup, which preserves a feature-wise decomposition. Our results should be interpreted more generally as opening up a new, unexplored avenue of research on generative additive models. Using diffusion models as the generative component avoids making a parametric assumption. Other flexible density estimators are an interesting direction for future work, which we will mention in the paper.

Reviewer_ZeudOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Diffusion Additive Models (DAMs), a generative framework that combines the interpretability of generalized additive models with the flexibility of diffusion-based probabilistic modeling. Unlike conventional additive models that represent feature contributions as deterministic functions or impose parametric assumptions on the outcome distribution, DAM models each feature contribution as a latent random variable with a flexible conditional distribution. To address the challenge that only the sum of feature contributions is observed, the authors develop an approximate posterior based on Gaussian conjugacy, together with variance routing and posterior moment calibration. Feature-specific stochastic differential equations (SDEs) are then trained using SDE Matching to model these latent contribution distributions.

Critical strengths
  • The paper introduces an interesting approach that combines the interpretability of generalized additive models with the flexibility of diffusion-based generative modeling. By representing individual feature contributions as latent random variables rather than deterministic functions, DAM extends conventional additive models to capture potentially complex, non-Gaussian contribution distributions.

  • A key technical challenge is that individual feature contributions are unobserved, with only their sum available during training. The authors address this underdetermined learning problem through a carefully designed approximate posterior based on Gaussian conjugacy, combined with variance routing, posterior moment calibration, and feature-specific SDE Matching.

Critical weaknesses
  • A central claim of DAM is its ability to learn interpretable distributions of individual feature contributions. However, these distributions are inherently underdetermined, and the synthetic experiments show that DAM fails to recover ground-truth higher-order structures, including skewness, heavy tails, and multimodality. This raises concerns about whether the learned distributions provide reliable feature-wise uncertainty interpretations rather than model-dependent decompositions.

  • Although DAM is motivated by overcoming the restrictive distributional assumptions of existing additive models, the regression experiments primarily compare against Gaussian-based additive baselines. More flexible alternatives, such as GAMLSS with skewed or heavy-tailed distributions and mixture-based distributional regression, are not evaluated.

Questions for authors
  • The synthetic experiments suggest that DAM's improvements mainly arise from refining the initial moment-based predictions rather than accurately recovering higher-order feature distributions. Could the authors compare DAM against a simpler approach that retains deterministic additive feature effects while modeling the remaining conditional residual distribution with diffusion or another flexible density estimator?

  • A key motivation for DAM is to overcome restrictive parametric distributional assumptions. However, the regression baselines primarily use Gaussian likelihoods, and the GAMLSS implementation further restricts log-scale effects to linear functions. Could the authors include stronger baselines using skewed, heavy-tailed, or mixture distributions?

  • The ablation results show that allowing feature-specific SDEs to collaborate improves predictive performance but sacrifices the proposed additive interpretability. However, interpretability is primarily demonstrated through visualizations of learned contribution distributions. Could the authors provide a quantitative evaluation of interpretability?

4 comments

Thank you very much for your time investment and your thoughtful comparison. Please fine below some comments and answers to your questions.

A central claim of DAM is its ability to learn interpretable distributions of individual feature contributions. However, these distributions are inherently underdetermined, and the synthetic experiments show that DAM fails to recover ground-truth higher-order structures, including skewness, heavy tails, and multimodality. This raises concerns about whether the learned distributions provide reliable feature-wise uncertainty interpretations rather than model-dependent decompositions.

This is correct, due to the underdetermined system, there is not enough information to recover the true distributions in general for any additive model, including DAM. To do so, the model would need additional information that allows to discern the feature contributions more directly. Similar to existing additive models, the learned distribution therefore already represents a model-dependent decomposition. The distribution decomposition is a direct result of the closed-form posterior, which itself depends on the first stage model. Hence, in the same manner as for existing additive models, the decomposition is determined by a feature-specific architecture. The feature-wise SDEs only learn the feature-wise distributions imposed by the closed-form posterior averaged over the data. They cannot further impact these distributions (e.g., by reshaping the probability density), and therefore cannot impact the decomposition. In fact, we found that letting the generative processes impact the decomposition leads to a posterior collapse due to the underdetermined system, as described in Section 3.3 on modelling challenges.

The synthetic experiments suggest that DAM's improvements mainly arise from refining the initial moment-based predictions rather than accurately recovering higher-order feature distributions. Could the authors compare DAM against a simpler approach that retains deterministic additive feature effects while modeling the remaining conditional residual distribution with diffusion or another flexible density estimator?

We interpret this as suggesting that we should model y=m(x)+ϵy = m(x) + \epsilon where m(x)m(x) is deterministic and ϵ∣x\epsilon | x is modeled via diffusion. This would not yield an interpretable, additive model, as the ϵ\epsilon is not decomposed into additive feature-wise attributions. Using feature-wise SDEs to learn ϵ\epsilon faces the same modeling challenges we discuss in Section 3.3, just with y~=y−m(x)\tilde{y} = y - m(x) instead of yy, since we still only observe the sum.

Indeed, our synthetic data experiments show that our closed-form posterior comes with trade-offs. One trade-off is that the true higher-order moments cannot be recovered and identified. The first two moments (for unimodal distributions) are identified by using noise-sink normalization and post-hoc centering. Due to the underdetermined system, it is unfortunately not possible to perfectly recover the whole true distributions. We will make sure to mention this in the final version of the paper.

A key motivation for DAM is to overcome restrictive parametric distributional assumptions. However, the regression baselines primarily use Gaussian likelihoods, and the GAMLSS implementation further restricts log-scale effects to linear functions. Could the authors include stronger baselines using skewed, heavy-tailed, or mixture distributions?

The reason for using Gaussian likelihoods is (1) this is the standard specification in previous papers for regression tasks (see e.g. [1]) and (2) GAMLSS can have great difficulties converging for more complex models. The latter point is also the reason why we chose linear log-scale effects for GAMLSS. Otherwise, GAMLSS would fail to converge on even more datasets. Lastly, the point of DAM is to avoid making strong parametric assumptions, so we compare it to the most popular parameterization for regression: the Gaussian likelihood. As an more complex baseline, we added NAMLSS with a four-parameter SHASHo distribution. GAMLSS and gamboostLSS models with the same distribution fail to converge on most datasets and are therefore not added. DAM remains competitive while providing better interpretability: Adding more distributional parameters leads to less interpretable baseline models. Each parameter θ\theta is modeled as g(θ)=α+∑ifi(xi)g(\theta) = \alpha + \sum_i f_i(x_i), for a link function g(⋅)g(\cdot). If the parameter is e.g. restricted to be positive, we have g(⋅)=log⁡g(\cdot) = \log, which implies that Var(y∣x)=exp⁡(α+∑ifi(xi))Var(y | x) = \exp(\alpha + \sum_i f_i(x_i)), which we can no longer decompose into additive feature-wise components. Also, more distributional parameters makes the existing additive models less interpretable because the distributional parameters interact in the likelihood. As an example: The variance of the four-parameter SHASHo distribution is a function of all four distributional parameters. On the other hand, DAM remains interpretable by providing full, feature-wise distributions that can easily be plotted and interpreted directly.

The ablation results show that allowing feature-specific SDEs to collaborate improves predictive performance but sacrifices the proposed additive interpretability. However, interpretability is primarily demonstrated through visualizations of learned contribution distributions. Could the authors provide a quantitative evaluation of interpretability?

We acknowledge that there is some confusion about the notion of interpretability (also raised by other reviewers). With interpretability we refer to the established inherent interpretability of additive models [1]. This stems from their additive decomposition of feature-wise functions fi(xi)f_i(x_i) that only act on a single feature xix_i. We essentially model y=α+∑ifi(xi)+ϵy = \alpha + \sum_i f_i(x_i) + \epsilon, where α\alpha is an intercept and ϵ\epsilon a feature-independent noise term. In this model, the total effect of xix_i can be summarized by plotting xix_i on the x-axis and fi(xi)f_i(x_i) on the y-axis for the entire support of xix_i. This is what our figures illustrate: For any given xix_i value, you directly see its contribution to the model prediction yy. While existing models can provide a similar average total effect (plotted as the model-specific mean functions), DAM is the first to provide the full distribution of that effect.

This also makes clear that there is no obvious quantitative metric for additive interpretability. Instead, following the standard practice in previous papers (see, e.g., [1], [2], [3]) we show qualitatively in figures 2, 3 and 4 that DAM comes relatively close to the true unimodel feature contributions but does not recover multimodal ones (caused by the limits of the closed-form posterior).

Lastly, collaboration between feature-specific SDEs changes the model significantly to y=α+∑ifi(x1,x2,… )+ϵy = \alpha + \sum_i f_i(x_1, x_2, \dots) + \epsilon, such that each feature contribution fif_i is now (implicitly) a function of all features. This leads to a complete collapse of the notion of interpretability explained above: We can no longer discern the total effect of feature xix_i and visualize it in a plot, as it is completely mixed in with the effects of other features. Hence, there is no quantitative evaluation necessary nor possible: Collaboration leads to a total loss of interpretability following the estbalished notion of interpretability in additive models.


[1] Neural Additive Models: Interpretable Machine Learning with Neural Nets. https://arxiv.org/abs/2004.13912

[2] Neural Additive Models for Location Scale and Shape: A Framework for Interpretable Neural Regression Beyond the Mean. https://arxiv.org/abs/2301.11862

[3] gamboostLSS: An R Package for Model Building and Variable Selection in the GAMLSS Framework. https://arxiv.org/abs/1407.1774

With the discussion period coming to an end, we would like to kindly follow up on our rebuttal. We believe that our responses have adequately addressed your concerns and clarified the questions raised in your review. We would greatly appreciate it if you could take a moment to review our responses and let us know whether your concerns have been resolved.

I appreciate the author for the detailed rebuttal. However, my concern still remains that the learned feature-wise uncertainty distributions are not necessarily faithful to the underlying feature contributions. The synthetic experiments further highlight this limitation. So I decide to maintain my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Updated Paper

Dear reviewers,

we updated the document with the following changes. Text changes are highlighted in blue.

  • We fixed issues with the citation style.
  • We added inference times of the diffusion-based models.
  • We removed the "insurance" dataset from the paper, as we found it to be a perfect copy of the "healthcare_insurance_expenses" data we retrieved from a different source. All tables are updated to reflect this change. This has no substantial impact on the results but only avoids double-counting the performance on a single dataset.
  • We added a NAMLSS variant that uses a four-parameter SHASHo distribution instead of a Gaussian distribution for regression tasks. GAMLSS and gamboostLSS models had substantial issues converging when imposing this distribution and are therefore not included.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.