- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3(+1)
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposed the Monotonic Transformer Additive Model (MTAM), an interpretable model for tabular data by combining monotonic additive model with differential transformer. They decompose the model output into two parts, one is the main effects, which remains to be additive, the other is the feature interactions, which is learned by the differential transformer. The main focus of the model design is to address monotonicity requirements for some pre-selected features through three parts: (1) a monotonic additive part for main effects; (2) a monotonicity-oriented differential transformer for interaction learning; (3) an augmented Lagrangian objective. Experiments include a synthetic ablation study and six regression/classification datasets.
Critical strengths
- The problem of comining domain-required monotonicity with interpretable model is well motivated and practivally relevant.
- The paper correctly identifies that requiring the interaction component itself to be monotonic would conflict with marginal clarity, and therefore constrains the total effect instead. This is a sensible design principle.
- All results are averaged over 10 random seeds with standard deviations reported.
Critical weaknesses
- Table 1-3 are used to demonstrate the usefulness of the inductive biases introduced in Section 4.1.1. However, all the details of the ablation experiments are missing and I have difficulty in understanding the reported results. Are they comparing MTAM with the variant without one specific inductive bias or are they comparing MTAM with the variant without several inductive biases? the paper does not state what the MTAM variant in Tables 1–3 actually removes.
- Section 5.1.2 is to evaluate the impact of MTAM with and without the proposed orthogonality constraint. The caption of Figure 3 says "NAMformer (top) and MTAM (bottom)", while the text describes the same figure as MTAM without (top) and with (bottom) the orthogonality constraint. It is therefore unclear which comparison is shown.
- The ablation study for differential transformer is weak. Comparing the weight for x1, x2 (~0.04) and x5, x6 (~0.08), the difference is not significant, which means the main effect features still contribute a lot to the interaction part. The separation of the contributions of different features is not clear, which hinders the final interpretability. The imperfect separation can also be reflected in the Figure 3 results. Comparing with GT in Figure 1, the results do not seem to be so faithful. Moreover, the paper notes that differential Attention can produce negative weights, which is an additional source of non-monotonicity. Its net benefit for the paper's goals is therefore unclear.
- The paper claims the model provides monotonicity guarantees, which is not guaranteed under such severe implementation compromises. The satisfcation metric is not clearly defined. Is satisfaction computed per pair of adjacent grid points or per whole sweep? Is a numerical tolerance used? How many test instances are sampled? These choices can change the numbers substantially.
- Limited novelty and missing related work and baselines. Main effect network is not new, it is a standard constrained monotonic network from [4]. The differential transformer for interaction part is also not original. Several related works are missing: [1][2][3].
- The experiments are limited to small scale datasets. The dataset sizes are not clearly indicated, and the largest feature dimension space is only 28. Would be good to have stronger experiments on more datasets / benchmarks, or datasets with higher dimensions.
- No plots or figures to show the learned shape functions for the main effect features or the interactions. All interpretability claims are only supported by the monotonicity satisfaction numbers, which is far from enough.
- The layout and display of the paper is not good: Table 1 to Table 3 are stacked together without clear gap; Table 4 and Table 5 have the same structure but with totally different size. Number fonts in the Tables are overly small.
- In general, I believe many details, both theoretically and experimentally, are not clearly stated, rendering the model itself and the conclusions not convincing enough.
References
[1] Xu, Shijian, Marcello Negri, and Volker Roth. "Nimo: a nonlinear interpretable model." International Conference on Learning Representations. Vol. 2026. 2026.
[2] Chang, Chun-Hao, Rich Caruana, and Anna Goldenberg. "Node-gam: Neural generalized additive model for interpretable deep learning." arXiv preprint arXiv:2106.01613 (2021).
[3] Lemhadri, Ismael, et al. "Lassonet: A neural network with feature sparsity." Journal of Machine Learning Research 22.127 (2021): 1-29.
[4] Sartor, Davide, Alberto Sinigaglia, and Gian Antonio Susto. "Advancing constrained monotonic neural networks: Achieving universal approximation beyond bounded activations." arXiv preprint arXiv:2505.02537 (2025).
Questions for authors
- First of all, why do we must have monotonicity? If is really a component of the dataset, isn't we also want to maintain this kind of main effect, instead of just interested in monotonicity? In another word, if the dataset has unknown monotonic features, how should this model handle it? Will the design choice / inductive bias bring benefits or will it be harmful?
- Personally, I do not feel that a model maintains the monotonicity is considered interpretable. Interpretability should be able to answer questions like: how much does the specific feature x_i contribute to the final prediction, as in traditional linear regression models. In MTAM, how can a user quantify how much contributes to a prediction through , analogous to a coefficient in linear regression? Without this, the total contribution of a feature remains opaque.
- is the feature interactin part still interpretable, or the so-called interpretability comes from the sparse attention of differential transformer? If the latter, how do the authors address the known limitations of attention as explanation?
- An important component of interpretability comes from sparse feature selection. Is this model able to do sparse feature selection? The optimization form in Eq. (9) seems to admit standard regularizers, but there is no experiments to directly support this claim, for example, with L1 or L2 penalty?
- In Eq. (1), there are two constraints: and , how are these two constraints specifically achieved? is claimed to be important for marginal clarity, but why is needed? The paper describes centering by its empirical mean. How is = 0 enforced?
- If I understand correctly, the main effects are monotonic by construct, the interaction part is from monotonic differential transformer, if so, then the whole model is monotonic without any further constraints. However, In section 4, the authors further mention that requiring itself to be monotonic would conflictt with marginal clarity. The authors argue that the monotonicity of the model output is addressed during training through an augmented Lagrangian, this further reduce the necessarity of the monotonicity of the differential transformer part. Can you clarify the role of the monotonicity of each part? Especially MTAM therefore does not require to be monotonic.
- The statement after Eq. (8), since , the inequality constraint reduces to the eqaultiy constraint. why, can you make it more clear?