acceptodds
Under review as a conference paper at ICLR 2027

ACCURACY WITHOUT OPACITY: MONOTONIC TRANSFORMER ADDITIVE MODELS

Abstract

Transformer-based models achieve strong performance on tabular data by learning complex feature interactions, but they remain black boxes and offer no guarantee that learned relationships respect domain-specific monotonicity constraints. Interpretable additive and monotonic models, conversely, provide transparent feature effects but lack the expressiveness to capture rich interactions. To bridge this gap, we introduce the Monotonic Transformer Additive Model (MTAM), which decomposes the model output into additive main effects and feature interactions learned by a Differential Transformer. MTAM addresses monotonicity at three levels: (a) the main effects are monotonic by construction; (b) the Differential Transformer is adapted to reduce sources of non-monotonicity in the interactions; and (c) an augmented Lagrangian scheme imposes monotonicity of the full model as a training constraint and supports an identifiable decomposition through marginal clarity. Across six regression and classification tasks, MTAM narrows the gap between interpretable models and unconstrained transformers while keeping monotonicity violations below % on average in public datasets, providing an accurate and transparent alternative to black-box transformers.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
interpretable machine learning, monotonic learning, monotonic neural networks, transformer interpretability, generalized additive models, neural additive models, monotonic transformers

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JNYzOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3(+1)
Soundness
2(+1)
Read review
Paper summary

This paper proposed the Monotonic Transformer Additive Model (MTAM), an interpretable model for tabular data by combining monotonic additive model with differential transformer. They decompose the model output into two parts, one is the main effects, which remains to be additive, the other is the feature interactions, which is learned by the differential transformer. The main focus of the model design is to address monotonicity requirements for some pre-selected features through three parts: (1) a monotonic additive part for main effects; (2) a monotonicity-oriented differential transformer for interaction learning; (3) an augmented Lagrangian objective. Experiments include a synthetic ablation study and six regression/classification datasets.

Critical strengths
  1. The problem of comining domain-required monotonicity with interpretable model is well motivated and practivally relevant.
  2. The paper correctly identifies that requiring the interaction component hh itself to be monotonic would conflict with marginal clarity, and therefore constrains the total effect instead. This is a sensible design principle.
  3. All results are averaged over 10 random seeds with standard deviations reported.
Critical weaknesses
  1. Table 1-3 are used to demonstrate the usefulness of the inductive biases introduced in Section 4.1.1. However, all the details of the ablation experiments are missing and I have difficulty in understanding the reported results. Are they comparing MTAM with the variant without one specific inductive bias or are they comparing MTAM with the variant without several inductive biases? the paper does not state what the MTAM variant in Tables 1–3 actually removes.
  2. Section 5.1.2 is to evaluate the impact of MTAM with and without the proposed orthogonality constraint. The caption of Figure 3 says "NAMformer (top) and MTAM (bottom)", while the text describes the same figure as MTAM without (top) and with (bottom) the orthogonality constraint. It is therefore unclear which comparison is shown.
  3. The ablation study for differential transformer is weak. Comparing the weight for x1, x2 (~0.04) and x5, x6 (~0.08), the difference is not significant, which means the main effect features still contribute a lot to the interaction part. The separation of the contributions of different features is not clear, which hinders the final interpretability. The imperfect separation can also be reflected in the Figure 3 results. Comparing with GT in Figure 1, the results do not seem to be so faithful. Moreover, the paper notes that differential Attention can produce negative weights, which is an additional source of non-monotonicity. Its net benefit for the paper's goals is therefore unclear.
  4. The paper claims the model provides monotonicity guarantees, which is not guaranteed under such severe implementation compromises. The satisfcation metric is not clearly defined. Is satisfaction computed per pair of adjacent grid points or per whole sweep? Is a numerical tolerance used? How many test instances are sampled? These choices can change the numbers substantially.
  5. Limited novelty and missing related work and baselines. Main effect network is not new, it is a standard constrained monotonic network from [4]. The differential transformer for interaction part is also not original. Several related works are missing: [1][2][3].
  6. The experiments are limited to small scale datasets. The dataset sizes are not clearly indicated, and the largest feature dimension space is only 28. Would be good to have stronger experiments on more datasets / benchmarks, or datasets with higher dimensions.
  7. No plots or figures to show the learned shape functions for the main effect features or the interactions. All interpretability claims are only supported by the monotonicity satisfaction numbers, which is far from enough.
  8. The layout and display of the paper is not good: Table 1 to Table 3 are stacked together without clear gap; Table 4 and Table 5 have the same structure but with totally different size. Number fonts in the Tables are overly small.
  9. In general, I believe many details, both theoretically and experimentally, are not clearly stated, rendering the model itself and the conclusions not convincing enough.

References

[1] Xu, Shijian, Marcello Negri, and Volker Roth. "Nimo: a nonlinear interpretable model." International Conference on Learning Representations. Vol. 2026. 2026.

[2] Chang, Chun-Hao, Rich Caruana, and Anna Goldenberg. "Node-gam: Neural generalized additive model for interpretable deep learning." arXiv preprint arXiv:2106.01613 (2021).

[3] Lemhadri, Ismael, et al. "Lassonet: A neural network with feature sparsity." Journal of Machine Learning Research 22.127 (2021): 1-29.

[4] Sartor, Davide, Alberto Sinigaglia, and Gian Antonio Susto. "Advancing constrained monotonic neural networks: Achieving universal approximation beyond bounded activations." arXiv preprint arXiv:2505.02537 (2025).

Questions for authors
  1. First of all, why do we must have monotonicity? If y=sin⁡x1+...y=\sin x_1 + ... is really a component of the dataset, isn't we also want to maintain this kind of main effect, instead of just interested in monotonicity? In another word, if the dataset has unknown monotonic features, how should this model handle it? Will the design choice / inductive bias bring benefits or will it be harmful?
  2. Personally, I do not feel that a model maintains the monotonicity is considered interpretable. Interpretability should be able to answer questions like: how much does the specific feature x_i contribute to the final prediction, as in traditional linear regression models. In MTAM, how can a user quantify how much xjx_j contributes to a prediction through hh, analogous to a coefficient in linear regression? Without this, the total contribution of a feature remains opaque.
  3. is the feature interactin part still interpretable, or the so-called interpretability comes from the sparse attention of differential transformer? If the latter, how do the authors address the known limitations of attention as explanation?
  4. An important component of interpretability comes from sparse feature selection. Is this model able to do sparse feature selection? The optimization form in Eq. (9) seems to admit standard regularizers, but there is no experiments to directly support this claim, for example, with L1 or L2 penalty?
  5. In Eq. (1), there are two constraints: E[fj(Xj)]=0\mathbb E[f_j(X_j)] = 0 and E[h(X)]=0\mathbb E[h(X)] = 0, how are these two constraints specifically achieved? E[h(X)]=0\mathbb E[h(X)] = 0 is claimed to be important for marginal clarity, but why is E[fj(Xj)]=0\mathbb E[f_j(X_j)] = 0 needed? The paper describes centering fjf_j by its empirical mean. How is E[h(X)]\mathbb E[h(X)] = 0 enforced?
  6. If I understand correctly, the main effects are monotonic by construct, the interaction part is from monotonic differential transformer, if so, then the whole model is monotonic without any further constraints. However, In section 4, the authors further mention that requiring hh itself to be monotonic would conflictt with marginal clarity. The authors argue that the monotonicity of the model output is addressed during training through an augmented Lagrangian, this further reduce the necessarity of the monotonicity of the differential transformer part. Can you clarify the role of the monotonicity of each part? Especially MTAM therefore does not require hh to be monotonic.
  7. The statement after Eq. (8), since vj≥0v_j \ge 0, the inequality constraint reduces to the eqaultiy constraint. why, can you make it more clear?
Reviewer_HfCtOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims at building an interpretable transformer model using additive components. Interesting perspective but unclear why existing interpretability methods could not be applied. Examples are [1,2] - even though [1] does not apply directly.

Overall missing benchmarking against state-of-the-art. Missing user evaluation to qualify how useful is the explanation.

[1] https://arxiv.org/abs/2012.09838 [2] https://arxiv.org/abs/1908.04626

Critical strengths
  • good presentation
Critical weaknesses
  • no benchmarking at scale
  • no user evaluation for interpretation
  • achievement is over-stated - empirical evaluation requires more.
  • look like only the single-feature effects are monotonic by construction.
  • accuracy is not maintained - there is clearly a cost associated to it.
  • ablation only on one synthetic function
Questions for authors
  • no benchmarking at scale: why?
  • no user evaluation for interpretation: why?
  • look like only the single-feature effects are monotonic by construction: why?
  • accuracy is not maintained - there is clearly a cost associated to it: why
  • ablation only on one synthetic function: why?
Reviewer_W1K6OpenReview ↗
Rating
3
Confidence
1
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MTAM, which combines univariate main-effect networks with a Differential Transformer interaction component. Monotonicity is encouraged through constrained main effects, architectural modifications, and an augmented Lagrangian objective. The paper also imposes empirical orthogonality constraints to separate main and interaction effects, and evaluates the method on synthetic data and six tabular datasets.

Critical strengths
  1. The problem is well motivated, and the decomposition into main effects and an interaction component provides a useful starting point.
  2. The synthetic experiments attempt to examine interaction recovery and the effect of Differential Attention, rather than reporting predictive accuracy alone.
  3. Reporting predictive performance together with monotonicity satisfaction is useful for assessing the trade-off.
Critical weaknesses

The paper seems to rely a lot on AI tools, for example, there are too many bolded/italic words (seems from ChatGPT) which are not really necessary.

The introduction is sloppily written; it should be rewritten.

The marginal clarity condition is not equivalent to the implemented constraint. Equation (2) requires E[h(X) | X_k] = 0. Equation (3) is a consequence of this condition, but the converse does not follow. Equation (7) only imposes empirical orthogonality to the particular learned function f_k. For example, a centered h depending on X_k can be orthogonal to f_k and still contain a purely univariate effect. The authors should justify identifiability under the actual training constraints, or weaken the claim. This is a central issue because identifiable separation is presented as a main contribution.

Questions for authors
  1. What establishes identifiability when Equation (7), rather than Equation (2), is enforced?
  2. Is there any global monotonicity certificate for the complete model?
  3. How much interaction information can a reader recover from h beyond the univariate main-effect plots?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.