acceptodds
Under review as a conference paper at ICLR 2027

CAST: Learning Compact Decision Trees via Compound Additive Splits

Abstract

Decision trees expose their routing rules, but simple splits can require large trees to represent nonlinear boundaries. Nonlinear splits can form richer boundaries, but generic multivariate node functions may obscure how individual features affect routing. We introduce Compound Additive Sparse Trees (CAST), a differentiable single-tree model that forms each split by summing learned univariate feature responses. Node-specific combinations of a shared nonlinear basis and direct linear terms give these responses flexible shapes while preserving an exact feature-wise decomposition of the routing score. Stochastic gates sparsify the feature responses used at each node, and exact folding replaces constant subtrees with leaves. On the complete 20-dataset benchmark, CAST achieves the best mean rank (1.100) in the primary five-method comparison. It ranks first at each matched depth setting from three through six. Plots of selected learned nodes show their nonlinear split boundaries and individual feature contributions in original feature coordinates. Across the primary runs, exact folding removes 30.3% of the starting trees' internal nodes in aggregate while preserving the selected gated trees' predictions.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Decision Trees, Differentiable Decision Trees, Interpretable Machine Learning, Nonlinear Splits, Shape Functions

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BHYwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Compound Additive Sparse Trees (CAST), a differentiable single-tree model whose node scores are sums of learned univariate feature responses. A shared nonlinear basis and node-specific linear/nonlinear coefficients provide flexible split boundaries while retaining an exact feature-wise decomposition of each routing score. The method further uses stochastic feature gates for sparsification and exact subtree folding to remove constant subtrees without changing predictions.

CAST is evaluated on 20 tabular classification and regression datasets against NODE, RADDT, GradTree, and SGT-T. At depth 6, it achieves the best mean rank (1.100), and it ranks first in mean rank at depths 3--6. Ablations support the contribution of nonlinear node responses, while additional analyses visualize learned splits and quantify the structural reduction from folding.

Critical strengths
  1. A technically clean combination of expressive and inspectable node rules. The additive split formulation permits nonlinear multivariate decision boundaries while retaining an exact decomposition into per-feature contributions. The exact folding operation is also well defined and prediction-preserving under the stated soft-routing assumptions.

  2. Strong and reasonably extensive empirical validation. CAST performs consistently well across 20 classification/regression datasets and multiple starting depths. The same-framework linear-node ablation provides useful evidence that the nonlinear responses, rather than only the training procedure or topology, contribute materially to predictive performance.

Critical weaknesses
  1. The evidence for “compactness” is weaker than the predictive evidence. Compactness is primarily measured through the number of retained tree nodes, but inference still requires the shared nonlinear basis network, whose parameters and computational cost are not reflected in this measure. Moreover, at depth 6 the folded models still retain 43.88 internal nodes per tree on average, and only 18.1% of retained nodes use at most two features. Reporting full deployed parameter count, inference cost, and accuracy--complexity trade-offs would make the compactness claim substantially more convincing.

  2. The strongest predictive comparison is not fully conclusive. The comparison matches methods by declared depth rather than model complexity; in particular, NODE is an ensemble and SGT-T may stop early, so equal depth does not imply an equal node or parameter budget. Furthermore, at depth 6 the reported Nemenyi comparison between CAST and NODE has adjusted p=0.080. Thus, CAST clearly has the best descriptive mean rank, but the evidence that it significantly outperforms the strongest baseline is more limited than the headline result may suggest.

Questions for authors
  1. Can the authors report the total deployed parameter count and inference cost of CAST, including the shared nonlinear basis, and provide an accuracy-versus-model-complexity comparison with single-tree baselines? This would materially affect my assessment of the paper's central compactness claim.

  2. How do the main conclusions change under a more closely complexity-matched comparison with NODE or other strong baselines? In particular, given the reported Nemenyi adjusted p=0.080 for CAST versus NODE at depth 6, would the authors characterize the result as statistically superior or primarily as competitive with NODE?

7 comments
AuthorsOpenReview ↗

Response to Reviewer BHYw

Thank you for recognizing CAST’s core design and empirical performance. As you point out, there is room to strengthen some aspects of our evaluation. Your comments broadened our perspective on compactness and helped us make the experimental reporting and comparisons more rigorous.

In the revised manuscript, changes addressing your comments are highlighted in light purple. Changes addressing multiple reviewers are highlighted in light gray.

W1 & Q1. Deployment costs and accuracy versus model complexity.

Thank you for pointing out the limitation in our evidence for compactness. In the original paper, we assessed model complexity mainly through the number of retained tree nodes. However, a small number of nodes does not necessarily mean that the nodes use few features or that the model has few parameters. As you note, only 18.1% of the retained nodes in the depth-six CAST use at most two features.

Your comment inspired us to examine CAST’s compactness from additional perspectives, including deployed parameter count, numerical state, inference cost, and their trade-offs with predictive performance. Figure 11 and Table 25 in Appendix A.8 report these comparisons, with the shared nonlinear basis included in the original CAST’s deployed parameter count.

To further examine the trade-off between CAST’s predictive performance and model complexity, we also explore more compact variants of CAST. First, CAST† selects its starting depth from 2 to 6 using validation data for each dataset and seed. Compared with the fixed-depth CAST in Table 1, this reduces the average number of retained internal nodes from 43.88 to 19.45.

With the tree topology unchanged, we then approximate the learned feature responses using piecewise-linear functions fitted on the training data. CAST’s additive node scores allow us to approximate each feature response separately. These functions can retain nonlinear shapes and can still be combined across features to form non-affine splits. This replaces the shared neural basis and high-dimensional node coefficients with a more compact deployment, while preserving the soft-routing form and leaf values. Appendix A.8 describes the construction.

Figure 11 shows the performance and deployment cost trade-offs for different numbers of knots. Among the variants shown, CAST† with at most two knots per feature has the fewest deployed parameters while retaining the best mean rank against the 11 baselines.

The table below compares this two-knot CAST† with fixed-depth CAST and representative single-tree baselines. It reports predictive performance alongside node counts, deployed parameters, numerical state, and inference cost measured by batch-1 latency. Appendix A.8 describes how these quantities are computed.

The two-knot CAST† achieves higher average Accuracy and R² than these single-tree baselines while using 0.308K deployed parameters and 1.95 kB of numerical state. It also uses fewer parameters and has lower batch-1 latency than SGT-T, RADDT, and CPT. And its deployment cost is competitive with simple trees such as CART.

PredictorAccuracy ↑R² ↑Mean internal nodes ↓Deployed parameters (K) ↓Numerical state (kB) ↓Batch-1 latency ↓
CAST, fixed depth93.090%0.812843.8827.778120.560.427 ms
CAST†, 2 knots92.494%0.800919.450.3081.950.017 ms
SGT-T88.042%0.6517241.830.348519.712.628 ms
RADDT91.615%0.723226.270.4351.740.162 ms
CPT86.184%0.694114.955.32121.450.887 ms
CART87.454%0.671934.330.0915.090.080 ms

Together, these comparisons provide the requested view of CAST’s predictive performance and deployment cost alongside single-tree baselines, and show how a more compact deployment trades performance for lower complexity. We hope these results address your concern about CAST’s compactness.

Due to the character limit, we continue our response in the next comment. We apologize for the inconvenience.

AuthorsOpenReview ↗

Response to Reviewer BHYw (continued)

W2 & Q2. Performance under matched model complexity.

We compare CAST with ten single-tree baselines at exactly matched numbers of internal nodes, using pruning to obtain candidates at each target count. Appendix A.8.2 describes the procedure. Some trees initially have fewer nodes than the target, so an exact match is not available for every dataset and seed. Figure 12 and Table 31 report the fixed-depth CAST comparisons. Each entry reports wins, ties, and losses across datasets where all three seeds have exact matches. A dash indicates no complete matches.

Baseline5 nodes10 nodes15 nodes20 nodes25 nodes30 nodes
SGT-T1/0/010/0/013/0/010/0/0—9/0/0
RADDT19/0/017/1/116/0/215/0/214/1/113/0/1
CPT13/0/010/0/09/0/08/0/03/0/03/0/0
CART18/0/017/0/016/0/014/0/013/0/013/0/0

We also compare the two-knot deployment of CAST† at the same target node counts, with the full results in Table 32. Its starting trees are generally smaller, so fewer datasets have complete matches, particularly at larger target node counts.

Baseline5 nodes10 nodes15 nodes20 nodes25 nodes30 nodes
SGT-T—7/0/07/0/06/0/0—3/0/0
RADDT9/0/08/0/08/0/06/0/06/0/03/0/0
CPT8/0/06/0/06/0/04/0/0——
CART9/0/08/0/08/0/06/0/06/0/03/0/0

Across these exact-node comparisons, CAST and its more compact two-knot deployment generally score higher than the selected baselines on the matched datasets.

To compare performance across all 20 datasets, we relax exact matching and use common upper limits on internal nodes. For example, models with 8 and 15 nodes can both be compared under a 15-node ceiling. Table 19 reports the 13-method comparison at ceilings of 5, 10, 15, 20, 25, and 30 nodes. As shown below, CAST has the best mean rank at every ceiling, while its two-knot deployment ranks second at the first five ceilings and third at 30 nodes. Figure 10 also shows a finer sweep of node ceilings in a separate single-tree comparison.

Node ceilingCASTCAST†, 2 knotsSGT-TRADDTCPTCART
52.5253.32511.8755.2506.5258.975
102.2753.57511.2505.6505.5508.975
152.2003.77511.0006.1505.9008.600
202.3503.62511.0256.2005.5008.700
252.3503.67511.0506.2505.3508.775
302.5003.85010.1506.5755.7009.100

For NODE, Table 30 compares CAST trees with NODE ensembles at nearby actual node counts. It uses an exact match when available and otherwise reports the closest smaller and larger candidates. Across 19 datasets with complete matches, CAST has a median of 51 nodes. The closest NODE candidates have medians of 31 nodes on the smaller side and 60 nodes on the larger side. CAST scores higher on 17 and 12 datasets, respectively.

To complement the single-tree comparison, we also compare NODE with an ensemble of 16 CAST trees at d=2d=2, combined using stacking. The CAST ensemble achieves 92.87% mean classification Accuracy and 0.8079 mean regression R², compared with 92.16% and 0.7964 for NODE at the same depth. Figure 13 shows how CAST’s performance changes as the number of trees increases, and Table 36 in Appendix A.10 reports the per-dataset results.

Regarding statistical significance, the Nemenyi-adjusted p in the original submitted paper is 0.080, which does not establish statistical superiority over NODE at the 0.05 level. We apologize for the imprecision in how we characterized this comparison in the original paper. In the revised main comparison, CAST scores higher than NODE on 19 of 20 datasets and achieves a better observed mean rank. The dataset-level wins show how often CAST scores higher across datasets, whereas the Nemenyi test assesses mean-rank differences across methods. We have revised the manuscript to report these results without claiming statistical superiority.

Thank you again for your careful review and insightful suggestions. They have helped us strengthen the evidence for our claims and improve the manuscript. We sincerely hope that the revision addresses your concerns and supports a more favorable assessment of our work. If you have any further questions or suggestions, we would be pleased to continue the discussion with you.

The authors have provided a substantial response to my concerns and added useful experiments on deployment costs and comparisons under matched model complexity. In particular, the additional analysis of deployed parameters, inference latency, and node-budget-controlled comparisons makes the evaluation more comprehensive. I also appreciate the authors' clarification regarding the statistical significance of the comparison with NODE, as they appropriately avoid claiming statistical superiority.

Nevertheless, some limitations remain. The comparisons with NODE are still based on nearby rather than strictly matched model sizes, and the evidence for a consistent advantage over this strong baseline remains inconclusive. Furthermore, the improved deployment efficiency of the compact CAST variants comes with some trade-off in predictive performance. While the proposed method is technically interesting and the experimental evaluation is reasonably extensive, I believe its overall methodological contribution and empirical evidence do not yet fully justify a stronger recommendation.

Overall, I appreciate the authors' efforts in addressing the review comments. However, I prefer to maintain my original assessment.

Reviewer_UzzoOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CAST, a single differentiable decision tree based on soft routing. The splitting score at each internal node is computed as a weighted sum of univariate responses derived from a combination of shared non-linear bases, node-specific coefficients, and linear anchors. This formulation ensures that the node score can be precisely decomposed on a per-feature basis.

Critical strengths
  1. The design of retaining linear anchors for node scores allows the model to encompass both oblique linear splits and purely non-linear splits as parametric endpoints; this renders ablation a single-parameter adjustment within the same framework—rather than requiring a separate implementation—which is methodologically elegant.
Critical weaknesses
  1. Table 1 conflates "differences in model families" with "differences in hyperparameter tuning or computational budget," and it fails to provide the actual number of nodes or parameters for each baseline, leaving readers unable to make their own assessments.
  2. The five methods in Section 4.3 are all differentiable trees. The selling point of this paper is "compact single tree," yet it does not compare with ordinary CART (including cost-complexity pruning) or optimal tree search under a matched number of nodes. Consequently, readers cannot determine whether the gain comes from the additive splitting itself or from gradient-based end-to-end training. For example, On the Consistency Rate of Decision Tree Learning Algorithms, AISTATS'23.
  3. Proposition 1 only states that the function family strictly contains the affine family; it does not imply that the trained network actually utilizes the shape characterized by double-zero crossings. The paper does not provide statistics on the proportion of non-affine shapes among the fitted nodes.

Some typoes:

  1. The title and abstract use the term "compound additive splits," whereas the heading for Section 3.1 refers to "anchored additive splits." Furthermore, the term "compound" is never defined in the main text. I suggest standardizing the terminology and providing a brief definition to clarify the distinction between "compound" (referring to the composition of multiple feature responses within a single node) and "anchored" (referring to the use of a linear anchor).
  2. In the input space, use Rp,RBR_p, R_B (rather than Rp,RB\mathbb{R}_p, \mathbb{R}_B); in line 10 of Algorithm 1, use cL,cRc_L, c_R; and in Eq. (7), use cL,cRc_\text{L}, c_\text{R}. Please ensure consistent subscript formatting throughout.
Questions for authors
  1. After matching the actual number of nodes (rather than the declared depth), does CAST maintain the same ranking relative to NODE and GradTree?
  2. What proportion of the fitted nodes truly exhibit the non-affine shape shown in Proposition 1? Is this proportion correlated with the gain from linear node ablation?
  3. Shallow trees are forced to support individual nodes with more features; could this be precisely why CAST lags behind NODE at d = 2?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Uzzo

Thank you for your careful review and for recognizing the value of CAST’s unified parameterization of linear and nonlinear splits. Your suggestions have helped us strengthen the experimental design by making the comparisons more controlled and accounting for differences in model size. They have also encouraged us to examine more closely how the learned non-affine node responses contribute to CAST’s predictive performance.

In the revised manuscript, changes addressing your comments are highlighted in light green. Changes addressing multiple reviewers are highlighted in light gray.

W1. Clarifying comparison settings and model sizes.

We would like to clarify that we do not use a single fixed configuration for each baseline in our comparison. For each dataset and seed, we perform hyperparameter tuning using validation data and evaluate the selected model on held-out test data. The search includes depth and, where applicable, other settings that affect model size or computational budget, such as pruning, growth penalties, and split complexity. This allows each baseline to select a configuration suited to the dataset, helping make the comparison across model families fairer. Table 4 summarizes the method-specific search spaces, and Appendix A.2.1 describes the selection procedures.

Following your suggestion, we report the actual number of nodes and deployed model parameters for each method. Figure 11 and Table 25 in Appendix A.8 show these model sizes together with mean rank, while Tables 27 and 28 provide the per-dataset node and parameter counts.

To further examine the trade-off between CAST’s predictive performance and model size, we also explored more compact variants of CAST. First, CAST† selects its starting depth from 2 to 6 using validation data. This reduces the median number of internal nodes from 49 to 14 compared with the fixed-depth CAST in Table 1. With the tree topology unchanged, we then approximate the learned feature responses using piecewise-linear functions, varying the maximum number of knots per feature to control the degree of approximation. These piecewise-linear responses can retain nonlinear shapes and can still be combined across features to form complex splits. Figure 11 shows the performance–complexity trade-offs across different approximation levels for both CAST and CAST†. With at most two knots per feature, CAST† achieves a better mean rank than all baselines in Table 25 with only 0.308K median deployed parameters.

W2 & Q1. Comparisons at matched actual node counts.

We compared CAST with ten single-tree baselines at matched actual numbers of internal nodes (Figure 12 and Table 31). For the methods you highlighted, these include ordinary CART, cost-complexity-pruned CART (pCART), optimal tree search with STreeD, GridCART from the paper you suggested, and GradTree as requested in Q1.

Baseline5 nodes10 nodes15 nodes20 nodes25 nodes30 nodes
CART18/0/017/0/016/0/014/0/013/0/013/0/0
pCART17/0/017/0/016/0/014/0/013/0/013/0/0
STreeD18/0/118/0/016/0/015/0/014/0/013/0/0
GridCART19/0/018/0/017/0/016/0/014/0/013/0/0
GradTree18/0/015/0/011/0/011/0/09/0/07/0/0

Each entry gives CAST’s wins/ties/losses across datasets where all three seeds have exact matches. Some trees initially have fewer nodes than the target, so an exact match is not available for every dataset and seed.

To compare performance across all 20 datasets, we relax exact matching and use common upper limits on internal nodes. Table 19 reports the 13-method comparison at ceilings of 5, 10, 15, 20, 25, and 30 nodes. CAST still has the best mean rank at every ceiling.

To examine additive splitting within the same gradient-based soft-tree framework, we further compare CAST with CAST-MLP, a variant that replaces the additive node function with a non-additive multivariate MLP. We match their training parameter counts as closely as possible to support a fair comparison. CAST achieves 93.09% mean classification accuracy and 0.8128 mean regression R², compared with 91.93% and 0.8122 for CAST-MLP. Their two-method mean ranks are 1.300 and 1.700, respectively (Table 33). The comparison supports the effectiveness of CAST’s additive node design. Appendix A.8 describes the implementation differences between the two node functions.

(Q1) For NODE, we compare CAST with the closest available NODE model with at least as many internal nodes. CAST scores higher on most of the 19 matched datasets (Table 30). Because NODE is an ensemble, we also compare it with CAST ensembles of increasing sizes in our response to Q3.

Because of the character limit, we continue our response to your remaining points in the next comment. We apologize for any inconvenience.

AuthorsOpenReview ↗

Response to Reviewer Uzzo (continued)

W3 & Q2. Non-affine shapes in fitted nodes.

(W3, Q2) To examine whether the fitted nodes use the non-affine shapes described in Proposition 1, we analyzed the 2,633 retained nodes in the 60 depth-six models. Of these, 46.15% exhibit two score zero crossings as one feature varies with the others fixed. To capture non-affine responses beyond this particular pattern, we also measure deviations from affine feature responses. By this broader criterion, 96.81% of nodes are non-affine (Appendix A.9 and Table 34).

(Q2) We also examined whether the proportion of non-affine nodes is associated with the gain from the linear node ablation. Using the broader criterion and weighting nodes by their validation arrival probabilities, we observe a positive association between the proportion of non-affine nodes and the gain from the linear node ablation in this exploratory analysis (Spearman ρ = 0.384, p = 0.094). Appendix A.9.1 and Table 34 report the analysis.

To examine the contribution of non-affine responses within the same CAST framework, we compared two interventions. During training, we fixed the nonlinear terms at zero. The resulting linear-node variant has 0.0346 lower mean classification accuracy and 0.0156 lower mean regression R² (Table 3). After training, we replaced the node scores in the fitted trees with affine approximations, keeping the topology and leaf outputs fixed. Restoring, on average, 29.49% of the original node responses in validation-loss order recovered 81.94% of the performance gap between the original and fully replaced trees, compared with 28.88% under random restoration (Table 35). Together, these comparisons show the contribution of nonlinear responses when learned during training and when used in the fitted trees.

W4. Clarifying compound and anchored splits.

We have standardized the term compound additive splits and added its definition in Section 3.1. Compound refers to combining multiple feature responses within a node, while anchored refers to the explicit linear term in each feature response.

W5. Notation.

We have checked the notation throughout. The dimensions in Rp\mathbb{R}^{p} and RB\mathbb{R}^{B} are written as superscripts, and Equation 7 and Algorithm 1 consistently use cLc_{\mathrm L} and cRc_{\mathrm R} for the left and right child outputs.

Q3. Analyzing CAST’s performance gap at d=2d=2.

The fitted CAST trees support your observation that shallow nodes use more features. At d=2d=2, each retained node uses 14.49 features on average, compared with 6.99 at d=6d=6 (Table 12). Yet this does not always prevent CAST from outperforming NODE. Among the 15 datasets where the shallow nodes use more features, CAST scores higher than NODE on five (Table 37). For example, on Balance Scale, CAST uses 4.00 features per node at d=2d=2 versus 3.34 at d=6d=6, and its d=2d=2 accuracy is 97.87%, compared with 92.80% for NODE. These examples suggest that feature count alone may not explain the gap. We suspect that the difference in ensemble size may be a major factor. CAST uses a single tree, while NODE combines 2,048 trees for classification and 1,024 for regression.

To examine this hypothesis, we conduct an exploratory comparison using stacking to combine 1, 2, 4, 8, and 16 independently trained CAST trees at d=2d=2. The following table compares the single CAST tree and the 16-tree ensemble with NODE across 20 datasets and three seeds.

ModelMean classification AccuracyMean regression R²
CAST, one tree90.25%0.7966
CAST, stacking (16 trees)92.87%0.8079
NODE92.16%0.7964

With 16 shallow trees, the CAST ensemble exceeds NODE in both mean classification accuracy and mean regression R², scoring higher on 14 datasets and lower on six. This result suggests that the ensemble setting is an important factor in the comparison at d=2d=2. Figure 13 shows that classification and regression performance improve as the number of CAST trees increases from 1 to 16, and Appendix A.10 reports the per-dataset results and stacking procedure.

Thank you again for your thoughtful suggestions on the comparison design and the role of CAST’s learned node responses. Your guidance has helped us make the evaluation more rigorous and clarify the contribution of our method. We sincerely hope that the revised paper addresses your concerns and helps you assess our work more favorably. If you have any further questions or suggestions, we would be pleased to continue the discussion with you.

Reviewer_UzzoOpenReview ↗

Official Comment by Reviewer Uzzo

Thank you for the comparisons at matched node counts and the additional analyses of nonlinear responses and deployment costs. These results substantially address my main concerns about comparison fairness and the value of the proposed design, and I am raising my rating from 2 to 3 (weak acceptance).

AuthorsOpenReview ↗

Thank You for Your Positive Feedback

Thank you for your positive feedback and for recognizing our efforts to address your concerns. We would be happy to continue discussing the paper with you throughout the discussion period.

Reviewer_ornpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes to learn a soft decision tree where decision nodes use generalized additive models (GAMs) with a sigmoid link function. Shape functions per each feature take the form a shared basis function with basis function coefficients being separate at decision node. Basis functions take the form of a two hidden-layer MLP with GELU activations. As in traditional soft decision trees, the final prediction uses a combination of all route-to-leaf probabilities. For initialization, some information from the fitted CART tree is used. There are also stochastic gates used to learn sparsity, i.e. to remove feature contributions at decision nodes. The model is learned end-to-end with stochastic gradient based optimization.

Critical strengths

Though the idea of using GAMs at decision nodes is simple, the final learnt model can be more accurate than traditional decision trees. However, interpretability of this model class is quite questionable, and should be carefully demonstrated.

Critical weaknesses
  • The paper is learning soft decision trees where all route-to-leaf paths are used during inference. Despite motivating for interpretability, such soft trees are hard to explain unlike traditional axis-aligned decision trees.
  • Experimental comparison with the baselines seem unfair. For example, RADDT and SGT-T use hard decision trees with simpler routing functions while this paper uses soft routing. Comparing with traditional soft decision trees is important to claim the accuracy improvements. Other potential baselines can include "Convex Polytope Trees" by M. Armandpour et al.
  • Interpretability of the model is questionable. Even with hard decision splits, interpreting multidimensional GAM shape features at each decision nodes is not simple for a human eye. Even sparse oblique decision trees are hard to interpret.
  • Some of the equations are hard to understand. For example, in eq. 1, why is the softmax function used? What is πi\pi_i? It is not defined in the paper. There is no clear objective function written with clear learnable parameters.
Questions for authors
  • Why are neural networks used as basis functions? And why are they shared across each decision node?
  • For GAMs, one of the most accurate methods are explainable boosting machines. Can those be used in the decision nodes?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ornp

Thank you for recognizing the simplicity and accuracy potential of CAST. Your comments on interpretability prompted us to revisit how we present the model: our original decision-boundary visualizations did not sufficiently demonstrate how individual features affect a multi-feature split or how all route-to-leaf paths contribute to a soft prediction. We have therefore developed a quantitative explanation that builds on CAST’s decomposable node scores, tracing the contributions of the original features through routing to the full prediction.

In the revised manuscript, changes addressing your comments are highlighted in light orange. Changes addressing multiple reviewers are highlighted in light gray.

W1 & W3. Demonstrating CAST’s interpretability.

We now use CAST’s additive node scores to quantify each original feature’s contribution to a multi-feature split, making the routing decision interpretable in terms of the features that form it. We then trace how the resulting path probabilities weight the leaf outputs to form the tree’s prediction. This gives us a quantitative way to inspect the soft prediction by following how a change in an original feature affects routing, redistributes weight among the leaves, and ultimately changes the prediction. Appendix A.4 develops this explanation through a held-out Iris example. We have also added an interactive viewer to the anonymous repository (https://anonymous.4open.science/r/cast_anonymous/cast_interactive_routing.html), where the feature contributions and routing paths can be explored across additional examples. You can switch JS Off to JS On in the preview, or download the HTML file and open it in a browser to use the viewer.

(W3) At the node level, CAST’s additive split lets us decompose a multi-feature routing score into contributions from the original features, even when their responses are nonlinear. For the held-out Iris example, Table 17A shows how these contributions combine at node 2 relative to its mean score over the training data: petal length contributes −1.00916, pulling the score toward the left branch, while the other three features contribute positively. Adding all four contributions to the baseline gives a score of −0.13571, which the sigmoid then maps to a 46.61% probability of routing to the right branch. Thus, instead of asking the reader to infer each feature’s role from a joint boundary, the decomposition shows how the features jointly produce the routing probability.

(W1) For the tree level, building on the feature-wise explanation of node routing, we show how the original features contribute to CAST's prediction through route-to-leaf paths (Figure 8, Table 17B, and Table 18). Each leaf’s contribution to the margin between versicolor and virginica is its arrival probability multiplied by its own margin. The figure highlights four leaves that together carry 86.30% of the arrival weight. Among them, leaves 38 and 33 contribute +0.10486 and −0.06437, respectively, pulling the prediction toward different classes. Summing the contributions of all leaves gives a margin of +0.06740 in favor of versicolor, which is also the class predicted by the full model. To connect the full tree’s prediction back to the original features, Figure 9 also attributes the change in its prediction margin from a reference input to the sample’s features.

Table 18 further shows how an input change affects the full prediction. When petal width increases from 1.5 to 1.6 cm, the tree’s routing changes. For example, its contribution at node 2 decreases by 0.71351, and the probability of taking the right branch at that node falls from 46.61% to 29.96%. The arrival probability of leaf 38 falls from 44.87% to 28.85%, while that of leaf 33 rises from 22.67% to 43.43%. Combining all leaf outputs changes the margin from +0.06740 to −0.03593, and the predicted class changes from versicolor to virginica. This example connects a change in an original feature to changes in routing and the full tree’s prediction.

Due to the character limit, we have to continue our response to your remaining comments in the next comment. We apologize for the inconvenience.

AuthorsOpenReview ↗

Response to Reviewer ornp (continued)

W2. Further comparisons with routing and split controls.

To address the concern that routing and split complexity change together in the original comparisons, we added Budding Trees (BT) and the suggested Convex Polytope Trees (CPT). CAST and BT both retain soft routing at inference, but use nonlinear and affine splits, respectively. CAST and CPT both use nonlinear splits, but CPT uses hard routing at inference after probabilistic training. Table 4 summarizes these properties. Their results are summarized below, with per-dataset comparisons added to Table 1.

MethodMean AccuracyMean R²Mean rank
CAST93.09%0.81281.675
BT92.01%0.78643.650
CPT86.18%0.69417.225

CAST achieves higher mean Accuracy and R² and a better mean rank than both BT and CPT. Accuracy and R² are averaged across 11 classification and 9 regression datasets, respectively.

Considering that these baselines also differ from CAST in their model frameworks, we further compare split types within CAST’s own soft-tree framework. Table 3 compares CAST with a linear-node variant obtained by removing the nonlinear node terms. At the final endpoint, the variant has 0.0346 lower mean classification accuracy and 0.0156 lower mean regression R². This supports the contribution of CAST’s nonlinear splits to predictive performance beyond the use of soft routing.

W4. Notation and objectives.

We clarified that πₜ,ᵢ is the logit for feature i at node t. Softmax normalizes the feature weights to control the relative scales of their responses, while separate gates remove feature contributions at individual nodes. In Sections 3.4–3.5, we have added and clarified the learnable parameters and objective functions for tree training and gate continuation.

Q1. Neural bases and sharing.

Appendix A.6 explains why CAST uses neural basis functions and shares them across nodes. CAST constructs its nonlinear feature responses from a set of basis functions. We compared three choices for this basis: a neural network, fixed RBFs, and cubic B-splines with fixed knots. After the full training pipeline, the neural basis achieves the best mean rank of 1.625, followed closely by B-splines at 1.775 and RBFs at 2.600 (Table 20). We choose the neural basis because, alongside its overall predictive performance, the basis functions themselves are learned jointly with the tree’s prediction objective, without requiring predefined spline knots. It also has lower batch-128 inference latency than the spline basis, with parameter counts of the same order of magnitude (Table 21).

Having selected a neural basis, we considered how to use it across the tree. A separate basis network at each node would allow nodes to learn their basis shapes independently, but it would also increase the model’s computational cost. We therefore compared three designs: a separate network per node, a shared network per level, and a globally shared network. Under the same 300-epoch training protocol, the per-node design achieves the best mean rank of 1.725, followed closely by global sharing at 1.850. However, the per-node design uses 10.52 times as many parameters and 11.73 times the single-sample inference time (Tables 22–23). Therefore, we choose global sharing, which retains similar predictive performance at substantially lower cost.

Q2. EBM-based routing alternative.

Yes. Following your suggestion, we implemented an EBM-based routing alternative in which boosted feature functions form the additive score at each node. The functions are trained using the whole-tree prediction loss, while the tree retains soft routing (Appendix A.7). We compared this alternative with CAST’s neural basis on the same 20 datasets and three seeds. At the initial 300-epoch stage, the neural basis and EBM routing achieve mean ranks of 1.35 and 1.65, respectively (Table 24). These results suggest that EBM-based routing is a promising direction for future research on CAST.

Thank you again for your valuable suggestions on interpretability, the experimental comparisons, and the model design. They have helped us improve our work and strengthen the manuscript. We sincerely hope that our response and the revised paper could address your concerns and provide a stronger basis for reassessing our work. If you have any further questions or suggestions, we would be glad to discuss them with you.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer ornp,

Thank you again for your thoughtful review. As the discussion period draws to a close, we would like to briefly follow up on our responses.

In response to your comments, we developed quantitative explanations of CAST’s node routing and soft-tree predictions, added comparisons with soft trees and the Convex Polytope Trees you suggested, and examined basis-function choices and EBM-based routing.

If you feel that our responses have addressed your concerns, we would appreciate it if you could update your review or rating if you consider it appropriate. If you have any further questions, we would be glad to discuss them with you.

Thank you again for your time and consideration. We wish you a wonderful day.

Best regards, The authors

AuthorsOpenReview ↗

A Brief Follow-Up Before the Discussion Closes

Dear Reviewer ornp,

Thank you again for your original comments. They gave us a fresh perspective on how to present CAST’s interpretability and predictive performance, and we’re grateful for the opportunity to strengthen the paper. We found that your comments were insightful and beneficial to us. We took each of your concerns seriously and did our best to address them. We responded promptly in the hope of leaving more time to discuss your feedback with you, as we believe that a fuller exchange could have helped us improve the paper further.

Unfortunately, as the discussion period draws to a close, we may no longer have the opportunity to hear any further constructive feedback from you. Still, we sincerely hope that our final response fully addresses your concerns. If any remain, we’d be grateful to hear your thoughts. We’d also appreciate an update to your rating if you consider it appropriate.

Thank you again for your time and consideration. We hope you a wonderful life.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.