- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Compound Additive Sparse Trees (CAST), a differentiable single-tree model whose node scores are sums of learned univariate feature responses. A shared nonlinear basis and node-specific linear/nonlinear coefficients provide flexible split boundaries while retaining an exact feature-wise decomposition of each routing score. The method further uses stochastic feature gates for sparsification and exact subtree folding to remove constant subtrees without changing predictions.
CAST is evaluated on 20 tabular classification and regression datasets against NODE, RADDT, GradTree, and SGT-T. At depth 6, it achieves the best mean rank (1.100), and it ranks first in mean rank at depths 3--6. Ablations support the contribution of nonlinear node responses, while additional analyses visualize learned splits and quantify the structural reduction from folding.
Critical strengths
-
A technically clean combination of expressive and inspectable node rules. The additive split formulation permits nonlinear multivariate decision boundaries while retaining an exact decomposition into per-feature contributions. The exact folding operation is also well defined and prediction-preserving under the stated soft-routing assumptions.
-
Strong and reasonably extensive empirical validation. CAST performs consistently well across 20 classification/regression datasets and multiple starting depths. The same-framework linear-node ablation provides useful evidence that the nonlinear responses, rather than only the training procedure or topology, contribute materially to predictive performance.
Critical weaknesses
-
The evidence for “compactness” is weaker than the predictive evidence. Compactness is primarily measured through the number of retained tree nodes, but inference still requires the shared nonlinear basis network, whose parameters and computational cost are not reflected in this measure. Moreover, at depth 6 the folded models still retain 43.88 internal nodes per tree on average, and only 18.1% of retained nodes use at most two features. Reporting full deployed parameter count, inference cost, and accuracy--complexity trade-offs would make the compactness claim substantially more convincing.
-
The strongest predictive comparison is not fully conclusive. The comparison matches methods by declared depth rather than model complexity; in particular, NODE is an ensemble and SGT-T may stop early, so equal depth does not imply an equal node or parameter budget. Furthermore, at depth 6 the reported Nemenyi comparison between CAST and NODE has adjusted p=0.080. Thus, CAST clearly has the best descriptive mean rank, but the evidence that it significantly outperforms the strongest baseline is more limited than the headline result may suggest.
Questions for authors
-
Can the authors report the total deployed parameter count and inference cost of CAST, including the shared nonlinear basis, and provide an accuracy-versus-model-complexity comparison with single-tree baselines? This would materially affect my assessment of the paper's central compactness claim.
-
How do the main conclusions change under a more closely complexity-matched comparison with NODE or other strong baselines? In particular, given the reported Nemenyi adjusted p=0.080 for CAST versus NODE at depth 6, would the authors characterize the result as statistically superior or primarily as competitive with NODE?
7 comments
Response to Reviewer BHYw (continued)
W2 & Q2. Performance under matched model complexity.
We compare CAST with ten single-tree baselines at exactly matched numbers of internal nodes, using pruning to obtain candidates at each target count. Appendix A.8.2 describes the procedure. Some trees initially have fewer nodes than the target, so an exact match is not available for every dataset and seed. Figure 12 and Table 31 report the fixed-depth CAST comparisons. Each entry reports wins, ties, and losses across datasets where all three seeds have exact matches. A dash indicates no complete matches.
| Baseline | 5 nodes | 10 nodes | 15 nodes | 20 nodes | 25 nodes | 30 nodes |
|---|---|---|---|---|---|---|
| SGT-T | 1/0/0 | 10/0/0 | 13/0/0 | 10/0/0 | — | 9/0/0 |
| RADDT | 19/0/0 | 17/1/1 | 16/0/2 | 15/0/2 | 14/1/1 | 13/0/1 |
| CPT | 13/0/0 | 10/0/0 | 9/0/0 | 8/0/0 | 3/0/0 | 3/0/0 |
| CART | 18/0/0 | 17/0/0 | 16/0/0 | 14/0/0 | 13/0/0 | 13/0/0 |
We also compare the two-knot deployment of CAST† at the same target node counts, with the full results in Table 32. Its starting trees are generally smaller, so fewer datasets have complete matches, particularly at larger target node counts.
| Baseline | 5 nodes | 10 nodes | 15 nodes | 20 nodes | 25 nodes | 30 nodes |
|---|---|---|---|---|---|---|
| SGT-T | — | 7/0/0 | 7/0/0 | 6/0/0 | — | 3/0/0 |
| RADDT | 9/0/0 | 8/0/0 | 8/0/0 | 6/0/0 | 6/0/0 | 3/0/0 |
| CPT | 8/0/0 | 6/0/0 | 6/0/0 | 4/0/0 | — | — |
| CART | 9/0/0 | 8/0/0 | 8/0/0 | 6/0/0 | 6/0/0 | 3/0/0 |
Across these exact-node comparisons, CAST and its more compact two-knot deployment generally score higher than the selected baselines on the matched datasets.
To compare performance across all 20 datasets, we relax exact matching and use common upper limits on internal nodes. For example, models with 8 and 15 nodes can both be compared under a 15-node ceiling. Table 19 reports the 13-method comparison at ceilings of 5, 10, 15, 20, 25, and 30 nodes. As shown below, CAST has the best mean rank at every ceiling, while its two-knot deployment ranks second at the first five ceilings and third at 30 nodes. Figure 10 also shows a finer sweep of node ceilings in a separate single-tree comparison.
| Node ceiling | CAST | CAST†, 2 knots | SGT-T | RADDT | CPT | CART |
|---|---|---|---|---|---|---|
| 5 | 2.525 | 3.325 | 11.875 | 5.250 | 6.525 | 8.975 |
| 10 | 2.275 | 3.575 | 11.250 | 5.650 | 5.550 | 8.975 |
| 15 | 2.200 | 3.775 | 11.000 | 6.150 | 5.900 | 8.600 |
| 20 | 2.350 | 3.625 | 11.025 | 6.200 | 5.500 | 8.700 |
| 25 | 2.350 | 3.675 | 11.050 | 6.250 | 5.350 | 8.775 |
| 30 | 2.500 | 3.850 | 10.150 | 6.575 | 5.700 | 9.100 |
For NODE, Table 30 compares CAST trees with NODE ensembles at nearby actual node counts. It uses an exact match when available and otherwise reports the closest smaller and larger candidates. Across 19 datasets with complete matches, CAST has a median of 51 nodes. The closest NODE candidates have medians of 31 nodes on the smaller side and 60 nodes on the larger side. CAST scores higher on 17 and 12 datasets, respectively.
To complement the single-tree comparison, we also compare NODE with an ensemble of 16 CAST trees at , combined using stacking. The CAST ensemble achieves 92.87% mean classification Accuracy and 0.8079 mean regression R², compared with 92.16% and 0.7964 for NODE at the same depth. Figure 13 shows how CAST’s performance changes as the number of trees increases, and Table 36 in Appendix A.10 reports the per-dataset results.
Regarding statistical significance, the Nemenyi-adjusted p in the original submitted paper is 0.080, which does not establish statistical superiority over NODE at the 0.05 level. We apologize for the imprecision in how we characterized this comparison in the original paper. In the revised main comparison, CAST scores higher than NODE on 19 of 20 datasets and achieves a better observed mean rank. The dataset-level wins show how often CAST scores higher across datasets, whereas the Nemenyi test assesses mean-rank differences across methods. We have revised the manuscript to report these results without claiming statistical superiority.
Thank you again for your careful review and insightful suggestions. They have helped us strengthen the evidence for our claims and improve the manuscript. We sincerely hope that the revision addresses your concerns and supports a more favorable assessment of our work. If you have any further questions or suggestions, we would be pleased to continue the discussion with you.
The authors have provided a substantial response to my concerns and added useful experiments on deployment costs and comparisons under matched model complexity. In particular, the additional analysis of deployed parameters, inference latency, and node-budget-controlled comparisons makes the evaluation more comprehensive. I also appreciate the authors' clarification regarding the statistical significance of the comparison with NODE, as they appropriately avoid claiming statistical superiority.
Nevertheless, some limitations remain. The comparisons with NODE are still based on nearby rather than strictly matched model sizes, and the evidence for a consistent advantage over this strong baseline remains inconclusive. Furthermore, the improved deployment efficiency of the compact CAST variants comes with some trade-off in predictive performance. While the proposed method is technically interesting and the experimental evaluation is reasonably extensive, I believe its overall methodological contribution and empirical evidence do not yet fully justify a stronger recommendation.
Overall, I appreciate the authors' efforts in addressing the review comments. However, I prefer to maintain my original assessment.
Response to Reviewer BHYw
Thank you for recognizing CAST’s core design and empirical performance. As you point out, there is room to strengthen some aspects of our evaluation. Your comments broadened our perspective on compactness and helped us make the experimental reporting and comparisons more rigorous.
In the revised manuscript, changes addressing your comments are highlighted in light purple. Changes addressing multiple reviewers are highlighted in light gray.
W1 & Q1. Deployment costs and accuracy versus model complexity.
Thank you for pointing out the limitation in our evidence for compactness. In the original paper, we assessed model complexity mainly through the number of retained tree nodes. However, a small number of nodes does not necessarily mean that the nodes use few features or that the model has few parameters. As you note, only 18.1% of the retained nodes in the depth-six CAST use at most two features.
Your comment inspired us to examine CAST’s compactness from additional perspectives, including deployed parameter count, numerical state, inference cost, and their trade-offs with predictive performance. Figure 11 and Table 25 in Appendix A.8 report these comparisons, with the shared nonlinear basis included in the original CAST’s deployed parameter count.
To further examine the trade-off between CAST’s predictive performance and model complexity, we also explore more compact variants of CAST. First, CAST† selects its starting depth from 2 to 6 using validation data for each dataset and seed. Compared with the fixed-depth CAST in Table 1, this reduces the average number of retained internal nodes from 43.88 to 19.45.
With the tree topology unchanged, we then approximate the learned feature responses using piecewise-linear functions fitted on the training data. CAST’s additive node scores allow us to approximate each feature response separately. These functions can retain nonlinear shapes and can still be combined across features to form non-affine splits. This replaces the shared neural basis and high-dimensional node coefficients with a more compact deployment, while preserving the soft-routing form and leaf values. Appendix A.8 describes the construction.
Figure 11 shows the performance and deployment cost trade-offs for different numbers of knots. Among the variants shown, CAST† with at most two knots per feature has the fewest deployed parameters while retaining the best mean rank against the 11 baselines.
The table below compares this two-knot CAST† with fixed-depth CAST and representative single-tree baselines. It reports predictive performance alongside node counts, deployed parameters, numerical state, and inference cost measured by batch-1 latency. Appendix A.8 describes how these quantities are computed.
The two-knot CAST† achieves higher average Accuracy and R² than these single-tree baselines while using 0.308K deployed parameters and 1.95 kB of numerical state. It also uses fewer parameters and has lower batch-1 latency than SGT-T, RADDT, and CPT. And its deployment cost is competitive with simple trees such as CART.
Together, these comparisons provide the requested view of CAST’s predictive performance and deployment cost alongside single-tree baselines, and show how a more compact deployment trades performance for lower complexity. We hope these results address your concern about CAST’s compactness.
Due to the character limit, we continue our response in the next comment. We apologize for the inconvenience.