- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes NADT, a soft decision tree ensemble that uses learned prototypes to incorporate neighborhood information into routing. A Neighborhood Context Module converts prototype affinities into input-dependent biases at internal tree nodes, without retrieving training samples during inference. Experiments on 30 classification datasets report the best average rank among the compared methods.
Critical strengths
-
The method provides a practical way to combine prototype-based neighborhood modeling with hierarchical routing. Sharing a compact prototype bank across the ensemble allows local adaptation without storing the training set for inference.
-
The evaluation includes repeated cross-validation, component ablations, runtime measurements, and a parameter-matched MLP router. These controls help examine the contribution of prototype-based routing beyond a standard soft tree ensemble.
Critical weaknesses
-
The expressivity proof contains a concrete gap. Lemma 4 claims that any nonzero context projection preserves differences between affinity vectors, but a matrix with identical columns produces a constant output because the affinities sum to one. Furthermore, showing different routing from one fixed baseline configuration does not establish that the resulting classifier cannot be represented by another baseline configuration. The claimed strict function-class containment therefore remains unproven.
-
The baseline tuning leaves uncertainty about the reported advantage. Table 4 limits XGBoost, LightGBM, and CatBoost to at most 150 boosting iterations, which may restrict their performance. The paper also does not clearly report NADT’s tuning search space or the number of Optuna trials per method, making it difficult to assess whether the comparison uses comparable tuning effort.
-
The claim that the gains specifically come from neighborhood structure is stronger than the ablation evidence supports. NADT and the MLP router have average ranks of 2.37 and 3.60, a difference below the reported Nemenyi critical difference of 1.38. Since the controls also use shared default hyperparameters rather than separate tuning, the results suggest a benefit but do not establish a statistically significant advantage over generic input-dependent routing.
Questions for authors
-
Can you provide a corrected strict-containment proof that accounts for degenerate nonzero context projections and LayerNorm, and rules out alternative baseline parameterizations?
-
Does NADT retain its advantage when the boosting baselines receive broader search spaces and all methods receive a comparable tuning budget? Please specify NADT’s search space and the trial budget for each method.
-
Does the prototype-based module outperform the MLP router when both are tuned separately and use matched parameter counts, normalization, and training schedules?
4 comments
Response to Q3 (MLP router under independent tuning)
Q3
We have re-tuned and re-run all ablation variants independently, with the same search space, protocol, and data preprocessing as the full model, so the comparison no longer shares hyperparameters. The updated results are reported in Section 4.3, Appendix E (Table 5), and Appendix F (Tables 6–8), and are summarized below. Under these conditions, NADT still outperforms the MLP router.
| Variant | W/L/T vs NADT | Avg F1 | Best | Avg Rank |
|---|---|---|---|---|
| NADT (full) | — | 0.852 | 11 | 2.32 |
| w/o NCM | 21/8/1 | 0.812 | 2 | 3.78 |
| w/ Random δ | 21/9/0 | 0.819 | 3 | 3.28 |
| w/o Projector | 23/7/0 | 0.848 | 5 | 2.98 |
| w/ Axis-Aligned | 24/5/1 | 0.822 | 4 | 4.63 |
| w/ MLP Router | 22/8/0 | 0.828 | 7 | 3.90 |
Regarding statistical significance. After independent tuning, the ranks of Full and w/ MLP Router are 2.32 and 3.90. The difference of 1.58 exceeds the Nemenyi critical difference of 1.38 (k = 6, N = 30, α = 0.05), establishing a significant advantage. This result is now stated in Appendix F of the revision.
Thank you for the detailed response and the revised manuscript.
My concerns about the proof, the baseline tuning, and the MLP router comparison are now resolved. However, in the new ablation table, Random δ (3.28) now ranks better than w/o NCM (3.78) and the MLP router (3.90), and it is not significantly different from full NADT (gap 0.96 < CD 1.38). But Section 4.3 still says Random δ performs nearly identically to w/o NCM. Also, in Table 1, the OVA_Uterus and OVA_Lung rows have the same scores for NADT, DOFEN, GRANDE, SoHoT, and DTSemNet. I'm curious if this is a coincidence or a copy error.
I also agree with the other reviewers that the novelty is somewhat incremental and that the performance gain is not significantly better than TabR and ModernNCA.
Response to the new comment: Random δ statement correction, OVA_Lung data fix, and novelty positioning
We thank the reviewer for the reply and are glad that the concerns from the review stage are now resolved. We reply to the issues raised in your latest comment below.
R1(Inconsistency between the Random δ statement and the data): We checked the manuscript and confirmed that the original sentence in Section 4.3 is inconsistent with the updated ablation results. We have corrected this statement in the manuscript. Regarding significance, we provide the average rank to give a more comprehensive comparison across all variants. The updated summary table (Section 4.3, Table 2; W/L/T counts are against NADT) reads:
| Method | W/L/T | Avg F1 | #Best | Rank |
|---|---|---|---|---|
| NADT (full) | -- | 0.857 | 12 | 2.25 |
| w/o NCM | 21/8/1 | 0.816 | 1 | 3.72 |
| w/ Random | 22/8/0 | 0.824 | 3 | 3.32 |
| w/o Projector | 24/6/0 | 0.852 | 4 | 3.08 |
| w/ Axis-Aligned | 24/5/1 | 0.826 | 3 | 4.67 |
| w/ MLP Router | 22/8/0 | 0.833 | 7 | 3.87 |
NADT leads every variant on all four metrics. It beats Random on 22 of 30 datasets, with a higher average F1 (0.857 vs 0.824) and the highest score on 12 datasets versus 3. A non-significant rank gap should not be taken as evidence that the methods perform the same.
R2(Identical OVA_Uterus and OVA_Lung rows): Thank you for this careful check. We traced the cause and confirmed a data loading error. The data-loading code for OVA_Lung mistakenly used the OpenML id of OVA_Uterus (1138), while the correct id is 1130. As a result, the OVA_Lung row in Table 1 actually reported results on OVA_Uterus. We have corrected the id and report the corrected OVA_Lung results in the manuscript (Sections 4.2 and 4.3). The overall conclusion is unchanged. NADT remains the best method. The average ranks of all 11 methods after the correction are listed below (ProtoGate is excluded due to missing results on 8 datasets).
| Method | Avg. rank |
|---|---|
| NADT | 2.50 |
| TabR | 4.12 |
| modernNCA | 4.25 |
| LightGBM | 4.83 |
| CatBoost | 4.88 |
| XGBoost | 4.90 |
| DOFEN | 5.82 |
| Random Forest | 6.27 |
| DTSemNet | 8.48 |
| SoHoT | 10.13 |
| GRANDE | 10.33 |
R3(Novelty positioning and non-significant gaps within the top group): We agree with this assessment. NADT is not statistically separated from TabR and ModernNCA in the CD diagram, and we do not claim a statistical advantage over them. What we want to emphasize is that NADT achieves the best overall performance. The strong results of TabR and ModernNCA confirm the effectiveness of incorporating sample-wise local information, which is consistent with our motivation. Building on this, NADT further internalizes local information into the splits themselves and achieves stronger expressive power and performance.
After broadening the GBDT tuning budget and correcting the data, NADT attains an average rank of 2.50, ahead of TabR (4.12) and ModernNCA (4.25). It achieves the highest score on 13 datasets, more than TabR (5) and ModernNCA (4). In per-dataset counts, NADT records 20 wins and 10 losses against TabR, and 24 wins and 6 losses against ModernNCA (Section 4.2, Table 1 and Figure 3). Likewise, the rank gaps below the CD should not be taken as evidence that NADT performs on par with TabR and ModernNCA.
This mechanism also yields a separation in expressive power. The corrected proof, which accounts for LayerNorm and degenerate projections, shows that there exist NADT functions that no baseline parameterization can represent (Theorem 1, Appendix B.1). In addition, the inference cost of NADT differs. TabR must retrieve instances from the training set at inference, and ModernNCA must score all training samples. In contrast, NADT's biases depend only on a fixed prototype bank, so the training set is no longer needed after training (Appendix I).
Response to Q1 (strict-containment proof) and Q2 (baseline tuning budget)
We thank the reviewer for the careful reading and the constructive suggestions. We reply point by point below.
Q1
We checked the manuscript and confirmed the issue you raised, and have completed the correction in the revised manuscript.
For matrices with identical columns, we tighten the "nonzero" assumption into an explicit rank condition (Lemma 5, Eq. (13), in Appendix B.1), requiring that the projection of W_ctx on the zero-sum subspace of the affinities be non-degenerate. Matrices with identical columns are directly excluded under this condition.
For the LayerNorm collapse, we first prove a lemma (Lemma 4 in Appendix B.1). For the standard layer normalization
LN(u) = (u − ū·1)/√(v + ε)(where ū and v are the mean and variance of u, 1 is the all-ones vector, and ε > 0 is any fixed constant),LN(u₁) = LN(u₂)if and only ifu₁ − u₂ ∈ span{1}. That is, LN nullifies exactly the translations with equal components and preserves every other difference. This rank condition combined with this characterization guarantees that the projected difference falls outside the null space of LN, so the node biases are non-constant.For "one configuration does not imply strict containment", we construct a NADT function with a specific asymptotic profile (softmax saturation gives its restriction along any ray an exponentially decaying term). We then prove that the restriction of any baseline function along any ray is a rational function and cannot carry such an exponential decay term. This argument holds simultaneously for every baseline parameterization, thereby ruling out all alternative parameterizations and giving
F_base ⊊ F_NADT.The corrected lemma statements and proofs are given in Appendix B.1.
Q2
We have re-run the baselines you mentioned with broadened search spaces, and completed the tuning protocol in the revised manuscript (Appendix E, Table 5; the trial budget is added in Section 4.2). The broadened search spaces are as follows:
All methods use the Optuna TPE sampler with 20 trials per method per dataset. NADT's search space is now explicitly reported in Appendix E, Table 5.
With the broadened budget, the GBDT methods do become stronger on several datasets. For example, LightGBM improves from 0.9732 to 0.9754 on AP_Uterus_Kidney (becoming the best method in that row), XGBoost improves from 0.7190 to 0.7534 on SVHN, and CatBoost improves from 0.4957 to 0.5330 on tumors_C. The full per-dataset results are reported in Section 4.2, Table 1.
The overall conclusion is unchanged. NADT remains the best method. The average ranks of all 11 methods after the re-run are listed below. The best GBDT (LightGBM) still trails NADT. Table 1 and Figure 3 in Section 4.2 have been updated to the re-run results.
In addition, the broadened budget also substantially lengthens the training time of the GBDTs. For example, on SVHN, XGBoost takes 5,569 s and LightGBM takes 3,295 s, versus 595 s for NADT (Section 4.6 and Appendix I, Tables 15–16).