- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces the transport-margin profile, which couples the class-conditional feature distributions at a tree node using one-dimensional monotone optimal transport. The profile provides a unified view of several split notions: its value at zero recovers KS-style separation and is closely related to Gini gain, its survival radius recovers a hard-margin notion, and its values at positive radii correspond to robust balanced-accuracy advantages. The proposed transport-margin splitting (TMS) criterion integrates this profile up to a radius cap and can be evaluated for all thresholds in linear time after sorting. The paper further provides consistency/stability results and evaluates TMS through matched random-forest experiments, synthetic mechanism studies, fixed-budget robust baselines, and exact/certified worst-case robustness experiments.
Critical strengths
-
A strong conceptual and theoretical unification. The transport-margin profile connects impurity/KS-style splitting, geometric margins, fixed-radius robustness, and Wasserstein geometry through a single object. The layer-cake and Wasserstein decomposition results make the proposed criterion more than a heuristic robustness objective.
-
Careful validation of the proposed mechanism. The experiments largely isolate the effect of the split criterion through matched forests, include fixed-budget robust baselines and cheap alternatives, and complement average-case noise experiments with exact worst-case evaluation and certification. The paper also explicitly studies important failure regimes rather than hiding them.
Critical weaknesses
-
The statistical treatment of the exact-robustness experiments appears inconsistent with the stated experimental unit. The appendix states that the dataset is the experimental unit, but several exact-robustness comparisons report Wilcoxon tests over individual train/test splits. Since splits from the same dataset are not independent experimental units, these p-values may be anti-conservative. The qualitative direction of the results appears strong, so this seems fixable rather than fatal.
-
The demonstrated generality is narrower than the overall framing. Exact worst-case experiments are limited to relatively small depth-limited forests on six numeric binary datasets, and the method can degrade substantially in interaction-dominated/high-noise-feature regimes. The proposed signal-density fallback is sensible, but its empirical validation is still relatively limited. The method also currently excludes categorical features and missing values.
Questions for authors
-
For the exact worst-case experiments, can the authors report statistical inference with the dataset, rather than individual data splits, as the experimental unit (or alternatively use an appropriate hierarchical procedure)? Does the main significance conclusion remain unchanged?
-
How sensitive are the recommended defaults, particularly the signal-density cutoff and , to dataset selection? Clarifying whether these values were fixed before evaluating the corresponding held-out datasets, or providing a leave-one-dataset-out sensitivity check, would materially strengthen the claim that these are validation-free defaults.
Thank you for the careful reading. Your summary is accurate, and both questions led to analyses that we report here.
Q1. Dataset-level inference for the exact worst-case experiments. You are right: Appendix G names the dataset as the experimental unit, and these comparisons should follow it. Recomputed with the dataset as the unit (five-split mean per dataset, six datasets):
With six datasets, 0.031 is the smallest p-value an exact test can reach. The bootstrap resamples datasets, then splits within each dataset. The main conclusion is unchanged: TMS beats the matched Gini forest on every dataset at ε = 0.05 and ε = 0.1, and the gain survives dataset-level inference. Without spambase, the mean gain at ε = 0.1 is still +3.2 points. The one change is that GROOT trained at the attack radius leads TMS at that radius with p = 0.06 instead of p < 0.001. The ties stay ties. The revised PDF reports the dataset-level tests in place of the split-level ones.
Q2. Sensitivity of the defaults to dataset selection.
Radius cap. R = 0.1 is one value used throughout, never tuned per dataset. A leave-one-dataset-out choice of R from {0.05, 0.1, 0.2} on the 19-dataset benchmark picks R = 0.2 in all 19 folds. Held-out robustness AUC rises by 0.4 points on average (the largest change on any dataset is 1.5), and the gain over Gini rises from +1.6 points (14 of 19 wins, p = 0.006) to +2.1 points (17 of 19, p = 0.0002). R = 0.05 gives +1.6 (p = 0.004). The default is conservative rather than selected, and the conclusion holds across the window. Dropping any one dataset leaves a mean gain of at least +0.95 points.
Density cutoff. The cutoff 0.1 was set on the 19 benchmark datasets and frozen before the out-of-sample checks were run: eight OpenML datasets outside the benchmark (Table 17) and six constructed interaction families. On the external datasets the rule flags the only one where TMS clearly trails (dexter, −3.4 AUC points), costs at most 1.3 points where falling back was unnecessary, and beats always using TMS by 0.5 clean and 0.4 AUC points. The decision is not knife-edge: every cutoff from 0.04 to 0.15 makes the same decision on all 19 benchmark datasets and all 24 external splits. The external check is small (three splits per dataset), so we read it as bounding the rule's cost rather than measuring its gain.
On generality. Exact verification of tree ensembles is NP-complete in general (Kantchelian et al., 2016; Chen et al., 2019b), and depth-6 forests of 20 trees are where the MILP stays exact at under a second per instance. The polynomial LP certificate is within 0.011 of the exact value at ε ≤ 0.05 for every forest tested and within 0.01 at every budget for TMS forests (Table 18), so it can carry the evaluation to larger forests. The average-case robustness study uses forests without a depth limit on all 19 datasets, and the multiclass study adds five datasets with up to ten classes. The interaction-dominated regime limits every per-feature criterion, Gini included. Proposition 5 shows that when each feature's class conditionals coincide, all such criteria sit at their uninformative value. We report this regime rather than hide it, as you note, and the density test identifies it before training. Categorical features and missing values are outside the scope we tested, as Section 4 states.