- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper builds a benchmark for decision tree representations. These are different ways to represent a single tree into an object you can compute a similarity on. It covers five existing representations and one new hand-built vector ("Tree Descriptor"). There are three evaluations:
- Perturbation: edit trees synthetically and correlate representation distance with change in OOB MCC and change in MDI feature importance.
- Subforest selection: use the distance matrix inside simulated annealing (objective = diversity + OOB MCC) to pick k trees from a 1000-tree RF, then compare against random, top-k OOB accuracy, and top-k OOB MCC.
- Runtime/memory.
The authors find that perturbation sensitivity doesn't predict subforest utility, and that the authors' representation needs the fewest trees (46 on average) to hit 99% of full-RF MCC. The work uses 25 UCI datasets and evaluates on numeric features with random forset.
Critical strengths
- The paper is easy to read and follow. The three benchmark axes are laid out cleanly, and the appendix documents every representation and selection strategy.
- The problem is well-motivated. Comparing trees comes up constantly (Rashomon sets, drift across retrains, ensemble pruning), and the existing representations were each evaluated only on their own narrow task.
- Having the existing representations in one codebase with a plug-in interface is useful, and the code is released. The validation/test split of datasets for choosing the selection strategy is done properly.
Critical weaknesses
- The perturbation benchmark is partly circular. PP is the tree's predictions, so of course its distance tracks ΔMCC. TD literally contains the normalized feature importance vector, so of course it tracks ΔFI. LP is built from leaf class distributions, which is close to the prediction. The benchmark ends up rewarding representations for containing the target quantity. On top of that, correlations are pooled across intensities lanbda in {0.2,...,1.0}, and both distance and |ΔMCC| grow with lambda. Some of every r in Table 1 is just "both go up with lambda". Within-intensity correlations, or partial correlations controlling for lambda, would be much more convincing.
- In the subforest benchmark I can't tell how much the representation matters versus the SA objective. SA-S optimizes diversity plus OOB MCC, while the baselines are top-k by individual OOB score, and those are well known to pick correlated trees. Missing: SA-S with the distance term removed (MCC only), and SA-S with a random distance matrix. Also, the differences are tiny: in the medium regime everything sits between 1.001 and 1.014 relative MCC with no error bars. In the tiny regime, PP (0.818) and TD (0.855) are no better than random (0.856). Random selection needs about 100 trees to hit the 99% threshold in Fig. 2a/7a, which beats ID, PP, TOA and TOM.
- TD is put forward as the new representation, but its weights (0.5 structure / 0.3 FI / 0.2 thresholds) are picked a priori with no sensitivity analysis. Half the weight goes to structural statistics (size, depth, width, path lengths, balance), which have little to do with what the tree computes. Thresholds are averaged per feature, which throws away where they sit in the tree. It's hard to say why TD wins compression and not the other axes, and the paper doesn't really try to explain it. Basically, I'm not sure this representation is something that is genuinely useful rather than benchmaxxed on the authors' particular evaluation choices.
Smaller note: no discussion of the Rashomon-set / tree-space literature, where comparing many near-equivalent trees is the central problem (e.g., Xin et al. 2022 on enumerating sparse-tree Rashomon sets, TimberTrek), or of SIRUS (stable rule extraction from RFs) and born-again tree ensembles, both of which bear on "compress an RF while keeping FI stable".
Questions for authors
- Can you report the subforest results for SA-S with (a) no distance term and (b) a shuffled distance matrix? If those come close to TD/FG, the representation isn't doing much.
- Perturbation correlations computed within each lambda, and with permutation FI instead of MDI. Does the ranking of representations survive?
- Any sensitivity analysis on TD weights?
2 comments
W3. TD is put forward as the new representation, but its weights (0.5 structure / 0.3 FI / 0.2 thresholds) are picked a priori with no sensitivity analysis. Half the weight goes to structural statistics (size, depth, width, path lengths, balance), which have little to do with what the tree computes. Thresholds are averaged per feature, which throws away where they sit in the tree. It's hard to say why TD wins compression and not the other axes, and the paper doesn't really try to explain it. Basically, I'm not sure this representation is something that is genuinely useful rather than benchmaxxed on the authors' particular evaluation choices.
Q3. Any sensitivity analysis on TD weights?
A3 (W3 & Q3). We thank the reviewer for highlighting these design choices. The original weights were set a priori and deliberately left unoptimized, to show that even a simple, intuitive configuration can provide a useful representation. We agree that this alone does not establish the robustness or usefulness of TD, and we have therefore added a sensitivity analysis (Appendices A.2.6 and A.3.5). We evaluated 67 weight configurations, comprising the full simplex grid at resolution plus equal weights, each with two threshold encodings: the mean threshold per feature and a new dominant-split encoding that retains the shallowest split per feature, partially preserving its position in the tree. Since the grid includes zero weights, it also covers all one- and two-component ablations. Configurations were selected on the validation datasets using the perturbation benchmark only (due to the computational needs of the subforest benchmark), and the selected configuration was then evaluated on the held-out test datasets in both the perturbation and subforest benchmarks. The selected configuration reduces the structural weight and improves all four correlations on the test datasets (Table 8). Without being tuned for this task, it also reaches % of full-forest MCC with trees on average instead of (Figure 10).
Smaller note. No discussion of the Rashomon-set / tree-space literature, where comparing many near-equivalent trees is the central problem (e.g., Xin et al. 2022 on enumerating sparse-tree Rashomon sets, TimberTrek), or of SIRUS (stable rule extraction from RFs) and born-again tree ensembles, both of which bear on "compress an RF while keeping FI stable".
A4. We also thank the reviewer for pointing us to this relevant literature, which helps position our work within recent research on decision trees. We have added an opening paragraph to the Related Work section (Section 2) that discusses the need for tree comparison across settings.
W1. The perturbation benchmark is partly circular. PP is the tree's predictions, so of course its distance tracks ΔMCC. TD literally contains the normalized feature importance vector, so of course it tracks ΔFI. LP is built from leaf class distributions, which is close to the prediction. The benchmark ends up rewarding representations for containing the target quantity. On top of that, correlations are pooled across intensities lanbda in {0.2,...,1.0}, and both distance and |ΔMCC| grow with lambda. Some of every r in Table 1 is just "both go up with lambda". Within-intensity correlations, or partial correlations controlling for lambda, would be much more convincing.
Q2. Perturbation correlations computed within each lambda, and with permutation FI instead of MDI. Does the ranking of representations survive?
A1 (W1 & Q2). We thank the reviewer for pointing out these limitations. To reduce circularity, MCC is now computed on the held-out test fold rather than on OOB samples, since PP encodes training-set predictions. We also replaced MDI with permutation feature importance, since TD encodes MDI scores. In addition, we added test and volume disagreement as direct measures of prediction change (Section 3.3, Appendix A.2.3). Volume disagreement is computed exactly over the input space and is therefore independent of the samples on which PP and LP are built. Some alignment remains inherent, as representations encoding predictions are expected to track prediction changes.
As the reviewer noted, both distance and functional change grow with intensity. We now report within-intensity correlations (Table 1) and, as a robustness check, partial correlations controlling for intensity as a discrete factor (Table 5). The ranking reported in the previous analysis survives. Both identify the same best representation in every perturbation–metric pair, with one tie between PP and LP in Table 5. The repeated significance analysis confirms that PP and LP consistently form the top group, significantly outperforming most other representations (Figure 4).
W2. In the subforest benchmark I can't tell how much the representation matters versus the SA objective. SA-S optimizes diversity plus OOB MCC, while the baselines are top-k by individual OOB score, and those are well known to pick correlated trees. Missing: SA-S with the distance term removed (MCC only), and SA-S with a random distance matrix. Also, the differences are tiny: in the medium regime everything sits between 1.001 and 1.014 relative MCC with no error bars. In the tiny regime, PP (0.818) and TD (0.855) are no better than random (0.856). Random selection needs about 100 trees to hit the 99% threshold in Fig. 2a/7a, which beats ID, PP, TOA and TOM.
Q1. Can you report the subforest results for SA-S with (a) no distance term and (b) a shuffled distance matrix? If those come close to TD/FG, the representation isn't doing much.
A2 (W2 & Q1). We thank the reviewer and agree that these controls are necessary. We repeated the subforest benchmark with (i) SA-S as originally proposed, (ii) SA-S with the distance term removed (MCC only), and (iii) SA-S with a shuffled distance matrix, obtained by randomly permuting its upper-triangular off-diagonal entries while preserving symmetry and the zero diagonal (new Appendix A.3.3, Table 6). Removing the distance term never yields the best compression, showing that optimizing predictive performance alone is not sufficient. However, the shuffled matrix compresses as well as or better than the original distances for most representations. This indicates that much of the benefit stems from the diversity pressure of the distance term rather than from representation-specific pairwise relations. TD is the exception: with its original distances it achieves a clearly better mean relative subforest size (0.046 vs. 0.089 shuffled), lower variability across datasets (0.084 vs. 0.229), and a better AUC (0.112 vs. 0.117). We report this finding in the main text and have moderated our claims accordingly.