acceptodds
Under review as a conference paper at ICLR 2027

DTRBench: A Benchmarking Framework for Decision Tree Representations

Abstract

Due to their efficiency, interpretability, and ensemble capabilities, decision trees are foundational to machine learning, with recent benchmark studies showing that tree ensembles remain competitive with deep learning on tabular data. Yet, systematically comparing individual trees remains difficult due to structural and semantic variability. While decision tree representations enable pairwise comparisons, existing representation methods have only been developed in isolation on narrow tasks. Consequently, their utility and broader applicability cannot be judged. To address this gap, we present DTRBench, a modular benchmarking framework for evaluating decision tree representations across three axes: task-independent sensitivity to controlled perturbations, utility for downstream subforest selection, and computational efficiency. Across 25 datasets, we benchmark five representations from the literature, spanning vector-, graph-, and neural-based paradigms, alongside an illustrative new hybrid vector representation we call “Tree Descriptor”. The results show that local perturbation sensitivity does not translate to downstream subforest selection utility, highlighting the need for measuring them as decoupled properties. Furthermore, DTRBench reveals that subforest selection based on Tree Descriptor representations requires up to 73% fewer trees than standard heuristics to recover 99% of the full random forest's predictive performance, thereby substantially accelerating inference. Ultimately, DTRBench provides an extensible, standardized, and open-source foundation for evaluating and selecting decision tree representations, revealing distinct strengths and limitations across representations.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
decision tree representation, decision tree, representation, benchmark, random forest, subforest

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fwrwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper builds a benchmark for decision tree representations. These are different ways to represent a single tree into an object you can compute a similarity on. It covers five existing representations and one new hand-built vector ("Tree Descriptor"). There are three evaluations:

  1. Perturbation: edit trees synthetically and correlate representation distance with change in OOB MCC and change in MDI feature importance.
  2. Subforest selection: use the distance matrix inside simulated annealing (objective = diversity + OOB MCC) to pick k trees from a 1000-tree RF, then compare against random, top-k OOB accuracy, and top-k OOB MCC.
  3. Runtime/memory.

The authors find that perturbation sensitivity doesn't predict subforest utility, and that the authors' representation needs the fewest trees (46 on average) to hit 99% of full-RF MCC. The work uses 25 UCI datasets and evaluates on numeric features with random forset.

Critical strengths
  • The paper is easy to read and follow. The three benchmark axes are laid out cleanly, and the appendix documents every representation and selection strategy.
  • The problem is well-motivated. Comparing trees comes up constantly (Rashomon sets, drift across retrains, ensemble pruning), and the existing representations were each evaluated only on their own narrow task.
  • Having the existing representations in one codebase with a plug-in interface is useful, and the code is released. The validation/test split of datasets for choosing the selection strategy is done properly.
Critical weaknesses
  1. The perturbation benchmark is partly circular. PP is the tree's predictions, so of course its distance tracks ΔMCC. TD literally contains the normalized feature importance vector, so of course it tracks ΔFI. LP is built from leaf class distributions, which is close to the prediction. The benchmark ends up rewarding representations for containing the target quantity. On top of that, correlations are pooled across intensities lanbda in {0.2,...,1.0}, and both distance and |ΔMCC| grow with lambda. Some of every r in Table 1 is just "both go up with lambda". Within-intensity correlations, or partial correlations controlling for lambda, would be much more convincing.
  2. In the subforest benchmark I can't tell how much the representation matters versus the SA objective. SA-S optimizes diversity plus OOB MCC, while the baselines are top-k by individual OOB score, and those are well known to pick correlated trees. Missing: SA-S with the distance term removed (MCC only), and SA-S with a random distance matrix. Also, the differences are tiny: in the medium regime everything sits between 1.001 and 1.014 relative MCC with no error bars. In the tiny regime, PP (0.818) and TD (0.855) are no better than random (0.856). Random selection needs about 100 trees to hit the 99% threshold in Fig. 2a/7a, which beats ID, PP, TOA and TOM.
  3. TD is put forward as the new representation, but its weights (0.5 structure / 0.3 FI / 0.2 thresholds) are picked a priori with no sensitivity analysis. Half the weight goes to structural statistics (size, depth, width, path lengths, balance), which have little to do with what the tree computes. Thresholds are averaged per feature, which throws away where they sit in the tree. It's hard to say why TD wins compression and not the other axes, and the paper doesn't really try to explain it. Basically, I'm not sure this representation is something that is genuinely useful rather than benchmaxxed on the authors' particular evaluation choices.

Smaller note: no discussion of the Rashomon-set / tree-space literature, where comparing many near-equivalent trees is the central problem (e.g., Xin et al. 2022 on enumerating sparse-tree Rashomon sets, TimberTrek), or of SIRUS (stable rule extraction from RFs) and born-again tree ensembles, both of which bear on "compress an RF while keeping FI stable".

Questions for authors
  1. Can you report the subforest results for SA-S with (a) no distance term and (b) a shuffled distance matrix? If those come close to TD/FG, the representation isn't doing much.
  2. Perturbation correlations computed within each lambda, and with permutation FI instead of MDI. Does the ranking of representations survive?
  3. Any sensitivity analysis on TD weights?
2 comments

W1. The perturbation benchmark is partly circular. PP is the tree's predictions, so of course its distance tracks ΔMCC. TD literally contains the normalized feature importance vector, so of course it tracks ΔFI. LP is built from leaf class distributions, which is close to the prediction. The benchmark ends up rewarding representations for containing the target quantity. On top of that, correlations are pooled across intensities lanbda in {0.2,...,1.0}, and both distance and |ΔMCC| grow with lambda. Some of every r in Table 1 is just "both go up with lambda". Within-intensity correlations, or partial correlations controlling for lambda, would be much more convincing.

Q2. Perturbation correlations computed within each lambda, and with permutation FI instead of MDI. Does the ranking of representations survive?

A1 (W1 & Q2). We thank the reviewer for pointing out these limitations. To reduce circularity, MCC is now computed on the held-out test fold rather than on OOB samples, since PP encodes training-set predictions. We also replaced MDI with permutation feature importance, since TD encodes MDI scores. In addition, we added test and volume disagreement as direct measures of prediction change (Section 3.3, Appendix A.2.3). Volume disagreement is computed exactly over the input space and is therefore independent of the samples on which PP and LP are built. Some alignment remains inherent, as representations encoding predictions are expected to track prediction changes.

As the reviewer noted, both distance and functional change grow with intensity. We now report within-intensity correlations (Table 1) and, as a robustness check, partial correlations controlling for intensity as a discrete factor (Table 5). The ranking reported in the previous analysis survives. Both identify the same best representation in every perturbation–metric pair, with one tie between PP and LP in Table 5. The repeated significance analysis confirms that PP and LP consistently form the top group, significantly outperforming most other representations (Figure 4).

W2. In the subforest benchmark I can't tell how much the representation matters versus the SA objective. SA-S optimizes diversity plus OOB MCC, while the baselines are top-k by individual OOB score, and those are well known to pick correlated trees. Missing: SA-S with the distance term removed (MCC only), and SA-S with a random distance matrix. Also, the differences are tiny: in the medium regime everything sits between 1.001 and 1.014 relative MCC with no error bars. In the tiny regime, PP (0.818) and TD (0.855) are no better than random (0.856). Random selection needs about 100 trees to hit the 99% threshold in Fig. 2a/7a, which beats ID, PP, TOA and TOM.

Q1. Can you report the subforest results for SA-S with (a) no distance term and (b) a shuffled distance matrix? If those come close to TD/FG, the representation isn't doing much.

A2 (W2 & Q1). We thank the reviewer and agree that these controls are necessary. We repeated the subforest benchmark with (i) SA-S as originally proposed, (ii) SA-S with the distance term removed (MCC only), and (iii) SA-S with a shuffled distance matrix, obtained by randomly permuting its upper-triangular off-diagonal entries while preserving symmetry and the zero diagonal (new Appendix A.3.3, Table 6). Removing the distance term never yields the best compression, showing that optimizing predictive performance alone is not sufficient. However, the shuffled matrix compresses as well as or better than the original distances for most representations. This indicates that much of the benefit stems from the diversity pressure of the distance term rather than from representation-specific pairwise relations. TD is the exception: with its original distances it achieves a clearly better mean relative subforest size (0.046 vs. 0.089 shuffled), lower variability across datasets (0.084 vs. 0.229), and a better AUC (0.112 vs. 0.117). We report this finding in the main text and have moderated our claims accordingly.

W3. TD is put forward as the new representation, but its weights (0.5 structure / 0.3 FI / 0.2 thresholds) are picked a priori with no sensitivity analysis. Half the weight goes to structural statistics (size, depth, width, path lengths, balance), which have little to do with what the tree computes. Thresholds are averaged per feature, which throws away where they sit in the tree. It's hard to say why TD wins compression and not the other axes, and the paper doesn't really try to explain it. Basically, I'm not sure this representation is something that is genuinely useful rather than benchmaxxed on the authors' particular evaluation choices.

Q3. Any sensitivity analysis on TD weights?

A3 (W3 & Q3). We thank the reviewer for highlighting these design choices. The original weights (0.5,0.3,0.2)(0.5, 0.3, 0.2) were set a priori and deliberately left unoptimized, to show that even a simple, intuitive configuration can provide a useful representation. We agree that this alone does not establish the robustness or usefulness of TD, and we have therefore added a sensitivity analysis (Appendices A.2.6 and A.3.5). We evaluated 67 weight configurations, comprising the full simplex grid at resolution 0.10.1 plus equal weights, each with two threshold encodings: the mean threshold per feature and a new dominant-split encoding that retains the shallowest split per feature, partially preserving its position in the tree. Since the grid includes zero weights, it also covers all one- and two-component ablations. Configurations were selected on the validation datasets using the perturbation benchmark only (due to the computational needs of the subforest benchmark), and the selected configuration was then evaluated on the held-out test datasets in both the perturbation and subforest benchmarks. The selected configuration (0.3,0.6,0.1)(0.3, 0.6, 0.1) reduces the structural weight and improves all four correlations on the test datasets (Table 8). Without being tuned for this task, it also reaches 9999% of full-forest MCC with 2424 trees on average instead of 4646 (Figure 10).

Smaller note. No discussion of the Rashomon-set / tree-space literature, where comparing many near-equivalent trees is the central problem (e.g., Xin et al. 2022 on enumerating sparse-tree Rashomon sets, TimberTrek), or of SIRUS (stable rule extraction from RFs) and born-again tree ensembles, both of which bear on "compress an RF while keeping FI stable".

A4. We also thank the reviewer for pointing us to this relevant literature, which helps position our work within recent research on decision trees. We have added an opening paragraph to the Related Work section (Section 2) that discusses the need for tree comparison across settings.

Reviewer_9cveOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DTRBench, a benchmarking framework for comparing different decision tree representations. The framework evaluates multiple representations along three dimensions, including perturbation sensitivity, downstream utility for subforest selection, and computational efficiency. Experiments on 25 UCI datasets show that different representations perform differently across these evaluation dimensions.

Critical strengths
  1. Comparing different representations of decision trees could be useful for downstream tasks such as tree similarity analysis, ensemble compression, and model stability analysis.

  2. The paper conducts a relatively comprehensive empirical study across multiple representations and datasets. The proposed framework also provides a unified implementation that could be used for future study.

Critical weaknesses
  1. The significance of the proposed benchmark is not fully established. It is unclear what important new insights the benchmark provides beyond the observation that different representations are suitable for different evaluation objectives, and what kind of downstream applications can be better addressed by this study.

  2. The perturbation benchmark is built upon the assumption that representation distances should correlate with changes in MCC and feature importance. However, it's unclear why this assumption is valid or why it is important to compare different representations. For example, two trees can have similar predictive performance but make very different predictions on individual samples. Feature importance captures only one aspect of model behavior. Therefore, it's unclear why the proposed benchmark can meaningfully evaluate how well a representation captures functional differences between trees.

  3. Subforest selection is evaluated as a separate benchmark. However, if the downstream task is clearly defined, such as selecting a small subset of trees that maintains predictive performance similar to that of the full random forest, why not directly optimize this objective? In other words, if the objective is clear, why not use an appropriate representation to directly address the task?

Questions for authors
  1. Random forest with max depth 10 is used for most experiments. However, different representations may have different sensitivities to tree depth and complexity, which could limit the generalizability of the obtained conclusion. For example, the computational cost of some representations may increase substantially with tree depth.

  2. The study discards all categorical features. But decision trees are well suited for handling categorical features, and removing these features can substantially change the underlying prediction tasks and resulting trees. The implementation limitations of the selected library do not seem to fully justify this problem.

3 comments

W1. The significance of the proposed benchmark is not fully established. It is unclear what important new insights the benchmark provides beyond the observation that different representations are suitable for different evaluation objectives, and what kind of downstream applications can be better addressed by this study.

A1. We thank the reviewer for the opportunity to clarify the significance of this work. To our knowledge, DTRBench is the first framework to collect the main decision tree representations from the literature and evaluate them under common criteria. It does so along three complementary axes: intrinsic evaluation through controlled perturbations, extrinsic evaluation on a downstream task (subforest selection), and computational efficiency. It also contributes a new representation (TD), showing that even a simple, intuitive design can outperform existing representations and strong baselines in a given task.

Beyond evaluation, DTRBench supports the development of representations. In the new Appendices A.2.6 and A.3.5, we show how the benchmark can be used to improve a representation. We test different weights for combining the three components of TD and two threshold encoding strategies, select the configuration on the validation datasets, and confirm the improvement on the held-out test datasets. The same protocol applies to any new representation or tunable component.

Regarding downstream applications, subforest selection is currently the only implemented task, but the framework is designed to accommodate new ones. We have added an interface for registering new downstream tasks (Appendix A.1.4), analogous to those for representations, datasets, perturbations, and selection strategies. A registered task reuses the framework’s existing machinery: dataset loading and CV splits, RF training, representation construction, pairwise distance computation, and the validation/test protocol. It therefore only needs to define its own task-specific population of trees and evaluation. This allows tasks such as anomalous tree detection, drift detection, or functional clustering of trees to be evaluated under the same standardized conditions as subforest selection.

W2. The perturbation benchmark is built upon the assumption that representation distances should correlate with changes in MCC and feature importance. However, it's unclear why this assumption is valid or why it is important to compare different representations. For example, two trees can have similar predictive performance but make very different predictions on individual samples. Feature importance captures only one aspect of model behavior. Therefore, it's unclear why the proposed benchmark can meaningfully evaluate how well a representation captures functional differences between trees.

A2. We thank the reviewer and agree that trees can achieve similar performance while disagreeing on many individual samples, so ΔMCC and ΔFI alone do not measure functional difference. We have therefore added two direct measures to the perturbation benchmark (Section 3.3, Appendix A.2.3). Test disagreement is the fraction of held-out samples on which the trees predict different classes, which captures exactly the sample-level differences the reviewer describes. Volume disagreement is the fraction of the input space on which they disagree. It is computed exactly via leaf-region intersections and equals zero if and only if the trees are functionally equivalent. The results reflect the reviewer’s point: correlations with MCC loss are consistently lower than with prediction disagreement, because changed predictions can either harm or improve performance and thus partially cancel out in MCC (Table 1). We retain ΔMCC and ΔFI not as measures of functional difference, but as proxies for two aspects that matter in practice: how well a model predicts and which inputs drive its decisions.

W3. Subforest selection is evaluated as a separate benchmark. However, if the downstream task is clearly defined, such as selecting a small subset of trees that maintains predictive performance similar to that of the full random forest, why not directly optimize this objective? In other words, if the objective is clear, why not use an appropriate representation to directly address the task?

A3. We thank the reviewer for this question. DTRBench is designed for comparative evaluation rather than as a task-specific optimizer. Its goal is to assess how well different representations capture tree characteristics across objectives, and subforest selection serves as one downstream task among the many the framework can host. The framework's new register_task interface (Appendix A.1.4) allows researchers to plug in any custom downstream task while reusing DTRBench's standardized machinery. Representations designed for subforest selection are nevertheless part of the comparison: TF was originally developed for this task.

Directly optimizing the subforest objective is also not always possible or desirable. Evaluating candidate subforests requires access to data and repeated predictions from the ensemble for every candidate. In contrast, most representations can be computed from the fitted trees alone, together with information stored in them, such as the number of samples reaching each node, or with summary statistics such as feature ranges or correlations. Similarly, the baselines TOM and TOA require only a stored performance score per tree. Representation-based selection therefore also applies when the original data are unavailable, e.g., for privacy reasons, in federated settings, or when trees are retrieved from model repositories. Moreover, a representation is computed once per tree and can be reused for any subforest size or downstream task, whereas direct optimization must be repeated for every target size.

Q1. Random forest with max depth 10 is used for most experiments. However, different representations may have different sensitivities to tree depth and complexity, which could limit the generalizability of the obtained conclusion. For example, the computational cost of some representations may increase substantially with tree depth.

A4. We appreciate the reviewer’s concern regarding the impact of tree depth and model complexity. To test whether our conclusions on computational cost generalize beyond a maximum depth of 10, we conducted additional resource benchmarks across maximum depths d∈{4,6,8,10}d \in \{4, 6, 8, 10\}. For each representation, we fit a power law relating per-operation cost to maximum depth via linear regression in log-log space, yielding a scaling exponent γ\gamma. The methodology and results are reported in Appendices A.2.5 and A.3.4 (Table 7). Runtime shows modest depth dependence, with exponents of 0.30.3–0.40.4 for representation construction and 0.00.0–0.50.5 for similarity computation, while peak RAM and representation size remain nearly constant.

Q2. The study discards all categorical features. But decision trees are well suited for handling categorical features, and removing these features can substantially change the underlying prediction tasks and resulting trees. The implementation limitations of the selected library do not seem to fully justify this problem.

A5. We thank the reviewer for this comment and agree that discarding categorical features required further justification. Our default configuration restricts datasets to numerical features because scikit-learn’s DecisionTreeClassifier, as well as threshold-based representations such as TD and ID, support only binary threshold splits. To assess whether our conclusions depend on this choice, we repeated the analyses with categorical features included (new Appendix A.3.6). We avoided one-hot encoding because it can bias split selection and dilute or inflate feature importance. Instead, each categorical feature is mapped to a single ordinal feature by ranking its levels by their class distribution, estimated on the training fold only. For binary tasks, levels are ordered by the proportion of the positive class. A threshold split on this ordering is then equivalent to the optimal binary partition of the levels under the Gini criterion [1], so trees can represent the same categorical splits as a native implementation. For multiclass tasks, levels are ordered by the first principal component of their class-proportion vectors, which approximates the optimal partition [2]. This affects 13 of the 25 datasets, for which we reran the perturbation and subforest benchmarks.

The perturbation results are essentially unchanged. PP and LP remain the top two representations in every perturbation–metric pair except r-MCC for remove nodes, and the only three ranking changes are swaps between adjacent lower-ranked representations (Table 11). In the subforest benchmark, all representations still outperform all performance-based baselines in compression, both in AUC and in the mean subforest size required to recover 99% of the full forest’s MCC. The advantage of the refined TD (TDO) over other representations is weaker with categorical features, but it remains best in median subforest size (Table 12). Overall, our main conclusions hold when categorical features are included.

[1] Walter D Fisher. On grouping for maximum homogeneity. Journal of the American Statistical Association, 53(284):789–798, December 1958. doi: 10.1080/01621459.1958.10501479.

[2] Don Coppersmith, Se June Hong, and Jonathan R M Hosking. Partitioning nominal attributes in decision trees. Data Mining and Knowledge Discovery, 3(2):197–217, June 1999. doi: 10.1023/a:1009869804967.

Reviewer_J1kBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the lack of standardized evaluation for decision tree representations used to compare individual decision trees. These representations were developed and tested in isolation on single tasks, making it unclear how well they capture tree behavior, how fast they run, or how useful they are for broader applications. To solve this, the authors introduce DTRBench, an open-source benchmark across 25 classification datasets that evaluates representations in three distinct dimensions: sensitivity to controlled tree modifications (Perturbation Benchmark), effectiveness in shrinking large Random Forests into smaller subforests (Subforest Benchmark), and computational runtime and memory usage (Resource Benchmark). The authors evaluate five existing representations and new representation they propose called Tree Descriptor (TD), which combines tree structure, feature importance, and split thresholds. Their experiments show that sensitivity to tree modifications does not predict how well a representation performs in subforest selection, demonstrating that representations must be tested across multiple tasks.

Critical strengths
  • Authors proposed first unified multi-axis benchmark for DT representations
  • Authors show empirically that that local sensitivity to tree perturbations does not translate to downstream utility in subforest selection
  • While the representations inside DTRBench are heuristic, the Perturbation Benchmark itself is a valid diagnostic tool
Critical weaknesses
  • the representations, similarity metrics, and the proposed Tree Descriptor (TD) rely heavily on ad-hoc, lossy heuristics rather than exact functional or geometric equivalence
  • TD computes three separate similarity scores and combines them via an arbitrary, hardcoded linear combination
  • In threshold averaging, the decision boundaries of a tree are encoded by taking the average split threshold for each feature across the entire tree. In a decision tree, what matters is the conditional hierarchy. Averaging those into a single number destroys the actual spatial partitioning of the feature space, making it useless for true interpretability.
  • In Table 1 (Perturbation Benchmark), when a split feature is changed (Change feature), the correlation between representation distance and the actual change in predictive performance (MCC) is 0.08 for TF, 0.25 for ID, 0.31 for FG, and 0.38 for the authors' own TD. Even the best-performing methods (LP and PP) only reach 0.56 and 0.54. This directly validates the claim that these metrics are largely inaccurate at tracking how tree modifications alter actual predictions
Questions for authors

Please address weaknesses

2 comments

W1. The representations, similarity metrics, and the proposed Tree Descriptor (TD) rely heavily on ad-hoc, lossy heuristics rather than exact functional or geometric equivalence

A1. Our focus is benchmarking DT representations and providing a platform to develop improved representations. Exact pairwise distances can be computed by comparing the partitions trees induce, but such algorithmic methods do not produce intermediate representations that can be stored, transported, communicated, or reused for other downstream tasks (Section 2). Representations are abstractions by design, and DTRBench quantifies how much functional behavior each one retains.

We agree that performance and FI differences are indirect proxies of functional change. We therefore added two direct measures to the perturbation benchmark (Section 3.3, Appendix A.2.3). Test disagreement is the fraction of test samples on which the two trees predict differently. Volume disagreement is the fraction of input space on which they disagree, computed exactly via leaf-box intersections; it is zero if and only if the trees are functionally equivalent. Rankings are consistent with the proxy measures (Table 1).

For FG, TF, ID, LP, and PP, the representations follow the original works. New choices were made only where no similarity measure was defined, and these were applied consistently across representations, e.g., the same correlation adjustment in FG and TD. The TD weights were initially set a priori, without optimization, to show that even a simple representation can be competitive. In the new Appendices A.2.6 and A.3.5, we optimize TD’s two weak points, the threshold encoding and the component weights, on the validation datasets. The optimized TD improves all four correlations on the test datasets (Table 8) and improves relative test MCC in subforest selection (Table 10). This analysis exemplifies how DTRBench can guide representation development.

W2. TD computes three separate similarity scores and combines them via an arbitrary, hardcoded linear combination.

A2. The original weights (0.5,0.3,0.2) were set a priori, reflecting the assumed relative importance of structure, features, and thresholds. They were intentionally left unoptimized to show that even a simple, intuitive representation can be competitive with existing ones. We agree, however, that this choice should be examined systematically, and we have used DTRBench to do so (new Appendix A.2.6 and A.3.5). We evaluated 67 weight configurations, comprising the full simplex grid at resolution 0.1 plus equal weights, each combined with two threshold encodings (mean threshold and dominant split). Because the grid includes zero weights, it also covers all one- and two-component ablations. Configurations were scored on the perturbation benchmark using the validation datasets only and perturbation and subforest benchmark results for the optimised configuration were reported for the test datasets.

W3. In threshold averaging, the decision boundaries of a tree are encoded by taking the average split threshold for each feature across the entire tree. In a decision tree, what matters is the conditional hierarchy. Averaging those into a single number destroys the actual spatial partitioning of the feature space, making it useless for true interpretability.

A3. We thank the reviewer and agree that averaging split thresholds per feature discards the conditional hierarchy and can dilute the signal of the threshold component. Using DTRBench, we compared this encoding with a new dominant-split encoding (new Appendices A.2.6 and A.3.5). For each feature, the dominant-split encoding retains the threshold of its most influential split: the shallowest one, with ties broken by the number of samples reaching the node, expressed on the feature’s training-quantile scale. This partially reintroduces the hierarchy while keeping TD a compact, fixed-size vector. All four metric-specific optima use the dominant-split encoding (Table 8, Figure 9). The joint weight analysis, which includes all ablations, assigns a non-zero threshold weight in three of the four metric-specific optima and a weight of 0.1 in the selected configuration, indicating that thresholds contribute complementary information. Further refinements, such as sample-weighted threshold quantiles or multiple thresholds per feature, can be evaluated within the same pipeline and are left for future work.

W4. In Table 1 (Perturbation Benchmark), when a split feature is changed (Change feature), the correlation between representation distance and the actual change in predictive performance (MCC) is 0.08 for TF, 0.25 for ID, 0.31 for FG, and 0.38 for the authors' own TD. Even the best-performing methods (LP and PP) only reach 0.56 and 0.54. This directly validates the claim that these metrics are largely inaccurate at tracking how tree modifications alter actual predictions

A4. We have revised the perturbation analysis as follows. First, we added two direct functional measures, test disagreement and volume disagreement (Section 3.3, Appendix A.2.3), so that representation distance is evaluated against the actual change in predictions rather than only performance proxies. Second, we reduced potential circularity between representations and reference measures: MCC is now computed on the held-out test fold rather than OOB samples, since PP encodes predictions on the training set, and MDI feature importance was replaced with permutation importance, since TD directly encodes MDI scores. Third, because both representation distance and functional change grow with perturbation intensity, pooled correlations could be inflated by intensity alone. We therefore report correlations computed within each intensity (Table 1) and, as a robustness check, partial correlations controlling for intensity as a discrete factor (Table 5), without assuming a linear effect. Both analyses yield the same best representation in every perturbation–metric pair.

Consistent with our previous results, PP and LP clearly lead, typically with statistically significant margins (Figure 4), followed by TD and FG, with TF and ID last. PP and LP achieve medium to high correlations across perturbations; for instance, PP exceeds 0.8 with test disagreement for every perturbation. Correlations with MCC loss are lower than with prediction disagreement for all representations because changed predictions can either harm or improve performance and are therefore only partially reflected in MCC.

We agree that several representations track functional change poorly, but we consider this a key result of the benchmark rather than a limitation of it. Notably, PP, which tracks functional change best, performs worst in subforest selection at small sizes (Table 2), whereas TF, among the least sensitive, performs best. A representation that faithfully captures local functional change is thus not necessarily the one that captures the heterogeneity relevant for a given downstream task. This supports our central claim that intrinsic sensitivity and extrinsic utility are distinct properties that must be evaluated jointly, which is the purpose of DTRBench.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.