- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a new way to measure sensitivity in decision tree ensembles weighting input regions based on how likely they actually occur in real data, rather than treating all inputs equally. The topic is relevant for trustworthy AI because models often encounter data distributions different from their training data. The authors introduce an approximate algorithm called Weighted-XCount using decision diagrams to compute these weighted sensitivity scores. The experiments are conducted using 8 tabular datasets and considering distinct model sizes. Results shows that the method scales better than an exact counting baseline and concludes that weighting significantly improves result.
Critical strengths
-
Overall paper is well written with a logical structure and easy to follow.
-
The shift from traditional formal verification and sensitivity analysis treats all parts of the input space uniformly (unweighted) to consider a data-aware approach that accounts for the actual likelihood or rarity of different regions under the data distribution. The approach is also theoretically sound.
-
Authors demonstrates their approach are able to solve more instances (1,669) instances compared to the exact baseline's 84. Thus they demonstrate that the algorithmic architecture is a significant performance step-up for symbolic reasoning over tree ensembles.
Critical weaknesses
-
Manuscript also omits main characteristics of the datasets. How many features they have? Do they suffer from class imbalance? Are they noisy? Do they have categorial features? Only continuous? With a few datasets and a more in-depth analysis regarding their characteristics (more importantly the number of features) it is hard to see how they affect results. In addition, the source of the dataset are not explicitly available (also code and data are not available).
-
Furthermore, the evaluation arbitrarily fixes the sensitive feature subset size to 8 features. For several datasets, 8 features represents the entirety of the feature space, making it impossible to evaluate how the approach scales or behaves as dimensionality increases.
-
The paper lacks an analysis of dataset dimensionalities as well as other dataset characteristics and feature-space scaling. With a small amount of datasets used for the benchmark (8 datasets only). To demonstrate the scalability and practical utility of Weighted-XCount, the authors should consider incorporating more diverse and challenging tabular benchmarks such as the ones from the TabZilla suite [1]
-
The paper frames its scope broadly around Decision Tree Ensembles (DTEs) Random Forests. However, its mathematical preliminaries and formulation are focused on additive/boosting methods (lines 133 to 136). This can cause confusions and would be better to use a different, more narrow term if a subcategory is considered.
-
I find the answer to question 2 too strong given the fact it was computed over the 84 instances that were provided with exact count. So a question still remains whether the results are good for these because these are easy to solve. And whether there is a strong bias in this analysis.
-
In my opinion, the paper lacks an more in depth analysis in a controlled scenario prior to the real world dataset analysis using like SCM [2, 3] to synthetically generate datasets with different characteristics and have more control in the simulaitions. IN particular since having the bayesian network must be empirically estimated from the original dataset which introduces estimation errors and noise. Therefore, the paper would benefit greatly from utilizing Structural Causal Models (SCMs) to generate synthetic environments allowing the authors to precisely control feature correlations, noise levels, and specific distribution shifts to rigorously isolate how these variables impact weighted sensitivity.
Refs: [1] When Do Neural Nets Outperform Boosted Trees on Tabular Data? [2] Accurate predictions on small data with a tabular foundation model [3] Elements of causal inference: foundations and learning algorithms.
Questions for authors
My main concern about the manuscript are regarding the experimental protocol which several design decisions are not clear, important details (dataset configurations) as well as more ablations were missing and whether the answers of some RQ (in particular #2) is well fundamental or could be biased (see critical weakness section for more details). So far, due to these problems my recommendation would be rejection. I would like the authors to provide a in-depth answer to the following points during the rebuttal and I am willing to revise my score if my concerns are properly addressed:
-
Can the authors provide details on the total feature dimensionality of each benchmark dataset, and clarify how Weighted-XCount behaves when the number of sensitive features varies relative to the total feature space?
-
Given the complexities of learning accurate Bayesian networks on noisy real-world data, have the authors considered evaluating their framework on synthetic data generated via Structural Causal Models (SCMs) ?
-
Can you justify why 8 dimension was used? Or run with multiple values to see how it impact the conclusions taken and the sensitivity of this choice regarding different dimensionalities? This is important in particular since at least 50% of the dataset used have dimensionality <= 8 which makes the experiments a bit strange due to the fact not all datasets had the same amount of instances.
-
Can you add main dataset characteristics (like number of instances features, types of features, etc) and also analyze the impact of such characteristics in the method's performance?
-
Regarding RQ2, the relative error bounds are evaluated exclusively on the 84 instances where the exact baseline successfully finished. Can the authors provide empirical evidence or proxy evaluations showing how the estimation error behaves on unsolved or harder instances where exact counts are unavailable?
-
Given that the current suite relies on datasets without source citations, have the authors considered evaluating their framework on more challenging, tabular benchmarks such as TabZilla?