- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SACA, a framework that builds a distance–direction probability landscape around a query to map out counterfactual evidence across three angles: Prevalence, Direction, and cumulative Reach. It uses empirical Bernstein bounds to decide when to report an estimate and when to abstain, and sets up a monotone submodular objective based on local sample deficits to guide active data collection.
Critical strengths
-
Moving from chasing a single counterfactual point to mapping the surrounding evidence landscape makes a lot of sense.
-
The paper is well-written and easy to follow.
-
The theoretical analysis is solid, and the experiments are meaningful and sufficient.
Critical weaknesses
-
The impact of preprocessing and geometric representation is under-explored: SACA's distances, direction vectors , kernel weights, and regional bins all rely heavily on post-encoded Euclidean geometry. The real-world tabular contain mixed-type features, but the appendix mostly just mentions min–max scaling and sticking with default benchmark encodings. There is no analysis of whether throwing these mixed-type features into a standard Euclidean distance is appropriate, and I couldn't find a clear explanation of how missing values are handled.
-
On real-world data, how features are preprocessed can significantly alter the local geometry. For example, different categorical encoding schemes yield different pairwise Euclidean distances. Duplicate features will also inflate the weights of certain variables without adding any real information, and strongly correlated feature groups can cause similar distortions.
-
There seems to be a gap between the theoretical assumptions and actual tabular data: Theorem 2 relies on Assumption (A2), which assumes the polar distribution has a density with a strictly positive lower bound across all directions within the specified distance range. But in tabular data with discrete/one-hot constraints, samples only ever fall on a restricted set of valid combinations. Many angular directions will simply have zero probability mass and cannot contain any legitimate records, so it's hard to see how this assumption holds in practice for these datasets.
-
Table 2 in the main text highlights a ~22% error reduction over a binned proxy, but the comparisons there are mainly against weak smoothing ablations and logistic regression. IN appendix, Tables 12 & 13 , SACA does not really beat standard off-the-shelf options like MLPs or k-NN. Does this mean the designed estimator itself isn't particularly essential to the framework?
Questions for authors
See Weaknesses.
7 comments
Responses for W2
W2: Encoding changes, duplicate features, and correlated groups
You are right that duplicate features "inflate the weights of certain variables": duplicating a feature changed the case count in 46.0% of probe regions. Weighting each copy by removes the inflation exactly, restoring every count and the estimated landscape to numerical precision, and regularized whitening addresses strongly correlated groups. Table R3-1 addresses categorical encoding, and Table R3-2 isolates duplication and correlation effects across six datasets, using three independent pilot splits and 20 queries per split.
Table R3-2. Effect of feature duplication and covariance adjustment.
| Transformation | Regions whose counts changed | Mean directional change | Prevalence rank correlation | Entropy rank correlation |
|---|---|---|---|---|
| Uncorrected duplicate | 46.0% | 0.051 | 0.984 | 0.943 |
| Weight-corrected duplicate | 0.0% | 0.000 | 1.000 | 1.000 |
| Regularized whitening | 70.4% | 0.182 | 0.805 | 0.664 |
Duplicate features. For a displacement , the two weighted copies contribute to squared distance. The mapping preserves inner products, so corresponding probes retain their angles, kernel weights, and region memberships. The maximum observed landscape difference was , consistent with the geometric invariance discussed in Proposition 1.
Correlated features. Strongly correlated groups can be handled by regularized whitening, which measures distance relative to the feature covariance. The method itself does not change: whitening is a fixed linear map fitted on the pilot sample and applied to all features before SACA runs, and SACA then operates on the transformed features as usual. In our experiment, it changed case counts in 70.4% of regions and raised the reportable fraction from 12.0% to 22.4%. Theorem 5 holds unchanged for the transformed regions, because they are fixed in advance and the certification sample remains i.i.d. Theorem 2 also continues to apply: the regularized whitening map is invertible, so it preserves Lipschitz continuity and a positive bounded density, changing only the constants in the rate.
So preprocessing does alter the local geometry, as you point out; SACA's guarantees then refer to the representation the user fixes in advance, and explicit weights remove the distortion from duplicated features.
[Revised PDF] Appendix E.8 adds Table R3-2.
Responses for W3
W3: Positive density and structurally invalid directions
You are right that with discrete and one-hot constraints, "many angular directions will simply have zero probability mass," so A2 can fail. Theorem 2's rate then does not apply, but the regional guarantee of Theorem 5 does not use A2 and still holds for every fixed region of positive probability, even when other directions are structurally impossible. This matters for the whole task: every regional estimate SACA reports, and every abstention, rests on Theorem 5, not on Theorem 2.
A one-hot constraint can rule out an entire set of displacements. The data do not determine the pointwise landscape there, even if the kernel returns a numerical estimate. An average over uniformly probed directions can therefore depend on how the landscape is extended outside the data support. A valid but rare region presents a different problem: its population probability is positive, but the available sample may contain too few cases to report an estimate. An empty sample region alone cannot distinguish these two situations, so SACA handles them separately. Structurally impossible displacements are excluded in advance, either through the admissible set of Eq. (25) or by probing the actionable or pilot-fitted subspace, so no probe is placed where legitimate records cannot occur. Rare but valid regions stay on the grid, abstain until enough cases are observed, and are exactly the regions that support-guided acquisition targets.
Why Theorem 5 holds without A2. The proof conditions on which reference cases fall in a fixed region . Given this membership, the outcome indicators of the cases inside are independent Bernoulli variables with the same mean , because the cases are i.i.d. and is fixed before the sample is drawn. The empirical-Bernstein inequality of Audibert et al. (2009) then bounds by the width in Eq. (10) with probability at least for every membership pattern with , and integrating over the membership pattern removes the conditioning. A union bound over the fixed grid makes the statement simultaneous for all regions. The argument uses only three facts: , independence of the cases, and a region fixed in advance. It never uses a density, so A2, smoothness, and the geometry of the other directions play no role. A one-hot constraint only changes which regions have positive mass; a region with zero mass contains no cases and is never reported. Proposition 4 carries the same bounds to prevalence and direction summaries formed from certified regions.
Probe placement then decides how many regions can be reported. In a controlled 16-dimensional setting where half the population shares the query's immutable context, restricting probes to the actionable subspace raised the reportable fraction from 14.1% to 90.3% with two actionable features.
So on tabular data, SACA relies on Theorem 5 for what it reports, and restricting probes to admissible displacements places them where legitimate records can occur.
[Revised PDF] Appendix B.3 now notes that discrete and one-hot features can violate A2 while Theorem 5 still applies, and Appendix E.8 reports the probe-placement results.
Responses for W1
Thank you for finding the theoretical analysis solid and the experiments meaningful. Your questions led to new analyses of categorical encoding, duplicated and correlated features, missing values, and probe placement.
W1: Mixed-feature geometry and missing values
Putting mixed-type features into a standard Euclidean distance is appropriate once categorical changes receive an explicit cost: weighting each one-hot group by 1/2 makes every category switch cost one unit, the convention of standard distances for mixed data [4, 5]. On German, query rankings and the counterfactuals SACA reports stay stable across a range of weights, whereas arbitrary ordinal codes change the nearest counterfactual for 80% of queries. On missing values, the twelve processed benchmark files contain none, so SACA performed no imputation.
Mixed-feature distances. Switching between two one-hot categories changes two coordinates, so weighting the group by 1/2 gives every category switch a unit squared Euclidean distance. This unit cost is standard: Gower's coefficient [4] and the heterogeneous Euclidean-overlap metric [5] assign it to a category mismatch, and DiCE [1] uses it for counterfactuals. Table R3-1 varies the weight on German, compares it with arbitrary ordinal codes, and reports how often the nearest counterfactual record and the dominant direction change.
Table R3-1. Encoding of categorical features on German, relative to plain one-hot encoding (weight 1).
The results show that the unit weight is a natural choice rather than a tuned one: across weights from 0.5 to 2, query rankings stay stable and the nearest counterfactual changes for at most 27% of queries. Arbitrary ordinal codes impose an ordering and spacing on nominal categories: they change the nearest counterfactual for 80% of queries and the dominant direction for 83%.
Missing values. None of the twelve processed files contains blanks, missing-value markers, or nonfinite entries. For data with missing values, the literature offers five families of approaches [2, 9]. 1) Deletion removes incomplete records, which is valid only when values are missing completely at random. 2) Single imputation fills each missing value with the training mean or median or with the value of nearest neighbours [6]. 3) Model-based imputation predicts each feature from the others, for example by chained equations (MICE) [7], random forests (MissForest) [8], or optimization [3]. 4) Multiple imputation draws several completed datasets to reflect the uncertainty of imputation [10]. 5) Missingness indicators add a feature that records whether a value was missing [2]. SACA can be combined with any of them. When the imputation rule is fitted on an independent pilot sample and then fixed, Theorem 5 applies to the regional outcome probability in the imputed feature space. With multiple imputation, the spread of SACA's estimates across completed datasets shows how much they depend on the imputed values. A missingness indicator keeps records with and without the value in different regions.
In short, explicit group weights handle mixed-type features, and with imputation the guarantee refers to the imputed feature space.
[Revised PDF] Appendix E.8 adds Table R3-1 with the weight sweep, the missing-value audit, and these imputation options.
References
[1] [FAT* 2020] Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations.
[2] [GigaScience 2022] Benchmarking Missing-Values Approaches for Predictive Models on Health Databases.
[3] [JMLR 2018] From Predictive Methods to Missing Data Imputation: An Optimization Approach.
[4] [Biometrics 1971] A General Coefficient of Similarity and Some of Its Properties.
[5] [JAIR 1997] Improved Heterogeneous Distance Functions.
[6] [Bioinformatics 2001] Missing Value Estimation Methods for DNA Microarrays.
[7] [JSS 2011] mice: Multivariate Imputation by Chained Equations in R.
[8] [Bioinformatics 2012] MissForest—Non-Parametric Missing Value Imputation for Mixed-Type Data.
[9] [Wiley 2019] Statistical Analysis with Missing Data, 3rd ed.
[10] [Wiley 1987] Multiple Imputation for Nonresponse in Surveys.