acceptodds
Under review as a conference paper at ICLR 2027

Support-Aware Counterfactual Analysis

Abstract

Data-grounded counterfactual methods often retrieve one or a few nearby instances with different outcomes, but finite samples may not reveal whether these instances reflect a recurring local pattern or isolated observations. We introduce Support-Aware Counterfactual Analysis (SACA) to characterize the surrounding evidence and determine which conclusions the data support. SACA estimates a distance–direction landscape of different-outcome probabilities and characterizes the evidence through three perspectives: prevalence measures how common different outcomes are nearby, direction identifies where they concentrate, and reach measures how far one must look to accumulate a prescribed amount of evidence. It reports regional estimates only when finite-sample bounds meet the requested accuracy and otherwise abstains. These support deficits guide acquisition of additional outcomes from a pool of unlabeled cases. We establish a near-optimal rate for landscape estimation, finite-sample guarantees for reach and reported regions, and approximation guarantees for acquisition. Experiments on six real-world datasets and five synthetic settings show that SACA reduces landscape estimation error by 22%, makes about 19% more regions reportable, and selects cases 22 times faster.

Primary area:
interpretability and explainable AI
Keywords:
Counterfactual Analysis, Uncertainty Quantification, Explainable AI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rniQOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SACA, a framework that builds a distance–direction probability landscape around a query to map out counterfactual evidence across three angles: Prevalence, Direction, and cumulative Reach. It uses empirical Bernstein bounds to decide when to report an estimate and when to abstain, and sets up a monotone submodular objective based on local sample deficits to guide active data collection.

Critical strengths
  1. Moving from chasing a single counterfactual point to mapping the surrounding evidence landscape makes a lot of sense.

  2. The paper is well-written and easy to follow.

  3. The theoretical analysis is solid, and the experiments are meaningful and sufficient.

Critical weaknesses
  1. The impact of preprocessing and geometric representation is under-explored: SACA's distances, direction vectors u(z)∈Sm−1u(z) \in \mathcal{S}^{m-1}, kernel weights, and regional bins all rely heavily on post-encoded Euclidean geometry. The real-world tabular contain mixed-type features, but the appendix mostly just mentions min–max scaling and sticking with default benchmark encodings. There is no analysis of whether throwing these mixed-type features into a standard Euclidean distance is appropriate, and I couldn't find a clear explanation of how missing values are handled.

  2. On real-world data, how features are preprocessed can significantly alter the local geometry. For example, different categorical encoding schemes yield different pairwise Euclidean distances. Duplicate features will also inflate the weights of certain variables without adding any real information, and strongly correlated feature groups can cause similar distortions.

  3. There seems to be a gap between the theoretical assumptions and actual tabular data: Theorem 2 relies on Assumption (A2), which assumes the polar distribution V=(r,u)V = (r, u) has a density with a strictly positive lower bound fmin>0f_{min} > 0 across all directions within the specified distance range. But in tabular data with discrete/one-hot constraints, samples only ever fall on a restricted set of valid combinations. Many angular directions will simply have zero probability mass and cannot contain any legitimate records, so it's hard to see how this assumption holds in practice for these datasets.

  4. Table 2 in the main text highlights a ~22% error reduction over a binned proxy, but the comparisons there are mainly against weak smoothing ablations and logistic regression. IN appendix, Tables 12 & 13 , SACA does not really beat standard off-the-shelf options like MLPs or k-NN. Does this mean the designed estimator itself isn't particularly essential to the framework?

Questions for authors

See Weaknesses.

7 comments
AuthorsOpenReview ↗

Responses for W1

Thank you for finding the theoretical analysis solid and the experiments meaningful. Your questions led to new analyses of categorical encoding, duplicated and correlated features, missing values, and probe placement.

W1: Mixed-feature geometry and missing values

Putting mixed-type features into a standard Euclidean distance is appropriate once categorical changes receive an explicit cost: weighting each one-hot group by 1/21/\sqrt{2} makes every category switch cost one unit, the convention of standard distances for mixed data [4, 5]. On German, query rankings and the counterfactuals SACA reports stay stable across a range of weights, whereas arbitrary ordinal codes change the nearest counterfactual for 80% of queries. On missing values, the twelve processed benchmark files contain none, so SACA performed no imputation.

Mixed-feature distances. Switching between two one-hot categories changes two coordinates, so weighting the group by 1/21/\sqrt{2} gives every category switch a unit squared Euclidean distance. This unit cost is standard: Gower's coefficient [4] and the heterogeneous Euclidean-overlap metric [5] assign it to a category mismatch, and DiCE [1] uses it for counterfactuals. Table R3-1 varies the weight on German, compares it with arbitrary ordinal codes, and reports how often the nearest counterfactual record and the dominant direction change.

Table R3-1. Encoding of categorical features on German, relative to plain one-hot encoding (weight 1).

One-hot group weightPrevalence rank correlationEntropy rank correlationNearest counterfactual changedDominant direction changed
0.50.9330.79626.7%31.7%
1/21/\sqrt{2} (unit category switch)0.9790.9206.7%13.3%
2\sqrt{2}0.9740.9770.0%15.0%
20.9440.9460.0%26.7%
Arbitrary ordinal codes0.6130.36480.0%83.3%

The results show that the unit weight is a natural choice rather than a tuned one: across weights from 0.5 to 2, query rankings stay stable and the nearest counterfactual changes for at most 27% of queries. Arbitrary ordinal codes impose an ordering and spacing on nominal categories: they change the nearest counterfactual for 80% of queries and the dominant direction for 83%.

Missing values. None of the twelve processed files contains blanks, missing-value markers, or nonfinite entries. For data with missing values, the literature offers five families of approaches [2, 9]. 1) Deletion removes incomplete records, which is valid only when values are missing completely at random. 2) Single imputation fills each missing value with the training mean or median or with the value of nearest neighbours [6]. 3) Model-based imputation predicts each feature from the others, for example by chained equations (MICE) [7], random forests (MissForest) [8], or optimization [3]. 4) Multiple imputation draws several completed datasets to reflect the uncertainty of imputation [10]. 5) Missingness indicators add a feature that records whether a value was missing [2]. SACA can be combined with any of them. When the imputation rule is fitted on an independent pilot sample and then fixed, Theorem 5 applies to the regional outcome probability in the imputed feature space. With multiple imputation, the spread of SACA's estimates across completed datasets shows how much they depend on the imputed values. A missingness indicator keeps records with and without the value in different regions.

In short, explicit group weights handle mixed-type features, and with imputation the guarantee refers to the imputed feature space.

[Revised PDF] Appendix E.8 adds Table R3-1 with the weight sweep, the missing-value audit, and these imputation options.

References

[1] [FAT* 2020] Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations.
[2] [GigaScience 2022] Benchmarking Missing-Values Approaches for Predictive Models on Health Databases.
[3] [JMLR 2018] From Predictive Methods to Missing Data Imputation: An Optimization Approach.
[4] [Biometrics 1971] A General Coefficient of Similarity and Some of Its Properties.
[5] [JAIR 1997] Improved Heterogeneous Distance Functions.
[6] [Bioinformatics 2001] Missing Value Estimation Methods for DNA Microarrays.
[7] [JSS 2011] mice: Multivariate Imputation by Chained Equations in R.
[8] [Bioinformatics 2012] MissForest—Non-Parametric Missing Value Imputation for Mixed-Type Data.
[9] [Wiley 2019] Statistical Analysis with Missing Data, 3rd ed.
[10] [Wiley 1987] Multiple Imputation for Nonresponse in Surveys.

AuthorsOpenReview ↗

Responses for W2

W2: Encoding changes, duplicate features, and correlated groups

You are right that duplicate features "inflate the weights of certain variables": duplicating a feature changed the case count in 46.0% of probe regions. Weighting each copy by 1/21/\sqrt{2} removes the inflation exactly, restoring every count and the estimated landscape to numerical precision, and regularized whitening addresses strongly correlated groups. Table R3-1 addresses categorical encoding, and Table R3-2 isolates duplication and correlation effects across six datasets, using three independent pilot splits and 20 queries per split.

Table R3-2. Effect of feature duplication and covariance adjustment.

TransformationRegions whose counts changedMean directional L1L_1 changePrevalence rank correlationEntropy rank correlation
Uncorrected duplicate46.0%0.0510.9840.943
Weight-corrected duplicate0.0%0.0001.0001.000
Regularized whitening70.4%0.1820.8050.664

Duplicate features. For a displacement aa, the two weighted copies contribute (a/2)2+(a/2)2=a2(a/\sqrt{2})^2+(a/\sqrt{2})^2=a^2 to squared distance. The mapping preserves inner products, so corresponding probes retain their angles, kernel weights, and region memberships. The maximum observed landscape difference was 3.44×10−153.44\times10^{-15}, consistent with the geometric invariance discussed in Proposition 1.

Correlated features. Strongly correlated groups can be handled by regularized whitening, which measures distance relative to the feature covariance. The method itself does not change: whitening is a fixed linear map fitted on the pilot sample and applied to all features before SACA runs, and SACA then operates on the transformed features as usual. In our experiment, it changed case counts in 70.4% of regions and raised the reportable fraction from 12.0% to 22.4%. Theorem 5 holds unchanged for the transformed regions, because they are fixed in advance and the certification sample remains i.i.d. Theorem 2 also continues to apply: the regularized whitening map is invertible, so it preserves Lipschitz continuity and a positive bounded density, changing only the constants in the rate.

So preprocessing does alter the local geometry, as you point out; SACA's guarantees then refer to the representation the user fixes in advance, and explicit weights remove the distortion from duplicated features.

[Revised PDF] Appendix E.8 adds Table R3-2.

AuthorsOpenReview ↗

Responses for W3

W3: Positive density and structurally invalid directions

You are right that with discrete and one-hot constraints, "many angular directions will simply have zero probability mass," so A2 can fail. Theorem 2's rate then does not apply, but the regional guarantee of Theorem 5 does not use A2 and still holds for every fixed region of positive probability, even when other directions are structurally impossible. This matters for the whole task: every regional estimate SACA reports, and every abstention, rests on Theorem 5, not on Theorem 2.

A one-hot constraint can rule out an entire set of displacements. The data do not determine the pointwise landscape there, even if the kernel returns a numerical estimate. An average over uniformly probed directions can therefore depend on how the landscape is extended outside the data support. A valid but rare region presents a different problem: its population probability is positive, but the available sample may contain too few cases to report an estimate. An empty sample region alone cannot distinguish these two situations, so SACA handles them separately. Structurally impossible displacements are excluded in advance, either through the admissible set of Eq. (25) or by probing the actionable or pilot-fitted subspace, so no probe is placed where legitimate records cannot occur. Rare but valid regions stay on the grid, abstain until enough cases are observed, and are exactly the regions that support-guided acquisition targets.

Why Theorem 5 holds without A2. The proof conditions on which reference cases fall in a fixed region RR. Given this membership, the outcome indicators I[Yi≠y]\mathbb{I}[Y_i\neq y] of the NN cases inside RR are independent Bernoulli variables with the same mean pR=Pr⁡(Y≠y∣X∈R)p_R=\Pr(Y\neq y\mid X\in R), because the cases are i.i.d. and RR is fixed before the sample is drawn. The empirical-Bernstein inequality of Audibert et al. (2009) then bounds ∣p^R−pR∣|\hat p_R-p_R| by the width in Eq. (10) with probability at least 1−δ1-\delta for every membership pattern with N≥2N\ge 2, and integrating over the membership pattern removes the conditioning. A union bound over the fixed grid makes the statement simultaneous for all regions. The argument uses only three facts: Pr⁡(X∈R)>0\Pr(X\in R)>0, independence of the cases, and a region fixed in advance. It never uses a density, so A2, smoothness, and the geometry of the other directions play no role. A one-hot constraint only changes which regions have positive mass; a region with zero mass contains no cases and is never reported. Proposition 4 carries the same bounds to prevalence and direction summaries formed from certified regions.

Probe placement then decides how many regions can be reported. In a controlled 16-dimensional setting where half the population shares the query's immutable context, restricting probes to the actionable subspace raised the reportable fraction from 14.1% to 90.3% with two actionable features.

So on tabular data, SACA relies on Theorem 5 for what it reports, and restricting probes to admissible displacements places them where legitimate records can occur.

[Revised PDF] Appendix B.3 now notes that discrete and one-hot features can violate A2 while Theorem 5 still applies, and Appendix E.8 reports the probe-placement results.

Reviewer_Db2kOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Support-Aware Counterfactual Analysis (SACA), which targets a real limitation of standard counterfactual explanations: a nearby instance with a different outcome tells you nothing about whether it is an isolated case or part of a common local trend. SACA characterizes the surrounding evidence with three quantities: prevalence (how common alternative outcomes are nearby), direction (where they concentrate), and reach (how far you have to look before enough evidence accumulates). A smooth estimator is used for the probability landscape around the query, while a separate regional estimator with confidence bounds decides whether a result gets reported or the method abstains. Where the regional data is too thin, an acquisition step selects additional unlabeled instances to improve support. The authors give theory for estimation, reporting, reach and acquisition. Experiments use six real tabular datasets and five synthetic ones, covering evidence characterization, estimation accuracy, regional reporting, endpoint selection and acquisition.

Critical strengths
  1. The main strength of this paper is the research problem. Existing methods return a nearby counterfactual without saying how representative it is of the surrounding data, and SACA attacks exactly that with prevalence, direction and reach. Figures 1 and 3 make a good poi: the same nearest-counterfactual distance can sit on top of very different evidence. The report-or-abstain rule is a practical addition too, since it makes sample limitations visible instead of hiding them.
  2. The paper pairs the theory with a very broad experimental section. Using synthetic data with known probabilities is a good choice, as it lets landscape estimation be checked against something true rather than a proxy. There are also several baselines, ablations, sensitivity analyses, and a lot of additional results in the appendix.
Critical weaknesses
  1. Prevalence, direction and reach are informative individually, but the benefit of using them together for assessing or generating counterfactual explanations is less clear. RQ3 uses direction and reach for endpoint selection, and prevalence does not really enter. I would have liked to see an evaluation of how the three perspectives together help assess or select actual counterfactual candidates in practice.
  2. The abstract reports a 22% reduction in estimation error but does not say against what; Sec. 5 and Table 2 show it is the binned estimator. Appendix Tables 12 and 13, though, show that generic estimators such as MLPs and Euclidean kernels reach lower MAE in several settings. This does not take away from the contribution, but the comparison baseline should be stated in the abstract, and I think it is worth saying more about what SACA trades off against raw point-estimation accuracy.
Questions for authors
  1. In the real-data experiments of RQ4 the outcomes are predictions from the supplied classifiers, and held-out training labels are only revealed once a case is acquired. Since the classifier is there the whole time, what is the acquisition budget standing in for? Is the cost meant to represent getting a real outcome in deployment?
  2. Appendix C.5 argues that selection depends only on case counts and not on outcomes, so Theorem 5 still covers the acquired cases. Sec. 5 describes the post-acquisition width threshold as an operational reporting criterion rather than a confidence guarantee. Could you clarify how these two statements relate? Separately, how do the smoothness assumptions in the convergence analysis sit with the deterministic classifiers used on the real data?
  3. Is SACA mainly meant to characterize counterfactual evidence, or could the three perspectives also be used together to evaluate or select candidates? What role would prevalence play next to direction and reach?
10 comments
AuthorsOpenReview ↗

Responses for W1 and Q3

Thank you for the careful reading of both the theory and the broad experimental section. Your questions led to a joint candidate-selection evaluation, an acquisition replay with observed outcomes, and a corrected statement in Appendix C.5.

W1 and Q3: Joint use of prevalence, direction, and reach

Used together, the three perspectives "help assess or select actual counterfactual candidates" in ways no single perspective does: direction and reach serve more than twice as many queries as reach alone at a quarter of its cost, and prevalence decides whether a query should receive a candidate at all. Combining the three means that a candidate is returned only when all three criteria hold, so the joint policy returns fewer candidates, each backed by stronger evidence. SACA is mainly meant to characterize counterfactual evidence, and the same perspectives also select among existing candidates. Direction indicates where nearby different-outcome evidence concentrates, reach checks whether enough of it accumulates by the candidate's distance, and prevalence tells whether different outcomes are common enough around the query to return a candidate.

The original RQ3 compares direction and reach for endpoint selection. Table R2-1 evaluates their individual and joint use on the same generator-produced candidates for 90 Adult queries.

Table R2-1. Individual and joint candidate-selection policies on Adult.

Selection policyQueries receiving a candidateAudit eligible among acceptedAudited different-prediction fractionMean costMean feature groups changed
Nearest distance90/90 (100.0%)2/900.4090.1081.044
Direction only84/90 (93.3%)2/840.9090.2531.190
Reach only37/90 (41.1%)17/370.5741.3571.946
Direction + reach84/90 (93.3%)5/840.5680.3191.238
Direction + reach + prevalence gate40/90 (44.4%)3/400.6170.2581.200
Matched random-distance control40/90 (44.4%)0/40—0.0981.050
Joint + certificate gate (separate ablation)3/90 (3.3%)3/30.3401.2712.000

A selected candidate is audit eligible when its region contains at least 10 independent audit observations. The audited fraction measures how often predictions in those regions differ from the query’s prediction. A dash indicates that no selected region qualified for audit.

What joint use adds. Compared with reach only, direction + reach serves 84 instead of 37 queries, at about a quarter of the cost (0.319 versus 1.357) and with fewer changed feature groups (1.24 versus 1.95). Compared with nearest distance and direction only, it yields more audit-eligible candidates (5 versus 2), so more of its selections can be checked against independent data. Reach alone remains the choice when audit eligibility matters most, at a much higher cost.

What role prevalence plays next to direction and reach. All candidates of a query share one prevalence, so prevalence decides whether to return a candidate, while direction and reach rank candidates within the query. The validation-selected gate keeps 40 of the 84 queries; its selections include 3 audit-eligible candidates with an audited different-prediction fraction of 0.617, whereas 40 randomly chosen queries yield no audit-eligible candidate. A candidate can lie in a well-supported direction and satisfy the reach criterion even when different outcomes are uncommon around the query; the gate withholds such candidates.

So, to "What role would prevalence play next to direction and reach?": direction and reach rank the candidates within a query, and prevalence decides whether the query receives one.

[Revised PDF] Section 5 adds RQ5 on assessing and selecting generated candidates, and Appendix E.9 gives Table R2-1 with the full protocol.

AuthorsOpenReview ↗

Responses of W2 and Q1

W2: The 22% claim and the estimator's tradeoff

The comparison baseline is now stated in the abstract: the 22% reduction is against the binned regional proxy. We also agree that generic estimators "reach lower MAE in some cases."

What SACA trades off against raw point-estimation accuracy. SACA gives up some pointwise accuracy (mean MAE 0.029 versus 0.020 for the MLP) in exchange for three properties. First, its query-centered kernel sets radial and angular smoothing separately, which lowers the error of the perspectives themselves: adapting angular concentration to distance reduces prevalence-curve MAE from 0.019 to 0.013 (Table 14). Second, it has a proven rate, optimal up to logarithmic factors (Theorems 2–3), which tells the user how much directional detail a given sample supports. Third, SACA's regional estimates carry sample-computable error bounds (Theorem 5, Proposition 4), which pointwise predictors do not provide. Because reach, regional reporting, and acquisition do not use the pointwise estimate, a user who prefers an MLP can plug it in for the descriptive summaries and keep the rest.

In short, SACA trades some raw point-estimation accuracy for perspective-level accuracy, a proven rate, and regional error bounds.

[Revised PDF] The abstract now reads: “On synthetic data with known posteriors, SACA reduces mean pointwise landscape MAE from 0.037 for the binned regional proxy to 0.029, a relative reduction of approximately 22%.” RQ2 now states the trade-off with generic estimators.

Q1: What does the acquisition budget represent?

The acquisition budget "stands in for" the cost of "getting a real outcome in deployment": it counts outcome reveals. In the original RQ4 experiment, B=300B=300 allowed each policy to reveal withheld classifier predictions for up to 300 selected cases; in deployment, it limits how many real outcomes can be collected or verified. If the supplied classifier’s predictions are freely available, the original replay has no genuine labeling bottleneck. “Withheld classifier predictions” therefore describes that experiment more precisely than “held-out training labels.”

This setting follows pool-based active learning (Lewis and Gale, 1994; Roy and McCallum, 2001; Sener and Savarese, 2018): the data are split into a labeled reference sample and an unlabeled pool whose features are known, and revealing the outcome of a pool case costs one unit of a fixed budget. Uncertainty Herding (Bae et al., 2025) likewise reveals held-out labels only upon acquisition.

What the cost stands for in practice. In lending, a real outcome is whether a loan was repaid. It is observed only after the loan has run for a performance period, and only for applicants who were approved; observing it for a denied applicant requires approving the loan. This is the selective-labels problem [1], studied in credit scoring as reject inference [2]. Verifying a past application by hand, through income and employment verification or a credit investigation, also takes staff time. Active learning summarizes such costs as a budget on the number of labels [3]; the budget BB plays this role here.

To evaluate the intended setting, we added a retrospective replay using observed outcomes on all six datasets. Each run used a pool of at most 2,000 cases and revealed up to 300 recorded outcomes. The geometry-fitting data, initial reference sample, acquisition pool, and audit sample were disjoint. Table R2-2 reports results across three seeds and four queries per dataset, with ε=0.5\varepsilon=0.5 and total δ=0.1\delta=0.1.

Table R2-2. Acquisition with observed outcomes. The SACA row uses batched selection.

Selection policyFinal width-qualified coverage ↑Budget AUC ↑Audit discrepancy ↓Audit discrepancy on reported regions ↓Selection time (ms) ↓
SACA0.3830.3390.0580.0514.35
Farthest-first coverage0.3040.2570.0720.04739.90
Margin uncertainty0.3040.2600.0770.05842.22
EWR0.3830.3300.0580.05190.62
Random0.3020.2560.0710.04511.70

With recorded outcomes, SACA reaches the same final width-qualified coverage as EWR (38.3%, versus 30.2% for random acquisition), reaches it sooner (budget AUC 0.339 versus 0.330), and selects cases about 21 times faster (4.35 ms versus 90.62 ms).

Audit discrepancy compares regional estimates with observed outcome frequencies in independent audit regions containing at least 20 cases. The reported-region column is conditional on each policy’s own reporting set.

So the budget stands in for the number of real outcomes a deployment can afford to collect or verify. With that budget, SACA reached the same final coverage as EWR sooner and about 21 times faster.

[Revised PDF] RQ4 and Appendix D.4 now use this wording. RQ4 summarizes the observed-outcome replay, and Appendix E.10 reports it in full.

AuthorsOpenReview ↗

Responses for Q2(a)

Q2(a): Outcome-independent selection and post-acquisition confidence

The two statements relate through their targets, and the method in Section 5 is consistent with Theorem 5 and its assumptions. Theorem 5 gives simultaneous confidence bounds for population regional means on a fixed grid when the certification sample is i.i.d. Section 5 therefore treats the post-acquisition width threshold as an operational reporting criterion. Theorem 5 covers the population means on an independent i.i.d. sample. Acquisition that uses features alone does not adapt to outcomes, but it can still change the distribution of features within a region, so the acquired cases are not such a sample. The revised Appendix C.5 states this directly; no result or assumption changes.

For example, suppose a region contains two equally common subgroups with outcome probabilities 0.1 and 0.9, so the population regional mean is 0.5, and a feature-based policy acquires cases only from the first subgroup. Theorem 5 applies to an independent i.i.d. sample of the region: its interval covers 0.5 with the stated confidence. Appendix C.5, read literally, would claim the same for the acquired cases, but their mean is 0.1, so a narrow interval around it need not cover 0.5. Section 5 only states whether the width meets the tolerance, which remains true for the acquired cases. The two means answer different questions: 0.5 is the approval rate among all comparable cases, which is what a user wants reported, while 0.1 describes only the cases that the policy selected. They coincide when selection within a region does not favor any subgroup, and they differ when it does. SACA therefore uses the acquired cases to decide which regions meet the width threshold, and reports population rates from an independent i.i.d. sample. Table R2-3 tests this distinction against known population means.

Table R2-3. Post-acquisition inference over 100 runs with 12 fixed regions and known population means. SACA uses the batched implementation evaluated in RQ4. Each acquisition run starts with 500 cases and reveals 500 additional outcomes from a pool of 3,000. The error tolerance is ε=0.2\varepsilon=0.2, and nominal joint confidence is 90%.

Selection policyWidth-qualified regionsMean absolute selection biasJoint population interval coverageRuns with a reported error >0.2>0.2Joint selected-target interval coverage
SACA100.0%0.03784%0%100%
Random84.1%0.006100%0%100%
EWR100.0%0.0810%2%100%
Low-uu selection (bias control)100.0%0.1170%39%100%
Independent i.i.d. audit100.0%—100%0%—

SACA met the width threshold in every region, and every reported estimate was within 0.2 of its population mean. Yet its intervals covered all 12 population means in only 84 of 100 runs. Both facts hold at once because they measure different things: the tolerance concerns the error of each reported estimate, while interval coverage also requires the i.i.d. condition, so an estimate can be within 0.2 of the population mean while its narrower interval misses it. The independent audit used 2,000 fresh i.i.d. cases and covered all 12 population means in all 100 runs.

The corrected interpretation of C.5 depends on the inferential target. Theorem 5 covers population regional means under independently fixed geometry and i.i.d. certification sampling. Theorem 7 addresses the running mean of the selected cases’ conditional outcome probabilities; the last column evaluates that target with a separate count-uniform Hoeffding confidence sequence. Proposition 5 gives an optimization guarantee for exact sequential greedy reduction of fixed count deficits. RQ4 accordingly uses width-qualified coverage to describe the fraction of regions that meet the operational reporting threshold. So the two statements relate as follows: after acquisition, Section 5's width criterion tells which regions have enough cases, and Theorem 5's population guarantee is then obtained on an independent i.i.d. sample; the acquired cases alone estimate the selected-case mean, which equals the population mean only when selection within a region favors no subgroup.

[Revised PDF] Appendix C.5 now states this interpretation in place of “applies verbatim,” and Appendix E.10 adds Table R2-3.

Reviewer_cXi2OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce Support-Aware Counterfactual Analysis, which characterizes the local decision landscape around a query through three complementary perspectives: prevalence, direction, and reach. To ensure statistical reliability, their method also evaluates discrete probe regions, where, for each region, they can either report a regional estimator if there is sufficient evidence in the region, or a regional deficit if the method chooses to abstain in the region. When abstention occurs, it further defines a monotone submodular optimization problem to greedily acquire unlabeled pool data that maximizes regional deficit reduction.

Critical strengths
  • The work identifies a clear shortcoming in current approaches to counterfactual explanations by focusing on support, and addressing this gap is interesting and relevant.
  • The paper's claims are backed up by theoretical properties.
Critical weaknesses
  • I actually quite like the general idea, but the presentation suffers from (likely AI-induced) redundancy, where concepts and solution parts are mentioned several times at a high level before they are actually explained (e.g. reach in Theo.4). The paper should be reworked to be much more concise and precise, cutting redundancy and bringing in core details from the appendix.
  • The discussion of related work is far too brief, and fails to clearly position the proposed approach with respect to the state of the art. The appendix F provides additional information, it is mainly a long list of properties and references, lacking an actual (human) delineation.
  • Similarly, the empirical evaluation is weak, as it is not clear that the setup and baselines give an accurate assessment of the state of the art.
    • If there is no principled way to select the directions where the data lies, then how can the method find it? Wouldn't the solution just abstain all the time?
  • An important property of a counterfactual explanation is that it is sparse, meaning that it should change few features in the interest of interpretability and actionability. Using L_0 norm alone might result in too dense feature modification vectors, especially when using the top-10 PCA trick on high dimensional data.
  • The scope seems overly large. The paper attempts to combine continuous kernel smoothing, discrete statistical certificates, and submodular active learning into a single framework.
  • While the paper makes theoretical statements, it does not sufficiently connect these formal guarantees to any practical benefit.
Questions for authors
  • How are the T directions selected? It says that "Angular neighborhoods become sparse with dimension" in C.2, but still in the m = 16 case the directions are sampled uniformly from the unit sphere, which (if T is something like 32) would almost surely miss any reference data?
10 comments
AuthorsOpenReview ↗

Responses for W1

Thank you for the careful and thorough review, and we are glad you "quite like the general idea." Your comments led us to cut redundancy, explain reach before Theorem 4 uses it, show how directions are chosen, and connect each guarantee to a practical decision.

W1: Presentation, redundancy, and the definition of reach

Reach, your example, is now defined once in Section 3.1, alongside Eq. (8) and Figure 2c, before Theorem 4 uses it; the introduction mentions it only in a one-sentence overview.

[Revised PDF] Specific edits (locations in the submitted manuscript):

  • Introduction and Figure 1. Shortened: the overview of the three perspectives in the introduction (p. 1, lines 046–052) is now one sentence that points to Section 3.1. Shortened: the Figure 1 caption (p. 2, lines 066–074) now only defines the symbols. Removed: the second walkthrough of the same panels in Section 3.1 (p. 5, lines 224–232).
  • Sections 2 and 3. Removed: the repeated motivation at the start of Section 2 (p. 2, lines 101–103) and of Section 3.2 (p. 5, lines 235–237), the walkthrough of Figure 2 that repeated its caption (p. 3, lines 141–144), and the opening sentence of Section 3.1 (p. 4, lines 184–185). Removed and merged: the preview "From estimation to acquisition" (p. 3, lines 145–152); its definitions of the accuracy, confidence, pool, and budget now appear in Sections 3.2–3.3, where they are used. Removed: the paragraph on the roles of the two estimators (p. 4, lines 177–181), which repeated Section 3.2 (p. 5, lines 238–243).
  • Theory and related work. Shortened: the theory summary in the introduction (p. 2, lines 084–092) is now one sentence. Rewritten: the reach paragraph of Section 4 (p. 6, lines 313–318) now starts from Eq. (8). Moved: the comparison with active learning (p. 6, lines 280–285) to Related Work.
  • Guarantees. Removed: restatements of the guarantees in Sections 3.2–3.3, Section 4, and RQ2 (pp. 5–8, lines 258–259, 264–265, 286–288, 320–321, 332–333, and 413–415). Each guarantee is now stated once, in Section 4, and the other places refer to it.

Details brought into the main text. The revision introduces direction selection immediately after Eq. (3), the feasibility filter in Section 3, the reporting-count rule in Section 3.2, and the comparison with related work in Section 6. Together, these edits cut the redundancy you pointed out and bring in the core details from the appendix.

AuthorsOpenReview ↗

Responses for W2

W2: Related work and delineation from the state of the art

We agree that the related work was "far too brief" and that Appendix F read as "a long list of properties and references." The revised Section 6 and Appendix F give an actual delineation: they organize prior work by the question it answers, split each question into finer families, and state what each family returns, what it guarantees, and what SACA adds. In brief:

1) Which alternatives and changes are available?

  • Optimization against the model (Wachter et al., 2018; Lucic et al., 2022) returns a nearby input that the model assigns a different outcome.
  • Retrieval and case-based adaptation (Keane and Smyth, 2020; Brughmans et al., 2024) returns or adapts one observed case.
  • Diverse sets (Mothilal et al., 2020; Furman et al., 2025) return several alternatives that trade off proximity and diversity.
  • Plausibility through paths, densities, or generative models (Poyiadzi et al., 2020; Kanamori et al., 2020; Wielopolski et al., 2024; Redelmeier et al., 2024) prefers alternatives in dense, connected parts of the data.
  • Actionable and causal recourse (Ustun et al., 2019; Karimi et al., 2021) restricts or models which changes are possible and effective.
  • Population-level rules (Kanamori et al., 2022; Ley et al., 2023) return actions shared by a group.

Their guarantees concern the validity or feasibility of the returned alternative; none estimates how often different outcomes actually occur around it.

2) Is a prediction robust or locally supported?

  • Robustness to input perturbations (Maragno et al., 2024) certifies a region in which the prediction stays valid.
  • Robustness to model changes (Hamman et al., 2023; Jiang et al., 2024a; Turbal et al., 2025) keeps the prediction valid across retrained models.
  • Conformal and uncertainty-aware explanations (Altmeyer et al., 2024; Schut et al., 2021) attach coverage or uncertainty to the model's prediction.
  • Local data support (Pawelczyk et al., 2020b; Gemou et al., 2026) scores how close observed target-class cases are.

Their guarantees concern the model's prediction rather than how often the outcome is observed in the data, and local-support scores carry no finite-sample error bound.

3) Which cases should be labeled?

  • Uncertainty-based selection (Lewis and Gale, 1994; Roy and McCallum, 2001) labels the cases the classifier is least sure about or whose labels most reduce expected error.
  • Coverage-based selection (Sener and Savarese, 2018; Yehuda et al., 2022; Hacohen et al., 2022) labels cases that represent or cover the pool.
  • Hybrid selection (Ash et al., 2020; Bae et al., 2025) combines uncertainty with diversity or coverage.

Their objective is a better classifier, not more evidence around a particular query.

An example in banking. A bank denies a loan applicant, who asks what would change the decision (numbers are illustrative).

  • Counterfactual generation and recourse return alternative applicant profiles. DiCE (Mothilal et al., 2020) returns a few, such as "raise annual income by $8k" or "raise income by $5k and lower the debt-to-income ratio by 10 points"; FACE (Poyiadzi et al., 2020) returns one profile reached through a chain of similar past borrowers; actionable recourse (Ustun et al., 2019) returns the cheapest change the bank permits, never age or marital status.
  • Robustness and local support return a score for one profile. A robust region (Maragno et al., 2024) returns a radius within which the approval stays valid if the reported income or the credit model changes slightly; a local-support score (Gemou et al., 2026) returns the distance to the kk-th nearest approved past applicant.
  • Active learning (Sener and Savarese, 2018; Bae et al., 2025) returns past applications whose repayment outcomes would most improve the credit-scoring model.

SACA reports the observed evidence around the query and each alternative. For "income +$8k", it may report that 10 of 200 comparable past applicants were approved (5%, within ±0.1 at 90% confidence). For another alternative at the same distance, 1 of 2 comparable applicants was approved (50%), too few to report. None of the three groups tells the loan officer how often comparable applicants were actually approved: generators guarantee only that the model approves the suggested profile, robustness and local-support scores describe the model's prediction or a single distance without an error bound that reflects the sample size, and active learning targets the credit model rather than the evidence for one applicant. SACA adds exactly this information.

[Revised PDF] Section 6 groups prior work into these three questions, and Appendix F splits each into finer families.

AuthorsOpenReview ↗

Responses for W3

W3: Experimental setup and strength of the baselines

To give "an accurate assessment of the state of the art," every experiment follows a published protocol, and each output is compared with published methods that produce the same kind of output from the same inputs.

Setup. Data and queries: the six OpenXAI benchmarks and their pretrained classifiers (Agarwal et al., 2022), with training cases as references and test cases as queries, as in CARLA (Pawelczyk et al., 2021) and Guidotti (2024). Synthetic data: Gaussian mixtures with closed-form posteriors, following Wielopolski et al. (2024) and Nemecek et al. (2025). Local scale: ρ\rho from the ten-neighbor convention of Smith et al. (2014), DACE (Kanamori et al., 2020), and FACE (Poyiadzi et al., 2020). Evidence profiles (RQ1): compared with data-complexity measures (Ho and Basu, 2002; Smith et al., 2014) and hubness (Radovanović et al., 2010). Acquisition: pool-based active learning (Lewis and Gale, 1994; Sener and Savarese, 2018), with outcomes revealed only when acquired (Bae et al., 2025). Certificates: empirical-Bernstein bounds (Audibert et al., 2009; Maurer and Pontil, 2009).

Landscape estimation (RQ2). SACA's kernel reduces MAE from 0.037 to 0.029 relative to the binned proxy, and the comparison includes the strongest generic estimator, an MLP, which is lower (Tables 2, 12–13). All estimators use the same 8,000 labeled cases per seed and are scored against the closed-form posterior: a binned proxy as in histogram regression (Györfi et al., 2002), two ablations of the Nadaraya–Watson estimator (Nadaraya, 1964; Watson, 1964), a kk-NN proportion (Brughmans et al., 2024; Smith et al., 2014), a Euclidean kernel (Artelt and Hammer, 2020; Wielopolski et al., 2024), a logistic posterior (Kothari et al., 2024; Cheon et al., 2025), and an MLP.

Endpoint selection (RQ3). SACA direction attains the highest mean endpoint posterior (0.791 versus 0.770 for a posterior-gradient oracle; Table 4) and, on matched query–seed pairs, beats every direction-selection baseline (Table 18). Eight counterfactual methods are evaluated at their returned points (Table 19): nearest retrieval (Keane and Smyth, 2020), NICE (Brughmans et al., 2024), DiCE (Mothilal et al., 2020), DACE (Kanamori et al., 2020), FACE (Poyiadzi et al., 2020), MCCE (Redelmeier et al., 2024), PPCEF (Wielopolski et al., 2024), and robust regions (Maragno et al., 2024). Five direction-selection rules share the fixed-level crossing protocol: random directions (Marsaglia, 1972), local support (Gemou et al., 2026), path geometry (Sokol et al., 2025), feature responsiveness (Cheon et al., 2025), and a posterior-gradient oracle (Baehrens et al., 2010).

Acquisition (RQ4). SACA qualifies 40.7% of probes versus 34.3% for EWR, selects cases about 22 times faster (Table 6), and leads on each of the six datasets (Table 22). Each strategy receives the same pool, probe grid, and budget of 300 reveals: Uncertainty Herding (Bae et al., 2025), BADGE (Ash et al., 2020), classifier margin (Lewis and Gale, 1994), TypiClust (Hacohen et al., 2022), ProbCover (Yehuda et al., 2022), farthest-first coverage (Gonzalez, 1985; Sener and Savarese, 2018), expected width reduction (Roy and McCallum, 2001; Maurer and Pontil, 2009), and random selection.

Why some methods cannot be compared. Each excluded family returns a different kind of output or needs inputs the benchmarks lack. Population-level rules (Kanamori et al., 2022; Ley et al., 2023) return one action shared by a group of applicants, so they produce no per-query endpoint, direction, or regional estimate to score. Causal recourse (Karimi et al., 2021) needs a structural causal model of how features affect each other; the benchmarks provide only observational records and a classifier, so any comparison would depend on a causal model we would have to invent. Robustness and conformal methods (Jiang et al., 2024a; Altmeyer et al., 2024) certify that the model's prediction for a given counterfactual stays valid, which has no common metric with an estimated outcome frequency; robust regions (Maragno et al., 2024), their closest relative that returns an endpoint, are included in Table 19. Boundary-query methods (Guerrero et al., 2026) spend their budget on classifier evaluations rather than on outcomes of pool cases, so under the same budget they add no observations to any region.

We hope this shows that the setup and baselines give an accurate assessment of the state of the art.

[Revised PDF] Section 5 states how each comparison matches its output and budget, and Appendix D.5 explains why the excluded families are not directly comparable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.