acceptodds
Under review as a conference paper at ICLR 2027

FlowCF: Sparse Counterfactual Explanations for Mixed-Type Tabular Data using Flow Matching

Abstract

In the field of Explainable AI (XAI), counterfactual (CF) explanations interpret a model's decision by suggesting the changes to the input that would lead to a more favourable outcome. To be useful in practice, such an explanation should change few features and change them as little as possible, properties known as sparsity and proximity. However, for numerical features, existing methods typically encourage small changes rather than leaving them exactly unchanged, whether they are model-agnostic and amortised, or gradient-based with full access to the model. In this paper, we propose FlowCF, a model-agnostic generative method that frames CF generation as sparse transport from the factual to the target class. We solve this transport with flow matching, which we extend to mixed feature types with a novel mixed flow operator, and exploit the resulting geometry to optimise for exact numerical sparsity through a gating network that leaves selected features exactly unchanged. Extensive experiments on six benchmark datasets demonstrate that FlowCF produces the best numerical sparsity and proximity, changing 16% of the numerical features where the best baseline changes 90%, while reducing numerical displacement by 70% relative to the best baseline and remaining comparable on the other desiderata.

Primary area:
interpretability and explainable AI
Keywords:
Counterfactual Explanations, Explainable AI, Flow Matching, Mixed-Type Tabular Data, Generative Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Lt7GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FlowCF is a model-agnostic, amortized method for sparse counterfactual generation on mixed-type tabular data. It combines a mixed-type flow-matching transport with a feature-wise gating network that can keep selected numerical features exactly unchanged. The method is trained in three stages for transport, sparsity, and proximity, and is evaluated on six tabular datasets against six baselines.

Critical strengths

Exact numerical sparsity by construction. The gating mechanism can leave selected features exactly unchanged, which is useful for actionable counterfactual explanations.
Efficient and model-agnostic inference. FlowCF is amortized and does not require black-box queries at explanation time.

Critical weaknesses

The headline sparsity gain is partly by construction. FlowCF reports much lower exact numerical sparsity than the baselines, but the paper itself proves that continuous generators almost surely modify every real-valued feature under an exact-equality metric. I would like to see results under several ϵ\epsilon-sparsity thresholds to assess whether the practical advantage remains large.

The proximity gain is not fully disentangled from sparsity. The proximity metric averages displacement over all numerical features. Since FlowCF sets many feature changes exactly to zero, this automatically improves proximity. Reporting displacement only over changed features would make the comparison more informative.

Questions for authors

Can the authors report numerical ϵ\epsilon-sparsity at several thresholds and compare against the baselines? Can the authors report the average displacement per changed feature, rather than averaging over all numerical features? Can the authors provide a quantitative ablation for the three training stages, especially the effect of gating and the final joint fine-tuning?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Lt7G

We sincerely thank the reviewer for the clear and helpful review, and for recognising the value of exact numerical sparsity and of amortised, model-agnostic inference. Your suggestions made our evaluation more transparent.

W1 / Q1. ϵ\epsilon-sparsity at several thresholds. Thank you. We agree that this is essential to assess the practical advantage. The advantage holds at every threshold. Table 5 (Section 5.1) reports numerical sparsity exactly, at the precision each feature is recorded in, and at ϵ=0.01,0.1,0.5\epsilon = 0.01, 0.1, 0.5 training-set standard deviations, and Appendix C.1 gives per-dataset results. Baselines do benefit (DiCE: 0.897 to 0.416 at ϵ=0.1\epsilon = 0.1), but FlowCF remains the sparsest method on average at every tolerance (0.146 at ϵ=0.1\epsilon = 0.1, against 0.218 for the next best method, NICE, a new baseline suggested by Reviewer 6WFu) and is first on five of six datasets.

W2 / Q2. Displacement per changed feature. Thank you, this is a good suggestion. We added it as the last column of Table 5. Per changed feature, FlowCF moves 1.2 standard deviations, as much as DiCE and TABCF (baselines: 0.8–1.2), and less than NICE (1.7), the other exactly sparse method. We agree with your point: the proximity gain comes mainly from changing fewer features, which is what the gate is designed for. Section 5.1 now states this.

Q3. Ablation of the three training stages. Thank you. We agree that this was missing, and we added it in Appendix C.4 (Table 10), summarised in Section 5.1. Step 1 alone changes every numerical feature (sparsity 1.000, Prox Num 1.998, validity 0.943). The gate (Step 2) gives exact sparsity (0.155) and raises validity to 0.957. The joint fine-tuning (Step 3) reduces Prox Num from 0.298 to 0.229 and raises validity to 0.970. The table also ablates the main components (multiple heads, pairing, validity and drift terms), and Appendix C.5 varies the hyperparameters.

Thank you again for the helpful feedback. We hope the revision addresses your concerns, and we would be glad to answer any further questions.

Thank you again for the clear and helpful review. We wanted to check whether the new tolerance-based sparsity results, displacement per changed feature, and ablation studies address your concerns. If anything remains unclear before the discussion closes, we would be very happy to clarify.

Thank you for the additional experiments and detailed response. I appreciate the authors' efforts to address my concerns. However, the new results confirm that the proximity improvement mainly comes from changing fewer features, rather than reducing the displacement of individual changed features. The underlying sparsity–displacement trade-off therefore remains unresolved, particularly on continuous-only datasets. Therefore, I will maintain my original rating.

Thank you for engaging with our response. We agree that Prox Num combines how many numerical features change with how far they move, and that against dense methods, the gain of FlowCF comes mainly from changing fewer features. We see this as the intended behaviour of a sparse method, and with the points below, we hope to convince you of this and that you would consider raising your score.

We would also like to recall Table 1: every baseline queries the black box when it explains an instance, and all except DiCoFlex optimise or search each instance anew. FlowCF is the only method with all the properties of Table 1. It only needs black-box labels during training, explains a new instance in a single forward pass, and still achieves the best numerical sparsity and proximity on average.

  • Prox Num vs. displacement per changed feature. Prox Num is the standard L1 proximity, the type of distance that the CARLA benchmarking framework reports. Wachter et al., who introduced counterfactual explanations, chose an L1 distance specifically because it favours counterfactuals that change few features. Unchanged features enter Prox Num with their true change of zero. Displacement per changed feature instead leaves out exactly the features that a sparse method does not change, and therefore favours dense changes. We would like to make this point with a simple example. For a linear classifier that weighs dd features equally, moving one feature by mm or all dd features by m/dm/d crosses the boundary with the same total change, but the dense counterfactual has a dd times smaller displacement per changed feature while asking the user for dd changes instead of one. In general, the two factors pull in opposite directions: the fewer features change, the further the changed ones typically have to move. We agree that this measure was missing from our original results, and we added it as you suggested (Table 5, and per dataset in Table 7). However, judging FlowCF on this measure alone would, in our opinion, be unfair to any sparse method, as it ignores exactly the features such a method leaves unchanged.

  • Both factors (numerical sparsity and displacement) together: FlowCF is Pareto-optimal on every dataset. Nonetheless, we added Figure 7 (Appendix C.9), which plots numerical sparsity against your measure, using only results already in the paper. FlowCF lies on the Pareto front on all six datasets and on the mean, and it is the only method that does so on every dataset. NICE and DiCE lie on it for four of the six datasets, and DiCoFlex, methodologically our closest competitor, for none. No baseline changes fewer features and also moves them less. Every baseline that moves the changed features less changes at least 86% of the numerical features, except NICE on the two continuous-only datasets.

  • Continuous-only datasets: FlowCF vs. NICE. With Prox Num (Table 4), FlowCF and NICE are the only Pareto-optimal methods on MAGIC and HTRU2, and neither dominates the other (table below). NICE moves less, but it relies on stronger access: it queries the black-box scores and searches each instance anew. With the same label-only access as FlowCF, the validity of NICE drops from 1.000 to 0.763, against 0.970 for FlowCF (Appendix C.3).

Features changed (MAGIC / HTRU2)Prox Num (MAGIC / HTRU2)Access at explanation time
FlowCF15.8% / 32.6%0.327 / 0.616none (one forward pass)
NICE22.8% / 41.7%0.275 / 0.509black-box scores, per-instance search
DiCE99.9% / 100%0.369 / 0.710gradients, per-instance optimisation
  • Across all datasets. On average, FlowCF changes the fewest numerical features (0.159) with the lowest Prox Num (0.229, Table 3), and on every dataset, it changes fewer features with a smaller Prox Num than every baseline except NICE. Compared with NICE, it also moves each changed feature less on average (1.2 vs. 1.7 standard deviations, Table 5). Step 3 shortens the retained changes directly: it lowers Prox Num from 0.298 to 0.229 at nearly unchanged sparsity (Appendix C.4).

  • An adjustable operating point. Like other counterfactual methods, FlowCF is tunable. NICE, for example, comes in variants that favour sparsity, proximity or plausibility (NICE-sparsity, NICE-proximity and NICE-plausibility). In FlowCF, a four times larger λprox\lambda_{\mathrm{prox}} lowers Prox Num from 0.229 to 0.183 and numerical sparsity to 0.139 (Appendix C.5).

We hope this shows the trade-off as a deliberate and adjustable operating point of FlowCF rather than an unresolved limitation. Thank you again for the constructive review.

Reviewer_rqbFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FlowCF, a model-agnostic and amortized method for generating sparse counterfactual explanations on mixed-type tabular data. It formulates counterfactual generation as a sparse transport problem and combines mixed-type flow matching with a feature-wise gating network so that unnecessary numerical features remain exactly unchanged. The method is trained in three stages to learn target-directed transport, enforce sparsity, and then improve proximity. Experiments on benchmark datasets show that FlowCF achieves substantially better exact numerical sparsity and numerical proximity than the baselines while remaining competitive in validity and categorical sparsity.

Critical strengths

-Clear practical motivation: The paper targets exact numerical sparsity, which is directly relevant to producing CFs that require fewer actual feature changes rather than merely small perturbations. -Strong empirical results: FlowCF consistently achieves substantially better numerical sparsity and proximity across six datasets while maintaining competitive validity, supporting the practical effectiveness of the proposed design.

Critical weaknesses
  • FlowCF mainly combines existing ingredients (flow matching, mixed-type transport, multi-head generation, and feature-wise gating) into a counterfactual pipeline. While the integration is sensible, I am not fully convinced that the technical novelty is sufficient for ICLR.
  • FlowCF formulates counterfactual generation as a transport between factual and target-class distributions, while constructing the transport through instance-wise nearest-neighbour pairings. However, counterfactual explanations are inherently instance-specific, and the paper does not clearly establish when such a distribution-level transport should recover the appropriate minimal changes for a particular individual. In particular, proximity to an observed target-class neighbour does not necessarily imply that the corresponding direction is the most relevant or minimal one for flipping the black-box decision.
  • While the paper claims that FlowCF performs strongly on categorical data, Appendix A does not seem to support that.
Questions for authors

Since FlowCF uses a frozen surrogate S both to optimize validity and to rank the final candidates, how closely does the surrogate agree with the original black-box classifier on the generated counterfactuals? Could the authors provide an ablation study to help reviewers better understand the individual contribution of each component of the proposed method? If FlowCF hyper parameters are ranked manually, is it fair with respect to baselines? Could the authors provide a fairer and more comprehensive comparison in terms of sparsity than the one currently presented? Why does FlowCF obtain better proximity than the closely related DiCoFlex e.g. in Table 3? It is not clear that FlowCF would obtain better results in this metric when optimising for sparsity. Are the examples in Figure 3 representative or hand picked?

5 comments
AuthorsOpenReview ↗

Response to Reviewer rqbF (1/2)

We sincerely thank the reviewer for the review and the many important questions, which helped us to state the contribution more precisely and to strengthen the evaluation. All changes are in the revised version.

Novelty. We appreciate this concern. In our view, the mixed flow operator, which transports numerical and categorical features jointly in one step, is a novel and useful architecture for mixed-type tabular data, and so is the three-step training (transport, sparsity gate, joint fine-tuning) that turns this flow into a sparse counterfactual generator. Our architecture also has multiple heads, trained winner-takes-all, so that a factual with several valid target directions is not pulled towards their average. This matters in practice: with a single head, validity drops from 0.970 to 0.925 and numerical sparsity almost doubles (Appendix C.4). We also believe that the focus on exact numerical sparsity is important for the field. Most existing methods only encourage small changes, and we show that samplers from a continuous density change every real-valued feature almost surely (Proposition 1). Our gate instead leaves features exactly unchanged by construction (Proposition 2). Together, this gives a combination no other method has (Table 1), and the ablation (Appendix C.4) confirms that each component contributes.

Distribution-level transport vs. instance-specific minimal changes. Thank you for this point. The transport is learned between the class distributions, but each factual is paired with its KK nearest target-class neighbours, so the map is conditioned on the instance and ends near target-class points close to it: random pairing instead raises Prox Num from 0.229 to 1.745 (Appendix C.4). We agree that this does not guarantee the minimal change, i.e. the closest point that the black box assigns to the target class. Like other amortised generators, FlowCF does not solve this optimisation for each individual, so we do not claim that its counterfactuals are minimal. Instead, it approximates it: the gate and Step 3 remove and shorten unnecessary changes, and FlowCF achieves the best numerical proximity on average (Table 3).

Appendix A. Thank you for pointing this out. We agree that this claim was a bit too strong. Section 4.2 now says FlowCF "remains competitive" on the all-categorical dataset, and Appendix A states that NICE, DiCE and CCHVAE change fewer features there, while FlowCF, at validity 0.995, is much sparser than the other amortised method, DiCoFlex (17.3% vs. 39.0%). NICE is added to Table 6.

AuthorsOpenReview ↗

Response to Reviewer rqbF (2/2)

Q1. Surrogate agreement on the generated counterfactuals. Thank you for this question. The surrogate and the black box agree on 97.5% of the returned counterfactuals (98.4% on held-out training rows), so relying on SS costs FlowCF less than two points of validity (0.988 by SS, 0.970 by the black box), see Appendix C.6 and Section 3.

Q2. Ablation. Thank you. We agree that this was missing. Appendix C.4, summarised in Section 5.1: the gate reduces numerical sparsity from 1.000 to 0.155 while raising validity, and Step 3 reduces Prox Num from 0.298 to 0.229 and raises validity to 0.970. Removing multiple heads, the pairing, or the validity or drift term lowers validity or worsens proximity.

Q3. Manual tuning and fairness. Thank you for raising the question of fairness. FlowCF uses one configuration for all datasets. Appendix C.5 halves and doubles λsparse\lambda_{\mathrm{sparse}} and λprox\lambda_{\mathrm{prox}} (also quadrupled) and varies KK: validity stays between 0.910 and 0.975, and every setting remains, on average, sparser and more proximal than all baselines. FlowCF's advantage therefore does not depend on carefully chosen hyperparameters.

Q4. A fairer sparsity comparison. Thank you. We agree and added three complementary views: ϵ\epsilon-sparsity at several tolerances and at data precision (Table 5, Appendix C.1), total sparsity over all features (Appendix C.2), and NICE as a new baseline, suggested by Reviewer 6WFu, which also achieves exact sparsity. FlowCF remains the sparsest on numerical features at every tolerance (0.146 vs. 0.218 for NICE at ϵ=0.1\epsilon = 0.1), and over all features changes 26%, second only to NICE (19%).

Q5. Proximity vs. DiCoFlex. Thank you for this question. The gain comes mainly from changing fewer features: Prox Num averages over all numerical features, and DiCoFlex changes 95% of them against 16% for FlowCF. Per changed feature, FlowCF moves about as much as DiCE and TABCF (1.2 standard deviations, Table 5), and Step 3 further shortens the displacement (Appendix C.4).

Q6. Figure 3. Thank you for asking. The two images were selected for illustration, as the caption and Section 5.2 now state. The new Appendix C.8 therefore evaluates all test images over three seeds: FlowCF changes a single latent axis on 98% and 82% of the test images, with 99% and 93% of its displacement on the target axis, more than any other method, while DiCoFlex, TABCF, REVISE, CCHVAE and Wachter change 4.1–6.0 axes on average. Since it only uses labels, its validity (0.76 and 0.89) is lower than for the methods that query the MLP directly. The new Figures 4–5 in the same appendix show further examples. (Edited: these numbers were updated after we corrected the setup of the image experiment, as explained in our reply to Reviewer 6WFu.)

Thank you again for the constructive feedback. We hope the revision addresses your concerns, and we would be glad to answer any further questions.

Thank you again for your review! We wanted to check whether our response and the revised paper address your concerns, particularly regarding our overall contribution, the instance-conditioned pairing, surrogate fidelity, and the expanded sparsity evaluation. If any concerns remain before the discussion closes, we would be very happy to clarify.

Reviewer_6WFuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

At its core, this paper proposes a new counterfactual generation technique for tabular data called FlowCF. The core insight is that existing methods often change several features (i.e. are not sparse), whereas FlowCF leverages flow matching to produce sparse explanations where only a few features are changed. This makes it easier for the end-user to comprehend the required change. FlowCF's explanations are also closer to the original / factual input (i.e. better proxmity).

The paper evaluates on 6 datasets and shows FlowCF outperforms baselines on numerical sparsity and proximity, but underperforms on categorical sparsity.

Critical strengths
  • Practical setting. FlowCF is model-agnostic, needs only labels from the black box, and is amortised. Training plus inference takes under 35 seconds per dataset.
  • Interesting extension of flow matching to mixed-type tabular data
  • Comprehensive evaluation across 6 datasets
Critical weaknesses

Please refer to questions below.

Questions for authors

1. The sparsity metric only measures whether any change was made to a feature, i.e. a change of 1e-6 would also be counted against a method. While I understand the purpose of measuring "exactly unchanged" features, please report the sparsity numbers at epsilon-sparsity distance too (i.e. sensitive to magnitude). The results would be weaker if the absolute magnitude of change is infinitesimal. This is my main concern with the paper. Report the sparsity at different tolerance thresholds.

To elaborate: most existing becnhmarks (CARLA, Dice-ML, etc) ignore tiny differences, esp. if the change is smaller than the data's precision (i.e. if say bank balance is in $0.01 increments, then a $0.0006 change is not significant). Is there any empirical justification for treating tiny difference as "feature changes"? Prior work suggests users implement changes imprecisely (Pawelczyk et al., ICLR 2023) and are indifferent to rounded vs. precise values (Choudhury et al., 2025)

2. Missing baseline. Doesn't work such as NICE [1] directly optimize for sparsity? It has been evaluated on the same datasets as yours (Adult, Magic) with good results. Please add it as a baseline and to Table 1. NICE requires logits/scores as input, so you can use your surrogate S instead to ensure it only needs labels.

3. Weak performance on categorical features. For instance, on the Adult dataset, adding both feature types up, DiCE changes fewer features overall: about 3.2 of 12 versus 4.3 for FlowCF. Can you provide an intuition for why flow matching performs worse at categorical features? Can this be mitigated? Is the gate simply moving changes from numerical to categorical features?

4. Robustness of counterfactuals. Sparse, minimal counterfactuals tend to lie close to the decision boundary, where small errors in how users carry them out can invalidate them (Pawelczyk et al). Please report the "recourse invalidation rate" metric that Pawelczyk et al propose, with noise applied (a) only to changed features and (b) to all features.

5. Categorical proximity. Understandably, the paper use categorical sparsity as categorical proximity and thus only reports numerical proximity. To get a sense of changes across both numerical and categorical features, please report total features changed, or a mixed-type distance such as Gower.

6. No quantitative ablations were run re: gating, training steps, etc.

7. (minor) Could you confirm empirically that the surrogate model (MLP) behaves similar to the black-box classifier?

[1] https://arxiv.org/pdf/2104.07411

8 comments
AuthorsOpenReview ↗

Response to Reviewer 6WFu (1/2)

We sincerely thank the reviewer for the very detailed and constructive review. The concrete suggestions, in particular the tolerance-based evaluation, the NICE baseline and the robustness metric, have substantially strengthened the paper, and we have followed all of them in the revised version.

Q1. ϵ\epsilon-sparsity at several tolerances (main concern). Thank you for raising this important point. We agree that the exact metric should be complemented by tolerance-based results. In short, FlowCF remains the sparsest method on average at every tolerance we test. Table 5 (Section 5.1) reports numerical sparsity exactly, at the precision each feature is recorded in (rounding, as you suggested), and at ϵ=0.01,0.1,0.5\epsilon = 0.01, 0.1, 0.5 training-set standard deviations, and Appendix C.1 gives per-dataset results. Many baseline changes are indeed small: at ϵ=0.1\epsilon = 0.1, DiCE drops from 0.897 to 0.416. FlowCF reaches 0.146, against 0.218 for NICE, and is first on five of six datasets. At data precision, FlowCF is unchanged (0.158), while DiCE only drops to 0.800.

We keep exact sparsity as the main metric because it does not depend on a threshold. Benchmarks set the tolerance differently (10−510^{-5} in CARLA, 5% of the range in DiCoFlex), and one tolerance treats features differently (on Adult, 0.1 standard deviations covers 1.4 years of age or USD 740 of capital gain). This agrees with Choudhury et al. [1], who find that users weigh changes differently per feature and do not generally prefer rounded over precise values. You also note that users implement changes imprecisely (Pawelczyk et al. [2]). We measure this directly with the recourse invalidation rate (Q4): when users implement the suggested changes imprecisely, FlowCF's counterfactuals are the most robust of all methods.

Q2. NICE as a baseline. Thank you very much for pointing us to NICE. It is a highly relevant method that we had missed, and adding it has made our evaluation considerably more complete. It is now included throughout: Sections 2 and 4.3, Tables 1 and 3–6, and the image experiment (Figure 3, Appendix C.8). The main results use NICE as published, with black-box scores. On average, FlowCF changes fewer numerical features (0.159 vs. 0.220) and moves them less (Prox Num 0.229 vs. 0.328), while NICE reaches full validity. Following your suggestion, Appendix C.3 also runs NICE label-only on our surrogate SS: its validity drops from 1.000 to 0.763, against 0.970 for FlowCF with the same access (Section 5.1). As you anticipated in Q4, NICE's counterfactuals lie close to the decision boundary and are among the least robust to imprecise implementation, whereas FlowCF's are among the most robust.

Q3. Categorical sparsity. Thank you for this thoughtful question. We agree that it is our weakest metric and now say so in Section 5.1. Intuition: the gate and Step 3 act feature by feature on the continuous displacement of flow matching. A categorical feature has no such displacement: it either keeps its value or switches in full, and choosing the category is an argmax that training can only relax. Is the gate moving changes to categorical features? No: it also lowers categorical sparsity, from 0.649 without the gate to 0.417 (Appendix C.4). Mitigation: doubling λsparse\lambda_{\mathrm{sparse}} lowers it to 0.350 at validity 0.952 (Appendix C.5). On your Adult example: DiCE does change fewer features in total there (0.263 vs. 0.361), but over all datasets FlowCF changes 26% of all features against 67% for DiCE (Q5).

References

[1] F. A. Choudhury, E. Leicht, J. E. Bislig, H. Guo, A. Yadav. Designing user-centric metrics for evaluation of counterfactual explanations. arXiv:2507.15162, 2025.

[2] M. Pawelczyk, T. Datta, J. van den Heuvel, G. Kasneci, H. Lakkaraju. Probabilistically robust recourse: Navigating the trade-offs between costs and robustness in algorithmic recourse. ICLR, 2023.

AuthorsOpenReview ↗

Response to Reviewer 6WFu (2/2)

Q4. Recourse invalidation rate. Thank you for this suggestion, which gives a very useful complementary view. We added it in Appendix C.7 (σ2=0.01\sigma^2 = 0.01 on [0,1][0,1]-scaled features), with noise on (a) the changed numerical features, (b) all numerical features, and (c) all features, the original setup. FlowCF is the most robust method under (a) (0.091), i.e. when users implement the suggested changes imprecisely. Under (b) and (c) it is the most robust model-agnostic method (0.217, 0.242), behind only the gradient-based DiCE, TABCF and REVISE. NICE, whose counterfactuals lie close to the decision boundary, is among the least robust (0.406, 0.428).

Q5. Total features changed / Gower. Thank you. We added both in Appendix C.2. FlowCF changes 26% of all features, second only to NICE (19%) and far below all other methods (67–79%). We do not use the Gower distance as a main metric, because in its standard unweighted form the contributions of the features are unbalanced (D'Orazio [3]). A single categorical change counts as much as moving a numerical feature across its entire range: changing a person's occupation weighs as much as changing their age from the youngest to the oldest value in the data. Many small numerical changes therefore barely register, and the distance is dominated by categorical changes. This is why the main results report numerical and categorical sparsity separately, with total sparsity as a summary that counts every changed feature equally. We report Gower for completeness: for FlowCF it is higher than for most baselines, as it is dominated by its categorical changes, our weakest metric, and on the two continuous-only datasets it is second only to NICE.

Q6. Ablations. We agree that these were missing, and thank you for asking for them. Appendix C.4 ablates the training steps and components, and Appendix C.5 varies the hyperparameters. Section 5.1 summarises both. Without the gate, every numerical feature changes. The gate brings numerical sparsity to 0.155 and raises validity, and Step 3 reduces Prox Num from 0.298 to 0.229 and raises validity to 0.970. Removing any component (multiple heads, nearest-neighbour pairing, the Step-1 validity term, the drift term) lowers validity or worsens proximity, and every hyperparameter setting remains, on average, sparser and more proximal than all baselines.

Q7. Surrogate vs. black box. Thank you for this question. Appendix C.6 and Section 3 now report it: the surrogate agrees with the black box on 98.4% of held-out training rows, 89.8% of rows near the decision boundary, and 97.5% of FlowCF's counterfactuals. It costs FlowCF less than two points of validity (0.988 according to SS, 0.970 according to the black box).

Thank you again for the very helpful feedback. We hope that the revision addresses your concerns, and we would be glad to answer any further questions.

References

[3] M. D'Orazio. Distances with mixed type variables, some modified Gower's coefficients. arXiv:2101.02481, 2021.

Thank you. I have a preliminary follow-up: in Figure 3's qualitative study, it seems FlowCF changes the hair style (from blonde to black) as well, but it is not counted as a feature change. NICE produces almost the same image but it's feature change is counted. Why does this happen?

Comments on the paper

2 comments
AuthorsOpenReview ↗

Thank you to all reviewers and summary of the revision

We sincerely thank all three reviewers for their time and effort, and for their careful and constructive reviews. Their suggestions have substantially improved the paper, most importantly the addition of NICE as a baseline, the evaluation of sparsity under a tolerance (ϵ\epsilon-sparsity), and the ablation studies. We have incorporated all of them in the revised version. We hope that it addresses the reviewers' concerns, and we would be happy to answer any further questions during the discussion.

We are encouraged that the new experiments confirm, and in several respects strengthen, the main conclusions of the paper:

  • ϵ\epsilon-sparsity: FlowCF remains the sparsest method on numerical features at every tolerance we test, including the precision of the data (0.146 vs. 0.218 for the next best method at ϵ=0.1\epsilon = 0.1).
  • NICE: FlowCF changes fewer numerical features than NICE (0.159 vs. 0.220) and moves them less (Prox Num 0.229 vs. 0.328). NICE, which uses the black-box scores, reaches full validity; with the same label-only access as FlowCF, its validity drops to 0.763, against 0.970 for FlowCF.
  • Robustness: the recourse invalidation rate revealed a further strength. FlowCF is the most robust method when users implement the suggested changes imprecisely, and the most robust model-agnostic method under noise on all features.
  • Ablation and sensitivity: every component contributes, and every hyperparameter setting we test remains, on average, sparser and more proximal than all baselines.

The individual replies give the details; here is an overview of the changes:

  • NICE baseline (6WFu): added throughout (Sections 2, 4.3 and 5.1, Tables 1 and 3–6, Figure 3), with a label-only variant in Appendix C.3.
  • ϵ\epsilon-sparsity and displacement per changed feature (all reviewers): Table 5 and Section 5.1, per dataset in Appendix C.1.
  • Ablation and sensitivity (all reviewers): Appendices C.4 and C.5, summarised in Section 5.1.
  • Surrogate fidelity (6WFu, rqbF): Section 3 and Appendix C.6.
  • Robustness (6WFu): recourse invalidation rate, Appendix C.7.
  • Total sparsity and Gower distance (6WFu): Appendix C.2.
  • Categorical sparsity (6WFu): discussed in Section 5.1.
  • Figure 3 (rqbF): marked as selected for illustration; results over all test faces and further examples in Appendix C.8.
  • Image experiment (6WFu): corrected setup, with one MLP classifier that every method explains (Section 4.2, Figures 3–5, Table 14).
  • Wording (rqbF): the Appendix A claim is now "remains competitive"; abstract, introduction and conclusion updated for NICE.

The updated code is available at https://anonymous.4open.science/r/FlowCF and in the supplementary material. The supplementary material also contains a diff PDF that shows all changes against the initially submitted version.

AuthorsOpenReview ↗

Summary of the discussion phase for the AC and all reviewers

Dear Area Chair, dear Reviewers,

We would first like to thank all three reviewers once more: their careful and constructive reviews have made the paper and its results considerably stronger. As the discussion closes, we would like to briefly summarise where it stands.

  • Reviewer 6WFu asked how sparse FlowCF is under a tolerance, for NICE as a baseline, and about the image experiment. We added numerical sparsity at several tolerances and at data precision, per dataset for all methods (Table 5, Table 7), added NICE both as published and label-only (Appendix C.3), and corrected the image experiment so that all methods explain the same classifier (Section 5.2, Table 14). In two follow-up exchanges, we also explained which NICE variant we use and how FlowCF differs from it. FlowCF remains the sparsest method on average at every tolerance, and with the same label-only access as FlowCF, the validity of NICE drops to 0.763, against 0.970 for FlowCF. The reviewer raised their score from 2 to 3.
  • Reviewer rqbF asked about novelty, an ablation of the components, the tuning and fairness of the comparison, the agreement between the surrogate and the black box, and a fairer sparsity comparison. We added an ablation of every component (Appendix C.4), a sensitivity analysis (Appendix C.5), the surrogate agreement (97.5% on the returned counterfactuals, Appendix C.6), and sparsity under tolerances and over all features (Table 5, Appendices C.1 and C.2). The reviewer found that the additional results support the improved sparsity across settings and raised their score from 2 to 3, while keeping a concern about novelty.
  • Reviewer Lt7G asked for sparsity at several tolerances, the displacement per changed feature, and an ablation of the three training steps. We added all three (Table 5, Appendix C.4). The reviewer acknowledged the new experiments, but noted that the proximity gain of FlowCF comes mainly from changing fewer features. We agree, and we see this as the intended behaviour of a sparse method. Prox Num is the standard L1 proximity of the CARLA benchmarking framework, and with the reviewer's own measure, FlowCF is the only method that lies on the Pareto front of numerical sparsity and displacement on all six datasets (Figure 7, Appendix C.9). The reviewer kept their score of 2 in a comment that came before our final reply. Since that reply answers the remaining concern with the reviewer's own measure, we believe it resolves it, and we hope it can be taken into account in the final assessment.

We believe that the revision addresses the concerns raised in the reviews, and we hope that this summary is helpful for the Area Chair. Thank you all for your time and for a constructive discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.