acceptodds
Under review as a conference paper at ICLR 2027

Diagnosing Counterfactual Behavior through Local Representation Geometry

26 views

Abstract

Counterfactual explanations seek small, semantically meaningful changes to an input that alter a model's prediction, helping interpret and audit machine learning systems. In modern vision, language, and multimodal systems, pretrained encoders map inputs to representation spaces, where task-specific classifier heads impose decision boundaries. Crossing a boundary changes the prediction, but the resulting representation may lie far from observed target-class examples. In applications such as biomedical research, a prediction change can mislead if it pushes the representation far from observed alternatives. To address this gap, this work uses local representation geometry to improve counterfactual search and establish a model-level diagnostic framework. A standardized local search probe characterizes each encoder–classifier pair by its counterfactual success, minimum displacement, search effort, and endpoint support. Across multiple domains and widely-used pretrained model families in vision, medical imaging, text, multimodal data, and single-cell biology, models with comparable predictive performance exhibit different counterfactual outcomes. Our framework associates these differences with classifier boundary proximity and local representation geometry, quantified by target-class support and local curvature. Incorporating support into the search objective and adapting step sizes to curvature increases counterfactual success under matched constraints. In a real-world biology case study, the generated counterfactuals align with experimentally observed differences between low- and high-immune response in single cells. Together, these findings establish counterfactual behavior as a distinct dimension beyond predictive performance and provide a framework for model diagnosis and more reliable counterfactual search.

Primary area:
interpretability and explainable AI
Keywords:
counterfactual explanations, representation geometry, decision boundaries, target-class support, local curvature, model selection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_b63HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper deals with incorporating a local geometry module for counterfactual generation. Authors provide an analysis of the faults of generating counterfactuals while only taking into account the boundary crossing of a classifier. Results showcase that adapting the generation to emphasize local geometry in the encoder space creates better supported counterfactuals. Authors also provide an analysis of different model combinations that showcases varying performance and validate all framework choices both experimentally and mathematically.

Critical strengths

Strengths:

  • The geometry proposition is well-founded and addresses a real problem. The recognition that despite the samples crossing the decision boundary they might still be unrelated to the desired class is known and the proposed method seemingly addresses it.
  • The paper is easily readable and proposes all required information in a good way. There is no bloat and each module has an explanation for why it earns it's place in the framework.
Critical weaknesses

Weaknesses:

  • Neural networks are not optimal and might be biased to information stemming from the dataset. Despite counterfactual research mainly dealing with classifiers and encoders, there is also a gap into criticizing the bias of the original models themselves. How can we be sure the geometry learned here is not searching for a geometric bias in the learned representation and instead looks at geometry? Is there an experiment that validates that such geometric topology, so to say, corresponds to real information?

  • There is no comparison with global geometry. There is a significant emphasis on the importance of the locality of the clusters, but there is no comparison with incorporating the distance or morphology between the clusters themselves. For example, incorporating global geometry, how clusters differ between themselves, might provide information about differences between them.

  • There is no comparison with prior work, all the results are different model combinations. There is no quantification of what the geometry module achieves due to the lack of baselines and the gap metrics are not particularly indicative of how well the model does. There are also no cross model ablations. While explanation is given for why these model combinations are chosen, the results are not reported and it is difficult to evaluate how dependent the performance is to the combination

  • It is also not clear apart from proposing the local geometry for counterfactuals if the authors have made an effort to adapt the geometry losses specifically for the counterfactual generation or if they propose changes. Most modules are relatively simple and already exist in some capacity in literature and the design is not substantially altered.

Questions for authors

Questions:

  • Is there a comparison with global information that could better isolate the gain from incorporating local geometry?
  • Are there comparisons with prior works that deal with similar problems? Are there results that could be reported by utilizing already established metrics, along with the proposed ones? The proposed metrics being only evaluated on the baseline and not on prior works makes the comparison difficult as is.
  • Could you provide the results of different model combinations, so that the performance difference can be gauged numerically?
5 comments

We thank the reviewer for finding the geometry module well-motivated and the framework clear. We clarify each concern below.

W1: Does the geometry capture real information in the data?

Yes. Encoders can carry bias, so we test the geometry against two sources of truth independent of the encoder:

  • Lab measurements. On Papalexi, decoded changes correlate with measured expression differences (r=0.91r=0.91) and oppose measured knockout effects (r=−0.78r=-0.78).
  • True labels. With shuffled labels, support stops predicting counterfactual success (R2R^2 0.59 → 0.54; boundary alone: 0.53).

W2, Q1: Comparison with global geometry

To test whether global geometry adds information, we added global features (e.g., class-center distance and overlap, Figure 9). Local geometry adds more predictive value: it raises R2R^2 by 0.10 over boundary + global, while global features add only 0.02 over local geometry.

FeaturesHeld-out R2R^2 (CF-Suc)
Global only0.27
Boundary only0.53
Boundary + global0.55
Boundary + support0.59
Boundary + support + curvature0.63
All0.65

W3, Q2: Comparison with prior work, using established metrics

Our method achieves the highest Supp-Suc and beats DiCE and REVISE on distance, LOF, and IM1. DiCE flips more often, and FACE and retrieval score better on LOF [4] and IM1 [5] but succeed less and move much farther (Figure 5). We compare with DiCE [1], FACE [2], REVISE [3], and retrieval on all six datasets, on the same frozen embeddings, 12 models, and starting points. Running all methods in embedding space is the most direct comparison of search strategies on our task, though not the native setting of DiCE, FACE, and REVISE. To check this choice, we also run DiCE and FACE natively on Papalexi gene expression and score their encoded outputs with the same classifier and metrics (ours keeps higher Supp-Suc and rr; Appendix C.4). Such native runs are possible only for some datasets, because these methods are tied to a specific input type, whereas ours applies to any encoder. LOF and IM1 do not depend on our support definition. Norm. CF-Dist divides distance by the encoder's RMS spread. Lower is better for all three.

MethodCF-SucNorm. CF-DistLOFIM1Supp-Suc
Baseline search0.810.041.210.890.67
Ours0.910.061.140.860.79
DiCE0.940.091.370.980.47
FACE0.610.661.050.830.34
REVISE0.830.211.190.910.55
Retrieval0.720.581.010.810.37

W3, Q3: Results for each model combination

We read this as asking for (i) absolute results per model rather than gaps, (ii) the effect of the geometry module on each model, and (iii) whether the conclusions depend on which pairs are selected.

(i, iii) How much does performance depend on the combination? Within each pair, accuracy differs by at most 0.03, while baseline CF-Suc differs by 0.10 to 0.65 and Supp-Suc by 0.09 to 0.57. Pairs were chosen by validation accuracy, before computing any counterfactuals.

(ii) What does the geometry module achieve? It improves or preserves both metrics for all 12 models, most where baseline search fails most (e.g., MNIST ViT). Per-model values (Appendix C.3):

ModelACCCF-Suc: base → geomSupp-Suc: base → geom
Shapes ResNet500.851.00 → 1.000.96 → 0.97
Shapes ViT0.860.47 → 0.940.45 → 0.70
IMDb BERT0.800.81 → 0.890.72 → 0.74
IMDb RoBERTa0.801.00 → 1.000.88 → 0.88
MNIST ResNet500.951.00 → 1.000.90 → 0.95
MNIST ViT0.960.35 → 0.930.33 → 0.70
Chest X-ray ResNet500.830.80 → 0.910.68 → 0.77
Chest X-ray ViT0.820.53 → 0.660.44 → 0.55
MM-IMDb CLIP0.711.00 → 1.000.87 → 0.92
MM-IMDb SigLIP20.690.90 → 0.950.78 → 0.86
Papalexi scGPT0.920.97 → 0.970.30 → 0.62
Papalexi Geneformer0.890.41 → 0.630.40 → 0.62

W4: Is the geometry adapted to counterfactual generation?

Each component is designed for counterfactual search in learned representations. The support loss (Eqs. 11, 24) recomputes target-class neighbors at every step, keeping the search near observed target-class data. The curvature-based step size (Eq. 25) shrinks updates where the representation is less linear. Table 2 shows each component contributes. Our contribution is how these measures are combined and used: together, they diagnose why similarly accurate models differ in counterfactual behavior and improve counterfactual search. To our knowledge, no prior work has combined them in this way, and we benchmark the framework against established methods across six datasets spanning diverse modalities.

[1] Mothilal et al. 2020. [2] Poyiadzi et al. 2020. [3] Joshi et al. 2019. [4] Breunig et al. 2000. [5] Van Looveren & Klaise 2021.

AuthorsOpenReview ↗

Followup on Rebuttal

We thank the reviewer again for their feedback. We have revised the paper based on their comments and those of the other reviewers, as summarized here (summary of revisions comment). We would appreciate hearing whether our responses address their concerns and would be happy to clarify any remaining ones.

Reviewer_b63HOpenReview ↗

Reply to rebuttal

Thank you to the authors for the rebuttal. While the new experiments shed some more light on how the methodology performs, I feel that the answer to W4 confirms that the contribution is the analysis of the paper and not a new module. Also, in the answer to W3 the baseline is already better than the rest of the methodologies on Supp-Suc, the one metric where the framework ranks first. Why is the framework behind prior methods on the other metrics, and could the authors clarify what is the overall value of the module is?

Reviewer_urZSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for diagnosing counterfactual behavior in pretrained representation spaces, distinguishing prediction changes from outcomes supported by observed target-class data. It relates search success, displacement, effort, and endpoint support to decision-boundary proximity and local representation geometry.

Across six datasets, similarly accurate models exhibit substantially different counterfactual outcomes. Incorporating target-class support and curvature into search increases average supported success from 0.67 to 0.79. A single-cell case study further shows that decoded counterfactual changes align with observed gene-expression differences.

Critical strengths
  1. A useful diagnostic perspective supported by broad and partially controlled experiments. The paper makes a consequential distinction between predictive accuracy, successful prediction changes, and data-supported prediction changes. The empirical study goes beyond isolated counterfactual examples: encoder pairs are selected by validation performance without consulting counterfactual outcomes, and classifier heads are varied while representations remain fixed. Table 1 reports counterfactual-success gaps of 10–65 percentage points despite accuracy gaps of at most three percentage points. The appendix further considers loss matching, search sensitivity, retrieval, and partial fine-tuning. Together, these experiments provide substantial evidence that accuracy alone does not characterize outcomes under the evaluated counterfactual procedures.

  2. The biological evaluation goes beyond the support metric used to guide search. The Papalexi case study evaluates decoded changes against observed expression differences across 4,000 genes, rather than relying exclusively on target-class neighborhood proximity. The reported improvements in correlation—from 0.74 to 0.91—and directional agreement—from 62.7% to 76.2%—provide additional evidence that the proposed search can recover meaningful population-level structure. Training the decoder without the evaluated test cells is a useful design choice.

Critical weaknesses

W1. Further validation is needed to establish how much the observed counterfactual differences reflect model properties.

The paper shows that similarly accurate models can have substantially different counterfactual success rates and relates these differences to decision-boundary placement and representation geometry. However, the current search objective increases the target-class score without directly encouraging it to exceed competing class scores. Consequently, search may fail to find a nearby counterfactual because it follows an ineffective direction, even when such a counterfactual exists. The appendix provides useful sensitivity analyses for optimizers, step sizes, and iteration budgets, but these retain the same objective and therefore do not fully resolve this concern. It would strengthen the paper to repeat the main model comparisons using a target-probability or margin-based objective and examine whether the success-rate gaps persist under matched locality radii and iteration budgets. This comparison would help distinguish limitations of the particular search procedure from the geometric model properties emphasized by the paper.

W2. The incremental contribution of curvature needs further validation. The paper reports improvements in diagnostic prediction and counterfactual search when incorporating curvature, but the experiments do not fully separate its contribution from other components. In Figure 4, boundary proximity alone is compared against boundary proximity combined with both target-class support and curvature. Adding a boundary-plus-support baseline would clarify whether curvature provides predictive information beyond these two factors. For search, the curvature-only ablation is useful, but it remains unclear whether the improvement requires curvature information or simply better step-size control. It would strengthen the paper to compare curvature-dependent updates against a validation-tuned constant step and a standard adaptive step-size method, while keeping the support objective, locality radius, and iteration budget fixed. Reporting computational costs would also help assess the additional neighborhood and tangent-space computations. These comparisons would establish whether curvature offers a distinct benefit beyond target-class support and ordinary step-size adjustment.

Questions for authors

Q1. Can the framework help choose a model for unseen data? The paper shows that similarly accurate models can have different counterfactual profiles. If a model is selected based on its supported-success rate on validation data, does it also perform better on held-out test data? This would clarify the practical value of the framework for model selection.

Q2. How much do the results depend on the available reference data? The support criterion relies on nearby observed target-class examples. Does the framework give consistent conclusions when fewer reference examples are available? This matters because a region may appear unsupported simply because the relevant examples were not collected.

Q3. How does the biological result compare with using an observed high-response cell directly? The decoded counterfactual changes align well with observed expression differences. How does this compare with using the nearest high-response reference cell as the endpoint? This would help clarify what the proposed search adds beyond selecting an existing target-class example.

5 comments

We thank the reviewer for recognizing the value of our controlled experiments and biological evaluation. We clarify each concern below.

W1: Do the gaps reflect the models or the search?

The gaps reflect the models: they persist with a margin-based objective under the same radius and budget. We agree that our original objective raises the target score without pushing it above competing classes, so search can miss a counterfactual that exists. As suggested, we repeated the Table 1 comparisons with the same radius τ\tau and budget T=300T=300, under two controls:

  • Margin objective: minimize max⁡c≠y′hc−hy′\max_{c\neq y'} h_c-h_{y'}, which pushes the target above all competing classes.
  • Fixed margins: rescale heads to match typical logit gaps without changing decision boundaries.

The mean gap decreases only slightly, from 0.38 to 0.33 with the margin objective and 0.35 after rescaling. The search objective therefore accounts for a small part of each gap, and the differences between models remain. The gaps also persist with encoder-scaled radii (mean 0.31), a common 32-dimensional PCA space (0.29), and shared target classes (0.33). These results support our original finding: similarly accurate models differ in counterfactual behavior because of the models themselves, not because of how we search.

CF-Suc gap (Table 1 pair)OriginalMarginFixed margins
Shapes0.530.470.49
IMDb0.190.150.16
MNIST0.650.590.57
Chest X-ray0.270.220.23
MM-IMDb0.100.090.11
Papalexi0.560.480.53
Mean0.380.330.35

W2: Curvature contribution

Curvature adds information beyond support and beyond step-size control. We ran both suggested comparisons.

Prediction: curvature adds a further 0.04 beyond boundary + support (0.59 to 0.63). We added the boundary + support baseline to Figure 4 (mean held-out R2R^2 for CF-Suc):

PredictorsHeld-out R2R^2
Boundary0.53
Boundary + support0.59
Boundary + support + curvature0.63

Search: setting the step size from curvature works better than simpler step-size rules. With everything else held fixed, it reaches the highest Supp-Suc (0.79, vs 0.76 for a tuned fixed step and 0.75 for Adam) and is best on 4 of 6 datasets. The cost is speed: it takes about 1.5 seconds per example instead of 0.6, because it fits a local plane at every step.

Step ruleSupp-SucTime per example (s)
Tuned constant0.760.6
Adam0.750.33
Curvature0.791.5

Q1: Can the framework guide model selection on unseen data?

Yes. We tested this as a model-selection task. Each dataset has several encoders (3 for most, 11 for MM-IMDb, 2 for Papalexi). Using only validation data, we pick one at random, by highest accuracy, or by highest Supp-Suc, then check on held-out test data whether it has the highest test Supp-Suc. Picking by Supp-Suc finds the best encoder in 5 of 6 datasets, versus 3 of 6 by accuracy. The one miss was Chest X-ray, where the picked encoder ranked second, only 0.020.02 below the best. The picked encoders also lost almost no accuracy (within 0.030.03 of the most accurate candidate).

How the encoder is pickedDatasets where it picked the test-best encoder
Random choice (average)~2 / 6
Validation accuracy3 / 6
Validation Supp-Suc5 / 6

Q2: How much do the results depend on the available reference data?

The conclusions stay consistent with half the reference data and weaken only modestly with a quarter. To test this, we kept a random 100%, 50%, or 25% of the reference examples, reran the search, and evaluated support on held-out data (means over six datasets). With half the references, the Supp-Suc gain is unchanged (0.07) and the ordering of models stays close to the full-data ordering (rank correlation 0.89). With a quarter, the gain shrinks to 0.04 but stays positive, and the ordering stays broadly similar (0.71). We agree that missing examples can make a region look unsupported; this makes the criterion conservative, but the overall comparisons hold.

Reference dataSupp-Suc gain over baseline searchRank correlation with full-data model ranking
100%0.071.00
50%0.070.89
25%0.040.71

Q3: Using an observed high-response cell

Our counterfactual beats the nearest observed high-response cell on every measure while moving only about half as far. For each low-response cell, we decoded the nearest observed high-response cell as an alternative endpoint and compared both with lab-measured changes across 4,000 genes.

EndpointrrDirectionTop 100 up / down
Baseline search0.7462.7%64.5% / 56.8%
Nearest observed cell0.7971.8%63.1% / 61.9%
Ours0.9176.2%75.0% / 67.0%
AuthorsOpenReview ↗

Followup on Rebuttal

We thank the reviewer again for their feedback. We have revised the paper based on their comments and those of the other reviewers, as summarized here (summary of revisions comment). We would appreciate hearing whether our responses address their concerns and would be happy to clarify any remaining questions.

AuthorsOpenReview ↗

Follow-up before the discussion closes

With the discussion period coming to an end, we would appreciate hearing whether the reviewer has any remaining concerns. We have addressed each point raised in their review and would welcome further questions. If our responses and paper revisions resolve their concerns, we would appreciate them considering an update to their scores.

Reviewer_TH4UOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a local representation geometry framework for diagnosing and improving counterfactual behavior in pretrained encoder–classifier systems. It analyzes decision-boundary proximity, target-class support, and local curvature, and incorporates support-aware optimization and curvature-adaptive step sizes into counterfactual search. Experiments across six datasets suggest that models with similar predictive performance can exhibit substantially different counterfactual behavior, while geometry-guided search improves counterfactual success and endpoint support. A single-cell case study further compares generated counterfactual expression changes with experimentally observed transcriptional differences. However, questions remain regarding methodological novelty, cross-model evaluation fairness, the independence of the support metric, and the strength of the empirical evidence.

Critical strengths
  1. Relevant problem formulation: The paper highlights an important limitation of conventional counterfactual evaluation: successfully changing a model's prediction does not necessarily produce a plausible alternative supported by observed data. Explicitly distinguishing CF-Suc from Supp-Suc provides a useful perspective for evaluating counterfactual behavior.
  2. A coherent diagnostic framework. The paper considers classifier-side decision-boundary proximity alongside representation-side target-class support and local curvature.
  3. A connection between diagnosis and optimization. Rather than limiting the contribution to post hoc analysis, the authors use target-class proximity to guide the search objective and local curvature to adapt optimization step sizes.
  4. Broad empirical evaluation with useful sensitivity analyses and a biological case study.
Critical weaknesses
  1. The proposed framework combines k-nearest-neighbor-based target-class support, a local curvature proxy, and curvature-adaptive optimization. Although reasonable, these components are relatively straightforward, and the paper does not sufficiently establish how the overall approach advances beyond existing manifold-aware or latent-space counterfactual methods. More direct comparisons with competitive counterfactual-generation baselines under matched computational budgets are needed to clarify the contribution.
  2. Potential confounding factors in cross-model evaluation: Different encoders produce representations with potentially different dimensionalities, scales, and local distance distributions. Consequently, using the same absolute locality radius and step size does not necessarily impose equivalent search constraints, making cross-model comparisons potentially unfair. In addition, each model selects its own highest-scoring alternative class, so different models may solve counterfactual tasks of different difficulty. More systematic representation normalization, normalized displacement metrics, and experiments with shared target classes are needed to establish that the reported differences genuinely reflect counterfactual behavior rather than representation scaling or target-selection effects.
  3. Limitations of the proposed geometric diagnostics and evaluation criteria. The supported-counterfactual metric is closely related to the target-neighbor proximity term optimized during search, which may favor the proposed method by construction without independently establishing broader counterfactual plausibility. The local curvature proxy may also reflect noise, neighborhood heterogeneity, or representation scaling rather than curvature alone. The paper would benefit from independent support evaluation, comparisons with simpler geometric alternatives, and sensitivity analyses over the tangent-subspace dimension. These additions would help establish whether the proposed geometric quantities provide robust information beyond the specific optimization objective and existing boundary-based diagnostics.
  4. Limited validation of biological applicability and incomplete reproducibility details. Although the paper evaluates six datasets and includes a single-cell case study, the biological results do not establish individual-level counterfactual validity or causal feasibility. Agreement between decoded expression changes and population-level transcriptional differences is encouraging, but does not demonstrate that the generated representations correspond to biologically realizable states for individual cells. The authors should clarify the scope of these conclusions and provide more complete implementation details, including the tangent-subspace dimension, optimization hyperparameters, dataset construction procedures, and criteria for defining low- and high-response cells, to facilitate reproducibility.
Questions for authors

Please see the weaknesses.

5 comments

We thank the reviewer for recognizing the value of our framework and its broad evaluation. We clarify each concern below.

W1: Contribution beyond existing methods

Our contribution has two parts. First, the joint use of boundary proximity, target-class support, and curvature: together, they explain why similarly accurate models behave differently and improve counterfactual search. Second, the framework applies unchanged across six datasets spanning images, text, multimodal, and single-cell data. To our knowledge, this is the first work to evaluate counterfactual behavior across models and modalities in this way.

As requested, we compared with DiCE [1], FACE [2], REVISE [3], and retrieval on all six datasets, on the same frozen embeddings, 12 models, starting points, and radius τ\tau. Our method achieves the highest Supp-Suc and beats DiCE and REVISE on distance, LOF, and IM1. DiCE flips more often, and FACE and retrieval score better on LOF [4] and IM1 [5] but succeed less and move much farther (Figure 5). Running all methods in embedding space is the most direct comparison of search strategies, though not the native setting of DiCE, FACE, and REVISE. To check this choice, we also run DiCE and FACE natively on Papalexi gene expression and score their encoded outputs with the same classifier and metrics (ours keeps higher Supp-Suc and expression-change correlation; Appendix C.4). Native runs are possible only for some datasets, as these methods are tied to one input type, whereas ours applies to any encoder. LOF and IM1 do not depend on our support definition, and Norm. CF-Dist divides distance by the encoder's RMS spread; lower is better for all three.

MethodCF-SucNorm. CF-DistLOFIM1Supp-Suc
Baseline search0.810.041.210.890.67
Ours0.910.061.140.860.79
DiCE0.940.091.370.980.47
FACE0.610.661.050.830.34
REVISE0.830.211.190.910.55
Retrieval0.720.581.010.810.37

(Means over six datasets)

W2: Fairness across encoders

The gaps between models persist after controlling for scale, dimension, and target class: the mean CF-Suc gap goes from 0.38 to 0.29-0.33, while accuracy gaps stay at most 0.03. We repeated Table 1 under three controls:

  • Scaled budget: the radius scales with each encoder’s embedding spread, and the step size lets search cover it within T=300T=300 steps.
  • Common dimension: all embeddings are reduced to the same 32 dimensions with PCA.
  • Shared targets: both models get the same starting example and target class.
CF-Suc gapOriginalScaledPCAShared
Shapes0.530.440.310.47
IMDb0.190.110.150.12
MNIST0.650.510.480.58
Chest X-ray0.270.240.290.22
MM-IMDb0.100.140.070.11
Papalexi0.560.390.420.49
Mean0.380.310.290.33

W3: Geometric diagnostics and evaluation

Supp-Suc by construction. The gain holds when support is evaluated independently of search. Using held-out references, the Supp-Suc gain over baseline search is 0.07 (0.12 with the search's own references, Table 2). Our method also beats DiCE and REVISE on LOF and IM1, which it does not optimize (W1 table).

Curvature proxy. On a Swiss roll, where true curvature is known, the proxy tracks it well (rank correlation 0.67). This holds under rescaling by 0.1 or 10 (0.67–0.70) and uneven sampling (0.73).

Simpler alternatives. Curvature adds information that a simpler measure does not. Held-out R2R^2 for CF-Suc rises from 0.53 (boundary) to 0.59 (with support) and 0.63 (with our proxy), while intrinsic dimension [6] adds nothing (0.58).

Tangent dimension. For mm from 2 to 16, held-out R2R^2 stays between 0.61 and 0.63.

W4: Biological scope and reproducibility

The single-cell case study demonstrates a real-world application, not individual-cell validity: sequencing destroys each cell, so no cell is measured in both states. We present the counterfactuals as population-level hypotheses about gene programs, supported by three tests:

  • Measured changes: decoded changes correlate with measured differences (r=0.91r=0.91, vs 0.74 for baseline search).
  • Real cells: for scGPT, the share of counterfactuals near real high-response cells (Supp-Suc) rises from 30% to 62%.
  • Knockouts: our counterfactuals oppose measured interferon-pathway effects (r=−0.78r=-0.78; −0.06-0.06 for other genes), and the interferon-γ Hallmark set [7] ranks first of 50.

Implementation details. Appendix B.6 lists the hyperparameters and response criteria; B.1 describes the datasets.

[1] Mothilal et al. 2020. [2] Poyiadzi et al. 2020. [3] Joshi et al. 2019. [4] Breunig et al. 2000. [5] Van Looveren & Klaise 2021. [6] Levina & Bickel 2004. [7] Liberzon et al. 2015.

AuthorsOpenReview ↗

Followup on Rebuttal

We thank the reviewer again for their feedback. We have revised the paper based on their comments and those of the other reviewers, as summarized here (summary of revisions comment). We would appreciate hearing whether our responses address their concerns and would be happy to clarify any remaining ones.

AuthorsOpenReview ↗

Follow-up before the discussion closes

With the discussion period coming to an end, we would appreciate hearing whether the reviewer has any remaining concerns. We have addressed each point raised in their review and would welcome further questions. If our responses and paper revisions resolve their concerns, we would appreciate them considering an update to their scores.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response: Summary of Revisions

We thank the reviewers for their constructive feedback and appreciate the positive assessments, especially on:

  • Problem and framing: "a real problem" with a "well-founded" geometry proposition (b63H); distinguishing CF-Suc from Supp-Suc "provides a useful perspective" (TH4U).
  • Evidence: "broad empirical evaluation" (TH4U) providing "substantial evidence that accuracy alone does not characterize outcomes" (urZS).
  • Biological validation: it "goes beyond the support metric used to guide search" (urZS).
  • Clarity: "easily readable" with "no bloat" (b63H).

Reviewers asked how our method compares with established counterfactual methods, whether the differences between models reflect the models rather than the search or representation scale, and whether Supp-Suc favors our method by construction. They also asked what curvature adds beyond support, and requested per-model results and a comparison with global geometry.

We believe we have addressed all comments, and we accordingly made the following revisions (new text in blue). In the main text (Section 5), we added comparisons with DiCE, FACE, REVISE, and retrieval per dataset (Fig. 5), controls for differences between models, independent support evaluation, and a boundary + support baseline for curvature (Fig. 4). The appendices add per-model results (C.3), comparison details and runtime (C.4), Swiss-roll validation of curvature (A.2), global geometry (B.4), and implementation details (B.6). Importantly, the paper's original findings hold under every new control and comparison.

We want to highlight two points. First, our framework uses the geometry of the model’s own representation both to diagnose and to improve counterfactual search: it explains why encoders with similar accuracy differ in counterfactual behavior, predicts when standard search will fail, and guides search past those failures. Prior methods neither explain this variation across models nor use the geometry of the model’s own representation to guide search. Second, prior methods depend on the input type: DiCE and FACE need input features, which is hard for text or image–text pairs, and REVISE needs a task-specific generative model. Our framework works in embedding space, which every model has, so it generalizes unchanged to images, text, multimodal, and single-cell data. To our knowledge, this is the first work to evaluate and improve counterfactual behavior across models and modalities in this way.

We are happy to provide any additional clarifications the reviewers might need.

AuthorsOpenReview ↗

Update after discussion

We sincerely thank all reviewers for their time and for engaging so constructively with our responses. Your feedback has substantially strengthened the paper. We are glad that urZS found their main concerns sufficiently addressed and raised their rating, and that TH4U confirmed that three of their four concerns resolved. We have answered every remaining question in our individual replies, including TH4U’s point on contribution and computational budget and b63H’s follow-up on the module’s contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.