acceptodds
Under review as a conference paper at ICLR 2027

Peering over the Manifold Vistas: A Unified Geometric Framework for Counterfactual XAI

14 views

Abstract

Counterfactual recourse identifies minimal predictive interventions on complex data manifolds, yet existing methods either yield off-manifold shortcuts via Euclidean descent or incur prohibitive computational overhead via Riemannian curvature pullbacks. Grounded in the local differential structure of deep representations, we propose a unified geometric framework: we derive a metric-weighted local right-inverse of the task differential, prove mathematically that it yields the strictly metric-optimal update direction, and bound manifold distortion between intrinsic pullback and Euclidean extremes. By systematically truncating task-irrelevant singular directions, our solver generates sparse, stable updates that suppress off-target attribute drift. Across cross-domain benchmarks, our method consistently outperforms prior baselines: in latent counterfactual steering, it reduces off-target distraction entropy by 30% and perceptual deviation by 29% relative to explicit Riemannian optimization while running 2.5 faster. Furthermore, on blackbox semiconductor metrology with a tabular foundation model, our formulation resolves nanoscale physical geometries 2.8 faster than standard descent under stochastic zeroth-order Jacobians.

Primary area:
interpretability and explainable AI
Keywords:
Explainable A.I., Counterfactual recourse, manifold optimization, differential geometry

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EvhLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the challenge of generating realistic counterfactual explanations without the high computational cost of Riemannian optimization. By casting counterfactual explanation generation in a generative model's latent space as a local minimum-cost problem, the paper derives an update expressed as a metric-weighted pseudoinverse of the task Jacobian. To further reduce computational cost, the paper introduces two refinements and analyzes their error bounds: retaining only task-relevant outputs and truncating weak singular values. The authors evaluate the method on StyleGAN2 facial-expression and cat-breed counterfactuals, as well as semiconductor metrology inversion using a black-box tabular foundation model, and find that it matches or outperforms prior baselines.

Critical strengths

Strength 1: I appreciate that the paper is theoretically grounded and that the authors provide a clean formulation combining several important concepts: counterfactual explanations, decoder-induced Riemannian geometry, and local inverse optimization. In particular, the paper formulates the local approximation of the latent counterfactual problem in Problem P1 and derives its closed-form solution in Theorem 1.

Strength 2: The paper introduces refinements to the closed-form solution in Theorem 1 through active-output selection and spectral truncation. These refinements can reduce computational cost, and, more importantly, the authors derive error bounds for them. For example, Proposition 1 bounds the error introduced by restricting the update to an active set of outputs, while Proposition 2 quantifies the error from retaining only the top-k singular directions.

Strength 3: I appreciate that the authors' experiments cover several different settings: facial-expression counterfactuals, cat-breed transformations, and black-box semiconductor metrology using a tabular foundation model. The evaluation results show that the proposed method consistently matches or outperforms the baselines considered.

Critical weaknesses

Weakness 1: In this work, the authors compare their proposed method with only two baselines: unconstrained Euclidean gradient descent (Wachter et al.) and Riemannian Stochastic Gradient Descent (RSGD). Moreover, in the introduction, the authors specifically point out that many prior studies have shown that Euclidean latent geometry often provides a distorted view of displacements in the generated data space. Therefore, unconstrained Euclidean gradient descent may be a relatively weak baseline. As a result, I believe that including additional and more recent methods would further strengthen the paper's claims.

Weakness 2: While I appreciate that the paper is theoretically grounded, the theoretical contribution seems rather incremental. For example, the derived metric-weighted pseudoinverse appears to be a standard solution to a weighted minimum-norm constrained optimization problem. Similarly, Proposition 2 relies on standard properties of truncated SVD.

Questions for authors

Question 1: Could the authors provide additional experimental comparisons with relevant baselines from more recent studies, beyond Euclidean gradient descent and RSGD?

Question 2: Could the authors better emphasize the theoretical and algorithmic contributions of this work relative to existing inverse-based optimization methods?

1 comment
AuthorsOpenReview ↗

New Benchmarks (Adam, RAdam, GN) & Prop. 1 vs. Prop. 2

We thank the reviewer for their encouragement. Upon the reviewer’s suggestion, we have evaluated the Cats experiments (our main benchmark) with four more methods, Adam, Riemannian Adam (RAdam), Gauss Newton (GN), and Metric induced Gauss Newton (Theorem 1, GN +G). We picked Adam as the modern counterpart to Wachter (baseline) and we chose GN (vanilla + Proposition 2) to help properly contrast our method and highlight the strength of Proposition 1. We computed the metric versions too, matching the existing setting and using the same loss and hyperparameters: Wachter to Adam, and, Ours to GN, to keep the comparisons fair. The results for the overall performance of all the methods are as follows:

Target BreedMethodTarget Prob. (%) ↑\uparrowFlip Rate (%) ↑\uparrowSteps to 50% ↓\downarrow∥Δw∥2\lVert\Delta w\rVert_2D(xcf)\mathcal{D}(x_{\text{cf}}) ↑\uparrowΔD\Delta\mathcal{D} ↑\uparrowLPIPS ↓\downarrowCompute Time (s) ↓\downarrow
OverallWachter60.0 ± 34.866.5%258 ± 1993.3 ± 0.5-8.3 ± 0.90.2 ± 0.70.23 ± 0.0828
RSGD43.7 ± 34.950.5%334 ± 1862.7 ± 0.6-8.3 ± 0.90.2 ± 0.80.35 ± 0.10196
Adam97.8 ± 9.299.0%20 ± 368.3 ± 2.1-8.4 ± 0.60.2 ± 1.00.69 ± 0.0827
RAdam0.5 ± 0.90.0%500 ± 00.1 ± 0.0-8.5 ± 0.90.0 ± 0.10.01 ± 0.00195
GN31.0 ± 42.233.0%392 ± 1733.7 ± 0.4-8.4 ± 0.80.1 ± 0.50.20 ± 0.09263
GN (+G)68.4 ± 37.174.0%137 ± 18118.9 ± 5.7-8.6 ± 0.5-0.0 ± 1.00.78 ± 0.19361
Ours57.6 ± 36.360.0%302 ± 1854.7 ± 0.5-8.4 ± 0.90.2 ± 0.60.23 ± 0.0673
Ours (+G)72.0 ± 37.477.5%116 ± 16816.6 ± 5.6-8.5 ± 0.60.0 ± 1.00.76 ± 0.20196

*Reran experiment 2 on A40 GPUs due to revision traffic on GPU cluster.

Proposition 1 vs Proposition 2 : Based on the reviewers comments, we compared GN with (Prop 2. Spectral Truncation) against Ours (Prop1. Class Truncation) to better highlight the difference. Before starting, a full GN is actually not feasible to be computed for these problems given the Jacobian would be 1000x512. For argument sake, we added all the cat classes and the top 15 common occurring classes of the input images. Comparing the two, firstly, Ours performs better than GN in overall target prob., flip rate, ΔD\Delta\mathcal{D}: 57.6 vs 31, 60% vs 33%, 0.2 vs 0.1, respectively. However, GN performed better than Ours on ∥Δw∥2\|\Delta w\|_2, D(xcf), LPIPS: 3.7 vs 4.7, -8.4 for both, 0.2 vs 0.23, respectively. These features scored more given the GN method didn't move far from its original image and rarely caused a flip. The reason can be easily seen if we contrast the bound of Prop.1 and Prop.2, in Prop.2 even if we have some residual direction in the main class in the tail of the spectrum, the inversion of the Jacobian will add noise causing stagnation, whereas, in Prop.1, the direction b is in the numerator and non critical dimensions push the term to zero. This is the key insight of Proposition 1.

Adam vs Ours : We gave Wachter (baseline) a significant advantage, firstly, a cross entropy loss instead of standard euclidean and also a cosine scheduler. However, we also computed Adam to contrast the latent method. The same loss and weights were used as Wachter. We observed that Adam excelled in target prob, flip rate, steps to 50%: 97.8 vs 57.6, 99 vs 60, 20 vs 302, respectively. However, when we observed the LPIPS scores and ∥Δw∥2\lVert\Delta w\rVert_2, 0.69 vs 0.23, 8.3 vs 4.7, respectively; and the reconstructions, we noticed that Adam transformed the original image significantly. We also note that RAdam did not even leave the origin. That said, we can't say there don't exist parameter configurations where Adam might do better, but in this experimental setting, giving it the same landscape as the baseline, Adam gave good statistics but poor counterfactuals.

Key Insights The new computations helped more clearly contrast our approach, we thank the reviewer for the suggestion. We want to go back to the assumption of local invertibility in foundation models, which is the core motivation for our work, and little commented on. In this setting, if locally there is a subspace, with lower intrinsic dimension, then naturally, restricting to the entangled classes spanned by this subspace induces a local inverse. There is growing evidence that this is how large models pack critical information. Prop. 1 then gives us a clear bound. Inverse-based methods are no match for GD methods, because of the need for the inverse. But in this counterfactual setting, we need to disentangle the classes before taking our update step. It is computationally more expensive, but we have shown, we find better counterfactuals (given there are many).

Reviewer_egtqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies counterfactual explanation in the latent space of generative models, i.e. StyleGAN2. It formulates the local counterfactual update as a minimum-norm optimization problem under the decoder-induced perturbation (P1) and derives a closed-form solution (Eq. 6). It also truncates inactive output classes to obtain sparse updates and gives error bounds for this truncation (Eq. 8). Experiments cover facial expression editing, cat counterfactuals, and parameter inversion in semiconductor with a tabular foundation model.

Critical strengths
  1. The method is evaluated not only on image generation but also on a real-world problem (physical semiconductor grating profile reconstruction). This shows that the formulation can be applied to a quite general domain.

  2. The paper formulates the counterfactual update as a metric-weighted pseudoinverse and gives error bounds for truncating output classes, giving a clean view of the update.

Critical weaknesses
  1. The paper motivates the method by arguing that Euclidean distance in the latent space is distorted by the decoder, that is, G≠IG \neq I. However, the main method sets G=IG = I, and the paper gives no justification that this is reasonable for StyleGAN. Also, the truncation acts as implicit manifold regularization is not supported.

  2. The novelty seems limited since the solution of P1 is the standard weighted minimum-norm solution, which corresponds to a Gauss–Newton step, and the authors themselves state that the closed form is not the contribution. The remaining theory (Corollary 1 and Proposition 1) mainly reinterprets this solution or bounds the error of truncating it. So it is not clear what the main technical contribution is beyond applying a known solution to counterfactual explanation.

  3. The paper does not explain how Eq. 7 is actually computed, for example how JhJ_h and GG are obtained in practice. The main text only gives Algorithm 1, so more details are needed on how these quantities are computed or approximated

  4. The experiments seems not support the proposed method, and the results are not clearly better than the Euclidean baseline, which also does not use GG. In Table 1, Ours has a similar LPIPS but takes more than twice as long. In addition, the explanation of the results in Section 5.1 is not convincing. The comparison is based on a single image, and claims such as preserving identity or producing localized edits are not supported by any quantitative evaluation.

Questions for authors

Please refer to weaknesses.

3 comments
AuthorsOpenReview ↗

Geometric Formulation, Active-Set Updates, and Empirical Evaluation

We thank the reviewer for their questions. We address W1–W4 and present new analysis of the metric, Eq. 7, and our empirical results.

Choice of G(z)=IG(z)=I and implicit manifold regularization. We do not claim that the latent space of StyleGAN2 is Euclidean: the decoder-induced metric motivates our framework (Eq. 3), and Ours is its isotropic special case G=cIG=cI (Corollary 1), which avoids estimating GG. StyleGAN2's path-length regularization encourages more uniform generator-Jacobian sensitivities (Karras et al., 2020), but does not establish a globally scaled-isometric Jacobian; we will clarify this limitation. We also withdraw the claim that truncation acts as implicit manifold regularization: Proposition 1 bounds the distance between the active-set step and the full step; to first order, the active step affects inactive outputs through their gradient correlation with active classes (EIAE_{\mathcal{IA}}), the second term of Eq. 8.

Novelty. Eq. 6 is the classical weighted minimum-norm solution of P1. Our contribution is the surrounding counterfactual framework: (i) we formulate counterfactual recourse as a task-restricted local right-inverse problem under the decoder-induced metric; Corollary 1 restricts the classifier differential JfJ_f to the generator's tangent space and recovers the task pseudoinverse for isotropic metrics; (ii) selecting active outputs at every step reduces the linear solve to an ∣At∣×∣At∣|\mathcal A_t|\times|\mathcal A_t| system, with ∣At∣≪q|\mathcal A_t|\ll q (q=1000q=1000 in the cat benchmark), and Proposition 1 bounds the selection error; (iii) Proposition 2 bounds the error of the alternative, which truncates singular directions rather than classes; and (iv) we demonstrate applicability to a black-box tabular foundation model using zeroth-order Jacobian estimates.

A cat-benchmark comparison supports class over spectral truncation. We ran Gauss–Newton (GN) with spectral truncation (Proposition 2) on a fixed set of 22 classes (the 12 ImageNet cat classes 281–292 and the 10 most frequent non-cat classes), using the same residual b=etarget−pb=e_{\mathrm{target}}-p and step length as Ours, over 50 origins × 4 targets. Ours achieves a 60.0% flip rate versus 33.0% for GN, with about 3.6× lower runtime (73 s vs. 263 s per trajectory), while GN has slightly lower LPIPS (0.20 vs. 0.23).

Computing Eq. 7 and implementation details. For Ours (G=IG=I), each step computes only the ∣At∣|\mathcal A_t| active rows of JhJ_h (one reverse-mode pass per active class) and solves

\qquad \lambda=10^{-4},$$ followed by $\delta z=J_{\mathcal A_t}^{\top}\alpha$. This damped SVD step avoids forming an $m\times m$ matrix. For **Ours (+G)**, $G(z)$ is estimated at every step using $K=16$ Rademacher vector–Jacobian products of the generator: $$G\approx\tilde J^{\top}\tilde J+10^{-3}I$$ (Appendix B.3). We invert this estimate by eigendecomposition, discarding eigenvalues below $0.02\lambda_{\max}$; in practice, at most 16 directions are retained. The update is $$\delta z=G^{-1}J_{\mathcal A_t}^{\top} \big(J_{\mathcal A_t}G^{-1}J_{\mathcal A_t}^{\top}+\lambda'I\big)^{-1}b_{\mathcal A_t},$$ where $\lambda'$ is $\lambda$ multiplied by the mean diagonal entry of $J_{\mathcal A_t}G^{-1}J_{\mathcal A_t}^{\top}$. **Ours** uses a fixed step length $\eta=0.08$, whereas **Ours (+G)** uses the full step, capped at $\|\delta z\|\leq1.5$. The additional generator passes and the $512\times512$ eigendecomposition contribute to the higher runtime of **Ours (+G)** (196 s vs. 73 s per trajectory in the cat-benchmark rerun). **Experimental support.** We agree that the experiments do not establish uniform superiority over Euclidean latent-space gradient descent. In the extended cat-benchmark rerun (50 origins × 4 targets), **Ours** matches Wachter on LPIPS (0.23), has a lower flip rate (60.0% vs. 66.5%), and is 2.6× slower (73 s vs. 28 s per trajectory). These results differ from the submitted Table 1 (72.5% flip rate for **Ours**); we will clarify and report the rerun consistently. The runs support a more targeted-update interpretation. Across all 1000 classes at steps 0 and 50, **Ours** changes the probability mass assigned to non-cat classes only marginally (0.357 to 0.361), whereas Wachter raises it to 0.420 and RSGD to 0.448. Its $L_1$ shift of non-cat probabilities is also lowest (0.084 vs. 0.117–0.290 for RSGD, Wachter, and Adam). **Ours** also outperforms RSGD (60.0% vs. 50.5% flip rate; 73 s vs. 196 s) and spectrally truncated GN (W2) in flip rate and runtime, and reaches the target in 69 vs. 198 steps on metrology inversion (Sec. 5.3). Section 5.1 uses a single seed. We therefore withdraw the claims of identity preservation and localized edits and present that example as a qualitative illustration only.

Thank you for the detailed response and the additional experiments. However, my overall impression has not changed. Most of the main claims, such as the geometric justification and its advantage over the Euclidean baseline, are not supported by the new results. For these reasons, I will keep my original score.

AuthorsOpenReview ↗

A Thought Experiment on the Geometric Rationale

We thank the reviewer for taking the time to reconsider our work and for the valuable feedback. We understand that the additional experiments have not changed the reviewer's overall assessment. While the rebuttal phase is still open, we would like to clarify one aspect of our geometric rationale through a simple thought experiment.

Thought Experiment. Imagine we have a latent space which is the surface of a 3D sphere. Over this sphere, we have a classifier which maps to 5 classes. It is clear that this latent space does not have Euclidean geometry. Furthermore, if we take the Jacobian, we get 5 directions, but given we only have 2 intrinsic directions on the manifold, the 5 directions are an over-parametrization of the tangent space, and the individual directions get mixed together. Hence, at most two directions can be linearly independent, forcing classes to share and mix directions in the Jacobian. In the 5-dimensional class space, the cross-class gradient correlation matrix therefore has rank at most two, meaning correlations are confined to a low-dimensional subspace rather than spanning all classes independently. Proposition 1 in essence is hinting towards this phenomenon.

How Proposition 1 Relates to StyleGAN. We do not claim that the true latent metric of StyleGAN is the identity—that is not true; instead, Proposition 1 gives the bounds for the intrinsic semantic subspace. Not all classes are correlated with all others; training groups related classes together that it resolves by local hyperplane partitions. In the case of StyleGAN, we have a 512-dimensional latent space and a 1,000-class classifier. Not all 512 directions are for cats, but the classifier inherently encodes the directional subspace for the cats, and this is what we exploit. We can only do this because of the classifier, since inside it is also information regarding the classes.

We again emphasize that this does not mean that the Euclidean metric is the true latent metric, nor that the resulting updates are geodesic. Rather, the identity-metric variant provides a task-directed local update that can exploit dependencies among classifier outputs even without explicitly estimating the decoder-induced metric. When the pullback metric is available, the metric-weighted variant additionally accounts for the local geometric distortion induced by the decoder.

We hope this helps clarify Proposition 1 and our geometric rationale a bit better for the reviewer. We thank the reviewer again for their feedback and quick response.

Reviewer_JFTuOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a novel method for computing counterfactual explanations (CE). CEs are minimal perturbations of a sample that change the prediction of that sample (e.g. flip a classification class) that are semantically meaningful. The proposed method uses a generative model to model the data distribution and optimizes the CE in the latent space of the generative model using Gauss-Newton updates. To improve performance, instead of optimizing all output dimensions, only selected dimensions can be optimized, leading to a sparse version of the method. The technique is validated on two vision datasets (changing emotions shown by a face and changing the breed of a cat) an well as on an inverse problem from semiconductor manufacturing.

Critical strengths
  1. To the my knowledge, the idea of using a Gauss-Newton step to optimize the output of the predictive model is novel, as is restricting these updates to selected dimensions of the output (Proposition 1).
  2. The application of the method to semiconductor manufacturing is novel and interesting.
Critical weaknesses
  1. The conceptual novelty of the proposed method is limited. Optimization in the latent space of a generative model has been analyzed theoretically and empirically before, see e.g. [1]. Flat geometries have been assumed both on Z ("Ours", e.g. [1]) and X ("Ours + G", e.g. [2]) before. Hence, the novelty lies in using a Gauss-Newton step and selecting "active" output dimensions. The authors point out that the significance of this contribution lies in the fact that it enables the bound in Proposition 1 (l. 207). This proposition is indeed novel and interesting but not a substantial conceptual contribution by itself. I would characterize the contribution therefore as mostly an engineering improvement, rather than a significant conceptual novelty.
  2. I do not find the empirical results fully convincing. In Section 5.1, only one sample is shown, so it remains unclear if the method performs well on many samples or just on selected ones. Section 5.2 reports quantitative measures and shows clear improvements for one of the methods ("Ours") in the distraction entropy (Figure 3b), but compared with latent gradient descent (“Wachter et al.”), “Ours” shows no improvement in the realism measures: D(xcf)\mathcal{D}(x_{cf}), ΔD\Delta\mathcal{D} and LPIPS are essentially identical, while latent drift is larger and runtime is longer. I also have some questions about the selection of the baseline methods, see below.

References:
[1] Dombrowski, A.-K., Gerken, J. E., Müller, K.-R., & Kessel, P. (2024). Diffeomorphic Counterfactuals With Generative Models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(5), 3257–3274. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/TPAMI.2023.3339980
[2] Arvanitidis et al., ICLR 2018, https://openreview.net/forum?id=SJzRZ-WCZ

Questions for authors
  1. Did you run the baseline methods with the same objective bb as your method?
  2. Did you use an optimizer with an adaptive learning rate (e.g. Adam) in the baseline methods (as done e.g. in [1])? If you did not, it would be a straightforward improvement over a fixed learning rate that would be the correct comparison to using the Gauss-Newton step you propose.
  3. How do the methods compare at a fixed compute budget?
  4. Do you have an explanation for why "Ours" is essentially on par with latent gradient descent (“Wachter et al.”) for D(xcf)\mathcal{D}(x_{cf}), ΔD\Delta\mathcal{D} and LPIPS but outperforms the baselines on the distraction entropy?
3 comments
AuthorsOpenReview ↗

Answers to Q1–Q4 & Distraction Entropy Analysis

We thank the reviewer for their questions and suggestions. We address Q1–Q4 (Part 1), We present some new analysis which hopefully shed more light on the conceptual novelty of Proposition 1.

Q1) Baseline objective The two families, Gradient Based (GB) and Inverse Based (IB), differ in their objective types, but within the family, we gave the same objective. For GB, we gave cross-entropy loss, and for IB methods, it was the directional vector to the desired counterfactual target.

Q2) Adaptive Learning Rate The reviewer raises a very important point here, and we did not use Adam, however, we implemented Cosine Annealing learning rate schedule for Wachter so as to keep the baseline strong. However, during the rebuttal phase, we included Adam as a solver too, keeping the same objective as Wachter. (please see the reply to Reviewer EvHL, and subsection titled Adam vs Ours for our analysis). In summary, we found that Adam performed really strongly in maximising the target probability, very fast, however, at the cost of transforming the original image. That said, we do not state that there is no possible configuration where Adam could perform better, but rather, we argue that our Wachter baseline is a fair and counterfactually meaningful baseline for our analysis.

Q3) Fixed Budget We are not clear on which factor the reviewer was suggesting to budget. Given Ours always needs two forward passes of the model to do a single update (Proposition 1, and then Jacobian), we will always take longer for the same number of steps, as seen by the 28s vs 73s per trajectory in the text. That said, counterfactual generation is not a global optimization problem; the methods do not converge to the same point, hence, we feel it is hard to gauge w.r.t. fixed costs.

Q4) Distraction Entropy Difference A notable finding in the Cats experiment is that although Wachter and Ours use different tangent-space geometries and achieve comparable overall realism (D(xcf)D(x_{\text{cf}}), ΔD\Delta\mathcal{D}, and LPIPS), Ours achieves significantly lower distraction entropy. When considered alongside the fact that Ours exhibits a larger latent displacement ∥Δw∥2\|\Delta w\|_2 (traveling further in latent space under the Euclidean norm), lower distraction entropy means fewer background pixels are altered despite moving further in latent space; suggesting a much more targeted semantic change.

We attribute this behavior directly to Prop. 1. At Step 0, the classifier assigns high probability to the source breed (e.g., Tabby cat), while residual probability mass remains distributed across non-cat classes (such as carton, basket, or paper towel) triggered by background context away from the animal. Because standard gradient descent penalizes all non-target logits, early updates push down these background classes and perturb off-cat pixels. In contrast, Ours (1) prunes low-probability background classes from the active set, and (2) projects the target probability step bb onto the Jacobian subspace spanned almost exclusively by competing cat breeds, decoupling breed transitions from unrelated background features.

To verify this empirically on the new runs, We evaluating ResNet-50 across all 1,000 classes at Step 0 and Step 50 (avoiding flip-rate selection bias):

SettingActive Set ∥A∥\|A\| (p>0.01p > 0.01)Cat Classes in ∥A∥\|A\|Non-Cat Classes in ∥A∥\|A\|Non-Cat Prob. MassNon-Cat L1L_1 Shift
Start (Step 0)4.2 ± 0.44.2 ± 0.40.0 ± 0.20.357 ± 0.037—
Wachter (Step 50)4.6 ± 1.24.3 ± 1.00.2 ± 0.60.420 ± 0.1270.141 ± 0.066
RSGD (Step 50)4.7 ± 1.04.5 ± 0.70.2 ± 0.60.448 ± 0.0920.117 ± 0.072
Adam (Step 50)2.4 ± 1.32.2 ± 0.50.2 ± 1.10.115 ± 0.1470.290 ± 0.088
Ours (Step 50)4.5 ± 0.84.4 ± 0.70.1 ± 0.30.361 ± 0.0880.084 ± 0.049

Active Jacobian Dimensions: At Step 0, the active set contains on average 4.2 dimensions, all corresponding to cat breeds. By Step 50, 4.3 of Wachter's 4.6 active dimensions are cat classes (with 0.2 non-cat), whereas Ours retains 4.4 cat classes out of 4.5 active dimensions; meaning $\sim$98% (4.4/4.54.4/4.5) of our active Jacobian rows correspond to cat breeds, with the lowest non-cat count (0.1 vs. 0.2). Notably, Adam collapses quickly to 2.4 active dimensions, yet 0.2 (~8%) are still non-cat classes compared to ~2% for Ours.

Preserving Background Mass: Ours keeps total non-cat probability virtually unchanged (0.361 vs. 0.357) with the lowest non-cat L1L_1 drift (0.084 vs. 0.117–0.290). Background regions remain undisturbed, directly explaining the lower distraction entropy, that is, probability mass is reallocated among cat breeds.

Thank you for the additional experiments and responding to my questions.

I think your new results on Adam sound interesting and mostly convincing, although I find the poor performance of Adam surprising, given that it has performed better in the literature (e.g. in [1] from my original review).

What I meant with fixed compute budget would e.g. be to fix the number of forward passes. I don’t think that the fact that the different methods do not converge to the same point means that compute-fixed comparisons are meaningless. After all, the authors also compare different methods to each other which also converge to different points. Therefore, a comparison in mean over many samples for the target metrics after a fixed number of forward steps would be meaningful.

Thank you for digging deeper into the distraction entropy, I find your analysis convincing.

The rebuttal alleviated some of the concerns in weakness 2 of my original rebuttal, but not all of them. In particular, my concerns about Section 5.1 remain unaddressed. Since also weakness 1 persists, I will retain my score.

AuthorsOpenReview ↗

Clarification on Section 5.1 Scope and Compute Budget

We thank the reviewer for taking the time to go through our revisions and for the constructive follow-up, particularly the encouraging feedback on our distraction entropy analysis. We apologize for overlooking your concern regarding Section 5.1; while the rebuttal phase is still open, we would like to clarify two brief points:

Section 5.1 (Baby Expressions). As noted in the text, Section 5.1 was intended purely as a toy exemplar to demonstrate that an inverse-based update can produce a valid counterfactual result, rather than to make broader claims about visual quality. Because random seeds naturally vary in visual artifacts, we selected one clear illustrative seed for the main text; we will explicitly clarify its scope as a single-seed qualitative showcase and are very happy to include all generated samples in the supplement for full transparency.

Fixed Compute Budget. Thank you for clarifying that you meant matching the total number of forward passes. Our method performs two forward passes per step compared to Wachter. Hence, for a fixed budget scenario, our step tt should be compared to step 2t2t of the others. However, RSGD, for example, requires many forward passes to evaluate the metric, yet we perform on par with or better than it across metrics. We are happy to rescale the scores of all the methods w.r.t a forward pass rather than steps.

We thank the reviewer again for taking the time to engage with our work and for their helpful suggestions.

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback. In response, we have made the following revisions:

We conducted an additional experiment and incorporated the corresponding results and discussion.

We added two relevant references to better position our work with respect to existing literature.

We revised the single-seed example in Section 5.1, removing claims of identity preservation and localized edits and clarifying its role as a qualitative illustration.

We refined the presentation of our method and contributions to better distinguish them from existing approaches and clarify their scope.

For the convenience of the reviewers and program committee, all changes in the rebuttal revision are marked in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.