- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the challenge of generating realistic counterfactual explanations without the high computational cost of Riemannian optimization. By casting counterfactual explanation generation in a generative model's latent space as a local minimum-cost problem, the paper derives an update expressed as a metric-weighted pseudoinverse of the task Jacobian. To further reduce computational cost, the paper introduces two refinements and analyzes their error bounds: retaining only task-relevant outputs and truncating weak singular values. The authors evaluate the method on StyleGAN2 facial-expression and cat-breed counterfactuals, as well as semiconductor metrology inversion using a black-box tabular foundation model, and find that it matches or outperforms prior baselines.
Critical strengths
Strength 1: I appreciate that the paper is theoretically grounded and that the authors provide a clean formulation combining several important concepts: counterfactual explanations, decoder-induced Riemannian geometry, and local inverse optimization. In particular, the paper formulates the local approximation of the latent counterfactual problem in Problem P1 and derives its closed-form solution in Theorem 1.
Strength 2: The paper introduces refinements to the closed-form solution in Theorem 1 through active-output selection and spectral truncation. These refinements can reduce computational cost, and, more importantly, the authors derive error bounds for them. For example, Proposition 1 bounds the error introduced by restricting the update to an active set of outputs, while Proposition 2 quantifies the error from retaining only the top-k singular directions.
Strength 3: I appreciate that the authors' experiments cover several different settings: facial-expression counterfactuals, cat-breed transformations, and black-box semiconductor metrology using a tabular foundation model. The evaluation results show that the proposed method consistently matches or outperforms the baselines considered.
Critical weaknesses
Weakness 1: In this work, the authors compare their proposed method with only two baselines: unconstrained Euclidean gradient descent (Wachter et al.) and Riemannian Stochastic Gradient Descent (RSGD). Moreover, in the introduction, the authors specifically point out that many prior studies have shown that Euclidean latent geometry often provides a distorted view of displacements in the generated data space. Therefore, unconstrained Euclidean gradient descent may be a relatively weak baseline. As a result, I believe that including additional and more recent methods would further strengthen the paper's claims.
Weakness 2: While I appreciate that the paper is theoretically grounded, the theoretical contribution seems rather incremental. For example, the derived metric-weighted pseudoinverse appears to be a standard solution to a weighted minimum-norm constrained optimization problem. Similarly, Proposition 2 relies on standard properties of truncated SVD.
Questions for authors
Question 1: Could the authors provide additional experimental comparisons with relevant baselines from more recent studies, beyond Euclidean gradient descent and RSGD?
Question 2: Could the authors better emphasize the theoretical and algorithmic contributions of this work relative to existing inverse-based optimization methods?
New Benchmarks (Adam, RAdam, GN) & Prop. 1 vs. Prop. 2
We thank the reviewer for their encouragement. Upon the reviewer’s suggestion, we have evaluated the Cats experiments (our main benchmark) with four more methods, Adam, Riemannian Adam (RAdam), Gauss Newton (GN), and Metric induced Gauss Newton (Theorem 1, GN +G). We picked Adam as the modern counterpart to Wachter (baseline) and we chose GN (vanilla + Proposition 2) to help properly contrast our method and highlight the strength of Proposition 1. We computed the metric versions too, matching the existing setting and using the same loss and hyperparameters: Wachter to Adam, and, Ours to GN, to keep the comparisons fair. The results for the overall performance of all the methods are as follows:
*Reran experiment 2 on A40 GPUs due to revision traffic on GPU cluster.
Proposition 1 vs Proposition 2 : Based on the reviewers comments, we compared GN with (Prop 2. Spectral Truncation) against Ours (Prop1. Class Truncation) to better highlight the difference. Before starting, a full GN is actually not feasible to be computed for these problems given the Jacobian would be 1000x512. For argument sake, we added all the cat classes and the top 15 common occurring classes of the input images. Comparing the two, firstly, Ours performs better than GN in overall target prob., flip rate, ΔD: 57.6 vs 31, 60% vs 33%, 0.2 vs 0.1, respectively. However, GN performed better than Ours on ∥Δw∥2, D(xcf), LPIPS: 3.7 vs 4.7, -8.4 for both, 0.2 vs 0.23, respectively. These features scored more given the GN method didn't move far from its original image and rarely caused a flip. The reason can be easily seen if we contrast the bound of Prop.1 and Prop.2, in Prop.2 even if we have some residual direction in the main class in the tail of the spectrum, the inversion of the Jacobian will add noise causing stagnation, whereas, in Prop.1, the direction b is in the numerator and non critical dimensions push the term to zero. This is the key insight of Proposition 1.
Adam vs Ours : We gave Wachter (baseline) a significant advantage, firstly, a cross entropy loss instead of standard euclidean and also a cosine scheduler. However, we also computed Adam to contrast the latent method. The same loss and weights were used as Wachter. We observed that Adam excelled in target prob, flip rate, steps to 50%: 97.8 vs 57.6, 99 vs 60, 20 vs 302, respectively. However, when we observed the LPIPS scores and ∥Δw∥2, 0.69 vs 0.23, 8.3 vs 4.7, respectively; and the reconstructions, we noticed that Adam transformed the original image significantly. We also note that RAdam did not even leave the origin. That said, we can't say there don't exist parameter configurations where Adam might do better, but in this experimental setting, giving it the same landscape as the baseline, Adam gave good statistics but poor counterfactuals.
Key Insights The new computations helped more clearly contrast our approach, we thank the reviewer for the suggestion. We want to go back to the assumption of local invertibility in foundation models, which is the core motivation for our work, and little commented on. In this setting, if locally there is a subspace, with lower intrinsic dimension, then naturally, restricting to the entangled classes spanned by this subspace induces a local inverse. There is growing evidence that this is how large models pack critical information. Prop. 1 then gives us a clear bound. Inverse-based methods are no match for GD methods, because of the need for the inverse. But in this counterfactual setting, we need to disentangle the classes before taking our update step. It is computationally more expensive, but we have shown, we find better counterfactuals (given there are many).