Peering over the Manifold Vistas: A Unified Geometric Framework for Counterfactual XAI

Counterfactual recourse identifies minimal predictive interventions on complex data manifolds, yet existing methods either yield off-manifold shortcuts via Euclidean descent or incur prohibitive computational overhead via Riemannian curvature pullbacks. Grounded in the local differential structure of deep representations, we propose a unified geometric framework: we derive a metric-weighted local right-inverse of the task differential, prove mathematically that it yields the strictly metric-optimal update direction, and bound manifold distortion between intrinsic pullback and Euclidean extremes. By systematically truncating task-irrelevant singular directions, our solver generates sparse, stable updates that suppress off-target attribute drift. Across cross-domain benchmarks, our method consistently outperforms prior baselines: in latent counterfactual steering, it reduces off-target distraction entropy by 30% and perceptual deviation by 29% relative to explicit Riemannian optimization while running 2.5$\times$ faster. Furthermore, on blackbox semiconductor metrology with a tabular foundation model, our formulation resolves nanoscale physical geometries 2.8$\times$ faster than standard descent under stochastic zeroth-order Jacobians.