Diagnosing Counterfactual Behavior through Local Representation Geometry

Counterfactual explanations seek small, semantically meaningful changes to an input that alter a model's prediction, helping interpret and audit machine learning systems. In modern vision, language, and multimodal systems, pretrained encoders map inputs to representation spaces, where task-specific classifier heads impose decision boundaries. Crossing a boundary changes the prediction, but the resulting representation may lie far from observed target-class examples. In applications such as biomedical research, a prediction change can mislead if it pushes the representation far from observed alternatives. To address this gap, this work uses local representation geometry to improve counterfactual search and establish a model-level diagnostic framework. A standardized local search probe characterizes each encoder–classifier pair by its counterfactual success, minimum displacement, search effort, and endpoint support. Across multiple domains and widely-used pretrained model families in vision, medical imaging, text, multimodal data, and single-cell biology, models with comparable predictive performance exhibit different counterfactual outcomes. Our framework associates these differences with classifier boundary proximity and local representation geometry, quantified by target-class support and local curvature. Incorporating support into the search objective and adapting step sizes to curvature increases counterfactual success under matched constraints. In a real-world biology case study, the generated counterfactuals align with experimentally observed differences between low- and high-immune response in single cells. Together, these findings establish counterfactual behavior as a distinct dimension beyond predictive performance and provide a framework for model diagnosis and more reliable counterfactual search.