Distributional Causal Probing of Vision Models via Self-Controlled Interventions

Occlusion studies often summarize model responses by average performance changes, obscuring heterogeneous effects across images and their relationship to representation changes. We introduce Self-Controlled Intervention, a paired design for probing frozen vision models with deterministic interventions at prespecified doses and directions. Building on this design, we establish a statistical framework that characterizes heterogeneous decision effects, representation changes, and their dependence. We conduct extensive numerical experiments in image classification and object detection to evaluate the framework. The results reveal substantial effect heterogeneity and task-dependent directional vulnerability. These findings challenge robustness assessments based on average performance and caution against inferring decision-level vulnerability from representation drift alone.