Aligning Deviations, Not States, in CLIP-based Few-Shot Anomaly Detection

Few-shot Generalist Anomaly Detection (GAD) detects and localizes anomalies from novel categories using only a few normal references without target-class retraining. Existing CLIP-based methods typically align query features directly with normal and anomalous textual semantics, while using normal references separately for visual matching. However, anomalies are inherently deviations from an expected normal state rather than absolute visual states, making state-based alignment sensitive to normal variations across regions and categories. We therefore exploit normal references in few-shot GAD to shift CLIP alignment from states to deviations. Specifically, we calibrate the query's semantic response with that of a matched normal reference, measuring how the query deviates from its matched normal state and relating this visual deviation to the semantic shift from normal to anomalous text. This naturally yields Residual-Residual Alignment. Based on this insight, we propose Res${}^2$CLIP, which combines visual deviation magnitude with its semantic alignment for training-free anomaly scoring and further performs progressive spatial calibration followed by semantic refinement when auxiliary supervision is available. Experiments on multiple benchmarks demonstrate the effectiveness of Res${}^2$CLIP in both training-free and fine-tune settings.