What makes a good anchor? Offline Anchor Selection for Test-Time Adversarial Robustness of CLIP

Test-time defense techniques improve the zero-shot adversarial robustness of CLIP without retraining. Among them, rectification-based methods provide a direct and low-overhead strategy by seeking a correction that counteracts the attack by moving the adversarial representation toward an anchor, a transformation of the input intended to approximate the unavailable clean representation. Because this anchor serves as the reference for estimating the correction, its selection is fundamental to successful rectification. Yet existing methods typically prescribe an anchor a priori, with no criterion for comparing it against alternatives, and justify it only through the performance of the resulting defense. In this paper, we formulate anchor selection as an offline measurement problem and ask: what makes a good anchor? We identify four complementary properties required for effective rectification and quantify them in CLIP feature space to construct an Anchor Quality Score (AQS). We rank representative anchors from 10 transformation families based on their AQS and validate this ordering against the ranking obtained by building the full defense around each anchor. This experiment selects JPEG compression at quality 50 as the strongest anchor. Building on this selection, we introduce Anchor-Guided Rectification using Compression (ARC), a lightweight test-time defense that combines the selected anchor with a soft anchor-consistency scaling of the rectification budget. Across ten zero-shot classification benchmarks, ARC delivers substantial improvements in adversarial robust accuracy over SOTA test-time defenses while maintaining comparable clean accuracy and inference cost, and remains effective under adaptive attacks that explicitly target the complete defense.