What CLIP Sees When Structure Breaks: Patch-Shuffle Response as a Generalizable Cue for AI-Generated Image Detection

Existing CLIP-based detectors identify AI-generated images from the static representation of a single image, which is inevitably entangled with image semantics and thus generalizes poorly to unseen generators. In this work, we shift the focus from asking what an image is to how its representation reacts when its global structure is deliberately broken. Specifically, we compare an image's representation with that of its patch-shuffled counterpart, where global spatial arrangement is destroyed while local patches remain intact. We observe that real images exhibit significantly stronger response shifts than generated ones within a frozen CLIP feature space, suggesting that real representations rely heavily on globally coherent structures, whereas generated ones are dominated by local patterns. To harness this observation, we propose SPARE Structural Perturbation-Aware REpresentation), a framework that converts structural responses into a reliable discriminative signal, including a Patch-Shuffle structural response module constructs the perturbed view and computes layer-wise responses; a layer-wise response gating module adaptively weights each layer's contribution based on its reliability; and a dual-branch response fusion detector integrates the gated responses with the original representations as a content reference. Notably, the CLIP encoder remains entirely frozen throughout. Extensive experiments across multiple benchmarks demonstrate that SPARE consistently surpasses state-of-the-art detectors in cross-generator generalization.