No Shortcuts to Truth: Exposing and Eliminating Shortcuts in AI-Generated Image Detection via Cross-Generator Consistency

Current synthetic image detectors frequently saturate on training generators but collapse on unseen ones. We trace this generalization failure to shortcut learning, a phenomenon where models capture spurious dataset correlations rather than genuine forgery evidence. We identify a fundamental structural asymmetry between these two feature types. Authenticity evidence is strictly confined to one side of the real and fake boundary, whereas shortcuts can manifest in both classes. Consequently, a shortcut-dependent detector yields fluctuating verdicts on the exact same real image when the paired synthetic generator changes. In contrast, an evidence-based model maintains prediction consistency. Building upon this insight, we propose Real Anchored Invariance, a novel data efficient multiple source framework that eliminates shortcuts without requiring prior knowledge of specific artifacts. Our method penalizes environment specific residuals along high variance directions of shared real features, dynamically upweights authentic instances that expose prediction disagreements, and simulates generator-level distribution shifts via episodic training. Extensive evaluations demonstrate that our approach achieves exceptional robustness and transferability across diverse unseen benchmarks and strictly held-out cutting-edge generators, even under extreme data scarcity.