TALA: Surrogate-Free Query-Based Black-Box Attacks on Image Forgery Localization

Cross-detector adversarial transfer for image forgery localization varies across surrogate–target pairs. Our analysis identifies substantial source-to-target effectiveness gaps on initially well-localized images, while attribution analysis suggests that the forensic evidence used for localization is both model- and input-dependent. These observations motivate us to explore whether victim localization feedback alone can guide effective attacks, without relying on cross-detector transfer. To this end, we propose Target-Adaptive Localization Attack ($\text{TALA}$), to the best of our knowledge, $\text{the first}$ query-based attack specifically designed for image forgery localization that requires neither a surrogate detector nor a surrogate-trained attack policy. TALA iteratively updates the image through local perturbations, using victim feedback to adapt their location, spatial extent, and type. It combines victim-guided patch-center sampling with failure-aware spatial penalties and determines patch sizes from the entropy-based effective support of the guided distribution. A shared rejection-feedback mechanism coordinates spatial search and switching between complementary smooth and high-frequency perturbation engines. Across three heterogeneous detectors, five datasets, and four evaluation conditions, TALA achieves the highest mean attack success rate in $\text{58 of 60}$ settings and ties for the highest in one additional setting against four baselines. It simultaneously achieves higher attack success, lower LPIPS, and higher PSNR than the strongest baseline in attack success rate in 51 settings. These findings establish surrogate-free target-side adaptation as an effective complementary approach to black-box robustness evaluation and expose the susceptibility of forgery localization detectors to local perturbations guided directly by their own feedback.