One Calibration, Many Requests: Amortized Black-Box Multimodal Jailbreaking via Cross-Model Refusal Coordinates

Multimodal foundation models couple visual and textual information, which can make jailbreak behavior transferable across architectures. Many universal attacks rely on surrogate-side optimization without target-specific adaptation, whereas query-based black-box methods often require fresh target queries for every request. We study whether open models can provide a reusable low-dimensional interface for a stricter setting: one calibration on a closed target, followed by reuse of the same multimodal carrier on unseen requests. We propose UniCal, a black-box multimodal jailbreak method that realizes this calibrate-once, reuse-many paradigm. UniCal first aligns refusal and compliance structures from heterogeneous surrogates into a shared coordinate space and tests whether the learned coordinates predict refusal and compliance behavior under leave-one-model-out registration. A coordinate-conditioned generator then maps a fixed benign seed and a low-dimensional coordinate to a request-independent image and text carrier. For a new target, the pretrained generator is frozen, and only a sparse coordinate offset is selected through an evaluator-mediated three-way decision-label interface. The selected carrier is then reused without per-request updates. Across the five coordinate-pool models, UniCal achieves 89.6% mean strict success under one-time decision-label adaptation. Without any target-side calibration, the same surrogate-prepared carrier transfers to the two external unseen architectures with 80.1% strict success on Molmo and 77.9% on Phi-Vision. On closed targets, independently calibrating each dated API evaluation yields 81.7% mean strict success, retains 93.5% benign utility, and amortizes the 192-query calibration after eight reused requests against label-only Random Search.