Bending Early, Curbing Deep: Test-Time Adversarial Defense for Large Vision-Language Models Based on Residual Curvature

Although Large Visual-Language Models (LVLMs) have achieved significant advances, their vulnerability to adversarial perturbations poses a persistent challenge to their reliable deployment in the real world. Prevalent test-time defenses largely focus on mitigating adversarial effects while overlooking how adversarial deviations propagate through the ViT-based visual encoder. In particular, their reliance on multi-view transformations and closed-set class space limits their applicability to generative tasks in LVLMs. In this paper, we find that adversarial perturbations first induce a geometric abnormality of early residual trajectories by shifting them from differential toward common component, with the resulting deviations subsequently amplified predominantly through deep MLP branches. Therefore, we propose Curvature-based Universal Residual Bending (CURB), a test-time defense that selectively bends abnormally flattened residual trajectories through input updates to curb adversarial growth. CURB does not depend on downstream outputs space, allowing universal and direct operation across open-ended generation and zero-shot classification. Experiments on LLaVA-1.5-7B and InternVL3.5-8B show consistent robustness gains in various tasks, together with substantial reductions in adversarial representation deviations.