RobustGenBench: A Benchmark for Robust Generalization to Adversarial and Common Perturbations, with Applications to Vision and Vision-Enabled Large Language Models

Robust generalization (i.e., how models behave across diverse perturbation settings) is poorly understood for modern classification techniques such as robust fine-tuning from pretrained backbones and zero-shot classification with vision-enabled large language models (Ve-LLMs). We introduce RobustGenBench, a standardized benchmark of six classification datasets with reproducible evaluation splits and a unified perturbation protocol covering adversarial (${\ell }_1,{\ell }_2,{\ell }_{\infty }$) and common perturbations. Using RobustGenBench, we conduct one of the most comprehensive studies of robust fine-tuning to date (40 pretrained backbones, 2 robust losses, and 3 adaptation protocols yielding 7,200 robustness measurements). We find that convolutional architectures with TRADES perform best at base scale, and the advantage of TRADES over Classic AT widens as size grows, and that hybrid architectures unlock competitive robust generalization. We further evaluate two frontier Ve-LLMs (GPT-4o mini, Gemini 3 Flash) and show that they exceed robust fine-tuned vision models on clean and common perturbations, and exhibit a striking resilience to ${\ell }_1$ perturbations that fine-tuned models lack. By evaluating both modeling techniques on the same tasks and perturbation settings, RobustGenBench provides a unifying view of robust generalization across these regimes.