Connector–Learner Coupling for Safe Domain Specialization of Vision–Language Models

Specializing vision–language models (VLMs) for professional domains requires improving domain competence while preserving multimodal safety. Balancing these objectives is challenging: domain adaptation can weaken safety behavior, while safety tuning can degrade performance on legitimate tasks. We study this interaction through Connector–Learner Coupling, which captures how the visual connector shapes the representations used for language-side professional learning and how the resulting learner state, in turn, shapes professional and safety gradients for connector updates. This perspective leads to a hierarchical formulation in which professional adaptation defines a connector-conditioned learner response, while professional and safety objectives are evaluated on the same coupled model state. Based on this formulation, we develop CoLead, a first-order co-adaptation method. In each cycle, CoLead first adapts the learner through multiple professional updates with the connector fixed, then jointly updates both components using normalized professional and safety gradients evaluated at the adapted state. The updated connector–learner pair carries forward across cycles without differentiating through the adaptation trajectory. Across finance, radiology, and autonomous driving, CoLead achieves lower mean validation negative log-likelihood on both professional and safety data than mixed fine-tuning and flattened gradient coordination under matched data exposure. Generation-based evaluations further show competitive professional performance alongside improved safety-related measures on multiple benchmarks, while revealing limitations in hallucination control and cross-dataset transfer.