Conditional Role Allocation for Heterogeneous Supervision in Source-Free Domain Adaptation

Source-Free Domain Adaptation (SFDA) seeks to adapt a pre-trained source model to an unlabeled target domain without access to source data. Foundation models (FMs) offer new opportunities for SFDA through their rich and heterogeneous pre-trained knowledge, with their predictions serving as pseudo-supervision. Such approaches often treat additional supervision as uniformly beneficial across target samples. We find that this assumption does not always hold: supervision from a newly introduced model, such as a multimodal large language model (MLLM), can disrupt already reliable pseudo-labels, while providing valuable complementary corrections for samples not yet reliably covered. Motivated by this finding, we propose Conditional Role Allocation (CRA), a framework that coordinates FMs by assigning supervision roles according to existing reliable coverage. Specifically, CRA proceeds in three stages: (1) building reliable anchors by combining CLIP’s semantic confidence with DINOv2’s visual structural support; (2) expanding the supervision coverage through confidence ranking within predicted classes and class-balanced selection; and (3) applying MLLM adjudication constrained by CLIP-derived candidate classes to samples not covered by preceding stages. In this way, CRA preserves reliable supervision while directing complementary model knowledge toward samples where it is most needed. Extensive experiments demonstrate the effectiveness and versatility of CRA in closed-set, partial-set, and open-set SFDA, achieving state-of-the-art results on multiple benchmarks—all without accessing source data or fine-tuning foundation models.