Decoupled Streams for Open-Set Test-Time Adaptation of Vision-Language Models

Open-set test-time adaptation requires a model to improve classification on in-distribution (ID) samples while reliably rejecting unknown out-of-distribution (OOD) inputs. Existing approaches for vision-language models optimize these two objectives through the same trainable parameters, implicitly assuming that their updates remain compatible throughout adaptation. We show that this compatibility progressively weakens during adaptation: classification and ID/OOD separation gradients become less aligned, while recent gradient-rebalancing and modular adapter-routing strategies do not consistently improve both objectives jointly. This motivates a simple, yet effective alternative: rather than reconciling the objectives within a shared adaptation space, we separate their optimization. We introduce DSTA (Decoupled Streams for Open-Set Test-Time Adaptation), which equips vision-language models with two independent sets of low-rank adapters for classification and ID/OOD separation over a shared frozen backbone, while retaining their interaction through OOD-guided sample selection. In this decoupled regime, DSTA stabilizes the separation stream with a novel margin-based objective that prevents excessive ID/OOD gap increase, and proposes automatic OOD logit-scale calibration to avoid compressed confidence scores. Across clean and corrupted open-set benchmarks, DSTA improves the joint behavior of ID classification and OOD detection, with particularly strong gains in unknown-sample rejection under distribution shift. These benefits remain evident across backbone sizes, supporting parameter-space decoupling as an effective design principle for open-set test-time adaptation. Code will be released upon acceptance.