Learning Where to Adapt: Sparse Delta Learning for Online Test-Time Adaptation

Modern learning systems are often trained in controlled or simulated environments but deployed under shifting real-world conditions, creating a persistent training–deployment gap. Online Test-Time Adaptation (OTTA) addresses this gap by updating models from unlabeled target data, yet existing methods mainly focus on how to adapt while largely fixing where updates occur. We introduce Sparse Delta Learning (SDL), which formulates OTTA as budgeted support learning over structured parameter groups. SDL separates support discovery from parameter adaptation. In the first stage, a dense functional update is coupled with a group-sparse structural variable, and Group Split-LBI traces an inverse-scale-space path to discover a data-dependent support under an explicit group budget. In the second stage, SDL initializes adaptation with the Stage-1 change inside the discovered support, applies the original TTA optimizer only within that support, and writes the refined update back to the persistent model. This formulation leaves the base adaptation objective unchanged and extends to fully connected, convolutional, and Transformer parameters through architecture-specific group definitions. On VisDA-C with DeiT-S/16 and SHOT, SDL improves final offline mean per-class accuracy by $14.29$ percentage points over the strongest budget-matched sparse baseline while updating only $0.0213$% of the model.