acceptodds
Under review as a conference paper at ICLR 2027

Learning Where to Adapt: Sparse Delta Learning for Online Test-Time Adaptation

Abstract

Modern learning systems are often trained in controlled or simulated environments but deployed under shifting real-world conditions, creating a persistent training–deployment gap. Online Test-Time Adaptation (OTTA) addresses this gap by updating models from unlabeled target data, yet existing methods mainly focus on how to adapt while largely fixing where updates occur. We introduce Sparse Delta Learning (SDL), which formulates OTTA as budgeted support learning over structured parameter groups. SDL separates support discovery from parameter adaptation. In the first stage, a dense functional update is coupled with a group-sparse structural variable, and Group Split-LBI traces an inverse-scale-space path to discover a data-dependent support under an explicit group budget. In the second stage, SDL initializes adaptation with the Stage-1 change inside the discovered support, applies the original TTA optimizer only within that support, and writes the refined update back to the persistent model. This formulation leaves the base adaptation objective unchanged and extends to fully connected, convolutional, and Transformer parameters through architecture-specific group definitions. On VisDA-C with DeiT-S/16 and SHOT, SDL improves final offline mean per-class accuracy by percentage points over the strongest budget-matched sparse baseline while updating only % of the model.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Test-Time Adaptation, Inverse Scale Space, Split Linearized Bregman Iteration

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_J797OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Sparse Delta Learning (SDL) learns which parameter groups to update for each unlabeled target batch in online test-time adaptation. Group Split-LBI first identifies a budget-constrained sparse support, and the base adaptation optimizer then refines only the selected parameters. On VisDA-C with DeiT-S/16 and SHOT, SDL raises final mean per-class accuracy by 14.29 percentage points over the strongest budget-matched sparse baseline while updating at most 0.0213% of the model parameters per batch. These gains, however, depend on the adaptation objective, and support discovery incurs substantial computational overhead.

Critical strengths
  1. A principled formulation of where to adapt. The paper casts the adaptation support as a variable learned from each incoming target batch under an explicit group budget. Its two-stage design decouples support discovery, performed by Group Split-LBI, from the subsequent parameter update, while preserving the underlying TTA objective. The same formulation extends to fully connected, convolutional, and Transformer parameters through architecture-specific grouping. The result is a coherent framework for studying how limited update capacity should be allocated—one that gives the contribution methodological reach across architectures.

  2. Strong empirical evidence for the value of learned update locations. On VisDA-C with DeiT-S/16 and SHOT, SDL reaches 61.50% final mean per-class accuracy, exceeding the strongest budget-matched sparse baseline by 14.29 percentage points while updating at most 0.0213% of the model parameters per batch (Table 2). It also surpasses candidate-dense adaptation by a further 8.56 points and comes within 1.08 points of full-dense adaptation. Together these results indicate that learning the update support makes a small parameter budget considerably more effective, and they directly support the paper's central claim.

Critical weaknesses
  1. The comparisons do not fully isolate the benefit of support selection. The reported gains are confounded with differences in optimization settings. For instance, the DeiT + SHOT controls use a learning rate of 10−510^{-5}, whereas SDL's second stage runs at 10−210^{-2} on VisDA-C with ρ=0.001\rho=0.001 and also inherits the parameter increment produced by the first stage (Appendices A.4 and A.6). Absent controls that hold these settings fixed, the sizeable improvement cannot be confidently attributed to the quality of the learned support. This weakens the empirical case for the paper's central methodological claim.

  2. Support discovery incurs substantial computational cost in online adaptation. In the Office-31 efficiency benchmark, SDL requires approximately 7.05 seconds per batch, making it 150–166 times slower than the sparse baselines; support discovery accounts for 99.22% of adaptation time (Appendix A.3). Its peak memory allocation also exceeds that of candidate-dense adaptation and of the other sparse methods. These costs considerably undercut the practical value of the small update footprint. Comparisons under matched compute or latency budgets are therefore needed to determine whether the accuracy gains justify the added online processing.

  3. Outdated and limited baselines. The comparison set omits strong work from the past two years; the most recent competitive baseline, COME, is from ICLR two years ago, so the empirical positioning of the method is unclear.

  4. Missing a standard benchmark. The evaluation does not include ImageNet-C, a widely used benchmark in this area, which prevents direct comparison with a large body of prior work.

  5. No released code. No code is provided in the appendix or as supplementary material, which limits reproducibility and makes the reported gains difficult to verify.

Questions for authors
  1. Clarification: Hyperparameter selection. Appendix A.6 states that configurations were selected according to target-domain FO accuracy. Were they chosen on an independent validation set, or on the same target data used to report the final results? Which target labels were available during selection, and did all compared methods receive equivalent tuning?

  2. Evidence needed for assessment: Attribution of the accuracy gains. How much of SDL's improvement can be attributed to support selection, rather than to the first-stage parameter increment or to the different second-stage learning rates? Could you provide ablations or matched optimization controls that isolate these effects, particularly for the main VisDA-C + DeiT + SHOT result?

  3. Evidence needed for assessment: Accuracy under comparable computational budgets. Given the reported 150–166× runtime overhead, does SDL retain its accuracy advantage when methods are given comparable adaptation time or gradient-evaluation budgets per batch? Such a comparison would materially inform the assessment of its practical value for online adaptation.

Reviewer_z1x1OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies which parameters to update in online test-time adaptation (TTA). The proposed method, Sparse Delta Learning (SDL), treats the choice of parameter-update locations as a variable to be learned from the current target batch, subject to an explicit budget on parameter groups. The method first uses Group Split-LBI to couple dense updates with sparse structure and identify the parameter support to be adapted. It then initializes local adaptation with the first-stage updates at the selected locations and performs online updates on the selected parameters. Experiments on two datasets with two TTA methods show only marginal improvements.

Critical strengths

The paper addresses a practically relevant problem. Both the number and the locations of parameters updated during TTA affect GPU memory requirements, runtime, and adaptation performance. Prior TTA methods often update affine parameters by default. Investigating which parameters should be adapted is therefore a worthwhile research direction.

Critical weaknesses
  1. The benefits of the proposed method are limited, and the experimental evidence is insufficient. SDL is integrated with only two TTA methods, SHOT and COME. Moreover, when applied to COME, it fails to outperform randomly selecting parameters for adaptation.

  2. The method lacks a clear justification. The paper formulates the selection of parameter-update locations as a complex dynamical system and uses Group Split-LBI to solve it. This choice is not supported by empirical observations or theoretical arguments, making the approach difficult for readers to understand.

  3. The method has poor runtime efficiency. Appendix A3 shows that, despite updating only a small subset of parameters, its runtime is nearly 100 times that of standard full-parameter adaptation. In particular, 99.22% of the runtime is spent searching for the parameters to update. This substantially limits the method's practical applicability.

Questions for authors
  1. Please explain the effectiveness of the proposed method, particularly why it fails to outperform random parameter selection when applied to COME. Does this indicate that the method has limited applicability?

  2. Please provide a clear justification for the method's design, for example through empirical observations about parameter selection in TTA.

  3. Can the substantial runtime overhead introduced by the method be reduced?

Reviewer_HEBJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents an online test-time domain adaptation method in which the model updates on each batch of target data before predicting on it. The authors argue that which parameters to update is as important as how to update them, and propose sparse delta learning (SDL) to select a subset of parameter groups for updating. Experiments on two benchmarks show that SDL outperforms fixed selectors but lags behind dense update

Critical strengths
  • Within the budget-matched comparison, SDL clearly outperforms other fixed selectors like random, magnitude, and saliency.
  • Honest reporting of the latency drawback and the full COME results.
Critical weaknesses
  • The paper shows SDL is the best selector within a family (selecting a few groups inside a hand-picked region), but the whole family is worse than plain dense adaptation (updating the whole network). Compared to dense update, SDL uses more compute and achieves lower accuracy. The paper never explains why a sparse update is needed in the first place.
  • The hand-picked candidate region is unjustified. That SDL beats candidate-dense and other selection methods within the region says little about its practical value. The authors should at least explain how the region was selected, and ideally compare against alternatives.
Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.