acceptodds
Desk rejected at ICLR 2027

Gradient Coherence Is Not Enough: Separating Influence from Utility in Continual Test-Time Adaptation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Continual test-time adaptation (CTTA) updates a model on unlabeled test data, making it difficult to determine which gradient directions are useful for adaptation. Recent approaches exploit consistency in gradient history to identify structured update directions, based on the intuition that gradients sharing consistent structure provide more reliable adaptation signals. But does a consistent gradient direction actually lead to better predictions? We study this question through a mechanistic analysis of gradient-based CTTA, separating three properties of an update direction: its coherence with past gradients, its influence on model behavior, and its utility for the task. We first establish that gradient coherence is not merely a geometric property: differences in gradient structure propagate through the tracked subspace to actual updates, leading the model toward substantially different parameter and prediction trajectories. Having established that coherent gradient structure can meaningfully influence adaptation, we then ask whether this influence translates into task improvement by comparing candidate directions from identical model states. Across five independently permuted trajectories and 15 frozen states, the more coherent direction in a pair is also the more influential one in 68.8% of comparisons. In contrast, the more coherent direction provides greater supervised improvement in only 47.5% of comparisons, showing no corresponding positive utility-ranking pattern. These results show that gradient coherence captures real influence over adaptation, but influence is not the same as utility: a consistent direction can strongly change the model without being a better direction for improving its predictions.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual test-time adaptation, gradient coherence, gradient geometry, distribution shift, online adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zZQoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents an analysis that the common assumption in continual test-time adaptation that gradient updates aligned with past gradient history provide useful adaptation signals. The authors formalize three separate properties of candidate directions, which are historical coherence, predictive influence measured by prediction change, and supervised utility measured by cross-entropy loss reduction. Using a controlled directional evaluation on identical model states with equal perturbation norms, the authors compare candidate directions under the historical-subspace adaptation mechanism from LCoTTA on ImageNet-C. The experimental results show that coherence consistently ranks predictive influence with around 69% to 81% concordance, but fails to rank supervised utility, achieving only around 48% to 51% concordance across ResNet-50 and ViT models.

Critical strengths
  1. The paper introduces a well-controlled same-state directional evaluation protocol that isolates update direction from gradient norm, optimizer state, and trajectory drift.

  2. The conceptual and empirical separation between predictive influence and supervised utility effectively questions whether gradient coherence actually improves task accuracy.

  3. The experiments carefully account for normalization statistics and demonstrate that measured gradient geometry depends heavily on batch norm computation choices.

Critical weaknesses

While the empirical analysis is carefully executed, the paper has several critical weaknesses regarding its overall contribution and scope:

  1. The paper points out the failure of historical coherence as a proxy for utility, but it does not provide an alternative criterion or a constructive method to select better update directions. I hope to see how the authors' findings can be applied to developing a better CTTA algorithm.

  2. In my understanding, the empirical scope is largely limited to the specific historical-subspace projection mechanism of LCoTTA (please let me know if i misunderstood something here), which makes it unclear whether these conclusions generalize to other alignment-based CTTA algorithms.

  3. Utility is evaluated exclusively through one-step cross-entropy reduction on a local probe, which does not necessarily reflect long-horizon stability or cumulative performance over an extended adaptation stream.

  4. While Appendix E shows that subspace projection substantially shrinks the gradient norm, the paper does not test a simple baseline that only dampens the update norm without altering the gradient direction, leaving open whether the benchmark stability of subspace methods is largely driven by implicit regularization rather than directional subspace filtering.

Questions for authors
  1. Since one-step utility may penalize conservative updates that prevent catastrophic error accumulation over long sequences, do you have experimental evidence showing whether directional coherence correlates with multi-step or long-horizon cumulative accuracy?

  2. Given that historical coherence fails to rank supervised utility, what specific unsupervised proxy signal do you suggest researchers use instead to identify beneficial gradient directions in continual test-time adaptation?

  3. Did you test this same-state directional comparison on other alignment-based CTTA methods beyond the LCoTTA subspace formulation, such as parameter displacement alignment or consensus-based approaches?

  4. In Appendix E.2, you show that the native projection retains only about 41% of the raw gradient norm. Did you check whether a naive baseline that simply scales down the original gradient norm by this factor yields comparable adaptation stability on benchmark trajectories, without relying on the historical subspace at all?

3 comments

Thank you for recognizing the controlled same-state protocol and the separation of predictive influence from utility. Your questions concern long-term performance, the contribution beyond diagnosis, generalization to other methods, and the role of norm reduction. We address each below.

Q1. Does coherence predict long-horizon improvement?

The paper does not establish that relationship. Section 8 explicitly limits directional utility to one-step improvement. A conservative update could benefit later adaptation while producing little immediate improvement; that possibility is compatible with our findings.

The question tested in Section 6 is whether coherence ranks influence and immediate utility similarly when state, perturbation norm, and probe are held fixed:

R50 / Gaussian noiseResult
Coherence–influence concordance68.84%
Coherence–utility concordance47.52%
Paired difference21.32 percentage points
Additional run-level bootstrap: 95% interval for the difference[19.06, 25.10] points
Difference after leaving out each run in turn19.49–22.01 points

The bootstrap uses five run-level summaries and is exploratory. The contrast remains after removing any single run. These results support stronger ranking of influence than immediate improvement, without implying that coherent updates cannot help over longer horizons or that utility ranking is statistically equivalent to random ordering.

Q2. What is the contribution without a replacement proxy?

The contribution is to establish what an existing structural signal predicts under controlled comparisons. For R50, 63.54% of candidate directions improve probe cross-entropy, yet coherence–utility concordance is 47.52%. Useful directions are therefore present, but coherence does not consistently order them by the amount of immediate improvement in this setting.

This yields a concrete implication for method development: an increase in coherence or predictive change is insufficient evidence that a direction-selection rule identifies greater immediate utility. The same-state, equal-norm protocol provides an offline test of that interpretation before it is used to explain an algorithm’s performance. Labels enter only the utility evaluation.

No replacement unsupervised proxy has been validated here. Such a proxy would be an additional algorithmic contribution; its absence does not remove the empirical distinction established by the controlled comparison.

Q3. Does the result extend beyond LCoTTA?

The empirical evidence concerns LCoTTA’s historical-subspace mechanism. Parameter-displacement alignment and consensus-based methods have not been tested. Section 8 states this scope explicitly.

The comparison protocol can be applied to other alignment scores, but that applicability is not evidence that those methods will produce the same result. Likewise, changing the backbone does not establish generalization across method families. The claim is therefore about the tested historical-subspace score, not a universal failure of alignment-based adaptation.

We base the quantitative argument above on R50. The ViT observations concern the reported ToTensor-only input regime; they do not establish replication under checkpoint-prescribed preprocessing.

Q4. Could norm reduction explain the findings?

This is a relevant question for native adaptation trajectories, but its role differs between Sections 5 and 6:

ExperimentMagnitude controlInterpretation
Section 5: projector interventionsProjection changes direction and magnitudeTheir separate contributions to trajectory behavior are unresolved
Section 6: candidate-direction comparisonAll candidates at an anchor have identical perturbation normBetween-candidate norm differences cannot explain the ranking contrast

Thus, the Section 6 result already controls the magnitude difference raised in this concern. It does not establish whether directional filtering or norm reduction explains the benchmark stability of native subspace adaptation.

A norm-only trajectory baseline has not been evaluated. The reported retention of 0.4167 is a median across measured anchors, with a range of 0.0867–0.9938. Scaling every raw gradient by 0.4167 would provide a useful fixed-shrinkage baseline, but would not match the varying norm reduction produced by projection.

We appreciate the questions about how these findings extend to algorithm design and long-term behavior. We ask you to reconsider the soundness assessment based on the controlled ranking result, distinguishing it from the broader mechanisms and method families that remain untested.

Thank you to the authors for the detailed response. The authors' clarifications and additional results for Q1 and Q4 helped me better understand the specific contribution of this paper. However, as the authors also noted and acknowledged in their response, the paper has clear fundamental limitations. The analysis is restricted to the specific mechanism of LCOTTA, and the paper does not provide any alternative criteria or constructive solutions to improve CTTA. In my view, these limitations mean that the current contribution is not yet sufficient for acceptance at ICLR. Therefore, I will maintain my rating as Weak Reject. Personally, I believe the core findings here are interesting. If the authors build on these observations to propose a better method, or generalize their analysis across broader algorithms, the work would have strong novelty for a future top-tier conference submission.

AuthorsOpenReview ↗

Response to reviewer's follow-up

Thank you for considering our response and recognizing that the core findings are interesting.

We understand your reservations about the scope and the absence of a new method. The contribution here is a controlled test showing that historical coherence ranks predictive influence more consistently than immediate improvement. This helps distinguish evidence that an adaptation mechanism changes predictions from evidence that it selects better directions.

A new method or broader evaluation would extend this contribution, but the present finding also has value as an analysis of an existing mechanism.

Since the clarifications on Q1 and Q4 helped explain the contribution, we ask you to reconsider whether the original soundness assessment still reflects the remaining concerns, which your follow-up emphasizes as scope and significance. We would also appreciate reconsideration of the overall rating on that basis.

Reviewer_HQ7YOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper looks at whether gradient coherence leads to useful updates for continual test time adaptation. It finds that normalization changes gradient geometry and adaptation trajectories, and that coherence can predict influence (i.e. how much predictions change from an update) more consistently than utility (defined as a one-step supervised improvement). The paper conducts its experiments with ResNet-50 on ImageNet-C, with some supporting experiments with Vision Transformers.

Critical strengths

I think the work asks a useful question, whether the underlying assumption behind several TTA methods holds. They also have a few interesting findings, specifically that gradient coherence does not necessarily lead to the immediate largest improvement on the task.

Critical weaknesses

Clarity
This paper is very difficult to read and understand. At several points, random notation or ideas are referred to or talked about with essentially no context (e.g. "condition b"); figures use confusing coloring (Figure 1a and 1b use the same colors for different things right next to each other); experiments are introduced without giving proper context for what they are supposed to test and why it makes sense (why did you select the normalization conditions? How does Section 4 relate to the main argument being put forth in the paper?); the analysis simply mentions the numbers, but does not properly explain what their implications are. Overall, I think the paper needs a major rewrite.

Experiments While there is some initial evidence for it, I am not sure there is enough evidence to support the main claim of the paper, that coherence does or does not lead to better adaptation. The evidence shows that at one step, coherence does not necessarily lead to the largest reduction in loss. That does not mean that overall a more coherent subspace does not over the full trajectory reach a better state. There is some acknowledgement of this fact, but the claim about "influence" may be an overclaim.

Questions for authors
  • Why did you use the normalization conditions as the controlled computation change in Section 4?
  • In the appendix you mention you don't use input normalization for ViT. Why is that? That checkpoint was trained with (.5, .5, .5) input mean/sd.
1 comment

Thank you for recognizing the value of the research question and the findings on immediate improvement. Your comments raise three issues: the evidence for influence, the purpose and presentation of the normalization experiments, and ViT preprocessing. We address each below.

1. Evidence for influence and the scope of the utility claim

Your concern about long-term improvement is distinct from the paper’s finding that coherence ranks predictive influence more consistently than immediate utility. Section 8 explicitly states that the one-step comparison does not establish long-horizon accuracy.

The evidence for influence does not rest on one-step loss measurements. Section 5 intervenes on the historical projector and follows the resulting adaptation trajectories:

MeasurementResult
Images with different predictions between branches8.45–10.43%
Net accuracy difference+0.036 to +0.178 percentage points
Disagreements where both predictions are wrong75.9–77.5%

These measurements establish changes in model behavior and explain why those changes need not yield comparable accuracy gains: most disagreements do not change correctness. The suffixes overlap and are not treated as independent replications; the small accuracy differences are not an equivalence claim.

Section 6 separately compares directions from the same state, at equal perturbation norm and on the same probe:

R50 / Gaussian noiseResult
Coherence–influence concordance68.84%
Coherence–utility concordance47.52%
Paired difference21.32 percentage points
Additional run-level bootstrap: 95% interval for the difference[19.06, 25.10] points
Difference after leaving out each run in turn19.49–22.01 points

The additional bootstrap uses five run-level summaries and is exploratory. The leave-one-run-out results show that the contrast is not driven by a single run.

Thus, behavioral influence is measured directly, while immediate utility is evaluated separately. A coherent subspace could still benefit long-term adaptation without contradicting either result. We do not use the one-step comparison to argue otherwise.

2. Why normalization was varied, and how Section 4 supports the argument

Normalization statistics allow gradient computation to be changed while keeping the pre-update weights and input batch fixed. BATCH uses current-batch statistics; FROZEN uses pretrained running statistics.

Section 4 measurementResult
ET–ED coherence gap, BATCH → FROZEN+0.0182 → −0.0222
Mean cosine between paired BATCH/FROZEN gradients0.0110
Mean FROZEN/BATCH gradient-norm ratio2.525

The paired gradients change in direction as well as magnitude, so the effect is not merely rescaling. The ET–ED gap illustrates the dependence of the measured contrast on computation; it does not isolate correctness from adaptation time because the groups occupy different stream regions.

This establishes the role of Section 4: measured geometry depends on gradient computation. Section 5 tests whether differences in tracked geometry affect adaptation, and Section 6 tests whether historical coherence ranks influence and utility similarly. These are separate, connected questions.

To make this connection easier to follow, we will state it before the experiments, explicitly define condition bb as BATCH/FROZEN, correct inconsistent color meanings across panels, and place the interpretation beside each principal result.

3. ViT input normalization

Thank you for raising the preprocessing question. As specified in the appendix, the ViT results were obtained with a ToTensor-only input pipeline, rather than the checkpoint-prescribed normalization with mean and standard deviation both set to (0.5, 0.5, 0.5). Their interpretation is therefore specific to that input regime.

Within each anchor, preprocessing, the restored state, perturbation norm, and evaluation probe were held fixed across candidates. These controls support the comparison within the evaluated regime; they do not establish that the ranking contrast would persist under checkpoint-prescribed preprocessing, which remains to be tested.

The main influence–utility finding does not rely on the ViT extension. The R50 experiments in Sections 4–6 were conducted separately and do not depend on the ViT pipeline. The preprocessing concern limits the interpretation of the ViT extension without, by itself, invalidating the R50 evidence above.

We appreciate the opportunity to clarify these points. We ask you to reconsider the overall recommendation based on the directly measured influence and the controlled R50 ranking results, while accounting for the scope of the ViT observations and the presentation changes specified above.

Reviewer_9sEvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This draft studied whether historical gradient coherence can indicate useful update directions in continual test-time adaptation (CTTA). The authors distinguished gradient coherence, predictive influence, and supervised utility, using the historical-subspace mechanism of LCoTTA as the experimental setting. Specifically, the authors first examined how changes in normalization computation affect gradient geometry, projected gradients, and adaptation trajectories. Then, candidate gradient directions were compared from the same restored model state, with equal perturbation norms and the same evaluation data. The experiments were conducted on ImageNet-C using ResNet-50 and ViT-B/16. The reported results showed a more consistent relationship between coherence and predictive influence than between coherence and one-step supervised improvement in the tested settings.

Critical strengths
  1. The authors carefully controlled the starting model state, perturbation magnitude, and evaluation data when comparing different gradient directions. Labels were only used to evaluate utility in these comparisons. These controls help separate the effect of direction from differences in model states and update magnitudes.

  2. The distinction between predictive influence and supervised utility is interesting. The projector interventions show that substantial changes in predictions can coexist with small net accuracy differences, while the directional comparisons show a stronger relationship between coherence and influence than between coherence and utility. In my view, these observations provide a useful perspective for interpreting historical gradient structure in CTTA.

Critical weaknesses
  1. The connection between the main directional comparison and the native adaptation updates is not sufficiently established. In Section 6, the candidates follow unprojected gradient directions without an optimizer step, while LCoTTA passes projected gradients to the optimizer. I understand that this is a deliberate diagnostic choice, but the implications of the resulting utility rankings for actual adaptation updates are still unclear to me.

  2. The robustness of the main utility-ranking results needs further clarification. The comparison uses one perturbation scale and one fixed evaluation probe at each state. Although per-run results are reported, I did not see sensitivity analyses for these choices or uncertainty estimates for the aggregate concordances. Since these results support the central conclusion, I would like to understand how stable they are under the evaluation choices and across runs.

Questions for authors
  1. Firstly, I want to discuss how the findings in Section 6 relate to the studied projection mechanism. I understand that Eq. (6) deliberately evaluates original gradient directions. As a complementary check, would the influence–utility contrast persist after projection? Could the authors compare −gj/∥gj∥2-g_j/\|g_j\|_2 and −Pgj/∥Pgj∥2-Pg_j/\|Pg_j\|_2, where the latter is defined, using the same restored state, perturbation norm, and probe while retaining the original coherence score CjC_j? Appendix D.3 provides relevant evidence, but uses a different structural score and compares observations from different model states.

  2. How sensitive is the influence–utility contrast to the perturbation magnitude and the choice of probe samples? A small sensitivity analysis would help assess the robustness of the findings. Could the authors also report the typical magnitudes of influence and utility, utility differences between candidates, and the proportion of candidates with positive utility? These summaries would clarify the practical meaning of the rankings beyond their relative ordering.

  3. The trajectory-level summaries already show a consistent contrast between influence and utility ranking. Could the authors additionally provide uncertainty estimates for the aggregate coherence–utility concordance and the paired difference between coherence–influence and coherence–utility concordance, accounting for dependence within trajectories? Given the five runs per setting, the limitations of these estimates should be stated. For ViT, they would remain conditional on the shared warm-up within each target.

3 comments

Thank you for highlighting the controlled design and the distinction between influence and utility. We have analyzed the saved outputs further to answer your questions.

Q1. Connection to native updates and projected directions

The Section 6 comparison is tied to native adaptation in three respects:

ComponentConnection to native adaptation
Reference stateTaken from the adaptation trajectory
Candidate gradientComputed using the native adaptation objective
Perturbation normMatched to the native anchor-update displacement

Applying the original directions at equal norm tests whether historical coherence ranks their predictive influence and immediate utility when state and magnitude are held fixed. This is the question answered by Section 6.

The proposed comparison with normalized projected directions has not been performed. Appendix D.3 does not replace it, since its score and comparison states differ. Section 5 shows that projector interventions affect actual adaptation trajectories; Section 6 tests the ranking of original directions. We do not infer the utility ranking of projected optimizer updates from the latter.

Q2. Probe sensitivity and magnitudes

We resampled the 64 probe batches within each anchor 2,000 times, keeping batch membership fixed and using identical resampling weights across candidates and outcomes.

ResNet-50 / Gaussian noiseResult
Original C–I minus C–U concordance21.32 percentage points
Central 95% range under probe resampling18.70–22.20 points
C–U concordance, first / second probe half48.20% / 47.13%

The contrast persists under these reweightings of the observed probe. Alternative perturbation magnitudes and new probe images have not been evaluated. The tested magnitude is calibrated to the native update, as described above.

The requested outcome summaries are:

SettingMedian IIMedian UUMedian within-anchor ∣Uj−Uk∣\lvert U_j-U_k\rvertU>0U>0
R50 / GN2.6812.6640.1563.54%
ViT / GN†15.5734.43312.8053.54%
ViT / MB†22.1651.67329.1356.46%

The three magnitude columns are in units of 10−510^{-5}, using natural-log KL and CE differences averaged over images. GN/MB denote Gaussian noise/motion blur. For R50, the utility interquartile range is [−12.62,46.16]×10−5[-12.62,46.16]\times10^{-5}.

These are small local changes. However, 63.54% of R50 candidates improve CE, so weak utility ranking does not mean that all candidate directions are harmful. The distinction is between producing improvement and ranking its magnitude.

Q3. Uncertainty across runs

We average the three anchors within each run and jointly resample the five run-level C–I/C–U summaries. Candidate pairs and anchors are not treated as independent replications.

SettingC–U concordance [95% bootstrap interval]C–I minus C–U, percentage points [95% bootstrap interval]
R50 / GN47.52% [44.27, 50.36]21.32 [19.06, 25.10]
ViT / GN†51.53% [46.77, 55.78]29.88 [24.01, 35.66]
ViT / MB†51.38% [48.06, 55.55]30.09 [27.38, 32.59]

These exploratory percentile intervals enumerate all 555^5 empirical bootstrap resamples. With only five runs, their precision is limited. For R50, leaving out each run in turn gives a difference of 19.49–22.01 points: the contrast is not driven by a single run. These results do not establish equivalence of C–U ranking to random ordering.

†ViT results describe the original input regime, which omitted checkpoint-prescribed normalization, and remain conditional on the shared warm-up within each target. They do not establish replication under standard ViT preprocessing. The R50 results are separate from this issue.

The additional analyses support the distinction identified in your review: coherence ranks predictive influence more consistently than immediate utility under the tested controls. We ask you to reconsider your assessment in light of these results and clarifications.

Reviewer_9sEvOpenReview ↗

Follow-up after author rebuttal

Thanks for the detailed response.

The additional experimental results are helpful and address much of my concern about the stability of the reported concordance differences. I also understand the authors' clarification regarding Q1. Section 6 is intended to study original gradient directions under controlled conditions, rather than the projected updates used in actual adaptation. I agree that this is a valid diagnostic setup.

My remaining concern is mainly about the scope of the conclusions. The results support a distinction between predictive influence and immediate supervised utility, but the connection to actual adaptation updates and longer-term performance remains unclear.

However, I am still open to discussions with the authors or other reviewers.

AuthorsOpenReview ↗

Response to reviewer's follow-up

Thank you for confirming that the diagnostic setup is valid and that the additional analyses address much of your concern about the stability of the results.

We would like to clarify the connection to adaptation. Section 5 changes the historical projector during adaptation and measures the resulting trajectories. Predictions differ on 8.45-10.43% of suffix images, while net accuracy differences are +0.036 to +0.178 percentage points. This shows that the projector affects model behavior, but the resulting prediction changes do not necessarily translate into comparable accuracy gains.

Section 6 then examines candidate directions at states reached during adaptation, using the native objective and a perturbation norm matched to the native anchor update. These choices connect the diagnostic to the adaptation process while allowing direction to be compared at a fixed state and magnitude. The result concerns original directions; it does not establish the utility ranking of projected optimizer updates.

Together, these experiments show why influence and utility need separate evaluation. A structural signal can be associated with substantial predictive change without ranking immediate improvement equally well. This matters when interpreting why an adaptation mechanism works: evidence that it changes the model is not sufficient evidence that it identifies better update directions.

Coherent updates may still help over longer horizons. Our finding is compatible with that possibility, and Section 8 states this limit. The contribution is the measured distinction under controlled conditions, rather than a claim against the long-term value of coherence.

We hope this clarifies what the findings establish and why they matter, and ask you to reconsider your rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.