- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents an analysis that the common assumption in continual test-time adaptation that gradient updates aligned with past gradient history provide useful adaptation signals. The authors formalize three separate properties of candidate directions, which are historical coherence, predictive influence measured by prediction change, and supervised utility measured by cross-entropy loss reduction. Using a controlled directional evaluation on identical model states with equal perturbation norms, the authors compare candidate directions under the historical-subspace adaptation mechanism from LCoTTA on ImageNet-C. The experimental results show that coherence consistently ranks predictive influence with around 69% to 81% concordance, but fails to rank supervised utility, achieving only around 48% to 51% concordance across ResNet-50 and ViT models.
Critical strengths
-
The paper introduces a well-controlled same-state directional evaluation protocol that isolates update direction from gradient norm, optimizer state, and trajectory drift.
-
The conceptual and empirical separation between predictive influence and supervised utility effectively questions whether gradient coherence actually improves task accuracy.
-
The experiments carefully account for normalization statistics and demonstrate that measured gradient geometry depends heavily on batch norm computation choices.
Critical weaknesses
While the empirical analysis is carefully executed, the paper has several critical weaknesses regarding its overall contribution and scope:
-
The paper points out the failure of historical coherence as a proxy for utility, but it does not provide an alternative criterion or a constructive method to select better update directions. I hope to see how the authors' findings can be applied to developing a better CTTA algorithm.
-
In my understanding, the empirical scope is largely limited to the specific historical-subspace projection mechanism of LCoTTA (please let me know if i misunderstood something here), which makes it unclear whether these conclusions generalize to other alignment-based CTTA algorithms.
-
Utility is evaluated exclusively through one-step cross-entropy reduction on a local probe, which does not necessarily reflect long-horizon stability or cumulative performance over an extended adaptation stream.
-
While Appendix E shows that subspace projection substantially shrinks the gradient norm, the paper does not test a simple baseline that only dampens the update norm without altering the gradient direction, leaving open whether the benchmark stability of subspace methods is largely driven by implicit regularization rather than directional subspace filtering.
Questions for authors
-
Since one-step utility may penalize conservative updates that prevent catastrophic error accumulation over long sequences, do you have experimental evidence showing whether directional coherence correlates with multi-step or long-horizon cumulative accuracy?
-
Given that historical coherence fails to rank supervised utility, what specific unsupervised proxy signal do you suggest researchers use instead to identify beneficial gradient directions in continual test-time adaptation?
-
Did you test this same-state directional comparison on other alignment-based CTTA methods beyond the LCoTTA subspace formulation, such as parameter displacement alignment or consensus-based approaches?
-
In Appendix E.2, you show that the native projection retains only about 41% of the raw gradient norm. Did you check whether a naive baseline that simply scales down the original gradient norm by this factor yields comparable adaptation stability on benchmark trajectories, without relying on the historical subspace at all?
3 comments
Thank you to the authors for the detailed response. The authors' clarifications and additional results for Q1 and Q4 helped me better understand the specific contribution of this paper. However, as the authors also noted and acknowledged in their response, the paper has clear fundamental limitations. The analysis is restricted to the specific mechanism of LCOTTA, and the paper does not provide any alternative criteria or constructive solutions to improve CTTA. In my view, these limitations mean that the current contribution is not yet sufficient for acceptance at ICLR. Therefore, I will maintain my rating as Weak Reject. Personally, I believe the core findings here are interesting. If the authors build on these observations to propose a better method, or generalize their analysis across broader algorithms, the work would have strong novelty for a future top-tier conference submission.
Response to reviewer's follow-up
Thank you for considering our response and recognizing that the core findings are interesting.
We understand your reservations about the scope and the absence of a new method. The contribution here is a controlled test showing that historical coherence ranks predictive influence more consistently than immediate improvement. This helps distinguish evidence that an adaptation mechanism changes predictions from evidence that it selects better directions.
A new method or broader evaluation would extend this contribution, but the present finding also has value as an analysis of an existing mechanism.
Since the clarifications on Q1 and Q4 helped explain the contribution, we ask you to reconsider whether the original soundness assessment still reflects the remaining concerns, which your follow-up emphasizes as scope and significance. We would also appreciate reconsideration of the overall rating on that basis.
Thank you for recognizing the controlled same-state protocol and the separation of predictive influence from utility. Your questions concern long-term performance, the contribution beyond diagnosis, generalization to other methods, and the role of norm reduction. We address each below.
Q1. Does coherence predict long-horizon improvement?
The paper does not establish that relationship. Section 8 explicitly limits directional utility to one-step improvement. A conservative update could benefit later adaptation while producing little immediate improvement; that possibility is compatible with our findings.
The question tested in Section 6 is whether coherence ranks influence and immediate utility similarly when state, perturbation norm, and probe are held fixed:
The bootstrap uses five run-level summaries and is exploratory. The contrast remains after removing any single run. These results support stronger ranking of influence than immediate improvement, without implying that coherent updates cannot help over longer horizons or that utility ranking is statistically equivalent to random ordering.
Q2. What is the contribution without a replacement proxy?
The contribution is to establish what an existing structural signal predicts under controlled comparisons. For R50, 63.54% of candidate directions improve probe cross-entropy, yet coherence–utility concordance is 47.52%. Useful directions are therefore present, but coherence does not consistently order them by the amount of immediate improvement in this setting.
This yields a concrete implication for method development: an increase in coherence or predictive change is insufficient evidence that a direction-selection rule identifies greater immediate utility. The same-state, equal-norm protocol provides an offline test of that interpretation before it is used to explain an algorithm’s performance. Labels enter only the utility evaluation.
No replacement unsupervised proxy has been validated here. Such a proxy would be an additional algorithmic contribution; its absence does not remove the empirical distinction established by the controlled comparison.
Q3. Does the result extend beyond LCoTTA?
The empirical evidence concerns LCoTTA’s historical-subspace mechanism. Parameter-displacement alignment and consensus-based methods have not been tested. Section 8 states this scope explicitly.
The comparison protocol can be applied to other alignment scores, but that applicability is not evidence that those methods will produce the same result. Likewise, changing the backbone does not establish generalization across method families. The claim is therefore about the tested historical-subspace score, not a universal failure of alignment-based adaptation.
We base the quantitative argument above on R50. The ViT observations concern the reported ToTensor-only input regime; they do not establish replication under checkpoint-prescribed preprocessing.
Q4. Could norm reduction explain the findings?
This is a relevant question for native adaptation trajectories, but its role differs between Sections 5 and 6:
Thus, the Section 6 result already controls the magnitude difference raised in this concern. It does not establish whether directional filtering or norm reduction explains the benchmark stability of native subspace adaptation.
A norm-only trajectory baseline has not been evaluated. The reported retention of 0.4167 is a median across measured anchors, with a range of 0.0867–0.9938. Scaling every raw gradient by 0.4167 would provide a useful fixed-shrinkage baseline, but would not match the varying norm reduction produced by projection.
We appreciate the questions about how these findings extend to algorithm design and long-term behavior. We ask you to reconsider the soundness assessment based on the controlled ranking result, distinguishing it from the broader mechanisms and method families that remain untested.