acceptodds
Under review as a conference paper at ICLR 2027

When and Why Can Test-Time Adaptation Succeed?

18 views

Abstract

Test-time adaptation (TTA) modifies deployed models using only unlabeled target data, yet a fundamental theoretical question remains open: when can unlabeled observations determine a better target predictor, and why can entropy-based up- dates correct errors rather than merely sharpen them? We identify recoverable semantic structure as a common principle underlying both questions. First, we characterize conditions under which target semantics are recoverable from unla- beled data: semantic components must be recoverable and oriented by the source model, with within-semantic support dominating cross-semantic leakage. Under these conditions, we establish pointwise component-resolution learnability in the probably approximately correct (PAC) sense and derive finite-sample guarantees for exact component-level semantic identification. We then ask how entropy-based TTA algorithms exploit this information. For finite-width two-layer networks with a fixed output head, we show that entropy on an isolated prediction carries no cor- rectness signal; correction instead arises from parameter sharing, which propagates source-oriented evidence through a parameter-gradient coupling kernel. Under a positive support-over-leakage condition controlling finite-width dynamics, we prove early-stage target-risk reduction. Guided by the same principle, we finally introduce PRojected Invariant Spectral Matching (PRISM), an optimization-free method that explicitly preserves shared task-relevant evidence while suppressing view-specific nuisance in the classifier-visible subspace. Across five distribution- shift benchmarks, PRISM consistently outperforms strong backpropagation-based and backpropagation-free TTA baselines. Code is available here.

Primary area:
learning theory
Keywords:
test-time adaptation, distribution shift, PAC learnability, entropy minimization, learning theory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YVvhOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies two fundamental questions of test-time adaptation (TTA): when unlabeled target data suffice to determine a better predictor, and why entropy minimization can correct errors rather than merely reinforce them. The contributions consist of three parts. (1) The paper proposes the principle of "recoverable semantic structure" and establishes a PAC learnability framework for TTA with finite-sample guarantees. (2) It proves that entropy itself carries no correctness signal, and that error correction instead arises from parameter sharing, which propagates source-oriented evidence across samples through a coupling kernel. (3) Guided by the same principle, the paper designs PRISM, an optimization-free method that fuses multiple deterministic views of each target sample via spectral matching in the classifier-visible subspace. PRISM achieves 69.36% average accuracy across five benchmarks, clearly surpassing the strongest baseline at 63.33%.

Critical strengths
  1. The large-scale structural audit and the intervention experiments directly verify the theoretical assumptions, effectively closing the loop between theory and experiment.
  2. PRISM requires no backpropagation, yet outperforms the strongest baseline by about 6 points on average across five benchmarks, and works consistently over four different backbones.
Critical weaknesses
  1. The scope of the theory (two-layer ReLU networks with a component-constant hypothesis class) is far from the ViT/ResNet architectures used in the evaluation, so it serves more as a design heuristic than a rigorous guarantee for practical methods.
  2. The structural audit is conducted only on CIFAR-C, and the ImageNet-level benchmarks lack equivalent verification, which weakens the link between the theory and the observed performance there. In addition, the main table reports no paired significance tests.
Questions for authors
  1. How sensitive are the results of PRISM to the choice of augmentation view types and the number of views?
  2. When labels vary gradually over a semantic continuum, so that no pure partition exists, does the PAC framework still apply?
5 comments
AuthorsOpenReview ↗

Response to Reviewer YVvh (1/3)

We thank you for your positive assessment and for noting that the structural audit and intervention experiments close the loop between theory and experiment. Below we address the scope of the theory, its verification on the ImageNet benchmarks, and your two questions.

Weakness 1: Scope of the theory relative to the ViT and ResNet models in the evaluation

Response: We agree for Theorem 4.1, which gives a design principle for ViT and ResNet, but the PAC result places no restriction on the architecture, and for ViT-B PRISM, the empirical bound of its output-level Corollary E.2 is below one in 16 of 17 and 12 of 15 positive-margin domains, at 512 and 4,096 samples.

Theorem 4.1 analyzes binary two-layer ReLU networks with a fixed output head, and we have revised contribution (2) and Section 4 to state this scope.

The PAC result of Section 3 places no restriction on the architecture, since the reachable class HsΓ\mathcal H^\Gamma_s contains whatever the protocol can reach in any network. We agree that it is component-constant in one respect. Both the comparator and the output of the recover-identify-project learner are constant on resolved components, and Theorem A.19 shows why the comparator cannot be relaxed, because unlabeled data cannot tell which subregion of a component to relabel. Corollary E.2 removes this restriction for evaluation. It bounds the error of any predictor, from any architecture and not necessarily constant on components, through the realization gap e(p)e(p) of its posterior pp,

L_Q(\arg\max p)\le \min\Big\\{1,\ \zeta+\alpha_{\rm tot}+\frac{2e(p)}{\Delta_{\rm id}}\Big\\},

whenever the identification margin Δid\Delta_{\rm id} is positive and the corollary's structural conditions hold on a fixed component graph. Here ζ\zeta is the unresolved mass and αtot\alpha_{\rm tot} is the within-component minority-label mass. Our response to Weakness 2 reports how often these conditions hold for ViT-B on ImageNet-C, ImageNet-R, and ImageNet-Sketch. The module-specific results of Appendix E.4 are likewise stated in the classifier-visible feature space of a frozen network with a linear head, so they apply to PRISM on ViT and ResNet whenever their conditions on the feature geometry hold.

To test whether the mechanism of Theorem 4.1 appears in practical networks, we apply a Tent-style entropy update to the normalization parameters of ResNet-50 and ViT-B on severity-5 ImageNet-C, at 720 receiver positions per backbone. For each initially wrong prediction, we compute the first-order change of its true-class margin from the entropy gradients of the other samples alone (cross-sample) and from the full batch update. Labels are used only to compute margins after the update. As shown in the table below, the other samples push the true-class margin up for 25.4% of wrong predictions on ResNet-50 and 42.9% on ViT-B under SGD, and removing each receiver's own term raises this fraction in every setting.

BackboneInitially wrongSGD, cross-sample > 0SGD, full update > 0Adam, cross-sample > 0Adam, full update > 0
ResNet-50457116 (25.4%)103 (22.5%)64 (14.0%)58 (12.7%)
ViT-B343147 (42.9%)132 (38.5%)140 (40.8%)128 (37.3%)

In these networks, we observe a positive first-order cross-sample contribution to the true-class margin for a subset of initially incorrect predictions. For the remaining errors, the first-order contribution is non-positive, because normalization-parameter coupling, unlike the nonnegative fixed-head kernel of Section 4, can take either sign. We present these results as first-order evidence for the mechanism, not as a guarantee, and have included them in Section 4 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer YVvh (2/3)

Weakness 2: ImageNet-level verification of the theory, and paired significance tests

Response: Appendix E.5 already audits the identification margin on all 17 ImageNet-C, ImageNet-R, and ImageNet-Sketch domains, we now add a held-out test of the semantic transfer of label-free partitions, and paired tests show that PRISM's gains over NEO-5V are significant on all three corruption benchmarks after Holm correction.

The four-condition audit of Section 3.3 remains CIFAR-C only, and the revised paper states this.

Identification margin. Appendix E.5 already audits 47 ViT-B target domains with the component graph and source anchors held fixed, 15 each from CIFAR-10-C, CIFAR-100-C, and ImageNet-C, plus ImageNet-R and ImageNet-Sketch. At 512 samples, 8 of the 17 ImageNet domains (15 ImageNet-C corruptions, ImageNet-R, and ImageNet-Sketch) have a strictly positive empirical margin, and PRISM's empirical bound is below one in 7 of these 8. Over all 47 domains, the margin is positive in 17 at 512 samples and 15 at 4,096, with PRISM's empirical bound below one in 16/17 and 12/15. On the same graphs, PRISM improves mean accuracy over the source by 8.90 pp (512) and 8.95 pp (4,096).

Held-out semantic transfer. We fix the partition rule in advance, fit it on 4,096 unlabeled images per domain, and apply it to a disjoint set of 4,096 images. Only after both sets of assignments are frozen do we use labels. Labels of the fit set name each component by its majority class, and labels of the held-out set score whether the name is correct. As shown in the table below, label-free components transfer their semantics to new images, but resolution is partial, and 30% to 55% of held-out images are both resolved and correctly named.

BenchmarkHeld-out images resolvedName correct, among resolvedResolved and correct, among all
ImageNet-C (15 corruptions)41.8%72.9%30.5%
ImageNet-R67.4%81.2%54.7%
ImageNet-Sketch57.2%60.4%34.6%

Paired significance. We compare PRISM with NEO-5V, which receives the same five views and averages its view-specific corrected posteriors. Predictions are paired by image and stream order, the bootstrap resamples groups defined by corruption and shared underlying image, and Holm correction is applied across the five benchmarks. As shown in the table below, the three corruption benchmarks are significant after correction. ImageNet-R and ImageNet-Sketch have positive estimates and unadjusted intervals that exclude zero, but they do not reach significance after Holm correction, and we report them as such. The corruption benchmarks pool all five severities, so their estimates are smaller than the severity-5 full-stream gaps (+3.98, +4.92, and +3.69 pp).

Benchmark and protocolPRISM − NEO-5V (pp)95% CI (pp)Holm-adjusted pp
CIFAR-10-C, full streams, all five severities+2.83[1.34, 4.32]0.00079
CIFAR-100-C, full streams, all five severities+3.80[2.00, 5.60]0.00018
ImageNet-C, full streams, all five severities+3.31[1.47, 5.15]0.00129
ImageNet-R, one order of 512 images+4.10[0.43, 7.78]0.0574
ImageNet-Sketch, one order of 512 images+5.27[0.50, 10.05]0.0574

We have included the margin coverage, the semantic transfer results, and this table in Section 5 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer YVvh (3/3)

Question 1: Sensitivity of PRISM to the type and number of views

Response: PRISM is insensitive to the type of views, gains most from the second view, and keeps its advantage over NEO when both receive the same views.

For view type, Table 9 replaces one transformation at a time with a 95% crop, Gaussian blur at radius 1.5, box blur, or a sharpness adjustment, and the five-benchmark macro accuracy changes by −0.21 to +0.04 pp from 69.36%.

For view count, we evaluate all 31 nonempty subsets of the five default views and recompute the online statistics for each. As shown in the table below, which averages the macro accuracy over all subsets of each size, the second view adds 3.41 pp, and later views add 1.09, 0.63, and 0.31 pp.

Number of views12345
Five-benchmark macro accuracy63.92%67.33%68.42%69.05%69.36%

In a separate control, every method receives the same four views (the original image, a 90% center crop, a horizontal flip, and the flipped crop) on severity-5 ImageNet-C with 512 images per corruption. PRISM outperforms NEO with the same views by 1.46 to 4.43 pp across ResNet-50, ViT-S, ViT-B, and ViT-L, so its advantage does not come from using more views.

Question 2: Whether the PAC framework applies when labels vary over a semantic continuum

Response: Yes, the framework applies without pure components, since impurity enters the bound through the within-component minority-label mass, and the bound loosens as the overlap grows.

Theorem 3.2 does not require pure components. It requires a finite partition whose components are near-pure, recoverable from unlabeled data, and oriented by the source, together with a positive identification margin and protocol stability. Impurity enters the approximation term ζ+2αtot\zeta+2\alpha_{\rm tot}, and the guarantee is relative to the component-resolution oracle.

A simple example shows what the bound gives when no pure partition exists. Let X∼Unif[0,1]X\sim{\rm Unif}[0,1] and Pr⁡(Y=1∣X=x)=σ(8(2x−1))\Pr(Y=1\mid X=x)=\sigma(8(2x-1)), so both labels have positive probability everywhere. Suppose the source score thresholds at x=1/2x=1/2 and induces the components [0,1/2)[0,1/2) and [1/2,1][1/2,1], with positive within-side affinity, zero cross-side affinity, and source-anchor mass only on the majority label of each side. Then ζ=0\zeta=0, the margin is positive, and the component-majority predictor, which here is the Bayes classifier, has risk αtot=(log⁡2−log⁡(1+e−8))/8≈8.66\alpha_{\rm tot}=\big(\log 2-\log(1+e^{-8})\big)/8\approx 8.66\\%. For a protocol that can realize this oracle, the bound is 3αtot≈25.983\alpha_{\rm tot}\approx 25.98\\% plus finite-sample terms.

The bound loosens as the overlap grows. Finer components lower αtot\alpha_{\rm tot} but shrink the minimum component mass and raise the cost of recovery, and when no finite partition balances impurity, recoverability, and orientation, the bound becomes vacuous. The comparator cannot be strengthened in general. Theorem A.19 constructs two worlds with the same unlabeled observations and source information, in which a protocol that can relabel a subregion inside a component admits a zero-risk predictor in each world, but the two predictors disagree on that subregion. No method can therefore match the unrestricted reachable oracle uniformly over these worlds. We have added the example and this discussion to Appendix A.7, referenced after Theorem 3.2.

In conclusion, we hope our responses resolve your concerns:

  • Theorem 4.1 is now scoped to its setting, while the PAC result places no restriction on the architecture and Corollary E.2 bounds any predictor. In ResNet-50 and ViT-B, the SGD cross-sample entropy field contributes positively to the true-class margin, to first order, for 25% to 43% of errors (W1).
  • On the three ImageNet benchmarks, we report margin coverage and held-out semantic transfer, and PRISM's gains over NEO-5V are significant on all three corruption benchmarks (W2).
  • PRISM is insensitive to the view type, gains most from the second view, and outperforms NEO under equal views (Q1).
  • The framework covers semantic continua through the impurity term, as a worked example shows (Q2).

We sincerely thank you for your time and effort in reviewing our paper, and we are happy to address any follow-up questions.

Reviewer_GYSmOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates when and why test-time adaptation can succeed using unlabeled target data. It develops a PAC learnability framework for recoverable semantic structures and analyzes how parameter sharing enables entropy minimization to correct prediction errors. Based on these theoretical insights, the authors propose PRISM, an optimization-free TTA method that achieves strong performance across five distribution-shift benchmarks.

Critical strengths

The paper investigates a potential underlying principle of TTA, offering useful theoretical perspectives for understanding existing multi-view and parameter-optimization-based TTA methods.

Critical weaknesses
  1. Theorem 4.1 analyzes entropy minimization in two-layer networks, but PRISM is an optimization-free method. The theoretical analysis does not directly establish PRISM's effectiveness.
  2. The empirical validation of A2 mainly measures clustering stability, which does not guarantee semantic recoverability. Moreover, some theoretical audit configurations were selected using label-based diagnostics.
  3. Agreement across augmented views does not necessarily indicate semantic correctness, as multiple views may share the same prediction errors. The proposed spectral weighting and JS-based fusion may reinforce such errors.
  4. PRISM's feature centering and covariance correction may remove discriminative information, especially under class-prior shifts. Their robustness under more general target distributions remains insufficiently validated.
Questions for authors
  1. How does Theorem 4.1 specifically justify the design of PRISM, given that PRISM does not perform entropy minimization or gradient updates?
  2. Can the authors provide stronger evidence for Assumption A2 beyond clustering stability? In particular, how can semantic recoverability be established without relying on label-informed configuration selection?
4 comments
AuthorsOpenReview ↗

Response to Reviewer GYSm (1/4)

Thank you for noting that our framework offers useful perspectives on both multi-view and optimization-based TTA. Your questions on how Theorem 4.1 relates to PRISM and when multi-view statistics can mislead helped us state this link precisely and measure its failure modes. We address each point below.

Weakness 1 & Question 1: How Theorem 4.1 justifies PRISM, which performs no entropy minimization or gradient update

Response: Theorem 4.1 motivates PRISM's design but does not by itself establish its effectiveness. In the theorem's setting, errors are corrected by cross-sample propagation of source-oriented evidence, not by the entropy objective. PRISM keeps shared cross-sample statistics and drops the entropy term. The guarantee that applies to PRISM's output is Corollary E.2, and Appendix E.4 states when PRISM's operations tighten it.

Section 4 separates three roles: entropy amplifies confidence, source-oriented cores supply correctness, and parameter sharing supplies propagation. Correction requires the coupling field from correctly oriented samples (support) to dominate the field from oppositely oriented or uncertain samples (leakage).

Element of Theorem 4.1Role in the theoremCounterpart in PRISM
Entropy force on a sample's own predictionAmplifies confidence, carries no correctness signalOmitted, as PRISM has no entropy objective
Parameter sharing (coupling kernel)Propagates evidence from other samples to each predictionStream-shared statistics (centering, covariance transport) apply the same stream-estimated correction to each sample
Fixed output headOnly classifier-visible changes affect predictionsAll corrections act in the classifier-visible subspace, with ker⁡Wc\ker W_c removed
Support over leakage, Δcorr(Λ)>0\Delta_{\rm corr}(\Lambda)>0Support must dominate leakage and within-component variationCorrectly oriented evidence is preserved or strengthened under the conditions of Appendix E.4

Appendix E.4 models target features as a source-oriented latent representation distorted by a view offset, a shared anisotropic distortion, and view noise, with a frozen head that classifies the latent representation with a positive margin (Eq. 123). At the first level, Propositions E.5–E.8 control the feature restoration error. View centering removes exactly the classifier-visible offset error when the estimated mean is a nuisance offset (Proposition E.5). Total-covariance transport, with PRISM's shrinkage and damping, does not increase the restoration error when the latent covariance is isotropic and the distortion preserves source coordinates (Proposition E.6), and Example E.7 shows that it can correct a genuine decision error. Within-orbit suppression reduces this error when the nuisance it removes outweighs the signal it attenuates (Proposition E.8). At the second level, a smaller restoration error strictly reduces the realization gap only below the explicit threshold of Theorem E.9 (Eq. 140). Spectral and JS fusion lower the gap when the signal shared across views is correctly oriented (Proposition E.10) and the spectral and posterior perturbations are sufficiently small (Proposition E.11).

These conditions are needed because PRISM's operations do not control leakage by themselves. Within-orbit suppression acts on variation among views of one image, not on cross-sample leakage, and neither spectral agreement nor JS confidence implies correctness, since a shared wrong factor can produce a large eigengap. Our responses to Weaknesses 3 and 4 measure what happens when these conditions fail.

Two measurements support this reading. First, the omitted term does not help. In a Tent-style update of ResNet-50 and ViT-B on ImageNet-C, removing each sample's own entropy term raises the fraction of initially wrong predictions with a positive first-order true-class margin contribution, from 22.5% to 25.4% and from 38.5% to 42.9%. Second, the retained components account for PRISM's gains. In Table 2(a), the shared-statistics steps add 5.38 pp macro accuracy over uncorrected five-view fusion, and the suppression and fusion steps add a further 1.60 pp.

AuthorsOpenReview ↗

Response to Reviewer GYSm (2/4)

Theorem 4.1 does not by itself guarantee PRISM's accuracy, since its dynamics concern gradient updates. The guarantee that applies to PRISM's output is Corollary E.2, which holds for any predictor, whatever its architecture and however its posterior is produced. When the identification margin Δid\Delta_{\rm id} is positive, the correct component labeling minimizes the convex relaxation JTVJ_{\rm TV} of the graph-anchor objective, and Theorem E.1 bounds the soft mismatch of the component-averaged posterior by e(p)/Δide(p)/\Delta_{\rm id}. This gives

L_Q(\arg\max p) \leq \min\left\\{1,\\; \zeta+\alpha_{\mathrm{tot}}+\frac{2e(p)}{\Delta_{\mathrm{id}}}\right\\}.

where ζ\zeta is the unresolved mass, αtot\alpha_{\rm tot} is the within-component minority-label mass, and the realization gap e(p)=JTV(Π(p))−min⁡ΠJTV(Π)e(p)=J_{\rm TV}(\Pi(p))-\min_\Pi J_{\rm TV}(\Pi) measures how far the component-averaged posterior Π(p)\Pi(p) is from that correct labeling. The first two terms are fixed by the target distribution and the partition, so a predictor tightens the bound only by lowering e(p)e(p).

We measure this on 47 ViT-B target domains across the five benchmarks, with the component graph and source anchors held fixed (Appendix E.5). Raw5 averages the same five views with equal weights and no correction.

Samples per domainSourceRaw5PRISM
5125.694.802.03
4,0968.386.653.39

Mean realization gap e(p)e(p), ×10⁻³.

On these graphs, PRISM improves mean accuracy over the source by 8.90 pp at 512 samples and 8.95 pp at 4,096, and its gap is lower than Raw5's in 42/47 and 45/47 domains. The empirical identification margin is positive in 17/47 and 15/47 domains, and in those domains PRISM's empirical bound is below one in 16/17 and 12/15, compared with 2/17 and 4/15 for the source model. These post-hoc results show that PRISM moves its posterior toward the identified labeling on a fixed graph, and that the bound becomes informative where the margin condition holds. They do not cover the domains without a positive margin.

We have added this correspondence, the conditions of Appendix E.4, and these measurements to Section 5 of the revised paper, with the full correspondence table in Appendix G.14.

AuthorsOpenReview ↗

Response to Reviewer GYSm (3/4)

Weakness 2 & Question 2: Evidence for A2 beyond clustering stability, and label-informed configuration selection

Response: We agree that stability alone is insufficient, and in two analyses where the partition rule is fixed before any label is used, the components pass the post hoc semantic check in 87/95 and 89/95 CIFAR-C environments and keep their class identity for 60% to 81% of resolved held-out images on ImageNet-C, ImageNet-R, and ImageNet-Sketch.

Our assumptions treat stability and semantics as separate requirements. A2 only requires that the component partition be recoverable from unlabeled data. That the recovered components carry the correct labels is required by near-purity and source orientation in Assumption 3.1, and this needs its own test, independent of labels used for configuration selection.

We agree that the audit in Section 3.3 and Appendix B.1 uses label-based diagnostics to choose dataset-level configurations, as Section 3.3 states. Appendix B.6 measures the effect of this choice. Selecting the configuration with each corruption family held out gives joint satisfaction in 87/95 and 78/95 environments, compared with 86/95 and 78/95 for the reported configurations. Since the candidate set was still developed on these benchmarks, Section 3.3 now pairs this audit with the two analyses below, which fix the partition rule before any label is used and use labels only to score the result.

Merge-gap audit (Appendix B.4). The merge scale is set by an unlabeled persistence gap in a kernel built from source outputs. All 95 environments of each CIFAR-C benchmark pass the geometric stability check, and the semantic alignment check, computed afterwards, passes in 87/95 environments on CIFAR-10-C and 89/95 on CIFAR-100-C.

Held-out semantic transfer on ImageNet-C, ImageNet-R, and ImageNet-Sketch (new). We fix the partition rule of Appendix E.5 in advance, fit it on 4,096 unlabeled images per domain, and apply it to a disjoint set of 4,096 images. Only after both sets of assignments are frozen do we use labels. Labels of the fit set name each component by its majority class, and labels of the held-out set score whether the name is correct. As shown in the table below, when a held-out image falls in a resolved component, the class name learned on the fit set is correct for 73% of images on ImageNet-C, 81% on ImageNet-R, and 60% on ImageNet-Sketch.

BenchmarkHeld-out images in a resolved componentName correct, among resolvedResolved and correct, among all
ImageNet-C (15 corruptions)25,681/61,440 (41.8%)18,732/25,681 (72.9%)30.5%
ImageNet-R2,759/4,096 (67.4%)2,241/2,759 (81.2%)54.7%
ImageNet-Sketch2,343/4,096 (57.2%)1,416/2,343 (60.4%)34.6%

Resolution is partial, so 30% to 55% of all held-out images are both resolved and correctly named. Components formed without labels are therefore semantically coherent, and their class identity carries over to new images on all three benchmarks, within the limits of partial coverage. We have included both the coverage and the accuracy in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer GYSm (4/4)

Weakness 3: Shared errors across views, and whether spectral and JS weighting reinforce them

Response: The weighting does reinforce errors shared by all views, but only in confidence, since no weighting can change a prediction that every view makes, and where views disagree its net effect is small and positive.

Fusion helps only when the signal shared across views points to the correct class (Proposition E.10), and Appendix G.6 shows that a shared wrong factor can produce a large eigengap. To measure the effect, we hold PRISM's corrected ImageNet-C posteriors fixed and vary only the fusion weights.

When all five views rank the same wrong class first, a nonnegative average keeps that class on top. This happens at 2,537 of 23,040 positions (11.01%). There the mean probability of the shared wrong class rises from 6.23% under equal weights to 6.35% under spectral and JS weights, and by 2.11 pp on CIFAR-100-C.

When views disagree, the weighting can change the prediction. At positions where at least three views share a wrong class and at least one view is correct, joint weighting corrects 62 predictions and introduces 47 errors, while JS weighting alone is slightly harmful. Over all samples, the two weighting stages add 0.75 pp macro accuracy over equal fusion (68.61% to 69.36%, Table 24).

Fusion weightsWrong → correctCorrect → wrongNet
Spectral only3731+6
JS only4245−3
Spectral and JS6247+15

We report both directions of change and discuss shared errors as a failure mode in Section 5.

Weakness 4: Loss of discriminative information from centering and covariance correction under class-prior shift

Response: We agree. Under strong global prior shift PRISM falls below the source model, with centering the largest single contributor, while under local label imbalance it remains stable.

A change in class proportions moves the marginal feature mean and the between-class covariance even when every class-conditional distribution is fixed. Centering then removes part of the semantic signal, the case Et≠0\mathbb E t\neq 0 discussed after Proposition E.5. At batch size 8 on ImageNet-C, Table 20 reports 62.81% for Dirichlet(0.1) streams (not sample-matched) and 62.93% for i.i.d. streams, and class-blocked and sticky (0.95) orderings cost 0.98 and 0.82 pp. To test a global prior shift on the 15 severity-5 CIFAR-10-C corruptions, PRISM estimates its statistics from 960 images, drawn class-balanced or with one class making up 30% or 60% of the stream. The dominant class is fixed in advance and rotated across corruptions. The statistics are then frozen and scored on a common class-balanced set of 1,000 images per corruption. The last two rows replace one statistic with its balanced-stream estimate.

Accuracy on the balanced evaluation setBalanced stream30% one class60% one class
Frozen source79.24%79.24%79.24%
Five views, equal weights, no correction81.28%81.28%81.28%
PRISM89.57%87.41%78.56%
PRISM, balanced means restored89.57%88.25%83.08%
PRISM, balanced covariance transform restored89.57%87.89%79.77%

At 30% concentration, PRISM loses 2.16 pp and stays 6.13 pp above uncorrected fusion. At 60%, it drops to 78.56%, 2.72 pp below uncorrected fusion and 0.68 pp below the source. Of the 11.01 pp loss, restoring the balanced means recovers 4.52 pp and restoring the covariance transform recovers 1.21 pp. We report this result, including the drop below the source, in Section 5.

In conclusion, we hope our responses resolve your concerns:

  • Theorem 4.1 motivates PRISM, which keeps its error-correcting roles without the entropy objective, Appendix E.4 states when its operations preserve correctly oriented evidence, and Corollary E.2 bounds its output (W1, Q1).
  • The label-informed audit is now paired with two analyses that use label-free partitions, in which the components are semantically coherent on CIFAR-C and on the three ImageNet benchmarks, with partial coverage (W2, Q2).
  • Fusion reinforces fully shared errors only in confidence and gives a small net gain where views disagree (W3).
  • PRISM is stable under local label imbalance but falls below the source under strong global prior shift, with centering the largest single contributor (W4).

We sincerely thank you for your time and effort in reviewing our paper, and we are happy to address any follow-up questions.

Reviewer_waL2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks when test-time adaptation (TTA) from unlabeled data can succeed. It formalizes recoverable semantic structure: near-pure components recoverable without labels, source orientation, and within-semantic support dominating cross-semantic leakage. Under these conditions and protocol stability, it proves a finite-sample bound for a recover-identify-project learner and constructs leave-one obstructions. It also derives an Assouad lower bound matching the leading margin and sample-size dependence. For binary pure-component pools and two-layer ReLU networks with a fixed output head, it derives an exact entropy-GD update and proves early risk descent when support outweighs leakage. PRISM is an optimization-free multi-view method. CIFAR-C audits find the four conditions jointly in 86/95 and 78/95 environments, with recovery in all 169 positive-margin environments. Across five ViT-B benchmarks, PRISM averages 69.36%, against 63.33% (Table 1's NEO average, mixing published and in-house values), 63.03% (matched NEO), and 63.96% (equal-view NEO-5V).

Critical strengths
  • The theory appears correct where I checked it. I followed the proofs of Theorem 3.2 (A.1-A.12), Theorem 4.1 (C.2-C.6), Theorem E.1 and Corollary E.2, and the KL step of Theorem A.20 without finding fatal errors.
  • PRISM's improvement survives the strongest controls: +6.33 over protocol-matched NEO and +5.40 over equal-view NEO-5V at 512 samples.
Critical weaknesses
  • Main concern is that the headline empirical claim is validated mainly on short streams. The abstract states that PRISM "consistently outperforms strong backpropagation-based and backpropagation-free TTA baselines", but the multi-baseline comparison uses NEO's 512-sample protocol. The 4,096-sample stream is compared only with NEO and NEO-5V, on a single seed.
  • Second concern is that the finite-width mechanism is stated more generally than its evidence. The abstract states the two-layer, fixed-head scope, but contribution (2) and Section 4.1 read as explaining entropy-based correction in practical TTA. Theorem 4.1 certifies strict risk reduction on only 11 of 1,018 audited unions (D.4), and this number appears only in the appendix.
Questions for authors
  • Under a standard full-stream protocol (all 50,000 ImageNet-C or 10,000 CIFAR-C images per corruption), how does PRISM compare with FOA, PEA, NEO, and Tent under the same implementation?
  • Is protocol stability exactly the inclusion stated in Lemma A.11? Must it hold only for the realized partition and bijection π\pi, or for all partitions and bijections within δC\delta_C?
  • Theorem 3.2 states probability "over the draw of D2∼μmD_2 \sim \mu^m", but Theorem A.12 bounds the probability over both D1D_1 and D2D_2, with δC=δC(m1,δ/3)\delta_C = \delta_C(m_1, \delta/3) and m1=⌊m/2⌋m_1 = \lfloor m/2 \rfloor. Will the main-text theorem be restated as in Theorem A.12?
  • Among the 11/1,018 unions with certified risk reduction, how many errors were actually corrected, and were their margins below ΛΔcorr\Lambda \Delta_{\rm corr} (Λ=10−3\Lambda = 10^{-3})? Has the coupling-field sign also been measured in a multiclass network adapting normalization parameters, as in Tent?
  • What are the five-benchmark results for single-view PRISM (V=1V = 1, centering plus total-covariance transport only)? Please also give the numbers for the cross-backbone equal-view control (G.10) and the leave-one-view-out analysis (G.16), which are described without results.
  • G.9 states that the view family and inference-time settings were fixed "without using target labels". Which data guided the choice of view family, κn=n/(n+r)\kappa_n = n/(n+r) damping, the one-sided rule, and the eigenvalue floor: a separate validation split, or Table 1 accuracy?
5 comments
AuthorsOpenReview ↗

Response to Reviewer waL2 (1/4)

We thank you for checking the proofs of Theorems 3.2 and 4.1 and Corollary E.2 so carefully, and for questions that pinpoint where our claims needed tighter scoping and stronger evidence. We address each point below.

Weakness 1 & Question 1: Multi-baseline comparison on 512-image streams, and a full-stream comparison with Tent, FOA, PEA, and NEO under the same implementation

Response: On full streams, PRISM leads Tent, FOA, NEO, and NEO-5V in all 12 comparisons, by at least 1.64 pp. Against PEA, which uses source-domain statistics that PRISM does not, it leads by 1.11 pp on CIFAR-10-C and is within 0.7 pp on CIFAR-100-C (+0.66) and ImageNet-C (−0.68). PRISM's own accuracy after image 4,096 is within 0.06 pp of its full-stream accuracy.

We run all methods with the same ViT-B source model on the 15 standard severity-5 corruptions using every image (10,000 per CIFAR-C corruption and 50,000 per ImageNet-C corruption). All methods share three fixed stream orders and the same batches of 64, and each method is reset only at the start of a corruption stream. As shown in the table below, PRISM does not fall behind later in the stream. On images 4,097 onward, its accuracy stays within 0.06 pp of its full-stream accuracy, and it still leads Tent, FOA, and both NEO variants by at least 1.39 pp.

Full stream / images 4,097 onwardPRISM (%)vs. Tent (pp)vs. FOA (pp)vs. PEA (pp)vs. NEO (pp)vs. NEO-5V (pp)
CIFAR-10-C89.89 / 89.83+3.04 / +3.25+3.98 / +3.72+1.11 / +1.13+4.61 / +4.70+3.98 / +4.04
CIFAR-100-C74.89 / 74.89+20.27 / +23.36+7.42 / +7.04+0.66 / +0.08+7.57 / +7.56+4.92 / +4.90
ImageNet-C64.27 / 64.32+5.38 / +5.20+1.64 / +1.39−0.68 / −0.63+5.08 / +5.12+3.69 / +3.72

The PRISM column is its accuracy, and the other columns are PRISM minus the baseline. NEO is the original single-view NEO, and NEO-5V receives PRISM's five views.

Note that PEA and PRISM differ in the information they use. PEA aligns target features with statistics computed on source-domain data, while PRISM uses only the frozen model and the unlabeled target stream, with no access to source images or source feature statistics. Even without this information, PRISM outperforms PEA on CIFAR-10-C and is within 0.7 pp of it on CIFAR-100-C and ImageNet-C.

We have revised the abstract to state the full-stream result, and have included this comparison in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer waL2 (2/4)

Weakness 2 & Question 4: Scope of the finite-width mechanism, the errors corrected in the 11 certified unions, and the coupling sign in multiclass Tent

Response: We have narrowed the claim to the two-layer, fixed-head setting of Theorem 4.1, in which 15 of the 18 errors corrected in the 11 certified unions meet the condition of Theorem 4.1 and all 18 start within depth ΛΔcorr(Λ)\Lambda\Delta_{\rm corr}(\Lambda), and in multiclass Tent the SGD cross-sample field contributes positively to the true-class margin, to first order, for 25% to 43% of initial errors.

Contribution (2) and Section 4.1 now state this setting, binary pure-component pools and two-layer ReLU networks with a fixed output head, and Section 4.1 now reports the 11/1,018 count from Appendix D.4. Within this setting, the section shows that entropy on an isolated prediction carries no correctness signal, and that correction comes from parameter sharing across samples. The certificate is conservative. On archived trajectories with union-specific schedules, the exact field d0d_0 separates improving from non-improving unions with AUROC 0.9985 (Appendix D.3). We do not claim that this mechanism explains most of the gains of entropy-based TTA in practice.

The 11 certified unions correct 18 errors, counted as sample-union occurrences. 15 of them meet the condition −ηTΔcorr(Λ)<z0≤0-\eta T\Delta_{\rm corr}(\Lambda)<z_0\le 0 of Theorem 4.1. All 18 have initial depth ∣z0∣|z_0| below the outer threshold ΛΔcorr(Λ)\Lambda\Delta_{\rm corr}(\Lambda) at Λ=10−3\Lambda=10^{-3}, an upper bound on the certified depth since η(T+1)≤Λ\eta(T+1)\le\Lambda, so the other 3 are corrected beyond the certified depth, as expected of a conservative certificate. No initially correct sample becomes incorrect.

For the multiclass case, we apply a Tent-style entropy update to the normalization parameters of ResNet-50 and ViT-B on severity-5 ImageNet-C, at 720 receiver positions per backbone. Under SGD, the cross-sample field of an initially wrong receiver ii with true-class margin mim_i removes the receiver's own entropy term and keeps the 1/B1/B normalization,

Δmicross=−ηB∑j≠i⟨∇θmi,∇θH(pj)⟩.\Delta m_i^{\rm cross}=-\frac{\eta}{B}\sum_{j\ne i}\big\langle\nabla_\theta m_i,\nabla_\theta H(p_j)\big\rangle .

Under Adam, we take one step from the same optimizer state with and without the receiver's term and report the sign of ∇θmi⊤δθ\nabla_\theta m_i^\top\delta\theta. Labels are used only to evaluate margins.

BackboneInitially wrongSGD, cross-sample > 0SGD, full update > 0Adam, cross-sample > 0Adam, full update > 0
ResNet-50457116 (25.4%)103 (22.5%)64 (14.0%)58 (12.7%)
ViT-B343147 (42.9%)132 (38.5%)140 (40.8%)128 (37.3%)

As shown in the table above, the other samples push the true-class margin up for 25.4% of initially wrong predictions on ResNet-50 and 42.9% on ViT-B under SGD, and removing the receiver's own term raises this fraction in every setting. Thus we observe a positive first-order cross-sample contribution to the true-class margin for a subset of initially incorrect predictions. The contribution is not uniformly positive: for the remaining errors, the first-order contribution is non-positive, as expected because normalization-parameter coupling, unlike the nonnegative fixed-head kernel of Section 4, can take either sign. We have added these measurements to Section 4 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer waL2 (3/4)

Question 2: Quantifiers in protocol stability

Response: Yes, protocol stability is exactly the inclusion in Lemma A.11, and because the realized partition and matching depend on the data, the revised paper requires it for all partitions and matchings within δC\delta_C.

The inclusion is TπYΓ(C)⊆YΓ(C^)T_\pi\mathcal Y_\Gamma(C)\subseteq\mathcal Y_\Gamma(\widehat C), where YΓ(C)\mathcal Y_\Gamma(C) is the set of component labelings reachable under protocol Γ\Gamma and TπT_\pi transfers a labeling along the matching π\pi. The proof uses it only for the realized pair (C^,π)(\widehat C,\pi) on the D1D_1 recovery event. Requiring it for every pair with max⁡uμ(Cu△C^π(u))≤δC\max_u\mu(C_u\triangle\widehat C_{\pi(u)})\le\delta_C covers this pair and is a sufficient condition. The recovery condition of Theorem 3.2 is unchanged apart from the restatement in our answer to Question 3.

Question 3: Probability space in Theorem 3.2

Response: Yes, Theorem 3.2 now states the probability over the joint draw of (D1,D2)(D_1,D_2), exactly as in Theorem A.12.

With m1=⌊m/2⌋m_1=\lfloor m/2\rfloor, m2=m−m1m_2=m-m_1, and independent D1∼μm1D_1\sim\mu^{m_1} and D2∼μm2D_2\sim\mu^{m_2}, the bound holds with probability at least 1−δ1-\delta, with δC=δC(m1,δ/3)\delta_C=\delta_C(m_1,\delta/3). Partition recovery on D1D_1, and component-mass estimation and semantic identification on D2D_2, each fail with probability at most δ/3\delta/3, and a union bound combines them. The proof in Appendix A is unchanged.

Question 5: Single-view PRISM, and the results of G.10 and G.16

Response: Single-view PRISM reaches 65.09% macro accuracy and outperforms single-view NEO on every benchmark, PRISM outperforms NEO under four equal views on all four backbones, and no single view is critical in G.16. We have added all three results to Appendices G.10 and G.16.

Single-view PRISM receives only the original image. With one view, nuisance suppression and fusion reduce to the identity, so the method uses only centering and total-covariance transport. As shown in the table below, it is 2.06 pp above protocol-matched NEO (63.03%, Table 11), which also uses the original image, and it leads on every benchmark by 1.38 to 3.39 pp.

BenchmarkCIFAR-10-CCIFAR-100-CImageNet-CImageNet-RImageNet-SketchMacro
PRISM, 1 view85.95%69.07%60.46%60.87%49.09%65.09%

This run differs from the five-view default (69.36%) in both its input and its number of views, since none of the five default views is the original image. The view-subset analysis keeps the input family fixed. A single default view gives 63.92% on average, and all five give 69.36%.

In the equal-view control (G.10), PRISM and NEO receive the same four views (the original image, a 90% center crop, a horizontal flip, and the flipped crop), on severity-5 ImageNet-C with 512 images per corruption.

BackboneNEO-4VPRISM-4VGain (pp)
ResNet-5030.29%34.71%+4.43
ViT-S43.53%45.47%+1.94
ViT-B60.79%62.43%+1.64
ViT-L63.28%64.74%+1.46

For G.16, we omit each view in turn and recompute all statistics and fusion weights. The largest drop comes from omitting the 90% center crop (0.54 pp).

Omitted view90% cropBlur, r = 0.5Blur, r = 1.03×3 medianUnsharp maskMean
Macro change (pp)−0.54−0.16−0.24−0.27−0.36−0.31

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.