- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies two fundamental questions of test-time adaptation (TTA): when unlabeled target data suffice to determine a better predictor, and why entropy minimization can correct errors rather than merely reinforce them. The contributions consist of three parts. (1) The paper proposes the principle of "recoverable semantic structure" and establishes a PAC learnability framework for TTA with finite-sample guarantees. (2) It proves that entropy itself carries no correctness signal, and that error correction instead arises from parameter sharing, which propagates source-oriented evidence across samples through a coupling kernel. (3) Guided by the same principle, the paper designs PRISM, an optimization-free method that fuses multiple deterministic views of each target sample via spectral matching in the classifier-visible subspace. PRISM achieves 69.36% average accuracy across five benchmarks, clearly surpassing the strongest baseline at 63.33%.
Critical strengths
- The large-scale structural audit and the intervention experiments directly verify the theoretical assumptions, effectively closing the loop between theory and experiment.
- PRISM requires no backpropagation, yet outperforms the strongest baseline by about 6 points on average across five benchmarks, and works consistently over four different backbones.
Critical weaknesses
- The scope of the theory (two-layer ReLU networks with a component-constant hypothesis class) is far from the ViT/ResNet architectures used in the evaluation, so it serves more as a design heuristic than a rigorous guarantee for practical methods.
- The structural audit is conducted only on CIFAR-C, and the ImageNet-level benchmarks lack equivalent verification, which weakens the link between the theory and the observed performance there. In addition, the main table reports no paired significance tests.
Questions for authors
- How sensitive are the results of PRISM to the choice of augmentation view types and the number of views?
- When labels vary gradually over a semantic continuum, so that no pure partition exists, does the PAC framework still apply?
5 comments
Response to Reviewer YVvh (2/3)
Weakness 2: ImageNet-level verification of the theory, and paired significance tests
Response: Appendix E.5 already audits the identification margin on all 17 ImageNet-C, ImageNet-R, and ImageNet-Sketch domains, we now add a held-out test of the semantic transfer of label-free partitions, and paired tests show that PRISM's gains over NEO-5V are significant on all three corruption benchmarks after Holm correction.
The four-condition audit of Section 3.3 remains CIFAR-C only, and the revised paper states this.
Identification margin. Appendix E.5 already audits 47 ViT-B target domains with the component graph and source anchors held fixed, 15 each from CIFAR-10-C, CIFAR-100-C, and ImageNet-C, plus ImageNet-R and ImageNet-Sketch. At 512 samples, 8 of the 17 ImageNet domains (15 ImageNet-C corruptions, ImageNet-R, and ImageNet-Sketch) have a strictly positive empirical margin, and PRISM's empirical bound is below one in 7 of these 8. Over all 47 domains, the margin is positive in 17 at 512 samples and 15 at 4,096, with PRISM's empirical bound below one in 16/17 and 12/15. On the same graphs, PRISM improves mean accuracy over the source by 8.90 pp (512) and 8.95 pp (4,096).
Held-out semantic transfer. We fix the partition rule in advance, fit it on 4,096 unlabeled images per domain, and apply it to a disjoint set of 4,096 images. Only after both sets of assignments are frozen do we use labels. Labels of the fit set name each component by its majority class, and labels of the held-out set score whether the name is correct. As shown in the table below, label-free components transfer their semantics to new images, but resolution is partial, and 30% to 55% of held-out images are both resolved and correctly named.
| Benchmark | Held-out images resolved | Name correct, among resolved | Resolved and correct, among all |
|---|---|---|---|
| ImageNet-C (15 corruptions) | 41.8% | 72.9% | 30.5% |
| ImageNet-R | 67.4% | 81.2% | 54.7% |
| ImageNet-Sketch | 57.2% | 60.4% | 34.6% |
Paired significance. We compare PRISM with NEO-5V, which receives the same five views and averages its view-specific corrected posteriors. Predictions are paired by image and stream order, the bootstrap resamples groups defined by corruption and shared underlying image, and Holm correction is applied across the five benchmarks. As shown in the table below, the three corruption benchmarks are significant after correction. ImageNet-R and ImageNet-Sketch have positive estimates and unadjusted intervals that exclude zero, but they do not reach significance after Holm correction, and we report them as such. The corruption benchmarks pool all five severities, so their estimates are smaller than the severity-5 full-stream gaps (+3.98, +4.92, and +3.69 pp).
| Benchmark and protocol | PRISM − NEO-5V (pp) | 95% CI (pp) | Holm-adjusted |
|---|---|---|---|
| CIFAR-10-C, full streams, all five severities | +2.83 | [1.34, 4.32] | 0.00079 |
| CIFAR-100-C, full streams, all five severities | +3.80 | [2.00, 5.60] | 0.00018 |
| ImageNet-C, full streams, all five severities | +3.31 | [1.47, 5.15] | 0.00129 |
| ImageNet-R, one order of 512 images | +4.10 | [0.43, 7.78] | 0.0574 |
| ImageNet-Sketch, one order of 512 images | +5.27 | [0.50, 10.05] | 0.0574 |
We have included the margin coverage, the semantic transfer results, and this table in Section 5 of the revised paper.
Response to Reviewer YVvh (3/3)
Question 1: Sensitivity of PRISM to the type and number of views
Response: PRISM is insensitive to the type of views, gains most from the second view, and keeps its advantage over NEO when both receive the same views.
For view type, Table 9 replaces one transformation at a time with a 95% crop, Gaussian blur at radius 1.5, box blur, or a sharpness adjustment, and the five-benchmark macro accuracy changes by −0.21 to +0.04 pp from 69.36%.
For view count, we evaluate all 31 nonempty subsets of the five default views and recompute the online statistics for each. As shown in the table below, which averages the macro accuracy over all subsets of each size, the second view adds 3.41 pp, and later views add 1.09, 0.63, and 0.31 pp.
| Number of views | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| Five-benchmark macro accuracy | 63.92% | 67.33% | 68.42% | 69.05% | 69.36% |
In a separate control, every method receives the same four views (the original image, a 90% center crop, a horizontal flip, and the flipped crop) on severity-5 ImageNet-C with 512 images per corruption. PRISM outperforms NEO with the same views by 1.46 to 4.43 pp across ResNet-50, ViT-S, ViT-B, and ViT-L, so its advantage does not come from using more views.
Question 2: Whether the PAC framework applies when labels vary over a semantic continuum
Response: Yes, the framework applies without pure components, since impurity enters the bound through the within-component minority-label mass, and the bound loosens as the overlap grows.
Theorem 3.2 does not require pure components. It requires a finite partition whose components are near-pure, recoverable from unlabeled data, and oriented by the source, together with a positive identification margin and protocol stability. Impurity enters the approximation term , and the guarantee is relative to the component-resolution oracle.
A simple example shows what the bound gives when no pure partition exists. Let and , so both labels have positive probability everywhere. Suppose the source score thresholds at and induces the components and , with positive within-side affinity, zero cross-side affinity, and source-anchor mass only on the majority label of each side. Then , the margin is positive, and the component-majority predictor, which here is the Bayes classifier, has risk . For a protocol that can realize this oracle, the bound is plus finite-sample terms.
The bound loosens as the overlap grows. Finer components lower but shrink the minimum component mass and raise the cost of recovery, and when no finite partition balances impurity, recoverability, and orientation, the bound becomes vacuous. The comparator cannot be strengthened in general. Theorem A.19 constructs two worlds with the same unlabeled observations and source information, in which a protocol that can relabel a subregion inside a component admits a zero-risk predictor in each world, but the two predictors disagree on that subregion. No method can therefore match the unrestricted reachable oracle uniformly over these worlds. We have added the example and this discussion to Appendix A.7, referenced after Theorem 3.2.
In conclusion, we hope our responses resolve your concerns:
- Theorem 4.1 is now scoped to its setting, while the PAC result places no restriction on the architecture and Corollary E.2 bounds any predictor. In ResNet-50 and ViT-B, the SGD cross-sample entropy field contributes positively to the true-class margin, to first order, for 25% to 43% of errors (W1).
- On the three ImageNet benchmarks, we report margin coverage and held-out semantic transfer, and PRISM's gains over NEO-5V are significant on all three corruption benchmarks (W2).
- PRISM is insensitive to the view type, gains most from the second view, and outperforms NEO under equal views (Q1).
- The framework covers semantic continua through the impurity term, as a worked example shows (Q2).
We sincerely thank you for your time and effort in reviewing our paper, and we are happy to address any follow-up questions.
Response to Reviewer YVvh (1/3)
We thank you for your positive assessment and for noting that the structural audit and intervention experiments close the loop between theory and experiment. Below we address the scope of the theory, its verification on the ImageNet benchmarks, and your two questions.
Weakness 1: Scope of the theory relative to the ViT and ResNet models in the evaluation
Response: We agree for Theorem 4.1, which gives a design principle for ViT and ResNet, but the PAC result places no restriction on the architecture, and for ViT-B PRISM, the empirical bound of its output-level Corollary E.2 is below one in 16 of 17 and 12 of 15 positive-margin domains, at 512 and 4,096 samples.
Theorem 4.1 analyzes binary two-layer ReLU networks with a fixed output head, and we have revised contribution (2) and Section 4 to state this scope.
The PAC result of Section 3 places no restriction on the architecture, since the reachable class HsΓ contains whatever the protocol can reach in any network. We agree that it is component-constant in one respect. Both the comparator and the output of the recover-identify-project learner are constant on resolved components, and Theorem A.19 shows why the comparator cannot be relaxed, because unlabeled data cannot tell which subregion of a component to relabel. Corollary E.2 removes this restriction for evaluation. It bounds the error of any predictor, from any architecture and not necessarily constant on components, through the realization gap e(p) of its posterior p,
L_Q(\arg\max p)\le \min\Big\\{1,\ \zeta+\alpha_{\rm tot}+\frac{2e(p)}{\Delta_{\rm id}}\Big\\},whenever the identification margin Δid is positive and the corollary's structural conditions hold on a fixed component graph. Here ζ is the unresolved mass and αtot is the within-component minority-label mass. Our response to Weakness 2 reports how often these conditions hold for ViT-B on ImageNet-C, ImageNet-R, and ImageNet-Sketch. The module-specific results of Appendix E.4 are likewise stated in the classifier-visible feature space of a frozen network with a linear head, so they apply to PRISM on ViT and ResNet whenever their conditions on the feature geometry hold.
To test whether the mechanism of Theorem 4.1 appears in practical networks, we apply a Tent-style entropy update to the normalization parameters of ResNet-50 and ViT-B on severity-5 ImageNet-C, at 720 receiver positions per backbone. For each initially wrong prediction, we compute the first-order change of its true-class margin from the entropy gradients of the other samples alone (cross-sample) and from the full batch update. Labels are used only to compute margins after the update. As shown in the table below, the other samples push the true-class margin up for 25.4% of wrong predictions on ResNet-50 and 42.9% on ViT-B under SGD, and removing each receiver's own term raises this fraction in every setting.
In these networks, we observe a positive first-order cross-sample contribution to the true-class margin for a subset of initially incorrect predictions. For the remaining errors, the first-order contribution is non-positive, because normalization-parameter coupling, unlike the nonnegative fixed-head kernel of Section 4, can take either sign. We present these results as first-order evidence for the mechanism, not as a guarantee, and have included them in Section 4 of the revised paper.