acceptodds
Withdrawn from ICLR 2027

The Space Between Views: A Simplex Approach to Test-Time Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We hypothesize that the region spanned by augmented views contains useful structure for test-time adaptation (TTA). Can TTA exploit it? Existing methods typically treat augmented views as a finite set, aggregating, filtering, or regularizing their predictions. We propose SimplexTTA, which models the convex hull of augmented test features and minimizes the expected adaptation loss over the resulting simplex. To make this objective practical, we derive a second-order approximation around the simplex barycenter. This, in turn, yields a curvature-weighted correction that captures the interaction between feature spread and the local curvature of the loss. The resulting objective can directly replace the standard adaptation loss in existing gradient-based TTA methods with minimal modification. Across multiple image classification benchmarks, including corruption benchmarks, SimplexTTA consistently improves state-of-the-art TTA methods, providing empirical support for our hypothesis.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Computer Vision, Test-Time Adaptation, Distribution Shift, Data Augmentation, Feature-Space Simplex.

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_f6jsOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SimplexTTA, a framework that applies a test-time adaptation objective to Dirichlet-weighted convex combinations of augmented visual features. A second-order expansion around the mixture barycenter yields a surrogate consisting of a barycenter loss and a covariance-weighted curvature correction. The correction is evaluated using Hessian–vector products. The paper also presents an approximation-error statement and an exact interpolation identity for quadratic losses. The framework is instantiated with TENT, CLIPTTA, BATCLIP, and BITTA using CLIP ViT-B/16. Evaluation covers 13 clean classification datasets, natural ImageNet shifts, corruption benchmarks, and cross-domain classification. The reported results include positive clean-data averages and improvements in all 16 natural-shift comparisons, alongside substantial failures for TENT under corruption. Importantly, the controlled BATCLIP ablation finds that barycenter-only adaptation matches the complete simplex objective, leaving the empirical contribution of the curvature correction unresolved.

Critical strengths
  1. The paper provides a coherent formulation connecting feature averaging, augmentation covariance, and the adaptation objective. The Dirichlet formulation yields an interpretable decomposition into a loss at the barycenter and a curvature-weighted covariance term. The quadratic special case makes the relationship between barycenter and vertex objectives explicit. This is a useful conceptual framework for distinguishing adaptation at a representative feature from adaptation that retains augmentation variability. Its value is not limited to the particular implementation, although the practical benefit of the correction remains to be demonstrated.

  2. The experiments include a meaningful component control and acknowledge important negative results. Table 4 distinguishes prediction-time averaging, discrete-view adaptation, barycenter-only adaptation, and the complete objective under a shared view budget and barycenter readout. This control shows that changing the adaptation objective can improve performance beyond prediction averaging alone. The paper also explicitly reports TENT’s corruption failures and acknowledges that the curvature correction contributes little in the controlled CLIP experiment. These observations make the study more informative than a comparison of aggregate accuracies alone.

Critical weaknesses
  1. The central empirical contribution is not distinguished from substantially simpler barycenter-only adaptation. The main methodological contribution is the continuous simplex objective and its covariance-weighted curvature correction. However, Table 4(a) reports 89.52/69.42 for the full objective versus 89.60/69.43 for barycenter-only adaptation on clean CIFAR-10 and the four-corruption subset. The concentration sweep produces very small changes, and the paper reports a TENT curvature-gradient magnitude of only 0.14% of the barycentric gradient in a matched diagnostic. These results support barycentric adaptation, but do not establish an advantage from retaining the curvature correction or approximating the full simplex expectation. The quadratic mechanism study demonstrates that an intermediate covariance level can be beneficial in a controlled setting. It does not establish that this mechanism explains the real-data improvements, especially when the paper itself finds full and barycenter-only adaptation nearly identical. A broader paired comparison against barycenter-only adaptation, including computational cost, is necessary to justify the additional method complexity.
  2. Essential implementation and evaluation details are missing, preventing verification that the reported results correspond to the stated objective and a consistently matched protocol. Section 3.3 states that implementations may detach the HVP during backpropagation, but does not specify which quantities are detached or give the resulting update formula. This matters because detaching HδiH\delta_i generally changes the gradient of δi⊤Hδi\delta_i^\top H\delta_i. The treatment of normalization and batch-coupled objectives also requires an explicit definition. The objective-value approximation bound does not, by itself, justify these altered gradients. The supplied PDF does not include the appendix that is said to contain these conventions, complete proofs, and experimental configurations. Moreover, the description that all ablation variants use barycenter prediction is difficult to reconcile with the different one-step baseline values in Table 4(a) and Table 4(c). The reuse of cached features also leaves the timing of prediction relative to parameter updates unclear Finally, the natural-shift and corruption tables report a single seed, and no end-to-end runtime or peak-memory comparison is provided in the available version. These omissions limit confidence in the smaller improvements and in the practical value of the additional curvature computation.
Questions for authors

Q1. Incremental value of the simplex correction — targeted validation. What real-data evidence demonstrates an advantage of the full objective over barycenter-only adaptation? Table 4(a) currently shows no improvement from the correction. Please extend this paired comparison to representative settings for the other base methods and to the benchmarks supporting the main claims, with variability across seeds and test-stream orders. Please also report latency and peak memory for the full and barycenter-only variants under the same view budget. This comparison would directly affect whether the extra curvature computation is justified.

Q2. Actual gradients and method-specific objectives — clarification and validation. Which tensors are detached in each implementation: the Hessian, the HVP, the deviations, or the complete correction? Please provide the exact update formula, including any coefficient compensation. Is the Hessian computed with respect to the pre-renormalization barycenter through the complete normalized-logit loss? For batch-coupled objectives such as CLIPTTA, how is the joint simplex expectation defined, and are mixing coefficients independent across images? A comparison with a Monte Carlo estimate of the original Dirichlet objective and its gradient, using the same cached views, would help establish whether the implemented surrogate faithfully approximates the proposed objective.

Q3. Prediction and caching protocol — clarification and correction. Why does Table 4(a) report 88.80/66.88 for Base+Bary readout, while the one-update Base entry in Table 4(c) is 90.12/68.33? Does panel (c) use original-view prediction for the baseline, despite the statement that all ablation variants use barycenter prediction? Please also specify whether predictions are produced before or after adapting on the current batch. When visual-encoder parameters are updated, are the cached embeddings recomputed? How is this handled for multiple updates, and is the same timing used for both sides of every comparison?

Q4. Missing supporting material and configuration selection — clarification. Could you provide or identify the appendix referenced throughout the manuscript? The version available to me ends after the references and does not contain the promised proofs, method-specific gradients, complete augmentation recipe, or full optimizer configurations. Please clarify how the update counts and the EuroSAT-specific augmentation recipe were selected, including whether a separate validation protocol was used. Please also provide repeated-run results for the main comparisons, especially where improvements are small or adaptation becomes unstable. eg: page5 line 262 there exit "???"

Reviewer_ikpYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes SimplexTTA, which aims to improve unlabeled test-time adaptation by exploiting the continuous convex region formed by multiple augmented views of the same test sample in feature space. It uses the augmented feature embeddings as vertices, defines convex combinations through a Dirichlet distribution, and minimizes the expected adaptation loss under this distribution. The experiments cover multiple datasets and a range of distribution shifts.

Critical strengths
  1. The motivation is clear.
  2. The experiments cover multiple datasets and a range of distribution shifts.
Critical weaknesses
  1. The writing is unclear and contains many broken references, particularly in the method and experiment sections.
  2. The contributions are not clearly disentangled, and key ablations are missing. In particular, the effectiveness of the central contribution—second-order curvature correction—has not been demonstrated.
  3. The core assumption is that the continuous convex region spanned by augmented views captures useful local data structure, but the paper provides no experimental or theoretical support for this assumption.
Questions for authors

See Weaknesses.

Reviewer_bFjXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SimplexTTA, a method that uses the continuous feature space between augmented views for test-time adaptation (TTA). Instead of using individual augmented views or their averaged predictions, the authors construct a simplex from augmented features and minimize the expected adaptation loss over this region.

Since directly computing this expectation is difficult, they derive a second-order approximation using the loss at the barycenter and a Hessian-based curvature correction. The proposed objective can be applied to existing gradient-based TTA methods without changing their overall framework.

The authors evaluate SimplexTTA with TENT, CLIPTTA, BATCLIP, and BITTA on several classification and distribution shift benchmarks. The results show improvements in many settings, especially for BATCLIP and BITTA, although some settings show performance degradation.

Critical strengths
  1. Interesting idea with a reasonable mathematical formulation

I think the main idea of this paper is interesting. Instead of treating augmented views as separate samples, the authors consider the continuous feature space between them. This seems to be a reasonable direction, especially for TTA where only limited unlabeled data is available.

The second-order approximation is also interesting. It provides a way to approximate the expected loss over the simplex using the barycenter loss and curvature information, without explicitly sampling many intermediate features.

Another advantage is that the proposed method can be applied to existing TTA methods by modifying their loss functions, without requiring a new model architecture.

  1. Experiments on different TTA methods and benchmarks

The authors evaluate the method on four TTA approaches and several datasets, including clean classification, natural distribution shifts, and corruption benchmarks.

The improvements on BATCLIP and BITTA are encouraging, and the method also improves all 16 comparisons on the natural ImageNet shifts. I also appreciate that the authors include ablation studies and report the settings where the proposed method does not work well.

These experiments are useful for understanding both the advantages and limitations of the method.

Critical weaknesses
  1. The contribution of the curvature correction is not clear from the ablation results

My main concern is that the experimental results do not strongly support the importance of the proposed curvature correction, which seems to be one of the main methodological contributions of this paper.

In Table 4(a), using only the barycenter loss gives almost the same performance as the full SimplexTTA objective. For example, the barycenter-only method achieves 89.60% on CIFAR-10 and 69.43% on the selected CIFAR-10-C corruptions, while the full method achieves 89.52% and 69.42%, respectively.

The sensitivity to the Dirichlet concentration parameter is also very small.

From these results, it seems that most of the improvement comes from adapting the model using the averaged feature representation, rather than from the curvature correction itself.

I understand that the authors also acknowledge this observation. However, I am still not convinced why the full second-order objective is necessary, considering the additional computation for Hessian-vector products.

I think this is an important issue because the curvature-based formulation is a central part of the proposed method. It would be more convincing if the authors could show some settings where the curvature correction provides a clear benefit over simply using the barycenter loss.

  1. Unclear distinction from general feature interpolation methods and the motivation for focusing on TTA

Another concern is about the methodological novelty and the scope of the proposed approach.

The idea of using convex combinations of feature representations is related to existing methods such as Mixup and Manifold Mixup. Of course, this paper is different in that it formulates an expected loss over the simplex and applies it to unlabeled test-time adaptation.

However, I wonder why this approach is specifically motivated for TTA. The proposed formulation seems quite general, and I think a similar idea could also be applied to conventional supervised or self-supervised learning.

I agree that TTA is a reasonable setting, since the model has to adapt using limited unlabeled samples. Still, it is not very clear to me what makes this simplex-based formulation particularly suitable for TTA, beyond applying it to existing TTA loss functions.

I am not suggesting that the authors must evaluate all other learning settings. However, I think the paper could better explain what is specific to TTA in the proposed method, and how its contribution differs from existing feature interpolation approaches.

Also, the performance is not consistent across all settings. In particular, SimplexTTA considerably degrades TENT performance on corruption benchmarks, and the results on OfficeHome are also negative. This makes me wonder under what conditions the proposed approach is actually beneficial.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.