acceptodds
Under review as a conference paper at ICLR 2027

Focus on Likely Classes for Test-Time Prediction

Abstract

We ask: Can focusing on likely classes of a single, in-domain sample improve accuracy? Prior work argued “no”, we answer “yes” on average. Standard entropy minimization yields largest gains, and we further dissect why by looking at its two core mechanisms: increasing likely and decreasing unlikely class predictions. Simply maximizing logits of the two most likely classes explains most of its benefits and, in some cases, even outperforms entropy minimization. Thus, decreasing classes is a secondary concern. Our controlled experiment and theory investigate the most puzzling case, why simply optimizing the logits of the two most likely classes leads to accuracy gains. Our small scale networks suggest that the degree of uncertainty of a prediction, followed by model choice, and to a lesser extent dataset choice moderate the success chances of the optimization. We show that the true class tends to have larger gradients independent of whether the prediction is correct or not. Our theory thoroughly investigates one of multiple possible explanations from the angle of shared features. Our evaluation focuses on 26 large language models and 12 text datasets, and 18 image recognition models trained on ImageNet. It demonstrates gains of 0.6% on average for entropy minimization on uncertain samples using no extra data aside from the given test sample using a fixed learning rate. We also suggest gradient-ray ensembling. When traversing input/embedding space by applying the same gradient with differing learning rates and aggregating probabilities for the same sample further improves accuracy significantly to 0.75% relying on a coarse range for learning rates, thus improving accuracy and eliminating the need to find a precise learning rate at the expense of computation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
test time adaptation, single sample, entropy minimization, SGD

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_AtdcOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks whether prediction accuracy can be improved by adjusting the input or its embeddings using only the current test sample, without labels, augmentations, or other samples. The authors compare entropy minimization with increasing likely-class outputs and decreasing unlikely-class outputs, and investigate gradient magnitudes and shared features as possible explanations. However, I find the evidence for the proposed mechanisms and their connection to the main empirical results insufficient.

Critical strengths
  1. The question is worth studying, particularly whether input or embedding optimization can improve predictions without additional supervision.

  2. The evaluation covers many models and datasets, and the authors report negative results rather than presenting only favorable cases.

  3. Decomposing entropy minimization into increasing likely-class outputs and decreasing unlikely-class outputs offers a useful perspective.

Critical weaknesses
  1. The main evaluation uses LLMs and ImageNet classifiers, but the gradient analysis relies on small EMNIST networks and samples whose true class is in the top two. Whether this mechanism explains the main results remains unverified.

  2. The theory relies on restrictive assumptions whose relevance to real models is not established. The shared-feature explanation remains a conjecture without direct measurements or feature interventions.

  3. The theory mainly analyzes unweighted top-two logit maximization, while entropy minimization performs best experimentally. The connection between these objectives is insufficiently established.

  4. A trivial solution to single-sample entropy minimization does not rule out benefits from finite-step updates. Framing the results as overturning prior understanding is therefore overstated.

  5. The method illustrations and experimental figures are insufficiently polished. Clearer explanations and more self-contained captions are needed to make the results easy to assess.

Questions for authors

See the weaknesses above.

3 comments
AuthorsOpenReview ↗

Removed non-essential theory and ensured easier to see linking of theory and experiments

The main evaluation uses LLMs and ImageNet classifiers, but the gradient analysis relies on small EMNIST networks and samples whose true class is in the top two. Whether this mechanism explains the main results remains unverified.

We removed the claim on gradient analysis from the abstract and moved the entire controlled experiment to the Appendix. We also removed some theorems to make the essential theorem stand out more clearly. We directly state the empirical evidence using LLMs (rather than small EMNIST networks) for the main theorem after the theory section. We disagree with the implied demand that a controlled experiment should explain particular LLM classifiers. By definition, a controlled experiment aims at deeper insights rather than verifying the behavior of the exact LLM architectures.

The concern that the correct class is among the top-2 is unwarranted as we aim to understand when the method helps and when it hurts. It can only hurt if the prediction is correct, i.e., it is among the top-1 and thus by definition among the top-2. If the true class is not among the top-2 but among the top-k (k>2k > 2), our method can only help. If the true class is not among the top-2, the method much more often neither hurts nor helps, so we left out these cases to reduce computation.

The theory relies on restrictive assumptions whose relevance to real models is not established. The shared-feature explanation remains a conjecture without direct measurements or feature interventions.

The general theory "General Theory for input tuning", already in the initial submission, derives the margin dynamics for arbitrary differentiable neural networks. We removed the shared feature conjecture and other distracting analysis and now focus on the key proven mechanism that is also established empirically on LLMs: What the theory shows is that (input) gradient sensitivity is a key factor.

The theory mainly analyses unweighted top-two logit maximization, while entropy minimization performs best experimentally. The connection between these objectives is insufficiently established.

We updated the paper with a new Figure 2. It clearly shows in words and mathematically the through simplification and problem isolation how we move from entropy minimization all the way to top-2 logit maximation.

A trivial solution to single-sample entropy minimization does not rule out benefits from finite-step updates. Framing the results as overturning prior understanding is therefore overstated.

We rephrased. Abstract: "The trivial solution of single-prediction entropy minimization motivated prior work to avoid optimizing predictions individually"

Intro: "entropy minimization has a trivial solution confirming just the most likely class, motivating forgoing single sample optimization in the favor of batches"

For reference the original paper stated: "Note that optimizing a single prediction has a trivial solution: assign all probability to the most probable class. We prevent this by jointly optimizing batched predictions over parameters that are shared across the batch."

The method illustrations and experimental figures are insufficiently polished. Clearer explanations and more self-contained captions are needed to make the results easy to assess.

We updated all descriptions and reworked all illustrations in the updated manuscript.

Reviewer_AtdcOpenReview ↗

Response to the Rebuttal

Thank you for the response and revisions. I appreciate the efforts to clarify the claims and improve the presentation. However, removing claims and moving analyses to the appendix do not, by themselves, resolve the concerns about the connection between the theory and the main empirical results. The stated additions would need to establish that connection convincingly.

The response still leaves me with the impression that the manuscript was not sufficiently developed at submission. The extensive revisions to its central explanation reinforce this concern. Obviously, this paper was not sufficiently prepared.

Overall, I remain unconvinced that the main concerns have been adequately addressed, and I maintain my original score.

Thanks for your assessment.

removing claims and moving analyses to the appendix do not, by themselves, resolve the concerns about the connection between the theory and the main empirical results. The extensive revisions to its central explanation reinforce this concern.

  • The main empirical result stated in the abstract is that incrementing classes is more valuable than decrementing. The latter is often associated with "noise" removal. There is no theory related to incrementing likely classes is better than decrementing unlikely.
  • We don't consider a conjecture the central explanation but rather the theorem that was already underpinned by empirical evidence in the initial version, which was connected through the gradient magnitude mentioned in the abstract . Eq. (22) connects the change in the logit margin to the difference in squared input-gradient norms.
  • This connection was reinforced in the new version by adding Figure 2, adding more text and also adding more experiments. These additions go beyond removing claims or relocating analyses.
Reviewer_1HR8OpenReview ↗
Rating
2(+1)
Confidence
2(−1)
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether prediction accuracy on a single in-domain test sample can be improved without ground-truth labels by reinforcing the class the model already considers most likely. It decomposes the entropy-minimization gradient update into two components: strengthening likely classes and suppressing the others. It then examines explicit optimization of the top two classes. The paper claims to evaluate over 300 model–dataset pairs.

Critical strengths
  1. The paper provides a theoretical analysis of test-time adaptation.
  2. The authors disclose the study’s limitations and their use of AI assistance.
Critical weaknesses
  1. The writing is unclear and overloaded with jargon.
  2. Despite claiming experiments on over 300 model–dataset pairs, the paper provides insufficient results and experimental details. The experimental evidence is severely lacking.
  3. The theoretical analysis is difficult to follow and does not adequately substantiate the proposed explanation of the mechanism.
  4. The citation for “Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factors” is incorrect: all author names are wrong.
  5. The approach appears similar to “Multi-label Test-Time Adaptation with Bound Entropy Minimization.” The authors should discuss the differences between the two methods and their respective strengths and weaknesses.
Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

Code/ general theory (already) in appendix/supplement, other concerns addressed

Ad 2)

Despite claiming experiments on over 300 model–dataset pairs, the paper provides insufficient results and experimental details. The experimental evidence is severely lacking.

We disagree. Code is in the supplement and multiple pages of results are in the Appendix. We shall add a reference in the main paper and a link to a public repo as this can indeed be overlooked.

Ad 3)

The theoretical analysis is (i) difficult to follow and (ii) does not adequately substantiate the proposed explanation of the mechanism.

We already provide two analyses. Each might be easy or hard depending on one's preferences. One is using a concrete network with few neurons (main paper) and one is general (in the appendix, Section "General Theory for input tuning"). We are not sure if the reviewer's criticism applies to both or just one.

Either way:

ad i) We simplified the model in the main paper allowing us to simplify the analysis a lot at a small expense of generality. We removed also everything except the key claim that is also empirically validated. In particular, we removed conjectures.

ad ii) We placed the empirical validation of the main theorem directly after it. The removal of the conjecture and non-essential analysis should help to make the main claim stand out more.

Ad 4)

The citation for “Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factors” is incorrect: all author names are wrong.

We apologize and fixed that. The last names were correct, but only one of the first names was correct.

Ad 5)

The approach appears similar to “Multi-label Test-Time Adaptation with Bound Entropy Minimization.” The authors should discuss the differences between the two methods and their respective strengths and weaknesses.

We added the ref pointing out that it is similar to the cited PASLE but the provided ref treats a different problem to PASLE and our paper. Table 2 and 3 (in the Appendix) already compare PASLE. In summary, our method involves different losses, we optimize inputs (needed less parameters and leading to better results - see Appendix D.3), we introduce and show benefits of self-ensembling (gradient-ray ensembling), also we strongly focus on understanding the core mechanism of many TTA papers: entropy minimization by analyzing it systematically.

It is understandable why the reference emerged. In the related work we wrote Conceptually, iFo and doFo are multi-class optimizations. This statement is true (also for PASLE), but the similarity is only "conceptual" regarding the optimization method. The problem solved is different, as we (and PASLE) optimize for uncertain samples towards two labels as for multi-label optimization, but in the end, we predict just one class. We deleted the above statement as it seems to cause confusion.

Thank you for the authors’ response. Most of my concerns have been addressed, and I am willing to increase my score to 2.

Reviewer_r1nLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether prediction accuracy can be improved solely by optimizing on a single in-domain test sample. The authors decompose entropy minimization into iFo, which increases the logits of likely classes, and doFo, which decreases the logits of unlikely classes. Experimental results show that iFo contributes more substantially to accuracy improvements and that simultaneously increasing the logits of the top two classes is sufficient to improve performance. To understand this phenomenon, the authors conduct controlled experiments to examine the gradient magnitude associated with the ground-truth class and provide a theoretical analysis based on shared features. They also propose Gradient-ray Ensembling, which combines predictions obtained at different learning rates. The method is evaluated on 26 LLMs, 12 text datasets, and 18 image classification models. The authors report average accuracy improvements of 0.60 percentage points with entropy minimization and 0.75 percentage points with Gradient-ray Ensembling.

Critical strengths

S1. Extensive experiments across diverse models and datasets:

The paper evaluates the effectiveness of single-sample optimization on more than 300 model–dataset combinations. The observation of accuracy improvements across diverse vision and language models provides important empirical evidence supporting the potential of the proposed approach.

S2. Interesting analysis of the mechanism underlying Entropy Minimization:

The authors decompose Entropy Minimization into iFo and doFo and show that increasing the logits of likely classes contributes more substantially to accuracy improvements. In particular, the observation that accuracy can be improved simply by increasing the logits of the Top-2 classes simultaneously is interesting.

Critical weaknesses

W1. Lack of logical connections among the individual research contributions:

The paper addresses several topics, including Single-sample TTA, Entropy Decomposition, Top-k Optimization, Gradient Analysis, and Gradient-ray Ensembling. However, it does not sufficiently explain how these topics are connected under a single coherent research question. Although the progression from Entropy Decomposition to the analysis of Top-k Optimization is presented to some extent, it is not sufficiently clear why the paper extends from its initial problem setting of Single-sample TTA to these analyses of the underlying mechanisms, or how each analysis contributes to the central claim of the overall study. In particular, Gradient-ray Ensembling is presented as a separate contribution aimed at addressing learning-rate sensitivity rather than being directly connected to the preceding analyses of the underlying mechanisms. As a result, although the individual empirical findings and methods are interesting, the central research message that they collectively aim to convey remains unclear. This goes beyond an issue of paper organization or presentation, making it difficult to identify the central academic contribution of the study.

W2. Lack of essential comparative experiments for assessing the generalizability and relative utility of the proposed method:

The main experiments focus on Input/Embedding Optimization. Although Section 4.1 presents results for parameter optimization in small neural networks, it remains unclear whether similar results hold for large-scale pretrained models, as this has not been sufficiently evaluated. Therefore, it is unclear whether the observed relative effects of iFo and doFo are preserved in Parameter-based TTA. Furthermore, although the use of only a single sample is emphasized as a key characteristic of the proposed approach, the paper does not directly compare its accuracy and computational cost with existing Multi-sample TTA methods under the same conditions. While the Appendix provides a qualitative comparison with existing methods, it is difficult to quantitatively assess the trade-off between performance and efficiency associated with using a single sample. Since the proposed method requires additional forward and backward passes, it is also important to compare its performance with existing methods under the same inference compute budget. The lack of such comparisons makes it difficult to fully assess the practical advantages and applicability of the proposed method. In particular, these comparative experiments are necessary not merely for additional performance validation, but also for determining what advantages single-sample optimization offers over existing approaches and whether the observed mechanisms remain valid in more general TTA settings. Therefore, the current experimental evidence is insufficient to fully substantiate the generalizability of the proposed approach and its distinct contributions relative to existing methods.

Questions for authors

Q1. Single-sample vs. Multi-sample TTA Could the authors provide quantitative results comparing the accuracy and computational cost of Single-sample TTA and Multi-sample TTA under the same conditions?

Q2. Model Parameter Optimization Could the authors provide quantitative results for optimizing model parameters rather than inputs/embeddings in large-scale pretrained models? In particular, I would like to see a performance comparison among Entropy, iFo, and doFo.

Q3. Computational Efficiency Could the authors compare the performance of the proposed method with existing Test-time Optimization methods under the same inference compute budget?

1 comment
AuthorsOpenReview ↗

Thanks, added experiments and clarified research rationale

Thanks, we worked day and night to address the reviewer's comment.

W1. Lack of logical connections among the individual research contributions: Single-sample TTA, Entropy Decomposition, Top-k Optimization, Gradient Analysis, and Gradient-ray Ensembling. It is not sufficiently clear why the paper extends from its initial problem setting of Single-sample TTA to these analyses of the underlying mechanisms

We added a Figure 2 (in the updated paper), which also adds sub-research questions and shows mathematically and conceptually how we move from Single-Sample TTA optimizing entropy to entropy decomposition to Top-k optimization onto theory and its empirical confirmation. We also added more explanation on Gradient-ray ensembling.

Q1. Single-sample vs. Multi-sample TTA Could the authors provide quantitative results comparing the accuracy (computational cost in Q3)

We showed that both can be applied jointly, which also compares gains of each method individually. But our S-TTA is not in competition with M-TTA. We added a brief statement in the main part of the paper for the main result and the rest in the appendix (D.2). We found that S-TTA + M-TTA (+0.012) leads to larger gains (p<0.001p < 0.001) than M-TTA (+0.01) and S-TTA (+0.007) alone. Thus, our single-sample TTA (S-TTA) is not a competitor to multi-sample TTA (M-TTA), but it can be combined with almost any method to improve predictions (not just M-TTA). Complementarity is also intuitive.

Why M-TTA + S-TTA and not M-TTA vs. S-TTA?

  • We focus on highly uncertain samples only, which might harm continual learning in M-TTA setups and is sometimes not considered.

  • M-TTA updates use small learning rates. We use very high learning rates and reset parameters after every sample. Our extreme updates improve outcomes for one sample.

  • Objective: M-TTA primarily targets continual adaptation under distribution shift, whereas our method does not require a distribution shift and can be applied independently to individual samples.

Q2 + W2. Lack of comparative experiments (tuning only some, not all parameters)

We did a comparison, adding a brief statement in the paper and details in the appendix (D.3), as it is reasonable to ask for one, but we disagree with the reasons the reviewer mentioned, in particular, that generalizability seems to require tuning all input parameters. Earlier M-TTA works also tuned only a few parameters, such as normalization parameters in the well-cited Tent (ICLR 2021). So tuning all parameters is not a requirement for a technique to be general.

Tuning only input (parameter)s leads to significantly larger improvements than tuning all model parameters (p<0.01p < 0.01), though both methods lead to significant gains. We added a statement in the main paper and details in the Appendix.

Q3: Computational efficiency given the same budget

Given many samples, per-sample computation costs are essentially the same for M-TTA (2.6ms +/- 1.71) and S-TTA (2.6ms+/- 1.69) under the same setup. We mention it now briefly in the main part and discuss it in the appendix (D.4). The key point is that both require one forward and one backward pass. But this does not hold for our gradient-ray tracing, which does ensembling to improve accuracy and eliminate the need to tune the learning rate. This comes at a steep cost, i.e., ensembling over xx iterations means xx times more forward passes, so, as a conservative bound, the method becomes xx times slower.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your reviews. The authors have submitted their rebuttal. Could you please take a look and check whether it addresses your questions and concerns? Please leave a comment for the authors, and update your review if your assessment has changed.

Thank you again for your time and effort!

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.