- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper asks whether prediction accuracy can be improved by adjusting the input or its embeddings using only the current test sample, without labels, augmentations, or other samples. The authors compare entropy minimization with increasing likely-class outputs and decreasing unlikely-class outputs, and investigate gradient magnitudes and shared features as possible explanations. However, I find the evidence for the proposed mechanisms and their connection to the main empirical results insufficient.
Critical strengths
-
The question is worth studying, particularly whether input or embedding optimization can improve predictions without additional supervision.
-
The evaluation covers many models and datasets, and the authors report negative results rather than presenting only favorable cases.
-
Decomposing entropy minimization into increasing likely-class outputs and decreasing unlikely-class outputs offers a useful perspective.
Critical weaknesses
-
The main evaluation uses LLMs and ImageNet classifiers, but the gradient analysis relies on small EMNIST networks and samples whose true class is in the top two. Whether this mechanism explains the main results remains unverified.
-
The theory relies on restrictive assumptions whose relevance to real models is not established. The shared-feature explanation remains a conjecture without direct measurements or feature interventions.
-
The theory mainly analyzes unweighted top-two logit maximization, while entropy minimization performs best experimentally. The connection between these objectives is insufficiently established.
-
A trivial solution to single-sample entropy minimization does not rule out benefits from finite-step updates. Framing the results as overturning prior understanding is therefore overstated.
-
The method illustrations and experimental figures are insufficiently polished. Clearer explanations and more self-contained captions are needed to make the results easy to assess.
Questions for authors
See the weaknesses above.
3 comments
Response to the Rebuttal
Thank you for the response and revisions. I appreciate the efforts to clarify the claims and improve the presentation. However, removing claims and moving analyses to the appendix do not, by themselves, resolve the concerns about the connection between the theory and the main empirical results. The stated additions would need to establish that connection convincingly.
The response still leaves me with the impression that the manuscript was not sufficiently developed at submission. The extensive revisions to its central explanation reinforce this concern. Obviously, this paper was not sufficiently prepared.
Overall, I remain unconvinced that the main concerns have been adequately addressed, and I maintain my original score.
Thanks for your assessment.
removing claims and moving analyses to the appendix do not, by themselves, resolve the concerns about the connection between the theory and the main empirical results. The extensive revisions to its central explanation reinforce this concern.
- The main empirical result stated in the abstract is that incrementing classes is more valuable than decrementing. The latter is often associated with "noise" removal. There is no theory related to incrementing likely classes is better than decrementing unlikely.
- We don't consider a conjecture the central explanation but rather the theorem that was already underpinned by empirical evidence in the initial version, which was connected through the gradient magnitude mentioned in the abstract . Eq. (22) connects the change in the logit margin to the difference in squared input-gradient norms.
- This connection was reinforced in the new version by adding Figure 2, adding more text and also adding more experiments. These additions go beyond removing claims or relocating analyses.
Removed non-essential theory and ensured easier to see linking of theory and experiments
We removed the claim on gradient analysis from the abstract and moved the entire controlled experiment to the Appendix. We also removed some theorems to make the essential theorem stand out more clearly. We directly state the empirical evidence using LLMs (rather than small EMNIST networks) for the main theorem after the theory section. We disagree with the implied demand that a controlled experiment should explain particular LLM classifiers. By definition, a controlled experiment aims at deeper insights rather than verifying the behavior of the exact LLM architectures.
The concern that the correct class is among the top-2 is unwarranted as we aim to understand when the method helps and when it hurts. It can only hurt if the prediction is correct, i.e., it is among the top-1 and thus by definition among the top-2. If the true class is not among the top-2 but among the top-k (k>2), our method can only help. If the true class is not among the top-2, the method much more often neither hurts nor helps, so we left out these cases to reduce computation.
The general theory "General Theory for input tuning", already in the initial submission, derives the margin dynamics for arbitrary differentiable neural networks. We removed the shared feature conjecture and other distracting analysis and now focus on the key proven mechanism that is also established empirically on LLMs: What the theory shows is that (input) gradient sensitivity is a key factor.
We updated the paper with a new Figure 2. It clearly shows in words and mathematically the through simplification and problem isolation how we move from entropy minimization all the way to top-2 logit maximation.
We rephrased. Abstract: "The trivial solution of single-prediction entropy minimization motivated prior work to avoid optimizing predictions individually"
Intro: "entropy minimization has a trivial solution confirming just the most likely class, motivating forgoing single sample optimization in the favor of batches"
For reference the original paper stated: "Note that optimizing a single prediction has a trivial solution: assign all probability to the most probable class. We prevent this by jointly optimizing batched predictions over parameters that are shared across the batch."
We updated all descriptions and reworked all illustrations in the updated manuscript.