acceptodds
Withdrawn from ICLR 2027

Training-Free Few-Shot Adaptation via Exponential Tilting of Latent Representations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Adapting frozen foundation models to new few-shot tasks is often infeasible when parameters cannot be modified e.g., under shared checkpoints or deployment policies barring mutation. We study few-shot classification under this fully frozen-model regime: no gradients, no parameter updates, no upstream data. We show that under these constraints, adaptation reduces to choosing a weighting over the support embeddings used to form class prototypes, and that exponential (Gibbs) tilting is the unique such weighting minimizing KL divergence from the empirical support distribution subject to a task-relevance constraint. We instantiate this constraint with simple support-set scores label consistency, geometric alignment, or classifier confidence yielding closed-form tilted prototypes with no backward pass or iterative solving. Across three frozen encoders (DINOv3, I-JEPA, CLIP) and several few-shot and cross-domain benchmarks, tilted prototypes improve over uniform-average baselines in most settings and are competitive with training-free adaptation methods that use strictly more information (query streams, upstream statistics), while gradient-based fine-tuning remains an upper bound under weaker constraints. This positions principled latent-distribution reweighting as a lightweight, gradient-free alternative to parameter-based few-shot adaptation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
foundation models, tilting, latent representation, few-shot adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_7c26OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper formulates few-shot test-time adaptation under fully frozen-model constraints as a change-of-measure problem over the latent distribution and proposes exponential tilting as its closed-form solution. The method reweights support-set embeddings to correct predictive marginals without parameter updates, gradients, or iterative optimization. Experiments across three frozen encoders and several few-shot benchmarks report consistent gains over uniform-average baselines.

Critical strengths

The paper is clearly written and professionally presented. The problem setting is compact, and the derivation from the constrained optimization problem to exponential tilting is complete and easy to follow.

Critical weaknesses
  1. The comparison table omits multiple 2025 works at the same constraint level, including FreeTTA, UL-TTA, PAF, and ProtoMM. FreeTTA explicitly models the test distribution online in a training-free manner, and UL-TTA updates prototypes and class priors through closed-form Bayesian steps. Both are operationally close to the present method. A table covering only seven baselines and missing several direct competitors cannot support the claim that the method outperforms every constraint-compliant baseline.

  2. A paper already used exponential tilting to learn training-data importance weights under distribution shift by minimizing the divergence between source and target data. The present paper does not discuss this closer precedent, nor does it explain whether transferring the reweighting target from training samples to test-time latent embeddings constitutes a substantive conceptual advance.

  3. The core operation assembles exponential tilting with normalized exponential weighting, both classical tools. The confidence-only score trails the label-aware score by only one to two percentage points in most settings. If a simpler reweighting scheme recovers a substantial fraction of the gain, the incremental value of the exponential-tilting framework over plain weighted prototypes requires stronger justification.

  4. The theorem numbering in the main text and appendix is inconsistent, and the main-text equation said to follow the appendix theorem is not explicitly derived from that theorem. The recommended tilting strength is justified circularly: the main text points to a section whose entropy-based heuristic is never evaluated in the main experiments. The ablation table repeats column headers, making the mapping between columns and shot counts ambiguous. The transductive variant uses query embeddings to construct the reference measure but is reported alongside inductive baselines as achieving the highest accuracy, which is not a like-for-like comparison.

  • Enhancing Flexibility in Test-Time Adaptation with Online EM
  • Ultra-Light Test-Time Adaptation for Vision Language Models
  • PAF: Prototype Adaptive Fusion for Test-Time Adaptation of Vision-Language Models
  • Dynamic Multimodal Prototype Learning in Vision-Language Models
  • Understanding new tasks through the lens of training data via exponential tilting
  • Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization
  • NEO: No-Optimization Test-Time Adaptation through Latent Re-Centering
  • Optimization-Free Test-Time Adaptation for Cross-Person Activity Recognition
  • DOTA: Distributional Test-Time Adaptation of Vision-Language Models
  • Ranked Entropy Minimization for Continual Test-Time Adaptation
  • I-divergence geometry of probability distributions and minimization problems -Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling
Questions for authors

See Weaknesses

Reviewer_Lgd2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents an approach to few-shot classification that does not require training and makes use of frozen encoders. It assigns exponential weights to the support embeddings by means of label consistency, geometric alignment, or classifier confidence and then creates adapted class prototypes for classifying the query samples. The reason for this weighting is that it represents a change of measure which minimises the KL divergence subject to a task-relevance constraint. The experiments include the use of DINOv3, I-JEPA, and CLIP on standard few-shot and cross-domain benchmarks.

Critical strengths
  • It offers practical motivation and a simple implementation. This approach of adapting frozen representations without the need for backpropagation or access to the pretraining data is suitable for situations involving limited computational resources or immutable checkpoints.
  • A simple and easy to understand mechanism; by using exponential weighting we are able to clearly give more weight to examples that receive high reliability scores.
  • It has extensive empirical coverage; the evaluation involves three encoders that were pretrained using different objectives and a number of classification benchmarks. In some cases the improvements are considerable, for example the increase from 91.03 per cent to 94.85 per cent on the DINOv3 CIFAR-FS 4-shot classification task.
Critical weaknesses
  1. The constraint given leads to a trivial solution. Since in equation (2) the target expectation is equal to the average score obtained under the uniform empirical support distribution, this average is then taken as P0P_0. As a result, P0P_0 already fulfils the constraint and has a zero KL divergence. It follows that the optimum of equation (3) is P⋆=P0P^\star = P_0, not the non-uniform distribution that was used in the experiments. The relationship between the constrained formulation and the fixed positive tilting strength must therefore be corrected.
  2. The rule based on theoretical prediction does not agree with the actual classifier. Equations (8) and (10) specify a marginal prediction that is independent of the query, whereas the implementation classifies the queries by referring to recomputed prototypes. The paper fails to show that these two methods are equivalent, and Theorem .3 only applies to input-independent marginal predictors. It thus cannot justify the real few-shot classifier.
  3. The experiments do not separate out the effect of exponential tilting; even when λ=0\lambda=0, the proposed soft-assignment prototype construction is still different from the original class-mean baseline. In order to find out whether the improvements are due to exponential weighting or to the change in prototype construction, a comparison matching those involving soft prototype refinement without tilting is necessary.
  4. There are issues that need to be resolved regarding several of the baseline results. Although ordinary positive-temperature scaling maintains the argmax of the fixed logits, Table 2 shows different classification accuracies. Furthermore, the method referred to as Ranked Entropy Minimization involves parameter optimization, which contradicts how it is listed in the table. These inconsistencies reduce confidence in the comparison procedure.
  5. The conclusions go beyond what can be claimed with regard to cross-domain robustness. As shown in Table 8, there is a considerable decline, for example a drop from 71.19 per cent to 59.04 per cent for I-JEPA on Quickdraw when using ten shots. These findings contradict the assertion of consistent improvement and therefore require a detailed analysis of the failures.
  6. The novelty is not adequately established; exponential tilting under KL constraints is a well-known technique. The paper should provide a more clear distinction from existing training-free weighting and prototype-refinement methods, together with more substantial comparisons and ablation studies at the component level.
Questions for authors
  • As the expected value given by equation (2) is equal to EP0[s]\mathbb{E}_{P_0}[s] with respect to the empirical reference distribution, why isn't the KL-minimizing solution just P⋆=P0P^\star = P_0? What constraint is satisfied by the experimental choice of λ=0.25\lambda = 0.25?

  • Regarding the connection between theory and implementation: while equations (8) and (10) yield predictions that are independent of the query, the implementation relies on the similarities between the query and the prototype. Could you please give the complete rule for the query-conditioned prediction and explain which of the theoretical results apply to it?

  • What is the performance achieved by the proposed soft-assignment prototype construction at λ=0\lambda=0, when using the same episodes? How much extra gain is due specifically to the exponential weighting?

  • Why does the top-1 accuracy change in Table 2 with ordinary temperature scaling in the baseline implementation? And since the cited Ranked Entropy Minimization method involves parameter updates, which implementation of Ranked is the one that has been evaluated?

  • What can account for the significant decline observed on several DomainNet tasks, especially I-JEPA on Quickdraw when using only ten shots? Does this affect the validity of the claims regarding robust cross-domain adaptation?

  • Can you share executable pseudocode or code, the set classifier temperature, details about covariance regularization, and confidence intervals for paired accuracy differences across shared evaluation episodes?

Reviewer_kwMNOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors look at few-shot image classification when the encoder and a nearest-prototype classifier's gradients are frozen. They apply a previous work, exponential tilting, to "reweight the frozen encoder’s latent distribution at inference". They run comparison experiments with DINOv3, I-JEPA and CLIP on CIFAR-FS, CUB and mini-ImageNet, one comparison with training-free methods, lambda and score ablations, and cross-domain benchmarks.

Critical strengths
  • Gains for all encoders in the domains tested. Shows that the method helps with the 3 encoders presented.
  • Method is simple and computationally cheap to implement. This reduces the cost to reproduction and means fewer barriers to applying this in other settings
Critical weaknesses
  • Wide error bands appear to overlaps significantly in much of the data presented. It's not obvious the presented results are significant or real. The overlap is also not thin, it appears to be many orders of overlap in some cases.
  • It's unclear if the method generalizes across domains based on the evidence presented. This limits the usefulness and application of the method.
  • The theory presented seems to suggest a setting which isn't fully empirically tested.
Questions for authors
  • Why is lambda 0.25 and not 0 as the theory presented seems to suggest is ideal?
  • Was lambda chosen without using the test episodes?
  • What temperature is used?
  • How were the Table 2 baselines produced?
  • Why do so many of the cross-domain 1-shot cells report identical standard deviations for the baseline and the tilted method?
  • Why do the numbers in Tables 6-8 not appear to support the cross-domain claims?
  • Why does the method hurt more on DomainNet as the number of shots grows?
  • How did you construct the cross-domain episodes?
  • what's driving the gain?
  • can you show that the gains are greater than chance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.