acceptodds
Withdrawn from ICLR 2027

Hierarchical Ensemble Adapter Tuning for Few-Shot Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Parameter-efficient few-shot adaptation usually trains one lightweight adapter of fixed bottleneck rank. That single rank commits adaptation to one hypothesis, which often lacks the robustness required for few-shot scenarios. In this paper, we propose Hierarchical Ensemble Adapter Tuning (HEAT), a framework that trains an ensemble of lightweight adapters with varying bottleneck ranks, each imposing a different structural inductive bias. Based on multiple adaptation hypotheses, HEAT expands the model's representational capacity by learning nested, rank-constrained features: a shared core of dominant task directions that each higher-rank adapter refines. To leverage this hierarchical structure at inference time, we introduce an uncertainty-aware aggregation strategy that dynamically weights each adapter's output according to its instance-wise predictive entropy. Analysis of the learned representations reveals that the adapters form nested subspaces where each of them sharing a common core subspace and that entropy weighting systematically exploits this structure, amplifying the most confident adapter on a per-sample basis. Across eleven few-shot benchmarks, HEAT achieves state-of-the-art accuracy all within the same parameter budget as a single CLIP-Adapter.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
adapter, few-shot adaptation, PEFT, ensembling

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_NQhNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HEAT, a few-shot adaptation framework that replaces a single CLIP-Adapter with an ensemble of adapters of different ranks trained from a shared initialization. It introduces an entropy-based instance-wise weighting scheme and argues that rank variation under shared initialization yields nested adapter subspaces. The claimed result is state-of-the-art average accuracy across eleven benchmarks from four shots onward at a comparable parameter budget.

Critical strengths

The paper is clearly organized and generally well written.

Critical weaknesses
  1. The empirical comparison omits directly relevant few-shot CLIP-Adapter ensemble methods, including Soup-Adapter and HOSO-Adapter, both of which address the same adaptation setting and report gains over CLIP-Adapter. Because the claimed margin over CLAP at four shots is small, these omissions materially weaken the empirical case.

  2. The central claim that rank variation under shared initialization induces nested subspaces, and that an uncertainty-aware criterion exploits this hierarchy, has been explored in adjacent PEFT work. The authors should clarify what HEAT contributes beyond these works, especially whether nesting is enforced or merely observed.

  3. Independent adapter training, rank diversity, shared initialization, and entropy-based confidence weighting are all individually well established. The paper does not provide a constructive mechanism that guarantees nesting, nor does it show through a controlled intervention that the observed nesting is causally responsible for the gains.

  4. The aggregation is described as parameter-free, yet the temperature in the entropy softmax is an additional hyperparameter whose sensitivity is not analyzed. The rank allocation is presented without a principled criterion, the shared-initialization procedure is not fully specified, and the subspace-overlap analysis is descriptive rather than causal. The efficiency claim also relies mainly on a single table and would benefit from clearer separation between adapter-only and end-to-end latency.

Improving Robustness of Foundation Models in Domain Adaptation with Soup-Adapters

Hold-One-Shot-Out for Validation-Free Few-Shot CLIP Adapters

Deep Hierarchical Learning with Nested Subspace Networks for Large Language Models

Hierarchical Subspace Adaptation for Efficient Fine-Tuning of Large Language Models

E2LoRA: Efficient and Effective Low-Rank Adaptation with Entropy-Guided Adaptive Sharing

Questions for authors

See Weaknesses

2 comments

W1. Soup-Adapter and HOSO-Adapter. Both are now cited and discussed.

  • HOSO-Adapter (published, RN50, 11 datasets): 63.03 / 65.13 / 71.78 / 75.25 at 2 / 4 / 8 / 16 shots, against 65.85 / 69.21 / 72.44 / 75.47 for HEAT (Tab. 1). HEAT is ahead at 2 shots (+2.82). The methods are close at 8 and 16 shots (+0.66 and +0.22). The 4-shot gap is +4.08, of which most is HOSO’s EuroSAT result of 35.00.

  • Soup-Adapter averages N independently trained full-size adapters (N× parameters) with uniform weights. We could not finish a faithful reproduction in the rebuttal period. The closest controls at equal budget are equal-rank ensembles (Tab. 4) and mean aggregation (App. G); HEAT is +0.25 to +0.98 above the former and +0.2 to +0.7 above the latter.

W2. Nested subspaces in prior PEFT work; enforced or observed? Nested Subspace Networks and DyLoRA enforce nesting inside one module, by re-parameterisation or rank truncation, to obtain a single model whose rank can be changed at inference. Hierarchical Subspace Adaptation (HSSA; Dong et al., MIND 2025) enforces a parameter-sharing hierarchy across the modules of an LLM. Each LoRA update is factorised as ΔW = v′_m B S′_n v_m S_n A, with A and B shared by all modules (as in VeRA), projections S_n, S′_n specific to each module type, and per-unit scaling vectors. Its ranks are set by a NAS-style search, and it also yields a single fine-tuned model. E²LoRA uses a gradient-based proxy entropy to decide layer sharing and rank allocation, which is unrelated to test-time predictive entropy. In HEAT the adapters are independent networks on CLIP’s output embeddings. They share no parameters, and their ranks are fixed rather than searched. Nesting is seeded by the initialization and not enforced afterwards (G1). It is used for ensembling, with the adapters’ predictions combined per sample by entropy weighting, rather than for rank elasticity or parameter sharing. We thank the reviewer for the HSSA reference.

W3. Constructive mechanism and controlled intervention. The constructive mechanism is the shared seed: it makes the adapters exactly nested at step 0 (G1), and decorrelating the initializations removes the structure monotonically (θ sweep: 28.9% → 4.7% shared fraction at 16 shots). LoRA A matrices give the same contrast: 42.6% for a shared seed and 7.0% for independent seeds at 16 shots, with equal movement 0.800 (G2). For accuracy, the HEAT ranks from a shared seed give the best ensemble at every shot: +0.33 over independently seeded equal ranks (p < 1e-9) and +0.46 over the single adapter (p < 1e-10), pooled over all five shots (G2). With bottleneck adapters, HEAT also outperforms equal-rank ensembles at equal budget at every shot (+0.25 to +0.98, Tab. 4). .

Shared spectral fraction (%) of the bottleneck down-projections (from G1):

Shared spectral fraction (%) of down-projectionsInitK=1K=4K=16
HEAT ranks, shared seed51.039.533.728.9
HEAT ranks, independent seeds2.12.43.24.7
4×64, independent seeds1.62.43.55.3

HEAT-LoRA accuracy, 10 datasets (from G2; K=8 includes one dataset with two seeds):

HEAT-LoRAK=124816
Single r=25660.2363.6467.2471.8676.63
4×64, independent seeds59.8763.8467.4672.1376.98
HEAT ranks, independent seeds59.8963.8967.6172.3377.05
HEAT ranks, Shared seed60.2664.0367.8772.5077.26

Shared spectral fraction (%) of the concatenated LoRA A matrices (from G2):

Shared spectral fraction (%)K=124816
HEAT ranks, shared seed49.948.847.345.442.6
HEAT ranks, independent seeds5.15.25.56.17.0
4×64, independent seeds5.65.76.06.57.6

W4. Temperature, rank allocation, initialization, efficiency.

  • Temperature: β = 1 for every dataset and shot and it was not tuned. Gain or loss in average accuracy relative to β = 1, on 10 datasets:
βK=124816
0.1−0.16−0.44−0.26−0.19−0.14
0.5−0.08−0.18+0.02+0.02+0.02
100000
1.2−0.07−0.10+0.04−0.01+0.03
1.4−0.09−0.10+0.05−0.01+0.02

β between 0.5 and 1.4 stays within 0.18 of β = 1. β = 0.1 is 0.14–0.44 below. The two limits are in App. G: mean aggregation (β → ∞) is 0.2–0.7 below, and most-confident selection (β → 0) is 3.5–5.3 below. We add these values to the appendix and replace "parameter-free" with "no learned parameters".

  • Rank allocation: The ranks {28, 36, 64, 128} were chosen by an ablation on Food101, DTD, and EuroSAT, then held fixed for every dataset and shot. Alternative allocations at the same budget differ by at most 0.42 (Tab. 14).
  • Initialization: now fully specified for the bottleneck (G1) and measured on the LoRA A matrices (G2: 42.6% versus 7.0% at 16 shots).
  • Efficiency: Tab. 3 separates the two. Adapter-only, HEAT takes 1.66 µs against 0.61 µs for one adapter; end-to-end, 238.9 µs against 232.4 µs (+2.8%). We have corrected "no extra inference time".
Reviewer_vKqeOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the problem of few-shot adaptation using CLIP. It proposes a variant of CLIP-Adapter [A] where, instead of learning a single adapter, multiple adapters with different bottleneck dimensions are learned independently. At inference, the adapters’ predictions are ensembled and weighted based on the entropy of predictions (giving more weight to more confident adapters and vice versa). Experiments on few-shot classification are performed across 11 datasets and performance is compared to prompt learning (CoOp), adapters (CLIP-Adapter, TaskRes) and linear probing (CLAP, cross-modal LP), showing an advantage compared to CLAP starting from K=4 shots. The domain generalization experiment is conducted on 8-shot ImageNet (with ViT-B/16 as backbone) and the proposed HEAT is compared to CLIP-Adapter showing large improvements. Ablations show the benefits of starting from the same seed and having unequal ranks of adapters, and the paper includes an analysis supporting how HEAT induces a nested subspace structure.

[A] Gao et al., CLIP-Adapter: Better Vision-Language Models with Feature Adapters. IJCV 2024.

Critical strengths

The method appears to improve CLIP-Adapter with a very small full-pass overhead while retaining the same number of parameters.

Critical weaknesses

While the paper presents improvements compared to CLIP-Adapter, there are several concerns related to the experiments. First of all, for the few-shot classification results (Table 1), the method underperforms CLAP [B] on K=1 and K=2 shots, and only slightly improves the performance for K=4 and K=8 (respectively 0.1 and 0.3 percentage points gain compared to CLAP). This problem is amplified by the fact the CLAP results are reported from the corresponding paper, and it is not clear whether all methods use the same support sets across seeds. It is also crucial to include the standard deviation since improvements are marginal, to assess whether it is really significant. One other concern is that the paper lacks comparison to other methods that outperform CLAP and that have established new state-of-the-art, for example: CLIP-LoRA [C], ProLIP [D], 2SFS [E]. Comparison with CLIP-LoRA and 2SFS requires showing the results of Table 1 using a ViT backbone, and ProLIP already outperforms both CLAP and HEAT for ResNet-50.

Another major concern is about the claim in lines 155-156 suggesting that using a single adapter hurts domain generalization, which is supported in Table 2 showing significantly lower domain generalization results with CLIP-Adapter compared to HEAT. I am skeptical of such a claim. How was CLIP-Adapter implemented in this setting? What is the weight α\alpha that is used? In other words, using a low α\alpha value (relying more on the zero-shot features and less on the adapted features), for instance α=0.1\alpha=0.1, might bring the results very close to zero-shot CLIP. This problem is coupled with the fact that comparing to CLIP-Adapter on 1 dataset using the single 8-shot setting is not sufficient to assess the importance of HEAT on domain generalization. For instance, [D] appears to report stronger results on this setting both on ImageNet and its out-of-distribution counterparts while using less data (K=4 instead of K=8). It is also crucial to discuss and position the proposed method with respect to methods like MaPLe [F], MMA [G], MMRL [H], VaMP [I]...

[B] Silva-Rodríguez et al., A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models. CVPR 2024
[C] Zanella & Ben Ayed, Low-Rank Few-Shot Adaptation of Vision-Language Models. CVPRW 2024
[D] Fahes et al., CLIP's Visual Embedding Projector is a Few-shot Cornucopia. WACV 2026
[E] Farina et al., Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages. CVPR 2025
[F] Khattak et al., MaPLe: Multi-modal Prompt Learning. CVPR 2023
[G] Yang et al., Mma: Multi-modal adapter for vision-language models. CVPR 2024
[H] Guo & Gu. MMRL: Multi-Modal Representation Learning for Vision-Language Models. CVPR 2025
[I] Cheng & Han. VaMP: Variational Multi-Modal Prompt Learning for Vision-Language Models. NeurIPS 2025

Questions for authors
  • The authors state that α\alpha is kept fixed and initialized with 1- (Acczs/100Acc_{zs}/100) (Lines 174-175). While this is a clever way to automatically adjust the contribution of zero-shot features, was it also applied to CLIP-Adapter?

  • Could the authors clarify how they ensure the initialization of all adapters is the same, given that these adapters have different ranks?

  • Another variant of CLIP-Adapter called "regularized linear adapter (RLA)" was proposed in [D], and it appears to outperform CLIP-Adapter. How does HEAT perform compared to this approach?

  • Is there any specific reason ImageNet-V2 is not included in the domain generalization experiment?

Suggestions

  • The title would be more informative if it mentions that CLIP is used for few-shot adaptation. Suggestion: Hierarchical Ensemble Adapter Tuning for CLIP-based Few-Shot Adaptation.

Minor

Line 081: face -> faces

2 comments

W1. Same support sets, standard deviation, margin over CLAP. All runs use the CoOp few-shot splits with seeds 1–3, the protocol CLAP follows; the CLAP numbers are from its paper. Per-dataset std is in App. Tab. 8; averaged over the 11 datasets (mean accuracy ± mean per-dataset std over seeds):

We share the table below with the mean per-dataset standard deviation over seeds 1–3, for all shots (11 datasets, from App. Tab. 8).

MethodK=124816
CLAP62.79±0.5866.07±0.5869.13±0.5472.08±0.6074.57±0.28
HEAT62.18±0.9065.85±0.5169.21±0.4572.44±0.5375.47±0.37

We agree that the margins over CLAP at K=4 and K=8 (+0.1 and +0.3) are small. Small average margins are typical in this setting, because with only a few labelled images per class the gains between strong methods are bounded: CLAP improves over TaskRes by +0.2 to +0.7 at K ≥ 4 (Tab. 1), and ProLIP + RLA improves over ProLIP by +0.05 to +0.11 (Fahes et al.). We have softened our claims to match the results and emphasise the fixed-budget gains over a single adapter and over equal-rank ensembles (G2).

W1. ProLIP, CLIP-LoRA, 2SFS. Thank you; all are now included.

  • ProLIP. Published validation-free ProLIP (11 datasets): 64.33 / 67.19 / 69.94 / 72.82 / 75.46, against 62.18 / 65.85 / 69.21 / 72.44 / 75.47 for HEAT (Tab. 1). ProLIP is ahead at K ≤ 8 and the two are level at K=16. ProLIP retrains CLIP's visual projection (2.1M weights, needs pre-projection features); HEAT is a 0.52M adapter on output embeddings. The two are complementary: on our splits (8 datasets), adding HEAT's logits to ProLIP's improves ProLIP by +0.55, +0.51 and +0.58 at K = 4, 8 and 16 (p < 1e-4).

  • 2SFS and CLIP-LoRA tune parameters inside the encoders and are ViT-only. On ViT-B/16 (new, 9 datasets, 3 seeds), HEAT reaches 76.55 / 79.59 / 82.70 at K = 4 / 8 / 16, against published 2SFS at 79.86 / 82.16 / 84.61 on the same datasets. HEAT trails encoder-tuning methods by 2–3 points while never back-propagating through the backbone. We will report this table.

W2. CLIP-Adapter in the domain-generalization table. It was run with the official implementation and its default ImageNet setting. The concern is fair: in-distribution, the residual ratio alone changes a single adapter by up to 5 points (table under Q1; G2). We will add CLIP-Adapter with our α rule to Tab. 2 to separate the effect of α from that of the ensemble.

W3. MaPLe, MMA, MMRL, VaMP. These learn prompts or adapters inside both encoders and target base-to-new generalization on ViTs. HEAT is a black-box adapter on frozen embeddings and is orthogonal to where adaptation happens. We have added this positioning; it was missing.

Q1. Was the α rule applied to CLIP-Adapter? No. Tab. 1 used CLAP's reported CLIP-Adapter numbers (fixed α = 0.2). With our α rule (taken from CLAP), a single r=256 adapter reaches 62.47 / 65.98 / 69.28 / 72.86 / 76.31 on 10 datasets. HEAT (App. Tab. 13) is above it at every shot. We add this baseline to Tab. 1 and correct Sec. 4.2.

Single adapter with our α rule, 10 datasets (from G2):

MethodK=124816
Single r=256, α=0.2 (new)61.0462.0764.0768.1973.19
Single r=256, our α rule (new)62.4765.9869.2872.8676.31
HEAT (App. Tab. 13)62.5366.5370.1573.4776.60

Q2. Same initialization across ranks. See G1: with the Shared seed, each smaller down-projection is an exact row prefix of the larger ones. The LoRA measurement is in G2: shared-seed trained HEAT ranks keep 42.6% at 16 shots, and independent seeds keep 7.0%.

Shared spectral fraction (%) of the trained bottleneck down-projections (from G1):

Shared spectral fraction (%) of down-projectionsInitK=1K=4K=16
HEAT ranks, shared seed51.039.533.728.9
HEAT ranks, independent seeds2.12.43.24.7
4×64, independent seeds1.62.43.55.3

Shared spectral fraction (%) of the concatenated traiend LoRA A matrices (from G2):

Shared spectral fraction (%)K=124816
HEAT ranks, shared seed49.948.847.345.442.6
HEAT ranks, independent seeds5.15.25.56.17.0
4×64, independent seeds5.65.76.06.57.6

Q3. RLA. Published RLA (11 datasets, 1.05M parameters): 63.41 / 65.80 / 68.27 / 70.91 / 73.35. HEAT (Tab. 1, 0.52M): 62.18 / 65.85 / 69.21 / 72.44 / 75.47. RLA is ahead at 1 shot; HEAT is level at 2 shots and ahead from 4 shots (+2.1 at 16) with half the parameters.

Q4. ImageNet-V2. We apologize for that. Because of compute related restrictions, we were unable to finish it in time. We have now tried to evaluate our model on imagenet v2. Imagenet v2 ZSL accuracy is 60.76 while HEAT is at 61.40 which gives it an improvement of 0.64.

Title and typo. Few-shot problem is not well defined within MLLM domain which leaves us with CLIP based methods to validate our hypotheses. However, we have Adopted the title and fixed the typo.

Reviewer_CpgVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an ensemble based few-shot learner. Essentially, instead of relying on a single adapter with fixed capacity (or in the terminology used by the authors, rank), the authors propose to train a collection of adapters with varying ranks. Specifically, adapters of different ranks can learn and adapt to different features and levels of details for various tasks. To aggregate the adapters' outputs, the authors propose an instance-wise entropy-based weighted combining.

Critical strengths

The notable strength of the paper comes from the empirical performance. Specifically, under fixed parameter count, the proposed method HEAT achieves the top classification accuracies on 4-or-above shot settings, and stays competitive for cross domain generalization.

Critical weaknesses
  1. The novelty of the method is very limited. Its essentially a normal ensemble method based on the prior CLIP-Adapter work. The authors provide theoretical discussion in Proposition 1, to support the claim that adapters with different ranks should learn hierarchical subspaces. However, the theoretical discussions rely heavily on classical low-rank matrix approximation, assuming linear adapters without ReLU or other nonlinearity, leaving the gap very noticeable between theory and the actual method trained adapters.
  2. The authors advocate shared initialization for a common target operator, however each adapter has different ranks, and thus different optimation landscapes. It is highly questionable that what author claimed about the shared initialization would gurantee the claimed convergence.
Questions for authors

One interesting point the authors advocate is, as oppsoed to traditional ensemble methods that encourage diversity among individual learners, the proposed HEAT exploits structural similarity. The results show higher subspace overlap lead to better accuracy, which is a bit counter-intuitive than the conventional ensemble idea. Can the authors provide a theoretical justification, or intuition on the reason behind this?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.