- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes HEAT, a few-shot adaptation framework that replaces a single CLIP-Adapter with an ensemble of adapters of different ranks trained from a shared initialization. It introduces an entropy-based instance-wise weighting scheme and argues that rank variation under shared initialization yields nested adapter subspaces. The claimed result is state-of-the-art average accuracy across eleven benchmarks from four shots onward at a comparable parameter budget.
Critical strengths
The paper is clearly organized and generally well written.
Critical weaknesses
-
The empirical comparison omits directly relevant few-shot CLIP-Adapter ensemble methods, including Soup-Adapter and HOSO-Adapter, both of which address the same adaptation setting and report gains over CLIP-Adapter. Because the claimed margin over CLAP at four shots is small, these omissions materially weaken the empirical case.
-
The central claim that rank variation under shared initialization induces nested subspaces, and that an uncertainty-aware criterion exploits this hierarchy, has been explored in adjacent PEFT work. The authors should clarify what HEAT contributes beyond these works, especially whether nesting is enforced or merely observed.
-
Independent adapter training, rank diversity, shared initialization, and entropy-based confidence weighting are all individually well established. The paper does not provide a constructive mechanism that guarantees nesting, nor does it show through a controlled intervention that the observed nesting is causally responsible for the gains.
-
The aggregation is described as parameter-free, yet the temperature in the entropy softmax is an additional hyperparameter whose sensitivity is not analyzed. The rank allocation is presented without a principled criterion, the shared-initialization procedure is not fully specified, and the subspace-overlap analysis is descriptive rather than causal. The efficiency claim also relies mainly on a single table and would benefit from clearer separation between adapter-only and end-to-end latency.
Improving Robustness of Foundation Models in Domain Adaptation with Soup-Adapters
Hold-One-Shot-Out for Validation-Free Few-Shot CLIP Adapters
Deep Hierarchical Learning with Nested Subspace Networks for Large Language Models
Hierarchical Subspace Adaptation for Efficient Fine-Tuning of Large Language Models
E2LoRA: Efficient and Effective Low-Rank Adaptation with Entropy-Guided Adaptive Sharing
Questions for authors
See Weaknesses
2 comments
- Rank allocation: The ranks {28, 36, 64, 128} were chosen by an ablation on Food101, DTD, and EuroSAT, then held fixed for every dataset and shot. Alternative allocations at the same budget differ by at most 0.42 (Tab. 14).
- Initialization: now fully specified for the bottleneck (G1) and measured on the LoRA A matrices (G2: 42.6% versus 7.0% at 16 shots).
- Efficiency: Tab. 3 separates the two. Adapter-only, HEAT takes 1.66 µs against 0.61 µs for one adapter; end-to-end, 238.9 µs against 232.4 µs (+2.8%). We have corrected "no extra inference time".
W1. Soup-Adapter and HOSO-Adapter. Both are now cited and discussed.
HOSO-Adapter (published, RN50, 11 datasets): 63.03 / 65.13 / 71.78 / 75.25 at 2 / 4 / 8 / 16 shots, against 65.85 / 69.21 / 72.44 / 75.47 for HEAT (Tab. 1). HEAT is ahead at 2 shots (+2.82). The methods are close at 8 and 16 shots (+0.66 and +0.22). The 4-shot gap is +4.08, of which most is HOSO’s EuroSAT result of 35.00.
Soup-Adapter averages N independently trained full-size adapters (N× parameters) with uniform weights. We could not finish a faithful reproduction in the rebuttal period. The closest controls at equal budget are equal-rank ensembles (Tab. 4) and mean aggregation (App. G); HEAT is +0.25 to +0.98 above the former and +0.2 to +0.7 above the latter.
W2. Nested subspaces in prior PEFT work; enforced or observed? Nested Subspace Networks and DyLoRA enforce nesting inside one module, by re-parameterisation or rank truncation, to obtain a single model whose rank can be changed at inference. Hierarchical Subspace Adaptation (HSSA; Dong et al., MIND 2025) enforces a parameter-sharing hierarchy across the modules of an LLM. Each LoRA update is factorised as ΔW = v′_m B S′_n v_m S_n A, with A and B shared by all modules (as in VeRA), projections S_n, S′_n specific to each module type, and per-unit scaling vectors. Its ranks are set by a NAS-style search, and it also yields a single fine-tuned model. E²LoRA uses a gradient-based proxy entropy to decide layer sharing and rank allocation, which is unrelated to test-time predictive entropy. In HEAT the adapters are independent networks on CLIP’s output embeddings. They share no parameters, and their ranks are fixed rather than searched. Nesting is seeded by the initialization and not enforced afterwards (G1). It is used for ensembling, with the adapters’ predictions combined per sample by entropy weighting, rather than for rank elasticity or parameter sharing. We thank the reviewer for the HSSA reference.
W3. Constructive mechanism and controlled intervention. The constructive mechanism is the shared seed: it makes the adapters exactly nested at step 0 (G1), and decorrelating the initializations removes the structure monotonically (θ sweep: 28.9% → 4.7% shared fraction at 16 shots). LoRA A matrices give the same contrast: 42.6% for a shared seed and 7.0% for independent seeds at 16 shots, with equal movement 0.800 (G2). For accuracy, the HEAT ranks from a shared seed give the best ensemble at every shot: +0.33 over independently seeded equal ranks (p < 1e-9) and +0.46 over the single adapter (p < 1e-10), pooled over all five shots (G2). With bottleneck adapters, HEAT also outperforms equal-rank ensembles at equal budget at every shot (+0.25 to +0.98, Tab. 4). .
Shared spectral fraction (%) of the bottleneck down-projections (from G1):
HEAT-LoRA accuracy, 10 datasets (from G2; K=8 includes one dataset with two seeds):
Shared spectral fraction (%) of the concatenated LoRA A matrices (from G2):
W4. Temperature, rank allocation, initialization, efficiency.
β between 0.5 and 1.4 stays within 0.18 of β = 1. β = 0.1 is 0.14–0.44 below. The two limits are in App. G: mean aggregation (β → ∞) is 0.2–0.7 below, and most-confident selection (β → 0) is 3.5–5.3 below. We add these values to the appendix and replace "parameter-free" with "no learned parameters".