acceptodds
Under review as a conference paper at ICLR 2027

Exemplar-Derived Soft Prompts for Efficient and Effective Domain Adaptation

Abstract

Adapting foundation models to new domains is challenging and computationally expensive. While parameter-efficient fine-tuning (PEFT) methods allow models to acquire domain-specific skills, they require updating deployed models and incur additional training and deployment overhead. In contrast, In-context Learning (ICL) avoids model updates and improves over off-the-shelf models by leveraging similar exemplars, but it often fails to achieve competitive accuracy on specialised tasks. Motivated by the success of in-context exemplars and the need for fine-tuning-level adaptation, we propose Multi-Head Attention-based Exemplar Soft Prompting (MHA-ESP), which uses an attention mechanism to learn soft prompts from retrieved exemplars, with multiple attention heads controlling prompt generation. Across multiple benchmarks and model scales, MHA-ESP performs on par with Low-Rank Adaptation (LoRA), outperforms standard ICL by an average of 18.85 points, and reduces inference cost by up to 10×GFLOPs, enabling efficient, high- accuracy domain adaptation without updating the foundation model

Primary area:
foundation or frontier models, including LLMs
Keywords:
In-context learning, Domain Adaptation, Prompt tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_M2iNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MHA-ESP, which aggregates retrieved input–output exemplars through query-conditioned attention heads. Each head generates one soft-prompt vector prepended to the input embeddings of a frozen foundation model.

Critical strengths

The method has a clear motivation and separates the number of retrieved examples from the soft-prompt length. The exemplar-order-invariance argument is sound under independent encoding without exemplar positional information.

Critical weaknesses
  1. The 100-example molecular and ChEBI tests lack a clearly documented independent validation protocol and uncertainty estimates.
  2. Prompt-generator training still requires gradients through the frozen backbone, so user-side adaptation needs an explicit local-model or server-gradient access arrangement. Please specify that arrangement, support for continuous prompt inputs.
  3. Increasing the head count simultaneously expands parameter capacity and prompt length, so the ablations do not isolate multi-head attention.
  4. Improve figure readability.
Questions for authors
  1. What model access and client–server communication are required during training and inference?
  2. Does multi-head aggregation retain its advantage under matched capacity and end-to-end inference budgets?
2 comments

W1: Validation protocol and uncertainty

Protocol. We follow the protocol of Guo et al. (2023), from which these benchmarks are taken. Each 100-example test set is sampled from the original MoleculeNet or ChEBI-20 test split and is disjoint from the training data, which also serves as the retrieval pool. Following the same protocol, configuration choices are made on a held-out validation set drawn from the training split, never on the test set. Selected hyperparameters are listed in Appendix F. We will state this protocol explicitly in Section 4.1.

Uncertainty. All results are averaged over 3 seeds. The training-seed standard deviations on BACE are:

  • Qwen3-4B: 76.27 ± 0.43
  • Llama3.2-3B: 67.62 ± 4.62
  • Mistral-7B: 76.26 ± 0.52

Variance is largest on the limited-data tasks, up to ±6.73 (Qwen3-4B, BBBP), but the main margins over ICL exceed it by a wide factor, e.g. +17.73 against ±4.62 on Llama3.2-3B BACE. Exemplar order contributes exactly 0.0 variance for MHA-ESP, against up to 8.52 for ICL (Table 4).


W2 / Q1: Model access and client–server communication

Training can run against a server that exposes a forward and backward pass with respect to input embeddings. The client sends its H soft-prompt vectors, the query, and the target tokens. The server returns only the gradient with respect to the soft prompt (H×d), and the client uses it to update its prompt generator locally. The backbone stays frozen, and the server stores no per-tenant parameters.

Inference follows the same pattern. The client sends its H soft-prompt vectors and the question to the server, and receives the answer back. The server only needs to accept prompt embeddings alongside the query, an interface that already exists in standard stacks: inputs_embeds in Hugging Face generate, and prompt_embeds on vLLM's OpenAI-compatible endpoints.

This is the same deployment setting assumed by the multi-tenant LoRA-serving systems we discuss (S-LoRA, dLoRA, FLoRA). Their per-tenant interface is more invasive: the server must store, page, and route weights per tenant, whereas MHA-ESP's server stores nothing per tenant. MHA-ESP, like every soft-prompt method (PT, P-tuning, prefix tuning, IDPG), is not applicable to closed, text-only APIs.


W3 / Q2: Isolating multi-head aggregation

The reviewer is right that raising H increases both parameters and prompt length. Three results separate these factors from aggregation.

  • Capacity alone does not help. The same H sweep adds identical parameters and prompt length at K = 1 and K = 5. Yet accuracy improves only at K = 5, where there is a set to aggregate (Figure 6). Adding capacity in other ways also fails to help: deeper attention and more tokens per exemplar bring no gain (Figure 3, right).

  • Aggregation wins at matched prompt length and inference cost. With a single prompt vector, MHA-ESP (H = 1) scores 75.30 / 78.37 / 80.60 against U-ESP’s 45.50 / 68.69 / 72.54 (Qwen3-4B, K = 5). The two methods have near-identical latency and memory (Table 13; e.g. BACE: 0.0536 vs 0.0511 ms, 7880 vs 7832 MB). MHA-ESP also outperforms P-ESP while emitting fewer vectors than P-ESP’s five. For example, on Qwen3-4B ClinTox MHA-ESP with H = 4 scores 96.32 against 53.22.

  • Heads specialise rather than duplicate. On ClinTox (H = 4), per-head attention entropies range from 0.45 to 0.84, against a uniform maximum of 1.61. Mean pairwise JS divergence is 0.348, so different heads select different exemplars for the same query.

Finally, the query, key and value projections are intrinsic to attention: they are what make the aggregation query-aware. Widening U-ESP’s MLP to the same size would add capacity without query-awareness.

Thank you for clarifying the validation and deployment protocols; however, the benefit of multiple heads remains confounded with parameter capacity and prompt length. These core concerns remain unresolved, so I maintain my original score.

Reviewer_mVRfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Multi-Head Attention-based Exemplar Soft Prompting (MHA-ESP) for domain adaptation of frozen language models. For each query, the method retrieves relevant labeled exemplars and employs a trainable multi-head attention module to aggregate their representations into a compact set of soft prompt tokens, which are prepended to the input embeddings of the frozen language model. The approach aims to combine the effectiveness of parameter-efficient fine-tuning with the inference efficiency of compressed in-context learning. Experiments across molecular prediction, molecular generation, and language understanding benchmarks using models ranging from 0.6B to 7B parameters demonstrate improvements over standard ICL and several prompt-based baselines.

Critical strengths

The paper addresses a practically relevant problem of efficient domain adaptation without modifying the foundation model. The proposed approach achieves encouraging improvements over conventional ICL and several prompt-based baselines across multiple benchmarks, while reducing the number of context tokens processed by the foundation model.

Critical weaknesses

The proposed method essentially applies standard query-conditioned multi-head attention to aggregate retrieved exemplars into soft prompt embeddings. Similar ideas have been extensively explored in dynamic prompt generation, retrieval-conditioned prompting, and context compression.

The paper does not sufficiently establish what fundamentally distinguishes its mechanism from existing approaches.

LoRA exhibits unusually poor performance on several benchmarks, including zero effective accuracy, raising concerns about baseline optimization and comparison fairness

TRhe proposed approach still requires training task-specific prompt generators and fine-tuning embedding encoders, while gradients must propagate through the frozen LLM. However, comprehensive end-to-end training cost comparisons are missing.

Questions for authors

Please provide additional evidence of proper baseline optimization, including per-class prediction statistics and relevant hyperparameter settings.

3 comments

(W1) "The proposed method essentially applies … distinguishes its mechanism from existing approaches."

We agree that multi-head attention is a standard primitive, and we do not claim it as new. Our contribution answers a representation question: how should labeled exemplars be represented for domain adaptation? Each family the reviewer names appears in Table 2 under an identical backbone, retriever, and exemplar set. None of them achieves what MHA-ESP does.

  • Dynamic prompt generation (PT, IDPG) ignores exemplars. PT uses a fixed prompt, and IDPG conditions only on the query. Discarding exemplars is what costs them performance: PT averages 14.58 / 28.06 / 25.04, against 66.38 / 66.60 / 67.59 for MHA-ESP.

  • Retrieval-conditioned compression (xRAG) maps exemplars into embeddings, and adapting it directly to labeled exemplars yields U-ESP / P-ESP. If MHA-ESP were simply this, U-ESP should recover most of the benefit, and it does not. Its G-Mean gain over ICL is +0.0 / +10.06 / +4.68, versus +14.37 / +26.54 / +15.99 for MHA-ESP (Qwen3-4B / Llama3.2-3B / Mistral-7B).

  • The objectives also differ. xRAG self-distills the full-context answer distribution. We never reconstruct the exemplars. We maximize answer likelihood on a new query, so the encoder keeps only what transfers to answering.

Three properties distinguish the mechanism:

  • Query-aware aggregation. U-ESP and P-ESP apply a fixed transformation, so each exemplar contributes identically regardless of the query. MHA-ESP weights each exemplar by its relevance to the current query. This mechanism, not added capacity, drives the gain.

    • At one prompt vector, with matched length and inference cost, MHA-ESP scores 75.30 / 78.37 / 80.60 against U-ESP's 45.50 / 68.69 / 72.54 (Qwen3-4B, K = 5).
    • Raising H helps at K = 5 but not at K = 1, where there is no set to aggregate (Figure 6).
    • Heads also specialize. On ClinTox with H = 4, per-head entropies are 0.45–0.84 against a uniform maximum of 1.61, and mean pairwise JS divergence is 0.348.
  • Order invariance by construction. MHA-ESP shows exactly 0.0 std under exemplar shuffling (Appendix A), against up to 3.02 for U-ESP, 4.91 for P-ESP, and 8.52 for ICL (Table 4).

  • Prompt length decoupled from K. U-ESP is fixed at one vector and P-ESP at exactly K. In MHA-ESP, H sets the length per task, for example, 8 heads for Mistral-7B on BACE/ClinTox and 4 on BBBP (Table 11).

The U-ESP → P-ESP → MHA-ESP ladder holds retrieval and backbone fixed, so it attributes the gain to these specific properties. To our knowledge, this decomposition has not been isolated before.


W2 / Q1: "LoRA exhibits unusually poor performance … Please provide additional evidence … and relevant hyperparameter settings."

LoRA’s low scores reflect overfitting in the limited-data regime, not under-tuning.

Same protocol for all methods. LoRA and MHA-ESP were trained on the same data splits (Section 4.1), and each was swept over a grid to reach its own best configuration. The grid was:

  • Learning rates {1, 3, 5} × {1e-3, 1e-4, 1e-5}.
  • Epochs {1, 2, 3, 5, 10, 15}, with shorter budgets acting as early stopping.
  • LoRA ranks {16, 32, 64} with alpha = 2 × rank, and MHA-ESP heads H ∈ {1, 2, 4, 8}.

For LoRA, we also tried class weighting proportional to the label distribution, without improvement. Reported numbers are the best obtained.

The zeros are prediction collapse. On BACE, the per-class rates are:

  • Qwen3-4B: TPR 1.85 / TNR 82.00 (G-mean 12.32).
  • Llama3.2-3B: TPR 0.00 / TNR 100.00, meaning the model answers “no” to every molecule.

Balanced accuracy would report Llama’s collapse as 50.0, indistinguishable from chance. Effective accuracy reports 0.0, which is why we use it under skewed labels. MHA-ESP’s predictions on the same tasks are non-degenerate.

LoRA is strong where data suffices. The same LoRA implementation is the strongest method on several medium-data tasks:

  • Qwen3-4B, MMLU-Pro: 50.68 vs 45.96 for MHA-ESP.
  • Mistral-7B, ChEBI-Design: 85.22 vs 78.11.

Collapse appears only on the ~1.4–2k-example property-prediction sets. The same pattern holds on Qwen3-0.6B (Table 9), where LoRA scores 0.0 on BACE, BBBP and ClinTox but 66.50 on Design. On the medium-data tasks, MHA-ESP narrows the gap to LoRA to −2.99, against ICL’s −17.22.

W3: “The proposed approach still requires training task-specific prompt generators … end-to-end training cost comparisons are missing.”

We measured end-to-end training wall-clock and peak GPU memory for all methods using Mistral-7B-Instruct-v0.3 as the backbone. Peak memory is the maximum reserved memory over the run. Measurements cover data loading, encoder fine-tuning, exemplar encoding and optimisation. All methods used the same splits, K = 5, and their best per-task configuration.

DatasetxRAGLoRA (r=64)MHA-ESP (H=1)MHA-ESP (H=8)
ChEBI-Captioning1h54m / 36.13 GB1h27m / 36.34 GB35m56s / 37.88 GB37m59s / 38.31 GB
ChEBI-Design1h48m / 31.69 GB1h7m / 39.27 GB1h8m / 37.23 GB1h11m / 37.72 GB
ClinTox5m9s / 24.88 GB6m50s / 30.37 GB5m18s / 25.34 GB5m26s / 24.36 GB
BACE4m46s / 21.45 GB6m17s / 25.97 GB4m54s / 21.50 GB5m4s / 22.00 GB
BBBP5m58s / 21.43 GB8m1s / 26.13 GB6m13s / 21.95 GB6m23s / 21.96 GB

The table supports four points:

  • Compared to LoRA, MHA-ESP trains faster or is on par across all five tasks, with lower peak memory usage on four of them.
  • Compared with xRAG, MHA-ESP costs nearly the same and is faster on ChEBI.
  • Scaling heads from 1 to 8 adds at most ~0.5 GB.
  • Backpropagating through the frozen backbone is a cost LoRA also pays. MHA-ESP updates fewer parameters: 87.02M client-side, including the fine-tuned encoder, versus 1.0B server-side for LoRA (Table 3).

This cost is paid once and locally, whereas ICL’s context overhead is paid on every query. We will add this table to Appendix G.

Dear Reviewer mVRf, thank you for your feedback. As the discussion period is closing soon, we wanted to check whether our rebuttal addressed your concerns. We would be glad to clarify anything further.

Reviewer_kobPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a method that combines in-context learning, soft prompting, and retrieval-augmented generation to achieve LLM domain adaptation. Experiments show that the method outperforms several domain adaptation methods including in-context learning, LoRA adaptation and some soft prompt methods.

Critical strengths
  1. This paper proposes an interesting approach that combines in-context learning, soft prompting, and retrieval-augmented generation to achieve domain adaptation. Experimental results demonstrate the effectiveness of the proposed method.
Critical weaknesses
  1. Classic approaches to LLM domain adaptation also include adjusting the model's output token probabilities and using a scoring model to score and search for answers. However, the authors neither discuss nor compare against these methods, which is a significant omission for a paper on domain adaptation.
  2. The paper suffers from poor formatting. Narrative citations are used throughout the paper, even though parenthetical citations would be more appropriate in many cases. In addition, there is an obvious text compression issue in line 209, while in line 204, a single letter occupies an entire line. These issues make me question whether the authors have carefully review the manuscript.

[1] Black-box tuning for language-model-as-a-service. ICML 2022

[2] BBox-Adapter: Lightweight Adapting for Black-Box Large Language Models. ICML 2024

Questions for authors

see weaknesses

2 comments

(W1) Comparison with black-box adaptation methods

We thank the reviewer and will extend the discussion on Related Work. They target a different access regime. BBox-Adapter’s own taxonomy classifies BBT and logit-adjustment methods (e.g., proxy-tuning) as grey-box, with access to output probabilities only, and BBox-Adapter itself as black-box. MHA-ESP targets a third regime: the model’s weights stay closed, but the provider accepts input embeddings and returns gradients with respect to them. The adaptation then lives entirely with the client, and the provider stores no per-tenant adapters.

BBT [1] optimizes a continuous prompt prepended to the input without gradients. This is the same interface MHA-ESP uses, but BBT’s prompt is static, one per task. Table 2 shows that static (PT) and query-only (IDPG) prompts trail MHA-ESP, whose prompt is conditioned on the query and its retrieved exemplars. This conditioning is orthogonal to the optimizer, and derivative-free training of it is a natural extension when gradients are unavailable.

BBox-Adapter [2] scores candidates sampled from the frozen LLM during beam search. It can only select among what the base model generates, and on our tasks those candidates are often unusable. Only 3.29% of off-the-shelf Mistral-7B outputs on ChEBI-Design are valid SMILES, and off-the-shelf models score 0.0 on ClinTox for all three backbones, and on BACE for Qwen3-4B and Llama3.2-3B. MHA-ESP instead changes what the model generates, raising SMILES validity to 83.50–85.71% with a single forward pass. BBox-Adapter’s authors also find that LoRA outperforms it when model parameters are accessible. MHA-ESP matches LoRA’s accuracy without touching the weights at all. The approaches are complementary: an output-side scorer could re-rank MHA-ESP’s outputs.


(W2) Formatting concerns

We thank the reviewer for pointing out the citation formatting issue, which went unnoticed during submission due to a PDF rendering issue. We have fixed it, along with the spacing, on line 208 in the revision. One clarification we want to make regarding line 204: the standalone letter should be read together with line 203, pointing to "A formal proof is given in Appendix A."

Dear Reviewer kobP, thank you for your feedback. As the discussion period is closing soon, we wanted to check whether our rebuttal addressed your concerns. We would be glad to clarify anything further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.