- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MHA-ESP, which aggregates retrieved input–output exemplars through query-conditioned attention heads. Each head generates one soft-prompt vector prepended to the input embeddings of a frozen foundation model.
Critical strengths
The method has a clear motivation and separates the number of retrieved examples from the soft-prompt length. The exemplar-order-invariance argument is sound under independent encoding without exemplar positional information.
Critical weaknesses
- The 100-example molecular and ChEBI tests lack a clearly documented independent validation protocol and uncertainty estimates.
- Prompt-generator training still requires gradients through the frozen backbone, so user-side adaptation needs an explicit local-model or server-gradient access arrangement. Please specify that arrangement, support for continuous prompt inputs.
- Increasing the head count simultaneously expands parameter capacity and prompt length, so the ablations do not isolate multi-head attention.
- Improve figure readability.
Questions for authors
- What model access and client–server communication are required during training and inference?
- Does multi-head aggregation retain its advantage under matched capacity and end-to-end inference budgets?
2 comments
Thank you for clarifying the validation and deployment protocols; however, the benefit of multiple heads remains confounded with parameter capacity and prompt length. These core concerns remain unresolved, so I maintain my original score.
W1: Validation protocol and uncertainty
Protocol. We follow the protocol of Guo et al. (2023), from which these benchmarks are taken. Each 100-example test set is sampled from the original MoleculeNet or ChEBI-20 test split and is disjoint from the training data, which also serves as the retrieval pool. Following the same protocol, configuration choices are made on a held-out validation set drawn from the training split, never on the test set. Selected hyperparameters are listed in Appendix F. We will state this protocol explicitly in Section 4.1.
Uncertainty. All results are averaged over 3 seeds. The training-seed standard deviations on BACE are:
Variance is largest on the limited-data tasks, up to ±6.73 (Qwen3-4B, BBBP), but the main margins over ICL exceed it by a wide factor, e.g. +17.73 against ±4.62 on Llama3.2-3B BACE. Exemplar order contributes exactly 0.0 variance for MHA-ESP, against up to 8.52 for ICL (Table 4).
W2 / Q1: Model access and client–server communication
Training can run against a server that exposes a forward and backward pass with respect to input embeddings. The client sends its H soft-prompt vectors, the query, and the target tokens. The server returns only the gradient with respect to the soft prompt (H×d), and the client uses it to update its prompt generator locally. The backbone stays frozen, and the server stores no per-tenant parameters.
Inference follows the same pattern. The client sends its H soft-prompt vectors and the question to the server, and receives the answer back. The server only needs to accept prompt embeddings alongside the query, an interface that already exists in standard stacks: inputs_embeds in Hugging Face generate, and prompt_embeds on vLLM's OpenAI-compatible endpoints.
This is the same deployment setting assumed by the multi-tenant LoRA-serving systems we discuss (S-LoRA, dLoRA, FLoRA). Their per-tenant interface is more invasive: the server must store, page, and route weights per tenant, whereas MHA-ESP's server stores nothing per tenant. MHA-ESP, like every soft-prompt method (PT, P-tuning, prefix tuning, IDPG), is not applicable to closed, text-only APIs.
W3 / Q2: Isolating multi-head aggregation
The reviewer is right that raising H increases both parameters and prompt length. Three results separate these factors from aggregation.
Capacity alone does not help. The same H sweep adds identical parameters and prompt length at K = 1 and K = 5. Yet accuracy improves only at K = 5, where there is a set to aggregate (Figure 6). Adding capacity in other ways also fails to help: deeper attention and more tokens per exemplar bring no gain (Figure 3, right).
Aggregation wins at matched prompt length and inference cost. With a single prompt vector, MHA-ESP (H = 1) scores 75.30 / 78.37 / 80.60 against U-ESP’s 45.50 / 68.69 / 72.54 (Qwen3-4B, K = 5). The two methods have near-identical latency and memory (Table 13; e.g. BACE: 0.0536 vs 0.0511 ms, 7880 vs 7832 MB). MHA-ESP also outperforms P-ESP while emitting fewer vectors than P-ESP’s five. For example, on Qwen3-4B ClinTox MHA-ESP with H = 4 scores 96.32 against 53.22.
Heads specialise rather than duplicate. On ClinTox (H = 4), per-head attention entropies range from 0.45 to 0.84, against a uniform maximum of 1.61. Mean pairwise JS divergence is 0.348, so different heads select different exemplars for the same query.
Finally, the query, key and value projections are intrinsic to attention: they are what make the aggregation query-aware. Widening U-ESP’s MLP to the same size would add capacity without query-awareness.