acceptodds
Under review as a conference paper at ICLR 2027

Taming Attention to Compose Soft Prompt Experts

Abstract

Large language models can be adapted into specialized experts through parameter or soft-prompt tuning. Combining independently trained soft-prompt experts without joint retraining is challenging because direct joint readout can misbalance both the expert group's total influence and the allocation among experts. We propose TAME(Two-level Attention Mixing of Experts), a training-free, input-conditioned composition method. TAME uses two-level attention to separately control the expert group's total readout and the within-group contribution of each expert while keeping all experts independently available. On GLUE, using the same set of soft-prompt experts, TAME achieves 94.4% mean capability retention, improves direct joint reading by 11.4 percentage points, and outperforms the evaluated context-space composition baselines. In a unified comparison with parameter merging, TAME achieves 1.90 percentage points higher GLUE Macro accuracy despite a lower single-expert reference, matches or exceeds capability retention on mathematics, code, and instruction following, and exhibits complementary capability-retention patterns with parameter merging. Overall, TAME provides a competitive training-free route for composing soft-prompt experts.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
model merging, soft prompt tuning, KV cache composition, task interference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uXQeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes TAME, a training-free method that composes independently trained soft prompt experts. Each prompt is encoded as a layer-wise KV Bank. An input-level attention probe estimates expert preference and selects a reference Bank. Two attention controls then set the total readout strength of the expert group and the share of each expert. Experiments compare context-space composition with parameter merging on six GLUE tasks, GSM8K, MBPP and IF, and report 94.4% mean capability retention.

Critical strengths

The two-level readout is a useful decomposition, and the mass-only ablation shows the role of group-level control.

The cross-space comparison gives a more informative view than a single average score.

Critical weaknesses

The method is tested on one backbone. The GLUE comparison drops MNLI, MRPC and STS-B because the experts were not strong enough. The conclusion therefore depends on a selected task subset.

The comparison with LoRA merging mixes two expert families. Soft prompt experts and LoRA experts have different single-expert references and different training procedures. A direct Macro comparison is not convincing.

Mean retention hides the absolute drop on individual tasks.

The intro is not clear and the paper's writing can't reach the standard.

The intro and framework pictures contain a lot of meaningless information.

Questions for authors

Are Table 1 and Table 2 reported on the development set or the test set? Does gamma selection use the same samples?

How is the calibration set built? Does it contain in-domain data from each task?

Why are MNLI, MRPC and STS-B excluded? Do the conclusions hold when they are added?

How does the method behave when the number of experts grows beyond six?

6 comments
AuthorsOpenReview ↗

Evaluation Protocol and Calibration

We thank the reviewer for recognizing the two-level decomposition of attention readout and the analysis of capability retention across merging spaces. Following your comments, we clarify the scope of our study, the purpose of the comparisons, and the evaluation protocol. We have added an eight-task evaluation covering the previously omitted tasks, official test-set results, generalization experiments on additional tasks, and experiments with more experts. We have also redrawn the framework diagram to make it simpler and clearer. Below, we respond to the questions and weaknesses in order.

Question 1

Are Table 1 and Table 2 reported on the development set or the test set? Does gamma selection use the same samples?

Response

Table 1 in the revised manuscript uses the official GLUE development set, and Table 3 uses the public evaluation sets for the cross-domain tasks. We have also added eight-task results returned by the official GLUE test server. For the main GLUE experiments, we fixed γ=1 in advance. For the cross-domain experiments, the comparisons of γ configurations and the reported results use the same evaluation samples.

The original GLUE results are evaluated on the official development set (validation split), with the same evaluation samples used for all methods. Reporting results on this split is consistent with published work in related areas, including the model merging methods DARE [1] and CAT Merging [2], and the parameter-efficient fine-tuning methods LoRA-GA [3] and Flat-LoRA [4]. To address your concerns about task coverage and evaluation reliability, we have added official GLUE test-set evaluations on CoLA, SST-2, QQP, QNLI, RTE, MNLI, MRPC, and STS-B, including the three previously omitted tasks you identified. This eight-task setting follows the common practice of excluding WNLI, consistent with the task coverage used by BERT [5] and related work when reporting official test results. All new aggregate results are calculated from scores returned by the official GLUE evaluation server. TAME achieves a mean capability retention of 88.24% and a worst-task retention of 60.57%, providing further evidence of capability retention beyond the original task setting. The detailed results appear in Appendix C (Table 6), highlighted in blue, and in our response to Question 3 below.

Table 3 in the revised manuscript uses the public evaluation sets for GSM8K, MBPP, and IF Strict Prompt. All configurations use the same evaluation samples and fixed calibration inputs, with no separate set for selecting γ. The evaluation protocol is clarified in Section 4.1 and Appendix E.4, and the configuration comparisons are reported in Section 4.3.2 and Appendix E.3, with the corresponding revisions highlighted in blue.

Question 2

How is the calibration set built? Does it contain in-domain data from each task?

Response

The calibration set contains an equal number of in-domain training inputs from each task or domain being combined. Calibration uses only the inputs to estimate each Bank's mean probe score, without accessing target answers, and the resulting statistics remain frozen throughout evaluation. All calibration inputs come from the corresponding training splits. In both the GLUE experiments (Table 1) and the three-task cross-domain experiments (Table 3), we use a fixed set of 128 training inputs per task.

The calibration statistics center the probe scores to reduce differences in the average response of different Banks. During evaluation, these statistics remain frozen, and the probe results for the current input determine the expert preferences and reference Bank. The model receives no additional task or domain labels. The calibration details are provided in Section 4.1 and Appendix E.4, highlighted in blue.

References

[1] Le Yu et al. Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch. ICML, 2024.

[2] Wenju Sun et al. CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging. ICML, 2025.

[3] Shaowen Wang et al. LoRA-GA: Low-Rank Adaptation with Gradient Approximation. NeurIPS, 2024.

[4] Tao Li et al. Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape. ICML, 2025.

[5] Jacob Devlin et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT, 2019.

AuthorsOpenReview ↗

GLUE Coverage and Cross-domain Baselines

Question 3 and Weakness 1

Why are MNLI, MRPC and STS-B excluded? Do the conclusions hold when they are added?

The method is tested on one backbone. The GLUE comparison drops MNLI, MRPC and STS-B because the experts were not strong enough. The conclusion therefore depends on a selected task subset.

Response

We have added an eight-task evaluation on the official test set, including MNLI, MRPC, and STS-B, and report per-task scores and capability retention. Using the same SP experts, TAME substantially outperforms the context-space baseline Plain joint. It also maintains higher mean and worst-task retention than the evaluated parameter-space merging methods, consistent with the observations in the original manuscript. We have further expanded the comparisons in the three-task cross-domain merging setting from Section 4.4.

Our study asks how to combine independently trained soft-prompt experts and reduce composition loss, without additional fusion training, when the individual soft prompts can already adapt effectively to the corresponding tasks. The original experiments therefore prioritize tasks where SP and LoRA single experts achieve similar performance, reducing the influence of differences in initial capabilities on the comparison of composition loss. This rationale is clarified in Appendix B.1, with the revision highlighted in blue.

To expand task coverage, we have added an eight-task experiment on CoLA, SST-2, QQP, QNLI, RTE, MRPC, MNLI, and STS-B. All test scores come from the official GLUE evaluation server.

MethodCoLA MCCSST-2 Acc.QQP Acc. / F1QNLI Acc.RTE Acc.MRPC Acc. / F1MNLI m / mm Acc.STS-B Pearson / SpearmanEight-task averageMean retentionWorst-task retention
LoRA single experts56.896.390.0 / 73.594.585.683.1 / 87.489.7 / 89.290.9 / 92.685.18——
TA30.392.786.5 / 61.582.780.971.8 / 76.272.4 / 74.083.5 / 92.374.4685.82%53.35%
TIES25.694.688.5 / 67.188.179.375.4 / 80.580.9 / 82.178.2 / 94.676.4187.63%45.07%
SP single experts55.895.688.1 / 70.792.484.679.4 / 84.988.5 / 88.088.0 / 90.283.41——
Plain joint0.089.082.8 / 21.357.062.268.2 / 73.562.2 / 64.944.9 / 63.256.0964.10%0.00%
TAME33.891.382.9 / 59.582.577.871.9 / 78.178.0 / 78.281.5 / 95.274.7688.24%60.57%

The eight-task average is calculated by first averaging the reported metrics within each task and then taking an equally weighted average across the eight tasks. Retention is calculated relative to the corresponding single-expert score for each task; we report the mean and minimum of these task-level retention rates.

For this broader task combination, TAME achieves a mean retention of 88.24%, substantially above direct joint readout in context space and also above TIES (87.63%) and TA (85.82%). LoRA experts retain stronger single-task performance, as their larger trainable parameter budget enables them to fit the tasks more effectively. After composition, TAME has a lower absolute macro score than TIES and higher mean and worst-task retention than both TA and TIES, consistent with the observations in the original manuscript. These test results are now in Appendix C (Table 6), highlighted in blue.

We have also expanded the baseline comparisons for the cross-domain combination of mathematics, coding, and instruction following in Section 4.4 (Table 3), with the additions highlighted in blue. With the same frozen SP experts, TAME achieves higher mean capability retention than direct concatenation, single-expert selection, and attention-mass-only control, as shown below.

MethodMathCodeIFCross-domain averageMean retention
Independent LoRA experts73.2445.5668.0262.27—
TA68.7642.2240.6750.5582.12%
TIES73.2443.3358.9658.5193.93%
Independent SP experts70.1338.8967.4758.83—
Plain joint52.0841.1141.2244.8180.36%
Centered top-170.0535.5653.9753.1990.44%
Mass-only (γ=0)64.2932.2257.1251.2186.40%
TAME (γ=1)69.9038.8956.1954.9994.32%
TAME (γ=2)70.4338.8960.0756.4696.49%

With the same frozen SP experts, TAME achieves higher cross-domain average scores and mean retention than the evaluated context-space composition baselines.

AuthorsOpenReview ↗

Cross-domain Comparison, Expert Scaling, and Shared-Expert Baselines

Question 3 and Weakness 1 (continued)

In the comparison with parameter-space merging, TIES achieves a higher absolute average score after composition, while TAME achieves higher mean retention relative to the respective single experts. This is consistent with the results from the expanded GLUE setting. These findings illustrate the trade-offs between final performance and capability retention across composition methods and support TAME's effectiveness in reducing composition loss in this cross-domain setting. The added discussion is in Section 5.1, highlighted in blue.

To evaluate another backbone and additional tasks, we have also conducted a larger experiment on multimodal few-shot object detection, detailed in our response to Question 4 below.

Question 4

Original Question 4

How does the method behave when the number of experts grows beyond six?

Response

Our response to Question 3 includes the combination of eight experts across eight tasks, where TAME achieves 88.24% mean retention, providing evidence of capability retention beyond six experts (Appendix C, Table 6; additions highlighted in blue). Following completion of the original manuscript, we also conducted RF20 multimodal experiments to evaluate TAME in a broader expert-composition setting.

Prior work [6] shows that, in RF20's 10-shot object detection setting, training only 1–3 soft tokens with suitable prompt placement and initialization can match the average performance of the best LoRA configuration evaluated in that study. Roboflow20-VL (RF20) is an object detection subset of Roboflow-100-VL, comprising 20 tasks across seven categories: aerial, documents, flora and fauna, industrial, medical, sports, and other. We therefore selected this few-shot vision-language setting and used Qwen3-VL-4B-Instruct. Following the soft-prompt tuning procedure in that work, we trained one SP expert with 1–3 soft-prompt tokens per domain and then combined all 20 experts using TAME. The results are shown below.

MethodMean mAP@50:95Overall retentionWorst-domain retention
SP single experts16.820——
LoRA single experts17.013——
TA14.28383.95%2.62%
TIES13.83481.31%3.09%
Plain Joint8.92253.04%0.00%
TAME14.38485.51%12.41%

With equal weighting across the 20 domains, SP single experts achieve a mean mAP close to that of LoRA single experts. TAME outperforms TA and TIES in mean mAP, overall retention, and worst-domain retention. Compared with direct concatenation in context space (Plain Joint), TAME improves mean mAP from 8.922 to 14.384. The complete RF20 results are now in Appendix D (Table 7), highlighted in blue.

Weaknesses 2 and 3

The comparison with LoRA merging mixes two expert families. Soft prompt experts and LoRA experts have different single-expert references and different training procedures. A direct Macro comparison is not convincing.

Mean retention hides the absolute drop on individual tasks.

Response

We evaluate TAME's composition effectiveness by comparing methods that use the same frozen SP experts. For comparisons across merging spaces, we report absolute scores, relative retention, and per-task changes together. Table 5 and Figure 4 in the revised manuscript present task-level absolute performance and changes relative to the respective single experts, making the losses and trade-offs behind the averages directly visible.

In Table 1 of the revised manuscript, TAME and context-space composition methods such as Plain joint, Centered top-1, and StructuredPrompting use the same frozen soft-prompt experts. TAME improves Macro Accuracy from 66.67% with Plain joint to 77.01%, a gain of 10.34 percentage points, achieving the highest score among the context-space composition methods. This improvement demonstrates the effect of changing the composition method. The additional cross-domain comparisons in our response to Question 3 also use the same frozen experts, with TAME achieving the best overall performance among methods using the same soft-prompt experts. These results further evaluate capability retention across different task combinations. The shared-expert comparison protocol is clarified in Section 4.1 and Appendix E.4, highlighted in blue.

References

[6] Gautam Rajendrakumar Gare et al. Your Model Already Knows Don't Teach It, Learn to Ask It: Soft Prompting for Few-Shot Adaptation of Vision-Language Models. arXiv:2609.11310, 2026.

Reviewer_UqJeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TAME, a training-free method for composing soft-prompt experts without requiring additional fusion training. The authors identify two potential sources of attention imbalance when independently trained soft prompts are jointly read: an imbalance in the total attention mass allocated to the expert group relative to the textual context, and an imbalance in attention allocation among individual experts. TAME first employs an input-level attention probe to estimate expert preferences and select a reference expert. During inference, it then applies two-level attention control to adjust the total readout strength of the expert group and the relative allocation among experts, aiming to mitigate performance degradation caused by expert composition. Experimental results show that TAME outperforms the evaluated context-space composition baselines on six GLUE subtasks and achieves competitive results compared with several static parameter-merging methods.

Critical strengths
  1. Well-motivated approach with a promising design.

The paper investigates the capability degradation arising from the composition of independently trained soft-prompt experts through the lens of attention allocation imbalance. It further decomposes this issue into two subproblems: Expert-group Mass Control and Within-group Expert Allocation. Based on this observation, the authors propose a two-level attention control method that requires no additional fusion training and dynamically adjusts expert contributions based on the input while preserving the independence of individual experts. This design offers a promising approach to soft-prompt expert composition.

  1. Extensive baseline comparisons on GLUE and supporting ablation studies.

The paper compares TAME against several context-space composition methods on six GLUE subtasks. TAME achieves a Macro Accuracy of 77.01%, representing an improvement of 10.34 percentage points over Plain Joint (66.67%). It also achieves a Mean Capability Retention of 94.4% and demonstrates strong Worst-task Retention compared with the evaluated non-TAME baselines.

Furthermore, the ablation studies show that Expert-group Mass Control alone improves Macro Accuracy to 76.05%, while incorporating Within-group Allocation further increases it to 77.01%. These results support the effectiveness of the proposed attention control mechanism in mitigating performance degradation during expert composition.

  1. Valuable comparison with parameter-space model merging methods.

Beyond evaluating the composition of soft-prompt experts, the paper also compares TAME with parameter-merging methods such as Task Arithmetic and TIES under a unified experimental setting. The results reveal different capability-retention patterns across tasks between the two approaches, providing valuable empirical insights into the trade-offs between Context-space Composition and Parameter-space Merging.

Critical weaknesses
  1. Insufficient baseline comparisons on mathematics, coding, and instruction-following tasks limit the strength of the cross-domain conclusions.

Although the authors evaluate TAME on GSM8K, MBPP, and IF Strict Prompt, Table 2 primarily reports results for single experts and TAME under different hyperparameter settings, without including Plain Joint, Centered Top-1, or other context-space composition baselines on the same tasks.

Moreover, TAME does not consistently outperform the single-expert references in terms of absolute performance on these tasks. For example, with γ = 2, its performance on GSM8K and MBPP is approximately on par with that of the corresponding single experts, while its performance on IF Strict Prompt decreases from 67.47% to 60.07%.

Therefore, while the current experiments demonstrate that TAME can preserve a substantial proportion of single-expert capabilities across these domains, they do not sufficiently establish its consistent advantages over simpler composition or routing strategies. Given that the paper also discusses TAME's competitiveness with Parameter-space Merging in these domains, more comprehensive baseline comparisons would help substantiate these claims.

  1. Evaluation on a single backbone and a limited task set leaves the generalizability of the method unclear.

All experiments are conducted using Qwen3-1.7B, leaving it unclear whether the proposed attention control mechanism can achieve similar improvements across different model architectures and parameter scales.

Additionally, the main GLUE experiments cover only six of the eight subtasks in the full benchmark, excluding two tasks for which sufficiently strong single-task experts could not be obtained. Although this experimental choice is understandable, it limits the extent to which the findings can be generalized to broader task combinations.

  1. The necessity of certain key design choices is not sufficiently validated.

The existing ablation studies indicate that most of the performance improvements come from Expert-group Mass Control, while Within-group Allocation provides a relatively modest additional gain.

However, TAME sets the total attention mass of the expert group to the native mass of the reference Bank selected by the input-level probe. The paper does not sufficiently compare this strategy with simpler alternatives, such as a fixed attention budget, the average Bank mass, or a fixed reference expert.

Consequently, the current experiments do not fully establish whether input-conditioned reference selection and the corresponding total-mass constraint represent the most appropriate design choices.

  1. Lack of quantitative analysis of inference efficiency and resource overhead.

TAME requires an additional input-level probe, reprocesses the input before controlled inference, and maintains multiple KV Banks. As the number of experts and prompt lengths increase, the additional computational and GPU memory requirements may become significant limitations for practical deployment.

However, the paper does not provide sufficient quantitative comparisons of inference latency, throughput, or GPU memory usage. This makes it difficult to assess the practical trade-off between the performance gains and the additional inference costs introduced by TAME.

Questions for authors
  1. Cross-domain comparisons and generalization. Could the authors provide additional comparisons with context-space composition baselines, such as Plain Joint and Centered Top-1, on GSM8K, MBPP, and IF Strict Prompt, consistent with the GLUE experiments? Furthermore, could the method be evaluated on backbones of different scales or architectures to better demonstrate its generalizability?

  2. Inference efficiency. TAME requires an initial uncontrolled probe followed by reprocessing the input for controlled inference. Could the authors provide a comparison of inference efficiency against other composition methods, including Time to First Token (TTFT), generation throughput, and peak GPU memory usage? In particular, it would be useful to understand how the additional overhead scales with the number of experts and soft-prompt lengths.

  3. Necessity of reference-based mass control. The authors use the native attention mass of the reference Bank to determine the total readout strength of the expert group. Could the authors further explain the rationale behind this design choice and compare it with alternatives such as a fixed or average attention budget or a fixed reference Bank? Such comparisons would help clarify the actual contributions of the input-level probe and reference selection.

  4. Figure clarity. Some text and annotations in Figure 2 are relatively small, and the diagram contains a high density of information, making certain parts difficult to read at a normal viewing scale. It would be helpful to provide a clearer vector-based figure with larger labels to improve readability.

7 comments
AuthorsOpenReview ↗

Cross-domain Composition Baselines

We thank the reviewer for recognizing the two-level attention control design, the baseline comparisons, and the analysis of capability retention across merging spaces. Following your comments, we have added cross-domain composition baselines, an eight-task evaluation on the official test set, and inference efficiency measurements. We also further explain the rationale for the reference Bank. Below, we respond to Questions 1–4 in order, together with the corresponding weaknesses.

Question 1 (Cross-domain Comparisons) and Weakness 1

  1. Cross-domain comparisons and generalization. Could the authors provide additional comparisons with context-space composition baselines, such as Plain Joint and Centered Top-1, on GSM8K, MBPP, and IF Strict Prompt, consistent with the GLUE experiments?
  1. Insufficient baseline comparisons on mathematics, coding, and instruction-following tasks limit the strength of the cross-domain conclusions.

Although the authors evaluate TAME on GSM8K, MBPP, and IF Strict Prompt, Table 2 primarily reports results for single experts and TAME under different hyperparameter settings, without including Plain Joint, Centered Top-1, or other context-space composition baselines on the same tasks.

Moreover, TAME does not consistently outperform the single-expert references in terms of absolute performance on these tasks. For example, with γ = 2, its performance on GSM8K and MBPP is approximately on par with that of the corresponding single experts, while its performance on IF Strict Prompt decreases from 67.47% to 60.07%.

Therefore, while the current experiments demonstrate that TAME can preserve a substantial proportion of single-expert capabilities across these domains, they do not sufficiently establish its consistent advantages over simpler composition or routing strategies. Given that the paper also discusses TAME's competitiveness with Parameter-space Merging in these domains, more comprehensive baseline comparisons would help substantiate these claims.

Response

We have added comparisons with Plain joint, Centered top-1, Mass-only, and the parameter-space merging methods TIES and TA in the cross-domain combination of mathematics, coding, and instruction following in Section 4.4 (Table 3), with the added comparisons and discussion highlighted in blue.

The results are shown below. Math, Code, and IF correspond to GSM8K, MBPP, and IF Strict Prompt, respectively. The context-space methods use the same SP experts, while TA and TIES use the same LoRA experts. Retention is calculated relative to the respective single experts.

MethodMathCodeIFCross-domain averageMean retention
Independent LoRA experts73.2445.5668.0262.27—
TA68.7642.2240.6750.5582.12%
TIES73.2443.3358.9658.5193.93%
Independent SP experts70.1338.8967.4758.83—
Plain joint52.0841.1141.2244.8180.36%
Centered top-170.0535.5653.9753.1990.44%
Mass-only (γ=0)64.2932.2257.1251.2186.40%
TAME (γ=1)69.9038.8956.1954.9994.32%
TAME (γ=2)70.4338.8960.0756.4696.49%

With the same frozen SP experts, TAME achieves higher cross-domain average scores and mean retention than the evaluated context-space composition baselines.

In the comparison with parameter-space merging, TIES achieves a higher absolute average score after composition, while TAME achieves higher mean retention relative to the respective single experts. This is consistent with the results from the expanded GLUE setting. These findings illustrate the trade-offs between final performance and capability retention across composition methods and support TAME's effectiveness in reducing composition loss in this cross-domain setting.

AuthorsOpenReview ↗

GLUE Task Coverage and Generalization

Question 1 (Backbone Generalization) and Weakness 2

Furthermore, could the method be evaluated on backbones of different scales or architectures to better demonstrate its generalizability?

  1. Evaluation on a single backbone and a limited task set leaves the generalizability of the method unclear.

All experiments are conducted using Qwen3-1.7B, leaving it unclear whether the proposed attention control mechanism can achieve similar improvements across different model architectures and parameter scales.

Additionally, the main GLUE experiments cover only six of the eight subtasks in the full benchmark, excluding two tasks for which sufficiently strong single-task experts could not be obtained. Although this experimental choice is understandable, it limits the extent to which the findings can be generalized to broader task combinations.

Response

We have added an eight-task evaluation on the official test set, including MNLI, MRPC, and STS-B, and have further conducted RF20 multimodal few-shot object detection experiments. In the new eight-task setting, TAME substantially improves over Plain joint and achieves higher mean and worst-task retention than TA and TIES.

Our study asks how to combine independently trained soft-prompt experts and reduce composition loss, without additional fusion training, when the individual soft prompts can already adapt effectively to the corresponding tasks. The original experiments therefore prioritize tasks where SP and LoRA single experts achieve similar performance, reducing the influence of differences in initial capabilities on the comparison of composition loss. This rationale is clarified in Appendix B.1, with the revision highlighted in blue.

To expand task coverage, we have added an eight-task experiment on CoLA, SST-2, QQP, QNLI, RTE, MRPC, MNLI, and STS-B. All test scores come from the official GLUE evaluation server. The complete results are now in Appendix C (Table 6), highlighted in blue.

MethodCoLA MCCSST-2 Acc.QQP Acc. / F1QNLI Acc.RTE Acc.MRPC Acc. / F1MNLI m / mm Acc.STS-B Pearson / SpearmanEight-task averageMean retentionWorst-task retention
LoRA single experts56.896.390.0 / 73.594.585.683.1 / 87.489.7 / 89.290.9 / 92.685.18——
TA30.392.786.5 / 61.582.780.971.8 / 76.272.4 / 74.083.5 / 92.374.4685.82%53.35%
TIES25.694.688.5 / 67.188.179.375.4 / 80.580.9 / 82.178.2 / 94.676.4187.63%45.07%
SP single experts55.895.688.1 / 70.792.484.679.4 / 84.988.5 / 88.088.0 / 90.283.41——
Plain joint0.089.082.8 / 21.357.062.268.2 / 73.562.2 / 64.944.9 / 63.256.0964.10%0.00%
TAME33.891.382.9 / 59.582.577.871.9 / 78.178.0 / 78.281.5 / 95.274.7688.24%60.57%

The eight-task average is calculated by first averaging the reported metrics within each task and then taking an equally weighted average across the eight tasks. Retention is calculated relative to the corresponding single-expert score for each task; we report the mean and minimum of these task-level retention rates.

With the same SP experts, TAME improves the eight-task average from 56.09 with Plain joint to 74.76, and mean retention from 64.10% to 88.24%. In the comparison across merging spaces, LoRA single experts have higher average performance than SP single experts. After composition, TAME has a lower absolute average score than TIES and higher mean and worst-task retention than both TA and TIES. This is consistent with the original observations on reducing composition loss.

AuthorsOpenReview ↗

Multimodal Results and Inference Efficiency

Question 1 (Backbone Generalization) and Weakness 2 (continued)

To evaluate another backbone and additional tasks, we have further conducted RF20 multimodal few-shot object detection experiments. Prior work [1] shows that, in RF20's 10-shot object detection setting, training only 1–3 soft tokens with suitable prompt placement and initialization can match the average performance of the best LoRA configuration evaluated in that study. Roboflow20-VL (RF20) is an object detection subset of Roboflow-100-VL, comprising 20 tasks across seven categories: aerial, documents, flora and fauna, industrial, medical, sports, and other. We therefore selected this few-shot vision-language setting and used Qwen3-VL-4B-Instruct. Following the soft-prompt tuning procedure in that work, we trained one SP expert with 1–3 soft-prompt tokens per domain and then combined all 20 experts using TAME. The results are shown below.

MethodMean mAP@50:95Overall retentionWorst-domain retention
SP single experts16.820——
LoRA single experts17.013——
TA14.28383.95%2.62%
TIES13.83481.31%3.09%
Plain Joint8.92253.04%0.00%
TAME14.38485.51%12.41%

With equal weighting across the 20 domains, SP single experts achieve a mean mAP close to that of LoRA single experts. TAME outperforms TA and TIES in mean mAP, overall retention, and worst-domain retention. Compared with direct concatenation in context space (Plain Joint), TAME improves mean mAP from 8.922 to 14.384. The complete RF20 results are now in Appendix D (Table 7), highlighted in blue.

Question 2 and Weakness 4

  1. Inference efficiency. TAME requires an initial uncontrolled probe followed by reprocessing the input for controlled inference. Could the authors provide a comparison of inference efficiency against other composition methods, including Time to First Token (TTFT), generation throughput, and peak GPU memory usage? In particular, it would be useful to understand how the additional overhead scales with the number of experts and soft-prompt lengths.
  1. Lack of quantitative analysis of inference efficiency and resource overhead.

TAME requires an additional input-level probe, reprocesses the input before controlled inference, and maintains multiple KV Banks. As the number of experts and prompt lengths increase, the additional computational and GPU memory requirements may become significant limitations for practical deployment.

However, the paper does not provide sufficient quantitative comparisons of inference latency, throughput, or GPU memory usage. This makes it difficult to assess the practical trade-off between the performance gains and the additional inference costs introduced by TAME.

Response

We have added inference efficiency comparisons and experiments varying the number of experts, using a lightweight optimized implementation that retains the input-level probe and two-level control algorithms. These additions appear in Section 4.5 (Table 4) and Appendix F, highlighted in blue. We first compare TAME with the parameter-space merging method TIES. Both methods use a static KV cache and fused attention-weight computation. The two efficiency tables use the same measurement run, with the method comparison corresponding to the eight-expert configuration. TAME's decoding throughput is approximately 2.47% lower than that of TIES, and its time to first token increases by 21.16 ms. All measurements use Qwen3-1.7B on a single RTX A6000 48GB GPU, with BF16, batch size=1, and 128 soft tokens per SP expert. Each configuration is fully warmed up and measured three times. TTFT includes TAME's input-level probe and controlled prefill; decoding throughput measures generation after the first token. The results are shown below.

MethodDecoding throughput / token/sTime to first token / ms
TIES54.6519.89
TAME53.3041.05

The two methods have similar decoding throughput, with TAME incurring a higher time to first token. For each input, TAME performs an additional input-level probe before generation and then reprocesses the input. These measurements show similar throughput during the optimized generation stage, with the additional overhead reflected in TTFT.

References

[1] Gautam Rajendrakumar Gare et al. Your Model Already Knows Don't Teach It, Learn to Ask It: Soft Prompting for Few-Shot Adaptation of Vision-Language Models. arXiv:2609.11310, 2026.

Reviewer_uGF5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This is an interesting and well written paper that tackles the issue of training experts within an MoE configuration. The authors look at how soft prompts can be generated, selected and combined in a disciplined way that minimised composition loss whilst preserving expert independence. LoRA is used as the main comparitor, and evals are run on the GLUE benchmark. Results indicate a 1.9 pp improvement on mean accuracy and a reduction in task-level drop off that results from LoRA parameter merging.

Critical strengths
  • Nice range of comparitor methods (Plain joint, Structured Prompting ,InstABoost, SpotLight, centered top-1)
  • Results are presented as macro, mean, worst which gives an honest view of performance
  • Results are presented for sweeping over preference strength gamma but it's good to see that performance is relatively invariant (about 1% GLUE macro from low to high). I'd highlight though that performance seems more sensitive to gamma when we look at Table 2 so the positive spin around "Increasing γ does not turn the method into hard selection" needs qualifying.
  • Comparison to parameter-space merging method is informative and meaningful. As expected the parameter merging is more powerful on GLUE but comes with imbalance (Figure 4) in how it distributes its attention.
Critical weaknesses
  • Relatively older benchmarks (GLUE, GSM8K,MBPP), one base model (Qwen3-1.7B) raises questions about contamination and generalisability;
  • Results for comparitor methods (e.g. Plain joint, centered top-1) are only presented for GLUE. Does the trend remain consistent for the other benchmarks?
  • The results for GLUE are based on 6 of 8 task splits. It would be better to qualify this in headline reporting. At the same time it would be good also to report results (e.g. in an appendix) for all 8 tasks so we can have an apples-apples comparison for methods that can accommodate all 8 tasks.
  • Results on TAME 6-tasks do look like an improvement but there are variations among the tasks. Are these sufficiently highlighted anywhere?
  • Not critical but some of the tables and results figures are small and hard to read.
Questions for authors

One of the most interesting aspects of the work for me was in the method to select the best soft prompt. Currently you provide a centered top-1 baseline but I'd be curious to know whether the reference chosen by the probe is optimal for your two-level control. If you could show a comparison against an oracle reference (e.g. based on known task identity), that would strengthen the paper.

5 comments
AuthorsOpenReview ↗

Oracle Reference Selection

We thank the reviewer for recognizing the merits of our method. Following your comments, we have added cross-domain composition baselines and an eight-task GLUE evaluation, and have further extended the experiments to multimodal tasks. Below, we first address your question and then discuss the experimental scope and results in the order of the weaknesses raised.

Question 1

One of the most interesting aspects of the work for me was in the method to select the best soft prompt. Currently you provide a centered top-1 baseline but I'd be curious to know whether the reference chosen by the probe is optimal for your two-level control. If you could show a comparison against an oracle reference (e.g. based on known task identity), that would strengthen the paper.

Response

Thank you for this suggestion. We have added a Task-identity oracle reference comparison using known task identities in the original six-task validation setting, directly examining how the choice of reference Bank affects two-level control.

This comparison retains joint readout from all experts and replaces only the reference Bank used for mass control with the expert Bank corresponding to the input's task. All other parts of the control procedure remain unchanged. Centered top-1 retains only the single Bank selected by the probe. The added comparison therefore examines the effect of reference selection more directly.

MethodMacro AccuracyMean retentionWorst-task retention
SP single82.19%——
Centered top-174.55%91.65%70.35%
TAME (Probe reference)77.01%94.40%84.09%
TAME (Task-identity oracle reference)77.93%95.39%87.60%

Using the task-identity reference improves TAME's Macro Accuracy from 77.01% to 77.93%, a gain of 0.92 percentage points. When checked against the known task identities, the probe selects the corresponding task expert for 35,048 of 48,156 samples, a match rate of 72.78%. This comparison identifies a performance gap between the probe-selected reference and the task-identity reference under TAME's two-level control. In practical inference, the probe estimates a suitable reference Bank directly from the input, providing a practical choice when task identity is unavailable.

The comparison with Centered top-1 shows that TAME improves macro-average task performance and increases tolerance to reference mismatches. We further analyze the 13,108 samples for which the selected reference differs from the corresponding task expert:

Reference matches task expertNumber of samplesCentered wrong, TAME correctCentered correct, TAME wrongNet gain in correct answers
No13,1081,465434+1,031

Among these samples, 1,899 have different correctness outcomes under the two methods. On the mismatched-reference subset, TAME yields 1,031 more correct answers than Centered top-1, showing that controlled joint readout can compensate for some errors when the reference differs from the corresponding task expert.

Probe-based reference selection thus provides an effective alternative when task identity is unavailable, with six-task macro accuracy only 0.92 percentage points below the task-identity oracle. Controlled joint readout further compensates for losses associated with reference mismatches. The oracle comparison and paired-correctness analysis are now in Appendix E.2 (Tables 9–10), highlighted in blue.

AuthorsOpenReview ↗

Multimodal, Cross-domain, and GLUE Evaluation

Weakness 1

Relatively older benchmarks (GLUE, GSM8K,MBPP), one base model (Qwen3-1.7B) raises questions about contamination and generalisability;

Response

To broaden the models and tasks covered by our evaluation, we conducted additional RF20 multimodal few-shot object detection experiments after completing the original manuscript. Our study focuses on composition loss when SP can already produce effective experts, and the new experiments follow this objective by using a setting well suited to SP.

Prior work [1] shows that, in RF20's 10-shot object detection setting, training only 1–3 soft tokens with suitable prompt placement and initialization can match the average performance of the best LoRA configuration evaluated in that study. Roboflow20-VL (RF20) is an object detection subset of Roboflow-100-VL, comprising 20 tasks across seven categories: aerial, documents, flora and fauna, industrial, medical, sports, and other. We therefore selected this few-shot vision-language setting and used Qwen3-VL-4B-Instruct. Following the soft-prompt tuning procedure in that work, we trained one SP expert with 1–3 soft-prompt tokens per domain and then combined all 20 experts using TAME. The results are shown below.

MethodMean mAP@50:95Overall retentionWorst-domain retention
SP single experts16.820——
LoRA single experts17.013——
TA14.28383.95%2.62%
TIES13.83481.31%3.09%
Plain Joint8.92253.04%0.00%
TAME14.38485.51%12.41%

With equal weighting across the 20 domains, SP single experts achieve a mean mAP close to that of LoRA single experts. TAME outperforms TA and TIES in mean mAP, overall retention, and worst-domain retention. Compared with direct concatenation in context space (Plain Joint), TAME improves mean mAP from 8.922 to 14.384. The complete RF20 results are now in Appendix D (Table 7), highlighted in blue.

We have also expanded and refined the original experiments to provide further support for our method, as detailed below.

Weakness 2

Results for comparitor methods (e.g. Plain joint, centered top-1) are only presented for GLUE. Does the trend remain consistent for the other benchmarks?

Response

We have added comparisons with Plain joint, Centered top-1, Mass-only, and parameter-space merging methods in the cross-domain combination of mathematics, coding, and instruction following in Section 4.4 (Table 3; added comparisons highlighted in blue). With the same frozen SP experts, TAME achieves higher cross-domain average scores and mean retention than these context-space composition baselines, consistent with the observations on GLUE.

Math, Code, and IF correspond to GSM8K, MBPP, and IF Strict Prompt, respectively. The results are shown below. Each composition method's retention is calculated relative to its corresponding single experts.

MethodMathCodeIFCross-domain averageMean retention
Independent LoRA experts73.2445.5668.0262.27—
TA68.7642.2240.6750.5582.12%
TIES73.2443.3358.9658.5193.93%
Independent SP experts70.1338.8967.4758.83—
Plain joint52.0841.1141.2244.8180.36%
Centered top-170.0535.5653.9753.1990.44%
Mass-only (γ=0)64.2932.2257.1251.2186.40%
TAME (γ=1)69.9038.8956.1954.9994.32%
TAME (γ=2)70.4338.8960.0756.4696.49%

With the same frozen SP experts, TAME achieves higher cross-domain average scores and mean retention than the evaluated context-space composition baselines.

In the comparison with parameter-space merging, TIES achieves a higher absolute average score after composition, while TAME achieves higher mean retention. This is consistent with the new eight-task GLUE results and further illustrates how the composition methods differ in final performance and capability retention.

Weakness 3

The results for GLUE are based on 6 of 8 task splits. It would be better to qualify this in headline reporting. At the same time it would be good also to report results (e.g. in an appendix) for all 8 tasks so we can have an apples-apples comparison for methods that can accommodate all 8 tasks.

Response

We have added an eight-task evaluation on the official test set, including MNLI, MRPC, and STS-B, and report complete per-task scores and capability retention. These results are now in Appendix C (Table 6), highlighted in blue.

References

[1] Gautam Rajendrakumar Gare et al. Your Model Already Knows Don't Teach It, Learn to Ask It: Soft Prompting for Few-Shot Adaptation of Vision-Language Models. arXiv:2609.11310, 2026.

AuthorsOpenReview ↗

GLUE Results, Task Variation, and Presentation

Weakness 3 (continued)

The new experiments cover CoLA, SST-2, QQP, QNLI, RTE, MRPC, MNLI, and STS-B. All test scores come from the official GLUE evaluation server. The task coverage, which excludes WNLI, is consistent with the test reporting practice of BERT [2] and related work.

MethodCoLA MCCSST-2 Acc.QQP Acc. / F1QNLI Acc.RTE Acc.MRPC Acc. / F1MNLI m / mm Acc.STS-B Pearson / SpearmanEight-task averageMean retentionWorst-task retention
LoRA single experts56.896.390.0 / 73.594.585.683.1 / 87.489.7 / 89.290.9 / 92.685.18——
TA30.392.786.5 / 61.582.780.971.8 / 76.272.4 / 74.083.5 / 92.374.4685.82%53.35%
TIES25.694.688.5 / 67.188.179.375.4 / 80.580.9 / 82.178.2 / 94.676.4187.63%45.07%
SP single experts55.895.688.1 / 70.792.484.679.4 / 84.988.5 / 88.088.0 / 90.283.41——
Plain joint0.089.082.8 / 21.357.062.268.2 / 73.562.2 / 64.944.9 / 63.256.0964.10%0.00%
TAME33.891.382.9 / 59.582.577.871.9 / 78.178.0 / 78.281.5 / 95.274.7688.24%60.57%

The eight-task average is calculated by first averaging the reported metrics within each task and then taking an equally weighted average across the eight tasks. Retention is calculated relative to the corresponding single-expert score for each task; we report the mean and minimum of these task-level retention rates.

For this broader task combination, TAME achieves a mean retention of 88.24%, substantially above direct joint readout in context space and also above TIES (87.63%) and TA (85.82%). LoRA experts retain stronger single-task performance, as their larger trainable parameter budget enables them to fit the tasks more effectively. After composition, TAME has a lower absolute average score than TIES and higher mean and worst-task retention than both TA and TIES. With the previously omitted tasks included, these results continue to support TAME's effectiveness in reducing composition loss in this setting. These test results are now in Appendix C (Table 6), highlighted in blue.

Weakness 4

Results on TAME 6-tasks do look like an improvement but there are variations among the tasks. Are these sufficiently highlighted anywhere?

Response

Differences across tasks also motivate our per-task analysis. Table 5 and Figure 4 in the revised manuscript present task-level absolute performance and changes relative to the respective single experts. The added discussion in Section 5.1 is highlighted in blue.

Figure 4 shows task performance changes relative to the respective single experts. The four parameter merging methods exhibit similar patterns of large drops on tasks such as CoLA and RTE. This is consistent with the observation by Cao et al. [3] that severe degradation can recur for certain task combinations across different parameter merging methods.

TAME also exhibits regressions on tasks such as QQP. In the original six-task setting, its overall degradation pattern is milder, with better preservation of capabilities on the vulnerable tasks identified above. Its worst-task retention is 84.1%, compared with 64.0%–73.2% for the four parameter merging methods. This observation is consistent with preserving independent KV Banks and coordinating their readout through two-level control, supporting the interpretation that this design mitigates composition interference in settings where SP adapts effectively.

The per-task scores for the new eight-task setting are also listed above, where TAME achieves higher mean and worst-task retention than the evaluated TA and TIES baselines. Appendix C (Table 6) reports these metrics together, with the additions highlighted in blue.

Weakness 5

Not critical but some of the tables and results figures are small and hard to read.

Response

Thank you for this suggestion. We have redrawn a simpler framework diagram that clearly separates input-level preference probing from two-level attention control. Figure 2 in Section 3 now uses a vector format with larger labels and a clearer layout; its revised caption is highlighted in blue.

References

[2] Jacob Devlin et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT, 2019.

[3] Yuan Cao et al. An Empirical Study and Theoretical Explanation on Task-Level Model-Merging Collapse. arXiv:2603.09463, 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Additional Evidence

Dear Area Chair and Reviewers,

Thank you for your careful review and constructive feedback. We have responded to all comments and revised the manuscript with additional experiments, clearer protocols, and an improved method diagram. We summarize the main responses and evidence below.

Our central question is how to reduce composition loss without additional fusion training when independently trained soft prompts already serve as effective experts. TAME coordinates joint readout by separately controlling the expert group's total readout strength and the allocation among experts while preserving expert independence.

1. Clarifying comparison scope and adding cross-domain baselines

Related reviewer comments: uGF5’s Weakness 2; UqJe’s Question 1 and Weakness 1; uXQe’s Weaknesses 2–3.

To isolate composition effects, we compare methods using the same frozen SP experts. On the original six-task GLUE setting, TAME improves Macro Accuracy from 66.67% with Plain Joint to 77.01%. We also add Plain Joint, Centered top-1, and Mass-only baselines for mathematics, coding, and instruction following. TAME achieves higher average scores and mean retention than these shared-expert baselines (Section 4.4, Table 3).

For LoRA merging comparisons, we report absolute scores, relative retention, and per-task changes together. We explicitly acknowledge that TIES achieves higher absolute average scores in the added eight-task GLUE and cross-domain experiments, while TAME achieves higher mean retention.

2. Expanding task coverage and adding multimodal, 20-expert experiments

Related reviewer comments: uGF5’s Weaknesses 1 and 3; UqJe’s Question 1 and Weakness 2; uXQe’s Questions 3–4 and Weakness 1.

The added eight-task official GLUE test evaluation includes MNLI, MRPC, and STS-B. TAME substantially outperforms Plain Joint and achieves higher mean and worst-task retention than TA and TIES (Appendix C, Table 6).

We further evaluate 20-expert composition on RF20 few-shot object detection using Qwen3-VL-4B-Instruct. TAME improves mean mAP from 8.922 with Plain Joint to 14.384 and outperforms TA and TIES in mean mAP, overall retention, and worst-domain retention (Appendix D, Table 7). These experiments extend the evaluation to another backbone, multimodal tasks, and more experts.

3. Examining budget and reference strategies

Related reviewer comments: uGF5’s Question 1; UqJe’s Question 3 and Weakness 3.

We add fixed-budget, average-budget, and all six fixed-reference comparisons. Mass control provides most of the gain, with within-group allocation offering further improvement. Budgets adapting to the current computation outperform the tested fixed budget. We also report reference-strategy trade-offs: the fixed QQP reference achieves accuracy close to TAME and higher worst-task retention.

Using the known task identity as the reference yields accuracy only 0.92 percentage points above the probe reference. On samples with mismatched references, TAME produces 1,031 more correct answers than Centered top-1, supporting the ability of controlled joint readout to compensate for some reference mismatches (Section 4.3.1; Appendices E.1-E.2).

4. Quantifying overhead and clarifying protocols

Related reviewer comments: UqJe’s Question 2 and Weakness 4; uXQe’s Questions 1–2.

With Qwen3-1.7B, batch size 1, and eight experts, optimized TAME has 2.47% lower decoding throughput than TIES, while time to first token increases from 19.89 to 41.05 ms. With 128 soft tokens per expert, increasing expert count from 1 to 8 adds 134.6 MiB of peak allocated GPU memory, with approximately stable latency and throughput (Section 4.5; Appendix F).

Calibration uses 128 training inputs per task without target answers, and statistics remain frozen during evaluation. GLUE uses γ=1 fixed in advance. Cross-domain γ configurations are compared on the same evaluation set, without a separate selection set (Section 4.1; Appendix E.4).

We also expand the per-task degradation discussion (uGF5’s Weakness 4; uXQe’s Weakness 3), clarify the introduction's research framing, and redraw the vector method diagram to address presentation concerns (uGF5’s Weakness 5; UqJe’s Question 4; uXQe’s Weaknesses 4–5).

We hope these additions clarify the evidence and scope of our conclusions and assist your assessment of the submission and responses. Thank you for your time and consideration.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.