- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes TAME, a training-free method that composes independently trained soft prompt experts. Each prompt is encoded as a layer-wise KV Bank. An input-level attention probe estimates expert preference and selects a reference Bank. Two attention controls then set the total readout strength of the expert group and the share of each expert. Experiments compare context-space composition with parameter merging on six GLUE tasks, GSM8K, MBPP and IF, and report 94.4% mean capability retention.
Critical strengths
The two-level readout is a useful decomposition, and the mass-only ablation shows the role of group-level control.
The cross-space comparison gives a more informative view than a single average score.
Critical weaknesses
The method is tested on one backbone. The GLUE comparison drops MNLI, MRPC and STS-B because the experts were not strong enough. The conclusion therefore depends on a selected task subset.
The comparison with LoRA merging mixes two expert families. Soft prompt experts and LoRA experts have different single-expert references and different training procedures. A direct Macro comparison is not convincing.
Mean retention hides the absolute drop on individual tasks.
The intro is not clear and the paper's writing can't reach the standard.
The intro and framework pictures contain a lot of meaningless information.
Questions for authors
Are Table 1 and Table 2 reported on the development set or the test set? Does gamma selection use the same samples?
How is the calibration set built? Does it contain in-domain data from each task?
Why are MNLI, MRPC and STS-B excluded? Do the conclusions hold when they are added?
How does the method behave when the number of experts grows beyond six?
6 comments
GLUE Coverage and Cross-domain Baselines
Question 3 and Weakness 1
Why are MNLI, MRPC and STS-B excluded? Do the conclusions hold when they are added?
The method is tested on one backbone. The GLUE comparison drops MNLI, MRPC and STS-B because the experts were not strong enough. The conclusion therefore depends on a selected task subset.
Response
We have added an eight-task evaluation on the official test set, including MNLI, MRPC, and STS-B, and report per-task scores and capability retention. Using the same SP experts, TAME substantially outperforms the context-space baseline Plain joint. It also maintains higher mean and worst-task retention than the evaluated parameter-space merging methods, consistent with the observations in the original manuscript. We have further expanded the comparisons in the three-task cross-domain merging setting from Section 4.4.
Our study asks how to combine independently trained soft-prompt experts and reduce composition loss, without additional fusion training, when the individual soft prompts can already adapt effectively to the corresponding tasks. The original experiments therefore prioritize tasks where SP and LoRA single experts achieve similar performance, reducing the influence of differences in initial capabilities on the comparison of composition loss. This rationale is clarified in Appendix B.1, with the revision highlighted in blue.
To expand task coverage, we have added an eight-task experiment on CoLA, SST-2, QQP, QNLI, RTE, MRPC, MNLI, and STS-B. All test scores come from the official GLUE evaluation server.
| Method | CoLA MCC | SST-2 Acc. | QQP Acc. / F1 | QNLI Acc. | RTE Acc. | MRPC Acc. / F1 | MNLI m / mm Acc. | STS-B Pearson / Spearman | Eight-task average | Mean retention | Worst-task retention |
|---|---|---|---|---|---|---|---|---|---|---|---|
| LoRA single experts | 56.8 | 96.3 | 90.0 / 73.5 | 94.5 | 85.6 | 83.1 / 87.4 | 89.7 / 89.2 | 90.9 / 92.6 | 85.18 | — | — |
| TA | 30.3 | 92.7 | 86.5 / 61.5 | 82.7 | 80.9 | 71.8 / 76.2 | 72.4 / 74.0 | 83.5 / 92.3 | 74.46 | 85.82% | 53.35% |
| TIES | 25.6 | 94.6 | 88.5 / 67.1 | 88.1 | 79.3 | 75.4 / 80.5 | 80.9 / 82.1 | 78.2 / 94.6 | 76.41 | 87.63% | 45.07% |
| SP single experts | 55.8 | 95.6 | 88.1 / 70.7 | 92.4 | 84.6 | 79.4 / 84.9 | 88.5 / 88.0 | 88.0 / 90.2 | 83.41 | — | — |
| Plain joint | 0.0 | 89.0 | 82.8 / 21.3 | 57.0 | 62.2 | 68.2 / 73.5 | 62.2 / 64.9 | 44.9 / 63.2 | 56.09 | 64.10% | 0.00% |
| TAME | 33.8 | 91.3 | 82.9 / 59.5 | 82.5 | 77.8 | 71.9 / 78.1 | 78.0 / 78.2 | 81.5 / 95.2 | 74.76 | 88.24% | 60.57% |
The eight-task average is calculated by first averaging the reported metrics within each task and then taking an equally weighted average across the eight tasks. Retention is calculated relative to the corresponding single-expert score for each task; we report the mean and minimum of these task-level retention rates.
For this broader task combination, TAME achieves a mean retention of 88.24%, substantially above direct joint readout in context space and also above TIES (87.63%) and TA (85.82%). LoRA experts retain stronger single-task performance, as their larger trainable parameter budget enables them to fit the tasks more effectively. After composition, TAME has a lower absolute macro score than TIES and higher mean and worst-task retention than both TA and TIES, consistent with the observations in the original manuscript. These test results are now in Appendix C (Table 6), highlighted in blue.
We have also expanded the baseline comparisons for the cross-domain combination of mathematics, coding, and instruction following in Section 4.4 (Table 3), with the additions highlighted in blue. With the same frozen SP experts, TAME achieves higher mean capability retention than direct concatenation, single-expert selection, and attention-mass-only control, as shown below.
| Method | Math | Code | IF | Cross-domain average | Mean retention |
|---|---|---|---|---|---|
| Independent LoRA experts | 73.24 | 45.56 | 68.02 | 62.27 | — |
| TA | 68.76 | 42.22 | 40.67 | 50.55 | 82.12% |
| TIES | 73.24 | 43.33 | 58.96 | 58.51 | 93.93% |
| Independent SP experts | 70.13 | 38.89 | 67.47 | 58.83 | — |
| Plain joint | 52.08 | 41.11 | 41.22 | 44.81 | 80.36% |
| Centered top-1 | 70.05 | 35.56 | 53.97 | 53.19 | 90.44% |
| Mass-only (γ=0) | 64.29 | 32.22 | 57.12 | 51.21 | 86.40% |
| TAME (γ=1) | 69.90 | 38.89 | 56.19 | 54.99 | 94.32% |
| TAME (γ=2) | 70.43 | 38.89 | 60.07 | 56.46 | 96.49% |
With the same frozen SP experts, TAME achieves higher cross-domain average scores and mean retention than the evaluated context-space composition baselines.
Cross-domain Comparison, Expert Scaling, and Shared-Expert Baselines
Question 3 and Weakness 1 (continued)
In the comparison with parameter-space merging, TIES achieves a higher absolute average score after composition, while TAME achieves higher mean retention relative to the respective single experts. This is consistent with the results from the expanded GLUE setting. These findings illustrate the trade-offs between final performance and capability retention across composition methods and support TAME's effectiveness in reducing composition loss in this cross-domain setting. The added discussion is in Section 5.1, highlighted in blue.
To evaluate another backbone and additional tasks, we have also conducted a larger experiment on multimodal few-shot object detection, detailed in our response to Question 4 below.
Question 4
Original Question 4
How does the method behave when the number of experts grows beyond six?
Response
Our response to Question 3 includes the combination of eight experts across eight tasks, where TAME achieves 88.24% mean retention, providing evidence of capability retention beyond six experts (Appendix C, Table 6; additions highlighted in blue). Following completion of the original manuscript, we also conducted RF20 multimodal experiments to evaluate TAME in a broader expert-composition setting.
Prior work [6] shows that, in RF20's 10-shot object detection setting, training only 1–3 soft tokens with suitable prompt placement and initialization can match the average performance of the best LoRA configuration evaluated in that study. Roboflow20-VL (RF20) is an object detection subset of Roboflow-100-VL, comprising 20 tasks across seven categories: aerial, documents, flora and fauna, industrial, medical, sports, and other. We therefore selected this few-shot vision-language setting and used Qwen3-VL-4B-Instruct. Following the soft-prompt tuning procedure in that work, we trained one SP expert with 1–3 soft-prompt tokens per domain and then combined all 20 experts using TAME. The results are shown below.
| Method | Mean mAP@50:95 | Overall retention | Worst-domain retention |
|---|---|---|---|
| SP single experts | 16.820 | — | — |
| LoRA single experts | 17.013 | — | — |
| TA | 14.283 | 83.95% | 2.62% |
| TIES | 13.834 | 81.31% | 3.09% |
| Plain Joint | 8.922 | 53.04% | 0.00% |
| TAME | 14.384 | 85.51% | 12.41% |
With equal weighting across the 20 domains, SP single experts achieve a mean mAP close to that of LoRA single experts. TAME outperforms TA and TIES in mean mAP, overall retention, and worst-domain retention. Compared with direct concatenation in context space (Plain Joint), TAME improves mean mAP from 8.922 to 14.384. The complete RF20 results are now in Appendix D (Table 7), highlighted in blue.
Weaknesses 2 and 3
The comparison with LoRA merging mixes two expert families. Soft prompt experts and LoRA experts have different single-expert references and different training procedures. A direct Macro comparison is not convincing.
Mean retention hides the absolute drop on individual tasks.
Response
We evaluate TAME's composition effectiveness by comparing methods that use the same frozen SP experts. For comparisons across merging spaces, we report absolute scores, relative retention, and per-task changes together. Table 5 and Figure 4 in the revised manuscript present task-level absolute performance and changes relative to the respective single experts, making the losses and trade-offs behind the averages directly visible.
In Table 1 of the revised manuscript, TAME and context-space composition methods such as Plain joint, Centered top-1, and StructuredPrompting use the same frozen soft-prompt experts. TAME improves Macro Accuracy from 66.67% with Plain joint to 77.01%, a gain of 10.34 percentage points, achieving the highest score among the context-space composition methods. This improvement demonstrates the effect of changing the composition method. The additional cross-domain comparisons in our response to Question 3 also use the same frozen experts, with TAME achieving the best overall performance among methods using the same soft-prompt experts. These results further evaluate capability retention across different task combinations. The shared-expert comparison protocol is clarified in Section 4.1 and Appendix E.4, highlighted in blue.
References
[6] Gautam Rajendrakumar Gare et al. Your Model Already Knows Don't Teach It, Learn to Ask It: Soft Prompting for Few-Shot Adaptation of Vision-Language Models. arXiv:2609.11310, 2026.
Evaluation Protocol and Calibration
We thank the reviewer for recognizing the two-level decomposition of attention readout and the analysis of capability retention across merging spaces. Following your comments, we clarify the scope of our study, the purpose of the comparisons, and the evaluation protocol. We have added an eight-task evaluation covering the previously omitted tasks, official test-set results, generalization experiments on additional tasks, and experiments with more experts. We have also redrawn the framework diagram to make it simpler and clearer. Below, we respond to the questions and weaknesses in order.
Question 1
Response
Table 1 in the revised manuscript uses the official GLUE development set, and Table 3 uses the public evaluation sets for the cross-domain tasks. We have also added eight-task results returned by the official GLUE test server. For the main GLUE experiments, we fixed γ=1 in advance. For the cross-domain experiments, the comparisons of γ configurations and the reported results use the same evaluation samples.
The original GLUE results are evaluated on the official development set (validation split), with the same evaluation samples used for all methods. Reporting results on this split is consistent with published work in related areas, including the model merging methods DARE [1] and CAT Merging [2], and the parameter-efficient fine-tuning methods LoRA-GA [3] and Flat-LoRA [4]. To address your concerns about task coverage and evaluation reliability, we have added official GLUE test-set evaluations on CoLA, SST-2, QQP, QNLI, RTE, MNLI, MRPC, and STS-B, including the three previously omitted tasks you identified. This eight-task setting follows the common practice of excluding WNLI, consistent with the task coverage used by BERT [5] and related work when reporting official test results. All new aggregate results are calculated from scores returned by the official GLUE evaluation server. TAME achieves a mean capability retention of 88.24% and a worst-task retention of 60.57%, providing further evidence of capability retention beyond the original task setting. The detailed results appear in Appendix C (Table 6), highlighted in blue, and in our response to Question 3 below.
Table 3 in the revised manuscript uses the public evaluation sets for GSM8K, MBPP, and IF Strict Prompt. All configurations use the same evaluation samples and fixed calibration inputs, with no separate set for selecting γ. The evaluation protocol is clarified in Section 4.1 and Appendix E.4, and the configuration comparisons are reported in Section 4.3.2 and Appendix E.3, with the corresponding revisions highlighted in blue.
Question 2
Response
The calibration set contains an equal number of in-domain training inputs from each task or domain being combined. Calibration uses only the inputs to estimate each Bank's mean probe score, without accessing target answers, and the resulting statistics remain frozen throughout evaluation. All calibration inputs come from the corresponding training splits. In both the GLUE experiments (Table 1) and the three-task cross-domain experiments (Table 3), we use a fixed set of 128 training inputs per task.
The calibration statistics center the probe scores to reduce differences in the average response of different Banks. During evaluation, these statistics remain frozen, and the probe results for the current input determine the expert preferences and reference Bank. The model receives no additional task or domain labels. The calibration details are provided in Section 4.1 and Appendix E.4, highlighted in blue.
References
[1] Le Yu et al. Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch. ICML, 2024.
[2] Wenju Sun et al. CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging. ICML, 2025.
[3] Shaowen Wang et al. LoRA-GA: Low-Rank Adaptation with Gradient Approximation. NeurIPS, 2024.
[4] Tao Li et al. Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape. ICML, 2025.
[5] Jacob Devlin et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT, 2019.