acceptodds
Under review as a conference paper at ICLR 2027

Compositional Learnware Identification and Gradient-Guided Assembly

Abstract

Tasks and query batches often require diverse capabilities, motivating complementary model reuse from growing repositories. The learnware paradigm—Learnware = Model + Specification—provides reusable capability representations, supporting such reuse without evaluating every candidate on the target task. PAVE (Parameter Vector) specifications encode model capabilities and user requirements as parameter updates in a common space. However, independently ranking models by PAVE similarity can favor overlapping capabilities, while collection-level PAVEs provide limited evidence for assigning individual inputs. We address these challenges through complementary identification and gradient-guided sample-level assignment. Inspired by task arithmetic, we formulate identification as sparse PAVE reconstruction, using the unexplained user-PAVE residual to guide complementary model selection. For assignment, we exploit sample gradients as local adaptation signals, calibrated using collection-level statistics, to match individual inputs with selected model PAVEs. Under a local update approximation, our analysis relates gradient–PAVE matching to average gradient-kernel similarity between a sample and a specification's fitting distribution. Across 72 mixed LLM batches, 720 mixed CV/NLP runs, and 41 single-dataset benchmarks, our method consistently improves model reuse without training a parametric selector. It exceeds the strongest compared learning-based router by 1.06 points on mixed LLM workloads and improves over hindsight Best-Single by 10.62/2.52 points on mixed CV/NLP tasks, while invoking one candidate per input. Extensive controlled ablations further validate the contributions of complementary identification and gradient-guided assembly, as well as the effects of gradient calibration and common-direction removal.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Machine learning, Learnware, PAVE specifications, Compositional model identification, Model reuse

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SsQNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To counter the two failure modes—independent similarity ranking favoring overlapping capabilities, and collection-level specifications lacking per-sample evidence, this paper proposes a two-stage method: first, complementary models are identified via sparse linear reconstruction of the user PAVE (solved with OMP) after common-direction removal, inspired by the linear compositionality of task arithmetic; second, sample-level assignment matches gradients of individual inputs, calibrated by collection-level statistics, against the selected models' hierarchical PAVEs, with a soft-teacher transfer from limited labeled data for the task scenario. Under an NTK/local-update approximation, the paper interprets gradient–PAVE matching as average gradient-kernel similarity between a sample and a specification's fitting distribution. Experiments on systems of 10 black-box LLMs, 12 CV, and 15 NLP models—covering 72 mixed LLM batches, 720 mixed CV/NLP runs, and 41 single-dataset tasks, with balanced/long-tail mixtures, out-of-system benchmarks, leave-one-out protocols, and component-level ablations—show that, without training a parametric selector, the method outperforms the strongest learned router by 1.06 points and hindsight Best-Single by 10.62/2.52 points on mixed CV/NLP, invoking one candidate per input.

Critical strengths

S1: Theoretically grounded: sparse reconstruction of the user PAVE is inspired by task-arithmetic linearity, and gradient–PAVE matching is analyzed as a kernel query (average gradient-kernel similarity) under a local-update approximation.

S2: Common-direction removal is well motivated and validated: ablating it costs 1.80 Utility points across 72 LLM mixed-query conditions.

S3: Gradient calibration is isolated and evidenced: it contributes +2.00 points, and improves every ranking metric plus top-1 accuracy in the matching-only analysis.

S4: Well-positioned contributions across learnware, routing, task arithmetic, and gradient representations, with cleanly separated claims.

Critical weaknesses

W1: The central NTK/local-update approximation (gradients frozen at initialization, ignoring gradient evolution and adaptive scaling) is unverified empirically, yet it underpins the entire interpretation.

W2: Statistical power is limited: "3 seeds" only randomize mixture construction; the 1.06-point LLM gain is small relative to a ±1.15 SD, and no significance tests are reported.

W3: Modest gains over learned routers: 1.06 points on mixed LLM workloads and only 9 wins vs 5 losses on single-dataset LLM benchmarks, with near-ties against several baselines.

W4: No sensitivity analysis for key hyperparameters (budget K, 95% explainability threshold, common rank r_c=1, calibration ε), even though K directly caps achievable performance.

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer SsQN

We sincerely appreciate your time and input. We are grateful for your recognition of the theoretical motivation, common-direction removal, calibration evidence, and positioning of our contributions.

W1. Local-update interpretation and evidence for matching.

Thank you for raising this concern. In practice, the local-update approximation is not always tight. We examine this by comparing fitted PAVEs with the negative mean initial gradient on their respective fitting samples. The 1,565 distinct specifications include full and cluster model PAVEs and user PAVEs.

ModalityNumber of PAVEsMedian cosine
CV5720.606
NLP3040.433
LLM6890.467

These similarities show differences between fitted PAVEs and the initial mean descent direction. Gradient evolution and adaptive coordinate scaling can affect the local-update approximation. Given these simplifications, our local-update analysis is intended to help explain gradient–PAVE matching, rather than provide a theoretical guarantee of the method’s effectiveness. This simplified gradient-kernel interpretation helps us understand the matching mechanism and guides its design and calibration.

Meanwhile, targeted experiments provide empirical support for the effectiveness of gradient–PAVE matching. In particular, we test whether matching can select suitable models for labeled support samples in CV and NLP mixed conditions. Each sample selects the highest-matching model, without auxiliary scores or teacher transfer. Results average conditions equally and use informative classification samples with both correct and incorrect candidates; random expectation is each sample's correct-candidate proportion.

ModalityRandom expectation (%)Matching top-1 accuracy (%)AUC
CV46.3294.100.801
NLP57.0069.030.647

Matching exceeds the random reference in both modalities, supporting its value for sample-level model selection. The paper also reports improved pure-matching rankings after calibration (Section 4.3.1; Appendix E.3.1, Table A21) and a 2.00-point drop in LLM mixed Utility when calibration is removed with candidates fixed (Section 4.3.2, Table 2).

W2 & W3. LLM gains and statistical reliability.

Thank you for raising concerns about the statistical reliability of the LLM gains. We have added paired significance tests and confidence intervals for both single-dataset and mixed evaluation, together with an average-rank analysis across the 14 single-dataset benchmarks.

The table below reports paired Utility differences against Random and UniRoute under the unchanged configurations, with 95% confidence intervals and two-sided pp-values.

SettingBaselineOurs − baseline (pp)95% CI (pp)pp-value
SingleRandom+3.84[+3.14, +4.51]0.00001
SingleUniRoute+2.62[+1.98, +3.23]0.00001
MixedRandom+2.37[+1.09, +3.70]0.00058
MixedUniRoute+1.06[-0.69, +2.82]0.23904

We use 2,000 dataset-stratified paired query-ID cluster bootstrap draws for 95% percentile intervals and 100,000 cluster-level method-label swaps for two-sided pp-values. All occurrences of the same query are grouped together during resampling and permutation. Significance requires p<0.05p < 0.05.

Our method significantly outperforms Random in both mixed and single-dataset evaluation. Although the mixed-setting gain over UniRoute is not statistically significant, the single-dataset gain is significant (+2.62 points; p=0.00001p = 0.00001).

The reported nine wins and five losses refer specifically to the comparison with UniRoute across the 14 single-dataset benchmarks. To assess our method's overall standing, we rank all nine methods on each dataset and average these ranks equally across datasets, assigning average ranks to ties. Our method achieves the best average rank of 2.75, ahead of UniRoute (4.39) and Random (6.18), as shown below.

MethodAverage rank
Ours2.75
UniRoute4.39
Random6.18
AuthorsOpenReview ↗

Response to Reviewer SsQN (cont'd)

W4. Hyperparameter sensitivity and candidate budget.

Thank you for requesting a sensitivity analysis. We have added sensitivity analysis of calibration ϵ\epsilon, common rank, explained-fraction threshold, and candidate budget across all six settings.

Each parameter is varied separately; all other settings follow the paper. The results show limited sensitivity to calibration ϵ\epsilon and common rank, while the explained-fraction threshold and KK have larger effects through candidate coverage.

ϵ\epsilonCV mixedCV single datasetNLP mixedNLP single datasetLLM mixedLLM single dataset
10−1010^{-10}83.8137 ± 0.628866.478873.7341 ± 0.133570.897180.64 ± 0.8481.44
10−810^{-8} (default)83.8139 ± 0.628766.502073.7341 ± 0.133570.896981.16 ± 1.1581.78
10−610^{-6}83.8175 ± 0.629266.628873.7336 ± 0.131570.889981.25 ± 1.0081.14

Utility remains stable across four orders of magnitude in calibration ϵ\epsilon, with negligible changes on CV/NLP and modest changes on LLMs.

rcr_cCV mixedCV single datasetNLP mixedNLP single datasetLLM mixedLLM single dataset
1 (default)83.81 ± 0.6366.5073.73 ± 0.1370.9081.16 ± 1.1581.78
283.01 ± 0.2066.4974.00 ± 0.3369.8480.69 ± 1.2480.15
483.57 ± 0.3366.4874.08 ± 0.0770.2780.97 ± 1.4381.22

Utility shows limited sensitivity to the common rank within the tested range.

ρ\rhoCV mixedCV single datasetNLP mixedNLP single datasetLLM mixedLLM single dataset
85%76.73 ± 0.1366.1172.11 ± 0.2870.5681.16 ± 1.4379.01
90%78.38 ± 0.3466.2572.62 ± 0.3870.8381.16 ± 1.1580.75
95%83.81 ± 0.6366.5073.73 ± 0.1370.9080.97 ± 1.0181.60
98%87.04 ± 0.2066.5074.52 ± 0.0470.9081.47 ± 1.0081.78

With the original candidate caps, higher explained-fraction thresholds improve CV/NLP mixed Utility but generally increase the number of candidates processed.

KKCV mixedCV single datasetNLP mixedNLP single datasetLLM mixedLLM single dataset
274.96 ± 0.3265.8470.05 ± 0.3269.8279.06 ± 1.8980.39
482.41 ± 0.3265.9473.73 ± 0.3269.2180.23 ± 1.9080.81
685.82 ± 0.3066.5474.31 ± 0.0971.0981.10 ± 1.1881.46
887.29 ± 0.0966.5874.45 ± 0.1270.5881.45 ± 0.9081.46
1087.57 ± 0.0766.6674.49 ± 0.0969.7980.69 ± 1.5379.67

With threshold stopping disabled, larger KK benefits CV/NLP mixtures, with diminishing returns on NLP beyond KK=6. NLP single-dataset Utility peaks at KK=6, and LLM mixed Utility peaks at KK=8. We use a bounded candidate budget to balance these gains against matching and assembly costs.

Reviewer_SsQNOpenReview ↗

Maintain my original score.

Thanks for your response. It addresses my questions. However, it does not change my evaluation, and I maintain my original score.

Thank you for carefully considering our response and confirming that it addresses your questions. We sincerely appreciate your time and constructive suggestions, which helped us clarify the scope of our analysis and strengthen the experimental evaluation.

Reviewer_1Gu3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The main contribution of this paper is to improve model reuse without parameter selectors. First, they use the OMP in PAVE space to choose the models (+complement models). Next, they use gradient matching to choose the detail models.

The authors evaluate their method on mixed and single dataset workloads in black-box LLM and CV/NLP tasks.

Critical strengths
  1. In the CV experiments, OMP performs better than cosine Top-K even with the same number of selected models.

  2. In the NLP experiments, the assembly method outperforms the supervised MLP, using the same candidate models and labeled data.

Critical weaknesses

The topic is interesting, but I think the figures and experiments need improvement. (In Fig.3, the text is too small, and the image quality is so low that it is difficult to read..)

  1. I think the experiments don't clearly show the contribution. In black-box LLM, gain over Random model is small and only 1.06 point over UniRoute. I think that more seeds and a significance test are needed. In CV/NLP, each dataset already has its own trained model, so this may explain the gain. So, I think more experiments are needed in these parts. Also, please compare with Wu et al. (2023), Tan et al. (2024b), and Liu et al. (2026b), or explain why these are not the baseline models.

  2. I also have concerns about reproducibility. I cannot find the exact model h(⋅,θ)h(\cdot, \theta) in Eq. (2). Also, no code is provided, so it is hard to check the results.

Questions for authors
  1. In the black-box LLM, can you use more seeds and check if the gains over Random and UniRoute are statistically significant?

  2. Can you show how much gradient–PAVE matching and the Quality score each contribute to LLMs? Since the Quality score also depends on the selected dataset, can you compare with a method that selects the dataset without gradient matching? Then we can know which part actually improves the performance.

  3. In the mixed CV/NLP, can you test again after removing the models trained on the component datasets? The paper already has single-dataset leave-one-task-out experiments, this may help show whether the method works without dataset-specific models.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1Gu3

We sincerely appreciate your time and input. We are grateful for recognizing the CV identification gains at matched candidate counts and the NLP assembly gains with fixed candidates and labeled data.

Additional presentation comment. Figure 3 readability.

We appreciate the suggestion. We have updated Figure 3 in the revised PDF, enlarging the text and shortening the panel headings to improve readability. The six cases and numerical values are unchanged.

W1-1 & Q1. LLM gains and statistical reliability.

Thank you for requesting a statistical assessment of the LLM gains. We have added paired significance tests and confidence intervals for both single-dataset and mixed evaluation.

The table below reports paired Utility differences against Random and UniRoute under the unchanged configurations, with 95% confidence intervals and two-sided pp-values.

SettingBaselineOurs − baseline (pp)95% CI (pp)pp-value
SingleRandom+3.84[+3.14, +4.51]0.00001
SingleUniRoute+2.62[+1.98, +3.23]0.00001
MixedRandom+2.37[+1.09, +3.70]0.00058
MixedUniRoute+1.06[-0.69, +2.82]0.23904

We use 2,000 dataset-stratified paired query-ID cluster bootstrap draws for 95% percentile intervals and 100,000 cluster-level method-label swaps for two-sided pp-values. All occurrences of the same query are grouped together during resampling and permutation. Significance requires p<0.05p < 0.05.

Our method significantly outperforms Random in both mixed and single-dataset evaluation. Although the mixed-setting gain over UniRoute is not statistically significant, the single-dataset gain is significant (+2.62 points; p=0.00001p = 0.00001).

W1-2 & Q3. Component-trained models in CV/NLP.

Thank you for raising the role of component-trained models. The mixed CV/NLP experiments test whether our method can identify suitable models and allocate inputs accurately when the model repository contains strong models for the component tasks. Single-dataset leave-one-task-out evaluation further tests robustness without the target-trained model. Excluding every component-trained model from a mixture instead tests transfer without those experts, which differs from the intended identification-and-allocation setting. We therefore do not include mixed-dataset leave-one-task-out in the main paper.

Nevertheless, to address your question, we report this additional exclusion test across all 720 conditions below. We removed each mixture's component-trained models and specifications.

MethodCV UtilityNLP Utility
Ours55.43 ± 0.1661.00 ± 0.33
Random51.77 ± 0.0652.76 ± 0.09
PAVE-Single51.98 ± 0.2158.10 ± 0.26
Best-Single (hindsight)59.84 ± 0.0963.67 ± 0.07
PAVE Top-8 Ensemble63.13 ± 0.1060.07 ± 0.13

Our method remains above Random and PAVE-Single in both modalities. After the component-trained experts are removed, the strongest remaining models have closer cross-task transfer performance. Our specification-guided identification and gradient-based assembly do not consistently resolve these smaller performance differences, making it harder to distinguish and favor the best transferable model. We interpret this reduced discrimination as a reason for the gap to Best-Single.

On CV, PAVE Top-8 Ensemble achieves higher Utility but requires predictions from up to eight candidate models per input. Compared with our method, which invokes one candidate model per input, ensembling requires substantially more candidate model evaluations. On NLP, our method exceeds the ensemble by 0.93 points while still invoking one candidate model per input.

AuthorsOpenReview ↗

Response to Reviewer 1Gu3 (cont'd)

W1-3. Comparisons with additional baselines.

Thank you for suggesting these additional comparisons. We now report RKME results across all six settings and Constructive Specification results in both LLM settings, alongside the rationale for our original baseline selection.

RKME-based methods (Wu et al., 2023; Tan et al., 2024b).

These works study RKME-based learnware specification, identification, and reuse. RKME was originally developed primarily for tabular models, whereas our experiments address CV, NLP, and black-box LLMs using PAVE specifications. The PAVE study (Shi et al., 2026) already compares PAVE with RKME, reports its advantages in the evaluated settings, and provides a detailed discussion of the two specification families. Our contribution focuses on complementary model identification and assembly using PAVE. We therefore initially prioritized PAVE-based reuse methods and routing baselines that directly assess these components.

Nevertheless, we additionally report RKME results in response to your request. The table below reports Utility across all six settings.

SettingModalityRKME SingleRKME MultipleOurs
MixedCV62.43 ± 0.2182.08 ± 0.2683.81 ± 0.63
MixedNLP62.34 ± 0.0770.67 ± 0.1373.73 ± 0.13
MixedLLM78.39 ± 2.8878.43 ± 3.0981.16 ± 1.15
SingleCV63.9363.7166.50
SingleNLP66.7067.9570.90
SingleLLM79.2779.2781.78

Our method exceeds both RKME variants in all six settings, by 1.74–3.07 points over the stronger variant.

Constructive Specification (Liu et al., 2026b).

Liu et al. (2026b) propose Constructive Specification (CS) for black-box learnware agent identification. The table below compares our implementation of this method with our method in both LLM settings.

SettingBaselineOurs Utility (%)Baseline Utility (%)Ours − baseline (pp)
MixedCS81.1680.64+0.52
SingleCS81.7879.77+2.00

Our method leads by 0.52/2.00 points in mixed/single evaluation. Beyond these accuracy gains, our method also reduces specification construction costs. CS fits a PAVE for each query, whereas our method fits one user PAVE per batch and uses per-query gradients, avoiding repeated optimization and reducing computation time.

W2. Reference model and reproducibility.

Our reference-model configuration is strictly identical to that used in PAVE (Shi et al., 2026, Appendix C.3): CLIP ViT-B/32 for CV, RoBERTa-base for NLP, and Qwen-0.5B for LLMs. These instantiate the shared reference model hh for PAVE construction and gradient computation. We have explicitly added these model choices in Appendix D.1.3 (Method Configuration), under “Shared reference models,” in the revised PDF. Our code, fixed experimental configurations, and reproduction instructions are available in the anonymous repository.

AuthorsOpenReview ↗

Response to Reviewer 1Gu3 (cont'd)

Q2. Contributions of matching and Quality.

Thank you for suggesting controls that separate gradient matching from Quality. We have added score-component ablations and a control that selects Quality records using embeddings instead of gradient matching.

We compare the full R+G+QR + G + Q score with RR only, QQ only, and QQ only with embedding-based benchmark selection. RR is per-input gradient matching; GG is global matching. QQ reads each model's historical accuracy on its query-matched PAVE's benchmark. Thus, QQ only still uses gradient matching; RR only uses similarity directly.

The embedding control uses the frozen Qwen3-Embedding-0.6B semantic text encoder for queries and historical profiling inputs. Each benchmark centroid is the normalized mean of its normalized historical-input embeddings. For each candidate model, we select the benchmark with the highest query–centroid cosine similarity to compute QQ. The table below reports these controls in both settings.

SettingScore variantUtility (%)Ours − variant (pp)
MixedOurs81.16—
MixedRR only79.21+1.95
MixedQQ only81.73-0.56
MixedQQ only (embedding)80.73+0.43
SingleOurs81.78—
SingleRR only79.71+2.06
SingleQQ only80.28+1.49
SingleQQ only (embedding)79.98+1.80

Our method exceeds RR only by 1.95/2.06 points for mixed/single evaluation and QQ only by 1.49 points on single datasets. Although QQ only is 0.56 points higher on mixtures, it still relies on gradient–PAVE matching to identify each candidate model's relevant benchmark, whose historical accuracy supplies QQ for final model selection. Its 1.00/0.31-point gains over embedding-based benchmark selection support the contribution of our gradient–PAVE design within Quality-based model selection itself.

Reviewer_9f44OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies multi-model reuse under the learnware paradigm, where a task or query batch may require diverse capabilities. Building on PAVE, it first performs compositional learnware identification: after common-direction removal, it approximately solves a sparse reconstruction of the user PAVE with OMP, using the unexplained residual to select complementary models. Gradient-guided assembly then calibrates sample gradients with collection-level second moments and assigns each input to one candidate by cosine matching against its full and cluster PAVEs. Experiments cover systems of different black-box LLMs, CV and NLP models, and 41 single-dataset benchmarks, and show gains on mixed workloads.

Critical strengths
  • Replacing independent similarity ranking with sparse PAVE reconstruction is a well-motivated way to address overlapping capabilities.
  • The experiments are large in scale and reasonably controlled. Identification ablations match candidate counts, and assembly ablations fix the candidate sets and labeled user data. Analysis such as the candidate-set oracle and dataset–model assignment agreement help clarify the mechanism.
  • The report is transparent. The appendices document mixture construction, sampling, aggregation, and timing protocols in detail.
Critical weaknesses
  1. The discriminative power of gradient–PAVE matching itself is not demonstrated. In the gradient-calibration diagnostic, ROC-AUC is 0.429 before calibration, below chance, and only 0.498 after, still near chance. The reported 81.50% top-1 accuracy is largely driven by the 457/ 800 queries on which every candidate is correct. On the 309 informative queries, pooled top-1 accuracy rises from 58.25% to 63.11%, but no random-selection reference is provided. I suggest reporting the chance-level top-1 accuracy on informative queries, as well as mixed-batch Utility under pure matching.

  2. In the LLM setting, the gains may stem largely from auxiliary signals. The standardized score in Eq15 assigns wQ=0.75w_Q=0.75 to the quality signal, which uses each candidate's benchmark accuracy on the dataset of its best-matching PAVE. This is essentially a historical-performance prior, similar in nature to the information exploited by K-NN and UniRoute. Tab 2 ablates only gradient calibration and common-space removal. An ablation over wGw_G and wQw_Q is needed to attribute the perf margin over competitors.

  3. The LLM improvements are modest relative to their variance, and the contribution of identification is limited. Eg, in Table A22, majority voting over the identified models, 81.32, outperforms the proposed assembly of 81.16. Against Similarity Top-K with the same assembly, identification gains only 0.39 (0.11 under the oracle), within seed-level SD. Random already reaches 78.80, so the headroom between random selection and learned routers is narrow. Besides, in single-dataset evaluation, the advantage is dominated by Lendingclub (65.00 vs 36.70 for most learned routers). Excluding it, the proposed method averages 79.69 over the remaining five datasets, behind Cluster-PAVE's 80.17 and roughly on par with UniRoute.

  4. Hyperparameter selection and sensitivity are not reported. The configurations in Tables A6 and A7 differ substantially across modalities.

Questions for authors
  1. Eq. 8 treats a multi-step, possibly adaptively optimized PAVE as an average of initial gradients. How well does τm,j\tau_{m,j} align with the mean-gradient prototype gˉm,j=1∣Dm,j∣∑z∈Dm,jg(z)\bar g_{m,j}=\frac{1}{|D_{m,j}|}\sum_{z\in D_{m,j}} g(z) at θ0\theta_0? Would matching against gˉm,j\bar g_{m,j} perform comparably?
  2. In Tab2, removing common-space projection changes Kˉ\bar K from 4.88 to 3.03. What is the result when candidate counts are matched?
  3. In the CV/NLP mixtures, how do non-component models produce predictions in other datasets' label spaces via target-compatible output heads? How does PAVE Top-8 Ensemble align class probabilities across heterogeneous label spaces?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 9f44

We sincerely appreciate your time and input. We are grateful for your recognition of the complementary identification design, controlled experiments, and transparent reporting.

W1. Chance accuracy and pure matching.

Thank you for pointing out that aggregate top-1 accuracy can obscure performance on informative queries. We have added the exact random-selection reference and expanded the pure-matching comparison to all test queries in both LLM settings.

We use the same test queries and identified candidate sets as in the main LLM experiments. Random expectation is computed from the proportion of candidates that answer each query correctly. The table below compares calibrated pure matching with uncalibrated matching and this exact random reference on all and informative queries.

SettingSubsetRandom expectation (%)Uncalibrated Utility (%)Calibrated Utility (%)
MixedAll78.6277.8479.21
MixedInformative58.2555.7260.17
SingleAll78.3078.8279.71
SingleInformative58.8559.4663.29

Calibration improves pure-matching Utility by 1.37/0.89 points in mixed/single evaluation. In mixed evaluation, calibration lifts Utility from below to above the random expectation, both overall and on informative queries.

On informative queries, where candidates include both correct and incorrect predictions, calibrated matching exceeds the exact random reference by 1.91/4.44 points. These gains demonstrate the practical value of gradient–PAVE matching for top-1 selection.

W2. Contributions of matching and Quality.

Thank you for requesting direct ablations of the global and Quality terms. We have added the requested comparisons and isolated the gradient-based benchmark selection within Quality.

RR is per-input gradient matching, GG is global PAVE matching, and QQ is query-matched historical accuracy. The table below compares the full score with the requested wG=0w_G = 0 (R+QR + Q) and wQ=0w_Q = 0 (R+GR + G) ablations in both settings.

SettingScore variantUtility (%)Ours − variant (pp)
MixedOurs (R+G+QR + G + Q)81.16—
MixedR+QR + Q (wG=0w_G = 0)81.32-0.15
MixedR+GR + G (wQ=0w_Q = 0)79.56+1.61
SingleOurs (R+G+QR + G + Q)81.78—
SingleR+QR + Q (wG=0w_G = 0)81.69+0.08
SingleR+GR + G (wQ=0w_Q = 0)79.77+2.00

Removing QQ reduces Utility by 1.61/2.00 points in mixed/single evaluation. Removing GG changes it by 0.15/−0.08 points, indicating a smaller, workload-dependent contribution.

QQ uses historical performance, but its record is selected through our gradient–PAVE mechanism: each calibrated query gradient matches a model's full/cluster PAVEs, retrieves accuracy on the best-matching PAVE's benchmark.

To evaluate benchmark selection for QQ, we set wR=wG=0w_R = w_G = 0 and compare gradient–PAVE matching with embedding similarity. The embedding control uses the frozen Qwen3-Embedding-0.6B semantic text encoder for queries and historical profiling inputs. Each benchmark centroid is the normalized mean of its normalized historical-input embeddings; QQ selects the benchmark with the highest query–centroid cosine similarity.

SettingQQ-only: gradient–PAVE selectionQQ-only: embedding selectionGradient–PAVE − embedding (pp)
Mixed81.7380.73+1.00
Single80.2879.98+0.31

Although Quality contributes substantially to performance, it also benefits from our gradient–PAVE matching design for selecting the relevant benchmark. The 1.00/0.31-point gains over embedding-based selection in mixed/single evaluation demonstrate this contribution within Quality-based model selection itself.

AuthorsOpenReview ↗

Response to Reviewer 9f44 (cont'd)

W3. Practical gains and statistical reliability.

Thank you for raising this concern. We clarify the practical benefits and report additional significance tests.

Compared with majority voting, our assembly invokes one candidate model per input, while the ensemble evaluates every identified model. This provides comparable Utility with substantially fewer candidate model evaluations.

The candidate-set Oracle measures ideal coverage. Both identification methods already achieve high mixed Oracle Utility, so the 0.11-point gap alone does not capture how effectively a practical selector can realize that coverage. Under the same assembly, our identification improves mean Utility over similarity Top-KK by 0.39/2.14 point in mixed/single evaluation, highlighting its compatibility with the downstream assignment rule.

We additionally report paired Utility differences and significance tests under the paper's configurations.

SettingBaselineOurs − baseline (pp)95% CI (pp)pp-value
SingleRandom+3.84[+3.14, +4.51]0.00001
SingleUniRoute+2.62[+1.98, +3.23]0.00001
MixedRandom+2.37[+1.09, +3.70]0.00058
MixedUniRoute+1.06[-0.69, +2.82]0.23904

We use 2,000 dataset-stratified paired query-ID cluster bootstrap draws for 95% percentile intervals and 100,000 cluster-level method-label swaps for two-sided pp-values. All occurrences of the same query are grouped together during resampling and permutation. Significance requires p<0.05p < 0.05.

Our method significantly outperforms Random in both mixed and single-dataset evaluation. Although the mixed-setting gain over UniRoute is not statistically significant, the single-dataset gain is significant (+2.62 points; p=0.00001p = 0.00001).

W4. Parameter sensitivity.

Thank you for requesting a sensitivity analysis. We have added sensitivity analysis of calibration ϵ\epsilon, common rank, explained-fraction threshold, and candidate budget across all six settings.

Each parameter is varied separately; all other settings follow the paper. The results show limited sensitivity to calibration ϵ\epsilon and common rank, while the explained-fraction threshold and KK have larger effects through candidate coverage.

ϵ\epsilonCV mixedCV single datasetNLP mixedNLP single datasetLLM mixedLLM single dataset
10−1010^{-10}83.8137 ± 0.628866.478873.7341 ± 0.133570.897180.64 ± 0.8481.44
10−810^{-8} (default)83.8139 ± 0.628766.502073.7341 ± 0.133570.896981.16 ± 1.1581.78
10−610^{-6}83.8175 ± 0.629266.628873.7336 ± 0.131570.889981.25 ± 1.0081.14

Utility remains stable across four orders of magnitude in calibration ϵ\epsilon, with negligible changes on CV/NLP and modest changes on LLMs.

rcr_cCV mixedCV single datasetNLP mixedNLP single datasetLLM mixedLLM single dataset
1 (default)83.81 ± 0.6366.5073.73 ± 0.1370.9081.16 ± 1.1581.78
283.01 ± 0.2066.4974.00 ± 0.3369.8480.69 ± 1.2480.15
483.57 ± 0.3366.4874.08 ± 0.0770.2780.97 ± 1.4381.22

Utility shows limited sensitivity to the common rank within the tested range.

ρ\rhoCV mixedCV single datasetNLP mixedNLP single datasetLLM mixedLLM single dataset
85%76.73 ± 0.1366.1172.11 ± 0.2870.5681.16 ± 1.4379.01
90%78.38 ± 0.3466.2572.62 ± 0.3870.8381.16 ± 1.1580.75
95%83.81 ± 0.6366.5073.73 ± 0.1370.9080.97 ± 1.0181.60
98%87.04 ± 0.2066.5074.52 ± 0.0470.9081.47 ± 1.0081.78

With the original candidate caps, higher explained-fraction thresholds improve CV/NLP mixed Utility but generally increase the number of candidates processed.

KKCV mixedCV single datasetNLP mixedNLP single datasetLLM mixedLLM single dataset
274.96 ± 0.3265.8470.05 ± 0.3269.8279.06 ± 1.8980.39
482.41 ± 0.3265.9473.73 ± 0.3269.2180.23 ± 1.9080.81
685.82 ± 0.3066.5474.31 ± 0.0971.0981.10 ± 1.1881.46
887.29 ± 0.0966.5874.45 ± 0.1270.5881.45 ± 0.9081.46
1087.57 ± 0.0766.6674.49 ± 0.0969.7980.69 ± 1.5379.67

With threshold stopping disabled, larger KK benefits CV/NLP mixtures, with diminishing returns on NLP beyond KK=6. NLP single-dataset Utility peaks at KK=6, and LLM mixed Utility peaks at KK=8. We use a bounded candidate budget to balance these gains against matching and assembly costs.

AuthorsOpenReview ↗

Response to Reviewer 9f44 (cont'd)

Q1. Initial-gradient prototypes and fitted PAVEs.

Thank you for this question. We clarify the interpretation and compare the two representations below.

(a) Alignment with the initial mean-gradient prototype.

In practice, the local-update approximation in Eq. 8 is not always tight. We examine it through the median cosine similarity cos⁡(τ,−gˉ)\cos(\tau,-\bar{g}) within each modality, where τ\tau is a fitted PAVE and gˉ\bar{g} is the mean gradient at initialization on the same fitting samples. Comparisons use the full PAVE space and cover 1,565 distinct specifications, including full and cluster model PAVEs and user PAVEs.

ModalityNumber of PAVEsMedian cosine
CV5720.606
NLP3040.433
LLM6890.467

These similarities show differences between fitted PAVEs and the initial mean descent direction. Gradient evolution and adaptive coordinate scaling can affect the local-update approximation. Given these simplifications, our local-update analysis is intended to help explain gradient–PAVE matching, rather than provide a theoretical guarantee of the method’s effectiveness. This simplified gradient-kernel interpretation helps us understand the matching mechanism and guides its design and calibration.

Meanwhile, targeted experiments provide empirical support for the effectiveness of gradient–PAVE matching. In particular, we test whether matching can select suitable models for labeled support samples in CV and NLP mixed conditions. Each sample selects the highest-matching model, without auxiliary scores or teacher transfer. Results average conditions equally and use informative classification samples with both correct and incorrect candidates; random expectation is each sample's correct-candidate proportion.

ModalityRandom expectation (%)Matching top-1 accuracy (%)AUC
CV46.3294.100.801
NLP57.0069.030.647

Matching exceeds the random reference in both modalities, supporting its value for sample-level model selection. The paper also reports improved pure-matching rankings after calibration (Section 4.3.1; Appendix E.3.1, Table A21) and a 2.00-point drop in LLM mixed Utility when calibration is removed with candidates fixed (Section 4.3.2, Table 2). Part (b) below directly addresses whether matching against initial-gradient prototypes performs comparably.

(b) Matching against initial-gradient prototypes.

We replace only assembly's full/cluster PAVEs with initial-gradient prototypes and compare Utility across all six settings, keeping identification and all other settings fixed.

SettingModalityFitted PAVEsInitial gradientsInitial − fitted (pp)
MixedCV83.81 ± 0.6383.81 ± 0.62-0.01
MixedNLP73.73 ± 0.1374.02 ± 0.08+0.28
MixedLLM81.16 ± 1.1580.62 ± 0.66-0.54
SingleCV66.5067.05+0.55
SingleNLP70.9070.07-0.83
SingleLLM81.7881.21-0.56

The Utility gaps are small (at most 0.83 points), but initial gradients are slightly weaker overall: fitted PAVEs lead by 0.19 points when the six settings are weighted equally.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary for Paper 20460

Dear PCs, SACs, ACs, and Reviewers,

We sincerely thank the reviewers for their careful feedback and consideration of our additional experiments. Below, we summarize the paper's main contributions, the reviewer feedback, and the evidence added during rebuttal.

Main contributions

Complementary model identification from existing PAVE specifications. Sparse reconstruction of the user PAVE after common-direction removal identifies complementary models beyond independent similarity ranking. With candidate counts and assembly fixed, it improves CV/NLP Utility by 5.81/1.08 points.

Gradient-guided sample-level assembly without training a parametric router. Calibrated sample gradients match full and cluster PAVEs to guide model assignment for individual inputs. With other conditions fixed, NLP assembly exceeds the supervised MLP by 4.06 points.

Broad validation of practical multi-model reuse. Evaluation covers 72 LLM mixed batches, 720 CV/NLP mixed conditions, and 41 single-dataset benchmarks. On CV/NLP mixtures, our method improves over hindsight Best-Single by 10.62/2.52 points and exceeds PAVE Top-8 Ensemble while invoking one candidate model per input, versus up to eight for the ensemble.

Reviewer feedback

Motivation and conceptual connections (9f44, SsQN). Both reviewers recognized the rationale for sparse PAVE reconstruction, with 9f44 emphasizing overlapping capabilities and SsQN highlighting connections to task arithmetic and gradient-kernel similarity.

Controlled experimental evidence (1Gu3, 9f44, SsQN). 1Gu3 and 9f44 highlighted the identification and assembly controls. In particular, 1Gu3 noted that CV identification outperforms similarity Top-KK with matched candidate counts and that NLP assembly outperforms a supervised MLP with the same candidates and labeled data. SsQN emphasized the isolated evidence for common-direction removal and gradient calibration.

Clear positioning and transparent reporting (9f44, SsQN). SsQN recognized the positioning across learnware, routing, and gradient representations; 9f44 highlighted the evaluation's scale, mechanism analyses, and detailed protocol documentation.

In their follow-ups, 9f44 maintained positive score and acknowledged the breadth of the additional experiments; SsQN confirmed that our response addresses the questions raised; and 1Gu3 acknowledged that the rebuttal addresses several concerns and expressed willingness to participate in further discussion.

Evidence added during rebuttal

  1. Gradient–PAVE matching improves model selection. On informative classification support samples from 36 CV/NLP mixed conditions, matching alone reaches 94.10%/69.03% accuracy against random expectations of 46.32%/57.00%, with candidates and PAVEs fixed and auxiliary scores and teacher transfer disabled. In LLMs, replacing gradient–PAVE benchmark selection with semantic embeddings lowers Quality-only Utility by 1.00/0.31 points in mixed/single evaluation, demonstrating the matching design's contribution within Quality.

  2. Effective reuse without component-trained experts. After removing component-trained models and their specifications from all 720 CV/NLP mixed conditions, our method still exceeds PAVE-Single by 3.45/2.90 points. On NLP, it also exceeds PAVE Top-8 Ensemble by 0.93 points while invoking just one candidate per input.

  3. Consistent performance gains in LLM evaluation. Paired tests accounting for repeated queries confirm single-dataset LLM gains over Random and UniRoute of 3.84/2.62 points (p=0.00001p=0.00001 for both), and a mixed LLM gain over Random of 2.37 points (p=0.00058p=0.00058). On the 14 single-dataset LLM benchmarks, our method achieves the best average rank (2.75) among nine methods.

  4. Limited sensitivity to calibration and common rank. Sensitivity tests across all six settings show limited impact from calibration ϵ\epsilon and common rank within tested ranges, while candidate budgets and explained-fraction thresholds control coverage and computation.

In the revised PDF, we clarified the reference-model configuration, target-compatible output heads, and probability alignment across heterogeneous label spaces, and improved Figure 3's readability. Our code is also available in the anonymous code repository.

Our original experiments and the additional rebuttal evidence support the value of complementary model identification and gradient-guided assembly. We would be grateful for your consideration of these contributions in your overall assessment.

Thank you for your time and consideration.

The Authors of Paper 20460

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.