- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
To counter the two failure modes—independent similarity ranking favoring overlapping capabilities, and collection-level specifications lacking per-sample evidence, this paper proposes a two-stage method: first, complementary models are identified via sparse linear reconstruction of the user PAVE (solved with OMP) after common-direction removal, inspired by the linear compositionality of task arithmetic; second, sample-level assignment matches gradients of individual inputs, calibrated by collection-level statistics, against the selected models' hierarchical PAVEs, with a soft-teacher transfer from limited labeled data for the task scenario. Under an NTK/local-update approximation, the paper interprets gradient–PAVE matching as average gradient-kernel similarity between a sample and a specification's fitting distribution. Experiments on systems of 10 black-box LLMs, 12 CV, and 15 NLP models—covering 72 mixed LLM batches, 720 mixed CV/NLP runs, and 41 single-dataset tasks, with balanced/long-tail mixtures, out-of-system benchmarks, leave-one-out protocols, and component-level ablations—show that, without training a parametric selector, the method outperforms the strongest learned router by 1.06 points and hindsight Best-Single by 10.62/2.52 points on mixed CV/NLP, invoking one candidate per input.
Critical strengths
S1: Theoretically grounded: sparse reconstruction of the user PAVE is inspired by task-arithmetic linearity, and gradient–PAVE matching is analyzed as a kernel query (average gradient-kernel similarity) under a local-update approximation.
S2: Common-direction removal is well motivated and validated: ablating it costs 1.80 Utility points across 72 LLM mixed-query conditions.
S3: Gradient calibration is isolated and evidenced: it contributes +2.00 points, and improves every ranking metric plus top-1 accuracy in the matching-only analysis.
S4: Well-positioned contributions across learnware, routing, task arithmetic, and gradient representations, with cleanly separated claims.
Critical weaknesses
W1: The central NTK/local-update approximation (gradients frozen at initialization, ignoring gradient evolution and adaptive scaling) is unverified empirically, yet it underpins the entire interpretation.
W2: Statistical power is limited: "3 seeds" only randomize mixture construction; the 1.06-point LLM gain is small relative to a ±1.15 SD, and no significance tests are reported.
W3: Modest gains over learned routers: 1.06 points on mixed LLM workloads and only 9 wins vs 5 losses on single-dataset LLM benchmarks, with near-ties against several baselines.
W4: No sensitivity analysis for key hyperparameters (budget K, 95% explainability threshold, common rank r_c=1, calibration ε), even though K directly caps achievable performance.
Questions for authors
See weaknesses.
4 comments
Response to Reviewer SsQN (cont'd)
W4. Hyperparameter sensitivity and candidate budget.
Thank you for requesting a sensitivity analysis. We have added sensitivity analysis of calibration , common rank, explained-fraction threshold, and candidate budget across all six settings.
Each parameter is varied separately; all other settings follow the paper. The results show limited sensitivity to calibration and common rank, while the explained-fraction threshold and have larger effects through candidate coverage.
| CV mixed | CV single dataset | NLP mixed | NLP single dataset | LLM mixed | LLM single dataset | |
|---|---|---|---|---|---|---|
| 83.8137 ± 0.6288 | 66.4788 | 73.7341 ± 0.1335 | 70.8971 | 80.64 ± 0.84 | 81.44 | |
| (default) | 83.8139 ± 0.6287 | 66.5020 | 73.7341 ± 0.1335 | 70.8969 | 81.16 ± 1.15 | 81.78 |
| 83.8175 ± 0.6292 | 66.6288 | 73.7336 ± 0.1315 | 70.8899 | 81.25 ± 1.00 | 81.14 |
Utility remains stable across four orders of magnitude in calibration , with negligible changes on CV/NLP and modest changes on LLMs.
| CV mixed | CV single dataset | NLP mixed | NLP single dataset | LLM mixed | LLM single dataset | |
|---|---|---|---|---|---|---|
| 1 (default) | 83.81 ± 0.63 | 66.50 | 73.73 ± 0.13 | 70.90 | 81.16 ± 1.15 | 81.78 |
| 2 | 83.01 ± 0.20 | 66.49 | 74.00 ± 0.33 | 69.84 | 80.69 ± 1.24 | 80.15 |
| 4 | 83.57 ± 0.33 | 66.48 | 74.08 ± 0.07 | 70.27 | 80.97 ± 1.43 | 81.22 |
Utility shows limited sensitivity to the common rank within the tested range.
| CV mixed | CV single dataset | NLP mixed | NLP single dataset | LLM mixed | LLM single dataset | |
|---|---|---|---|---|---|---|
| 85% | 76.73 ± 0.13 | 66.11 | 72.11 ± 0.28 | 70.56 | 81.16 ± 1.43 | 79.01 |
| 90% | 78.38 ± 0.34 | 66.25 | 72.62 ± 0.38 | 70.83 | 81.16 ± 1.15 | 80.75 |
| 95% | 83.81 ± 0.63 | 66.50 | 73.73 ± 0.13 | 70.90 | 80.97 ± 1.01 | 81.60 |
| 98% | 87.04 ± 0.20 | 66.50 | 74.52 ± 0.04 | 70.90 | 81.47 ± 1.00 | 81.78 |
With the original candidate caps, higher explained-fraction thresholds improve CV/NLP mixed Utility but generally increase the number of candidates processed.
| CV mixed | CV single dataset | NLP mixed | NLP single dataset | LLM mixed | LLM single dataset | |
|---|---|---|---|---|---|---|
| 2 | 74.96 ± 0.32 | 65.84 | 70.05 ± 0.32 | 69.82 | 79.06 ± 1.89 | 80.39 |
| 4 | 82.41 ± 0.32 | 65.94 | 73.73 ± 0.32 | 69.21 | 80.23 ± 1.90 | 80.81 |
| 6 | 85.82 ± 0.30 | 66.54 | 74.31 ± 0.09 | 71.09 | 81.10 ± 1.18 | 81.46 |
| 8 | 87.29 ± 0.09 | 66.58 | 74.45 ± 0.12 | 70.58 | 81.45 ± 0.90 | 81.46 |
| 10 | 87.57 ± 0.07 | 66.66 | 74.49 ± 0.09 | 69.79 | 80.69 ± 1.53 | 79.67 |
With threshold stopping disabled, larger benefits CV/NLP mixtures, with diminishing returns on NLP beyond =6. NLP single-dataset Utility peaks at =6, and LLM mixed Utility peaks at =8. We use a bounded candidate budget to balance these gains against matching and assembly costs.
Maintain my original score.
Thanks for your response. It addresses my questions. However, it does not change my evaluation, and I maintain my original score.
Thank you for carefully considering our response and confirming that it addresses your questions. We sincerely appreciate your time and constructive suggestions, which helped us clarify the scope of our analysis and strengthen the experimental evaluation.
Response to Reviewer SsQN
We sincerely appreciate your time and input. We are grateful for your recognition of the theoretical motivation, common-direction removal, calibration evidence, and positioning of our contributions.
Thank you for raising this concern. In practice, the local-update approximation is not always tight. We examine this by comparing fitted PAVEs with the negative mean initial gradient on their respective fitting samples. The 1,565 distinct specifications include full and cluster model PAVEs and user PAVEs.
These similarities show differences between fitted PAVEs and the initial mean descent direction. Gradient evolution and adaptive coordinate scaling can affect the local-update approximation. Given these simplifications, our local-update analysis is intended to help explain gradient–PAVE matching, rather than provide a theoretical guarantee of the method’s effectiveness. This simplified gradient-kernel interpretation helps us understand the matching mechanism and guides its design and calibration.
Meanwhile, targeted experiments provide empirical support for the effectiveness of gradient–PAVE matching. In particular, we test whether matching can select suitable models for labeled support samples in CV and NLP mixed conditions. Each sample selects the highest-matching model, without auxiliary scores or teacher transfer. Results average conditions equally and use informative classification samples with both correct and incorrect candidates; random expectation is each sample's correct-candidate proportion.
Matching exceeds the random reference in both modalities, supporting its value for sample-level model selection. The paper also reports improved pure-matching rankings after calibration (Section 4.3.1; Appendix E.3.1, Table A21) and a 2.00-point drop in LLM mixed Utility when calibration is removed with candidates fixed (Section 4.3.2, Table 2).
Thank you for raising concerns about the statistical reliability of the LLM gains. We have added paired significance tests and confidence intervals for both single-dataset and mixed evaluation, together with an average-rank analysis across the 14 single-dataset benchmarks.
The table below reports paired Utility differences against Random and UniRoute under the unchanged configurations, with 95% confidence intervals and two-sided p-values.
We use 2,000 dataset-stratified paired query-ID cluster bootstrap draws for 95% percentile intervals and 100,000 cluster-level method-label swaps for two-sided p-values. All occurrences of the same query are grouped together during resampling and permutation. Significance requires p<0.05.
Our method significantly outperforms Random in both mixed and single-dataset evaluation. Although the mixed-setting gain over UniRoute is not statistically significant, the single-dataset gain is significant (+2.62 points; p=0.00001).
The reported nine wins and five losses refer specifically to the comparison with UniRoute across the 14 single-dataset benchmarks. To assess our method's overall standing, we rank all nine methods on each dataset and average these ranks equally across datasets, assigning average ranks to ties. Our method achieves the best average rank of 2.75, ahead of UniRoute (4.39) and Random (6.18), as shown below.