acceptodds
Under review as a conference paper at ICLR 2027

Complementary PAVE Specifications for Characterizing and Identifying LLM Capabilities

10 views

Abstract

Parameter Vector (PAVE) *specifications* encode each candidate large language model (LLM) and each task as fine-tuning updates of a common auxiliary model, enabling LLM selection without evaluating candidates on the target task. Extending PAVE beyond models derived from a common base, we find that auxiliary models provide complementary views of model–task compatibility: the same behavioral observations can yield accurate model comparisons in one parameter space but misleading comparisons in another. To connect these views to actual model performance, we prove that, for a fixed auxiliary model and under explicit assumptions, the probability that two tasks agree on a model pair's true performance ordering is non-decreasing in their nonnegative PAVE cosine similarity, up to a quantified approximation error. This result motivates Reference-Guided Multi-Auxiliary Voting (RG-MAV), which uses evaluated reference tasks that are most informative for the target in each auxiliary model's parameter space to weight and combine model comparisons separately for each candidate pair. Extensive experiments on heterogeneous LLMs and target tasks from six domains support the benefit of reference locality and show robustness to specification-training seeds and individual auxiliary-model removal. RG-MAV achieves an average Kendall's of , which exceeds the of a fixed best auxiliary model selected on reference tasks and the of EmbedLLM, and reaches over 96% of the attained by a post-hoc task-wise oracle. These results demonstrate that RG-MAV can effectively select and combine accurate views, and that reusable parameter-space characterizations, guided by reference-task evidence, can support effective model selection before any candidate is run on the target task.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Learnware, PAVE specification, Task vectors, LLM selection, Weight-space representation, Rank aggregation, Spherical harmonics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uy2pOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies model selection for a new task without running all candidate LLMs on the target data. It extends Parameter Vector (PAVE) specifications to heterogeneous LLMs by using multiple auxiliary models. Each candidate model and task is represented by a parameter-efficient fine-tuning update of the same auxiliary model, and cosine similarity between the resulting vectors is used to induce model rankings. The paper shows empirically that different auxiliary models provide complementary views of model-task compatibility. It then proposes Reference-Guided Multi-Auxiliary Voting (RG-MAV), which selects informative reference tasks separately in each auxiliary-model parameter space and aggregates pairwise model comparisons using task- and model-pair-specific weights. The paper also provides a theoretical analysis connecting task-PAVE cosine similarity to the probability that two tasks agree on the true ordering of a candidate model pair.

Critical strengths
  1. The paper gives a formal characterization of model-pair decision boundaries in an auxiliary-model PAVE space and derives an approximate monotonic relationship between nonnegative task-PAVE alignment and cross-task agreement on the true ordering of a model pair. This provides a principled explanation for why local reference tasks may be informative, rather than using task similarity only as an intuitive heuristic.

  2. RG-MAV is not merely an ensemble over auxiliary models. The theorem motivates local reference neighborhoods; the orientation factor accounts for whether the auxiliary's target and reference decisions point in compatible directions; and pair-specific reliability weights use reference-task correctness to determine which auxiliary view should be trusted for each candidate pair. This alignment between theoretical analysis and algorithmic design is a notable strength.

  3. The experiments demonstrate that no single auxiliary model is uniformly reliable across domains and tasks. This observation is important for heterogeneous LLM selection and motivates multi-view aggregation in a technically meaningful way.

  4. The reuse of cached model PAVEs and evaluated reference-task information addresses a practically relevant deployment constraint, especially when target data are private or candidate inference is expensive.

Critical weaknesses
  1. The cross-task agreement theorem assumes isotropic/exchangeable task geometry and a conditional-independence/mediation structure for pairwise order labels. Real tasks across finance, medicine, law, code, and science are unlikely to satisfy these assumptions. The paper reports small approximation errors under the fitted empirical expansion, but this does not validate the population assumptions or establish that the theorem quantitatively explains the observed gains. The authors should discuss the scope of the theorem more explicitly and provide empirical diagnostics or robustness tests for assumption violations.

  2. The transition from the single-auxiliary theoretical result to the full pair-specific multi-auxiliary voting procedure is intuitive but not formally justified. In particular, it is unclear under what conditions the weighted aggregation preserves or improves pairwise ranking accuracy, and how errors or dependence between auxiliary votes affect the final Copeland ranking.

  3. The paper would benefit from an explicit formal definition of the deployment setting: what is known before target-task selection, what is cached, what target information is allowed, and which quantities are strictly unavailable during inference. At present, the reader must infer these distinctions from the PAVE construction and evaluation protocol. A concise problem formulation and an information-availability diagram would improve both rigor and readability.

  4. The paper relies on PAVE as the central representation, but the motivation, exact construction, relation between model PAVEs and task PAVEs, and dependence on the auxiliary model are not explained sufficiently before the main contribution. In particular, the reader needs a clearer explanation of why fine-tuning an auxiliary model on probe responses produces a useful capability specification, and what information is lost or distorted by the LoRA parameterization.

  5. RG-MAV assumes a maintained pool of evaluated reference tasks. Its performance depends on the coverage, selection, and distribution of this pool, yet the cost and governance of creating and maintaining it are not fully analyzed. The robustness experiments with alternative reference pools are useful, but the paper should state more clearly how many reference tasks are needed and how performance changes under domain shift or poor reference coverage.

Questions for authors
  1. Can the authors provide a concise formal statement of the information available at deployment time versus the information used only for post-hoc evaluation? In particular, what target-task labels or examples are required to construct the target PAVE?

  2. How sensitive is RG-MAV to the number and domain coverage of reference tasks? Is there a practical rule for detecting that the reference pool is insufficient or mismatched to a target task?

  3. Can the authors provide a theoretical or empirical analysis of dependence among auxiliary-model votes? The current aggregation appears to treat the auxiliary views through reliability weights, but correlated errors may affect the claimed benefit of multi-auxiliary voting.

6 comments
AuthorsOpenReview ↗

Response to uy2p - Part 1/5

W 1: The authors should clarify the scope and limitations of the theorem and provide empirical diagnostics or robustness tests to assess the impact of assumption violations.

A:

Thanks for the reviewer for this constructive suggestion. The scope of the theorem is a mathematical basis for RG-MAV. The core limitation is that the error from violation of assumptions or the approximation is hard to measure directly, thus needs to be validated through implicit experiments.

  • As a matter of fact, it is hard to even approximate the true distribution of PAVE trained on all possible tasks. If normalized task PAVEs are sampled independently from a nonuniform density f∈L2(ς)f\in L^2(\varsigma), where ς\varsigma is the uniform spherical measure, the probability expression becomes

    1+\frac{\displaystyle\sum_{k\ge0}|\Pi_k(fm)|{L^2(\varsigma)}^2P_k^{(d)}(\rho)} {\displaystyle\sum{k\ge0}|\Pi_k f|_{L^2(\varsigma)}^2P_k^{(d)}(\rho)} \right] .$$ Due to the lack of sufficient tasks, we cannot assess directly how much of the probability is affected by violation of assumptions.
  • So the effectiveness of the theorem is implicitly validated by the experiment and ablations. The gain in average performance and robustness across tasks is significant, and consistent across all five suites. That said, assume that assumptions are violated in application, it should be a small violation.

  • Also, the theoretical analysis is not meant to set up a strictly correct rule that we can follow. Instead, it carries important theoretical value of giving a condition upon which the intuition of 'greater task similarity, more likely ranking agreement' holds. This condition is hard to give using conventional theoretical approaches in analysis of task vectors.

AuthorsOpenReview ↗

Response to uy2p - Part 2/5

W 2 + Q 3: The conditions under which weighted aggregation preserves or improves pairwise ranking accuracy remain unclear, and the paper does not sufficiently analyze how errors or statistical dependence among auxiliary votes affect the final Copeland ranking.

A:

  • The condition aggregation preserves or improves pairwise ranking accuracy is that the majority of the auxiliaries filtered out by weights is correct. The single-auxiliary theoretical result can be seen as a filter that filter out bad auxiliaries to attain majority correctness, especially when originally the majority of auxiliaries are bad.

  • We directly measured dependence between auxiliary errors against the measured model-score orderings across all 22 tasks. Mean spearman error correlation is 0.347, indicating positive but imperfectly overlapping errors.

  • Notice in the component ablation from 4.3, the gain from uniform voting is already high. This means using multiple auxiliaries is a validated essential part of RG-MAV.

AuthorsOpenReview ↗

Response to uy2p - Part 3/5

W 3 + Q 1: Can the authors provide a concise formal statement of the information available at deployment time versus the information used only for post-hoc evaluation, including what target-task labels or examples are required to construct the target PAVE?

A:

We provide a concise formal statement of the information available both at deployment time and only for post-hoc evaluation as follows:

  1. Offline preparation. The system maintains the auxiliary pool A\mathcal{A}, candidate pool M\mathcal{M}, and reference pool R\mathcal{R}. It caches model PAVEs qa,i\\{\boldsymbol{q}_{a,i}\\}, reference task PAVEs qa,r\\{\boldsymbol{q}_{a,r}\\}, and reference evaluation scores sr(i)\\{s_r(i)\\}. Model PAVEs are constructed from candidate responses on the shared probe dataset P\mathcal{P}.

  2. Online identification. For a new task tt, its labeled dataset Dt=(xj,yj)j=1Nt\mathcal{D}_t=\\{(x_j,y_j)\\}_{j=1}^{N_t} is used to construct target task PAVEs qa,t\\{\boldsymbol{q}_{a,t}\\} by fine-tuning the auxiliary models. RG-MAV then runs RG-MAV with these PAVEs and cached information to produce the candidate ranking.

  3. Post-hoc evaluation. Candidate models are evaluated on the target task to obtain evaluation scores st(i)\\{s_t(i)\\}, which are used only to construct the evaluation-induced ranking and compute Kendall's τb\tau_b.

To make it more clear, we further provide an information-availability diagram in Fig. 3 [Appendix B.4, p. 24]. Before target-task selection, the system gains candidate, auxiliary, reference pools and the probe dataset, caches model PAVEs, reference-task PAVEs, and candidate evaluation scores on reference tasks; For a new target task, labeled examples are required to construct its task PAVEs by fine-tuning the auxiliary models with the corresponding ground-truth labels, but candidate predictions and evaluation scores on the target task are not permitted; During system inference, candidate predictions y^i(x)\\{\hat{y}_i(x)\\}, evaluation scores st(i)\\{s_t(i)\\}, and the evaluation-induced pairwise ordering σt(i,j)\sigma_t(i,j) are strictly unavailable, they are obtained only in the post-hoc experimental evaluation to compute metrics such as Kendall's τb\tau_b. We add the information-availability diagram to the revised paper.

Reviewer_iYTSOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies complementary rankings of heterogeneous LLMs across auxiliary-model PAVE spaces and establishes an approximate relationship between task similarity and agreement on true pairwise model orderings under explicit population assumptions.

Critical strengths
  1. The complementarity study is well controlled. The 13 auxiliary models use identical candidate responses and task data, with shared frozen LoRA A factors within each auxiliary. This makes ranking differences more attributable to the auxiliary representation spaces than to differences in observations.
  2. The method uses local reference evidence for individual model pairs.RG-MAV does not require a single auxiliary to be optimal for an entire target task; it identifies reliable views separately for each candidate pair. In Table 4, removing pair-specific weighting reduces average τ_b from 0.677 to 0.598, supporting this design choice.
  3. Improvements appear in both ranking and selection metrics. In Table 3, average τ_b exceeds U-MAV's 0.590 and EmbedLLM's 0.600, while average Top-1 regret is approximately 0.016. Thus, the gains are not restricted to better ordering of weaker candidates.
Critical weaknesses
  1. Theorem 3.3 concerns agreement between true pairwise model orderings on two tasks under isotropy and conditional independence. RG-MAV instead uses auxiliary correctness on reference tasks and orientation-adjusted neighborhood weights. Appendix F.8 explicitly acknowledges that agreement in true orderings does not directly imply transfer of auxiliary prediction accuracy. The theory therefore provides motivation rather than a guarantee for this weighting rule.
  2. The ε values in Table 2 come from spectral fits on a finite collection of task pairs, with degree selected by cross-validation. They do not directly bound omitted high-degree energy in the true propensity function, and pairs sharing tasks are dependent. Estimates below 3×10^-8 therefore do not establish that deviations from monotonicity are similarly negligible in real data.
Questions for authors

see in the weakness

5 comments
AuthorsOpenReview ↗

Response to iYTS - Part 1/2

W 1: Agreement in true orderings does not directly imply transfer of auxiliary prediction accuracy, so the theory provides motivation rather than a guarantee for this weighting rule.

A:

We appreciate this insightful question about the transfer from theory on single auxiliary to multi-auxiliary voting. We agree that the detailed relation between theorem and RG-MAV deserves a more clear explanation, and make clear the importance of theorem 3.3 here.

Note that in RG-MAV, the orientation factor is already fixing this concern. The gap from direct transfer is from the fact that when the auxiliary model makes opposite predictions on two tasks, we want the true ordering on the target task also be the opposite of that on the historical task, which is exactly what the orientation factor is meant to do; only then the correctness on historical task and target task aligns. So RG-MAV is actually strictly following Theorem 3.3 by introducing orientation factor, more than a mere motivation, and the gain of doing so is validated in component ablation in Section 4.3.

AuthorsOpenReview ↗

Response to iYTS - Part 2/2

W 2: The ϵ\epsilon values in Table 2 do not directly bound omitted high-degree energy in the true propensity function, and pairs sharing tasks are dependent.

A:

We sincerely thank the reviewer for this insightful and constructive comment. We can examine the error term from the following perspectives.

  • Approximating the true propensity function requires a sufficiently large and representative sample of tasks. Our finite task collection does not allow us to recover this function or directly measure its approximation error.

  • The space in which PAVE resides, namely the weight space of an LLM, is extremely high-dimensional, which means to make the error affect the comparisons between probabilities, a term of at least about m∗10−1\sqrt{m}*10^{-1} degree should contribute a great part of ε\varepsilon. That means the true signal shold contain great energy, which rarely happens.

  • As empirical evidence, consider the reference-loacality ablation results shown in Figure 2. Increasing reference neighbor size kk, generally degrades performance, with broadly similar trends across the four alternative suites. If the true approximation error were substantial, we would expect much more pronounced fluctuations in performance as kk increases, rather than the consistent trend observed. This is convincing enough that the error is tolerable when applying the theorem.

In addition, as for task pairs may share tasks, we additionally conduct strict task-disjoint LTTO(leave-two-tasks-out) validation to examine the robustness of degree selection, as shown in Table 3. The selected degree changes in only 17 out of 1,560 configurations (1.09%), suggesting that degree selection is robust to task overlap in our finite task collection. We also add the corresponding result and analysis in Appendix F.8 of the revised paper.

Table 3. Selected-degree distributions under original LOTO and strict task-disjoint LTTO, over 1,560 auxiliary–model-pair configurations.

Selected KKOriginal LOTOStrict LTTO
221550 (99.36%)1549 (99.29%)
3310 (0.64%)11 (0.71%)
44–12120 (0%)0 (0%)
AuthorsOpenReview ↗

Gentle Follow-up

Dear Reviewer iYTS,

We appreciate the care you took in reviewing our work and the thoughtful suggestions you shared.

In our responses above, we explained how the orientation factor aligns reference-task correctness with the target-task comparison and pointed to its ablation result. We acknowledged that finite task data struggle to directly bound the population approximation error. We also added a task-disjoint validation of degree selection, which changed the selected degree in only 1.09% of configurations, suggesting that our degree selection is robust to task dependence.

With the discussion period nearing its end, we would like to follow up on our rebuttal and see whether it has adequately addressed your concerns. We hope our responses and additional analyses have provided useful clarification. If you have any remaining questions or suggestions, we would be happy to address them before the discussion closes.

Thank you again for your valuable feedback, time, and consideration.

Best regards,

The authors

Reviewer_VHCvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies LLM selection for a new task without evaluating the candidates on it, extending Parameter Vector (PAVE) specifications to heterogeneous LLMs. Its method, RG-MAV, combines the pairwise model comparisons of several auxiliary models. Each auxiliary’s vote on a candidate pair is weighted by whether it ordered that pair correctly on the most similar reference task in its own PAVE space. This design is motivated by a theorem linking PAVE similarity to cross-task agreement.

Experiments show that the best auxiliary varies from task to task and is not predicted by scale or domain specialization. RG-MAV outperforms the best fixed auxiliary, uniform voting and EmbedLLM, comes close to a post-hoc task-wise oracle, and is robust to specification-training seeds and to removing individual auxiliaries.

Critical strengths

[C1] Clear evidence that the choice of auxiliary model matters. The candidates' probe responses and the task data are kept fixed and only the auxiliary model is changed. Under this setup, the quality of the PAVE-induced ranking varies widely from task to task; Every auxiliary performs poorly on some tasks, and neither auxiliary scale nor domain specialization predicts which one works best. The earlier LLM study of PAVE used one fixed auxiliary, so this is a useful finding for anyone building such specifications.

[C2] A simple method that needs no evaluation of the candidates on the target task. Once the model PAVEs are cached, RG-MAV only needs the target task's PAVEs and the stored scores on the reference tasks.

Critical weaknesses

[W1] The two simplest baselines that use the reference-task scores are missing, so it is unclear whether the PAVEs add value. RG-MAV uses the evaluation scores of all candidates on the 11 reference tasks. The comparison includes learned methods that also use these scores (Model Spider and EmbedLLM; Appendix B.5), but not the two simplest ones. One would rank candidates by their mean score over the reference tasks. The other would use the ranking of the reference task most similar to the target, for example under the prompt-embedding similarity already used by PE-AS. Without these baselines, it is not clear how much of RG-MAV's performance comes from the PAVEs rather than from the reference scores themselves.

[W2] RG-MAV's advantage over the strongest baselines is small and not consistent across the task splits, and no uncertainty is reported. The paper evaluates five different splits of the 22 tasks into targets and references: the main split (Table 3) and four alternative splits in Appendix D.5. Across these five splits, RG-MAV's gain in average Kendall's τb is +0.088, +0.073, +0.049, −0.005 and +0.025 over uniform voting (U-MAV). Its gain over EmbedLLM is +0.077, +0.103, +0.081, +0.039 and +0.008. On one split it is slightly below U-MAV, and on another it is nearly tied with EmbedLLM. The paper gives no confidence intervals or significance tests for these comparisons.

For scale, the seed study (Table 4, with 8 of the 13 auxiliaries) shows that RG-MAV alone varies by 0.029 τb across three PAVE training seeds (0.658 to 0.687). That is larger than three of the ten differences above. The results therefore do not establish that RG-MAV reliably beats U-MAV and EmbedLLM. The same applies to the ablations. Removing the orientation factor lowers τb by 0.019 (0.677 to 0.658), less than the seed spread, yet the introduction calls the orientation factor indispensable.

Questions for authors

[Q1]Theorem 3.3 predicts that, for nonnegative similarity, the probability that two tasks agree on a model pair's ordering does not decrease (up to a small error) as their PAVEs become more similar. Could you plot the observed agreement against PAVE cosine similarity for a few auxiliaries and model pairs, together with the fitted expansion mentioned in Section 3.2? Please also show the distribution of cosine similarities between task PAVEs, which bears on Assumption (I).

4 comments
AuthorsOpenReview ↗

Response to VHcv - Part 1/3

W 1: The two simplest baselines that use the reference-task scores are missing, so it is unclear whether the PAVEs add value.

A:

We sincerely thank the reviewer for the valuable suggestion. We agree that with these two critical baselines, PAVE's contribution to the method will be clearer.

  • We have conducted the two baselines, with results listed below.

  • As shown in Table 1, RG-MAV ranks across all five suites and consistently outperforms both baselines. Though prompt-embedding achieves competitive performance on the Main suite, it is sensitive to the reference task suite, with notably lower performance than RG-MAV on Suites 01, 03, and 04. Mean reference score generally performs worse than prompt-embedding and RG-MAV across the suites. These comparisons quantify the information provided by reference tasks and show that simply selecting the nearest task or averaging reference scores does not fully exploit such information.

    Table 1. Mean Kendall's τb\tau_b across the reference-task suites.

    SuitePrompt-embedding nearest taskRG-MAVMean Reference Score
    010.56150.66300.5493
    020.69050.72790.6358
    030.56310.64030.5175
    040.65860.77020.6631
    Main0.63530.67750.5155
  • Besides the positive gain, this approach has high exploratory value in stabilizing PAVE characterization. Also, a crucial condition for weighted aggregation to work well is the complementarity of PAVEs. Therefore, PAVE is necessary in RG-MAV.

  • We add the suggested baselines and corresponding discussion to the Appendix D.5 of the revised paper.

AuthorsOpenReview ↗

Response to VHcv - Part 2/3

W 2: RG-MAV's advantage over the strongest baselines is small and not consistent across the task splits, and no uncertainty is reported.

A:

Table 2. Paired performance differences of RG-MAV over U-MAV and EmbedLLM across five task splits. Values are mean differences with pointwise 95% bootstrap CIs.

SplitRG-MAV vs. U-MAVRG-MAV vs. EmbedLLM
Main+0.0877+0.0877 [0.00770.0077, 0.17500.1750]+0.0770+0.0770 [0.02830.0283, 0.12760.1276]
Suite 01+0.0726+0.0726 [−0.0061-0.0061, 0.15020.1502]+0.1031+0.1031 [0.01980.0198, 0.19470.1947]
Suite 02+0.0490+0.0490 [0.00160.0016, 0.10030.1003]+0.0814+0.0814 [0.01580.0158, 0.14890.1489]
Suite 03−0.0054-0.0054 [−0.0898-0.0898, 0.08870.0887]+0.0394+0.0394 [−0.0200-0.0200, 0.09440.0944]
Suite 04+0.0249+0.0249 [−0.0483-0.0483, 0.09900.0990]+0.0085+0.0085 [−0.0491-0.0491, 0.07780.0778]

Note. Each entry reports the mean paired difference in Kendall's τb\tau_b across 11 target tasks and its 95% percentile bootstrap CI, based on 10,000 paired task-level resamples. Intervals are pointwise and are not adjusted for multiple comparisons.

We appreciate the concern that RG-MAV's improvements vary across task splits and may be comparable to the variation induced by PAVE training seeds. We therefore quantify task-level uncertainty and examine seed robustness and the contribution of the orientation factor.

Main comparison and uncertainty. We quantify task-level uncertainty using paired bootstrap confidence intervals, on our main split. RG-MAV achieves mean τb\tau_b improvements of 0.08770.0877 over U-MAV and 0.07700.0770 over EmbedLLM, with 95% bootstrap CIs of [0.0077,0.1750][0.0077,0.1750] and [0.0283,0.1276][0.0283,0.1276], respectively. Both intervals lie above zero. We further conduct two-sided centered paired bootstrap tests with 10710^7 resamples. The Holm-adjusted pp-values are 0.036880.03688 for U-MAV and 0.003950.00395 for EmbedLLM, respectively, providing statistically significant evidence of improvement over EmbedLLM and U-MAV.

About alternative task splits. Table 2 reports the mean paired differences and corresponding 95% bootstrap CIs across all five splits, quantifying task level uncertainty within each split.

PAVE training seeds. Using the same eight auxiliary models, RG-MAV consistently achieves higher average τb\tau_b than U-MAV across three seeds (3407, 42, and 1337), with mean improvements of 0.0490.049(0.666−0.6170.666 - 0.617), 0.0510.051(0.658−0.6070.658 - 0.607), and 0.0890.089(0.687−0.5980.687 - 0.598), while at the task level, RG-MAV wins on 6, 5, and 8 out of 11 tasks under the three seeds, respectively.

Role of the orientation factor. The orientation factor follows the pair specific decision boundary geometry underlying our analysis, accounting for whether target and reference tasks induce the same or opposite ordering of a candidate pair. Removing it reduces average τb\tau_b from 0.6770.677 to 0.6580.658. More importantly, the orientation factor substantially improves Math-MCQA from 0.2860.286 to 0.5880.588 and raises the worst-task performance from 0.2860.286 to 0.3900.390, as shown in Table 11 in appendix D.3. This highlights its complementary role in mitigating severe task-specific ranking degradation, more valuable than its contribution to average performance.

The corresponding experiments and discussion have been added to Appendix~D.5.

AuthorsOpenReview ↗

Response to VHcv - Part 3/3

Q 1: Could you plot the observed agreement against PAVE cosine similarity for a few auxiliaries and model pairs, together with the fitted expansion, and show the distribution of cosine similarities between task PAVEs?

A:

Following your valuable suggestion, we provide empirical diagnostics of the relationship between task-PAVE similarity and pairwise-order agreement. Figure 11 [Appendix F.8, p. 47] presents four representative settings, covering two auxiliary models and two candidate model pairs, with the observed agreement and the fitted spectral expansion P5#(ρ)P^\#_5(\rho) from Section 3.2. The results show an overall increasing trend of agreement with task-PAVE similarity, consistent with the theoretical prediction.

Figure 10 [Appendix F.8, p. 46] further shows the distributions of cosine similarities over all 231 task pairs for both auxiliaries. All observed similarities are nonnegative, providing an empirical diagnostic relevant to Assumption (I).

AuthorsOpenReview ↗

Gentle Follow-up

Dear Reviewer VHCv,

We appreciate the care you took in reviewing our work and the thoughtful suggestions you shared.

In our responses above, we added the two suggested reference-score baselines, against which RG-MAV outperforms across all five task splits; we reported paired bootstrap intervals for the comparisons and performance across PAVE training seeds; we clarified the orientation factor’s effect and showed its critical benefit lies in the worst-performing tasks; we also added plots of observed pairwise-order agreement and the task-PAVE similarity distributions.

With the discussion period nearing its end, we would like to follow up on our rebuttal and see whether it has adequately addressed your concerns. We hope our responses and additional analyses have provided useful clarification. If you have any remaining questions or suggestions, we would be happy to address them before the discussion closes.

Thank you again for your valuable feedback, time, and consideration.

Best regards,

The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Key Revisions

Summary of Rebuttal

We thank the reviewers for their thoughtful feedback. We appreciate their recognition of the controlled evidence that auxiliary PAVE spaces provide complementary rankings (VHCv, iYTS, uy2p), RG-MAV’s reference-guided, pair-specific voting design (iYTS, uy2p), and model selection without evaluating candidate models on the target task (VHCv, uy2p). They also highlighted the theoretical connection between task similarity and pairwise-order agreement (uy2p) and the gains in both ranking and Top-1 selection (iYTS). Below, we summarize how we addressed the main concerns.

  1. Value of PAVE and strength of comparisons (VHCv). We show that RG-MAV outperforms both suggested reference-score baselines on all five task splits, indicating RG-MAV's performance comes from the PAVEs. We also report paired confidence intervals against U-MAV and EmbedLLM and clarify the orientation factor’s effect on difficult target tasks.
  2. Theory and aggregation (VHCv, iYTS, uy2p). We find that observed pairwise-order agreement generally rises with task-PAVE similarity and that all observed similarities are nonnegative for the two auxiliaries examined. We also confirm stable spectral-degree selection under task-disjoint validation and examine how orientation factor and auxiliary-error dependence affect voting.
  3. Deployment and reference coverage (uy2p). We clarify what is cached, what labeled target examples are needed, and what is used solely for evaluation. We further provide an information-availability diagram to improve both rigor and readability of the manuscript. We find that RG-MAV remains ahead of U-MAV and EmbedLLM with smaller or domain-reduced reference pools, and that the PAVE-based coverage indicator is positively associated with target-task ranking quality.
  4. PAVE construction and auxiliary dependence (uy2p). We clarify how model and task PAVEs are constructed in the same auxiliary space and explain how different auxiliary spaces reveal complementary ranking views.

Key revisions to the manuscript

  • Introduction and Section 3.1 strengthen the motivation for PAVE’s vector representation and clarify the auxiliary-specific LoRA adaptation space; Section 6 connects the baseline, theory, robustness, and reference-coverage findings to explain PAVE’s role in RG-MAV.
  • Sections 4.2–4.3 and Appendices D.3/D.5 (Tables 11–12) add the reference-score baselines and paired confidence intervals, and clarify the orientation factor’s contribution on difficult targets.
  • Appendix B.4 (Fig. 3) and Appendix D.5 (Table 13) specify what information is available at each stage and examine reference-pool size, domain coverage, and the PAVE-based coverage indicator.
  • Appendix F.8 (Figs. 10–11; Table 16) adds observed similarity and ordering-agreement plots and task-disjoint validation of spectral-degree selection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.