- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies model selection for a new task without running all candidate LLMs on the target data. It extends Parameter Vector (PAVE) specifications to heterogeneous LLMs by using multiple auxiliary models. Each candidate model and task is represented by a parameter-efficient fine-tuning update of the same auxiliary model, and cosine similarity between the resulting vectors is used to induce model rankings. The paper shows empirically that different auxiliary models provide complementary views of model-task compatibility. It then proposes Reference-Guided Multi-Auxiliary Voting (RG-MAV), which selects informative reference tasks separately in each auxiliary-model parameter space and aggregates pairwise model comparisons using task- and model-pair-specific weights. The paper also provides a theoretical analysis connecting task-PAVE cosine similarity to the probability that two tasks agree on the true ordering of a candidate model pair.
Critical strengths
-
The paper gives a formal characterization of model-pair decision boundaries in an auxiliary-model PAVE space and derives an approximate monotonic relationship between nonnegative task-PAVE alignment and cross-task agreement on the true ordering of a model pair. This provides a principled explanation for why local reference tasks may be informative, rather than using task similarity only as an intuitive heuristic.
-
RG-MAV is not merely an ensemble over auxiliary models. The theorem motivates local reference neighborhoods; the orientation factor accounts for whether the auxiliary's target and reference decisions point in compatible directions; and pair-specific reliability weights use reference-task correctness to determine which auxiliary view should be trusted for each candidate pair. This alignment between theoretical analysis and algorithmic design is a notable strength.
-
The experiments demonstrate that no single auxiliary model is uniformly reliable across domains and tasks. This observation is important for heterogeneous LLM selection and motivates multi-view aggregation in a technically meaningful way.
-
The reuse of cached model PAVEs and evaluated reference-task information addresses a practically relevant deployment constraint, especially when target data are private or candidate inference is expensive.
Critical weaknesses
-
The cross-task agreement theorem assumes isotropic/exchangeable task geometry and a conditional-independence/mediation structure for pairwise order labels. Real tasks across finance, medicine, law, code, and science are unlikely to satisfy these assumptions. The paper reports small approximation errors under the fitted empirical expansion, but this does not validate the population assumptions or establish that the theorem quantitatively explains the observed gains. The authors should discuss the scope of the theorem more explicitly and provide empirical diagnostics or robustness tests for assumption violations.
-
The transition from the single-auxiliary theoretical result to the full pair-specific multi-auxiliary voting procedure is intuitive but not formally justified. In particular, it is unclear under what conditions the weighted aggregation preserves or improves pairwise ranking accuracy, and how errors or dependence between auxiliary votes affect the final Copeland ranking.
-
The paper would benefit from an explicit formal definition of the deployment setting: what is known before target-task selection, what is cached, what target information is allowed, and which quantities are strictly unavailable during inference. At present, the reader must infer these distinctions from the PAVE construction and evaluation protocol. A concise problem formulation and an information-availability diagram would improve both rigor and readability.
-
The paper relies on PAVE as the central representation, but the motivation, exact construction, relation between model PAVEs and task PAVEs, and dependence on the auxiliary model are not explained sufficiently before the main contribution. In particular, the reader needs a clearer explanation of why fine-tuning an auxiliary model on probe responses produces a useful capability specification, and what information is lost or distorted by the LoRA parameterization.
-
RG-MAV assumes a maintained pool of evaluated reference tasks. Its performance depends on the coverage, selection, and distribution of this pool, yet the cost and governance of creating and maintaining it are not fully analyzed. The robustness experiments with alternative reference pools are useful, but the paper should state more clearly how many reference tasks are needed and how performance changes under domain shift or poor reference coverage.
Questions for authors
-
Can the authors provide a concise formal statement of the information available at deployment time versus the information used only for post-hoc evaluation? In particular, what target-task labels or examples are required to construct the target PAVE?
-
How sensitive is RG-MAV to the number and domain coverage of reference tasks? Is there a practical rule for detecting that the reference pool is insufficient or mismatched to a target task?
-
Can the authors provide a theoretical or empirical analysis of dependence among auxiliary-model votes? The current aggregation appears to treat the auxiliary views through reliability weights, but correlated errors may affect the claimed benefit of multi-auxiliary voting.
6 comments
Response to uy2p - Part 2/5
W 2 + Q 3: The conditions under which weighted aggregation preserves or improves pairwise ranking accuracy remain unclear, and the paper does not sufficiently analyze how errors or statistical dependence among auxiliary votes affect the final Copeland ranking.
A:
-
The condition aggregation preserves or improves pairwise ranking accuracy is that the majority of the auxiliaries filtered out by weights is correct. The single-auxiliary theoretical result can be seen as a filter that filter out bad auxiliaries to attain majority correctness, especially when originally the majority of auxiliaries are bad.
-
We directly measured dependence between auxiliary errors against the measured model-score orderings across all 22 tasks. Mean spearman error correlation is 0.347, indicating positive but imperfectly overlapping errors.
-
Notice in the component ablation from 4.3, the gain from uniform voting is already high. This means using multiple auxiliaries is a validated essential part of RG-MAV.
Response to uy2p - Part 3/5
W 3 + Q 1: Can the authors provide a concise formal statement of the information available at deployment time versus the information used only for post-hoc evaluation, including what target-task labels or examples are required to construct the target PAVE?
A:
We provide a concise formal statement of the information available both at deployment time and only for post-hoc evaluation as follows:
-
Offline preparation. The system maintains the auxiliary pool , candidate pool , and reference pool . It caches model PAVEs , reference task PAVEs , and reference evaluation scores . Model PAVEs are constructed from candidate responses on the shared probe dataset .
-
Online identification. For a new task , its labeled dataset is used to construct target task PAVEs by fine-tuning the auxiliary models. RG-MAV then runs RG-MAV with these PAVEs and cached information to produce the candidate ranking.
-
Post-hoc evaluation. Candidate models are evaluated on the target task to obtain evaluation scores , which are used only to construct the evaluation-induced ranking and compute Kendall's .
To make it more clear, we further provide an information-availability diagram in Fig. 3 [Appendix B.4, p. 24]. Before target-task selection, the system gains candidate, auxiliary, reference pools and the probe dataset, caches model PAVEs, reference-task PAVEs, and candidate evaluation scores on reference tasks; For a new target task, labeled examples are required to construct its task PAVEs by fine-tuning the auxiliary models with the corresponding ground-truth labels, but candidate predictions and evaluation scores on the target task are not permitted; During system inference, candidate predictions , evaluation scores , and the evaluation-induced pairwise ordering are strictly unavailable, they are obtained only in the post-hoc experimental evaluation to compute metrics such as Kendall's . We add the information-availability diagram to the revised paper.
Response to uy2p - Part 1/5
W 1: The authors should clarify the scope and limitations of the theorem and provide empirical diagnostics or robustness tests to assess the impact of assumption violations.
A:
Thanks for the reviewer for this constructive suggestion. The scope of the theorem is a mathematical basis for RG-MAV. The core limitation is that the error from violation of assumptions or the approximation is hard to measure directly, thus needs to be validated through implicit experiments.
As a matter of fact, it is hard to even approximate the true distribution of PAVE trained on all possible tasks. If normalized task PAVEs are sampled independently from a nonuniform density f∈L2(ς), where ς is the uniform spherical measure, the probability expression becomes
1+\frac{\displaystyle\sum_{k\ge0}|\Pi_k(fm)|{L^2(\varsigma)}^2P_k^{(d)}(\rho)} {\displaystyle\sum{k\ge0}|\Pi_k f|_{L^2(\varsigma)}^2P_k^{(d)}(\rho)} \right] .$$ Due to the lack of sufficient tasks, we cannot assess directly how much of the probability is affected by violation of assumptions.So the effectiveness of the theorem is implicitly validated by the experiment and ablations. The gain in average performance and robustness across tasks is significant, and consistent across all five suites. That said, assume that assumptions are violated in application, it should be a small violation.
Also, the theoretical analysis is not meant to set up a strictly correct rule that we can follow. Instead, it carries important theoretical value of giving a condition upon which the intuition of 'greater task similarity, more likely ranking agreement' holds. This condition is hard to give using conventional theoretical approaches in analysis of task vectors.