- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
The paper addresses the problem of estimating the performance of all models in a given set on a new task without running every single model on it. It does so assuming that the model-task performance matrix is low rank, and so can be factorized as , in which and , with being the rank of the approximation. A new task’s performance column is then approximated with , where is a new -dimensional column of , the only unknown quantity that needs to be computed to estimate the performances of all models. From selected so-called anchor models the linear system can be solved by least squares to estimate the scores of the remaining , thus requiring only evaluations. The problem of the basis in which is written is solved by proving that only the span matters and it is fixed to the one that minimizes the norm of the orthogonal projection residual (the truncated SVD). Under this basis, picking linearly independent models suffices to reconstruct the performances of the other models not yet evaluated. The paper then bounds the error between the reconstructed performances and the true ones by three components: the low-rank approximation error of the true performance, the displacement of the approximated subspace from the true one, and the measurement noise over the anchor models.
Critical strengths
The math is correct and coherent. The paper addresses a concrete and relevant problem providing a bound over performances in an interpretable way, separating it in three sources. The experimental setting proposed is satisfying and broad, covering task and model reconstruction. The protocol is robust and principled.
Critical weaknesses
- From my understanding the method is a truncated SVD followed by row selection and least squares, however the pipeline is spread across sections 3 and 4, and the heavy notation makes it hard to follow. I understand that this might be inherited from the literature, but at least an algorithm to make the reader understand what's happening practically would simplify the understanding.
- The notation is very heavy, and sometimes the discussion is overly complicated, with, in my opinion, little effort to make it easily understandable, while an algorithm or a framing in terms of SVD theory would be much easier to understand.
- The bound of Theorem 1makes use of the idempotence of Q, but this is never acknowledged and appears only in the appendix, at least to me from reading the main matter it was not obvious.
- Anchor selection is not analysed or disclosed. Please report the model family, size, whether they are adapters or base models, the corresponding compute saving...
- No OOD task experiment. App. C.12 removes the target's family but keeps semantically related tasks in the history.
- More baselines would have helped (naive ones like weighted averages, interpolations, or previous works like SimBA [1]).
Questions for authors
See weaknesses. Corollary 1 requires which from my understanding can be checked; can the authors test this empirically?
[1] Subramani, Nishant, Alfredo Gomez and Mona T. Diab. “SimBA: Simplifying Benchmark Analysis Using Performance Matrices Alone.” Conference on Empirical Methods in Natural Language Processing (2025).
16 comments
W5
We agree that the original family holdout may still retain semantically related historical tasks. We therefore added three wide-domain holdout tests to examine the target-subspace mismatch mechanism. We fix , E-optimal anchor selection, and subspace least-squares reconstruction, using the same 132 models, five model folds, and budgets . Under each historical condition, the subspace is re-estimated and anchors are selected using only the corresponding historical matrix.
We added strict domain holdout experiments and random controls with the same number of historical tasks. Let denote the task set. For a target domain , let denote its related task set (see the table below). For each target task , we predict using the performance matrix corresponding to . For the random control, we randomly select tasks from and predict using the corresponding performance matrix, averaging over 10 random samples per target.
| Test domain and scope | Related tasks |
|---|---|
| Mathematics: numerical problems and mathematics multiple-choice questions | Physics/chemistry/astronomy and electrical engineering, CS/ML/security, logic, economics/business/public relations, and ARC, OpenBookQA, QASC |
| Medicine/biology/health | Chemistry, psychology, and ARC, OpenBookQA, QASC |
| Code: HumanEval/MBPP and CS/ML/security | Numerical mathematics and mathematics multiple-choice questions, logic, and electrical engineering |
We then compute the projection residual and average MAE for both groups. MAE refers to clipped MAE, averaged equally over the six budgets . The difference is defined as strict holdout minus random history.
| Domain | Projection residual (random history / strict holdout / difference) | Average MAE (random history / strict holdout / difference) |
|---|---|---|
| Mathematics | 0.09989 / 0.12126 / +0.02137 | 0.09494 / 0.11491 / +0.01997 |
| Medicine/biology/health | 0.06840 / 0.07025 / +0.00184 | 0.06652 / 0.06785 / +0.00133 |
| Code/CS/ML | 0.06802 / 0.06969 / +0.00167 | 0.06579 / 0.07040 / +0.00461 |
| Equal-weight macro-average across the three domains | 0.07877 / 0.08706 / +0.00829 | 0.07575 / 0.08439 / +0.00864 |
Both the projection residual and budget-averaged MAE increase in all three domains, consistent with the subspace mismatch mechanism.
W6
We implemented the suggested weighted average, kNN interpolation, and task-side adaptations of SimBA (Ridge / kNN). Following the task-side reconstruction setting in Sec. 6.1, we evaluated these additional baselines using the same data splits, budgets, and metrics. Below, we provide an expanded version of Table 5 from Appendix C.3. Newly added baselines are underlined, and bold indicates the best clipped MAE / ranking Spearman at each budget.
| L | Rule / complete anchor selection and prediction pipeline | Out-of-bounds (%) ↓ | Mean exceedance ↓ | Unclipped MAE ↓ | Clipped MAE ↓ | Ranking Spearman ↑ |
|---|---|---|---|---|---|---|
| 5 | Ours | 0.04 | 0.0480 | 0.04824 | 0.04823 | 0.650 |
| 5 | E-optimal | 0.12 | 0.0421 | 0.05991 | 0.05986 | 0.559 |
| 5 | D-optimal | 0.20 | 0.0516 | 0.06195 | 0.06185 | 0.554 |
| 5 | Q-DEIM | 0.20 | 0.0516 | 0.06195 | 0.06185 | 0.554 |
| 5 | Leverage | 17.17 | 1.9693 | 0.52390 | 0.18584 | 0.196 |
| 5 | Random | 19.04 | 5.9247 | 1.32021 | 0.19223 | 0.225 |
| 5 | <u>weighted average</u> | 0.01 | 0.0015 | 0.05687 | 0.05687 | 0.607 |
| 5 | <u>interpolation</u> | 0.00 | 0.0000 | 0.06409 | 0.06409 | 0.552 |
| 5 | <u>SimBA–Ridge</u> | 0.01 | 0.0066 | 0.05136 | 0.05136 | 0.625 |
| 5 | <u>SimBA–kNN</u> | 0.00 | 0.0000 | 0.06050 | 0.06050 | 0.552 |
| 10 | Ours | 0.10 | 0.0370 | 0.04200 | 0.04196 | 0.710 |
| 10 | E-optimal | 0.20 | 0.0539 | 0.05582 | 0.05571 | 0.579 |
| 10 | D-optimal | 0.08 | 0.0312 | 0.05555 | 0.05552 | 0.590 |
| 10 | Q-DEIM | 0.15 | 0.0411 | 0.05863 | 0.05857 | 0.583 |
| 10 | Leverage | 0.59 | 0.2671 | 0.07288 | 0.07130 | 0.549 |
| 10 | Random | 0.81 | 0.1465 | 0.07264 | 0.07144 | 0.484 |
| 10 | <u>weighted average</u> | 0.03 | 0.0121 | 0.05372 | 0.05371 | 0.608 |
| 10 | <u>interpolation</u> | 0.00 | 0.0000 | 0.06059 | 0.06059 | 0.548 |
| 10 | <u>SimBA–Ridge</u> | 0.01 | 0.0031 | 0.04562 | 0.04562 | 0.681 |
| 10 | <u>SimBA–kNN</u> | 0.00 | 0.0000 | 0.05937 | 0.05937 | 0.555 |
Question
Thank you for this suggestion. In controlled experiments with known perturbations, this condition can indeed be checked directly. Appendix C.6 already uses the empirical matrix as a reference and constructs perturbations satisfying , sweeping . A dashed line marks the half-gap boundary at , allowing us to examine subspace displacement and reconstruction error both within and beyond the condition.
W1
Sorry for not presenting the overall pipeline clearly in the original manuscript. Your understanding is correct: the reconstruction pipeline can be summarized as truncated SVD, anchor selection, and least-squares reconstruction. We have added pseudocode for the complete pipeline to the PDF (Algorithm 1, Appendix C.2), followed by descriptions of the specific anchor selection rules.
W2
We agree that the original notation and presentation increased the reading burden. From an SVD perspective, we form U using the top K left singular vectors of the historical matrix, estimate the least-squares coefficients using the anchor rows UA and observed scores, and multiply by U to reconstruct the full prediction, i.e., pu=UUA†pAu. We adopt the subspace formulation to make the theoretical analysis independent of the choice of basis and provide a unified characterization of identifiability, stability, and the propagation of the three error sources.
W3
We agree that the main text did not explain the idempotence of Q and its role clearly enough. Due to space limitations, we placed the detailed theoretical derivations in the appendices. The complete proof of Theorem 1 is provided in Appendix B.2, where this property is explicitly stated and used.
W4
Taking the task-side reconstruction experiment in Sec. 6.1 as an example, at K=5, we aggregate the five-fold results across 97 tasks and average equally over the six budgets L=5,⋯,10. The anchors selected by Ours cover four backbone families: Qwen, Llama, Phi, and Gemma. Adapters account for 73.6% of the selected anchor slots, the mean full-backbone parameter count is 1.88B, and 99.6% of the slots correspond to models with no more than 4B parameters. Frequently selected examples include MediPhi-Guidelines (3.8B, non-adapter) and Qwen3.5-2B / arc_easy / small_lr_r8 (a LoRA adapter with a 2B backbone). Overall, the selected anchors primarily combine small models from different backbone families with specialized adapters.
During data construction, we retained the inference time for each model-query pair. Under the common reference setting of A100 80GB, BF16, and single-query inference, the aggregate online inference cost of the anchors is approximately 3.05 GPU hours, compared with 78.80 GPU hours for evaluating all 132 models, yielding a saving of 96.13%. This shows that the selected combination supports reconstruction of overall performance while also incurring a low online evaluation cost.