acceptodds
Under review as a conference paper at ICLR 2027

Anchor Reconstruction for Efficient Model-Task Evaluation: Identifiability and Stability

Abstract

As the numbers of models and tasks grow rapidly, dense evaluation becomes increasingly costly since each new model or task may require many additional evaluations. Anchor reconstruction reduces this burden by using approximately low-rank model-task structure to estimate the full performance vector of a new task or model from evaluations on a few selected models or tasks, called anchors. This setting raises three questions: what determines reconstruction across nonunique factorizations, when do sparse anchor scores uniquely determine the full vector, and how do errors arise, propagate, and become amplified? For a fixed rank- approximation, we show that the performance subspace rather than a particular basis is the intrinsic object. Anchor reconstruction is identifiable exactly when coordinate restriction is injective on this subspace, while stability is governed by the anchor minimum gain. We derive an end-to-end reconstruction error bound separating target-subspace mismatch, historical-matrix perturbation, and anchor-evaluation noise, with inverse gain amplifying all three. This explains why approximate low rank and enough anchors alone do not ensure stable reconstruction. Experiments on a matrix and a public leaderboard illustrate the rank-budget boundary for identifiability and support the predicted roles of error pathways and inverse-gain amplification in reconstruction stability.

Primary area:
datasets and benchmarks
Keywords:
Model-task evaluation, anchor reconstruction, LLM-benchmark evaluation, identifiability, stability analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aLeXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper addresses the problem of estimating the performance of all models in a given set on a new task without running every single model on it. It does so assuming that the model-task performance matrix is low rank, and so can be factorized as P=CW+RP=CW+R, in which C∈RN×KC\in \mathbb{R}^{N \times K} and W∈RK×MW\in \mathbb{R}^{K \times M}, with KK being the rank of the approximation. A new task’s performance column is then approximated with CwuCw_u, where wuw_u is a new KK-dimensional column of WW, the only unknown quantity that needs to be computed to estimate the performances of all models. From L≥KL \geq K selected so-called anchor models the linear system can be solved by least squares to estimate the scores of the remaining N−LN-L, thus requiring only LL evaluations. The problem of the basis in which CC is written is solved by proving that only the span matters and it is fixed to the one that minimizes the norm of the orthogonal projection residual (the truncated SVD). Under this basis, picking linearly independent models suffices to reconstruct the performances of the other models not yet evaluated. The paper then bounds the error between the reconstructed performances and the true ones by three components: the low-rank approximation error of the true performance, the displacement of the approximated subspace from the true one, and the measurement noise over the anchor models.

Critical strengths

The math is correct and coherent. The paper addresses a concrete and relevant problem providing a bound over performances in an interpretable way, separating it in three sources. The experimental setting proposed is satisfying and broad, covering task and model reconstruction. The protocol is robust and principled.

Critical weaknesses
  • From my understanding the method is a truncated SVD followed by row selection and least squares, however the pipeline is spread across sections 3 and 4, and the heavy notation makes it hard to follow. I understand that this might be inherited from the literature, but at least an algorithm to make the reader understand what's happening practically would simplify the understanding.
  • The notation is very heavy, and sometimes the discussion is overly complicated, with, in my opinion, little effort to make it easily understandable, while an algorithm or a framing in terms of SVD theory would be much easier to understand.
  • The bound of Theorem 1makes use of the idempotence of Q, but this is never acknowledged and appears only in the appendix, at least to me from reading the main matter it was not obvious.
  • Anchor selection is not analysed or disclosed. Please report the model family, size, whether they are adapters or base models, the corresponding compute saving...
  • No OOD task experiment. App. C.12 removes the target's family but keeps semantically related tasks in the history.
  • More baselines would have helped (naive ones like weighted averages, interpolations, or previous works like SimBA [1]).
Questions for authors

See weaknesses. Corollary 1 requires ∥Ep∥≤γk/2\lVert E_p \rVert \leq \gamma_k/2 which from my understanding can be checked; can the authors test this empirically?

[1] Subramani, Nishant, Alfredo Gomez and Mona T. Diab. “SimBA: Simplifying Benchmark Analysis Using Performance Matrices Alone.” Conference on Empirical Methods in Natural Language Processing (2025).

16 comments

W1

Sorry for not presenting the overall pipeline clearly in the original manuscript. Your understanding is correct: the reconstruction pipeline can be summarized as truncated SVD, anchor selection, and least-squares reconstruction. We have added pseudocode for the complete pipeline to the PDF (Algorithm 1, Appendix C.2), followed by descriptions of the specific anchor selection rules.

W2

We agree that the original notation and presentation increased the reading burden. From an SVD perspective, we form UU using the top KK left singular vectors of the historical matrix, estimate the least-squares coefficients using the anchor rows UAU_{\mathcal A} and observed scores, and multiply by UU to reconstruct the full prediction, i.e., p^u=UUA†pAu\widehat p_u=UU_{\mathcal A}^{\dagger}p_{\mathcal A u}. We adopt the subspace formulation to make the theoretical analysis independent of the choice of basis and provide a unified characterization of identifiability, stability, and the propagation of the three error sources.

W3

We agree that the main text did not explain the idempotence of QQ and its role clearly enough. Due to space limitations, we placed the detailed theoretical derivations in the appendices. The complete proof of Theorem 1 is provided in Appendix B.2, where this property is explicitly stated and used.

W4

Taking the task-side reconstruction experiment in Sec. 6.1 as an example, at K=5K=5, we aggregate the five-fold results across 97 tasks and average equally over the six budgets L=5,⋯ ,10L=5,\cdots,10. The anchors selected by Ours cover four backbone families: Qwen, Llama, Phi, and Gemma. Adapters account for 73.6% of the selected anchor slots, the mean full-backbone parameter count is 1.88B, and 99.6% of the slots correspond to models with no more than 4B parameters. Frequently selected examples include MediPhi-Guidelines (3.8B, non-adapter) and Qwen3.5-2B / arc_easy / small_lr_r8 (a LoRA adapter with a 2B backbone). Overall, the selected anchors primarily combine small models from different backbone families with specialized adapters.

During data construction, we retained the inference time for each model-query pair. Under the common reference setting of A100 80GB, BF16, and single-query inference, the aggregate online inference cost of the anchors is approximately 3.05 GPU hours, compared with 78.80 GPU hours for evaluating all 132 models, yielding a saving of 96.13%. This shows that the selected combination supports reconstruction of overall performance while also incurring a low online evaluation cost.

W5

We agree that the original family holdout may still retain semantically related historical tasks. We therefore added three wide-domain holdout tests to examine the target-subspace mismatch mechanism. We fix K=5K=5, E-optimal anchor selection, and subspace least-squares reconstruction, using the same 132 models, five model folds, and budgets L=5,⋯ ,10L=5,\cdots,10. Under each historical condition, the subspace is re-estimated and anchors are selected using only the corresponding historical matrix.

We added strict domain holdout experiments and random controls with the same number of historical tasks. Let T=tjj=1M\mathcal{T}=\\{t_j\\}_{j=1}^M denote the task set. For a target domain D⊆T\mathcal D\subseteq\mathcal{T}, let D′⊆T\mathcal{D}'\subseteq\mathcal{T} denote its related task set (see the table below). For each target task u∈Du\in\mathcal{D}, we predict uu using the performance matrix corresponding to Tstrict=T\(D∪D′)\mathcal{T}_{\rm strict}=\mathcal{T}\backslash(\mathcal{D}\cup\mathcal{D}'). For the random control, we randomly select ∣Tstrict∣|\mathcal{T}_{\rm strict}| tasks from T\{u}\mathcal{T}\backslash\{u\} and predict uu using the corresponding performance matrix, averaging over 10 random samples per target.

Test domain and scopeRelated tasks
Mathematics: numerical problems and mathematics multiple-choice questionsPhysics/chemistry/astronomy and electrical engineering, CS/ML/security, logic, economics/business/public relations, and ARC, OpenBookQA, QASC
Medicine/biology/healthChemistry, psychology, and ARC, OpenBookQA, QASC
Code: HumanEval/MBPP and CS/ML/securityNumerical mathematics and mathematics multiple-choice questions, logic, and electrical engineering

We then compute the projection residual and average MAE for both groups. MAE refers to clipped MAE, averaged equally over the six budgets L=5,⋯ ,10L=5,\cdots,10. The difference is defined as strict holdout minus random history.

DomainProjection residual (random history / strict holdout / difference)Average MAE (random history / strict holdout / difference)
Mathematics0.09989 / 0.12126 / +0.021370.09494 / 0.11491 / +0.01997
Medicine/biology/health0.06840 / 0.07025 / +0.001840.06652 / 0.06785 / +0.00133
Code/CS/ML0.06802 / 0.06969 / +0.001670.06579 / 0.07040 / +0.00461
Equal-weight macro-average across the three domains0.07877 / 0.08706 / +0.008290.07575 / 0.08439 / +0.00864

Both the projection residual and budget-averaged MAE increase in all three domains, consistent with the subspace mismatch mechanism.

W6

We implemented the suggested weighted average, kNN interpolation, and task-side adaptations of SimBA (Ridge / kNN). Following the task-side reconstruction setting in Sec. 6.1, we evaluated these additional baselines using the same data splits, budgets, and metrics. Below, we provide an expanded version of Table 5 from Appendix C.3. Newly added baselines are underlined, and bold indicates the best clipped MAE / ranking Spearman at each budget.

LRule / complete anchor selection and prediction pipelineOut-of-bounds (%) ↓Mean exceedance ↓Unclipped MAE ↓Clipped MAE ↓Ranking Spearman ↑
5Ours0.040.04800.048240.048230.650
5E-optimal0.120.04210.059910.059860.559
5D-optimal0.200.05160.061950.061850.554
5Q-DEIM0.200.05160.061950.061850.554
5Leverage17.171.96930.523900.185840.196
5Random19.045.92471.320210.192230.225
5<u>weighted average</u>0.010.00150.056870.056870.607
5<u>interpolation</u>0.000.00000.064090.064090.552
5<u>SimBA–Ridge</u>0.010.00660.051360.051360.625
5<u>SimBA–kNN</u>0.000.00000.060500.060500.552
10Ours0.100.03700.042000.041960.710
10E-optimal0.200.05390.055820.055710.579
10D-optimal0.080.03120.055550.055520.590
10Q-DEIM0.150.04110.058630.058570.583
10Leverage0.590.26710.072880.071300.549
10Random0.810.14650.072640.071440.484
10<u>weighted average</u>0.030.01210.053720.053710.608
10<u>interpolation</u>0.000.00000.060590.060590.548
10<u>SimBA–Ridge</u>0.010.00310.045620.045620.681
10<u>SimBA–kNN</u>0.000.00000.059370.059370.555

Question

Thank you for this suggestion. In controlled experiments with known perturbations, this condition can indeed be checked directly. Appendix C.6 already uses the empirical matrix as a reference and constructs perturbations satisfying ∣EP∣2=τγ^K\\|\mathcal{E}_P\\|_2=\tau\widehat\gamma_K, sweeping τ=0.1,⋯ ,1.0\tau=0.1,\cdots,1.0. A dashed line marks the half-gap boundary at τ=0.5\tau=0.5, allowing us to examine subspace displacement and reconstruction error both within and beyond the condition.

Reviewer_ygmgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper provides a theoretical and empirical analysis of linear anchor reconstruction for efficient model-task evaluation, rather than proposing a novel algorithm. It formalizes reconstruction as basis-independent, selected-coordinate recovery on a low-rank performance subspace. The authors derive an end-to-end error bound separating target-subspace mismatch, historical-matrix perturbation, and evaluation noise. Crucially, they demonstrate that the inverse "anchor minimum gain" amplifies all three error sources. The claims are validated through empirical evaluations and detailed ablation studies.

Critical strengths
  1. The paper clarifies when anchor reconstruction is identifiable, when it is stable, and how different error sources affect its reliability. This perspective can help the community understand the conditions and limitations of low-rank evaluation methods.

  2. The paper is well structured. The relationships between subspace structure, anchor selection, and reconstruction error are easy to follow.

  3. The experiments are systematically designed, offering thorough investigations into rank-budget behavior, anchor conditioning, and the distinct error pathways predicted by the theory.

  4. I appreciate how the paper formalizes the intuition that good anchors should collectively preserve all relevant directions of performance variation and analyzes its implications for reconstruction reliability. This provides a useful perspective on benchmark selection: individually informative tasks may still be redundant as a group, leaving important differences between models unobserved.

Critical weaknesses
  1. Despite the motivation for efficient evaluation, the experiments do not establish the computational savings–risk trade-off relative to direct evaluation. Comparisons under matched computational budgets, accounting for heterogeneous evaluation costs and amortization of historical data collection, would better substantiate the practical benefit.

  2. The analysis is coherent, but its main ingredients, including the rank-based identifiability, pseudoinverse amplification, and subspace perturbation bounds, are classical. It remains unclear which conclusions go substantially beyond combining existing results, and whether these conclusions enable new evaluation procedures or guarantees.

  3. The theory explains reconstruction error but does not provide an actionable procedure for choosing the rank, determining the number of anchors, or deciding when to abandon reconstruction and perform additional evaluations.

Questions for authors
  1. Can a target’s anchor scores be consistent with the estimated subspace while its unobserved scores deviate substantially from the reconstruction? Are there observable diagnostics for detecting such “silent failures,” or would detecting them require additional evaluations beyond the selected anchors?

  2. Geometrically informative anchors may nevertheless yield high-variance scores due to small test sets or evaluation randomness. How does anchor noise affect reconstruction reliability? Could uncertainty-aware anchor selection or weighted reconstruction improve robustness under the same total evaluation budget?

4 comments

W1

Thank you for pointing out that the original manuscript did not sufficiently present the trade-off between computational savings and reconstruction error. The original experiments primarily expressed the budget in terms of the number of anchors, without directly reflecting differences in evaluation costs across models and tasks.

When constructing the data, we kept the inference time of each model query (A100 80GB, BF16, single query). Taking the experiment in Fig. 1b as an example, the table reports equally weighted averages over the budgets L=5,…,10L=5,\ldots,10. All six anchor selection methods save more than 90% of the estimated cost, with Ours saving 96.13%. This indicates that the selected anchor set supports reconstruction of the full performance vector while incurring a low online evaluation cost.

MethodMean estimated online inference cost (GPU hours) ↓Fraction of full evaluation cost ↓Cost savings ↑Mean clipped MAE ↓
Ours3.053.87%96.13%0.04450
E-optimal3.945.00%95.00%0.05745
D-optimal5.346.78%93.22%0.05827
Q-DEIM6.548.30%91.70%0.06005
Leverage7.379.35%90.65%0.11179
Random4.485.69%94.31%0.10882
Full evaluation reference (132 models)78.80100.00%0.00%0.00000

W2

Thank you for reminding us to clarify the scope of our contributions. We agree that rank-based identifiability, pseudoinverse amplification, and singular-subspace perturbation analysis have classical foundations; we do not claim that these components were individually first proposed in this work. Our analytical contribution is a basis-independent formulation of the complete model-task evaluation pipeline, from estimating the performance structure using historical data to selecting anchors and performing online reconstruction. We distinguish target-subspace mismatch, historical-matrix perturbation, and anchor-evaluation noise, and explain how historical perturbation simultaneously changes target coverage and the stability margin of the anchors.

This error-propagation perspective directly yields the anchor selection method driven by historical reconstruction risk that we have added to the revised manuscript, directly addressing your question of whether the analysis can lead to new evaluation procedures. Worst-case control via inverse gain focuses on the most unfavorable directions, whereas realized average error also depends on the directional structure of historical performance and its interaction with the reconstruction operator. To use this information, we construct a risk proxy for MAE from historical reconstruction residuals and select anchors accordingly, using only historical data (Sec. 5.4 and Appendix C.2 of the revised manuscript). Fig. 1b and Table 5 show that, under the same anchor budget, Ours achieves lower average reconstruction error than the geometric baselines considered.

W3

Thank you for pointing out the limitations of our configuration guidance. We provide the following practical suggestions.

Rank. KK can be selected based on the singular spectrum and SVD residuals of the historical performance matrix to retain the main performance structure.

Number of anchors. L⩾KL\geqslant K is a necessary condition for the identifiability of rank-KK subspace reconstruction, while stability also depends on the conditioning of the selected anchors. Fig. 1b, Fig. 4, Fig. 9b, Fig. 10, and Fig. 21b show that relatively low MAE can already be achieved at L=KL=K using anchor selection methods such as Ours and E-optimal. Therefore, we suggest first selecting KK using SVD and then starting with L=KL=K using an appropriate anchor selection method to obtain good reconstruction accuracy with a small evaluation budget.

Configuration adjustment or additional evaluations. If the anchor set is not identifiable, is poorly conditioned, or fails to meet the historical validation error requirement, one can adjust KK, add anchors, or switch to direct evaluation.

Q1

Yes. Consistency between the anchor scores and the estimated subspace does not rule out deviations at unobserved coordinates. Theorem 1 explicitly retains the target-subspace mismatch term, while identifiability guarantees recovery only within the subspace. Therefore, silent failure cannot generally be detected using the original anchor scores alone. Checking the reliability of a new target requires additional independent audit observations or explicit distributional assumptions, with the audit cost included in the budget.

Q2

Anchor noise propagates through the reconstruction operator, with its worst-case amplification controlled by inverse anchor gain. The finite-sample bound in Sec. 5.2 and the experiments in Fig. 2c and Appendix C.7 show that increasing the number of samples per anchor can reduce noise and excess reconstruction error.

Noise-aware anchor selection and weighted reconstruction are reasonable extensions. However, when L=KL=K and the anchor set is identifiable, simply changing the positive-definite weights in least squares does not change the solution; weighting can be further investigated in settings with redundant observations.

Thanks for the authors' rebuttal and new experiments. The cost estimates and improved reconstruction results are helpful. However, given the largely classical theoretical foundations, I would expect stronger evidence of what the approach enables specifically for model evaluation. Lower reconstruction MAE and substantial online savings do not by themselves establish that meaningful model comparisons are preserved, particularly among closely performing models. My concern about the overall significance therefore remains, and I maintain my rating.

Thank you for raising the question of model comparison. For a new task, we can evaluate a few anchor models and reconstructs the remaining candidates’ scores, which can be used to rank and recommend models for that task.

Table 5 (Appendix C.3) already reports Spearman correlations of 0.650 and 0.710 at L=5L=5 and L=10L=10, respectively, versus 0.559 and 0.590 for the best baselines in that table. In the expanded comparison, SimBA–Ridge reaches 0.625 and 0.681.

To test how well the method distinguishes closely performing models, we compared all ten methods on pairs of held-out models within the same target task and evaluation fold. We grouped pairs by observed score gap, excluded observed ties, and averaged accuracy equally across tasks. The table reports strict pairwise accuracy for Ours, counting predicted ties as incorrect, with 95% confidence intervals from resampling tasks.

Observed score gapPair comparisonsL=5L=5L=10L=10
All non-tied pairs143,91177.31% [75.87%, 78.69%]80.33% [79.10%, 81.52%]
Greater than 5 percentage points106,46582.25% [80.56%, 83.93%]85.41% [83.80%, 87.01%]
Greater than 0 and up to 1 percentage point6,10153.51% [52.02%, 54.96%]53.60% [51.86%, 55.29%]

Ours has the highest overall pairwise accuracy at both budgets, exceeding SimBA-Ridge by 1.38 and 1.55 percentage points. We suggest using reconstructed rankings to shortlist candidates and directly evaluating candidates with similar predicted scores when their order determines the final recommendation.

More broadly, prior studies have also used partial evaluations to support model recommendation and ranking. Maimon et al. [1, Sec. 6.3] evaluate a small set of representative models on a new task and predict the remaining models’ scores to recommend candidates. BenchPress [2, Sec. 5.2 and Appendix D.2] evaluates whether completing missing benchmark scores preserves pairwise order and recovers top-ranked models. Across these different methods and evaluation protocols, the shared idea is to use a limited set of evaluations to estimate unobserved model-task scores and use those estimates to guide model comparison and selection.

References

[1] Aviya Maimon, Amir David Nisan Cohen, Gal Vishne, Shauli Ravfogel, and Reut Tsarfaty. From Benchmarks to Skills: Low-Rank Factors for LLM Evaluation. Transactions of the Association for Computational Linguistics, 14:1660–1684, 2026.

[2] Yuchen Zeng and Dimitris Papailiopoulos. You Don’t Need to Run Every Eval. arXiv preprint arXiv:2606.24020, 2026.

Reviewer_cGN7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to estimate the performance of a new model or task using a small number of anchor evaluations. The authors analyze anchor reconstruction from the perspective of low-dimensional subspace geometry. They establish the conditions for unique reconstruction and derive an error bound that considers target-subspace mismatch, historical-matrix perturbation, and anchor-evaluation noise. Experiments on a model-task matrix and a public LLM leaderboard support the theoretical analysis and show the importance of anchor selection and reconstruction stability.

Critical strengths
  1. The paper studies an important practical problem in LLM evaluation. Reducing the number of required model-task evaluations can help lower evaluation costs as the number of models and benchmarks grows.
  2. The theoretical analysis is clear and well organized. The authors distinguish between identifiability and stability, showing that having enough anchors does not necessarily guarantee reliable reconstruction.
  3. The paper provides a systematic error analysis that considers three different error sources. It also explains how poor anchor conditioning can amplify reconstruction errors, offering useful insights into anchor selection.
  4. The experiments cover both model-side and task-side reconstruction, different anchor-selection strategies, and multiple error settings. The additional public leaderboard experiments further support the main findings.
Critical weaknesses
  1. The theoretical novelty is somewhat limited. Many key results, including basis invariance, rank conditions, and inverse-conditioning bounds, are already established in low-rank reconstruction and sensor selection. The paper mainly applies and combines these results in the model-task evaluation setting.
  2. The paper does not propose a new reconstruction or anchor-selection algorithm. Its practical recommendations mainly rely on existing methods, such as E-optimal selection. This limits the methodological contribution beyond the theoretical analysis.
  3. The experimental evaluation has some limitations. The main performance matrix includes many related model checkpoints and adapters, while the independent public leaderboard contains only six benchmark tasks. It remains unclear whether the findings generalize to more diverse and independent models and tasks.
  4. Although the paper provides theoretical error bounds, these bounds can be loose for individual reconstruction cases. More practical guidance on how to select the rank and anchor budget based on the error bounds would strengthen the contribution.
Questions for authors
  1. What are the main theoretical contributions of this paper compared with existing studies on low-rank reconstruction, gappy POD, and sensor selection?
  2. Can the proposed analysis provide a practical method for selecting the reconstruction rank and anchor budget under a given accuracy requirement or evaluation cost?
  3. Would the main findings remain consistent on larger collections of independent models and more diverse benchmark tasks?
1 comment

W1 & W2 & Q1

Thank you for your suggestions. We agree that these theoretical analyses have classical foundations, and the original manuscript already discusses the relevant connections in the related work section. Building on these classical results, we analyze the complete model-task evaluation pipeline of estimating the performance structure from historical data, selecting anchors, and reconstructing the performance vector for a new target. We distinguish target-subspace mismatch, historical-matrix perturbation, and anchor-evaluation noise, and characterize how historical perturbation simultaneously affects target coverage and the stability margin of the anchors.

This error-propagation analysis also directly yields an anchor selection algorithm for minimizing historical reconstruction residuals (we have updated the PDF; details are provided in Sec. 5.4 and Appendix C.2 of the revised manuscript). We have updated the experiments in the revised manuscript accordingly: under the same anchor budget, the proposed method achieves the lowest MAE in the main comparison (Fig. 1b and Table 5), demonstrating that the analysis can lead to an effective evaluation method.

W3 & Q3

Thank you for pointing out the limitations in data coverage. We have added experiments on a new dataset containing 55 models, including recent frontier models such as GPT-5.6-sol, Kimi-K3, Claude-Fable-5, DeepSeek-V4-Pro, and Qwen3.8-Max, and 33 benchmark tasks spanning six domains: Code, General Reasoning, Knowledge QA, Long Context, Mathematical Reasoning, and Reading Understanding.

The experiments on the new dataset reproduce the patterns observed in the main experiments: the rank-budget boundary, the effect of anchor selection on reconstruction accuracy, and the inverse-gain amplification trend shared by the three error pathways. These results further support the consistency of our main findings across different model compositions and broader task coverage. The complete dataset description, experimental settings, and results are provided in Appendix F of the revised manuscript.

W4 & Q2

Thank you for your suggestions. We agree that the error bound may be conservative for individual reconstruction instances and primarily serves to explain the three error sources and their amplification mechanisms. In practice, KK can be selected based on the singular spectrum and SVD residuals of the historical performance matrix to retain the main performance structure.

For the anchor budget, L⩾KL\geqslant K is a necessary condition for the identifiability of rank-KK subspace reconstruction, while stability also depends on the conditioning of the selected anchors. Fig. 1b, Fig. 4, Fig. 9b, Fig. 10, and Fig. 21b show that relatively low MAE can already be achieved at L=KL=K using anchor selection methods such as Ours and E-optimal. Therefore, we suggest first selecting KK using SVD and then starting with L=KL=K using an appropriate anchor selection method to obtain good reconstruction accuracy with a small evaluation budget.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response and Manuscript Updates

Revision summary

We added an anchor selection algorithm (Ours) derived from our analysis, its experimental evaluation, and experiments on a new dataset of 55 models (including recent frontier models such as GPT-5.6-sol, Kimi-K3, Claude-Fable-5, DeepSeek-V4-Pro, and Qwen3.8-Max) and 33 tasks (Appendix F). We also added end-to-end pseudocode (Algorithm 1), clarified the idempotence of QQ and its role in deriving the error bound (Sec. 5.2), and added a Limitations section (Sec. 8).

We thank all reviewers for their careful reading and constructive suggestions. We have supplemented the method and experiments and address the following shared concerns here. Other specific questions are addressed in our individual responses.

Theoretical Contributions and the New Algorithm (cGN7 W1 & W2 & Q1; ygmg W2)

We agree that rank-based identifiability, pseudoinverse amplification, and singular-subspace perturbation analysis have classical foundations. Our work analyzes the complete model-task evaluation pipeline, from estimating historical performance structure to selecting anchors and reconstructing new targets. We provide a basis-independent formulation, distinguish target-subspace mismatch, historical-matrix perturbation, and anchor-evaluation noise, and characterize how historical perturbation simultaneously affects target coverage and the anchor stability margin. This explains why approximate low rank and enough anchors alone do not ensure stable reconstruction.

This error-propagation analysis directly yields an anchor selection algorithm. Inverse gain controls worst-case amplification, whereas realized average reconstruction error also depends on the directional structure of historical residuals and their interaction with the reconstruction operator. We therefore construct an MAE-oriented risk proxy from historical reconstruction residuals to guide anchor selection, using only historical data for selection and hyperparameter tuning (Sec. 5.4 and Appendix C.2 of the revised manuscript). Fig. 1b and Table 5 show that, under the same anchor budget, the complete anchor selection and reconstruction pipeline of Ours achieves the lowest average MAE in the main comparison.

Experimental Coverage and Validation (cGN7 W3 & Q3; aLeX W4 & W5 & W6)

We added a dataset containing 55 models, including frontier models such as GPT-5.6-sol, Kimi-K3, Claude-Fable-5, DeepSeek-V4-Pro, and Qwen3.8-Max, and 33 tasks spanning code, general reasoning, knowledge question answering, long context, mathematical reasoning, and reading comprehension. Appendix F of the revised manuscript reports the data, protocol, and experiments reproducing the rank-budget boundary, the effect of anchor selection on reconstruction accuracy, and the inverse-gain amplification mechanism shared by the three error sources.

During the discussion, we added weighted average, kNN interpolation, and task-side adaptations of SimBA (Ridge / kNN) as baselines. In the expanded comparison at L=5L=5 and L=10L=10, the complete anchor selection and reconstruction pipeline of Ours achieves the lowest clipped MAE and highest ranking Spearman among the compared pipelines.

To further address the OOD question, we added code-output format holdout experiments on the original 132-model pool and constructed a holdout setting involving 26 VL models from Appendix F, two VL tasks, and 33 text tasks. In the format-holdout experiment on the original 132-model pool, both the rank-5 projection residual and reconstruction MAE are higher than under full history and size-matched code-containing history. This empirical result is consistent with the target-subspace mismatch mechanism: when the historical subspace cannot adequately represent an OOD target's performance structure, reliable anchor reconstruction based on that subspace is not guaranteed.

Despite these additional experiments, we still acknowledge limitations. We added a Limitations section to the revised manuscript (Sec. 8, p. 10), explicitly acknowledging two limitations: (1) model pool composition and adapters sharing backbones may affect the historical subspace and effective model diversity; (2) reliable subspace reconstruction is not guaranteed for OOD targets poorly represented by the historical subspace.

Evaluation Cost and Savings (ygmg W1; aLeX W4)

The original experiments express the budget through the number of anchors and do not sufficiently reflect cost differences across models and tasks. During data construction, we retained the inference time for each model-query pair under a common reference setting of A100 80GB, BF16, and single-query inference. Taking the task-side reconstruction experiment in Sec. 6.1 / Fig. 1b as an example, we use the same target tasks, five model folds, and evaluation coordinates, and average equally over the six budgets L=5,⋯ ,10L=5,\cdots,10, obtaining the results below. All six methods save more than 90% of the online inference cost; Ours saves 96.13%, with an average clipped MAE of 0.04450.

MethodAverage online cost (reference GPU hours) ↓Fraction of full evaluation cost ↓Cost savings ↑Average clipped MAE ↓
Ours3.053.87%96.13%0.04450
E-optimal3.945.00%95.00%0.05745
D-optimal5.346.78%93.22%0.05827
Q-DEIM6.548.30%91.70%0.06005
Leverage7.379.35%90.65%0.11179
Random4.485.69%94.31%0.10882
Full evaluation reference: 132 models78.80100.00%0.00%0.00000

Selecting the Rank and Anchor Budget (cGN7 W4 & Q2; ygmg W3)

We agree that the error bound may be conservative for individual reconstruction instances and primarily serves to explain the error sources and amplification mechanisms. For practical use, we discuss the selection of rank KK and anchor budget LL separately.

Rank KK. For rank-KK subspace reconstruction, candidate values of KK can be obtained from the singular spectrum and SVD residuals of the historical matrix. Historical holdout validation can then compare reconstruction errors across ranks to retain the main performance structure.

Anchor budget LL. L⩾KL\geqslant K is a necessary condition for identifiable subspace reconstruction, while stability also depends on the anchor minimum gain. Fig. 1b, Fig. 4, Fig. 9b, Fig. 10, and the corresponding experiments in Appendix F support using anchor selection methods under small budgets. For subspace reconstruction, one can start with L=KL=K and adjust the budget based on historical validation error and evaluation cost. If conditioning is poor or validation error does not meet the requirement, one can adjust KK, add anchors, or switch to direct evaluation.

Dear Reviewers,

Thank you for your thoughtful reviews and contributions. The authors have now submitted their rebuttals. Please take some time to review their responses and assess whether your concerns have been adequately addressed. If appropriate, please update your comments and ratings accordingly.

Thank you for your time and continued efforts.

Best regards,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.