acceptodds
Under review as a conference paper at ICLR 2027

CollabEval: Statistically Efficient Collaborative Evaluation via Matrix Completion

Abstract

Evaluating generative AI models is a routine, but resource-intensive, process that is conducted over and over again during the course of model development. In this work, we propose Collaborative Evaluation (CollabEval), a simple, effective, and principled method for exploiting dependencies between historical runs of different models on the same tasks to improve statistical efficiency. Specifically, our approach treats model evaluation as a matrix completion problem over an matrix of evaluation scores, where is the total number of models and is the total number of evaluation prompts. We assume that a subset of these models are targeted for evaluation. For these target models only a small fraction, , of prompts has been annotated with evaluation scores. Leveraging recent results in prediction-powered statistical inference, we build a low-rank approximation of the score matrix, and use the reconstructed values as control variates in a manner that guarantees unbiased estimates of the true evaluation metric mean, in addition to statistically valid confidence intervals. Empirically, across a wide range of datasets, models, and sparsity levels , we find that CollabEval substantially reduces the mean confidence interval size, and the mean squared error of the point estimate, compared to baseline methods at the same annotation budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
llms, model evaluation, matrix completion, efficient statistical inference, control variates, prediction-powered inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5e3bOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

CollabEval uses historical model evaluations to reduce the number of prompts needed to evaluate new models. It completes a partially observed model-by-prompt score matrix and uses the predictions as control variates with bias correction. It estimates both model scores and differences between models. Tests on five benchmarks show narrower confidence intervals and lower estimation error at the same evaluation budget.

Critical strengths

Historical evaluations can reduce both generation and scoring costs for new models. Using imputed scores as control variates avoids treating potentially biased predictions as ground truth. The experiments across five benchmarks show that this gives useful reductions in estimation error and confidence-interval width at the same evaluation budget.

Critical weaknesses

The PPI baseline using an untrained average of anchor scores is not strong enough to establish the need for matrix completion. A learned, target-specific regression on anchor scores, using the same cross-fitting and bias correction, would be a more informative comparison. The results demonstrate the value of historical information more clearly than the added value of the proposed matrix-completion approach.

Questions for authors

What does matrix completion add over a learned, target-specific regression on anchor scores, using the same cross-fitting, bias correction, and power tuning? Is this advantage established theoretically or isolated in any reported comparison?

1 comment
AuthorsOpenReview ↗

Author response

Thank you for the review and the positive assessment! We’d like to clear up some of your questions and concerns to hopefully help raise your confidence in our work.

With respect to your main question: what does matrix completion add over a target-specific regression on anchor scores? Our framework is not tied to a particular matrix completion algorithm. Any stable method (see Assumption A.1) that predicts the missing scores can be plugged in, and cross-fitting, bias correction, power tuning, and our handling of overlapping samples for pairwise comparisons keep inference asymptotically valid regardless. A target-specific regression on anchor scores (that is appropriately regularized) is one such method (the PPI baseline with anchor score is meant as an example of a baseline that just uses a control variate that is already provided without any fitting).

Appendix D.2 already compares IterativeSVD to a neural network, nuclear norm minimization, and a 2-PL IRT model. We use IterativeSVD because it is simple and robust, and we do not claim it is the best choice. Following your suggestion, we also ran CollabEval with a per-target ridge regression as the completion method (CollabEval-Ridge), with everything else unchanged (now in Appendix D.3). It performs about the same as IterativeSVD, provided that it is properly regularized.

Average 5 task performance. Individual model scores (θ), % MSE reduction relative to Classic (IID), with CI coverage in parentheses:

Methodp = 0.20p = 0.35p = 0.50
CollabEval-Ridge, α = 0.130.7 (88.7)26.7 (89.4)20.1 (89.5)
CollabEval-Ridge, α = 134.1 (89.1)29.1 (89.3)22.6 (89.6)
CollabEval-Ridge, α = 1037.0 (89.2)30.0 (89.3)23.0 (89.4)
CollabEval (iSVD, ranks up to 16, the default used in the paper)36.0 (89.6)30.4 (89.6)23.3 (89.8)

Additionally, note that matrix completion falls more generally under prediction with missing features, which IterativeSVD handles quite naturally. We have added an experiment to the appendix (Appendix C.6) that hides a random fraction of the anchor entries, and show that coverage is maintained at every sparsity level while efficiency degrades gracefully. Ridge and similar regression methods would require choosing how to impute the missing features separately (though this is of course possible -- even, for example, by first imputing with iSVD, and then running Ridge).

Average 5 task performance of CollabEval (iSVD, from a separate set of runs) when a random fraction of the anchor entries is hidden (columns). Individual model scores (θ), % MSE reduction relative to Classic (IID), with CI coverage in parentheses:

p \ anchors hidden0%10%25%50%75%
p = 0.2035.8 (89.4)34.8 (89.3)33.0 (89.5)30.2 (89.6)22.0 (89.6)
p = 0.3031.6 (89.7)31.5 (89.6)30.0 (89.6)26.9 (89.7)19.1 (89.8)
p = 0.4027.9 (89.8)27.1 (89.9)26.6 (90.0)23.9 (89.9)16.3 (90.0)
p = 0.5023.1 (89.8)23.1 (90.0)22.0 (89.8)20.1 (90.0)14.3 (90.0)
Reviewer_mEfUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CollabEval, which leverages historical model evaluation results through matrix completion to predict target model scores on unevaluated samples. By combining Cross-Prediction-Powered Inference and Power Tuning, it constructs statistical estimators for both individual model performance and pairwise performance differences. Experiments on five benchmarks demonstrate that CollabEval reduces estimation errors and confidence interval widths while maintaining coverage close to the nominal level.

Critical strengths
  1. This paper addresses the high cost of repeated model evaluations by leveraging historical per-sample evaluation scores to predict unobserved results, thereby reducing inference and scoring costs for target models.

  2. The authors evaluate both individual model performance estimation and pairwise model comparisons across five different tasks. The results demonstrate that CollabEval improves statistical efficiency while maintaining confidence interval coverage close to the nominal level.

  3. The proposed framework considers not only the estimation of individual model performance but also pairwise performance comparisons, and further incorporates Power Tuning to improve the efficiency of evaluation resource utilization.

Critical weaknesses
  1. The validity of the confidence intervals in Section 4.4 relies on a stability assumption for the matrix completion algorithm. However, the authors do not rigorously establish that the adopted IterativeSVD algorithm satisfies this assumption.

  2. The methodological novelty is somewhat incremental, as the proposed approach primarily combines existing techniques, including Matrix Completion, CrossPPI, and Power Tuning.

  3. The improvements in Figure 2 are primarily reported relative to Classic IID. However, for pairwise model comparisons, Classic Paired is a more natural and competitive baseline, and the incremental gains over this stronger baseline appear relatively limited.

  4. The experiments lack a dedicated ablation study to isolate the contribution of Power Tuning.

  5. Although the paper emphasizes reducing inference and evaluation costs for target models, it provides insufficient analysis of the additional computational overhead introduced by matrix completion, cross-prediction, Power Tuning, and confidence interval estimation. As the number of models, evaluation samples, or the size of the historical evaluation repository increases, the runtime and memory costs of IterativeSVD and repeated cross-fitting could become bottlenecks. The lack of end-to-end cost measurements, runtime analysis, and scalability experiments makes the practical efficiency gains less clear.

  6. One more minor presentation issue: In the first column of Figure 2, the legends obscure some of the plotted curves. Moving the legends below the plots would improve the figure's clarity.

Questions for authors
  1. Could the authors further clarify how the adopted IterativeSVD algorithm satisfies Assumption A.1?

  2. Could the authors more explicitly quantify the improvements of CollabEval over Classic Paired, particularly in terms of CI width, MSE, and computational cost?

  3. Could the authors provide additional ablation studies to better understand the contributions of individual components, such as Power Tuning?

4 comments
AuthorsOpenReview ↗

Author response (1/2)

Thank you for the review! We are glad that you found the paper clear and easy to follow. It also sounds like you agree that the problem setting is important (especially as the field moves towards longer and longer horizon evals), and that our method is a simple way to improve the efficiency of evaluation resource utilization. Please see our responses below to your raised questions and weaknesses.

W1: Stability of IterativeSVD

We had provided a small proof sketch of the stability of IterativeSVD in the Appendix (albeit in a footnote, sorry!), but did not rigorously prove it (we thought that it added more technical distraction to the paper than it was worth, as it represents only one such possible matrix completion technique, and the empirical results showed good coverage). That said, per the reviewer’s request, we have replaced the proof sketch with a formal proof. Please see Proposition A.8 in Appendix A.2.

W2: Methodological novelty

We agree that CollabEval builds on existing tools, and we view its simplicity as a strength. Applying them to this setting, however, still requires new technical work, which we describe below. Prior work on benchmark prediction (e.g., Anchor Points, IRT-based methods, and Zhang et al., 2025) targets the same goal of predicting a new model's scores from existing ones, but does not provide valid inference. CrossPPI / PPI provides valid inference, but had not been explicitly applied to matrix-completion-based predictors, which requires additional care (e.g., the stability condition required for matrix completion, the covariance under partially overlapping prompts, and joint power tuning; see Appendix A). CollabEval brings the two together. It gives valid intervals for individual scores and pairwise comparisons in this setting. The closest prior estimator (Zhang et al., 2025) uses a similar regression without cross-fitting or power tuning. It comes with no confidence intervals or validity guarantee, the standard PPI interval undercovers substantially when applied to it, and it is far more sensitive to its regularization strength (Appendix D.3).

W3: Empirical contribution

We also agree that classic inference with paired sampling is a good baseline for pairwise comparisons (at least, when one has control over the joint sampling), and that’s why we include it in the main paper. That said, pointwise evals are also still important for evaluating absolute metrics (and paired sampling does not help here). CollabEval handles both settings, in addition to pairwise settings with unequal sample sizes, such as the target vs. anchor comparisons. Even within target vs. target pairwise comparisons, where Classic (Paired) is most competitive, there are still clear gains. Averaged across the five tasks at a 20% labeling rate, CollabEval (Paired) reduces CI width by 10.4% and MSE by 17.5% relative to Classic (Paired) (6.3% and 12.5% at a 50% labeling rate). Matching that MSE would require roughly 23% more labels with Classic (Paired) on average (from 8% on AlpacaEval to 46% on AQA). Paired sampling and CollabEval are complementary, and we recommend using them together.

W4: Power-tuning ablation

Thanks for the suggestion to ablate the impact of power tuning. We report the results for CollabEval without power tuning, with (standard) univariate power tuning, and with joint power tuning using our pairwise differences derivation below for p = 0.2, across all datasets. We’ve included a plot across p in Appendix C.5.

Pairwise differences between target models at p = 0.2 with paired sampling. Each cell is % MSE reduction / % CI size reduction / CI coverage (90% nominal), relative to Classic (Paired).

DatasetNo power tuningUnivariateJoint (ours)
SWE-bench-0.6 / 3.1 / 90.08.7 / 6.5 / 88.512.7 / 8.3 / 88.6
AQA29.8 / 19.1 / 90.229.5 / 20.2 / 89.231.6 / 20.7 / 90.1
AlpacaEval6.7 / 3.6 / 89.66.9 / 4.0 / 89.57.1 / 4.7 / 89.4
WMT11.6 / 4.8 / 89.511.9 / 5.1 / 89.313.3 / 6.2 / 89.0
MMLU23.4 / 11.4 / 90.523.5 / 12.0 / 90.022.8 / 12.2 / 90.0
Average14.2 / 8.4 / 90.016.1 / 9.6 / 89.317.5 / 10.4 / 89.4

On average, power tuning (of any sort) is helpful, though the gains are modest. Joint power tuning then further helps over univariate power tuning.

AuthorsOpenReview ↗

Author response (2/2)

W5: Computational overhead

The overhead of CollabEval is negligible compared to modern LLM inference costs. It runs as a post-processing step, and does not require specialized hardware (i.e., it does not require GPUs or TPUs). To make this point explicit, we benchmarked the end-to-end estimation time and peak memory for a single target model on a single CPU core (mean over 20 runs) while varying the number of models M and examples N (we keep p fixed at 0.2, and use synthetic data). Results are below (and now in Appendix D.4). Even for the largest matrices, estimation costs are minimal (all less than 60s).

M × N1001k10k100k
80.03 s (0.1 MB)0.06 s (0.9 MB)0.32 s (9.1 MB)2.93 s (91 MB)
160.04 s (0.2 MB)0.20 s (1.6 MB)1.38 s (16 MB)14 s (158 MB)
320.11 s (0.3 MB)0.32 s (2.6 MB)1.31 s (26 MB)17 s (255 MB)
640.13 s (0.8 MB)0.42 s (4.8 MB)2.12 s (47 MB)26 s (472 MB)
1280.09 s (2.6 MB)0.77 s (9.6 MB)6.44 s (91 MB)53 s (911 MB)

Then, on the actual score matrices used in the experiments, the timing / memory are as follows:

TaskModels × promptsTargetsTimePeak memory
SWE-bench12 × 50020.06 s0.6 MB
AQA21 × 3,61020.66 s6.9 MB
AlpacaEval221 × 790170.98 s40 MB
WMT825 × 9981105.5 s668 MB
MMLU108 × 14,04267.1 s184 MB

W6: Presentation

Thanks for the suggestion. We fixed this in the updated draft.

Hopefully this, and the changes in the updated draft, addresses your questions and concerns! If so, we hope you will consider revising your score.

Thanks again for your review. Did our response address your questions? Do you have any additional questions?

Thank the authors for the detailed response and the additional experiments. The newly added theoretical analysis, experiments, and computational cost analysis have addressed some of my concerns. I truly appreciate the effort the authors have put into improving the paper. However, I still have reservations regarding the novelty of the proposed method. Specifically, the paper primarily integrates existing techniques, e.g., matrix completion, crossPPI, and power tuning, into a model evaluation task. While the statistical inference for partially overlapping observations and the "joint power tuning" approach represent some technical contributions, the new experiments show that the gains from joint power tuning over "univariate power tuning" are quite limited. Furthermore, a simple ridge regression can achieve results comparable to those of IterativeSVD. Consequently, I believe the paper’s primary contribution leans more toward the application and extension of existing methods, with relatively limited methodological innovation. Taking everything into account, I have decided to maintain my current rating.

Reviewer_rqNTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CollabEval, an approach that leverages historical data to reduce the cost of evaluating LLM model performance on prompts. The idea is related to collaborative filtering where ratings are predicted by historical ratings of the user and the similar users. Instead of directly using the predictions, the paper leverages the idea of CrossPPI and uses the predictions as control variates. It derives covariance estimates and power-tuning weights for performance estimates and pairwise comparison between targeted models for model comparison and selection.

Critical strengths
  1. The paper addresses a practical problem, i.e., how to reduce the cost of evaluating and selecting models. The proposed method aims to reduce the redundancy caused by repeated evaluation of related models.

  2. Correcting prediction errors using observed target evaluations is a principled way to leverage imperfect performance predictions.

  3. The experimental study covers CI width, MSE, and sample coverage and shows performance gain over the selected baselines.

Critical weaknesses
  1. The extension made by the paper compared to AIPW estimator (Zhang et al 2025) includes cross fitting and power tuning, but the major components of the contribution are similar (such as both of them use SVD for matrix completion, use CrossPPI for rectifying estimate). This makes the paper's novelty/contribution incremental.

  2. The theoretical analysis mainly focus on the confidence interval width, which shows that CollabEval can make the estimation more calibrated, but there is no guarantee on the evaluation accuracy. This could be the common issue of the evaluation methods that leverage both unlabelled and labelled data. But it would be more interesting if the paper can examine the impact of some important factors to the accuracy, such as the number and coverage of prompts for observed evaluation, the selection of anchor models, the diversity of the test models, etc.

  3. The paper should clarify the end-to-end complexity/cost of the proposed method and compare it with the baselines.

  4. The experimental study only covers the comparison of three simple baselines. The improvement over other baselines, such as AIPW using the same settings identical anchors should also be examined.

Questions for authors

Please see the listed weaknesses above.

3 comments
AuthorsOpenReview ↗

Author response (1/2)

Thank you for the review! We are glad that you agree that the problem CollabEval addresses is practical, and that our method is a principled solution.

With respect to the questions / weaknesses raised, please see our responses below.

W1: Contribution over AIPW (Zhang et. al., 2025).

We discuss the connection between our paper and Zhang et. al., 2025 in the related work section (lines 117–120). In particular, Zhang et. al. do not construct confidence intervals in their paper, and their estimator is biased. This is because they fit the ridge regression and compute the debiasing term on the same labeled data, with neither cross-fitting nor power tuning (this can be seen from either their paper, or implementation. When we compute the standard PPI interval for their estimator, it severely undercovers (again, since their estimator is biased).

The tables below compare Zhang et. al.’s AIPW estimator to a version of CollabEval where we use the same Ridge regressor as Zhang et. al. (we label this result as CollabEval-Ridge), but within our cross-fitting and power-tuning framework. It also shows CollabEval with our default IterativeSVD completion. Results are averaged over all 5 datasets. In terms of MSE alone, with the best regularization strength picked in hindsight (α\alpha = 10, which also happens to be their default), AIPW estimates individual model scores nearly as accurately as CollabEval. But its 90% intervals are invalid (77.6% coverage at p = 0.20, against 89.6% for CollabEval), and it is less accurate for pairwise differences. It is also not robust. With weaker regularization the in-sample fit overfits, and AIPW degrades sharply. On the other hand, CollabEval-Ridge stays close to CollabEval (IterativeSVD) and to nominal coverage on average at every α.

We view the cross-fitting, debiasing, and CI framework -- rather than the specific choice of model completion function -- as the main contribution of this work. Relative to CrossPPI, what is new is the matrix setting (models labeled on different, partially overlapping prompts), joint power tuning for pairwise comparisons, and the stability condition required of the matrix completion algorithm, which we now prove holds for IterativeSVD in Proposition A.8 (we reworded the related work section accordingly).

Individual model scores (θ), % MSE reduction relative to Classic (IID), with CI coverage (90% nominal) in parentheses:

Methodp = 0.20p = 0.35p = 0.50
AIPW, α = 0.16.3 (77.1)6.4 (83.9)8.3 (85.6)
AIPW, α = 130.2 (76.1)24.0 (83.8)17.2 (86.5)
AIPW, α = 1035.8 (77.6)29.4 (85.1)20.1 (87.1)
CollabEval-Ridge, α = 0.130.7 (88.7)26.7 (89.4)20.1 (89.5)
CollabEval-Ridge, α = 134.1 (89.1)29.1 (89.3)22.6 (89.6)
CollabEval-Ridge, α = 1037.0 (89.2)30.0 (89.3)23.0 (89.4)
CollabEval (IterativeSVD)36.0 (89.6)30.4 (89.6)23.3 (89.8)

Pairwise differences between target models (Δ), % MSE reduction relative to Classic (Paired), with CI coverage in parentheses:

Methodp = 0.20p = 0.35p = 0.50
AIPW, α = 0.1-32.3 (72.7)-25.1 (81.3)-12.6 (84.1)
AIPW, α = 15.4 (71.1)3.7 (81.4)1.6 (84.7)
AIPW, α = 1013.9 (73.0)11.8 (82.7)5.9 (85.6)
CollabEval-Ridge, α = 0.115.4 (89.3)14.5 (89.2)11.2 (89.8)
CollabEval-Ridge, α = 117.7 (89.3)16.0 (89.4)14.2 (90.1)
CollabEval-Ridge, α = 1018.7 (89.4)15.8 (89.3)12.5 (89.7)
CollabEval (IterativeSVD)17.5 (89.4)16.1 (89.5)12.5 (89.9)

W2: Scope of theoretical analysis

Our theoretical analysis shows that CollabEval is asymptotically unbiased and has lower variance than the classical estimator. Reducing the variance of an unbiased estimator also reduces its mean-squared error. The “Fractional MSE” metric we report in Figure 2 (last column) measures exactly this. CollabEval obtains substantially lower MSE than the baselines (see also Figure 4 in the Appendix for even more baselines) at the same eval budget. For the factors you mention, Figure 2 varies the number of labeled prompts, Appendix C.4 varies the anchor set, and the new Appendix C.6 hides a fraction of the anchor scores.

AuthorsOpenReview ↗

Author response (2/2)

W3: End-to-end complexity and cost

In general, the overhead of CollabEval is negligible relative to LLM inference. To make this point more explicit, we benchmarked the end-to-end estimation time and peak memory for a single target model on a single CPU core (mean over 20 runs) while varying the number of models M and examples N (we keep p fixed at 0.2, and use synthetic data). Results are below (and now in Appendix D.4). Even for the largest matrices, estimation costs are minimal (all less than 60s).

M × N1001k10k100k
80.03 s (0.1 MB)0.06 s (0.9 MB)0.32 s (9.1 MB)2.93 s (91 MB)
160.04 s (0.2 MB)0.20 s (1.6 MB)1.38 s (16 MB)14 s (158 MB)
320.11 s (0.3 MB)0.32 s (2.6 MB)1.31 s (26 MB)17 s (255 MB)
640.13 s (0.8 MB)0.42 s (4.8 MB)2.12 s (47 MB)26 s (472 MB)
1280.09 s (2.6 MB)0.77 s (9.6 MB)6.44 s (91 MB)53 s (911 MB)

Then, on the actual score matrices used in the experiments, the timing / memory are as follows:

TaskModels × promptsTargetsTimePeak memory
SWE-bench12 × 50020.06 s0.6 MB
AQA21 × 3,61020.66 s6.9 MB
AlpacaEval221 × 790170.98 s40 MB
WMT825 × 9981105.5 s668 MB
MMLU108 × 14,04267.1 s184 MB

W4: Baselines

Please see our response to W1, which includes the results for the Zhang et. al. estimator. We have added this result to the paper (Appendix D.3). In addition to the classic estimator baselines in the main paper, Appendix C.2 includes results for naive imputation, Anchor Points, and PPI using anchor model means. Appendix D.2 also includes results for other debiased matrix completion techniques (including IRT).

Hopefully this addresses your questions and concerns! If so, we hope you will consider revising your score.

Thanks again for your review. Did our response address your questions? Do you have any additional questions?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author response (general)

Thank you to the reviewers for their reviews. We have uploaded a revised draft, and have replied to individual reviewer comments and questions below.

Here is a summary of the main points.

Novelty (rqNT, mEfU). CollabEval builds on existing tools, and we view its simplicity as a strength. Applying them to this setting, however, still requires new technical work. Prior work on benchmark prediction (e.g., Anchor Points, IRT-based methods, and Zhang et al., 2025) targets the same goal of predicting a new model's scores from existing ones, but does not provide valid inference. CrossPPI / PPI provides valid inference, but had not been explicitly applied to matrix-completion-based predictors, which requires additional care (e.g., the stability condition required for matrix completion, the covariance under partially overlapping prompts, and joint power tuning; see Appendix A).

Comparison to Zhang et al., 2025 (rqNT). This is a good example of where the technical contribution of this work is important. Zhang et al. do not construct confidence intervals in their paper, and their estimator is biased, because they do not use cross-fitting. When we compute the standard PPI interval for their estimator, it severely undercovers (see Appendix D.3).

Regression instead of matrix completion (5e3b). Our framework is not tied to a particular matrix completion algorithm. Any stable method (see Assumption A.1) that predicts the missing scores can be plugged in, and cross-fitting, bias correction, power tuning, and our handling of overlapping samples for pairwise comparisons keep inference asymptotically valid regardless. A per-target ridge regression is one such method, and we now include it in Appendix D.3.

Additional results and ablations. We have now included a proof that IterativeSVD satisfies the stability condition required, and have also added ablations for power tuning and sparsely observed anchors. We also added some runtime analysis for CollabEval (and show that it is negligible compared to modern LLM inference costs).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.