- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a method to estimate the performance of a model using a small selection of benchmarks within the entire collection of benchmarks, to grade the model performance across a variety of domains. The authors compile a sparse matrix from a large collection of public results, discovering that variations can be captured by a low-dimensional representation. Their approach BENCHPRESS uses this information to define a logit space rank 2 matrix completion method that can predict missing scores based on whichever benchmark scores are provided. They also define a smaller set of low-cost benchmarks to apply their technique to. The authors provide a wide set of ablations, covering prediction reliability, agreement, examining model rankings, generalizability to new unseen models and more.
Critical strengths
- The method offers a simple and elegant solution to estimating model performance based on a subset of benchmarks. The method relies on benchmark scores, making it easy to apply.
- The paper spends ample time discussing the reliability of the predictive score, defining good uncertainty estimates to help define when the score is useful, and where it may warrant further benchmark scores.
Critical weaknesses
- Table 21 questions the significance of this method, by comparing it to simpler baselines. I did have trouble parsing this piece of information, but I believe that their method in sparse public score settings does not guarentee benefit over simpler methods such as model-mean. I still believe this is a good finding to report.
Questions for authors
- Could the authors provide more information on how the 25-benchmark low-cost allowlist constructed? What criteria were used to classify benchmarks as inexpensive?
3 comments
I thank the authors for their detailed explanation and analysis. The explanation for Table 21 has been very helpful for me to understand the results and learnings from this ablation.
In short I have decided to increase my score, as I believe this work is valuable research.
Thank You for Your Thoughtful Review and Support
Thank you for carefully reading our rebuttal and for your support. We are glad our responses helped address your concerns. Thank you again for your encouraging feedback and constructive suggestions!
Response to Reviewer 5ChR: Baseline Comparisons and Low-Cost Benchmark Selection
We thank the reviewer for finding (1) the method simple and elegant, relying only on benchmark scores and easy to apply, and (2) the reliability analysis useful for deciding when a prediction warrants running more benchmarks.
Your reading is correct: on these new models, BenchPress is not guaranteed to beat the model mean when only one or two scores are known. The main reason is a distribution shift between the older, weaker models in the main matrix and the newer, stronger ones, rather than a general weakness of the method. BenchPress is fit mostly on the older models and knows only one or two scores of a new model, while the model mean is fit directly on the new model's known scores, which come from our probes, chosen as the most predictive benchmarks. BenchPress therefore needs more than two known scores to adapt to the new distribution: with three known scores, it is already more accurate.
The original Table 21 made this hard to see for two reasons. First, its rows were indexed by k, the number of probes requested, but many new models reported only one or two of the probes, so a larger k often added no known scores. Second, its rows were scored on different test cells, for two reasons: a model entered a row only if it reported at least one of the first k probes, so the set of models changed with k; and as k grew, more of a model's scores were revealed and were no longer tested. The rows therefore could not be compared directly. We have replaced Table 21 in Appendix D.3 with a comparison of the same predictions grouped by the number of revealed scores, and clarified the evaluation setup. Among the models added in the later matrix, 11 have at least 3 benchmarks in the probe set reported under the cost-unaware probes, and 8 under the cost-aware probes. The table below reports MedAE in score points (BenchPress / model mean) on these models, on the same test cells at every count. It stops at three because, within one ordering, only one or two models reveal every count up to four. These samples are small, which is why we keep this comparison in the appendix; we do not over-interpret it and present it only to clarify the part that was hard to parse.
Together, these show that BenchPress is the better predictor once a few scores are known.
We constructed the allowlist in two steps:
Note that the cost still varies across users, depending on how many turns and how much reasoning effort they choose to spend on each benchmark. The allowlist is therefore not fixed: users can rerun the pipeline with costs from their own models and reasoning settings to obtain their own allowlist and cost-aware probes via our algorithm. Following the reviewer's comment, we have added this explanation to Appendix D.1.
Final Note: Thank you again for your support and constructive feedback. We have incorporated your suggestions into the revised manuscript, marked in navy blue. We would be grateful if you could champion our paper during the discussion.
References