acceptodds
Under review as a conference paper at ICLR 2027

Precision Thresholds for Shared Evaluation Caches

Abstract

Model evaluation is often staged: a library is scored once on a shared set of examples, and later reports ask for more precise comparisons without rebuilding the entire evaluation table. We ask how far such a finite cache can be reused before fresh evaluation must scale with the size of the library. We consider complete i.i.d. score vectors in , arbitrary unknown cross-model dependence, and a fixed collection of mean differences that must satisfy a uniform marginal MSE target under a hard cap on fresh work. The key obstruction is residual candidate uncertainty: additional measurements of a shared reference improve every contrast, but cannot uniformly remove the uncertainty left in each candidate's finite cache. For a unit-price reference star with candidates, this yields an exact threshold separating bounded, , and linear fresh work in ; matching lower bounds hold for fully adaptive acquisition, with critical work and a transition window. A stronger layout-wise accuracy contract changes the critical law to , with an integer crossover governed by . For partial observations on general comparison graphs, we give an exact bounded-score reduction to edgewise union counts and asymptotically exact high-precision work limits.

Primary area:
learning theory
Keywords:
Minimax Estimation, Finite-Sample Phase Transitions, Adaptive Data Acquisition, Statistical Decision Theory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JoUyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the minimum additional evaluation cost required to improve estimates of model performance differences when a shared evaluation cache is already available. For multiple candidates compared against a common reference, Theorems 1–2 identify an exact precision threshold. With a fixed cache size, the optimal additional work is bounded above the threshold, grows as Θ(√M) at the threshold, and grows linearly below it. The authors also establish matching lower bounds for adaptive acquisition and determine the critical leading constant when both the cache size and the number of candidates grow. The paper further examines different accuracy requirements, derives an exact minimax risk characterization for balanced partial observations, and establishes high-precision work bounds for general comparison graphs. Numerical and frozen-score experiments illustrate these results.

Critical strengths

The main theoretical results are solid. For a fixed cache size, Theorem 1 establishes tight asymptotic orders for additional evaluation work in three precision regimes. Theorem 2 further determines the critical cost as (2+o(1))h√M when both h and M tend to infinity. Importantly, the lower bounds allow adaptive sampling, randomization, and stopping. Thus, the paper characterizes the optimal cost under its model, rather than merely analyzing a particular sampling method. This is my main reason for supporting acceptance.

The theoretical analysis has meaningful novelty. Although the minimax estimation result for a single bounded mean is classical, the paper shows how the remaining uncertainty in a finite cache determines the evaluation cost of an entire model library. The proof uses a common product prior to control the adaptive observation counts of all candidates, which is important for obtaining the lower bounds. The exact threshold also matters beyond constant-factor improvements, since replacing it with a first-order approximation can lead to the wrong cost regime.

Critical weaknesses

The strongest results still rely on a specific model. The tight three-regime characterization is established for equally sized complete caches, comparisons against a common reference, and marginal MSE. Although the paper also considers general comparison graphs, the asymptotically optimal work bound in Theorem 5 assumes d→0 and dh→0, and does not extend the full finite-cache transition to general comparison structures. Therefore, I regard this as a solid result for the proposed model, but its implications for more general evaluation allocation problems remain limited.

The broader significance of the critical regime is not sufficiently established. The square-root scaling occurs at the exact precision threshold, within a narrow transition region. Moreover, Theorem 3 shows that requiring every realized allocation to satisfy the error target can increase the critical cost to linear order. In Table 1, for M=10 and M=19, randomized and deterministic plans have the same evaluation work. The savings mainly come from sharing the reference rather than randomization. Larger controlled examples do demonstrate a separation, but it remains unclear how representative this regime is of practical model evaluation. This is my main reservation about the significance of the paper.

Questions for authors

The square-root scaling at the precision threshold is one of the most interesting results of the paper, but it depends on the precision target and the accuracy requirement. Could the authors clarify which natural model evaluation tasks would require precision close to d0(h), and why marginal MSE is the appropriate guarantee in those settings? In particular, since Table 1 does not show an additional work saving from randomization at the tested library sizes, how should we understand the practical significance of the critical regime?

Reviewer_pKYkOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

After very difficult process of trying to understand the terms, I find the work about:

  1. Trying to evaluate (termed comparison graph) ai models (termed model library), given observed model scores (termed evaluation cache).
  2. Assume cost for each edge (comparing two different model) is known, and how to find the optimal experiment design, and how the evaluation cost grow the size of graph (comparison graph), or sometimes called candidates M.
Critical strengths

The writing is very organized, visually clean, structurally complete.

Critical weaknesses

There is a chance that my opinion stems from me being outside of the intended audience so that I lack the background, so I will keep a low confidence.

However, if I view the work as theoretician, this work is clearly relavant to experiment design. The exact math formulation in section 2 probably is a new setup, and hence previous methods are not directly applicable. Yet, one should provide clean examples for what all the terms mean, following the experiment design convention; and provide background on how this setup extends / generalize known setups, with practical motivation; discuss why the analyses is nontrivial (instead of just applying concetrations).

If I view the work as a practitioner, I would expect a more practical experiment, rather than synthetic experiment. As the work is motivated by models being expensive to evaluate, which aligns well with LLMs especially in agentic / sandbox setups, it would be much better if the experiments show a more precise / efficient evaluation than things like AA or LLMArena.

Last, the alien terms used in this work seem to suggest very heavy use of AI (which is fine) but almost no human supervision.

Questions for authors

Please see weaknesses.

Reviewer_K93TOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper considers the following version of model evaluation. Given MM models with means, say, μ1,…,muM\mu_1,\dots,mu_M as well as their performance on hh samples and additional reference model say with unknown mean μ0\mu_0, the goal of the paper is to take additional samples from either the reference models or the MM models to provide estimates Δi\Delta_i of the squared differences (μi−μ0)2(\mu_i-\mu_0)^2 within small error. The main question is the tradeoff between number of samples and the maximum of these errors. A main idea is that the number of samples taken from the reference distribution provides information about all MM models, so we should probably take more samples from the reference distribution. The remaining samples they distribute evenly among the MM other distributions and they show that this given an asymptotically optimal performance (they prove a lower bound even when the distributions we sample from a picked adaptively.)

Critical strengths

Although I don't know a lot about model evaluation, the question asked in the paper seems quite natural and interesting.

Critical weaknesses

The paper is heavily AI generated and it seems like no serious effort has been made to make it readable to humans. I don't mind authors using AI generated proofs but in this day and age I think such papers are of little value without humans putting in a good amount of work to understand the content and improve the presentation. E.g., for me, the abstract was almost incomprehensible to me.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.