- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Existing work often rank policy performance in “realistic” simulators that demonstrate a real-to-sim performance correlation. The author challenge the notion that a real-and-sim performance correlation is enough, and instead propose to rank policies based on selection regret. The paper found on 11 existing evaluation benchmarks that correlation alone gives weak selection guarantees, and that MMRV-based metric is better, and prove that it bounds policy-selection regret.
Critical strengths
- The paper studies a very relevant topic for realistic simulated evaluation, and ask “what metrics benchmarks should pulish”, which is very informative for future research
Critical weaknesses
- The writing of the paper is extremely hard to understand. There are often undefined or unintroduced terms that are used abruptly in the paper. For example, the first sentence “A world model that correlated at .97 … the best single policy on that board”, where it’s unclear what “the board” refers to. The second sentence refers to a “printed table of real outcomes”, but it is unclear what it means. Similarly, the third sentence refers to an unclear reference of “cone of score matrices”. A major rewrite is needed.
- Novelty: the paper recommends reporting MMRV instead of correlation. However, MMRV is a metric already proposed and used in previous work. The novelty of this paper is that it establishes that MMRV bounds policy-selection regret. This should be made explicit in the paper.
Questions for authors
- What is the actionable recommendation for future benchmarks to publish? Is it just MMRV? How is this different from the MMRV reported in previous benchmarks (e.g. SIMPLER, AutoEval)
4 comments
We have updated our general comment with a further revision (16 November). It corrects one statement from our response to you: a statistic computed within tasks cannot certify shipping from its value alone, but on some tables, such as WMT23, a perfect MMRV does. It also addresses your point on novelty with references that can be checked directly. SIMPLER introduced the MMRV within each task, without a guarantee, and its benchmark is still used to rank new policies by average success across tasks, a decision our results show no within-task statistic can certify from its value alone.
I thank the authors for their response.
The rewrite has improved over the initial submitted version, and has removed the undefined terms that causes a lot of confusion. However, I still find the new version very difficult to parse and understand. For example, the abstract mentions "We compute the largest loss in real outcome...", but it's unclear what "loss" is referring to. I think another major rewrite is still needed.
And thanks for clarifying your contribution, I now understand it much better.
Given the improvements, I will raise my score accordingly. Given that I still find the paper very hard to understand, I still recommend the paper for rejection.
Thank you for your comment and for raising your score. In the abstract, the loss is the real success rate given up by following the evaluator instead of the true outcomes, so a guarantee of .05 means the evaluator can never cost more than five points of real success per task. We did not think the paper needed a full rewrite, but we have uploaded a revision in which sentences of this kind are simpler and quicker to follow throughout the paper. If time allows, we would appreciate a quick look at the parts you found hardest to follow in the previous version.
We would like to state the novelty more precisely, since the revision makes it clearer than our first response did.
The MMRV is SIMPLER's statistic, and the bound R≤Kv is one consequence of the paper, not its contribution. The contribution is a theory of what a validation statistic certifies about a decision, with exact answers in both regimes a reader faces.
Without the tables, we give the worst-case regret as a function of the published value, with tight constants, for the pooled correlation, Kendall's τ, Spearman's rS, the MMRV and the average of within-task correlations (Theorem 1). The correlation's guarantee scales as K(1−r2)/2 and is exactly vacuous below a threshold (r≤.707 for five policies), while the MMRV's is linear with tight factors K and K/2. With the tables, the guarantee is an exact multiple-choice knapsack over tasks, solved in rational arithmetic (Prop. 1, Eq. 3), and the same program certifies any statistic that decomposes over tasks, including the pairwise accuracy used to rank machine-translation metrics and language-model judges (Prop. 5).
The gap is not an artefact of the worst case. For an evaluator with errors of size σ, the correlation's guarantee is of order σ while the regret and the MMRV's guarantee are O(σ2) (Theorem 2). On the published tables the correlation overstates the regret by a factor of 31 to 180, the MMRV by 4.6 to 6.5 (Table 2).
The theory also bounds what any statistic can do. A statistic computed within tasks, the MMRV and the averages SIMPLER and AutoEval publish included, cannot certify which policy to ship even at its perfect value (Theorem 7). On Ctrl-World an evaluator with an MMRV of 0 can ship a policy that loses .30 (Table 8). No scalar can certify both the per-task choice and shipping (Prop. 10), and for each class of decisions we identify the least certificate any published summary can give (Theorem 8). Finally, no bound computed from K validated policies extends to a new policy with confidence above K/(K+1), so a 95% claim needs at least 19 policies, more than any validation we found (Theorem 10).
These results are what our recommendation rests on. They explain why the correlation should not be the reported summary, why the MMRV alone is not enough, and what a validation must publish instead.