acceptodds
Withdrawn from ICLR 2027

Correlation Does Not Certify a Policy Choice: What Benchmarks Should Publish Instead

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A world model that correlates at .97 with real robot trials can still pick a worse policy per task than the best single policy on that board. Evaluators like it are validated against that printed table of real outcomes by publishing a correlation; we compute what one certifies about the choice. That certificate is an attained maximum over a cone of score matrices, depending on the decision. For ranking the policies and shipping one, the usual claim, four of eleven stay within the cost of shipping the best one, and shortlisting two for hardware improves eight pairs. For choosing a policy per task, a decision we price against a correlation rather than observe, exactly one is adequate; ten leave open a regret above that baseline and two certify nothing at all. A slightly higher correlation would not rescue the finer decision. Two numbers per task and two for the board bracket the correlation ruling out a five-point regret, wherever the ranges clear the budget. That bar lies between .978 and .998 here and above .97 on 498 of 499 random boards, and its mean did not fall as more tasks of the same kind were added. A different summary converts better, and two of these benchmarks already publish it. An evaluator whose margin-weighted rank violation is over policies gives up at most choosing a policy per task, and half that where its taskwise argmax is unique. Priced exactly, the statistic beats the correlation on all eleven pairs, but for a board-plus-noise evaluator it does nearly as well: the finding is the conversion, not the corpus. With no board, the correlation's bound gives nothing on six settings of thirteen, the statistic's on one. We recommend publishing it, with trial counts, per-task margins and ranges, and a declared accuracy.

Primary area:
datasets and benchmarks
Keywords:
evaluation, benchmarks, policy selection, world models, worst-case regret, sim-to-real correlation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Jyv8OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Existing work often rank policy performance in “realistic” simulators that demonstrate a real-to-sim performance correlation. The author challenge the notion that a real-and-sim performance correlation is enough, and instead propose to rank policies based on selection regret. The paper found on 11 existing evaluation benchmarks that correlation alone gives weak selection guarantees, and that MMRV-based metric is better, and prove that it bounds policy-selection regret.

Critical strengths
  1. The paper studies a very relevant topic for realistic simulated evaluation, and ask “what metrics benchmarks should pulish”, which is very informative for future research
Critical weaknesses
  1. The writing of the paper is extremely hard to understand. There are often undefined or unintroduced terms that are used abruptly in the paper. For example, the first sentence “A world model that correlated at .97 … the best single policy on that board”, where it’s unclear what “the board” refers to. The second sentence refers to a “printed table of real outcomes”, but it is unclear what it means. Similarly, the third sentence refers to an unclear reference of “cone of score matrices”. A major rewrite is needed.
  2. Novelty: the paper recommends reporting MMRV instead of correlation. However, MMRV is a metric already proposed and used in previous work. The novelty of this paper is that it establishes that MMRV bounds policy-selection regret. This should be made explicit in the paper.
Questions for authors
  1. What is the actionable recommendation for future benchmarks to publish? Is it just MMRV? How is this different from the MMRV reported in previous benchmarks (e.g. SIMPLER, AutoEval)
4 comments

We would like to state the novelty more precisely, since the revision makes it clearer than our first response did.

The MMRV is SIMPLER's statistic, and the bound R≤KvR\le Kv is one consequence of the paper, not its contribution. The contribution is a theory of what a validation statistic certifies about a decision, with exact answers in both regimes a reader faces.

Without the tables, we give the worst-case regret as a function of the published value, with tight constants, for the pooled correlation, Kendall's τ\tau, Spearman's rSr_S, the MMRV and the average of within-task correlations (Theorem 1). The correlation's guarantee scales as K(1−r2)/2\sqrt{K(1-r^2)/2} and is exactly vacuous below a threshold (r≤.707r\le.707 for five policies), while the MMRV's is linear with tight factors KK and K/2K/2. With the tables, the guarantee is an exact multiple-choice knapsack over tasks, solved in rational arithmetic (Prop. 1, Eq. 3), and the same program certifies any statistic that decomposes over tasks, including the pairwise accuracy used to rank machine-translation metrics and language-model judges (Prop. 5).

The gap is not an artefact of the worst case. For an evaluator with errors of size σ\sigma, the correlation's guarantee is of order σ\sigma while the regret and the MMRV's guarantee are O(σ2)O(\sigma^2) (Theorem 2). On the published tables the correlation overstates the regret by a factor of 31 to 180, the MMRV by 4.6 to 6.5 (Table 2).

The theory also bounds what any statistic can do. A statistic computed within tasks, the MMRV and the averages SIMPLER and AutoEval publish included, cannot certify which policy to ship even at its perfect value (Theorem 7). On Ctrl-World an evaluator with an MMRV of 0 can ship a policy that loses .30 (Table 8). No scalar can certify both the per-task choice and shipping (Prop. 10), and for each class of decisions we identify the least certificate any published summary can give (Theorem 8). Finally, no bound computed from KK validated policies extends to a new policy with confidence above K/(K+1)K/(K+1), so a 95% claim needs at least 19 policies, more than any validation we found (Theorem 10).

These results are what our recommendation rests on. They explain why the correlation should not be the reported summary, why the MMRV alone is not enough, and what a validation must publish instead.

We have updated our general comment with a further revision (16 November). It corrects one statement from our response to you: a statistic computed within tasks cannot certify shipping from its value alone, but on some tables, such as WMT23, a perfect MMRV does. It also addresses your point on novelty with references that can be checked directly. SIMPLER introduced the MMRV within each task, without a guarantee, and its benchmark is still used to rank new policies by average success across tasks, a decision our results show no within-task statistic can certify from its value alone.

I thank the authors for their response.

The rewrite has improved over the initial submitted version, and has removed the undefined terms that causes a lot of confusion. However, I still find the new version very difficult to parse and understand. For example, the abstract mentions "We compute the largest loss in real outcome...", but it's unclear what "loss" is referring to. I think another major rewrite is still needed.

And thanks for clarifying your contribution, I now understand it much better.

Given the improvements, I will raise my score accordingly. Given that I still find the paper very hard to understand, I still recommend the paper for rejection.

Thank you for your comment and for raising your score. In the abstract, the loss is the real success rate given up by following the evaluator instead of the true outcomes, so a guarantee of .05 means the evaluator can never cost more than five points of real success per task. We did not think the paper needed a full rewrite, but we have uploaded a revision in which sentences of this kind are simpler and quicker to follow throughout the paper. If time allows, we would appreciate a quick look at the parts you found hardest to follow in the previous version.

Reviewer_7oLdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether a high correlation between an evaluator’s scores and real-world results is enough to trust its policy choices. It shows that high correlation alone does not guarantee good choices. The authors develop theoretical bounds on worst-case performance loss and analyze evaluator–benchmark pairs, finding that correlation often gives weak guarantees for selecting a policy for each task. They recommend reporting rank-based error measures, trial counts, and per-task performance gaps alongside correlation.

Critical strengths
  1. The paper identifies an important problem with using correlation as the main metric for evaluation. A high correlation with real-world performance does not necessarily mean that the evaluator will make the correct policy-selection decisions.

  2. The paper provides a theoretical framework connecting an evaluation metric to worst-case policy-selection regret on a fixed benchmark table. This makes the argument more systematic than simply showing a few counterexamples where correlation fails.

  3. The paper analyzes several existing benchmarks across robotics and other domains and recommends reporting additional information, such as rank-based statistics, trial counts, and per-task margins, that may be more informative for policy selection.

Critical weaknesses
  1. The practical recommendations would benefit from stronger empirical support through carefully controlled experiments at a larger scale. Although the paper includes synthetic experiments, its evidence from actual evaluators relies on a relatively small collection of published benchmark tables. A larger study across learned evaluators and tasks would help establish whether the recommended metrics are useful indicators of actual policy-selection performance, beyond providing tighter worst-case guarantees.

  2. As the authors acknowledge in Section 6, the evaluation corpus is small and mostly robotic: nine benchmark tables, on five of which one policy leads at least three tasks in four, and two others share cells. This limits how broadly the empirical conclusions can be generalized to other benchmarks and domains.

  3. I found the presentation difficult to follow. The paper introduces many technical quantities and terms, and I struggled to connect them to the practical evaluation problem. An early worked example showing the real performance table, evaluator scores, selected policies, and resulting regret would make the main contribution easier to understand.

Questions for authors

Could the authors provide additional evaluation on real benchmarks where different policies perform best on different tasks, to test whether the same empirical conclusions still hold?

1 comment

Thank you again for your review. Your main concern was whether the statistics we recommend track the loss that evaluators actually cause, beyond the worst case and beyond a small robotic corpus. In our response of 8 November we said that a single validation cannot test this. The current revision does, with four audits whose hypotheses were registered in advance, in which many evaluators score the same outcome table: translation metrics (WMT22 to WMT25, 702 metric and table pairs), proxies for pretraining data, reward models and architecture-search proxies. Across the WMT tables the MMRV orders the metrics by the loss they cause with a median Kendall τb\tau_b of .52.52 against .32.32 for the correlation, a difference that a bootstrap over tables places between .07.07 and .30.30, and it also does better on DataDecide (.87.87 against .79.79) and PPE (.66.66 against .64.64). Architecture search is the exception, and we report it (Sec. 4). Our updated general comment adds independent evidence from Wang et al. (2026) and corrects one statement from our earlier responses. We hope this addresses your concern about empirical support.

Reviewer_WWdiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper provides a formal analysis of several existing benchmarks and their leaderboards to expose inconsistencies between aggregate multi-task performance scores of policies / candidates on those benchmarks and their correlations with real world deployment. The paper's primary contribution is concrete recommendations for what summary and nuanced statistics should be published alongside benchmark leaderboards to improve correlations with later realized deployment performance.

===

Update after rebuttal: The abstract, introduction, and Figure 1 are much more clearly written and illustrate the the MMRV metric argument. I have raised my overall score from 2 -> 3.

Critical strengths

Reading through section 2, the contribution of this paper feels like something that would be valuable (an audit of existing benchmarks where their claims are to enable the choice of a correct policy for real-world deployment). The writing of the paper needs heavy revision, however, for comprehensibility to a human reader.

The paper contribution seems to be the call to community action "Benchmarks should publish the statistic, with trial counts, per-task margins and ranges, and a declared per-cell accuracy", though it would be really helpful if even here the authors could provide a definition of "the statistic" that doesn't require us to do reference resolution.

The community recommendation is the main contribution of this paper, I believe. Even then, it uses such bizarre language it's hard to understand. For example, when the recommendation arrives in section 4, it's introduced by "What follows binds in two cases: the three source tables that state no trial count, and the benchmarks that print a correlation and a statistic and no table." What does "binds" mean here?? It sounds like the authors mean to say "here's where our recommendation would apply to the benchmark boards we have analyzed" but instead they're using this odd LLM-like, LinkedIn influencer-sounding language everywhere. This is an academic paper submission, not a blog post.

Critical weaknesses

The abstract is thick with LLM-generated sentences, jargon that is not defined, and an odd mix of formal and informal language that includes repeated unresolved and ambiguous pronouns. While there are several obvious LLM-generated sentences, the most egregious run-on with bizarre word choices is "Priced exactly, the statistic beats the correlation on all eleven pairs, but for a board-plus-noise evaluator it does nearly as well: the finding is the conversion, not the corpus."

The paper structure and assumptions about reader knowledge of the paper content are difficult to understand. The first sentence of the Introduction opens "A world model that correlates at .97 with real robot trials can tell you which policy to ship: on three of the nine boards here, that correlation rules out any aggregate regret at all" without telling us what "nine boards here" are, what "aggregate regret" is, or what a "correlates at .97" value means (correlates by what measure? What values are being correlated?? A world model is typically an estimate of the transition function; is the correlation implied here about transition dynamics between world modeling and the robot trials??)

"It does not tell you, on any of those three, which policy to use on each task." What does not tell me? The world model??

"Both halves of that claim are computable exactly, and this paper computes them." what claims??

The middle of the introduction suddenly dives into methodology and terminology definitions, which are helpful but oddly placed. For example, page one ends defining a "board" (referred to in para 1), but the level of detail here is close to methodology. "Fix a published outcome table p of K policies on T tasks (a board), and let an evaluator choose, on each task, the policy its scores rank first. The largest regret that choice can incur at correlation r with p is the exchange rate ε*(r): a maximum over score matrices, attained, free of any distributional assumption."

Figure 1 is inscrutable. The opening of the caption "What the published evidence certifies about the decision." doesn't tell us what certification means, or what decision is being made. The caption goes on to describe the legend of the figure, but it's really hard to tell what the authors are trying to communicate. Again, the language is weirdly mixed informal and formal, with formal language using undefined terms and informal languae introducing unresolved references; (e.g., "The band runs to the regret of shipping the board’s best fixed policy, so a mark inside it means the evidence resolves the decision.")

I encourage the authors (and maybe AC) to read, for example, the paragraph from lines 071 to 081 on page 2 for an example of how inscrutable the writing in this paper is, even during the introduction. I am truly at a loss for how to parse large parts of this writing as a human reader who has worked in the field for over 15 years.

"The bar is high" paragraph is similarly inscrutable, and includes measurements (of what?), new terminology and notation, and maybe new baselines? All mixed together with statements that sound like conclusions but don't seem to have interpretable internal semantics (e.g., "In our scan, drawing more uniform random tasks never lowered the mean bar, which converges"; what is a "scan"? What is the "mean bar"? Converges to what? What tasks are being drawn and for what purpose??)

There are a lot of them, but I want to point out when a sentence is inscrutable. For example, section 2's "In practice a laboratory needs the smallest shortlist its correlation certifies at five points: one to five policies here, median two." is even syntactically difficult to walk through, and its meaning is a mystery.

Similarly difficult to parse, "A maximum over every real matrix invites the objection that its maximiser is not a plausible evaluator, and the sources print the evaluator column, so it is testable. Write δ for the largest per-cell error after the best affine calibration."

Some parts of the paper are directed at the reader, but also seem odd to me. For example, "The restriction a reader may try first, a board plus Gaussian noise, is no restriction at all: its support contains the unrestricted maximiser on every pair here, so it returns ε⋆(r) unchanged." Why do the authors state this?

Questions for authors

The last para of the intro feels related to the main claims of the paper, but is also thick with LLM-generated sentences and very difficult to parse. The opening lines are "A margin-weighted rank-violation statistic converts to a per-task regret guarantee by a factor of K, and of K/2 when the published score table has a unique taskwise argmax. The factor is attained, so no constant improves it. That closed form needs no board and is weaker for it." What does it mean to say "the factor is attained"?? What does "no constant improves it" mean? What is the meaning of "needs no board and is weaker for it"? Did the authors read the introduction themselves before submitting this manuscript?

"Proposition 1 is Moreau’s decomposition with Pythagoras, true of any closed convex cone holding the constants; the product structure over tasks and the boundary at d(a) = S are ours. Apps. A and B prove every statement of this paper." Surely the claim that Appendices A and B "prove every statement of this paper" is overstated, yes? Did the authors mean that the appendices give proofs for propositions and lemmas?

1 comment

Thank you for reading the revision and for raising your score. We have updated our general comment with a further revision (16 November). It states the contribution against how these statistics are used in current practice, with references that can be checked directly, and corrects one statement from our earlier responses.

Comments on the paper

1 comment

Dear Reviewers and Area Chair,

We have uploaded a new revision. Beyond rewriting the appendices for readability and making terminology consistent, it adds two results that sharpen the paper's claims. Theorem 2 shows that on the scale of an evaluator's errors the correlation's guarantee falls in proportion to their size while the regret falls at least as their square (Sec. 3, App. A.4). App. B.2 to B.6 extend the exact guarantee to any statistic computed task by task, including the pairwise accuracy used for machine-translation metrics and language-model judges, and identify the least certificate available to each kind of reader. No earlier result or reported number changes. Every number in the manuscript is now bound to the released output it reports (App. G.1). References in our earlier responses point to the 8 November version, which remains in the revision history.

The authors


Update, 16 November. We have uploaded a further revision, which corrects a few statements described below. References in our responses of 13 November match its numbering.

A correlation remains the standard way to validate an evaluator. It is reported by 51 of the 52 published robot tables we collected (App. E.2), and it is how translation metrics, reward models and proxies for pretraining data and architecture search are validated. SIMPLER introduced the mean maximum rank violation (MMRV), which averages over policies the largest real gap to a policy the evaluator misorders, to address known weaknesses of the correlation. It computed both numbers within each task as measures of ranking consistency, but it states no guarantee for either. The decision these numbers end up supporting is often not the one they measure. SIMPLER is still used to rank new policies by their average success across tasks, for instance in MemoryVLA (arXiv 2508.19236, ICLR 2026) and in preprints of this October and November (arXiv 2610.37381 and 2611.16589), and that is choosing which policy to ship, that is to deploy on every task. A statistic computed within tasks cannot certify that choice from its value alone (Theorem 7), and on SIMPLER's own Google Robot tables even an MMRV of 0 cannot rule out a shipping loss of 8.7 points of success rate (Table 8). Most policies ranked this way were also never part of the validation, and Theorem 10 shows that no informative bound computed from the validated policies covers a new one with confidence above K/(K+1)K/(K+1) for every exchangeable population of policies. Independent measurements point the same way. Wang et al. (2026, arXiv 2606.10366, Table 2) evaluate five recent policies under perturbations of vision, layout and language, and find that SIMPLER's rankings agree with the real ones with Spearman correlations of only .2 to .7.

The correlation does not fill the gap. On the published tables it certifies the shipped policy within five points on two of eleven validations (Figure 3). For choosing a policy separately for each task, it guarantees anything at all only if 1−r2<2.92/K1-r^2 < 2.92/K (Theorem 1), so that 766766 of 798798 correlations in four audits registered in advance guarantee nothing from rr and KK alone (Sec. 4). These results follow from a general theory, with tight guarantees from a published value for the usual correlations and rank statistics, exact guarantees from published tables for any statistic that splits over tasks, and limits that no statistic can overcome. They are why we recommend publishing the tables, or failing that the MMRV together with the policy means. Every number from our own analysis above is in the manuscript and is regenerated by the released code (App. G.1), so each can be checked directly.

In our response to Reviewer Jyv8 of 13 November we wrote that a statistic computed within tasks cannot certify shipping even at its perfect value. That was too strong. It depends on the table, and on WMT23 a perfect MMRV does certify shipping within five points. The paper now says so. We also state that the MMRV is never looser than the correlation as an observation on the pairs we examine rather than as a general result, and App. F.3 shows a table where it fails. Finally, a sharper lower bound for three policies raises the census tables on which the MMRV's guarantee is strictly tighter from 3030 to 3232 of 4747.

The authors


Update on the AI use statement. Now that the revisions are complete, we have updated the AI use statement so that it describes accurately the assistance we received, including on parts of the theory and on the experimental work added during the discussion. Earlier versions understated it; the paper now gives the details. The submission form's AI field cannot be edited at this stage, so the statement in the paper is the reference.

The authors


Final update. Final revision: wording simplified throughout (Reviewer Jyv8), proofs rechecked, and Thms. 1 and 10 stated exactly as proved (scope only). No number or conclusion changed.

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.