acceptodds
Under review as a conference paper at ICLR 2027

The Metanym Game: An LLM Benchmark Without Ground Truth That Rises With the Models It Measures

Abstract

We introduce a benchmark that is fully self-contained, needs no ground truth, and rises with the models it measures. Language models compete at making analogies and subjectively grade one another; nothing enters from outside. The benchmark reproduces GPQA Diamond, a keyed benchmark of expert-written questions, at r = 0.98, audited for a leak and found clean. We hypothesize that both benchmarks measure the same thing in different ways: a language model holds its knowledge as archetypal contexts, relationship patterns valid across many topic domains. GPQA instantiates the required knowledge in one domain; the Metanym Game instantiates one archetype into several domains, generating analogies, no reasoning required. The reasoning feature of an LLM hardly changes the game's ratings, while it lifts GPQA, which requires derivations. In the game, a player writes a context template whose slots, filled with a set of keywords from a topic domain, instantiate a factually true description of that domain; the instantiations are each other's metaphors, and keywords filling the same slot are metanyms, metaphorically synonymous. Correctness is settled sentence by sentence. Ground truth is replaced by the SVD of the factual rating matrix: its left and right singular vectors rate the players as judges and as generators, two ratings from one factorisation, to our knowledge a first for an LLM council of peers. On the subjective criteria, judges are weighted by their rating consistency under a swept calibration anchor. Generating and judging are different skills: on this roster the strongest generators were middling judges. A council of the five best issues the official ratings; its contestable seats keep it current, a candidate steering signal for self-improving AI. The paper is accompanied by a validating package that recomputes every number.

Primary area:
datasets and benchmarks
Keywords:
LLM evaluation, benchmark without ground truth, LLM-as-judge, peer evaluation, council of peers, spectral aggregation, singular value decomposition, eigenvector centrality, analogical reasoning, structural intelligence, contamination-resistant benchmark, self-improving AI, knowledge representation, archetypal contexts, compression

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LQKBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents the construction and usage of a benchmark for LLMs, that relies on the construction, instantiation and rating of analogies in several domains. The models to be ranked each generates a number of analogies as templates with several slots, a table with fillers for each slot from several domains, and instantiations -- as generated texts -- in each domain for the corresponding templates. Each model rates the data generated by the other models along given axes, without a reference. A reference model is chosen (whose ratings are highest according to the other models in the first evaluation round), and then each model is evaluated against the reference model. The ratings are used to score the models as both data generators and as evaluators. The ranking produced with this method is compared to a rating on a human-validated benchmark, consisting of multiple choice questions in several scientific domains at the graduate level. The Pearson correlation obtained is high. The authors emphasize that the benchmark is produced without the necessity of rhuman intervention and evaluation, and can be reconstructed as it (if it) becomes obsolete.

Critical strengths

Benchmarking the ever evolving models is an issue, particularly as existing benchmarks are incorporated in the training data of subsequent model generations. The paper proposes a benchmark that the authors describe as easy to build anew, as it does not require human feedback.

Critical weaknesses

The paper is built on several assumptions that are not obviously true:

  1. agreement among models is indicative of their goodness. This can also be the consequence of shared biases, which cannot be dismissed considering that all these models have very similar architectures, share huge amounts of training data, and have very similar training objectives.

  2. the left and right singular vectors of the factuality scores matrix have specific interpretation that reflect a model's competence. Why is this a reasonable assumption?

One good feature of the benchmark, according to the authors, is that this is free of human intervention, and as such it can be rebuilt on demand. While it showed very high correlation with the ranking obtained on a human-validated benchmark, it would still be useful to know how good are the generated templates, as well as a bit more benchmark analysis: how similar are the templates generated by the different models (within and across model families)? how good are the model evaluations?

Another good feature is considered to be the lack of contamination, as the templates, tables and instantiations are generated on the run. Given the high amounts and varied topics and forms of the data used for training, this does not seem like a justifiable assumption.

In section 6, lines 410-412: "Our hypothesis is that ... and GPQA runs on it." -- this was not proven, although the discussion of the comparison (and the way Figure 3 is drawn) uses this as an assumption. It is not clear why this is a reasonable assumption, answering questions (and multiple choice questions in particular) does not necessarily rely on an underlying model of the field.

Questions for authors

Added after discussion period with authors finished, in reply to their last comment:

This is all extremely interesting, and definitely should be added to the paper, but together with a manual validation (on a sample, of course). Without an external reference point, these numbers do not mean very much. I understand that you want to push your paper, but all the work done during the reviewing period is rushed, and patchy, and you should take your time and put it all together, and add some external validation of these numbers and analyses. The scores stand.


Please justify the two assumptions mentioned in the "Critical Weaknesses" section: the assumption of goodness, and the interpretation of the SVD decomposition.

Please clarify Figure 3. This seems to show that there is shared processing between generating templates and their instantiations and answering the multiple choice questions, but this is not obvious. Why should we assume that in the GPQA task the archetype is given. Answering questions doesn't necessarily assume an internal organization of the knowledge of the domain, just the capacity to retrieve the right answer (or to plug in each candidate answer in a sentence and choose the best according to some criterion).

Some analysis of the generated data would be very helpful and interesting, in particular: how different are the templates and instantiations generated by each model? Do they generate the same data on different runs? Do the models choose the domains for each template? Are the generated templates good (some human evaluation)? Have the models generated something particularly novel or surprising?

3 comments

Thank you for the review. Your questions about the two assumptions led us to the plain average and to the comparison with many public benchmarks. Point by point:

  • Assumption 2: why the singular vectors would mean competence. The spectral estimator is gone. It weighs each judge by its eigenvector centrality in the network of the judges' agreement with one another, so the more central judges count more. That mattered when weaker models also judged; with a council of strong judges the weights come out close to equal and it gives nearly the same leaderboard as the plain average, so the score is now the simpler one: the plain average of the factual ratings, as in a panel of judges (Verga et al., 2024).
  • "GPQA runs on it" (lines 410–412). The claim is gone. The Discussion now mentions GPQA only as a contrast: a GPQA reply works out a derivation before it picks an answer, while a metanym portfolio comes with no construction phase in the reply; it is served at once. From this, and from the game's format as a compression, the Discussion offers one clearly marked speculation: that a model keeps archetypes and topic domains apart.
  • The quality and novelty of the templates; human evaluation. Figure 1 shows a complete play by GPT-5.5 and §2 gives the size of the average template. A human evaluation is not in this revision; §3 explains why a human cannot rate whole portfolios.
  • Assumption 1: agreement may be shared bias. The submitted version named this risk, but it was hard to find; the revision states it plainly in the Ethics statement and the Limitations (§7): judges trained on similar data can share blind spots, as a human panel trained in the same school can, and addressing it is next on our agenda. §5 checks the score against public benchmarks the game never sees, Table 3 shows that the game agrees with the common ability of those benchmarks about as well as the public benchmarks do, and the council is balanced across three companies, with no judge rating its own portfolio.
Reviewer_LQKBOpenReview ↗

A new paper

Thank you for the clarifications. The authors have done much work, but the methodology is radically changed, the evaluation is changed (possibly ... it is not clear whether section 3.4 from the original paper is still used), the new draft is quite different from what was submitted.

The new draft also carries rather strong assumptions about the level of intelligence needed to play this game (section 2).

The issue of bias remains however -- acknowledging it does not make it acceptable, it's an important issue that can invalidate the premise of the work here. It may be at least partially addressed by an analysis of the portfolios that the models create: the more different they are, the better case one can make that the underlying models have different biases. This can be done automatically, using some (several would be best) text similarity measures and lexical overlap, particularly since the models conveniently generate templates, which could be more objectively compared, so to speak, as they succinctly capture the analogy and its fillers in various domains (rather than lengthier and more freeform text). I do not mean for this to replace some human checking, as the authors say is in the works. But this automatic analysis would help provide a better view of what and how varied the output of the models is.

For these reasons, my rating remains the same.

Thank you. On "a new paper", here is what is unchanged and what changed.

Unchanged

  • The Metanym Game itself: everything the models do when they play, and everything the judges do when they rate.
  • It is a benchmark without ground truth.
  • It rises with the models it measures.

Changed

  • How the ratings become a score: now the plain average of the factual ratings. Section 3.4 is no longer used.
  • More models and more benchmarks.

The analysis you suggested. We ran it. It also answers a question from your first review: how similar are the templates within and across model families, and across runs?

We compared all 581 rated context templates (39 models, 15 companies) by wording (TF-IDF cosine) and by meaning (cosine of sentence embeddings, all-mpnet-base-v2). 1 means identical; lower means less alike.

1. Different models write different templates.

Two templates fromWording: mean (SD)Meaning: mean (SD)
The same model, in another run0.07 (0.18)0.47 (0.19)
The same portfolio0.03 (0.07)0.42 (0.14)
Two models of one company0.01 (0.02)0.38 (0.13)
Two companies0.01 (0.01)0.38 (0.12)

By the rules of the game, the templates of one portfolio must represent different archetypes. Templates from different models are as far apart in meaning as those (0.38 against 0.42).

2. Families show a little, in the wording. We grouped the 39 models by hierarchical clustering (average linkage) on the similarity of their templates, cut into 15 groups, as many as there are companies. Then we compared the groups with the companies (adjusted Rand index: 1 is a full match, 0 is chance).

Grouping byMatch with the companiesWhat the groups show
Wording0.19The five newest GPT models form one group, the two Claude Opus models another
Meaning−0.04Most models fall into one large mixed group

3. More models give more variety. When we draw 10 templates at random, how many different ones do we effectively get, by meaning? 1 if they are all the same, 10 if they have nothing in common (Vendi Score; Friedman & Dieng, 2023).

Where the 10 templates come fromEffectively different templates
All from the same model5.0
All from models of the same company5.6
From any of the 39 models6.2

What this means. By your argument, the models have different biases. Models of one company are somewhat closer to each other, which is a reason to spread the council over several companies. The analysis does not rule out blind spots that all models share. A human check of a sample remains to be done.

Your other points

  • Domains. The models choose them. Portfolios of different models share about 1% of their domain names.
  • Section 2. We will reword the sentence on types of intelligence to say which abilities the game draws on. The score does not rest on it.

We will add the analysis to the paper.

Reviewer_pQk2OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose an evaluation approach where a panel of tested models is scored based on their ability to create new analogical phrases starting from a pattern of roles. In addition to the tested models the authors introduce an admin role, but it is unclear what it does. The models iterate on a template that is generated, and are compared against a reference submission that seems to be written by the admin role (Opus 4.5). The scoring criteria are fixed and ambiguous (e.g. beauty is a criterion but it is not specified)

Critical strengths

I can not find any

Critical weaknesses

The authors claim that this approach is informative of model capability because it scores models with r=0.96 correlations with GPQA Diamond. This claim is insufficiently explored:

  1. There are no experiments over multiple runs to measure variance or changing the admin model
  2. There is no analysis of errors or judgement results to indicate the scores map to capabilities, so it is unclear what model gap this evaluation measures
  3. Despite receiving no input data, the evaluation is still subject to model training biases and the choice of the fixed grading rubric
Questions for authors

How have you decided on the evaluation rubrics? How have you evaluated errors? Have you compared against other benchmarks except GPQA?

1 comment

Thank you for the review. Your questions led to some of the main changes in the revision: it is built around one criterion and one simple score, and it now compares the score with seven public benchmarks besides GPQA, as you asked. Point by point:

  • The administrator, the reference written by the administrator, the unspecified beauty criterion (W1). None of these is part of the revised protocol. The judges rate one criterion, factual correctness, from one to ten per parallel context, each with a written justification (§3). The anchor is GPT-5.5's portfolio and counts as a seven.
  • How the rubric was decided, and the fixed rubric (Q, W3). Factual correctness is the one criterion that can be checked against the world, so it is now the only one; the creative criteria are no longer used. The score is checked against public benchmarks the game never sees (§5, Tables 2 and 3), and the Ethics statement names the remaining risk, a misunderstanding the judges share.
  • Other benchmarks besides GPQA (Q). Table 2 and Figure 3 compare the score with seven public benchmarks and with release date, and Table 3 shows where the game stands among them.
  • What the scores map to (W2). Tables 2 and 3 show it: the score orders the models much as public benchmarks of knowledge, coding and agentic tasks do.
  • An analysis of the errors and the judgement results (W2, Q). The new Appendix B analyses the judgement results. Two halves of the council with no judge in common, one judge per company each, order the 38 rated models at Pearson r = 0.97; within a single model, two judges agree on which parallel contexts are weaker at r = 0.59 on average; and when one judge rates a parallel context 4 or lower, another judge does too in 68% of cases, against 27% for any parallel context (Table 5). Figure 2 shows how the judges find the errors, with three judges' complete justifications for one parallel context. A systematic classification of the errors is not in this revision.
  • Variance over runs (W1). This was in the submitted version, which was hard to read; the revision presents it more clearly. Every model plays three runs. Each score in Table 1 carries a 95% interval from resampling the model's context templates across the runs, and each correlation in Tables 2 and 3 an interval from resampling the models. The Limitations section states that the vendors' services do not answer identically twice.
Reviewer_RRx3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes the Metanym Game, an LLM benchmark that needs no ground truth.

  1. Each model writes a portfolio of five "archetypal contexts". Each one is a slotted paragraph template plus five "metanym sets" that instantiate it as a factually true description of five distant domains. Each model then rates the others' portfolios on six axes: per-instantiation factuality; beauty, intelligence, domain distinctness, and length per archetype; and structural diversity per portfolio. Ratings are relative to a fixed anchor portfolio (Claude Opus 4.5's) pinned at 7.
  2. (i) A rank-one SVD of the row-centred evaluator × instantiation factual-rating matrix. Its left vector gives evaluator factual competence E_F, and its right vector gives generator factuality G_F. (ii) An anchor-value sweep (5–8) whose test–retest Pearson correlation gives a consistency score E_C. This score also weights the subjective votes that form G_C.
  3. The total is T=1/4(G_F+G_C+E_F+E_C). The top five models by T form a "council" that issues the official ratings. Seats are contestable, protected by two "ballast" portfolios and spectral-gap guards.
  4. Twelve models from three vendors, three runs pooled. T tracks a self-administered GPQA Diamond at r=0.98. The authors report a generation–judging dissociation: the Opus models are the strongest generators but middling judges, and Gemini 3.1 Pro is the strongest judge.

A new generation-style analogy task. Models need to make a template, and the filled words must keep true in five very different fields (Para 2, A.B). This is complementary with the choice-style analogy benchmarks. The test items are generated newly every run, so it can reduce static data leakage. A two-sided spectral estimator method. Just one SVD can separate judge ability (left vector) and generator factuality (right vector), and using row-centering to delete leniency problem. The math details are very clear to every equation, include clamp, sign, self-entry filling, and Procrustes bootstrap (A.2, A.4). They also state the licensing assumption and what it cannot see very clearly (A.8, Table 8). Besides, the authors test the limit of consensus ability by making an artificial dissenter (A.7). Very honest about bad results and problems.- In Run 3, the fact axis cannot be identified, and sigma_1/sigma_2=1.29 (A. F).

  • Gemini's GPQA answers were cut off, so they have to ask it again in step two (D.2).
  • The authors clearly say what their checking cannot rule out (D.3).
  • All prompts are given word by word, original data are published, and they provide a reproduce.sh that can run directly.
Critical strengths
  1. An original production-style analogy task. Models must build a template whose substitutions stay true in five distant domains (Para 2, A.B). This complements selection-style analogy benchmarks. Items are freshly generated each run, which reduces static contamination.
  2. A two-sided spectral estimator. One SVD separates judge competence (left vector) from generator factuality (right vector), with row-centering to remove leniency. It is specified to the equation level, including clamp, sign, self-entry filling, and Procrustes-aligned bootstrap (A.2, A.4). The licensing assumption and its blind spots are stated explicitly (A.8, Table 8). The authors also probe the limits of consensus-based competence with a synthetic dissenter (A.7).
  3. Unusual transparency about negative evidence.
    • Run 3's factual axis is unidentified, with sigma_1/sigma_2=1.29 (A. F).
    • Gemini's GPQA answers were truncated and re-asked in a second stage (D.2).
    • The authors state what their audit cannot rule out (D.3).
    • Prompts are given verbatim, raw records are released, and a deterministic reproduce.sh is provided.
Critical weaknesses
  1. The headline validity evidence (r=0.98 with GPQA) cannot tell this benchmark apart from a generic capability ordering. The paper's central claims all rest on it: that the benchmark is valid, that it "measures the same thing" as GPQA, and the archetype hypothesis.
  • samples are small and clustered.12 models from three vendors, spanning gpt-4o-mini to Opus 4.5 and Gemini 3.1 Pro. The ordering is dominated by model family and generation; the GPT family is at the bottom on both instruments. Benchmark scores across LLMs are known to lie in a low-dimensional capability space. Without reference points, a correlation of 0.98 says little.
  • the machinery's added value is not shown. A.D.1 reports that the plain anchored average already reaches r=0.93. Para 4.5 concedes the differences between aggregators are "not individually resolved at n=12". So the GPQA criterion does not show that the SVD, council, and weighting add value over a naive average.
  • this correlation does not validate the factual axis. D.1 itself says GPQA agreement "is not an axis-specific claim", because G_C alone already reaches 0.95.
  • overclaims in the text.
    • "Audited for a leak and found clean" (abstract) and "as close as their measurement error allows" (Para 4.5) go beyond the evidence, need results.
    • D.3 states that differential contamination of the public GPQA set cannot be excluded. The audit only certifies the path from raw records to published numbers.
  1. The factual axis is never checked against any external truth signal; its identification is fragile, and it may be confounded by vendor blocs and by the anchor. This axis underlies E_F, G_F, council selection and the dissociation claim.
  • no external check. Humans or tools check none of the 805 parallel contexts. A small expert-labelled subset would directly test whether the singular vectors track factual errors. "No ground truth needed at run time" does not exempt the validation study from needing one.
  • the protocol does not measure what the paper says.
    • The abstract and Para 1 say correctness is "settled sentence by sentence". The protocol (B.2) actually collects one holistic 1–10 score per parallel context, relative to the reference.
    • The verbatim judgments in C.1 cite "awkward phrasing" and "lacks the depth and precision of the Reference". The axis therefore mixes factuality with style and precision relative to the Claude anchor.
  • fragile identification.
    • The per-run spectral gap is 2.57, 1.67, and 1.29.
    • Seat E_F values swing across runs: 4.68 / 3.81 / 0.01 for gemini-2.5-flash, and 3.55 / 6.78 / 5.17 for opus-4.1 (Table 15).
    • A.4 notes the leading axis is "near-degenerate between the Anthropic and Google blocs". With the council alone, the column is scrambled (Table 7).
    • The pooling policy and the gap threshold of 2 were fixed after these runs were seen. App. F calls the threshold a rule of thumb.
  • possible vendor bloc. All five "competent" judges are Claude or Gemini; every GPT judge has a loading \le 0.10. The licensing assumption ("the only thing competent judges share is the truth") is exactly what a bloc would violate. The same-vendor check in para 4.2 covers only the G_F ordering, not E_F, G_C or T. In fairness, a Claude-free judge set still places Claude generators at the top (\ge 7.0), which partially answers this concern for G_F.
Questions for authors
  1. For the same 12 models, what is the correlation with GPQA of: (a) other public benchmarks (e.g., MMLU-Pro, Arena Elo); (b) trivial proxies (release date, response length, the un-anchored plain average)? If T clearly beats these, my concern would largely be resolved.
  2. Can you have domain experts label a stratified sample (e.g., 100–200) of parallel contexts for sentence-level factual errors? Please report how well \hat r_j and the judge loadings agree with those labels.
  3. Can you run at least one evaluation pass with a non-Claude anchor (Gemini or GPT)? Do the Claude generation lead and the dissociation in para 4.4 survive?
  4. Recompute E_F using only Claude judges and only Gemini judges as the "consensus". Does Gemini 3.1 Pro remain the top judge?
3 comments

Thank you for a detailed and constructive review. Your questions shaped much of the revision: the comparison with more benchmarks and with release date, the balanced council and the non-Claude anchor all come from them. Point by point:

  • "r = 0.98 cannot tell the benchmark apart from a generic capability ordering." The revision drops the claim. The 0.98 is correct for the submitted twelve models; on the larger roster the score agrees with GPQA Diamond at r = 0.67 (Table 2), and we have no decisive explanation for the difference. Table 3 asks your question directly: each benchmark is compared with the common ability of the other public benchmarks. The game sits among them (r = 0.83, against 0.76–0.95 for the public benchmarks) and well above release date (0.63) on the same 27 models.
  • The added value of the machinery over the plain average. The spectral machinery is gone. The score is now the plain average you pointed to.
  • Q1: MMLU-Pro, Arena Elo, release date. All three are in Table 2: MMLU-Pro r = 0.81, LMArena 0.53, release month 0.53.
  • W2 and Q2: an external check of factuality. Each model submits three portfolios of about 6,000 words, some 200 sentences each; a judge fact-checks every sentence, and how they work together as a whole, against the anchor portfolio (§3). Across 39 models this is more than a human check can cover. The external check in the paper is the agreement with public benchmarks the game never sees (§5, Tables 2 and 3); the Ethics statement names the remaining risk, a misunderstanding all judges share. A labelled sample, as you suggest, would complement this. Figure 2 and the supplementary material show the judges' full justifications for one portfolio.
  • Q3: a non-Claude anchor. The anchor is now GPT-5.5's portfolio.
  • Q4: a vendor bloc. The council is balanced, two judges each from OpenAI, Anthropic and DeepSeek, and no council member rates its own portfolio. We also checked own-company ratings. OpenAI's two judges rate OpenAI's models higher than the outside judges do; Anthropic's and DeepSeek's judges show no such bonus. In the plain average this lifts OpenAI's models by about 0.2 points against the field. A production version of the benchmark can remove it, for instance by having no judge rate its own company's models, corrected for each judge's general toughness. The revision states this in its Limitations (§7).
  • Overclaims. "Audited for a leak and found clean" and "as close as their measurement error allows" are gone.

It seems like the rating axes,estimator are removed, while the anchor, judges, and roster are changed. Some of my questions are answered through modification, instead of testing or experiments. This remains my concern about council selection criteria and council model dependency, as the judging processes are all different but still seem to work. I will keep my score for now.

Thank you. You are right that tests answer your questions better than changes to the setup. We have now run them:

  • One company's judges alone (your question 4). Each company's two judges rated all models by themselves. The three leaderboards agree at r = 0.90 or more. Only the order among the top few models shifts, toward the judges' own company (§7).
  • Two halves of the council with no judge in common agree at r = 0.97 (Appendix B).
  • Two anchors (your question 3). The same judges rated the same portfolios against Claude Opus 4.5's portfolio and against GPT-5.5's. The two orders of the 22 models agree at r = 0.97 (Spearman 0.95).
  • The estimator. We recalculated the SVD with the new council. Its leaderboard agrees with the plain average at r = 1.00.

The company, two-halves and SVD tests can be recomputed from the ratings in the supplementary material.

We also looked again at the data of the submitted version. It began with a round without an anchor. In that round the judges gave nearly every portfolio 9 to 9.5 for factual correctness, weak and strong models alike, and the ratings did not follow GPQA (r = 0.12). With an anchor, the same models spread from 5.0 to 7.2, and the plain average of the factual ratings followed GPQA (r = 0.85). The submitted 0.98 was for the total score over three runs, which also used the subjective criteria, the SVD and each model's score as a judge. We removed those three parts because they did not prove themselves on the larger roster. The SVD changes nothing with the new council. The subjective criteria do not improve the agreement with the seven benchmarks together (0.80 with them, 0.83 without). And a score kept for its agreement with one benchmark over twelve models would be fitted to that benchmark. So the anchor gives the judges a common scale. It does not decide the order: two different anchors give the same one. Which strong judges rate and how the ratings are combined matter little.

On council selection: the revision has no formal rule. We chose six strong models that play with randomness and reasoning switched off, two from each of three companies (§3). The tests above show that the choice among such judges changes the ranking little.

Not yet tested: a sample checked by experts (your question 2).

We will add these tests to the appendix.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the revision

We thank the reviewers for their careful reading and their constructive critique. We have revised the manuscript in response: it is rewritten for clarity, and the game is played again on a larger roster. The contribution is the same: the metanym game as a benchmark without ground truth, scored by a council of models against a fixed anchor portfolio. The title is unchanged. Each change below names, in brackets, the points it answers.

1. One simple score, no formulas (RRx3: the added value of the machinery; LQKB: why the singular vectors would mean competence; pQk2: an unclear protocol). The revision describes the game, the council and one score: the plain average of the council's factual ratings of a model's portfolios (§3). The spectral estimator is gone, and so is the administrator role. The ballast portfolios, which spanned the rating scale for the spectral estimator, are no longer needed and are gone too.

2. One criterion, factual correctness (pQk2: the unspecified beauty criterion and how the rubric was decided; RRx3 W2: style mixed into factuality). The judges rate each parallel context on whether every sentence is factually correct, with a written justification. The creative criteria are no longer part of the score.

3. A larger and balanced run (RRx3 W1: twelve models clustered by vendor; RRx3 Q4 and LQKB: a vendor bloc or shared bias; RRx3 Q3: a non-Claude anchor). The leaderboard has grown from the twelve models of the submitted version to 39 models from 15 companies (Table 1). The council has six members, two each from OpenAI, Anthropic and DeepSeek, and the anchor is GPT-5.5's portfolio; Google and xAI offer no model strong enough for the council that plays with randomness and reasoning switched off (§3). Every model plays three runs with randomness and reasoning switched off. The Limitations (§7) state the remaining risks: judges trained on similar data can share blind spots, and the two OpenAI judges rate OpenAI's models somewhat higher than the outside judges do.

4. GPQA Diamond in proportion (RRx3 W1 and the overclaims; LQKB: "GPQA runs on it"). The submitted r = 0.98 is correct for its twelve models: it recomputes from the submitted version's package, and an audit found no error and no leak. On the larger roster the score agrees with GPQA Diamond at r = 0.67 [0.41, 0.89] over twenty models (Table 2). The roster, the council and the GPQA source (now a public listing) all changed, and we have no decisive explanation for the difference; as RRx3 pointed out, twelve models clustered by vendor cannot tell the score apart from a generic capability ordering. The revision no longer claims that the game reproduces GPQA or that GPQA runs on the same mechanism; point 5 shows where the game stands among seven benchmarks.

5. More benchmarks (RRx3 Q1: MMLU-Pro, Arena Elo, release date; RRx3 W1: a generic capability ordering). The submitted version compared the score with GPQA Diamond alone; the revision compares it with seven public benchmarks, among them MMLU-Pro and LMArena, and with release date (Table 2, Figure 3). Table 3 compares every benchmark, the game among them, with the common ability of the other benchmarks: the game agrees with it about as well as the public benchmarks do, and more than release date does.

6. What the score measures (pQk2 W2: no analysis of errors or of what the scores map to). Tables 2 and 3 show that the score orders the models much as public benchmarks of knowledge, coding and agentic tasks do. Figure 2 shows how the judges find errors: one parallel context with the complete justifications of three judges and a summary of all six. The new Appendix B shows how far the judges agree: two halves of the council with no judge in common put the models in nearly the same order (Table 5).

As in the submitted version, the supplementary material lets anyone check every number: it holds every factual rating the council gave and the public benchmark values with their sources, and one command rebuilds every table and Figure 3 and checks the tables against the paper.

The TL;DR and keywords at the top of this page are those of the submitted version: the revision form changes only the abstract, the PDF and the supplementary material.

Area_Chair_kJamOpenReview ↗

AC Reminder: Please Confirm Whether Your Concerns Are Addressed

Dear Reviewers,

The authors posted a summary of the revision and individual responses on 11 Nov, including a simplified scoring protocol and comparisons against additional public benchmarks. Please read the response addressed to your review and post a short reply indicating whether your concerns are now resolved. If issues remain, please state briefly which ones and what would settle them, so the authors have a fair chance to respond before the discussion period closes. A confirmation, or an updated rating where appropriate, would be very helpful for the final assessment.

Thank you for your time. Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.