acceptodds
Under review as a conference paper at ICLR 2027

MOSAIC: Dynamic Benchmarking as Mechanism Design

Abstract

A benchmark scores each model on its own and ranks the scores. Those scores now decide which models get deployed and which outputs train the next round, so the leaderboard also shapes what developers build. Standalone scoring pays the same for a capability whether every other model already has it or none does. Duplicating what already scores well is then each developer's best response, and collapse onto one mode is an equilibrium: a property of the score, not of the developers. We introduce MOSAIC (Mechanism for Output Scoring that Aligns Incentives with Coverage), which pays a valid response the log-determinant volume its features add to what the record already spans: responses that the declared feature map sends to one direction, a *mode*, count as equivalent, and each further one earns less. On a shared orthogonal menu, below an explicit quality–novelty threshold, MOSAIC's equilibria are exactly the no-repeat allocations of the highest-quality modes, all of them globally optimal, where standalone scoring's unique equilibrium puts every developer on the best mode; the threshold is worst-case tight. Quality plus coverage is an exact potential: a developer's gain from replacing its own entry equals the population's gain, so every maximal improvement path ends at an equilibrium. By submodularity every such equilibrium attains at least half the optimum, and below the threshold a globally optimal population. Keeping every past response in the comparison set makes coverage grow linearly in the horizon while fresh directions remain, against standalone scoring's . In separate experiments MOSAIC raises pooled effective modes from to over twelve adaptation rounds, and the gain comes from keeping the record, whether each developer keeps its own or they share one. The same mechanism raises the share of held-out tasks some developer solves by points at unchanged per-slot accuracy, and closes of the baseline-to-SOTA gap on RSI-Exam's sealed grader.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI Evaluation, Benchmark Design, Diversity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8ewNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MOSAIC rewards responses that add new features to existing records to encourage different solutions. The paper studies when model choices become stable and overall feature coverage improves. Experiments study the effects of keeping past responses and updating scores.

Critical strengths
  • The score function rewards features that other responses do not cover well. This encourages models to output different solutions.
  • The theory connects reward score to overall feature coverage. Each answer is compared with the latest record to get its score.
  • The paper makes careful comparisons. The experiments compare private and shared records, keeping and clearing the history.
  • The paper combines theory and practice well.
Critical weaknesses
  • A new feature direction need not be a new capability. Appendix G shows that dead code can earn extra coverage, so the score still depends on a representation that rejects meaningless variation.
  • The theory pays developers for score gains, not leaderboard rank. Real developers may value beating a rival more than increasing their score; that setting falls outside the stated guarantees.
  • More correct-response modes in the training experiment do not increase held-out task coverage. The reward can encourage new ways to solve known tasks without teaching the population to solve new ones.
  • The overall structure of the paper could be further improved. For example, moving some key experimental results from the appendix to the main text would help readers better understand the paper's core contributions at an earlier stage.
Questions for authors

Please respond to the concerns raised in Critical Weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 8ewN

A new feature direction need not be a new capability. Appendix G shows that dead code can earn extra coverage, so the score still depends on a representation that rejects meaningless variation.

In App. G we pad the solutions to 164 HumanEval problems with unreachable code. A program together with its four padded versions scores 10% of the coverage range between five copies of one program (1.79 nats) and five orthogonal directions (3.47 nats), while five distinct verified solutions score 48% and beat the padded set on every problem. Deleting constant-false branches and empty-range loops before we compute the signature returns each of these padded programs to its base (App. G). For this response we also built a signature from the bytecode each program executes on its tests, and added pads that run but change no output. Under this signature padded programs score 3.5% of the range when the pads never run and 5.9% when they run, compared with 37.7% for distinct solutions.

The theory pays developers for score gains, not leaderboard rank. Real developers may value beating a rival more than increasing their score; that setting falls outside the stated guarantees.

We added Proposition 4 (App. B.7 and §4) for rank rewards on orthogonal action sets with any number d≥2d \ge 2 of modes and any non-increasing rank schedule, such as a fixed number of leaderboard slots. With equal qualities and developers ranked by MOSAIC score, every path of strict improvements made one developer at a time is finite and a pure Nash profile exists. If in addition N≥3N \ge 3 and the schedule is not constant, every pure Nash profile occupies min⁡(N,d)\min(N, d) modes. When d≥Nd \ge N these profiles repeat no mode and attain optimal coverage. Suppose instead that the mode qualities are strictly ordered, N≥3N \ge 3 and the schedule is not constant. Ranking by MOSAIC score then rules out full collapse onto one mode once w2>w1/(1+(N−2)w1)w_2 > w_1/(1+(N-2)w_1) at unit coverage scale, whereas ranking by quality alone makes full collapse the unique equilibrium.

More correct-response modes in the training experiment do not increase held-out task coverage. The reward can encourage new ways to solve known tasks without teaching the population to solve new ones.

In the training experiment the selector's features count arithmetic operators, adjacent operator pairs and chain length. The reward therefore targets how a model computes the answers to problems it already solves (§6). Training on responses selected against each developer's own persistent reference changes those computations in later models. For every Qwen2.5 size the correct answers cover more operation modes than under uniform selection, and on all five models in Fig. 3 they contain more executed intermediate-value patterns per problem. These executed values are not among the selector's features.

The overall structure of the paper could be further improved.

We made a new revision following your request, please see the new draft.

Reviewer_jxsJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MOSAIC, which addresses the tendency of standalone scoring to reward developers for duplicating capabilities that already score well. It rewards a valid response according to the log-determinant volume its features add to what the existing record already spans, so repeated modes receive diminishing rewards. The authors show that, below a quality-novelty threshold, its equilibria are globally optimal no-repeat allocations, while standalone scoring collapses onto the best mode. Quality plus coverage is an exact potential, and a persistent reference produces linear coverage growth while fresh directions remain. Experiments show that MOSAIC increases pooled effective modes and preserves coverage under concurrent updates.

Critical strengths
  • The paper provides a very important and forward-looking problem that benchmark scores increasingly affect model deployment and future training, but standalone scoring does not reward capabilities that complement the current model population.
  • The idea of designing benchmark as a mechanism has high originality.
  • Theoretical analysis and mechanism design are well-established and completed.
Critical weaknesses
  • The theory in the paper relies on a simplified setting with fixed action sets, feature maps, and developer payoffs, and the results assumes orthogonal modes and a quality-novelty threshold. But it’s unclear to what extend these assumptions reflect the real world model development.
  • The continual-adaption experiments uses only a single small base model Qwen2.5-0.5B-instruct, which limits the generalizability of the results
Questions for authors
  • Can the no-repeat and global-optimality guarantees be extended to non-orthogonal modes?
  • Do you expect the effect for continual adaption to change as model scales?
2 comments
AuthorsOpenReview ↗

Response to Reviewer jxsJ

The theory in the paper relies on a simplified setting with fixed action sets, feature maps, and developer payoffs, and the results assumes orthogonal modes and a quality-novelty threshold. But it's unclear to what extend these assumptions reflect the real world model development.

We see the feature map and the validity test as design choices of the evaluator. The evaluator announces them and keeps them fixed within each epoch (Assumption 2 in §4 and App. D.2), and our guarantees are stated for whatever map it chooses. On payoffs, developers do not need to care about the score itself. In Proposition 3 (App. B.4) each developer's utility can be any strictly increasing function of its own score. The pure Nash profiles, the finite improvement paths and the half-optimum bound all stay the same. If developers care only about rank, Proposition 4 (App. B.7) applies. Suppose N≥3N \ge 3 developers on orthogonal modes have strictly ordered mode qualities that meet Theorem 1's threshold, and they are paid by a non-constant rank schedule. Ranking them by MOSAIC score then rules out full collapse onto one mode, whereas ranking them by quality makes full collapse the only equilibrium. Over repeated rounds in which the developers themselves change, Proposition 1 (§5) needs neither a learning rule nor an equilibrium. It assumes that at each insertion some valid output stays partly new to the reference and the chosen output scores a fixed fraction of the best available.

Theorems 2 and 5 do not need orthogonality and apply to any finite action set. At w1=1w_1 = 1 the threshold wN>w1/(1+w1)w_N > w_1/(1+w_1) asks that each of the top NN modes have more than half the quality of the best one. With a binary validity test every passing response has weight one, and for d≥Nd \ge N equally weighted orthogonal modes over an empty background Proposition 2 needs no threshold at all. In that case the pure Nash profiles are exactly the profiles that repeat no mode, and each of them maximizes coverage.

Can the no-repeat and global-optimality guarantees be extended to non-orthogonal modes?

Global optimality does extend to a continuum of developers. Theorem 5 (App. B.10) assumes neither orthogonality nor a threshold. Every Nash state then maximizes quality-weighted coverage, and the Brown–von Neumann–Nash and Smith dynamics converge to these maximizers while coverage increases. For finitely many developers on any finite action set, Theorem 2 (§4) shows that quality-weighted coverage is an exact potential. Every path of strict improvements made one developer at a time is therefore finite. Every pure Nash equilibrium also attains at least half the maximum coverage, and in our experiments this bound is loose. In 12 four-slot games on frozen GSM8K responses with arithmetic-operator features (§4 and App. H.3) all 18,432 improvement paths end at a pure Nash equilibrium and 18,072 of them at a global maximizer. The worst equilibrium there keeps 0.992 of the optimal coverage.

The continual-adaption experiments uses only a single small base model Qwen2.5-0.5B-instruct, which limits the generalizability of the results Do you expect the effect for continual adaption to change as model scales?

We repeated the twelve-round adaptation protocol on Qwen2.5-1.5B, 3B and 7B and on Llama-3.2-3B (§6 and App. H.8). Selection against each developer's own persistent reference ends with more effective modes than uniform selection at every Qwen2.5 size, and the paired 95% interval excludes zero (Table 3). Llama-3.2-3B already reaches 63 modes under uniform selection, far more than any of the Qwen models.

Model (Instruct)SeedsUniform selectionOwn persistent − uniform (modes)95% CI
Qwen2.5-0.5B2414.4+18.214 to 23
Qwen2.5-1.5B2427+6.42.1 to 10.7
Qwen2.5-3B2417+18.612.1 to 25.1
Qwen2.5-7B1216+29.721.1 to 38.3
Llama-3.2-3B2463+0.1−3.8 to 4.0

We therefore expect the effect to persist as models get larger, since the gain is positive at every Qwen2.5 size we ran. Here a mode groups correct answers that use the same ordered sequence of arithmetic operators, truncated after eight operations (§6). Executed intermediate values are part of neither this count nor the selector's features, and Fig. 3 shows that their diversity increases on all five models.

Thank you for clarifying the theoretical guarantees and extending the adaptation experiments to more models. The new results address my concern about relying on a single small model. My overall assessment remains unchanged.

Reviewer_7b2VOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies mechanism design for benchmarking. The motivation is that incentives in the standsard benchmark design may implicitly push competitors toward similar models. The authors propose a payment scheme (based on the log determinant volume increase of a submitted model relative to previous submissions). Theorems and simulations show that this objective indeed promotes diversity.

Critical strengths

Nice to see work on dynamic benchmarking and mechanism design. This is generally a good direction.

The idea to pay competitors based on an objective function that promotes differentiation is reasonable.

Critical weaknesses

I was excited to see work on the relationship between dynamic benchmarking and mechanism design. Dynamic benchmarking in a competitive setting certainly involves incentives and mechanism design may help align those incentives toward better outcomes.

Unfortunately, I quickly got bogged down trying to read this paper, because it strikes me as largely AI-generated.

Take the second paragraph of the intro:

"Every one of those scores is standalone: it scores one model at a time, never the responses already on record."

Here we have a typical Claude-ism. A colon-construction followed by a forced contrast. Characteristically, the second part of the contrast is vacuous. Why would we score the responses already on record? What does that even mean?

The writing continues: "A model that solves a suite of problems is paid the same whether a hundred deployed models already solve them or nobody does." But never before was there a mention of payment. In a benchmarking setting, there are no payments to model builders typically. What does this even refer to?

The paragraph then drifts into freely associative AI writing: "Economics has the same convergence as minimum differentiation, and algorithmic decision-making has its monoculture..."

What are we even talking about at this point?

By the time we get to the third paragraph, Claude has lost the plot: "We put the question on one task in Fig. 1: return the sum 1..n". How did we get from economics' minimum differentiation to "return the sum 1..n" in one turn?

The list of contributions is so Claude-written that I understand virtually nothing, e.g.: "2. Mechanism. MOSAIC pays a valid response the log-determinant coverage it adds beyond the record, under a declared evaluation unit, validity gate, and feature map."

Beyond the record, declared evaluation unit, validity gate — what does any of this mean?

Cutting through the noise, the paper does seem to have a meaningful core idea: Reward competitors based on how much a new feature vector that they submit increases the log determinant spanned by all features submitted so far. The objective is monotone and submodular and has nice mathematical properties, standard in log-determinant arguments. However, it's not exactly clear to me what this feature vector is in an actual benchmarking context where competitors submit models or model weights. Model weights may be nearly orthogonal while encoding functionally very similar models due to permutational invariance various parts of the model weights.

I tried to decipher Theorem 1, Theorem 2, and Corollary 1. The writing keeps using the term "menu", as in "We take the menu of Section 1." I don't know what this means. My most generous reading of the theorem is that the log-determinant payments give Nash equilibrium where model providers pick orthogonal feature vectors, whereas in the standard payment game (again, not sure there is such a thing) providers pick identical vectors. Something like this at least makes and quite plausibly follows from typical log-determinant arguments.

To summarize, I think this paper contains one reasonable idea, worked out and written up by Claude. I would recommend developing and explaining the idea more thoroughly in human terms. This could become a nice paper, but it's not there yet. Aside from the jarring AI slop throughout, the theorems and simulations aren't particularly strong or illuminating. This needs more work.

Questions for authors

I recommend a major revision for clarity and human readability. This would be a revision from the ground up that I don't expect is feasible during the rebuttal period.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 7b2V [1]

Beyond the record, declared evaluation unit, validity gate — what does any of this mean?

We replaced or removed every passage you quoted, and the table below lists each one next to the revised text.

SubmittedRevised
"Every one of those scores is standalone: it scores one model at a time, never the responses already on record.""We call a score standalone when it depends on a submission alone. Such a score gives every developer with the same choices the same incentive to submit the highest-quality option."
"A model that solves a suite of problems is paid the same whether a hundred deployed models already solve them or nobody does."Removed. §1 continues with "To reward a complementary option, the evaluator must make its reward depend on what the other developers already provide."
"Economics has the same convergence as minimum differentiation (Hotelling, 1929), and algorithmic decision-making has it as monoculture, ..."Removed.
"We put the question on one task in Fig. 1: return the sum 1..n.""The sum 1 + 2 + ⋯ + n gives a concrete example (Fig. 1). A loop, a recursion and the identity n(n + 1)/2 all return the correct answer."
"2. Mechanism. MOSAIC pays a valid response the log-determinant coverage it adds beyond the record, under a declared evaluation unit, validity gate, and feature map."The list of contributions is gone. §1 says the evaluator "first checks whether a response is valid, for example by running a program's unit tests. It then computes how much coverage the response adds to a reference, the collection of responses against which it is compared."
"We take the menu of Section 1 with strict quality differences ..." (§3.1)"We now find when quality-weighted coverage makes complementary populations stable." Assumption 1 then states the setting.

What the submitted paper called the record is now the reference. From §2.1 to §4 it consists of a fixed background plus the other developers' current responses, and in §5 it contains every response accepted so far.

The validity gate is simply the first step in §2, where the evaluator runs a test such as a task's unit tests. Each response then gets a quality weight, zero if it is invalid and one if it passes a binary test. An evaluation unit is one response to one prompt (Table 2 in App. D.1), since each developer's action gives one response per prompt (§2.1).

The feature map turns each valid response into a vector. In the HumanEval example of §2 it maps a program to a signature of its syntax tree that ignores identifiers and comments (App. H.2). A copy of a solution with renamed variables therefore gets exactly the same vector. A solution built on a different algorithm can point in a new direction, and we call such a direction a mode.

But never before was there a mention of payment. In a benchmarking setting, there are no payments to model builders typically. What does this even refer to?

We use payoff for what a developer gains from its score through deployment, a higher leaderboard position or the selection of its outputs to train the next round of models (§1). In the theory this payoff is the developer's MOSAIC score. It equals the coverage that the developer's responses add to the other developers' responses, with each contribution weighted by quality (§2.1). By standalone scoring we mean the benchmark scores used today, which grade each model's responses without looking at what other models already produce.

The writing keeps using the term "menu", as in "We take the menu of Section 1." I don't know what this means.

We dropped this word from the main text. It only meant that each developer chooses one action from a finite action set (§2.1). In Fig. 1 those actions are the loop, the recursion and the closed-form identity.

AuthorsOpenReview ↗

Response to Reviewer 7b2V [2]

it's not exactly clear to me what this feature vector is in an actual benchmarking context where competitors submit models or model weights. Model weights may be nearly orthogonal while encoding functionally very similar models due to permutational invariance various parts of the model weights.

MOSAIC does not look at weights at all. It scores the responses a model gives to the benchmark's prompts (§1), so two models whose weights differ only by a permutation of hidden units produce the same distribution of responses and hence the same distribution of features. In our experiments the features are a syntax signature for programs (§2) and counts of arithmetic operators, operator pairs and chain length (§4 and §6). In the LiveCodeBench release replay of §4 we use the set of tasks each release solves. Which differences between responses count as distinct is therefore up to the evaluator when it picks the feature map. 8ewN asked the same question about dead code, and we give the numbers in our reply there.

My most generous reading of the theorem is that the log-determinant payments give Nash equilibrium where model providers pick orthogonal feature vectors, whereas in the standard payment game (again, not sure there is such a thing) providers pick identical vectors.

Theorem 1 makes this reading precise by adding one condition. Suppose N≥2N \ge 2 developers each pick one of d≥Nd \ge N orthogonal unit modes over an empty background, and mode kk has quality weight wkw_k with w1>⋯>wd>0w_1 > \cdots > w_d > 0. If wN>w1/(1+w1)w_N > w_1/(1+w_1) then the MOSAIC pure Nash equilibria are exactly the profiles with one developer on each of the NN best modes, and each of them maximizes quality-weighted coverage. The standard payment game does correspond to current practice. In what we call standalone scoring each developer maximizes its own benchmark score as on a leaderboard, and its only pure Nash equilibrium puts every developer on the best mode. Proposition 4(iii) in App. B.7 shows the same collapse when the mode qualities are strictly ordered and developers are ranked by quality under any non-constant rank schedule, for example when there are fewer leaderboard slots than developers.

Something like this at least makes and quite plausibly follows from typical log-determinant arguments.

Theorem 2 rests on earlier work, which we cite in §4 and §7. The potential follows from the marginal-contribution utility of Wolpert & Tumer (2001) and the potential games of Monderer & Shapley (1996), and the half-optimum bound is due to Vetta (2002). The rest of the theory is about decisions a benchmark designer has to make. With Theorem 1 we show how much quality spread the score tolerates. In Theorem 4 (App. B.9) we compare leave-one-out with scores that use a random subset of the other developers as the reference, and find that leave-one-out gives a second copy of the same quality the largest discount among them (twice the Shapley discount). On the orthogonal action set it is also the only one of these scores that makes coverage an exact potential. For a persistent reference, Proposition 1 bounds the mass any single mode can accumulate. As a result the effective mode count grows at least linearly, as long as some valid output stays partly new to the reference and each chosen output scores a fixed fraction of the best available. Standalone scoring with a unique best response on one fixed shared action set grows coverage only logarithmically (Corollary 3). When developers propose at the same time, the evaluator takes the proposals one at a time and scores each again against the other developers' current models. It accepts a proposal only if the proposal still scores higher than the model it replaces (§4). Over 18 replayed LiveCodeBench release schedules this never let total coverage fall, whereas scores computed when a proposal was prepared let coverage cycle in 8 of the 12 schedules where proposals overlapped.

the theorems and simulations aren't particularly strong or illuminating.

All experiments use real model outputs. In 12 four-slot games built on frozen GSM8K responses even the worst pure Nash equilibrium keeps 0.992 of the optimal coverage, where Theorem 2 only guarantees half (§4). When we fine-tune Qwen2.5-0.5B-Instruct for twelve rounds and select against each developer's own persistent reference, the pooled effective modes rise from 14.4 under uniform selection to 32.6 (positive on all 24 seeds). On Qwen2.5-7B they rise from 16 to 46. The result we find most convincing concerns the intermediate values that a correct answer computes, which are not part of the selector's features. Their diversity still increases on all five models in Fig. 3, and every paired interval stays above zero after we match the number of correct responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.