- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MOSAIC rewards responses that add new features to existing records to encourage different solutions. The paper studies when model choices become stable and overall feature coverage improves. Experiments study the effects of keeping past responses and updating scores.
Critical strengths
- The score function rewards features that other responses do not cover well. This encourages models to output different solutions.
- The theory connects reward score to overall feature coverage. Each answer is compared with the latest record to get its score.
- The paper makes careful comparisons. The experiments compare private and shared records, keeping and clearing the history.
- The paper combines theory and practice well.
Critical weaknesses
- A new feature direction need not be a new capability. Appendix G shows that dead code can earn extra coverage, so the score still depends on a representation that rejects meaningless variation.
- The theory pays developers for score gains, not leaderboard rank. Real developers may value beating a rival more than increasing their score; that setting falls outside the stated guarantees.
- More correct-response modes in the training experiment do not increase held-out task coverage. The reward can encourage new ways to solve known tasks without teaching the population to solve new ones.
- The overall structure of the paper could be further improved. For example, moving some key experimental results from the appendix to the main text would help readers better understand the paper's core contributions at an earlier stage.
Questions for authors
Please respond to the concerns raised in Critical Weaknesses.
Response to Reviewer 8ewN
In App. G we pad the solutions to 164 HumanEval problems with unreachable code. A program together with its four padded versions scores 10% of the coverage range between five copies of one program (1.79 nats) and five orthogonal directions (3.47 nats), while five distinct verified solutions score 48% and beat the padded set on every problem. Deleting constant-false branches and empty-range loops before we compute the signature returns each of these padded programs to its base (App. G). For this response we also built a signature from the bytecode each program executes on its tests, and added pads that run but change no output. Under this signature padded programs score 3.5% of the range when the pads never run and 5.9% when they run, compared with 37.7% for distinct solutions.
We added Proposition 4 (App. B.7 and §4) for rank rewards on orthogonal action sets with any number d≥2 of modes and any non-increasing rank schedule, such as a fixed number of leaderboard slots. With equal qualities and developers ranked by MOSAIC score, every path of strict improvements made one developer at a time is finite and a pure Nash profile exists. If in addition N≥3 and the schedule is not constant, every pure Nash profile occupies min(N,d) modes. When d≥N these profiles repeat no mode and attain optimal coverage. Suppose instead that the mode qualities are strictly ordered, N≥3 and the schedule is not constant. Ranking by MOSAIC score then rules out full collapse onto one mode once w2>w1/(1+(N−2)w1) at unit coverage scale, whereas ranking by quality alone makes full collapse the unique equilibrium.
In the training experiment the selector's features count arithmetic operators, adjacent operator pairs and chain length. The reward therefore targets how a model computes the answers to problems it already solves (§6). Training on responses selected against each developer's own persistent reference changes those computations in later models. For every Qwen2.5 size the correct answers cover more operation modes than under uniform selection, and on all five models in Fig. 3 they contain more executed intermediate-value patterns per problem. These executed values are not among the selector's features.
We made a new revision following your request, please see the new draft.