- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents the construction and usage of a benchmark for LLMs, that relies on the construction, instantiation and rating of analogies in several domains. The models to be ranked each generates a number of analogies as templates with several slots, a table with fillers for each slot from several domains, and instantiations -- as generated texts -- in each domain for the corresponding templates. Each model rates the data generated by the other models along given axes, without a reference. A reference model is chosen (whose ratings are highest according to the other models in the first evaluation round), and then each model is evaluated against the reference model. The ratings are used to score the models as both data generators and as evaluators. The ranking produced with this method is compared to a rating on a human-validated benchmark, consisting of multiple choice questions in several scientific domains at the graduate level. The Pearson correlation obtained is high. The authors emphasize that the benchmark is produced without the necessity of rhuman intervention and evaluation, and can be reconstructed as it (if it) becomes obsolete.
Critical strengths
Benchmarking the ever evolving models is an issue, particularly as existing benchmarks are incorporated in the training data of subsequent model generations. The paper proposes a benchmark that the authors describe as easy to build anew, as it does not require human feedback.
Critical weaknesses
The paper is built on several assumptions that are not obviously true:
-
agreement among models is indicative of their goodness. This can also be the consequence of shared biases, which cannot be dismissed considering that all these models have very similar architectures, share huge amounts of training data, and have very similar training objectives.
-
the left and right singular vectors of the factuality scores matrix have specific interpretation that reflect a model's competence. Why is this a reasonable assumption?
One good feature of the benchmark, according to the authors, is that this is free of human intervention, and as such it can be rebuilt on demand. While it showed very high correlation with the ranking obtained on a human-validated benchmark, it would still be useful to know how good are the generated templates, as well as a bit more benchmark analysis: how similar are the templates generated by the different models (within and across model families)? how good are the model evaluations?
Another good feature is considered to be the lack of contamination, as the templates, tables and instantiations are generated on the run. Given the high amounts and varied topics and forms of the data used for training, this does not seem like a justifiable assumption.
In section 6, lines 410-412: "Our hypothesis is that ... and GPQA runs on it." -- this was not proven, although the discussion of the comparison (and the way Figure 3 is drawn) uses this as an assumption. It is not clear why this is a reasonable assumption, answering questions (and multiple choice questions in particular) does not necessarily rely on an underlying model of the field.
Questions for authors
Added after discussion period with authors finished, in reply to their last comment:
This is all extremely interesting, and definitely should be added to the paper, but together with a manual validation (on a sample, of course). Without an external reference point, these numbers do not mean very much. I understand that you want to push your paper, but all the work done during the reviewing period is rushed, and patchy, and you should take your time and put it all together, and add some external validation of these numbers and analyses. The scores stand.
Please justify the two assumptions mentioned in the "Critical Weaknesses" section: the assumption of goodness, and the interpretation of the SVD decomposition.
Please clarify Figure 3. This seems to show that there is shared processing between generating templates and their instantiations and answering the multiple choice questions, but this is not obvious. Why should we assume that in the GPQA task the archetype is given. Answering questions doesn't necessarily assume an internal organization of the knowledge of the domain, just the capacity to retrieve the right answer (or to plug in each candidate answer in a sentence and choose the best according to some criterion).
Some analysis of the generated data would be very helpful and interesting, in particular: how different are the templates and instantiations generated by each model? Do they generate the same data on different runs? Do the models choose the domains for each template? Are the generated templates good (some human evaluation)? Have the models generated something particularly novel or surprising?
3 comments
A new paper
Thank you for the clarifications. The authors have done much work, but the methodology is radically changed, the evaluation is changed (possibly ... it is not clear whether section 3.4 from the original paper is still used), the new draft is quite different from what was submitted.
The new draft also carries rather strong assumptions about the level of intelligence needed to play this game (section 2).
The issue of bias remains however -- acknowledging it does not make it acceptable, it's an important issue that can invalidate the premise of the work here. It may be at least partially addressed by an analysis of the portfolios that the models create: the more different they are, the better case one can make that the underlying models have different biases. This can be done automatically, using some (several would be best) text similarity measures and lexical overlap, particularly since the models conveniently generate templates, which could be more objectively compared, so to speak, as they succinctly capture the analogy and its fillers in various domains (rather than lengthier and more freeform text). I do not mean for this to replace some human checking, as the authors say is in the works. But this automatic analysis would help provide a better view of what and how varied the output of the models is.
For these reasons, my rating remains the same.
Thank you. On "a new paper", here is what is unchanged and what changed.
Unchanged
- The Metanym Game itself: everything the models do when they play, and everything the judges do when they rate.
- It is a benchmark without ground truth.
- It rises with the models it measures.
Changed
- How the ratings become a score: now the plain average of the factual ratings. Section 3.4 is no longer used.
- More models and more benchmarks.
The analysis you suggested. We ran it. It also answers a question from your first review: how similar are the templates within and across model families, and across runs?
We compared all 581 rated context templates (39 models, 15 companies) by wording (TF-IDF cosine) and by meaning (cosine of sentence embeddings, all-mpnet-base-v2). 1 means identical; lower means less alike.
1. Different models write different templates.
| Two templates from | Wording: mean (SD) | Meaning: mean (SD) |
|---|---|---|
| The same model, in another run | 0.07 (0.18) | 0.47 (0.19) |
| The same portfolio | 0.03 (0.07) | 0.42 (0.14) |
| Two models of one company | 0.01 (0.02) | 0.38 (0.13) |
| Two companies | 0.01 (0.01) | 0.38 (0.12) |
By the rules of the game, the templates of one portfolio must represent different archetypes. Templates from different models are as far apart in meaning as those (0.38 against 0.42).
2. Families show a little, in the wording. We grouped the 39 models by hierarchical clustering (average linkage) on the similarity of their templates, cut into 15 groups, as many as there are companies. Then we compared the groups with the companies (adjusted Rand index: 1 is a full match, 0 is chance).
| Grouping by | Match with the companies | What the groups show |
|---|---|---|
| Wording | 0.19 | The five newest GPT models form one group, the two Claude Opus models another |
| Meaning | −0.04 | Most models fall into one large mixed group |
3. More models give more variety. When we draw 10 templates at random, how many different ones do we effectively get, by meaning? 1 if they are all the same, 10 if they have nothing in common (Vendi Score; Friedman & Dieng, 2023).
| Where the 10 templates come from | Effectively different templates |
|---|---|
| All from the same model | 5.0 |
| All from models of the same company | 5.6 |
| From any of the 39 models | 6.2 |
What this means. By your argument, the models have different biases. Models of one company are somewhat closer to each other, which is a reason to spread the council over several companies. The analysis does not rule out blind spots that all models share. A human check of a sample remains to be done.
Your other points
- Domains. The models choose them. Portfolios of different models share about 1% of their domain names.
- Section 2. We will reword the sentence on types of intelligence to say which abilities the game draws on. The score does not rest on it.
We will add the analysis to the paper.
Thank you for the review. Your questions about the two assumptions led us to the plain average and to the comparison with many public benchmarks. Point by point: