- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The Token Games (TTG) evaluates language models through pairwise duels in which they alternate between proposing Python puzzles and solving their opponents’ puzzles. Solutions are verified by execution, and duel outcomes yield Elo ratings. Across 918 duels involving 18 models, the study reports positive correlations with established benchmarks and reproducible rankings. Further analyses examine puzzle difficulty, proposer behavior, and the relationship between problem posing and solving.
Critical strengths
- The framework automates both challenge generation and solution verification. Evaluating problem posing alongside solving offers a useful alternative to fixed, human-authored question sets. The complete round-robin evaluation, with three repetitions of each ordered pairing, provides substantial evidence beyond a proof of concept (§§3–4).
- The validation extends beyond benchmark correlations. Repeated tournaments achieve pairwise Spearman rank correlations of 0.95. Evaluation with a common set of solvers and re-solving self-proposed puzzles without the original context provide complementary evidence about puzzle difficulty and proposer behavior (§§4.3–4.6).
Critical weaknesses
- The statistical significance claim conflicts with the reported results. Section 4.1 states that all 15 correlations survive Bonferroni correction, yet Table 8 reports adjusted p-values of 0.0981 for match Elo–HLE and 0.0567 for match Elo–SWE-Bench Pro. Table 2 also uses role-specific win rates, whereas Table 8 uses role-specific Elo ratings, so the latter does not directly establish the robustness of the former’s correlations. These discrepancies require correction but do not negate the positive associations.
- The effect of formatting-only failures on rankings remains unquantified. Appendix D.12 shows a solver deriving the intended solution but losing because of Markdown backticks. Format compliance is a legitimate evaluation target, but such failures are distinct from reasoning errors. Their frequency and model-specific impact should be quantified to clarify what TTG rankings measure.
Questions for authors
- Statistical reporting: Please reconcile §4.1 with Tables 2 and 8, clarify the metrics used, and report adjusted p-values for a consistent set of correlations. Which conclusions hold under Bonferroni correction, and which under Benjamini–Hochberg correction? The existing scores should suffice for this analysis.
- Formatting sensitivity: Using saved solver outputs, could the authors quantify failures recoverable by removing only unambiguous formatting wrappers? Please report recovered solutions by model and any changes in offline duel scores or Elo rankings. Retain the original protocol as the primary result and present normalization as a sensitivity analysis; no new tournament is requested.
2 comments
Thank you to the authors for their response. Most of my concerns have been addressed.
Thank you for the very careful reading. Both requests were feasible with existing data, and we have done both.
Statistical reporting. You are right, and we thank you for catching this. The claim in §4.1 that all 15 correlations survive Bonferroni correction is incorrect; this was an error in our text. Using a consistent family of 15 tests per table:
All failures under Bonferroni involve HLE or SWE-Bench Pro. We will report raw, Bonferroni-adjusted, and BH-adjusted p-values for Table 2, state that the leave-one-out analysis uses unadjusted p-values, and revise §4.1 to claim only what these results support. As you note, the positive associations remain.
Formatting sensitivity. Following your protocol, we re-verified the saved solver outputs after removing only unambiguous formatting wrappers: enclosing backticks, a stray trailing backtick (from wrapping the whole
SOLUTION: xline in backticks), and Markdown bold markers. A wrapper is removed only if the original answer is not valid Python and the stripped answer is, so no valid answer is ever changed.We keep the original protocol as the primary result and report normalization as a sensitivity analysis in the appendix.
We’ll update the paper shortly.