- Rating
- 2(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies whether language models can discover useful facts about an unfamiliar mathematical theory before seeing specific questions. It generates theories and exams with automatically checked answers. Models explore each theory freely, then compress their findings into short notes. The authors compare exam performance with and without those notes.
Critical strengths
- Studying what models discover before seeing a problem goes beyond ordinary question-answering benchmarks.
Critical weaknesses
-
The evaluation coverage and selection need clarification. Structural filtering reduces 1,953 candidate theories to 373, but the final exam experiment uses only 21, and the diversity experiment uses five. The full selection process is not clear to me. The small number is not automatically a statistical flaw, but it limits confidence in broader conclusions, especially because difficulty screening uses one model's performance.
-
It is unclear whether the improvement comes from open-ended exploration or simply giving the model more time to prepare. A comparison with equally long, guided preparation would help. Also, asking different checker questions does not necessarily mean discovering useful ideas. The paper should show whether those different questions lead to useful findings.
-
I also could not find a clear release plan or a complete summary of question counts. I also found the paper hard to follow. An early worked example would help explain the setup and what the measurements establish.
Questions for authors
-
How were the final 21 theories selected from the 373 filtered candidates? Please report exclusions at each stage, total question counts, and counts by theory and question type.
-
How does open-ended exploration compare with equally costly directed preparation, such as deriving consequences of the axioms and summarizing them?
-
How do you distinguish meaningful idea diversity from variation in query wording or equivalent equations? Can you show whether less common queries lead to useful findings?
-
Will the theories, exams, generation code, and evaluation scripts be released as a reusable benchmark?
7 comments
(2/4)
2 - "It is unclear whether the improvement comes from open-ended exploration or simply giving the model more time to prepare. A comparison with equally long, guided preparation would help."
This is an important point! We will try to address your concern on two levels: first, by providing the comparisons that you and the other reviewers helpfully suggested, and second, from a high-level research motivation perspective.
Comparisons
For better intuition of the results, we added two document comparisons with matched preparation budgets: "cheatsheet" which is prompted with the exam task and question formats and "derivation", as you suggested, which is prompted to derive comparisons. The prompts are given in Appendix A.11. We ran this on a subset of 15 of the theories for budget reasons and supply the completion curves in Figure S3 (Appendix A.11), which would be moved to Fig. 2b for the camera-ready. The results are summarized in the table below.
Table 5. Matched 15-theory preparation comparison: mean completion %.
| Model | Token threshold | Baseline | Explore | Derive | Cheatsheet |
|---|---|---|---|---|---|
| GPT | 2k | 45.9 | 68.1 | 69.1 | 66.4 |
| GPT | 5k | 70.6 | 80.4 | 81.7 | 78.5 |
| GPT | 10k | 91.0 | 89.5 | 89.5 | 87.0 |
| Claude | 2k | 19.3 | 44.6 | 45.7 | 38.3 |
| Claude | 5k | 20.6 | 45.7 | 46.6 | 39.6 |
| Claude | 10k | 20.8 | 46.1 | 46.9 | 40.0 |
| GLM | 2k | 22.8 | 42.5 | 42.2 | 40.7 |
| GLM | 5k | 35.1 | 54.7 | 53.3 | 51.9 |
| GLM | 10k | 46.2 | 62.7 | 60.1 | 60.0 |
The explore, derive, and cheatsheet CI's mostly overlap, with mean cheatsheet performance notably underperforming relative to explore and derive.
Research motivation
These comparisons also help clarify what our evaluation is intended to measure.
The capability we wanted to study in models was their ability to discover "new math", or more generally, "new ideas", without being given any direction other than to explore something. Although this capability has not been characterized because research efforts have focused on problem-solving, it is arguably the next frontier of research AI's.
Therefore, the goal of our exam experiments is not to show that an open-ended exploration document is the best-performing preparation out of all possible document preparations. Instead, our goal is to isolate and characterize different components of exploration capability using quantifiable methods.
We believe our exam results demonstrate the usefulness of the method even with just the naive baseline. It shows that (1) open-ended exploration consistently discovers useful information for exam performance; (2) tokens of general-purpose exploration documents are more helpful than additional problem-solving tokens, which can be seen by the greater initial steepness of the exploration curves; and (3) different model explorations have different effects on exam performance, based on both curve height and shape.
(3) crucially means that we can identify markers of strengths and failure modes of model exploration with our quantified method, which can then be used for targeted qualitative analysis. These model differences are shown more clearly in our updated Fig. 2, which shows cross-model exploration document transfer--how well models do on exams using exploration docs from other models.
We see that Claude explore documents consistently underperform compared to the GPT and GLM. Briefly, our follow-up initial qualitative analysis suggests that Claude tends to overly focus on specific facts or models rather than bringing them together to general conclusions or laws. In one example, it constructs a family containing noncommutative models but still lists commutativity as an open conjecture. This discussion would be expanded in a final paper to demonstrate how our method reveals differences in exploration characteristics of different models.
(3/4)
2 - "Also, asking different checker questions does not necessarily mean discovering useful ideas. The paper should show whether those different questions lead to useful findings."
One aspect of this is canonicalization of queries. We implement this but missed documenting this in the manuscript. We regularize by whitespace formatting and for equivalent statements by variable renaming, equality reversal, and order of assumptions. Countermodel queries with the same statement are treated as the same even with different countermodel tables. We will definitely include this in a final writeup.
The other aspect of this is the quality/usefulness of the queries, particularly the less studied ones proposed by various models.
First of all, we can establish that some of GLM's rare queries were mathematically-sensible and motivated, with clear interpretations. Two such questions were: "Does squaring preserve multiplication? " - Refuted and "Can four-factor expressions be rewritten into paired products? - Proved.
Even without deeper investigations of the immediate usefulness of these long-tail queries, the capability of generating ideas without getting stuck in a small distribution already gets at what we want to measure. In human society, creative ideation generates many non-useful products and maybe one or two good ideas. In an AI setting, we could imagine wanting to develop a collective of agents that are inherently capable of having different ideas in order to innovate together.
There is much more that can be done to study the intersection of diversity and utility with the data generated by our method: by doing further qualitative analysis, reading, and comparing checker queries to what a model explores without the checker. However, this follow-up work will take a significant amount of time for expert review of equational theory examples. As such, we believe this work is better suited for a follow-up project rather than this paper, which aims to establish the method.
3 - "I also could not find a clear release plan or a complete summary of question counts."
Good catch! Releasing code and artifacts for transparency and reusability was always part of our plan once we had time to neatly package things. We will release the whole codebase, including code for generating the set of theories, filtering theories, generating the exams, sending API queries, plotting results, and scripts for reproducing the results. We also plan to release the exploration documents and exam responses to be available for further qualitative analysis.
Table 6. Unique exam questions in the original 21-theory evaluation.
| Theory | Basic | Derived structure | Assumptions | Axiom ablation | Equational | Total |
|---|---|---|---|---|---|---|
| pair-0038 | 2 | 10 | 8 | 10 | 4 | 34 |
| pair-0218 | 2 | 10 | 10 | 8 | 10 | 40 |
| pair-0584 | 4 | 10 | 10 | 10 | 10 | 44 |
| pair-0610 | 2 | 10 | 10 | 10 | 10 | 42 |
| pair-0676 | 2 | 10 | 8 | 10 | 10 | 40 |
| pair-0691 | 4 | 10 | 10 | 10 | 10 | 44 |
| pair-0775 | 2 | 10 | 10 | 10 | 10 | 42 |
| pair-0845 | 2 | 10 | 8 | 8 | 10 | 38 |
| pair-0848 | 2 | 10 | 10 | 8 | 10 | 40 |
| pair-0849 | 2 | 10 | 10 | 8 | 10 | 40 |
| pair-1020 | 7 | 10 | 10 | 10 | 9 | 46 |
| pair-1030 | 7 | 10 | 10 | 8 | 9 | 44 |
| pair-1036 | 2 | 10 | 10 | 10 | 10 | 42 |
| pair-1104 | 2 | 10 | 10 | 10 | 10 | 42 |
| pair-1147 | 4 | 10 | 8 | 10 | 10 | 42 |
| pair-1181 | 7 | 10 | 10 | 10 | 9 | 46 |
| pair-1199 | 7 | 10 | 8 | 10 | 9 | 44 |
| pair-1201 | 7 | 10 | 10 | 10 | 9 | 46 |
| pair-1227 | 2 | 10 | 10 | 8 | 10 | 40 |
| pair-1458 | 6 | 10 | 10 | 10 | 10 | 46 |
| pair-1461 | 2 | 10 | 4 | 10 | 6 | 32 |
| Total | 77 | 210 | 194 | 198 | 195 | 874 |
(Note. All question types are required to have balanced T/F responses except for the Basic type because we can't control the distribution of basic properties in randomly-sampled theories.)
(1/4)
Thank you for reading our paper and for your clear, constructive criticism! We hope that the following response will address your concerns.
Weaknesses
We raised the number of theories in our evaluation set to 30 for the exam utility experiment and 15 for the diversity analysis and found that our results are very stable. We are constrained to expand beyond this sample by budget. We share the details, discussion, and updated results below as well as in updates to Fig. 2 and 4C. We also want to highlight that our paper introduces a general method for creating theories and exams for model evaluation, so it can be extended to arbitrary theory instantiations beyond the 30 documented in our paper. We share a pilot study of an expanded signature exam evaluation in Appendix A.6 to demonstrate this extensibility and will build support for it into the final code release.
We summarize the filtration steps from 373 => 21 in a table below. API costs were 544 USD for the experiments in our manuscript and an additional 980 USD for follow-up experiments for rebuttal.
Table 1. Original exam selection.
Exam generation could be rejected for missing a question type subject to balanced T/F questions and light difficulty filters (e.g. must be more than one rewrite away). We will describe these details in the final version of the paper.
We summarize the updated exam results in a table and an updated Fig. 2, and note that adding 9 theories changes the trends very little.
Table 2. Completion percentages: original 21 → expanded 30 theories (means).
For diversity experiments, we randomly selected 5 theories from the first 100/373 theories that we screened. We have now added 10 theories from the selected theory-exam bank in order to compare analyses across metrics.
Table 3. Rank-frequency concentration: original 5 → expanded 15 theories (mean share of calls, %).
Here, we see the observation that GLM has the longer tail out of the other two models in terms of canonicalized checker queries still holds.
We wanted to run exams that would show a spread in model performance across question budgets of up to 10k tokens, which would not be possible if models could solve exams too quickly. We based this performance on only GPT because it was a rough screening metric that we did not want to consume much budget.
We ran follow-up exam experiments on two theories that were filtered out based on exam difficulty and found that the results were not sensitive to filtering. For pair-0412, with 23/38 correct (60.5%) in screening, and pair-0089, whose exam screening showed a minimum 21/34 (61.8%) and max possible accuracy 100%, the completion curves look the same as on our selected theories, with a slight shift up and to the left showing improved performance.
Figure S4 (Appendix A.12) shows the completion curves for these two theories; compare to Figure 3A. Table 4 summarizes performance at 2k, 5k, and 10k tokens.
Table 4. Completion percentages on theories excluded by difficulty screening (means).