acceptodds
Under review as a conference paper at ICLR 2027

Mathematica Incognita: Characterizing Open-Ended Mathematical Exploration in Language Models

Abstract

Mathematics has long depended on problem-solving as a guidepost for structuring and measuring progress—a methodology that is strongly reflected in the development of mathematical AIs. However, open-ended exploration, as well as the related activities of theory building and asking new research questions, remain under-studied capabilities of language models. In order to evaluate open-ended exploration of language models, we develop a suite of problem-solving tasks within equational theories—sets of equational axioms that define an algebraic structure. This area of math is particularly useful for model evaluation due to the ease of instantiating diverse, self-contained theories that are rarely described in literature, as well as the possibility of generating problems within these theories which remain difficult for frontier reasoning models without external tool use. Our experimental tasks evaluate two desiderata of successful explorations: future utility and idea diversity. As a proxy for future utility, we measure the improvement in problem-solving performance when models are provided with an exploration document, and observe notable but uneven benefits from exploration across models and theories. To measure idea diversity, we implement a limited automated reasoning query interface for theory explorations that enables unambiguous comparison of research ideas across model runs, and observe a common core and long tail of ideas across models. As language models gain a greater role in solving research-grade math problems, studying their behavior during open-ended exploration will become increasingly important for comprehensive understanding of their research capabilities.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
automated mathematics, AI for math, mathematical reasoning, open-ended exploration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KCcnOpenReview ↗
Rating
2(+1)
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies whether language models can discover useful facts about an unfamiliar mathematical theory before seeing specific questions. It generates theories and exams with automatically checked answers. Models explore each theory freely, then compress their findings into short notes. The authors compare exam performance with and without those notes.

Critical strengths
  1. Studying what models discover before seeing a problem goes beyond ordinary question-answering benchmarks.
Critical weaknesses
  1. The evaluation coverage and selection need clarification. Structural filtering reduces 1,953 candidate theories to 373, but the final exam experiment uses only 21, and the diversity experiment uses five. The full selection process is not clear to me. The small number is not automatically a statistical flaw, but it limits confidence in broader conclusions, especially because difficulty screening uses one model's performance.

  2. It is unclear whether the improvement comes from open-ended exploration or simply giving the model more time to prepare. A comparison with equally long, guided preparation would help. Also, asking different checker questions does not necessarily mean discovering useful ideas. The paper should show whether those different questions lead to useful findings.

  3. I also could not find a clear release plan or a complete summary of question counts. I also found the paper hard to follow. An early worked example would help explain the setup and what the measurements establish.

Questions for authors
  1. How were the final 21 theories selected from the 373 filtered candidates? Please report exclusions at each stage, total question counts, and counts by theory and question type.

  2. How does open-ended exploration compare with equally costly directed preparation, such as deriving consequences of the axioms and summarizing them?

  3. How do you distinguish meaningful idea diversity from variation in query wording or equivalent equations? Can you show whether less common queries lead to useful findings?

  4. Will the theories, exams, generation code, and evaluation scripts be released as a reusable benchmark?

7 comments

(1/4)

Thank you for reading our paper and for your clear, constructive criticism! We hope that the following response will address your concerns.

Weaknesses

1 - "The evaluation coverage and selection need clarification... the final exam experiment uses only 21, and the diversity experiment uses 5... The full selection process is not clear to me."

We raised the number of theories in our evaluation set to 30 for the exam utility experiment and 15 for the diversity analysis and found that our results are very stable. We are constrained to expand beyond this sample by budget. We share the details, discussion, and updated results below as well as in updates to Fig. 2 and 4C. We also want to highlight that our paper introduces a general method for creating theories and exams for model evaluation, so it can be extended to arbitrary theory instantiations beyond the 30 documented in our paper. We share a pilot study of an expanded signature exam evaluation in Appendix A.6 to demonstrate this extensibility and will build support for it into the final code release.

We summarize the filtration steps from 373 => 21 in a table below. API costs were 544 USD for the experiments in our manuscript and an additional 980 USD for follow-up experiments for rebuttal.

Table 1. Original exam selection.

StageExcluded / not runRemaining
Structurally filtered catalogue—373
First 100 in random seeded shuffle273 not attempted100
Exam generation rejected3961
Fewer than 32 questions259
Screening score above 60%3227
First 21 accepted theories evaluated6 deferred21

Exam generation could be rejected for missing a question type subject to balanced T/F questions and light difficulty filters (e.g. must be more than one rewrite away). We will describe these details in the final version of the paper.

We summarize the updated exam results in a table and an updated Fig. 2, and note that adding 9 theories changes the trends very little.

Table 2. Completion percentages: original 21 → expanded 30 theories (means).

ModelToken thresholdBaseline: original → expandedExplore: original → expanded
GPT2k48.5 → 47.073.7 → 71.1
GPT5k76.1 → 74.184.9 → 83.0
GPT10k92.9 → 92.192.2 → 91.2
Claude2k20.0 → 21.945.0 → 44.7
Claude5k21.1 → 23.046.1 → 45.8
Claude10k21.4 → 23.246.4 → 46.0
GLM2k23.8 → 24.147.4 → 44.4
GLM5k36.7 → 36.160.0 → 56.5
GLM10k48.0 → 47.467.7 → 65.4

For diversity experiments, we randomly selected 5 theories from the first 100/373 theories that we screened. We have now added 10 theories from the selected theory-exam bank in order to compare analyses across metrics.

Table 3. Rank-frequency concentration: original 5 → expanded 15 theories (mean share of calls, %).

ModelTop 1 statementTop 10 statementsTop 20 statements
GPT10.6 → 9.262.3 → 54.083.8 → 75.1
Claude10.3 → 8.153.6 → 46.072.8 → 65.0
GLM9.3 → 7.849.3 → 43.765.4 → 60.5

Here, we see the observation that GLM has the longer tail out of the other two models in terms of canonicalized checker queries still holds.

1 - "...difficulty screening uses one model's performance."

We wanted to run exams that would show a spread in model performance across question budgets of up to 10k tokens, which would not be possible if models could solve exams too quickly. We based this performance on only GPT because it was a rough screening metric that we did not want to consume much budget.

We ran follow-up exam experiments on two theories that were filtered out based on exam difficulty and found that the results were not sensitive to filtering. For pair-0412, with 23/38 correct (60.5%) in screening, and pair-0089, whose exam screening showed a minimum 21/34 (61.8%) and max possible accuracy 100%, the completion curves look the same as on our selected theories, with a slight shift up and to the left showing improved performance.

Figure S4 (Appendix A.12) shows the completion curves for these two theories; compare to Figure 3A. Table 4 summarizes performance at 2k, 5k, and 10k tokens.

Table 4. Completion percentages on theories excluded by difficulty screening (means).

ModelToken thresholdpair-0089: baselinepair-0089: explorepair-0412: baselinepair-0412: explore
GPT2k92.294.161.481.6
GPT5k100.099.086.094.7
GPT10k100.0100.098.299.1
Claude2k21.636.37.037.7
Claude5k23.539.28.838.6
Claude10k23.539.28.838.6
GLM2k38.256.933.352.6
GLM5k68.674.544.764.9
GLM10k84.381.459.670.2

(2/4)

2 - "It is unclear whether the improvement comes from open-ended exploration or simply giving the model more time to prepare. A comparison with equally long, guided preparation would help."

This is an important point! We will try to address your concern on two levels: first, by providing the comparisons that you and the other reviewers helpfully suggested, and second, from a high-level research motivation perspective.

Comparisons

For better intuition of the results, we added two document comparisons with matched preparation budgets: "cheatsheet" which is prompted with the exam task and question formats and "derivation", as you suggested, which is prompted to derive comparisons. The prompts are given in Appendix A.11. We ran this on a subset of 15 of the theories for budget reasons and supply the completion curves in Figure S3 (Appendix A.11), which would be moved to Fig. 2b for the camera-ready. The results are summarized in the table below.

Table 5. Matched 15-theory preparation comparison: mean completion %.

ModelToken thresholdBaselineExploreDeriveCheatsheet
GPT2k45.968.169.166.4
GPT5k70.680.481.778.5
GPT10k91.089.589.587.0
Claude2k19.344.645.738.3
Claude5k20.645.746.639.6
Claude10k20.846.146.940.0
GLM2k22.842.542.240.7
GLM5k35.154.753.351.9
GLM10k46.262.760.160.0

The explore, derive, and cheatsheet CI's mostly overlap, with mean cheatsheet performance notably underperforming relative to explore and derive.

Research motivation

These comparisons also help clarify what our evaluation is intended to measure.

The capability we wanted to study in models was their ability to discover "new math", or more generally, "new ideas", without being given any direction other than to explore something. Although this capability has not been characterized because research efforts have focused on problem-solving, it is arguably the next frontier of research AI's.

Therefore, the goal of our exam experiments is not to show that an open-ended exploration document is the best-performing preparation out of all possible document preparations. Instead, our goal is to isolate and characterize different components of exploration capability using quantifiable methods.

We believe our exam results demonstrate the usefulness of the method even with just the naive baseline. It shows that (1) open-ended exploration consistently discovers useful information for exam performance; (2) tokens of general-purpose exploration documents are more helpful than additional problem-solving tokens, which can be seen by the greater initial steepness of the exploration curves; and (3) different model explorations have different effects on exam performance, based on both curve height and shape.

(3) crucially means that we can identify markers of strengths and failure modes of model exploration with our quantified method, which can then be used for targeted qualitative analysis. These model differences are shown more clearly in our updated Fig. 2, which shows cross-model exploration document transfer--how well models do on exams using exploration docs from other models.

We see that Claude explore documents consistently underperform compared to the GPT and GLM. Briefly, our follow-up initial qualitative analysis suggests that Claude tends to overly focus on specific facts or models rather than bringing them together to general conclusions or laws. In one example, it constructs a family containing noncommutative models but still lists commutativity as an open conjecture. This discussion would be expanded in a final paper to demonstrate how our method reveals differences in exploration characteristics of different models.

(3/4)

2 - "Also, asking different checker questions does not necessarily mean discovering useful ideas. The paper should show whether those different questions lead to useful findings."

One aspect of this is canonicalization of queries. We implement this but missed documenting this in the manuscript. We regularize by whitespace formatting and for equivalent statements by variable renaming, equality reversal, and order of assumptions. Countermodel queries with the same statement are treated as the same even with different countermodel tables. We will definitely include this in a final writeup.

The other aspect of this is the quality/usefulness of the queries, particularly the less studied ones proposed by various models.

First of all, we can establish that some of GLM's rare queries were mathematically-sensible and motivated, with clear interpretations. Two such questions were: "Does squaring preserve multiplication? (xx)(yy)=(xy)(xy)(xx)(yy)=(xy)(xy)" - Refuted and "Can four-factor expressions be rewritten into paired products? ((xy)z)w=(xw)(yz)((xy)z)w=(xw)(yz) - Proved.

Even without deeper investigations of the immediate usefulness of these long-tail queries, the capability of generating ideas without getting stuck in a small distribution already gets at what we want to measure. In human society, creative ideation generates many non-useful products and maybe one or two good ideas. In an AI setting, we could imagine wanting to develop a collective of agents that are inherently capable of having different ideas in order to innovate together.

There is much more that can be done to study the intersection of diversity and utility with the data generated by our method: by doing further qualitative analysis, reading, and comparing checker queries to what a model explores without the checker. However, this follow-up work will take a significant amount of time for expert review of equational theory examples. As such, we believe this work is better suited for a follow-up project rather than this paper, which aims to establish the method.

3 - "I also could not find a clear release plan or a complete summary of question counts."

Good catch! Releasing code and artifacts for transparency and reusability was always part of our plan once we had time to neatly package things. We will release the whole codebase, including code for generating the set of theories, filtering theories, generating the exams, sending API queries, plotting results, and scripts for reproducing the results. We also plan to release the exploration documents and exam responses to be available for further qualitative analysis.

Table 6. Unique exam questions in the original 21-theory evaluation.

TheoryBasicDerived structureAssumptionsAxiom ablationEquationalTotal
pair-0038210810434
pair-02182101081040
pair-058441010101044
pair-061021010101042
pair-06762108101040
pair-069141010101044
pair-077521010101042
pair-0845210881038
pair-08482101081040
pair-08492101081040
pair-10207101010946
pair-1030710108944
pair-103621010101042
pair-110421010101042
pair-11474108101042
pair-11817101010946
pair-1199710810944
pair-12017101010946
pair-12272101081040
pair-145861010101046
pair-1461210410632
Total77210194198195874

(Note. All question types are required to have balanced T/F responses except for the Basic type because we can't control the distribution of basic properties in randomly-sampled theories.)

Reviewer_xCCfOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Mathematica Incognita, a framework for evaluating open-ended mathematical exploration inLLMs. It generates unfamiliar equational theories and verified mathematical exams to evaluate two aspects: future utility and idea diversity. Experiments on three frontier models show that exploration generally improves problem-solving performance, although gains vary across theories and models. The diversity analysis reveals a common core of frequently explored ideas and a long tail of less common queries.

Critical strengths
  1. Addresses an interesting and relatively underexplored aspect of mathematical reasoning beyond conventional problem-solving benchmarks.

  2. Automated generation of equational theories and verified problems provides a useful evaluation framework.

  3. The experiments provide interesting insights into the utility and diversity of mathematical exploration.

Critical weaknesses
  1. The exploration-guided condition receives additional information, while the baseline does not. A control using a non-exploratory summary would better isolate the contribution of open-ended exploration.

  2. Idea diversity is measured through distinct checker queries, which may not reflect mathematically distinct or meaningful ideas.

  3. The evaluation covers only 21 theories for utility and five for diversity.

  4. The related work could better discuss recent benchmarks of open-ended scientific exploration [1, 2].

[1] Carlsson et al. (2026). SEDRAS: Symbolically Evaluated Deep Research And Science. ICML 2026.

[2] Xu et al. (2026). EarthSE: A Benchmark Evaluating Earth Scientific Exploration Capability for Large Language Models. ICLR 2026.

Questions for authors
  1. How much improvement comes specifically from exploration rather than additional information or reasoning?

  2. How are mathematically equivalent queries handled in the diversity analysis?

  3. How sensitive are the results to theory selection and difficulty filtering?

3 comments

(1/2) Thank you for your review of our paper and your thoughtful comments!

Weaknesses

1 - A control using a non-exploratory summary would better isolate the contribution of open-ended exploration.

This is an important point! We will try to address your concern on two levels: first, by providing the comparisons that you and the other reviewers helpfully suggested, and second, from a high-level research motivation perspective.

Comparisons

We added two document comparisons with matched preparation budgets for better intuition: "cheatsheet" which is prompted with the exam task and question formats and "derivation", which is prompted to derive comparisons. The prompts are given in Appendix A.11. We ran this on a subset of 15 of the theories for budget reasons and supply the completion curves in Figure S3 (Appendix A.11), which would be moved to Fig. 2b for the final paper.

Table 1. Matched 15-theory preparation comparison: mean completion %.

ModelToken thresholdBaselineExploreDeriveCheatsheet
GPT 5.6 Sol2,00045.968.169.166.4
GPT 5.6 Sol5,00070.680.481.778.5
GPT 5.6 Sol10,00091.089.589.587.0
Claude Sonnet 52,00019.344.645.738.3
Claude Sonnet 55,00020.645.746.639.6
Claude Sonnet 510,00020.846.146.940.0
GLM 5.32,00022.842.542.240.7
GLM 5.35,00035.154.753.351.9
GLM 5.310,00046.262.760.160.0

The explore, derive, and cheatsheet CI's mostly overlap, with mean cheatsheet performance notably underperforming relative to explore and derive.

Research motivation

These comparisons also help clarify what our evaluation is intended to measure.

The capability we wanted to study in models was their ability to discover "new math", or more generally, "new ideas", without being given any direction other than to explore something.

Therefore, the goal of our exam experiments is not to show that an open-ended exploration document is the best-performing preparation out of all possible document preparations. Instead, our goal is to isolate and characterize different aspects of exploration capability using quantifiable methods. So, we are more interested in how exam performance changes when given an exploration document.

Our results show that (1) open-ended exploration consistently discovers useful information for exam performance; (2) tokens of general-purpose exploration documents are more helpful than additional problem-solving tokens, which can be seen by the greater initial steepness of the exploration curves; and (3) different model explorations have different effects on exam performance, based on both curve height and shape.

(3) crucially means that we can identify markers of strengths and failure modes of model exploration with our method, which can then be used for targeted qualitative analysis. These model differences are shown clearly in our updated Fig. 2, which shows cross-model exploration document transfer. We see that Claude Sonnet 5 consistently underperforms the explore documents of GPT 5.6 Sol and GLM 5.3.

Our follow-up initial qualitative analysis suggests that Claude tends to overly focus on specific facts or models rather than bringing them together to general conclusions or laws. In one example, it constructs a family containing noncommutative models but still lists commutativity as an open conjecture. This discussion would be expanded in a final version of our paper to demonstrate how our method reveals differences in exploration characteristics of different models.

2 - Idea diversity is measured through distinct checker queries, which may not reflect mathematically distinct or meaningful ideas.

One aspect of this is canonicalization of queries. We implement this but missed documenting this in the manuscript. We regularize by whitespace formatting and for equivalent statements by variable renaming, equality reversal, and order of assumptions. Countermodel queries with the same statement are treated as the same even with different countermodel tables. We will include this in a final writeup.

The other aspect of this is the quality/usefulness of the queries.

First of all, we can establish that some of GLM's rare queries were mathematically-sensible and motivated, with clear interpretations. Two such questions were: "Does squaring preserve multiplication? (xx)(yy)=(xy)(xy)(xx)(yy)=(xy)(xy)" - Refuted and "Can four-factor expressions be rewritten into paired products? ((xy)z)w=(xw)(yz)((xy)z)w=(xw)(yz)" - Proved.

(2/2) Even without deeper investigations of the immediate usefulness of these long-tail queries, the capability of generating ideas without getting stuck in a small distribution already gets at what we want to measure. In human society, creative ideation generates many non-useful products and maybe one or two good ideas. In an AI setting, we could imagine wanting to develop a collective of agents that are inherently capable of having different ideas in order to innovate together.

There is much more that can be done to study the intersection of diversity and utility with the data generated by this procedure: by doing further qualitative analysis, reading, and comparing checker queries to what a model explores without the checker. However, we believe this work is better suited for a follow-up project rather than this paper, which aims to establish the method.

3 - The evaluation covers only 21 theories for utility and five for diversity.

We agree that the number of theories in our manuscript is low! To address these concerns, we raised the number of theories in our evaluation set to 30 for the exam utility experiment and 15 for the diversity analysis and found that our results remain very stable. We share the details, discussion, and updated results below as well as in an updates to Fig. 2 and 4C. We also want to highlight that our paper introduces a general method for creating theories and exams for model evaluation, so it can be extended to arbitrary theories beyond the 30 we document. We run a pilot in Appendix A.6 with an expanded signature as an example.

Table 2. Completion percentages: original 21 → expanded 30 theories (means).

ModelToken thresholdBaseline: original → expandedExplore: original → expanded
GPT 5.6 Sol2,00048.5 → 47.073.7 → 71.1
GPT 5.6 Sol5,00076.1 → 74.184.9 → 83.0
GPT 5.6 Sol10,00092.9 → 92.192.2 → 91.2
Claude Sonnet 52,00020.0 → 21.945.0 → 44.7
Claude Sonnet 55,00021.1 → 23.046.1 → 45.8
Claude Sonnet 510,00021.4 → 23.246.4 → 46.0
GLM 5.32,00023.8 → 24.147.4 → 44.4
GLM 5.35,00036.7 → 36.160.0 → 56.5
GLM 5.310,00048.0 → 47.467.7 → 65.4

4 - The related work could better discuss recent benchmarks of open-ended scientific exploration [1, 2].

Thank you for bringing our attention to these papers! Although our paper focuses on AI4math, we agree that the field has exchanged many ideas with scientific open-ended exploration researchers. We added a section of related work in Appendix B.1 and cite the papers that you mention.

Questions

  1. The comparison to derivation and cheatsheet document performance shows that exam performance generally benefits from having a previous reasoning round. However, the point of exploration in this study is not to outperform other reasoning, but rather to characterize the behaviors of models when faced with an open-ended task. Exploration does surprisingly well compared with more exam-aware baselines. We believe that we discovered interesting behaviors and failure modes with our methods.

  2. Mathematically equivalent queries in diversity analysis are canonicalized as described in the discussion of weakness 2!

  3. The difficulty filtering was a rough metric for us to make sure the exams were hard enough to see variance in the performance. We used only GPT-5.6-Sol and not a panel in order to save on API credit costs, but we chose a non-stringent barrier of 60% for this reason.

We ran follow-up exam experiments on two theories that were filtered out based on exam difficulty and found that the results were not sensitive. For pair-0412, with 23/38 correct (60.5%) in screening, and pair-0089, whose exam screening showed a minimum 21/34 (61.8%) and max possible accuracy 100%, the completion curves look the same as on our selection of theories, with a slight shift up and to the left showing improved performance.

Figure S4 (Appendix A.12) shows the completion curves for these two theories; compare to Figure 3A. Table 3 summarizes performance at 2k, 5k, and 10k tokens.

Table 3. Completion percentages on theories excluded by difficulty screening (means).

ModelToken thresholdpair-0089: baselinepair-0089: explorepair-0412: baselinepair-0412: explore
GPT 5.6 Sol2,00092.294.161.481.6
GPT 5.6 Sol5,000100.099.086.094.7
GPT 5.6 Sol10,000100.0100.098.299.1
Claude Sonnet 52,00021.636.37.037.7
Claude Sonnet 55,00023.539.28.838.6
Claude Sonnet 510,00023.539.28.838.6
GLM 5.32,00038.256.933.352.6
GLM 5.35,00068.674.544.764.9
GLM 5.310,00084.381.459.670.2

Dear reviewer,

We found your feedback on our paper's weaknesses important and we were hoping to hear back in the next day about if any of our additional explanation and experiments were helpful. Our most important additions to your points were (1) new matched token budget preparation baselines to compare exam performance, (2) explanation of our canonicalization process for checker queries, (3) increased theory sample size and a pilot for sampling broader theory signatures, and (4) added scientific exploration citations.

Otherwise, we really appreciate your careful reading of our manuscript and positive evaluation of our research ideas. Thank you for your time!

Reviewer_Lu3VOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a benchmark for open-ended mathematical exploration based on automatically generated two-axiom equational theories with one binary operation. It evaluates future utility through unseen binary entailment questions, comparing same-family solvers with and without self-generated exploration notes. It evaluates diversity through distinct checker queries over repeated explorations. All three models benefit on average, unevenly across theories and token budgets; query distributions show a shared core and long tail. Automated provers and countermodel searches certify answers.

Critical strengths
  1. The automated generation and verification pipeline yields relatively unfamiliar, reproducible mathematical environments, avoiding dependence on manually authored problems.

  2. The same-family exploration–exam design demonstrates reusable knowledge and exposes substantial heterogeneity across models, theories, and budgets. The appendix provides a concrete example of derived structural insights.

Critical weaknesses
  1. The paper frames its study as mathematical exploration, but exams primarily reduce to binary entailment judgments within two-axiom, single-operation systems. This setting does not directly assess question significance, original concept formation, or broader mathematical creativity; conclusions about general research ability should be correspondingly limited.

  2. Exploration notes add information and upfront compute absent from the baseline. No matched-summary or equal-budget comparison isolates the value of autonomous exploration. Cross-family transfer is also untested. GPT-5.6-based difficulty screening may shape model comparisons, and the filtering path to 21 theories is unspecified.

  3. Counting distinct checker queries does not establish semantic idea diversity or usefulness. Variable renaming, equivalent identities, and isomorphic countermodels may inflate counts; the quality of long-tail queries is unevaluated. Table 2 omits clear denominators for invalid/truncated outputs.

Questions for authors
  1. How many of the 373 theory candidates passed each subsequent selection stage, and is 21 exhaustive? Could you add a matched-summary/equal-compute baseline or results without GPT-5.6 screening?

  2. Are equivalent checker queries canonicalized, and can you evaluate the value—not just the count—of distinct ideas? Please clarify Table 2 denominators and whether cross-model exploration documents transfer.

5 comments

(1/3)

Thank you for reading our paper and for your insightful comments!

Weaknesses

1 - The paper frames its study as mathematical exploration, but [...] conclusions about general research ability should be limited.

We agree that these experiments do not directly measure mathematical significance or general research ability. Our intended claim is narrower: we operationalize two aspects of exploration—-future utility and query diversity—-in controlled mathematical environments.

We initially wanted our exams to be based on writing proofs. However, we found that there were many challenges to automatically grading proofs--translating text into Lean for verification often led to ill-formed statements, sorrys, and other edge cases and bugs. Most AIs that write good Lean code do so in an agentic tool-use setup, which would not work for our experimental setup. So, we decided that entailment questions were the most robust way to evaluate a model's theory understanding without these correctness concerns. We developed different question types and different generation methods to try to squeeze as much intuition into programmatically generated questions as we could, as detailed in Appendix A.4.

Despite the simple question format, we believe that our evaluation method isolates model behaviors that are fundamental to open-ended exploration. Thanks to the cross-model exam evaluation you kindly suggested, we can see in Fig. 2 of the updated paper and in Table 2 of this rebuttal that Claude underperforms GLM and GPT in generating useful exploration documents, which we analyze in the cross-model section. The diversity study, once we establish that the long tail of queries of a model is mathematically sensible, suggests that many instantiations of a model would have wide-ranging ideas.

Of course, the holy grail of this field would be mathematical significance and creativity. However, these are difficult qualitative analyses that we don't believe LLM-Judges or other automated methods are capable of handling. Until the next step-change in AI capabilities, we believe frameworks like ours that, although somewhat constrained, isolate model behaviors and are interpretable, quantifiable, and repeatable for running statistics.

Finally, the method is extensible beyond two-axiom, single-operation signatures. The theory filtration steps described in our paper are for computational and budget tractability. To demonstrate extensibility, we ran a pilot study of the exam evaluation completion curves using an exam on a three-axiom, two-operation theory, written by GPT-6-Astra Codex on high reasoning. The exam and results are documented in Appendix A.6. For our final version, we will update the code to make it possible to programatically generate theories and exams beyond the signature in our paper and release it for others to use.

2 - Exploration notes add information and upfront compute absent from the baseline. No matched-summary or equal-budget comparison isolates the value of autonomous exploration.

Comparisons

We agree that document comparisons with matched preparation budgets help with intuition. We added "cheatsheet" which is a model prompted with the exam task and question formats, and "derivation" which is a model prompted to derive consequences. The prompts are given in Appendix A.11. We ran this on a subset of 15 of the theories for budget reasons and supply the completion curves in Figure S3, which would be moved to Fig. 2b for the final version. The results are summarized in the table below.

Table 1. Matched 15-theory preparation comparison: mean completion %.

ModelToken thresholdBaselineExploreDeriveCheatsheet
GPT 5.6 Sol2,00045.968.169.166.4
GPT 5.6 Sol5,00070.680.481.778.5
GPT 5.6 Sol10,00091.089.589.587.0
Claude Sonnet 52,00019.344.645.738.3
Claude Sonnet 55,00020.645.746.639.6
Claude Sonnet 510,00020.846.146.940.0
GLM 5.32,00022.842.542.240.7
GLM 5.35,00035.154.753.351.9
GLM 5.310,00046.262.760.160.0

The explore, derive, and cheatsheet CI's mostly overlap, with mean cheatsheet performance notably underperforming relative to explore and derive.

Research motivation

At a higher-level, rather than trying to show that an open-ended exploration document is the best-performing preparation out of all possible document preparations, we wanted a method that would allow us to evaluate the quality of the exploration, and the exploration document itself. So, we want to know how an exam-taker's performance improves when given exploration documents from different models.

(2/3)

Our question-answering results show that (1) open-ended exploration consistently discovers useful information for exam performance; (2) tokens of general-purpose exploration documents are more helpful than additional problem-solving tokens, which can be seen by the greater initial steepness of the exploration curves; and (3) different model explorations have different effects on exam performance, based on both curve height and shape.

(3) crucially means that we can identify markers of strengths and failure modes of model exploration, which can then be used for targeted qualitative analysis. These model differences are shown in our cross-model exploration document transfer, discussed in the cross-model section of our rebuttal.

2 - Cross-family transfer is also untested.

This was our next step to isolate exploration quality from affinity with the same model! We evaluated all 9 explorer/test-taker combinations on 30 theories, with 3 runs per combination and theory. Fig. 2 in our updated paper now has same-model and cross-model performance. Table 2 summarizes their means at 2k, 5k, and 10k tokens.

The most striking takeaway is that Claude exploration underperforms across all test-takers. Our initial follow-up qualitative analysis suggests that Claude tends to overly focus on specific facts or models rather than bringing them together to general conclusions or laws. In one example, it constructs a family containing noncommutative models but still lists commutativity as an open conjecture. This discussion would be expanded in a final paper to demonstrate how our method reveals differences in exploration characteristics of different models.

Table 2. Cross-model exploration transfer: mean completion %.

Test takerToken thresholdBaselineGPT explorationClaude explorationGLM exploration
GPT 5.6 Sol2,00047.071.160.268.6
GPT 5.6 Sol5,00074.183.079.383.7
GPT 5.6 Sol10,00092.191.291.290.9
Claude Sonnet 52,00021.949.444.751.9
Claude Sonnet 55,00023.050.245.852.6
Claude Sonnet 510,00023.250.346.052.7
GLM 5.32,00024.142.533.244.4
GLM 5.35,00036.156.746.056.5
GLM 5.310,00047.465.755.565.4

2 - GPT-5.6-based difficulty screening may shape model comparisons

For practical analysis, we wanted to run exams that would show a spread in model performance across question budgets of up to 10k tokens, which would not be possible if models could solve exams quickly. We based this performance on only GPT-5.6 because it was a rough screening metric that we did not want to consume much budget. We chose a lax cutoff of 60% performance to account for model differences.

We ran follow-up exam experiments on two theories that were filtered out based on exam difficulty: pair-0412, with 23/38 correct (60.5%) in screening, and pair-0089, whose exam screening showed a minimum 21/34 (61.8%) and max possible accuracy 100%. The completion curves look the same as on our selection of theories, with a slight shift up and to the left for improved performance, so we believe bias is not an issue.

Figure S4 (Appendix A.12) shows the completion curves for these two theories; compare to Figure 3A.

Table 3. Completion percentages on theories excluded by difficulty screening (means).

ModelToken thresholdpair-0089: baselinepair-0089: explorepair-0412: baselinepair-0412: explore
GPT 5.6 Sol2,00092.294.161.481.6
GPT 5.6 Sol5,000100.099.086.094.7
GPT 5.6 Sol10,000100.0100.098.299.1
Claude Sonnet 52,00021.636.37.037.7
Claude Sonnet 55,00023.539.28.838.6
Claude Sonnet 510,00023.539.28.838.6
GLM 5.32,00038.256.933.352.6
GLM 5.35,00068.674.544.764.9
GLM 5.310,00084.381.459.670.2

2 - and the filtering path to 21 theories is unspecified.

We summarize the filtration steps from 373 => 21 in a table below. The selectiveness of the first 100 for exam generation and for evaluating theories was a result of budget and time constraints, otherwise we would have run our method on all theory-exams accepted from the 373 theories. Recorded API costs were 544 USD for the experiments in our manuscript and an additional 980 USD for various follow-up experiments for rebuttal.

In consideration of the small numbers of theories we had in our experiments, we added 9 theories for exam evaluation and 10 theories for diversity evaluation, totaling 30 for exam eval and 15 for diversity eval. We updated Figs. 2 and 4C and will update the rest of the paper in the final version.

(3/3)

Table 4. Original exam selection.

StageExcluded / not runRemaining
Structurally filtered catalogue—373
First 100 in random seeded shuffle273 not attempted100
Exam generation rejected3961
Fewer than 32 questions259
Screening score above 60%3227
First 21 accepted theories evaluated6 deferred21

Exam generation could be rejected for missing a question type, subject to balanced T/F questions and very lax difficulty filters (e.g. must be more than one rewrite away). We will describe all these details in our final version of the manuscript.

3 - Counting distinct checker queries does not establish semantic idea diversity or usefulness. Variable renaming, equivalent identities, and isomorphic countermodels may inflate counts;

We did canonicalize queries but missed documenting this in the manuscript! We regularize by whitespace formatting and for equivalent statements by variable renaming, equality reversal, and order of assumptions. Countermodel queries with the same statement are treated as the same concept even with different countermodel tables. We will include this in a final writeup.

3 - the quality of long-tail queries is unevaluated.

Initial inspection shows that some of GLM's rare queries were mathematically-sensible and motivated, with clear interpretations. Two such questions were: "Does squaring preserve multiplication? (xx)(yy)=(xy)(xy)(xx)(yy)=(xy)(xy)" - Refuted and "Can four-factor expressions be rewritten into paired products? ((xy)z)w=(xw)(yz)((xy)z)w=(xw)(yz) - Proved.

Even without deeper investigations of the immediate usefulness of these long-tail queries, the capability of generating ideas without getting stuck in a small distribution already gets at what we want to measure. In human society, creative ideation generates many non-useful products and maybe one or two good ideas. In an AI setting, we could imagine wanting to develop a collective of agents that are inherently capable of having different ideas in order to innovate together.

There is much more that can be done to study the intersection of diversity and utility with the data generated by our method: by doing further qualitative analysis, reading, and comparing checker queries to what a model explores without the checker. However, this follow-up work will take a significant amount of time for expert review of equational theory examples. As such, we believe this work is better suited for a follow-up project rather than this paper, which aims to establish the method.

3 - Table 2 omits clear denominators for invalid/truncated outputs.

Our caption was unclear: invalid and truncated outputs remain in the denominators. The table below reproduces the paper's Table 2 at the 2k token threshold, showing percentages to one decimal place and explicit counts for the original 21 theories and three runs per theory.

Table 5. Reproduction of paper Table 2, with explicit counts. Each cell reports mean % (pooled numerator/denominator). Percentages average over runs within each theory and then weight theories equally, whereas counts are pooled across all theory–run pairs.

MetricAnswerGPT 5.6 Sol BaseGPT 5.6 Sol GuidedClaude Sonnet 5 BaseClaude Sonnet 5 GuidedGLM 5.3 BaseGLM 5.3 Guided
AccuracyProof56.8 (750/1,311)73.6 (977/1,311)18.7 (246/1,311)38.8 (515/1,311)35.9 (472/1,311)60.3 (810/1,311)
AccuracyCountermodel40.2 (514/1,311)73.8 (972/1,311)21.2 (278/1,311)51.2 (663/1,311)11.7 (148/1,311)34.5 (455/1,311)
Response proportionProof28.4 (751/2,622)36.9 (981/2,622)9.7 (254/2,622)23.0 (609/2,622)20.1 (528/2,622)33.1 (888/2,622)
Response proportionCountermodel20.4 (523/2,622)37.3 (982/2,622)15.7 (411/2,622)39.6 (1,025/2,622)8.0 (204/2,622)19.2 (503/2,622)

Questions

  1. We describe the filtering to 21 theories in the fourth part of our response to Weakness 2; 21 theories is not exhaustive! We add baselines with matched preparation budgets in the first part of our response to Weakness 2. We add some test cases of theory exam evaluations that were not included as a result of GPT-5.6 screening.

  2. The checker queries are canonicalized! We qualitatively evaluated the mathematical sensibility of several of the long-tail queries. Further analysis of the value of these ideas would rely on more in-depth qualitative analysis and comparison to exam cases as described in the second part of our response to Weakness 3. The denominators for the paper's Table 2 are reviewed in Table 5 of this rebuttal, in the third part of our response to Weakness 3. Cross-model exploration documents transfer as described in the second part of our response to Weakness 2.

Thank you again for reading our paper and please let us know of any other concerns!

Comments on the paper

1 comment

Dear Reviewers and AC’s,

Thank you again for your time and thoughtful suggestions! We would like to summarize our paper’s strengths and our comprehensive response to the cited weaknesses.

We used equational theories as a testing ground for evaluating open-ended mathematical exploration of frontier language models. Equational theories represent varied instantiations of unfamiliar math. These theories and challenging exams about them can be generated and graded using automated reasoning techniques. Enabled by these non-neural computational methods, we developed approaches for evaluating two exploration desiderata: usefulness for future problem-solving and idea diversity. Our work provides a framework for characterizing frontier AI exploration capabilities and behaviors and is novel for not requiring model knowledge ablation for evaluation. We believe it is highly relevant to characterizing the status of research-grade AI's.

Our additional experiments address the reviewers' concerns.

1 - Theory sample size and extension to richer mathematical settings. Reviewers raised concerns about theory sample size and restriction to two-axiom single-operation theories. We expanded our experiments from 21 => 30 theories for problem-solving utility and 5 => 15 theories for diversity, finding that the main trends remained stable. We also ran a pilot study on a theory with 3 axioms and 2 operations, demonstrating that the evaluation method can extend far beyond our sample of simple theories. Lu3V (W1, W2); xCCf (W3); KCcn (W1)

2 - Theory selection procedure and GPT-based difficulty screening. Reviewers asked how we selected our theories and wondered if GPT exam screening difficulty could influence findings. We explained the complete, reproducible selection procedure and the budget considerations that prevented a larger theory set. We ran a pilot exam study on 2 theories excluded by the difficulty screening which demonstrated similar results to our paper. Lu3V (W2, Q1); xCCf (Q3); KCcn (W1, Q1)

3 - Matched-token budget comparisons. Reviewers noted that exploration provides additional preparation budget absent from the baseline. We added 2 conditions with matched preparation budgets: “cheatsheet”, which was given the exam task and question formats and “derivation”, which was instructed to derive consequences of the axioms. Exploration, derivation, and cheatsheet performed comparably; cheatsheet had lowest mean performance.

Cheatsheet and derivation apply strong priors to generating exam-useful information. The goal of exploration is to discover similarly useful ideas with no such prior rather than outcompeting the comparisons. Therefore, overlapping performance gives a positive result that exploration can produce useful preparation comparable to guided search. Lu3V (W2, Q1); xCCf (W1, Q1); KCcn (W2, Q2)

4 - Query canonicalization and the usefulness of rare ideas. Reviewers noted that different query strings need not represent different mathematical ideas, and that a long tail of queries does not establish usefulness. We clarified that our diversity analysis already canonicalizes queries that differ only in formatting, variable names, equality direction, model table, or assumption order.

We also added LLM-judge analysis of query usefulness on 5/15 diversity-study theories. We set up and did preliminary eval of a constrained judge framework wherein an LLM assigns one of a small set of category labels (e.g. advance, useful check, redundant, etc.) to a checker query. Our LLM-judge classified ~50% of rarest queries within a theory (1 occurrence) as advances or useful checks, demonstrating that the long tail contains useful mathematical inquiries. Lu3V (W3, Q2); xCCf (W2, Q2); KCcn (W2, Q3)

5 - Cross-model exploration transfer. Reviewers asked whether exploration documents transfer to other models (study exploration quality vs. model compatibility).

We evaluated all explorer–exam-taker combinations. Exploration documents transferred across models and Claude’s documents generally provided smaller benefits than GPT’s or GLM’s across exam takers. This comparison also revealed Claude's issues with articulating general principles in exploration, illustrating how our framework can identify differences in exploration behavior and guide closer examination of their causes. Lu3V (W2, Q2)

6 - Non-structural concerns. We clarified our release plan for data, benchmark, and code. We counted exam questions by theory and question type. We clarified Table 2’s denominators. We added discussion of related scientific exploration benchmarks and a worked illustration of the exam-based evaluation. Lu3V (W3, Q2); xCCf (W4); KCcn (W3, Q1, Q4)

Thank you again for the careful feedback! We hope these discussions demonstrate the value of Mathematica Incognita as an evaluation and we are excited to polish our paper into a final version based on your help.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.