acceptodds
Under review as a conference paper at ICLR 2027

A Simple Statistic for Discovery Capacity in Language Models

Abstract

Large language models (LLMs) show growing promise for scientific discovery, yet basic questions remain about what enables reliable discovery. Can a simple statistic of the training data reveal a model's capacity for discovery? We study this question through missing mass and develop a theoretical account in which monofacts–the facts observed once during training–set a budget for discovery. In materials science and regulatory genomics, we directly manipulate monofact prevalence and find that more monofacts consistently yield more discoveries across two open-weight model families, with and without reasoning traces, and across sampling strategies and reasoning budgets. Further experiments isolate the effect of monofact prevalence from key confounds, show that higher monofact prevalence increases mass placed on unobserved outputs, and find that stronger conditioning can substantially increase discovery. Together, our results identify monofact prevalence in training data as a key factor governing LLM discovery.

Primary area:
learning theory
Keywords:
AI for Science

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fvbFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks if we can predict how good an LLM will be at discovering just by counting how many facts only appear once in the training data (monofacts). The authors borrow the missing-mass framework (Kalai and Vempala, 2024) and propose that training dataset with more monofacts provide LLMs with much more room to generate unseen possibilities, hallucinations and discoveries. The authors show that discovery and hallucinations are related, and also show an upper and lower bound on discovery capacities using monofact rates. These were tested on GPT-OSS 20B and Gemma 3 12B with monofact rates from 0 to 100% on materials science and regulatory genomics. The authors find that higher monofact rates consistently improve recovery of held-out targets across model families, trainingstages, and sampling settings.

Critical strengths
  1. Potentially important theoretical perspective. Extending the missing-mass account of hallucination to correct, previously unseen statements offers an interesting connection between training-data and scientific discovery. The frontier formulation and associated bounds also provide a framework for reasoning about the relationship between discovery and hallucination.

  2. The authors directly manipulate monofact presence across two scientific domains and two model families, observing consistent improvements across training stages and sampling settings. The additional reasoning-budget, repeated-exposure, and cross-domain experiments provide useful evidence about the robustness and limitations of this relationship.

Critical weaknesses
  1. The theoretical results do not establish the claimed dependence of discovery on monofact prevalence. Theorem 6 provides an upper bound on discovery probability, but increasing an upper bound does not imply increasing realized discovery. Theorem 7 additionally depends on frontier leakage, calibration error, and parameters that may change across training conditions. The theoretical results do not guarantee the monotonic relationship observed experimentally. The discovery-frontier assumptions and quantities are also not directly validated in the experiments. This leaves a substantial gap between the mathematical results and the paper's central claim that monofacts govern discovery capacity.

  2. The causal interpretation of discovery remains insufficiently supported. Monofact prevalence and training-set diversity are confounded in the main experiments. In section 5.2, repeating examples changes optimization and memorization without changing the underlying distribution's missing mass. Diversity, training exposure, etc. are all possible confounds that were not removed.

  3. Connection between improvements in monofacts and scientific-discovery capacity is unclear. Discovery is evaluated through exact recovery of held-out targets rather than independently validated scientific findings. For example if a model that predicts another chemically valid material satisfying the requested properties could be a “real discovery,” but is counted as a failure here. While predicting the exact answer based on structural clues without scientific reasoning will be counted as a discovery here.

  4. In materials discovery, the model also receives structural information derived from the target material itself. The authors remove direct answer-related hints but the remaining information still includes geometry, parameters and positions hinting at the answer.

Questions for authors
  1. Theorems 6 and 7 provide bounds on discovery probability, but do not establish that realized discovery increases monotonically with monofact prevalence. Can the authors demonstrate that theoretical bounds meaningfully predict the results?

  2. In Section 5.2, repetition does not change the missing mass distribution, but changes optimization and memorization. How could this be interpreted as isolating monofact prevalence rather than training exposure? Could the authors distinguish these explanations?

  3. How much of the reported improvement remains when a target’s structural information is removed? Could the authors provide simple nearest-neighbor baselines, etc. to show that the gains reflect science rather than target-specific cues?

  4. How are alternative materials/DNA modifications that satisfy the constraints but differ from target treated? What is the true utility of exact matching in science?

5 comments

Thank you for the careful review and for finding the theoretical perspective potentially important and our experiments robust.

W1/Q1: Do the bounds predict the results?

TL;DR: Thm. 7 adds a floor, and with every other term fixed or flat, both bounds rise with μ^\hat\mu.

We agree that an upper bound alone does not imply that realized discovery increases, which is why we pair Thm. 6 with the floor in Thm. 7. Sec. 4.1 (From theory to experiments) now states this argument in the paper.

Both bounds are linear in μ^\hat\mu, so whenever the remaining terms are stable, realized discovery is bracketed between two quantities that increase with the monofact rate. α\alpha, β\beta, mm are properties of the world prior and the domain (Def. 4 requires the band to hold for all DD; mm is the sparsity of true statements, assumption (iii)), and we fix NN and δ\delta, so the concentration terms do not move. The model-dependent terms, CE and ρf\rho_f, could vary across SFT runs, so we measure both. ECE is nearly constant across monofact rates (5.6–6.2%; General Response A3 and App. B.1). Good–Turing estimates of ρf\rho_f (Sec. 3) are also flat (0.772–0.789, no significant trend; General Response A4, App. B.5). Every term other than μ^\hat\mu is thus fixed or empirically stable, and discovery indeed rises monotonically across two domains, two model families, and both pass@1 and pass@16.

The theory makes two further directional predictions that our results confirm. Discovery and hallucination rise together in-distribution (Fig. 2, App. C). Memorization also lowers the floor even at fixed μ^\hat\mu (General Response A3 and revised Sec. 5.2).

W2/Q2: Confounds and repetition.

TL;DR: Exposure is fixed, discovery rises at fixed diversity, and repetition now tests calibration rather than diversity.

Training exposure is fixed across monofact rates: every run within an epoch condition uses the same NN and number of steps (now stated in Sec. 4.1). For diversity, we ran the direct control (General Response A5, App. B.3). At fixed NN, the same 5K chemical systems, and an identical one-epoch training, discovery still rises at every step from rr = 0 to 0.4 (pass@16 17.7→20.8%). Two results in the paper align with this finding. The multi-epoch runs (revised Sec. 5.2, Fig. 4) also hold diversity fixed, yet the trend disappears at 3 and 5 epochs, and discovery falls at high rr instead of strengthening, ruling out undertraining. A coverage account also predicts more discovery with fewer hallucinations, whereas both rise together in-distribution (Fig. 2, App. C).

We agree that repetition changes ff while leaving the data statistics untouched. In this context, repeated passes push ff toward the empirical distribution over OO, shrinking f(U)f(U) and raising CE, which lowers the floor via f(U)≥q(U)−CEf(U) \geq q(U) - \mathrm{CE}. We now confirm this: under repetition, ECE rises with rr (Spearman p<10−4p<10^{-4} at 3 and 5 epochs), reversing its slight decrease at one epoch (General Response A3, App. B.1, Fig. 6b).

Repetition is an established way to inject miscalibration at a fixed monofact rate [6]. The revised Sec. 5.2 states that repetition leaves μ^\hat\mu unchanged, and it now presents the multi-epoch experiment as a direct test of the calibration rather than a diversity control.

W3/Q3: Structural cues and baselines.

TL;DR: The input contains no composition, a nearest-neighbor baseline scores 0%, and removing the CIF leaves the monofact gain unchanged.

The structural input contains no element identities, formula, labels or occupancies (App. D). Within a chemical system, many compounds remain possible (e.g., H$_2$O vs. H$_2$O$_2$ for H-O), so the model must still infer the stoichiometry. A nearest-neighbor baseline scores 0%, and 72% of test materials come from chemical systems absent from training (General Response A1, App. B.4).

We test this directly in App. B.6 (Table 3). Removing the CIF lowers pass@16 at each monofact rate tested, but the gain from r=0r=0 to r=1r=1 is unchanged (+6.5 vs. +6.7 points), so the CIF cannot explain the trend over rr. Cumulatively removing the remaining properties at r=1r=1 lowers discovery at every step, from 27.8% to 13.9% (App. B.6, Fig. 10), so the model uses the requested properties rather than geometry alone.

W4/Q4: Alternative valid answers.

Exact match is the only criterion with confirmed ground truth, since the test materials were later recorded in Materials Project, a curated database, and each MPRA variant’s activity was experimentally measured (see Protocol in General Response). Unlike prior monofact work on synthetic facts [6], we use real post-cutoff scientific targets.

We hope the revision addresses your concerns. If so, we would be grateful if this could be reflected in your assessment, and we are happy to clarify anything further.

References are listed in the comment following the general response.

Thank you again for your careful review. We have added several new analyses addressing your concerns, including controls for diversity and structural cues. We would greatly appreciate your thoughts on these results and welcome any further questions.

I appreciate the authors' efforts to resolve the concerns above. I find the empirical findings of the paper to be more solid now. My remaining concern, however, is that there are still some overclaims in the paper.

Specifically, as in W1, the theoretical bounds do not imply a monotonic increase in actual scientific discovery. Furthermore, only counting the exact matches in W4 may still not fully capture scientific discovery.

With that said, I believe the empirical findings are interesting enough and the remaining problems are largely an issue of interpretation and framing. The responses from the authors have positively influenced my assessment of the paper.

Reviewer_EQHgOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript extends the missing mass framework to try and investigate the discovery capacity of large language models, contextualized by facts that appear only once in the training data. The authors correctly note that in scientific discoveries all unseen statements are not equal and provide a well-defined framework to model their proposed setting. There is a neat problem setup, explaining the connection between monofacts and discoveries, ensuring that other confounding effects like diversity of the data does not influence the outcomes. Two problem settings, material science and genomics, are investigated and both are of importance to the larger scientific contributions. The proposition 5 and theorems 6 and 7 are clean and provide a good base to build the experiments on. The ablations and results are convincing in terms of proving the relationship between monofacts and novel discoveries and the only caveat in the pitch seems that the scope is quite narrow in terms of a specific phenomenon being investigated

Critical strengths

[S1]: The investigative setup is neat. I really like the re-purposing of the missing mass framework to the discovery task and it seems like a natural extension and provides a clean lens into understanding model discovery from this pov at the very least. The ablations generally answer the first questions that pop to mind when one looks at the presented hypothesis, and Sec. 5.2 and App. B.2 are crisp and negate the natural concern of the confounding of the dataset diversity. The temporal split albeit natural, is well adapted for this investigation and lends itself to a lot of other scenarios which might be of use to the researchers in other fields.

Critical weaknesses

[W1]: Task formulation inconsistencies: Genomics and Material Science are not my areas of expertise, but even with the description provided by the authors it seems that the claim of material discovery is a bit stretched? As in line 1237 aren't the reference_cif's provided during supervision? It is unclear what the exact task definition is and how hard of a discovery problem it truly is? Likewise for Example 2, Line 1584, the reference DNA is provided which has 199 bases and the model is asked to edit the same, whereas Line 1176 has the model output exactly 200 DNA bases? In general I feel the two tasks could be better defined and explained, especially given the usual audience might not understand these exact applications.

[W2]: Theoretical applicability: From Line 76, Thm. 6 and 7 model an unconditional distribution f∈Δ(S)f \in \Delta(S), and each document is a single statement, but the experimental tasks are conditional generation and the monofacts are domain specific (e.g. H-O appearing once). It isn't clear why the outcomes of the theoretical framework are applicable in the experimental settings.

Questions for authors

[Q1]: See [W1] for details. Could you please elaborate on the task definitions and what the models are expected to discover and what these discoveries imply in terms of novel ideas? What would be helpful is to provide a small example that could be understood by a layperson as to what could constitute a discovery and what information was provided to the LLM to make that discovery.

[Q2]: See [W2] for details. Could you clarify why the experimental settings fit the theoretical model and assumptions and how the outcomes from Sec. 3 carry forward to the results observed in Sec. 5?

5 comments

Thank you for the thoughtful review and for finding our setup neat, theory clean, and ablations convincing.

W1/Q1: Task definitions.

TL;DR: Models must name an exact, unseen compound or DNA edit from properties alone.

Materials. The model receives (i) a chemical system, i.e. a set of elements, (ii) target properties (e.g. stability), and (iii) a composition-blind view of the crystal that includes lattice geometry, with every element identity, formula, label and occupancy removed to prevent leakage (App. D). The models must name the exact compound, whose formula is never in the input. Removing the CIF lowers absolute rates (pass@16 at rr=1: 27.8→20.6%) but leaves the monofact gain unchanged (+6.5 vs. +6.7 points from rr=0 to 1; App. B.6, Table 3).

For instance, in the first example in App. E.1, the model reasons from cell volume and site counts to CeMg$_6$B. This is far from trivial, because one element set admits many compounds with different structures and properties (e.g., H-O contains both H$_2$O and H$_2$O$_2$). Moreover, 72% of test materials come from chemical systems never seen in training (General Response A1, App. B.4).

This is a real and valuable discovery problem. A wide range of applications need materials with specific combinations of properties, and proposing candidate compounds that satisfy them is a core step in computational materials design. LLMs are already used for exactly this kind of property-conditioned generation [7, 8].

Importantly, every test target was first publicly released after the models’ knowledge cutoffs, making this a discovery task (see Protocol in General Response). As further evidence that the targets are new to the models, both models find test CIFs much more surprising than training CIFs (Cohen’s d of 1.77 and 0.64; General Response A2, App. B.4).

Genomics. The model receives a reference DNA sequence and the requested MPRA activity changes in three cell types. It must predict a single edit (i.e., substitution or insertion), its position, and the new base(s). The 199-base reference in Ex. 2 is an insertion variant (T→TA). As the trace notes (App. E.2, Example 2), the insertion restores the 200-nt MPRA oligo length. The “exactly 200 DNA bases” instruction in the App. D example was a leftover from an earlier task format we did not use. We have corrected it in the revision so that it matches the edit format used in all experiments, and we thank the reviewer for catching it.

This task is equally valuable, since designing regulatory sequences with cell-type-specific activity is central to gene therapy and computational biology, the very purpose of our dataset [9].

W2/Q2: From the unconditional theory to conditional tasks.

TL;DR: Thms. 6–7 apply to the joint distribution over prompts and generations. All three predictions are supported by the results.

Each training example is a single (input, target) pair, which matches assumption (i) of one statement per document (Sec. 2). For conditional generation, the model and the prompt distribution together induce a joint distribution over pairs, f(x,y)=ν(x)f(y∣x)f(x, y) = \nu(x) f(y \mid x). Thms. 6-7 apply to this joint, with discovery averaged over prompts.

Sec. 3 carries over to Secs. 4-5 because our claims are comparative, not numeric. Across runs, α\alpha, β\beta, mm are properties of the world prior and the domain (Def. 4 requires the band to hold for all DD; mm is the sparsity of true statements, assumption (iii)), and NN and δ\delta are fixed by design. The model-dependent terms are also stable, since CE is nearly constant across monofact rates (General Response A3 and App. B.1), and frontier leakage ρf\rho_f is flat (General Response A4, App. B.5). The theory therefore predicts directional effects, and we test three of them. First, discovery should rise with μ^\hat\mu (Thms. 6-7). It does so monotonically across two domains, two model families, and both pass@1 and pass@16. Second, discovery and hallucination should rise together, since both draw on the same unobserved mass (Prop. 5). We observe this in-distribution (Fig. 2, App. C). Under temporal shift, where unobserved mass is already saturated at 0.98-0.99, higher μ^\hat\mu instead shifts mass from hallucination toward discovery. Third, memorization should lower the floor even at a fixed μ^\hat\mu. Repeated epochs push ff toward the empirical distribution over OO, shrinking f(U)f(U) through f(U)≥q(U)−CEf(U) \geq q(U) - \mathrm{CE}, and discovery drops at high monofact rates accordingly in the multi-epoch runs (revised Sec. 5.2, Fig. 4), as CE rises with rr (General Response A3).

We hope the revision addresses your concerns. If so, we would be grateful if this could be reflected in your assessment, and we are happy to clarify anything further.

References are listed in the comment following the general response.

Thank you again for your constructive feedback. We have revised the paper and provided detailed clarifications addressing your questions. We would be grateful for your thoughts, and we are happy to discuss any remaining concerns.

I appreciate the detailed responses provide by the authors, but still have a few outstanding doubts, clearing which would positively influence my opinion.

My [W2], [Q2] concern is still unresolved. In Theorems 6–7, μ^\hat{\mu} counts statements observed exactly once, and the author rebuttal states "Each training example is a single (input, target) pair," The experiments, however, manipulate the fraction of examples whose chemical system (or, in genomics overlap cluster) appears once. If each (input, target) pair in the training set is unique, would the statement-level monofact rate not be close to 1 at every value of rr? It is still not clear to me why the claim μ^=r\hat{\mu} = r holds true, what is the backing or the justification for the same?

With regards to [W1], [Q1] I appreciate the clarification about the domains, and the discovery problem, and the correction of the Example 2, but all genomics examples in the paper (edit position 99 or 100? Is this also the case for the test set? Shouldn't the edit positions also vary to make the task more challenging? Or a naive baseline that predicts a random base would not perform terribly either? Some more insight into the data construction and evaluation would help strengthen the empirical results and applicability of the proposed method.

Reviewer_AhN1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether monofact prevalence in training data predicts LLM discovery capacity. It extends a missing-mass framework with a discovery frontier and derives bounds connecting monofact rates to discovery probability. Experiments in materials science and regulatory genomics show that higher monofact rates generally improve held-out target recovery across two model families.

Critical strengths

The connection between discovery and hallucination through missing mass is an interesting perspective. The empirical trend is also reasonably consistent across two domains, model families, and training stages, making the observation worth investigating further.

Critical weaknesses

The contribution is narrower than the paper claims. The theoretical extension builds on the existing Good–Turing framework and depends on frontier and calibration assumptions that are not established experimentally. The experiments measure exact recovery of held-out answers, rather than independently validated scientific discoveries. Together, these results offer an interesting observation, but do not yet establish monofact prevalence as a governing factor in scientific discovery.

The main alternative explanation remains unresolved. Monofact prevalence and training-set diversity increase together. The multi-epoch experiment in §5.2 does not isolate them: additional epochs change optimization and overfitting, while repeated exposure to a fixed dataset is not equivalent to drawing additional independent samples for missing-mass estimation. The disappearance of the trend therefore does not rule out diversity as an explanation. This substantially weakens the paper’s central interpretation.

Questions for authors

Do you have comparisons that vary monofact rate while holding the number of distinct chemical systems, dataset size, and training budget fixed?

How should the theoretical monofact statistic be interpreted under repeated epochs, given that replaying examples does not change the underlying dataset’s coverage?

3 comments

Thank you for the careful review and for finding the discovery-hallucination link interesting and our empirical findings consistent.

W1: Frontier and calibration assumptions.

TL;DR: Every term except μ^\hat\mu is fixed by design or measured to be flat.

Our claim is comparative. We now make this mapping explicit in Sec. 4.1 (From theory to experiments). Across our SFT runs, each term falls into one of two groups:

  • Fixed by design: α\alpha, β\beta, mm are properties of the world prior and the domain, so they do not depend on the model: Def. 4 requires the [α,β][\alpha,\beta] band to hold for all DD, and mm is the sparsity of true statements in the domain (Sec. 2 assumption (iii)). NN and δ\delta are identical across runs, since we fix NN at every monofact rate within each training stage and δ\delta is our chosen confidence level.
  • Model-dependent: CE and ρf\rho_f. CE is nearly constant across monofact rates (5.6–6.2%; General Response A3 and App. B.1), and we now estimate ρf\rho_f directly and find it flat (0.772–0.789; General Response A4, App. B.5).

So μ^\hat\mu is the only term that varies meaningfully across runs, in both the capacity bound (Thm. 6) and the floor (Thm. 7). Consistently, discovery rises monotonically with μ^\hat\mu across two domains, two families, and both pass@1 and pass@16. We do not estimate α\alpha, β\beta directly (as we already acknowledge in Limitations), but they are fixed across runs due to being constants set by the data generating process for each task.

W1: Exact recovery vs. validated discovery.

TL;DR: Recovering post-cutoff results is the standard test of discovery, and DFT validation would cost >500K USD.

Our test sets postdate the models’ knowledge cutoffs, so each target is a later-confirmed result the models never saw, the standard retrospective test of discovery [1, 2, 3, 4, 5] (General Response A, Sec. 4.1). Externally validating every generation via DFT would cost >500K USD with QUANTUM ESPRESSO on AWS for just pass@1 across all 66 models. Moreover, open-ended frontier discovery (e.g., the Navier–Stokes result) has so far required frontier-scale systems, not 12-20B models.

Two new analyses support the validity of this setup: test targets are far from the training data (72% come from chemical systems never seen in training; General Response A1, App. B.4), and both models find them much more surprising than training materials (Cohen’s dd of 1.77 and 0.64; General Response A2, App. B.4).

Q1: Holding distinct systems, dataset size, and budget fixed.

TL;DR: At fixed diversity, discovery still rises at every step in rr.

We ran this control (General Response A5), now in App. B.3 (Fig. 8). With NN = 10K, the same 5K chemical systems (dd = 0.5), and an identical one-epoch budget, discovery rises at every step from rr = 0 to 0.4 (pass@1 1.3→2.7%, pass@16 17.7→20.8%). Diversity therefore cannot explain the trend.

Two results already in the paper agree with our new experiments:

  1. The multi-epoch runs (Sec. 5.2) also hold diversity exactly fixed. If diversity drives discovery, the ordering over monofact rate rr should survive repetition, but it does not. If the one-epoch trend was an undertraining artifact, more training should strengthen it. Instead, discovery falls at high monofact rates, where one epoch beats three and five epochs.
  2. Discovery and hallucination move together (Fig. 2, App. C). If diversity worked through better coverage, it should increase discovery while reducing hallucination. Missing mass predicts both rise together, since they draw on the same unobserved mass (Prop. 5). In-distribution, we observe this joint rise.

W2 / Q2: Monofacts under repetition.

TL;DR: Repetition leaves μ^\hat\mu unchanged and raises CE, which lowers the discovery floor.

We agree that μ^\hat\mu is a statistic of the i.i.d. sample, and replaying examples does not change it. Repetition does not change OO, UU, or the frontier CC either. The revised Sec. 5.2 states this explicitly: repetition “leaves the training sets, and therefore μ^\hat\mu, unchanged.”

Repeated passes raise CE, which lowers the floor via f(U)≥q(U)−CEf(U) \geq q(U) - \mathrm{CE}. This experiment is now the subject of the revised Sec. 5.2 (General Response A3; Fig. 4, App. B.1 Fig. 6b), which shows that ECE rises with rr under repetition and discovery falls at high rr as the bound predicts. Repetition is thus an intervention on ff that leaves the data statistics untouched, not a substitute for new samples. Diversity is identical at 1, 3 and 5 epochs, yet the trend appears only at one, and A5 confirms the trend at fixed diversity without repetition.

We hope the revision addresses your concerns. If so, we would be grateful if this could be reflected in your assessment, and we are happy to clarify anything further.

References are listed in the comment following the general response.

Thank you again for your thoughtful review. We have added new experiments addressing your main concerns, particularly the fixed-diversity control. We would greatly appreciate your thoughts on our response and are happy to clarify anything further.

We understand this is a busy period and are grateful for your time. Reviewer fvbF, who shared your concern about diversity, noted that the revised empirical findings are now more solid. If our response addresses your remaining concerns as well, we would be grateful if you would consider reflecting this in your assessment.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their insightful feedback. All three value our missing-mass perspective on discovery, describing it as interesting (AhN1, fvbF), potentially important (fvbF), and a clean lens (EQHg). AhN1 and fvbF both find the results consistent across domains, models, and training stages, and EQHg finds the setup neat. We have uploaded a revised PDF, with changes in blue, that addresses the shared concerns with five new analyses using gpt-oss-20b on materials discovery. Below, we go through each concern and point to the revised sections that resolve it.

TL;DR: Test targets are post-cutoff and far from training ( A ); μ^\hat\mu is the only term in the bounds that varies ( B ); discovery rises with rr at fixed diversity ( C ).

( A ) Our tasks measure discovery.

Protocol (fvbF W4/Q4; AhN1 W1; EQHg Q1). Our evaluation follows the standard for offline discovery, where models are evaluated on whether they recover discoveries made after their training cutoff. It is well established across domains, including materials [1, 2, 4], chemistry [3], and protein structure [5]. We now state this protocol in Sec. 4.1 (Evaluation), with citations. Externally validating every generation via DFT would cost >500K USD on QUANTUM ESPRESSO on AWS for just pass@1 across all 66 models.

A1. Test targets are far from the training data (fvbF Q3; EQHg Q1; AhN1 W1) → App. B.4, App. C. No test formula appears in training, so a nearest-neighbor baseline scores 0%. Moreover, 72.3% of test materials come from chemical systems absent from training, and among the remaining 277 only 1 shares a stoichiometry pattern with any training material. The train and test element distributions also differ substantially, as JS divergence is 0.10 for the test set vs. 0.008 for held-out training data (App. C).

A2. Test targets are novel to the models (AhN1 W1; EQHg Q1) → App. B.4, Table 2. Before our training, both base models find test CIFs significantly more surprising than training CIFs (per-token surprisal; Cohen’s dd = 1.77 for gemma-3-12b-it and 0.64 for gpt-oss-20b; Welch p<10−43p < 10^{-43}).

( B ) μ^\hat\mu is the only term in the bounds that changes → Sec. 4.1 (From theory to experiments).

The terms of Thms. 6-7 fall into three groups.

  1. NN and δ\delta are fixed by design: within each training stage, every monofact rate uses the same NN, and δ\delta is a confidence level that we choose, not a property of the data or model.
  2. α,β,m\alpha, \beta, m are fixed by the task: Def. 4 requires the band to hold for every DD, and the statement space and targets are identical across rr.
  3. CE and ρf\rho_f are the only terms that depend on the model, and A3-A4 show both are flat, as measured in App. B.1 and B.5.

This leaves μ^=r\hat\mu=r as the only term that varies across monofact rates.

A3. Calibration error is flat at one epoch, but repetition raises it with rr (AhN1 W1/W2; fvbF W1/Q1, W2/Q2; EQHg Q2) → Sec. 5.2 (retitled “Miscalibration Weakens the Monofact–Discovery Relationship”), App. B.1 (Fig. 6b).

At one epoch, ECE stays nearly flat across rr (6.12% → 5.56% from r=0r=0 to 1). With three and five epochs, ECE instead grows with rr (6.19% → 7.35% and 6.04% → 7.58%; Spearman p<10−4p < 10^{-4} for both). Since higher calibration error lowers the discovery floor (Thm. 7), repetition lowers discovery at high rr, which matches the multi-epoch results (Fig. 4). Sec. 5.2 now presents this experiment as a direct test of the theory’s calibration prediction.

A4. Frontier leakage is flat (AhN1 W1; fvbF W1/Q1; EQHg Q2) → App. B.5 (Fig. 9).

We estimate ρf\rho_f by Good–Turing from hallucinations that occur once among an input’s 16 samples. ρ^f\hat\rho_f stays near 0.78 across rr (slope −0.008, p=0.18p = 0.18). Together with A3, both bounds are affine in μ^\hat\mu with fixed coefficients and a constant ceiling-to-floor ratio, so μ^\hat\mu is the only term that explains the rise in discovery with rr (Fig. 1).

( C ) Diversity does not explain discovery, but monofacts do.

A5. Discovery rises with rr at fixed diversity (AhN1 W2/Q1; fvbF W2/Q2) → App. B.3 (Fig. 8).

We build new training sets with fixed NN = 10K drawn from the same KK = 5K chemical systems (fixed diversity d=0.5d = 0.5), and vary r∈[0,0.4]r \in [0, 0.4] (each monofact is its own system, so r<dr < d). Each rate uses one epoch and three seeds. Discovery increases at every step in rr (with pass@1 1.3% → 2.7%, and pass@16 17.7% → 20.8%), even though diversity, dataset size, and training budget are fixed. Diversity therefore cannot explain the trend.

We will move the fixed-diversity control (App. B.3) into the main text in the camera-ready version, using the additional page. We hope these revisions address the reviewers’ concerns, and if so, we would be grateful if they could be reflected in the final assessment. We are happy to clarify any remaining questions during the discussion period.

AuthorsOpenReview ↗

References

References

[1] V. Tshitoyan et al. Unsupervised word embeddings capture latent knowledge from materials science literature. Nature, 571:95–98, 2019.

[2] J. Riebesell et al. A framework to evaluate machine learning crystal stability predictions. Nat. Mach. Intell., 7:836–847, 2025.

[3] Z. Yang et al. MOOSE-Chem: Large language models for rediscovering unseen chemistry scientific hypotheses. ICLR, 2025.

[4] J. Sourati and J. A. Evans. Accelerating science with human-aware artificial intelligence. Nat. Hum. Behav., 7:1682–1696, 2023.

[5] A. Kryshtafovych et al. Critical assessment of methods of protein structure prediction (CASP)—Round XIV. Proteins, 89(12):1607–1617, 2021.

[6] M. M. Miao and M. Kearns. Hallucination, monofacts, and miscalibration: An empirical investigation. arXiv:2502.08666, 2025.

[7] N. Gruver et al. Fine-tuned language models generate stable inorganic materials as text. ICLR, 2024.

[8] V. Mishra et al. Foundational large language models for materials research. arXiv:2412.09560, 2024.

[9] S. J. Gosai et al. Machine-guided design of cell-type-targeting cis-regulatory elements. Nature, 634:1211–1220, 2024.

[10] R. Jiao et al. Crystal structure prediction by joint equivariant diffusion. NeurIPS, 2023.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.