acceptodds
Under review as a conference paper at ICLR 2027

Search Allocation Is Not Determined by Loss, Validity, or Unique Yield in a Laboratory

Abstract

Search allocation is not determined by loss, validity, or unique yield. Verifier-backed discovery is commonly scored by those three, yet none of them determines how much verified mass a search puts on a target family. In a laboratory with an exact verifier, withholding a two-variable family from pretraining yields zero verified proposals on six training seeds and on 12 further generation runs, even though every prompt contains . When the family is included, twenty-four matched initializations all visit it, yet parse-conditional allocation runs from 4.6% to 72.4%. The decisive comparison is inside that band: Seed 8 has higher teacher-forced loss than Seed 2 (0.2526 vs. 0.2428) and allocates 34.4% rather than 4.1% ( 37.0% vs. 4.6%). The gap remains under penalty 1.0 (0.4%), a healed prompt prime (7.2%), no feedback (2.0%), six structural exemplars (1.3%), and temperature 1.0 without nucleus sampling (8.5%; 13.2% parse-conditional, against 46.2% and 41.9%). At the low-allocation seed recovers 49/50 catalog keys while allocation stays 5.7%. Allocation and parse-conditional allocation are the quantities a verifier-backed report has to add.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
language model discovery, verifier-backed generation, search allocation, proposal support, formal verification, synthetic reasoning, neurosymbolic AI, evaluation metrics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Vd6BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether learning a knowledge domain during pretraining leads to effective exploration of that domain during autonomous generation. Using a controlled mathematical environment over Z7\mathbb{Z}_7 with an exact verifier, the authors examine the relationships among teacher-forced loss, verified search allocation, and unique discovery coverage. The experiments show that models trained without the target knowledge family fail to generate verified proposals from that family, while models trained with it exhibit substantial variation in search allocation despite similar training losses. The authors also find that increasing the sampling budget can improve unique coverage without resolving low search allocation. Based on these findings, the paper argues that search allocation should be evaluated separately from loss, validity, and unique yield.

Critical strengths
  1. This paper demonstrates that models with similar training losses can exhibit substantially different search allocation behaviors. It also shows that increasing the sampling budget can improve unique coverage without resolving inefficient search allocation.

  2. This paper uses a controlled mathematical environment with an exact verifier to investigate search behavior. It also conducts experiments across multiple training seeds, sampling budgets, and decoding configurations, providing empirical support for its main observations.

Critical weaknesses
  1. Limited novelty compared with existing research. The main finding that low teacher-forced loss does not necessarily lead to effective generation or exploration is closely related to existing studies on the mismatch between likelihood and generation behavior [1, 2]. More recent work has also investigated diversity collapse in LLM-based scientific hypothesis search [3]. Although this paper provides a controlled mathematical environment to study these phenomena, its main conclusions largely reinforce existing observations. Furthermore, the proposed search allocation metrics are relatively straightforward measures of the proportion of verified proposals. It is therefore unclear what fundamentally new insights this paper provides beyond previous research. The authors should more clearly distinguish their contributions from these studies and explain the novel understanding gained from their experiments.

  2. Limited experimental settings and unclear generalizability. The main experiments are conducted exclusively in a simplified mathematical environment over Z7\mathbb{Z}_7, using relatively small language models trained from scratch. The discovery task mainly involves generating and verifying algebraic identities within a predefined set of mathematical structures, which differs substantially from realistic scientific discovery tasks. Although this controlled setting is useful for isolating experimental factors, it remains unclear whether the findings generalize to modern pretrained LLMs or more complex discovery systems. This limitation is particularly important given the paper's broader motivation concerning LLM-based scientific discovery. Evaluating the main findings on larger models and more realistic discovery tasks would substantially strengthen the paper.

  3. Insufficient analysis of the underlying causes of search allocation divergence. In Section 4.1 and Figure 3, the paper demonstrates substantial differences in search allocation among models with similar teacher-forced losses. However, the most severe example of structural proposal starvation is primarily observed in Seed 2. Although the authors conduct additional experiments with different decoding configurations and prompts, these experiments mainly demonstrate that the phenomenon persists for this particular checkpoint. They do not sufficiently explain why such a large difference emerges between independently trained models or establish how frequently this failure occurs. The discussion in Section 5 attributes the phenomenon to preferences for unary identity patterns, but the underlying training dynamics remain unclear. A more systematic investigation of the causes and prevalence of this behavior would strengthen the paper's conclusions.

[1] "The Curious Case of Neural Text Degeneration", Holtzman et al., ICLR 2020.

[2] "Neural Text Generation with Unlikelihood Training", Welleck et al., ICLR 2020.

[3] "Towards Diverse Scientific Hypothesis Search with Large Language Models", Wang et al., ICML 2026.

Questions for authors
  1. Could the authors further clarify the novelty of this work compared with existing studies on the mismatch between likelihood and generation behavior, particularly regarding the new insights provided by the proposed experimental setting?

  2. Could the authors further explain why severe search allocation divergence occurs in certain training seeds, particularly Seed 2, and provide additional evidence on how frequently this phenomenon occurs?

Reviewer_BCJyOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

NA

Critical strengths

NA

Critical weaknesses

NA

Questions for authors

NA

Reviewer_WjuGOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper trains small GPT models (about 42M parameters) from scratch on a toy domain of linear equations over ℤ₇, with an exact verifier. Leaving the x+y family out of pretraining means the model never generates it. Including it gives seeds with similar loss but widely varying verified allocation A_N (4.1–59.8%), and one seed stays starved under every ablation. At large sampling budgets, unique-key coverage saturates even when A_N is low. The authors conclude that discovery reports should include A_N alongside loss and yield.

Critical strengths

None

Critical weaknesses
  1. Figures don't match the reported data, and the main figures are partly synthetic. The supplement's figure script (scripts/analysis/render_all_sota_figures.py) reads no result files; all plotted values are hardcoded constants.
  • Figure 3 values contradict Table 2 for several seeds:
SeedFigure 3 (A_N / A_N^par)Table 2 (A_N / A_N^par)
032.6 / 36.7%33.1 / 45.2%
136.6 / 40.5% (48 keys)37.0 / 41.9% (49 keys)
323.5 / 23.9%23.6 / 49.2%
110 / 0 (shown as "Parse")19.2 / 61.7%
  • The abstract's A_N^par maximum of 72.4% can't be traced. It appears in no table, and the figure data tops out at 67.2%. Seeds 12–23 are never tabulated.
  • The Figure 3b regression line is hardcoded (y = −189.9x + 83.1). That's a clearly negative slope, yet the badge reads "R² = 0.000 (Orthogonal)". By my rough estimate, a slope that size over the plotted loss range is inconsistent with R² = 0.000. The central "loss is orthogonal to allocation" claim needs to be recomputed from the raw data and reported with the actual fit.
  • Figure 1B's coverage curves are closed-form exponentials, not observed trajectories, and nothing in the figure says so. Its labels (A_N = 20% vs 5%) also don't match the caption's "8.5× gap".
  • Figure 1A shows an equation the system can't express. It illustrates "active exploration" with (x+y)² = x²+2xy+y², but the DSL is linear (A.1) and cannot multiply variables.
  1. The reproducibility claims can't be checked.
  • Appendix A.16 points to results/round2_, frozen catalogs, and dated research/.md records. None are in the supplement, so submission_audit.py can't be run.
  • The supplement README describes directories that don't exist (src/grammar/, src/verifier/, src/models/).
  • Most reported runs are a single generation run per checkpoint, and the historical discovery runs weren't RNG-seeded (L751).
  1. Presentation.
  • The core idea is never stated plainly.
  • The abstract is a dense list of numbers.
  • There's a lot of notation (A_N, A_N^par, Vis_N, U_N, U_surface, U_canonical, U_sem, L_tf, L_P, B, D, R, S, I, C, P).
  • Undefined internal jargon ("historical," "legacy key," "frozen dumps," "round2_t2") and loaded labels ("mirage," "starvation collapse regime," "methodological firewall").
  • Appendices A.11 (ophthalmology retrieval) and A.13–A.15 (pilots) don't support the claims and should be removed or justified.
  • I found the paper very hard to follow, even after several readings.
Questions for authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.