acceptodds
Under review as a conference paper at ICLR 2027

Learning what to preserve: Grammar constrained search across scientific domains

Abstract

Scientific strings encode structure directly in their notation. A chemical formula such as specifies elemental identities and stoichiometric quantities, while a protein mutation such as M105K specifies residues and a positional coordinate. Yet pretrained models typically process such inputs using tokenizers learned from corpus statistics rather than from the structure of the notation itself. Existing solutions often introduce domain-specific parsers, vocabularies, or tokenizers, requiring a new representation strategy for each scientific domain. We ask whether a shared input representation rule can instead be discovered from downstream tasks across domains with different structural requirements. We introduce GCMS, a grammar-constrained multi-domain search framework that uses a policy gradient search over interpretable input rewrites. Joint search on crystal-system classification and protein mutation fitness discovers a common representation without an element inventory, residue classifier, or domain-specific parser. The learned rule transfers across unseen tasks and notation systems, including held-out materials properties, additional protein assays, modified peptides, and genomic variants. GCMS reduces band-gap MAE from to relative to WordPiece and reaches Spearman on the held-out ProteinGym BLAT assay. These results show that transferable structure in scientific notation can be discovered from task supervision rather than engineered separately for each domain.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
AI for Science, Multi-domain learning, Materials informatics, Domain-specific tokenization, Life sciences

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_GyJMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether one input rule can represent scientific strings across domains. GCMS uses grammar-constrained policy-gradient search and a noise-normalized soft-min reward. The rule is learned on materials and protein tasks, then evaluated on held-out properties and biological notations. Results show improved material predictions but mixed biological transfer.

Critical strengths
  1. The noise-normalized soft-min reward addresses conflicts between domains. Ablations show its advantages over mean and hard-min objectives.
  2. The paper compares multiple tokenizers and search strategies and includes ablations on reward design, grammar choices, and cross-domain supervision.
  3. GCMS reduces band-gap MAE from 0.6161 to 0.4918 compared with WordPiece and performs well across several materials benchmarks.
Critical weaknesses
  1. On ClinVar, GCMS only improves accuracy from 0.8183 to 0.8221. Peptide performance also requires manually relaxing the learned length bound from the search setting.
  2. GCMS performs worse than WordPiece on BRCA1 (0.5874 vs. 0.6092). Simple character splitting also matches or exceeds GCMS on several materials tasks.
Questions for authors

1.GCMS does not clearly outperform identity on ClinVar (0.8221 vs. 0.8183), performs worse than WordPiece on BRCA1, and requires changing the learned length bound for its best peptide result. Do these results support a general transfer claim, or only transfer under specific conditions? 2. Figure 5 suggests that much of GCMS's search advantage comes from behavioral deduplication and diversity. Can the authors show that policy learning itself provides additional gains over uniform random search with the same deduplication and number of unique rule evaluations? 3. The grammar already includes operations for preserving numeric spans and separating letters. How much of the discovered structure comes from learning rather than these predefined choices? What evidence shows that the learned segments are scientifically meaningful?

1 comment
AuthorsOpenReview ↗

We demonstrate that GCMS learns structural transformations that generalize across scientific domains and held-out tasks. Comparisons with uniform sampling show improved proposal quality, while grammar ablations and expanded search experiments demonstrate that the learned behaviour does not depend on domain-specific heuristics or narrowly defined grammar choices.

We thank the reviewers for the questions. The answers are given below.

1. GCMS learns a shared input transformation using a noise-normalized soft-min reward across two scientific domains, discouraging gains in one domain at the expense of another and encouraging structural choices that generalize across scientific notations. For Cs6Na2Nb6H36O37, WordPiece fragments include ##2n, ##h3, and ##o3, mixing element symbols and stoichiometric coefficients. GCMS separates symbolic units while preserving numeric spans such as 36 and 37. For protein mutation A123G, WordPiece gives ['a1', '##23', '##g'], compared to GCMS ['a', '123', 'g'] which preservs the residue position.

Further, the rule learned from crystal classification and ProteinGym PSEAI transfers to held-out tasks: band-gap MAE decreases from 0.6161 to 0.4918 and modified-peptide iRT Spearman correlation improves from 0.8907 to 0.9649 after relaxing the detector length bound for longer sequences, without changing the learned segmentation policy. Additionally, applied without re-search to molecular SMILES on BBBP, the rule improves accuracy from 0.8846 to 0.9024 over WordPiece. The authors would like to mention that we do not claim universal improvements. We acknowledge that the ClinVar gain is within evaluation variability, and GCMS underperforms WordPiece on BRCA1. Empirically, the results show transfer when the learned structural primitives apply to the target notation.

2. We compared GCMS with true uniform sampling, holding grammar, behavioural deduplication, caching, exclusions, and evaluation budgets identical across three search seeds. For crystal + PSEAI, mean proposal reward improves from -2.581 (uniform) to -0.863 (GCMS). Proposals within 1.5 reward units of the best observed reward rise from 7.6% to 31.2% and proposal-reward standard deviation decreases by a factor of 2.09 during GCMS optimization versus 1.21 for uniform, showing concentration toward higher-reward rules (table below).

Search settingΔ\Delta Mean RStd. gainPaired t(2)t(2)Δ\Delta Best R
Crystal + PSEAI+1.718+0.5629.86-0.012
Crystal + Peptide iRT+0.767+0.1581.30+0.644
Crystal only (blind)+1.687+0.3353.28+0.028
Crystal only (case-aware)+2.355+0.3647.33+0.062

In the table, Δ\Delta denotes the gain from GCMS and the standardized gains use pooled proposal-reward SD. Each of the four settings uses three matched-budget search seeds. Importantly, GCMS raises mean proposal reward in all 12 runs, with standardized gains in the range 0.158 to 0.562. Further, for case-aware crystal search, proposals with normalized reward z≥14z\geq14 increase from 8.2% to 17.9%. From the first to second half of optimization, mean proposal reward improves by 3.71 for GCMS versus 1.08 for uniform sampling. The results support the finding that although the best of budget rewards remain comparable, policy learning improves sampled-rule quality beyond shared caching and deduplication.

3. The authors would like to mention that the grammar contains no domain-specific knowledge, chemical element lists, residue definitions, or scientific parsing heuristics. It constrains search through generic string operations and GCMS learns useful operations and combinations from downstream scientific task performance. Interestingly, even after removing four digit-handling operations (group_digits, numeric_suffix, by_prev_class, preserve_numeric) while retaining generic separation and \d+ matching, GCMS is able to recover the same functional segmentation behaviour, showing it does not depend on explicitly providing the preserve_numeric grammar.

Across 2,900 evaluations, 2,225 distinct rules, and 50 soft-min searches, GCMS achieves a maximum reward of 2.729 via multiple digit-handling strategies: preserve_numeric, separate, and group_digits. Their probabilities of appearing among the top 10% of evaluated rules are 17.1%, 21.7%, and 21.0% respectively. Thus, useful numeric-span preservation can be recovered via different grammar configurations rather than one predefined operation.

Further, we enlarged the grammar with five generic, case-blind distractor transitions (letter_punct, punct_letter, alnum_punct, punct_alnum, punct_punct) and eight distractor regex pattern sets: [aeiou]+ / [^aeiou]+; .{3,4}; .{1,3}; [a-m]+ / [n-z]+; [0-4]+ / [5-9]+; [^0-9]+; .{2,3}; [a-z]{1,2} / [^a-z]+. These have no domain-specific knowledge and expand reachable behaviours from 842 to 1,096. Despite this larger search space, GCMS recovers the same digit-preserving rule family in all three search seeds and achieves higher mean evaluated-rule reward than uniform sampling (-0.869 vs. -2.182). The evidence supports recovery of useful structural transformations without a narrowly specified grammar by GCMS.

Reviewer_KTUCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GCMS (stands for grammar constrained multi-domain search), which uses policy gradient over input rewrites to arrive at a grammar over inputs. Previous work is usually based on frequency tokenization. This new approach starts with a 'bag' of rules. A policy is then evolved over input writes such that it increases the probability of high-reward rules. The approach is test on two families of tasks in materials and life sciences.

Critical strengths
  • The approach can allow for interpretable rules that respect the data for each domain. It can also compose multiple rules.
  • The approach is still plug and play in the sense that it modifies the input string before it is tokenized, so it doesn't change the vocabulary, and the transformer architecture.
  • The experimental procedure and space is comprehensive for the two groups of settings that they consider.
Critical weaknesses
  • The advantage here is also the downside. The interpretability that the rule design space provides could also be the limitation. Perhaps a free form rewrite and RL search would lead to the emergence of rules that are more computationally performant and relevant for the area.
  • I have some other questions relating to the results that I ask in the next section.
Questions for authors
  • I noticed that the improvement over unigram is quite small. For example, on protein mutation fitness, Unigram is very competitive. There is not a convincing general advantage for GCMS across protein fitness assays. GCMS generally works across multiple domains, but a well-established tokenizer already achieves comparable results on many tasks. Given the complexity of the RL procedure, what is the advantage, in practice of using this approach?
  • Following on the above, what is the justification for keeping the regular tokenizer if the input should be edited? Shouldn't the tokenizer change to take advantage of the new grammar that is applied on the inputs?
  • to what extent are the learned rules specific to WordPiece? Could the authors evaluate whether the same discovered grammar improves other tokenizers, such as Unigram, without retraining the rule-selection policy? This would help distinguish transferable scientific structure from preprocessing rules that compensate for limitations of a particular tokenizer.
1 comment
AuthorsOpenReview ↗

We show that Unigram can merge chemically distinct elements, leading to larger prediction errors. GCMS improves downstream performance while retaining pretrained tokenizers. We further demonstrate that the learned transformation transfers across different tokenizers without additional search or rule modification.

We thank the reviewers for the questions. Our responses are below.

1. Unigram performs competitively on several tasks, but can merge chemically distinct elements, which is associated with larger prediction errors. For Er2Fe14B, Unigram gives ['e', 'r2fe1', '4', 'b'], mixing symbols and numbers and splitting 14. While, WordPiece + GCMS gives ['e', 'r', '2', 'f', 'e', '14', 'b'], separating symbolic units and preserving numeric spans. Unigram also splits 20 in Cs3Cu20Te13 into ['cu2', '0'].

Among 9,646 unique formulas from matbench_perovskite, Unigram incorrectly combines multiple elements for 41.7% formulas, example of in Ag2NOF, mono in AgMoNO2, and are in BaReN2O. Among 920 held-out band-gap formulas, Unigram merges element symbols in 170 formulas (18.5%). On this subset, WordPiece + GCMS reduces MAE from 0.6168 to 0.4585 eV while, on the other 750 formulas, performance is similar (0.4586 vs. 0.4684 eV). Further, the difference between subgroup performance gap is 0.1682 eV. For NaLaS3 (true band gap 2.610 eV), Unigram predicts −0.084 eV, versus 2.563 eV for GCMS. Thus, GCMS is beneficial for chemical formulas where Unigram merges distinct elements.

In the main benchmarks, GCMS improves over Unigram on crystal classification (0.6147 vs. 0.6063), band-gap MAE (0.4918 vs. 0.5230), and peptide iRT Spearman (0.9649 vs. 0.9459, after relaxing the detector length bound). Importantly, GCMS is complementary to Unigram. Applying the discovered rule directly, without any additional search, improves Unigram crystal accuracy from 0.6087 to 0.6323, exceeding either method alone.

2. Retaining the pretrained tokenizer is a central GCMS design choice, not a limitation. Scientific literature mixes natural language and structured notation. Frequency-based tokenizers work well for natural language but may fail to preserve scientific structure. GCMS learns task-guided input transformations while keeping the existing tokenizer vocabulary and pretrained encoder initialization, avoiding training a new tokenizer or pretraining an encoder from scratch.

Moreover, training a tokenizer on scientific text does not necessarily preserve scientific structure. For the formula Ca12Al11Si3Cl5O32, MATTER gives ['ca1', '##2al', '##11', '##si', '##3c', '##l', '##5', '##o3', '##2'] while the proposed GCMS method gives ['c', 'a', '12', 'a', 'l', '11', 's', 'i', '3', 'c', 'l', '5', 'o', '32'], separating symbolic units and preserving numeric spans. For B10H14S, MATTER gives ['b1', '##0', '##h1', '##4s'] whereas GCMS gives ['b', '10', 'h', '14', 's']. GCMS also outperforms MATTER on all seven evaluated materials tasks, including crystal classification (0.6147 vs. 0.5804) and band-gap MAE (0.4918 vs. 0.6144).

Moreover, separate scientific tokenizers need suitable training corpora and domain-specific vocabulary choices that may not generalize across notations. An advantage of learning input transformations is that the same rule can be applied across different tokenizers without retraining the rule-selection policy. For example, applying the WordPiece-discovered rule directly to Unigram improves crystal accuracy from 0.6087 to 0.6323. The rule also transfers to scientific notations not used during search, including modified peptides and genomic variants.

3. We applied the same WordPiece-discovered rule to MATTER, BPE, and Unigram without retraining the search policy or modifying the rule. Each uses its own pretrained encoder, fine-tuned on rewritten input. The reported results (table) are averaged over three evaluation seeds. Identity is the original tokenizer input while +GCMS uses the same fixed rewrite before tokenization.

TokenizerCrystal ↑ IdentityCrystal ↑ +GCMSPSEAI ↑ IdentityPSEAI ↑ +GCMSClinVar ↑ IdentityClinVar ↑ +GCMS
WordPiece0.58480.61470.85660.87540.81830.8221
Unigram0.60870.63230.87280.87330.80450.8057
BPE0.59160.61090.82830.87180.77050.7988
MATTER0.58040.61450.85330.87210.82100.8223

The results show that the learned transformation is not specific to WordPiece. On crystal classification, GCMS improves MATTER (+0.0341), WordPiece (+0.0299), Unigram (+0.0236), and BPE (+0.0193). On ProteinGym PSEAI, GCMS gives the greatest gain for BPE (+0.0435), followed by WordPiece and MATTER (+0.0188 each). Unigram improves marginally (+0.0005), consistent with its competitive baseline. This supports the finding that frequency-based tokenizers can obscure meaningful boundaries in scientific notation while GCMS addresses this with downstream scientific supervision. Larger BPE gains on PSEAI suggest particular utility when tokenizer segmentation is less effective. Importantly, a rule discovered using one tokenizer improves independently trained tokenizers without additional search.

Comments on the paper

2 comments
AuthorsOpenReview ↗

We show that GCMS learns to sample better rules than uniform sampling, although best-of-budget results are comparable. We compare GCMS with fixed heuristics, verify numeric-span preservation, and show why soft-min aggregation better handles failures across domains.

We thank the reviewers for the questions. The answers are given below.

1. We compared the learned GCMS policy with true uniform sampling and its frozen initialization using matched grammar, evaluation budgets, caching, and behavioural deduplication across three search seeds. For crystal + PSEAI, mean proposal reward is -0.863 for GCMS, compared to -2.581 for uniform sampling and -2.667 for frozen initialization, positive in all three seeds. Further, proposals within 1.5 reward units of the best observed reward increase from 7.6% to 31.2%. Proposal-reward spread decreases by a factor of 2.09 during optimization, versus 1.21 for uniform sampling (table below).

Search settingΔ Proposal RStd. gainPaired t(2)Δ Best R
Crystal + PSEAI+1.718+0.5629.86-0.012
Crystal + Peptide iRT+0.767+0.1581.30+0.644
Crystal only (blind)+1.687+0.3353.28+0.028
Crystal only (case-aware)+2.355+0.3647.33+0.062

In the table, Δ\Delta denotes the gain from GCMS and standardized gains use pooled proposal-reward standard deviation. Importantly, Proposal gains are positive in all 12 paired runs. For the case-aware crystal-only search, the fraction of proposals with crystal reward z≥14z\geq14 rises from 8.2% to 17.9%. Between the first and second halves of search runs, proposal reward improves by 3.71 for GCMS against 1.08 for uniform sampling. Enlarging the grammar with generic case-blind distractor transitions and regex patterns increases reachable behaviors from 842 to 1,096, GCMS recovers digit-preserving behavior in all three seeds and its mean evaluated-rule reward exceeds uniform sampling. The proposal-quality gains across four settings and improvement during optimization show that GCMS learns to focus evaluations on higher-reward rules rather than relying solely on uniform exploration.

The authors acknowledge that uniform sampling can find comparable best-of-budget rules (table below), possibly because the grammar contains multiple high-performing transformations.

MetricGCMS learned policyUniform sampling
Mean best-so-far reward2.4378 ± 0.12912.5339 ± 0.3364
Final crystal accuracy0.6094 ± 0.00460.6224 ± 0.0294
Final PSEAI Spearman (ρ\rho)0.8735 ± 0.00160.8726 ± 0.0000

2. The learned rule uses an alphabetic detector (input length 4–32 characters), character-level segmentation for eligible inputs, and grouping of consecutive digits into intact numeric spans. It preserves balanced expressions enclosed in parentheses, square brackets, or braces and allows boundary splitting. The authors acknowledge that a fixed character-splitting heuristic preserving contiguous digit runs performs comparably on most tasks. However, GCMS is marginally better on ProteinGym NEIME (0.1593 vs. 0.1459) and PSEAI (0.8754 vs. 0.8726). The authors would like to point out that the heuristic was specified retrospectively based on the discovered rule. Further, digit-run preservation does not preserve decimals: F e 0.5 N i 0.5 becomes F e 0 . 5 N i 0 . 5, potentially losing fractional-stoichiometry information.

Actual WordPiece outputs after GCMS rewriting confirms that all of the measured numeric spans remain single tokens for crystal formulas (5,384 spans) and PSEAI mutations (972 spans), although we acknowledge preservation in rewritten strings does not guarantee token-level preservation in general. The tokenizer vocabulary is unchanged as shown in the table below.

InputRewritten inputWordPiece tokens
A123GA 123 Ga, 123, g
Fe12O19F e 12 O 19f, e, 12, o, 19

3. Table 7 shows the limitation of mean aggregation wherein plain character splitting gains +9.53σ on crystal classification but degrades ProteinGym by -3.58σ. Mean aggregation still assigns it +2.97, above identity, whereas soft-min assigns -3.58. Another rule attaining a higher mean reward of 6.32 does not change the argument: mean aggregation can offset a major failure in one domain with gains in another. Soft-min penalizes this failure while remaining sensitive to improvements across both domains. Repeated fixed-rule evaluations show consistent behaviour across search seeds (table below):

Rulen runsCrystalProteinGymMeanSoft-min
Charsplit + preserve_numeric6+9.845 ± 0.096+2.729 ± 0.000+6.287 ± 0.048+2.728 ± 0.000
Plain charsplit3+9.836 ± 0.000-3.579 ± 0.000+3.128 ± 0.000-3.579 ± 0.000

Values are mean ± standard deviation. Our claim is not that mean aggregation necessarily selects character splitting, but that it permits cross-domain compensation.

4. The standard deviation values reported in Table 9 are correct. We will modify the manuscript accordingly.

AuthorsOpenReview ↗

Summary of Revisions and Thank you for all the constructive reviews.

We thank all reviewers for their constructive feedback. We have addressed the concerns raised by all three reviewers through additional experiments, analyses, and clarifications. The main findings are summarized below.

1. Scientific notation and pretrained tokenizers (Reviewers KTUC, GyJM): We analyzed 9,646 chemical formulas and found that 41.7% contain Unigram tokens spanning multiple elements. For Er2Fe14B, we observed that Unigram splits the coefficient 14, whereas WordPiece + GCMS preserves it. In 920 held-out band-gap formulas, we found that Unigram merges elements in 170 (18.5%). On this subset, WordPiece + GCMS reduces MAE from 0.6168 to 0.4585 eV. Performance is similar on the remaining 750 formulas (0.4586 vs. 0.4684 eV), with a 0.1682 eV difference between subgroup performance gaps. In our benchmarks, GCMS improves over Unigram on crystal classification (0.6147 vs. 0.6063), band-gap MAE (0.4918 vs. 0.5230), and peptide iRT Spearman (0.9649 vs. 0.9459, after relaxing the detector length bound). We show that GCMS outperforms MATTER across all seven evaluated materials tasks. We clarify that retaining the pretrained vocabulary and encoder initialization is a central design choice of GCMS, avoiding tokenizer retraining or encoder pretraining from scratch.

2. Cross-tokenizer and cross-domain transfer (Reviewers KTUC, GyJM): We learn a shared transformation using a noise-normalized soft-min reward across two scientific domains. We applied the WordPiece-discovered rule to Unigram, BPE, and MATTER without re-search or rule modification. Crystal classification improves from 0.6087 to 0.6323 for Unigram, 0.5916 to 0.6109 for BPE, and 0.5804 to 0.6145 for MATTER. The Unigram combination exceeds either method alone. On PSEAI, BPE improves from 0.8283 to 0.8718. We evaluated transfer to held-out tasks. Band-gap MAE decreases from 0.6161 to 0.4918. Modified-peptide iRT Spearman improves from 0.8907 to 0.9649 after relaxing the detector length bound, without changing the rule. BBBP accuracy improves from 0.8846 to 0.9024 without re-search. We acknowledge that GCMS does not improve on every task. ClinVar gains are within evaluation variability, and GCMS underperforms WordPiece on BRCA1.

3. Policy learning versus uniform sampling (Reviewers GyJM, 3cPd): We compared GCMS against uniform sampling using matched search grammar, evaluation budgets, caching, and deduplication. GCMS improves mean proposal reward in all 12 paired runs across four settings. For crystal + PSEAI, mean proposal reward is -0.863 for GCMS, compared with -2.581 for uniform sampling and -2.667 for frozen initialization. Proposals within 1.5 reward units of the best observed reward increase from 7.6% to 31.2%. Mean proposal reward improves by 3.71 between the first and second halves of GCMS optimization, compared with 1.08 for uniform sampling. The proposal-reward spread decreases by a factor of 2.09 during GCMS optimization, compared with 1.21 for uniform sampling. We also acknowledge that best-of-budget rewards remain comparable.

4. Grammar robustness (Reviewer GyJM): We clarify that our grammar uses generic string operations without chemical element lists, residue definitions, domain-specific knowledge, or scientific parsers. We removed four digit-handling operations and recovered the same functional segmentation behavior. Across 2,900 evaluations, 2,225 distinct rules, and 50 soft-min searches, we found that multiple digit-handling strategies reach the maximum reward of 2.729. We enlarged the grammar from 842 to 1,096 reachable behaviors and recovered the digit-preserving rule family across all three seeds. GCMS also achieves higher mean evaluated-rule reward than uniform sampling (-0.869 vs. -2.182).

5. Fixed heuristics and token verification (Reviewer 3cPd): We compared GCMS against a fixed digit-preserving character-splitting heuristic and found comparable performance on most tasks. GCMS performs better on NEIME (0.1593 vs. 0.1459) and PSEAI (0.8754 vs. 0.8726). We clarify that the fixed heuristic was specified retrospectively based on the discovered rule. We verified that 100% of measured numeric spans remain single WordPiece tokens, covering 5,384 crystal and 972 PSEAI spans. We clarify that preserving consecutive digits does not preserve decimal values.

6. Reward aggregation and reporting (Reviewer 3cPd): We examined the limitation of mean aggregation using plain character splitting, which gains +9.53σ on crystal classification but loses -3.58σ on ProteinGym. Mean aggregation still assigns it +2.97, compared with -3.58 for soft-min. We verified this behaviour through repeated fixed-rule evaluations. We clarify that our claim is not that mean aggregation necessarily selects character splitting, but that it permits cross-domain compensation. We confirm that the standard deviation values in Table 9 are correct.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.