acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking Symbolic Rule Discovery for Biological Motifs

Abstract

Existing methods for symbolic discovery primarily focus on recovering mathematical equations from numerical data, leaving it unclear whether they can induce valid logical rules from discrete sequences. We introduce SeqRule-Bench, a benchmark designed for symbolic rule discovery in biological motifs. Unlike conventional biological sequence analysis tasks formulated to classify or score sequences, it formulates each task as rule induction from labeled positive and negative sequences, requiring outputs in a human-readable domain-specific language. SeqRule-Bench integrates real biological data with synthetic tasks featuring ground-truth rules and informative negatives, addressing the scarcity of verified rules and the difficulty of obtaining negatives. To support systematic evaluation, we propose comprehensive metrics to analyze the trade-offs between accuracy, interpretability, and cost, including a novel boundary accuracy metric. By evaluating a broad range of methods, the benchmark reveals that even when these methods achieve high accuracy on some tasks, they still exhibit a gap in boundary accuracy and incur higher costs. The benchmark will be fully open-sourced upon publication.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
rule discovery, rule induction, biological motifs, domain-specific language, counterfactual consistency, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RhFxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a novel benchmark - SeqRule-Bench - that revolves around testing the ability of a wide variety of systems to inductively generate a rule for classifying whether or not a given sequence fulfills a criterion from positive and negative examples. SeqRule-Bench includes real biological sequences (both DNA and proteins) with ground-truth classification rules, along with synthetically generated counterparts inspired by real biological-motifs. The synthetic examples used in evaluation vary widely in difficulty and vocabulary size. The paper additionally introduces the concept of boundary accuracy - the ability of a system's learned rule to distinguish cases at the edges of the ground-truth rule that a heuristic pattern-matcher would miss. This is designed to distinguish statistical learning from hard rule induction. The paper then evaluates the aforementioned systems on SeqRuleBench and analyzes how both general accuracy and boundary accuracy change with respect to system type (LLM few-shot, LLM in a harness, traditional rule inductors, and black box predictors), and problem difficulty (rule complexity, vocabulary size). The paper concludes that while LLM's can approach the accuracy of black box predictors for rule induction, this doesn't necessarily lead to the learning of the true underlying rules.

Critical strengths
  • The paper evaluates a diverse variety of systems - including real Agentic harnesses, classic LLM setups from the literature, and symbolic rule inductors. They also run black-box controls such as finetuning DNABert-2 to learn each rule. This addresses a common confound in genomic-LM research: conventional baselines performing nearly as well if not better than more extensive LLM setups.
  • The boundary accuracy metric is extremely clever way of distinguishing rule induction from pattern-matching - it is the strongest point of the paper. It is intuitive and easy to understand, yet clearly delineates the difference - a true rule will properly distinguish cases that vary only slightly but in meaningful ways, while a global pattern matcher will fail to notice the crucial but small differences.
Critical weaknesses
  • The presented results in Table 3 have wildly overlapping confidence intervals - for the real subset of SeqRule-Bench, these are typically around ~0.1 in either direction for most results - making it hard to distinguish between the middle performers in the 0.3-0.5 range. However, the confidence intervals are far wider in the synthetic subsets - often ~0.25 in either direction. This makes distinguishing performance very difficult, beyond distinguishing methods that completely fail from those that have partial success. Running more trials to reduce these intervals, or setting up a more controlled comparison, could help. Throughout the rest of the paper, confidence intervals remain a significant issue - Figure 5 (an accuracy comparison) lacks them entirely. As does Figure 7 - which tries to analyze Few Shot GPT-5.4 performance with respect to vocab size and data complexity. Given the extremely large confidence intervals in the Table 3 for the synthetic regime, it is unclear whether the trends in Table 7 mean anything - or are just reflective of noise. Proper statistical analysis would address this - which the paper does not include.
  • The overall conclusion of the paper is fairly banal: that models in agentic coding harnesses do best on rule induction. The more interesting analyses - like properties of boundary accuracy or how model performance interacts with vocab size/complexity - are relegated to the appendix or lack statistical power. Furthermore, the paper fails to tell a convincing story and instead appears as a collection of observations, only some of which are connected - and are often listed like a laundry-list of findings rather than merged into a coherent conclusion.
Questions for authors
  • Did you test the black-box baselines for boundary accuracy? I know they technically can't be tested as described, as they don't output an inductive rule - but nevertheless, you could treat their classification of the boundary points as an application of their learned 'rule' and report that. I'd be extremely interested.
  • You remark, in 4.2, that "furthermore, although high accuracy is correlated with better boundary accuracy to some extent, they are not strictly proportional." I'd be very interested to know to what extent the correlation exists, whether its statistically significant, etc. - this is worth elaborating on!
  • Furthermore, in 4.2, you remark "Finally, increasing the number of input examples brings only a modest improvement in accuracy." Quantify this, with appropriate statistical tests - it would make 4.2 much more compelling. Similarly, regress accuracy against logic constraints and input alphabet.
  • Finally, in Table 3, the confidence intervals could be narrowed by running more trials per-model. In some cases, however, I understand that this might be prohibitively expensive. I could see either running more trials for cheaper baselines to distinguish them, or adjusting the table to directly test paired differences - which may offer more power than raw confidence intervals.

If all the above concerns were addressed, I could easily see my assessment going from "weak reject" to "weak accept".

5 comments

Question 1: Did you test the black-box baselines for boundary accuracy? I know they technically can't be tested as described, as they don't output an inductive rule - but nevertheless, you could treat their classification of the boundary points as an application of their learned 'rule' and report that. I'd be extremely interested.

Response: Thanks for the question, here we complement the black-box baselines experiments. All values are reported as percentages (%).

DatasetESM BoundaryAccCNN BoundaryAccResNet BoundaryAccBoundaryCov
ps_structured (PS-S)4.665.223.5590.92
V1-s—30.8127.3180.95
V1-m—35.430.45100
V1-c—24.820.7490.71
V3-s—55.6851.4591.43
V3-m—20.0919.7885.78
V3-c—18.217.6281.71
V5-s—39.739.5480
V5-m—41.2437.8893.56
V5-c—33.3531.4488.63

The remaining rows retain the previously retrained results; V3 uses the corrected vocabulary. “—” indicates that the model was not evaluated.

Question 2: You remark, in 4.2, that "furthermore, although high accuracy is correlated with better boundary accuracy to some extent, they are not strictly proportional." I'd be very interested to know to what extent the correlation exists, whether its statistically significant, etc. - this is worth elaborating on!

Response: Thank you for the question. We have quantitatively analyzed the relationship between the two metrics using the latest data. The Spearman rank correlation is ρ = 0.8434, indicating a strong positive monotonic association.

Accuracy and BoundaryAcc are highly correlated: Accuracy reflects the overall trend in boundary performance, but their rankings are not identical. Higher Accuracy does not always correspond to higher BoundaryAcc, and differences remain in both scores and rankings. Kendall’s τ-b = 0.6722 further supports broad agreement between the rankings. When comparing results within the same subset and under the same input budget, 483 of the 3,356 pairwise comparisons excluding ties exhibit ranking reversals, yielding a reversal rate of 14.39%.

These quantitative results support the conclusion that the two metrics are highly correlated but not fully interchangeable. Beyond producing numerically different scores, BoundaryAcc captures distinctions that Accuracy does not fully reflect.

Question 3: Furthermore, in 4.2, you remark "Finally, increasing the number of input examples brings only a modest improvement in accuracy." Quantify this, with appropriate statistical tests - it would make 4.2 much more compelling. Similarly, regress accuracy against logic constraints and input alphabet.

Response: For the statistical test of how accuracy scales with the number of samples, we reanalyzed the archived input-size experiments, evaluating the same 250 ground-truth (GT) tasks under three training-set sizes: 400, 800, and 1,600 examples. Increasing the input from 400 to 1,600 examples yielded a mean accuracy change of +0.00556, with a task-level bootstrap 95% confidence interval of [−0.006100, 0.017372]. A paired t-test yielded p = 0.351. In this sample, the change from 800 to 1,600 examples was near zero (paired t-test, p = 0.998).

Thus, within the tested input range, the experiment did not detect a statistically significant improvement in accuracy.

And for the second suggestion, using the records from Figure 7 in Section 4.2, we controlled for input size and computed standard errors clustered by rule. The regression results are as follows:

Predictor changeChange in Accuracy [95% CI]p-valueChange in BoundaryAcc [95% CI]p-value
One additional logical constraint−0.520 [−1.271, 0.232]0.1744−0.253 [−1.511, 1.005]0.6923
One additional alphabet symbol+0.173 [0.014, 0.333]0.0335+0.360 [0.090, 0.630]0.0091

The regression model has R2=0.020R^2 = 0.020, explaining approximately 2.08% of the variation in Accuracy.

As shown in the table, the number of logical constraints shows no statistically significant linear association with accuracy, while alphabet size shows a weak positive linear association.

A very helpful baseline showing the power of LLM-based systems relative to black-box classifiers - thank you for providing! Maybe include as an appendix in final paper?

Reviewer_6zkDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose SeqRule-Bench, a benchmark for inducing human-readable symbolic rules for biological sequence motifs (DNA and proteins) from labeled positive and negative examples. In SeqRule-Bench, methods express rules using a DSL (a small language for rule description) that extends the pattern notation of the PROSITE protein motif dictionary, writing them as pairs of local patterns and logical constraints (e.g., identical characters at two positions, complementary characters, conditional constraints, and upper and lower limits on character count) (§3.1, Figure 2).

The SeqRule-Bench dataset consists of sequences generated from real-world data This includes the protein task (PS) derived from PROSITE and the binding site task (MA) derived from the JASPAR database of transcription factor binding motifs, as well as synthetic data (V1–V5, consisting of 5 to 26 letters) generated by constructing sequences from DSL rules produced by LLMs and supplemented with challenging negative examples near the boundaries (§3.2–3.3). As evaluation metrics, the authors propose a new metric, BoundaryAcc, in addition to accuracy, rule complexity, and token cost. BoundaryAcc is the proportion of pairs, where a positive example sequence is modified by only 1–2 characters to invert the classification on the correct rule, that the learned rule classifies correctly (Section 3.4, Appendices A.2–A.3).

The methods compared by the authors include few-shot LLMs, iterative refinement approaches (Self-Refinement, Hypothesis Refinement), LLM-SR (an adaptation of formula discovery methods that use LLMs to explore programs), coding agents (Claude Code is based on Claude Sonnet 4.6, and Codex is based on GPT-5.4), the grammar inference tool FlexFringe, genetic algorithms and simulated annealing (GA/SA) on a DSL, and black-box predictors (such as the protein language model ESM-2). The few-shot method runs on GPT-5.4, Claude Sonnet 4.6, and Gemini 3.1 Pro, while the iterative refinement and LLM-SR methods run on GPT-5.4 (§4, p. 7). According to the authors’ results, among methods that generate symbolic rules, Claude Code achieves the highest BoundaryAcc in many sequences and approaches the accuracy of black-box predictors in synthesis tasks. On the other hand, the authors report that while accuracy improves in the Few-shot setting, BoundaryAcc remains low (Figure 7); that the token cost of Claude Code is one order of magnitude higher than that of non-agent-based methods; and that LLM-SR exhibits low performance and high cost (Table 3, Figures 5–7).

Critical strengths
  1. The authors established an evaluation environment that integrates a DSL, an executor, a synthesis rule generator, and a boundary pair generator for the induction of symbolic rules from sequences. In this environment, users can independently adjust the alphabet size and the number of logical constraints in synthesis tasks where the correct rules are known; negative examples are also categorized into four layers: random, logical violations, single-character substitutions, and those near the boundaries (§3.3, Figure 4). Why this is effective for comprehensive evaluation: While benchmarks for regular expression induction (REI) and mathematical formula discovery already exist, there has never been a setting where real-world biological sequence data and synthesis tasks with ground-truth rules containing logical constraints could be evaluated using the same DSL (Table 2).

  2. This paper concretely demonstrated differences that cannot be captured by accuracy alone. Even within the same Claude Sonnet 4.6 model, the BoundaryAcc differs significantly between Few-shot (providing one answer based on context) and Agent (Claude Code) modes (0.28 → 0.64 on the PS-S real-world dataset derived from PROSITE, and 0.38 → 0.83 on the V3-s synthetic dataset with 15 letters and no logical constraints; Table 3, p. 8). Furthermore, LLM-SR performs poorly even when using approximately 9×1069\times10^6 tokens per rule (Figure 6, p. 8). Why this matters for overall evaluation: The observations in this paper can help researchers using LLMs for scientific rule discovery in selecting methods and estimating costs.

Critical weaknesses
  1. In this paper, regarding the central new metric BoundaryAcc, it is unclear which definition and which generator were used to calculate the values in the main result (Table 3), so the meaning of the numbers and their reproducibility cannot be determined. Equation (9) in the main text (§3.4, p. 7) uses a symmetric definition that counts matches at both ends, whereas Equation (20) in the appendix (A.3.1, p. 17) uses a definition that counts only in the positive-to-negative direction. If we apply the description immediately following Equation (20) (“swap the order of the two sequences within each pair (x, x′), yielding both positive-to-negative and negative-to-positive pairs,” p. 17) literally, even correct rules do not count pairs in the opposite direction, and the score does not equal 1. On the other hand, Appendix A.3.6 (p. 25) reports that all 73 correct rules yield a score of 1, creating a contradiction between the statements in the paper. In Appendix A.3.6 (pp. 25–26), the authors show that the impact of the definition change is small (average 0.462 → 0.457) for the 2,920 candidate rules that were corrupted for verification purposes; however, in the same section, they acknowledge that the old generator did not fix the seed and did not preserve boundary pairs, and Table 3 does not specify which generator was used. Why this affects the overall evaluation: The paper’s claim, that high accuracy does not necessarily mean the correct rules have been recovered, relies almost entirely on this metric and Table 3.

  2. In §4.1 (p. 8), the authors conclude that since accuracy improves when using an agent compared to the same base LLM, the harness (the agent’s execution environment) and the agent’s reasoning are important (“suggesting that harnessing and stronger agentic reasoning are important sources of improvement”). This conclusion is not distinguished from differences in the amount of input data. While few-shot and iterative improvement approaches incorporate only 800 examples into the context, the agent can analyze the entire training dataset using scripts (§4.1, p. 8; Table 15, p. 28). The authors themselves explain that the reason the agent is computationally expensive is “mainly because the agent reads the full training data, executes analysis scripts, and inspects intermediate outputs” (p. 9). The authors demonstrate that changing the input size for the few-shot method to 400, 800, or 1,600 examples yields only a small increase in accuracy (§4.2, Figure 7, p. 9); however, they tested only a single few-shot/GPT-5.4 method and did not conduct comparisons where the agent’s input was reduced. Furthermore, since the agent outperforms GA/SA and FlexFringe, which use the full dataset, the reviewer does not dispute the agent’s strength per se. Reason for impact on the overall evaluation: Since the main findings of benchmark papers are derived from comparisons between methods, confounding in the comparison conditions directly undermines the reliability of this paper’s conclusions.

Questions for authors
  1. [Clarification] Did the authors calculate the BoundaryAcc values in Table 3 and Figure 7 using Equation (9)/(13) (symmetric), the traditional “any-flip” method (where a pair is counted if the learning rule’s decision changes between the two arrays, regardless of orientation), or Equation (20) (directional). How should Equation (20) be combined with the description on p. 17, which states that the order of the pair is reversed to create pairs in both directions, to yield a score of 1 for the correct rule? Does Table 3 result from the original generator without a seed or the regenerated generator? The authors should present the scores for Table 3 recalculated using the regenerated boundary pairs. (Critical Weaknesses 1)

  2. [Clarification] Which subset of PROSITE (number of entries, selection criteria) does the real-data sequence PS-S in Table 3 represent, and are the correct rules in Table 3 the PROSITE patterns themselves? What is the average of how many rules for each column in Table 3? Furthermore, did the authors exclude rules for which no boundary pairs could be generated and the values became undefined, or did they set them to 0? (In Appendix Table 7, for some partitions of the synthetic domain “virtual1” (26 letters of the alphabet), 5–6 out of 10 rules were undefined.) Which LLM generated the synthetic rules? (If that LLM belongs to the same family as the one under evaluation, there may be a bias toward rules that are easier for the LLM to generate.) (Critical Weaknesses 1)

  3. [Clarification] (Affects evaluation) Even in a comparison with standardized input volumes (i.e., providing only 800 examples to the agents, or providing the agents in the iterative improvement system with a statistical summary of the entire training dataset), does the agents’ advantage over the same base LLM persist? Since the authors’ input volume experiment in Figure 7 is limited to few-shot scenarios and does not vary the agents’ input volume, I would like to ask the authors to supplement this comparison. If such a comparison is difficult, the authors may instead revise the conclusion in §4.1 to account for differences in input volume. Additionally, I ask the authors to specify the number of training data points per rule used in this experiment. In Table 14 (p. 27), the number of sequences per task ranges from approximately 45,000 to 100,000 for synthetic data, 300 to 17,683 for real data (PS), and 300 to 7,926 for real data (MA) (all figures represent the total for training and testing combined); therefore, the proportion of the training data represented by 800 to 1,600 examples is likely to vary significantly depending on the task. (Critical Weaknesses 2)

  4. [Clarification] In this paper, boundary pairs are generated by using training positive examples as seeds for the positive side (Appendix A.3.1, Step 2, p. 18). Therefore, the reviewers believe that rules that are too specific, tailored to the training positive examples, can achieve high BoundaryAcc, whereas rules that are too general, accepting distant negative examples, are unlikely to be penalized. Do the authors agree with this interpretation? If so, do they intend to explicitly state that BoundaryAcc is positioned as a complementary metric to be interpreted in conjunction with accuracy? Additionally, I ask the authors to explain the conceptual difference between BoundaryAcc and the “consistency” (the proportion of cases where both the original example and its edited version were correctly classified) of the contrast set used to measure boundaries with minimum-edit pairs (Gardner et al., 2020). This response relates to the reviewer’s evaluation of the novelty of BoundaryAcc (Informativeness). (Informativeness)

  5. [Suggestion] (Does not affect the evaluation) (1) Section 2 (p. 3) of this paper cites LLM-SRBench as Cava et al., 2021, and SIRBench-V1 as Hua et al., 2025; however, these are actually Shojaee et al. (ICML 2025) and Lin et al. (EMNLP 2025), respectively. The reference cited in this paper as “Deep Symbolic Regression (Landajuela et al., 2021)” is also a paper on symbolic strategies rather than symbolic regression (p. 11). (2) The authors state that DSL is designed to avoid soft rules such as symbol frequency (§3.1, p. 4: “designed to avoid soft rules, e.g., symbol frequency”); however, since DSL includes operators that impose upper and lower limits on the number of characters (MIN/MAX_SYMBOL_COUNT) and the generated prompts also contain constraints on GC content (Appendix A.8.1), the authors should align the wording of their design principles accordingly.

  6. [Suggestion] For the PS and MA tasks, there are established methods such as Pratt (Jonassen, Collins & Higgins, Protein Science 1995), which searches for PROSITE-type patterns in unaligned sequences, and STREME (Bailey, Bioinformatics 2021), which searches for motifs enriched in positive examples. Since the only methods used for comparison with the symbolic approach in this paper are FlexFringe and the authors’ own GA/SA, the authors should at least include Pratt for the PS task to clarify the positioning of LLM-based methods.

  7. [Suggestion] Since labels in the PROSITE task are determined by matches to publicly available patterns (§3.2, Figure 3), there is a possibility that the LLM being evaluated is answering based on memory. If the authors were to isolate the contribution of memory using a version where symbols are randomly substituted, and present a few examples where the rules generated by each method using PS-S and JASPAR are compared side-by-side with official PROSITE patterns or JASPAR consensus sequences, the paper’s claim that human-readable rules can be obtained would become more concrete.

  8. [Suggestion] As a supplement to Questions For Authors 3, the reliability of the comparison would be enhanced if the authors also demonstrated the impact of budget differences (Claude Code: 1,800 seconds, $5; Codex: 900 seconds; Table 16, p. 29) and the variability between runs when the experiment is repeated multiple times (the intervals in Table 3 are based on rule-level bootstrapping and do not include inter-run variability).

4 comments

Question 1: BoundaryAcc formulation used in Table 3 and Figure 7, the consistency between Equation (20) and the reversed-pair construction, and whether Table 3 uses the original or regenerated boundary pairs, with recalculated scores requested for the regenerated pairs.

Response: Thanks for pointing it out. Table 3 and Figure 7 report directional BoundaryAcc, as defined in Equation (20), which is exactly equivalent to the “both endpoints correctly classified” criterion in Equations (9) and (13) under a fixed pair ordering. The boundary pairs were generated using the reproducible generator.

All boundary pairs are stored in a positive-to-negative order, with ground-truth labels of (1, 0). A pair contributes to the numerator only when the model predicts exactly (1, 0). The reverse prediction, (0, 1), receives no credit. Therefore, the directional score and the symmetric “both endpoints correct” score are numerically identical.

We have removed the reluctant description of any-flip from Appendix.

Question 2: Clarification is needed on the size and selection criteria of PS-S, the ground-truth rules and number of rules averaged in each column of Table 3, the handling of undefined BoundaryAcc values, and the model used to generate synthetic rules.

Response: PS-S is a subset of PROSITE: PS-S in Table 3 comprises 22 independent tasks selected from PROSITE entries expressed in a hard-rule syntax. The selection criteria were that the tasks use only hard-rule syntax, cover as many types of PROSITE entries as possible (domains, families, etc.), have high precision and recall, and contain at least 150 positive examples after deduplication. PS-S can be extended to include other PROSITE entries that satisfy these criteria.

The correct rules are the PROSITE patterns themselves: The correct rules in Table 3 are the official patterns from the corresponding PROSITE entries, used directly as ground truth without modification.

Average number of rules: Each reported mean is computed over 22 rules for PS-S and 5 rules for each synthetic subset.

Handling undefined values: Rules for which no valid boundary pairs can be generated, leaving BoundaryAcc undefined, are excluded rather than assigned a score of zero.

Synthetic rule generation and potential generator advantage

We used Gemini Pro 3.1, the model used to generate the rules,and was also included among the evaluated models. The rules were therefore jointly constrained by the prompt requirements and human review, helping reduce implausible content and some potential biases. Thus, although the same model was used for both rule generation and evaluation, it served different functions.

The experimental results below provide indirect evidence against a clear advantage arising from this overlap.

Accuracy on real-world and synthetic tasks (%)

DatasetNumber of tasksGeminiClaudeGPT
AMP356.8353.4953.9
Gene1353.1956.4952.3
MA4656.1667.9762.25
PS4378.2673.8453.22
SP365.537459.71
All real-world tasks10864.8868.6957.16
Synthetic tasks4565.7967.8259.84
PS-S (subset)2285.9473.4257.29

The overall Accuracy ranking is identical across the 45 synthetic tasks and the 108 real-world tasks, with Claude > Gemini > GPT.

The BoundaryAcc ranking is also consistent, with Gemini > Claude > GPT. On PS-S, the only real-world subset for which BoundaryAcc can be computed, Gemini’s lead over the other models is even larger than on the synthetic tasks. This provides indirect evidence that the overlapping model does not show a clear advantage specific to the synthetic tasks.

Question 3: Clarification is needed on whether the agents’ advantage over the same base LLM persists under matched input volumes.

Response: Regarding differences in the information available to each method, we divide the methods into two categories: 1) requires placing examples directly into the context and cannot process the full dataset, so we uniformly sample the same number of examples. 2) can summarize patterns over the full dataset, so we retain access to all available data. We consider the ability to be part of a method’s capability.

We clarify that the agent does not directly read the raw data. Instead, it generates code to obtain statistical summaries of the data and then makes decisions based on those summaries. The answer is therefore yes: even when the agent has access only to data summaries, it significantly outperforms LLMs in the few-shot setting.

Given LLM context-window limits, directly including tens or hundreds of thousands of raw samples is impractical. Under simple random sampling, the sample mean is an unbiased estimator of the population mean. Thus, a small sampling fraction does not imply poor estimation precision by itself.

Question 4: Clarification is needed on whether BoundaryAcc can reward overly specific rules and fail to penalize overly general rules, whether it should be interpreted alongside accuracy as a complementary metric, and how it conceptually differs from contrast-set consistency (Gardner et al., 2020).

Response: We agree that it should be treated as a complementary metric to standard Accuracy, because accuracy on a conventional test set does not fully capture this local boundary behavior.

We agree that BoundaryAcc cannot detect all over-wide or over-narrow rules. However, expanding it to distant examples would not ensure a local edit distances with Hamming distance no larger than 2, and would alter background composition and change the quantity being measured. In this sense, fully closing the coverage gap within BoundaryAcc itself would contradict its intended role.

Compared with the “consistency” metric of Gardner et al. (2020), both approaches evaluate boundaries via local edits and require the original and edited instances to be classified correctly at the same time. However, their goals and formalizations differ fundamentally. Gardner et al. focus on whether a model has fully captured a particular local linguistic phenomenon and adopt a set-level binary consistency criterion (the entire contrast set must be correct). BoundaryAcc, by contrast, measures the extent to which a symbolic rule recovers the local boundary of an executable ground-truth rule. Its novelty lies primarily in its application to the recovery and assessment of symbolic rules under controlled statistical patterns.

Question 5: Corrections are suggested for the references to LLM-SRBench, SIRBench-V1, and Deep Symbolic Regression, along with clarification of the DSL’s stated avoidance of soft rules in light of its symbol-count operators and the perceived GC-content constraints in the generation prompt.

Response: Thank you for pointing this out. We have revised the references accordingly. However, the rule-generation prompt does not impose GC-content constraints; it explicitly states that such constraints must not be included. The operators that impose lower and upper bounds (MIN/MAX_SYMBOL_COUNT) themselves also define hard constraints.

Question 6: Additional comparisons with established methods such as Pratt for PS and STREME for MA are suggested, with at least Pratt requested for PS to clarify the positioning of LLM-based methods.

Response: Thanks for this suggestion. We selected PRATT, which generates regular-expression-like patterns, and obtained the results below. STREME does not align with the definition of rules used in this paper, so we do not include it.

Under the default settings (C%=100), we selected the top-ranked pattern according to PRATT’s native information score. Tasks returning no pattern were retained and evaluated using constant-negative predictions.

DatasetNumber of tasksTasks returning patternsAccuracyBoundaryAcc
PS-S2215 of 220.83410.3724
Remaining PS tasks217 of 210.5817—
Full PS set4322 of 430.7109—

Question 7: A random symbol-substitution control is suggested to assess potential memorization in the PROSITE task, alongside examples comparing rules generated by each method with official PROSITE patterns or JASPAR consensus sequences to substantiate the claim of human readability.

Response: For PROSITE tasks, publicly available biological patterns may confer an advantage through memorization or prior knowledge. However, the core capabilities assessed in this work—particularly BoundaryAcc, BoundaryCov, and rule inspectability—have already been validated on synthetic data without the bias of memorization, as no corresponding public knowledge exists that could be memorized. Furthermore, the grammar follows the PROSITE style and incorporates more complete character settings and logical constraints, we believe this fulfills the assessment.

We have complement some rule discovered for elaboration in Appendix A.9.

[Suggestion] As a supplement to Questions For Authors 3, the reliability of the comparison would be enhanced if the authors also demonstrated the impact of budget differences (Claude Code: 1,800 seconds, $5; Codex: 900 seconds; Table 16, p. 29) and the variability between runs when the experiment is repeated multiple times (the intervals in Table 3 are based on rule-level bootstrapping and do not include inter-run variability).

Response: We agree that both resource budgets and run-to-run variability are important for interpreting the comparison.

Budget selection and comparability. In preliminary experiments, we observed that the 900-second limit sometimes caused Claude Code to time out before completing its iterative analysis and rule generation. We therefore increased its wall-clock limit to 1,800 seconds while retaining the $5 cost cap per rule. Codex retained the 900-second limit used in its original configuration. These settings served as operational upper limits. We acknowledge that the budgets were not matched.

Run-to-run variability. We conducted two additional runs of selected baselines, bringing the total to three runs, and added the results to Appendix A.3. We report mean Accuracy and BoundaryAcc with 95% run-level bootstrap confidence intervals. Selected V1-m results are reproduced below.

MethodBoundaryAccAccuracy
Claude Code0.71 [0.62, 0.76]0.92 [0.88, 0.97]
Few-shot Claude0.45 [0.40, 0.54]0.71 [0.60, 0.78]
Few-shot Gemini0.43 [0.35, 0.51]0.73 [0.72, 0.75]
Few-shot GPT0.31 [0.20, 0.41]0.63 [0.58, 0.68]
Hypothesis Refinement0.41 [0.37, 0.46]0.68 [0.64, 0.73]
Self-Refinement0.45 [0.44, 0.47]0.68 [0.66, 0.68]

The observed variability depends on the method and subset. For example, few-shot GPT achieved Accuracy values of 60.28% and 59.47% on PS-S in the two additional runs, a difference of 0.81 percentage points. These supplementary run-level intervals should be distinguished from the original rule-level bootstrap intervals in Table 3: repeated runs do not increase the number of independent rules, and the intervals do not consistently narrow. The repeated-run analysis covers Claude Code and the few-shot and refinement methods; a temporary environment limitation prevented us from repeating Codex under its original configuration.

Reviewer_jCsTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SeqRule-Bench. Each task gives labeled positive and negative sequences, and a method must output a rule in a PROSITE-style DSL (a pattern plus logic constraints). The real tasks are PROSITE, JASPAR, methylation/promoter, signal-peptide and AMP. The synthetic tasks use LLM-generated ground-truth rules over five alphabets, with tiered negatives. The paper introduces BoundaryAcc, which is accuracy on minimally perturbed pairs that cross the ground-truth rule's boundary. It evaluates few-shot LLMs, refinement methods, LLM-SR, agentic coders (Claude Code, Codex), FlexFringe/GA/SA and black-box predictors. Claude Code performs best. The authors conclude that high accuracy does not guarantee rule recovery, and that the agents cost roughly an order of magnitude more tokens than non-agentic LLM baselines (Sec. 4.1, Fig. 6).

Critical strengths

The problem framing is new and well motivated: interpretable, executable rule induction for biological sequences, which sits in the gap between SR, ILP and LLM-induction benchmarks (Table 2). The synthetic tasks have ground-truth rules and tiered negatives, from random pattern-violating sequences to logic-violating and minimally edited near-boundary sequences (Sec. 3.3, Fig. 4). This allows a controlled test of rule recovery versus shortcut fitting. BoundaryAcc is carefully validated in App. A.3: identity/rewrite checks (Table 9), seed stability (Table 7), and invariance to label noise and class imbalance (Tables 11 and 13). It exposes a real gap between accuracy and rule recovery. The baselines are broad, include agentic LLMs, and report token cost (Fig. 6).

Critical weaknesses
  1. Domain-relevant baselines are insufficient. Learning motifs or rules from positive and negative sequences has been studied for a long time in bioinformatics, and the related work section already mentions ILP and regex inference. However, the only non-LLM symbolic baselines are FlexFringe, GA and SA, and all of them achieve low boundary accuracy in Table 3. Without stronger domain-specific baselines, it remains unclear how established methods perform on this benchmark.
  2. The real-data half is only loosely connected to rule discovery. According to Table 1, only PROSITE has partial ground-truth rules, so boundary accuracy is reported on just one real subset. The hard-logic DSL may also not suit tasks such as JASPAR and AMP, which Appendix A.5 describes as relying on frequency matrices and soft physicochemical patterns. As a result, most conclusions about rule recovery are drawn from synthetic data.
  3. Evaluation scale and comparability are limited. Each synthetic domain contains only 50 rules, and the confidence intervals in Table 3 are wide, for example 0.31 to 0.97 for Claude Code on V1-m. The methods are also given different amounts of data: the few-shot and refinement baselines receive 800 in-context examples, while the agents use the full training set and can execute scripts. Reporting variance across repeated runs and specifying which LLM generated the synthetic rules would strengthen the evaluation.
Questions for authors
  1. Can you add established domain-specific baselines, such as discriminative motif discovery methods and ILP or regex-induction systems?
  2. Which LLM generated the synthetic rules? Does it overlap with the evaluated models?
  3. What is the run-to-run variance of the LLM and agent methods?
5 comments

Question 1: Can you add established domain-specific baselines, such as discriminative motif discovery methods and ILP or regex-induction systems?

Response: Thank you for the questions. We have added the motif discovery method PRATT [1] as a baseline for the PS tasks. PRATT can automatically discover conserved patterns from a set of unaligned protein sequences using a PROSITE-like pattern syntax. However, ILP requires explicitly defined sequence predicates and background knowledge, while incorporating regex-induction systems would require addressing the issue of overlap between the training and test sets, as noted in the related work. We therefore did not incorporate these two classes of methods into our setting.

Under the default settings (C%=100), we selected the top-ranked pattern according to PRATT’s native information score. Tasks returning no pattern were retained and evaluated using constant-negative predictions.

DatasetNumber of tasksTasks returning patternsAccuracyBoundaryAcc
PS-S2215 of 220.83410.3724
Remaining PS tasks217 of 210.5817—
Full PS set4322 of 430.7109—

References

[1] Jonassen, I., Collins, J. F., & Higgins, D. G. (1995). Finding flexible patterns in unaligned protein sequences. Protein Science, 4(8), 1587–1595. https://doi.org/10.1002/pro.5560040817

Question 2: Which LLM generated the synthetic rules? Does it overlap with the evaluated models?

Response: We used Gemini Pro 3.1 to generate the rules, and it was also included among the evaluated models. The rules were therefore jointly constrained by the prompt requirements and human review, helping reduce implausible content and some potential biases. Thus, although the same model was used for both rule generation and evaluation, it served different functions.

Question 3: What is the run-to-run variance of the LLM and agent methods?

Response: Thank you for the question. We conducted two additional runs of selected LLM and agent baselines, bringing the total to three runs, and added a run-to-run variability analysis in Appendix A.3. We report mean Accuracy and BoundaryAcc with 95% run-level bootstrap confidence intervals; selected results on V1-m are reproduced below.

MethodBoundaryAccAccuracy
Claude Code0.71 [0.62, 0.76]0.92 [0.88, 0.97]
Few-shot Claude0.45 [0.40, 0.54]0.71 [0.60, 0.78]
Few-shot Gemini0.43 [0.35, 0.51]0.73 [0.72, 0.75]
Few-shot GPT0.31 [0.20, 0.41]0.63 [0.58, 0.68]
Hypothesis Refinement0.41 [0.37, 0.46]0.68 [0.64, 0.73]
Self-Refinement0.45 [0.44, 0.47]0.68 [0.66, 0.68]

The observed variability depends on both the method and subset. For example, few-shot GPT achieved Accuracy values of 60.28% and 59.47% on PS-S in the two additional runs, a difference of 0.81 percentage points. Some small differences between few-shot runs may be related to their low temperature of 0.025. Although Hypothesis Refinement and Self-Refinement use a temperature of 0.7, the results do not show consistently greater variability than few-shot methods.

Domain-relevant baselines are insufficient. The real-data half is only loosely connected to rule discovery. Evaluation scale and comparability are limited.

Response: These are good questions. We added PRATT as a domain-specific motif-discovery baseline. Other related methods require changes to their design or assumptions to fit our setting, as detailed in our response to Question 1. Under its default settings, PRATT did not outperform the coding-agent baselines on PS-S and returned no pattern for some tasks.

Biological datasets with explicit ground-truth rules remain scarce, motivating our combination of real biological tasks and controlled synthetic tasks. The shared rule language enables consistent comparisons across methods and controlled evaluation of whether models recover rule-defined decision boundaries beyond fitting statistical patterns. It builds on familiar motif descriptions such as ATxxxGC and extends them with logical constraints, allowing more precise and readable hypotheses about sequence–function relationships. On real data, these explicit hypotheses provide a direct connection to rule discovery.

To strengthen reliability, we added repeated runs to assess run-to-run variability and extended the sample-size analysis (Appendix A.3). The results show limited additional gains at larger sample sizes, supporting 800 in-context examples as a practical budget for the evaluated setting.

Comments on the paper

4 comments

Question: The lack of accuracy changes with respect to training examples is very intriguing—have you measured smaller shot counts like 10/40/100? Perhaps the accuracy curve increases there, and you’re already far enough out for it to have plateaued.

Response: Indeed, our experimental results support your observation: performance reaches a plateau at around 400 training examples, while showing an upward trend at smaller sample sizes.

Training examplesBoundaryAcc (%)Acc (%)
108.3154.02
4016.258.05
10017.6558.1
40018.2859.66
80019.4460.3
160018.5360.19

Overall, performance improves noticeably as the training set grows from very small to moderately sized. Once the training set is already relatively large, further increases provide only limited gains in final performance.

Question 3: Finally, in Table 3, the confidence intervals could be narrowed by running more trials per-model. In some cases, however, I understand that this might be prohibitively expensive. I could see either running more trials for cheaper baselines to distinguish them, or adjusting the table to directly test paired differences—which may offer more power than raw confidence intervals.

Response: We conducted additional baseline runs, bringing the total to three runs. Claude Code was allowed up to three attempts per run. Since our current environment temporarily prevents us from invoking Codex under the original settings, and LLM-SR performed poorly in the original experiments, this round of analysis focused mainly on Claude Code, few-shot, and refinement methods. Selected results are shown below:

MethodPS-SV1-sV1-mV1-cV3-sV3-mV3-c
Claude Code0.56 [0.45, 0.67]0.35 [0.24, 0.47]0.71 [0.34, 1.00]0.96 [0.87, 1.00]0.77 [0.61, 0.92]0.50 [0.28, 0.78]—
Few-shot Claude0.21 [0.12, 0.32]0.13 [0.05, 0.21]0.45 [0.17, 0.73]0.29 [0.14, 0.50]0.37 [0.18, 0.56]0.15 [0.09, 0.24]0.13 [0.07, 0.22]
Few-shot Gemini0.42 [0.31, 0.54]0.09 [0.02, 0.17]0.43 [0.14, 0.73]0.23 [0.06, 0.50]0.39 [0.12, 0.66]0.27 [0.11, 0.47]0.11 [0.02, 0.24]
Few-shot GPT0.11 [0.04, 0.20]0.09 [0.04, 0.14]0.31 [0.15, 0.51]0.20 [0.08, 0.40]0.32 [0.10, 0.53]0.13 [0.05, 0.21]0.06 [0.02, 0.10]
Hypothesis Refinement0.51 [0.43, 0.59]0.21 [0.13, 0.30]0.41 [0.22, 0.64]0.23 [0.14, 0.32]0.47 [0.27, 0.67]0.18 [0.12, 0.24]0.17 [0.09, 0.26]
Self-Refinement0.30 [0.19, 0.42]0.15 [0.07, 0.26]0.45 [0.21, 0.68]0.35 [0.23, 0.51]0.43 [0.21, 0.60]0.24 [0.16, 0.37]0.12 [0.10, 0.15]

Compared with the original results, some values have changed, but the confidence intervals have not consistently narrowed. Repeated runs do not increase the number of independent rules, and the revised intervals also account for differences between runs.

For the few-shot methods, we observed relatively small differences between some runs, which may be related to the low temperature (0.025) in the original settings. This is also reflected in Acc: for example, few-shot GPT achieved Acc values of 60.28% and 59.47% on PS-S in the two additional runs, a difference of 0.81 percentage points. Hypothesis Refinement and Self-Refinement iteratively generate and adjust candidate solutions, and their original temperature setting of 0.7 may also introduce run-to-run variability. However, the present results do not show that they consistently exhibit greater variability than few-shot methods.

Under the specified run budget, Claude Code sometimes failed to return a parsable rule. For example, one additional run on V1-m/VIRT1-004 still returned empty or unparsable output after three attempts.

AuthorsOpenReview ↗

Rebuttal Summary

We have responded to the reviewers’ comments and revised the manuscript accordingly. The main changes are summarized below, primarily in the Experiments section and Appendices A.3–A.4.

  • Added repeated runs of selected baselines to assess run-to-run variability and report uncertainty estimates (Appendix A.3).
  • Extended the sample-size analysis, showing limited gains beyond 800 examples and supporting this input budget for the evaluated setting (Experiments; Appendix A.3).
  • Added BoundaryAcc evaluations of black-box models (Appendix A.3).
  • Resolved inconsistencies in the BoundaryAcc definitions and notation and corrected related typos (Appendix A.4).
  • Expanded analyses of rule complexity, negative sampling, and metric properties to strengthen the justification of our rule-based benchmark and support fairer comparisons across methods (Appendices A.3–A.4).
  • Added PRATT as a domain-specific motif-discovery baseline to broaden the comparison (Appendix A.3).
  • Revised the conclusion to emphasize the distinction between predictive accuracy and rule recovery (Conclusion).

We thank the reviewers and editors for their constructive comments and suggestions.

Adding black-box baseline analyses to the appendix and strengthening the conclusion.

Response: We have uploaded the revised manuscript and incorporated your additional suggestions, including the BoundaryAcc evaluation of black-box models, into Appendix A.3 and the corresponding parts of the Experiments section for your review.

We have also revised the conclusion to better reflect the findings of our additional analyses:

Our experiments reveal a gap between predictive success and recovery of local ground-truth boundaries. Accuracy and BoundaryAcc are strongly correlated, yet their rankings can disagree, and high test accuracy can coexist with poor boundary recovery. Controlled validation shows that BoundaryAcc respects semantic equivalence and decreases overall under rule corruption; it is invariant to stored-label noise and class-ratio changes when rules and boundary pairs are fixed. The difficulty analyses further suggest that logical-constraint counts and alphabet size have limited explanatory power of learning difficulty, and negatives closer to positives in rule space are not consistently harder to classify.

Thank you very much for your constructive suggestions, which have helped us improve the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.