- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a novel benchmark - SeqRule-Bench - that revolves around testing the ability of a wide variety of systems to inductively generate a rule for classifying whether or not a given sequence fulfills a criterion from positive and negative examples. SeqRule-Bench includes real biological sequences (both DNA and proteins) with ground-truth classification rules, along with synthetically generated counterparts inspired by real biological-motifs. The synthetic examples used in evaluation vary widely in difficulty and vocabulary size. The paper additionally introduces the concept of boundary accuracy - the ability of a system's learned rule to distinguish cases at the edges of the ground-truth rule that a heuristic pattern-matcher would miss. This is designed to distinguish statistical learning from hard rule induction. The paper then evaluates the aforementioned systems on SeqRuleBench and analyzes how both general accuracy and boundary accuracy change with respect to system type (LLM few-shot, LLM in a harness, traditional rule inductors, and black box predictors), and problem difficulty (rule complexity, vocabulary size). The paper concludes that while LLM's can approach the accuracy of black box predictors for rule induction, this doesn't necessarily lead to the learning of the true underlying rules.
Critical strengths
- The paper evaluates a diverse variety of systems - including real Agentic harnesses, classic LLM setups from the literature, and symbolic rule inductors. They also run black-box controls such as finetuning DNABert-2 to learn each rule. This addresses a common confound in genomic-LM research: conventional baselines performing nearly as well if not better than more extensive LLM setups.
- The boundary accuracy metric is extremely clever way of distinguishing rule induction from pattern-matching - it is the strongest point of the paper. It is intuitive and easy to understand, yet clearly delineates the difference - a true rule will properly distinguish cases that vary only slightly but in meaningful ways, while a global pattern matcher will fail to notice the crucial but small differences.
Critical weaknesses
- The presented results in Table 3 have wildly overlapping confidence intervals - for the real subset of SeqRule-Bench, these are typically around ~0.1 in either direction for most results - making it hard to distinguish between the middle performers in the 0.3-0.5 range. However, the confidence intervals are far wider in the synthetic subsets - often ~0.25 in either direction. This makes distinguishing performance very difficult, beyond distinguishing methods that completely fail from those that have partial success. Running more trials to reduce these intervals, or setting up a more controlled comparison, could help. Throughout the rest of the paper, confidence intervals remain a significant issue - Figure 5 (an accuracy comparison) lacks them entirely. As does Figure 7 - which tries to analyze Few Shot GPT-5.4 performance with respect to vocab size and data complexity. Given the extremely large confidence intervals in the Table 3 for the synthetic regime, it is unclear whether the trends in Table 7 mean anything - or are just reflective of noise. Proper statistical analysis would address this - which the paper does not include.
- The overall conclusion of the paper is fairly banal: that models in agentic coding harnesses do best on rule induction. The more interesting analyses - like properties of boundary accuracy or how model performance interacts with vocab size/complexity - are relegated to the appendix or lack statistical power. Furthermore, the paper fails to tell a convincing story and instead appears as a collection of observations, only some of which are connected - and are often listed like a laundry-list of findings rather than merged into a coherent conclusion.
Questions for authors
- Did you test the black-box baselines for boundary accuracy? I know they technically can't be tested as described, as they don't output an inductive rule - but nevertheless, you could treat their classification of the boundary points as an application of their learned 'rule' and report that. I'd be extremely interested.
- You remark, in 4.2, that "furthermore, although high accuracy is correlated with better boundary accuracy to some extent, they are not strictly proportional." I'd be very interested to know to what extent the correlation exists, whether its statistically significant, etc. - this is worth elaborating on!
- Furthermore, in 4.2, you remark "Finally, increasing the number of input examples brings only a modest improvement in accuracy." Quantify this, with appropriate statistical tests - it would make 4.2 much more compelling. Similarly, regress accuracy against logic constraints and input alphabet.
- Finally, in Table 3, the confidence intervals could be narrowed by running more trials per-model. In some cases, however, I understand that this might be prohibitively expensive. I could see either running more trials for cheaper baselines to distinguish them, or adjusting the table to directly test paired differences - which may offer more power than raw confidence intervals.
If all the above concerns were addressed, I could easily see my assessment going from "weak reject" to "weak accept".
5 comments
Question 2: You remark, in 4.2, that "furthermore, although high accuracy is correlated with better boundary accuracy to some extent, they are not strictly proportional." I'd be very interested to know to what extent the correlation exists, whether its statistically significant, etc. - this is worth elaborating on!
Response: Thank you for the question. We have quantitatively analyzed the relationship between the two metrics using the latest data. The Spearman rank correlation is ρ = 0.8434, indicating a strong positive monotonic association.
Accuracy and BoundaryAcc are highly correlated: Accuracy reflects the overall trend in boundary performance, but their rankings are not identical. Higher Accuracy does not always correspond to higher BoundaryAcc, and differences remain in both scores and rankings. Kendall’s τ-b = 0.6722 further supports broad agreement between the rankings. When comparing results within the same subset and under the same input budget, 483 of the 3,356 pairwise comparisons excluding ties exhibit ranking reversals, yielding a reversal rate of 14.39%.
These quantitative results support the conclusion that the two metrics are highly correlated but not fully interchangeable. Beyond producing numerically different scores, BoundaryAcc captures distinctions that Accuracy does not fully reflect.
Question 3: Furthermore, in 4.2, you remark "Finally, increasing the number of input examples brings only a modest improvement in accuracy." Quantify this, with appropriate statistical tests - it would make 4.2 much more compelling. Similarly, regress accuracy against logic constraints and input alphabet.
Response: For the statistical test of how accuracy scales with the number of samples, we reanalyzed the archived input-size experiments, evaluating the same 250 ground-truth (GT) tasks under three training-set sizes: 400, 800, and 1,600 examples. Increasing the input from 400 to 1,600 examples yielded a mean accuracy change of +0.00556, with a task-level bootstrap 95% confidence interval of [−0.006100, 0.017372]. A paired t-test yielded p = 0.351. In this sample, the change from 800 to 1,600 examples was near zero (paired t-test, p = 0.998).
Thus, within the tested input range, the experiment did not detect a statistically significant improvement in accuracy.
And for the second suggestion, using the records from Figure 7 in Section 4.2, we controlled for input size and computed standard errors clustered by rule. The regression results are as follows:
| Predictor change | Change in Accuracy [95% CI] | p-value | Change in BoundaryAcc [95% CI] | p-value |
|---|---|---|---|---|
| One additional logical constraint | −0.520 [−1.271, 0.232] | 0.1744 | −0.253 [−1.511, 1.005] | 0.6923 |
| One additional alphabet symbol | +0.173 [0.014, 0.333] | 0.0335 | +0.360 [0.090, 0.630] | 0.0091 |
The regression model has , explaining approximately 2.08% of the variation in Accuracy.
As shown in the table, the number of logical constraints shows no statistically significant linear association with accuracy, while alphabet size shows a weak positive linear association.
A very helpful baseline showing the power of LLM-based systems relative to black-box classifiers - thank you for providing! Maybe include as an appendix in final paper?
Response: Thanks for the question, here we complement the black-box baselines experiments. All values are reported as percentages (%).
The remaining rows retain the previously retrained results; V3 uses the corrected vocabulary. “—” indicates that the model was not evaluated.