acceptodds
Under review as a conference paper at ICLR 2027

Coding LLMs can be harnessed to learn Interpretable Models from Data

Abstract

Since compact, interpretable classifiers correspond to simple programs, it is natural to ask if LLM-based coding agents can be used to learn interpretable models from data. In this paper we present a simple harness that repeatedly prompts a coding LLM to produce the next rule in a rule set, given a sample of labeled examples. We show that this approach is competitive with, and often superior to, state-of-the-art rule-learning methods on tabular data, especially with small training sets, and wins by large margins on text, including against another LLM- based rule learner; that this approach outperforms standard harnesses, and that the approach immediately leads to new kinds of interpretable models, since coding agents are not restricted to search a highly restricted hypothesis space. In addition to the advances made in learning interpretable models, these results are of interest because they introduce a new class of coding benchmarks—construction of interpretable classifiers—which includes quantitative measurements of output quality (i.e., test-set error) and existing mature “specialist coding algorithms” (i.e., existing learning methods).

Primary area:
interpretability and explainable AI
Keywords:
rule induction, neurosymbolic AI, LLM-based rule learners, hybrid AI systems, interpretable AI, classifiers, interpretability, boosting, program synthesis, text classification, tabular classification, clinical NLP, coding agents, interpretable classifiers

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_V49yOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes CLIPPER, which replaces SLIPPER's discrete inner-loop rule search with a coding LLM. At each iteration, the LLM proposes one rule, the harness evaluates it, and then reweights examples. The whole flow resembles GEPA at the level of LLM proposes -> evaluate for the loop.

The method is elegant and evaluated broadly across a wide range of domains. The empirical evidence supports the conclusion that LLMs are useful rule proposers within an iterative learner.

Critical strengths
  • Simple technical idea. The paper proposes to replace a symbolic learner's search procedure instead of the whole algorithm, provides a clear role for LLM's world knowledge.
  • Strong empirical results. The gains over SLIPPER, HS-TREE and RuleChef are large and significant.
  • Inference remains deterministic. LLMs are used for finding rules only.
Critical weaknesses
  • Limited novelty. The paper's main argument is the LLM's world knowledge can be better to extract useful rules used on traditional algorithms. This is not a surprising result.
  • Several experiments may be unfair in the sense that LLM has access to a richer representation. CLIPPER reads raw text and can use regexes, phrases, arithmetic, while the baseline often just receive word set or TD-IDF. I have to admit though this fact might be hard to isolate and fully tested. Can we get better rules without the use of LLM?
Questions for authors

See weaknesses.

1 comment

Thank you for your review, and for describing the paper as "elegant and evaluated broadly across a wide range of domains". Figures R6 and R7 are linked below and in Appendix H.

W1 - Novelty

Of course this is a subjective matter. While the bottom line "LLMs help with X" is very familiar, we found several of the specific results surprising:

  • Even long-context models do not extract statistics well from larger samples: showing the LLM 200 or 400 examples per call instead of 100 raises error by up to 0.5 points.
  • Giving the LLM full-data statistics (hints and coverage tools) can degrade it with little data, relative to its parametric knowledge alone: error rises by 0.8–1.0 points at n = 100 and 250, and falls by 0.3–1.0 from n = 600.
  • The same model, used more directly, does not beat a 1999 rule learner. Asked for a complete classifier in one prompt, it is 1.7–2.9 points worse than SLIPPER from n = 600. Inside a general-purpose coding agent with all the data it is 2.5 points worse. RuleChef, which proposes, evaluates and revises rules with the same model on the same raw text, is 3.5–19.8 points worse than CLIPPER.

On the resemblance to GEPA: the two share what every generate-and-test method shares. GEPA optimizes the prompts of an LLM program by reflecting on its traces and keeping a Pareto front of candidates, and the optimized program still calls the LLM at prediction time. CLIPPER has no reflection, population or revision: each call writes one new rule from a reweighted sample, the harness sets the rule's weight in closed form as SLIPPER does, and the final classifier runs without an LLM. We will add this comparison to the related work.

W2 - Richer representation

It is either unfair, or else a new intrinsic advantage of LLMs. Writing a discrete search for regexes is very difficult, and the ability to expand a rule language by just changing a prompt is a big win! We do measure this in the paper: Figure 4b includes a CLIPPER restricted to SLIPPER's input and rule language (each document as a set of words; a rule tests whether words appear). Even with this restriction CLIPPER has 4.9–9.6 points lower error than SLIPPER at every training size. Reading the raw text adds 0.5–3.0 more, so about 80% of the gain on text comes from identical input (Figure R6).

We also parsed the learned rules. On tabular data both learners read the same attribute values, and 98.9% of 4,699 CLIPPER rules are conjunctions of exactly SLIPPER's threshold and equality tests; none uses arithmetic. Of 1,897 text rules, none uses arithmetic either. The word-set rules never use a regex or a phrase; the raw-text rules do (88% a regex, 83% a phrase), which is the extension the paper intends. We will add a table stating what each learner reads in each experiment.

Can we get better rules without an LLM?

We tried. In 2,220 new fits on the same text subsets, with no LLM, we strengthened SLIPPER in three ways:

  • Phrases: every two- and three-word sequence as a feature, so a rule can test "not good".
  • Surface features: twelve counts taken from the raw string (digits, capitals, punctuation, links, length).
  • Two-sided search: in the paper, SLIPPER and CLIPPER learn rules for one class only. We let SLIPPER's rules vote for either class; CLIPPER is unchanged.

Macro test error (%), five text datasets:

n = 100n = 750
SLIPPER (paper)32.026.0
+ phrases32.025.8
+ phrases + surface30.622.6
Two-sided SLIPPER28.019.5
Two-sided + phrases + surface27.719.4
CLIPPER, same word sets26.216.4
CLIPPER, raw text23.314.8

Phrases do not help, surface features help a little, and the two-sided search helps most (2.4–6.5 points); we will add this stronger SLIPPER to Figure 4b. CLIPPER still has lower error than the best LLM-free learner at every training size: by 1.5–3.2 points reading the same word sets as SLIPPER, and by 2.8–4.6 reading the raw text (R6). The gap is on the three sentiment datasets (3.0–6.0 points on identical input), where a rule must know what a word means. On the two spam datasets, where digits and links carry the signal, the strengthened SLIPPER matches CLIPPER on raw text to within about one point.

Better features do not close the gap on sentiment because the useful words are rare (Figure R7). Take the words that occur once in 100 training documents, in a document of the target class. Counts cannot tell them apart, and as a group they indicate that class in only 59% of the test documents that contain them. The ones CLIPPER put in its rules do so 92% of the time. Among words seen once in 400 Yelp reviews, it used "mediocre", "overcooked" and "stale", and left out "bamboo", "cashier" and "soup". The LLM picks among the words the data offers by what they mean.

Reviewer_1vsbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CLIPPER, which replaces SLIPPER’s discrete rule search with a coding LLM. At each iteration, the LLM receives weighted labeled examples and proposes an executable rule; the harness evaluates its coverage, assigns a confidence, and updates example weights. Optional tools provide full-data statistics and candidate conditions. The method is evaluated on tabular classification, text classification, and value extraction.

Critical strengths
  • The approach is well motivated and simple. Combining LLM-generated hypotheses with conventional statistical evaluation offers a practical way to incorporate semantic knowledge into executable symbolic models.
  • The empirical study is substantial, covering multiple datasets, training sizes, seeds, model families, and harness variants. Developing prompts and configurations on a separate benchmark suite strengthens the evaluation.
  • The idea of including LLMs in predictive modeling, leveraging its rich prior information in reducing the hypotheses space is interesting, intuitive, and inspiring.
Critical weaknesses
  1. Important modern symbolic baselines are missing.
    The tabular evaluation omits other stronger rule-based models. One example is FastSRS, a recent method directly targeting sparse rule-set classification and the accuracy–sparsity tradeoff. Its experiments include overlapping application domains and comparisons with RIPPER, Bayesian Rule Sets, BRCG, and LIBRE. A modern sparse-tree optimizer, such as GOSDT with the reference-ensemble techniques of McTavish et al., would also strengthen the broader interpretable-model comparison. The current experiments therefore do not sufficiently establish competitiveness with the strongest relevant non-LLM approaches.

  2. Complexity is reported, but its comparison across the baseline suite is incomplete. The paper reports retained-rule counts for SLIPPER and CLIPPER variants, but not for other rule-based baselines, including RIPPER, CORELS, RuleFit, and SkopeRules. Table 8 provides configuration limits rather than measured sizes of the resulting models. Consequently, the experiments do not establish CLIPPER’s accuracy–complexity tradeoff against the broader baseline suite. Within the reported comparison, CLIPPER with hints averages 16.0 rules at n=2,000n=2{,}000, versus SLIPPER’s 8.9, while their errors are 21.6% and 21.3%, respectively. Moreover, one Python predicate can contain several operations, and the conjunction restriction is imposed through prompting rather than syntax validation. Comparisons should account for total conditions and internal branching, alongside rule counts.

  3. The mechanism evidence supports semantic priors, but the applicability of the resulting advantage remains unclear. At n=250n=250 across all eight datasets, the coverage-tool variant obtains 22.7% error with meaningful names and 23.8% after anonymization (Table 51), compared with the reported full-data SLIPPER reference of 23.1% (Table 23). Although SLIPPER is not rerun under anonymization, its comparisons do not exploit linguistic meanings. These results are consistent with semantic information explaining CLIPPER’s modest advantage in this setting. This is a plausible and useful mechanism. The remaining question is when it produces an advantage over strong conventional search. On tabular data, the intended rules largely combine existing attributes and thresholds; the paper should clarify how often generated predicates introduce different operations and whether improvements persist within a validated common grammar. The experiments also reveal boundaries: the advantage diminishes with additional training data, full-data SLIPPER wins at n=2,000n=2{,}000, and increasing the prompt sample does not reliably improve performance. The generalizable contribution appears to be a method for incorporating semantic priors into rule learning. The paper should articulate the conditions under which this is useful more clearly, rather than broadly characterizing CLIPPER as a better rule learner.

  4. The harness experiments do not isolate the benefit of adaptive iteration. The one-shot comparison changes model-call count, cumulative example exposure, hypothesis composition, and acceptance behavior simultaneously. It therefore evaluates the complete system without isolating whether adaptive construction improves over spending the same resources on independent rule proposals.The comparable performance of set covering and boosting is informative and suggests that boosting itself is not essential. The general-purpose-agent comparison is additionally conditional on restrictions against fitting existing learners, which materially affect performance. These qualifications should be reflected more prominently in the harness claims.

  5. Cost is not adequately compared with conventional learners. The approximately $0.04 per learning run reported at n=250n=250 is compared with other LLM coding agents. There is no systematic comparison of training wall time, CPU resources, memory, and monetary cost against the specialist baselines. CLIPPER additionally requires SLIPPER cross-validation, repeated rule execution, and potentially tool calls and retries. Its lack of inference-time LLM calls is useful, but conventional rule learners share this property. The practical benefit of adding an LLM therefore remains difficult to assess.

Questions for authors

How does CLIPPER compare with FastSRS under matched training subsets and training-only parameter selection?

Can the authors report realized model complexity for the other rule-based baselines and compare accuracy across complexity levels?

2 comments

We thank the reviewer for the constructive review. We ran the experiments requested; Figures R1, R2, R4 and R5 are linked below and in Appendix H of the revised PDF.

W1, Q1 - FastSRS, GOSDT We added FastSRS and GOSDT with reference-ensemble thresholds, trained on the same subsets as CLIPPER and SLIPPER (7 datasets, 10 seeds), with regularization chosen by 3-fold CV on the training subset alone (Figure R4). CLIPPER with hints has lower error than FastSRS at all five sizes under both regularization grids we ran: by 1.0 points at n = 100 and 250 and 0.3 at n = 2,000, against FastSRS's better grid at each size. Against GOSDT it is level at n = 100 and 0.1–0.6 points lower from n = 250 on. The lowest error of any learner at n = 100 and 250 is CLIPPER with set covering (24.4% and 23.0%). FastSRS is level with full-data SLIPPER at n = 100 and behind it at larger sizes. On text, given the same word sets as SLIPPER, FastSRS and GOSDT are 5–11 and 4–11 points behind CLIPPER (Figure R5).

W2, Q2 - Realized complexity Measured rules at n = 100–2,000: CORELS 1.8–2.1, FastSRS 1.8–2.3, GOSDT 2.2–2.6 leaves, RIPPER 1.9–6.4, SkopeRules 31–55, RuleFit 62–77. The larger baselines are no more accurate: RuleFit only matches CLIPPER with hints (12–16 rules), and SkopeRules is 3–6 points worse. Figure R1 and Figure R2 give the fronts of error against rules, training time and rule depth. Pooled over training sizes, CLIPPER is at both ends of the size frontier: with set covering it learns 2.1–2.6 rules, the size of FastSRS and GOSDT, with lower error than both up to n = 600; with hints it is the most accurate non-ensemble learner and the shallowest in the paper (longest rule: 1.7 conditions).

On 16.0 rules against 8.9: CLIPPER's rules are short (1.3 conditions on average), and boosting can accept the same rule twice, which SLIPPER removes automatically and CLIPPER does not yet. With repeats merged, CLIPPER's n = 2,000 models have 13.5 rules and 17.9 conditions, against SLIPPER's 8.8 and 20.6

To check the conjunction restriction, we parsed all 4,699 saved tabular rules: 99.9% are plain conjunctions, 98.9% use only SLIPPER's own threshold and equality tests, and none uses arithmetic, compares two attributes or calls a function. So rule counts measure CLIPPER's size fairly.

W3 - Semantic priors We agree that this is very likely CLIPPER's advantage and that it is, as you say, a plausible and useful mechanism; we will say so in the paper. The improvement persists within the common grammar: deleting every rule outside SLIPPER's rule language from the saved models changes error by at most 0.01 points. And SLIPPER, RIPPER and CORELS, rerun on the anonymized data, return the identical model on all 80 fits, while CLIPPER loses 0.4–1.1 points.

When the advantage appears is addressed in the paper: there is a clear advantage over a similar data-driven rule search on text everywhere, and on tabular data for small training sets (n below 200–300) with meaningful attribute names and values. The advantage on text is intuitive, since LLMs understand text well, while tabular data is less represented in pre-training. We will state these conditions explicitly. Whether other rule learners can be improved with LLM-derived semantic priors is future work (we suspect so).

W4 - Harness We agree that the alternative harnesses do not completely isolate adaptive iteration. Separating each factor you list would require extensive experimentation, which is beyond our scope, and we will restrict our claims accordingly. Our data argue against a resource explanation: set covering matches boosting with 3.2 LLM calls per model instead of 21.4, and showing the LLM 2–4× as many examples per call raises error. One-shot remains 0.9 points worse on the fits where it returns a usable model. Both conditions for general-purpose agents are in the paper (Table 2, Appendix E); we will state the restriction with the harness claims.

W5 - Cost Indeed, as practitioners know, any LLM use is expensive compared with nearly any algorithm developed in the 1990s, as SLIPPER was. At n = 250 and 2,000, CLIPPER with hints takes 29 and 43 s and 7c and 9c per model, including SLIPPER's CV and all retries; SLIPPER takes 2 and 13 s, FastSRS 7 and 19 s, GOSDT 1–2 s, RIPPER and CORELS under 0.3 s (R1b). CLIPPER's local process uses 300 MB of memory (others: 55–477 MB). We will note in the revision that LLMs, even "fast" ones, are much slower than discrete search. What this buys is prior knowledge that discrete search cannot supply: about 1 point over FastSRS and 3 over RIPPER on tabular data with n ≤ 250, and 5–11 points over FastSRS on text.

Thank you for your suggestions. Above we responded to your questions - in summary, the new experiments show that CLIPPER does indeed generally outperform more modern rule learners (FastSRS and GOSDT), even considering a range of regularization/rule-size constrains. Some of these experiments were also suggested by reviewer 1, and we agree they are informative. Will you consider changing your score, as Reviewer 1 did?

Reviewer_xqv4OpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper proposed CLIPPER, a LLM-assisted supervised learning approach to learn rule set. The propose method take as input a labelled binarized training set and repeatedly prompts a coding LLM to produce the rules that form the rule set. More precisely, in each iteration, it reweights the training data to focus on examples that are not explained by the accumulated rules. The authors evaluate CLIPPER on several tabular and text classification datasets, and compare it performance against existing rule learning algorithms such as CORELS.

Critical strengths

None

Critical weaknesses

Lack of Pareto front analysis. Most interpretable learning are multi-objective in nature (e.g., error, sparsity, running time). The propose learning strategy does not seem to have explicit control on sparsity, which is an important property for interpretability. How does one prevent the rule set from being very large (rule cardinality, rule set depth). Appendix F results show rather big number in comparison to what other interpretable models like CORELS typically exhibit. Ideally, it would have been interesting to have error = f(rule depth) Pareto front analysis. Similarly, since baseline learning technique (e.g., CORELS), it would have been interesting to also present error/training time trade-offs analysis.

Data contamination. The reliance on frontier backend models introduces several data contamination concerns, such as exposition to the evaluation datasets (e.g., COMPAS), or description of specific rule based models learned from these datasets (e.g., COMPAS rule list description in CORELS paper). The paper needs a stronger evaluation protocol to rule out potential data contamination risks.

Questions for authors
  1. Could authors report error/complexity and error/runing time Pareto front analysis ?
  2. Could authors explain how data contamination risks are mitigated in the current experiments?
  3. Another reason one typically aims for interpretability is transparency at algorithmic level (e.g., optimality certificate in CORELS). Could authors discuss what are the trade-off between understanding of the learned model and performance gains?
  4. Arbitrariness (e.g., predictive multiplicity) is also an emerging concern in the responsible AI literature. Could the authors discuss consistency (or minimally, label agreement) of learned rule sets across backends (e.g., Sonnet, Gemini, Haiku, Qwen) and prompt variants, as well as potential implication for accountability in decision-making?
5 comments

We thank the reviewer for their specific questions. To clarify, CLIPPER does not take a binarized training set: the LLM reads raw attribute values and text and writes predicates over them. CORELS, FPSkope, FastSRS and GOSDT are the learners that receive binarized features. Figures R1–R3 are linked below and in Appendix H of the revised PDF.

W1, Q1 - Pareto fronts CLIPPER lies on every front requested: error against rules, conditions, rule depth and training time (Figure R1, Figure R2). Sparsity can be controlled explicitly: each boosting round adds at most one rule, so limiting the rounds limits model size. We used SLIPPER's number of rounds (set by internal 5-fold CV) only for a fair comparison with SLIPPER, since our interest is the benefit of replacing discrete rule search with a coding LLM. Table 1 already has a sparse CLIPPER: with set covering it learns 2.1–2.6 rules (CORELS: 1.9) and has the lowest error of any learner at n = 100 and 250.

CLIPPER is at both ends of the size frontier (R1a). With set covering (2.4 rules) it is more accurate than SLIPPER, and than CORELS, FastSRS and GOSDT at the same size. With hints it is the most accurate non-ensemble learner. CLIPPER is the slowest learner (24–36 s and $0.06–0.09 per model, against 7.7 s for SLIPPER); it spends that time buying accuracy where labels are scarce (R1b). CLIPPER has more rules than SLIPPER partly because boosting can produce the same rule repeatedly. SLIPPER's rules are data structures, so it removes duplicates automatically; deduplicating CLIPPER's code is not yet implemented. With repeats merged, CLIPPER's n = 2,000 models have 13.5 rules and 17.9 conditions, against 8.8 and 20.6 for SLIPPER. Rule counts measure CLIPPER's size fairly: we parsed all 4,699 saved tabular rules, and 99.9% are plain conjunctions, 98.9% use only SLIPPER's own threshold and equality tests, and none uses arithmetic, compares two attributes or calls a function.

On the error = f(rule depth) front (R2), CLIPPER with hints is the shallowest and most accurate learner in the paper (1.7 conditions; 0.41 points below SLIPPER). Only GOSDT's one- or two-split trees are shallower, and they are less accurate. Three datasets skew Appendix F's means (median: 5.5 rules at n = 250, 10 at n = 2,000); we will add medians, distinct rules, conditions, depths and these fronts.

W2, Q2 - Data contamination We agree this is important and was underemphasized. The benchmarks are public; our claim is that the model uses what attribute names mean, which is the point of using an LLM, rather than recalling the data or a published model. For COMPAS this is testable: the rule list printed in the CORELS paper has 32.8% error on our test split. A learner recalling it would score 32.8% from the start; with 100 examples CLIPPER is at 36.4–38.0% and improves with data like SLIPPER and GOSDT. None of our 40 COMPAS models reproduces it; they agree with it on 80–82% of test rows, as do SLIPPER (80%) and CORELS (79%). On a text dataset published after Gemini 3.1 Flash-Lite's training cutoff (Appendix A.5), CLIPPER beats all text baselines at every size. With all names replaced by arbitrary identifiers, following Stechly et al. [1], CLIPPER with hints loses only 0.4 points and still matches full-data SLIPPER (Figure 4d).

Q3 - Transparency This is outside this paper's scope: to our knowledge there is no clear data on how optimality certificates for syntactic properties of a model affect people's trust, and measuring it would need a user study of its own. An LLM makes the learner much simpler: a simple harness replaces a complex search. The complexity moves into the LLM, but coding LLMs are widely used and trusted to generate code, which is the use here.

Q4 - Predictive multiplicity CLIPPER with three Gemini backends (one run twice), SLIPPER and CORELS were trained on the same 250 rows (7 datasets, 3 seeds). CLIPPER's backends disagree no more than two conventional learners do (Figure R3a): different backends agree on 87–92% of test labels, SLIPPER and CORELS on 88%, and two runs of one backend on 95%. Prompt variants agree as often as SLIPPER and RIPPER (87%). Every backend is more accurate than SLIPPER. Across training subsets, CLIPPER's error varies less than SLIPPER's from n = 250 on, and less than CORELS's with little data (R3b). This multiplicity comes from the data, on which many small models are nearly equally accurate, not from the LLM. A trained CLIPPER model is a saved Python program that runs without an LLM, so it can be audited like any rule set.

[1] Stechly et al., Chain of Thoughtlessness?, NeurIPS 2024.

Thank you for the detailed experiments. The responses have addressed my concerns on data contamination and multtiplicity. The pareto front experiments is still unclear. Could you please focus on one dataset, then, for each method, sweep various constraints on timing (for Corels, you can use the number of iteration) and rules; collect corresponding (error, #rules) , (error, time); report non-dominated solutions. So it should be one curve per method.

Could you also report error = f(training set size) beyond 2k, (ideally > 10k samples)

Thank you for the follow-up. We ran both analyses you asked for: Figure R8 and Figure R9, also in Appendix H of the revised PDF.

Scope of the paper. We would first like to recall what the paper claims, since it affects how these figures should be read. Our main interest is the benefit of using a coding LLM to replace discrete rule search, and that benefit comes from the LLM's semantic priors: it knows what the attribute names and values mean. This matters most when labeled data is scarce - the abstract says "especially with small training sets", and our tabular experiments use 100 to 2,000 examples. We do not claim that an LLM should replace exact search when tens of thousands of labeled examples are available; with that much data, the data itself can stand in for the prior.

One dataset, one curve per method (R8). We used Adult (one of our two datasets with over 10,000 training rows) at n = 250 and n = 2,000, with ten training subsets each. For each method we swept its own size and time limits and plot its non-dominated configurations: boosting rounds for CLIPPER and SLIPPER (1 to 64), the per-rule penalty and node budget for CORELS (10^2 to 10^6 nodes), the per-leaf penalty and time limit for GOSDT, the rule penalty and search iterations for FastSRS, and the rule limit and optimization passes for RIPPER.

As we said in our first response, CLIPPER does allow one to explicitly control sparsity: each boosting round produces at most one new rule, so limiting the rounds limits rule complexity. At n = 2,000 its error goes from 19.8% with one rule to 16.6% with four, 15.3% with eleven and 14.7% with 37.

With 250 examples, CLIPPER has the lowest curve from two rules up (R8a). With set covering it reaches 18.3% with two rules and 17.2% with three, where CORELS is at 19.4% (two rules), GOSDT at 18.8% (four leaves) and SLIPPER at 19.3% (2.6 rules). With hints it continues to 16.6% at 14 rules, while SLIPPER's best is 18.3%.

With 2,000 examples the methods are closer (R8c). Below about ten rules SLIPPER and GOSDT are ahead (GOSDT reaches 15.4% with eight leaves, CLIPPER 15.3% with eleven rules). CLIPPER then keeps improving to 14.7%, the lowest error of any method; SLIPPER's best is 15.2%. CORELS stays at 17.7% with two rules for every node budget, FastSRS reaches 17.4% and RIPPER 18.8%.

On training time (R8b, d): indeed, as practitioners generally know, any use of an LLM is expensive compared to a conventional learner. The conventional learners train in about a second or less on Adult (FastSRS: up to 9 s), while CLIPPER takes 2.5 s for one rule and 23 to 93 s for 16 to 64 rounds, nearly all of it waiting for the model. CLIPPER is the slowest curve, and at both sizes it is the one that reaches the lowest error.

Beyond 2,000 examples (R9). This regime is outside what the paper claims, but we agree it is useful to see where the advantage ends. Adult (24,421 training rows) and Bank marketing (33,909) are the datasets in our suite that allow it; we ran every learner at 5,000, 10,000 and 20,000 examples.

On Adult, CLIPPER with hints and coverage tools is at 14.6%, 14.4% and 14.3% (14.6%, 14.6% and 14.5% with hints alone). We note that FastSRS and GOSDT are the modern symbolic baselines that Reviewer 1vsb asked us to add, and CLIPPER stays ahead of both at every size: 1.1 to 1.3 points below GOSDT (15.6–15.7%) and 2.9 to 3.0 below FastSRS (17.2–17.6%). It is also 3.1 to 3.4 points below RIPPER (17.4–17.7%) and CORELS (17.7%). SLIPPER, which searches all rows directly, passes it from 5,000 examples (14.4%, 14.0% and 13.6%). Both use the round budget chosen by SLIPPER's cross-validation, which gives models of about 30 to 60 rules at these sizes; CLIPPER with set covering stays at four rules and 15.9–16.0%. On Bank marketing every learner is within 0.8 points; CLIPPER, SLIPPER and FastSRS are all at 10.8%.

This is what the claim above predicts. On Adult, CLIPPER is 1.9 points below SLIPPER at n = 250 and 0.4 below at n = 2,000; with tens of thousands of labels, exact search over all rows uses the data better than an LLM that reads 100 examples and a table of candidate conditions per call. Even there, CLIPPER stays ahead of every sparse-model learner. We will add both figures and state this range in the paper.

Finally, we note that the trends favor CLIPPER. Its training time is almost entirely model latency, and its accuracy comes from the model's prior knowledge, so both curves move as LLMs get faster, cheaper and better, with no change to the harness (swapping the model is a one-line change; in Table 3 the current Flash-Lite generation is already 0.5 points better than the previous one). The curves of the conventional learners are fixed by their search, while CLIPPER's improve with each model generation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.