- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes CLIPPER, which replaces SLIPPER's discrete inner-loop rule search with a coding LLM. At each iteration, the LLM proposes one rule, the harness evaluates it, and then reweights examples. The whole flow resembles GEPA at the level of LLM proposes -> evaluate for the loop.
The method is elegant and evaluated broadly across a wide range of domains. The empirical evidence supports the conclusion that LLMs are useful rule proposers within an iterative learner.
Critical strengths
- Simple technical idea. The paper proposes to replace a symbolic learner's search procedure instead of the whole algorithm, provides a clear role for LLM's world knowledge.
- Strong empirical results. The gains over SLIPPER, HS-TREE and RuleChef are large and significant.
- Inference remains deterministic. LLMs are used for finding rules only.
Critical weaknesses
- Limited novelty. The paper's main argument is the LLM's world knowledge can be better to extract useful rules used on traditional algorithms. This is not a surprising result.
- Several experiments may be unfair in the sense that LLM has access to a richer representation. CLIPPER reads raw text and can use regexes, phrases, arithmetic, while the baseline often just receive word set or TD-IDF. I have to admit though this fact might be hard to isolate and fully tested. Can we get better rules without the use of LLM?
Questions for authors
See weaknesses.
Rebuttal
Thank you for your review, and for describing the paper as "elegant and evaluated broadly across a wide range of domains". Figures R6 and R7 are linked below and in Appendix H.
W1 - Novelty
Of course this is a subjective matter. While the bottom line "LLMs help with X" is very familiar, we found several of the specific results surprising:
On the resemblance to GEPA: the two share what every generate-and-test method shares. GEPA optimizes the prompts of an LLM program by reflecting on its traces and keeping a Pareto front of candidates, and the optimized program still calls the LLM at prediction time. CLIPPER has no reflection, population or revision: each call writes one new rule from a reweighted sample, the harness sets the rule's weight in closed form as SLIPPER does, and the final classifier runs without an LLM. We will add this comparison to the related work.
W2 - Richer representation
It is either unfair, or else a new intrinsic advantage of LLMs. Writing a discrete search for regexes is very difficult, and the ability to expand a rule language by just changing a prompt is a big win! We do measure this in the paper: Figure 4b includes a CLIPPER restricted to SLIPPER's input and rule language (each document as a set of words; a rule tests whether words appear). Even with this restriction CLIPPER has 4.9–9.6 points lower error than SLIPPER at every training size. Reading the raw text adds 0.5–3.0 more, so about 80% of the gain on text comes from identical input (Figure R6).
We also parsed the learned rules. On tabular data both learners read the same attribute values, and 98.9% of 4,699 CLIPPER rules are conjunctions of exactly SLIPPER's threshold and equality tests; none uses arithmetic. Of 1,897 text rules, none uses arithmetic either. The word-set rules never use a regex or a phrase; the raw-text rules do (88% a regex, 83% a phrase), which is the extension the paper intends. We will add a table stating what each learner reads in each experiment.
Can we get better rules without an LLM?
We tried. In 2,220 new fits on the same text subsets, with no LLM, we strengthened SLIPPER in three ways:
Macro test error (%), five text datasets:
Phrases do not help, surface features help a little, and the two-sided search helps most (2.4–6.5 points); we will add this stronger SLIPPER to Figure 4b. CLIPPER still has lower error than the best LLM-free learner at every training size: by 1.5–3.2 points reading the same word sets as SLIPPER, and by 2.8–4.6 reading the raw text (R6). The gap is on the three sentiment datasets (3.0–6.0 points on identical input), where a rule must know what a word means. On the two spam datasets, where digits and links carry the signal, the strengthened SLIPPER matches CLIPPER on raw text to within about one point.
Better features do not close the gap on sentiment because the useful words are rare (Figure R7). Take the words that occur once in 100 training documents, in a document of the target class. Counts cannot tell them apart, and as a group they indicate that class in only 59% of the test documents that contain them. The ones CLIPPER put in its rules do so 92% of the time. Among words seen once in 400 Yelp reviews, it used "mediocre", "overcooked" and "stale", and left out "bamboo", "cashier" and "soup". The LLM picks among the words the data offers by what they mean.