- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper wants to find interpretable, decision-relevant features in a set of unstructured items without hand-engineering them. The pipeline has four steps:
An LLM compares items in pairs along a sparse 40-regular comparison graph. For each pair it names the features that differ and which item shows more of each. One strong LLM call groups these free-text rationales into K canonical features. A two-channel choice model is fitted to the citations. A “gate” models whether a feature gets cited (more likely when the score gap is larger), and a “direction” channel models which item wins, with a shared position-bias term. This is a gated Bradley–Terry model and generalizes Davidson’s tie model. Condition-number and reliability diagnostics filter out poorly identified features before a final refit.
There is a consistency theorem (uniform ℓ∞ rate, local MLE) for K = 1 under a dense design.
Results:
Synthetic cover letters: on 200 templated letters with 16 planted “dials,” the method tracks all 16, versus 13 for direct GPT-5.4 extraction. It is also better on attribution and leakage, at about 13× the cost. ChangeMyView: it finds 29 features (16 reliable) whose held-out accuracy (about 62%) is comparable to hand-engineered features and embeddings. Appendix extension: the same machinery audits holistic LLM grading on two essay corpora.
Critical strengths
-
The core idea is clean and well motivated: treat LLM rationales as data and model citation and direction separately. “Silence is informative” is a nice modeling point, and the link to Davidson’s tie model in B.1 is well done.
-
The statistics are taken seriously: bias terms, sandwich covariance under misspecification, identifiability diagnostics, out-of-fold calibration, and Brier skill against a habit-only submodel.
-
The synthetic benchmark has real ground truth and careful controls. Dials are near-orthogonal, length is decorrelated, and an irrelevant fluency dial serves as a control. The comparisons are paired, with explicit caveats about variance.
-
The ablations separate the two contributions honestly: pairwise elicitation versus statistical scoring and filtering.
Critical weaknesses
-
Related work is missing a whole line of research. LLM-based discovery of natural-language features and hypotheses from data is directly relevant: Zhong et al.’s D5 and describing differences between distributions, VisDiff, LLooM, HypotheSAEs, text bottleneck and concept bottleneck models with LLM-proposed concepts, and LLM hypothesis generation. These are the natural baselines, not just “dump the dataset into one prompt.” This is the biggest gap.
-
The statistical model contributes less to discovery than the framing suggests. The ablation “pairwise comparisons + LLM scoring, multi-score” already reaches 100% Tracked and Sem.R. So recovery comes from pairwise elicitation plus the LLM taxonomy. The model mainly improves attribution and leakage and filters features. Features are actually defined by a single LLM clustering call that the model doesn’t account for, and its uncertainty is only partly captured by re-running it in the bootstrap.
Questions for authors
N/A
2 comments
Concern 2: the statistical model contributes less to discovery than the framing suggests.
We agree with much of this point, and revised the framing accordingly. Finding the features comes mainly from pairwise elicitation followed by the LLM taxonomy step, which is one of the paper's messages: comparing items surfaces features that direct prompting misses. The statistical model is a second step, which measures the features, attributes them, and selects them. Section 4.1 now states this division of labour, and the new ICLR experiment is consistent with your reading: features discovered pairwise and scored pointwise, without the model, transfer well. We added three analyses that quantify the model's role, each answering one part of your point.
(a) "Pairwise comparisons + LLM scoring already reaches 100% Tracked and Sem.R." Correct: tracking comes from pairwise elicitation. To isolate what the model adds, we compared statistical and LLM scoring of the same 21 features over 50 paired resamples. Statistical scoring raises attribution in 100% of resamples: +0.11 [0.05, 0.19] against one-shot LLM scoring and +0.12 [0.05, 0.19] against 20-repetition LLM scoring. It lowers leakage against 20-repetition LLM scoring (-0.009 [-0.017, -0.000]). In return, 20-repetition LLM scoring reaches slightly higher statistical recovery (-0.045 Stat.R.), and tracking is equivalent.
(b) "The model mainly improves attribution and leakage and filters features." We agree, and tested whether its filter beats a simple alternative. On the synthetic benchmark, replacing the / reliability filter with a frequency filter (the 21 most-cited features) lowers Tracked from 100% to 93.8%, Sem.R. from 100% to 87.5% and attribution from 76.2% to 71.4%. On CMV, the / filter halves the feature set (29 to 16) without a significant loss of accuracy (-0.5 pp, p = 0.76), and the 13 discarded features are about 4 pp less predictive than a size-matched subset of the kept ones (58.0% vs 62.0%). On CMV, though, a frequency filter does as well, which we report.
(c) "Features are defined by a single LLM clustering call that the model does not account for." We quantified the uncertainty of this call. Re-running the taxonomy step in every bootstrap replicate, our method tracks 14.8/16 dials on average, against 12.6 for Direct LLM on the same resamples. The drop from 16/16 comes from the comparison budget, not from the clustering: a bootstrap replicate keeps only the comparisons whose two letters are both drawn (average degree 25.3 instead of 40), and at that budget with the taxonomy frozen, our method tracks 14.6 dials. Re-discovering the taxonomy therefore adds no measurable loss. The model still conditions on the taxonomy and does not propagate its uncertainty beyond this resampling; the revision now states this limitation explicitly.
We hope these revisions address both concerns, and remain available for any further questions.
Thank you for these constructive points. You raised two concerns: missing related work and baselines, and the statistical model contributing less to discovery than our framing suggested. We agree with both. Below, we take each in turn and explain how we revised the paper (changes in color).
Concern 1: missing related work, and baselines beyond "dump the dataset into one prompt".
Related work. The revision adds a related-work paragraph and a positioning table (Section 2, Table 1) covering the lines of work you listed: describing differences between distributions (D3/D5), VisDiff, LLooM, HypotheSAEs, text and concept bottleneck models with LLM-proposed concepts, and LLM hypothesis generation (NLParam, HypoGeniC). It also covers ICAI and VibeCheck, which, like us, have an LLM describe how two items differ. The table makes the differences explicit. Most of these methods need labels or label-defined groups; LLooM and the clustering models of NLParam do not. What sets our method apart is how features are found and measured. Even ICAI and VibeCheck, which also work from pairs, use pairs fixed by preference labels or by model identity, and score each feature in a separate LLM pass. We instead compare arbitrary unlabelled items along a comparison graph, and reuse the same rationales to score items and assess each feature's reliability.
Baselines. We added two of the methods you named as baselines in all three prediction or recovery tasks: LLooM, which, like ours, needs no labels, and HypotheSAEs, which selects features with labels.
(continued below)