acceptodds
Under review as a conference paper at ICLR 2027

Interpretable Feature Discovery from Unstructured Data Using LLMs and Statistical Choice Models

58 views

Abstract

Feature extraction from unstructured data typically relies on opaque embeddings unsuited to interpretable decision-making. Interpretable features, on the other hand, typically require costly, and often infeasible, hand-engineering. Large language models (LLMs) offer a natural bridge, as they can read raw data and describe it in interpretable terms. One could simply ask an LLM to extract features from a large sample of the dataset, but context limitations and diluted attention can lead to unsatisfactory results. We introduce a method that automatically discovers interpretable, decision-relevant features from unstructured data. Our key idea is to perform pairwise comparisons between items, asking an LLM to articulate, in interpretable terms, the features distinguishing each pair. We treat these explanations, or *rationales*, as data: we develop a statistical choice model that combines these rationales across the comparisons to identify relevant features and estimate per-item, per-feature scores. We first test our method on a synthetic hiring dataset of cover letters with known ground-truth features and show that it significantly outperforms direct LLM feature extraction in feature recovery. We then apply it to the ChangeMyView (Reddit) debate dataset, where it discovers several interpretable features associated with persuasion while reaching predictive accuracy comparable to established baselines. We also show how the same rationales can be used to audit LLM decisions, yielding a holistic score whose decomposition over interpretable features reveals what the LLM rewards.

Primary area:
interpretability and explainable AI
Keywords:
Interpretable feature discovery, Large language models, LLM-as-a-judge, Pairwise comparisons, Bradley-Terry, Interpretable machine learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qCtoOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper wants to find interpretable, decision-relevant features in a set of unstructured items without hand-engineering them. The pipeline has four steps:

An LLM compares items in pairs along a sparse 40-regular comparison graph. For each pair it names the features that differ and which item shows more of each. One strong LLM call groups these free-text rationales into K canonical features. A two-channel choice model is fitted to the citations. A “gate” models whether a feature gets cited (more likely when the score gap is larger), and a “direction” channel models which item wins, with a shared position-bias term. This is a gated Bradley–Terry model and generalizes Davidson’s tie model. Condition-number and reliability diagnostics filter out poorly identified features before a final refit.

There is a consistency theorem (uniform ℓ∞ rate, local MLE) for K = 1 under a dense design.

Results:

Synthetic cover letters: on 200 templated letters with 16 planted “dials,” the method tracks all 16, versus 13 for direct GPT-5.4 extraction. It is also better on attribution and leakage, at about 13× the cost. ChangeMyView: it finds 29 features (16 reliable) whose held-out accuracy (about 62%) is comparable to hand-engineered features and embeddings. Appendix extension: the same machinery audits holistic LLM grading on two essay corpora.

Critical strengths
  • The core idea is clean and well motivated: treat LLM rationales as data and model citation and direction separately. “Silence is informative” is a nice modeling point, and the link to Davidson’s tie model in B.1 is well done.

  • The statistics are taken seriously: bias terms, sandwich covariance under misspecification, identifiability diagnostics, out-of-fold calibration, and Brier skill against a habit-only submodel.

  • The synthetic benchmark has real ground truth and careful controls. Dials are near-orthogonal, length is decorrelated, and an irrelevant fluency dial serves as a control. The comparisons are paired, with explicit caveats about variance.

  • The ablations separate the two contributions honestly: pairwise elicitation versus statistical scoring and filtering.

Critical weaknesses
  • Related work is missing a whole line of research. LLM-based discovery of natural-language features and hypotheses from data is directly relevant: Zhong et al.’s D5 and describing differences between distributions, VisDiff, LLooM, HypotheSAEs, text bottleneck and concept bottleneck models with LLM-proposed concepts, and LLM hypothesis generation. These are the natural baselines, not just “dump the dataset into one prompt.” This is the biggest gap.

  • The statistical model contributes less to discovery than the framing suggests. The ablation “pairwise comparisons + LLM scoring, multi-score” already reaches 100% Tracked and Sem.R. So recovery comes from pairwise elicitation plus the LLM taxonomy. The model mainly improves attribution and leakage and filters features. Features are actually defined by a single LLM clustering call that the model doesn’t account for, and its uncertainty is only partly captured by re-running it in the bootstrap.

Questions for authors

N/A

2 comments

Thank you for these constructive points. You raised two concerns: missing related work and baselines, and the statistical model contributing less to discovery than our framing suggested. We agree with both. Below, we take each in turn and explain how we revised the paper (changes in color).

Concern 1: missing related work, and baselines beyond "dump the dataset into one prompt".

Related work. The revision adds a related-work paragraph and a positioning table (Section 2, Table 1) covering the lines of work you listed: describing differences between distributions (D3/D5), VisDiff, LLooM, HypotheSAEs, text and concept bottleneck models with LLM-proposed concepts, and LLM hypothesis generation (NLParam, HypoGeniC). It also covers ICAI and VibeCheck, which, like us, have an LLM describe how two items differ. The table makes the differences explicit. Most of these methods need labels or label-defined groups; LLooM and the clustering models of NLParam do not. What sets our method apart is how features are found and measured. Even ICAI and VibeCheck, which also work from pairs, use pairs fixed by preference labels or by model identity, and score each feature in a separate LLM pass. We instead compare arbitrary unlabelled items along a comparison graph, and reuse the same rationales to score items and assess each feature's reliability.

Baselines. We added two of the methods you named as baselines in all three prediction or recovery tasks: LLooM, which, like ours, needs no labels, and HypotheSAEs, which selects features with labels.

  • Synthetic cover letters (Section 4.1). We planted 16 ground-truth features ("dials") in 200 letters. LLooM's 15 concepts track half of them, and HypotheSAEs' 28 hypotheses track one (14 of them name the applicant in the signature; with no outcome to predict here, we ran it without label-based selection). Ours track all 16 in the full-data run, and 14.8 on average when discovery is re-run on resamples.
  • CMV (Section 5.1), predicting which of two Reddit replies changed the poster's view. LLooM (4 concepts) reaches 59.25% and HypotheSAEs (29 hypotheses) 58.13%, against 62.63% for ours (29 features), although HypotheSAEs uses the labels. The gap is not significant on the full sets, but at equal size HypotheSAEs is below ours at every k, by 3.8 points on average (95% CI [1.8, 5.8]).
  • A new task: AI-written ICLR reviews (Section 5.2). For 1,481 ICLR 2020-2022 papers, we pair a human review written before ChatGPT's release with a review GPT-5.4 mini writes from the same paper, and predict which is AI-written. All feature sets are near ceiling on held-out papers (ours: 99.7%). When the AI reviews come from another model, or from a prompt we wrote after discovery to remove the differences our features capture, ours keep 89.5% to 99.2%, and lead LLooM by 2.2 to 8.5 points and HypotheSAEs by 15.6 to 37.3. At equal numbers of features, ours trail LLooM on three of four test versions: the advantage is breadth (LLooM, allowed 30 concepts, returned 9).

(continued below)

Concern 2: the statistical model contributes less to discovery than the framing suggests.

We agree with much of this point, and revised the framing accordingly. Finding the features comes mainly from pairwise elicitation followed by the LLM taxonomy step, which is one of the paper's messages: comparing items surfaces features that direct prompting misses. The statistical model is a second step, which measures the features, attributes them, and selects them. Section 4.1 now states this division of labour, and the new ICLR experiment is consistent with your reading: features discovered pairwise and scored pointwise, without the model, transfer well. We added three analyses that quantify the model's role, each answering one part of your point.

(a) "Pairwise comparisons + LLM scoring already reaches 100% Tracked and Sem.R." Correct: tracking comes from pairwise elicitation. To isolate what the model adds, we compared statistical and LLM scoring of the same 21 features over 50 paired resamples. Statistical scoring raises attribution in 100% of resamples: +0.11 [0.05, 0.19] against one-shot LLM scoring and +0.12 [0.05, 0.19] against 20-repetition LLM scoring. It lowers leakage against 20-repetition LLM scoring (-0.009 [-0.017, -0.000]). In return, 20-repetition LLM scoring reaches slightly higher statistical recovery (-0.045 Stat.R.), and tracking is equivalent.

(b) "The model mainly improves attribution and leakage and filters features." We agree, and tested whether its filter beats a simple alternative. On the synthetic benchmark, replacing the κ\kappa/ρ\rho reliability filter with a frequency filter (the 21 most-cited features) lowers Tracked from 100% to 93.8%, Sem.R. from 100% to 87.5% and attribution from 76.2% to 71.4%. On CMV, the κ\kappa/ρ\rho filter halves the feature set (29 to 16) without a significant loss of accuracy (-0.5 pp, p = 0.76), and the 13 discarded features are about 4 pp less predictive than a size-matched subset of the kept ones (58.0% vs 62.0%). On CMV, though, a frequency filter does as well, which we report.

(c) "Features are defined by a single LLM clustering call that the model does not account for." We quantified the uncertainty of this call. Re-running the taxonomy step in every bootstrap replicate, our method tracks 14.8/16 dials on average, against 12.6 for Direct LLM on the same resamples. The drop from 16/16 comes from the comparison budget, not from the clustering: a bootstrap replicate keeps only the comparisons whose two letters are both drawn (average degree 25.3 instead of 40), and at that budget with the taxonomy frozen, our method tracks 14.6 dials. Re-discovering the taxonomy therefore adds no measurable loss. The model still conditions on the taxonomy and does not propagate its uncertainty beyond this resampling; the revision now states this limitation explicitly.

We hope these revisions address both concerns, and remain available for any further questions.

Reviewer_LTzgOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes discovering interpretable features by asking an LLM to compare item pairs, grouping the cited rationales into shared features, and fitting a two-stage choice model. For each feature, the model uses both whether it was cited and which item was favored to estimate item scores, a citation propensity, and position bias. It provides score-reliability diagnostics and proves local consistency in a dense, single-feature comparison regime (pp. 3–5). In one full-data run on 200 synthetic cover letters with planted attributes, it recovers all 16 selected targets, versus 13 for direct full-dataset LLM feature extraction (Table 1). On ChangeMyView (CMV) replies, 29 discovered features, or 16 after reliability filtering, yield held-out persuasion prediction near hand-engineered and embedding baselines (Table 2). The CMV prediction uses fresh LLM pointwise or per-feature Bradley–Terry scores; the proposed joint choice model contributes discovery and filtering there, rather than the prediction scores. An appendix applies a related model to explain several LLM judges' holistic essay comparisons.

Critical strengths

The graph <-> interpretable features make sense to me.

Following are the statements made by agents that I reviewed and agreed on The statistical model makes pairwise rationales aggregable and auditable. It combines feature citation and direction, explicitly models position bias, and filters poorly estimated scores. This is a substantive addition to pairwise rubric elicitation, especially where a per-item score matrix is needed (pp. 3–5, Appendix B). The evaluation separates several useful comparisons. The synthetic data make feature recovery measurable, with planted dials designed to reduce correlations and length shortcuts (Appendix C). The paper compares direct and distributed LLM baselines at approximately matched cost, tests scoring and filtering ablations in one full-data run (Table 1), and reports paired synthetic resampling (Fig. 15, Table 7) and CMV accuracy intervals (Table 2). In CMV, the filtered discovered features outperform sample-elicited features by 3.6 points on average across matched feature counts (95% CI [1.8, 5.4]; p. 9). The real-data protocol topic-matches reply pairs and probes outcome memorization (Appendix D).

Critical weaknesses

Lack of showing effectiveness of the method on 1) simulated dataset to show the power of theory cleanly 2) more datasets/tasks

Figure 4: "An open marker denotes a construct absent from that judge's reliable basis..." needs figure improvement. The open marker is not clear to me.

Following are the statements made by agents that I reviewed and agreed on The real-data results establish prediction, but only weakly establish feature validity or the added value of the joint estimator. CMV has no ground-truth feature labels or independent human ratings of the discovered constructs. The best discovered-feature accuracy is 62.63% versus 63.75% for hand-engineered features; the paper does not report a paired test of that gap. A matched-training-size embedding baseline is 61.63% (Table 2), and a full-training-set embedding arm reaches 64.88% with more labeled pairs (Table 9). On CMV, the proposed joint estimator does not produce the scores used for prediction or the Fig. 3 feature associations: these come from separate LLM pointwise ratings or per-feature Bradley–Terry fits. The filtered 16-feature set scores slightly below the unfiltered 29-feature set in full-set pointwise accuracy (62.13% versus 62.63%); at matched feature count the filtered subsets average about one point higher, with 95% intervals [−0.1, 2.2] for pointwise and [0.0, 2.1] for BT (p. 9). The Fig. 3 “effectiveness” associations are observational and unadjusted for reply length, which alone predicts 59.25% of CMV outcomes (Table 2). The conclusion's claim of significantly better feature recovery on real data exceeds what CMV can test without ground-truth features (p. 9). Thus the real-data study supports useful feature elicitation more clearly than it validates the joint estimated scores as measurements of persuasion mechanisms.

The formal guarantees and strongest statistical ablations do not directly establish the estimator's practical benefit. Theorem 1 is for a dense, single-feature design, while the experiments use 40 comparisons per item and many features (pp. 4–5; Appendices C–D); the authors acknowledge this limit. Appendix B.3 gives fixed-item, increasing-comparison asymptotics for sandwich intervals. Out-of-fold checks show calibration of predicted citation and direction probabilities, but not finite-sample score-interval coverage at degree 40. The 16/16 synthetic recovery is a single-run result; Table 6's matching-resampling band for tracked dials is 87.5–93.8%, corresponding to 14–15/16. This reuses the existing judge outputs and is a run-variance summary, not a confidence interval for 16/16. Table 7's paired corpus bootstrap also reuses existing judgments, reduces average graph degree to 25.3, and supports better tracking and attribution than Direct LLM; its leakage interval reaches zero after rounding. In Table 7(b), the method's discovered features are held fixed. Neither resampling analysis tests the marginal gain of statistical scoring over LLM scoring of the same discovered features; Table 1's attribution and leakage advantages on that comparison are single-run estimates, while tracked dials and statistical reliability are no better. Published pairwise rubric-discovery methods also narrow the broad “no approach” novelty statement on p. 3. This matters because the score estimator is the distinctive contribution.

Questions for authors

Does table 1 include GPT 5.4 models only? On the 300 CMV discovery exchanges, do the proposed joint-model feature scores predict the matched persuasion outcomes under cross-validation?

Theory related good questions from my agent, selected by human: Under paired resampling, does statistical scoring improve attribution and leakage over “Reliable features + LLM scoring” on the same discovered features? At degree 40, what is the coverage of the per-item score intervals and how stable are κ/ρ filter decisions across freshly judged graphs?

4 comments

Thank you for this detailed review. You asked us to show the method's effectiveness on more datasets and tasks, and to isolate cleanly what the statistical model contributes. We have uploaded a revised paper (changes in color) with two new experiments on real data, two new published baselines, and the statistical analyses you asked for. We answer each point below.

New experiment 1: recovering an expert taxonomy on real data (Section 4.2). This experiment provides the real-data ground truth that CMV lacks.

  • The original study. Cemri et al. ("Why Do Multi-Agent LLM Systems Fail?", NeurIPS 2025 Datasets and Benchmarks) released execution traces, i.e. full logs of the messages exchanged by cooperating LLM agents, from two systems: two GPT-4o agents solving maths word problems (AG2), and four agents trying to fix GitHub bugs (HyperAgent, SWE-bench Lite). Six experts read the traces and, over more than 120 hours, built a taxonomy of failure categories (MAST). On 54 traces, they recorded which categories occur.
  • Our experiment. We keep the 249 released traces whose task failed. Without access to the experts' annotations, our method compares failed traces pairwise, with the judge asked why each failed and how the failures differ, and discovers features from these explanations. We run it on the 54 annotated traces, then compare the discovered features with the 16 categories the experts flagged on them.
  • Results. Our method finds a counterpart to 12 of the 16 categories (5 direct, 7 partial where our feature is broader), which together account for 94 of the 108 flags the experts ticked. The 4 it misses all concern communication between agents. It also surfaces failure types the categories do not cover: substantive errors (wrong file, misdiagnosed root cause, a fix that does not match the bug), which the experts' notes mention but MAST leaves out by design, and environment and tool failures, which are within MAST's scope but have no category. Cost: USD 2.78 (USD 20.08 on all 249 traces) and no expert time, against 120+ expert-hours.

New experiment 2: detecting AI-written peer reviews, at scale (Section 5.2).

  • Why this task. Conferences now receive reviews drafted by LLMs. A useful detector should say why a review looks AI-written, and keep working when the model or prompt that wrote it changes. The task also tests our method on a new domain, at a larger scale (1,481 papers, 7,405 reviews), and under distribution shift.
  • Setting. For 1,481 ICLR 2020-2022 submissions, we pair one human review, written before ChatGPT's release, with a review that GPT-5.4 mini writes from the same submitted PDF, prompted as a reviewer of that year. The task is to predict which review of a pair is AI-written, and, separately, to classify each review on its own.
  • Protocol. Our method discovers 30 features from the 300 reviews of 150 training papers. The judge is asked only how two reviews differ as pieces of reviewing and writing, and is never told that authorship is at stake. All features are then scored on every review by a cheap pointwise scorer, and a sparse logistic regression is trained on training papers and tested on 1,081 held-out papers. To check that the features capture something general rather than one model's habits, we also test on AI reviews written by another model (Claude Haiku 5.5), and by a prompt we wrote after discovery to remove the differences our features capture.

(continued below)

  • Results. The discovered features reach 99.7% held-out paired accuracy, and keep 89.5% to 99.2% when the model or the prompt changes, including under the prompt designed against them.
  • What it shows. Features found without labels, from only 300 reviews, transfer across generators and prompts on a practical task. The costly pairwise step ran on 300 items, and only the cheap scoring grows with the dataset. For transparency: 18 features that GPT-5.4 proposes without seeing data do as well on this task, and better under the adversarial prompt (by 2.8 and 4.1 points), which the paper reports. Here, the advantage of our method is over other data-driven methods (next item).

Two published baselines, on all three tasks. LLooM (Lam et al., 2024) has an LLM summarize and cluster a corpus into concepts, without labels; HypotheSAEs (Movva et al., 2025) interprets sparse-autoencoder neurons selected with outcome labels.

  • Synthetic letters. LLooM tracks 8 of the 16 planted dials and HypotheSAEs 1, against 16 for ours.
  • CMV. They reach 59.25% and 58.13%, against 62.63% (not significant on the full sets; at equal size, HypotheSAEs is below ours at every k).
  • ICLR reviews. Under a change of model or prompt, ours lead them by 2.2 to 8.5 and 15.6 to 37.3 points.

Figure 4, absent constructs. Constructs a judge did not surface are now drawn as crosses, offset from the center so they cannot be read as a zero estimate, and the caption says so.

CMV: are the features real and correctly assessed? CMV has no ground truth, so it cannot establish validity on its own: the synthetic benchmark and the new agent-trace experiment do that. What CMV can show is that the features are measured consistently (Appendix D.1.4):

  • median per-feature reliability is 0.83 for pointwise scoring (test-retest) and 0.86 for Bradley-Terry (split-half, Spearman-Brown);
  • the two independent protocols agree at a median Spearman of 0.75 over the 1,600 held-out replies (minimum 0.60).

CMV: paired tests between arms. We added exact McNemar tests on the 800 held-out pairs, paired bootstrap CIs, and a Holm correction over 16 contrasts. No difference between the interpretable sets and the high-dimensional baselines is significant: against Tan et al., -1.1 pp [-4.5, +2.3], p = 0.57; against embeddings, +1.0 pp, p = 0.65; against full-data embeddings, -2.3 pp [-5.9, +1.4], p = 0.26. The revision states that the discovered features are statistically indistinguishable from these baselines, and that with 800 pairs the minimum detectable gap is about 3-4 pp.

CMV: the joint estimator does not produce the prediction scores. This is correct, and Section 5.1 now describes this two-stage design. On CMV, the estimator selects the features (the κ\kappa/ρ\rho reliability filter), and the prediction scores come from separate pointwise or per-feature Bradley-Terry scoring. The design is intentional: features are discovered pairwise on 300 exchanges and then scored more cheaply on all 2,400, which is also how the method scales on the ICLR task.

CMV: filtered vs unfiltered features. The 0.5 pp gap between the unfiltered and filtered sets (62.63% vs 62.13%) is not significant (p = 0.76), and the filtered predictor uses 4 non-zero coefficients against 12. The 13 features the filter removes are, on their own, about 4 pp less predictive than a size-matched subset of the kept ones (58.0% vs 62.0% pointwise, 56.0% vs 60.7% Bradley-Terry). The filter thus halves the feature set without a significant loss of accuracy. For transparency, a citation-frequency filter keeps 14 of the same 16 features on CMV and performs comparably (identical pointwise predictions; -1.4 pp with Bradley-Terry, p = 0.35).

(continued below)

CMV: adjusting for reply length. You are right that Figure 3 should account for length. We refitted each feature's association with the difference in log reply length as a covariate (Appendix D.1.4):

  • The ranking is largely preserved: Spearman 0.88 with Figure 3's effectiveness under pointwise scoring, 0.86 under Bradley-Terry.
  • Fewer features remain significant: 8/29 (pointwise) and 4/29 (Bradley-Terry) at BH q < 0.05, against 21 and 17 unadjusted. No feature gains significance.
  • Sign changes occur only among features that are not significant after adjustment.

Since longer replies carry more evidence and examples, length is partly a mediator, so the adjusted coefficients are a conservative lower bound. The caption of Figure 3 now reports the adjusted result, and a new appendix figure shows both versions side by side.

CMV: the conclusion's claim about real data. We agree that CMV cannot support a recovery claim without ground-truth features. The conclusion now restricts "recovers features better than a direct LLM call" to the synthetic benchmark, and supports the real-data claim with the agent-trace experiment (Section 4.2), where the reference features are the experts' categories.

Practical benefit of the estimator. Two analyses address this. The first is the same-feature comparison you ask for (next item). The second replaces the κ\kappa/ρ\rho filter with a frequency filter on the synthetic benchmark (the 21 most-cited features, 19 of them shared with ours), and every recovery metric drops: Tracked from 100% to 93.8%, Sem.R. from 100% to 87.5%, attribution from 76.2% to 71.4%. For CMV, see "filtered vs unfiltered features" above.

Statistical vs LLM scoring of the same features (your question on paired resampling). We ran exactly this test, over the 50 paired letter resamples of Table 7 of the submission, with the 21 discovered features held fixed:

  • Attribution. Statistical scoring raises it in 100% of resamples: +0.11 [0.05, 0.19] against one-shot LLM scoring, and +0.12 [0.05, 0.19] against 20-repetition LLM scoring.
  • Leakage. It is lower than with 20-repetition LLM scoring (-0.009 [-0.017, -0.000], 98% of resamples); against one-shot scoring, the difference is not significant.
  • Statistical recovery. 20-repetition LLM scoring does better (statistical minus LLM: -0.045 Stat.R. [-0.063, -0.025]). Tracking is equivalent.

On the same features, the model's gain therefore lies in attribution and leakage, not in tracking, and Section 4.1 now states it that way. These resamples keep only comparisons whose two letters are both drawn (average degree 25.3 instead of 40), which handicaps only the statistical arm.

16/16 is a single run. We agree, and the main text now reports the resampling distribution. With the taxonomy re-discovered in every replicate, our method tracks 14.8/16 dials on average, and at least 15 in 76% of replicates under both resampling schemes; Direct LLM tracks 12.6 on the same letter resamples. The drop from 16 comes from the reduced comparison budget, not from the re-discovered taxonomy: with the taxonomy frozen at a similar budget (25 of the 40 matchings), our method tracks 14.6. The dial it then misses is employer prestige, which Direct LLM never tracks either, while two of the three dials Direct LLM misses in the full run are tracked by ours in 86-100% of replicates.

Novelty statement. The related-work section now states the novelty precisely. ICAI and VibeCheck also have an LLM describe how two items differ, but their pairs are fixed by labels or by model identity (two models' outputs on the same prompt), labels drive feature selection, and each feature is then scored in a separate LLM pass. Our novelty lies in comparing arbitrary unlabelled items along a comparison graph for feature discovery, and in reusing the same rationales, through the statistical model, to measure and select features with per-item scores and reliability diagnostics.

Does Table 1 of the submission (Table 2 in the revision) use GPT-5.4 models only? Yes. All 52,742 cached responses behind it come from gpt-5.4-mini-2026-03-17 with no reasoning (judge, all scorers, Direct LLM scoring) or gpt-5.4-2026-03-05 with high reasoning (taxonomies, rubric elicitation and fusion). text-embedding-3-large is used only for evaluation (semantic matching) and to pick medoid descriptions in the Local-LLM rubric fusion. The revision states this in Section 4.1.

We thank you again for this careful review, which led to most of the new analyses in the revision, and we remain available to discuss any further questions before the end of the discussion period.

Thank you for a thorough revision. You ran the specific tests I asked for, including the ones whose results do not favor the method, and you narrowed the claims accordingly.

Remaining requests

  1. Baselines on the agent-trace experiment. You report LLooM and HypotheSAEs on the synthetic letters, CMV, and ICLR reviews, but the agent-trace experiment is the one with real-data ground truth and appears to carry no comparison arm. What do Direct LLM, LLooM, and HypotheSAEs recover of the 16 expert categories on the same 54 annotated traces, at matched cost? Without that, the experiment establishes that the method works, not that it works better, and it is now load-bearing for the paper's central claim.
  2. How the feature-to-category matching was made. 7 of the 12 recoveries are "partial, where our feature is broader," which is the category most sensitive to the matcher's generosity. Please state who or what performed the matching, whether it was blind to which arm produced each feature, and whether the same procedure was applied to any baseline. If an LLM did the matching, a human spot-check or a second independent matcher with agreement reported would help.
  3. The four missed categories. That all four concern inter-agent communication reads as structural rather than incidental: pairwise "why did each fail and how do the failures differ" may systematically surface item-level properties over relational or process-level ones. I would rather see this named as a limitation of the elicitation design than attributed to sampling.
  4. Length adjustment on CMV (Figure 3). Significance falling from 21/29 to 8/29 (pointwise) and 17/29 to 4/29 (BT) is a large reduction, even granting that length is partly a mediator. Please make the adjusted figure the primary one in the main text, with the unadjusted version in the appendix, rather than reporting the adjustment in the caption.
  5. AI-review detection at 99.7%. Near-ceiling paired accuracy makes me want the floor established: what do length-only and simple stylometric baselines (punctuation, hedging, section structure, sentence-length variance) achieve on the same held-out pairs?
  6. Two items from my questions that I do not think were answered. (a) Coverage of the per-item score intervals at degree 40 (b) Stability of the kappa/rho filter decisions across freshly judged graphs; the resampling analyses reuse cached judgments, and the frequency-filter comparison answers a different question.
  7. Agreement of the joint-estimator scores with the prediction scores. The median Spearman of 0.75 you report is between the two separately elicited protocols (pointwise and BT). What I asked is whether the joint model's own scores track those, and whether they predict outcomes on the 300 discovery exchanges under cross-validation.
  8. Prior work from my review. Please confirm whether Rezaei et al., Online Rubrics Elicitation from Pairwise Comparisons (ICML 2026) and Liu et al., OpenRubrics (ACL 2026) are now cited and distinguished.

Updated assessment

Soundness: raised from 2 to 3. The remaining items are reporting and scoping requests, not unresolved soundness problems.

Reviewer_eHeJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a pipeline to automate "interpretable" feature discovery in unstructured text data. The basic idea is to have an LLM compare pairwise datapoints to extract different features and then collate them (in a data prep pipeline). The final evaluation puts these into a classifier to see their predictive performance, which almost does as well as hand engineered features.

Critical strengths
  • Automated feature discovery has large implications if solved, especially in domains where humans have little understanding, hence I think this is a fascinating topic that needs a lot more work.
  • The usage of Bradley-Terry etc. is smart and well motivated
  • Mostly reasonable baselines
Critical weaknesses
  • The paper evaluates a synthetic dataset, and a relatively simple reddit one, hence the paper is really a case study, with very limited data (a few hundred datapoints)
  • There is little practical impact demonstrated. The real impact of work in this areas would be to highlight scientifically novel features we didn't know existed before, but this paper is far from achieving that.
  • A baseline that asks an LLM to propose features without seeing the dataset achieves 61.25% accuracy, versus 62.63% for the proposed method. The modest improvement raises questions about whether the additional complexity of pairwise comparison and statistical estimation is justified in real-world applications.
  • In the synthetic experiment, the method costs USD 3.45 compared with USD 0.26 for direct prompting.

Overall, the paper's direction is extremely interesting, and I would encourage the authors to keep pursuing it. However, this currently paper has too little evidence of broader scientific significance for ICLR in my opinion.

Questions for authors

See weaknesses

2 comments

We thank the reviewer for this careful and encouraging review. Your main concern was that a synthetic dataset and one Reddit dataset of a few hundred items make the paper a case study, with little demonstrated practical impact. We agree, and we added two experiments on real data to the revised paper (changes in color). The first shows the kind of impact you describe: without labels and for under USD 3, our method recovers most of a failure taxonomy that six experts spent more than 120 hours building, and it surfaces failure types that their taxonomy does not cover. The second shows that the method scales to thousands of items, because the costly pairwise step only needs a few hundred.

1. Practical impact: recovering, and extending, an expert taxonomy (new Section 4.2).

The original study. Cemri et al. ("Why Do Multi-Agent LLM Systems Fail?", NeurIPS 2025 Datasets and Benchmarks) studied why systems of cooperating LLM agents fail. They released execution traces, i.e. full logs of the messages the agents exchange while working on a task, from two systems: AG2, where two GPT-4o agents solve maths word problems, and HyperAgent, where four agents try to fix bugs from real GitHub repositories (SWE-bench Lite). Six experts read the traces and, over more than 120 hours, built a taxonomy of failure categories (MAST). On 54 traces, they also recorded which categories occur, with a short note on what went wrong.

Our experiment. We keep the 249 released traces whose task failed, 54 of which carry the experts' annotations. Our method never sees these annotations. The LLM judge compares two failed traces at a time and explains why each failed and how the failures differ; our pipeline then turns these explanations into features. We run it on the 54 annotated traces and compare the discovered features with the 16 categories the experts flagged on them.

Results. Our method finds a counterpart to 12 of the 16 categories: 5 direct, and 7 partial, where our feature is broader (e.g. "trajectory restart", "step repetition" and "discontinued reasoning" all fall under one feature, Convergence and Completion). These 12 categories account for 94 of the 108 times the experts flagged a category on a trace. The 4 categories it misses all concern communication between agents, such as withholding information (14 flags in total). The method also surfaces failure types that the categories do not cover: substantive errors (editing the wrong file, misdiagnosing the root cause of a bug, writing a fix that does not match it, miscalculating), which the experts' notes mention but MAST leaves out of scope by design, and environment and tool failures, which are within MAST's scope but have no category. We assessed the matches ourselves, and Section 4.2 lists every pairing so that it can be checked.

Why it matters. This run cost USD 2.78 (USD 20.08 on all 249 traces) and no expert time, against more than 120 expert-hours. A researcher studying a new set of failures would get a draft taxonomy covering most of what the experts found, plus candidate failure modes they had not formalized, and could spend expert time checking and refining it rather than building it from scratch.

(continued after)

2. Scale and cost (USD 3.45 vs USD 0.26 on the synthetic letters).

  • Fewer comparisons suffice. Our method compares each of the 200 letters with 40 others. With about 15 comparisons per letter (USD 1.44), it already tracks as many of the 16 planted ground-truth features as Direct LLM, which gives a strong LLM the whole dataset in one prompt (13 of 16). Conversely, spending our budget on the direct approach does not close the gap: at matched cost, Direct LLM with multiple scorers still misses the same 3 features.
  • Scoring scales separately from discovery. Pairwise comparisons are needed to find the features; once found, every item can be scored by a cheap pointwise scorer. We test this on a new, larger task (Section 5.2): detecting AI-written peer reviews. For 1,481 ICLR 2020-2022 papers, we pair a human review, written before ChatGPT's release, with a review that GPT-5.4 mini writes from the same paper, and predict which of the two is AI-written. Features are discovered pairwise from only 300 reviews, without the judge ever being told that authorship is at stake. They are then scored on all 7,405 reviews (the human review and four AI versions per paper) by jev-1.13.0, a dedicated rubric-scoring model whose scores agree with GPT-5.4 mini's about as well as two GPT-5.4 mini scorings agree with each other (median Spearman 0.81 vs 0.79). Scoring cost USD 0.79 on the training version, whereas discovery on every review would take about 150,000 comparisons. In every experiment, discovery ran on at most 300 items: what grows with the dataset is only the scoring.
  • Results. The discovered features reach 99.7% held-out paired accuracy. When the AI reviews come from another model (Claude Haiku 5.5), or from a prompt we wrote after discovery specifically to remove the differences the features capture, they keep 89.5% to 99.2%, and lead LLooM and HypotheSAEs (data-driven baselines another reviewer pointed out) by 2.2 to 8.5 and 15.6 to 37.3 points respectively.

3. The gain over LLM-proposed features (62.63% vs 61.25%). We agree, and the new experiments reinforce your point, so we state it plainly. On the same 800 held-out CMV pairs, the difference is +1.4 pp (95% CI [-1.6, +4.5], exact McNemar p = 0.43), and every gap to Tan et al.'s 3,205 hand-engineered features and to 3,072-dimensional embeddings is at most 2.75 pp (p >= 0.17). On the ICLR task, 18 features that GPT-5.4 proposes without seeing any review do as well as ours under the original prompt, and better under the prompt written to evade our features (by 2.8 and 4.1 points). Persuasion and the style of AI-generated text are both widely studied, so an LLM's priors are already good there, and the revised paper reports that on these tasks our features match, rather than beat, features proposed without data. The added complexity pays off elsewhere:

  • When the features must be found in the data. On the synthetic letters, Direct LLM sees the whole dataset and still misses 3 of the 16 planted features, even at matched cost. On the agent traces, our method surfaces failure types that the experts' taxonomy had not formalized.
  • Less redundancy. On CMV, 16 LLM-proposed features span about four effective dimensions, against six to eight for 16 of ours (Figure 2).
  • Against other data-driven methods. On all three tasks, ours do at least as well as LLooM, which also needs no labels, and HypotheSAEs, which uses them (on CMV, the gaps on the full sets are not significant).
  • The statistical step. Holding the discovered features fixed, statistical scoring makes more features attributable to a single planted feature in every resample (+0.11 [0.05, 0.19]), and on CMV its reliability filter halves the feature set (29 to 16) without a significant loss of accuracy.

We hope these two experiments address your concerns about scope and practical impact. If you see a remaining gap, we would be glad to discuss it before the end of the discussion period.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Comment on possibly misdirected reply

Dear Reviewer,

Thank you again for responding to our rebuttal earlier in the discussion.

We think your most recent comment (the one this answer responds to) may have been meant for another submission. It discusses an attention-edge ablation, an x-substitution control, Figure 7a, Appendix A.3 filtering, and results on Qwen3-8B and Gemma-4-12B. None of these appear in our paper.

We’re mentioning it in case the other paper’s authors haven’t received your comment. We’d also ask you to confirm that the score shown on our submission still reflects your assessment of our paper.

Best regards, The Authors

Yes, I manually opened too many Chrome tags today with ARR review deadlines, and got it wrong. Corrected. Thank you for the reminder. I will make my agent double check my human operations in the future :). I am wondering if my agents would be better than me now regarding this manual task.

"The mathematical claims and proofs were developed by the authors without AI assistance; as an additional verification step, we used Claude Fable 5 (Anthropic) to audit the proofs, which surfaced an error; the authors diagnosed and corrected it by hand.

Regarding the appendix: I would suggest employing an agentic system to verify the proof, and briefly mention how you audit those proof. I recommend including formal verification methods such as Lean and Coq.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.