acceptodds
Under review as a conference paper at ICLR 2027

Conditional Hypothesis Generation for LLM-Based Text Analysis with Researcher-Specified Covariates

Abstract

An important goal of text analysis is to discover interpretable hypotheses about how language varies with outcomes of interest. Recent LLM-based methods select globally discriminative patterns that do not necessarily answer a researcher’s question. We introduce , the task of generating hypotheses that distinguish outcome groups within researcher-specified covariate strata. Covariates encode domain knowledge about which comparisons matter and can guide hypothesis discovery. We study two statistical challenges that arise in conditional discovery: relevant strata may be underrepresented within outcome groups (), and differences may reverse direction across strata (). Using sparse autoencoder features, we examine stratified regression, interaction models, and demeaning for covariate-aware selection. Controlled synthetic experiments show that these approaches can improve recovery of targeted differences overlooked by global selection, with effectiveness depending on the structure of conditional associations. On two real-world datasets, experts rate the covariate-aware method’s unique hypotheses as more helpful on average than the global baseline’s. We also illustrate how conditional comparisons can help researchers interpret generated hypotheses and formulate new ones.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
hypothesis generation, covariate adjustment, sparse autoencoders, computational social science

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yTxYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper introduces conditional hypothesis generation, where researchers specify covariates that define the contexts within which textual outcome-group differences should be discovered. This is to address the problem that globally discriminative patterns can be irrelevant or misleading when strata are imbalanced or when associations reverse direction across strata. It is built on an earlier approach of Movva et al. (2025), which first maps documents to SAE features and then selects discriminative features via LASSO. The paper modifies the feature selection stage with covariate-aware methods including stratified regression, interaction models, and demeaning. Selected SAE features are subsequently verbalized as natural-language hypotheses by an LLM. Evaluation results from synthetic experiments and small expert study verify the effectiveness of the proposed approach.

Critical strengths
  • The paper tackles an interesting problem in interpretable hypothesis discovery in text analysis. The globally discriminative features in data may not enable meaningful comparison a researcher wants to make. The paper addresses this by allowing researchers to define relevant groups using covariates, and introduces conditional hypothesis generation which generates discriminative hypotheses within researcher-specified covariate strata. It also distinguishes associations that hold across groups, appear only in particular groups, or reverse direction between groups.

  • The paper examines two statistical challenges in discovering conditional differences: unequal representation across groups and associations that reverse direction between groups. It extends a previous SAE feature selection approach (Movva et al., 2025) and proposes a few covariate-aware feature selection methods. The experimental results show that stratified-max and demeaned-reweighted LASSO perform best under group imbalance, while interaction LASSO performs best when associations reverse direction.

Critical weaknesses
  • The paper has limited methodological novelty. The proposed approach was built on (Movva et al., 2025). Both SAE training and LLM interpretation are the same as in the previous approach. The four proposed variants in the feature selection stage (stratified, interaction, demeaned, and reweighted LASSO) are standard techniques for confounding and effect modification. Also, the paper positions itself as the first covariate-aware hypothesis-generation method, but covariate-adjusted text discovery has been around for a long time. For example, structural topic models (Roberts et al., 2014) model topic prevalence and content as functions of covariates; and deconfounded lexicon induction (Pryzant et al., 2018) uses residualization which is similar to the proposed within-stratum demeaning. The paper should include at least one such method (e.g., structural topic models with LLM topic labeling) as a baseline.

  • The paper claims to address sign reversal, but it is unclear how it works. In the synthetic experimental setup, the target is the set of subtopics that characterize corpus A before 2011, even though these subtopics characterize the other corpus after 2011. Both Interaction-LASSO and stratified-max rank features by the strength of their coefficients or interaction effects, regardless of direction. In the feature verbalization phase, each selected feature is described using text that does not specify the period or which corpus it characterizes. It is therefore unclear whether the pipeline identifies the intended association before 2011 or simply detects that the association differs across periods.

Roberts et al., 2014. Structural topic models for open‐ended survey responses. American journal of political science, 58(4), pp.1064-1082.

Pryzant et al., 2018. Deconfounded lexicon induction for interpretable social science. NAACL.

Questions for authors
  1. Could the authors clarify the main technical contribution in comparison to the approach in (Movva et al., 2025)? Since the feature selection methods are standard techniques (interaction terms, fixed-effects demeaning, inverse-frequency weighting), what is adapted specifically for conditional hypothesis generation?

  2. Why not compare with existing covariate-aware text methods, such as structural topic models (Roberts et al., 2014) and deconfounded lexicon induction (Pryzant et al., 2018)?

  3. In the sign-reversal experiment, how does the proposed approach distinguish "A over B before 2011" from the reverse association after 2011? Are the evaluation metrics sensitive to direction?

  4. In real-world applications, the presence of sign reversal may not be known. In such a case, how should one choose among the feature selection methods mentioned in the paper?

3 comments

W1a: "The paper has limited methodological novelty. [...] standard techniques for confounding and effect modification."

Q1: "[...] main technical contribution in comparison to the approach in (Movva et al., 2025) [...]"

Our main contribution is defining the problem of conditional hypothesis generation. Like other LLM-based methods, Movva et al. (2025) rank attributes by the marginal difference DhD_h between outcome groups. The revised §3.1 decomposes DhD_h into a within-context term and a composition term (Kitagawa, 1955). Global selection can therefore favor an attribute through composition alone. It can also miss within-context differences that get little weight in DhD_h because their stratum is underrepresented (stratum imbalance), or that cancel across strata (sign reversal). We instead define the task by the conditional differences Δh(c)\Delta_h(c), with three targets that answer common research questions: the adjusted difference (what differs after accounting for CC), the context-specific difference (what differs in a particular context), and heterogeneity (how differences vary across contexts).

The statistical tools are standard by design. Covariate adjustment and heterogeneity analysis are well established for testing attributes specified in advance, and we bring their targets to discovery, where the attributes are not yet known. Each approach scores SAE features by their within-stratum associations instead of DhD_h, and §3.2 shows how each relates to the targets:

  • Stratified-max ranks features by their largest context-specific difference.
  • The demeaned variants remove the composition term, and reweighting keeps small strata from being dominated by large ones.
  • Interaction-lasso also scores how a feature's association changes across strata, so sign reversal becomes a signal instead of cancelling out.

Because the SAE features are fixed before selection, covariates can enter directly into the selection.


W1b: "the paper positions itself as the first covariate-aware hypothesis-generation method, [...] deconfounded lexicon induction (Pryzant et al., 2018) uses residualization which is similar to the proposed within-stratum demeaning. [...] the paper should include at least one such method (e.g., structural topic models with LLM topic labeling) as a baseline."

Q2: "compare with existing covariate-aware text methods, such as structural topic models (Roberts et al., 2014) [...]"

We will make our claim precise: it concerns LLM-based hypothesis generation. Our contribution brings the conditional targets of §3.1 into this setting. Covariate-adjusted text analysis has a long history, and we will discuss it more explicitly in related work, including STM and DLI. As the reviewer notes, DLI's residualization is close to our demeaning step.

We added STM as the covariate-aware baseline the reviewer suggested, in two versions: topic prevalence depends on the outcome only (STM), or on both the outcome and the covariate, including their interaction (STM + covariate). We select the topics whose prevalence differs most between outcome groups (overall for STM, within a stratum for STM + covariate). An LLM labels each selected topic by contrasting documents with a high proportion of that topic against documents from other topics, and the label serves as the hypothesis. To compare all methods on equal terms, each one selects its hyperparameters (the SAE config, or the number of topics for STM) by AUC on a validation set, without the reference hypotheses. The scores are therefore lower than in the paper's tables, which selected the SAE configs using the references, but our selectors' significant gains over lasso appear in the same settings.

Surface similarity / F1, mean over 10 seeds. † better than lasso, p<.05.

SettinglassoSTMSTM + covariatestratified-maxdemeaned-reweighted-lassointeraction-lasso
Imbalance, r=.50.330/.296.273/.257.537†/.454†.573†/.481†.610†/.558†.130/.085
Imbalance, r=.35.227/.150.163/.152.590†/.497†.677†/.643†.663†/.601†.147/.137
Imbalance, r=.20.253/.204.233/.212.543†/.424†.767†/.550†.730†/.636†.253/.151
Sign reversal.266/.254.354/.347.440†/.465†.382/.369†.260/.251.428†/.407†

Under stratum imbalance, stratified-max and demeaned-reweighted-lasso score higher than STM + covariate at every ratio, significantly so when the targeted strata are rarest (r=.20, p≤.016 on surface similarity). Under sign reversal, interaction-lasso is comparable to STM + covariate. As expected, conditioning also matters beyond SAE features: STM + covariate significantly improves on STM in every imbalance setting (p≤.008). We will add the STM baseline and these results to the revised paper.

W2: "The paper claims to address sign reversal, but it is unclear how it works. [...] rank features by the strength of their coefficients or interaction effects, regardless of direction. [...] whether the pipeline identifies the intended association before 2011 or simply detects that the association differs across periods."

Q3: "In the sign-reversal experiment, how does the proposed approach distinguish "A over B before 2011" from the reverse association after 2011? Are the evaluation metrics sensitive to direction?"

We distinguish two directions in the reviewer's question: which corpus an attribute characterizes (the outcome direction) and in which period (the covariate direction). For the outcome direction, we follow the convention in LLM-based hypothesis generation. A hypothesis describes an attribute of the text, and the corpus it characterizes is given by the sign of its association with the outcome, for global and covariate-aware methods alike.

The covariate direction matters in the sign-reversal setting. Our synthetic settings give the methods the covariate but not its direction, as when a researcher knows that the period matters but not which period holds the difference of interest. Without the target period, the methods find both the target subtopics (T1\mathcal{T}_1), which characterize corpus A before 2011, and the mirror-image subtopics (T2\mathcal{T}_2), which characterize corpus A after 2011. Both are genuine differences that reverse across periods.

That said, our evaluation metrics are sensitive to the covariate direction. The reference hypotheses are only the target subtopics, so a hypothesis describing a mirror-image subtopic receives almost no credit. The metrics therefore reward recovering the intended pre-2011 association, not merely detecting that the association differs across periods. The reported scores are also conservative, since the mirror-image hypotheses are valid differences under this setup but count as misses. This accounts for part of the gap between the oracle and the covariate-aware methods in the sign-reversal setting.

An additional setting can give the covariate direction (the target period), for example asking what characterizes corpus A before 2011. The covariate-aware methods then need only a small change to selection: they keep only features that favor corpus A in the target period, then rank them as before. We will consider adding this setting in the revision.


Q4: "In real-world applications, the presence of sign reversal may not be known. In such a case, how should one choose among the feature selection methods mentioned in the paper?"

We expand on the intended-use paragraph in Section 4.4. We envision our approach as an exploratory tool, like other hypothesis generation methods: researchers inspect the generated hypotheses and refine them for further analysis. In practice, researchers can start with stratified-max and demeaned-reweighted-lasso. When there is a plausible reason to expect sign reversal, they can also run interaction-lasso. Researchers can inspect each candidate's within-stratum differences on validation data and test the refined hypotheses on held-out data. Since the held-out data are not used for generation, running more selectors widens the candidate set without compromising that test.

Thank the authors for the thorough response. My main concerns have been largely addressed. I will update my score accordingly.

Reviewer_oZTnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Researchers use AI to find patterns in text that differ between groups, such as speeches by Republicans and Democrats. Current tools pick the most obvious overall differences, but these can reflect background factors and obscure differences within specific contexts. The method developed here lets researchers explicitly name background factors, like whether a speech covers policy or procedure, then searches for differences within each setting. On test data with known answers, the authors show their methods can recover helpful patterns. Experts reviewing congressional speeches and math classroom transcripts rated the new method’s unique findings as more helpful than another approach that applies Lasso to SAE features from text embeddings.

Critical strengths
  • The paper is well-written. I could follow the logic. The problem is also well-motivated in that the biggest overall differences between groups may not answer the researcher’s question. Framing the task around two familiar statistical problems, stratum imbalance and sign reversal, is a useful way to frame the problem.

  • Candid and useful evaluations. They show that the demeaned methods struggle under sign reversal and that interaction-lasso struggles under imbalance, which gives readers practical guidance on choosing a method. The NCTE breakdown is especially valuable. It shows how a covariate can improve interpretation (in the example relating to the link between eliciting student reasoning and remediation quality is much weaker in poorly managed classrooms).

  • The analysis of human raters is very useful validation and a strong point in the paper.

Critical weaknesses
  • The paper's specific methodological contribution is somewhat unclear. Demeaning is a standard fixed-effects technique across empirical work; conditioning on covariates is not new either. Applying these tools to SAE-based hypothesis generation may be new, but I am not fully convinced it produces substantive insight that existing approaches, or permutations thereof, could not. Indeed, the main change relative to existing work is that the work here replaces a global LASSO applied to SAE features (in predicting the outcome of interest, such as party affiliation or teacher remediation quality) with covariate-aware selectors (stratified-max, interaction-lasso, and the demeaned variants). The paper also finds that the best choice depends on whether sign reversal is likely, which a researcher may not know in advance, making the proposed approaches hard to apply confidently in practice.

  • The main empirical results come from synthetic corpora. The real-world evidence rests on four experts rating five method-unique hypotheses per method, which could be somewhat thin. The 3.10 versus 2.50 helpfulness gap rests on ten ratings per method, with, I think, overlapping confidence intervals; the comparison is limited to a subset defined after the fact by an LLM-assisted matching step (what is the population of interest here?). A larger expert panel, intervals or a permutation test on the ratings, and results across all hypotheses rather than only the unmatched ones would make this evidence far more persuasive.

  • Is it true that no existing method handles covariates (Section 4.3)? Structural topic models let covariates shape both topic prevalence and content, and Taddy's (2013) multinomial inverse regression conditions on covariates. As the paper cites both, I would narrow the claim a bit, for example to covariates in LLM-based hypothesis generation or similar.

  • Related to the question about the "target population" of interest in which we want to test our claims here -- the two synthetic-data metrics could use a clearer explanation. Both are computed after an LLM labels the test texts for each generated hypothesis, and the Hungarian algorithm pairs generated and reference hypotheses by how closely their labels correlate. Surface similarity then asks GPT-4.1 whether each pair means the same thing, and F1 measures how well their labels agree. Because the matching step already maximizes label agreement, it is unclear how much F1 adds. The matching also forces every generated hypothesis into a pair, even when an entity fits no reference. The LLM judge gives half credit to loosely related pairs, and the repeatability check shows consistent scores, but not necessarily alignment with human judgments. Interestingly, the random baseline scores are close to global LASSO and sometimes above it (e.g., F1 of 0.30 vs 0.24 at r = 0.35). The metrics may thus reward broad topic overlap rather than recovery of the targeted differences.

  • The paper offers relatively little theoretical analysis of what the various covariate selector approaches "do" here; there is, to my eyes, no clear/explicit link between selectors and the estimand.

Questions for authors

What quantity does each selector estimate, and under what conditions does a high selector score mean that the final hypothesis shows a large difference between outcome groups within the covariate strata?

6 comments

W1a: "The paper's specific methodological contribution is somewhat unclear. [...] I am not fully convinced it produces substantive insight that existing approaches [...]

W5: "The paper offers relatively little theoretical analysis of what the various covariate selector approaches "do" here; there is, to my eyes, no clear/explicit link between selectors and the estimand."

Our main contribution is defining the problem of conditional hypothesis generation: finding textual attributes that distinguish outcome groups conditional on researcher-specified covariates CC. Existing LLM-based methods select attributes by their overall difference between outcome groups, which does not necessarily answer a researcher's question. Covariates encode domain knowledge about which comparisons matter, so researchers can direct what the generation looks for without knowing the hypothesis in advance.

The revised §3.1 makes the estimands explicit and here's the summary. We further decompose the overall difference, DhD_h, into two terms (Kitagawa, 1955): a weighted average of the within-stratum differences Δh(c)\Delta_h(c), which compare the outcome groups among texts in stratum cc, and a composition term, which arises because the groups' texts are distributed differently across strata. Global selection can therefore favor an attribute through the composition term alone. It can also miss within-stratum differences when their stratum is underrepresented (stratum imbalance) or when they cancel across strata (sign reversal). We define the task by the conditional differences Δh(c)\Delta_h(c) rather than DhD_h, with three targets that answer common research questions: the adjusted difference (what differs after accounting for CC), the context-specific difference (what differs in a particular stratum), and heterogeneity (how differences vary across strata).

The statistical tools are standard by design, and we do not claim a new estimator. Covariate adjustment is established for testing attributes specified in advance, and we bring their targets to discovery, where the attributes are not yet known. A new paragraph in §3.2 states what each selector's score tracks and when a high score reflects a large Δh(c)\Delta_h(c):

  • Stratified-max scores a feature by its largest context-specific difference, so a high score reflects a large Δh(c)\Delta_h(c) in at least one stratum, even a rare one.
  • Demeaned-reweighted-lasso (DR-lasso) tracks an adjusted difference, with the composition term removed and small strata reweighted, so a high score reflects Δh(c)\Delta_h(c) that are large in the same direction across strata (opposite signs cancel).
  • Interaction-lasso scores how a feature's association changes across strata, so a high score reflects heterogeneity, and sign reversal becomes a signal instead of cancelling out.

Fitting the global selector separately in each stratum, the most direct permutation of existing approaches, reaches only the context-specific target (stratified-max), while the other two targets need the strata modeled jointly.

Kitagawa, Evelyn M. (1955). "Components of a Difference Between Two Rates". Journal of the American Statistical Association. 50 (272): 1168–1194


Q: "What quantity does each selector estimate, and under what conditions does a high selector score mean that the final hypothesis shows a large difference between outcome groups within the covariate strata?"

Our response above states the quantity each selector's score tracks and when a high score reflects a large within-stratum difference Δh(c)\Delta_h(c). To show empirically that a high score yields final hypotheses with a large difference between outcome groups within the covariate strata, we estimated the within-stratum differences Δh(c)\Delta_h(c) on held-out texts for every hypothesis generated in the synthetic settings. W2a below reports the same check on Congress.

Share of generated hypotheses with a statistically significant held-out within-stratum difference in a targeted stratum. Mean over 10 seeds. † better than lasso, p<.05.

Settinglassostratified-maxDR-lassointeraction-lasso
Imbalance, r=.50.47.93†.87†.23
Imbalance, r=.35.37.93†.90†.13
Imbalance, r=.20.30.80†.87†.27
Sign reversal (pre-2011 stratum).66.96†.70.92†

The results follow the targets: stratified-max holds in both scenarios, DR-lasso under imbalance, and interaction-lasso under sign reversal. We will add the held-out analysis to the appendix.

W1b: "The paper also finds that the best choice depends on whether sign reversal is likely, which a researcher may not know in advance, making the proposed approaches hard to apply confidently in practice."

We expand on the intended-use paragraph in Section 4.4. We envision our approach as an exploratory tool, like other hypothesis generation methods: researchers inspect the generated hypotheses and refine them for further analysis. In practice, researchers can start with stratified-max and demeaned-reweighted-lasso. When there is a plausible reason to expect sign reversal, they can also run interaction-lasso. Researchers can inspect each candidate's within-stratum differences on validation data and test the refined hypotheses on held-out data. Since the held-out data are not used for generation, running more selectors widens the candidate set without compromising that test.


W2a: "The main empirical results come from synthetic corpora. The real-world evidence rests on four experts rating five method-unique hypotheses per method, which could be somewhat thin."

We originally evaluated one covariate per dataset, fixed before the expert study. We have now added a held-out analysis on Congress, where we examined additional covariates and report the two with clear conditional structure: (1) whether a speech is procedural, and (2) whether it discusses cultural or social issues. Whether conditional differences can be found depends on the data as well as the method. For each method's 10 hypotheses, we check on held-out speeches whether the Republican–Democrat difference is concentrated in the covariate stratum: the within-stratum difference is statistically significant and at least twice as large as the difference across all speeches.

Number of hypotheses (out of 10 per method) whose difference is concentrated in the covariate stratum.

CovariatelassoDR-lassostratified-max
Procedural speech267
Cultural or social issues157

The covariate-aware selectors surface more such hypotheses than lasso, about half of which lasso does not produce at all. Some examples:

Republican − Democrat difference in the share of speeches with the attribute (percentage points).

HypothesisCovariateOverallIn stratumOutside
moves to suspend the rules and passProcedural speech+3.3+9.5+0.2
discusses illegal immigrationCultural or social issues+1.9+8.0+1.4
discusses poverty or the poorCultural or social issues−5.8−15.1−1.6

Within the stratum of speeches about cultural or social issues, Republicans emphasize illegal immigration and Democrats emphasize poverty, with gaps two to four times larger than overall. Outside this stratum, the parties barely differ on either. We will add this analysis to the paper.


W2b: "The 3.10 versus 2.50 helpfulness gap has overlapping confidence intervals. [...] The comparison is limited to a subset defined after the fact by an LLM-assisted matching step [...] A larger expert panel, intervals or a permutation test on the ratings, and results across all hypotheses [...] would make this evidence far more persuasive."

We want to clarify that experts rated the complete, method-blinded lists. The matching only split them afterwards for reporting. Because that split was made after the fact, we now report the comparison across all hypotheses, with the unique ones for reference.

Expert helpfulness ratings (1–5, two experts per hypothesis), pooled over Congress and NCTE. 95% bootstrap confidence intervals and p-values from exact permutation tests.

Hypotheses ratedlassoDR-lassoDifference (DR − lasso)p
All (20 per method)2.952.90−0.05 [−0.62, +0.50].94
Unique to each method (5 per method)2.503.10+0.60 [0.00, +1.20].23

Across all hypotheses, the two methods are rated about the same, as expected since three quarters of the hypotheses are found by both. The hypotheses unique to DR-lasso are rated at least as helpful as lasso's, though not significantly so due to small sample size. This fits the goal of the expert study: to surface hypotheses the global baseline misses that are as useful, not to outperform it. We will make this goal clearer in the revised paper. We agree that a larger expert panel would make this evidence more persuasive. Domain experts are hard to recruit, and two experts per domain is in line with prior evaluations of hypothesis generation (Zhong et al., 2023; Movva et al., 2025). We will note the panel size as a limitation in the revised paper.

W3: "Is it true that no existing method handles covariates (Section 4.3)? Structural topic models let covariates shape both topic prevalence and content, and Taddy's (2013) multinomial inverse regression conditions on covariates. [...] narrow the claim a bit [...]."

We agree and will make our claim precise: it concerns LLM-based hypothesis generation. We will discuss STM and Taddy's (2013) multinomial inverse regression, which condition on covariates when modeling topics or word counts, in related work. Our contribution brings the conditional targets of §3.1 to the discovery of natural-language hypotheses.

We also added STM as a covariate-aware baseline, in two versions: topic prevalence depends on the outcome only (STM), or on both the outcome and the covariate, including their interaction (STM + covariate). An LLM labels each selected topic, and the label serves as the hypothesis. For a fair comparison, each method selects its hyperparameters (the SAE config, or the number of topics for STM) by AUC on a validation set, without the reference hypotheses. Scores are therefore lower than in the paper's tables, which tuned SAE configs on the references, but our selectors' significant gains over lasso appear in the same settings.

Surface similarity / F1, mean over 10 seeds. † better than lasso, p<.05.

SettinglassoSTMSTM + covariatestratified-maxDR-lassointeraction-lasso
Imbalance, r=.50.330/.296.273/.257.537†/.454†.573†/.481†.610†/.558†.130/.085
Imbalance, r=.35.227/.150.163/.152.590†/.497†.677†/.643†.663†/.601†.147/.137
Imbalance, r=.20.253/.204.233/.212.543†/.424†.767†/.550†.730†/.636†.253/.151
Sign reversal.266/.254.354/.347.440†/.465†.382/.369†.260/.251.428†/.407†

Under stratum imbalance, stratified-max and DR-lasso score higher than STM + covariate at every ratio, significantly so when the targeted strata are rarest (r=.20, p≤.016 on surface similarity). Under sign reversal, interaction-lasso is comparable to STM + covariate. As expected, conditioning also matters beyond SAE features: adding the covariate significantly improves STM in every imbalance setting (p≤.008). We will add the STM baseline and these results to the revised paper.


W4: "[...] the two synthetic-data metrics could use a clearer explanation. [...] the Hungarian algorithm pairs generated and reference hypotheses by how closely their labels correlate. [...] Because the matching step already maximizes label agreement, it is unclear how much F1 adds. The matching also forces every generated hypothesis into a pair [...]. The LLM judge gives half credit to loosely related pairs [...] alignment with human judgments. [...] the metrics may thus reward broad topic overlap [...]"

For synthetic evaluation, the Hungarian matching only decides which generated hypothesis is compared with each reference. Therefore, a poorly matched pair still gets a low F1. Surface similarity and F1 then check each pair from two sides: what the hypotheses say, and which held-out texts they apply to. The two metrics agree closely at the pair level (Spearman ρ ≈ .82 in every setting).

We agree that metrics may reward broad topic overlap. We therefore added a strict score analysis that gives credit only to pairs which the judge consistently rates as the same hypothesis.

Strict surface similarity: share of matched pairs rated "same". Mean over 10 seeds. † better than lasso, p<.05.

Settingrandomlassostratified-maxDR-lassointeraction-lasso
Imbalance, r=.50.000.100.267.267.000
Imbalance, r=.35.000.000.333†.400†.000
Imbalance, r=.20.000.033.633†.533†.067
Sign reversal.080.060.140.000.220†

Under the strict score, random falls to zero at every imbalance ratio and lasso close to zero, because their hypotheses describe broad or non-targeted topics. Stratified-max and DR-lasso keep a clear lead under imbalance, as does interaction-lasso under sign reversal, so their advantage holds even when topic overlap gets no credit.

To check alignment with human judgment, one author rated 50 matched pairs, sampled evenly across the judge's three ratings and blind to method. Agreement with the judge is high (quadratic-weighted κ = .84). Human and LLM also never disagree between "same" and "distinct". The human agreed with 14 of the 16 pairs the judge rated "same", so the strict score rests on judgments a human confirms. We will add the full comparison to the appendix.

Our evaluation follows the established protocol for hypothesis generation on synthetic data (Zhong et al., 2024; Movva et al., 2025), but we agree that the reviewer's questions deserve a closer look. The analyses above give more insight into what the metrics capture, and we will explain the metrics more clearly in the revised paper.

Reviewer_d81jOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces conditional hypothesis generation, using researcher-specified covariates to discover textual differences that global comparisons may overlook. It incorporates stratified regression, interaction models, and demeaning into an existing SAE-based hypothesis-generation pipeline. Synthetic experiments examine stratum imbalance and sign reversal, while expert evaluations assess usefulness on congressional speeches and classroom transcripts. The problem is well motivated, although the methodological novelty is modest and the practical evidence remains limited.

Critical strengths
  1. The paper addresses a meaningful limitation of global hypothesis discovery. The controlled experiments clearly illustrate why different statistical approaches are needed for imbalance and sign reversal.

  2. The expert evaluation connects the methods to substantive research questions. The paper also clearly acknowledges that conditional associations do not establish causal effects.

Critical weaknesses
  1. The real-world comparison rests on very few unique hypotheses: two per method in Congress and three in NCTE, with two experts per domain. The helpfulness improvement is therefore preliminary. Reporting results across the complete hypothesis lists, repeated generation runs, and additional covariates would strengthen the practical contribution.

  2. Synthetic SAE hyperparameters are selected using similarity to known ground-truth hypotheses (Appendix C.6), which is unavailable during actual discovery. Moreover, each method selects its own SAE configuration, making it harder to isolate the benefit of covariate-aware selection. Results using a shared SAE and a tuning criterion that does not require reference hypotheses would provide stronger evidence.

Questions for authors
  1. Do the synthetic gains persist with a shared SAE and reference-free hyperparameter selection?

  2. Does the helpfulness advantage persist across complete hypothesis lists and repeated runs?

3 comments

W1a: "The real-world comparison rests on very few unique hypotheses. [...] The helpfulness improvement is therefore preliminary. Reporting results across the complete hypothesis lists [...] would strengthen the practical contribution."

We would like to clarify the goal of the expert study: conditional selection aims to surface hypotheses that the global baseline misses and that are as useful, not to outperform it. The hypotheses unique to the covariate-aware method were rated at least as helpful as lasso's (3.10 vs 2.50). This comparison is intended to be descriptive rather than a significance test.

Experts rated the complete lists (Appendix D.1, Tables 6–7) and we will add those ratings to the appendix. We agree that a larger expert panel would strengthen this evidence. Domain experts are hard to recruit, and two experts per domain is in line with prior evaluations of hypothesis generation (Zhong et al., 2023; Movva et al., 2025). We will note the panel size as a limitation in the revised paper.


W1b: "Reporting results across [...] repeated generation runs, and additional covariates would strengthen the practical contribution."

Additional covariates. Whether conditional differences can be found depends on the data as well as the method. We examined additional Congress covariates and report the two with clear conditional structure: (1) whether a speech is procedural, and (2) whether it discusses cultural or social issues. For each method's 10 hypotheses, we check on held-out speeches whether the Republican–Democrat difference is concentrated in the covariate stratum (e.g., procedural speeches): the within-stratum difference is statistically significant and at least twice as large as the difference across all speeches.

Number of hypotheses (out of 10 per method) whose difference is concentrated in the covariate stratum. DR-lasso: demeaned-reweighted-lasso.

CovariatelassoDR-lassostratified-max
Procedural speech267
Cultural or social issues157

The covariate-aware selectors surface more such hypotheses than lasso, about half of which lasso does not produce at all. Some examples:

Republican − Democrat difference in the share of speeches with the attribute (percentage points).

HypothesisCovariateOverallIn stratumOutside
moves to suspend the rules and passProcedural speech+3.3+9.5+0.2
discusses illegal immigrationCultural or social issues+1.9+8.0+1.4
discusses poverty or the poorCultural or social issues−5.8−15.1−1.6

Within the stratum of speeches about cultural or social issues, Republicans emphasize illegal immigration and Democrats emphasize poverty, with gaps two to four times larger than overall. Outside this stratum, the parties barely differ on either.

Repeated runs. Each run outputs free-text hypotheses, so run-to-run variation is hard to summarize directly. We instead checked the feature selection step by running it separately on two random, non-overlapping halves of the Congress training speeches. It selects 9 of the same 10 features for lasso and all 10 for DR-lasso. This check holds the SAE fixed, but SAEs trained with different seeds can learn different features (Paulo & Belrose, 2026). We will note this as a limitation of the real-data results. The synthetic results already train a separate SAE for each seed.

Gonçalo Paulo and Nora Belrose. Sparse Autoencoders Trained on the Same Data Learn Different Features. ICLR 2026.


W2a: "Synthetic SAE hyperparameters are selected using similarity to known ground-truth hypotheses [...]"

We agree that a reference-free tuning criterion would provide stronger evidence. We reran all SAE-based methods without reference hypotheses: each method selects its SAE configuration by validation AUC computed within the covariate strata, since we look for differences after controlling for the covariate.

Results with reference-free tuning criterion. Surface similarity / F1, mean over 10 seeds. † better than lasso, p<.05.

Settinglassostratified-maxdemeaned-reweighted-lassointeraction-lasso
Imbalance, r=.50.330/.296.573†/.481†.610†/.558†.130/.085
Imbalance, r=.35.227/.150.677†/.643†.663†/.601†.147/.137
Imbalance, r=.20.253/.204.767†/.550†.730†/.636†.253/.151
Sign reversal.266/.254.382/.369†.260/.251.428†/.407†

The significant gains over lasso (†) appear in exactly the same settings as in the submission. The numbers are lower than in the original results, but under imbalance the gaps over lasso are similar or larger. We will add these results to the revised paper.

W2b: "Moreover, each method selects its own SAE configuration [...] Results using a shared SAE and a tuning criterion that does not require reference hypotheses would provide stronger evidence."

Selecting the SAE config per method is a deliberate choice: methods use SAE features differently, so the best config can differ by method. But we do agree a shared config better isolates the selection method, so we ran the following sensitivity check: for each seed, all methods use the same SAE config, also chosen without reference hypotheses.

The imbalance results are also unchanged. Under sign reversal, interaction-lasso remains the best SAE method, but its gain over lasso is no longer significant (p=.31/.07).

Shared configuration. Surface similarity / F1, mean over 10 seeds. † better than lasso, p<.05.

Settinglassostratified-maxdemeaned-reweighted-lassointeraction-lasso
Imbalance, r=.50.257/.231.540†/.435†.593†/.483†.080/.075
Imbalance, r=.35.230/.158.670†/.624†.617†/.564†.143/.126
Imbalance, r=.20.273/.219.720†/.580†.730†/.589†.180/.109
Sign reversal.346/.308.382/.380.324/.319.428/.407

We will add this sensitivity check to the revised paper.


Q1: "Do the synthetic gains persist with a shared SAE and reference-free hyperparameter selection?"

Yes for stratum imbalance, under both reference-free protocols (W2a, W2b). Under sign reversal, interaction-lasso remains the best SAE method, but its gain over lasso is significant only with per-method tuning (W2b).


Q2: "Does the helpfulness advantage persist across complete hypothesis lists and repeated runs?"

Our goal is comparable helpfulness rather than a helpfulness advantage: conditional selection surfaces hypotheses the global baseline misses that are as useful (W1a). We will make this goal clearer in the revised paper. Experts rated the complete lists, and we will add those ratings to the appendix. Across the complete lists, the two methods are rated about the same (2.90 for DR-lasso vs 2.95 for lasso), as expected since most hypotheses are shared. For repeated runs, we show that feature selection is stable across disjoint halves of the data (W1b).

I thank the authors for those additional analysis. I will increase my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their thoughtful feedback. We especially appreciate that the reviewers found the paper clearly written (oZTn) and the problem well motivated (d81j, oZTn, yTxY), and found the framing around stratum imbalance and sign reversal useful (d81j, oZTn, yTxY).

We summarize the main changes below and give details, including result tables, in the individual responses.

Revised in the paper (shown in blue in the revised PDF).

  • §3.1–3.2 define the task by within-stratum differences, with three conditional targets, and state what each selector targets (oZTn, yTxY).
  • We narrowed our claim to covariates in LLM-based hypothesis generation (oZTn, yTxY).

New experiments and analyses (to be added to the revised paper).

  • Hyperparameter selection without reference hypotheses, both per method and with a shared SAE config. The gains of covariate-aware selection under stratum imbalance hold in both (d81j).
  • A structural topic model baseline, with and without the covariate (oZTn, yTxY). Our selectors score higher under stratum imbalance, significantly so when the targeted strata are rarest, and are comparable under sign reversal.
  • Held-out tests of whether the generated hypotheses show the targeted within-stratum differences, on synthetic data and on Congress with additional covariates (d81j, oZTn). Covariate-aware selectors surface more such hypotheses than global selection.
  • A strict surface-similarity score and a human check of the LLM judge (oZTn).
  • Expert ratings across the complete hypothesis lists, with confidence intervals and permutation tests (d81j, oZTn).
  • A clarification of direction under sign reversal and guidance on choosing a selector (oZTn, yTxY).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.