- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper introduces conditional hypothesis generation, where researchers specify covariates that define the contexts within which textual outcome-group differences should be discovered. This is to address the problem that globally discriminative patterns can be irrelevant or misleading when strata are imbalanced or when associations reverse direction across strata. It is built on an earlier approach of Movva et al. (2025), which first maps documents to SAE features and then selects discriminative features via LASSO. The paper modifies the feature selection stage with covariate-aware methods including stratified regression, interaction models, and demeaning. Selected SAE features are subsequently verbalized as natural-language hypotheses by an LLM. Evaluation results from synthetic experiments and small expert study verify the effectiveness of the proposed approach.
Critical strengths
-
The paper tackles an interesting problem in interpretable hypothesis discovery in text analysis. The globally discriminative features in data may not enable meaningful comparison a researcher wants to make. The paper addresses this by allowing researchers to define relevant groups using covariates, and introduces conditional hypothesis generation which generates discriminative hypotheses within researcher-specified covariate strata. It also distinguishes associations that hold across groups, appear only in particular groups, or reverse direction between groups.
-
The paper examines two statistical challenges in discovering conditional differences: unequal representation across groups and associations that reverse direction between groups. It extends a previous SAE feature selection approach (Movva et al., 2025) and proposes a few covariate-aware feature selection methods. The experimental results show that stratified-max and demeaned-reweighted LASSO perform best under group imbalance, while interaction LASSO performs best when associations reverse direction.
Critical weaknesses
-
The paper has limited methodological novelty. The proposed approach was built on (Movva et al., 2025). Both SAE training and LLM interpretation are the same as in the previous approach. The four proposed variants in the feature selection stage (stratified, interaction, demeaned, and reweighted LASSO) are standard techniques for confounding and effect modification. Also, the paper positions itself as the first covariate-aware hypothesis-generation method, but covariate-adjusted text discovery has been around for a long time. For example, structural topic models (Roberts et al., 2014) model topic prevalence and content as functions of covariates; and deconfounded lexicon induction (Pryzant et al., 2018) uses residualization which is similar to the proposed within-stratum demeaning. The paper should include at least one such method (e.g., structural topic models with LLM topic labeling) as a baseline.
-
The paper claims to address sign reversal, but it is unclear how it works. In the synthetic experimental setup, the target is the set of subtopics that characterize corpus A before 2011, even though these subtopics characterize the other corpus after 2011. Both Interaction-LASSO and stratified-max rank features by the strength of their coefficients or interaction effects, regardless of direction. In the feature verbalization phase, each selected feature is described using text that does not specify the period or which corpus it characterizes. It is therefore unclear whether the pipeline identifies the intended association before 2011 or simply detects that the association differs across periods.
Roberts et al., 2014. Structural topic models for open‐ended survey responses. American journal of political science, 58(4), pp.1064-1082.
Pryzant et al., 2018. Deconfounded lexicon induction for interpretable social science. NAACL.
Questions for authors
-
Could the authors clarify the main technical contribution in comparison to the approach in (Movva et al., 2025)? Since the feature selection methods are standard techniques (interaction terms, fixed-effects demeaning, inverse-frequency weighting), what is adapted specifically for conditional hypothesis generation?
-
Why not compare with existing covariate-aware text methods, such as structural topic models (Roberts et al., 2014) and deconfounded lexicon induction (Pryzant et al., 2018)?
-
In the sign-reversal experiment, how does the proposed approach distinguish "A over B before 2011" from the reverse association after 2011? Are the evaluation metrics sensitive to direction?
-
In real-world applications, the presence of sign reversal may not be known. In such a case, how should one choose among the feature selection methods mentioned in the paper?
3 comments
W2: "The paper claims to address sign reversal, but it is unclear how it works. [...] rank features by the strength of their coefficients or interaction effects, regardless of direction. [...] whether the pipeline identifies the intended association before 2011 or simply detects that the association differs across periods."
Q3: "In the sign-reversal experiment, how does the proposed approach distinguish "A over B before 2011" from the reverse association after 2011? Are the evaluation metrics sensitive to direction?"
We distinguish two directions in the reviewer's question: which corpus an attribute characterizes (the outcome direction) and in which period (the covariate direction). For the outcome direction, we follow the convention in LLM-based hypothesis generation. A hypothesis describes an attribute of the text, and the corpus it characterizes is given by the sign of its association with the outcome, for global and covariate-aware methods alike.
The covariate direction matters in the sign-reversal setting. Our synthetic settings give the methods the covariate but not its direction, as when a researcher knows that the period matters but not which period holds the difference of interest. Without the target period, the methods find both the target subtopics (), which characterize corpus A before 2011, and the mirror-image subtopics (), which characterize corpus A after 2011. Both are genuine differences that reverse across periods.
That said, our evaluation metrics are sensitive to the covariate direction. The reference hypotheses are only the target subtopics, so a hypothesis describing a mirror-image subtopic receives almost no credit. The metrics therefore reward recovering the intended pre-2011 association, not merely detecting that the association differs across periods. The reported scores are also conservative, since the mirror-image hypotheses are valid differences under this setup but count as misses. This accounts for part of the gap between the oracle and the covariate-aware methods in the sign-reversal setting.
An additional setting can give the covariate direction (the target period), for example asking what characterizes corpus A before 2011. The covariate-aware methods then need only a small change to selection: they keep only features that favor corpus A in the target period, then rank them as before. We will consider adding this setting in the revision.
Q4: "In real-world applications, the presence of sign reversal may not be known. In such a case, how should one choose among the feature selection methods mentioned in the paper?"
We expand on the intended-use paragraph in Section 4.4. We envision our approach as an exploratory tool, like other hypothesis generation methods: researchers inspect the generated hypotheses and refine them for further analysis. In practice, researchers can start with stratified-max and demeaned-reweighted-lasso. When there is a plausible reason to expect sign reversal, they can also run interaction-lasso. Researchers can inspect each candidate's within-stratum differences on validation data and test the refined hypotheses on held-out data. Since the held-out data are not used for generation, running more selectors widens the candidate set without compromising that test.
Thank the authors for the thorough response. My main concerns have been largely addressed. I will update my score accordingly.
Our main contribution is defining the problem of conditional hypothesis generation. Like other LLM-based methods, Movva et al. (2025) rank attributes by the marginal difference Dh between outcome groups. The revised §3.1 decomposes Dh into a within-context term and a composition term (Kitagawa, 1955). Global selection can therefore favor an attribute through composition alone. It can also miss within-context differences that get little weight in Dh because their stratum is underrepresented (stratum imbalance), or that cancel across strata (sign reversal). We instead define the task by the conditional differences Δh(c), with three targets that answer common research questions: the adjusted difference (what differs after accounting for C), the context-specific difference (what differs in a particular context), and heterogeneity (how differences vary across contexts).
The statistical tools are standard by design. Covariate adjustment and heterogeneity analysis are well established for testing attributes specified in advance, and we bring their targets to discovery, where the attributes are not yet known. Each approach scores SAE features by their within-stratum associations instead of Dh, and §3.2 shows how each relates to the targets:
Because the SAE features are fixed before selection, covariates can enter directly into the selection.
We will make our claim precise: it concerns LLM-based hypothesis generation. Our contribution brings the conditional targets of §3.1 into this setting. Covariate-adjusted text analysis has a long history, and we will discuss it more explicitly in related work, including STM and DLI. As the reviewer notes, DLI's residualization is close to our demeaning step.
We added STM as the covariate-aware baseline the reviewer suggested, in two versions: topic prevalence depends on the outcome only (STM), or on both the outcome and the covariate, including their interaction (STM + covariate). We select the topics whose prevalence differs most between outcome groups (overall for STM, within a stratum for STM + covariate). An LLM labels each selected topic by contrasting documents with a high proportion of that topic against documents from other topics, and the label serves as the hypothesis. To compare all methods on equal terms, each one selects its hyperparameters (the SAE config, or the number of topics for STM) by AUC on a validation set, without the reference hypotheses. The scores are therefore lower than in the paper's tables, which selected the SAE configs using the references, but our selectors' significant gains over lasso appear in the same settings.
Surface similarity / F1, mean over 10 seeds. † better than lasso, p<.05.
Under stratum imbalance, stratified-max and demeaned-reweighted-lasso score higher than STM + covariate at every ratio, significantly so when the targeted strata are rarest (r=.20, p≤.016 on surface similarity). Under sign reversal, interaction-lasso is comparable to STM + covariate. As expected, conditioning also matters beyond SAE features: STM + covariate significantly improves on STM in every imbalance setting (p≤.008). We will add the STM baseline and these results to the revised paper.