Conditional Hypothesis Generation for LLM-Based Text Analysis with Researcher-Specified Covariates

An important goal of text analysis is to discover interpretable hypotheses about how language varies with outcomes of interest. Recent LLM-based methods select globally discriminative patterns that do not necessarily answer a researcher’s question. We introduce $\text{conditional hypothesis generation}$, the task of generating hypotheses that distinguish outcome groups within researcher-specified covariate strata. Covariates encode domain knowledge about which comparisons matter and can guide hypothesis discovery. We study two statistical challenges that arise in conditional discovery: relevant strata may be underrepresented within outcome groups ($\text{stratum imbalance}$), and differences may reverse direction across strata ($\text{sign reversal}$). Using sparse autoencoder features, we examine stratified regression, interaction models, and demeaning for covariate-aware selection. Controlled synthetic experiments show that these approaches can improve recovery of targeted differences overlooked by global selection, with effectiveness depending on the structure of conditional associations. On two real-world datasets, experts rate the covariate-aware method’s unique hypotheses as more helpful on average than the global baseline’s. We also illustrate how conditional comparisons can help researchers interpret generated hypotheses and formulate new ones.