Predicting Real-World Quantities with LLMs through Learned Aggregation Of Scenarios

Large language models (LLMs) are increasingly used to estimate expectations of real-world quantities, yet their estimates are often inaccurate. Prior methods address this through case analysis, asking the LLM to estimate various more specific quantities (conditioned on additional facts) and aggregating the results. However, these more specific estimates may still be inaccurate, and it is not always clear how to aggregate them appropriately. We thus introduce LAoS, Learned Aggregation of Scenarios, which *learns* to predict the target quantity from a collection of specific LLM estimates. LAoS is an attention-based feedforward network that considers the various textual LLM queries (via embeddings) and the LLM's responses to them. On held-out conditional probability queries, LAoS outperforms prevalence-weighted averaging and repeated direct elicitation, also demonstrating generalizability across domains. Controlled stress tests with overlapping and unevenly distributed scenarios further demonstrate that LAoS learns to correct for these issues, as well as correcting for LLM estimation errors.