- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes Learned Aggregation of Scenarios (LAoS), a framework for improving LLM-based probability estimation by learning to aggregate predictions from multiple generated scenarios. Specifically, LAoS employs a self-attention-based neural ensembler that integrates scenario descriptions, estimated prevalences, and conditional probability estimates to produce more accurate predictions. Experiments on GSS and Com2Sense demonstrate improvements over several baselines, along with enhanced prediction coherence and robustness to overlapping or biased scenarios.
Critical strengths
- The paper addresses an important limitation of LLM-based probability estimation, where individual predictions can be inaccurate and inconsistent.
- LAoS consistently outperforms the evaluated baselines on both GSS and Com2Sense across multiple inference budgets and evaluation metrics.
- The experiments with overlapping and biased scenarios show that LAoS can effectively mitigate errors introduced by imperfect scenario generation.
- The paper investigates inference-time scaling, prediction coherence across multiple LLMs, and the contributions of different model components through ablation studies.
Critical weaknesses
- The proposed framework primarily applies a standard self-attention architecture to aggregate LLM-generated scenario estimates, with auxiliary reconstruction losses and heteroskedastic prediction.
- It remains unclear whether the proposed approach generalizes to continuous quantities, more complex probability distributions, or real-world forecasting tasks.
- Although the paper claims cross-domain generalizability, LAoS trained on Com2Sense performs substantially worse on GSS than its in-domain counterpart and even underperforms some simple baselines.
- Additional comparisons with simpler supervised aggregation methods under matched training conditions would help establish whether the gains primarily arise from the proposed architecture.
- Although LAoS explicitly models predictive uncertainty through heteroskedastic regression, the evaluation primarily focuses on point-estimation accuracy.
- The ablation experiments in Appendix J are based on single training runs and exhibit considerable sensitivity to checkpoint selection, while some reported numerical results appear inconsistent across the appendix.
Questions for authors
please see the weaknesses.
2 comments
(continued from previous message)
Besides these newly added experiments, we would like to point out that we already reported multiple ablation studies to validate the specific design of the architecture (Appendix G, lines 1082–1091; §§G.1–G.6, lines 1108–1187). The components that we ablated are:
(i) learned aggregation (we use the ensembler to process each scenario disjointly, then average the outputs using raw estimated prevalence. This ablated version does not have the capacity to learn a custom ensembling of scenarios)
(ii) learned correction of raw estimates (we let the ensembler output weights to average LLM raw target estimates. This ablated version does not have the capacity to correct errors in LLM raw estimates)
(iii) no longer supply raw target estimates as input
(iv) no longer supply raw prevalence estimates as input
(v) no longer supply scenario text embeddings as input
(vi) heteroskedastic prediction but with a fixed variance, so the model no longer learns an uncertainty quantification
(vii) supervise mean/variance jointly. We show that this results in uncalibrated uncertainty ( overconfidence)
We duplicate the ablation table below.
| 16 LLM calls | 32 LLM calls | 64 LLM calls | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | KL | MSE | MAE | RMSE | KL | MSE | MAE | RMSE | KL | MSE | MAE | RMSE |
| LAoS | 0.076 | 0.031 | 0.135 | 0.175 | 0.072 | 0.029 | 0.134 | 0.171 | 0.072 | 0.029 | 0.136 | 0.171 |
| 1-layer LAoS | 0.119 | 0.048 | 0.171 | 0.219 | 0.117 | 0.047 | 0.171 | 0.217 | 0.118 | 0.048 | 0.171 | 0.219 |
| w/o learned aggregation | 0.089 | 0.036 | 0.146 | 0.189 | 0.100 | 0.040 | 0.151 | 0.199 | 0.116 | 0.045 | 0.161 | 0.212 |
| w/o learned correction | 0.092 | 0.036 | 0.144 | 0.190 | 0.078 | 0.031 | 0.135 | 0.176 | 0.077 | 0.030 | 0.134 | 0.174 |
| target estimates r̂ᵢ = 1 | 0.254 | 0.088 | 0.232 | 0.296 | 0.154 | 0.061 | 0.190 | 0.246 | 0.121 | 0.050 | 0.175 | 0.223 |
| prevalences πᵢ = 1 | 0.080 | 0.031 | 0.137 | 0.177 | 0.079 | 0.031 | 0.137 | 0.177 | 0.079 | 0.031 | 0.136 | 0.177 |
| random embeddings | 0.086 | 0.034 | 0.143 | 0.184 | 0.084 | 0.033 | 0.140 | 0.181 | 0.079 | 0.031 | 0.136 | 0.176 |
| fixed variance (σ² = 1) | 0.071 | 0.028 | 0.135 | 0.167 | 0.081 | 0.032 | 0.145 | 0.180 | 0.091 | 0.037 | 0.155 | 0.191 |
| update mean and variance jointly | 0.077 | 0.031 | 0.141 | 0.176 | 0.080 | 0.032 | 0.143 | 0.179 | 0.086 | 0.034 | 0.148 | 0.183 |
Although LAoS explicitly models predictive uncertainty through heteroskedastic regression, the evaluation primarily focuses on point-estimation accuracy.
Competing methods do not have any uncertainty quantification; they only report point-estimation accuracy. To compare our method against them we need to evaluate through point estimation accuracy.
We believe that heteroskedastic modeling is justified even if the only goal is to improve point predictions: it reduces the influence of outliers in regions where the model is uncertain. As shown in Figure 10 (§G.6, lines 1183–1187; figure caption, lines 1287–1290), heteroskedastic modeling does improve point estimation. Our points in this case are probabilities, so we measure the predictive error by KL divergence.
We report that the variance produced by our separated method is more calibrated (achieves better log-loss) than the variance produced by either heteroskedastic or homoskedastic prediction with joint mean/variance supervision. (§G.6, lines 1183–1187; figure 10, lines 1275–1290)
The ablation experiments in Appendix J are based on single training runs and exhibit considerable sensitivity to checkpoint selection, while some reported numerical results appear inconsistent across the appendix.
Thank you for identifying these inconsistencies. We have made a submission error in the ablation section; we have corrected the error in the updated PDF. We also repeated the ablation experiments across five random seeds and reported the average result to reduce sensitivity to random effects, similar to our main experiments. Please see Appendix G (lines 1082–1091) for updated ablation description and Table 5 and §§G.1–G.7 (lines 1093–1187 and 1223–1233) for ablation results.
We consider the use of established components a strength. The ensembling of potentially biased or overlapping scenarios for case-analysis is a new task. Our results show that it can be addressed with readily available architectures.
LAoS learns to combine LLM-generated scenarios with unknown proposal distribution through a standard transformer architecture. The stress-test results indicate that it can handle imperfect LLM-proposed scenarios robustly.
It was not at all obvious in advance that this would work. If we had used some baroque new architecture, you would have suspected us of overfitting the architecture to the data domain.
On the other hand, we consider our method for training heteroskedastic prediction novel, which separates direct supervision of the mean and variance (§2.4, lines 262–269 and 286–292). We report in §G.6 that it yields significantly higher held-out log-likelihood and a modest improvement in point accuracy (lines 1170–1187). As a future work we may develop this method further in a separate paper.
We already predict continuous real-world quantities in the GSS dataset: the targets are empirical probabilities (§2.3.1, lines 190–193). However, we acknowledge that our experiments are limited to estimating probabilities (expectations of 0-1 boolean quantities), while our task formulation allows conditional expectations of real-valued quantities, such as expected prices / heights.
We consider forecasting a related but different evaluation task from the conditional expectation estimation task studied by this work. We would welcome testing LAoS in that setting, but improving LLM estimation of probabilities or real-world expected values is useful independently of its application to forecasting.
We acknowledge that our original transfer claim was too strong. We have updated the argument in §3.3 (lines 372–377 and 402–405), which now describes it as asymmetric transfer between domains. At 64 calls, GSS-to-Com2Sense transfer gives KL of 0.588, better than repeated prompting and prevalence weighting (0.716 and 0.742). In the reverse direction, KL is 0.151, worse than those baselines (0.117 and 0.106; Table 1, lines 378–398). We interpret this asymmetry as reflecting the broader coverage of GSS: learning to aggregate diverse survey queries supports transfer to commonsense judgments, whereas Com2Sense’s narrower coverage provides less support for transfer in the reverse direction (§3.3, lines 403–405). Please also see our top-level response.
We appreciate the suggestion for a clearer comparison against a simpler supervised predictor under matched training conditions.
Thus, we added a MLP predictor baseline, which only sees the root embedding and raw predictions during training and testing (§3.2, lines 348–350; Appendix C, lines 929–937). It is trained under the same condition as LAoS, with the only difference being that it is not supplied with any scenario information to aggregate, and therefore uses a feedforward network instead of the transformer architecture in LAoS that encodes each scenario as a token. Please also see our top-level response. We report that at 64 calls, LAoS achieves KL of 0.072 on GSS and 0.536 on Com2Sense, compared with 0.124 and 0.575 for the corresponding in-domain MLP predictors (Table 1, lines 378–398).
We also added a 1-layer LAoS as a simpler model for comparison. At 64 calls, its KL-divergence on the GSS dataset is 0.118, compared with 0.072 for the five-layer LAoS model (Table 5, lines 1093–1096). It performs worse on all four metrics at every reported budget despite using the same training setup, showing that multiple self-attention layers helps (§G.7, lines 1227–1233). As a future work we could also tune the number of layers we use.