acceptodds
Under review as a conference paper at ICLR 2027

Predicting Real-World Quantities with LLMs through Learned Aggregation Of Scenarios

Abstract

Large language models (LLMs) are increasingly used to estimate expectations of real-world quantities, yet their estimates are often inaccurate. Prior methods address this through case analysis, asking the LLM to estimate various more specific quantities (conditioned on additional facts) and aggregating the results. However, these more specific estimates may still be inaccurate, and it is not always clear how to aggregate them appropriately. We thus introduce LAoS, Learned Aggregation of Scenarios, which *learns* to predict the target quantity from a collection of specific LLM estimates. LAoS is an attention-based feedforward network that considers the various textual LLM queries (via embeddings) and the LLM's responses to them. On held-out conditional probability queries, LAoS outperforms prevalence-weighted averaging and repeated direct elicitation, also demonstrating generalizability across domains. Controlled stress tests with overlapping and unevenly distributed scenarios further demonstrate that LAoS learns to correct for these issues, as well as correcting for LLM estimation errors.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
large language models, probability estimation, case analysis, inference-time scaling, calibration, learned aggregation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gJrROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper proposes Learned Aggregation of Scenarios (LAoS), a framework for improving LLM-based probability estimation by learning to aggregate predictions from multiple generated scenarios. Specifically, LAoS employs a self-attention-based neural ensembler that integrates scenario descriptions, estimated prevalences, and conditional probability estimates to produce more accurate predictions. Experiments on GSS and Com2Sense demonstrate improvements over several baselines, along with enhanced prediction coherence and robustness to overlapping or biased scenarios.

Critical strengths
  • The paper addresses an important limitation of LLM-based probability estimation, where individual predictions can be inaccurate and inconsistent.
  • LAoS consistently outperforms the evaluated baselines on both GSS and Com2Sense across multiple inference budgets and evaluation metrics.
  • The experiments with overlapping and biased scenarios show that LAoS can effectively mitigate errors introduced by imperfect scenario generation.
  • The paper investigates inference-time scaling, prediction coherence across multiple LLMs, and the contributions of different model components through ablation studies.
Critical weaknesses
  • The proposed framework primarily applies a standard self-attention architecture to aggregate LLM-generated scenario estimates, with auxiliary reconstruction losses and heteroskedastic prediction.
  • It remains unclear whether the proposed approach generalizes to continuous quantities, more complex probability distributions, or real-world forecasting tasks.
  • Although the paper claims cross-domain generalizability, LAoS trained on Com2Sense performs substantially worse on GSS than its in-domain counterpart and even underperforms some simple baselines.
  • Additional comparisons with simpler supervised aggregation methods under matched training conditions would help establish whether the gains primarily arise from the proposed architecture.
  • Although LAoS explicitly models predictive uncertainty through heteroskedastic regression, the evaluation primarily focuses on point-estimation accuracy.
  • The ablation experiments in Appendix J are based on single training runs and exhibit considerable sensitivity to checkpoint selection, while some reported numerical results appear inconsistent across the appendix.
Questions for authors

please see the weaknesses.

2 comments

The proposed framework primarily applies a standard self-attention architecture to aggregate LLM-generated scenario estimates, with auxiliary reconstruction losses and heteroskedastic prediction.

We consider the use of established components a strength. The ensembling of potentially biased or overlapping scenarios for case-analysis is a new task. Our results show that it can be addressed with readily available architectures.

LAoS learns to combine LLM-generated scenarios with unknown proposal distribution through a standard transformer architecture. The stress-test results indicate that it can handle imperfect LLM-proposed scenarios robustly.

It was not at all obvious in advance that this would work. If we had used some baroque new architecture, you would have suspected us of overfitting the architecture to the data domain.

On the other hand, we consider our method for training heteroskedastic prediction novel, which separates direct supervision of the mean and variance (§2.4, lines 262–269 and 286–292). We report in §G.6 that it yields significantly higher held-out log-likelihood and a modest improvement in point accuracy (lines 1170–1187). As a future work we may develop this method further in a separate paper.

It remains unclear whether the proposed approach generalizes to continuous quantities, more complex probability distributions, or real-world forecasting tasks.

We already predict continuous real-world quantities in the GSS dataset: the targets are empirical probabilities (§2.3.1, lines 190–193). However, we acknowledge that our experiments are limited to estimating probabilities (expectations of 0-1 boolean quantities), while our task formulation allows conditional expectations of real-valued quantities, such as expected prices / heights.

We consider forecasting a related but different evaluation task from the conditional expectation estimation task studied by this work. We would welcome testing LAoS in that setting, but improving LLM estimation of probabilities or real-world expected values is useful independently of its application to forecasting.

Although the paper claims cross-domain generalizability, LAoS trained on Com2Sense performs substantially worse on GSS than its in-domain counterpart and even underperforms some simple baselines.

We acknowledge that our original transfer claim was too strong. We have updated the argument in §3.3 (lines 372–377 and 402–405), which now describes it as asymmetric transfer between domains. At 64 calls, GSS-to-Com2Sense transfer gives KL of 0.588, better than repeated prompting and prevalence weighting (0.716 and 0.742). In the reverse direction, KL is 0.151, worse than those baselines (0.117 and 0.106; Table 1, lines 378–398). We interpret this asymmetry as reflecting the broader coverage of GSS: learning to aggregate diverse survey queries supports transfer to commonsense judgments, whereas Com2Sense’s narrower coverage provides less support for transfer in the reverse direction (§3.3, lines 403–405). Please also see our top-level response.

Additional comparisons with simpler supervised aggregation methods under matched training conditions would help establish whether the gains primarily arise from the proposed architecture.

We appreciate the suggestion for a clearer comparison against a simpler supervised predictor under matched training conditions.

Thus, we added a MLP predictor baseline, which only sees the root embedding and raw predictions during training and testing (§3.2, lines 348–350; Appendix C, lines 929–937). It is trained under the same condition as LAoS, with the only difference being that it is not supplied with any scenario information to aggregate, and therefore uses a feedforward network instead of the transformer architecture in LAoS that encodes each scenario as a token. Please also see our top-level response. We report that at 64 calls, LAoS achieves KL of 0.072 on GSS and 0.536 on Com2Sense, compared with 0.124 and 0.575 for the corresponding in-domain MLP predictors (Table 1, lines 378–398).

We also added a 1-layer LAoS as a simpler model for comparison. At 64 calls, its KL-divergence on the GSS dataset is 0.118, compared with 0.072 for the five-layer LAoS model (Table 5, lines 1093–1096). It performs worse on all four metrics at every reported budget despite using the same training setup, showing that multiple self-attention layers helps (§G.7, lines 1227–1233). As a future work we could also tune the number of layers we use.

(continued from previous message)

Besides these newly added experiments, we would like to point out that we already reported multiple ablation studies to validate the specific design of the architecture (Appendix G, lines 1082–1091; §§G.1–G.6, lines 1108–1187). The components that we ablated are:

(i) learned aggregation (we use the ensembler to process each scenario disjointly, then average the outputs using raw estimated prevalence. This ablated version does not have the capacity to learn a custom ensembling of scenarios)

(ii) learned correction of raw estimates (we let the ensembler output weights to average LLM raw target estimates. This ablated version does not have the capacity to correct errors in LLM raw estimates)

(iii) no longer supply raw target estimates as input

(iv) no longer supply raw prevalence estimates as input

(v) no longer supply scenario text embeddings as input

(vi) heteroskedastic prediction but with a fixed variance, so the model no longer learns an uncertainty quantification

(vii) supervise mean/variance jointly. We show that this results in uncalibrated uncertainty ( overconfidence)

We duplicate the ablation table below.

16 LLM calls32 LLM calls64 LLM calls
ModelKLMSEMAERMSEKLMSEMAERMSEKLMSEMAERMSE
LAoS0.0760.0310.1350.1750.0720.0290.1340.1710.0720.0290.1360.171
1-layer LAoS0.1190.0480.1710.2190.1170.0470.1710.2170.1180.0480.1710.219
w/o learned aggregation0.0890.0360.1460.1890.1000.0400.1510.1990.1160.0450.1610.212
w/o learned correction0.0920.0360.1440.1900.0780.0310.1350.1760.0770.0300.1340.174
target estimates r̂ᵢ = 10.2540.0880.2320.2960.1540.0610.1900.2460.1210.0500.1750.223
prevalences πᵢ = 10.0800.0310.1370.1770.0790.0310.1370.1770.0790.0310.1360.177
random embeddings0.0860.0340.1430.1840.0840.0330.1400.1810.0790.0310.1360.176
fixed variance (σ² = 1)0.0710.0280.1350.1670.0810.0320.1450.1800.0910.0370.1550.191
update mean and variance jointly0.0770.0310.1410.1760.0800.0320.1430.1790.0860.0340.1480.183

Although LAoS explicitly models predictive uncertainty through heteroskedastic regression, the evaluation primarily focuses on point-estimation accuracy.

Competing methods do not have any uncertainty quantification; they only report point-estimation accuracy. To compare our method against them we need to evaluate through point estimation accuracy.

We believe that heteroskedastic modeling is justified even if the only goal is to improve point predictions: it reduces the influence of outliers in regions where the model is uncertain. As shown in Figure 10 (§G.6, lines 1183–1187; figure caption, lines 1287–1290), heteroskedastic modeling does improve point estimation. Our points in this case are probabilities, so we measure the predictive error by KL divergence.

We report that the variance produced by our separated method is more calibrated (achieves better log-loss) than the variance produced by either heteroskedastic or homoskedastic prediction with joint mean/variance supervision. (§G.6, lines 1183–1187; figure 10, lines 1275–1290)

The ablation experiments in Appendix J are based on single training runs and exhibit considerable sensitivity to checkpoint selection, while some reported numerical results appear inconsistent across the appendix.

Thank you for identifying these inconsistencies. We have made a submission error in the ablation section; we have corrected the error in the updated PDF. We also repeated the ablation experiments across five random seeds and reported the average result to reduce sensitivity to random effects, similar to our main experiments. Please see Appendix G (lines 1082–1091) for updated ablation description and Table 5 and §§G.1–G.7 (lines 1093–1187 and 1223–1233) for ablation results.

Reviewer_3Ts8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Learned Aggregation of Scenarios (LAoS) for estimating real-world probabilities with LLMs. A learned attention-based aggregator combines scenario embeddings with LLM-estimated target probabilities and prevalences. Experiments on GSS and Com2Sense examine prediction accuracy, probabilistic coherence, robustness, and cross-domain transfer. Although the reported in-domain results are promising, contradictions concerning the training objective, ablation results, and evaluation populations prevent a reliable assessment of the central claims.

Critical strengths

S1. Relevant problem and plausible approach. Overlapping scenarios can cause prevalence-weighted averaging to double-count information. Learning interactions among scenarios is a reasonable approach to this limitation and addresses a meaningful challenge in LLM-based probability estimation.

S2. Promising results and relevant stress tests. Table 1 reports in-domain KL improvements over repeated prompting on Com2Sense and prevalence weighting on GSS. Evaluations of probabilistic coherence, scenario overlap, and biased generation also target relevant failure modes beyond standard prediction error.

Critical weaknesses

W1. Contradictory claims about the training objective. Section 2.4 claims that separate mean/variance supervision outperforms joint training and fixed variance, while Appendix J.6 explicitly concludes the opposite. J.6 also identifies joint training as the configuration used in Table 1, leaving it unclear whether the main results evaluate the proposed method.

W2. Conflicting ablation results undermine the conclusions. Table 5 shows that removing learned correction improves KL at 16 and 32 calls (0.0731/0.0697 versus 0.0757/0.0722), whereas Appendix J.2 reports different numbers and claims degradation at every budget. This discrepancy reverses the interpretation of a central component’s effectiveness.

W3. Inconsistent evaluation populations. Section 3.1 reports 169 GSS test queries, but Appendix J reports 371 while claiming to follow the Table 1 setup. Their relationship and separation from training data require clarification before the main experiments and ablations can be meaningfully compared.

W4. Benefits from supervision and scenario aggregation are insufficiently separated. LAoS receives task-specific labels, while several main baselines do not, and LoRA uses external training data. Although a single-correction ablation is provided, a clear comparison against a root-only predictor trained on the same labels is needed to establish the additional value of scenario aggregation.

W5. Scenario generation is documented inconsistently. Section 2.2 describes generating diverse scenarios in one prompt, whereas Appendix F.1 requests one scenario that makes the target event more likely. This mismatch obscures the actual generation procedure, its directional bias, and the call-budget accounting.

W6. Cross-domain claims exceed the evidence. At 64 calls, C2S-to-GSS transfer yields KL 0.1509, worse than repeated prompting (0.1165) and prevalence weighting (0.1062). These results indicate asymmetric transfer rather than broadly reliable cross-domain generalization.

Questions for authors

See weaknesses

2 comments

W4. Benefits from supervision and scenario aggregation are insufficiently separated. LAoS receives task-specific labels, while several main baselines do not, and LoRA uses external training data.

We compared against BIRD, which is a state-of-art method that receives the same task specific supervision as LAoS (Feng et al., 2025). We report that LAoS outperforms BIRD when trained with the same in-domain data. (§3.3, lines 354–359; Table 1, lines 378–398.) Regarding LoRA, LoRA was not trained on in-domain data because the size of the in-domain data is tiny (~80 target questions, paired with different conditions to form ~1000 queries). The original paper (Always tell me the odds, Wang et al., 2025) that proposed LoRA for probability estimation uses the same evaluation benchmark as us (C2S), and we use their suggested training data that yields their reported performance on C2S. (§3.2, lines 343–347; Appendix C, lines 922–927.)

W4 (continued). Although a single-correction ablation is provided, a clear comparison against a root-only predictor trained on the same labels is needed to establish the additional value of scenario aggregation.

We appreciate the suggestion for a clearer comparison against a simpler predictor with in-domain supervision. We added a MLP predictor baseline, which only sees the root embedding and raw predictions during training and testing (§3.2, lines 348–350; Appendix C, lines 929–937). We report that at 64 calls, LAoS achieves KL of 0.072 on GSS and 0.536 on Com2Sense, compared with 0.124 and 0.575 for the corresponding in-domain MLP predictors (Table 1, lines 378–398).

W5. Scenario generation is documented inconsistently. Section 2.2 describes generating diverse scenarios in one prompt, whereas Appendix F.1 requests one scenario that makes the target event more likely. This mismatch obscures the actual generation procedure, its directional bias, and the call-budget accounting.

Appendix F.1 incorrectly shows the single-scenario-generation prompt. Although querying for one scenario at a time is also a valid approach, in practice we use a single prompt to obtain all scenarios to save inference time cost, as described in main text (§2.2, lines 149–155). We have updated Appendix H.1 (lines 1304–1317 and 1418–1420) with the coupled scenario-generation prompt that we used for the reported experiments. We thank the reviewer for identifying this inconsistency.

W6. Cross-domain claims exceed the evidence. At 64 calls, C2S-to-GSS transfer yields KL 0.1509, worse than repeated prompting (0.1165) and prevalence weighting (0.1062). These results indicate asymmetric transfer rather than broadly reliable cross-domain generalization.

We acknowledge that our original transfer claim was too strong. We have updated the argument in §3.3 (lines 372–377 and 402–405), which now describes it as asymmetric transfer between domains. At 64 calls, GSS-to-Com2Sense transfer gives KL of 0.588, better than repeated prompting and prevalence weighting (0.716 and 0.742). In the reverse direction, KL is 0.151, worse than those baselines (0.117 and 0.106; Table 1, lines 378–398). We interpret this asymmetry as reflecting the broader coverage of GSS. Learning to aggregate diverse survey queries supports transfer to commonsense judgments, whereas Com2Sense’s narrower coverage provides less support for transfer in the reverse direction (§3.3, lines 403–405).

W1. Contradictory claims about the training objective. Section 2.4 claims that separate mean/variance supervision outperforms joint training and fixed variance, while Appendix J.6 explicitly concludes the opposite. J.6 also identifies joint training as the configuration used in Table 1, leaving it unclear whether the main results evaluate the proposed method.

Thank you for identifying these inconsistencies. We apologize for the confusion. Section 2.4 is correct. Appendix J.6 accidentally included figures from untuned runs and an incorrect description of the ablation setup. We have corrected the figures and the procedure description in §G.6 (lines 1170–1187), as summarized in our top-level response. In the corrected version, we clarified that:

  1. Separate mean/variance supervision achieves considerably higher held-out log-likelihood than joint supervision. We found that joint training produces overconfident variance estimates, which motivated us to develop separate supervision. The higher held-out log-likelihood under separate supervision indicates better-calibrated uncertainty on test queries (§G.6, lines 1177–1182). Separate supervision also modestly improves point accuracy over joint training and fixed variance (§G.6, lines 1183–1187).
  2. The main GSS experiments use separate mean/variance supervision. (§2.4, lines 262–269 and 286–292; Appendix G, lines 1082–1088 and 1170–1176).

W2. Conflicting ablation results undermine the conclusions. Table 5 shows that removing learned correction improves KL at 16 and 32 calls (0.0731/0.0697 versus 0.0757/0.0722), whereas Appendix J.2 reports different numbers and claims degradation at every budget. This discrepancy reverses the interpretation of a central component’s effectiveness.

This arose from the same appendix upload error described in the previous bullet point and in our top-level response; the corrected results are in Table 5 (lines 1093–1104), which we also show below. Learned correction improves KL at all three budgets, especially at 16 LLM calls; its benefit decreases as more scenarios become available (§G.2, lines 1124–1127). We apologize again for the confusion caused by the submission error.

16 LLM calls32 LLM calls64 LLM calls
ModelKLMSEMAERMSEKLMSEMAERMSEKLMSEMAERMSE
LAoS0.0760.0310.1350.1750.0720.0290.1340.1710.0720.0290.1360.171
1-layer LAoS0.1190.0480.1710.2190.1170.0470.1710.2170.1180.0480.1710.219
w/o learned aggregation0.0890.0360.1460.1890.1000.0400.1510.1990.1160.0450.1610.212
w/o learned correction0.0920.0360.1440.1900.0780.0310.1350.1760.0770.0300.1340.174
target estimates r̂ᵢ = 10.2540.0880.2320.2960.1540.0610.1900.2460.1210.0500.1750.223
prevalences πᵢ = 10.0800.0310.1370.1770.0790.0310.1370.1770.0790.0310.1360.177
random embeddings0.0860.0340.1430.1840.0840.0330.1400.1810.0790.0310.1360.176
fixed variance (σ² = 1)0.0710.0280.1350.1670.0810.0320.1450.1800.0910.0370.1550.191
update mean and variance jointly0.0770.0310.1410.1760.0800.0320.1430.1790.0860.0340.1480.183

W3. Inconsistent evaluation populations. Section 3.1 reports 169 GSS test queries, but Appendix J reports 371 while claiming to follow the Table 1 setup. Their relationship and separation from training data require clarification before the main experiments and ablations can be meaningfully compared.

The GSS test set contains 371 queries. We have corrected the test-set count in §3.1 (lines 300–301), consistent with Appendix G (line 1088).

Reviewer_SZF1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to improve LLM estimates of conditional probabilities through scenario analysis. Given a root query, an LLM generates more specific scenarios and estimates both the prevalence of each scenario and the target probability conditioned on it. The proposed method, LAoS, trains a self-attention-based ensembler that combines these numerical estimates with textual embeddings of the scenarios. It includes the direct root estimate and uses a likelihood-based training objective with auxiliary reconstruction losses.

Critical strengths

The paper addresses a genuine difficulty in scenario-based estimation: LLM-generated scenarios can overlap, be unrepresentative, and carry correlated estimation errors. Learning to combine scenario-level estimates and their descriptions is a reasonable and potentially useful approach when exact probabilistic aggregation is impractical.

Critical weaknesses
  1. The figures contain obvious layout problems that are immediately noticeable without careful inspection. In Figure 1, the red vertical line indicating the reference label directly overlaps the green "Point prediction" text, partially obscuring it. In Figure 2(a), several connector lines around the "Clipped logits" block appear duplicated or overlapping. These are conspicuous graphical errors that should have been apparent even from a quick glance at the figures. Their presence raises concerns about the overall care taken in preparing and proofreading the manuscript.

  2. These are inconsistencies in the ablation results. Table 5 and Appendix J.1 report substantially different KL divergences for the same methods on GSS. At 16, 32, and 64 LLM calls, Table 5 reports 0.0757 / 0.0722 / 0.0723 for LAoS, whereas J.1 reports 0.0910 / 0.0921 / 0.0975. Similarly, the results for w/o Learned Aggregation in Table 5 are 0.0994 / 0.1129 / 0.1309, compared with 0.1335 / 0.1481 / 0.1663 for the corresponding Single-Correction LAoS in J.1. I may have overlooked some differences in the experimental settings or misunderstood part of the evaluation protocol, and I would appreciate it if the authors could clarify the reasons for these discrepancies.

Questions for authors

Could the authors clarify the discrepancies between Table 5 and Appendix J.1?

1 comment

The figures contain obvious layout problems that are immediately noticeable without careful inspection. In Figure 1, the red vertical line indicating the reference label directly overlaps the green "Point prediction" text, partially obscuring it. In Figure 2(a), several connector lines around the "Clipped logits" block appear duplicated or overlapping. These are conspicuous graphical errors that should have been apparent even from a quick glance at the figures. Their presence raises concerns about the overall care taken in preparing and proofreading the manuscript.

Thank you for the feedback on the figures. We have updated the figures based on the feedback; please see the updated PDF (Figure 1, lines 67–72; Figure 2, lines 125–129). We also apologize for any confusions caused by writing, and have refined the writing throughout the paper. Please also see our top-level response.

These are inconsistencies in the ablation results. Table 5 and Appendix J.1 report substantially different KL divergences for the same methods on GSS. At 16, 32, and 64 LLM calls, Table 5 reports 0.0757 / 0.0722 / 0.0723 for LAoS, whereas J.1 reports 0.0910 / 0.0921 / 0.0975. Similarly, the results for w/o Learned Aggregation in Table 5 are 0.0994 / 0.1129 / 0.1309, compared with 0.1335 / 0.1481 / 0.1663 for the corresponding Single-Correction LAoS in J.1. I may have overlooked some differences in the experimental settings or misunderstood part of the evaluation protocol, and I would appreciate it if the authors could clarify the reasons for these discrepancies.

Thank you for identifying this discrepancy. It arose from a submission error; please see our top-level response. Table 5 (lines 1093–1104) and §G.1 (lines 1108–1118) now report corrected results. At 16/32/64 calls, KL is 0.076/0.072/0.072 for LAoS and 0.089/0.100/0.116 for the ablated method without learned aggregation. These results isolate the benefit of learning how to combine scenarios, particularly at larger budgets (§G.1, lines 1114–1118).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers

We thank the reviewers for their feedback and for carefully identifying detailed issues in the manuscript. We devoted over a year to the research and remain confident in its soundness, but left only a week for writing; we apologize for the imperfect presentation. We have continued revising the manuscript (see the updated PDF) and will keep improving it for readers. The major revisions we made since the reviewed submission are:

  • Corrected upload errors. Appendix J.6 (ablations) accidentally used old figures that had not tuned the hyperparameters. We had the tuned results before submission but forgot to put them into the appendix. Please see corrected results in the updated PDF, which provide a fair comparison of the ablated methods and are consistent with the claims in the main text. We also repeat the ablation experiments across multiple random seeds, as one of the reviewers suggested. (Table 5, lines 1093–1104; Appendix G, lines 1082–1091.)
  • Added additional baseline (MLP predictor). One of the reviewers suggested comparing against a simpler predictor that uses the same kind of in-domain supervision but does not aggregate over scenarios. We experimented with a simpler MLP predictor that only sees the root embedding and raw prediction, and learns to correct root raw predictions through the same in-domain training. We report that it is considerably worse than LAoS, thus isolating the effect of having in-domain supervision from the benefit of scenario ensembling. (§3.2, lines 348–350; Appendix C, lines 929–937; Table 1, lines 378–398.)
  • Adjusted domain transfer claim. We acknowledge that our original transfer claim was phrased too broadly. We have revised §3.3 to describe asymmetric transfer, with gains from the more diverse domain (GSS) to the one with narrower, less diverse questions (Com2Sense), but not the reverse. (lines 372–377 and 402–405.)
  • Other writing errors and presentation issues. We have corrected other typos, diagram glitches, and submission errors, and refined wording throughout the paper. As a major presentation change we rewrote the methodology section to make our approach clearer and easier to follow. We hope the revised manuscript reads better. (We would also expect to spend many further hours polishing the camera-ready paper - our usual practice before publishing anything with our names on it.)

Our takeaway point: We find the positive result surprising. A small network trained on a small set of textual questions can learn to discern useful relationships among scenarios, based only on their embeddings under a pretrained embedder, and use them to build an ensembling function that generalizes to unseen textual questions and potentially to other domains. It wasn’t clear this would actually work. As we explain in Appendix A, the underlying inference problem is hard to formalize, and hard to treat using statistical methods, since it relies on background knowledge about the world. So we tried this method of pretrained LLM embeddings plus learned aggregation, and were surprised to find that it behaved reasonably, e.g., with appropriate qualitative behavior in the "stress tests" of Appendix J. We believe this result is worth sharing with the research community. We again apologize that the presentation left aspects of our work unclear, and welcome further feedback on the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.