acceptodds
Under review as a conference paper at ICLR 2027

Can language models forecast inductively?

Abstract

Judgmental forecasting involves predicting uncertain events like elections or geopolitical conflicts where abundant historical data is lacking. A key challenge is induction: forecasters must generalize from sparse evidence and contextual knowledge to infer how available evidence predicts future outcomes. Here, we investigate whether language models (LMs) recover and use such predictive relationships when forecasting, rather than relying on base rates or surface cues. Across a battery of experiments spanning real-world prediction markets and controlled synthetic environments, we test whether models: 1) update their beliefs coherently; 2) correctly recover predictive relationships, and 3) transfer what they learn to new settings. Our results provide evidence for each capability. Models selectively update their forecasts in response to causally relevant evidence, use contextual information to infer predictive relationships when observations are sparse, and revise these inferences as direct evidence accumulates. Training on families of predictive relationships improves forecasting across new domains, although evidence for transfer to unseen predictive structures is weaker. Together, our experiments provide behavioral evidence that LMs can forecast inductively.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Large language models, inductive reasoning, judgmental forecasting, relational reasoning, generalization and transfer, world models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_j4M3OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper presents a study on LLM's ability to forecast inductively. They look at how LLMs infer from evidence and how it relates to outcomes. To do the following, the paper presents different experiments around LLMs belief update and look if models know in which direction and at what degree it should update. They conclude that LLMs are able to leverage information from relevant information, and revise their belief as evidence accumulates. The paper also show that we can use the predictive relationships as training signal for forecasting.

Critical strengths
  • The question in itself is well motivated. I agree that it is important to understand this ability of LLMs, but is outdated.
  • The controlled experiments are sound. The different ablated components in the experiments allow for isolated analysis and understanding of how each component interact and tie to the outcome.
Critical weaknesses
  • The research question seems a bit outdated. I am not sure if it is due to a gap in related works, but a lot of the conclusions/findings in the paper are already quite well established in current literature, or seems slightly trivial.
  • For example, assessing if model updates in the right direction is already established. In both https://arxiv.org/abs/2512.25070 and https://arxiv.org/abs/2604.15859, there is already evidence of how LLMs leverage new information to update their forecasts and how accumulating evidence shapes their probabilities. https://arxiv.org/abs/2605.15188 also looks at the dynamics of new source of information and its belief updates.
  • Experiment 2 seems slightly weird. To capture the extent of which a model update its belief, the setup should have been to directly look at difference in brier score. From my understanding, the setup in the paper assumes a binary label without a probability which already is very flawed.
  • Binary label questions are a very flawed and undesired format for forecasting. I would suggest looking at https://arxiv.org/abs/2506.00723
  • Models tested are not specific in the main text. Also, most models are heavily outdated e.g. Qwen-2.5 and Llama-3.1.
  • Related works is heavily missing recent references.
Questions for authors

I feel like the current work lies on heavily outdated assumptions. Improving performance of LLMs on different benchmarks already is proof that models are able to properly update their belief in the right direction and to a good extent. I am unsure what exactly are the contributions and what new insights comes out of the experiments.

Reviewer_UJfVOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper examines the forecasting capabilities of LLMs using four dedicated experiments. Exp 1 examines the ability of the model to classify, for a given piece of evidence, the direction in which it should affect the forecast, and when the evidence shouldn’t affect the forecast even if is related; Exp 2 tests the ability of the model to infer how strongly a piece of evidence should affect the forecast; Exp 3 and Exp 4 examine the transferability of forecasting skills learned through RL fine-tuning using synthetic and real-world data, respectively. The paper provides empirical evidence showing inductive forecasting skills across different closed-source and open-source models under the experimental settings, along with useful ablation studies to examine alternative performance explanations.

Critical strengths

The paper shows evidence for 3 complementary research questions regarding forecasting capabilities, each of which is answered using a dedicated experiment, backed by its own reasonable ablation analyses, and demonstrated using a variety of models. Altogether, I find the evidence convincing and justified.

Critical weaknesses

None

Questions for authors
  1. The prompt provided in Appendix A asks the models to reason without using any formulas or calculations. Did the authors check whether the models really didn’t use math in their reasoning?
  2. The paper doesn’t provide examples of the actual prompts used in their experiments. Could the authors provide some examples to help readers fully understand the experimental setup?
  3. Figure 7 shows that the base model did better than the two trained versions under the structure shift condition. Can the authors provide a hypothesis for why this is the case?
Reviewer_WCUGOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies if language models can perform inductive forecasting (update their forecasts based on incoming evidence). Through series of experiments spanning synthetic forecasting tasks separating specific capabilities needed for inductive forecasting, real world prediction market forecasting and RL fine tuning of model, this work provides a convincing argument for use of LLMs for forecasting tasks.

Critical strengths

The core strength of this work comes from the design and breadth of experiments. The tasks are well-constructed and isolate specific capabilities needed for inductive forecasting, with thoughtful controls to test reliance on superficial cues and shortcuts. The study also brings together multiple sources of evidence, including mechanistic interventions and fine-tuning experiments examining the learning and transfer of forecasting skills.

Critical weaknesses

My main concern is the interpretation of the real-world experiment. Models receive prediction market price time series and approach the accuracy already available from the latest market price, making it difficult to determine whether the training gains reflect improved inductive forecasting beyond better use of this signal. An analysis of whether fine-tuning improves responses to new evidence while holding market history fixed would strengthen this. A smaller concern is the limited transfer across predictive structures, where the additional benefit over shuffled-target training is inconsistent. Regardless, assessing this work as a whole, the weaknesses/suggestions for improvements mentioned here are not overshadowing the strengths, which is reflected in my score.

Questions for authors

No questions, only points that were mentioned in weaknesses.

2 comments

Thank you for your review. To know you have found value and learned something from our work means a lot to us!

Re: your comment on experiment 4 (“An analysis of whether fine-tuning improves responses to new evidence while holding market history fixed would strengthen this”) – we thought this was a really good idea, and we had not considered this prior to your comment. So, we ran it!

We test whether fine-tuning improves evidence updating while holding market history fixed. To set up this additional experiment (done on our small models for now, but will be in full under our camera-ready), we compared 5 seeds of base Qwen3-4B, 8B, and 14B models against their trained counterparts (under our prior protocol). We evaluated updating in the same way we did for experiment 1, creating updates for the 318 questions set aside for E.4. Note that here, we can only analyze the direction of the update, and not assess the magnitude (as in experiments 2 and 3).

Again, similar to experiment 1, we measure the probability that we see:

  1. Supporting evidence increasing the predicted p(YES)
  2. Opposing evidence decreasing the predicted p(YES)
  3. Irrelevant information leaving the probability of p(YES) unchanged

Each comparison is relative to the same model’s forecast without the “new evidence” (generated by us). We hold the question and market history fixed. The table reports the change across seeds (the delta between before and after fine-tuning). We show both the mean across seeds and 95% confidence interval.

ModelSeedsSupporting Evidence success ΔOpposing Evidence success ΔUnchanged under irrelevant information Δ
Qwen3-4B5−2.72 [−5.32, −0.06]+3.61 [+0.49, +6.81]+7.97 [+4.70, +11.23]
Qwen3-8B5−0.95 [−2.53, +0.48]+0.82 [−1.90, +3.65]+6.14 [+3.39, +9.02]
Qwen3-14B5−0.63 [−1.96, +0.62]+0.25 [−1.72, +2.31]+2.22 [−0.26, +4.70]

We find that while fine-tuning does improve the Brier score, it does not significantly improve directional updating for positive or opposing evidence. However, the models do improve at not changing their predictions under irrelevant evidence.

This finding better aligns experiment 4 with the results from experiment 3, where improvements from GRPO post-training did not transfer to new predictive structures. We hypothesize these two results may stem from the same issue: improved accuracy on existing structures does not beget better inductive reasoning.

We thank you for the constructive feedback, which we believe has helped improve our work. We will expand this to cover all of our models, include in-full in the appendix, and discuss it as part of experiment 4 and our limitations section. We have made changes to our main text in light of these changes and will finalize in the coming weeks (post-ICLR rebuttal).

Thank you for running the experiment. I’m glad the suggestion was useful and that the results clarify how fine-tuning affects the models’ responses to new evidence.

Comments on the paper

1 comment

As a result of this rebuttal process, we have produced (and provided) an updated PDF version of the paper that includes the following changes discussed with our reviewers:

  1. We explicitly clarify for experiment 2 that models predict a poll value from 0 to 100 and that MAE measures error in poll percentage points.
  2. We evaluated whether post-training (resulting in improved brier-score) resulted in better evidence updating (in the style of experiment 1). Post-training improved responses to orthogonal / unrelated evidence, but directional updating showed mixed results.
  3. We added the requested related work, and we clarified the relationship of our work against pre-existing forecasting benchmark studies.
  4. We added prompt examples from all four experiments to a newly created Appendix G.

Thank you for taking the time to assist us in our work during the peer review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.