- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper presents a study on LLM's ability to forecast inductively. They look at how LLMs infer from evidence and how it relates to outcomes. To do the following, the paper presents different experiments around LLMs belief update and look if models know in which direction and at what degree it should update. They conclude that LLMs are able to leverage information from relevant information, and revise their belief as evidence accumulates. The paper also show that we can use the predictive relationships as training signal for forecasting.
Critical strengths
- The question in itself is well motivated. I agree that it is important to understand this ability of LLMs, but is outdated.
- The controlled experiments are sound. The different ablated components in the experiments allow for isolated analysis and understanding of how each component interact and tie to the outcome.
Critical weaknesses
- The research question seems a bit outdated. I am not sure if it is due to a gap in related works, but a lot of the conclusions/findings in the paper are already quite well established in current literature, or seems slightly trivial.
- For example, assessing if model updates in the right direction is already established. In both https://arxiv.org/abs/2512.25070 and https://arxiv.org/abs/2604.15859, there is already evidence of how LLMs leverage new information to update their forecasts and how accumulating evidence shapes their probabilities. https://arxiv.org/abs/2605.15188 also looks at the dynamics of new source of information and its belief updates.
- Experiment 2 seems slightly weird. To capture the extent of which a model update its belief, the setup should have been to directly look at difference in brier score. From my understanding, the setup in the paper assumes a binary label without a probability which already is very flawed.
- Binary label questions are a very flawed and undesired format for forecasting. I would suggest looking at https://arxiv.org/abs/2506.00723
- Models tested are not specific in the main text. Also, most models are heavily outdated e.g. Qwen-2.5 and Llama-3.1.
- Related works is heavily missing recent references.
Questions for authors
I feel like the current work lies on heavily outdated assumptions. Improving performance of LLMs on different benchmarks already is proof that models are able to properly update their belief in the right direction and to a good extent. I am unsure what exactly are the contributions and what new insights comes out of the experiments.
Thank you for your review. To know you have found value and learned something from our work means a lot to us!
Re: your comment on experiment 4 (“An analysis of whether fine-tuning improves responses to new evidence while holding market history fixed would strengthen this”) – we thought this was a really good idea, and we had not considered this prior to your comment. So, we ran it!
We test whether fine-tuning improves evidence updating while holding market history fixed. To set up this additional experiment (done on our small models for now, but will be in full under our camera-ready), we compared 5 seeds of base Qwen3-4B, 8B, and 14B models against their trained counterparts (under our prior protocol). We evaluated updating in the same way we did for experiment 1, creating updates for the 318 questions set aside for E.4. Note that here, we can only analyze the direction of the update, and not assess the magnitude (as in experiments 2 and 3).
Again, similar to experiment 1, we measure the probability that we see:
Each comparison is relative to the same model’s forecast without the “new evidence” (generated by us). We hold the question and market history fixed. The table reports the change across seeds (the delta between before and after fine-tuning). We show both the mean across seeds and 95% confidence interval.
We find that while fine-tuning does improve the Brier score, it does not significantly improve directional updating for positive or opposing evidence. However, the models do improve at not changing their predictions under irrelevant evidence.
This finding better aligns experiment 4 with the results from experiment 3, where improvements from GRPO post-training did not transfer to new predictive structures. We hypothesize these two results may stem from the same issue: improved accuracy on existing structures does not beget better inductive reasoning.
We thank you for the constructive feedback, which we believe has helped improve our work. We will expand this to cover all of our models, include in-full in the appendix, and discuss it as part of experiment 4 and our limitations section. We have made changes to our main text in light of these changes and will finalize in the coming weeks (post-ICLR rebuttal).