- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TUCH, a temporal unfolding framework for LLM-based forecasting. Instead of aggregating retrieved evidence in a temporally flat manner, TUCH divides historical evidence into consecutive time intervals, maintains evolving causal hypotheses, and uses them to guide subsequent retrieval. The model is trained on hindsight-constructed reference trajectories through supervised fine-tuning and process-outcome reinforcement learning. Experiments on PROPHET and ForecastBench demonstrate improvements over several same-backbone forecasting baselines.
Critical strengths
TUCH provides a structured approach to forecasting by explicitly tracking evolving causal hypotheses across temporal stages. The authors make reasonable efforts to prevent information leakage, including separating training and evaluation periods, constructing evaluation evidence without access to outcomes, and manually validating synthesized trajectories. The experimental results are promising, with TUCH-RL achieving Brier scores of 21.70 and 17.19 on PROPHET and ForecastBench, respectively. The component-wise ablations also suggest that hypothesis memory, hypothesis-guided retrieval, and process rewards contribute to the reported improvements.
Critical weaknesses
1. The novelty and effectiveness of the central temporal unfolding mechanism are not sufficiently established.
Prior studies have already explored maintaining and updating linguistic belief states during sequential reasoning [1, 2]. While TUCH introduces hindsight-supervised causal-hypothesis trajectories, its contribution relative to these existing approaches could be more clearly distinguished.
More importantly, the experiments do not directly isolate the benefit of temporal partitioning. All ablations in Table 2 retain the same monthly temporal scaffold (T = 12), examining only hypothesis memory, retrieval, and process rewards. Without a temporally flat variant using identical evidence, supervision, and training configurations, it remains unclear whether the improvements arise from temporal unfolding itself or from additional supervision and structured retrieval.
2. Several experimental claims are not sufficiently supported by the reported comparisons.
In Table 1, the untrained Gemini 2.0 Flash achieves a Brier score of 21.13 on PROPHET, outperforming TUCH-RL at 21.70. This result complicates the claim that the temporal scaffold alone is insufficient, yet the discussion focuses primarily on models that TUCH outperforms.
Furthermore, the claimed best overall performance on ForecastBench is evaluated on a self-constructed subset of 274 questions rather than through the official benchmark protocol. The comparison omits important reference points, including simple base-rate forecasts, crowd or expert predictions, and prediction-market consensus. Given the importance of human forecasting baselines in ForecastBench [3], these omissions limit the interpretation of the reported performance.
3. The comparison with existing methods does not adequately control computational and retrieval budgets.
TUCH retrieves up to 60 facts across each of 11 temporal perception steps, resulting in approximately 660 evidence slots per question. However, the corresponding retrieval budgets, search rounds, and training costs of the baselines are not clearly reported. Since Search-R1 already employs multi-round retrieval and reinforcement learning, differences in computational resources may contribute to the observed performance gap.
Additionally, a contemporaneous study [4] reports a Brier score of approximately 0.18 using retrieval-augmented 7–8B models on a different set of 500 resolved ForecastBench questions. Although these results are not directly comparable, they further motivate stronger and resource-matched baselines.
References
[1] ABBEL: Learning Natural-Language Belief States for Memory-Efficient Interaction.
[2] Agentic Forecasting Using Sequential Bayesian Updating of Linguistic Beliefs.
[3] ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities.
[4] The Corroboration Illusion: When More News Makes LLM Forecasts Less True.
Questions for authors
-
Can the authors provide a temporally flat TUCH variant with identical evidence, supervision, and training configurations, along with an untrained Qwen2.5-7B baseline under the same temporal scaffold? These controls would help isolate the contributions of temporal partitioning and process learning.
-
Can the authors report the retrieval budgets, search rounds, and training/inference costs for all baselines, and provide resource-matched comparisons with Search-R1?
-
How do the authors interpret Gemini 2.0 Flash outperforming TUCH-RL on PROPHET? Can they also include base-rate, human, and prediction-market reference forecasts on the same ForecastBench subset to better contextualize the claimed improvements?
3 comments
Rebuttal for Weakness 2 and Question 3
Thank you for raising these concerns and giving us the opportunity to clarify our results. We address three points: Gemini’s performance, the scope of our ForecastBench evaluation, and comparisons with reference forecasts.
Gemini 2.0 Flash on PROPHET.
Gemini 2.0 Flash achieves a lower Brier score than TUCH-RL on PROPHET (21.13 vs. 21.70). TUCH-RL nevertheless outperforms most evaluated models, including the compared open-source models. Separately, our same-backbone comparison shows that TUCH-SFT reduces the Brier score of Qwen2.5-7B-Instruct from 41.37 to 24.80 under the same temporal scaffold, supporting the benefit of TUCH-specific training for this backbone. We will revise the abstract and Section 5.1 to explicitly acknowledge the Gemini result and qualify the corresponding claims.
Scope of the ForecastBench evaluation.
Our evaluation uses a subset of 274 resolved binary questions and a separately constructed evidence corpus. The reported results therefore apply to this evaluation setting. We will label it “ForecastBench (274-question subset)” and clarify that the results are obtained under our evaluation setup, rather than the official ForecastBench protocol.
Reference forecasts contextualize the reported improvements.
We have computed the following reference scores on our 274-question subset (Brier ×100; lower is better):
| Method | ForecastBench (274-question subset) ↓ |
|---|---|
| Base rate | 21.39 |
| Human crowd | 10.88 |
| Prediction-market consensus | 7.47 |
| TUCH-SFT | 19.37 |
| TUCH-RL | 17.19 |
We will include this table in the appendix.
Rebuttal for Weakness 3 and Question 2
We appreciate the reviewer's careful attention to the fairness of the computational comparison. In response, we detail the baseline budgets and costs, provide a resource-matched chronological-order ablation, and contextualize the results from [4] to address these concerns.
Baseline configurations and computational costs are explicitly reported below.
We reproduce Search-R1 and OpenForecaster following their original methods, with the same Qwen2.5-7B-Instruct backbone and training questions as TUCH. Their retrieval settings follow the original papers: Search-R1 retrieves the top 3 passages per search call with a maximum action budget of 4; OpenForecaster uses a single retrieval of 5 chunks, each containing 512 tokens, during evaluation.
Training, where applicable, and inference are performed on 8 NVIDIA H200 GPUs.
| Method | Training epochs | Training time | H200 GPU-hours | Inference GPU·s/question |
|---|---|---|---|---|
| Search-R1 | 5 | 5 h 56 min | 47.47 | 0.22 |
| OpenForecaster | 1 | 45 min | 6.00 | 0.55 |
| Vanilla RAG | — | — | 0.00 | 0.46 |
| TUCH-SFT | 2 | 63 min | 8.40 | 2.30 |
Vanilla RAG requires no task-specific training. TUCH-SFT outperforms Search-R1 on both datasets while requiring substantially less model-training compute (8.40 vs. 47.47 GPU-hours). Its advantage therefore cannot be attributed to a larger model-training budget. TUCH-SFT incurs higher inference costs, reflecting a trade-off between forecasting performance and inference cost.
Chronological organization contributes under fixed evidence and resource budgets.
Our chronological-order control preserves the backbone, step-level evidence and reference hypotheses, SFT configuration, and teacher and inference budgets, while shuffling the perception steps after the first. This increases Brier scores from 24.80 to 29.34 on PROPHET and from 19.37 to 24.54 on ForecastBench.
The performance difference in this controlled comparison cannot be explained by different resource budgets. It provides evidence that how evidence is organized along real-world time contributes to forecasting performance, beyond the amount of evidence and computation alone. Together with the lower training cost relative to Search-R1, this supports the framework’s contribution while making its inference-cost trade-off explicit. We will include the resource settings and cost table in the appendix.
The result in [4] requires its own baseline context.
Table II in [4] reports a Brier score of 0.184 for calibrated RAG, compared with 0.127 for its constant base-rate reference. On our subset, TUCH-SFT and TUCH-RL achieve 0.1937 and 0.1719, respectively, compared with a base-rate reference of 0.2139. These differences illustrate why raw Brier scores across different subsets do not establish a ranking between methods.
Rebuttal for Weakness 1 and Question 1
Thank you for highlighting these highly relevant studies. We appreciate the opportunity to clarify TUCH’s distinctions from prior linguistic belief-state methods and present additional experimental evidence for its temporal-unfolding process.
Differences from prior linguistic belief-state methods.
ABBEL [1] updates beliefs across environment interactions, while BLF [2] updates a structured belief state through iterative tool use under a forecast-date cutoff for leakage control. In TUCH, perception steps correspond to consecutive real-world intervals (Section 3.2). The current causal hypotheses guide retrieval from the next temporal interval, and the evidence and hypotheses are retained in an ordered core memory. Thus, state evolution and evidence acquisition progress jointly along the event’s real-world timeline, rather than only across interaction or search rounds.
ABBEL trains belief updates to preserve task-relevant information while improving memory efficiency. TUCH instead constructs forecasting-specific reference trajectories from resolved questions, using candidate upstream mechanisms and temporally organized evidence to synthesize step-level causal hypotheses (Section 3.3). These trajectories supervise perception-step hypothesis generation during SFT, followed by process-outcome RL for the unfolding process (Section 3.4). During RL and inference, hypothesis generation and retrieval are fully self-conditioned.
The contribution is therefore the joint formulation and learning of causal-state evolution and evidence acquisition along real-world time, extending linguistic belief updating to an explicitly supervised temporal-unfolding process for forecasting.
Isolating the benefit of temporal partitioning.
We have conducted a chronological-order ablation of TUCH-SFT to examine the contribution of chronological organization under matched training and inference conditions.
Experimental setup.
We retrain TUCH-SFT with the chronological order of its perception steps shuffled, starting from the same Qwen2.5-7B-Instruct backbone and using the same SFT configuration and number of optimization steps. We reuse the same step-level evidence and reference causal hypotheses and retain the same teacher and inference budgets.
We keep the first perception step fixed, preserving its question-based retrieval and initial hypothesis generation. We randomly shuffle the remaining perception steps using a fixed permutation for each example, keeping the retrieved evidence and reference causal hypotheses within each step together. The final judgment step remains last, conditions on the resulting shuffled core memory, and retains the same reference judgment target. This preserves the overall information content, the number of perception steps, and the evidence–hypothesis pairing within each step.
Results.
Brier scores ×100, where lower is better:
Shuffling increases the Brier score by 4.54 points on PROPHET and 5.17 points on ForecastBench. Under these matched conditions, chronological processing outperforms the shuffled-order sequential variant on both datasets. This within-TUCH comparison provides direct evidence for the contribution of chronological organization beyond performing the same number of sequential reasoning steps.
Untrained Qwen2.5-7B-Instruct under the same temporal scaffold.
We provide the results of the untrained Qwen2.5-7B-Instruct evaluated under the exact same temporal scaffold.
The results are as follows (Brier ×100; lower is better):
This comparison supports the contribution of learning the temporal-unfolding process beyond applying the scaffold alone.
We will add results to Table 1 and Table 2 in the revised manuscript.