- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces FutureMap, a training-free framework for omnimodal future prediction based on temporal hypergraphs. The paper first formulates the Omnimodal Future Prediction (OmniFP) task and releases OmniFP-Bench, consisting of 1,530 questions, 712 storylines, 7,422 frozen resources, five modalities, and eight domains. FutureMap addresses three challenges: integrating evidence across modalities, estimating the temporal and outcome regularities between precursors and future events, and avoiding deadline-anchored timing bias. Its three-stage pipeline constructs an omnimodal temporal hypergraph, backtests typed hyper-rules with temporal and magnitude kernels, and finally performs a deadline-blind deduction using a backbone LLM to generate dated branches followed by a deterministic read-out. Experiments with three frozen backbones and twelve training-free baselines show that FutureMap achieves the best overall accuracy on all three backbones, with gains of 4.1–5.3 points over the strongest baselines.
Critical strengths
-
The paper proposes a coherent and technically well-motivated framework for omnimodal forecasting. The separation into three stages is conceptually clear: the temporal hypergraph addresses cross-modal evidence aggregation, the known-zone deduction explicitly estimates rule reliability and temporal/magnitude distributions, and the unknown-zone deduction separates the forecasting process from the final deadline read-out.
-
The paper proposes a novel algorithm with strong empirical results. The paper proposes a novel training-free forecasting framework, FutureMap, and demonstrates its effectiveness through comprehensive experiments across three backbones and twelve baselines, consistently achieving the best performance with a 4.1–5.3 percentage-point improvement over existing methods.
Critical weaknesses
- FutureMap relies on many manually specified thresholds, horizons, and weighting choices. The author doesn't provide sufficient evidence to valid their choice, and the reported ablations do not sufficiently establish their robustness.
For example, the hypergraph construction uses a 3-day temporal matching window and cosine threshold of 0.86; the rule mining uses a 42-day maximum lag and a 14-day backtesting horizon; the multimodal validity function introduces manually chosen parameters such as η=2 and β=0.25; and the final read-out uses fixed weights such as α=1/4 and ω=0.75. These choices are central to the behavior of the system, but the ablation study mainly removes components rather than testing sensitivity to these numerical hyperparameters.
- The evaluation does not fully isolate the contribution of native multimodal understanding from the contribution of structured preprocessing.
Some modalities are passed through fixed parsers when a backbone cannot directly consume them, while the hypergraph itself is built using Gemini 2.5. Thus, the final comparison involves not only different forecasting strategies but also a substantial upstream multimodal processing pipeline. Although the paper reports that all methods use the same frozen archive and that parsers are fixed, it would be useful to disentangle errors caused by modality understanding from errors caused by forecasting/reasoning.
Questions for authors
-
How much of FutureMap's performance gain comes from the benchmark's repeated precursor–outcome structure rather than from genuinely transferable forecasting ability?
-
Can the authors report calibration metrics for the final probabilities, such as Brier score or expected calibration error, rather than evaluating probabilities mainly through binary accuracy?
-
I find the paper writing very crowded and difficult to read. The author defined a lot of variables, formulas, new definitions. The illustration figures are also filled with content, making readers hard to get the main flow of the author's method.
3 comments
Response to Reviewer 2ZR5 (Part 2/3)
W2: "The evaluation does not fully isolate the contribution of native multimodal understanding from the contribution of structured preprocessing."
Thank you very much. We now separate the two in three steps (§5.8), and the gain comes from the forecasting stages.
- ① The upstream pipeline is shared: every row of Table 1 reads the same frozen archive, and Gemini 2.5 Flash-Lite builds every store once for all backbones (App. E, Table 8), so a weaker backbone does not get a weaker store. Table 1 also states per backbone which media are read natively and shades every cell served through a fixed parser, so on each backbone every row sees the same media through the same parsers.
- ② Understanding fixed, forecasting changed ("Stage 3 on Identical Evidence" and "Without Non-Text Media or Fusion"): plain RAG gets exactly FutureMap's excerpts and media, through the same parsers, with the same answer contract and backbone (App. F.8), and articles only removes FutureMap's non-text inputs altogether:
| Overall (1,530) | Gemini | Qwen | Nemo |
|---|---|---|---|
| best baseline (Table 1) | 27.2 | 28.3 | 25.5 |
| plain RAG on FutureMap's own evidence | 24.4 | 28.2 | 21.6 |
| articles only (no non-text report, no media) | 29.4 | 32.7 | 29.5 |
- On identical evidence, FutureMap leads plain RAG by +5.6/+4.6/+7.3 (every paired interval above +2.3), and with articles only it still leads the best baseline on every backbone. If the advantage came from a richer parse of the media, plain RAG on FutureMap's own evidence would inherit it, yet it falls below the best baseline. If it came from the non-text media, articles only would lose it, yet it keeps the lead.
- ③ Errors split by evidence coverage (§5.8, "Calibrated and Grounded"): items are grouped by how many of their gold precursors reach FutureMap's prompt ("some" means at least one but not all; the remaining 119 items have none):
| Gold precursors in FutureMap's prompt | items | No source (G/Q/N) | FutureMap (G/Q/N) |
|---|---|---|---|
| some | 296 | 17.9 / 20.6 / 17.2 | 26.7 / 28.4 / 26.0 |
| all | 1,115 | 23.4 / 25.0 / 22.5 | 35.0 / 37.8 / 33.1 |
- When all gold precursors of an item reach the prompt, missing gold evidence cannot explain an error, and here FutureMap gains 11 to 13 points over the same backbone without sources. Accuracy is in single digits only on the 119 items whose gold precursors are not retrieved (7.6/6.7/7.6), a coverage problem that stage 1 already reduces, with 84.9% gold-precursor coverage, 12.1 points above flat retrieval of equal length (Figure 6b).
Thank you for prompting these controls; they make the source of the gain explicit. Hence, with multimodal understanding held fixed, the contribution that remains is that of the forecasting stages.
Q1: "How much of FutureMap's performance gain comes from the benchmark's repeated precursor–outcome structure ...?"
Thank you very much. Three checks in the paper show that the gain holds where that structure cannot help.
- ① The mining uses only the archive: stage 2 reads only the known-zone archive, never the questions, answers or storyline labels, and backtests at pseudo-cutoffs inside (§4.2). Each rule is scored on forecasts made at weekly dates before the real cutoff and checked against the days that followed in the archive. The outcomes lie after the cutoff, so stage 2 cannot learn which outcomes the benchmark asks about.
- ② The rules hold out of sample in time: re-mined up to , the 838 rules forecast the 14 days the miner never saw with ECE 0.014 against 0.093 for the base rate, and the 9,192 realised lags have near-uniform PIT (KS 0.074 against 0.203). At a cutoff, again has the lowest ECE (0.017) and the kernels the lowest KS (0.109) (§5.5, Figure 7, Table 11). ECE is the gap between a rule's predicted and observed hit rate, and a near-uniform PIT (low KS) means the realised lags fall where the kernels put them.
- ③ The margin holds across storylines and memory: resampling whole storylines, all 39 paired CIs of Table 13 still exclude zero (lowest bound +1.9; §5.8, "Clustered and Audited"). The margin over the best baseline is positive in all 18 stratum cells (deadline, domain-chain and memory strata on each backbone), including the 1,249 items not answerable from memory (Table 13), and FutureMap built from a third of already beats the best baseline using the full archive (Figure 6c).
Thank you for this question. Therefore, the gain rests on rules mined without the questions and calibrated on unseen days, and it survives storyline resampling and items the backbone cannot recall.
Response to Reviewer 2ZR5 (Part 3/3)
Q2: "Can the authors report calibration metrics for the final probabilities, such as Brier score or expected calibration error?"
Thank you very much. Yes, following your suggestion, §5.8 ("Calibrated and Grounded") now reports Brier score, ECE and AUC on the 333 yes/no items, and FutureMap has the lowest Brier score of all rows on every backbone, by 0.05 to 0.15.
- The three metrics are defined for yes/no forecasts, so we compute them on the yes/no items. For Brier and AUC, the baselines' probabilities are the stated ones of mini-prophet and LEAP, and every other answer counts as a hard Yes or No forecast (App. F.8). The first row gives, per metric and backbone, the best value reached by any baseline:
| Yes/no items (G/Q/N) | Brier ↓ | ECE ↓ | AUC ↑ |
|---|---|---|---|
| best baseline value | .384 / .384 / .286 | .347 / .367 / .181 | at most .55 |
| FutureMap | .233 / .233 / .235 | .093 / .087 / .112 | .64 / .64 / .66 |
- FutureMap's ECE is below the best baseline value on every backbone, and it ranks Yes above No items better than any baseline (AUC .64 to .66 against at most .55).
- Its probabilities are also informative. On Gemini, the observed Yes rate rises across the five probability bins of width 0.2, at 20%, 46%, 55%, 63% and 76%, so higher stated probabilities go with more frequent Yes outcomes.
Thank you for requesting these metrics; they are now part of RQ8. As a result, the final probabilities come with Brier score, ECE and AUC, and FutureMap is the best calibrated row on every backbone.
Q3(a): "I find the paper writing very crowded and difficult to read ..."
Thank you very much. We reworked the writing so that the main flow is easier to follow; these edits run through the whole paper, so they are not colour-marked.
- §4 (pp. 4–5): there is one displayed equation fewer, since the backbone's branch output is now one sentence. Symbols are glossed in words where they appear (e.g., , , ), and each Proposition now follows its plain-word claim with its formal condition in one sentence.
- §5: every result paragraph opens with a bold one-line finding. All new analyses sit in one section, §5.8, with the setting of each run in App. F.8, so a reader who skims gets each result from its first line.
- Appendix: App. A gives the full prompts of stages 1 and 3 (stage 2 runs in code), App. B the proofs and App. C the algorithm with every constant.
- Main flow, as §4 opens and Figure 4 shows: stage 1 builds a hypergraph once from the archive, stage 2 mines and backtests rules in code, and stage 3 makes one deadline-blind call per item and reads out the answer in code.
Thank you for this feedback; it made the paper easier to read. Therefore, the writing is now lighter, and each step of the method can be followed in words before its formula.
Q3(b): "... The illustration figures are also filled with content."
Thank you very much. We redrew Figures 1–4 in one consistent style, and each panel of Figure 4 now names the challenge it answers and when it runs (C1 built once, C2 mined in code, C3 one call per item).
- The caption explains each stage in one clause, so the figure reads from left to right as the three stages.
- The result figures share one palette and print their values (e.g., Figure 5).
Thank you for pointing this out. As a result, the illustration figures are now organised by stage and share one consistent style.
Summary of Revisions (all in the revised PDF)
| Where | Change |
|---|---|
| §5.8 (new RQ8), Figures 10c, 11a | One 480-item slice for all rows; constant sweeps; lead outside the slice |
| §5.8, Figure 10b | Articles-only and no-fusion variants; plain RAG on identical evidence |
| §5.8 | Brier, ECE and AUC on 333 yes/no items; accuracy when all gold precursors reach the prompt |
| §4, Figure 4, §5 | Readability: fewer displays, glossed symbols, panel tags, one-line findings |
| App. F.8, Table 14 | Setting, data and metric of every new run |
At last, we sincerely appreciate your valuable feedback. We have carefully considered all your suggestions and substantially improved and updated the paper accordingly. If our responses and revisions address your concerns, we would be deeply grateful if you could kindly reconsider raising the score to 3 or above. Thank you very much!
Response to Reviewer 2ZR5 (Part 1/3)
Thank you very much for your time and thorough review, and for recognizing that FutureMap is a coherent and technically well-motivated framework whose three stages are conceptually clear, and a novel algorithm with strong empirical results. Your concerns are the many manually specified thresholds, horizons and weighting choices (W1(a)), the sensitivity to these numerical hyperparameters (W1(b)), isolating native multimodal understanding from preprocessing (W2), how much of the gain comes from the benchmark's repeated precursor–outcome structure (Q1), calibration metrics for the final probabilities (Q2), the crowded writing (Q3(a)) and the illustration figures filled with content (Q3(b)).
We have substantially improved the paper's writing, figure and table layout, and supplementary experiments, among other details, and we kindly invite you to open the revised PDF to view them.
In the revised PDF, our revisions for you are in blue, together with those in red, which marks changes that address points raised by several reviewers, i.e., the overlapping concerns. They are in the new §5.8 (RQ8, p. 9) with Figures 10 and 11, and in Appendix F.8 with Table 14 (pp. 29–30).
The new runs (App. F.8) use the released code and data. They score FutureMap at 30.0/32.8/29.0 on Gemini/Qwen/Nemo (Table 1: 31.2/33.6/29.7) and grade 89 to 94% of the items alike. Rows marked with a paper table are the paper's own runs, and intervals are paired 95% CIs between runs of the same code, resampling the 712 storylines.
W1(a): "FutureMap relies on many manually specified thresholds, horizons, and weighting choices ..."
Thank you very much. All constants form one frozen setting: App. C lists every constant, and §5.8 ("Constants Fixed on One Slice") now states how the setting was fixed.
Thank you for raising this point. Therefore, the manually specified thresholds, horizons and weights are one frozen setting, and the lead holds on items that had no part in fixing it.
W1(b): "... testing sensitivity to these numerical hyperparameters."
Thank you very much. We added sensitivity sweeps for each constant you name (settings in App. F.8, results in Figure 10c), and every setting stays above the strongest baseline on every backbone.
Thank you for this suggestion; it added a sensitivity analysis to the paper. Hence, the sensitivity to each numerical hyperparameter is now tested, and the margin does not hinge on finely tuned constants.