- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TimeStone aims to produce a history-dependent collection of forecast scenarios, each associated with a group of statistical mechanisms, a probability, and a conditional future distribution. It separates uncertainty over mechanisms from future randomness and from the compression of predictions into a manageable number of scenarios. For each domain, the method specifies a simulator and prior, trains an amortized mechanism-posterior estimator on simulated histories, infers the current state, and simulates futures. It clusters sampled mechanisms using parameter/rule descriptors and selects the number of groups using distortion and complexity criteria. Optional real-data adaptation reweights mechanism probabilities using an energy-score objective and KL regularization. Experiments cover COVID-19, Amazon event sequences, and FRED-MD, using separate domain-specific models. The reported results include favorable predictive scores, scenario-distance comparisons, synthetic posterior checks, and extensive ablations. Here, a mechanism is a rule within a specified model family; it is not automatically an identified real-world causal mechanism. The nominal zero-shot setting also includes target-domain prior calibration.
Critical strengths
- A coherent and useful representation of multiple plausible futures. TimeStone provides more structure than an undifferentiated sample distribution by linking future trajectories to groups of compatible statistical mechanisms. History-dependent grouping and scenario probabilities form a clear methodological pipeline. The evaluation across three different data types suggests that this formulation has value beyond a single illustrative application, even though each domain requires its own simulator and model.
- Substantial empirical analysis with unusually useful negative results. The paper includes synthetic posterior coverage, wrong-history and prior-only controls, single-mechanism comparisons, out-of-family perturbations, clustered bootstrap intervals, and prior sensitivity. These are meaningful checks of the proposed interpretation. It also reports that Amazon adaptation selects step zero and that some multi-mechanism gains are not significant. Such evidence makes the paper more informative and credible than a uniformly positive presentation would be.
Critical weaknesses
- The contribution of neural mechanism inference is not separated from that of the engineered simulator and calibrated prior. The current controls establish that conditioning on the correct history helps, but not whether the proposed amortized inference improves accuracy, cost, or scalability over a strong inference method using the same generative family. This is particularly consequential for Amazon, where a same-prior analytic posterior is available and is already used in posterior diagnostics, but is not carried through the main real-data forecasting and scenario comparisons (Appendices B.2 and G.1.1). Several other baselines require substantial task adaptations or use degenerate probabilistic interfaces, further limiting what their rankings establish. A matched comparison holding the prior, simulator, state estimation, grouping, and sample budget fixed would directly identify the methodological value added by TimeStone.
- The scenario evaluation does not establish the validity of the mechanism-based interpretation. The main scenario metrics use eight nearest-history reference futures, whose relevance depends on manually chosen features, distances, and cross-entity or cross-time matching rules (Section 4.1; Appendices C.3 and D.1). They are empirical analogues, not samples known to follow the query's conditional future distribution. Equal weighting of displayed groups and evaluation of at most three scenarios can also obscure scenario probabilities and omitted mass. Meanwhile, prior changes produce mechanism-region probability shifts much larger than Monte Carlo variation, even when predictive losses change little. Thus, favorable energy scores and scenario distances do not establish stable real-world mechanism probabilities. The finding that mechanism clustering improves distance but worsens coverage error on COVID-19 and Amazon reinforces this limitation. The paper needs to distinguish model-conditional explanations from validated real-world scenarios more sharply.
Questions for authors
- Assessment-changing matched baseline: What forecasting and scenario results are obtained with the analytic Amazon posterior under exactly the same prior, simulator, grouping, and sample budget? For COVID-19 or FRED-MD, can a tractable same-family numerical inference or parameter-fitting baseline establish the accuracy–latency trade-off of amortization?
- Assessment-changing evaluation check: How sensitive are scenario rankings to the number of reference histories, distance features, temporal separation, and same-entity versus cross-entity rules? Please report the selected number of scenarios and the probability mass represented by the displayed top three.
- Clarification of mechanism interpretation: Which mechanism conclusions remain stable across the tested priors? Can the claims explicitly distinguish simulator-posterior calibration, Monte Carlo group-mass accuracy, and calibration of real-world scenario probabilities?
- Clarification of the reported comparisons: Since Amazon adaptation selects step zero, can the two branches be compared with identical clustering settings and random samples to isolate any adaptation effect? Please also resolve the reference-set rule stated as excluding the same split-local sequence in Appendix C.3 versus using a different split in Appendix D.1.
2 comments
Thank you again for your detailed feedback. We would greatly appreciate one final look at our latest response, which adds direct evaluation against observed real futures, complementing our earlier matched inference comparisons and reference-sensitivity checks. We hope you will reconsider your assessment in light of these results.
Thank you for the detailed review and targeted suggestions.
W1 / Q1. Value of learned mechanism inference
TimeStone explicitly learns qϕ(M∣D) and connects inferred mechanisms to alternative futures. We isolate inference by fixing the prior, simulator, state handling, grouping, and sampling budgets within each family.
Amazon retains analytic within-component updates; only mixture weights change. Its common-partition scenario-mass TV is 0.122. COVID-19 paired 95% CIs favor TimeStone for ES and both scenario metrics. Numerical references are approximate; timings exclude training and shared simulation/grouping.
With TRAIN-only residual uncertainty, ForecastPFN/TimePFN attain COVID-19 ES 1.2766/1.9211, versus 0.5771 for adapted TimeStone. Paired ES and Event-NLL CIs favor TimeStone against both. These are probabilistic extensions of the official point-forecasting interfaces.
W2 / Q2. Reference sensitivity and displayed mass
Reference-based rankings agree with oracle rankings at Spearman 0.90 for both geometric metrics on synthetic Amazon histories. We additionally vary reference counts, features, distances, and feasible temporal/entity rules.
Zero-shot TimeStone beats Moirai on both metrics in all 10 COVID-19 and seven Amazon settings, with paired 95% CIs excluding zero. Amazon scenario distance also beats Chronos in 7/7 settings (six significant). An extra two-horizon temporal gap strengthens both COVID-19 advantages over Chronos. Coverage rankings can reverse against Chronos with COVID shape-only or Amazon category-only retrieval.
The COVID same-state check permits 145 queries with four references, none with eight; coverage favors Chronos on this subset. Amazon calendar/customer matching cannot be tested from available metadata; FRED-MD is a single panel.
Displayed probabilities are not renormalized. Equal-weight scores assess geometry; observed-event Brier scores improve over all four evaluated Amazon baselines, with paired CIs excluding zero. In an additional comparison starting from common medoids, DEV-selected representative refinement reduces the mechanism–trajectory coverage gap by 48–63% on COVID-19/Amazon, preserving groups, probabilities, and full predictions.
Q3. Stable conclusions and probability validation
Selected mechanism properties show consistent average posterior-minus-prior support increases across tested settings:
All 95% CIs exclude zero across the tested observation-link, prior-concentration, or family changes. Numerical probabilities remain conditional on each model.
We distinguish simulator uncertainty (88.8–91.2% marginal coverage for nominal 90% intervals), Monte Carlo accuracy (mean group-mass TV below 1 pp at 4,096 samples in all six settings), and observed-event probability quality (direct Brier evaluation). The latter evaluates observable outcomes, not real-world mechanism calibration.
Q4. Step zero and reference rules
Amazon step zero preserves the base posterior. Identical samples and grouping settings give exactly identical metrics, group counts, and probabilities across 256 queries per reporting configuration. Earlier differences arose from sampling and grouping settings.
Retrieval uses TRAIN-scaled histories from DEV and eligible TEST, excluding the query and identical histories. All audited COVID-19/FRED-MD pairs satisfy the cutoff. We will align C.3/D.1: Amazon excludes the same sequence, not the same split.
Together, these checks isolate the accuracy–latency value of learned inference and identify which evaluation conclusions persist, while making mechanism probabilities and reference rules explicit.