- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ConflictCast, a benchmark for conflict-event forecasting that extends conventional event-count prediction to forecasting complete spatio-temporal event trajectories. It integrates four conflict-event datasets covering different reporting mechanisms and conflict settings.
The benchmark evaluates models under three updating regimes: fully conditioned, periodic update, and no update, using metrics for trajectory fidelity, event counts, probabilistic calibration, and next-event likelihood. It compares neural spatio-temporal point processes, Hawkes processes, tree-based regressors, and simple baselines, alongside two proposed conflict-tailored models.
The experiments show that Hawkes-based models perform well overall, but strong next-event prediction does not necessarily translate into reliable long-horizon forecasting. The results also reveal persistent calibration limitations across existing approaches.
Critical strengths
- Move Beyond Aggregate Event Counts Toward Fine-Grained Spatio-Temporal Conflict Forecasting. The paper addresses an important forecasting problem with applications in humanitarian response and early-warning systems. By moving beyond aggregated event counts to event-level spatio-temporal trajectories, it enables a richer assessment of conflict dynamics and forecasting capabilities.
- Provide unified evaluation across different updating regimes, forecasting models, and metrics. The benchmark provides a unified framework for comparing diverse forecasting approaches. The three updating regimes distinguish next-event prediction from longer-horizon simulation, while multiple evaluation metrics capture complementary aspects of forecasting quality. The standardized data processing and evaluation pipelines further enhance its practical utility.
- Reveals important limitations in long-horizon forecasting through extensive empirical evaluation. The paper evaluates a diverse set of models and includes additional analyses of forecast horizons, reporting delays, covariates, and model design choices. A particularly valuable finding is that strong next-event prediction does not necessarily lead to reliable long-horizon simulation, highlighting an important limitation of existing approaches.
Critical weaknesses
-
Insufficient comparison with existing forecasting benchmarks. Although the paper discusses related benchmarks such as ViEWS, WeatherBench, and ForecastBench, it lacks a systematic comparison establishing the novelty and necessity of ConflictCast. In particular, differences in forecasting targets, task granularity, updating regimes, and evaluation capabilities are not clearly summarized.
-
Unexamined ranking sensitivity and potential selection bias in test-Fold construction. (1)Sensitivity of the overall model ranking. The overall ranking equally weights five metric families, despite substantial differences in model performance across forecasting regimes. For instance, CAM-HP ranks first overall but performs poorly under the no-update regime. It remains unclear whether the overall rankings would remain consistent under alternative metric weights or greater emphasis on long-horizon forecasting. (2)Potential selection bias in test-fold construction. Forecast origins are selected using event-time quantiles, and only folds with at least 40 test events are retained. While this ensures sufficient observations for evaluation, it may systematically exclude low-activity periods and bias the evaluation toward more active conflict settings. The impact of these selection criteria on model rankings and benchmark representativeness is not examined.
-
Limited Dataset Diversity and Generalizability. Although the benchmark includes four datasets, they cover only two conflict settings, with three datasets focusing on Central Africa. Moreover, ACLED, UCDP-GED, and SAGE have overlapping geographic and temporal coverage and may record some of the same underlying conflict events. While their reporting mechanisms differ, the potential event overlap is not quantified or discussed, raising questions about the independence of the evaluation settings. These limitations may affect the generalizability of the findings, and further analysis of dataset overlap or evaluation across more diverse conflict settings would strengthen the benchmark.
-
Presentation Could Be Improved. The manuscript contains several presentation issues, including incorrect cross-references (e.g., Appendix L.1 for the Energy Score definition and Section E.6 in the kernel ablation discussion), an unresolved "??" placeholder in the Compute Statement, and occasional typographical and grammatical errors.
Questions for authors
- What distinguishes ConflictCast from existing forecasting benchmarks, particularly ViEWS, in terms of task formulation and evaluation capabilities?
- How sensitive are the overall model rankings to alternative metric weights or greater emphasis on long-horizon forecasting?
- How does the minimum requirement of 40 test events affect evaluation representativeness? Would alternative splitting strategies change the results?
- To what extent do ACLED, UCDP-GED, and SAGE contain overlapping events, and how might this affect the independence and generalizability of the evaluation?
- What does the evaluation-history fix mentioned in Table 8 involve, and does it affect any reported results or conclusions?
4 comments
Dear Reviewer,
Thank you again for your review! As the discussion period is coming to an end in a few days, we wanted to check whether our response addresses your concerns. Thank you.
After considering the authors’ rebuttal, I am inclined to revise my recommendation from Weak Reject to Weak Accept. The authors have addressed several of my main concerns, most importantly by expanding the evaluation to 38 countries and 149 folds and by removing the test-set event-count threshold, which alleviates the concerns about limited geographic generalizability and selection bias. The clarification of the distinction from ViEWS also makes the benchmark’s task-level contribution more convincing.
A few questions remain:
-
Sensitivity of overall rankings: The sensitivity of the overall ranking to alternative metric-family weights is still not directly examined.
-
Cross-dataset event overlap: The reported overlap is quantified only under a strict same-day/1-km matching criterion, which may not capture events recorded with temporal or spatial discrepancies.
-
Validation-fold stability: Some newly included validation folds contain very few events, which may affect the reliability of hyperparameter selection.
We thank the reviewer for the rebuttal and for the continued engagement.
We will take the additional points into account in the camera ready version of the paper or we will add them as limitations.
We thank the reviewer for their time. We concisely reply to the stated weaknesses:
Comparison to other benchmarks: Respectfully, we would like to note that WeatherBench and ForecastBench are benchmarks on other domains. Regarding ViEWs, it is limited to binned models on event counts, with grids that are not more granular than ~50km per side at the equator. Our benchmark evaluates the full predictive distribution over future event sequences (of which ‘expected counts’ are just a functional of the distribution), testing neural point processes previously never used. We expand on this already in our related work.
Unexamined ranking sensitivity: We don't think this is true. Ranks of table 1 summarize all metrics and regimes, hence we weight each column the same. However, we also provide other ranks: for example, figure 2 ranks models on the spatio-temporal energy score, showing simple baselines winning over longer horizons and ranking them first. Indeed, our result section highlights that different models have different strengths. On this point, we want to importantly remark that the objective of the paper is not to show the superiority of a class of models, but to provide 1) an understanding of which model class fails when, and 2) a benchmark (software, evaluation framework, website …) that can help bringing progress forward in neural point process models and their applicability.
Generalizability and evaluation folds / Selection Bias. We think this is the fairest criticism. Since the submission we did two things: i) Added additional 34 countries for ACLED and UCDP, with a total of 149 evaluation folds across countries, periods and data sources. We retrained on each fold-country-source. We note that the overall conclusions have stayed consistent. ii) We changed the fold selection criterion: we agree that filtering on test-set could be seen as a kind of 'leakage', hence we filtered on training instead, excluding training periods that had less than 50 events, to ensure that deep learning models would have enough data to train.
On overlapping events: the three sources are different because they come from different reporting strategies (media sources, newswire and hand collection), and these indeed give different data distributions. To get a sense of this: SAGE records 11.6 events per day over the region against 7.5 for ACLED and 1.7 for UCDP; only 4% of SAGE events have an ACLED event within 1 km on the same day and 1% a UCDP event.
This caption remained from a previous bug in the code. Although the bug had been fixed, the caption was inadvertently left unchanged. Thank you for catching this; we have now removed it.
We thank the reviewer again for their time and engagement with the work. If the reviewer was at least partially convinced by our responses, we would appreciate their reassessment of the work.