- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
RELY is a crime forecasting framework with probabilistic graph and hypergraph structures. It refines urban relation priors using recent observations. A reliability score summarizes structural support and uncertainty. Separate routers select relational information for the negative-binomial mean and excess-zero components of a ZINB distribution. Experiments cover Chicago and New York with eight baselines and five random seeds. The authors report MAE improvements of 6.9% and 4.5%. Additional analyses examine component contributions, sparse regimes, reliability–error relationships, and computational efficiency.
Critical strengths
-
The method has a clear task-specific motivation. Sparse crime counts require appropriate output distributions. Different urban relations may provide different predictive information. RELY addresses these properties through probabilistic relation refinement and separate routing for the two ZINB components. This combination offers a plausible approach to heterogeneous crime forecasting.
-
The results provide evidence of practical value. RELY reports the lowest mean aggregate errors on both datasets. The experiments include five random seeds and several component ablations. Removing the reliability bias increases Chicago MAE from 0.803 to 0.837. Sharing one router increases MAE to 0.830. These results support the usefulness of the proposed routing design under the evaluated setting.
Critical weaknesses
-
The evaluation does not establish generalization to future periods. Section 4.1 randomly splits forecast windows into training, validation, and test sets. This can place later observations in training than some test forecast origins. Overlapping windows may create additional dependence between the partitions. The paper does not report a chronological evaluation. This limits confidence in the claimed forecasting gains under deployment conditions. The exact overlap and any resulting information leakage require clarification.
-
The interpretation of structural reliability needs stronger validation. Section 4.5 shows lower errors for samples with higher reliability. The authors acknowledge a possible confound from crime count levels. Quieter regions may have lower absolute errors regardless of structural uncertainty. The current analysis does not establish whether reliability provides information beyond this effect. The ablations demonstrate a useful routing bias. They do not establish that the score reliably measures the quality of inferred relations. This distinction matters because structural reliability is a central contribution.
Questions for authors
-
Clarification: What is the window stride? Can observations from a test target date appear in a training window? Please specify the dataset dates and the data used to fit preprocessing statistics.
-
Essential validation: Does RELY retain its advantage under a strictly chronological train/validation/test split? A comparison against the strongest existing baselines would materially affect my assessment.
-
Essential validation: Does reliability still predict error within groups with similar historical count levels and occurrence frequencies? Does it provide additional information beyond simple count-based scores or reliability computed from the priors alone? These controls would help establish the meaning of the learned reliability scores.
3 comments
Response to Reviewer SXdz (2/2)
W2 & Q3: Reliability still predicts error within groups with similar occurrence frequencies, and beyond reliability computed from the priors alone it provides additional information that helps reduce forecast error. We agree that the available evidence is insufficient to directly validate the latent structural quality, and we therefore limit our conclusions to the score’s in-model diagnostic value for forecasting and its incremental routing utility, which we support with the frequency-stratified analysis and the comparison against prior reliability.
To further examine the effect of occurrence frequency, we have added an analysis stratified by crime occurrence frequency. The table below reports MAE, RMSE, and MAPE across reliability groups within each occurrence-frequency bin, where Q1 denotes the lowest reliability group and Q5 denotes the highest.
| Frequency | 0-0.25 | 0.25-0.5 | 0.5-0.75 | 0.75-1 | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Reliability | MAE | RMSE | MAPE | MAE | RMSE | MAPE | MAE | RMSE | MAPE | MAE | RMSE | MAPE |
| Q1(Lowest) | 0.175 | 0.553 | 0.080 | 0.534 | 0.945 | 0.248 | 0.873 | 1.264 | 0.373 | 1.553 | 2.119 | 0.438 |
| Q2 | 0.158 | 0.455 | 0.078 | 0.507 | 0.924 | 0.236 | 0.832 | 1.210 | 0.362 | 1.442 | 1.976 | 0.435 |
| Q3 | 0.118 | 0.413 | 0.057 | 0.489 | 0.903 | 0.228 | 0.835 | 1.186 | 0.356 | 1.364 | 1.839 | 0.433 |
| Q4 | 0.099 | 0.401 | 0.046 | 0.466 | 0.886 | 0.216 | 0.828 | 1.181 | 0.352 | 1.321 | 1.818 | 0.424 |
| Q5(Highest) | 0.084 | 0.328 | 0.042 | 0.458 | 0.879 | 0.213 | 0.755 | 1.082 | 0.336 | 1.205 | 1.649 | 0.412 |
All four occurrence-frequency bins show an overall trend of decreasing error as reliability increases. From Q1 to Q5, MAE decreases by approximately 51.9%, 14.2%, 13.5%, and 22.4%, respectively. This indicates that the reliability–error relationship persists among samples with similar occurrence frequencies, providing further support for the score’s diagnostic value for forecasting.
To address the question of additional information raised in Q3, we have added a comparison against reliability computed from the priors alone to examine the incremental predictive value of learned reliability. Both variants retain structure refinement and message propagation based on the learned structures, and both use the same reliability function. The full model supplies the routers with , derived from the learned structural distribution, whereas the control variant supplies , derived from the completed prior. This comparison differs from an ablation that directly removes structure refinement: it assesses the predictive utility of the two reliability sources while retaining structure learning.
The results within different occurrence-frequency groups are reported below. “Learned Reliability” denotes the full model, and “Prior Reliability” denotes the variant that uses prior reliability for routing.
| MAE | MAE | RMSE | RMSE | MAPE | MAPE | |
|---|---|---|---|---|---|---|
| frequency | Learned Reliability | Prior Reliability | Learned Reliability | Prior Reliability | Learned Reliability | Prior Reliability |
| [0,0.25) | 0.126±0.0002 | 0.126±0.0008 | 0.423±0.0003 | 0.424±0.0009 | 0.061±0.0001 | 0.061±0.0004 |
| [0.25,0.5) | 0.489±0.0068 | 0.501±0.0145 | 0.895±0.0061 | 0.899±0.0078 | 0.228±0.0035 | 0.234±0.0078 |
| [0.5,0.75) | 0.819±0.0148 | 0.835±0.0262 | 1.183±0.0262 | 1.212±0.0529 | 0.353±0.0067 | 0.360±0.0104 |
| [0.75,1] | 1.431±0.0259 | 1.571±0.1525 | 1.988±0.0458 | 2.275±0.3006 | 0.422±0.0048 | 0.432±0.0099 |
In the three intervals with occurrence frequencies of at least 0.25, the full model achieves lower MAE, RMSE, and MAPE than the prior reliability variant. In the highest-frequency group, the three metrics decrease by approximately 8.9%, 12.6%, and 2.3%, respectively. In the lowest-frequency group, the two variants have essentially the same mean errors. In addition, the full model has smaller standard deviations on all three metrics across all four frequency groups, indicating lower variability across random seeds.
Structural reliability is a model-internal confidence indicator derived from the support strength and decisiveness of the relation posteriors, used to assist the allocation of relational information and forecast diagnosis. The two analyses above support, respectively, the association between the score and forecast error and its incremental predictive utility over the prior score, but they do not directly test whether the latent relations are in fact correct. We therefore limit the corresponding conclusions to the score’s diagnostic and routing value within this model, and do not interpret it as direct validation of true structural quality or of probability calibration.
The author's experiments addressed my concerns to some extent, so I have decided to increase my score.
Response to Reviewer SXdz (1/2)
W1 & Q1 & Q2: The window stride is one day, and observations from test target dates do not appear in any training window. RELY retains its advantage under a strictly chronological train/validation/test split (Table 1). The table below provides the date ranges or versions of each data source and specifies the data used to estimate preprocessing statistics. Upon checking, we confirmed that an early implementation used random window splits, whereas all results ultimately reported in the paper were obtained using training, validation, and test splits ordered strictly by forecast target date. Thus, Table 1 shows that RELY’s gains hold when no training window occurs after a held-out forecast. The manuscript retained the description of the earlier implementation and therefore did not accurately reflect the final experimental setup. We thank the reviewer for identifying this oversight and will correct the description in the revised manuscript.