- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper aims at a simulator and forecasting benchmark. It provides clear support for simulation coherence (RQ1) and model comparisons under controlled conditions (RQ4). For behavioral fidelity (RQ2) and benchmark novelty (RQ3), it shows relevant supply-chain patterns and different forecasting challenges, but further evidence is needed to establish how well these reflect actual operations and what they add to existing benchmarks. The scenario analysis (RQ5) illustrates how forecasts vary with system settings, though this alone does not establish reliable uncertainty estimates. Its applicability to real-world operational decisions remains uncertain. The reported results do not show operator-specific calibration or whether using these forecasts reduces stockouts, costs or service failures. This limits the practical claims, but the simulator can still be a useful research testbed.
Critical strengths
- Mechanistically specified simulator with internal consistency checks. The released setup models a 13-node network with 3 sources, 9 warehouses, and 16 edges. Its state tracks inventory, outstanding orders, shipments, and demand estimates. The paper proves three pathwise conservation laws (Proposition C.1; §§2–3.2, App. C).
- Benchmark includes network-state targets and varied operating scenarios. The release includes 7,300-day trajectories, C=50 and C=200 datasets, and a 33-rollout scenario library. Forecast targets include demand, backlog, fill rate, and edge utilization; scenarios vary demand, inventory, capacity, lead times, and disruptions (§4; App. G).
- Broad, target-specific forecasting evaluation. The paper compares three zero-shot foundation models with ARIMA, ETS, and PatchTST across four targets and multiple forecast horizons. For example, TimesFM performs best among the foundation models on demand, while Moirai performs best on backlog (Table 2; Table 6).
Critical weaknesses
- Limited validation against real operations. The SupplyGraph comparison covers one company, 41 SKUs, and 221 days. Only 15 of 39 reported statistics fall within the simulator’s scenario range; the paper also notes differences in demand intermittency and bullwhip ratios (App. K).
- Narrow operational assumptions and release scope. The released datasets use one 13-node network. Runs hold the control policy fixed; the base configuration assumes unbounded source supply and deterministic edge transit. The paper identifies alternative replenishment policies and other extensions as future work (App. B; App. F).
- Uncertainty results do not establish decision value. The 20-scenario ensemble is a designed set, not a probability distribution over system conditions. Backlog and fill-rate intervals are often poorly calibrated, and the paper does not test whether forecasts improve inventory, service, or cost outcomes (App. E, Tables 7–8; App. F).
Questions for authors
Clarification requests
- Beyond the output-statistic comparison with SupplyGraph, did any operator data inform the network structure or parameter choices, such as capacities, lead times, or replenishment rules? This would affect how strongly the released simulator can be interpreted as representative of real operations.
- Were the disruption responses compared with observed operational disruption records? The paper’s comparison uses a synthetic dataset and different service measures; evidence from real events would affect the assessment of disruption realism.
- Could the authors clarify whether the reported results were generated with the supplementary scripts as provided? In particular, the edge-utilization code appears to use shipment counts instead of item volumes and to assign capacity use to different days than the simulator; the MASE confidence intervals and TimesFM CRPS calculations also appear inconsistent with the stated metrics. Corrections or clarification of their effect on reported results would affect reproducibility and confidence in those results.
Suggestions for future work
- Validate the simulator against multi-echelon operational and disruption data, including service and inventory outcomes.
- Evaluate whether forecasts or scenario analysis improve decisions—such as inventory, service levels, or costs—and compare them with current operational methods.
- For probabilistic UQ claims, define and justify scenario probabilities; otherwise, present the results explicitly as coverage across a designed scenario set.
Thank you for your response. All of my remaining concerns are addressed. I will retain my positive score.