- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LUSim, a multi-agent simulator. It shows that structural decision rules can be swapped in under the same environment, and validates this through specific experiments.
Critical strengths
Coupling an LLM decision rule with calibrated spatial-equilibrium parameters and market clearing is a quite novel idea.
Critical weaknesses
Weakness 1
One of my major concerns is that the historical validation does not establish positive predictive ability. Based on the reported results, the direction accuracy is indistinguishable from chance on all four variables, and the experiments also lack a permutation test on the average rank.
Weakness 2
Another major concern is that V1 is negatively correlated on all four variables. Does this really show that structural models fail here, or could it instead reflect the counterfactual setup?
Questions for authors
Please state how many East Berlin zones have usable 1986 values. How do the authors explain V1's below-chance direction accuracy?
2 comments
Thank you for the detailed response. However, my cocerns still remain after the rebuttal, and I therefoe maintain my rating.
Predictive significance, counterfactual construction, and the structural baseline
We thank the reviewer for these questions.
W1. Repeated historical runs show predictive ability above chance. We repeat V4 and V5 ten times with fresh LLM responses under the original settings (V5, K=25 archetypes, 20-iteration cap). We evaluate 1986–2006 log changes in 55 West Berlin zones, centering each predicted and observed series by subtracting its mean across these zones.
The table reports mean relative RMSE, which divides root mean squared error by the standard deviation of observed changes (so zero change scores 1.00). Direction accuracy is the fraction of zones with matching predicted and observed signs after centering. We obtain 95% confidence intervals (CI) by resampling paired runs and neighboring zone groups.
For all four outcomes, V5's error is below the zero-change benchmark and its direction-accuracy interval lies above 0.50, directly addressing whether its predictions outperform chance.
W1. The requested permutation test supports V5's average ranking. Following the original scoring rule, we rank V1–V4 and V5 with and without district names by Pearson correlation of predicted and observed changes, Spearman correlation of their zone rankings, direction accuracy, and relative RMSE, using the available metrics for each outcome.
We shuffle observed changes in neighboring zone groups, applying the same shuffle across outcomes and repetitions, and recompute V5's mean rank across metrics and repetitions. Lower ranks are better.
Fewer than 1% of permutations yield ranks as favorable as V5's, supporting its average ranking beyond chance with and without names.
W2. No Eastern zone has usable 1986 outcomes, and the additional counterfactual includes both halves of Berlin. Usable 1986 values exist for all four outcomes in 55 Western zones and none of the 41 Eastern zones, so historical accuracy is evaluated only in the West.
The additional scenario uses 1986 Western and 2006 Eastern economic inputs in both network states, following Section 6.3 of Ahlfeldt et al. (2015). Eastern values are simulation inputs, not observations of East Berlin in 1986. Floor supply, location characteristics, worker population, and within-East and within-West travel times remain fixed while only cross-Wall connectivity changes.
This explains why East Berlin participates in the simulation but not historical scoring, and why the additional counterfactual is correctly constructed to isolate cross-Wall connectivity. The repeated-run results above use the original scenario.
W2. V1's negative correlations partly reflect the counterfactual setup. The original scenario keeps Eastern employment and wages at zero, so opening cross-Wall connections does not restore access to Eastern jobs. The table compares this scenario with the additional construction above. Pearson r measures the correlation between predicted and observed centered log changes, and correct directions count zones with matching signs in the original scenario.
Residence employment and wage correlations become positive, directly supporting the reviewer's explanation that the setup affects V1's performance. Prices and workplace employment remain weakly predicted, so these results identify a setup effect without establishing a general failure of structural models.
W2. V1's housing-price direction errors reflect a reversed spatial pattern. Across the same 55 zones, we correlate predicted and observed housing-price changes separately with distance to the central business district (CBD), using the distance measure in the Berlin data.
V1 predicts larger relative price increases farther from the CBD, whereas observed changes (and our V5 predictions) show larger relative increases closer to it. This reversed spatial pattern helps explain why V1 predicts the correct housing-price direction in only 19 of 55 zones.