acceptodds
Under review as a conference paper at ICLR 2027

LUSim: An LLM-Driven Spatial-Equilibrium Simulator for Urban Planning

Abstract

Quantitative ex-ante urban-policy evaluation has historically relied on closed-form spatial-equilibrium and low-dimensional discrete-choice models that constrain analysis to homogeneous agents and fixed-form parametric utilities, and thereby to smooth proportional responses around the calibrated baseline. Integrating large language models into multi-agent simulation alleviates these constraints, but existing LLM-driven platforms lack the market-clearing mechanism and externally calibrated parameters that quantitative policy evaluation requires. We present LUSim (LLM-driven Urban Simulator), a multi-agent simulator in which heterogeneous households choose residence and workplace through a large language model and clear housing and labour markets inside an externally calibrated general-equilibrium city, with structural decision rules as baselines under identical conditions. We simulate a new rapid-transit line on a 96-zone Berlin instance calibrated from Ahlfeldt et al. (2015). The structural engines, which can only reweight the calibrated city in proportion, concentrate their response at the new stations, whereas with LLM-driven households approximately eight times as many relocate and the gains shift to peripheral districts. To test which engine predicts real urban change, we initialise every engine on the divided Berlin of 1986, let it predict the reunified city, and score, by criteria fixed in advance, its prediction of which districts gain or lose residents, jobs, prices, and wages against the changes observed by 2006. The simulation with LLM-driven households predicts these changes better than every structural baseline, also with district names withheld so that it cannot rely on memorised history, and the contrast persists across a second rail corridor, a zoning policy that expands floor space, and a second model family. We interpret these results as evidence that LLM-driven decision rules, disciplined by market clearing in a calibrated environment, predict urban change beyond the reach of structural models.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM agents, Multi-agent simulation, Spatial equilibrium, Urban planning, Counterfactual policy evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_W4YnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LUSim, a multi-agent simulator. It shows that structural decision rules can be swapped in under the same environment, and validates this through specific experiments.

Critical strengths

Coupling an LLM decision rule with calibrated spatial-equilibrium parameters and market clearing is a quite novel idea.

Critical weaknesses

Weakness 1

One of my major concerns is that the historical validation does not establish positive predictive ability. Based on the reported results, the direction accuracy is indistinguishable from chance on all four variables, and the experiments also lack a permutation test on the average rank.

Weakness 2

Another major concern is that V1 is negatively correlated on all four variables. Does this really show that structural models fail here, or could it instead reflect the counterfactual setup?

Questions for authors

Please state how many East Berlin zones have usable 1986 values. How do the authors explain V1's below-chance direction accuracy?

2 comments
AuthorsOpenReview ↗

Predictive significance, counterfactual construction, and the structural baseline

We thank the reviewer for these questions.

W1. Repeated historical runs show predictive ability above chance. We repeat V4 and V5 ten times with fresh LLM responses under the original settings (V5, K=25 archetypes, 20-iteration cap). We evaluate 1986–2006 log changes in 55 West Berlin zones, centering each predicted and observed series by subtracting its mean across these zones.

The table reports mean relative RMSE, which divides root mean squared error by the standard deviation of observed changes (so zero change scores 1.00). Direction accuracy is the fraction of zones with matching predicted and observed signs after centering. We obtain 95% confidence intervals (CI) by resampling paired runs and neighboring zone groups.

OutcomeV5 errorV4 errorV5 direction accuracy [95% CI]
Housing prices0.941.110.69 [0.58, 0.79]
Residence employment0.991.150.66 [0.55, 0.76]
Workplace employment0.991.080.64 [0.53, 0.74]
Wages0.971.010.62 [0.51, 0.72]

For all four outcomes, V5's error is below the zero-change benchmark and its direction-accuracy interval lies above 0.50, directly addressing whether its predictions outperform chance.

W1. The requested permutation test supports V5's average ranking. Following the original scoring rule, we rank V1–V4 and V5 with and without district names by Pearson correlation of predicted and observed changes, Spearman correlation of their zone rankings, direction accuracy, and relative RMSE, using the available metrics for each outcome.

We shuffle observed changes in neighboring zone groups, applying the same shuffle across outcomes and repetitions, and recompute V5's mean rank across metrics and repetitions. Lower ranks are better.

V5 inputsMean rankPermutations at least as favorablep-value
Names shown1.812519/10,0000.002
Names withheld2.125079/10,0000.008

Fewer than 1% of permutations yield ranks as favorable as V5's, supporting its average ranking beyond chance with and without names.

W2. No Eastern zone has usable 1986 outcomes, and the additional counterfactual includes both halves of Berlin. Usable 1986 values exist for all four outcomes in 55 Western zones and none of the 41 Eastern zones, so historical accuracy is evaluated only in the West.

The additional scenario uses 1986 Western and 2006 Eastern economic inputs in both network states, following Section 6.3 of Ahlfeldt et al. (2015). Eastern values are simulation inputs, not observations of East Berlin in 1986. Floor supply, location characteristics, worker population, and within-East and within-West travel times remain fixed while only cross-Wall connectivity changes.

This explains why East Berlin participates in the simulation but not historical scoring, and why the additional counterfactual is correctly constructed to isolate cross-Wall connectivity. The repeated-run results above use the original scenario.

W2. V1's negative correlations partly reflect the counterfactual setup. The original scenario keeps Eastern employment and wages at zero, so opening cross-Wall connections does not restore access to Eastern jobs. The table compares this scenario with the additional construction above. Pearson r measures the correlation between predicted and observed centered log changes, and correct directions count zones with matching signs in the original scenario.

OutcomeOriginal rCorrect directionsAdditional r
Housing prices-0.3219/55-0.046
Residence employment-0.0925/550.261
Workplace employment-0.0328/55-0.067
Wages-0.1420/550.208

Residence employment and wage correlations become positive, directly supporting the reviewer's explanation that the setup affects V1's performance. Prices and workplace employment remain weakly predicted, so these results identify a setup effect without establishing a general failure of structural models.

W2. V1's housing-price direction errors reflect a reversed spatial pattern. Across the same 55 zones, we correlate predicted and observed housing-price changes separately with distance to the central business district (CBD), using the distance measure in the Berlin data.

Housing-price changesPearson r with distance to CBD
V1 predictions+0.40
Observed changes-0.32

V1 predicts larger relative price increases farther from the CBD, whereas observed changes (and our V5 predictions) show larger relative increases closer to it. This reversed spatial pattern helps explain why V1 predicts the correct housing-price direction in only 19 of 55 zones.

Thank you for the detailed response. However, my cocerns still remain after the rebuttal, and I therefoe maintain my rating.

Reviewer_gYHvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents LUSim, an LLM-driven urban simulator that integrates LLM-based household location decisions with a calibrated spatial-equilibrium model. It simulates policy-induced changes in residential and workplace distributions, housing prices, and wages through iterative market clearing. Experiments on Berlin compare LLM-based decisions against structural economic models, including a historical validation using Berlin's reunification.

Critical strengths
  1. The paper addresses a meaningful problem by connecting LLM-based agent simulation with quantitative urban policy evaluation.

  2. The modular framework provides a unified interface for different decision engines and incorporates endogenous market feedback.

  3. The paper includes real-world economic data, historical validation, multiple policy scenarios, and detailed implementation and reproducibility materials.

Critical weaknesses

W1. Weak predictive performance and insufficient baselines.
Historical predictions are close to chance (near-zero correlations and ~50% directional accuracy). V5's relative RMSE (0.99, 1.25, 1.07, 1.00) barely matches a trivial zero-change baseline (~1). Stronger heterogeneous choice baselines are needed, and the exclusion of existing LLM simulators warrants clearer justification.

W2. Lack of statistical reliability.
No repeated independent runs, confidence intervals, or significance tests are provided. Cache replay ensures reproducibility, not robustness to LLM stochasticity. The observed sensitivity to prompt wording and archetype count further raises concerns about result stability.

W3. Confounded comparison of decision mechanisms.
V5 differs from the structural baselines not only in behavioral modeling but also in its two-stage selection and cluster-level assignment. These mechanisms may themselves produce larger population reallocations. Matched baselines are needed to isolate the contribution of LLM-based reasoning.

W4. Limited validity of historical evaluation.
Changes during 1986–2006 reflect many factors beyond transportation reunification, which mean-centering cannot eliminate. Anonymizing district names also cannot fully exclude historical knowledge effects: V5's housing-price correlation drops from 0.25 to -0.16 after anonymization.

W5. Insufficient behavioral calibration and equilibrium convergence.
External calibration applies to the economic environment, not the LLM's household decisions. V5's baseline housing-price correlation is -0.24. More critically, the historical simulation fails to reach the stated 10−310^{-3} convergence tolerance even after 50 iterations, undermining the central spatial-equilibrium claim.

W6. Limited methodological novelty and generalization.
The method largely integrates LLM prompting with an existing equilibrium model, without substantial new behavioral learning or equilibrium-solving techniques. Moreover, the Singapore and block-level experiments omit the core V5 engine, leaving its generalization and scalability unverified.

Questions for authors
  1. How does the framework validate or calibrate LLM-generated household preferences against observed individual-level residential or workplace choices, beyond calibrating the economic environment?

  2. Could the authors clarify the discrepancy in the reported V5 mean housing-price change at K=50 between Table 2 (0.0111) and Table 13 (0.0019)?

  3. It would also help to clarify the paper's intended scope as urban policy simulation/evaluation, rather than automated urban planning or policy optimization.

1 comment
AuthorsOpenReview ↗

Baseline comparisons, statistical reliability, and equilibrium convergence

We thank the reviewer for the detailed comments.

W1. Stronger heterogeneous choice baselines. We compare V5 with V4, which elicits heterogeneous household-specific weights from an LLM, under the same economic environment and market-clearing framework to test V5's gains beyond an LLM-based heterogeneous preference model. The table evaluates centered 1986–2006 changes in 55 West Berlin zones using relative RMSE, which divides root mean squared error by the standard deviation of observed changes (so zero change scores 1.00).

Outcome (original)V4V5 namedV5 anonymous
Housing prices1.110.991.10
Residence employment1.151.250.97
Workplace employment1.081.071.05
Wages1.011.001.01

Anonymous V5 improves residence employment prediction over heterogeneous V4 and zero change.

W2. Independent repetitions. We repeat V4 and V5 ten times with fresh LLM responses (V5, K=25, 20-iteration cap), reporting mean relative RMSE and paired gaps with 95% intervals from resampling run pairs and neighboring zone groups.

OutcomeV5V4V4 minus V5 [95% CI]
Housing prices0.941.110.17 [0.09, 0.25]
Residence employment0.991.150.16 [0.04, 0.28]
Workplace employment0.991.080.09 [0.03, 0.15]
Wages0.971.010.04 [0.01, 0.07]

The positive intervals support V5's lower errors across runs. Across 18 model, prompt, and archetype settings, the smallest mean gap over four outcomes is 0.055 with an adjusted 95% interval [0.012, 0.098], supporting V5's advantage across these settings.

W3. Matched decision mechanisms. To isolate the effect of LLM scores, we convert V1's residence probabilities and workplace probabilities given residence into V5's score format, retaining V5's grouping, two-stage selection, shortlist, sampling, and market clearing.

East-West ExpressNet reallocation, % [95% CI]
V51.12 [1.00, 1.24]
Structural control0.23 [0.19, 0.27]
V5, no policy0.08 [0.05, 0.11]

Reallocation halves the sum of absolute zone changes in employed residents to count each transfer once, then divides by their baseline total. V5 exceeds the structural control by 0.89 percentage points [0.76, 1.02], above no-policy variation, supporting an effect of LLM scores with decision mechanisms held fixed.

W4. District names and concurrent shocks. We test dependence on names using Pearson r between predicted and observed historical housing-price log changes across 55 zones, and V4 minus V5 relative RMSE averaged over four outcomes and ten runs.

MeasureNames shownNames withheld
Original manuscript's housing-price correlation0.25-0.16
Mean error gap0.1150.090 [0.040, 0.140]

The positive interval supports V5's average advantage over V4 without names despite the original price correlation's sensitivity. Since centering leaves local shocks, historical evaluation measures predictive fit, while policy simulations isolate transport effects by fixing other exogenous inputs.

W5 and Q1. Behavioral validation. We assess aggregate fit through Pearson r between simulated and observed 2006 levels across 96 zones.

ModelHousing pricesResidence employmentWorkplace employmentWages
V5-0.240.720.890.88

V5 fits employment better than prices, and we distinguish this evaluation from preference calibration, for which linked individual choice data are unavailable.

W5. Equilibrium convergence. The original simulation converges at a tolerance of 0.01, and Anderson acceleration reaches 0.001 by combining recent price and wage updates using their residuals while retaining V5's decision rules and economic equations.

For each market, the residual is the maximum across zones of the absolute difference between a zone's share of total citywide demand and its share of total citywide supply.

Reunified-city marketRequired toleranceFinal residual
Housing0.0010.00062
Labor0.0010.00078

The housing residual of 0.00062 and labor residual of 0.00078 both meet the 0.001 market-clearing threshold at the same solution, satisfying the stricter criterion.

W6 and Q3. Generalization and scope. We test full V5 in Singapore and at finer Berlin resolution.

Full V5ZonesHousing residualLabor residual
Singapore270.00092Not modeled
Berlin3840.000730.00091

Markets clear at 0.001, and Berlin's price changes aggregated to 96 zones have Pearson r=0.93 with the original resolution's changes, supporting V5 across settings and resolutions. LUSim couples heterogeneous LLM choices with market feedback to evaluate user-specified policies.

Q2. Different price measures.

TableMeasure at K=50Value
2Log price-index change with scenario-specific residence weights0.0111
13Price-level change averaged with baseline residence weights0.0019

Table 2 uses changing residence weights and log changes, while Table 13 averages level changes at fixed weights.

Reviewer_ouGoOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

LUSim is a spatial-equilibrium urban simulator in which LLM-driven households choose residence and workplace while housing and labor markets clear under an externally calibrated urban model. The main empirical claim is that the LLM decision engine produces qualitatively different policy responses from structural baselines and better predicts the spatial pattern of change around Berlin reunification, with additional checks on a second corridor, a zoning intervention, and a Singapore instance.

Critical strengths

-tackles a genuinely interesting question: whether an LLM-based behavioral layer can add something beyond standard structural urban models when embedded inside a market-clearing simulator. -controlled comparison against several structural variants is thoughtful and more serious than most LLM-agent simulation papers. -historical validation against Berlin reunification is the strongest part of the paper. Even though the effect sizes are modest, the authors do attempt a held-out prediction exercise rather than relying only on attractive counterfactual maps.

Critical weaknesses

-empirical case is still too fragile for the strength of the headline claims. Historical validation rests on one event and 55 zones, correlations are modest. -The paper itself shows sensitivity to prompt wording, archetype count, and provider snapshot. That makes the claimed advantage over models feel suggestive rather than well established.

-uncertainty quantification is thin where it matters most. Several key LLM results are effectively single-configuration point estimates, and the strongest qualitative claims about mechanism and policy response are not backed by seed-level or model-level variability analyses at the same level of prominence.

Questions for authors

Can you report uncertainty estimates for the main V5 results in Sections 5.1 and 5.2, not just robustness appendices, so that the gap to structural baselines can be judged statistically rather than visually?

The paper argues that V5 captures labor-market pooling rather than compensating differentials. What direct falsification test shows this is not an artifact of prompt-induced preferences or the archetype approximation?

1 comment
AuthorsOpenReview ↗

Uncertainty in the main results and tests of the wage mechanism

We thank the reviewer for requesting uncertainty estimates for Sections 5.1 and 5.2 and direct tests of the wage mechanism.

The East-West Express results survive independent runs. To assess whether the policy responses exceed LLM and sampling variation, we ran ten independent baseline-policy pairs per model with fresh LLM responses. V5 uses K=50 with district names shown. V4 elicits heterogeneous household-specific weights from an LLM, while the structural control converts V1's residence probabilities and workplace probabilities given residence into V5's score format and retains V5's grouping, two-stage selection, shortlist, sampling, and market clearing. Brackets give 95% confidence intervals from resampling baseline-policy pairs.

Policy responseV5V4Structural control
Net residential reallocation, %1.12 [1.00, 1.24]0.21 [0.18, 0.24]0.23 [0.19, 0.27]
Station share, %3.8 [2.6, 5.1]48.3 [46.0, 50.6]46.7 [42.9, 50.5]
Central wage change, %+0.84 [0.62, 1.06]-0.17 [-0.24, -0.10]-0.15 [-0.22, -0.08]
Log housing-price index change0.0108 [0.0086, 0.0130]0.0007 [0.0004, 0.0010]0.0012 [0.0008, 0.0016]

Reallocation sums absolute zone changes in employed residents, halves the sum, and divides by the citywide baseline total. Moving 100 residents adds 100 at the destination and removes 100 at the origin, so halving counts the transfer once, giving 1.12% for V5 and 0.23% for the control. Station share is the percentage of absolute residential change in the four station zones. Central wages average percentage changes in Mitte and Charlottenburg, and the price index uses each scenario's residence weights, as in Table 2.

Ten V5 pairs with identical travel times give 0.08% [0.05, 0.11] reallocation. The policy gap to the control is 0.89 percentage points [0.76, 1.02], exceeding this variation. The separated station-share intervals and opposite wage signs also persist, supporting differences in response size, location, and wage direction beyond a single run.

V5's historical advantage over heterogeneous V4 persists across repetitions. To assess the stability of the results in Section 5.2, we repeat both models ten times with fresh LLM responses and the original scenario and settings, including K=25 and a 20-iteration cap for V5. Scoring uses centered 1986–2006 changes in the 55 West Berlin zones with usable baseline data.

The table reports relative RMSE, which divides root mean squared error by the standard deviation of observed changes (so zero change scores 1.00). Positive V4 minus V5 gaps favor V5, with intervals obtained by resampling run pairs and neighboring zone groups.

OutcomeV5V4V4 minus V5 [95% CI]
Housing prices0.941.110.17 [0.09, 0.25]
Residence employment0.991.150.16 [0.04, 0.28]
Workplace employment0.991.080.09 [0.03, 0.15]
Wages0.971.010.04 [0.01, 0.07]

All four intervals exclude zero, supporting lower errors after accounting for run and spatial variation. With names withheld, the mean gap is 0.090 [0.040, 0.140], versus 0.115 with names shown, so V5's average gain also persists without explicit district names.

Model, prompt, and archetype changes preserve the historical advantage. To address configuration sensitivity, we vary K, model A (GPT 5.3) versus B (DeepSeek V4 Flash), and the original prompt P0 versus rewordings P1 and P2. Each cell averages V4 minus V5 relative RMSE over four outcomes and three fresh repetitions, with V4's settings fixed.

KA/P0A/P1A/P2B/P0B/P1B/P2
250.1170.1060.0950.0980.0880.076
500.1240.1150.1080.1030.0920.082
750.1090.1000.0890.0930.0730.055

All 18 gaps favor V5, and the smallest, 0.055 [0.012, 0.098], remains positive after adjustment for 18 comparisons. Thus V5's predictive advantage holds across the tested models, prompts, and archetype counts.

Removing productivity feedback reverses the central wage response at K=50. To test the pooling explanation against prompt and grouping artifacts, we hold productivity fixed, use neutral prompt wording, and compare grouped and individual queries for the same 500 simulated household types.

InterventionCentral wage change, % [95% CI]
Full V5+0.84 [0.62, 1.06]
Productivity held fixed-0.12 [-0.20, -0.04]
Neutral prompt wording+0.79 [0.57, 1.01]
500 household types grouped+0.81 [0.61, 1.01]
Same types queried individually+0.80 [0.60, 1.00]

Individual querying changes the wage response by -0.01 percentage points [-0.05, 0.03], while neutral wording preserves its positive sign. Only disabling the effect of employment density on productivity reverses the sign, supporting the productivity channel underlying the pooling interpretation at K=50.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.