- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ScaleCity addresses the learning of urban area representations across multiple tasks and spatial configurations, a problem of practical value. The reuse of atomic grids, the generation of conditional parameters, and the explicit modelling of area size are well-targeted; the experiments cover multiple cities, tasks and spatial configurations, and report significant performance improvements. However, the sources of these performance improvements have not yet been fully disentangled, and the evaluation independence and the accuracy–cost trade-off require further verification.
Critical strengths
1: The problem is closely aligned with practical needs, jointly addressing different spatial scales and area boundaries, and thus holds practical value.
2: The atomic grid reuse, conditional parameter generation and area size information are well-designed.
3: The experiments cover multiple cities, tasks and spatial configurations, and report significant performance improvements.
Critical weaknesses
1: Some baseline results differ significantly from those in the corresponding original papers. This paper has modified the implementation of certain baselines, and the rationale for these modifications requires explanation. Furthermore, the inputs for some baselines differ from the original settings; Appendix C.4 reports that the baselines utilised settings from their original papers, or default or fixed parameters, without conducting systematic hyperparameter search. It remains unverified whether these parameters remain applicable following changes to the inputs and implementation.
2: The independence of the evaluation set needs to be strengthened. Appendix C.2 and the open-source code (prompt_learning/train.py) indicate that the paper uses the same holdout fold for both checkpoint selection and final scoring, which may lead to over-optimistic results.
3: This paper and the baselines differ in terms of inputs, supervision methods and prediction architectures. To clarify the contribution of the hypernetwork, the first-stage representation must be fixed; independent predictors, simple multi-task models and the hypernetwork proposed in this paper should be compared under conditions where the same region-size information, aggregation methods and evaluation protocols are used, in order to isolate the contributions of each component.
4: The use of an overlap-weighted average for regional features may result in the loss of intra-region differences and spatial arrangements. The final aggregation may cause regions with different internal structures to receive similar representations.
5: There is a lack of efficiency comparisons under uniform conditions. The time taken per epoch is insufficient to demonstrate the overall cost advantage of the shared model. The total training cost, including pre-training, should be reported and compared with standalone predictors based on the same backbone to clarify the trade-off between accuracy and cost.
Questions for authors
This is a valuable piece of research. From a reviewer’s perspective, whilst I acknowledge its practical value and system design, I currently have reservations regarding the comparability of baselines, the independence of the evaluation, and the sources of performance gains. The aforementioned experiments will directly influence the assessment of the main conclusions.
9 comments
W2: Independent validation
Comment:
W2. The independence of the evaluation set needs to be strengthened. Appendix C.2 and the open-source code (prompt_learning/train.py) indicate that the paper uses the same holdout fold for both checkpoint selection and final scoring, which may lead to over-optimistic results.
Response:
We thank the reviewer for checking the released code and recommending independent validation.
1. Nested protocols. Outer folds are identical to Table 1, allowing direct comparison. Within each outer training portion (80%), 12.5% of L11 anchors form an inner validation set, approximately 10% of all anchors. Scheduling, early stopping, and checkpoint selection use only inner-validation metrics; ScaleCity uses the macro average over 72 task–configuration pairs. After training, the selected checkpoint is loaded for one inference pass on the outer test fold, which is excluded from downstream training and all selection decisions.
2. Baseline handling. All baselines requiring training-time selection—ViT, SpaBERT, GeoVectors, CityFM, HAFusion, and HREP—were retrained under the same nested protocol. ComSRE, Urban2Vec, and UrbanMFM use fixed settings and a single fit per fold. Their Table 1 scores already follow “fit on training folds, evaluate once on the test fold” and therefore remain unchanged.
3. Results (Overall R², submission → nested).
| Method | Beijing | Chengdu | Shenzhen |
|---|---|---|---|
| ScaleCity (ours) | 0.7833 → 0.7281 | 0.6707 → 0.5984 | 0.6441 → 0.5715 |
| ViT | 0.3450 → 0.2025 | 0.2707 → 0.0682 | 0.2953 → 0.0857 |
| SpaBERT | 0.2922 → 0.1824 | 0.3397 → 0.2242 | 0.1814 → 0.0354 |
| GeoVectors | 0.2712 → 0.1214 | 0.2795 → 0.1214 | 0.1793 → −0.0770 |
| CityFM | 0.2200 → 0.1449 | 0.1735 → 0.1265 | 0.1495 → −0.0288 |
| HAFusion | 0.2712 → 0.3039 | 0.3324 → 0.2925 | 0.2396 → 0.2104 |
| HREP | 0.0978 → 0.0304 | 0.1011 → −0.0167 | 0.0783 → 0.0038 |
| ComSRE (single-fit, unchanged) | 0.3255 | 0.3775 | 0.3348 |
| Urban2Vec (single-fit, unchanged) | 0.2466 | 0.2529 | 0.2439 |
| UrbanMFM (single-fit, unchanged) | 0.2358 | 0.1318 | 0.0531 |
ScaleCity remains first in all three cities. Relative gains over each protocol’s strongest baseline change from +127.0% / +77.7% / +92.4% to +123.7% / +58.5% / +70.7% in Beijing / Chengdu / Shenzhen.
4. Supplementary split robustness. Our earlier comparison of three random data splits yielded an R² standard deviation of 0.0332, providing additional evidence of stability across the tested splits.
W3: Matched predictor controls
Comment:
W3. This paper and the baselines differ in terms of inputs, supervision methods and prediction architectures. To clarify the contribution of the hypernetwork, the first-stage representation must be fixed; independent predictors, simple multi-task models and the hypernetwork proposed in this paper should be compared under conditions where the same region-size information, aggregation methods and evaluation protocols are used, in order to isolate the contributions of each component.
Response:
We thank the reviewer for asking us to clarify the baseline adaptations and distinguish the contributions of the inputs, joint supervision, and conditional prediction.
1. Matched inputs and training setup. Table 1 baselines use method-specific inputs and separate predictors, adapted to multiple regional configurations as explained in W1 and Appendix C.4. The controls below hold fixed the frozen SVI, POI, and AlphaEarth embeddings, extent covariate, overlap-weighted aggregation, sampling, five-fold splits, and evaluation metric:
- P: 72 independent MLPs, one per task–configuration pair (0.41M parameters each).
- J1/J2: a joint MLP (0.42M parameters), using learnable task/configuration ID embeddings or one-hot identifiers.
- H: a joint shared trunk with one head per task–configuration pair (1.59M parameters).
- S: the full ScaleCity hypernetwork with descriptive prompts.
Overall R² under the submission protocol (five-fold mean):
| Method | Beijing | Chengdu | Shenzhen |
|---|---|---|---|
| P: independent MLP | 0.5317 | 0.3690 | 0.3611 |
| J1: joint MLP (learnable ID embeddings) | 0.7082 | 0.6401 | 0.5847 |
| J2: joint MLP (one-hot) | 0.6654 | 0.6184 | 0.5693 |
| H: multi-head model | 0.6321 | 0.4711 | 0.5862 |
| S: ScaleCity (hypernetwork) | 0.7833 | 0.6707 | 0.6441 |
2. Joint training and conditional prediction. P→J1 gains +0.177 / +0.271 / +0.224 R² in Beijing / Chengdu / Shenzhen. ScaleCity exceeds J1 by +0.075 / +0.031 / +0.059, and also exceeds J2 and H in every city. AlphaEarth and extent are available to every predictor, so their availability does not explain these differences.
3. Sparse regional supervision. The coarsest road-network configuration L1 has 207 / 114 / 102 regions in Beijing / Chengdu / Shenzhen, with only 128 / 58 HFP-labeled regions in Beijing / Chengdu. Each L1 head sees about 41–102 training samples per fold. H has cross-task mean R² −0.62 / 0.03 in Chengdu / Beijing, versus J1's 0.546 / 0.505 and ScaleCity's 0.385 / 0.494, supporting sharing across sparsely labeled task–configuration pairs.
4. Parameter-generation components. The following ablations retain the frozen modalities, extent, descriptive prompts, and joint training. They replace generated projections and gates with shared trainable projections and uniform weights, or replace the generated readout with a shared static head:
| Setting | Beijing | Chengdu | Shenzhen |
|---|---|---|---|
| Full ScaleCity | 0.783 | 0.671 | 0.644 |
| Shared modality projections + uniform gates | 0.710 | 0.561 | 0.532 |
| Shared static prediction head | 0.632 | 0.585 | 0.578 |
Both replacements reduce accuracy, supporting conditional fusion and prediction with the input information and prompt form unchanged.
5. Identity versus descriptive text. Within the same hypernetwork, descriptions and symbolic identifiers achieve three-city mean R² of 0.699 and 0.695 on known conditions (Appendix G). Identity is therefore effective, with a modest descriptive-text benefit. The component controls retain descriptions while changing parameter generation. Descriptions also represent unseen configurations in the transfer experiments.
W1: Baseline implementations and hyperparameters
Response:
We thank the reviewer for asking us to clarify baseline performance and implementation choices.
1. Baseline performance and fairness. Each baseline is trained separately for every task–configuration–fold and evaluated on the same regions, labels, and metrics. Table 1’s Configuration R² (min–max) shows that baselines perform strongly at favorable scales, consistent with their original scale-specific evaluations. Exact scores also depend on cities, data sources, and regional definitions. Averaging across configurations captures weaknesses beyond those favorable scales. ScaleCity sustains performance across all evaluated shapes and scales with a single jointly trained city model.
2. Reasons for implementation adaptations. We adapted the baselines to accept our multi-scale regional inputs (e.g., removing positional encodings tied to fixed grids or replacing fusion modules incompatible with these inputs), while preserving each method’s core architecture. Appendix C.4 documents these implementation choices.
3. Hyperparameter applicability. We used the published, default, or fixed settings in Appendix C.4 without systematic per-method search. Baseline performance at favorable scales remains consistent with the original studies, supporting the applicability of published or default settings under our adapted inputs. The nested cross-validation rerun (W2) also preserves ScaleCity’s overall lead in all three cities.