acceptodds
Under review as a conference paper at ICLR 2027

Uncertainty-Aware Multimodal Fusion for Robust Urban Representation

Abstract

Urban representation learning increasingly relies on multiple sensing modalities to capture the diverse functional, behavioral, and physical characteristics of geographic regions. Existing methods improve multimodal representations through cross-view alignment or adaptive fusion, but largely treat each observed modality as deterministic evidence whose reliability is implicit in its features. This assumption is fragile in real urban sensing, where data quality and coverage can vary substantially across both modalities and regions. As a result, unreliable observations may distort cross-view semantic learning and propagate into the final representation. We introduce URGE, an uncertainty-aware framework built on a simple principle: a modality should be represented by both what it suggests about a region and how certain that suggestion is. URGE learns probabilistic view-specific semantics through Pairwise Gaussian Agreement, which evaluates cross-view compatibility relative to uncertainty without requiring different modalities to share the same uncertainty level. It then carries uncertainty into fine-grained token-level fusion, allowing reliability to condition how semantic evidence interacts across modalities. Finally, we introduce Relative Fusion Robustness, which uses controlled evidence degradation and counterfactual fusion to encourage the model to make productive use of uncertainty rather than merely predict it. Across real-world urban benchmarks covering multiple cities and downstream tasks, URGE yields more effective region representations and remains more robust when multimodal evidence is degraded.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Urban Representation Learning, Multimodal Learning, Uncertainty Modeling, Self-Supervised Learning, Geospatial Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QNYFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes URGE for multimodal urban region representation learning. It uses Gaussian embeddings to estimate uncertainty and guide alignment and fusion. Experiments on NYC and London show improvements in prediction results and robustness to degraded inputs.

Critical strengths
  1. The paper addresses a practical problem that urban data sometimes can be unreliable even when all modalities are available. Learning to handle this without downstream labels is useful.

  2. URGE performs better than the tested baselines across the reported tasks in both cities. The masking and replacement experiments also provide evidence of robustness to degraded inputs.

Critical weaknesses
  1. The PGA loss in Eqs. (6)–(7) has almost the same mathematical form as the negative mutual likelihood score in PFE (Shi and Jain, ICCV 2019). The paper does not cite or discuss this work, so it is unclear what is new in PGA.

  2. UrbanFusion (ICML 2026) also learns spatial representations from similar modalities and handles incomplete inputs, but the paper does not compare with it. The paper also states that it evaluates 29 tasks, while only 8 city–task combinations are reported. The remaining results and the reason for selecting these tasks are not provided.

  3. The results show that the model performs better under perturbations, but do not show whether its predicted variances actually identify unreliable observations. Maybe the variances simply work as learned fusion weights. A direct analysis of how they change when a modality becomes less reliable may help clarify this.

  4. The ablation study changes multiple components simultaneously. Removing probabilistic modeling also changes the alignment loss and removes uncertainty-aware fusion and RFR. Removing uncertainty-aware fusion also removes RFR. So, it is difficult to tell how much each component contributes.

Questions for authors
  1. How does PGA differ from the existing PFE objective?
  2. Can the authors provide results for all 29 tasks and compare with UrbanFusion under similar settings?
  3. When one modality is degraded, does its predicted uncertainty increase and its contribution to fusion decrease?
  4. Can the authors test each component separately while keeping the rest of the model unchanged?
1 comment
Area_Chair_G7u1OpenReview ↗

Invitation to Review and Discuss the Rebuttal

Dear Reviewer,

Thank you for your review. As the discussion continues, I’d welcome your thoughts on the authors’ response. If their reply answered your questions, a quick note to let them know would be appreciated. If anything remains unclear, please feel free to ask a follow-up question.

Best regards,

Area Chair

Reviewer_mCKHOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an uncertainty-aware multimodal framework for urban region representation learning, addressing variation in observation reliability across regions even when all modalities are available. It models modality-specific urban semantics as Gaussian distributions and uses the estimated uncertainty to guide alignment and fusion. A relative robustness objective further encourages stable representations under partial observations. Experiments show clear improvements on urban prediction tasks and greater robustness under token masking and semantic perturbations.

Critical strengths

S1. The paper addresses a practical problem in urban sensing: modalities can differ in reliability even when all are available. Learning to handle this variation without downstream labels is an interesting direction.

S2. The framework uses uncertainty to guide both cross-modal alignment and fusion, with a robustness objective that supports stable representations under partial observations. These components are consistent with the paper’s motivation.

S3. The method shows strong effectiveness and consistently outperforms the baselines.

Critical weaknesses

W1. The motivation for using diagonal Gaussian distributions to model modality-specific semantic uncertainty is not clearly explained. Could you briefly discuss this choice and its assumption?

W2. Section 4.4 only considers random token masking and replacement. It would be interesting to discuss how URGE might perform under structured reliability variations, such as uneven spatial coverage or modality-specific sampling biases, which may also arise in real-world urban data.

W3. The related work could better discuss prior studies on multimodal uncertainty, such as [1–3]. Clarifying how URGE differs from/aligns with these methods would help readers understand its contribution. [1] Beyond Unimodal: Generalising Neural Processes for Multimodal Uncertainty Estimation. NeurIPS 2023. [2] Multimodal Patient Representation Learning with Missing Modalities and Labels. ICLR 2024. [3] Probabilistic Embeddings for Cross-Modal Retrieval. CVPR 2021.

W4. Could you specify the values of hyperparamters used in experiments, such as the margin mm in Eq. 11 and the threshold τ\tau in Eq. 12?

Questions for authors

W1-W4

2 comments
Area_Chair_G7u1OpenReview ↗

Invitation to Review and Discuss the Rebuttal

Dear Reviewer,

Thank you for your review. As the discussion continues, I’d welcome your thoughts on the authors’ response. If their reply answered your questions, a quick note to let them know would be appreciated. If anything remains unclear, please feel free to ask a follow-up question.

Best regards,

Area Chair

Thank you for your efforts. The new experiments address my previous concerns about the paper motivation and the assumptions. Overall, I think this is a good paper that addresses an interesting problem in the field of urban representation learning. I maintain my recommendation for acceptance and will raise my score accordingly.

Reviewer_R3FwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

URGE proposes a self-supervised framework for urban region representation learning under heterogeneous observation reliability. Each modality is represented by a diagonal Gaussian whose mean encodes regional semantics and whose variance represents uncertainty. A Pairwise Gaussian Agreement (PGA) objective aligns modality-specific semantics, while uncertainty guides both precision-weighted initialization and attention-based fusion. A Relative Fusion Robustness (RFR) objective encourages recomputed uncertainty to improve representation stability under partial observations. Experiments evaluate frozen embeddings through downstream Ridge regression in NYC and London, alongside token-masking and token-replacement robustness tests.

Critical strengths
  1. The problem is practically relevant, and the architecture addresses it coherently. Distinguishing modality availability from observation reliability is well motivated. Carrying uncertainty through semantic alignment, region initialization, and fine-grained fusion provides a consistent design rather than an isolated confidence-weighting mechanism.
  2. The reported results are promising across multiple prediction domains and perturbation types. URGE outperforms the compared baselines on all eight displayed city–task combinations. The token-replacement experiment is particularly relevant because it changes semantic reliability while preserving observation counts, complementing the evidence-removal setting.
Critical weaknesses
  1. The uncertainty formulation has an unresolved degeneracy, and its interpretation as reliability is insufficiently established. For one dimension, PGA is the summed variance. When cross-modal means agree, reducing V toward zero drives this term toward negative infinity. The across-region variance regularizer constrains semantic means and does not exclude this case: means can vary across regions while agreeing across modalities within each region. The manuscript does not specify a variance floor, clipping, or another mechanism addressing this degeneracy. Moreover, cross-modal disagreement can reflect complementary information rather than unreliable observations; downstream improvements alone do not establish that learned variances identify observation reliability.
  2. The experiments do not sufficiently isolate uncertainty modeling from augmentation and architectural changes. URGE uses masking during training, but a deterministic fusion baseline with matched masking and consistency training is not reported. Several ablations remove multiple mechanisms simultaneously, limiting attribution. Direct diagnostics connecting corruption severity, predicted uncertainty, and fusion weights are also missing. Reporting is incomplete: the manuscript states that 29 downstream tasks are evaluated but provides results for only eight city–task combinations, without uncertainty estimates, and Figure 3 does not clearly identify its evaluated task/city or aggregation protocol.
Questions for authors
  1. What prevents predicted variances from approaching zero when modality means align? Please specify any variance bounds, clipping, or regularization, and report variance distributions and PGA training dynamics.
  2. When one modality is selectively corrupted, does its predicted uncertainty increase and its fusion contribution decrease? Can the method distinguish corrupted evidence from clean but complementary evidence?
  3. How does URGE compare with a capacity-matched deterministic model using identical masking augmentation and a comparable consistency objective? Can uncertainty-based initialization, attention conditioning, and RFR be ablated separately?
  4. Can you provide all 29 downstream results with repeated-run variability, specify the spatial cross-validation and model-selection procedures, and identify the task/city aggregation and downstream predictor protocol used in Figure 3?
1 comment
Area_Chair_G7u1OpenReview ↗

Invitation to Review and Discuss the Rebuttal

Dear Reviewer,

Thank you for your review. As the discussion continues, I’d welcome your thoughts on the authors’ response. If their reply answered your questions, a quick note to let them know would be appreciated. If anything remains unclear, please feel free to ask a follow-up question.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.