acceptodds
Under review as a conference paper at ICLR 2027

Where from Above, What from the Street: Learning Semantic Routing for Cross-View Segmentation under Pose Uncertainty

Abstract

Cross-view semantic segmentation aims to integrate satellite and street-view imagery to produce fine-grained urban semantic maps by exploiting complementary observations across perspectives. Satellite imagery provides a stable spatial reference for organizing predictions, while street-view imagery contributes additional fine-grained semantic evidence complementary to overhead observations. Existing approaches typically assume accurate spatial registration between the two views and exploit this correspondence for cross-view feature alignment and fusion. However, such accurate registration is difficult to guarantee in practical scenarios, as camera locations and orientations obtained from positioning sensors or pose estimation may contain errors. We argue that, for this task, exploiting street-view semantics does not necessarily require exact pixel-level geometric correspondence; instead, the key is to learn how semantic evidence should be associated with spatial entities in the satellite representation. Based on this perspective, we formulate cross-view fusion as a semantic routing problem and propose CVRoute, which learns to route street-view semantic evidence to relevant satellite regions under uncertain geometry. Specifically, CVRoute models satellite regions and street-view observations as cross-view entities and learns their semantic relationships to enable robust evidence transfer. To stabilize relationship learning, CVRoute introduces a training-only privileged relationship teacher and gradually transitions from teacher-guided routing to fully predicted routing through curriculum learning. Experiments on OmniCity and Brooklyn demonstrate that CVRoute consistently improves segmentation performance and exhibits substantially smaller degradation under increasing pose perturbations, reducing the dependence on precise spatial alignment.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Cross-View Semantic Segmentation, Cross-View Learning, Pose Uncertainty, Semantic Routing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gBibOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CVRoute for satellite-space building-attribute segmentation via paired satellite and ground-level street-view imagery under uncertain geometry. The key insight is to learn how semantic evidence should be associated with spatial entities in the satellite representation. Specifically, CVRoute models satellite regions and street-view observations as cross-view entities and learns their semantic relationships for robust evidence transfer, with a training-only privileged relationship teacher guiding early optimization through curriculum learning.

Critical strengths
  • The method is innovative. Anchoring predictions in satellite regions while allowing uncertain associations with street-view evidence is a sensible response to pose errors. The specific combination of pose marginalization and nullable entity routing is interesting. CVRoute achieves the highest Avg4 mIoU across all four level combinations on two benchmarks in Tables 1–2.

  • The paper provides several necessary ablation studies. Table 5 indicates the effectiveness of teacher-guided routing and demonstrates that objective contrastive learning consistently introduces slight degradation.

Critical weaknesses
  • The reproduction of the baseline SG-BEV requires stronger validation. SG-BEV achieves only 29.72 Clean mIoU on OmniCity floor prediction, whereas the original SG-BEV paper reports 40.64, despite Appendix A.5 stating that Clean follows the original nominal-pose protocol.

  • The ablation studies in Tables 5 and 7 primarily investigate teacher-guided routing and contrastive learning, while several key components of the proposed method remain insufficiently validated. In particular, the paper does not compare pose-marginalized routing against single-pose or MAP-based routing, making it unclear whether marginalization is necessary for improved robustness. Moreover, Appendix A.3 introduces a clean-to-noisy pose-consistency loss, which may itself contribute to robustness under pose perturbations, but its effect is not isolated. These missing ablations make it difficult to attribute the reported improvements specifically to the proposed semantic routing mechanism.

Questions for authors
  • What explains the gap between SG-BEV and the original published Clean results? Were baseline pose augmentation, initialization, training budget, label mapping, and metric computation matched or otherwise validated?

  • How much performance and robustness remain with single-pose/MAP routing, ordinary attention without null/capacity constraints, and removal of the clean/noisy consistency objective?

2 comments
AuthorsOpenReview ↗

Follow-up on our responses and revised manuscript

Thank you again for your time and valuable feedback. We have revised the manuscript according to your comments and provided point-by-point responses. Could you please let us know if these revisions have addressed your concerns and whether any new concerns have arisen? We would greatly appreciate your response and would be glad to make further revisions according to your feedback.

Thank the authors for the rebuttal. I would like to maintain my original rating.

Reviewer_7N2fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies satellite-space building-attribute segmentation using paired satellite images and street-view panoramas when camera position and orientation are uncertain. CVRoute represents the two views as entities, scores their compatibility over a bank of 45 pose hypotheses, and marginalizes capacity-constrained associations with a null option. Ground-view semantic logits are routed to satellite entities and combined with satellite-side predictions. A training-only teacher uses corrected poses and annotation-derived components to guide routing early in training, with its contribution removed from epoch six onward. Experiments cover floor and land-use prediction on Brooklyn and OmniCity under four levels of synthetic pose perturbation. The reported results show smaller degradation under increasing perturbation than the reproduced projection-based baselines.

Critical strengths
  1. The formulation addresses a practical weakness of cross-view fusion and produces consistent robustness gains. Keeping the output anchored to satellite regions while allowing uncertain or unmatched ground evidence is well motivated. In Table 1, Brooklyn floor mIoU drops from 44.24 to 42.54 between Clean and Hard, compared with 44.12 to 38.17 for SG-BEV†. Average performance also improves on all four dataset/task combinations in Tables 1–2. The value is primarily robustness under uncertain geometry rather than a large clean-pose improvement.

  2. The paper includes useful checks on the contribution of its training procedure. Removing teacher-guided routing reduces Brooklyn floor Avg4 from 43.63 to 41.73, while adding contrastive learning offers little benefit (Tables 5 and 7). The panorama replacement results establish that the trained system depends on the paired ground observation, and the three additional runs in Table 6 provide some evidence of stability for the principal Brooklyn floor result.

Critical weaknesses
  1. The main robustness comparison does not establish sufficiently matched training conditions. Section 4.1 explicitly describes pose-perturbed training for CVRoute, but the baseline description in Appendix A.5 does not establish whether SG-BEV† and GP receive the same training perturbations. Appendix A.1 also describes 60 epochs of base geometry training, 2 of bootstrapping, and 10 of joint geometry training before the 8-epoch CVRoute stage for Brooklyn floor, whereas SG-BEV† is trained for 42 epochs. The controlled CVRoute ablations share an initialization, which is useful, but this does not resolve the comparison with other methods. Since robustness is the principal claim, the baseline training noise, supervision, and optimization budget need to be made explicit. This is a fixable evaluation concern, but it materially limits attribution of the gains to semantic routing.

  2. The evidence does not isolate the main routing mechanisms. The ablations vary the teacher and a contrastive loss, while retaining pose marginalization, nullable transport, capacity constraints, and the semantic writeback machinery. They therefore support the curriculum more directly than the proposed routing design. In addition, Appendix A.4 describes a bounded satellite semantic prior and a gated blend with a dense prediction. Table 3 replaces the panorama only at evaluation time, so its large drops cannot establish the gain over a separately trained satellite-only system. Comparisons that isolate the pose bank, null state, transport constraints, and matched single-view or simpler fusion alternatives would clarify which mechanisms are necessary. Existing results for any of these controls would be especially helpful.

Questions for authors
  1. Were SG-BEV† and GP trained with the same pose perturbations as CVRoute? Please specify the training noise distribution and sampling procedure for each method, including whether perturbations are fixed across epochs.
  2. For Brooklyn floor, can you clarify the total training stages and supervision available to each comparison method, including the geometry warm start? Are there existing comparisons that control for these differences?
  3. Do you have existing results for a single nominal pose instead of pose marginalization, removal of the null/capacity mechanisms, or a separately trained satellite-only model? If not, which part of the current evidence most directly supports the routing mechanism beyond teacher-guided optimization?
1 comment
AuthorsOpenReview ↗

Follow-up on our responses and revised manuscript

Thank you again for your time and valuable feedback. We have revised the manuscript according to your comments and provided point-by-point responses. Could you please let us know if these revisions have addressed your concerns and whether any new concerns have arisen? We would greatly appreciate your response and would be glad to make further revisions according to your feedback.

Reviewer_Rw1bOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper studies cross view semantic segmentation under noisy camera poses. Prior methods warp street view features to the overhead plane using the camera pose, which fails when the pose is inaccurate. The authors propose CVRoute, which replaces geometric warping with entity level semantic routing. The model samples multiple pose hypotheses around the noisy prior, matches ground and satellite entities using Sinkhorn matching with a null column, and routes ground semantics to satellite masks. A privileged teacher guides the routing matrix during early training. The method is evaluated on the Brooklyn and OmniCity datasets under four noise levels.

Critical strengths

The formulation of entity level routing is conceptually appealing because it decouples spatial layout from ground semantic attributes. In addition, using a privileged teacher to replace the forward routing path during early training is an effective solution to the circular dependency between entity extraction and routing convergence. Table 4 demonstrates that this training strategy substantially improves entity recall.

Critical weaknesses
  1. Potential protocol leakage and unverified pose reasoning (Section 4, Figure 3, Table 3): The paper does not state whether pose noise is applied by recropping satellite images and rotating panoramas, or only by changing the numerical pose prior value. In Brooklyn and OmniCity, the true camera is fixed at the satellite image center and faces north. If only the numerical prior is modified, learnable modules can memorize the fixed geometry and ignore the prior. This concern is supported by Figure 3, where performance barely drops under Hard noise even though the perturbation exceeds the hypothesis search range. Furthermore, in Table 3, feeding corrupted or zero street views reduces accuracy far below a satellite only baseline. This shows that the null assignment fails to fall back to satellite features under severe noise.

  2. Weak baselines, unequal training, and undocumented architecture (Section 4.3, Section 5, Figure 2, Appendix A.4): In Table 1 and Table 2, the reproduced SG-BEV baseline performs substantially worse than reported in the original SG-BEV paper. The training conditions are also unequal. CVRoute is trained for 80 epochs with noise augmentation and consistency losses, while baselines are trained for 42 epochs on clean poses only. Most critically, Appendix A.4 describes an undocumented dense prediction branch that is blended with the entity predictions. This branch is omitted from the main text and Figure 2. Because the paper lacks ablations on pose marginalization, the null bin, and individual scoring terms, it is unclear whether the gains come from semantic routing, the hidden dense branch, or simply noisy training.

Questions for authors
  • Does your perturbation protocol physically transform the input images, or does it only modify the numerical pose prior vector?

  • Why does your reproduced SG-BEV baseline underperform the original paper, and how do baselines perform when trained with the same noise augmentation and epoch budget?

  • What is the individual contribution of the dense prediction branch described in Appendix A.4, and how does the model perform without it?

  • In Table 3, why does corrupting the street view input degrade performance well below a satellite only model if the null mechanism functions properly?

1 comment
AuthorsOpenReview ↗

Follow-up on our responses and revised manuscript

Thank you again for your time and valuable feedback. We have revised the manuscript according to your comments and provided point-by-point responses. Could you please let us know if these revisions have addressed your concerns and whether any new concerns have arisen? We would greatly appreciate your response and would be glad to make further revisions according to your feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Reviewers

We sincerely thank all reviewers for their careful evaluation and constructive feedback. The comments have helped us identify aspects of the original manuscript that required clearer explanations and stronger experimental validation. In response, we have substantially revised the manuscript and conducted additional controlled experiments to address the raised concerns.

Specifically, our revisions and additional experiments focus on four aspects:

  1. Clarification of the CVRoute architecture. We have clarified the complete framework, including the complementary dense and entity-routing pathways, their respective supervision, and the uncertainty-aware fusion mechanism.

  2. Robustness under pose and observation-geometry changes. We have evaluated the model under actual changes to panorama heading and satellite crop position, together with correctly updated and perturbed pose inputs, to examine whether the observed robustness extends beyond the original fixed observation geometry.

  3. Baseline reproduction and fairer comparisons. We have further documented our SG-BEV reproduction protocol and evaluated GP and SG-BEV† under pose-noise-matched training, including additional controls with clean-pose supervision and clean-to-noisy consistency regularization.

  4. Analysis of the proposed design choices. We have added controlled studies of the dense and entity prediction pathways, individual association-score components, pose-hypothesis aggregation, null assignment, capacity-constrained transport, and clean-to-noisy consistency training. We have also included an independently trained satellite-only reference and additional quantitative and qualitative analyses.

The corresponding methodological clarifications, experimental results, and implementation details have been incorporated into the revised manuscript and supplementary material. We believe these revisions provide a more complete description of CVRoute, strengthen the experimental evidence supporting its key design choices, and offer a clearer understanding of its robustness and limitations under pose uncertainty.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.