Where from Above, What from the Street: Learning Semantic Routing for Cross-View Segmentation under Pose Uncertainty

Cross-view semantic segmentation aims to integrate satellite and street-view imagery to produce fine-grained urban semantic maps by exploiting complementary observations across perspectives. Satellite imagery provides a stable spatial reference for organizing predictions, while street-view imagery contributes additional fine-grained semantic evidence complementary to overhead observations. Existing approaches typically assume accurate spatial registration between the two views and exploit this correspondence for cross-view feature alignment and fusion. However, such accurate registration is difficult to guarantee in practical scenarios, as camera locations and orientations obtained from positioning sensors or pose estimation may contain errors. We argue that, for this task, exploiting street-view semantics does not necessarily require exact pixel-level geometric correspondence; instead, the key is to learn how semantic evidence should be associated with spatial entities in the satellite representation. Based on this perspective, we formulate cross-view fusion as a semantic routing problem and propose CVRoute, which learns to route street-view semantic evidence to relevant satellite regions under uncertain geometry. Specifically, CVRoute models satellite regions and street-view observations as cross-view entities and learns their semantic relationships to enable robust evidence transfer. To stabilize relationship learning, CVRoute introduces a training-only privileged relationship teacher and gradually transitions from teacher-guided routing to fully predicted routing through curriculum learning. Experiments on OmniCity and Brooklyn demonstrate that CVRoute consistently improves segmentation performance and exhibits substantially smaller degradation under increasing pose perturbations, reducing the dependence on precise spatial alignment.