Kalman-Inspired Cross-Scale Selective Fusion in U-Nets

Standard U-Net decoding is globally multi-scale, yet each decoder stage usually receives only its matching encoder feature. We study whether aligned non-matching scales contain task-relevant information beyond that matching skip and how such evidence can be fused. Specifically, we formulate cross-scale fusion as sequential inference from complementary observations with varying reliability. Within a linear-Gaussian latent observation model along the scale axis, the classical Kalman recursion motivates prediction, innovation, and adaptive correction. Guided by this insight, we develop Cross-scale Selective Fusion U-Net (CSF U-Net), which replaces same-scale skip concatenation with a recurrent prediction–innovation–correction scan over aligned encoder scales. Experiments on four medical image segmentation benchmarks show that CSF U-Net consistently improves segmentation accuracy and boundary quality over vanilla U-Net and performs competitively with representative U-Net variants. Ablation studies further support the effectiveness of the recurrent scan, innovation subtraction, and adaptive correction for cross-scale fusion.