acceptodds
Under review as a conference paper at ICLR 2027

Kalman-Inspired Cross-Scale Selective Fusion in U-Nets

Abstract

Standard U-Net decoding is globally multi-scale, yet each decoder stage usually receives only its matching encoder feature. We study whether aligned non-matching scales contain task-relevant information beyond that matching skip and how such evidence can be fused. Specifically, we formulate cross-scale fusion as sequential inference from complementary observations with varying reliability. Within a linear-Gaussian latent observation model along the scale axis, the classical Kalman recursion motivates prediction, innovation, and adaptive correction. Guided by this insight, we develop Cross-scale Selective Fusion U-Net (CSF U-Net), which replaces same-scale skip concatenation with a recurrent prediction–innovation–correction scan over aligned encoder scales. Experiments on four medical image segmentation benchmarks show that CSF U-Net consistently improves segmentation accuracy and boundary quality over vanilla U-Net and performs competitively with representative U-Net variants. Ablation studies further support the effectiveness of the recurrent scan, innovation subtraction, and adaptive correction for cross-scale fusion.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Medical image segmentation, U-Net, cross-scale feature fusion, selective state-space models, Kalman-inspired inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CziFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CSF U-Net, which replaces same-scale skip concatenation with a bidirectional recurrent block that fuses all aligned encoder scales. The block uses gated prediction, innovation, and correction steps, motivated by a Kalman recursion under a linear-Gaussian model along the scale axis and an analysis of the squared-loss risk gap. On SegPC, ISIC, GLaS and MoNuSeg, CSF U-Net achieves the lowest HD95 on all four datasets and the highest Dice on three. It requires 4.0× the latency and 3.8× the training memory of a parameter-matched concatenation baseline.

Critical strengths
  1. A well-justified problem is presented in the paper: do non-matching encoder scales offer information additional to the matching skip? Frozen-feature probes with paired-and-shuffled controls show that they do.

  2. CSF outperforms vanilla U-Net on all four benchmarks and achieves the lowest HD95 for each one. The authors have also acknowledged the limits of the theory and the computational expense.

Critical weaknesses
  1. The novelty is limited, and the connection between the theory and the method is weak. Eq. (4) is very similar to a ConvGRU over scales and has most of the components already present in [A-C]. The Kalman analysis does little to distinguish the method; the gate does not receive the reliability Rs,pR_{s,p}, while retention and damping have no counterpart in the theory.
  2. Evaluation includes U-Net variants from 2015-2022 and FuseUNet (2025), but excludes the current state-of-the-art segmentation models, e.g., [D-F].
  3. The baselines may be under-tuned. Reproduction of UCTransNet's results is lower than the published results for GLaS (87.70 vs. 90.18) and MoNuSeg (73.53 vs. 79.08). TransUNet has also been trained without pre-training.
  4. CSF is substantially more computationally expensive than the baselines, while several accuracy gains are within the reported variation. An equal-cost comparison is required to determine the advantage of using it as a drop-in replacement.

[A] UNET 3+: A FULL-SCALE CONNECTED UNET FOR MEDICAL IMAGE SEGMENTATION

[B] Bi-Directional ConvLSTM U-Net with Densley Connected Convolutions

[C] FuseUNet: A Multi-Scale Feature Fusion Method for U-like Networks

[D] Scalefusionnet: transformer-guided multi-scale feature fusion for skin lesion segmentation.

[E] When Mamba Meets xLSTM: An Efficient and Precise Method with the xLSTM-VMUNet Model for Skin lesion Segmentation

[F] VM-UNet: Vision Mamba UNet for Medical Image Segmentation

Questions for authors
  1. Could you compare CSF with a ConvGRU/ConvLSTM scan over scales and with non-recurrent adaptive scale weighting? These comparisons would help clarify what the Kalman-inspired structure adds beyond gated recurrence.
  2. Gate does not directly obtain pp or Rs,pR_{s,p}. Does γs\gamma_s decrease with the addition of noise at scale ss, as expected due to reliability-driven gain?
  3. Explain the difference between the reproduced and published UCTransNet results, and report the results of TransUNet with pre-training.
  4. Could you add [D-F] and an equal-cost baseline? Is the 4×4\times latency at S=4S=4 a characteristic of the method, or is it due to the implementation?
  5. Where is the "decoder-stability analysis" referred to in the Reproducibility Statement? Please specify the data splits, model selection, metric calculations, and ablation implementations as well.
Reviewer_sgKMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address the medical image segmentation challenge, the authors propose CSF UNet, a method featuring cross-scale selective fusion. Specifically, they model cross-scale information fusion as a sequential inference process, replacing standard feature concatenation in same-scale skip connections. The proposed approach is evaluated on four 2D medical image datasets, SegPC, ISIC, GLaS, and MoNuSeg, and the experimental results validate its effectiveness.

Critical strengths

Overall, the method in this article is introduced in detail, and the experiments are comprehensive and demonstrate a higher quantitative score than compared methods. Furthermore, the authors conducted experiments using three random seeds and reported the average results, which is commendable.

Critical weaknesses

U-Net is a well-established baseline in medical image segmentation, and numerous variants have emerged in recent years. Methods such as FuseUNet, UCTransNet, and DSC already address skip connections or multi-scale fusion. In this context, I find the proposed method’s novelty limited. My overall concern is that the performance gains may primarily stem from additional parameters and computation rather than multi-scale information. More targeted ablation studies to distinguish these factors would substantially strengthen the paper.

Questions for authors
  1. The authors list the proof of a squared-loss risk-gap equality as a main contribution. Why is this proof necessary, and what role does it play in medical image segmentation? While theoretical support is valuable, I suggest placing greater emphasis on method design and empirical evaluation, as clinical applications ultimately require fast and accurate models.

  2. Taking decoder stage D2 as an example, how do the authors distinguish gains from the cross-scale information provided by F1, F3, and F4 from gains due to additional feature processing and computation? Could the authors include a control that uses only F2, with additional processing layers or repeated updates to match the full CSF module as closely as possible in parameter count, computational cost, and processing steps? This would help isolate the contribution of the additional scales.

  3. The baseline architecture underlying CSF U-Net is not clearly specified. Is it the standard U-Net? Please clarify this explicitly.

  4. Please include comparisons with additional established baselines and recent state-of-the-art methods published in leading journals or conferences to better demonstrate the advantages of the proposed module. Relevant candidates include Mamba-based methods, given the proposed method’s use of sequential modeling, and methods that specifically improve skip connections. Evaluation on 3D segmentation tasks would further strengthen the evidence.

  5. Why does Table 2 compare the computational complexity of CSF U-Net with only a subset of the competing methods? CSF U-Net has substantially more parameters and higher GPU memory consumption than U-Net. Since model size can affect performance, comparisons between models with substantially different parameter counts do not establish that the gains arise from architectural design. Please provide comparisons under similar parameter budgets or further discuss the trade-off between model size and performance.

Reviewer_5nKTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CSF U-Net, a Kalman-inspired framework for cross-scale feature fusion in U-Nets. Instead of using only the matching encoder feature at each decoder stage, CSF aligns features from multiple scales and integrates them through a recurrent prediction–innovation–correction process. The method is motivated by an estimation-theoretic analysis and uses adaptive gating to control how information from different scales updates the hidden state. Experiments on four medical image segmentation datasets show improvements over vanilla U-Net and competitive performance against several existing variants. However, I have concerns about the connection between the theoretical motivation and practical implementation, the complexity of the proposed design, and its computational efficiency.

Critical strengths
  1. Clear motivation. The paper identifies a meaningful limitation of standard U-Net skip connections and explores how information from different encoder scales can be better utilized. The estimation-based perspective provides an interesting way to motivate the proposed fusion mechanism.

  2. Useful experimental analysis. The authors evaluate CSF on four medical image segmentation datasets and compare it with several U-Net variants. The frozen-feature probes and component ablations are also useful, as they help examine the value of non-matching scales and the contribution of individual components. The paper also reports computational costs rather than focusing only on segmentation accuracy.

Critical weaknesses
  1. Presentation of Figure 1. Figure 1 lacks the visual polish expected of an ICLR submission. The layout is too spread out, with excessive whitespace and small fonts. I suggest redesigning the figure with a more compact layout, larger fonts, and better visual organization.

  2. Gap between theory and implementation. The theoretical motivation relies on observation reliability and the Kalman gain, but the actual CSF module uses a learned channelwise gate without explicitly modeling uncertainty. Although the ablations show that adaptive gating improves performance, they do not establish whether the learned gate actually reflects the reliability of different scales. Could the authors provide further analysis to support this connection? For example, how does the gate respond when individual scale features are corrupted or become less reliable?

  3. Questionable sequential modeling across scales. The proposed method treats encoder features at different resolutions as sequential observations. However, these features do not have a natural temporal or causal ordering. While the authors motivate this design through a scale-axis estimation model, the practical justification for imposing such a sequential structure remains unclear. The current ablations compare coarse-to-fine, fine-to-coarse, and bidirectional scanning, but do not explore other scale permutations. How sensitive is CSF to the ordering of scales? Would alternative orderings provide similar performance?

  4. High computational overhead. Table 2 shows that CSF requires substantially more computation than the parameter-matched All-scale Concatenation baseline. Its inference latency increases from 4.4 ms to 17.7 ms, while training memory increases from 7.4 GB to 28.0 GB. Is such a large computational cost justified by the reported performance improvements? Could the authors explore a more efficient implementation of the proposed fusion module?

  5. Unclear advantage over simpler adaptive fusion. The proposed CSF combines prediction, residual subtraction, adaptive gating, and recurrent updates. While the component ablations are useful, it remains unclear whether this relatively complex design is necessary. Could the authors compare CSF with a simpler, parameter-matched adaptive cross-scale fusion module, such as gated fusion or cross-scale attention, under the same backbone and training settings? This would help clarify what the proposed Kalman-inspired design actually contributes.

Questions for authors

Please refer to the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.