acceptodds
Under review as a conference paper at ICLR 2027

Channel-Spatial Gated and Contour Aligned Latent Diffusion for Medical Image Segmentation

Abstract

Diffusion models have recently been employed for medical image segmentation due to their strong generative priors and ability to model rational anatomical structure consistent with real distribution. However, existing latent diffusion segmentation methods are largely adapted from natural image synthesis and remain constrained by parameter-heavy U-Net architectures, static skip connections, and insufficient contour-aware supervision. To align the latent diffusion paradigm with medical image segmentation tasks, we propose , a channel-spatial ated and ontour Aligned atent iffusion egmentation framework, with three bottom-up improvements at the architecture, feature, and loss levels. Our method redesigns the diffusion backbone with a compressed bottleneck to reduce parameter count and computational overhead without compromising segmentation accuracy. We design a channel-spatial disentangled gated skip connection to enable adaptive, context- and timestep-aware feature fusion during the denoising process. In addition, a contour alignment optimization loss is derived to enhance boundary localization and improve anatomical plausibility. Extensive experiments on multiple benchmarks demonstrate that GCLDS achieves superior segmentation accuracy and boundary delineation while maintaining efficient computation consumption.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical Image Segmentation, Latent Diffusion Models, Gated Skip Connection, Contour Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_STuzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GCLDS, a latent diffusion framework for medical image segmentation. The method combines three changes: channel compression in the U-Net bottleneck, a channel-spatial disentangled gated skip connection conditioned on the diffusion timestep and encoder-decoder features, and a latent contour alignment loss based on Sobel gradients. Experiments on BTCV, ISIC2017, ISIC2018, and REFUGE2 report improvements in region-overlap and boundary metrics over the compared methods. The paper also reports reduced trainable parameters, training memory, and inference time relative to several diffusion-based baselines. The current framework is limited to binary segmentation.

Critical strengths
  1. The paper targets three practical limitations of latent diffusion segmentation: backbone cost, static skip-feature fusion, and weak boundary supervision. The proposed components are simple to implement and are evaluated through individual and combined ablations.

  2. The evaluation includes CT, dermoscopic, and retinal image datasets, with both region-based metrics and boundary-sensitive metrics. The reported results show improvements over the compared diffusion baselines, and the efficiency comparison suggests that the compressed bottleneck can reduce model cost without an obvious loss of segmentation accuracy.

Critical weaknesses
  1. The main technical components are relatively incremental. Bottleneck compression, gated skip connections, and gradient-based contour losses are established design ideas, and the paper combines them within a latent diffusion segmentation pipeline. The ablations support the usefulness of the components, but do not fully establish a methodological advance beyond this combination. The authors should better distinguish the proposed design from existing adaptive skip connections and boundary-aware segmentation objectives.

  2. The contour loss is computed on VAE-encoded mask latents rather than on the decoded segmentation boundaries. The paper shows qualitative latent-gradient maps, but it remains unclear how closely these gradients correspond to clinically relevant contours in pixel space. In addition, all tasks are formulated as binary segmentation, including BTCV, where the 13 foreground organs are merged into one class. This limits the evidence for the method’s usefulness in multi-class medical segmentation, a setting that is important for many clinical applications. The authors should clarify the relationship between latent contour alignment and actual boundary accuracy, and narrow the scope of their claims or provide multi-class validation.

Questions for authors
  1. How strongly do the Sobel gradients of the VAE latent masks correspond to boundaries in the decoded pixel-space masks? Can the authors quantify this relationship or compare the proposed loss with a contour loss applied in pixel space?

  2. Since the BTCV organs are merged into a single foreground class and all benchmarks are treated as binary segmentation, what evidence supports extending the method to multi-class segmentation? Can the authors clarify the intended scope of the method?

  3. What is the specific methodological novelty of the proposed channel-spatial gating and contour alignment loss compared with existing adaptive skip-connection and boundary-aware segmentation methods?

1 comment

Novelty and Differences from Existing Methods. We thank the reviewer for the constructive and insightful comments. We agree that the individual components of our method are related to established techniques. Our contribution lies in adapting and integrating these components for pre-trained latent diffusion medical image segmentation. Specifically, unlike existing adaptive skip-connection methods such as Attention U-Net, CBAM, and FreeU, our channel-spatial gating explicitly separates the sources of channel and spatial modulation. The channel gate is conditioned only on the diffusion timestep embedding, while the spatial gate is derived from paired encoder-decoder features. This design provides separate control over channel responses and spatial feature selection within the latent diffusion denoising framework. For contour alignment, unlike conventional boundary losses that operate on full-resolution masks, our loss applies gradient consistency directly to VAE-encoded mask latents. This provides a memory-efficient alternative to pixel-space contour supervision. We will expanded the related-work section to clarify these differences.

Effectiveness and Computational Efficiency of Latent-Space Contour Supervision. To investigate the effectiveness of latent-space contour supervision, we conducted controlled experiments comparing SDSeg, GCLDS with pixel-space contour loss, and GCLDS with latent-space contour loss. All experiments were performed using three independent random seeds under identical network architectures and training settings. As shown in Table R1, latent-space contour supervision achieves slightly better region-overlap performance and comparable or better boundary-distance metrics (HD95 and ASD) than pixel-space contour supervision across all three datasets. These results suggest that latent gradients provide useful structural information for improving segmentation boundaries, although they should not be interpreted as direct pixel-level boundary representations.

We also compared the computational costs of the two approaches. Although the VAE decoder weights are frozen, gradients must still propagate through the decoder when computing the pixel-space contour loss. This requires retaining full-resolution decoder activations during backpropagation, resulting in approximately 6,918 MB of additional training GPU memory consumption compared with latent-space contour supervision. Therefore, the proposed latent-space contour loss provides a more memory-efficient alternative while maintaining competitive segmentation and boundary performance.

Applicability to Multi-Class Segmentation We acknowledge that all experiments in the current study are formulated as binary segmentation, including BTCV, where the 13 organ labels are merged into a single foreground class. Therefore, the current experiments do not provide direct empirical evidence of the method's effectiveness in multi-class segmentation. Our framework is currently designed and validated for binary medical image segmentation, and extending the method to native multi-class segmentation requires addressing the fixed-channel output representation of the pre-trained mask VAE. Although sequential per-class inference is possible, it increases inference cost with the number of classes, while adapting the VAE to multi-channel mask representations requires additional training and computational resources. We have clarified these limitations and identified multi-class latent diffusion segmentation as an important direction for future work.

Table R1. Quantitative comparison of SDSeg, GCLDS with pixel-space contour loss, and GCLDS with latent-space contour loss on BTCV, ISIC2017, and ISIC2018. All results are averaged over three independent runs and reported as mean ± standard deviation. Pixel-space contour supervision requires an additional ~6,918 MB of training GPU memory compared with latent-space contour supervision.

DatasetModelDice ↑IoU ↑HD95 ↓ASD ↓
BTCVSDSeg92.10 ± 0.0685.42 ± 0.108.09 ± 0.271.90 ± 0.13
GCLDS + pixel-space contour92.68 ± 0.1186.44 ± 0.166.35 ± 0.291.68 ± 0.16
GCLDS + latent-space contour92.97 ± 0.0986.93 ± 0.185.93 ± 0.671.57 ± 0.21
ISIC2017SDSeg90.74 ± 0.1484.43 ± 0.2411.23 ± 0.264.23 ± 0.07
GCLDS + pixel-space contour91.36 ± 0.0885.27 ± 0.0910.57 ± 0.283.90 ± 0.12
GCLDS + latent-space contour91.95 ± 0.1486.15 ± 0.189.69 ± 0.443.55 ± 0.27
ISIC2018SDSeg87.86 ± 0.3880.23 ± 0.6818.57 ± 0.437.35 ± 0.38
GCLDS + pixel-space contour89.22 ± 0.0582.41 ± 0.3117.16 ± 0.436.26 ± 0.21
GCLDS + latent-space contour89.73 ± 0.0983.04 ± 0.1516.52 ± 0.186.07 ± 0.16
Reviewer_98FrOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows how a diffusion-based medical image segmentation model can be made both smaller and more accurate. The authors take an existing diffusion segmentation model and make a few changes: a smaller network core, learned gates that control what information passes between parts of the network and an extra training loss that targets object boundaries. Across medical segmentation benchmarks, the modified model is more accurate than a range of competing methods, including the model it builds on, while using less memory and fewer parameters. The authors attribute the gains to the gates adapting to how noisy the input is, the boundary loss sharpening edges and the original network being larger than needed and they recommend the changes as general improvements for diffusion-based segmentation.

Critical strengths
  1. Compared to SDSeg, the method improves Dice score on all four datasets while also being smaller, using less memory and running slightly faster (Tables 1, 4, 5). Changes that make a model both cheaper and more accurate are directly useful to anyone already using diffusion for segmentation.

  2. The gate is compared not only with removing it, but with three other gate designs (a gate driven only by image features, a gate mixing noise level and features, and a single combined gate). The finding that the middle layer can be cut to 75% of its width without losing accuracy, while 50% hurts, is a useful practical observation.

Critical weaknesses
  1. Each component is a known technique, and the paper's explanations of why they help do not fit how the model is actually run. Gated skip connections (Attention U-Net [1]), channel and spatial reweighting (SE [2], CBAM [3]), reweighting skip features inside diffusion networks (FreeU [4]) and boundary losses [5] are all prior work that the related-work section does not discuss, so what is new beyond the combination is unclear. The explanations also break down on close inspection:

    • The noise-level part of the gate cannot adapt at test time. Inference always uses the same single noise level, so this part of the gate outputs the same values for every test image and acts as a fixed per-channel scale. Its benefit therefore cannot come from adapting to the noise level, which is the stated motivation.

    • The edge loss does not focus on boundaries. Because the edge filter is linear, the loss is the edge-filtered version of the same prediction error the main loss already penalizes, applied equally at every position. It also operates on a 32 x 32 compressed grid where each cell covers an 8 x 8 pixel patch, which is too coarse to support the claim of fine boundary refinement. The only evidence (Figure 2) shows a trained model's edge maps but no model trained without the loss for comparison.

    • The narrower middle layer does not keep the pretrained knowledge the paper says it keeps. The resized layers are trained from random initialization (Appendix B.1.2), and the channel-importance analysis in Figure 4 is not used to choose which pretrained channels to retain.

  2. The experiments do not show that the gains are reliable or that the components add to each other.

    • Each setting appears to be trained once, with no standard deviations. Several design arguments rest on gaps under 0.1 Dice points (0.09 between the proposed and combined gate in Table 7; 0.05 between channel-only and spatial-only gating in Table 3), which a single re-run with a different random seed could reverse. Some results also contradict each other: a gate with the same spatial part plus an extra channel part scores lower (91.02, Table 7) than the spatial part alone (91.23, Table 3). The significance test in Appendix B.2.5 compares test images for one trained model per method, so it does not measure variation between training runs.

    • The appendix reports different numbers for the same comparison. On ISIC2017, Table 1 gives Dice of 90.77 (SDSeg [6]) and 92.11 (proposed method), while Table 8 gives 90.86 and 91.88. On ISIC2018 the gain is 1.68 points in Table 1 and 2.25 in Table 8.

    • The benchmarks are easier versions of the standard tasks. BTCV is normally a 13-organ, 3D benchmark; here all organs are merged into one class and segmented one 2D slice at a time, and REFUGE2 is reduced to the optic cup only. Results on these versions say little about per-organ accuracy, and limit the state-of-the-art claim in Section 1.

    • No validation set is described for any dataset, and the middle-layer width was chosen using BTCV test results (Table 6), so hyperparameters and checkpoints may have been selected on test data.

References

[1] O. Oktay et al., "Attention U-Net: Learning where to look for the pancreas," in Proc. Med. Imag. Deep Learn. (MIDL), 2018, arXiv:1804.03999.

[2] J. Hu, L. Shen, and G. Sun, "Squeeze-and-excitation networks," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2018, pp. 7132-7141.

[3] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, "CBAM: Convolutional block attention module," in Proc. Eur. Conf. Comput. Vis. (ECCV), 2018, pp. 3-19.

[4] C. Si, Z. Huang, Y. Jiang, and Z. Liu, "FreeU: Free lunch in diffusion U-Net," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024, pp. 4733-4743.

[5] H. Kervadec, J. Bouchtiba, C. Desrosiers, E. Granger, J. Dolz, and I. Ben Ayed, "Boundary loss for highly unbalanced segmentation," in Proc. Med. Imag. Deep Learn. (MIDL), PMLR, vol. 102, 2019, pp. 285-296.

[6] T. Lin, Z. Chen, Z. Yan, W. Yu, and F. Zheng, "Stable diffusion segmentation for biomedical images with single-step reverse process," in Proc. Med. Image Comput. Comput. Assist. Interv. (MICCAI), 2024, doi: 10.1007/978-3-031-72111-3_62.

Minor issues:

  • Typos: natrual (sec 3.2) and others.
  • HiDiff and MedSegLatDiff are discussed in Section 2 but not compared against.
Questions for authors
  1. Why do Table 1 and Table 8 report different numbers for the same methods and datasets? Which are the main results, and are metrics averaged per image, per volume, or over the whole test set?

  2. Are all results from one training run? Please report mean and standard deviation over at least three random seeds for SDSeg and your method in Table 1, and for Tables 2, 3 and 7.

  3. Since the noise-level gate is constant at test time, how does it compare with a simple learned per-channel scale?

  4. The parameter count drops by about 104M (329M to 225M, Table 4), which seems large for resizing only the middle layer and its neighbours. Please give a per-module breakdown, the parameters added by the gates, and the share of the network trained from scratch. Did you try keeping the pretrained weights of the channels Figure 4 ranks most important?

  5. The edge loss has no weight in Eq. 10. How sensitive are results to its weight, and does simply increasing the weight of the main loss give similar gains?

  6. For BTCV, are metrics computed per slice or per volume, and how are slices with no organ handled for the boundary-distance metrics? Was nnU-Net run in 2D or 3D? Were all baselines retrained with the same training budget and augmentation, or were some numbers copied from earlier papers?

  7. How were checkpoints and hyperparameters chosen, given that no validation set is described?

  8. Is the image encoder in Section 3.1 the frozen autoencoder (as in SDSeg) or a separate trained network, and is it counted in the trainable parameters in Table 4?

3 comments

We sincerely thank the reviewer for the thorough and constructive comments. We have carefully examined the concerns regarding the novelty and interpretation of our method, experimental reliability, numerical consistency, and evaluation protocol. In response, we have conducted additional experiments, corrected the experimental protocol, and will revise the manuscript. Our responses are organized below.

Novelty and Design of the Proposed Components. We agree that gated skip connections, channel and spatial feature reweighting, and boundary-related losses have been studied in prior work. We will therefore expande the related-work section to discuss Attention U-Net [1], SE-Net [2], CBAM [3], FreeU [4], and existing boundary-loss methods [5]. Our method is designed to adapt channel modulation, spatial feature selection, and contour regularization to the latent diffusion segmentation framework. Specifically, the channel gate generates channel-wise weights from the diffusion timestep embedding, whereas the spatial gate derives spatial weights from paired encoder-decoder features. This decoupled design separates channel modulation from spatial feature selection. In addition, instead of applying conventional boundary losses to full-resolution masks, we introduce gradient consistency directly in the VAE latent space. This design integrates contour-related structural regularization into latent diffusion while avoiding the additional memory overhead associated with pixel-space contour supervision. Together, these designs provide a framework-specific adaptation of existing techniques rather than simply combining independent modules.

Effectiveness of the Timestep-Conditioned Channel Gate. Regarding the channel gate, we agree that under single-step inference (t=T−1)(t=T-1), it produces a fixed channel-wise scaling vector for each layer. To examine whether its parameterization provides additional benefits, we conducted the controlled comparison suggested by the reviewer. Specifically, we replaced the timestep-conditioned channel gate with a timestep-independent learnable per-channel scale, implemented as sigmoid(s)\mathrm{sigmoid}(s), with one learnable parameter per channel and an initialization matching the initial channel scaling of the original gate. The spatial gate, bottleneck compression, and contour loss were kept unchanged. As shown in Table R1, these results suggest that generating channel weights from the timestep embedding provides a more effective parameterization than directly learning independent channel-wise scales in our framework. We emphasize that this advantage reflects the benefit of the timestep-conditioned parameterization learned during training. Due to limited computational resources and time, the learned per-channel scale baseline was evaluated using only one run. Therefore, we report its results without standard deviations and interpret this comparison as preliminary evidence.

Table R1. Comparison of channel modulation strategies under single-step inference.

DatasetChannel modulationDice ↑IoU ↑HD95 ↓ASD ↓
ISIC2017None (SDSeg)90.74 ± 0.1484.43 ± 0.2411.23 ± 0.264.23 ± 0.07
Learned scale91.4185.4710.343.84
Timestep-conditioned (GCLDS)91.95 ± 0.1486.15 ± 0.189.69 ± 0.443.55 ± 0.27
ISIC2018None (SDSeg)87.86 ± 0.3880.23 ± 0.6818.57 ± 0.437.35 ± 0.38
Learned scale88.7781.4517.626.98
Timestep-conditioned (GCLDS)89.73 ± 0.0983.04 ± 0.1516.52 ± 0.186.07 ± 0.16

Analysis of Latent-Space Contour Supervision. We also acknowledge the limitation of interpreting the latent contour loss as explicit boundary supervision. Since the Sobel operator is linear, ∥S(z0)−S(z^0)∥1=∥S(z0−z^0)∥1\|S(z_0) - S(\hat{z}_0)\|_1=\|S(z_0 - \hat{z}_0)\|_1, the difference between the filtered latent representations is equivalent to filtering their prediction error. Thus, the loss acts as a gradient-domain regularizer rather than directly enforcing pixel-level boundary alignment. We will revise the corresponding explanation to avoid overstating its ability to refine fine pixel-level boundaries. To further investigate the effectiveness of our contour‑supervision objective, we add controlled ablations comparing latent‑space contour supervision, pixel‑space contour supervision, and the SDSeg baseline. For the pixel‑space variant, gradients must flow backward through the VAE decoder even when decoder weights are frozen. This requires retaining full‑resolution decoder activations during backpropagation, resulting in an additional ~6,918 MB of training GPU memory consumption. As shown in Table R2, the latent‑space contour loss achieves slightly better region‑overlap performance, while delivering comparable or better boundary‑distance metrics, with substantially lower memory overhead. These results support adopting latent‑space contour supervision as a more memory‑efficient alternative that maintains competitive segmentation accuracy.

Experimental reliability and numerical consistency. The differences between Tables 1 and 8 in the original submission were caused by different random seeds. Table 1 reported results from one best-performing seed, whereas Table 8 used another seed for the per-sample Wilcoxon signed-rank test. We acknowledge that presenting results from different runs in these tables without sufficient clarification could cause confusion. More importantly, the original experiments did not adequately assess run‑to‑run variability. Given limited computational resources, we repeated experiments with three independent random seeds for SDSeg and GCLDS for the main comparisons in Table R2 and the ablation studies in Table R3. We clarify that the original Wilcoxon test quantifies per‑sample differences within individual trained models, rather than capturing variability across separate training runs.

Comparison between Table 3 and Table 7. We thank the reviewer for pointing this out; the two results are not directly comparable. In Table 7, the static encoder-decoder gate computes its channel gate from the pooled encoder and decoder features heh^e and hdh^d (Eq. 11), whereas the spatial-only variant in Table 3 applies no channel modulation at all. Therefore, the lower Dice score of the Static Encoder-Decoder Gate (91.02) compared with that of the Spatial-only variant (91.23) does not indicate an inconsistency between the two tables. Rather, this observation suggests that channel signals derived from pooled feature content may not provide effective guidance and could potentially introduce undesirable modulation. This motivates our design choice of conditioning the channel gate on the timestep embedding, allowing the encoder features to be adaptively modulated according to the current noise level.

Parameter analysis and Datasets. We acknowledge that the original description of the bottleneck modification was incomplete. The approximately 104M parameter reduction, from 329M to 225M, results from two modifications: (i) removing two residual blocks from the middle bottleneck block and (ii) reducing the deepest-layer channel width from 768 to 576. We will revise Section 3.2 to describe both changes. For metric computation, all BTCV metrics are computed at volume level. Our GCLDS and 2D nnU-Net run slice-wise inference, then stack predictions into 3D volumes. Empty slices are skipped for HD95 and ASD. We use this unified evaluation pipeline for all 2D methods. This evaluation protocol follows the standard practice in SDSeg. All baseline methods in the main comparisons were locally retrained using the same dataset splits and augmentation pipeline following SDSeg. No results copied directly from previous papers were used in the main comparison tables. For all experiments, we strictly follow the training and test splits adopted by SDSeg and evaluate the final checkpoint saved at the end of training on the test set.

Image encoder and implementation details. The image encoder (τθ)(\tau_\theta) in Section 3.1 is a separate trainable network. It shares the architecture of the VAE encoder (E)(\mathcal{E}) and is initialized with its pretrained weights. During training, (τθ)(\tau_\theta) is trainable, whereas the mask VAE remains frozen. The parameters of (τθ)(\tau_\theta) are included in the trainable parameter count reported in Table 4.

Table R2. Quantitative comparison of SDSeg, GCLDS with pixel-space contour loss, and GCLDS with latent-space contour loss on BTCV, ISIC2017, and ISIC2018. All results are averaged over three independent runs and reported as mean ± standard deviation. Pixel-space contour supervision requires an additional ~6,918 MB of training GPU memory compared with latent-space contour supervision.

DatasetModelDice ↑IoU ↑HD95 ↓ASD ↓
BTCVSDSeg92.10 ± 0.0685.42 ± 0.108.09 ± 0.271.90 ± 0.13
GCLDS + pixel-space contour92.68 ± 0.1186.44 ± 0.166.35 ± 0.291.68 ± 0.16
GCLDS + latent-space contour92.97 ± 0.0986.93 ± 0.185.93 ± 0.671.57 ± 0.21
ISIC2017SDSeg90.74 ± 0.1484.43 ± 0.2411.23 ± 0.264.23 ± 0.07
GCLDS + pixel-space contour91.36 ± 0.0885.27 ± 0.0910.57 ± 0.283.90 ± 0.12
GCLDS + latent-space contour91.95 ± 0.1486.15 ± 0.189.69 ± 0.443.55 ± 0.27
ISIC2018SDSeg87.86 ± 0.3880.23 ± 0.6818.57 ± 0.437.35 ± 0.38
GCLDS + pixel-space contour89.22 ± 0.0582.41 ± 0.3117.16 ± 0.436.26 ± 0.21
GCLDS + latent-space contour89.73 ± 0.0983.04 ± 0.1516.52 ± 0.186.07 ± 0.16

Table R3. Ablation of channel and spatial gating on ISIC2017 and ISIC2018. All results are reported as mean ± standard deviation over three independent runs.

DatasetSettingDice ↑IoU ↑HD95 ↓
ISIC2017Baseline (No Gate)90.74±0.1490.74\pm0.1484.43±0.2484.43\pm0.2411.23±0.2611.23\pm0.26
Channel-only Gate91.09±0.1291.09\pm0.1284.98±0.1784.98\pm0.1711.26±0.8211.26\pm0.82
Spatial-only Gate91.26±0.0791.26\pm0.0785.23±0.0485.23\pm0.0410.79±0.1110.79\pm0.11
Channel-Spatial Gate (Ours)91.52±0.0591.52\pm0.0585.48±0.0685.48\pm0.0610.49±0.0510.49\pm0.05
ISIC2018Baseline (No Gate)87.86±0.3887.86\pm0.3880.23±0.6880.23\pm0.6818.57±0.4318.57\pm0.43
Channel-only Gate88.44±0.1088.44\pm0.1080.99±0.1680.99\pm0.1617.98±0.6217.98\pm0.62
Spatial-only Gate88.58±0.0588.58\pm0.0581.05±0.2481.05\pm0.2417.80±0.2517.80\pm0.25
Channel-Spatial Gate (Ours)89.20±0.0489.20\pm0.0482.24±0.1482.24\pm0.1417.40±0.2317.40\pm0.23

[1] O. Oktay et al., "Attention U-Net: Learning where to look for the pancreas," in Proc. Med. Imag. Deep Learn. (MIDL), 2018, arXiv:1804.03999.

[2] J. Hu, L. Shen, and G. Sun, "Squeeze-and-excitation networks," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2018, pp. 7132-7141.

[3] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, "CBAM: Convolutional block attention module," in Proc. Eur. Conf. Comput. Vis. (ECCV), 2018, pp. 3-19.

[4] C. Si, Z. Huang, Y. Jiang, and Z. Liu, "FreeU: Free lunch in diffusion U-Net," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024, pp. 4733-4743.

[5] H. Kervadec, J. Bouchtiba, C. Desrosiers, E. Granger, J. Dolz, and I. Ben Ayed, "Boundary loss for highly unbalanced segmentation," in Proc. Med. Imag. Deep Learn. (MIDL), PMLR, vol. 102, 2019, pp. 285-296.

Reviewer_y5mrOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GCLDS, a latent diffusion model (LDM) designed for medical image segmentation. The approach employs a "bottom-up" strategy involving three key modifications: (1) Architectural adjustment: the channel dimension at the bottleneck layer of the LDM’s U-Net is reduced from 768 to 576 (targeting only the deepest layer to preserve as much of the pre-trained weights as possible). (2) Feature enhancement: skip connections are augmented with gated mechanisms that decouple channel and spatial information—using timestep embeddings for channel gating, and a combination of encoder and decoder features for spatial gating. (3) Loss function: a contour-aware loss based on Sobel gradient alignment is introduced within the latent space. Experiments conducted on the BTCV, ISIC2017, ISIC2018, and REFUGE2 datasets demonstrate that GCLDS outperforms existing diffusion-based segmentation models (such as SDSeg and TSLDSeg) across metrics including Dice, IoU, HD95, and ASD. Furthermore, the model achieves a reduction in parameter count (225M vs. 329M), training VRAM consumption, and inference time. The study also includes a paired Wilcoxon signed-rank test for statistical significance and a comprehensive ablation analysis.

Critical strengths
  1. The performance gains are robust—not merely the result of "gaming the leaderboard." The model consistently outperforms SDSeg across all metrics on four different datasets. Furthermore, the authors provide sample-wise paired significance tests (with p-values ​​as low as 1e-57 on the ISIC2018 dataset), a rigorous approach that goes well beyond the common practice of simply reporting mean values. Most importantly, the model achieves both higher accuracy and greater efficiency—reducing the parameter count by 30% and accelerating inference—a rare combination in the field of diffusion-based segmentation, and arguably the paper’s strongest contribution.

  2. The ablation studies are highly persuasive, particularly those presented in Table 7. Rather than stopping at the simple claim that "using a gating mechanism is better than not using one," the authors evaluate three alternative designs—static gating, mixed-signal gating, and coupled unified gating. By comparing these, they effectively justify their design choice: decoupling the gating mechanism so that the temporal dimension focuses on channels, while the spatial dimension focuses on local regions. Additionally, the performance curve—showing that the model maintains accuracy when the bottleneck is compressed to 75% of its channels, but collapses at 50%—along with the analysis of channel importance distribution, provides solid evidence for the conclusion that "moderate compression" is optimal. The overall logic is coherent and self-contained.

Critical weaknesses
  1. A critical comparison regarding the contour loss is missing. The paper repeatedly asserts that "calculating contours in the latent space is more efficient than in the pixel space," yet it fails to provide a controlled comparison against pixel-space contour/boundary losses, nor does it quantify the actual difference in computational overhead. This claim of "efficiency" is merely inferred rather than empirically measured—ironically, it serves as the core selling point for the paper’s contributions at the loss function level. Furthermore, while the authors claim that their generative prior yields "structurally coherent and anatomically plausible" results, the experiments lack metrics for connectivity or topological consistency; only HD95 and ASD are reported, which hardly substantiate such a broad claim.

  2. The performance gains are marginal, and the proposed innovations resemble an "engineering-style" combination of existing techniques. On the ISIC2017 dataset, the improvement in Dice score over the SDSeg baseline is barely over one percentage point—and this baseline itself was modified by the authors. Each of the three components—bottleneck pruning, sigmoid gating on skip connections, and an L1 loss based on Sobel gradients—represents a standard technique; moving the boundary loss into the latent space is a clever adaptation, but hardly a novel contribution. Additionally, the paper lacks results averaged over multiple random seeds (with standard deviations); given that the two compressed bottleneck layers are randomly initialized, the stability of the results remains unaddressed. On the BTCV dataset, the 13 organ classes are collapsed into a binary foreground/background segmentation task. While this follows the setup used in SDSeg, it undermines the authors' claim that their method is superior for multi-class abdominal organ segmentation.

Questions for authors
  1. Could you provide a direct comparison between the contour loss in latent space and the contour loss in pixel space? Specifically, what is the difference in computational overhead? This is the data point I am most interested in seeing.
  2. Could you report results across multiple random seeds? In particular, given the random initialization of the bottleneck compression layer, is the ~1-point improvement in Dice score stable?
  3. Is it possible to decouple the "diffusion paradigm" from the "gating/loss modifications" to evaluate them separately? For instance, could you use a deterministic segmenter (such as nnU-Net) combined with the same gated skip connections and contour losses to determine the true source of the performance gains?
2 comments

We thank the reviewer for the constructive and critical feedback, which has helped us improve the technical rigor and clarity of our manuscript. We have conducted additional experiments.

Contour Loss and Computational Efficiency. We agree that the original submission lacked a controlled comparison between latent-space and pixel-space contour losses. To address this concern, we conducted experiments using three independent random seeds under identical network architectures and training settings. Although the VAE decoder weights are frozen, gradients must still propagate through the decoder when computing the pixel-space contour loss. This requires retaining full-resolution decoder activations during backpropagation, resulting in an additional ~6,918 MB of training GPU memory consumption. As shown in Table R1, the latent-space contour loss achieves slightly better region-overlap performance, while providing comparable or better boundary-distance metrics, with substantially lower memory overhead. These results support the use of latent-space contour supervision as a more memory-efficient alternative.

Novelty and Contribution. We acknowledge that the individual components, including channel compression, sigmoid gating, and Sobel-based gradient matching, have been explored in previous studies. The contribution of GCLDS lies in their integration through design principles tailored to pretrained latent-diffusion segmentation. Specifically, channel compression is restricted to the U-Net bottleneck to reduce redundancy while preserving most pretrained encoder-decoder weights and skip-connection compatibility. The proposed gated skip mechanism separates the conditioning signals: timestep embeddings guide channel modulation, whereas encoder-decoder features determine spatial modulation. Our ablation results support this signal-disentanglement design. In addition, contour consistency is imposed directly in the VAE latent space, providing an efficient way to incorporate boundary information into the diffusion objective. We will revise the discussion to better distinguish these design contributions from the underlying operators and to provide a more balanced account of the performance improvements.

Robustness Across Random Seeds. We agree that single-run results cannot establish the stability of performance improvements, particularly when some differences are relatively small. We therefore repeated the experiments for the SDSeg baseline, GCLDS with pixel-space contour loss, and the full GCLDS with latent-space contour loss using three independent random seeds. The results are reported as mean ± standard deviation in Table R1. GCLDS consistently improves segmentation performance over SDSeg across the evaluated datasets, suggesting that the observed gains are not merely due to favorable initialization of the compressed bottleneck layers.

Table R1. Quantitative comparison of SDSeg, GCLDS with pixel-space contour loss, and GCLDS with latent-space contour loss on BTCV, ISIC2017, and ISIC2018. All results are averaged over three independent runs and reported as mean ± standard deviation. Pixel-space contour supervision requires an additional ~6,918 MB of training GPU memory compared with latent-space contour supervision.

DatasetModelDice ↑IoU ↑HD95 ↓ASD ↓
BTCVSDSeg92.10 ± 0.0685.42 ± 0.108.09 ± 0.271.90 ± 0.13
GCLDS + pixel-space contour92.68 ± 0.1186.44 ± 0.166.35 ± 0.291.68 ± 0.16
GCLDS + latent-space contour92.97 ± 0.0986.93 ± 0.185.93 ± 0.671.57 ± 0.21
ISIC2017SDSeg90.74 ± 0.1484.43 ± 0.2411.23 ± 0.264.23 ± 0.07
GCLDS + pixel-space contour91.36 ± 0.0885.27 ± 0.0910.57 ± 0.283.90 ± 0.12
GCLDS + latent-space contour91.95 ± 0.1486.15 ± 0.189.69 ± 0.443.55 ± 0.27
ISIC2018SDSeg87.86 ± 0.3880.23 ± 0.6818.57 ± 0.437.35 ± 0.38
GCLDS + pixel-space contour89.22 ± 0.0582.41 ± 0.3117.16 ± 0.436.26 ± 0.21
GCLDS + latent-space contour89.73 ± 0.0983.04 ± 0.1516.52 ± 0.186.07 ± 0.16

Scope of BTCV Evaluation. We fully agree that merging the 13 organ classes into a binary foreground-background task does not constitute genuine multi-class organ segmentation. Our BTCV experiments follow the binary segmentation protocol adopted by SDSeg and therefore cannot establish the effectiveness of GCLDS for 13-class segmentation. We have clarified this experimental setting and explicitly acknowledged native multi-class segmentation as a limitation of the current framework in supplementary material, partly due to the fixed-channel constraints inherited from the pretrained VAE.

Applicability to Deterministic Segmentation Networks. We appreciate the suggestion to evaluate the proposed components on deterministic architectures such as nnU-Net. However, directly transferring our modules to nnU-Net would not provide an equivalent comparison. The channel-spatial disentangled gating mechanism explicitly uses the diffusion timestep embedding to condition channel modulation, whereas deterministic segmentation networks do not have such a signal. Replacing it with a constant would alter the proposed design. Similarly, our contour loss operates on VAE latent representations, which are absent in nnU-Net. Applying a pixel-space Sobel loss to nnU-Net predictions would therefore evaluate a different objective. Instead, we conduct component-wise ablations within the latent-diffusion framework, progressively introducing bottleneck compression, gated skip connections, and latent-space contour alignment to assess their individual contributions.

Comments on the paper

1 comment

We sincerely thank all reviewers for their constructive feedback. Their comments helped us identify gaps in clarification and experimental validation. We have added new experiments and revised our method presentation accordingly. The major clarifications are summarized below.

Clarification of the methodological contributions and novelty.

We have revised the discussion of related work and clarified the distinction between our design and existing techniques, including Attention U-Net, SE, CBAM, and FreeU. Rather than claiming novelty in individual operators, we emphasize the framework-specific design of GCLDS for pretrained latent-diffusion segmentation. In particular, the channel gate is conditioned on diffusion timestep embeddings, while the spatial gate is derived from paired encoder-decoder features. This separates channel modulation from spatial feature selection. We also clarify the role of bottleneck compression in reducing model complexity while preserving the pretrained encoder-decoder structure and skip-connection compatibility.

To further validate the channel-gating design, we conducted an additional comparison with a timestep-independent learnable per-channel scaling strategy. The results provide preliminary evidence that timestep-conditioned channel modulation is more effective than directly learning independent channel-wise scales under our inference setting.

Additional experiments on contour supervision and computational efficiency.

To address concerns about the effectiveness of latent-space contour supervision, we conducted controlled experiments comparing SDSeg, GCLDS with pixel-space contour loss, and GCLDS with latent-space contour loss on three datasets. All three configurations were evaluated using three independent random seeds under identical network architectures and training settings.

The results show that latent-space contour supervision consistently achieves better region-overlap performance and comparable or better boundary-distance metrics than pixel-space contour supervision. More importantly, pixel-space contour supervision requires approximately 6,918 MB of additional training GPU memory because gradients must propagate through the frozen VAE decoder, whereas latent-space supervision avoids this overhead. These findings support the computational advantage of our design without sacrificing segmentation performance. We also refined our explanation of the contour objective. Specifically, we acknowledge that the latent-space Sobel loss acts as a gradient-domain regularizer rather than directly enforcing pixel-level boundary alignment.

Improved experimental reliability and additional ablation studies.

In response to concerns about single-run evaluation and the reliability of relatively small performance differences, we repeated the main comparisons and component-wise ablation experiments using three independent random seeds. The updated results are reported as mean ± standard deviation.

We also clarified the numerical differences between the original tables, which resulted from using different random seeds, and explained that the original Wilcoxon signed-rank test assessed per-sample differences within individual trained models rather than variability across independent training runs. Furthermore, we clarified the differences between the static encoder-decoder gate and the spatial-only gate, as well as the reasons why their results are not directly comparable.

Clarification of the applicability and limitations of the framework.

We acknowledge that the current experiments, including BTCV, follow a binary segmentation protocol and do not establish the effectiveness of GCLDS for native multi-class organ segmentation. We have explicitly clarified that the 13 BTCV organ labels are merged into a single foreground class and identified multi-class segmentation as an important direction for future work, particularly considering the fixed-channel representation of the pretrained mask VAE.

We also addressed the suggestion of evaluating the proposed components on deterministic segmentation architectures such as nnU-Net. Since our channel-spatial gating mechanism relies on diffusion timestep conditioning and our contour objective operates on VAE latent representations, directly transferring these components to deterministic networks would fundamentally change their formulation. We therefore provide controlled component-wise ablations within the intended latent-diffusion framework rather than presenting an inequivalent comparison.

We hope that these additional experiments and clarifications have strengthened the technical rigor of our manuscript. We have also explicitly acknowledged the current limitations of our framework. We will proofread the manuscript to fix remaining grammatical errors. We sincerely appreciate the reviewers' constructive feedback and hope that our revisions provide a clearer and more comprehensive account of our contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.