acceptodds
Withdrawn from ICLR 2027

From Prior to Precision: Unifying Global Structural Priors and Local Generative Refinement for Medical Image Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Medical image segmentation is acutely bottlenecked by the scarcity of pixel-level annotations, especially in the low-data regime. Vision foundation models(VFMs) such as DINO provide rich semantic and structural priors that can alleviate this dependency, yet cross-domain discrepancies erode their direct transfer to the fine-grained representations required by medical segmentation. In this work, we propose **P**rior-t**o**-**P**recision **Net**work (PoPNet), a unified framework that bridges generic foundation-model priors with task-specific segmentation precision through global structural regularization and local generative refinement. To capture global structures, PoPNet uses a spectral clustering regularization to anchor the task-specific latent manifold toward structural priors derived from VFM cross-layer attention affinities, encouraging the learned representation to preserve dominant structural patterns while reducing reliance on domain-specific feature details. To further improve local precision, PoPNet incorporates target-aware region-of-interest masked generation, which performs conditional reconstruction within target regions, compelling the network to capture fine-grained textures and subtle boundary details critical for accurate delineation. Extensive experiments across five supervised benchmarks and semi-supervised segmentation settings demonstrate that PoPNet achieves state-of-the-art performance with strong generalization across datasets and foundation-model priors.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
medical segmentation, spectral clustering regularization, target-aware masked generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QizFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PoPNet to bridge the gap between vision foundation model (VFM) representations and the task-specific features needed for medical image segmentation. It also introduces a reconstruction loss to support segmentation training. Although the VFM integration strategy seems a little boring, the reconstruction loss is somehow interesting.

Critical strengths
  1. Clear presentation. The paper is well organized, and the figures clearly illustrate the key components and workflow, making the method easy to follow.

  2. Broad experimental coverage. Although the choice of VFM (mainly using general VFM) limits the experiments to 2D medical image segmentation, the evaluation covers multiple datasets under both supervised and semi-supervised settings. The appendix also reports repeated runs with different random seeds, making the evaluation more complete.

Critical weaknesses
  1. Lack of 3D medical image segmentation setting.
  • Even if general VFMs are difficult to apply directly to 3D data, models pretrained on 3D medical images, such as VoCo, Hi-End-MAE, MambaMIM, and nnFoundation, could provide feature priors. The reconstruction loss also appears straightforward to extend to 3D. Given the importance of 3D segmentation in medical imaging, the lack of these experiments limits the evidence for PoPNet’s generality.
  1. Some ablations of key components.
  • The method introduces spectral clustering regularization to combine VFM representations with bottleneck features. However, how would performance change if spectral embedding prediction and feedback modulation were removed, leaving only gated fusion?
  • Is a VFM necessary? What happens if it is replaced with a plain ViT (or some hierarchical variants) trained from scratch? How do segmentation gains vary with VFM size, and do stronger VFM representations lead to better segmentation? Do these designs can extend to 3D medical image segmentation?
  • PoPNet appears to be a general framework whose designs could, in principle, transfer to U-shaped segmentation networks beyond U-Net. The authors should evaluate whether these designs yield consistent improvements across different network architectures.
Questions for authors

I hope the authors can address the concerns raised in the Weaknesses section above and provide further clarification on reproducibility:

  • The paper does not include a reproducibility statement, usually placed before the References section, or describe any plans to release the code.
Reviewer_fnhFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary
  • The paper proposes PoPNet, a medical image segmentation framework combining global structural priors with local generative refinement.
  • It uses frozen DINOv3 features and spectral clustering regularization to guide structural representation learning, together with ROI-guided masked image reconstruction to enhance local details.
  • The method is evaluated on multiple medical image segmentation benchmarks under fully supervised and semi-supervised settings, reporting improved segmentation performance and generalization.
Critical strengths
  • The authors conduct extensive experiments across multiple datasets and supervision settings to validate the effectiveness of the proposed method.
  • The paper incorporates spectral clustering regularization into medical image segmentation to leverage structural priors from pretrained vision models.
Critical weaknesses
  • Insufficient Controlled Ablations: Table 3 compares U-Net+DINO Concat with U-Net+SC to demonstrate the effectiveness of spectral clustering regularization. However, these configurations differ not only in the spectral loss but also in their feature integration and structural modulation mechanisms. Consequently, the observed improvements cannot be attributed solely to spectral regularization. More carefully controlled ablations are necessary to disentangle these contributions.
  • Unclear Necessity of Spectral Clustering: Although the proposed spectral regularization improves segmentation performance, the paper does not sufficiently establish why a spectral clustering objective is preferable to simpler approaches for transferring structural knowledge from DINOv3. The current comparison with feature concatenation is insufficient, as it does not evaluate alternative regularization objectives under comparable conditions.
  • Limited Novelty of ROI Masked Generation: The proposed ROI Masked Generation module largely follows conventional masked image reconstruction, with the primary modification being the restriction of reconstruction to predicted regions of interest. Although Table 4 demonstrates the benefits of ROI-based masking over global masking, the underlying reconstruction mechanism remains relatively standard.
Questions for authors
  • Can the authors separately evaluate the spectral regularization loss, gated DINO feature fusion, and spectral modulation while keeping the remaining architecture unchanged?
  • How does the proposed spectral objective compare with standard feature distillation (e.g., MSE-based feature alignment) and pairwise attention-affinity consistency regularization under the same backbone and training settings?
  • Are the improvements attributable to the spectral formulation itself, or simply to introducing additional features from DINOv3?
Reviewer_RQdiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PoPNet, a method for transforming the priors of general vision foundation models into task-specific representations for medical image segmentation. It is implemented through global structural regularization combined with local generative refinement and achieves competitive performance across various medical image datasets.

Critical strengths

1.The paper proposes an effective method for using knowledge from general vision models to regularize the representations of medical image segmentation models.

2.Addressing domain discrepancies with general models by focusing on the global and local representations of medical segmentation models is a highly targeted approach.

3.Extensive experiments validate the method’s segmentation performance.

Critical weaknesses

1.Limited innovation. It is common practice to transfer general vision models to medical image segmentation, and spectral clustering and generative methods are both widely used in segmentation. The paper lacks sufficient comparison with works addressing similar problems and employing similar methods. Additionally, it should be compared with work on medical foundation models.

2.The motivation is unclear. While the goal of improving global and local representations to mitigate domain differences is reasonably clear, and the formula effectively derives the method’s structural characteristics of high affinity and low redundancy, there is a lack of discussion regarding the direct connection between these characteristics and the practical clinical needs of medical image segmentation. Additionally, there is no substantive analysis of the generative model’s actual contribution to segmentation details.

3.The scope of the study and the conditions for method applicability are unclear. Overall, the paper proposes an effective method for transferring visual models to medical segmentation. However, it lacks theoretical and practical justification; this is not only due to insufficient descriptions in the first three chapters but also to the absence of an analysis of the experimental phenomena. It is necessary to analyze the practical problems addressed in the paper by considering the characteristics of multimodal medical images, and discuss the applicable imaging modalities and diseases, as well as the mechanisms underlying its effective representation.

4.The method description indicates that segmentation and generation share a common pipeline; it is difficult to understand how this is implemented, and a more detailed explanation is needed.

5.A sensitivity analysis of the weights of the loss function components is missing. This would be useful for understanding the effectiveness of injecting general visual model knowledge into medical segmentation models through spectral clustering regularization, as well as the complementary relationship between the generation and segmentation tasks.

6.In Figure 2(a), zθez_{\theta}^e and zqz_{q} no longer exhibit any discernible semantic components; under these circumstances, the actual contribution of SC to medical image segmentation is difficult to ascertain.

7.Visualizations are insufficient. There is a lack of visual comparisons with related methods.

Questions for authors

Please refer to the Critical Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.