acceptodds
Withdrawn from ICLR 2027

SHE-Net: Semantic-Guided Hierarchical Experts for Ultrasound Standard Plane Recognition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Ultrasound standard plane recognition is an essential yet challenging task in clinical screening and diagnosis, owing to substantial intra-class variations and inter-class similarities caused by imaging noise, operator dependency, and anatomical variability. Existing methods mainly rely on visual appearance modeling, which may struggle to capture subtle structural differences and often overlook explicit anatomical semantics relevant to standard plane recognition. To address these challenges, we propose SHE-Net, a Semantic-guided Hierarchical Experts Network for ultrasound standard plane recognition. We first construct multi-granularity textual descriptions for existing datasets, providing explicit anatomical semantics from coarse plane categories to fine-grained structural characteristics. Based on these descriptions, we develop a Hierarchical Vision-Language Alignment (HVLA) strategy to align visual representations at different hierarchical levels with corresponding textual semantics, introducing structured semantic supervision into visual representation learning. Furthermore, we propose a Semantic-Routed Anatomical Experts (SRAE) module, which uses semantic-aware routing to dynamically assign visual representations to specialized experts, enabling the learning of complementary anatomical patterns from local structures to global plane configurations. By jointly integrating hierarchical semantic alignment and adaptive expert specialization, SHE-Net enhances the discriminability and structural consistency of ultrasound standard plane representations. In addition, we construct and annotate a new kidney ultrasound standard plane dataset to evaluate the generalizability of the proposed framework. Extensive experiments on three ultrasound standard plane datasets demonstrate that SHE-Net consistently outperforms existing methods across multiple evaluation metrics, highlighting the effectiveness of hierarchical anatomical semantics and semantic-guided expert specialization for robust ultrasound standard plane recognition.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Ultrasound Standard Plane Recognition, Vision-Language Alignment, Anatomical Semantics, Multi-Expert Network

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FUm9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SHE-Net for ultrasound standard plane recognition. The method consists of two main components: (1) Hierarchical Vision-Language Alignment (HVLA), which aligns visual features at different network stages with local, relational, and plane-level anatomical text semantics; and (2) Semantic-Routed Anatomical Experts (SRAE), which uses semantic supervision to guide routing among local, relational, and recognition experts. Textual information is only required during training. The paper also introduces a new kidney ultrasound dataset, KU2026, containing 3,716 images. Experiments on FPUS23, LP2025, and KU2026 report consistent improvements over the selected baselines, with additional ablations and Grad-CAM visualizations.

Critical strengths
  1. The local/relational/plane-level semantic hierarchy is reasonably aligned with the clinical reasoning process, and using textual supervision only during training avoids additional inference-time requirements.

  2. The method is evaluated on fetal, liver, and kidney ultrasound datasets, and consistently improves over the reported baselines. The newly collected KU2026 dataset also broadens evaluation beyond commonly studied fetal ultrasound settings.

Critical weaknesses
  1. The paper does not clearly report the train/validation/test splits, the number of patients/examinations, or whether samples from the same patient or acquisition sequence are strictly separated across splits. This is consequential because image-level leakage could substantially inflate performance in ultrasound datasets.

  2. In particular, the related work cites SEMC, a recent mixture-of-experts contrastive method specifically developed for ultrasound standard plane recognition, but it is absent from the main quantitative comparison. Instead, several compared methods originate from more general long-tailed recognition, segmentation, few-shot learning, or other medical imaging tasks. This makes it difficult to determine whether the reported gains represent a meaningful advance over the most directly relevant prior work.

A related reproducibility issue is that Align(·) is introduced only abstractly as a “prototype-based visual-semantic alignment objective,” while the semantic scores sloc,srel,srecs_{\mathrm{loc}},s_{\mathrm{rel}},s_{\mathrm{rec}} used to supervise expert routing are also not explicitly defined.

Questions for authors
  1. What are the exact train/validation/test splitting protocols for FPUS23, LP2025, and KU2026? In particular, are the splits patient-/examination-/video-level, and can frames or images from the same subject appear in both training and test sets?

  2. Please provide the exact mathematical definition of Align(·) and of the semantic consistency scores sloc,srel,srecs_{\mathrm{loc}},s_{\mathrm{rel}},s_{\mathrm{rec}}. What text encoder, semantic prototypes, EMA momentum μ\mu, and routing temperature τr\tau_r are used?

  3. Why is SEMC (Cai et al., 2026), which is highly relevant as a mixture-of-experts method for ultrasound standard plane recognition, not included in the quantitative comparison? A comparison under the same data split and training protocol would materially affect my assessment.

  4. Are the reported results from a single run or averaged over multiple random seeds? Given that several improvements over the strongest baselines are below one percentage point, standard deviations or statistical significance would be helpful.

Reviewer_DdqPOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SHE-Net, a semantic-guided hierarchical expert framework for ultrasound standard plane recognition. Hierarchical Vision-Language Alignment (HVLA) aligns visual features from different backbone stages with class-specific textual descriptions at three semantic levels: local anatomy, relational anatomy, and plane recognition. Semantic-Routed Anatomical Experts (SRAE) employs convolutional, atrous spatial pyramid pooling, and self-attention branches. A visually conditioned gating network combines their outputs, while a semantic routing objective supervises the gate during training. The authors also introduce KU2026, a kidney ultrasound dataset containing 3,716 images classified into transverse, longitudinal, and non-standard planes. Experiments on FPUS23, LP2025, and KU2026 report accuracies of 95.98%, 84.68%, and 84.52%, respectively, outperforming the included baselines. Ablation studies investigate hierarchical alignment, coarse semantic alignment, routing supervision, and loss weighting.

Critical strengths

S1. Clinically meaningful problem and coherent design. Ultrasound standard plane recognition is important for automated examination assistance and standardized anatomical assessment. Combining local, relational, and plane-level descriptions provides an intuitive way to introduce anatomical knowledge into representation learning. The complementary expert architectures are reasonably motivated, and the absence of textual inputs at inference is practically attractive. S2. Evaluation across anatomical domains and an additional dataset. The experiments cover fetal, liver, and kidney ultrasound images, with reported improvements across the included benchmarks. KU2026 extends the evaluation to renal imaging. Component ablations, semantic-level ablations, parameter sensitivity analyses, and qualitative visualizations provide useful preliminary evidence of the framework's effectiveness.

Critical weaknesses

W1. Insufficient evidence for the incremental contribution and the source of the gains. SEMC, “Structure-Enhanced Mixture-of-Experts Contrastive Learning for Ultrasound Standard Plane Recognition” (Cai et al., AAAI 2026), already investigates hierarchical representations and expert-based learning for the same task. It is cited in the manuscript but absent from Table 2. SEMC reports an LP2025 F1-score of 79.32%, whereas SHE-Net reports 79.36%. These figures should not be directly compared without verifying identical protocols, but they make a controlled comparison particularly important. The ablations do not adequately separate meaningful anatomical information from additional class supervision, model capacity, and architectural complexity. The descriptions are class-specific and sampled using ground-truth labels, so removing semantic levels does not isolate their linguistic content. Controls using class-name-only or permuted descriptions, parameter-matched experts with standard visual gating but no semantic routing loss, and semantic alignment without experts would clarify the attribution. The reported accuracy gains of 0.16 percentage points on FPUS23 and 0.53 on KU2026, and the KU2026 F1 gain of 0.51 points, also lack repeated-run uncertainty estimates. These limitations weaken the evidence for the claimed methodological advance.

W2. Under-specified mechanisms and evaluation protocols limit assessment of the central claims. The function Align(v, t, y) in Equations 4 and 12 is undefined, and the computation of the semantic scores in Equation 8 and the EMA update scope in Equation 9 are unclear. If the EMA aggregates scores globally, the routing targets may reflect a dataset-level preference rather than sample-specific semantics. Equations 6 and 7 describe dense expert weighting, rather than sparse conditional activation. The paper also lacks direct evidence that the experts learn their intended anatomical specializations; alignment of globally pooled features does not itself establish localized concept grounding or learned spatial relations. The text encoder, class-specific versus dataset-specific coarse text pools, and the stated 512-dimensional expert representations versus subsequent pooling need clarification. Clinical data splits and patient- or examination-level independence are insufficiently documented, and KU2026 requires clearer acquisition and annotation information. These omissions do not establish leakage, but they limit confidence in evaluation validity. FPUS23 is a fetal phantom dataset, making the appendix's references to gestational-stage differences and fetal motion insufficiently justified. Separate supervised experiments on three datasets also do not alone establish out-of-domain clinical generalization. Together, these gaps limit reproducibility and support for the claimed mechanism.

Questions for authors

Q1. Relationship to SEMC and attribution of gains. How does SHE-Net differ substantively from SEMC, and can the authors provide an identical-protocol comparison? What are the results with the same experts and standard visual gating but no semantic routing loss, and with class-name-only or shuffled descriptions? Repeated-run statistics and parameter-matched controls would help determine whether the gains arise from anatomical semantics rather than capacity or training variation. Q2. Alignment and routing formulation. What is the exact definition of Align(v, t, y), and how are the three semantic consistency scores computed? Is the EMA maintained globally, per class, per mini-batch, or per sample? Please clarify the text encoder, coarse text pools, and feature dimensions needed to reproduce the method. Q3. Expert specialization. What evidence demonstrates that the experts capture their intended anatomical concepts rather than complementary generic visual features? How do the routing distributions vary across samples, and how does this support the claimed specialization? Q4. Dataset protocols. What are the exact splits and numbers of independent patients or examinations in LP2025 and KU2026? Are the clinical splits patient-independent? Please clarify KU2026 acquisition and annotation reliability and reconcile the phantom nature of FPUS23 with the variability claims in the appendix.

Reviewer_YGfDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SHE-Net, a semantic-guided hierarchical experts framework for ultrasound standard plane recognition. By introducing hierarchical vision-language alignment and semantic-aware expert routing, the proposed method incorporates anatomical semantics into visual representation learning and enables complementary modeling from local structures to global plane configurations. The hierarchical alignment strategy provides multi-level semantic supervision, while the expert routing mechanism adaptively promotes specialized feature learning for different anatomical patterns. In addition, the authors construct and annotate a new kidney ultrasound standard plane dataset. Experiments demonstrate improvements over existing methods.

Critical strengths
  • KU2026 dataset is valuable though this is dependent on release, also with many meta information currently missing.

  • Hierarchical formulation provides clues for distinguishing visually similar ultrasound planes

Critical weaknesses
  • Limited novelty, the method combines known components (CLIP-style alignment, MoE head)
  • Marginal performance improvement without any statistical significance tests and unclear about image or patient level partitioning.
  • Missing technical details. For example, how are the parameters are set? What is the shared backbone? What is Align() in (4)?
Questions for authors
  • Is the evaluation partitioning on patient or examination level?
  • Can you address the missing details in weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.