acceptodds
Under review as a conference paper at ICLR 2027

HCT-Net: Coordinated Heterogeneous Co-Training for Semi-Supervised 3D Medical Image Segmentation

Abstract

Heterogeneous learners can provide complementary training signals for semi-supervised segmentation, but their relative quality may change throughout training. We study this setting as a coordination problem involving three decisions: which learner provides supervision, when cross-branch learning is enabled, and how strongly each unlabeled sample contributes. We present HCT-Net, a heterogeneous Mamba–Transformer co-training framework in which smoothed labeled-batch performance serves as an online routing proxy. A relative-performance gate controls cross-branch activation, while training ramp-up and confidence coverage determine the strength of pseudo-label supervision. Prediction-level and feature-level transfer follow the same directed route with detached teacher targets. We evaluate HCT-Net on ACDC and LA under 10% and 15% labeled settings. At 15% labeling, HCT-Net achieves Dice scores of 83.01% and 85.00%, respectively, improving over symmetric cross-pseudo supervision using the same Mamba–Transformer pair by 1.09 and 2.20 percentage points. Across the evaluated backbone pairings, the largest observed CPS-to-HCT Dice improvements occur for the heterogeneous Mamba–Transformer pair on both datasets at 15% labeling. Epoch-wise diagnostics further characterize differences in branch diversity and error behavior across HCT configurations.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
semi-supervised medical image segmentation, heterogeneous co-training, dynamic teacher-student routing, reliability-aware learning, pseudo-labeling, 3D medical image segmentation, Mamba

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VPSuOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HCT-Net trains a SegMamba branch and a Swin UNETR branch for semi-supervised 3D segmentation. An exponential moving average of labeled-training-batch foreground Dice chooses the temporary teacher. A relative-performance gate activates cross-branch training, while a ramp-up schedule and the teacher's confidence coverage weight each unlabeled sample's pseudo-label loss. A detached feature-alignment objective follows the same direction. Both branches continue receiving supervised losses. At inference, a branch or a validation-selected probability fusion is used. Evaluation on ACDC and LA covers 10% and 15% labeling. At 15%, HCT-Net improves Dice over symmetric CPS with the same heterogeneous pair by 1.09 and 2.20 percentage points, respectively. Homogeneous controls and descriptive disagreement/error trajectories provide further analysis of the complete configuration.

Critical strengths

S1 The mathematical specification makes the training policy inspectable

Secs. 3.3-3.6 specify a score EMA rather than a parameter EMA, detached teacher targets, full-domain pseudo-label cross-entropy, and a separate feature-alignment loss. The text explicitly states that coverage weights a whole sample rather than filtering individual voxels. These details make it possible to identify what supervision is actually applied, and they are more useful than describing the method simply as reliability-aware co-training.

S2 The matched backbone control is more informative than an unmatched architecture comparison

Table 3 holds the Mamba-Transformer pair fixed and improves Dice from 0.8192 to 0.8301 on ACDC and from 0.8280 to 0.8500 on LA, with corresponding surface-metric improvements. The homogeneous controls also expose a negative result: HCT reduces ACDC Dice for the homogeneous Mamba pair from 0.8219 to 0.7871. Reporting that dependence is helpful and argues against treating the controller as universally beneficial. The parameter-count analysis further shows that the best heterogeneous result is not simply the result of having the most parameters. [Secs. 4.4-4.6; Table 3, p. 8.]

Critical weaknesses

W1 The reported comparison does not identify the benefit of coordination

Table 3 holds the backbone pair fixed, which is a useful control, but it changes the full training strategy. HCT replaces two-way supervision with one-way supervision, changes its effective weight, and adds directed feature matching. The gain can therefore reflect reduced pseudo-label pressure or feature regularization without establishing that coordinating the three decisions is beneficial. This matters because the paper itself identifies performance-based dynamic teaching in DiCo (Xu et al., CVPR 2025) as prior work; the new claim is the value of the coordination. The current contrast establishes a package-level improvement over symmetric CPS, not the proposed explanation for that improvement.

The inference protocol introduces another unresolved source of benefit. Sec. 3.7 allows a selected branch or validation-chosen probability fusion, together with optional TTA and connected-component processing. The actual choices for the CPS and HCT rows are not reported. A heterogeneous pair may improve through complementary inference errors, even if its training-time reliability policy adds little. Figure 3's larger disagreement is compatible with that alternative. The homogeneous Mamba degradation on ACDC, from 0.8219 to 0.7871, is also informative: lower performance under the controller cannot be explained solely by insufficient parameter count. The paper acknowledges this result, but does not connect it to routing behavior or supervision exposure. [Table 3; Secs. 3.7 and 4.4-4.6.]

W2 The reliability diagnostics do not measure the reliability claim on a common population

Eq. 8 compares the selected teacher's current labeled-training Dice with a fraction of its own smoothed score. A branch with persistently poor but stable predictions can pass this gate: once its current and smoothed positive scores agree, the inequality holds for either reported alpha. The gate detects a relative drop in training fit, not absolute pseudo-label quality. Coverage in Eq. 11 averages over all voxels, so confident background can give a sample a high weight even when its foreground predictions are unreliable. The full-domain loss still teaches low-confidence foreground voxels. Meanwhile, feature alignment bypasses the ramp and coverage factors entirely. Thus the mechanisms do not uniformly suppress unreliable cross-branch information, even when pseudo-label weighting is small.

Figure 3 does not resolve these limitations. For LA, disagreement is measured on validation volumes, whereas pseudo-label error is measured only on active pseudo-supervision mini-batches. Because each configuration has its own teacher ranking and gate, the error curves can condition on different crops and training stages. A lower curve can reflect easier selected examples rather than more accurate pseudo-labels on the same examples. For ACDC, shared error overlap is an intersection-over-union restricted to ground-truth foreground: it omits false-positive regions and changes when the sizes of either branch's error set change. Neither statistic establishes that the selected teacher is better than its peer on the routed unlabeled batch. The curves are useful descriptions, as the text calls them, but they cannot carry the causal explanation attributed to reliable heterogeneous coordination. [Sec. 4.5; Figure 3.]

Figure 1 also raises a provenance and accuracy concern. I would like clarification about possible generative-AI assistance in constructing this illustration; appearance alone cannot establish its provenance. The verifiable issue is that panel (b) draws a sequence from hard pseudo-label to confidence map to coverage mask to safety gate. Confidence requires the probability map, and Eq. 8's gate depends on labeled Dice and warm-up, not coverage. The caption partly acknowledges the schematic nature but leaves these arrows misleading. The AI Use Statement mentions ideation, literature retrieval, drafting, and polishing, but does not say whether figures were generated or edited. The authors should clarify this and make the figure agree with the actual computation; this concern is about faithful scientific communication, not proof that experimental data were fabricated.

Questions for authors

Q1 What changed between the matched CPS and HCT results

For the two heterogeneous rows in Table 3, were the supervised Hausdorff term, training budget, checkpoint criterion, and inference rule identical? Please identify the branch or fusion coefficient, TTA, and connected-component settings actually used for each. Which reported observation distinguishes improved training-time coordination from a change in regularization strength or inference-time ensembling?

Q2 What is the population behind Figure 3

For each LA configuration, which samples and epochs enter the active-batch error curve, and what fraction of candidate batches is excluded by its gate? Does the plotted error average first over crops, batches, or voxels? Please explain how the curves can be compared when their inclusion rule is method-dependent. For ACDC, what happens to the shared-error ratio when one branch improves while the other does not? These definitions are necessary to interpret the existing diagnostics.

Q3 What does the gate protect against in the observed runs

Do the available routing records show that the labeled-training-score winner provides better unlabeled targets than its peer, or only that it fits the labeled crops better? How do those records explain the homogeneous Mamba failure? Please also explain what limits unreliable feature transfer immediately after warm-up, since Eq. 15 lacks the ramp and coverage factors used in Eq. 12. A mechanistic explanation tied to the reported runs would be more informative than a general claim that the controller is adaptive.

Q4 Were generative tools used to produce the figures

Was Figure 1 generated or substantially edited using generative AI, and if so, which elements and how were their computational dependencies checked? Please reconcile the confidence-to-gate arrows with Eqs. 8 and 11. For clarity, distinguish any illustrative assets from Figures 2 and 3, which are presented as experimental evidence, and state whether those panels were rendered directly from segmentation outputs and recorded training diagnostics. This requests provenance and an accurate depiction; it does not presume that those experimental panels were generated.

3 comments

A0. Thank you for identifying the incorrect dependency in the original schematic and for asking us to clarify the training diagnostics. We corrected Figure 1 and its caption (p. 4), aligned the description in Sec. 3.4 (pp. 4–5) with the revised paths, clarified the LA curves in Sec. 4.5 (p. 7), and expanded the AI Use Statement (p. 9).

Q1. For the two heterogeneous rows in Table 3, were the supervised Hausdorff term, training budget, checkpoint criterion, and inference rule identical? Please identify the branch or fusion coefficient, TTA, and connected-component settings actually used for each. Which reported observation distinguishes improved training-time coordination from a change in regularization strength or inference-time ensembling?

A1. The shared control in Table 3 is the SegMamba–Swin UNETR backbone pair. At 15% labeling, the complete HCT configuration improves mean Dice over symmetric CPS from 81.92% to 83.01% on ACDC and from 82.80% to 85.00% on LA. HD95 and ASD are also lower on both datasets. The gain therefore extends across both overlap and surface-distance metrics at this labeling level.

For HCT-Net, Sec. 4.1 (p. 6) specifies a supervised Hausdorff coefficient of 0.1 and a schedule of 200 supervised warm-up epochs followed by 250 semi-supervised epochs. Sec. 3.7 (p. 5) describes the validation-based selection of the branch or fixed probability fusion and any test-time augmentation or post-processing. The quantitative claim in Table 3 (p. 8) is a comparison of the evaluated HCT and CPS configurations with the backbone pair held fixed. Its central result is the improvement of the complete HCT configuration on both benchmarks at 15% labeling.

Q2. For each LA configuration, which samples and epochs enter the active-batch error curve, and what fraction of candidate batches is excluded by its gate? Does the plotted error average first over crops, batches, or voxels? Please explain how the curves can be compared when their inclusion rule is method-dependent. For ACDC, what happens to the shared-error ratio when one branch improves while the other does not? These definitions are necessary to interpret the existing diagnostics.

A2. For LA, the two panels describe different populations. Foreground prediction disagreement is evaluated on validation volumes. Pseudo-label error is computed as one minus Dice against reference masks on training mini-batches with active pseudo-supervision. The reference masks are used only for the diagnostic. We added a clarification in Sec. 4.5 (p. 7) that each error curve summarizes the active batches of its own configuration. The relative heights of the curves therefore reflect pseudo-label accuracy together with the composition of the selected batches. We interpret the LA error curves as descriptive training trajectories conditional on each configuration’s active batches.

For ACDC, shared-error overlap measures the intersection-over-union of the two error sets within reference foreground. With one branch fixed, correcting a shared error shrinks the intersection while leaving the union unchanged. Correcting an error unique to the improving branch shrinks the union while leaving the intersection unchanged, which can increase the ratio. This explains why the metric depends on which errors are corrected as well as how many.

Q3. Do the available routing records show that the labeled-training-score winner provides better unlabeled targets than its peer, or only that it fits the labeled crops better? How do those records explain the homogeneous Mamba failure? Please also explain what limits unreliable feature transfer immediately after warm-up, since Eq. 15 lacks the ramp and coverage factors used in Eq. 12. A mechanistic explanation tied to the reported runs would be more informative than a general claim that the controller is adaptive.

A3. The gate controls when cross-branch transfer is active. After warm-up, Eq. (8) checks whether the selected teacher’s current labeled-batch Dice exceeds a fraction of its own EMA. A low but stable positive Dice can satisfy this relative criterion. The score thus has a specific role: it ranks the branches by their smoothed labeled-batch performance. Figure 3 characterizes the corresponding training behavior at the configuration level.

The shared gate disables feature alignment on iterations that fail this check. On active iterations, the feature loss uses its own coefficient and the detached teacher feature directs the update to the student side. The ramp-up and coverage factors instead scale the pseudo-label cross-entropy. The revised Figure 1 (p. 4) depicts these separate controls, which are defined in Eqs. (12)–(16).

Table 3 (p. 8) also shows why the backbone pairing matters empirically. On ACDC, Dice moves from 82.19% to 78.71% for homogeneous Mamba and from 81.92% to 83.01% for the heterogeneous pair when switching from CPS to HCT. The two outcomes support evaluating the training policy together with the backbone pairing. This is the distinction we emphasize in Sec. 4.7 (p. 8).

Q4. Was Figure 1 generated or substantially edited using generative AI, and if so, which elements and how were their computational dependencies checked? Please reconcile the confidence-to-gate arrows with Eqs. 8 and 11. For clarity, distinguish any illustrative assets from Figures 2 and 3, which are presented as experimental evidence, and state whether those panels were rendered directly from segmentation outputs and recorded training diagnostics. This requests provenance and an accurate depiction; it does not presume that those experimental panels were generated.

A4. We used generative AI to produce reference artwork for the original Figure 1 and redrew the framework in Draw.io. Some illustrative icons in the submitted figure came from the reference artwork. The revised figure uses editable vector elements drafted with AI assistance. Its volume icons are vector traces of example 2D slices with schematic depth. We added this disclosure to the AI Use Statement (p. 9).

Your comment identified a specific mismatch between the original diagram and Eq. (8). In revised Figure 1(b) (p. 4), the gate receives warm-up status and the teacher’s current labeled-batch Dice relative to its EMA-based threshold. The teacher’s probability map now branches separately into hard pseudo-labels and voxel-wise confidence, following Eqs. (10)–(11). Coverage and ramp-up enter the crop-level weight in Eq. (12), which scales cross-entropy in Eq. (13). Feature alignment shares the gate in Eq. (15) and uses its own coefficient in Eq. (16). We also updated Figure 1(a) to show labeled inputs reaching both branches and supervised loss entering the total objective. The revised caption and Sec. 3.4 (pp. 4–5) now follow the same computational paths.

Figure 2 was rendered directly from segmentation outputs and reference masks. Figure 3 was plotted directly from the training diagnostics described in Sec. 4.5. The AI-generated reference artwork was used for the framework illustration in Figure 1.

Thank you for the detailed response. The authors have partially addressed my concerns. However, the contribution of the proposed coordination mechanism and the reliability of teacher selection remain insufficiently demonstrated. Accordingly, I have updated my score from 1 (Clear Rejection) to 2 (Weak Rejection).

Reviewer_2cbmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes HCT-Net, a semi-supervised framework for 3D medical image segmentation using heterogeneous Mamba and Transformer branches. The motivation is that heterogeneous learners may evolve at different rates, making symmetric cross-pseudo supervision potentially harmful when a temporarily weaker branch provides unreliable pseudo-labels to the stronger branch.

HCT-Net introduces a controller that coordinates three decisions during training: which branch acts as the teacher, whether cross-branch supervision should be activated, and how strongly each unlabeled sample contributes. The teacher is dynamically selected according to an exponential moving average of labeled-batch Dice. A performance-based gate determines whether cross-branch supervision is enabled, while confidence coverage and training ramp-up control the strength of pseudo-label supervision. Prediction-level pseudo-label transfer and feature-level alignment follow the same dynamically selected teacher-to-student direction.

Critical strengths

The paper is clearly written and easy to follow. It addresses important design choices when using heterogeneous architecture branches: which branch should teach, whether teaching should occur, and how strongly each unlabeled sample should contribute. The method dynamically determines the teacher–student roles by estimating the relative performance of the two branches. The overall idea is intuitive and well motivated.

Critical weaknesses
  1. The methodological novelty appears relatively incremental. Heterogeneous cross-teaching, dynamic teacher–student role assignment, confidence-based pseudo-label regulation, and collaborative feature learning have all been explored in prior work. The paper itself acknowledges that DiCo dynamically assigns teacher–student roles according to supervised performance, while other methods already regulate pseudo-supervision using uncertainty, confidence, diversity, or role exchange. The primary contribution therefore appears to be integrating these existing ideas under a shared directed routing policy rather than introducing a fundamentally new semi-supervised learning mechanism.

  2. The empirical evaluation scope is relatively narrow, only on cardiac MRI. It remains unclear whether the benefit of the proposed heterogeneous coordination extends to other anatomies, modalities, or segmentation settings.

Questions for authors
  1. How the branches is selected should be described more clearly. The current description is vague.
  2. How often does the branch with the higher labeled-batch reliability score actually produce more accurate pseudo-labels, and how sensitive is this relationship to batch size and EMA smoothing?
1 comment

A0. Thank you for asking us to clarify branch selection and the role of the reliability score. The revised Figure 1(b) makes teacher assignment explicit, and Sec. 4.5 clarifies how to interpret the LA training diagnostics.

Q1. How the branches is selected should be described more clearly. The current description is vague.

A1. The two architectures are fixed before training. SegMamba provides a state-space branch, and Swin UNETR provides an attention-based branch. Their teacher and student roles are assigned dynamically. At each training iteration, we compute mean foreground Dice from both branches’ hard predictions on the same labeled mini-batch, update their score EMAs with decay 0.9, and select the branch with the higher updated score as the temporary teacher. When the gate opens, pseudo-label supervision and feature alignment both follow this teacher-to-student direction. Both branches continue to receive ground-truth supervision.

We made the routing rule explicit in Figure 1(b), steps 1–2 (p. 4). Branch A is shown as the teacher in the example where it ranks higher. Section 3.7 describes the separate validation-based selection of the prediction rule used at inference.

Q2. How often does the branch with the higher labeled-batch reliability score actually produce more accurate pseudo-labels, and how sensitive is this relationship to batch size and EMA smoothing?

A2. We use a fixed EMA decay of 0.9 on both datasets. The score provides an online ranking based on labeled-batch foreground Dice. Under this setting, Figure 3(b) shows that Hetero-HCT maintains greater foreground prediction disagreement than the homogeneous controls, while its LA pseudo-label error trajectory stays below Homo-Swin and close to Homo-Mamba.

We clarified in Sec. 4.5 (p. 7) that each LA error curve summarizes the active batches of its own configuration. The reported evidence therefore concerns prediction diversity and pseudo-label error under the evaluated settings. The teacher-versus-peer win rate and its sensitivity to batch size and EMA decay are separate quantities from those plotted in Figure 3.

Reviewer_MFH5OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the HCT-Net, which combines Mamba and Transformer branches for semi-supervised 3D segmentation. It selects a temporary teacher using smoothed Dice on labeled training batches, gates cross-branch learning by relative performance, and weights each unlabeled sample by a ramp-up and confidence coverage. Experimental results On ACDC and LA, show the full method improves Dice over symmetric CPS with the same backbone pair by 1.09 and 2.20 percentage points, respectively, at 15% labeling.

Critical strengths
  • The paper is clearly written and easy to follow.

  • The paper addresses a relevant problem: heterogeneous learners may provide supervision of different quality as training progresses.

Critical weaknesses
  1. the novelty on dynamic collaboration is unclear. The paper cites DiCo, which also changes teacher–student roles according to labeled-data performance, but offers no matched comparison.

  2. Lack component ablations. The experiments do not separate the contributions of routing, the gate, coverage weighting, and feature alignment.

  3. Dice on a labeled training batch may not rank the branches by pseudo-label quality on unlabeled crops; teacher-switch and gate-activation frequencies are also missing.

  4. The evidence for robustness is limited. Results use fixed labeled splits and report variation across test volumes, not training seeds or labeled-case draws. I also expect more experiemntal resutls on popular medical benchmarks like Pancreass, Brats, etc.

  5. where is the details on the warm-up length or ramp-up function?

Questions for authors

HCT-Net adds a controller and feature alignment to a dual-branch setup. How about the comparison with a plain single-model U-Net/V-Net.

2 comments

A0. Thank you for asking how HCT-Net compares with a conventional segmentation backbone. In the revised Sec. 4.1 (p. 6), we identify UA-MT as a V-Net-based semi-supervised baseline and specify the two-stage HCT-Net training schedule.

Q1. HCT-Net adds a controller and feature alignment to a dual-branch setup. How about the comparison with a plain single-model U-Net/V-Net.

A1. Tables 1–2 (p. 7) include UA-MT (Yu et al., MICCAI 2019) as the V-Net-based comparator in our semi-supervised evaluation. UA-MT uses V-Net for both the student and its EMA teacher. At 15% labeling, HCT-Net achieves 83.01% versus 79.27% mean Dice on ACDC and 85.00% versus 76.19% on LA. The gains over this V-Net-based semi-supervised baseline are 3.74 and 8.81 percentage points, respectively.

Table 3 (p. 8) provides a complementary comparison with the SegMamba–Swin UNETR backbone pair held fixed. The complete HCT configuration improves mean Dice over symmetric CPS by 1.09 points on ACDC and 2.20 points on LA, with lower HD95 and ASD on both datasets. DiCo established performance-based teacher assignment. HCT-Net extends this design space by using one teacher direction and one activation gate for both pseudo-label supervision and feature alignment, together with crop-level weighting of the pseudo-label loss. The revised Sec. 4.1 also specifies 200 epochs of supervised warm-up followed by 250 epochs of semi-supervised training.

Thanks for your rebuttal. My concerns and questions are not fully response or addressed. I will keep my score

Comments on the paper

2 comments

We thank all three reviewers for their thoughtful and constructive feedback. Your comments helped us clarify how HCT-Net coordinates pseudo-label supervision and feature alignment through a shared teacher direction and activation gate, with crop-level weighting of the pseudo-label loss. In the revised manuscript, we updated Figure 1 and its caption to make these relationships explicit, specified the two-stage training schedule in Sec. 4.1, and clarified the interpretation of the training diagnostics in Sec. 4.5. We respond to each reviewer below and identify the relevant changes in the revised manuscript.

Area_Chair_w73BOpenReview ↗

Gentle reminder: discussion closing soon

Dear Reviewer,

As the discussion period closes soon, we kindly encourage you to engage in the discussion and share any remaining thoughts or feedback.

Thank you for your time and valuable contributions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.