acceptodds
Withdrawn from ICLR 2027

Minimally supervised medical semantic segmentation via unshared reconstructions and shared feature vectors

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The research frontier in semi-supervised semantic image segmentation has recently shifted from the moderate supervision regimes towards minimal supervision, involving 1% to 5% labeled data. This is particularly relevant to medicine, where data annotation requires scarce expert clinical time. However, the extreme low-data setting introduces a unique set of mathematical and structural challenges that do not linearly scale down from their more moderate counterparts. In this work, we introduce a novel minimally supervised (1% labeled samples) framework for semantic the segmentation of medical images. Our model learns meaningful representations through image reconstruction in an encoder–decoder network architecture setting. However, unlike conventional methods that generate a unique latent embedding per image, our architecture utilizes a novel sub-module, named the Vectorizer, and outputs a small set of scalar coefficients. These coefficients serve the purpose of generating a linear combination of trainable feature vectors shared across the dataset. This allows the proposed model to rely on prior shape knowledge accumulated throughout training in order to reconstruct each class. We tested our framework on challenging and diverse medical image datasets, surpassing or being comparable with the minimally supervised state of the art.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Minimally Supervised Learning, Semantic Segmentation, Image Reconstruction, Medical Imaging

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ME2wOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses medical image segmentation with only ~1% labeled data. It proposes a reconstruction-based framework with class-specific Vectorizers that compress each image into a few coefficients used to combine learnable feature vectors shared across samples, thereby encouraging dataset-level shape and location priors. The small labeled set mainly associates each Vectorizer with a semantic class, while reconstruction on both labeled and unlabeled images provides most of the training signal. Experiments on seven medical imaging datasets show that the method is generally competitive with or better than eight semi-supervised baselines under the 1% label setting.

Critical strengths
  1. The method is well motivated. Reducing annotation requirements is important in medical imaging, and using a reconstruction signal that is computed identically on labeled and unlabeled images is a sensible alternative to pseudo-labeling and consistency-based approaches, as it does not depend on the quality of a teacher model.

  2. The evaluation is broad, covering seven datasets, five imaging modalities, and eight recent baselines, and is reasonably controlled. The same labeled subset and data-loading pipeline are used across all methods; the baselines are run using the authors’ official code, and the source code is provided. Notably, the proposed method achieves the lowest Hausdorff distance on all seven datasets.

Critical weaknesses
  1. The entire evaluation is one table and two sentences. No standard deviations or number of random seeds are reported; at 1% labels the variance across labeled subsets is large, so single-run numbers cannot distinguish the method from the baselines. There is no ablation study, so it is unknown whether the gains come from the Vectorizer or simply from adding a reconstruction loss. No qualitative segmentation results are shown.

  2. The exact number of labeled samples per dataset is not given. Several datasets are small (Drishti-GS has about 50 training images, RIM-ONE DL about 311), so "1%" cannot be realized literally. It is not stated whether sampling is per slice/frame or per patient for CAMUS, CardiacUDA and MSD Hippocampus, which changes task difficulty substantially.

  3. The Vectorizer (Figure 2) computes C=ΣjsjujC = Σ_j s_j u_j, which is a low-dimensional bottleneck followed by a linear layer; the "shared trainable feature vectors" are the weights of that layer. The segmentation-weighted per-class reconstruction loss (Eq. 7) is the design of MONet and Neural EM, which the authors cite. Per-class decomposition with reconstruction for semi-supervised medical segmentation was already proposed in SDNet (Chartsias et al., 2019), which is not discussed. Nothing in the method is specific to the 1% regime that motivates the paper.

  4. Since L_seg is only an anchor and L_rec is minimized by assigning each pixel to the component that reconstructs it best, on the 99% unlabeled images the segmentation head is effectively trained to cluster pixels by appearance, not anatomy. Structures with similar intensity but different semantics (left vs. right ventricular blood pool, optic disc vs. cup) are likely to be confused. The authors observed this collapse themselves (Section 3.5) but only patch it with a warm-up schedule, and no masks on unlabeled images are shown.

  5. The 1% target is justified by clinical annotation cost, but a percentage depends on dataset size and is not a meaningful unit of annotation effort. The paper does not explain why 1% rather than a fixed small number of cases is the right target.

  6. Writing can be improved. The introduction surveys general semi-supervised learning at length without connecting it to the method. The contributions appear only as three bullet points inside the related work section. Terms such as "moderate supervision regimes" and "minimally supervised" lack clarity.

Questions for authors

Q1. (W1) The reproducibility statement says all experiments were run with seed=1, i.e., a single run on a single labeled subset, while Section 4 itself states that the choice of labeled samples has a severe impact on results. Please report results over multiple seeds and labeled subsets with mean and standard deviation.

Q2. (W1) Section 3.3 states that the Vectorizer is the main novelty, but no experiment isolates its effect. Please provide an ablation that replaces the Vectorizer with a standard per-image latent vector of the same dimension, keeping everything else fixed, together with a sweep over the number of feature vectors d. Without this, the reported gains cannot be attributed to the proposed contribution.

Q3. (W2) For each dataset, how many labeled images or volumes were used? For CAMUS, CardiacUDA and MSD Hippocampus, is the 1% sampled per slice/frame or per patient?

Q4. (W2) Were baseline hyperparameters adapted to the 1% setting and to these datasets, or were the default settings for ACDC/LA at 5-10% used?

Q5. (W3) Page 5 acknowledges that the Vectorizer is “practically a matrix multiplication”, i.e., a bottleneck followed by a linear layer, yet Section 3.3 presents it as the main novelty. What is new about this module relative to a standard low-dimensional latent in an autoencoder?

Q6. (W1, W4) Please show predicted segmentation masks against ground truth and against the strongest baseline on test images from several datasets, in particular CAMUS, HC18, and ISIC-2017, where the method has lower Dice but lower HD. Please also show masks on unlabeled training images where structures have similar intensity (e.g., left vs. right ventricle), and report how often the collapse described in Section 3.5 occurs with the warmup.

Q7. (W4) In Figure 3, the four reconstructions look nearly identical despite visibly different inputs, which suggests the decoder outputs an average shape of the dataset rather than a reconstruction of each image. Can the authors show that the reconstruction actually varies with the input, for example, by comparing with a standard autoencoder trained with the same loss? How does the model handle pathological cases where chamber walls are truly absent?

Reviewer_1fF3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a minimally supervised framework for medical semantic segmentation using approximately 1% labeled training data. Unlike conventional semi-supervised approaches that rely on pseudo-labels or teacher-generated supervision, the proposed method leverages image reconstruction to learn class-specific representations. Its core component, termed the Vectorizer, generates compact representations by combining trainable feature vectors shared across training samples. These representations are processed by separate reconstruction and segmentation decoders, enabling the model to capture implicit anatomical priors. Training jointly optimizes a reconstruction loss on unlabeled images and a Dice-based segmentation loss on the limited labeled samples, with a progressive reconstruction warmup to stabilize optimization. The framework is evaluated on seven medical imaging datasets covering multiple modalities and anatomical structures. Experimental results demonstrate competitive performance against eight established semi-supervised segmentation methods, achieving the best Hausdorff Distance across all evaluated datasets and the highest Dice scores on four datasets.

Critical strengths

1.The paper addresses the challenging 1% labeled-data setting by leveraging reconstruction-based learning rather than relying on potentially unreliable pseudo-labels. The use of class-specific representations and shared feature vectors provides a reasonable approach to incorporating implicit anatomical priors.

2.The method is evaluated on seven datasets spanning multiple imaging modalities and anatomical structures. It achieves the lowest Hausdorff Distance on all seven datasets and the highest Dice score on four, demonstrating promising effectiveness, particularly in reducing spatially inconsistent segmentation errors.

Critical weaknesses

The proposed Vectorizer is mathematically similar to a learnable linear projection, yet its advantages over simpler alternatives remain unclear. The absence of essential ablation studies makes it difficult to determine whether the reported improvements originate from shared feature vectors, reconstruction supervision, or other architectural choices.

All experiments use a single random seed despite the sensitivity of the 1% labeled-data setting to sample selection. Without repeated trials or statistical analysis, the marginal improvements over existing methods on several datasets provide insufficient evidence of consistent superiority.

Questions for authors

The proposed Vectorizer appears mathematically equivalent to a learnable linear projection. Could the authors clarify its fundamental distinction from a standard projection layer and provide controlled comparisons or ablation results demonstrating the specific contributions of shared feature vectors and reconstruction supervision? Such evidence would help establish the necessity of the proposed design.

Given the sensitivity of the 1% labeled-data setting to sample selection, could the authors clarify whether the experiments were repeated with different random seeds and labeled subsets? If available, please report the mean and standard deviation, particularly for datasets where the improvements over competing methods are marginal. This would help assess the reliability of the reported gains.

Reviewer_rKYwOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a minimally supervised medical image segmentation framework that combines class-specific image reconstruction with a small amount of segmentation supervision (1% labelled data).

The method introduces a vectorizer module that represents each semantic/representative class through a linear combination of learnable feature vectors shared across training samples. These class-specific representations are processed by reconstruction and segmentation decoders, whose parameters are shared across classes. The framework also introduces a reconstruction objective that weights per-component reconstruction errors by the corresponding segmentation probabilities, rather than computing the reconstruction error after blending component outputs.

Experiments on seven medical imaging datasets demonstrate competitive performance against several semi-supervised segmentation baselines, particularly in terms of Hausdorff distance.

Critical strengths

This paper addresses the most important drawback for medical imaging -- the scarity of labeled data. The authors proposed a rather simple method to boost the segmentation performance given extremely small set of labeled dataset (1% of labeled data).

I would appreciate the simplicity of their proposed method. a) a vectoriser module with learnable representation set. b) two decoders to do independent segmentation tasks, and c) the proposed square-then-blend reconstruction loss providing an intuitive mechanism for encouraging individual components to explain different image regions.

The authors show good but limited experimental results. The method achieves the highest Dice score on four of seven datasets and the lowest reported Hausdorff distance on all seven datasets in Table 1.

Critical weaknesses

The paper addressed the scarcity of labels in the medical imaging field with their proposed "unshared reconstructions and shared feature vectors." I find this paper interesting, but I wish the authors could address the following concerns.

  1. The writing needs to be significantly improved. The method seemed to be easy, but when diving into it, one would find it hard to guess the implementation details, especially the shape and the motivation of each defined variable. The figures are not complete, saying they show segmentation results in number but never in image. Figs. 1 and 2 seem to be done with a rush and need huge effort to polish.

  2. The motivation is not well developped. The core idea is rather simple, and the structure appears easy to implement. However, one fails to understand why this would work in the particular setting, instead of a similar design of a simpler one (like using MLP with 64-node block neck) or higher complexity (like using attention with sigmoid as the activation).

Questions for authors

Q1. Choice of encoder. The framework uses a custom ResNet-style encoder. What motivated this choice over encoders pretrained on medical images? Could the authors clarify whether the proposed approach benefits from such pretrained representations [1, 2], and whether the reported gains remain consistent when using a medical-image-pretrained encoder?

Q2. The author proposed a Vectorizer module. The goal of this is to predict a few coefficients to blend with a set of learnable parameters. Why not use attention with sigmoid here? The image feature could provide the query, with class-specific learned keys and values. This would still preserve the shared basis vectors and the 64-coefficient bottleneck. I do not see a clear distinction between this formulation and the proposed Vectorizer.

Q3. Could you clarify the exact architecture of (V_{\mathrm{nn},i}), particularly its output activation? Are the 64 coefficients unconstrained, sigmoid-gated, or softmax-normalized? This is not clear from the paper and affects how we should interpret the combination of the learned basis vectors.

Q4. For the decoder, whose only input is a spatially compressed component vector (1x1x256), the reconstruction is to recover the segmentation and appearance. Personally, I have more concern about this claim, and I believe the benefit of the spatial bottleneck is insufficiently supported. The paper provides no controlled ablation showing that withholding spatial encoder features improves segmentation.

Q5. The paper provides quantitative segmentation metrics but no qualitative segmentation predictions. Figure 3 shows only input images and reconstructions, which do not directly demonstrate accurate segmentation of structures with unclear boundaries. Comparisons of predicted masks against ground truth, including difficult and failure cases, are needed to support this claim.

Q6. The choice of learnable set per segmentation class or shared across all classes? I believe they are class-specific, but I may suggest providing an additional experience showing per-class better than the shared one.

Q7. The experiments only include training with extremely small amounts of data. The authors should add a baseline of using all available data to see the upper bound of the proposed method.

Q8. Coupling reconstruction and segmentation. What motivates weighting reconstruction errors by the predicted segmentation probabilities, rather than using reconstruction as an independent auxiliary task with a shared encoder? Could the authors provide a controlled ablation against the independent auxiliary-task baseline, with comparable model capacity and identical data and training budgets, to demonstrate the segmentation gains attributable to this coupling?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.