- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a registration-aware in-context segmentation model that leverages the deformation relationship between the context and query to facilitate the transfer of context masks to query segmentation. The experimental results show that the proposed method achieves better performance than previous baselines. However, this work suffers from limited novelty and technical insight, as its core motivation has already been explored by several prior works on related topics. In addition, the proposed core modules, such as SRM and DCP, appear to mainly consist of relatively basic attention mechanisms and are not sufficiently validated. There are also potential concerns regarding the fairness of the experimental comparison. Overall, I believe this work is below the acceptance bar of ICLR.
Critical strengths
- Registration-driven alignment is a reasonable direction for addressing in-context medical image segmentation.
- The paper is well organized and easy to follow.
- The experimental results appear to be strong.
Critical weaknesses
-
Performing few-shot/in-context medical image segmentation using registration-like learning to align the context with the query is not a new idea. In fact, it has already been studied by several previous works, including but not limited to [1,2,3].
-
What if the context and query differ substantially? For example, the context may come from an edge slice while the query comes from a middle slice. Can SpotBlock still effectively align them? I think this could have a significant impact on the results, especially when only a few context examples are available, e.g., S=1. This may also be one reason why the model performance drops noticeably with fewer context examples in Fig. 5.
-
The proposed registration mechanism is not sufficiently justified and validated.
- For SRM, the registration mechanism (or local attention) should be more directly validated and interpreted, for example, by visualizing intermediate local DVFs or deformation grids. Otherwise, it is unclear whether the context-query alignment is actually achieved through the proposed registration mechanism, or is simply a result of the implicit attention computation that is subsequently interpreted with some fancy stories.
- It would also be useful to remove the operation in Eq. (3) and simply use vanilla attention to see whether the registration still works under the registration loss, which will help to validate the contribution of
-
I do not understand why the registration loss is applied only to the segmentation masks. Segmentation masks contain limited information. Even for aligning context and query features, the model could directly focus on semantic feature alignment rather than learning it indirectly through the final masks. Additional experiments would be beneficial to validate the superiority of the chosen design.
-
The proposed registration loss seems to provide only marginal improvement to the model, as shown in Table 2. This makes the role and necessity of registration less convincing, as the model already performs well without registration supervision. In addition, SRM, another registration-related module, brings only a small improvement compared with DCP. These results make the overall registration story weaker.
-
I am concerned that the experimental comparison may be unfair, as baselines such as UniverSeg and Tyche appear to use pretrained weights, whereas OmniSeg is trained on the dataset assembled in this work. This itself may provide a considerable advantage.
[1]Y. He et al., "Learning Better Registration to Learn Better Few-Shot Medical Image Segmentation: Authenticity, Diversity, and Robustness," in IEEE Transactions on Neural Networks and Learning Systems, vol. 35, no. 2, pp. 2588-2601, Feb. 2024 [2]Zhou Y, Zhou F, Xi F, Liu Y, Peng Y, Carlson DE, Tu L. Efficient few-shot medical image segmentation via self-supervised variational autoencoder. Med Image Anal. 2025 Aug;104:103637. doi: 10.1016/j.media.2025.103637. Epub 2025 May 26. [3]Li Y, Fu Y, Gayo I J M B, et al. Prototypical few-shot segmentation for cross-institution male pelvic structures with spatial registration[J]. Medical Image Analysis, 2023, 90: 102935.
Questions for authors
Please see Weakness for my questions.
10 comments
W1. Positioning w.r.t. registration learning in ICS literature
Registration as an auxiliary objective for segmentation. We fully acknowledge that using registration for ICS or few-shot segmentation is not novel, as atlas-based techniques are foundational to medical image analysis. Specifically, deep learning-based methods have frequently employed registration to perform or improve segmentation. However, in these approaches, registration is typically predicted by a dedicated module and applied in pixel space. In contrast, our work computes registration DVFs in closed form from the features of the OmniSeg network rather than predicting them. This enables OmniSeg to apply registration directly to feature maps as an auxiliary objective that guides segmentation. The benefits of this additional supervision term are supported by our experimental results (see the updated Table 2) and an analytical argument (Appendix B). To our knowledge, OmniSeg is the first ICS model that registers the context onto the query in the feature space at every level of its decoder, selects relevant context information pixel by pixel, supervises this registration end to end, and analyzes why this supervision benefits segmentation.
Positioning w.r.t. literature. We thank the reviewer for pointing out papers related to our work. Indeed, [1] and the more recent COVER method by the same author helped us to deepen our understanding of the interaction of registration and segmentation. Along with COVER, [1] is cited as a related work in Sec. 2 (He et al., 2022, early-access version of the TNNLS 2024 paper). In the revised paper, we include [2, 3] in our related works section as they are indeed highly relevant. However, we believe that [1,2] present major differences with our work: the available data samples of the few-shot scenarios are used at training time while OmniSeg uses them as input context at inference time.
[3] is the closest work: an episodic prototypical few-shot model that also tackles query/context misalignment, but only for cross-institution generalization. Like OmniSeg, it supervises alignment with a Dice loss on masks, but differs in three respects. [3] predicts a single global affine transform with a separate module and applies it once, whereas OmniSeg's alignment is dense, multi-scale and induced by the attention itself (Eq. (3), Eq. (18)). [3] relies on segmentations of the base classes of a single domain (pelvic MRI, 8 structures), whereas OmniSeg only uses the binary mask of the current task (Eq. (7)–(8)), which makes it modality-agnostic and trainable across 80 subsets and 8 modalities (Sec. 4.1). For $S > 1$, [3] keeps a single support image selected at the image level, whereas our DCP weights all context examples pixel by pixel (Eq. (4)–(5)).
Moreover, all three papers use registration in ways that differ fundamentally from ours, and are task-specific frameworks, while OmniSeg is a universal model.
To account for this discussion and explicitly acknowledge [1,2,3], we have updated the paragraph Local attention for registration of Sec. 2 and renamed it Registration and query–context alignment.
W2. Robustness to edge cases.
The reviewer questions how OmniSeg behaves when a small context contains samples very different from the query. The following paragraphs address this point both in the general case (i) and, more specifically for 3D imaging, in the scenario raised by the reviewer where the context image is a slice taken at a different level of the same anatomical target as the query image (ii).
ICS robustness to semantic misalignment (i). When the query and the context differ substantially, we acknowledge that cases with $\mathbf{S = 1}$, i.e., a single context example semantically unrelated to the query, are a limit case for our approach. No registration is meaningful there, and no other context example can compensate. This limitation is shared by every ICS method, since the context then carries no transferable information. We have added this limit case to the conclusion of the revised paper (in blue) as a direction for future work, complementing the limitation on strong topological variability across subjects discussed in App. F.3.
In cases $\mathbf{S>1}$, we design a new experiment to show that OmniSeg segmentation performance remains stable even when the context set contains semantically unrelated context examples. We compare MultiverSeg to the “SRM” and “SRM+DCP” variants of our model in an experiment where we inject 50% of examples unrelated to the query in the context. Our results show that adding the DCP mechanism makes OmniSeg more robust to the distractor examples with a stronger performance with the corrupted context. Indeed, DCP discards semantically misaligned context examples. This experiment is presented in Sec. 4.3 in paragraph Robustness to outlier context examples.
Table (new): Robustness to outlier context examples () on the cytoplasm task of WBC
| Model | Clean Ctxt | Corrupted Ctxt | Δ |
|---|---|---|---|
| MultiverSeg | 64.8 $\pm$ 1.9 | 42.8 $\pm$ 2.0 | -22.0 |
| SRM | 72.0 $\pm$ 1.6 | 58.8 $\pm$ 1.8 | -13.2 |
| SRM+DCP | 74.6 $\pm$ 1.8 | 63.4 $\pm$ 1.7 | -11.2 |
Constructing our dataset to handle varied slice positions in 3D (ii). For volumes, we construct both our training and evaluation datasets by extracting from each target on the three axes the largest section slice and two other slices in the middle between the largest section slice and edge slices, as detailed in App. C. This ensures our model is able to handle varying slice views of the same target.
OmniSeg’s architecture handles differences of shape and sizes between query and context (ii). Since SRM is not restricted to rigid displacements, it can also absorb changes in size and shape covering the differences between a middle slice and an edge-slice. DCP additionally weights context examples pixel by pixel according to their similarity with the query after registration (Eq. (4), Fig. 20), so that poorly matching examples contribute less.
We thank the reviewer for the detailed review and for acknowledging that registration-driven alignment is a reasonable direction for in-context segmentation (ICS), that the paper is clear, and that the results are strong. We address each weakness below, following the reviewer's numbering, and summarize the revisions at the end of our response. Table, figure and section numbers refer to the revised paper.