acceptodds
Under review as a conference paper at ICLR 2027

Local Attention Learns From Registration for In-Context Medical Image Segmentation

Abstract

In-Context Segmentation (ICS) uses a small set of annotated samples as a prompt and has proven to be a powerful paradigm for both domain and task generalization in medical image segmentation. However, current ICS models often assume that the query image and the provided context are spatially aligned, a condition frequently unmet in real-world clinical scenarios. This paper introduces OmniSeg, a novel ICS model that dynamically aligns context and query representations, with two core contributions: SpotBlock, a new module combining spatial and contextual dense attention, and a deep registration scheme that supervises this attention with a registration loss. Our method better leverages the context set, which is crucial for out-of-distribution scenarios, and thus achieves strong generalization. OmniSeg significantly outperforms state-of-the-art ICS methods, yielding an average gain of 9.8 Dice points against the best baseline across 10 held-out datasets. Our approach paves the way for a new generation of spatially-aware ICS models. Source code will be released upon publication.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
In-Context Segmentation, Few-Shot Segmentation, Medical Image Segmentation, Registration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ewVFOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a registration-aware in-context segmentation model that leverages the deformation relationship between the context and query to facilitate the transfer of context masks to query segmentation. The experimental results show that the proposed method achieves better performance than previous baselines. However, this work suffers from limited novelty and technical insight, as its core motivation has already been explored by several prior works on related topics. In addition, the proposed core modules, such as SRM and DCP, appear to mainly consist of relatively basic attention mechanisms and are not sufficiently validated. There are also potential concerns regarding the fairness of the experimental comparison. Overall, I believe this work is below the acceptance bar of ICLR.

Critical strengths
  1. Registration-driven alignment is a reasonable direction for addressing in-context medical image segmentation.
  2. The paper is well organized and easy to follow.
  3. The experimental results appear to be strong.
Critical weaknesses
  1. Performing few-shot/in-context medical image segmentation using registration-like learning to align the context with the query is not a new idea. In fact, it has already been studied by several previous works, including but not limited to [1,2,3].

  2. What if the context and query differ substantially? For example, the context may come from an edge slice while the query comes from a middle slice. Can SpotBlock still effectively align them? I think this could have a significant impact on the results, especially when only a few context examples are available, e.g., S=1. This may also be one reason why the model performance drops noticeably with fewer context examples in Fig. 5.

  3. The proposed registration mechanism is not sufficiently justified and validated.

  1. For SRM, the registration mechanism (or local attention) should be more directly validated and interpreted, for example, by visualizing intermediate local DVFs or deformation grids. Otherwise, it is unclear whether the context-query alignment is actually achieved through the proposed registration mechanism, or is simply a result of the implicit attention computation that is subsequently interpreted with some fancy stories.
  2. It would also be useful to remove the Δ\Delta operation in Eq. (3) and simply use vanilla attention to see whether the registration still works under the registration loss, which will help to validate the contribution of Δ\Delta
  1. I do not understand why the registration loss is applied only to the segmentation masks. Segmentation masks contain limited information. Even for aligning context and query features, the model could directly focus on semantic feature alignment rather than learning it indirectly through the final masks. Additional experiments would be beneficial to validate the superiority of the chosen design.

  2. The proposed registration loss seems to provide only marginal improvement to the model, as shown in Table 2. This makes the role and necessity of registration less convincing, as the model already performs well without registration supervision. In addition, SRM, another registration-related module, brings only a small improvement compared with DCP. These results make the overall registration story weaker.

  3. I am concerned that the experimental comparison may be unfair, as baselines such as UniverSeg and Tyche appear to use pretrained weights, whereas OmniSeg is trained on the dataset assembled in this work. This itself may provide a considerable advantage.

[1]Y. He et al., "Learning Better Registration to Learn Better Few-Shot Medical Image Segmentation: Authenticity, Diversity, and Robustness," in IEEE Transactions on Neural Networks and Learning Systems, vol. 35, no. 2, pp. 2588-2601, Feb. 2024 [2]Zhou Y, Zhou F, Xi F, Liu Y, Peng Y, Carlson DE, Tu L. Efficient few-shot medical image segmentation via self-supervised variational autoencoder. Med Image Anal. 2025 Aug;104:103637. doi: 10.1016/j.media.2025.103637. Epub 2025 May 26. [3]Li Y, Fu Y, Gayo I J M B, et al. Prototypical few-shot segmentation for cross-institution male pelvic structures with spatial registration[J]. Medical Image Analysis, 2023, 90: 102935.

Questions for authors

Please see Weakness for my questions.

10 comments

We thank the reviewer for the detailed review and for acknowledging that registration-driven alignment is a reasonable direction for in-context segmentation (ICS), that the paper is clear, and that the results are strong. We address each weakness below, following the reviewer's numbering, and summarize the revisions at the end of our response. Table, figure and section numbers refer to the revised paper.

AuthorsOpenReview ↗

W1. Positioning w.r.t. registration learning in ICS literature

Registration as an auxiliary objective for segmentation. We fully acknowledge that using registration for ICS or few-shot segmentation is not novel, as atlas-based techniques are foundational to medical image analysis. Specifically, deep learning-based methods have frequently employed registration to perform or improve segmentation. However, in these approaches, registration is typically predicted by a dedicated module and applied in pixel space. In contrast, our work computes registration DVFs in closed form from the features of the OmniSeg network rather than predicting them. This enables OmniSeg to apply registration directly to feature maps as an auxiliary objective that guides segmentation. The benefits of this additional supervision term are supported by our experimental results (see the updated Table 2) and an analytical argument (Appendix B). To our knowledge, OmniSeg is the first ICS model that registers the context onto the query in the feature space at every level of its decoder, selects relevant context information pixel by pixel, supervises this registration end to end, and analyzes why this supervision benefits segmentation.

Positioning w.r.t. literature. We thank the reviewer for pointing out papers related to our work. Indeed, [1] and the more recent COVER method by the same author helped us to deepen our understanding of the interaction of registration and segmentation. Along with COVER, [1] is cited as a related work in Sec. 2 (He et al., 2022, early-access version of the TNNLS 2024 paper). In the revised paper, we include [2, 3] in our related works section as they are indeed highly relevant. However, we believe that [1,2] present major differences with our work: the available data samples of the few-shot scenarios are used at training time while OmniSeg uses them as input context at inference time.

[3] is the closest work: an episodic prototypical few-shot model that also tackles query/context misalignment, but only for cross-institution generalization. Like OmniSeg, it supervises alignment with a Dice loss on masks, but differs in three respects. [3] predicts a single global affine transform with a separate module and applies it once, whereas OmniSeg's alignment is dense, multi-scale and induced by the attention itself (Eq. (3), Eq. (18)). [3] relies on segmentations of the base classes of a single domain (pelvic MRI, 8 structures), whereas OmniSeg only uses the binary mask of the current task (Eq. (7)–(8)), which makes it modality-agnostic and trainable across 80 subsets and 8 modalities (Sec. 4.1). For $S > 1$, [3] keeps a single support image selected at the image level, whereas our DCP weights all context examples pixel by pixel (Eq. (4)–(5)).

Moreover, all three papers use registration in ways that differ fundamentally from ours, and are task-specific frameworks, while OmniSeg is a universal model.

To account for this discussion and explicitly acknowledge [1,2,3], we have updated the paragraph Local attention for registration of Sec. 2 and renamed it Registration and query–context alignment.

AuthorsOpenReview ↗

W2. Robustness to edge cases.

The reviewer questions how OmniSeg behaves when a small context contains samples very different from the query. The following paragraphs address this point both in the general case (i) and, more specifically for 3D imaging, in the scenario raised by the reviewer where the context image is a slice taken at a different level of the same anatomical target as the query image (ii).

ICS robustness to semantic misalignment (i). When the query and the context differ substantially, we acknowledge that cases with $\mathbf{S = 1}$, i.e., a single context example semantically unrelated to the query, are a limit case for our approach. No registration is meaningful there, and no other context example can compensate. This limitation is shared by every ICS method, since the context then carries no transferable information. We have added this limit case to the conclusion of the revised paper (in blue) as a direction for future work, complementing the limitation on strong topological variability across subjects discussed in App. F.3.
In cases $\mathbf{S>1}$, we design a new experiment to show that OmniSeg segmentation performance remains stable even when the context set contains semantically unrelated context examples. We compare MultiverSeg to the “SRM” and “SRM+DCP” variants of our model in an experiment where we inject 50% of examples unrelated to the query in the context. Our results show that adding the DCP mechanism makes OmniSeg more robust to the distractor examples with a stronger performance with the corrupted context. Indeed, DCP discards semantically misaligned context examples. This experiment is presented in Sec. 4.3 in paragraph Robustness to outlier context examples.

Table (new): Robustness to outlier context examples (S=2S=2) on the cytoplasm task of WBC

ModelClean CtxtCorrupted CtxtΔ
MultiverSeg64.8 $\pm$ 1.942.8 $\pm$ 2.0-22.0
SRM72.0 $\pm$ 1.658.8 $\pm$ 1.8-13.2
SRM+DCP74.6 $\pm$ 1.863.4 $\pm$ 1.7-11.2

Constructing our dataset to handle varied slice positions in 3D (ii). For volumes, we construct both our training and evaluation datasets by extracting from each target on the three axes the largest section slice and two other slices in the middle between the largest section slice and edge slices, as detailed in App. C. This ensures our model is able to handle varying slice views of the same target.

OmniSeg’s architecture handles differences of shape and sizes between query and context (ii). Since SRM is not restricted to rigid displacements, it can also absorb changes in size and shape covering the differences between a middle slice and an edge-slice. DCP additionally weights context examples pixel by pixel according to their similarity with the query after registration (Eq. (4), Fig. 20), so that poorly matching examples contribute less.

Reviewer_2gykOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OmniSeg, an in-context medical image segmentation framework that addresses spatial misalignment between query and context images. The method combines local attention-based soft registration, dense context pooling, and deep registration supervision to improve feature alignment and context aggregation. Experiments on 10 held-out datasets demonstrate improved segmentation performance over existing methods, supported by ablation studies and generalization analyses.

Critical strengths
  1. The paper addresses a meaningful limitation of existing in-context segmentation methods, and the proposed framework provides a reasonable approach to improving query-context feature alignment.
  2. The experiments cover diverse medical imaging datasets and demonstrate substantial improvements over existing methods, with ablation studies supporting the effectiveness of the proposed components.
Critical weaknesses
  1. The methodological novelty beyond existing attention-based registration and feature aggregation techniques is not sufficiently clear. The paper should better distinguish its contributions from closely related approaches.
  2. The experiments do not fully establish whether improved spatial registration is responsible for the reported segmentation gains. Table 2 shows that dense context pooling contributes considerably more to segmentation performance than explicit registration supervision, leaving the paper's central claim insufficiently validated.
Questions for authors
  1. Could the authors clarify the key methodological differences between SRM and existing attention-based registration approaches, particularly COVER?
  2. Given the relatively modest segmentation improvement from explicit registration supervision in Table 2, can the authors provide stronger evidence that improved spatial alignment directly contributes to segmentation performance?
  3. Could the authors evaluate the method under controlled levels of spatial misalignment to more directly demonstrate the effectiveness of the proposed registration mechanism?
6 comments

We thank the reviewer for the positive assessment. We are happy to have convinced the reviewer that our approach brings a relevant solution to a meaningful limitation of existing in-context segmentation (ICS) methods. We address the weaknesses and questions below. Table, figure and section numbers refer to the revised paper.

AuthorsOpenReview ↗

W1 / Q1. Novelty with respect to attention-based registration and feature aggregation, in particular COVER.

Soft registration mechanism for ICS. In summary, our primary contribution is reformulating local cross-attention as a feature-space spatial registration mechanism between reference and target features for ICS (corresponding to the query and context examples in in-context segmentation). Although this connection has been noted in prior work, it has not yet been applied to general in-context segmentation (ICS) or medical image segmentation. Furthermore, both our experimental results (Table 2) and analytical derivations (Appendix B) demonstrate that explicitly supervising this registration behavior via an auxiliary loss produces a strong synergy with the primary ICS objective. Consequently, registration serves not as an end goal, but as an effective auxiliary supervisory signal that improves overall segmentation performance.

Positioning to literature. We detail the difference of OmniSeg with regard to other papers combining registration and segmentation for medical image analysis in our answer to Reviewer ewVF. Regarding precisely COVER (He & Li, 2025), which Sec. 2 cites as the inspiration for our design, it differs from OmniSeg in three respects:

  • Task and supervision. COVER is a representation-learning method: it supervises displacement fields between two augmented views of the same image, where the ground-truth displacement is known from the augmentation. Thus, COVER’s contribution is mainly a self-supervision scheme that can be used with different vision encoder backbones. OmniSeg registers different images (context examples onto the query, across patients and acquisition sites), where the target DVF is unknown. We therefore supervise it with a mask-based proxy loss (Eq. (7)–(8)) and derive its contrastive effect on the features (Eq. (9), App. B).
  • Role of the DVF. In COVER, the displacement field is a pretext task for learning features. In OmniSeg, it is part of the segmentation model: it is composed across decoder levels (Eq. (18)) and used in the forward pass to warp the context skip features (Eq. (15)), and the same attention weights context examples pixel by pixel (DCP, Eq. (4)–(5)).
  • Architecture. Another key difference is that the models trained via COVER’s scheme ultimately do not perform registration. While DVFs are computed from the features for the SSL supervision, these models do not register the feature maps by themselves contrary to OmniSeg. Indeed, SpotBlock's SRM re-embeds context features based on the spatial attention that is further supervised via the registration loss. Moreover, registering a query against S context examples within an ICS model required adapting the encoder (two separate branches for query and context), the decoder (SpotBlocks at every level) and the skip connections (registered context features), as described in App. A.

We have revised the related-work section (Sec. 2, in blue) to position OmniSeg more clearly with respect to registration-based few-shot segmentation, COVER, and recent ICS works addressing misalignment at test time.

AuthorsOpenReview ↗

W2 / Q2 / Q3. Benefit of registration supervision & evidence that alignment drives the segmentation gains.

The small gain in Table 2 reflects an overlap between registration supervision and DCP, not a lack of effect, and the two remain complementary where registration is hardest. The ablation of Table 2 is incremental (SRM, then DCP, then the registration loss): each row measures a component given those added before it, so the registration loss is only evaluated on top of DCP. To isolate its contribution, we ran a new experiment: SRM trained with the deep registration loss but without DCP (mean Dice over the 10 held-out datasets, 99% CI $\pm$0.1), now added to Table 2 and Table 11.

Without registration lossWith registration loss
SRM73.3 ("SRM", Table 2)77.9 (new)
SRM + DCP77.1 ("SRM+DCP", Table 2)77.9 ("Ours", Table 2)

Registration supervision has a substantial effect on its own. As shown in the submitted paper, SRM already learns to register without supervision: it raises the Alignment Dice from 21.8 to 41.2 and improves Dice over MultiverSeg retrained in the same setting (73.3 vs. 72.0, Table 2). In the new experiment, with the registration loss and without DCP, SRM reaches 77.9 Dice, i.e., +4.6 over unsupervised SRM and +5.9 over the retrained MultiverSeg.

On average, DCP and registration supervision overlap. Without supervision, DCP compensates for poor alignment by discarding misaligned context examples. With supervision, SRM registers most of these examples, so DCP has less left to correct. This may explain why their gains do not add up on the average Dice score. Where SRM partially fails, the combination is needed. The new experiment shows that adding DCP on top of supervised SRM on autoPET brings large gains. This is because lesions can lie anywhere in the body, so query–context misalignment often exceeds SRM's registration range (updated Table 11, Sec. 4.2):

SRMSRM+DCPSRM+Reg. LossSRM+DCP+Reg. Loss (Ours)
autoPET Dice23.524.124.6 (new)28.9

On autoPET, neither DCP nor registration supervision helps much on its own (+0.6 and +1.1 Dice over SRM), while their combination brings +5.4 Dice, i.e., +4.3 over supervised SRM alone. When SRM cannot register a context example, DCP down-weights it, and the registration loss makes the similarity used by DCP more reliable. The full combination therefore keeps the mean performance of supervised SRM while being more robust in the most misaligned cases, which is why OmniSeg combines all three components.

Feature spatial coherence for better segmentation. The relation between an improved query/context alignment and stronger segmentation performance is also supported by the impact of warping the skip connections. In the submitted paper, removing the warping of context skip features by the DVF (Eq. (15)) decreases Dice from 77.1 to 71.6 ("w/o Skip Co. Registration", Table 11): without it, the skip and decoder activations live on mismatched spatial grids.

OmniSeg's advantage grows with misalignment. In Fig. 1(b), already in the initially submitted paper, the gap with MultiverSeg increases steadily with query–context misalignment, reaching 14.2 Dice points in the most misaligned bin. As suggested by the reviewer, we performed an additional controlled displacement experiment. We extract center crops around the target and translate the query crop by an offset relative to the target's bounding box size (Sec. 4.3 and App. E.6). At the largest offset, MultiverSeg loses 57.4 Dice points while OmniSeg loses only 5.8, and the gain of the registration loss over SRM+DCP grows with the offset from +0.3 Dice points at 0% to +1.5 Dice points at 75% on AMOS.

Table (new): Robustness to query offsets on the (CT, axial, left kidney) task of AMOS (S=16)

OffsetInitial (Align. Dice)MultiverSeg (Dice)SRM+DCP (Dice)SRM+DCP (Align. Dice)SRM+DCP+Deep Reg. Loss (Dice)SRM+DCP+Deep Reg. Loss (Align. Dice)
0%82.995.7 $\pm$ 0.395.9 $\pm$ 0.482.5 $\pm$ 0.696.2 $\pm$ 0.392.1 $\pm$ 0.4
25%59.694.9 $\pm$ 0.495.6 $\pm$ 0.480.0 $\pm$ 0.696.0 $\pm$ 0.390.5 $\pm$ 0.5
50%34.278.3 $\pm$ 0.994.1 $\pm$ 0.573.0 $\pm$ 0.695.1 $\pm$ 0.483.6 $\pm$ 0.6
75%11.838.3 $\pm$ 1.188.9 $\pm$ 0.662.5 $\pm$ 0.690.4 $\pm$ 0.770.3 $\pm$ 0.9
Reviewer_pxUDOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary
  • The paper addresses spatial misalignment between query images and annotated context examples, which limits the generalization of existing in-context medical image segmentation methods.
  • It proposes OmniSeg, introducing SpotBlock to combine local spatial attention with context pooling, alongside registration-based deep supervision to improve query-context alignment.
  • Evaluations across eight imaging modalities and ten held-out datasets demonstrate strong generalization, with an average improvement of 9.8 Dice points over the best baseline.
Critical strengths
  • The paper addresses spatial misalignment in in-context medical image segmentation through a well-designed combination of local attention, context pooling, and registration-based supervision.
  • The authors conduct large-scale experiments across eight imaging modalities and ten held-out datasets, demonstrating substantial improvements over existing methods, particularly under spatial misalignment and domain shifts.
Critical weaknesses
  • Although registration-based supervision is a central contribution, Table 2 shows that adding the deep registration loss improves segmentation Dice only from 77.1% to 77.9%, despite a larger improvement in alignment accuracy. This raises questions about whether explicit registration supervision provides substantial benefits beyond the proposed attention architecture.
  • OmniSeg is trained on a large-scale dataset containing approximately 1.8 million image-mask pairs, whereas several baselines in the main comparison use their original pretrained weights. Although the authors retrain MultiverSeg for the ablation studies, it remains difficult to fully separate the benefits of the proposed architecture from those of the training data and computational budget. Moreover, OmniSeg introduces additional inference overhead.
Questions for authors
  • Can the authors compare the registration loss with simpler feature-consistency or contrastive alignment objectives to demonstrate its unique contribution?
  • Can the authors provide more comprehensive comparisons with baselines trained under matched data and computational budgets, and clarify the performance–efficiency trade-off?
5 comments

We thank the reviewer for the positive assessment, and for acknowledging the design of our architecture and the effort put into building a large-scale experimental setting. We address the weaknesses and questions below. Table, figure and section numbers refer to the revised paper.

AuthorsOpenReview ↗

W1. Benefit of the registration supervision

The small gain in Table 2 reflects an overlap between registration supervision and DCP, not a lack of effect, and the two remain complementary where registration is hardest. The ablation of Table 2 is incremental (SRM, then DCP, then the registration loss): each row measures a component given those added before it, so the registration loss is only evaluated on top of DCP. To isolate its contribution, we ran a new experiment: SRM trained with the deep registration loss but without DCP (mean Dice over the 10 held-out datasets, 99% CI $\pm$0.1), now added to Table 2 and Table 11 of the revised manuscript.

Without registration lossWith registration loss
SRM73.3 ("SRM", Table 2)77.9 (new)
SRM + DCP77.1 ("SRM+DCP", Table 2)77.9 ("Ours", Table 2)

Registration supervision has a substantial effect on its own. As shown in the submitted paper, SRM already learns to register without supervision: it raises the Alignment Dice from 21.8 to 41.2 and improves Dice over MultiverSeg retrained in the same setting (73.3 vs. 72.0, Table 2). In the new experiment, with the registration loss and without DCP, SRM reaches 77.9 Dice, i.e., +4.6 over unsupervised SRM and +5.9 over the retrained MultiverSeg.

On average, DCP and registration supervision overlap. Without supervision, DCP compensates for poor alignment by discarding misaligned context examples. With supervision, SRM registers most of these examples, so DCP has less left to correct. This may explain why their gains do not add up on the average Dice score. Where SRM partially fails, the combination is needed. The new experiment shows that adding DCP on top of supervised SRM on autoPET brings large gains. This is because lesions can lie anywhere in the body, so query–context misalignment often exceeds SRM's registration range (updated Table 11, Sec. 4.2):

SRMSRM+DCPSRM+Reg. LossSRM+DCP+Reg. Loss (Ours)
autoPET Dice23.524.124.6 (new)28.9

On autoPET, neither DCP nor registration supervision helps much on its own (+0.6 and +1.1 Dice over SRM), while their combination brings +5.4 Dice, i.e., +4.3 over supervised SRM alone. When SRM cannot register a context example, DCP down-weights it, and the registration loss makes the similarity used by DCP more reliable. The full combination therefore keeps the mean performance of supervised SRM while being more robust in the most misaligned cases, which is why OmniSeg combines all three components.

Feature spatial coherence for better segmentation. The relation between an improved query/context alignment and stronger segmentation performance is also supported by the impact of warping the skip connections. In the submitted paper, removing the warping of context skip features by the DVF (Eq. (15)) decreases Dice from 77.1 to 71.6 ("w/o Skip Co. Registration", Table 11): without it, the skip and decoder activations live on mismatched spatial grids.

Moreover, we performed an additional controlled displacement experiment as suggested by Reviewer 2gyk. It shows that the positive impact of the registration supervision grows when the misalignment widens (see our answer to Reviewer 2gyk and App. E.6).

AuthorsOpenReview ↗

Q1. Comparison to feature-consistency/contrastive alignment objectives

Our registration loss is itself a dense contrastive objective on the features. Since the DVF is computed in closed form from the query–context attention (Eq. (3)), the gradient of $\mathcal{L}_{\text{align}}$ flows directly into the features. Its analytical form (Eq. (9), App. B) pulls query foreground features toward context foreground features and pushes them away from context background features, and conversely. The positives and negatives are defined by the masks and evaluated at registered locations.

Direct feature or image alignment raises specific issues. A loss aligning warped context features with query features, both being learned, admits a trivial collapsed solution unless negatives or a stop-gradient are introduced. Image-level similarities (NCC, MI) are ill-posed in our setting: training spans eight modalities and includes task augmentations that flip intensities and replace labels with Sobel edges (Table 10b). Moreover, with SpotBlock’s current implementation, query and context features do not share the same dimension after the SRM block.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Authors’ Answers and Additional Experiments

We thank the reviewers for their constructive feedback. Below, we summarize the strengths they highlighted and our answers to their shared concerns. Details are in the individual responses, and changes are in blue in the revised paper.

Strengths. Reviewers judged query–context misalignment in ICS a meaningful problem (pxUD, 2gyk) and agreed that registration-driven alignment is a reasonable direction (ewVF). They found our method well designed (pxUD, 2gyk) to tackle the issue and have all highlighted the scale of our evaluation (8 modalities, 10 held-out datasets) and the strength of our results (pxUD, 2gyk, ewVF). Moreover, we are glad that they found the paper clear and easy to follow (2gyk, ewVF).

Main concerns.

1. Clarifying the contribution of the registration loss (pxUD, 2gyk, ewVF).

The impact of the registration is large; the initially submitted paper did not highlight it enough:

  • Table 2 is incremental, so the loss was only evaluated on top of DCP (+0.8 Dice). We added the ablation SRM + registration loss without DCP: supervision alone lifts SRM from 73.3 to 77.9 Dice, matching the full model (+5.9 over retrained MultiverSeg). Its smaller gain in the full model reflects an overlap with DCP. In combination with the registration loss, DCP still brings +4.8 on autoPET, +3.1 on BUID, two challenging tumor segmentation datasets.
  • Controlled misalignment (2gyk, ewVF). To better isolate the effect of spatial misalignment on performance, we added a new controlled displacement experiment (App. E.6). At the highest tested offset, MultiverSeg drops to 38.3 Dice while OmniSeg keeps 90.4. Moreover, OmniSeg performance decreases less with the registration loss than without it (-5.8 points vs -7 points).
  • Validating registration (ewVF). The DVF warps context skip features in the forward pass: removing this warping drops Dice from 77.1 to 71.6 (Table 11). Alignment Dice rises from 21.8 to 41.2 with SRM and 50.7 with the loss. We added per-level DVFs and deformation grids (App. E.10).
  • Feature-level loss (pxUD, ewVF). Our mask loss already acts as a feature alignment loss with mask-defined positives and negatives, whereas direct feature losses risk collapse.

2. Additional comparisons under matched data and inference budgets

  • Retrain more baselines on the same dataset (pxUD, ewVF). Table 1 of the submitted paper uses released weights, as ICS models are meant for off-the-shelf use. As reported in the initially submitted paper, the strongest baseline MultiverSeg retrained with our data and protocol already showed that the 9.8-point gain splits into +3.9 (training setup) and +5.9 (architecture). During the rebuttal period, we generalize the study to UniverSeg and Tyche, showing similar trends: OmniSeg stays more than 10 points above retrained UniverSeg and Tyche (Table 11, App. E.2).
  • Performance-efficiency analysis (pxUD): OmniSeg with S = 16 beats MultiverSeg with S = 64 by 1.9 Dice, while being 1.7× faster with 2.6× fewer GFLOPs (App. E.9).

3. Restating our Positioning (2gyk, ewVF).

We do not claim novelty for local attention or registration-based segmentation per se, but for bringing attention-based registration into ICS: DVFs are computed in closed form from the attention, used in the forward pass at every decoder level, and supervised end to end by a mask loss with a dense contrastive effect on features (App. B). We detail the differences with COVER (2gyk) and [1,2,3] (ewVF), which are not in-context models or use a single global affine alignment (revised Sec. 2).

Summary of revisions (in blue in the revised paper).

  • Sec. 2: the paragraph Local attention for registration is replaced by Registration and query–context alignment, which discusses [1,2,3] and recent ICS works on misalignment.
  • Fig. 4 and Sec. 4.3: additional experiments to highlight our model’s robustness to spatial misalignment.
  • Tab. 2 and Sec. 4.3: new ablation with SRM and the deep registration loss, without DCP.
  • Tab. 3 and Sec. 4.3: new experiment to highlight our model’s robustness to outliers in the context set.
  • Conclusion: the limit case of a single, semantically unrelated context example is added (ewVF W2).
  • App. E.2: retrained baselines Tyche and UniverSeg with our training set-up and dataset, and new ablation with SRM and the deep registration loss, without DCP, with an analysis paragraph.
  • Tab. 14, 15 and App. E.6: segmentation performance in controlled displacement setting, extending the paragraph Robustness to misalignment in Sec. 4.3 and Fig. 1(b).
  • App. E.9 & Fig. 22: additional analysis about segmentation performance and inference overhead trade-off.
  • App. E.10 & Fig. 24: additional visualizations of residual DVFs and the associated deformation grid, extending Fig. 7.
AuthorsOpenReview ↗

Discussion summary (follow-up)

Following our rebuttal, reviewers 2gyk and ewVF indicated that their primary technical concerns were addressed. Additionally, reviewer 2gyk acknowledged our clarification regarding the novelty of our contribution relative to COVER [1].

To address reviewer ewVF’s follow-up remark about the performance of few-shot learning methods at scale, we launched an additional training experiment using an architecture suggested by the reviewer (HSNet [2]). Validation results show that standard few-shot architectures do not trivially scale to large, diverse medical imaging datasets. At the time of this message, HSNet has reached four epochs (~2.5 days) and the held-out validation Dice score plateaus around 45.95 Dice points, under OmniSeg (81.62 Dice points) but also other retrained baselines by a large margin at the same training stage. This observation is further supported by findings in UniverSeg [3], where standard few-shot methods underperformed on a comparable dataset by more than 20 Dice points. Combined with our ablation studies (Table 2 & 11), these results confirm that a substantial portion of OmniSeg's performance gain stems from our proposed fusion module, SpotBlock, and our deep registration loss.

Additional revisions:

  • Abstract & Reproducibility statement: Clarified our commitment to publicly releasing the model weights, along with the source code.
  • Table 2: Added the performance of the retrained versions of UniverSeg and Tyche (complementing Table 11).
  • (Planned) Table 1: Will report the performance of HSNet on the held-out test set upon completion of the 5-epoch training run.

[1] He, Yuting, and Shuo Li. "Vector contrastive learning for pixel-wise pretraining in medical vision." International Conference on Computer Vision, 2025.
[2] Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrelation Squeeze for Few-Shot Segmentation. International Conference on Computer Vision, 2021.
[3] Victor Ion Butoi*, Jose Javier Gonzalez Ortiz*, Tianyu Ma, Mert R. Sabuncu, John Guttag, and Adrian V. Dalca. UniverSeg: Universal Medical Image Segmentation. International Conference on Computer Vision, 2023

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.