acceptodds
Under review as a conference paper at ICLR 2027

When to Unpair: Regulating Pairing Dependence in Medical Visual In-Context Learning

Abstract

Visual in-context learning (ICL), well suited to label-scarce medical imaging, uses support image–label pairs to demonstrate input–output mappings, while the labels collectively indicate the requested task. We diagnose dependence on individual pairings with a test-time derangement that reassigns every support label to another support image while preserving the query, support images, and label multiset. The resulting pairing gap, defined as shuffled-minus-matched performance, shows that all four released models depend on the pairing, to widely varying degrees. Further analysis of a paired-trained model reveals support-associated spurious regions and lesion-size biases even with real, unaltered supports, alongside sensitivity to mis-registered support labels. To regulate this dependence, we introduce a late unpairing curriculum (LUC), which starts with matched training and then applies random unpairing, replacing each support label with that of another support in the same episode. LUC nearly closes the pairing gap on two backbones while maintaining or improving matched-support performance across all evaluated task types, with gains extending to held-out tasks and cross-dataset episodes. It also mitigates these failure modes. On BraTS whole-tumor segmentation, matched-support DSC rises from to while the gap shrinks from to . In a released model, brief fine-tuning with random unpairing reduces the gap. A reversed curriculum that places the same number of unpairing epochs at the start of training leaves a large gap. This shows that pairing dependence is shaped by the order of training and not only by the amount of unpaired training.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual in-context learning, medical imaging, in-context learning, curriculum learning, few-shot learning, pairing dependence, dense prediction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PjykOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a black-box diagnostic for measuring how strongly medical visual in-context learners depend on individual support image-label pairings and reports failure modes in which support-label properties influence query predictions. It proposes a Late Unpairing Curriculum that first learns from correctly paired supports and subsequently regularises correspondence dependence through random label reassignment. The reported experiments show substantial reductions in pairing gap alongside improved matched-support, held-out-task, and cross-dataset performance. However, it remains unclear whether LUC encourages selective use of meaningful correspondence or partly causes the model to ignore pair-specific support information.

Critical strengths
  • Novel and interesting diagnostic based on test-time derangement that isolates dependence on individual support image-label correspondences while preserving the query, support images, and label set.

  • The paper links pairing dependence to interpretable errors, including spurious regions, lesion-size bias, spatial copying, and sensitivity to misregistered support labels.

  • LUC requires no architectural changes, additional annotations, training steps, parameters, or inference-time cost.

  • Strong internal evaluation comprising paired-only, always-unpaired, and reversed-curriculum comparisons show that timing matters. Results across five task types, two backbones, held-out tasks, and cross-dataset episodes show the method’s effectiveness.

Critical weaknesses
  • Comparisons focus mainly on LUC schedule variants, omitting methods for noisy demonstrations and label-aware support selection, such as Cheng et al., Exploring the Robustness of In-Context Learning with Noisy Labels; Kang et al., In-Context Learning with Noisy Labels; and Luo et al., Love Me, Love My Label.

  • A near-zero pairing gap could indicate selective use of correspondence, but it could also mean that the model largely ignores pair-specific support information; query-only, label-only, and unrelated-support controls are missing.

  • Most evidence concerns brain MRI, while detailed failure-mode analyses are concentrated on one primary model, dataset, and tumour-segmentation setting.

  • Random unpairing permits label duplication and omission as well as breaking correspondence, but the paper lacks a permutation-preserving baseline that would isolate the effect of correspondence disruption.

Questions for authors

1- Why is LUC evaluated against its own schedule variants rather than established approaches for noisy-label ICL and label-aware prompt selection?

2- What evidence demonstrates that LUC encourages selective correspondence use rather than support neglect?

3- How broadly do the conclusions apply beyond the predominantly brain-MRI experiments and the single setting used for detailed failure analysis?

4- Since random unpairing changes both pair correspondence and label-set composition, how can the authors attribute the gains specifically to reduced correspondence dependence without a permutation-preserving control?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Pjyk (1/2)

Thank you for the careful review. We ran the controls you asked for and compared LUC with the closest counterpart of each method family you mention.

Q1 (noisy-label ICL and label-aware prompt selection). Thank you for pointing us to these papers. They deal with demonstrations whose labels are wrong or that are poorly chosen. Kang et al. and Luo et al. select or down-weight demonstrations at inference, and Cheng et al. add label noise during training. Our setting differs in one respect: pairing dependence shows up with clean, valid supports, and the derangement does not alter any label. The three papers work on linear regression (Cheng et al.), text classification with language models (Kang et al.) and a learned retriever for 2D natural-image prompts (Luo et al.), so we implemented the closest counterpart of each on our setup (BraTS whole tumor, K=4).

Table R12. Training-side methods (protocol of the paper)

MethodMatched DSCGap
Paired0.733−0.184
Label perturbation in every epoch, 25% of supports0.731−0.025
Label perturbation in every epoch, 50%0.724−0.002
Label perturbation in every epoch, 75%0.724+0.003
Label perturbation in every epoch, 100% (our unpaired model)0.611+0.109
Learned support weighting0.709−0.013
LUC₀.₅0.857−0.008

Label perturbation in every epoch is the strategy of Cheng et al. (noise as augmentation throughout training). Learned support weighting follows the idea of noisy-demonstration methods: a scorer is trained to detect supports whose label was moved and to down-weight them.

Table R13. Support selection at inference (pool of 80 labeled cases, 30 queries)

Model and selectionMatched DSCGapInjected-mask response
Paired, random supports0.739−0.1320.82
Paired, nearest by image similarity0.743−0.1690.83
Paired, label-aware retrieval0.732−0.1860.81
LUC₀.₅, random supports0.875−0.0060.16

Label-aware retrieval picks the nearest images among the supports with a large labeled foreground. Table R13 is a separate evaluation, because the supports are drawn from the pool.

Both training-side alternatives close the gap, but their matched DSC stays at or below the paired model, and LUC is more than 0.12 DSC above them. Retrieval changes which supports are used and does not change how the paired model uses them: the gap does not shrink (−0.132 → −0.169 and −0.186) and the injected-mask response stays above 0.8, because a correct label on a different image still steers the paired model. We will add these comparisons and discuss the three papers in the related work.

Q4 (permutation-preserving control). Our test-time derangement is multiset-preserving: every label is kept and only moved to another image. For training, we ran the control you asked for. We trained LUC₀.₅ on UniverSeg with a derangement in place of random unpairing in the second half, so that the label set of every episode stays intact.

Table R14. UniverSeg: random unpairing vs. derangement in training

ModelBraTS, matched DSCBraTS, gapNon-brain, matched DSCNon-brain, gap
Paired0.593−0.2300.579−0.044
LUC₀.₅, random unpairing0.6030.0000.566−0.002
LUC₀.₅, derangement0.591−0.0010.577−0.002

BraTS: four tumor regions; the three rows were re-evaluated together, which is why the first two differ slightly from Table 8 (0.590 / −0.222 and 0.602 / 0.000). Non-brain: 34 tasks; paired and random unpairing are means over 3 seeds, the derangement run is one seed.

The multiset-preserving version closes the gap just as well, and at the same matched DSC. On Neuroverse3D we have a related comparison from a shorter, 50-epoch recipe with a constant rate (not the recipe of Table R12). Deranging whole episodes at a rate of 25% / 75% leaves gaps of −0.014 / −0.001 DSC on BraTS and −0.124 / −0.065 dB on translation, while per-support random unpairing at the same rates leaves −0.038 / −0.015 DSC and −0.455 / −0.090 dB. So the gap closes because the correspondence is broken, and not because labels are repeated or dropped.

AuthorsOpenReview ↗

Response to Reviewer Pjyk (2/2)

Q2 (selective use or support neglect; query-only, label-only and unrelated-support controls). We ran the three controls you list, plus one in which only the labels are removed (K=4, same queries and supports for all models). We use anatomical segmentation and modality translation, where one query admits several targets and the task can therefore only be read from the supports. The paired model and further details are in Tables R1 and R2 of the general response.

Table R15. LUC₀.₅ under the requested controls

TaskMatchedQuery-onlyLabel-only (support images blank)Labels zeroedUnrelated supports, vs. original targetUnrelated supports, vs. shown target
Anatomical segmentation (DSC)0.8270.0000.2240.0000.0120.822
Modality translation (PSNR, dB)21.646.7715.076.9818.4020.79

For "unrelated supports", the supports show a different structure (or target contrast) from the one evaluated. The unprocessed input scores 17.99 dB on translation.

LUC₀.₅ fails from the query alone, fails without the support labels, loses most of its accuracy without the support images, and when the supports show a different task it does that task. Whole-tumor segmentation gives the same picture: 0.857 with matched supports, 0.201 label-only, 0.125 with supports from an unrelated task (T2→T1 translation pairs), and 0.003 query-only. A zero pairing gap therefore means that the prediction no longer depends on which label is attached to which image. It still depends on the support set.

We also see that LUC uses the label content wherever it defines the target. Besides Table R15, this holds for held-out targets that only exist through the support labels, e.g. both lateral ventricles at once (0.893 vs. 0.817 DSC, Table 10), and for tumor data in the UniverSeg study, where the supports specify which of four tumor regions to segment and LUC₀.₅ keeps the matched DSC (0.602 vs. 0.590) with a zero gap. Dropping the assignment did not cost accuracy with matched supports either: LUC₀.₅ is above the paired model on all five trained task types (Table 2).

Q3 (beyond brain MRI; failure analysis beyond one setting). We added experiments for both (G2 and G3 of the general response, Tables R3 to R6).

Other modalities. On 34 tasks in CT, abdominal MRI, ultrasound and chest X-ray, the released UniverSeg has pairing gaps of −0.344, −0.322, −0.110 and −0.195. When we train UniverSeg from scratch on these tasks (3 seeds), LUC₀.₅ brings the gap from −0.044 to −0.002, with matched DSC 0.566 vs. 0.579.

Failure modes on another model and on other data. On the same non-brain tasks, an 8-pixel shift of the support labels costs the released UniverSeg 0.40 / 0.43 / 0.11 / 0.28 DSC (CT / MRI / ultrasound / X-ray), and its injected-mask response is 0.82 / 0.73 / 0.37 / 0.66. After 40 epochs of fine-tuning with random unpairing, the shift costs 0.15 instead of 0.31 (34-task means; 0.28 after the same fine-tuning with matched supports), and the injected-mask response is 0.14 (0.22 with matched supports). For UniverSeg trained from scratch on BraTS, the same shift costs the paired model 0.502 DSC on whole tumor and edema, LUC₀.₅ 0.001 and LUC₀.₂ 0.015.

Presentation. We will add a small table at the beginning of Sec. 3 that defines the three support conditions (matched, shuffled, unpaired) and the two operators (test-time derangement, training-time random unpairing). We will also break up the sentences in Sec. 4 and 5 that carry several numbers at once.

We hope this addresses your concerns, and we are happy to run further checks during the discussion period.

Reviewer_PjykOpenReview ↗

rebuttal comments

The rebuttal addresses my main concerns. In particular, the query-only, label-only, labels-zeroed, and unrelated-support controls show that LUC continues to use the support set rather than simply ignoring pair-specific information, while the permutation-preserving control isolates correspondence disruption from label duplication or omission. The additional comparisons and non-brain evaluations also improve the paper. Some caveats remain because the competing methods are adapted counterparts and parts of the new evidence use only one seed, but these are secondary limitations. I'd therefore like to increase my score to 3.

AuthorsOpenReview ↗

Thank you for the updated assessment

Thank you for reading the rebuttal so carefully, and for deciding to raise your score to 3. We are glad the support-set controls and the permutation-preserving experiment resolved your main concerns. The new controls, comparisons and non-brain evaluations will be included in the revised paper.

One practical note: the rating on your original review still reads 2 on our side. We look forward to seeing the update there when you have a moment. Thank you again for your time.

Reviewer_4W4jOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates the problem of excessive reliance on the pairing relationship between reference images and labels in medical visual ICL, and proposes to quantify this reliance by randomly shuffling support labels at test time. They also propose Late Unpairing Curriculum (LUC), which first trains with correctly paired examples and then randomly unpairs them in the later stage of training, in order to reduce the model's reliance on individual reference labels. The experiments cover tasks such as tumor segmentation, anatomical structure segmentation, modality transfer, image restoration, and bias field correction. The results show that LUC can significantly narrow the pairing gap while maintaining or improving performance under normal pairing.

Critical strengths
  1. The exploration angle of the paper is very interesting. Current models may rely too much on image-label pairings, thereby neglecting the processing of information from the images themselves. By shuffling labels, the authors force the model to pay more attention to the images. The method is simple and effective.
  2. The paper's narrative is reasonable, the problem definition is very clear, and the analysis of the solution is also fairly thorough.
  3. The experiments cover multiple tasks and model architectures, demonstrating the generalization of the method.
Critical weaknesses
  1. Does random unpairing truly improve the model's ICL capability? After shuffling the pairings between support images and labels, the model's performance actually improves. But is this because the model makes better use of the query image, or because it relies less on the support information? I am curious about the gap between the model's performance in the query-only setting and the pairing proposed by the method.
  2. Does the model still rely on the visual features of the reference images? Although random unpairing reduces the model's reliance on specific image-label pairings, the reference images themselves are unchanged. Does the model still use the visual features in the reference images, or does it gradually ignore the reference images and rely more on the query itself? I think it is good to further analyze the model's sensitivity to the content of the reference images.
Questions for authors

Please refer to the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 4W4j

Thank you for the positive assessment and for the two questions, which led us to run controls that the paper did not have. The short answer to both is that LUC makes better use of the query and still needs the supports.

W1 (better use of the query, or less reliance on the supports? The query-only setting). We ran the query-only setting you asked about: support images and labels are blank, so the model only has the query (K=4, same queries as with matched supports).

Table R10. Query-only vs. matched supports

TaskLUC₀.₅, query-onlyLUC₀.₅, matchedPaired, query-onlyPaired, matched
Whole tumor (DSC)0.0030.8570.0160.732
Anatomical segmentation (DSC)0.0000.8270.0000.789
Modality translation (PSNR, dB)6.7721.6411.7521.56

(The unprocessed input scores 17.99 dB on translation.) LUC cannot do any of these tasks from the query alone, so its score comes entirely with the supports. We also removed or changed only the support labels, on anatomical segmentation and translation, where the same query can be asked for several targets (Tables R1 and R2 in the general response). With zeroed labels LUC₀.₅ drops to 0.000 DSC and 6.98 dB. When the supports show another structure, it segments that structure (0.822 DSC) and not the original one (0.012). Its gain therefore does not come from relying less on the supports.

What changes with LUC is how the supports and the query are combined. The model takes from the supports what to produce, and from the query where the answer is. When we give it a support set whose tumor lies elsewhere, its predicted tumor centroid stays near the query tumor (median 0.19 of the way to the support tumor, compared with 0.66 for the paired model). A mask painted into every support label raises its prediction inside that mask by 0.095, compared with 0.846 for the paired model (Sec. 5.2).

On your broader question of whether unpairing really improves ICL: we think the tasks outside training are the most direct test, since there the model can only know the task from the supports. These are segmentation targets that combine structures, new translation directions, and episodes whose supports come from another dataset. Fig. 5 and Table 10 report 46 such tasks on which both LUC models are significantly better than the paired model. One example is segmenting both lateral ventricles at once, a target that only exists through the support labels, where LUC₀.₅ reaches 0.893 DSC and the paired model 0.817. LUC also improves with more supports (0.786 DSC at K=1, 0.870 at K=8).

W2 (does LUC still use the visual features of the support images?). Yes. Here we kept the support labels and blanked only the support images.

Table R11. Blank support images

TaskLUC₀.₅, matchedLUC₀.₅, images blankPaired, matchedPaired, images blank
Whole tumor (DSC)0.8570.2010.7320.100
Anatomical segmentation (DSC)0.8270.2240.7890.329
Modality translation (PSNR, dB)21.6415.0721.5613.44

Without the support images LUC₀.₅ loses 0.66 and 0.60 DSC on the two segmentation tasks, and on translation it falls below the unprocessed input (15.07 vs. 17.99 dB). The paired model behaves similarly. If the support images come from an unrelated task (T2→T1 translation pairs), whole-tumor DSC is 0.125. So LUC does read the content of the support images.

We will add these controls to the paper. Thank you again for suggesting them.

Thank you for the reply! Most of my concerns are well-addressed so I am happy to maintain my positive rating.

AuthorsOpenReview ↗

Follow-up to reviewer 4W4j

Thank you for the follow-up and for the positive assessment. The two questions in your review led to controls that the paper did not have before. They are now in the revised version we uploaded (Appendix H, with a summary paragraph in Sec. 5.1; changes are in blue). We appreciate the time you spent on our paper!

Reviewer_3PFBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates pairing dependence in medical visual in‑context learning: models may heavily rely on correct image‑label correspondence within each support example, rather than only learning task semantics from the whole support set. The authors propose a test‑time derangement diagnostic to quantify this effect across four released medical ICL models. They observe harmful failure modes: predictions copying lesion location/size from support labels, sensitivity to mis‑registered support masks. To mitigate this issue, they introduce LUC train normally on matched pairs first, then switch to random label‑unpairing inside each training episode. Controlled experiments show LUC nearly eliminates pairing gap, preserves or even improves matched‑support performance, and reduces the above failure modes. Reversed‑schedule EUC ablations prove training ending‑phase dominates pairing dependence, not just total unpairing epochs. Short fine‑tuning with unpairing can also retrofit pre‑trained models. Evaluations cover segmentation, modality translation, inpainting and bias correction across multiple brain MRI cohorts, including held‑out and cross‑dataset episodes.

Critical strengths
  1. Test‑time label derangement and pairing‑gap metric offer a clean, zero‑extra‑annotation black‑box probe to evaluate visual ICL’s over‑reliance on per‑example pairing.
  2. LUC curriculum requires no architecture changes, only modifying training episode label assignments. It suppresses pairing dependence without hurting (and often lifting) matched‑support performance.
  3. Rigorous comparison with reversed‑curriculum EUC, full‑unpaired and fully‑paired baselines, clearly demonstrates the critical role of training ending phase. Short fine‑tuning on off‑the‑shelf model further validates practical usability.
  4. Validated across segmentation, translation, inpainting, bias correction, on in‑distribution, held‑out tasks and cross‑dataset settings, with multiple perturbation probes (mask shift, injected artefact mask etc.) to dissect failure modes.
  5. Fixed hyper‑parameter setup across all model variants, open code, detailed appendix for all perturbation protocols and statistics.
Critical weaknesses
  1. All data are brain MR; it is unclear whether LUC’s benefits transfer to other medical modalities (ultrasound, CT, X‑ray). 2.Main controlled experiments are built on Neuroverse3D; only one small supplementary test on UniverSeg. Generalization to other popular ICL backbones needs more evidence.
  2. The paper demonstrates what changes under LUC but provides limited inside‑model analysis (e.g. attention/feature representation shifts) explaining why late‑phase unpairing alters pairing dependence. 4.Probes simulate mis‑registration / corruption, but real‑world support‑label noise patterns in clinical practice are more diverse and not exhaustively tested.
  3. How pairing gap and LUC effectiveness change under different numbers of support examples is underexplored.
Questions for authors
  1. All evaluations are confined to brain MRI datasets. Could you discuss whether you expect LUC to generalize to other modalities such as CT or ultrasound? Would you consider small‑scale supplementary tests on non‑brain medical data?
  2. You show limited result on UniverSeg. Can you add more evidence on whether the LUC effect holds across diverse ICL architectures beyond Neuroverse3D?
  3. Could you provide some internal model analysis (attention/feature statistics) to help explain why unpairing applied in the final training phase reshapes pairing dependence more strongly than early‑phase unpairing?
  4. Real‑world clinical support labels suffer diverse errors besides mis‑registration and injected masks. Could you comment on how LUC would cope with more realistic label noise e.g. partial annotation omission, inter‑rater label disagreement?
  5. How does support‑set size K influence pairing gap and LUC performance? Have you swept over different K values to test robustness?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 3PFB (1/2)

Thank you for the detailed review and for the concrete suggestions. We ran a new experiment for each of your five questions.

Q1 (other modalities). We followed your suggestion and ran supplementary tests on non-brain data: 34 segmentation tasks in abdominal CT, abdominal MRI, ultrasound and chest X-ray, all from public datasets. The details and Tables R3 to R5 are in G3 of the general response. In brief:

  1. The released UniverSeg has a pairing gap in every modality: −0.344 (CT), −0.322 (abdominal MRI), −0.110 (ultrasound) and −0.195 (X-ray). It also reproduces a mask that we paint into the support labels (injected-mask response between 0.37 and 0.82).
  2. When UniverSeg is trained from scratch on these tasks (3 seeds), LUC₀.₅ brings the gap from −0.044 ± 0.017 to −0.002 ± 0.001, with a matched DSC of 0.566 vs. 0.579 for the paired model. On the nine tasks where the paired gap is below −0.05, the gap goes from −0.120 to −0.004.
  3. Fine-tuning the released UniverSeg for 40 epochs with random unpairing brings its gap from −0.256 to −0.025, while the same fine-tuning with matched supports leaves it at −0.227 (matched DSC 0.524 vs. 0.548). The DSC lost under an 8-pixel shift of the support labels is halved (0.31 → 0.15).

So the dependence is present in CT, abdominal MRI, ultrasound and X-ray, and LUC reduces it in each of them.

Q2 (architectures). We extended the UniverSeg study of the paper. These models are trained from scratch on BraTS in a multi-task setting where the supports specify which of four tumor regions to segment. Table 8 of the paper has the paired model and LUC₀.₅; the LUC₀.₂ row and the last column are new.

Table R6. UniverSeg (2D) trained from scratch on BraTS

ModelMatched DSCGapDSC drop, 8-pixel label shift
Paired0.590−0.2220.502
LUC₀.₅0.6020.0000.001
LUC₀.₂0.591−0.0010.015

The shift is evaluated on whole tumor and edema, the two tasks of Table 1. The paired UniverSeg shows the mis-registration failure of Sec. 4, and LUC removes it. Taken together, LUC is now trained from scratch on two architectures (3D with a mean over K, 2D with averaging blocks) on brain and non-brain data, and used as a short fine-tune on two released models (Neuroverse3D in Fig. 7, UniverSeg in Table R5). The derangement test covers four released models with four fusion mechanisms.

Q5 (support-set size K). We swept K from 1 to 8 on BraTS whole tumor (30 subjects; this is a separate evaluation, so K=4 differs slightly from Table 2).

Table R7. Matched DSC for different K

ModelK=1K=2K=3K=4K=6K=8
Paired0.5530.7000.7250.7340.7380.740
LUC₀.₅0.7860.8460.8620.8560.8720.870
LUC₀.₂0.7610.8510.8730.8760.8890.886
EUC₀.₅0.5190.6790.7280.7410.7510.756

Table R8. Pairing gap for different K (a derangement needs K ≥ 2)

ModelK=2K=3K=4K=6K=8
Paired−0.259−0.185−0.151−0.104−0.096
LUC₀.₅−0.003−0.023−0.009−0.006−0.001
LUC₀.₂−0.001−0.024−0.006−0.005−0.005
EUC₀.₅−0.223−0.126−0.074−0.048−0.033

The paired gap is largest when there are few supports, which is the common situation when labels are scarce. Both LUC models stay within 0.024 of a zero gap and are more accurate than the paired model for every K.

AuthorsOpenReview ↗

Response to Reviewer 3PFB (2/2)

Q3 (why late unpairing changes the dependence more than early unpairing). Our explanation is that pairing dependence follows whatever the most recent phase of training rewarded. We have three pieces of evidence.

Training dynamics. We log the gap at every epoch (Fig. 6). After LUC₀.₅ switches to unpairing, the whole-tumor gap is within 0.02 of zero five epochs later, and on all five tracked tasks the gap shrinks by 71–97%. EUC₀.₅ behaves the other way round: its gap is about zero during the unpairing phase and opens again right after the switch to matched supports, about as fast as it closes in LUC (Appendix E). Early unpairing is thus overwritten by the matched training that comes after it, whereas late unpairing is what the final model ends with. The matched phase is still needed for accuracy: the model trained with unpaired supports throughout only reaches 0.611 matched DSC, compared with 0.857 for LUC.

Features. Neuroverse3D fuses the supports with an equal-weight mean (Eq. 1), so there are no attention maps to inspect, and we looked at the features. We computed linear CKA between LUC₀.₅ at 18 points during training and its final state, for the support pathway (the aggregated support features of Eq. 1) and for the query pathway (the bottleneck of the target encoder), using 60 episodes. The support pathway is already close to its final state at epoch 34, during the matched phase (CKA 0.84, and between 0.75 and 0.96 from then on). The query pathway only reaches its final state during the unpairing phase, which starts at epoch 75 (CKA 0.84 at epoch 104 and 0.97 from epoch 124). In other words, late unpairing is applied to a model whose support-reading features already exist. It keeps these features and changes the query pathway.

Outputs. For each query we chose a support set whose tumor is at a clearly different location, and projected the predicted tumor centroid onto the line from the query tumor (0) to the support tumor (1), on an 8 × 8 grid of queries and support sets. The median is 0.66 for the paired model, 0.68 for the model unpaired throughout, and 0.19 for LUC₀.₅. Unpairing from the start does not move the prediction back to the query, while unpairing after a matched phase does.

Q4 (realistic label errors). We tested partial annotation and inter-rater disagreement, as you suggested, together with a few other error types (BraTS whole tumor, 60 subjects, K=4). Unless stated otherwise, all four support labels carry the error.

Table R9. Whole-tumor DSC under support-label errors

Support-label errorPairedLUC₀.₅LUC₀.₂
None0.7370.8510.875
Inter-rater disagreement (random erosion or dilation, 1–3 voxels)0.7330.8480.873
Erosion, 1 voxel0.7370.8510.876
Erosion, 3 voxels0.7310.8490.875
Erosion, 5 voxels0.7060.8430.872
Dilation, 1 voxel0.7340.8510.872
Dilation, 3 voxels0.7250.8470.863
Dilation, 5 voxels0.7130.8400.854
Two labels eroded, two dilated (3 voxels)0.7320.8490.873
Partial annotation, half of each lesion unlabeled0.7220.8500.876
Partial annotation, every other slice labeled0.7370.8410.872
1 of 4 labels empty0.7310.8510.877
2 of 4 labels empty0.7110.8460.873
Mean DSC loss over the 12 error types0.0110.0040.004
Largest DSC loss0.0310.0110.021

On average LUC loses less DSC than the paired model (Wilcoxon over subjects, p = 0.002 for LUC₀.₅ and 0.001 for LUC₀.₂). Even its lowest score under any of these errors (0.840) is 0.10 above the paired model with clean labels. In the paper, Fig. 10 additionally shows replaced labels (0 to 4 of the 4 labels taken from another patient: paired 0.74 → 0.62, LUC 0.85 → 0.83) and an 8-voxel mis-registration (it costs the paired model 0.035 DSC and LUC 0.001). Because LUC does not take its answer from any single support label, an error in a label changes its prediction less.

We hope this addresses your concerns, and we are happy to run further checks during the discussion period.

Thank you to the authors for their detailed responses and extensive supplementary experiments, which have alleviated some of my concerns. However, the core generalization evidence and mechanism explanation are still insufficient, and I tend to maintain the original rating.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response (1/2)

We thank the three reviewers for their careful reading and constructive suggestions. During the rebuttal period we ran the additional experiments that were requested. Three points were raised by more than one reviewer, so we answer them here: whether LUC still uses the support set (G1), other architectures (G2), and data beyond brain MRI (G3). All other points are answered in the individual replies. Tables are numbered R1 to R15 across our replies.

G1. Does LUC still use the support set? (4W4j, Pjyk)

It does. We ran the controls the reviewers suggested on the models of the paper (K=4, same queries and supports for every model): a query-only setting, support labels removed, support images removed, and supports that ask for a different target. For the label and target controls we use anatomical segmentation and modality translation, since there one query admits several targets (ten structures on the same T1 scan, several target contrasts) and the task can only be read from the supports. These runs have their own subjects and support draws, so the matched scores differ a little from Table 2.

Table R1. Anatomical segmentation, DSC (24 subjects × 10 structures, three cohorts)

ModelMatchedShuffledQuery-onlyLabels zeroedImages blankOther structure, vs. shownOther structure, vs. original
Paired0.7890.7180.0000.0000.3290.7820.007
LUC₀.₅0.8270.8170.0000.0000.2240.8220.012

Table R2. Modality translation, PSNR in dB (20 subjects × 6 directions; the unprocessed input scores 17.99)

ModelMatchedShuffledQuery-onlyLabels zeroedImages blankOther contrast, vs. shownOther contrast, vs. original
Paired21.5620.3511.757.6013.4420.8218.44
LUC₀.₅21.6421.626.776.9815.0720.7918.40

Query-only: support images and labels are both blank. Other structure / contrast: the supports show a different structure (or target contrast) from the one evaluated, and we score the output against what the supports show and against the original target.

Without supports LUC₀.₅ cannot do either task (0.000 DSC, and 6.77 dB, which is below the unprocessed input). It also fails when only the labels are removed, it loses 0.60 DSC when only the support images are removed (the paired model loses 0.46), and when the supports show another structure it segments that structure (0.822) instead of the original one (0.012). We see the same on whole-tumor segmentation: 0.857 with matched supports, 0.201 with blank support images, 0.125 with supports from an unrelated task (T2→T1 translation pairs), and 0.003 query-only. LUC therefore still relies on the support set. What it no longer relies on is which label sits on which image. It also benefits from additional supports (0.786 DSC at K=1, 0.870 at K=8, Table R7).

G2. Other architectures (3PFB)

The derangement test is applied to four released models that fuse supports in four different ways (averaging, feature ensemble, mean over K, cross-attention), in 2D and 3D (Table 1). LUC itself is trained on two of them, Neuroverse3D (3D) and UniverSeg (2D). For UniverSeg the paper reports that the gap goes from −0.222 to 0.000 with LUC₀.₅ at a matched DSC of 0.602 vs. 0.590 (Table 8). We now add LUC₀.₂ (gap −0.001, matched DSC 0.591) and the failure analysis on this architecture: an 8-pixel shift of the support labels costs the paired UniverSeg 0.502 DSC on whole tumor and edema, compared with 0.001 for LUC₀.₅ and 0.015 for LUC₀.₂ (Table R6 in our reply to 3PFB). G3 adds UniverSeg experiments on four non-brain modalities, both trained from scratch and fine-tuned from the released weights.

AuthorsOpenReview ↗

General response (2/2)

G3. Beyond brain MRI (3PFB, Pjyk)

Pairing dependence itself is not tied to brain MRI. Three of the four released models in Table 1 were trained on broad, mostly non-brain collections (UniverSeg on 53 datasets, Medverse on 22 datasets covering several organs and modalities, SegGPT mainly on natural images), and their whole-tumor gaps are two to six times larger than that of the brain-only Neuroverse3D. To test LUC outside the brain, we put together a benchmark from public data with 34 segmentation tasks: abdominal CT (12 organs) and abdominal MRI (9 organs) from AMOS22, ultrasound (8 abdominal structures, fetal head, pubic symphysis) and chest X-ray (left lung, right lung, both lungs). It has 4,889 training slices and 1,453 held-out slices. We ran three experiments on it.

(a) The released UniverSeg, inference only, with 16 supports.

Table R3. Released UniverSeg on non-brain data

CTAbdominal MRIUltrasoundChest X-ray
Matched DSC0.5720.6060.3190.909
Pairing gap−0.344−0.322−0.110−0.195
Injected-mask response0.820.730.370.66
DSC drop, 8-pixel label shift0.400.430.110.28

The pairing gap and both failure modes show up in all four modalities. On CT, moving correct labels to other support images removes 60% of the released model's DSC.

(b) UniverSeg trained from scratch on these tasks, with the same recipe as the UniverSeg study in the paper.

Table R4. UniverSeg trained from scratch on non-brain data (3 seeds, mean ± s.d.)

ModelMatched DSC, 34 tasksGap, 34 tasksMatched DSC, 9 tasksGap, 9 tasks
Paired0.579 ± 0.005−0.044 ± 0.0170.725−0.120
LUC₀.₅0.566 ± 0.022−0.002 ± 0.0010.702−0.004
LUC₀.₂0.563 ± 0.005−0.003 ± 0.0040.716−0.008

The "9 tasks" are those on which the paired model has a gap below −0.05. LUC closes the gap in each modality where the paired model has one (CT −0.037 → −0.002, abdominal MRI −0.057 → −0.003, ultrasound −0.053 → −0.001), and matched DSC stays within 0.016 of the paired model. The tasks with the largest paired gap are ultrasound gallbladder (−0.292 → −0.008), MRI liver (−0.154 → −0.002) and CT spleen (−0.119 → 0.000).

(c) Fine-tuning the released UniverSeg on these tasks for 40 epochs, with random unpairing or, as a control, with matched supports. This mirrors the Neuroverse3D experiment of Fig. 7.

Table R5. Fine-tuning the released UniverSeg (means over the 34 tasks)

ModelMatched DSCGapInjected-mask responseDSC drop, 8-pixel label shift
Released0.536−0.2560.650.31
Fine-tuned, matched supports0.548−0.2270.220.28
Fine-tuned, random unpairing0.524−0.0250.140.15

A short fine-tuning run with unpairing removes about 90% of the released model's gap (CT −0.344 → −0.050, abdominal MRI −0.322 → −0.020, ultrasound −0.110 → −0.008, X-ray −0.195 → +0.001) and halves the DSC it loses under shifted labels. The control with matched supports keeps both.

Planned revision. We will add the results of this response and of the individual replies to the paper: the support controls, the non-brain experiments, the UniverSeg failure analysis, the sweep over K, the label-error tests, the multiset-preserving control, the comparison with noisy-label and retrieval methods (Cheng et al., Kang et al., Luo et al.), and the internal analysis.

Area_Chair_Sd4ROpenReview ↗

ICLR Review Discussion – Follow-up on Author Rebuttal

Dear Reviewers,

Thank you for your time and effort in reviewing this submission.

The authors have submitted their rebuttal. Could you please take a moment to review their responses and assess whether your concerns have been adequately addressed? If you have any remaining concerns or new questions, please feel free to raise them in the discussion.

If you have already responded to the authors, please kindly disregard this reminder.

Thank you again for your valuable contributions to the review process.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.