- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a black-box diagnostic for measuring how strongly medical visual in-context learners depend on individual support image-label pairings and reports failure modes in which support-label properties influence query predictions. It proposes a Late Unpairing Curriculum that first learns from correctly paired supports and subsequently regularises correspondence dependence through random label reassignment. The reported experiments show substantial reductions in pairing gap alongside improved matched-support, held-out-task, and cross-dataset performance. However, it remains unclear whether LUC encourages selective use of meaningful correspondence or partly causes the model to ignore pair-specific support information.
Critical strengths
-
Novel and interesting diagnostic based on test-time derangement that isolates dependence on individual support image-label correspondences while preserving the query, support images, and label set.
-
The paper links pairing dependence to interpretable errors, including spurious regions, lesion-size bias, spatial copying, and sensitivity to misregistered support labels.
-
LUC requires no architectural changes, additional annotations, training steps, parameters, or inference-time cost.
-
Strong internal evaluation comprising paired-only, always-unpaired, and reversed-curriculum comparisons show that timing matters. Results across five task types, two backbones, held-out tasks, and cross-dataset episodes show the method’s effectiveness.
Critical weaknesses
-
Comparisons focus mainly on LUC schedule variants, omitting methods for noisy demonstrations and label-aware support selection, such as Cheng et al., Exploring the Robustness of In-Context Learning with Noisy Labels; Kang et al., In-Context Learning with Noisy Labels; and Luo et al., Love Me, Love My Label.
-
A near-zero pairing gap could indicate selective use of correspondence, but it could also mean that the model largely ignores pair-specific support information; query-only, label-only, and unrelated-support controls are missing.
-
Most evidence concerns brain MRI, while detailed failure-mode analyses are concentrated on one primary model, dataset, and tumour-segmentation setting.
-
Random unpairing permits label duplication and omission as well as breaking correspondence, but the paper lacks a permutation-preserving baseline that would isolate the effect of correspondence disruption.
Questions for authors
1- Why is LUC evaluated against its own schedule variants rather than established approaches for noisy-label ICL and label-aware prompt selection?
2- What evidence demonstrates that LUC encourages selective correspondence use rather than support neglect?
3- How broadly do the conclusions apply beyond the predominantly brain-MRI experiments and the single setting used for detailed failure analysis?
4- Since random unpairing changes both pair correspondence and label-set composition, how can the authors attribute the gains specifically to reduced correspondence dependence without a permutation-preserving control?
4 comments
Response to Reviewer Pjyk (2/2)
Q2 (selective use or support neglect; query-only, label-only and unrelated-support controls). We ran the three controls you list, plus one in which only the labels are removed (K=4, same queries and supports for all models). We use anatomical segmentation and modality translation, where one query admits several targets and the task can therefore only be read from the supports. The paired model and further details are in Tables R1 and R2 of the general response.
Table R15. LUC₀.₅ under the requested controls
| Task | Matched | Query-only | Label-only (support images blank) | Labels zeroed | Unrelated supports, vs. original target | Unrelated supports, vs. shown target |
|---|---|---|---|---|---|---|
| Anatomical segmentation (DSC) | 0.827 | 0.000 | 0.224 | 0.000 | 0.012 | 0.822 |
| Modality translation (PSNR, dB) | 21.64 | 6.77 | 15.07 | 6.98 | 18.40 | 20.79 |
For "unrelated supports", the supports show a different structure (or target contrast) from the one evaluated. The unprocessed input scores 17.99 dB on translation.
LUC₀.₅ fails from the query alone, fails without the support labels, loses most of its accuracy without the support images, and when the supports show a different task it does that task. Whole-tumor segmentation gives the same picture: 0.857 with matched supports, 0.201 label-only, 0.125 with supports from an unrelated task (T2→T1 translation pairs), and 0.003 query-only. A zero pairing gap therefore means that the prediction no longer depends on which label is attached to which image. It still depends on the support set.
We also see that LUC uses the label content wherever it defines the target. Besides Table R15, this holds for held-out targets that only exist through the support labels, e.g. both lateral ventricles at once (0.893 vs. 0.817 DSC, Table 10), and for tumor data in the UniverSeg study, where the supports specify which of four tumor regions to segment and LUC₀.₅ keeps the matched DSC (0.602 vs. 0.590) with a zero gap. Dropping the assignment did not cost accuracy with matched supports either: LUC₀.₅ is above the paired model on all five trained task types (Table 2).
Q3 (beyond brain MRI; failure analysis beyond one setting). We added experiments for both (G2 and G3 of the general response, Tables R3 to R6).
Other modalities. On 34 tasks in CT, abdominal MRI, ultrasound and chest X-ray, the released UniverSeg has pairing gaps of −0.344, −0.322, −0.110 and −0.195. When we train UniverSeg from scratch on these tasks (3 seeds), LUC₀.₅ brings the gap from −0.044 to −0.002, with matched DSC 0.566 vs. 0.579.
Failure modes on another model and on other data. On the same non-brain tasks, an 8-pixel shift of the support labels costs the released UniverSeg 0.40 / 0.43 / 0.11 / 0.28 DSC (CT / MRI / ultrasound / X-ray), and its injected-mask response is 0.82 / 0.73 / 0.37 / 0.66. After 40 epochs of fine-tuning with random unpairing, the shift costs 0.15 instead of 0.31 (34-task means; 0.28 after the same fine-tuning with matched supports), and the injected-mask response is 0.14 (0.22 with matched supports). For UniverSeg trained from scratch on BraTS, the same shift costs the paired model 0.502 DSC on whole tumor and edema, LUC₀.₅ 0.001 and LUC₀.₂ 0.015.
Presentation. We will add a small table at the beginning of Sec. 3 that defines the three support conditions (matched, shuffled, unpaired) and the two operators (test-time derangement, training-time random unpairing). We will also break up the sentences in Sec. 4 and 5 that carry several numbers at once.
We hope this addresses your concerns, and we are happy to run further checks during the discussion period.
rebuttal comments
The rebuttal addresses my main concerns. In particular, the query-only, label-only, labels-zeroed, and unrelated-support controls show that LUC continues to use the support set rather than simply ignoring pair-specific information, while the permutation-preserving control isolates correspondence disruption from label duplication or omission. The additional comparisons and non-brain evaluations also improve the paper. Some caveats remain because the competing methods are adapted counterparts and parts of the new evidence use only one seed, but these are secondary limitations. I'd therefore like to increase my score to 3.
Thank you for the updated assessment
Thank you for reading the rebuttal so carefully, and for deciding to raise your score to 3. We are glad the support-set controls and the permutation-preserving experiment resolved your main concerns. The new controls, comparisons and non-brain evaluations will be included in the revised paper.
One practical note: the rating on your original review still reads 2 on our side. We look forward to seeing the update there when you have a moment. Thank you again for your time.
Response to Reviewer Pjyk (1/2)
Thank you for the careful review. We ran the controls you asked for and compared LUC with the closest counterpart of each method family you mention.
Q1 (noisy-label ICL and label-aware prompt selection). Thank you for pointing us to these papers. They deal with demonstrations whose labels are wrong or that are poorly chosen. Kang et al. and Luo et al. select or down-weight demonstrations at inference, and Cheng et al. add label noise during training. Our setting differs in one respect: pairing dependence shows up with clean, valid supports, and the derangement does not alter any label. The three papers work on linear regression (Cheng et al.), text classification with language models (Kang et al.) and a learned retriever for 2D natural-image prompts (Luo et al.), so we implemented the closest counterpart of each on our setup (BraTS whole tumor, K=4).
Table R12. Training-side methods (protocol of the paper)
Label perturbation in every epoch is the strategy of Cheng et al. (noise as augmentation throughout training). Learned support weighting follows the idea of noisy-demonstration methods: a scorer is trained to detect supports whose label was moved and to down-weight them.
Table R13. Support selection at inference (pool of 80 labeled cases, 30 queries)
Label-aware retrieval picks the nearest images among the supports with a large labeled foreground. Table R13 is a separate evaluation, because the supports are drawn from the pool.
Both training-side alternatives close the gap, but their matched DSC stays at or below the paired model, and LUC is more than 0.12 DSC above them. Retrieval changes which supports are used and does not change how the paired model uses them: the gap does not shrink (−0.132 → −0.169 and −0.186) and the injected-mask response stays above 0.8, because a correct label on a different image still steers the paired model. We will add these comparisons and discuss the three papers in the related work.
Q4 (permutation-preserving control). Our test-time derangement is multiset-preserving: every label is kept and only moved to another image. For training, we ran the control you asked for. We trained LUC₀.₅ on UniverSeg with a derangement in place of random unpairing in the second half, so that the label set of every episode stays intact.
Table R14. UniverSeg: random unpairing vs. derangement in training
BraTS: four tumor regions; the three rows were re-evaluated together, which is why the first two differ slightly from Table 8 (0.590 / −0.222 and 0.602 / 0.000). Non-brain: 34 tasks; paired and random unpairing are means over 3 seeds, the derangement run is one seed.
The multiset-preserving version closes the gap just as well, and at the same matched DSC. On Neuroverse3D we have a related comparison from a shorter, 50-epoch recipe with a constant rate (not the recipe of Table R12). Deranging whole episodes at a rate of 25% / 75% leaves gaps of −0.014 / −0.001 DSC on BraTS and −0.124 / −0.065 dB on translation, while per-support random unpairing at the same rates leaves −0.038 / −0.015 DSC and −0.455 / −0.090 dB. So the gap closes because the correspondence is broken, and not because labels are repeated or dropped.