acceptodds
Under review as a conference paper at ICLR 2027

Open-Ontology Gap in Free-Text Medical Segmentation

Abstract

Free-text medical segmentation lets users request clinical targets in ordinary language, but a plausible mask may represent the wrong concept. Prior work reports category errors and limited generalization, but lacks a controlled test of whether a model selects the requested target when a related alternative is present. We introduce Clinical Contrast Sets, which holds the image fixed, changes the requested concept, and measures whether the mask shifts toward the intended region and away from its competitor. On 40 MS3SEG development patients, Medical-SAM3 and VoxTell preferentially select multiple sclerosis (MS) lesions when asked for the dataset’s normal/non-MS white-matter hyperintensities (nWMH). A protocol frozen before evaluating the 20 official-test patients confirms this pattern: mean target-minus-confounder coverage is −0.308 and −0.363, with 0/20 correct canonical prompt pairs for each model. Positive descriptions and volume controls preserve the failure, local size matching confirms it for VoxTell, and published supervised results establish that nWMH is learnable. The failure is not a missing label: VoxTell’s published training inventory lists both MS lesions and white matter hyperintensities, and the model switches correctly between documented kidney and tumor concepts, yet it resolves nWMH prompts, and even the generic white-matter-hyperintensity prompt, to MS lesions. Incorrect nWMH masks retain a mean foreground probability of 0.962, and a prompt-overlap warning that flags all 20 official VoxTell failures performs near chance within BraTS. This matters because the failure is not a missing answer but a wrong one: a clinician requesting a benign, non-MS finding instead receives a confident, well-formed mask of MS pathology, with nothing in the model’s own output to flag the substitution. We call this mismatch between language flexibility and reliable visual target selection the open-ontology gap, and offer Clinical Contrast Sets as a controlled protocol for auditing it: with at least two annotated competitors, two prompts, and a superclass reference, developers can identify which clinical distinctions a model does not reliably select and verify whether an adaptation restores correct switching rather than only raising overlap.

Primary area:
datasets and benchmarks
Keywords:
medical image segmentation, foundation models, text-prompted segmentation, evaluation benchmarks, visual grounding, vision-language models, contrast sets, multiple sclerosis MRI, semantic faithfulness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YyGWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether free-text medical segmentation models select the intended clinical concept when related targets coexist in the same image. It introduces Clinical Contrast Sets, which changes the text prompt while holding the image fixed and measures whether predictions switch toward the requested target. Experiments with Medical-SAM3 and VoxTell show that prompts requesting non-MS white-matter hyperintensities often select MS lesions instead. The study includes wording and size controls, anatomical and tumor contrasts, confidence analysis, and confirmation on a held-out 20-patient cohort.

Critical strengths
  1. Keeping the image fixed while changing the requested concept helps distinguish target-selection errors from generic segmentation inaccuracies. The wording, volume, and component-size controls strengthen the central observation.

  2. Independent confirmation of the observed failure. The official-test protocol was frozen before evaluation, and the 20-patient cohort reproduces the development-set pattern. Reporting unsuccessful mitigation and limitations of the failure detector also helps establish the boundaries of the findings.

Critical weaknesses
  1. Limited novelty and scientific insight. The paper carefully documents a concept-selection failure, but the resulting insight appears incremental relative to known limitations in semantic grounding, prompt sensitivity, and confident incorrect predictions. The controls strengthen the observation without establishing a generalizable mechanism or explaining which conditions determine successful concept selection. The contrast-set protocol is useful, but its methodological advance and the depth of the resulting discoveries remain limited.

  2. Narrow empirical scope and unresolved interpretation. The strongest failure evidence concerns one annotation-defined distinction in MS3SEG. The kidney and tumor experiments mainly establish successful control cases rather than independently replicating the failure across other clinical distinctions. Furthermore, the presence of “white matter hyperintensities” in VoxTell’s training inventory does not establish alignment with MS3SEG’s specific nWMH annotation convention. The current evidence therefore does not adequately distinguish a broader semantic-grounding limitation from a dataset-specific mismatch in concept definitions.

Questions for authors

See weakness

1 comment

Thank you for asking what is learned beyond "category errors occur" and which conditions determine selection. Revised text is blue and tagged [YyGW] (Section 5.7, p. 9; Table 16; Appendix P; Section 6; Limitations); Table 10 marks every new analysis as post hoc or exploratory.

W1: Insight, and which conditions determine success

1. A mask can respond strongly without selecting the requested target. VoxTell v1.1 changes 41% of the canonical masks' union when the concept changes (repeat-inference Dice 0.9997), yet the paired preference shift S is -0.033 [-0.050, -0.016] and no official patient has a positive nWMH direction. v1.0 shifts positively in 18/20 cases but still has 0/20 pairs. The protocol separates responsiveness, progress toward the target and preferential selection.

2. What separates success from failure. Kidney laterality: 40/40 pairs, S +1.90 (40/40 positive). BraTS: S +0.74 [0.59, 0.89] (38/40 positive), 19/40 pairs; 18 of 21 failures are enhancing-prompt masks that segment the enhancing tumor well (median Dice 0.97) but also cover the small embedded core (median 72 voxels): high enhancing-tumor Dice can conceal coverage of the competing non-enhancing core, which our pair criterion exposes. MS3SEG: S -0.033, 0/20. Six co-labeled CT contrasts give 233/240, broadening competence checks (Appendix P).

3. A matched intervention identifies a measured contributor. A rank-16 text adapter on frozen VoxTell v1.1 is trained on development patients; all arms use the same patients, seeds 0-2, exactly 1,200 updates and the same patch rule.

ArmConcepts per patientCompetitor voxelsPairsnWMH margin
frozen--0/20-0.36
1oneignored53/60+0.16 [0.09, 0.21]
2onebackground56/60+0.37 [0.30, 0.43]
3both, one prompt/stepbackground58/60+0.48 [0.42, 0.53]
4both, both prompts/stepbackground58/60+0.44 [0.37, 0.49]
4, 2 patients (3 subsets)bothbackground57/60+0.34 [0.28, 0.39]
  • Competitor supervision strengthens selection: background vs ignored +0.22 [0.19, 0.25]; both concepts on the same patients +0.11 [0.09, 0.13].
  • Not necessary: arm 1 still recovers 53/60; the structure mainly sets the margin size.
  • Few labels suffice: margins plateau from 5 patients.
  • Tradeoff: the +0.22 gain comes with a held-out MS paraphrase change of -0.17 [-0.19, -0.14]. Arms 3 and 4 differ in prompt exposure (1,200 vs 2,400 prompt evaluations).

These are exploratory results on previously opened patients, not a universal rule.

4. Target-dataset recovery can conceal collateral failure. The first adapter recovers MS3SEG (57/60) but reduces unrelated BraTS pair success to 7/120. An embedding-retention anchor then gives 58/60 MS3SEG, 120/120 kidney and 64/120 BraTS pairs (19/40 frozen). The protocol shows where a repair damages another distinction. The anchor followed BraTS feedback, so this retention evaluation is exploratory (Tables 4 and 15).

W2: Definition mismatch and empirical scope

We agree that generic WMH in a training inventory does not establish MS3SEG's nWMH convention, and withdraw "not a missing label." The documented WMH source labels presumed vascular WMH; equivalence to MS3SEG's category is unknown, and v1.1 has unlisted additions. Absent distinction-specific supervision and a convention mismatch both remain viable; adaptation shows learnability but does not separate them.

The observation remains an operationally meaningful failure: the model accepts the clinical request and returns a confident mask covering its competitor. It persists on the frozen 20-patient confirmation, across both VoxTell versions, and within jointly size/intensity/ventricular-distance-matched components (-0.211 [-0.335, -0.092]). The strongest evidence is VoxTell on MS3SEG, with Medical-SAM3 corroborating. An independently specified second clinical distinction is the natural next test of breadth (Section 7).

Novelty relative to known limitations

  • Prompt sensitivity: here the model is prompt-sensitive (41% of the mask changes), yet still selects the competitor. Sensitivity checks would pass it; our test does not.
  • Semantic grounding: grounding benchmarks ask whether a named object is found. We test competitive selection between two coexisting annotated targets in the same image, with a superclass control, which Dice or grounding accuracy cannot reveal.
  • Confident errors: wrong masks have foreground probability 0.962, and an overlap warning is near chance within BraTS. Confidence-based safeguards miss this failure.
  • Partial-label training (DoDNet; CLIP-Driven Universal Model; now cited) trains across label sources but does not test selection. We supply the test, show which supervision components move selection, and catch collateral damage from repairs.

We hope these experiments address your concern. We would be glad to address any remaining questions.

Reviewer_dTS6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a free-text medical segmenter picks the concept a prompt names when a related concept is in the same image. Its test, Clinical Contrast Sets, fixes the image and asks for concept A, then concept B. The coverage margin is the share of the requested region covered minus that of the competing region. On MS3SEG, Medical-SAM3 and VoxTell are asked for multiple sclerosis (MS) lesions or for the dataset's normal, non-MS white matter hyperintensities (nWMH). Both models cover MS lesions whichever concept is requested. On a 20-patient test set fixed in advance, the nWMH margins are -0.308 and -0.363. Each model gets 0 of 20 prompt pairs correct. Size and wording controls leave the result unchanged. The test is sound and the effect is real. The claim that the failure is "not a missing label" uses checkpoints the paper does not name and a superclass entry. The failure is shown for one distinction on one dataset, in practice with one model. The paper tests models but does not improve one.

Critical strengths
  1. A same-image, two-prompt test that separates picking the target from finding it. Equations 1 and 2 score the requested region against its same-image competitor. A prompt pair is correct only if both prompts give the correct result in one patient. Dice on the requested target alone cannot tell a missed target from a swapped one. The test also shows success: VoxTell segments the requested kidney side in 40 of 40 cases. This extends contrast sets (Gardner et al., 2020) to pixel-level prediction. It is the paper's main reusable contribution.
  2. Held-out test with rules fixed in advance, size controls, and openly reported negative results. Endpoints and analysis rules were fixed before the official test was opened. Official results match development. For VoxTell, volume reversal and component matching (-0.102 [-0.198, -0.010]) show that lesion size in the test images does not explain the result. Failures are reported too: Medical-SAM3's matched interval includes zero. With VoxTell, the prompt-overlap detector has an area under the receiver operating characteristic curve (AUROC) of 0.916 pooled but 0.461 within the BraTS brain tumor cohort.
Critical weaknesses
  1. The paper only tests models. It does not use its own criterion to improve one. The paper calls its contribution "an evaluation procedure" and leaves "a reliable model-level repair" to future work. The one repair it tries, Contrastive Vote Exclusion, lowers macro Dice by 0.104 and 0.306 (Appendix G). For ICLR, a test of one distinction on one dataset is a narrow contribution. The criterion can also serve as a training loss. Equation 2 compares coverage of the requested region and of its competitor. With soft masks, this margin is differentiable. A loss can require a positive margin for both prompts of a pair, next to the usual segmentation loss. MS3SEG has 80 development patients, with both classes annotated. The paper also states that published supervised results show nWMH is learnable. I suggest:

    • Fine-tune VoxTell, or an adapter on it, with this margin loss on the development patients.
    • Rerun the frozen protocol on the 20 official-test patients.
    • Report Dice on the kidney and BraTS controls.

    A loss that restores correct switching would make the paper both a test and a fix.

  2. Checkpoints the paper does not name and a superclass entry in a concept list do not show that nWMH was a trained concept. The abstract says the failure is "not a missing label" because VoxTell's list of training concepts includes both. The evidence shows a listed superclass. It does not show a subclass learned by the model they ran.

    First, Table 6 marks nWMH only as "covered by the listed white matter hyperintensities category". The paper's own prompts count MS lesions as white matter hyperintensities (Appendix A). A listed superclass cannot show that training separated the two. Table 6 lists 60 superclass instances against 366 for MS lesions. These may be the white matter hyperintensity segmentation challenge labels (Kuijf et al., IEEE TMI 2019). That challenge marked lesions of presumed vascular origin in memory-clinic patients. MS3SEG defines nWMH as "age-related or CSF-contaminated regions". CSF is cerebrospinal fluid. The age-related part overlaps the challenge convention. The CSF-contaminated part may not. The Limitations admit that the conventions may differ. A model with one concept, bright white matter lesion, would give the same results if nWMH is fainter or lies nearer the ventricles.

    Second, the paper names neither checkpoint. I could not inspect the supplementary manifests. Appendix H gives shortened repository revisions. A revision does not determine which checkpoint is loaded. By my reading of the VoxTell model card, the concept list describes voxtell_v1.0. The tool downloads voxtell_v1.1 by default. That checkpoint was trained on 190 datasets, including VoxTell's own test sets. On GitHub, the VoxTell revision is a commit made after v1.1 became the default. If v1.1 was run, Table 6 lists only part of its training data. A rerun with v1.0 would make Table 6 match the model that was run. For Medical-SAM3, no model card names the training data of the 3D checkpoint. The repository's 3D training configuration lists BraTS and white matter hyperintensity sets on FLAIR (fluid-attenuated inversion recovery) magnetic resonance imaging (MRI).

    The rebuttal should:

    • Name both checkpoint files and download dates.
    • State whether MS3SEG is in their training sets.
    • Identify which dataset supplies the white matter hyperintensities label.
    • Say whether any training volume carries both labels.
    • Split the component-matching analysis by FLAIR intensity and distance to the ventricles.

    The claim is supported if one result is shown. Bright deep nWMH components must stay uncovered while matched MS components are covered. If not, the abstract, Section 6 and Table 6 should say only that the superclass is listed.

Questions for authors
  1. The MS3SEG dataset paper reports FLAIR slices of 5.73 +/- 0.52 mm (Bawil et al., 2026). Were volumes resampled for each model? Thick slices would reduce accuracy on thin periventricular nWMH more than on MS lesions.
  2. What share of nWMH volume is CSF-contaminated rather than age-related? The dataset paper reports consensus labels from two annotators and no agreement statistic. Is any agreement measure available for the MS versus nWMH split?
  3. The Medical-SAM3 paper trains and evaluates with single standard terms. Its Table 1 lists no computed tomography (CT) or MRI training set (Jiang et al., 2026). Which prompt format do the weights that were run expect?
1 comment

Thank you for proposing a model-level intervention and retention test; we implemented both: segmentation-only and segmentation-plus-margin adapters, evaluated on MS3SEG and on both frozen competence cohorts, kidney and BraTS-MEN, as you asked. Revised text is blue and tagged [dTS6]; Table 10 gives each analysis's script and output.

W1: Use the criterion to improve the model and test retention (Section 5.6, p. 8; Table 4; Appendix O)

A rank-16 adapter (82k parameters) changes only VoxTell v1.1's text embedding; image, fusion, decoder and Qwen3 weights remain frozen. Trained on the 80 development patients, three seeds. The margin arm adds the differentiable soft-coverage hinge from Eq. 2.

EndpointFrozenAnchored segAnchored seg+margin
MS3SEG canonical pairs0/2058/6058/60
nWMH family direction-0.36+0.44 [0.37, 0.50]+0.47 [0.41, 0.53]
Held-out nWMH paraphrase / superclass / desc.-0.36 / -0.34 / -0.34+0.42 / +0.42 / +0.42+0.46 / +0.46 / +0.45
nWMH / MS Dice0.19 / 0.540.54 / 0.590.54 / 0.59
Kidney pairs; laterality Dice40/40; 0.918120/120; 0.918120/120; 0.918
BraTS pairs; paired shift S19/40; +0.7464/120; +0.7564/120; +0.76
BraTS Dice: enhancing / non-enhancing / meningioma0.94 / 0.36 / 0.950.94 / 0.32 / 0.940.94 / 0.33 / 0.94

Counts of 60/120 are 20/40 patients across three seeds; intervals cluster by patient. The official patients are training-disjoint but previously opened.

The criterion works as the acceptance test you proposed. The initial segmentation-only adapter recovered MS3SEG (57/60) but reduced BraTS pair success to 7/120, failing on "non-enhancing" prompts; MS3SEG accuracy and kidney retention alone did not reveal this. We then added a penalty discouraging changes to 38 unrelated prompt embeddings (no evaluation prompt included; weight fixed before the anchored runs). This restored BraTS pair success (64/120) and paired shift (+0.75 vs +0.74 frozen), and kidney Dice to the frozen 0.918. Because BraTS feedback motivated the anchor, this is an exploratory repair, not fresh confirmation.

Your margin loss gives the larger nWMH margin; segmentation alone already restores most pairs. Remaining limits: non-enhancing BraTS Dice is slightly below frozen, and the held-out MS paraphrase stays negative (-0.17/-0.33; Table 15). A matched factorial (Section 5.7) adds which supervision matters: competitor-as-background +0.22 margin; two patients give 57/60. As you proposed, the paper is now both a test and a fix.

W2: Checkpoints, training provenance and image covariates

  • Weights: mrokuss/VoxTell/voxtell_v1.1/fold_0/checkpoint_final.pth (SHA-256 f45e61c3...) and ChongCong/Medical-SAM3/checkpoint_3D_checkpoint8.pt (31fd6a05...); full hashes and revisions in Table 9. Each file has exactly one published Hub version, so the hash identifies the weights that were run. Download dates were not logged; rerun copies were hash-verified on 17 Nov 2026.
  • Inventory-matched version: v1.0 gives official nWMH direction -0.339 [-0.415, -0.269], 0/20 pairs; development -0.304, 0/80.
  • MS3SEG exposure: not listed in the v1.0 tables, v1.1 card or Medical-SAM3 configuration (v1.1's 32 unlisted additions cannot be ruled out).
  • WMH source/co-labeling: the WMH Challenge (60 instances), as you expected. No joint MS/nWMH annotation is documented; 42/366 MS instances are unattributed. We withdraw "not a missing label" and adopt your wording (abstract, Section 6, Table 8).
  • Intensity / ventricle distance (Appendix N), testing your single "bright lesion" alternative: 80.5% of nWMH vs 5.7% of MS components touch the ventricles. Calipers: size ratio <=2, intensity <=0.25 z, ventricular distance <=2 mm; achieved 0.09 z and 0.8 mm. Direction remains -0.211 [-0.335, -0.092] (138 pairs, 19 official patients) and -0.274 [-0.321, -0.229] (592 pairs, 79 development patients); v1.0 gives -0.119/-0.113.
  • Bright/deep subgroup: development -0.139 [-0.428, 0.147] (16 pairs); official +0.53 [0.15, 0.91] (5 pairs). This does not meet your affirmative test, so we adopt your fallback wording.

Q1. Neither pipeline physically resamples MS3SEG. Voxel spacing is 0.90 mm in-plane and 6.0-8.1 mm through-plane (mean 7.1). VoxTell uses noResampling; Medical-SAM3 resizes slices internally to 504x504. The adapter recovers nWMH on the same grid (Dice 0.54), though resolution may still contribute.

Q2. The released binary nWMH mask does not separate CSF-related from age-related voxels, and no agreement statistic is available. Ventricular proximity is not a CSF-contamination share.

Q3. At b610242c, medical/inference_3d.py --prompt 1=<phrase> --instance accepts text prompts. The 3D vocabulary has standard WMH terms but no MS concept; accepted syntax does not establish training support, so Medical-SAM3 is corroborative (Appendix K).

Reviewer_eHPkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether text-prompted 3D medical segmenters actually select the requested clinical concept when a related, coexisting concept is present in the same volume. It proposes Clinical Contrast Sets: hold the image fixed, define two disjoint annotated regions A and B, prompt for each plus a superclass prompt, and score a size-normalized directional margin ΔCov, with "pair success" requiring both directions positive in the same case. Three contrast sets are used: MS3SEG as the target case, plus BraTS-MEN and TotalSegmentator as competence controls. With Medical-SAM3 and VoxTell, nWMH prompts yield negative margins, replicated under a protocol frozen and hashed before test inference, and persisting under volume reversal, connected-component size matching, positive rephrasings, and the generic WMH superclass prompt. The paper further audits VoxTell's published training-concept inventory, reports high foreground confidence for wrong masks, and shows a prompt-overlap detector that separates datasets but not failures within BraTS.

Critical strengths

The superclass-null + size-normalization design converts a vague worry into a falsifiable test. By pairing each child prompt with a superclass reference and normalizing each class by its own denominator, the protocol separates base-rate bias, prompt compression, generic localization failure, and target substitution. This is the reusable part of the paper and the part most likely to be adopted — including as an endpoint for adaptation methods. It matters because "category errors happen" is already known; "here is a controlled protocol that isolates which explanation holds" is not.

Critical weaknesses
  1. The headline numbers actually look more like "the prompt doesn't matter" than "the concept gets substituted," yet the paper never reports the quantity that would tell these apart. If the two canonical prompts produce the same mask M, then by construction ΔCov(qA) = cov_M(A) − cov_M(B) and ΔCov(qB) = −ΔCov(qA). The reported values are almost exactly antisymmetric: Medical-SAM3 official +0.310 vs. −0.308 (sum +0.002); VoxTell +0.335 vs. −0.363 (sum −0.028); development +0.349 vs. −0.368. A depolarization that small is exactly what you'd expect when Dice(P_qA, P_qB) is high and the mask barely moves when the prompt changes. PromptSwapDice is computed and threshold-calibrated (App. I), but it's never reported descriptively—no mean Dice(P_qA, P_qB), no per-case voxel-change fraction. This matters because the abstract and §3.1 say the mask shifts "toward the intended region and away from its competitor," and the framework's name implies the model resolves a different concept. But if the mask is essentially invariant, the more accurate description is a prompt-insensitive saliency prior that just hands back the most salient lesion—a weaker and different claim, and the "ontology" framing doesn't hold up. The missing analysis is just one table, and either way it would settle the question; until it's in, the central construct is still underdetermined.

  2. The inference "the concept was documented, so it's not a missing label" rests on the superclass, not the subclass. Appendix K lists "white matter hyperintensities" with 60 reported instances—but "WMH" in everyday usage already includes MS lesions, and in an MS cohort it's the natural superclass. Nothing in the audit shows that the non-MS subtype was ever a separate training category, nor which annotation convention VoxTell used. The paper concedes this in Limitations ("the inventory lists concept names, not annotation conventions"), yet the abstract and §6 still trot out the documented-concept audit as the argument that rules out a missing label, and §5.2 leans on it again ("even that documented prompt resolves to MS lesions"). Since the paper's own analysis shows the generic superclass prompt is also MS-biased (−0.307/−0.341), the simplest reading is that the model has just one visual concept ("WMH in MS patients") and the subclass was never separated—which is a training-label/convention issue, not an ontology-interface gap. Worse, only one model carries the claim: Medical-SAM3's canonical MS Dice is 0.198 and nWMH Dice 0.026, which the paper itself downgrades to "corroboration," so the abstract's "two 3D segmenters" oversells the evidence base—really it's one informative model, on one annotation-defined subclass, in one dataset.

Questions for authors
  1. Please report, per model and per contrast set, the descriptive mask-change statistics: mean Dice(P_qA, P_qB), mean symmetric volume difference, and the distribution of per-case ΔCov for the nWMH family. How many of the 20 official cases have a positive nWMH direction, and is the mask for qA distinguishable from the mask for qB beyond noise? Given the near-perfect antisymmetry of Dir. A and Dir. B, is there any evidence of a prompt-driven shift rather than a static mask?

  2. Can you provide any evidence that the MS3SEG nWMH class is separable from the model's "white matter hyperintensity" concept — e.g., evaluating on a WMH-only cohort without MS lesions, or probing whether the model can return non-MS WMH when no MS lesion is present in the volume? Without this, the "documented concept" argument concerns the superclass, not the subclass.

1 comment

Thank you for naming the analyses that decide between an invariant mask and failed target selection; we ran both with the frozen prompts and metrics. Revised text is blue and tagged [eHPk]; Table 10 marks each analysis as original, post hoc or exploratory.

W1/Q1: Static mask, or a prompt-driven shift?

Not static: VoxTell's mask responds strongly to the prompt, but the requested target never wins (Section 3.3; Section 5.2, p. 7; Table 3; Appendix M).

Official, 20 patientsMedical-SAM3VoxTell v1.1VoxTell v1.0
Canonical prompt-swap Dice0.4460.7340.324
Changed voxels / mask union67.1%41.0%79.9%
Symmetric rel. volume difference0.6670.3550.312
Repeat-inference Dice (noise)-0.9997-
Paired preference shift S [95% CI]+0.002 [-0.013, 0.018]-0.033 [-0.050, -0.016]+0.121 [0.078, 0.167]
Positive nWMH-family direction0/200/200/20

Two independent VoxTell v1.1 runs agree at Dice 0.9997 (one prompt per pass: 0.9991). The 0.734 cross-concept Dice is therefore a substantial prompt response, far beyond repeat variation. Within-MS paraphrase Dice is 0.956.

The exploratory paired shift S_i = DeltaCov(q_B) + DeltaCov(q_A) asks whether the prompt change moves preference toward nWMH. For v1.1, S is negative in 16/20 patients. For v1.0, S is positive in 18/20, but no canonical pair succeeds: the movement toward nWMH is too small to reverse the MS preference. Medical-SAM3's mask changes, but its shift is near zero. S complements the pair-success endpoint.

Antisymmetry does not establish mask identity: identical masks imply antisymmetric directions, not conversely. We now distinguish output change from correct target selection (Section 3.3).

Table 11 (Appendix M) gives all three measures for every model and cohort. VoxTell/Medical-SAM3 swap Dice is 0.000/0.336 on kidneys and 0.232/0.083 on BraTS: the masks change on every contrast. On kidneys and BraTS the change moves toward the requested target (S +1.90 and +0.74); on MS3SEG it does not (S -0.033). Official VoxTell nWMH-family directions have median -0.350, IQR [-0.436, -0.224], maximum -0.060; on all 80 development patients, v1.1 gives -0.376 and 0/80 pairs.

W2/Q2: Was the subclass trained, and can the model distinguish it?

We agree that a superclass entry cannot establish training on MS3SEG's nWMH convention, and we withdraw "not a missing label" (abstract; Section 4.3; Section 6; Appendix K). The documented WMH source is the WMH Challenge (presumed vascular WMH); equivalence to MS3SEG's category is undocumented. No documented source jointly annotates the distinction; 42 MS instances and v1.1's 32 additional datasets remain unattributed.

  1. Remove the competitor (synthetic, Table 13). After MS-lesion inpainting, the nWMH-prompt mask's share on nWMH rises from 0.21 to 0.51, but its coverage falls from 0.257 to 0.159 (no patient increases). MS prompts give coverage 0.151 and share 0.61, so the nWMH wording is not more nWMH-selective even without the competitor. A sham edit of equal shape changes nothing (0.000 [-0.002, 0.001]).
  2. Adapt only the text conditioning (Section 5.6, p. 8; Table 4). An anchored rank-16 adapter trained on the 80 development patients, with all visual weights frozen, restores canonical switching in 58/60 seed-patient evaluations (frozen 0/20) and raises nWMH Dice from 0.19 to 0.54. The held-out nWMH paraphrase, superclass and descriptions all turn positive; kidney (120/120) and BraTS (64/120 vs 19/40 frozen) pair success are retained. The frozen model can thus be steered to the dataset's convention; this does not prove the convention was learned before adaptation. The held-out MS paraphrase remains misdirected on average (Table 15).

Your training-label reading and ours largely coincide, and the MS-biased superclass prompt (-0.307/-0.341) fits it. Section 5.7 (p. 9) adds a measured contributor: in adaptation, explicit competitor supervision enlarges the nWMH margin (+0.16 to +0.48), and two patients suffice for 57/60. This cannot separate absent supervision from a convention mismatch.

Joint size/FLAIR-intensity/ventricular-distance matching retains a negative direction: -0.211 [-0.335, -0.092] (138 pairs, 19 official patients) and -0.274 [-0.321, -0.229] (592 development pairs).

On the framing. We now define the open-ontology gap operationally (Section 3): the interface accepts an open-vocabulary phrase and the mask responds to it, but the requested subclass is not bound to a distinct visual target. That is the behavior measured above, not a static saliency prior, whatever its training cause. VoxTell is now explicitly the primary evidence; Medical-SAM3 corroborates; v1.0 adds version robustness. The finding is the reusable distinction your review highlights: accepting and responding to a clinical phrase does not establish correct selection of its target.

We would be glad to address any remaining questions

Comments on the paper

1 comment

We thank the reviewers for detailed, constructive reviews. Every requested analysis has been run; each change in the revised manuscript is blue and tagged with the requesting reviewer, and Table 10 marks every analysis as original, post hoc or exploratory, with its script and output.

The problem and our contribution

Free-text segmenters accept any clinical phrase, but a plausible mask can cover the wrong finding. The question no existing evaluation answers is: when two related findings coexist in one image, does the model select the one that was requested? Clinical Contrast Sets answers it with

  • two coexisting annotated targets plus their superclass, prompted on the same image;
  • class-normalized coverage directions, robust to large size differences;
  • paired selection: both canonical prompts must prefer their own target in the same case;
  • controls that separate base-rate bias, prompt compression, localization failure and target substitution.

Why this is new

  • Prompt sensitivity is not selection. VoxTell changes 41% of its mask when the concept changes, so sensitivity tests pass it, yet it never selects the requested target.
  • Accuracy and grounding are not selection. Dice or grounding scores on each target cannot distinguish a missed target from a swapped one; the paired, same-image test can.
  • Confidence does not flag it. Wrong masks have foreground probability 0.962; an overlap warning is near chance within BraTS.
  • Partial-label training (DoDNet; CLIP-Driven Universal Model; now cited) trains across label sources but does not test competitive selection. We provide that test, and use it to diagnose, repair and audit a model.

Main findings

  • A complete failure under a prespecified protocol: 0/20 canonical MS/nWMH pairs for both models on the official patients (nWMH margins -0.308, -0.363); 0/80 in development; both VoxTell checkpoints; and after joint matching on size, FLAIR intensity and ventricular distance (-0.211 [-0.335, -0.092]).
  • Not a static mask: prompt-swap Dice 0.734 vs repeat-inference 0.9997, yet the requested target never wins (Section 5.2, Tables 3 and 11).
  • Learnable from frozen image features: a text-only adapter (82k parameters) restores switching in 58/60 seed-patient evaluations and raises nWMH Dice 0.19 to 0.54, while kidney (120/120) and BraTS (64/120 vs 19/40 frozen) pair success are retained (Section 5.6, Table 4).
  • An acceptance test for repairs: a first adapter fixed MS3SEG but broke the unrelated BraTS "non-enhancing" concept (7/120 pairs). Accuracy and kidney retention missed this; Clinical Contrast Sets caught it, and a retention anchor restored competence-cohort pair success.
  • Which supervision matters: in a matched factorial, treating the competitor as background adds +0.22 [0.19, 0.25] to the nWMH margin, and two labeled patients give 57/60 pairs (Section 5.7, Table 16).

How each review is addressed

  • eHPk: mask-change statistics for every model and contrast, repeat-inference noise, paired shift S, a synthetic competitor-removal probe with sham control, and text-only adaptation. "Not a missing label" is withdrawn; VoxTell is stated as the primary evidence.
  • dTS6: both suggested adapters, tested on MS3SEG and both competence cohorts; checkpoint files and hashes (Table 9); v1.0; WMH source; joint intensity/ventricle matching; answers to Q1-Q3.
  • YyGW: what separates success from failure across kidney, BraTS, MS3SEG and six CT distinctions; the controlled supervision factorial; and a precise account of what remains open.

On scope

All reviewers noted that the decisive failure concerns one distinction. We treat it as a precisely characterized case rather than a prevalence claim: the same protocol detects success on kidney laterality (40/40), BraTS sub-regions (S +0.74) and six CT distinctions (233/240), so it separates competence from failure instead of only finding errors. Applying it to further clinical distinctions requires only two coexisting annotations per image.

Integrity

The original confirmation is unchanged. New analyses use the frozen prompts, metrics and patient IDs; our pipeline reproduces the saved outputs to within 0.002 per patient. Every trained arm is reported per seed, with scripts and per-patient outputs in the supplement.

The revision turns a documented failure into a validated, reusable protocol that diagnoses whether a free-text segmenter selects the requested clinical target, identifies what supervision changes that, and checks that a repair does not break other concepts. We thank the reviewers; their questions shaped these experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.