acceptodds
Under review as a conference paper at ICLR 2027

Deciding in the Dark: Multimodal LLMs as Evidence Selectors for Few-Shot Ultrasound Video Segmentation

Abstract

Few-shot segmentation of ultrasound sweeps depends on which annotated examples the segmenter sees and from which frame a mask is propagated, yet both are usually chosen by visual similarity, which is unreliable when speckle, shadowing and probe position change how the same anatomy looks. We present LucidSeg, a training-free framework in which a multimodal large language model (MLLM) makes discrete decisions about the evidence while frozen segmenters compute every mask. At Level 1, the MLLM chooses a small support set from retrieved candidates of other subjects, shown either as annotated exemplars or as annotations transferred onto the query, and a frozen in-context segmenter uses it; at Level 2, it chooses the starting frame of a sweep by judging the predicted outlines of candidate frames, and SAM2 propagates the mask from there. Because the MLLM never outputs a mask, box or point, its decisions serve any in-context segmenter. On four ultrasound datasets, a locally deployed open-weight MLLM matched a reranker trained on the target data and improved Dice over visual similarity by up to 0.11, with the largest gains where the target's appearance depends on anatomical side or probe position. On thyroid sweeps, the chosen starting frame came within 0.05 Dice of the best candidate; on spine sweeps over repeating vertebrae, starting from all candidates was better.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
segmentation, ultrasound video understanding, few-shot segmentation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1T5jOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LucidSeg, a training-free framework that uses multimodal LLMs as evidence selectors rather than direct segmentation models. The MLLM makes two discrete decisions: (1) selecting annotated support examples for an in-context segmenter, and (2) selecting the anchor frame for SAM2 propagation in ultrasound sweeps. The actual masks are always produced by frozen segmentation models. The method is evaluated on four ultrasound datasets with multiple executors and MLLMs. The results show clear improvements over visual similarity on TG3K, modest gains on USST, and limited or no gains on MicroSeg/CardiacUDA; for video propagation, MLLM-selected anchors help on TG3K, while multi-start propagation works better on USST.

Critical strengths
  1. Instead of requiring an MLLM to generate masks, boxes, or points, LucidSeg restricts it to discrete evidence-selection decisions and leaves geometry to frozen segmentation models. This is conceptually clean and makes the approach executor-agnostic.

  2. The paper evaluates four ultrasound datasets, two in-context executors, several MLLMs, trained and training-free baselines, and multiple ablations. In particular, the substantial improvement over DINOv2 selection on TG3K supports the central hypothesis that semantic/anatomical judgment can help when visual similarity is misleading.

Critical weaknesses
  1. The strongest result is on TG3K. On MicroSeg, LucidSeg is below DINOv2 similarity; on CardiacUDA the differences among support-selection strategies are small; and on USST the improvement is modest and configuration-dependent. Moreover, with Gemma-4-31B, withholding the query performs similarly or even better on three of four datasets, suggesting that part of the gain may come from selecting generally representative supports rather than query-specific anatomical reasoning.

  2. The paper correctly uses cluster bootstrap confidence intervals because queries from the same unit are correlated, but the reported paired Wilcoxon tests appear to be performed over individual queries rather than units, which may produce overly optimistic p-values. In addition, Level 2 is evaluated on only 13 TG3K sweeps and 6 USST sweeps, and the authors explicitly note that full-system results on the single-vertebra USST setting are missing.

Questions for authors
  1. Could the authors recompute the main significance tests using the patient/scan/unit as the statistical unit, or another test that explicitly accounts for within-unit correlation? Do the main TG3K and USST conclusions remain significant?

  2. The paper states that TG3K scan units were reconstructed because patient/video identities are unavailable and therefore cannot guarantee that all supports come from different subjects. Could the authors quantify the potential leakage risk or provide results under a stricter exclusion strategy?

  3. Can the authors provide the full LucidSeg pipeline on the available single-vertebra USST sweeps, or otherwise provide additional sweep-level experiments? This would help determine whether the success of anchor selection on TG3K generalizes beyond a single anatomy/acquisition pattern.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 1T5j (1/2)

Thank you for the positive assessment. You saw right away what the method offers: a training-free MLLM that only makes discrete choices over annotated evidence, so that every mask comes from a frozen segmenter and any executor can be used.

W1. Improvements are concentrated on TG3K. With Gemma-4-31B, withholding the query is similar or better on three of four datasets.

The general response (CQ1) gives the full comparison. You are right. How much the MLLM improves on image features depends on its own ability to generalize, and it differs across models and datasets. A stronger MLLM goes further above the features. With DINOv3 candidates, GPT-6 Sol in the exemplar mode is +0.047 above DINOv3 similarity on TG3K (p_u = 0.017) and level with it on MicroSeg, while Gemma-4-31B is level on both.

Withholding the query is as good on TG3K, CardiacUDA and USST. Still, one fixed support set loses 0.016–0.163 on these three datasets, significantly on CardiacUDA and USST. So the decision without the query still follows each query's candidate list.

On the withheld query, we built a fixed-set baseline. It uses one support set for all queries. The set is the four annotated frames chosen most often as supports when the query was withheld. It needs no MLLM call at test time.

Dice UvS / MvSGemma E, query withheldone fixed set for all queriesΔ (p_u)
TG3K.483 / .622.467 / .595−0.016 / −0.027 (0.22 / 0.13)
MicroSeg.638 / .723.609 / .690−0.029 / −0.033 (< 1e-5)
CardiacUDA.682 / .778.636 / .751−0.046 / −0.027 (0.001 / 0.012)
USST.580 / .675.466 / .512−0.114 / −0.163 (< 1e-10)

W2 / Q1. Wilcoxon over queries may be optimistic. Recompute with the unit as the statistical unit.

The TG3K and USST improvements over DINOv2 stay significant at the unit level. On TG3K, all 13 scans are positive in the transfer mode. The improvement over the trained reranker on TG3K is no longer significant (p_u = 0.19). The paper now says "matched" instead of "improved on". The MicroSeg deficit becomes significant. The paper states it as such.

We recomputed every main test with the unit as the statistical unit. The unit is the scan on TG3K and the patient on USST and MicroSeg. CardiacUDA already has one query per patient. We average the paired differences within each unit. We then run a paired Wilcoxon over units (p_u) and a unit-bootstrap 95% CI.

LucidSeg vs.Δ Dice (UvS)p_q (paper)p_uunits with Δ > 095% CI (unit bootstrap)
TG3K, T vs DINOv2+0.1075e-110.0002413/13[+0.077, +0.139]
TG3K, E vs DINOv2+0.0831e-60.003411/13[+0.044, +0.124]
TG3K, T vs reranker+0.0360.0120.198/13[+0.004, +0.072]
USST, T vs DINOv2+0.0340.0050.03031/50[+0.009, +0.060]
USST, T vs reranker−0.0120.630.5724/50[−0.039, +0.014]
MicroSeg, E vs DINOv2−0.0230.190.01126/71[−0.039, −0.006]
MicroSeg, T vs DINOv2−0.0443e-41e-618/71[−0.059, −0.029]
CardiacUDA, all pairswithin 0.04n.s. except GPT-5.5_T vs DINOv2, UvS (+0.021, 0.007)same as p_q (one query per patient)–cover 0
TG3K, T (DINOv3 cand.) vs DINOv3+0.0130.500.248/13[−0.008, +0.032]
USST, T (DINOv3 cand.) vs DINOv3+0.0310.0050.02930/50[+0.005, +0.057]
MicroSeg, T (DINOv3 cand.) vs DINOv3−0.0542e-54e-620/71[−0.073, −0.034]
CardiacUDA, T (DINOv3 cand.) vs DINOv3+0.0080.100.1060/100[−0.015, +0.029]
TG3K, E (DINOv3 cand.) vs DINOv3+0.0170.0320.2210/13[−0.017, +0.047]
USST, E (DINOv3 cand.) vs DINOv3+0.0050.350.4228/50[−0.021, +0.031]
MicroSeg, E (DINOv3 cand.) vs DINOv3−0.0170.190.1530/71[−0.037, +0.005]
CardiacUDA, E (DINOv3 cand.) vs DINOv3−0.0320.110.1145/100[−0.061, −0.004]

The last eight rows also compare LucidSeg with DINOv3 similarity. LucidSeg_T is significantly above it on USST, significantly below it on MicroSeg and level with it on TG3K and CardiacUDA. LucidSeg_E is not significantly different from it on any dataset over units. Appendix D now defines the unit-level test. §5.1 gives both p-values for the key claims.

AuthorsOpenReview ↗

Response to Reviewer 1T5j (2/2)

Q2. Leakage from reconstructed TG3K scan units, and a stricter exclusion.

The improvement does not shrink as the exclusion widens. DINOv2 similarity stays at the level of random supports throughout. DINOv3 similarity falls slightly, and LucidSeg_E with DINOv3 candidates stays level with or above it at every R.

We ran two checks.

(i) Similarity by unit distance. The mean DINOv2 cosine between pool frames is .804 within a unit. It is .750 for adjacent units, which are already excluded. It is .747 at distance 2 and .742 at distance ≥ 5. A query's maximum cosine is .915 to its own unit, .840 to adjacent units and .874 to the allowed pool. The supports LucidSeg chose average .76–.78. The nearest allowed frame lies at distance 2 in 16.5% of queries (10.2% if independent of the unit). Its cosine is at the level of distant units.

(ii) Stricter exclusion. We removed the query's unit and R neighbours on each side. We then re-ran the Gemma decision (exemplar mode).

R (paper uses 1)pool framesrandomDINOv2DINOv3LucidSeg_E (DINOv2 cand.)LucidSeg_E (DINOv3 cand.)(DINOv2 cand.) Δ vs DINOv2, UvS / MvS (p_u)(DINOv3 cand.) Δ vs DINOv3, UvS / MvS (p_u)
1144.373 / .537.401 / .544.495 / .623.484 / .611.512 / .629+0.084 (0.0034) / +0.066 (0.01)+0.017 (0.22) / +0.006 (0.84)
2129.388 / .542.404 / .556.491 / .626.509 / .607.515 / .622+0.105 (0.00024) / +0.052 (0.08)+0.025 (0.24) / −0.005 (0.64)
3115.378 / .529.404 / .556.479 / .617.496 / .596.506 / .626+0.092 (0.0061) / +0.040 (0.094)+0.027 (0.31) / +0.009 (0.89)

Random supports are re-drawn for every R. So R = 1 gives .373 / .537, while the draw of Table 1 gives .384 / .547.

If one video had been split into non-adjacent units, the leak would help visual similarity, not the MLLM. DINOv3 similarity falls by 0.016 with UniverSeg and 0.006 with MultiverSeg from R = 1 to R = 3, so a leak, if there is one, favours the visual baseline. LucidSeg_E with DINOv3 candidates stays level with or above it at every R (p_u ≥ 0.22), and its value does not fall (.512, .515 and .506 with UniverSeg).


Q3. Full pipeline on the single-vertebra USST sweeps and more sweep-level experiments. Does anchor selection generalise beyond TG3K?

Every chain agrees. On the 50 vertebral segments, multi-start is best (+0.015 to +0.026 over the LLM anchor, p = 0.04–0.12). The LLM anchor is the best single start in every chain. It is +0.023 to +0.077 over the middle frame and +0.007 to +0.045 over confidence. It is +0.07 to +0.11 over a random candidate (p < 1e-5).

We ran the full system on all 50 USST test segments with all four support chains. We also ran it on the 14 single-vertebra segments of the Level-2 patients. Table 2 now reports the 50 segments.

USST, MultiverSeg, K = 8middleconfidenceLLM anchorall K (multi-start)anchor oracle
14 segments, DINOv2 supports.658.673.671.703.726
14 segments, Gemma supports.676.658.691.720.730
14 segments, GPT-5.5 supports.589.642.671.723.724
50 segments, DINOv2 supports.642.632.667.686.742
50 segments, Gemma supports.661.695.702.726.767
50 segments, GPT-5.5 supports.633.674.703.718.752
50 segments, GPT-6 Sol supports.630.683.699.725.759
6 per-patient scans of the original version (each joins one patient's segments), DINOv2 / Gemma / GPT-5.5 supports.623 / .640 / .633.589 / .630 / .622.678 / .665 / .632.712 / .726 / .724.706 / .706 / .695

So the anchor decision generalises from TG3K, but the single-anchor policy does not. One structure through the scan favours one well-chosen start. A structure that changes or repeats favours many starts. We made this choice a third discrete decision of the MLLM. One extra call per scan shows the MLLM the eight candidate frames with their predicted outlines. The MLLM answers whether the target is one continuous structure or a series of repeating structures. For "one" we propagate from the LLM anchor, and for "repeating" we use multi-start. It chose "one" on 92–100% of TG3K scans. It chose "repeating" on 50–67% of the 6 per-patient USST scans. This pipeline is at most 0.025 Dice below the better fixed policy (.764–.787 on TG3K, .688–.724 on the 6 per-patient USST scans). Supports chosen by Gemma-4-31B raise every anchor policy by 0.02–0.06 over DINOv2 supports on the 50 segments. TG3K has only 13 test scans, and the protocol fixes this number. The per-patient scans of the original version are given here for reference.

Reviewer_1YYwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LucidSeg, a training-free framework that uses MLLM to select evidence for few-shot ultrasound segmentation while leaving mask prediction to frozen segmentation models. At the first level, an MLLM selects annotated support examples from a retrieved candidate pool. At the second level, it evaluates predicted masks on candidate video frames and selects an anchor for SAM2 propagation. Experiments are performed across four ultrasound datasets.

Critical strengths
  1. The study addresses a relevant and interesting problem.
  2. The distinction between visual similarity and anatomical compatibility provides a reasonable motivation.
  3. The overall design is clear and easy to understand.
  4. The evaluation covers various organs/anatomical settings.
Critical weaknesses
  1. If none of the retrieved supports is suitable, or all candidate anchor masks are incorrect, the MLLM is still required to make a selection. How to handle this issue?
  2. In the Gemma exemplar setting on TG3K with UniverSeg, showing versus withholding the query yields nearly identical Dice scores, 0.484 and 0.483. This suggests that part of the improvement may come from selecting clear and representative examples rather than reasoning about the query’s anatomy.
  3. Support-set complementarity is not sufficiently examined. Four individually plausible supports may still be highly redundant.
  4. The patient-level exclusion protocol requires further clarification. The main text describes supports as coming from other subjects, whereas the appendix defines the USST exclusion unit as a vertebral segment, with multiple segments belonging to the same patient. It is therefore unclear whether all segments from the query patient are excluded from the support pool.
  5. The performance advantage over the strongest visual baseline is concentrated on TG3K.
  6. Comparisons with closely related recent approaches are needed, e.g., Context Matters, SAM-Veteran.
Questions for authors

Please see the weakness part.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1YYw (1/2)

Thank you for the review and for the six concrete points.


W1. If no retrieved support is suitable, or every candidate anchor mask is wrong, the MLLM is still forced to choose.

Such lists are rare (0–4.2%). On them every method fails, including the reranker trained on the target data. If the pool has no usable support, then no selector can recover.

We call a candidate list "without a good support" when its best 4-set is below 0.5 Dice. The best 4-set is the union-oracle, scored with ground truth.

lists without a good support (UniverSeg)TG3KMicroSegCardiacUDAUSST
share of queries4.2% (11/260)0.3% (1/355)1.0% (1/100)1.6% (4/250)
median union-oracle Dice.81.90.85.81
Dice on those queries, DINOv2 / reranker / LucidSeg_T.04 / .11 / .10.02 / .00 / .06.14 / .14 / .13.23 / .46 / .26

We then let LucidSeg_T answer "none suitable". It then falls back to DINOv2 top-4. Gemma-4-31B abstained on 6 of 965 queries (MicroSeg 4, CardiacUDA 2, none on TG3K or USST). Overall Dice changed by at most 0.011 (n.s.). On the abstained MicroSeg queries, the fallback improved Dice (.42 → .55 UvS, .59 → .73 MvS). The abstentions did not fall on the lists above. The model abstains when the transferred outlines look wrong. This is not the same as a pool without a usable support.

At Level 2, the best of the K anchors gets .825–.833 on TG3K and .742–.767 on the 50 USST segments. So no scan had all candidates fail. Multi-start uses all K predictions and does not rely on one anchor.


W2. Query shown vs. withheld is 0.484 vs 0.483 on TG3K. Part of the improvement may be clear, representative examples rather than reasoning about the query's anatomy.

Partly so. The TG3K improvement has two parts, the choice of frames and the side reading. Without the query, the model picks the supports without mirroring them. These frames alone already give +0.08 over DINOv2 (.483 vs .401), the same as the frames chosen with the query and mirrored by the side rule (.484). Most of this comes from one set of four frames that works for every query: using that set for all queries gives .467. DINOv2 similarity cannot find this set (+0.017 over random, p_u = 0.64). DINOv3 similarity can reach the same level (.495). Everything above DINOv3 on TG3K needs the query, either to read the side (.512–.542 with DINOv3 candidates) or to transfer the outlines (.501–.517). On the other three datasets, one set for all queries is significantly worse, so supports there have to be chosen for each query.

We show a fixed-set baseline as below, using the same support set for all queries. The fixed set is the four annotated frames that the decision model chose most often as supports when it did not see the query, but the candidates themselves are still retrieved by the query.

Dice UvS / MvSGemma E, query withheldone fixed set for all queriesΔ (p_u)
TG3K.483 / .622.467 / .595−0.016 / −0.027 (0.22 / 0.13)
MicroSeg.638 / .723.609 / .690−0.029 / −0.033 (< 1e-5)
CardiacUDA.682 / .778.636 / .751−0.046 / −0.027 (0.001 / 0.012)
USST.580 / .675.466 / .512−0.114 / −0.163 (< 1e-10)

W3. Complementarity. Four individually plausible supports may be redundant.

We call a set complementary when it does better than its best member used alone. How much of the support is used depends on the dataset and the executor, so it may be redundant. The number of supports is a design trade-off. We measure complementarity as the Dice of the set minus the Dice of its best single support (feed separately and take the best), as shown below:

UniverSeg, Dice(set) − best member (share of queries > 0)TG3KMicroSegCardiacUDAUSST
DINOv2 top-4+0.005 (50%)+0.102 (79%)+0.186 (95%)+0.015 (56%)
Reranker+0.013 (56%)+0.086 (76%)+0.155 (91%)+0.009 (62%)
LucidSeg_E / T, Gemma-4-31B+0.047 / +0.047 (62% / 67%)+0.096 / +0.099+0.158 / +0.178+0.010 / +0.020
LucidSeg_E / T, GPT-6 Sol+0.053 / +0.042 (66% / 63%)+0.108 / +0.093+0.160 / +0.171+0.016 / +0.028
Union-oracle+0.213 (99%)+0.207 (100%)+0.237 (100%)+0.127 (95%)

On TG3K, LucidSeg's sets add 0.04–0.05 over their best member, DINOv2 top-4 adds 0.005 and the reranker 0.013 (p_u ≤ 0.05 against DINOv2 for five of six configurations). On the other datasets all policies add about the same. The union-oracle sets add 0.13–0.24, so the remaining headroom is in how supports are combined.


W4. Patient-level exclusion on USST.

Sorry for the confusion. All segments of the query's patient are excluded. In the USST protocol, the group of each unit is its patient. The held-out rule removes every unit of the query's group from the pool. So a vertebral segment never takes supports from another segment of the same patient. The revision states this in Appendix B.

AuthorsOpenReview ↗

Response to Reviewer 1YYw (2/2)

W5. The advantage over the strongest visual baseline is concentrated on TG3K.

The scenes, applications and devices differ across datasets, so a regularity in the image features does not hold everywhere. The MLLM buys robustness where image features fail, and it needs no training. A feature-based selector works only where it holds. On USST, LucidSeg_T with DINOv3 candidates is above DINOv3 similarity (+0.031, p_q = 0.005, p_u = 0.029) and above the reranker. The table in our general response (1/2) supports this claim.


W6. Comparisons with Context Matters and SAM-Veteran.

Context Matters. It selects supports by cosine top-K on frozen DINOv3 embeddings. Its segmenter is MultiverSeg. There is no public code. We re-implemented its selection rule under our protocol (DINOv3 ViT-B/16, same pools and exclusions). It is now a row of Table 1. It gets .495 / .623 on TG3K, .728 / .808 on MicroSeg, .700 / .797 on CardiacUDA and .593 / .675 on USST. It is far above DINOv2 on TG3K (+0.09 / +0.08). LucidSeg_T with DINOv3 candidates is level with it on TG3K and CardiacUDA. It is above it on USST (Gemma, significant) and below it on MicroSeg. These cells are the "DINOv3 cand." rows of the general-response table.

Context Matters also has a failure detector. The detector is a transformer trained on segmentation outcomes of the target dataset. Our training-free setting has no such data.

SAM-Veteran. We found no public code or weights (ICLR page, OpenReview, third-party indexes). It is an RL-trained MLLM agent for natural images. It outputs a box and then adds points for SAM2. VisionReasoner belongs to the same class of methods, and is in Table 1 with its official weights (.14–.51 Dice).


Presentation. We have revised the manuscript. All changes are marked in blue.

Dear Reviewer 1YYw,

Thank you again for your review and the questions, which helped us strengthen or further clarify the paper.

As the discussion period ends soon, we would like to kindly ask whether our responses have addressed your concerns. We would greatly appreciate it if you could consider these results in your final assessment.

Thank you for your time.

Reviewer_DMbFOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes LucidSeg, a training-free framework for few-shot ultrasound segmentation that uses multimodal large language models (MLLMs) to select supporting evidence rather than directly generate segmentation masks. The framework makes two decisions: selecting annotated support examples for an in-context segmenter and choosing an optimal starting frame for SAM2-based video segmentation. Experiments across four ultrasound datasets show improvements of up to 0.11 Dice over visual similarity-based support selection, with particularly promising results on thyroid ultrasound. The main contribution is a practical way to incorporate MLLM-based anatomical reasoning into existing segmentation pipelines without additional training.

Critical strengths
  1. The paper proposes an interesting and practical use of MLLMs for segmentation. Instead of requiring an MLLM to directly predict masks or spatial prompts, the authors use it to select appropriate evidence for frozen segmentation models. This is a sensible design choice given the limitations of MLLMs in precise spatial localization. The framework is also flexible and can work with different segmentation backbones, which increases its potential usefulness.

  2. Comprehensive evaluation with meaningful ablations. The experiments cover four ultrasound datasets with different anatomical characteristics and compare against multiple support selection strategies, including a learned reranker. The authors also investigate different evidence presentation strategies, MLLM backbones, and video propagation approaches. These experiments provide useful insights into when MLLM-based evidence selection is beneficial and when simpler methods may be sufficient.

Critical weaknesses
  1. The performance improvements are inconsistent across datasets, raising questions about generalizability. The main gains are concentrated on TG3K, where anatomical side and orientation play important roles. On MicroSeg, the proposed method underperforms DINOv2 similarity, while improvements on CardiacUDA and USST are relatively small or inconsistent. Furthermore, the reported results suggest that the MLLM's anatomical understanding is not always reliable. For example, the anatomical side is correctly identified in only 68% and 77% of clear TG3K cases using Gemma-4-31B and GPT-5.5, respectively. These results make it difficult to determine whether the proposed framework provides a broadly applicable improvement or mainly addresses particular dataset-specific challenges.

  2. The benefit of MLLM reasoning over simpler selection strategies is not sufficiently established, particularly for video segmentation. In Table 2, the executor-confidence baseline achieves 0.802 Dice on TG3K with Gemma-selected supports, exceeding the MLLM anchor selection result of 0.781. Moreover, on USST, multi-start propagation consistently outperforms MLLM-based single-anchor selection. Since the proposed approach introduces additional MLLM inference and potentially substantial computational overhead, its practical advantage over simpler automated selection methods remains unclear. This is particularly important because the paper's central claim is that MLLMs provide useful evidence-selection capabilities beyond conventional approaches.

Questions for authors
  1. Could the authors provide a more detailed analysis of why MLLM-based support selection improves performance substantially on TG3K but offers limited or negative gains on other datasets? In particular, how much of the improvement can be attributed to anatomical side identification and mirroring rather than more general anatomical reasoning?

  2. In Table 2, the executor-confidence baseline outperforms MLLM-based anchor selection in some settings. Could the authors clarify under what conditions MLLM anchor selection is preferable to confidence-based selection or multi-start propagation? Would an adaptive strategy combining these approaches provide more consistent improvements?

  3. What is the computational overhead of the proposed framework compared with DINOv2 retrieval and executor-confidence-based selection, including candidate preparation and MLLM inference? Given the relatively modest improvements on several datasets, a comparison of segmentation accuracy against computational cost would help establish the practical value of the method.

3 comments
AuthorsOpenReview ↗

Response to Reviewer DMbF (1/2)

Thank you for the detailed review and for the three precise questions.


Q1. How much of the TG3K improvement is side identification and mirroring rather than general anatomical reasoning?

The TG3K improvement is one anatomical judgment plus a better choice of frames, not general anatomical reasoning. About half of the +0.08 over DINOv2 is the side reading and half is the choice of frames.

The table shown below keeps the same four frames in each row and changes only their orientation. The side rule mirrors every support whose gland lies on the other side from the query's, and the columns differ in who reads the query's side.

TG3K, Dice UvS / MvSno mirroringorientation by cosineside rule, Gemma sideside rule, GPT-5.5 sideside rule, GPT-6 Sol sideside rule, GT side
DINOv2 frames.405 / .549.401 / .544.443 / .576.463 / .599.466 / .610.473 / .610
Reranker frames.473 / .601.472 / .603.485 / .612.487 / .614.491 / .618.495 / .621
LucidSeg_E frames (Gemma).460 / .588n/a.484 / .611.504 / .625.513 / .642.514 / .649

Giving DINOv2's own frames Gemma's side reading recovers +0.042 / +0.032 of the +0.084 / +0.066. The frames LucidSeg_E chose add the rest (+0.041 / +0.035). They are above DINOv2 even without mirroring (.460 vs .401 UvS, p_u = 0.033).


W1. Improvements are concentrated on TG3K.

On MicroSeg and CardiacUDA, the target has one typical appearance, so retrieval by appearance already finds suitable supports and the MLLM has little to add. On CardiacUDA even random supports are only about 0.05 Dice below the best row (.663 / .757 vs .714 / .797). On USST the best visual baseline is the reranker, which requires training on the target dataset. LucidSeg_T matches it without training (.604 / .679 vs .616 / .678). The table in our general response (1/2) gives every row.


W1. The side is read correctly in only 68% / 77%.

The side reading does not need to be perfect. Gemma-4-31B / GPT-5.5 / GPT-6 Sol read the side correctly in 68% / 77% / 78% of the queries, and replacing their reading with the ground-truth side adds only 0.00–0.04 Dice (last column of the table below, Appendix H). The transfer mode does not use the side reading at all and reaches .508 / .635.


W2 / Q2. Multi-start wins on USST. When is which preferable?

A single anchor is preferable where one structure persists through the scan (TG3K). Multi-start is preferable where structures vary or repeat (USST). On TG3K the LLM anchor is 0.08–0.12 above multi-start (p ≤ 0.017). On the 50 USST segments (in the revision) multi-start is 0.015–0.026 above the LLM anchor (p = 0.04–0.12), and the LLM anchor is the best single start in every chain.

USST, MultiverSeg, K = 8middleconfidenceLLM anchorall K (multi-start)anchor oracle
50 segments, DINOv2 supports.642.632.667.686.742
50 segments, Gemma supports.661.695.702.726.767
50 segments, GPT-5.5 supports.633.674.703.718.752
50 segments, GPT-6 Sol supports.630.683.699.725.759

Q2. Would an adaptive strategy help?

We considered three kinds of strategies. The added design complexity has to be weighed against the benefit. None of the following intuitive strategies improves on the better fixed policy of each dataset. TG3K numbers below are for supports chosen by DINOv2 / Gemma-4-31B / GPT-5.5 / GPT-6 Sol.

  • Switching between one start and multi-start. The switch is one extra MLLM call per scan, deciding to use one start or multi-start. The MLLM sees all candidate frames with their predicted outlines. If it predicts the target is one continuous structure, the one start applies; if it predicts repeat structures (original USST), the multi-start applies. On TG3K, the single start gets .780 / .781 / .785 / .807. The switch gets .780 / .764 / .772 / .787. On USST, the switch was run on the 6 scans of the original version with the first three chains, where vertebrae repeat. The single start gets .678 / .665 / .632. Multi-start gets .712 / .726 / .724. The switch gets .688 / .701 / .724.
  • Switching between anchors. The LLM anchor alone gets .780 / .781 / .785 / .807. Rule 1 takes the most confident candidate when the executor's confidence on the LLM anchor is low. It gets .781 / .781 / .785 / .757. Rule 2 drops the LLM anchor when it is the first or last candidate. It gets .766 / .781 / .785 / .807. Neither rule is above the LLM anchor. The best candidate would get .825–.833. Confidence and frame position do not find it.
  • X starts from the K candidates. A prior rule could be applied, e.g., a threshold on the shape change, or an RL-trained policy. The possible benefit is small, because the best single candidate is 0.02–0.05 above the LLM anchor (TG3K) and 0.03–0.06 above multi-start on (USST) (anchor oracle, Table 2).
AuthorsOpenReview ↗

Response to Reviewer DMbF (2/2)

W2. Executor confidence beats the LLM anchor in one chain.

Confidence is ahead in one of the three TG3K chains. None of the three differences is significant. The chains differ in who chose the supports, DINOv2 / Gemma / GPT-5.5. The LLM anchor gets .780 / .781 / .785. Confidence gets .737 / .802 / .754. Confidence is ahead by 0.022 on the Gemma chain (p = 0.22). The LLM anchor is ahead by 0.042 and 0.032 on the other two (Table 7). With GPT-6 Sol as the anchor model, the LLM anchor gets .810 / .787 / .776 / .807 on the four TG3K chains. It is above confidence on three of them. On the 50 USST segments, the LLM anchor is above confidence in all four chains (.667 / .702 / .703 / .699 vs .632 / .695 / .674 / .683, supports by DINOv2 / Gemma / GPT-5.5 / GPT-6 Sol).


Q3. Computational overhead against DINOv2 retrieval and executor confidence, and accuracy vs. cost.

The MLLM adds one call per query. It buys robustness where image features fail, and it needs no training. DINOv2 similarity and the reranker are Level 1 baselines. Executor confidence is a Level 2 baseline. The scenes, applications and devices differ across datasets, so a regularity in the image features does not hold everywhere. A feature-based selector works only where it holds.

(1) On TG3K, the gland is brighter than its surroundings (105 vs. 73) and has an irregular shape (solidity 0.73). It splits into two parts in 46% of the frames. No simple appearance cue holds. DINOv2 similarity is at the level of random supports, and LucidSeg is 0.08–0.14 above it.

(2) On CardiacUDA, the chambers are dark, convex regions with a stable size (mean intensity 20 inside the mask vs. 80 around it). Appearance is a reliable cue. Even random supports get .663 / .757, close to DINOv2 similarity (.682 / .786).

(3) On MicroSeg, the prostate is one convex region that covers about 40% of the image (solidity 0.99, one component in every frame). Appearance is again a reliable cue.

(4) On USST, the target is very small (2.6% of the image) and splits into several parts in 96% of the frames. Appearance is an unreliable cue. DINOv2 similarity recovers −0.01 to 0.39 of the gap between random and oracle supports. LucidSeg_T recovers 0.18–0.29 on every dataset.

In practice, the choice is a trade-off between efficiency, cost, accuracy and robustness.

Per query on TG3K, on one shared RTX PRO 6000 (MLLM times from the logged runs):

stagetimeused by
DINOv2 feature of the query + cosine ranking32 mssimilarity, reranker, LucidSeg
template matching over the pool (CPU)0.45 sreranker, LucidSeg
rendering 17 candidate tiles (+ transfer warps)41 ms (91 ms)LucidSeg
one MLLM call, 18 images, Qwen3.5-9B / Gemma-4-31B W4A16 (local vLLM)5–6 s / 12–16 sLucidSeg
one MLLM call, GPT-5.5 / GPT-6 Sol without reasoning (API, median of 10 calls)5.4 s / 4.7 sLucidSeg (API variant)
UniverSeg / MultiverSeg forward9 / 40 msall support-based rows
reranker inference< 1 msreranker

One-time cost of the reranker, per dataset and executor (TG3K):

stagewhattime
1. labelsthe single-support Dice of every allowed candidate for every labelled query: 300 queries with ground-truth masks × ≈ 290 candidates ≈ 87k executor forwards3 min with UniverSeg, 16 min with MultiverSeg
2. training15 leave-one-unit-out fits of the gradient-boosted model on these labels≈ 2 s

Althogh the reranker trains fast, its needs ground-truth masks on the target dataset. Also, it is tied to one executor while does not always transfer: trained on UvS or MvS Dice, it is no better than random supports with Matcher (Appendix J). Its improvement also varies across datasets.

Through the API, one TG3K call uses about 5.8k input and 80–105 output tokens. The reranker's cost is in its labels: they need ground-truth masks on the target dataset, and they are recomputed for every executor.

At Level 2, one anchor call (7–10 s) takes less time than the propagation itself (17.8–33.3 s per scan). The executor takes 0.33 s per scan on the K = 8 candidates. SAM2.1-Large takes 17.8 s to propagate from one anchor and 33.3 s for multi-start.

Dear Reviewer DMbF,

Thank you again for your review and the questions.

As the discussion period ends soon, we would like to kindly ask whether our responses have addressed your concerns. We would greatly appreciate it if you could consider these results in your final assessment.

Thank you for your time.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response (1/2)

We thank the reviewers for the careful reading and thoughtful feedback. Here we answer the concern shared by all three reviewers; part 2/2 lists the changes to the paper, and individual points are answered in each thread.


Common Question 1 (DMbF W1/Q1, 1YYw W5, 1T5j W1): improvements concentrated on TG3K and below DINOv2 on MicroSeg; is the MLLM's advantage over visual selection established, and where does it apply?

We want to clarify that the paper's claim was never that visual similarity is bad. The scenes, applications and devices differ across datasets, so a regularity in the image features does not hold everywhere. A feature-based selector works only where its features cover the distinction that matters for the query, and this is not known in advance. The MLLM needs no training and makes its choice per query. The MLLM buys robustness where image features fail, and takes more of its supports from the feature-based candidates when they work.

We therefore added DINOv3 ViT-B/16 (Context Matters' rule, our implementation) as a baseline and as the proposer of LucidSeg's candidates, and GPT-6 Sol (same protocol as GPT-5.5):

Dice UvS / MvS (% picks: similarity / template / random-only)TG3KMicroSegCardiacUDAUSST
Random supports.384 / .547.582 / .702.663 / .757.545 / .604
DINOv2 similarity.401 / .544.718† / .802.682 / .786.570 / .646
DINOv3 similarity = Context Matters (our implementation).495 / .623.728 / .808.700 / .797.593 / .675
Reranker (trained on target).472 / .603.687 / .765.688 / .780.616 / .678†
LucidSeg_E, Gemma, DINOv2 cand..484 / .611 (45/25/31).694 / .771 (56/9/35).665 / .767 (45/29/29).576 / .650 (47/28/25)
LucidSeg_T, Gemma, DINOv2 cand..508 / .635 (38/38/27).674 / .748 (54/1/44).703 / .784 (40/43/22).604 / .679 (42/33/26)
LucidSeg_E, GPT-5.5, DINOv2 cand..498 / .618 (46/25/31).698 / .754 (59/7/34).687 / .776 (38/35/30).593 / .666 (47/29/24)
LucidSeg_T, GPT-5.5, DINOv2 cand..497 / .633 (40/36/28).663 / .738 (52/1/47).703 / .787 (40/40/24).593 / .654 (37/39/25)
LucidSeg_E, GPT-6 Sol, DINOv2 cand..508 / .626 (46/28/28).716 / .781 (62/5/33).676 / .777 (42/35/27).593 / .656 (50/33/18)
LucidSeg_T, GPT-6 Sol, DINOv2 cand..502 / .635 (39/37/27).669 / .748 (54/2/44).712 / .794† (40/42/24).615† / .673 (43/33/24)
LucidSeg_E, Gemma, DINOv3 cand..512 / .629 (54/23/26).712 / .786 (57/9/35).669 / .766 (44/34/27).598 / .669 (49/32/21)
LucidSeg_T, Gemma, DINOv3 cand..508 / .627 (44/35/25).674 / .752 (55/2/44).708 / .791 (37/46/23).623 / .691 (46/30/26)
LucidSeg_E, GPT-5.5, DINOv3 cand..523 / .647 (51/24/27).712 / .769 (63/6/31).684 / .773 (38/36/30).597 / .656 (51/25/25)
LucidSeg_T, GPT-5.5, DINOv3 cand..501 / .657 (42/35/27).670 / .742 (55/1/44).711† / .791 (37/42/25).609 / .663 (41/38/22)
LucidSeg_E, GPT-6 Sol, DINOv3 cand..542 / .654† (52/26/25).739 / .799† (66/7/28).680 / .765 (39/40/26).607 / .661 (59/27/16)
LucidSeg_T, GPT-6 Sol, DINOv3 cand..517† / .655 (43/34/27).669 / .743 (55/1/43).714 / .795 (38/43/23).623 / .678† (49/32/21)
Union-oracle (best set in list).779 / .860.875 / .921.824 / .869.789 / .838

Bold: best per column (support-based rows); italic: second; †: third.

Three points (§5.1, Tables 1 and 12).

1. The MLLM's improvement is steady across datasets and does not depend on the retriever. In recovered headroom (0 = random, 1 = oracle), DINOv2 similarity recovers −0.01 to 0.39 and the reranker 0.12–0.30, whereas LucidSeg_T with Gemma-4-31B recovers 0.18–0.29 on every dataset and executor. With DINOv3 instead of DINOv2 candidates, LucidSeg_T changes by at most 0.025 and reaches .623 / .691 on USST, above DINOv3 similarity (+0.031, p_u = 0.029) and the reranker. Against DINOv3 similarity it is level on TG3K and CardiacUDA and below on MicroSeg (−0.054 / −0.056, p_u < 1e-5), the stated exception.

2. A stronger MLLM closes the MicroSeg gap. GPT-6 Sol in the exemplar mode with DINOv3 candidates reaches .739 / .799 on MicroSeg, level with DINOv3 similarity (p_u ≥ 0.19), and .542 / .654 on TG3K (+0.047 over DINOv3 similarity, p_u = 0.017). The transfer mode differs by at most 0.03 between Gemma-4-31B, GPT-5.5 and GPT-6 Sol.

3. When the visual features are good, the MLLM takes more supports from their candidates (table; uniform ≈ 35 / 35 / 32). The share from the similarity candidates rises from 38% to 44% when DINOv3 replaces DINOv2 (Gemma_T, TG3K) and is largest on MicroSeg (52–66%). On CardiacUDA, where template matching gives the best frames, most picks come from it (40–46%). Where appearance retrieval already suffices, the MLLM adds nothing.

AuthorsOpenReview ↗

General response (2/2)

Changes to the paper (blue in the revised PDF).

We add new experiments, including the Context Matters baseline and DINOv3 candidates, GPT-6 Sol and DeepSeek-V4-Flash-Vision-Exp, Level 2 on all 50 USST segments with every model's anchor decision on every chain, unit-level tests and a fixed-support-set control, and revise the wording. Specifically:

  • Table 1. Context Matters row (our implementation, DINOv3 similarity); LucidSeg with DINOv3 candidates for Gemma-4-31B and GPT-5.5; four GPT-6 Sol rows; the share of picks from each candidate source under every LucidSeg row; Δ against the best visual-based row; best / second / third per column; same-video reference on CardiacUDA and USST.
  • Table 2. USST on all 50 test vertebral segments instead of 6 joined scans; a GPT-6 Sol support chain and anchor row; every model's anchor decision on every chain (no n/a cells); a new appendix table with paired differences, CIs and p-values; a new appendix table with the six joined scans of the original version.
  • Table 3. GPT-6 Sol rows and column; a fixed-support-set control.
  • Figure 3. (a) Context Matters column and the best configuration of each model (Gemma-4-31B transfer mode, GPT-6 Sol exemplar mode, DINOv3 candidates); (b) rows LLM anchor, executor confidence, middle frame, first frame.
  • §3. The three sources of the candidate list named; template matching described.
  • §4. SAM 3 in the baselines; the unit-level test and recovered headroom defined; DINOv3 variant and Level-2 models stated.
  • §5.1. Four observations (steadier improvement across datasets, independent of the proposer, a stronger MLLM closes the MicroSeg gap, picks follow the better features); unit-level p-values next to query-level ones; "matched" the reranker.
  • §5.2. 50-segment USST results with p-values; the anchor decision with GPT-6 Sol and across chains; the same comparison on the six joined scans of the original version.
  • §6. Query shown or withheld for GPT-6 Sol; the fixed-set control.
  • Appendices. A: limitations rewritten (a structure that changes along the scan; the exemplar mode depends on the anatomical reading). B: USST exclusion rule. D: unit-level test. I: Level 2 on the six joined scans of the original version, with paired differences; the USST rows of the raw-frame table labelled as joined scans from 6 patients. F: DINOv3 settings, DINOv3 candidates, per-stage timings, GPT-6 Sol and DeepSeek settings. G: Context Matters and SAM 3. K: GPT-6 Sol and DeepSeek-V4-Flash-Vision-Exp in the model comparison. N: recovered-headroom table, DINOv3 and GPT-6 Sol rows in the IoU/HD95 table, a third forest-plot panel.
  • Wording and layout. Abstract, contributions and conclusion state the comparison with the best visual policy per dataset, and the abstract says that on spine scans the structures change along the scan instead of "repeating vertebrae"; "sweep" → "scan", "gain" → "improvement"; appendix floats in reading order; figure rasters at 300 dpi.
AuthorsOpenReview ↗

Final Response and Summary to Reviewers and AC

Dear Reviewers and AC,

We sincerely thank all three reviewers and the Area Chair for their time and constructive feedback. We have carefully addressed every concern, conducted extensive additional experiments, and substantially strengthened the manuscript. We would like to highlight the main outcomes of the rebuttal.

1. Stronger baselines and a more precise characterization of the contribution. We added Context Matters (DINOv3), using DINOv3-based candidates, and stronger MLLMs (GPT-6). These results clarify that LucidSeg's advantage is not universal superiority over visual retrieval, but training-free evidence selection that remains useful when visual features are insufficient. With DINOv3 candidates, LucidSeg significantly improves over DINOv3 similarity on USST (+0.031 Dice, unit-level p = 0.029), while GPT-6 Sol improves TG3K (+0.047 Dice, p = 0.017). We also acknowledge the strong visual baseline on MicroSeg.

2. More rigorous statistical validation and leakage analysis. We recomputed significance tests at the patient/scan level. The main improvements over DINOv2 remain significant on TG3K and USST. We also conducted stricter support-pool exclusion experiments. The results shows that the TG3K advantage remains robust under wider exclusions. Claims not supported by the revised statistical analysis have been corrected.

3. Additional analyses of the MLLM's actual contribution. We disentangled anatomical side identification from support-frame selection, introduced fixed-support-set controls, analyzed support complementarity and failure cases, and provided detailed computational costs. These experiments clarify 1) what the MLLM contributes, 2) when simpler alternatives suffice, and 3) the associated efficiency trade-offs.

4. Expanded full-pipeline evaluation. We extended Level-2 experiments to all 50 USST test segments, evaluated every anchor-selection strategy across four support chains, and examined adaptive propagation. The MLLM consistently provides the strongest single-anchor selection on USST, and multi-start propagation performs better when anatomical structures change along the scan. These findings establish a clearer distinction between the settings in which each strategy is appropriate.

All changes are incorporated into the revised manuscript and highlighted in blue.

We believe these additional experiments and clarifications substantially strengthen the paper's empirical foundation. We hope this addresses the central concerns raised by reviewers.

We also appreciate that the reviewers recognized the practical and conceptually clear design of using MLLMs for evidence selection while leaving mask prediction to frozen segmentation models, as well as the broad experimental evaluation across different ultrasound datasets and segmentation settings.

We sincerely appreciate the reviewers' and Area Chair's consideration of the revised work.

Warm regards,

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.