acceptodds
Under review as a conference paper at ICLR 2027

Discover and Verify: Fine-Grained Video Object Segmentation with Weak-Text Prompts

Abstract

Language-guided video segmentation converts text queries into spatial prompts, reducing the need for manual spatial annotations. However, this paradigm commonly relies on an implicit assumption that semantic similarity is sufficient to determine the identity of the target. This assumption often fails in fine-grained settings where only an easily available category name is provided as a weak-text prompt. Visually similar objects from different categories may receive similar semantic similarity scores, making it difficult to reliably distinguish the target from confusing objects. To address this issue, we propose a selective weak-text prompted fine-grained segmentation framework based on a “discover-and-verify” mechanism. Confuser-aware Prompt Discriminative Grounding (CPDG) compares the target category with potential confuser categories to learn latent discriminative cues. These cues are then used to locate and aggregate local discriminative evidence within candidate regions, allowing CPDG to rerank the candidate regions. Semantic Consistency Verification (SCV) further combines semantic and visual evidence to verify the top-ranked candidate and rejects the candidate hypothesis when the available evidence is insufficient. Extensive experiments on two weak-text fine-grained datasets, EndoVis17 and EndoVis18, demonstrate the effectiveness of our method, which consistently improves weak-text prompted fine-grained segmentation performance and improves mcIoU, our primary evaluation metric, over the strongest compared method by 1.31% and 16.91%, respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Weak-Text Prompts, Fine-Grained Video Object Segmentation, Vision-Language Grounding, Discriminative Visual Evidence, Semantic Consistency Verification

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7QudOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies fine-grained surgical instrument segmentation from category-name-only (“weak-text”) queries. The proposed Discover-and-Verify (DaV) framework first obtains candidate boxes from GroundingDINO. Its Confuser-aware Prompt Discriminative Grounding (CPDG) module mines local evidence, models target–confuser differences, and reranks candidates. Semantic Consistency Verification (SCV) then assesses the top candidates and rejects unsupported target hypotheses. Accepted boxes are passed to a frozen SAM3 model for mask generation and video propagation. Evaluation on EndoVis17 and EndoVis18 reports segmentation, candidate discrimination, and absent-query rejection metrics; the paper reports mcIoU of 68.34% and 82.39%, respectively.

Critical strengths
  1. Relevant failure modes. Category-name-only queries are practical in the surgical setting, and the confusion between visually similar instruments and the rejection of absent targets are worthwhile problems to address.
  2. Diagnostic evaluation. The paper reports TSA/CER and rejection accuracy/AUROC in addition to IoU/mcIoU, and separates CPDG and SCV in ablations. This provides some insight into the roles of reranking and hypothesis verification.
Critical weaknesses
  1. Tables 1–2 compare only CLIPSeg (CVPR 2022), CRIS (CVPR 2022), TP-SIS (NeurIPS 2023), SurgicalSAM (AAAI 2024; the manuscript references its 2023 version), and RoSIS (2024). There is no 2025–2026 method in the main results, despite this being an ICLR 2027 submission.

  2. The paper itself discusses SAMWISE (CVPR 2025) and ReferDINO (ICCV 2025) but does not compare against either; MPG-SAM 2 (ICCV 2025) is another relevant recent text-driven video segmentation method.

  3. The methodological novelty appears incremental rather than a clearly differentiated new principle.

Questions for authors
  1. Could the authors provide comparisons with recent text-driven video segmentation methods, such as SAMWISE, ReferDINO, and MPG-SAM 2, under the same category-name-only prompts, dataset splits, and evaluation protocol?

  2. What is the core methodological novelty of the Discover-and-Verify framework? Beyond combining local evidence mining, target–confuser contrastive learning, candidate reranking, and learned rejection, do CPDG and SCV introduce a substantively new discrimination or verification mechanism? Please distinguish the methodological advances from adaptations specifically tailored to weak-text surgical instrument segmentation.

  3. Could the authors provide finer-grained ablations to quantify the individual contributions of dual-path local evidence mining, confuser modeling, and latent discriminative attribute prototypes in CPDG?

  4. What substantively distinguishes SCV’s “hypothesis verification” from standard supervised candidate classification or confidence-based rejection?

6 comments

We appreciate the reviewer’s recognition of the relevance of our diagnostic evaluations and the contributions of CPDG and SCV. We have addressed the concerns and revised the manuscript accordingly. Modifications are highlighted in blue.

W1, W2, Q1: Comparisons with recent methods

We thank you for this valuable suggestion. We have substantially expanded the experimental comparisons by including recent methods: SAMWISE (CVPR 2025), ReferDINO (ICCV 2025), MPG-SAM 2 (ICCV 2025), and SurgRef (AAAI 2026). These methods represent recent advances in text-driven video segmentation. To ensure a fair and controlled comparison, we evaluate all reproduced methods under consistent dataset splits, textual inputs, and evaluation protocols.

(1) Dataset and split. We follow the dataset-specific evaluation protocols adopted by TP-SIS. For EndoVis17, we conduct four-fold cross-validation over the eight annotated training sequences, using six sequences for training and two for validation in each fold. For EndoVis18, we follow the corresponding TP-SIS split, using 11 sequences for training and the remaining four sequences for evaluation.

(2) Evaluation protocol. We evaluate predictions on the same annotated video frames using the same ground-truth masks and IoU/mcIoU implementation. We train and recompute all results on both datasets. As shown in the revised Tables 1–2, DaV achieves 67.27% and 80.46% mcIoU on EndoVis17 and EndoVis18, respectively, compared with 32.19%/38.56% for SAMWISE, 42.01%/48.24% for ReferDINO, and 42.90%/61.07% for MPG-SAM 2. This corresponds to improvements of 24.37% and 19.39% over the strongest reproduced baseline, MPG-SAM 2, on the two datasets, respectively.

These comparisons provide a direct assessment of recent RVOS methods under the weak text input. They demonstrate that fine-grained surgical instrument segmentation with weak text prompts requires more than general text-video correspondence. By explicitly discovering target-versus-confuser local evidence and verifying candidate reliability, DaV provides additional benefits in visually ambiguous surgical scenes. We have incorporated these results and the corresponding experimental details into the revised manuscript.

 

Table 1: Overall comparison with existing methods on EndoVis18 under different prompting protocols. The highest result is bolded, while the second-highest result is underlined.

\hline \text{Prompt} & \text{Method} & \text{IoU} & \text{mcIoU} & \begin{array}{c} \text{Instrument Categories} \\\\ \begin{array}{rrrrrrr} \text{BF} & \text{PF} & \text{LND} & \text{SI} & \text{CA} & \text{MCS} & \text{UP} \end{array} \end{array} \\\\ \hline \text{Class Prototype} & \text{SurgicalSAM} & \underline{80.33} & 58.87 & \begin{array}{rrrrrrr} \underline{83.66} & 65.63 & 58.75 & 54.48 & 39.78 & 88.56 & 21.23 \end{array} \\\\ \hline \begin{array}{c} \text{} \\\\ \text{} \\\\ \text{Rich Text} \\\\ \text{} \end{array} & \begin{array}{l} \text{RoSIS} \\\\ \text{TP-SIS(448)} \\\\ \text{CLIPSeg} \\\\ \text{CRIS} \end{array} & \begin{array}{r} 76.16 \\\\ \mathbf{81.54} \\\\ 69.86 \\\\ 72.29 \end{array} & \begin{array}{r} 44.54 \\\\ 65.48 \\\\ 39.70 \\\\ 46.04 \end{array} & \begin{array}{rrrrrrr} \mathbf{84.11} & 42.73 & 35.26 & 46.36 & 0.00 & 88.10 & 15.22 \\\\ 81.53 & 70.18 & 71.54 & \mathbf{90.58} & 21.46 & 65.57 & 57.51 \\\\ 67.25 & 39.59 & 50.72 & 47.27 & 2.92 & 79.96 & 4.22 \\\\ 73.58 & 58.20 & 47.64 & 72.14 & 4.56 & 45.99 & 20.18 \end{array} \\\\ \hline \begin{array}{c} \text{} \\\\ \text{} \\\\ \text{} \\\\ \text{} \\\\ \text{Weak Text} \\\\ \text{} \\\\ \text{} \\\\ \text{} \\\\ \text{} \end{array} & \begin{array}{l} \text{TP-SIS} \\\\ \text{CLIPSeg} \\\\ \text{CRIS} \\\\ \text{SAMWISE} \\\\ \text{ReferDINO} \\\\ \text{MPG-SAM 2} \\\\ \text{SurgRef} \\\\ \text{\bf DaV (frame)} \\\\ \text{\bf DaV (video)} \end{array} & \begin{array}{r} 67.24 \\\\ 50.40 \\\\ 64.77 \\\\ 52.57 \\\\ 55.36 \\\\ 71.45 \\\\ 40.36 \\\\ \mathbf{77.65} \\\\ \underline{77.40} \end{array} & \begin{array}{r} 44.67 \\\\ 37.00 \\\\ 49.51 \\\\ 38.56 \\\\ 48.24 \\\\ 61.07 \\\\ 32.68 \\\\ \underline{80.46} \\\\ \mathbf{82.39} \end{array} & \begin{array}{rrrrrrr} 71.39 & 42.24 & 35.73 & 50.46 & 23.01 & 73.70 & 16.18 \\\\ 66.74 & 44.80 & 36.22 & 31.25 & 2.21 & 76.56 & 1.25 \\\\ 73.20 & 41.48 & 43.73 & 35.59 & 41.13 & 81.44 & 30.00 \\\\ 67.19 & 49.26 & 47.80 & 27.73 & 12.31 & 27.64 & 37.99 \\\\ 68.00 & 41.79 & 34.64 & 43.62 & 30.24 & 86.33 & 31.09 \\\\ 86.76 & 51.52 & 33.28 & 67.86 & 56.44 & 93.74 & 37.92 \\\\ 33.14 & 22.52 & 22.63 & 36.81 & 28.74 & 61.26 & 24.12 \\\\ 81.92 & 72.28 & \underline{80.01} & 78.82 & \underline{79.50} & \underline{90.48} & \underline{80.94} \\\\ 82.17 & \mathbf{72.43} & \mathbf{83.87} & \underline{79.00} & \mathbf{79.53} & \mathbf{94.80} & \mathbf{84.90} \end{array} \\\\ \hline \end{array}$$

Table 2: Overall comparison with methods on EndoVis17 under different prompting protocols. The highest result is bolded, while the second highest result is underlined. * denotes frame-wise evaluation under the four-fold cross-validation protocol on EndoVis17.

\hline \text{Prompt} & \text{Method} & \text{IoU} & \text{mcIoU} & \begin{array}{c} \text{Instrument Categories} \\\\ \begin{array}{rrrrrrr} \text{BF} & \text{PF} & \text{LND} & \text{VS} & \text{GR} & \text{MCS} & \text{UP} \end{array} \end{array} \\\\ \hline \text{Class prototype} & \text{SurgicalSAM} & \underline{69.94} & 67.03 & \begin{array}{rrrrrrr} 68.30 & 51.77 & 75.52 & 68.24 & \mathbf{57.63} & 86.95 & 60.80 \end{array} \\\\ \hline \begin{array}{c} \text{} \\\\ \text{} \\\\ \text{Rich Text} \\\\ \text{} \end{array} & \begin{array}{l} \text{RoSIS} \\\\ \text{TP-SIS(448)} \\\\ \text{CLIPSeg} \\\\ \text{CRIS} \end{array} & \begin{array}{r} 57.33 \\\\ \mathbf{76.45} \\\\ 65.02 \\\\ 67.83 \end{array} & \begin{array}{r} 40.65 \\\\ 54.78 \\\\ 33.42 \\\\ 38.21 \end{array} & \begin{array}{rrrrrrr} 67.34 & 39.95 & 55.85 & 45.32 & 5.10 & 26.00 & 44.99 \\\\ \underline{69.57} & \mathbf{68.91} & \mathbf{89.88} & \mathbf{82.60} & 0.00 & 72.53 & 0.00 \\\\ 51.29 & 42.27 & 49.56 & 30.12 & 9.96 & 30.69 & 20.05 \\\\ 54.87 & 50.21 & 68.33 & 50.12 & 0.00 & 43.97 & 0.00 \end{array} \\\\ \hline \begin{array}{c} \text{} \\\\ \text{} \\\\ \text{} \\\\ \text{} \\\\ \text{Weak Text} \\\\ \text{} \\\\ \text{} \\\\ \text{} \\\\ \text{} \end{array} & \begin{array}{l} \text{TP-SIS(448)} \\\\ \text{CLIPSeg} \\\\ \text{CRIS} \\\\ \text{SAMWISE} \\\\ \text{ReferDINO} \\\\ \text{MPG-SAM 2} \\\\ \text{SurgRef} \\\\ \text{\bf DaV* (frame)} \\\\ \text{\bf DaV (video)} \end{array} & \begin{array}{r} 51.97 \\\\ 42.17 \\\\ 46.23 \\\\ 45.45 \\\\ 48.33 \\\\ 50.75 \\\\ 39.73 \\\\ 67.22 \\\\ 67.91 \end{array} & \begin{array}{r} 39.28 \\\\ 29.25 \\\\ 31.41 \\\\ 32.19 \\\\ 42.01 \\\\ 42.90 \\\\ 31.20 \\\\ \underline{67.27} \\\\ \mathbf{68.34} \end{array} & \begin{array}{rrrrrrr} 58.83 & 43.26 & 70.03 & 51.63 & 0.00 & 51.23 & 0.00 \\\\ 46.49 & 33.50 & 42.61 & 28.50 & 5.83 & 29.00 & 18.83 \\\\ 51.85 & 42.30 & 69.29 & 43.71 & 0.00 & 12.70 & 0.00 \\\\ 62.33 & 15.05 & 16.69 & 39.09 & 23.81 & 58.56 & 9.80 \\\\ 52.06 & 50.74 & 68.83 & 36.41 & 35.73 & 64.25 & 36.06 \\\\ 55.11 & 48.16 & 60.80 & 30.21 & 29.74 & 46.44 & 29.84 \\\\ 35.06 & 36.73 & 41.97 & 26.60 & 28.11 & 30.04 & 19.89 \\\\ \underline{69.38} & \underline{55.09} & 67.51 & 66.62 & \underline{55.07} & \underline{94.83} & \underline{62.41} \\\\ \mathbf{70.09} & \mathbf{57.33} & \underline{69.42} & \mathbf{68.57} & 54.52 & \mathbf{94.99} & \mathbf{63.44} \end{array} \\\\ \hline \end{array}$$ ## **W2: Core Methodological Novelty** We thank you for this important question. We respectfully clarify that **our methodological contribution is an evidence-driven selective grounding principle for weak-text fine-grained video object segmentation.** In weak-text fine-grained video object segmentation, category-name-only prompts provide limited cues for distinguishing confusers (i.e., instruments that are both textually and visually similar). As shown in Fig. 1(b), GroundingDINO ranks a confuser above the true target in 51.8% of the evaluated target-confuser pairs. To address this challenge, **our method couples target-confuser-conditioned discriminative grounding with evidence-based candidate verification.** While existing studies have mitigated target ambiguity by enhancing semantic representations, cross-modal alignment, and category-level discrimination, there remains insufficient research on explicitly mining and evaluating the differential support of local evidence for targets versus confusers during candidate localization, and further verifying the validity of specific candidate hypotheses. **DaV addresses this gap through two complementary mechanisms: CPDG jointly models target-attribute support and confuser compatibility to mine and distinguish local candidate evidence, while SCV leverages this evidence to verify, re-select, or reject candidate hypotheses.** The experimental results validate the effectiveness of DaV. By jointly modeling target-attribute support and confuser compatibility, CPDG reduces the CER from 52.69%/51.80% to 33.43%/30.15% on EndoVis17/EndoVis18, indicating improved ranking of true targets over confusing candidates. By verifying multiple candidates using multi-source discriminative evidence, SCV improves the AUROC from 55.91% to 78.43% and the absent-query rejection accuracy from 80.22% to 94.13% on EndoVis17, with consistent gains on EndoVis18 (AUROC: 62.57%→80.43%; rejection accuracy: 84.09%→96.96%). **These results demonstrate the complementary contributions of CPDG and SCV: CPDG improves target-confuser discrimination, while SCV provides evidence-based candidate verification and rejection.**

Q2: Core Methodological Novelty of the Discover-and-Verify Framework

We thank the reviewer for raising this important question. We clarify that the methodological contribution of DaV lies in addressing two complementary yet distinct challenges in weak-text fine-grained video object segmentation: (1) distinguishing the true target from visually similar confusers, and (2) determining whether a candidate is sufficiently supported to be accepted rather than merely being the highest-ranked proposal.

(1) Target–confuser discrimination.

Category-name-only prompts provide limited cues for distinguishing visually similar instruments. Consequently, semantic similarity alone may favor a confuser over the true target. As shown in Fig. 1(b), GroundingDINO ranks a confuser above the true target in 51.8% of the evaluated target-confuser pairs. For example, for the PF query, LND receives a higher similarity score in 33% of the evaluated cases (n=269). This exposes a limitation of target-only semantic matching: high text-region similarity does not necessarily provide sufficient evidence of target identity. Existing methods address ambiguity primarily through richer textual prompts, cross-modal alignment, or category-level representation learning, but do not explicitly couple local target-specific attribute support with compatibility against query-related confusers during candidate grounding.

To address this challenge, CPDG introduces target-confuser-conditioned discriminative grounding. Specifically, it learns latent discriminative attribute prototypes conditioned on both target and confuser representations (Eq. 10), which are optimized using visual candidate evidence through the target-confuser contrastive objective (Eq. 19). More importantly, its asymmetric local scoring (Eq. 13) jointly evaluates target-attribute support and confuser compatibility, providing discriminative evidence for candidate reranking (Eq. 14). Thus, CPDG moves beyond target-only similarity matching toward explicit local discrimination based on relative target-confuser evidence. Experimentally, CPDG reduces the Confuser Error Rate (CER) from 52.69% to 33.43% on EndoVis17 and from 51.80% to 30.15% on EndoVis18.

(2) Candidate validity verification.

Even after candidate reranking, selecting the highest-scoring proposal does not guarantee that it is a valid target. This limitation is particularly important when the queried category is absent or when the top-ranked proposal is incorrect. Existing ranking-based methods generally select the best-matched candidate, whereas category-presence-based rejection, as explored by RoSIS, primarily addresses whether a queried category is present rather than whether a specific grounding hypothesis is sufficiently supported.

To address this challenge, SCV introduces evidence-based multi-hypothesis verification. Rather than simply thresholding the original grounding confidence or accepting the highest-ranked proposal, SCV evaluates the Top-3 hypotheses retained by CPDG using fused global, local, semantic, attribute, and discriminative evidence. This separates candidate ranking from candidate validity assessment, enabling the recovery of a valid lower-ranked candidate or abstention when no hypothesis receives sufficient target-specific support. Compared with CPDG-only thresholding, SCV improves Reject Accuracy from 80.22% to 94.13% and AUROC from 55.91% to 78.43% on EndoVis17, with consistent gains on EndoVis18 (Reject Accuracy: 84.09%→96.96%; AUROC: 62.57%→80.43%).

(3) Methodological advances versus task-specific adaptations.

The methodological contributions lie in coupling target-confuser-conditioned local evidence discrimination with evidence-based multi-hypothesis verification, addressing candidate identity and candidate validity, respectively. In contrast, category-name-only prompting, text-similarity-based confuser construction, surgical instrument categories, proposal configurations, and the frozen SAM3 backend are task-specific implementations of these mechanisms. The results demonstrate that the two stages provide complementary improvements in candidate discrimination and selective rejection.

Reviewer_JDoPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses text-based video object segmentation of surgical instruments, where text prompts are often underspecified. The authors focus on a “weak-text” setting, where the prompt is only a category name from a fixed set (e.g. “Prograsp Forceps”). They propose a “discover and verify” (DaV) framework on top of a GroundingDINO → SAM3 pipeline.

At a high level, the first module, Confuser-aware Prompt Discriminative Grounding (CPDG), identifies the confusers of each prompt: the categories with the most similar text embeddings (e.g. another type of forceps). It then learns visual cues that separate the target from these confusers, and uses them to rerank the candidate boxes proposed by GroundingDINO.

The second module, Semantic Consistency Verification (SCV), checks whether the top-ranked candidates really belong to the queried category. It selects the most likely box, or rejects the prompt (no mask) when the evidence is too weak, for example when the instrument is absent. The selected box is then passed to SAM3 for segmentation and tracking.

The method is evaluated on two surgical instrument segmentation benchmarks, EndoVis17 and EndoVis18. It reports the best mcIoU among compared methods, as well as better target selection and absent-prompt rejection than the baselines.

Critical strengths
  1. The problem tackled is practical an impactful. For specialized data such as medical data, segmentation models can struggle with predicting similar classes, and also to abstain predicting something that is not on the picture. Category-name prompts are cheap for surgeons, but several instrument classes look alike. Fig. 1b quantifies the issue: GroundingDINO scores a confuser above the target in 51.8% of target–confuser pairs on EndoVis18. The paper reports metrics for each stage: Segmentation: IoU, mcIoU Grounding: TSA, CER Rejection: Reject Acc., AUROC

  2. Absent-query rejection. The paper frames grounding as accept-or-abstain and evaluates rejection explicitly. Pipelines that take the argmax box (Eq. 1) cannot abstain. Against the CPDG-only baseline, Reject Acc. improves from 84.1 to 97.0 on EndoVis18 and from 80.2 to 94.1 on EndoVis17, and Table 4 links these gains to L_abs. SAM3 can already abstain through its presence head, however, so the value of this part depends on the comparison requested in W1.

Critical weaknesses
  1. Unclear comparison protocol. I have doubts about whether the comparisons in Tables 1–2 are on equal footing. The baseline numbers seem to match those reported in the original papers, which may use different evaluation settings. For example, SurgicalSAM prompts only the classes present in each frame, while RoSIS prompts all classes. The “IoU” column may also mix Challenge IoU and ISI IoU. Since the paper does not state the protocol used for DaV or the baselines, it is hard to judge the state-of-the-art claim.

  2. Missing baselines; gains hard to attribute. There is no SAM3 text-prompt baseline, zero-shot or fine-tuned, although SAM3 can segment and track from text and has a presence head to reject absent concepts. There is also no fine-tuned GroundingDINO baseline; GroundingDINO appears to be frozen while DaV is trained on EndoVis. In Table 3, GroundingDINO + SAM3 alone reaches 76.64 mcIoU on EndoVis18, versus 65.48 for the best compared method, so much of the gain may come from SAM3. Absent-prompt rejection is also already addressed by RoSIS, but the rejection results are not compared against it.

  3. The weak-text framing is not supported. Prompts are 7 fixed class names, all seen in training, so the task is close to closed-set class-prompted segmentation, and there is no test on unseen classes. Confusers are chosen by similarity of the name embeddings, which reflects wording more than appearance. Yet the paper’s motivating example (PF vs LND, Fig. 1a) is a visual confusion.

Questions for authors

W1: Were the baselines re-run under your protocol, or taken from their papers? If taken from their papers, I am unsure the settings match. Could you clarify which IoU is reported, the EndoVis17 split, and whether metrics are computed per frame or after propagation? Ideally, please report all methods under a single protocol.

W2: How does SAM3 with its native text prompt perform, zero-shot and fine-tuned on the same data, using its presence score for rejection? Is GroundingDINO frozen, and how would a fine-tuned GroundingDINO compare?

W2: How does DaV’s rejection compare with RoSIS’s existence prediction under the same absent-query setting? How are absent-query samples constructed?

W3: Which text encoder is used, and what is each class’s confuser set compared with Fig. 1b? Do results change with learned per-class embeddings instead of text embeddings, or on a held-out class?

6 comments

We appreciate the reviewer’s recognition of the practical importance of reliable surgical instrument segmentation under weak-text prompting, particularly in distinguishing visually similar instruments and rejecting absent queries. We have carefully addressed the reviewer’s concerns and provided detailed responses below. Modifications corresponding to the reviewer’s comments are highlighted in blue.

W1, Q1: Evaluation protocols and comparability of baselines

We thank you for pointing out the ambiguity in our original evaluation protocol.

(1) Re-running baseline comparisons.

The original Tables 1–2 included methods developed under different prompting protocols, including rich-text or multi-prompt methods (e.g., TP-SIS, RoSIS, CLIPSeg, and CRIS), as well as non-text-based class-prototype methods (e.g., SurgicalSAM). These settings differ from our category-name-only evaluation protocol. To ensure fair comparisons, we re-evaluated all reproducible text-promptable methods under a unified category-name-only protocol, using identical dataset splits and the same ISI IoU evaluation implementation.

Two baselines require separate treatment. SurgicalSAM uses native class-conditioned rather than natural-language text prompting, so its original results are retained only as references under its native protocol. RoSIS lacks a publicly available implementation, preventing faithful reproduction under our unified protocol; therefore, its published results are retained as literature-reported references rather than directly comparable results.

Importantly, DaV achieves the best performance in terms of mcIoU across all prompt settings in revised Tables 1 and 2. Notably, under the unified category-name-only evaluation protocol, DaV attains the highest mcIoU among all reproduced baselines, reaching 67.27% and 80.46% on EndoVis17 and EndoVis18, respectively, outperforming the strongest reproduced baseline, MPG-SAM 2, by 24.37 and 19.39 percentage points, respectively.

(2) Evaluation protocol and data splits.

For EndoVis17, we re-aligned our evaluation with the original four-fold protocol of TP-SIS and report the newly obtained frame-wise results, marked with * in Table 2. We use identical data splits and the ISI IoU evaluation protocol. For EndoVis18, we follow the corresponding dataset split protocol adopted by TP-SIS and also report newly obtained frame-wise results in Table 1. All reproduced methods are evaluated using the same category-name-only prompts and ISI IoU evaluation protocol, ensuring consistency in both input and metric definitions.

(3) Inference settings: Frame-wise vs. video-level evaluation.

Regarding inference, we distinguish frame-wise segmentation from video-level evaluation after temporal propagation. To enable a direct comparison with frame-wise baselines, we additionally evaluate a frame-wise variant of DaV using the same GroundingDINO–CPDG–SCV pipeline and the SAM3 single-frame inference interface, without temporal propagation. Its mcIoU results are 67.27% and 80.46% on EndoVis17 and EndoVis18, respectively. The full video-level results are obtained by initializing SAM3 with the verified box and performing temporal propagation, yielding 68.34% and 82.39% mcIoU, respectively. These two evaluation settings are reported separately to distinguish the contribution of frame-wise candidate grounding from subsequent temporal propagation.

Table 1: Overall comparison with methods on EndoVis18. The highest result is bolded, while the second-highest result is underlined.

\hline \text{Prompt} & \text{Method} & \text{IoU} & \text{mcIoU} & \begin{array}{c} \text{Instrument Categories} \\\\ \begin{array}{rrrrrrr} \text{BF} & \text{PF} & \text{LND} & \text{SI} & \text{CA} & \text{MCS} & \text{UP} \end{array} \end{array} \\\\ \hline \begin{array}{c} \text{} \\\\ \text{} \\\\ \text{Weak Text} \\\\ \text{} \\\\ \text{} \end{array} & \begin{array}{l} \text{TP-SIS} \\\\ \text{CLIPSeg} \\\\ \text{CRIS} \\\\ \mathbf{\text{DaV (frame)}} \\\\ \mathbf{\text{DaV (video)}} \end{array} & \begin{array}{r} 67.24 \\\\ 50.40 \\\\ 64.77 \\\\ \mathbf{77.65} \\\\ \underline{77.40} \end{array} & \begin{array}{r} 44.67 \\\\ 37.00 \\\\ 49.51 \\\\ \underline{80.46} \\\\ \mathbf{82.39} \end{array} & \begin{array}{rrrrrrr} 71.39 & 42.24 & 35.73 & 50.46 & 23.01 & 73.70 & 16.18 \\\\ 66.74 & 44.80 & 36.22 & 31.25 & 2.21 & 76.56 & 1.25 \\\\ 73.20 & 41.48 & 43.73 & 35.59 & 41.13 & 81.44 & 30.00 \\\\ \underline{81.92} & \underline{72.28} & \underline{80.01} & \underline{78.82} & \underline{79.50} & \underline{90.48} & \underline{80.94} \\\\ \mathbf{82.17} & \mathbf{72.43} & \mathbf{83.87} & \mathbf{79.00} & \mathbf{79.53} & \mathbf{94.80} & \mathbf{84.90} \end{array} \\\\ \hline \end{array}$$ *Table 2 continues on the next page.*

Table 2: Overall comparison with methods on EndoVis17. * denotes frame-wise evaluation under the four-fold cross-validation protocol on EndoVis17.

\hline \text{Prompt} & \text{Method} & \text{IoU} & \text{mcIoU} & \begin{array}{c} \text{Instrument Categories} \\\\ \begin{array}{rrrrrrr} \text{BF} & \text{PF} & \text{LND} & \text{VS} & \text{GR} & \text{MCS} & \text{UP} \end{array} \end{array} \\\\ \hline \begin{array}{c} \text{} \\\\ \text{} \\\\ \text{Weak Text} \\\\ \text{} \\\\ \text{} \end{array} & \begin{array}{l} \text{TP-SIS(448)} \\\\ \text{CLIPSeg} \\\\ \text{CRIS} \\\\ \mathbf{\text{DaV* (frame)}} \\\\ \mathbf{\text{DaV (video)}} \end{array} & \begin{array}{r} 51.97 \\\\ 42.17 \\\\ 46.23 \\\\ \underline{67.22} \\\\ \mathbf{67.91} \end{array} & \begin{array}{r} 39.28 \\\\ 29.25 \\\\ 31.41 \\\\ \underline{67.27} \\\\ \mathbf{68.34} \end{array} & \begin{array}{rrrrrrr} 58.83 & 43.26 & \mathbf{70.03} & 51.63 & 0.00 & 51.23 & 0.00 \\\\ 46.49 & 33.50 & 42.61 & 28.50 & 5.83 & 29.00 & 18.83 \\\\ 51.85 & 42.30 & 69.29 & 43.71 & 0.00 & 12.70 & 0.00 \\\\ \underline{69.38} & \underline{55.09} & 67.51 & 66.62 & \mathbf{55.07} & \underline{94.83} & \underline{62.41} \\\\ \mathbf{70.09} & \mathbf{57.33} & \underline{69.42} & \mathbf{68.57} & \underline{54.52} & \mathbf{94.99} & \mathbf{63.44} \end{array} \\\\ \hline \end{array}$$   ## **W2, Q2: SAM3 and GroundingDINO Baselines** We thank you for this important suggestion. **(1) SAM3 baseline comparison.** We evaluated SAM3 with weak text prompts and its presence head, both zero-shot and after fine-tuning. On EndoVis17, zero-shot and fine-tuned SAM3 achieve mcIoU of 23.60% and 38.50%, AUROCs of 51.50% and 55.10%, respectively, with Reject Accuracy of 83.00% and 64.90%. On EndoVis18, fine-tuning changes mcIoU from 37.20% to 41.70%, AUROC from 56.60% to 96.90%, and Reject Accuracy from 92.60% to 84.90%. **These results show that native SAM3 prompting, even after fine-tuning, remains insufficient for fine-grained surgical instrument segmentation.** In comparison, DaV achieves mcIoU of 68.34%/82.39% on EndoVis17/EndoVis18, respectively, with rejection accuracies of 94.13%/96.96%. **(2) GroundingDINO comparison.** We clarify that GroundingDINO is fine-tuned on the EndoVis training data, with all designated trainable modules enabled. It generates the top-5 proposals, while SAM3 remains frozen. The GroundingDINO + SAM3 baseline, without CPDG or SCV, achieves Reject Accuracy/AUROC of 70.92%/63.88% on EndoVis17 and 85.26%/74.93% on EndoVis18. Adding CPDG and SCV improves these to 94.13%/78.43% and 96.96%/80.43%, respectively. **The comparison isolates the additional contribution of CPDG and SCV beyond the fine-tuned GroundingDINO + frozen SAM3 pipeline, demonstrating their effectiveness in absent-query rejection.** **(3) RoSIS comparison.** Since public code is unavailable and some reproduction details are insufficiently specified, we developed an independent weak-text adaptation based on RoSIS paper description. This approximate reproduction achieves rejection accuracy/AUROC of 75.06%/42.35% on EndoVis17 and 88.81%/65.96% on EndoVis18, compared with 94.13%/78.43% and 96.96%/80.43% for DaV, respectively. Despite the approximate reproduction setting, DaV achieves higher rejection accuracy on both datasets, with improvements of 19.07% and 8.15%, respectively. Overall, **DaV achieves the highest absent-query rejection accuracy among the evaluated methods on both datasets**, demonstrating the effectiveness of its confuser-aware candidate discrimination and verification mechanism. The corresponding experimental results and detailed analysis are provided in **Appendix A.2 (Table 5)** and **Section 4.4** . **Table 5: Comparison with SAM3, GroundingDINO, and RoSIS on EndoVis17 and EndoVis18. FT is Fine-tuned** $$\begin{array}{llrrrr} \hline \text{Dataset} & \text{Method} & \text{Reject Acc.}\uparrow & \text{AUROC}\uparrow & \text{IoU}\uparrow & \text{mcIoU}\uparrow \\\\ \hline \text{EndoVis17} & \text{SAM3 (Zero-shot)} & 83.00 & 51.50 & 38.90 & 23.60 \\\\ & \text{SAM3 (FT)} & 64.90 & 55.10 & 37.40 & 38.50 \\\\ & \text{GroundingDINO (FT) + SAM3} & 70.92 & \underline{63.88} & \underline{59.42} & \underline{60.31} \\\\ & \text{RoSIS (Weak-text retrained)} & 75.06 & 42.35 & 51.66 & 32.49 \\\\ & \text{\bf DaV (Ours)} & \mathbf{94.13} & \mathbf{78.43} & \mathbf{67.91} & \mathbf{68.34} \\\\ \hline \text{EndoVis18} & \text{SAM3 (Zero-shot)} & \underline{92.60} & 56.60 & 40.10 & 37.20 \\\\ & \text{SAM3 (FT)} & 84.90 & \mathbf{96.90} & 66.90 & 41.70 \\\\ & \text{GroundingDINO (FT) + SAM3} & 85.26 & 74.93 & \underline{70.49} & \underline{76.64} \\\\ & \text{RoSIS (Weak-text retrained)} & 88.81 & 65.96 & 49.16 & 37.41 \\\\ & \text{\bf DaV (Ours)} & \mathbf{96.96} & \underline{80.43} & \mathbf{77.40} & \mathbf{82.39} \\\\ \hline \end{array}$$

W3, Q4: Weak-text formulation, confuser construction, and text encoder

We thank the reviewer for this insightful comment.

(1) Weak-text formulation.

We clarify that weak-text refers to category-name-only prompting, rather than open-vocabulary generalization. Our aim is to reduce the interaction burden of providing detailed appearance, motion, or relational descriptions, which may require specialized surgical expertise. Our current evaluation uses seven predefined instrument categories, and we acknowledge that it does not establish generalization to unseen categories.

(2) Confuser construction and visual confusion problem.

① Wording and appearance similarity confusion: visually similar candidates outside the text-derived confuser set.

SCV (Eq. 20) can reject visually similar candidates outside the text-derived confuser set through proposal-level validity supervision. Specifically, Eq. (20) assigns each GroundingDINO proposal a binary label based on its IoU with the ground-truth box of the queried category. Thus, a visually similar non-target proposal is labeled as invalid, regardless of whether it belongs to the text-derived confuser set. SCV learns to reject such proposals using their visual and query-conditioned evidence (Eq. 17), provided they are retained in the top-3 candidates.

② Wording and appearance similarity confusion: visually similar candidates within the text-derived confuser set.

When a visually similar category is selected as a confuser based on name-embedding similarity, its text embedding serves as an explicit negative semantic reference. CPDG then uses local visual evidence and target-confuser contrastive supervision (Eqs. 13 and 19) to learn discriminative cues that support the target while suppressing responses to the selected confuser.

③ Motivation for text-based confuser construction.

Under category-name-only prompting, text embeddings provide readily available semantic relationships among instrument categories. We use these relationships to establish informative contrastive references for visual discrimination, rather than treating textual similarity as a direct measure of visual similarity. This design is supported by prior studies that leverage language-derived semantic relationships to guide visual representation learning [1,2].

④ Clarification of the confuser definition.

We apologize for the ambiguity in our original definition of confusers. We have revised the corresponding sentence in the Introduction from:

“The target and a visually similar non-target object, which we refer to as a confuser, may receive similar text-region semantic similarity scores.”

to:

“We define confusers as non-target categories with text embeddings semantically similar to the target, whose instruments may also exhibit strong local visual similarity that coarse text-region matching fails to distinguish.”

We further clarify that Fig. 1(b) evaluates grounding confusers across all eligible target–non-target instrument pairs, rather than only the Top-2 text-selected confusers in Eq. 9. For PF, the evaluated confusers include BF (44%, n=669), LND (33%, n=269), SI (34%, n=507), CA (38%, n=154), MCS (35%, n=172), and UP (30%, n=44). These results demonstrate that PF can be confused with multiple non-target instruments, including LND, as illustrated in Fig. 1. More importantly, the corresponding image regions also exhibit visual similarity. As shown in Fig. 1(a), the text-region similarity score between the PF prompt and the LND image region (0.26) is higher than that between the PF prompt and the true PF region (0.23). This motivates our central observation that semantic similarity alone is insufficient for fine-grained target discrimination. Accordingly, CPDG learns local discriminative visual evidence through target-confuser contrastive supervision (Eqs. 10–14 and 19). Thus, text specifies what to contrast, while visual evidence determines how to distinguish.

(3) Text embeddings versus learned class embeddings.

We use the BERT text encoder within GroundingDINO to obtain category-name embeddings. We replaced the original text representations with independently learned per-class embeddings and compared the two formulations. The original DaV achieved 82.39% mcIoU on EndoVis18, compared with 81.08% for the learned-embedding variant. The original formulation also achieved higher TSA (86.81% vs. 84.15%) and lower CER (30.15% vs. 37.99%). These results indicate that the original text-based formulation provides better target-confuser discrimination and segmentation performance than independently learned class embeddings under the evaluated setting. The corresponding experimental results and detailed analysis are provided in Appendix A.4 (Table 7).

Continued on the next page.

Reviewer_hAJoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies fine-grained video object segmentation using weak text prompts. The goal is to solve the confusion between visually similar surgical instruments and queries for absent categories. To address this issue, the authors proposed Discover-and-Verify (DaV) framework, which augments GroundingDINO proposals with two novel modules: Confuser-aware Prompt Discriminative Grounding module, which extracts local features and reranks candidates, and Semantic Consistency Verification strategy to perform candidate selection. After which, the accepted box is passed to a frozen SAM3 for segmentation and temporal propagation. Experimental results on EndoVis17 and EndoVis18 show the effectiveness of the DaV framework.

Critical strengths
  1. The paper addresses a practically relevant problem in surgical video understanding, i.e., only using category names may be insufficient for distinguishing visually similar instruments.

  2. The reported improvements in target selection accuracy, confuser error rate, and verification AUROC provide evidence that the proposed components improve discrimination beyond the initial grounding scores.

  3. Two key components in the proposed DaV framework, CPDG and SCV are proven effective in the experiments.

Critical weaknesses
  1. The evaluation consistency is not clear enough. Several baseline results match their original papers’ richer-prompt settings, despite Tables 1–2 being presented as weak-text comparisons. It is unclear whether the weak-text definition is applied to every comparison method.

  2. The EndoVis17 split used in this work needs clarification relative to the four-fold evaluation used by TP-SIS.

  3. The ablations do not isolate CPDG’s numerous components (e.g., CPDG includes sparse localization, prototype-based aggregation, dual-path fusion, confuser selection, latent attribute induction, and evidence-based scoring) or compare against a simple ROI classifier under matched supervision and backbone settings. Consequently, the necessity of the proposed design and its interpretation as discovering distinct local attributes remain insufficiently supported.

  4. Equations 4 and 5 use learned attention scores only to select hard Top-k indices. As written, this operation does not provide gradients to the attention-scoring parameters through the selection. The paper should explain how this branch is trained with a bit more details.

  5. The methodology section contains too much details that are not the main contributions of this paper. Consider only keeping the main components.

Questions for authors
  1. Were all baselines evaluated using category-name-only prompts and identical data splits? If published results were reused, how were differences in prompting and evaluation protocols accounted for?
  2. How are the QGSL attention-scoring parameters trained through the hard Top-k selection?
4 comments

We appreciate the reviewer’s recognition of the practical relevance of addressing visual confusion among surgical instruments under weak-text prompting, and the effectiveness of our CPDG and SCV modules in improving candidate discrimination and verification. We have carefully addressed the reviewer’s concerns and provided detailed responses below. Modifications corresponding to the reviewer’s comments are highlighted in blue.

W1, Q1: Prompt consistency and evaluation protocols

We thank you for highlighting the importance of fair comparisons. The original Tables 1–2 included methods developed under different prompting protocols, including rich-text or multi-prompt methods (e.g., TP-SIS, RoSIS, CLIPSeg, and CRIS), as well as non-text-based class-prototype methods (e.g., SurgicalSAM). To reduce the influence of prompt richness on the comparison, we re-evaluated reproducible text-promptable methods using category-name-only (weak-text) prompts under a unified evaluation protocol. All directly compared methods use identical dataset splits and evaluation protocols.

Two baselines require separate treatment. SurgicalSAM uses native class-conditioned rather than text prompting, so its results are retained only as references under its original setting. RoSIS lacks a publicly available implementation, precluding reproduction under our unified protocol; thus, we report its published results only as literature references, not directly comparable results.

Revised Tables 1 and 2 clearly distinguish among the categories of class prototype, rich-text prompt, and weak-text prompt. Importantly, DaV achieves the best performance in terms of mcIoU across all prompt settings. Notably, under the unified category-name-only evaluation protocol, DaV attains the highest mcIoU among all reproduced baselines, reaching 67.27% and 80.46% on EndoVis17 and EndoVis18, respectively, outperforming the strongest reproduced baseline, MPG-SAM 2, by 24.37% and 19.39%, respectively.

Table 1: Overall comparison with methods on EndoVis18. The highest result is bolded, while the second-highest result is underlined.

\hline \text{Prompt} & \text{Method} & \text{IoU} & \text{mcIoU} & \begin{array}{c} \text{Instrument Categories} \\\\ \begin{array}{rrrrrrr} \text{BF} & \text{PF} & \text{LND} & \text{SI} & \text{CA} & \text{MCS} & \text{UP} \end{array} \end{array} \\\\ \hline \begin{array}{c} \text{} \\\\ \text{} \\\\ \text{Weak Text} \\\\ \text{} \\\\ \text{} \end{array} & \begin{array}{l} \text{TP-SIS} \\\\ \text{CLIPSeg} \\\\ \text{CRIS} \\\\ \mathbf{\text{DaV (frame)}} \\\\ \mathbf{\text{DaV (video)}} \end{array} & \begin{array}{r} 67.24 \\\\ 50.40 \\\\ 64.77 \\\\ \mathbf{77.65} \\\\ \underline{77.40} \end{array} & \begin{array}{r} 44.67 \\\\ 37.00 \\\\ 49.51 \\\\ \underline{80.46} \\\\ \mathbf{82.39} \end{array} & \begin{array}{rrrrrrr} 71.39 & 42.24 & 35.73 & 50.46 & 23.01 & 73.70 & 16.18 \\\\ 66.74 & 44.80 & 36.22 & 31.25 & 2.21 & 76.56 & 1.25 \\\\ 73.20 & 41.48 & 43.73 & 35.59 & 41.13 & 81.44 & 30.00 \\\\ \underline{81.92} & \underline{72.28} & \underline{80.01} & \underline{78.82} & \underline{79.50} & \underline{90.48} & \underline{80.94} \\\\ \mathbf{82.17} & \mathbf{72.43} & \mathbf{83.87} & \mathbf{79.00} & \mathbf{79.53} & \mathbf{94.80} & \mathbf{84.90} \end{array} \\\\ \hline \end{array}$$ **Table 2: Overall comparison with methods on EndoVis17. The highest result is bolded, while the second highest result is underlined. * denotes frame-wise evaluation under the four-fold cross-validation protocol on EndoVis17.** $$\begin{array}{llrrc} \hline \text{Prompt} & \text{Method} & \text{IoU} & \text{mcIoU} & \begin{array}{c} \text{Instrument Categories} \\\\ \begin{array}{rrrrrrr} \text{BF} & \text{PF} & \text{LND} & \text{VS} & \text{GR} & \text{MCS} & \text{UP} \end{array} \end{array} \\\\ \hline \begin{array}{c} \text{} \\\\ \text{} \\\\ \text{Weak Text} \\\\ \text{} \\\\ \text{} \end{array} & \begin{array}{l} \text{TP-SIS(448)} \\\\ \text{CLIPSeg} \\\\ \text{CRIS} \\\\ \mathbf{\text{DaV* (frame)}} \\\\ \mathbf{\text{DaV (ours)}} \end{array} & \begin{array}{r} 51.97 \\\\ 42.17 \\\\ 46.23 \\\\ \underline{67.22} \\\\ \mathbf{67.91} \end{array} & \begin{array}{r} 39.28 \\\\ 29.25 \\\\ 31.41 \\\\ \underline{67.27} \\\\ \mathbf{68.34} \end{array} & \begin{array}{rrrrrrr} 58.83 & 43.26 & \mathbf{70.03} & 51.63 & 0.00 & 51.23 & 0.00 \\\\ 46.49 & 33.50 & 42.61 & 28.50 & 5.83 & 29.00 & 18.83 \\\\ 51.85 & 42.30 & 69.29 & 43.71 & 0.00 & 12.70 & 0.00 \\\\ \underline{69.38} & \underline{55.09} & 67.51 & 66.62 & \mathbf{55.07} & \underline{94.83} & \underline{62.41} \\\\ \mathbf{70.09} & \mathbf{57.33} & \underline{69.42} & \mathbf{68.57} & \underline{54.52} & \mathbf{94.99} & \mathbf{63.44} \end{array} \\\\ \hline \end{array}$$

W2: EndoVis17 Evaluation Protocol

Thank you for raising this point. To ensure a consistent comparison with prior work, we additionally conducted four-fold cross-validation on EndoVis17, following the protocol established by TP-SIS. Specifically, the eight annotated training sequences are divided into four folds, with six sequences used for training and two for validation in each fold. Under this protocol, DaV achieves 67.27% mcIoU, which remains the best performance among the compared methods under the same evaluation protocol. We have added the four-fold evaluation results and detailed sequence-level split settings to the revised manuscript in Table 2.

W3: Ablation of CPDG and Comparison with a Simple ROI Classifier

We thank you for this insightful comment. CPDG addresses a limitation of conventional text-based grounding: global semantic matching alone may be insufficient to distinguish the queried target from visually similar confusers. It follows a unified target–confuser-conditioned evidence modeling process, extracting complementary local visual evidence, modeling category-discriminative cues, and evaluating the resulting evidence for candidate ranking. The following ablations are illustrated in Table 4.

(1) Dual-path local evidence mining. Removing QIPA (sparse localization) or QGSL (prototype-based aggregation) decreases TSA from 86.81% to 83.06% or 84.92%, respectively, while increasing CER from 30.15% to 38.87% or 35.01%. QGSL focuses on spatially localized discriminative regions, preventing shared structures from dominating candidate representations. QIPA complements this by aggregating query-relevant evidence across the ROI, providing more stable and distributed local representations. Removing either branch therefore weakens the complementary localization and aggregation of discriminative cues. Replacing the learnable dual-path fusion with simpler fusion also reduces performance (TSA: 85.13%, CER: 34.62%, mcIoU: 80.82%). These results demonstrate the complementary roles of sparse localization and prototype-based aggregation in capturing discriminative local evidence.

(2) Confuser selection and latent attribute induction. Removing confuser selection increases CER substantially from 30.15% to 49.61%, despite only a marginal decrease in TSA (86.81% to 86.58%). This is because TSA measures overall target selection, whereas CER specifically evaluates target-confuser ranking errors. Without explicit confuser modeling, the model can still recognize general target semantics but lacks counter-evidence to distinguish visually similar categories. Removing latent attribute induction also degrades performance (TSA: 84.99%, CER: 39.04%, mcIoU: 81.00%), as the model loses the target-confuser-conditioned prototypes that encode category-specific visual differences. These results highlight the importance of explicitly modeling confusers and learning category-discriminative attributes rather than relying solely on target semantics.

(3) Evidence-based scoring and comparison with ROI classification. Removing evidence-based scoring reduces TSA to 80.28% and mcIoU to 79.10%. Localized features and attribute activations alone do not establish which candidate is better supported: Eq. (13) contrasts target-attribute support against confuser compatibility, while Eq. (14) aggregates the resulting local evidence into candidate-level scores. The simple ROI classifier achieves only 81.40% TSA, 51.22% CER, and 76.16% mcIoU, compared with 86.81%, 30.15%, and 82.39% for the full method. Unlike CPDG, which explicitly identifies and contrasts local target-supporting and confuser-related evidence, the ROI classifier primarily relies on holistic candidate representations. These results demonstrate the importance of evidence-based candidate scoring beyond conventional global ROI classification.

Overall, these results demonstrate the complementary contributions of local evidence mining, target-confuser modeling, and evidence-based candidate ranking within CPDG. We have added the ablations and controlled ROI-classifier comparison, together with the corresponding analysis, in Appendix A.1 (Table 4) and Section 4.3.

**Table 4: Ablation of CPDG components and comparison with a simple ROI classifier on EndoVis18. **

\hline \text{Variant} & \text{TSA}\uparrow & \text{CER}\downarrow & \text{mcIoU}\uparrow \\\\ \hline \text{w/o QIPA} & 83.06 & 38.87 & 79.19 \\\\ \text{w/o QGSL} & 84.92 & 35.01 & 80.47 \\\\ \text{w/o Learnable Fusion} & 85.13 & \underline{34.62} & 80.82 \\\\ \text{w/o Confuser Selection} & \underline{86.58} & 49.61 & \underline{82.03} \\\\ \text{w/o Latent Attribute Induction} & 84.99 & 39.04 & 81.00 \\\\ \text{w/o Evidence-based Scoring} & 80.28 & 42.67 & 79.10 \\\\ \text{Simple ROI Classifier} & 81.40 & 51.22 & 76.16 \\\\ \text{\bf DaV (Ours)} & \mathbf{86.81} & \mathbf{30.15} & \mathbf{82.39} \\\\ \hline \end{array}$$

W4, Q2: Comment on QGSL Gradient Propagation

Thank you for pointing out this ambiguity. Although hard Top-k selection is non-differentiable, it only determines the selected indices, which remain fixed during each backward pass. The corresponding continuous attention weights are retained to modulate the selected visual features and are further used in the QGSL–QIPA fusion.Therefore, gradients propagate through these continuous operations to the QGSL attention parameters, without differentiating through the discrete Top-k indices. We apologize that the original Eq. (5) did not explicitly show the retention and use of the selected continuous attention weights. We have revised Eq. (5) and clarified the corresponding gradient-flow mechanism.

Specifically, for a fixed selected set of locations:

Si=((u1,v1),…,(uNp,vNp))S_i=((u_1,v_1),\ldots,(u_{N_p},v_{N_p}))

Each local representation is computed as:

Pisp=αisp⊙TopK⁡Np(Fi,Ai)P_i^{sp}=\alpha_i^{sp}\odot\operatorname{TopK}_{N_p}(F_i,A_i)

The corresponding continuous attention weights are defined as:

αisp=Ai(um,vm)m=1Np\alpha_i^{sp}=\\{A_i(u_m,v_m)\\}_{m=1}^{N_p}

Since the selected attention weights are retrieved from the differentiable spatial attention map generated by Eq. (4), downstream gradients propagate through the attention-weighted representations and spatial softmax to the QGSL attention-scoring parameters, while the discrete Top-k indices remain fixed during each backward pass. The selected locations are recomputed in subsequent forward passes as the attention parameters are updated.

We further verified this gradient pathway through autograd analysis on three training batches. The verification loss, L_ver, yielded finite, non-zero gradient norms of 2.863, 3.388, and 3.053 for the QGSL attention projection weights. The total training loss also produced non-zero gradients of 2.812, 3.326, and 3.109, respectively. These parameters are included in the AdamW optimizer.

Therefore, QGSL is optimized through differentiable attention-weighted feature aggregation and downstream task supervision, rather than through gradients of the discrete Top-k selection operation.

 

W5: Removing Unnecessary Methodological Details

Thank you for this helpful suggestion. We have streamlined Section 3 by condensing routine feature encoding and implementation details, as well as removing repetitive explanations. We retain the essential formulations of CPDG's target-confuser discrimination and SCV's evidence-based candidate verification to clarify the core methodological contributions.

Dear Reviewer,

Thank you for your valuable feedback and constructive suggestions. We have carefully addressed the concerns you raised through additional experiments and detailed clarifications in our rebuttal. We sincerely hope that our responses have adequately addressed your concerns, and we would greatly appreciate the opportunity to further discuss any remaining questions with you.

Thank you again for your time and consideration. We look forward to hearing your thoughts.

Comments on the paper

1 comment

Thank you very much for your thoughtful reconsideration and for taking the time to evaluate our responses alongside the other reviewers' comments. We sincerely appreciate your willingness to improve the rating. We are glad that our clarifications have helped address your concerns, and we greatly appreciate your constructive feedback and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.