- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces LD-RSVIS for referring surgical video instrument segmentation, addressing the limited scale and single-target scope of existing benchmarks. The dataset contains 3,536 clips, approximately 1.09 million frames, and 58,666 expressions covering 30 instrument categories and 25 procedures, including single-target, multi-target, and no-target settings. The paper also proposes Cascade-RSVIS, which combines target-context mining, context-guided segmentation, and multi-cue textual features. Against 12 baselines, it reports the highest overall J&F of 37.4%, compared with 33.1% for FlowRVS.
Critical strengths
-
Substantial scale and broader task coverage. LD-RSVIS substantially expands the scale of the benchmarks for the task. Its coverage of broader procedures and instruments make it a potentially valuable resource for training and evaluation.
-
Evaluation of multiple representative methods provides a useful reference.
Critical weaknesses
-
LD-RSVIS is built from exisiting source datasets. They may carry richer annotations than RSVIS requires. Yet, the resulted benchmark may support RSVIS only. The manuscript also does not state which source annotations are preserved, converted, or discarded.
-
The paper does not clarify which concrete clinical interaction RSVIS supports or how often referring expressions occur in real surgical workflows. The task significance seems to be moderate and would benefit from a more specific clinical use case.
Questions for authors
-
Baseline comparison details. Multi-target and no-target settings can be new for those existing methods. How these existing methods adapt to the benchmark should be provided.
-
The paper reports expression counts and examples but does not quantify whole-instrument versus part-level expressions, leaving unclear whether aggregate performance reflects part grounding difficulty. Is it possible to characterize and evaluate such expression granularity?
-
No-target expressions are treated as a single category, without distinguishing absent-category cases from incorrect-attribute/relation/action cases, which test different grounding capabilities. I was wondering how such cases are distributed in the benchmark and how they are addressed by the baseline/proposed methods.
6 comments
Rebuttal (Part 2/3)
Q4: The paper reports expression counts and examples but does not quantify whole-instrument versus part-level expressions, leaving unclear whether aggregate performance reflects part grounding difficulty. Is it possible to characterize and evaluate such expression granularity?
A4: Thank you for suggesting this analysis. Across the original train and test sets, STE contains 17,492 whole-instrument and 17,252 part-level expressions; the counts also appear in the appendix. Table A evaluates the two granularities separately using the same models and evaluator.
Table A: Comparison with existing methods by single-target expression granularity on the original test set.
| Method | Whole | Whole | Whole J\\&F | Part | Part | Part J\\&F |
|---|---|---|---|---|---|---|
| VIS-Net | 27.8 | 25.4 | 26.6 | 21.0 | 18.8 | 19.9 |
| ReferFormer | 23.4 | 28.2 | 25.8 | 19.0 | 24.0 | 21.5 |
| OnlineRefer | 24.7 | 28.9 | 26.8 | 18.2 | 23.0 | 20.6 |
| LoSh | 30.5 | 35.3 | 32.9 | 21.4 | 26.6 | 24.0 |
| LMPM | 21.7 | 24.3 | 23.0 | 20.2 | 23.2 | 21.7 |
| ReSurgSAM2 | 32.4 | 35.6 | 34.0 | 20.4 | 23.6 | 22.0 |
| SurgRef | 35.1 | 33.3 | 34.2 | 20.5 | 18.1 | 19.3 |
| DsHmp | 30.9 | 38.5 | 34.7 | 14.8 | 22.8 | 18.8 |
| SAMWISE | 34.4 | 38.0 | 36.2 | 33.2 | 37.4 | 35.3 |
| ReferDINO | 32.6 | 37.2 | 34.9 | 36.8 | 42.4 | 39.6 |
| ReferMo | 39.6 | 41.6 | 40.6 | 33.8 | 37.6 | 35.7 |
| FlowRVS | 37.3 | 42.3 | 39.8 | 34.6 | 40.8 | 37.7 |
| Cascade-RSVIS (ours) | 40.3 | 43.3 | 41.8 | 35.6 | 39.0 | 37.3 |
Cascade-RSVIS obtains 41.8 on Whole and 37.3 on Part, a 4.5-point difference. Compared with FlowRVS, it improves Whole by 2.0 points but is 0.4 points lower on Part. ReferDINO obtains the highest Part score, 39.6. Therefore, the combined STE score can obscure granularity-specific difficulty and method rankings; the aggregate lead does not imply the best performance at every granularity.
In the revision, we will report the whole-instrument and part-level expression counts and their separate evaluation results, including the differences hidden by the aggregate STE score.
Rebuttal (Part 3/3)
Q5: No-target expressions are treated as a single category, without distinguishing absent-category cases from incorrect-attribute/relation/action cases, which test different grounding capabilities. I was wondering how such cases are distributed in the benchmark and how they are addressed by the baseline/proposed methods.
A5: Thank you for this suggestion. We report the semantic subtype distribution and explain how both the baselines and Cascade-RSVIS handle no-target queries below.
NTE construction and subtype distribution. NTE expressions are constructed for the particular video using its instruments and scene context, rather than randomly imported from other videos. Candidate descriptions introduce a category, attribute, location, action/motion, or relation/interaction mismatch. Reviewers check the complete video to verify that no visible target satisfies the full description at any point. NTE expressions use the same affirmative referring style as positive queries, without words that reveal the absence label. All evaluated frames have all-zero ground-truth masks. Table B reports the distribution of these semantic subtypes across the full dataset.
Table B: Distribution of no-target expression subtypes in LD-RSVIS.
| NTE semantic subtype | Expressions | Percentage |
|---|---|---|
| Category mismatch | 2,387 | 24.41% |
| Attribute mismatch | 1,921 | 19.65% |
| Location mismatch | 1,684 | 17.22% |
| Action/motion mismatch | 2,413 | 24.68% |
| Relation/interaction mismatch | 1,373 | 14.04% |
| Total | 9,778 | 100.00% |
A target that appears in the sequence but later leaves the view or becomes fully occluded remains an STE/MTE target, with empty masks for its invisible frames. NTE instead requires no visible match anywhere in the complete sequence.
Handling by baselines and Cascade-RSVIS. As described in our response to Q3, the baselines handle NTE using their respective prediction mechanisms; these mechanisms apply across the NTE semantic subtypes.
In Cascade-RSVIS, TCM assesses text-related candidates and rejects clips without a matching target; other clips proceed to CGS.
Whole-video decision. The threshold for each method's empty-output decision follows its prediction mechanism and the corresponding adaptation to our benchmark. After assembling clip predictions into a complete video, an NTE sample is correct only if every evaluated frame contains no foreground; any foreground in any frame makes it incorrect. An always-empty predictor therefore receives a perfect NTE score by definition, but this does not reflect its ability to segment present targets. We report results for each setting and do not assess a method based on its NTE score alone. For NTE sample ,
\qquad J_i=F_i=A_i,$$ where $\widehat M_{i,t}$ is the final binary mask and $\mathcal T_i$ includes all evaluated frames. Clip predictions are assembled before this decision. In the revision, we will add the NTE semantic subtype distribution and explain how the baselines and Cascade-RSVIS handle no-target queries, including the whole-video empty-output decision.
Rebuttal (Part 1/3)
A1: Thank you for asking about the source annotations. Where available, we retain applicable instrument segmentation annotations for selected clips and subject them to multiple rounds of manual inspection and refinement. Non-segmentation labels, such as detection or phase/action recognition, are not converted into segmentation GT. For sources providing only such labels, instrument masks are manually annotated, inspected, and refined. Both retained and newly annotated masks therefore receive manual quality control.
LD-RSVIS is intended as a unified language-conditioned video instrument segmentation benchmark, rather than a replacement for every task supported by its sources. Detection, phase recognition, and action recognition labels are not part of the current RSVIS supervision/evaluation protocol; this does not remove their availability in the original datasets. Our addition is the referring expressions and their links to temporal target masklets, including whole/part granularity and single-, multi-, and no-target settings.
In the revision, we will distinguish retained and refined segmentation masks, newly annotated masks and expressions, and source labels excluded from the RSVIS task.
A2: Thank you for requesting a concrete use case. One intended interaction is language-guided target highlighting during postoperative video review and teaching. For example, an instructor reviewing a cholecystectomy video could query ``the silver hook in the upper right dissecting and cauterizing connective tissue along the gallbladder.'' The system would segment and highlight that instrument through the video, helping trainees follow the demonstrated manipulation. Position, action, and tissue interaction allow the query to identify the object of interest rather than every instrument of a predefined class.
STE supports selecting one instrument or part; MTE supports highlighting several related instruments; NTE tests whether an unmatched query produces an empty result instead of highlighting an unrelated instrument. Together, they evaluate target selection, segmentation, and handling of mismatched requests in such an interaction.
The intended expressions are user-initiated queries for review and teaching, rather than transcriptions of naturally occurring intraoperative dialogue. The dataset's expression counts and setting proportions provide evaluation coverage; they are not estimates of natural expression frequency in clinical workflows.
In the revision, we will add the postoperative review and teaching example and clarify that benchmark expression counts are not estimates of query frequency in clinical workflows.
A3: Thank you for asking how existing methods are adapted. Each method receives a video and one expression and returns an expression-level binary mask sequence. We adapt supervision and output interpretation to the three settings while retaining each method's prediction mechanism:
In the revision, we will describe multi-target union supervision and selection, as well as empty-mask supervision and the methods' respective empty-output mechanisms.