acceptodds
Under review as a conference paper at ICLR 2027

LD-RSVIS: A Large-Scale and Diverse Benchmark for Referring Surgical Video Instrument Segmentation

Abstract

Referring surgical video instrument segmentation (RSVIS) aims at segmenting the instrument in a surgical video, given a textual description. Despite recent progress, current models are trained and assessed on relatively small-scale benchmarks, hindering the development of more general RSVIS. In addition, existing benchmarks support only the single-target expression that refers to one instrument in the video, while overlooking multi-target and no-target referring expressions, restricting the applicability of RSVIS in practical scenarios. Addressing these issues, we propose LD-RSVIS, a new benchmark aiming to facilitate more robust and general RSVIS. Specifically, LD-RSVIS consists of 3,536 surgical videos with 1.09 million frames and covers a broad set of 30 instrument classes from 25 various procedures. By including abundant videos and classes, LD-RSVIS could benefit large-scale training and evaluation of more general RSVIS methods. Besides, unlike existing datasets, LD-RSVIS offers diverse referring settings, including no-target, single-target, and multi-target expressions, which enables the development of more practical RSVIS models in real applications. In order to ensure high-quality annotations, all videos in LD-RSVIS are manually labeled with multiple rounds of inspection and refinement. To our knowledge, LD-RSVIS is the largest and most diverse benchmark for RSVIS. To analyze LD-RSVIS and to provide comparison for future research, we evaluate 12 representative methods, and the results reveal that more efforts are required for improvements. To encourage future research, we present a simple yet effective RSVIS method, dubbed Cascade-RSVIS, that first mines target-specific cues using the complementary multi-cue text information and then employs such cues and textual information for segmentation, achieving promising performance. Through LD-RSVIS and Cascade-RSVIS, we hope to inspire more future research toward robust and general RSVIS. Our benchmark and code will be released.

Primary area:
datasets and benchmarks
Keywords:
Referring Video Instrument Segmentation, Surgical Video Understanding

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_32sdOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces LD-RSVIS for referring surgical video instrument segmentation, addressing the limited scale and single-target scope of existing benchmarks. The dataset contains 3,536 clips, approximately 1.09 million frames, and 58,666 expressions covering 30 instrument categories and 25 procedures, including single-target, multi-target, and no-target settings. The paper also proposes Cascade-RSVIS, which combines target-context mining, context-guided segmentation, and multi-cue textual features. Against 12 baselines, it reports the highest overall J&F of 37.4%, compared with 33.1% for FlowRVS.

Critical strengths
  • Substantial scale and broader task coverage. LD-RSVIS substantially expands the scale of the benchmarks for the task. Its coverage of broader procedures and instruments make it a potentially valuable resource for training and evaluation.

  • Evaluation of multiple representative methods provides a useful reference.

Critical weaknesses
  • LD-RSVIS is built from exisiting source datasets. They may carry richer annotations than RSVIS requires. Yet, the resulted benchmark may support RSVIS only. The manuscript also does not state which source annotations are preserved, converted, or discarded.

  • The paper does not clarify which concrete clinical interaction RSVIS supports or how often referring expressions occur in real surgical workflows. The task significance seems to be moderate and would benefit from a more specific clinical use case.

Questions for authors
  • Baseline comparison details. Multi-target and no-target settings can be new for those existing methods. How these existing methods adapt to the benchmark should be provided.

  • The paper reports expression counts and examples but does not quantify whole-instrument versus part-level expressions, leaving unclear whether aggregate performance reflects part grounding difficulty. Is it possible to characterize and evaluate such expression granularity?

  • No-target expressions are treated as a single category, without distinguishing absent-category cases from incorrect-attribute/relation/action cases, which test different grounding capabilities. I was wondering how such cases are distributed in the benchmark and how they are addressed by the baseline/proposed methods.

6 comments

Rebuttal (Part 1/3)

Q1: LD-RSVIS is built from exisiting source datasets. They may carry richer annotations than RSVIS requires. Yet, the resulted benchmark may support RSVIS only. The manuscript also does not state which source annotations are preserved, converted, or discarded.

A1: Thank you for asking about the source annotations. Where available, we retain applicable instrument segmentation annotations for selected clips and subject them to multiple rounds of manual inspection and refinement. Non-segmentation labels, such as detection or phase/action recognition, are not converted into segmentation GT. For sources providing only such labels, instrument masks are manually annotated, inspected, and refined. Both retained and newly annotated masks therefore receive manual quality control.

LD-RSVIS is intended as a unified language-conditioned video instrument segmentation benchmark, rather than a replacement for every task supported by its sources. Detection, phase recognition, and action recognition labels are not part of the current RSVIS supervision/evaluation protocol; this does not remove their availability in the original datasets. Our addition is the referring expressions and their links to temporal target masklets, including whole/part granularity and single-, multi-, and no-target settings.

In the revision, we will distinguish retained and refined segmentation masks, newly annotated masks and expressions, and source labels excluded from the RSVIS task.

Q2: The paper does not clarify which concrete clinical interaction RSVIS supports or how often referring expressions occur in real surgical workflows. The task significance seems to be moderate and would benefit from a more specific clinical use case.

A2: Thank you for requesting a concrete use case. One intended interaction is language-guided target highlighting during postoperative video review and teaching. For example, an instructor reviewing a cholecystectomy video could query ``the silver hook in the upper right dissecting and cauterizing connective tissue along the gallbladder.'' The system would segment and highlight that instrument through the video, helping trainees follow the demonstrated manipulation. Position, action, and tissue interaction allow the query to identify the object of interest rather than every instrument of a predefined class.

STE supports selecting one instrument or part; MTE supports highlighting several related instruments; NTE tests whether an unmatched query produces an empty result instead of highlighting an unrelated instrument. Together, they evaluate target selection, segmentation, and handling of mismatched requests in such an interaction.

The intended expressions are user-initiated queries for review and teaching, rather than transcriptions of naturally occurring intraoperative dialogue. The dataset's expression counts and setting proportions provide evaluation coverage; they are not estimates of natural expression frequency in clinical workflows.

In the revision, we will add the postoperative review and teaching example and clarify that benchmark expression counts are not estimates of query frequency in clinical workflows.

Q3: Baseline comparison details. Multi-target and no-target settings can be new for those existing methods. How these existing methods adapt to the benchmark should be provided.

A3: Thank you for asking how existing methods are adapted. Each method receives a video and one expression and returns an expression-level binary mask sequence. We adapt supervision and output interpretation to the three settings while retaining each method's prediction mechanism:

  • STE: each expression refers to a single target, and the model is supervised using the corresponding ground-truth mask.
  • MTE: the per-frame GT is the union of all visible referred targets. Direct expression-mask methods are supervised with that union. Methods retaining instance/candidate predictions include multiple matching targets in training matching or target selection, then union the selected masks at inference. They are not restricted to one selected instance.
  • NTE: all evaluated frames have all-zero GT masks and participate in training. Candidate-based methods can reject all candidates; presence-based methods use their presence outputs; direct-mask methods can return all-background masks under their original pixel binarization.

In the revision, we will describe multi-target union supervision and selection, as well as empty-mask supervision and the methods' respective empty-output mechanisms.

Rebuttal (Part 2/3)

Q4: The paper reports expression counts and examples but does not quantify whole-instrument versus part-level expressions, leaving unclear whether aggregate performance reflects part grounding difficulty. Is it possible to characterize and evaluate such expression granularity?

A4: Thank you for suggesting this analysis. Across the original train and test sets, STE contains 17,492 whole-instrument and 17,252 part-level expressions; the counts also appear in the appendix. Table A evaluates the two granularities separately using the same models and evaluator.

Table A: Comparison with existing methods by single-target expression granularity on the original test set.

MethodWhole JJWhole FFWhole J\\&FPart JJPart FFPart J\\&F
VIS-Net27.825.426.621.018.819.9
ReferFormer23.428.225.819.024.021.5
OnlineRefer24.728.926.818.223.020.6
LoSh30.535.332.921.426.624.0
LMPM21.724.323.020.223.221.7
ReSurgSAM232.435.634.020.423.622.0
SurgRef35.133.334.220.518.119.3
DsHmp30.938.534.714.822.818.8
SAMWISE34.438.036.233.237.435.3
ReferDINO32.637.234.936.842.439.6
ReferMo39.641.640.633.837.635.7
FlowRVS37.342.339.834.640.837.7
Cascade-RSVIS (ours)40.343.341.835.639.037.3

Cascade-RSVIS obtains 41.8 on Whole and 37.3 on Part, a 4.5-point difference. Compared with FlowRVS, it improves Whole by 2.0 points but is 0.4 points lower on Part. ReferDINO obtains the highest Part score, 39.6. Therefore, the combined STE score can obscure granularity-specific difficulty and method rankings; the aggregate lead does not imply the best performance at every granularity.

In the revision, we will report the whole-instrument and part-level expression counts and their separate evaluation results, including the differences hidden by the aggregate STE score.

Rebuttal (Part 3/3)

Q5: No-target expressions are treated as a single category, without distinguishing absent-category cases from incorrect-attribute/relation/action cases, which test different grounding capabilities. I was wondering how such cases are distributed in the benchmark and how they are addressed by the baseline/proposed methods.

A5: Thank you for this suggestion. We report the semantic subtype distribution and explain how both the baselines and Cascade-RSVIS handle no-target queries below.

NTE construction and subtype distribution. NTE expressions are constructed for the particular video using its instruments and scene context, rather than randomly imported from other videos. Candidate descriptions introduce a category, attribute, location, action/motion, or relation/interaction mismatch. Reviewers check the complete video to verify that no visible target satisfies the full description at any point. NTE expressions use the same affirmative referring style as positive queries, without words that reveal the absence label. All evaluated frames have all-zero ground-truth masks. Table B reports the distribution of these semantic subtypes across the full dataset.

Table B: Distribution of no-target expression subtypes in LD-RSVIS.

NTE semantic subtypeExpressionsPercentage
Category mismatch2,38724.41%
Attribute mismatch1,92119.65%
Location mismatch1,68417.22%
Action/motion mismatch2,41324.68%
Relation/interaction mismatch1,37314.04%
Total9,778100.00%

A target that appears in the sequence but later leaves the view or becomes fully occluded remains an STE/MTE target, with empty masks for its invisible frames. NTE instead requires no visible match anywhere in the complete sequence.

Handling by baselines and Cascade-RSVIS. As described in our response to Q3, the baselines handle NTE using their respective prediction mechanisms; these mechanisms apply across the NTE semantic subtypes.

In Cascade-RSVIS, TCM assesses text-related candidates and rejects clips without a matching target; other clips proceed to CGS.

Whole-video decision. The threshold for each method's empty-output decision follows its prediction mechanism and the corresponding adaptation to our benchmark. After assembling clip predictions into a complete video, an NTE sample is correct only if every evaluated frame contains no foreground; any foreground in any frame makes it incorrect. An always-empty predictor therefore receives a perfect NTE score by definition, but this does not reflect its ability to segment present targets. We report results for each setting and do not assess a method based on its NTE score alone. For NTE sample ii,

\qquad J_i=F_i=A_i,$$ where $\widehat M_{i,t}$ is the final binary mask and $\mathcal T_i$ includes all evaluated frames. Clip predictions are assembled before this decision. In the revision, we will add the NTE semantic subtype distribution and explain how the baselines and Cascade-RSVIS handle no-target queries, including the whole-video empty-output decision.
Reviewer_XdhWOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LD-RSVIS, a large-scale benchmark for referring surgical video instrument segmentation (RSVIS): 3,536 video clips, 1.09M frames, 30 instrument classes, and 25 procedures, with 58,666 referring expressions. Compared with existing RSVIS datasets, it adds no-target (NTE) and multi-target (MTE) referring settings in addition to single-target (STE). The authors select and split clips from 15 public surgical video datasets, build masklets through multiple rounds of human annotation and expert review, retrain and evaluate 12 representative methods (3 RSVIS and 9 general RVOS methods), and propose Cascade-RSVIS, a SAM 2-based baseline: in the TCM stage, text cross-attention predicts masks per frame and the top-KK masks by IoU form the target context; in the CGS stage, the context is encoded into a memory, fused with text features through memory attention, and decoded into segmentation. Cascade-RSVIS reports an overall J&F of 37.4, compared with 33.1 for FlowRVS. Overall, this is a substantial and practically useful benchmark paper.

Critical strengths
  • Problem level: the paper targets an overlooked dimension—target cardinality—and provides usable scale and diversity. Existing RSVIS datasets contain at most a few hundred videos and support only single-target referring. LD-RSVIS scales to 3,536 clips, 1.09M frames, 30 instrument classes, and 25 procedures, while providing NTE/STE/MTE. For the community, this is likely to have more long-term value than another method that improves by one point; the NTE example in Figure 2 also makes the clinical meaning of "empty output" concrete.
  • Method/evaluation level: 12 methods are retrained and evaluated under one protocol, with results broken down by MTE/STE/NTE. Retraining 12 methods is a large amount of work and easy to get wrong; the authors deserve credit for doing it. Table 3 shows that FlowRVS/ReferMo are strong on MTE, ReSurgSAM2 is strong on NTE, and Cascade leads clearly on NTE; these structural observations are useful for future work.
Critical weaknesses
  1. The evaluation details could be more complete, since they affect how interpretable the headline numbers are. The main text does not specify a validation split, and K=3K=3, Nq=3N_q=3 in Tables 5/8 are selected as the "best" test J&F ("K=3 achieves the best J&F of 37.4", "Nq=3 achieves the best"). If hyperparameters are indeed selected on the test set, a held-out validation version would make 37.4 and the ranking more reliable. Another detail is that overall J&F mixes two metrics with different semantics: STE/MTE use per-frame IoU/contour, while NTE is an all-or-nothing binary decision (1 if the entire sequence is correct, 0 otherwise). An always-empty model would score 100 on NTE and would noticeably pull up the overall number. I would recommend defining "empty" precisely (all-zero? threshold? clip-level or video-level?) and reporting an always-empty baseline as a sanity check. These are common and easy-to-add details in benchmark papers.
  2. The method attribution could be cleaner: Cascade's gain is concentrated in NTE, and a CGS-only control is missing. In Table 3, Cascade is 39.8 vs 38.7 for FlowRVS on MTE (+1.1) and 39.6 vs 38.8 on STE (+0.8); the large gap is on NTE (30.6 vs 16.2). Thus the overall +4.3 headline is driven mainly by whether the model outputs empty, not by segmentation quality. This does not mean the method is ineffective, but it suggests describing it as "more sensitive to target absence" rather than "better segmentation overall." Table 4 compares only TCM alone (30.7) with the cascade (37.4); adding a CGS-only variant (without the context mined by TCM) or a GT-first-frame-mask upper bound would show whether +6.7 comes from context mining or from the extra parameters/training of the second stage. In addition, Table 3 currently has no error bars; for differences of about one point on MTE/STE, multiple seeds would make the conclusion more reliable.
Questions for authors
  • Is there a validation split? On which split were KK, NqN_q, the context fusion strategy, and the number of training epochs selected? If they were selected on the test set, could you add a held-out validation version?
  • Could you define the NTE "empty" decision precisely: is it no foreground in any frame of the whole video, or a per-clip decision? What is the threshold? What do an always-empty model score on NTE and overall? This sanity check would help readers interpret the metric.
  • For STE/MTE, are frames in which the target temporarily leaves the view or is occluded annotated as empty and included in J/F? Based on the numbers in Tables 1/2, 58,666 expressions correspond to 3.85M masks, i.e., about 66 masks per expression on average (about 79 even if only ST/MT are counted), while the average video length is 308 frames. How are frames without a mask supervised and evaluated? If these frames are excluded, false positives on target-absent frames are not penalized, which would weaken the long-term tracking challenge the benchmark aims to measure.
  • How were NTE constructed: by pairing an existing expression with a video that does not contain the target, or by writing them independently? Could you report the distribution of NTE types (never present / leaves the view / occluded) and the distribution of the number of targets in multi-target expressions? This would help readers judge the difficulty of NTE.
  • Could you add a CGS-only variant (without TCM context) and a GT-first-frame-mask upper bound? If CGS-only is already close to the cascade, then +6.7 is better attributed to the capacity of the second stage than to context mining. This would not invalidate the method, but it would make the contribution description more accurate.
  • What are the training details for the 12 baselines (resolution, clip length, learning rate, epochs, and whether each was tuned according to its official recipe)? Were all methods trained with the same 896×896, 8-frame input protocol? Providing these details would make the ranking in Table 3 more convincing.
  • The examples in Figures 2 and 6 look fairly templated (color + instrument + location + motion + tissue + neighboring instrument). Could you report the number of templates, lexical diversity, and whether the expressions were written freely by clinicians or generated by annotators from templates? If templating is substantial, it would be worth stating this in the paper and discussing whether models may be largely learning the templates.
  • The "expert group" in the annotation team is described as "students who work on related surgical video understanding tasks." Could you clarify whether clinically qualified surgeons participated in the review? This is important for instrument semantics and boundary quality, and would give readers more confidence in the annotation quality.
8 comments

Rebuttal (Part 1/7)

Q1: Is there a validation split? On which split were (K), (N_q), the context fusion strategy, and the number of training epochs selected? If they were selected on the test set, could you add a held-out validation version?

A1: Thank you for raising this important point. Following the train/test organization of existing surgical RVOS benchmarks such as Ref-EndoVis and UniPVOS, the original LD-RSVIS protocol did not include a separate validation set. Initial settings were empirical, and the final configuration was selected using the reported sensitivity and ablation comparisons on the original test set. We agree that this selection was not separated from final evaluation and should have been stated clearly.

Following the reviewer's suggestion, we additionally reserve 100 videos from the original test pool for validation, leaving 583 videos as a new test set and keeping the 2,853 training videos unchanged. On the 100-video validation set, we compare the three configuration choices: KK, NqN_q, and context fusion. Tables A-C favor K=3K=3, Nq=3N_q=3, and IoU-weighted addition, respectively, consistent with the original configuration.

Table A: Ablation on KK in TCM on the held-out validation set.

Top KK predictionsOverall JJOverall FFOverall J\\&F
K=1K=136.640.038.3
K=3K=338.040.639.3
K=5K=537.939.438.7

Table B: Ablation on NqN_q on the held-out validation set.

Num. of queries NqN_qOverall JJOverall FFOverall J\\&F
Nq=1N_q=136.439.738.1
Nq=3N_q=338.040.639.3
Nq=5N_q=537.139.338.2
Nq=7N_q=736.940.138.5

Table C: Ablation on context fusion in CGS on the held-out validation set.

Context fusion strategyOverall JJOverall FFOverall J\\&F
Addition38.338.638.5
Gated fusion37.839.838.8
IoU-weighted addition38.040.639.3

In the revision, we will identify the original test-set selection, describe how the 100-video validation set and 583-video test set were formed, and add the three validation comparisons for KK, NqN_q, and context fusion while labeling Table 3 with its original test set.

Rebuttal (Part 2/7)

Q2: Could you define the NTE "empty" decision precisely: is it no foreground in any frame of the whole video, or a per-clip decision? What is the threshold? What do an always-empty model score on NTE and overall? This sanity check would help readers interpret the metric.

A2: Thank you for asking about the empty-output criterion and its interpretation. We clarify the video-level decision and the aggregation below.

(a) Whole-video decision. The threshold for each method's empty-output decision follows its prediction mechanism and the corresponding adaptation to our benchmark. After assembling clip predictions into a complete video, an NTE sample is correct only if every evaluated frame contains no foreground; any foreground in any frame makes it incorrect. An always-empty predictor therefore receives a perfect NTE score by definition, but this does not reflect its ability to segment present targets. We report results for each setting and do not assess a method based on its NTE score alone. Formally, Ai=1[∀t∈Ti,M^i,t=∅]A_i=\mathbf{1}[\forall t\in\mathcal T_i,\widehat M_{i,t}=\varnothing], and Ji=Fi=AiJ_i=F_i=A_i for NTE.

(b) Meaning of Overall. Overall combines region/boundary segmentation with whole-sequence absence correctness; it is not a pure segmentation-quality measure. We clarify the aggregation and provide a sample-weighted comparison below.

STE, MTE, and NTE contain different numbers of queries. To account for this imbalance, we use fixed setting weights to control their contributions to Overall, rather than allowing those contributions to be determined solely by query frequency. Specifically, the submitted paper assigns weights of 0.50 to STE and 0.25 each to MTE and NTE:

0.25X_{\mathrm{MTE}}+0.50X_{\mathrm{STE}}+0.25X_{\mathrm{NTE}}, \qquad X\in\\{J,F\\}.$$ These coefficients are predefined evaluation weights, not the exact proportions of queries in the test set. The same weights and aggregation procedure are applied to all baselines and Cascade-RSVIS, providing a consistent basis for comparing methods under this metric. As a supplement, we provide sample-weighted aggregation, in which each video--expression sample has equal weight. For raw sample scores $X_i$, its definition is $$X_{\mathrm{overall}}^{\mathrm{sample}}=\frac{1}{N}\sum_{i=1}^N X_i =\sum_g\frac{N_g}{N}\overline X_g, \quad g\in\\{\mathrm{STE},\mathrm{MTE},\mathrm{NTE}\\},\quad X\in\\{J,F\\}.$$ Here, $N_g$ denotes the number of samples in setting $g$, and $\overline X_g$ is their sample-level mean score. The original test set contains 6,573 STE, 2,732 MTE, and 1,861 NTE samples ($N=11,166$). Table D supplements, rather than replaces, the original setting-weighted results. Cascade-RSVIS obtains 38.2 compared with FlowRVS's 35.0 under this supplementary calculation. **Table D: Comparison with existing methods on the original test set using sample-weighted average.** | Method | Overall $J$ | Overall $F$ | Overall $J\\&F$ | | --- | --- | --- | --- | | VIS-Net | 21.7 | 19.3 | 20.5 | | ReferFormer | 19.9 | 24.9 | 22.4 | | OnlineRefer | 21.3 | 25.4 | 23.4 | | LoSh | 23.7 | 29.7 | 26.7 | | LMPM | 25.3 | 27.3 | 26.3 | | ReSurgSAM2 | 27.1 | 29.9 | 28.5 | | SurgRef | 29.7 | 28.0 | 28.9 | | DsHmp | 24.6 | 33.7 | 29.2 | | SAMWISE | 30.3 | 34.1 | 32.2 | | ReferDINO | 30.7 | 35.5 | 33.1 | | ReferMo | 33.6 | 34.3 | 34.0 | | FlowRVS | 31.5 | 38.5 | 35.0 | | Cascade-RSVIS (ours) | 36.7 | 39.6 | 38.2 | STE/MTE measure region overlap and boundary quality, whereas NTE measures whether a complete video--expression sample is correctly predicted as empty. For NTE sample $i$, $$A_i=\mathbf{1}\\!\left[\forall t\in\mathcal T_i,\\ \widehat M_{i,t}=\varnothing\right], \qquad J_i=F_i=A_i,$$ where $\widehat M_{i,t}$ is the final binary mask and $\mathcal T_i$ includes all evaluated frames. Clip predictions are assembled before this decision. Thus, Overall combines positive-target segmentation and target-absence handling; it should not be interpreted as a measure of positive-target segmentation quality alone. In the revision, we will define the whole-video empty-output decision, explain how it contributes to Overall, and present the setting-specific scores and sample-weighted results alongside the original aggregation.

Rebuttal (Part 3/7)

Q3: For STE/MTE, are frames in which the target temporarily leaves the view or is occluded annotated as empty and included in J/F? Based on the numbers in Tables 1/2, 58,666 expressions correspond to 3.85M masks, i.e., about 66 masks per expression on average (about 79 even if only ST/MT are counted), while the average video length is 308 frames. How are frames without a mask supervised and evaluated? If these frames are excluded, false positives on target-absent frames are not penalized, which would weaken the long-term tracking challenge the benchmark aims to measure.

A3: Thank you for these questions. Temporarily absent frames are supervised and evaluated; the reported mask count uses a different unit from the number of video--expression frames.

(a) Annotation and supervision. A temporarily absent target is represented by an all-zero GT mask, rather than an unlabeled or excluded frame. Partial occlusion is annotated over the visible region; leaving the field of view or complete occlusion produces empty GT. In MTE, the per-frame GT unions the visible masks of all referred targets, so it is empty only when none is visible. These empty frames participate in training and in the sequence-level J/FJ/F averages. Predicting foreground on an empty-GT frame is penalized.

(b) Counting unit. The reported 3,847,264 masks count non-empty instance masks: 1,941,650 whole-instrument masks and 1,905,614 part masks. This counts an annotated instrument/part instance in a frame, not one GT frame for every video--expression sample. Expression-level sequences are constructed from the corresponding temporal masklets; MTE unions reuse whole-instrument annotations rather than adding new manually annotated instance masks. Empty GT frames are supervised and evaluated but are not counted as non-empty instance masks.

Therefore, dividing 3.85M by 58,666 does not estimate the number of supervised frames per expression.

In the revision, we will rename the mask count as non-empty instance masks and explain masklet reuse, empty-frame supervision, and their inclusion in evaluation.

Q4: How were NTE constructed: by pairing an existing expression with a video that does not contain the target, or by writing them independently? Could you report the distribution of NTE types (never present / leaves the view / occluded) and the distribution of the number of targets in multi-target expressions? This would help readers judge the difficulty of NTE.

A4: Thank you for this suggestion. We clarify how NTE queries are constructed, distinguish sequence-level from temporary absence, and report the requested target-cardinality distribution.

(a) Construction. NTE expressions are constructed for the particular video, rather than randomly imported from other videos. Using the video's instruments and scene context, candidate descriptions introduce a category, attribute, location, action/motion, or relation/interaction mismatch. Reviewers check the complete video to verify that no visible target satisfies the complete description at any point. NTE uses the same affirmative referring style as positive queries, without words such as ``absent'' that reveal the label. Its GT target set is empty in every evaluated frame.

(b) Sequence-level absence versus temporary absence. A referred target that appears in the sequence but later leaves the view or is fully occluded remains an STE/MTE target; its invisible frames have empty masks. Such events are not separate NTE subtypes under our definition. NTE instead requires no visible match anywhere in the complete sequence. We characterize its semantic mismatch types in Table E.

Table E: Distribution of no-target expression subtypes in LD-RSVIS.

NTE semantic subtypeExpressionsPercentage
Category mismatch2,38724.41%
Attribute mismatch1,92119.65%
Location mismatch1,68417.22%
Action/motion mismatch2,41324.68%
Relation/interaction mismatch1,37314.04%
Total9,778100.00%

(c) MTE cardinality. Table F reports the number of distinct instrument identities referred to over the full sequence. Different instruments of the same class count separately; the same instrument leaving and re-entering does not count again. This is the size of the referred target set, not the maximum simultaneous on-screen count.

Table F: Distribution of target-set cardinality for multi-target expressions in LD-RSVIS.

Referred targetsExpressionsPercentage
28,91763.04%
34,20629.74%
≥4\geq 41,0217.22%
Total14,144100.00%

In the revision, we will describe video-specific NTE construction and full-sequence verification, distinguish temporary absence from NTE, and add the NTE subtype and MTE target-cardinality distributions.

Reviewer_uAK9OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces LD-RSVIS, a benchmark with 3,536 surgical video clips and 58,666 referring expressions covering single-target, multi-target, and no-target cases. It also proposes Cascade-RSVIS, a SAM 2-based method that mines target-specific context before refining segmentation. Validation includes comparisons with 12 methods, component ablations, annotation-quality checks, and evaluations on existing surgical benchmarks.

Critical strengths

A substantial benchmark contribution. LD-RSVIS provides 3,536 video clips spanning 30 instrument classes and 25 procedures, with manual annotation and an independent quality check. This broader coverage gives the community a valuable resource for training and evaluating surgical referring segmentation models and is the main reason I support acceptance.

A more complete task setting. Including multi-target and no-target expressions extends evaluation beyond identifying a single, always-present instrument. This tests whether models can select all matching targets and avoid false detections when none exists, making the benchmark more relevant to practical use.

Critical weaknesses

The model-selection protocol is unclear. Only training and test splits are described, while the choices of (K) and (N_q) are justified using test-set comparisons. If these settings were selected on the test set, the reported performance could be optimistic. Clarifying this is important for a benchmark intended to support reliable future comparisons.

The independence of the external evaluation is not established. LD-RSVIS includes EndoVis17/18 videos, which also underpin the benchmarks in Table 11. The paper does not clearly explain the training setup or how overlap is excluded. This limits how confidently those results can be interpreted as evidence of generalization.

Questions for authors

The paper describes training and test splits, but the validation setup is unclear. Since the choices of (K) and (N_q) are discussed alongside test-set results, please explain how these settings were selected and whether a separate validation split was used.

The overall scores in Table 3 appear to use weights of 0.25, 0.50, and 0.25 for multi-target, single-target, and no-target cases. Please state this rule and explain the weighting. Also, the single-target J&F is 39.6 in Table 3 but 41.2 in Section 5.1.

For Table 11, please clarify whether models are trained separately on each benchmark, fine-tuned from LD-RSVIS, or evaluated directly. Since LD-RSVIS also draws from EndoVis17/18, it would help to explain how overlap with the external test videos is excluded.

Training all baselines on the same dataset is a good starting point. A short description of how methods originally designed for single-target segmentation handle multiple targets and empty outputs would make the comparison easier to interpret.

The independent reannotation study is reassuring. Please clarify whether it also checks the meaning of the expressions, particularly instrument-part descriptions and no-target cases, and briefly describe how ambiguous expressions are resolved.

The benchmark covers many procedures and instrument categories, but the results are mostly aggregated. A breakdown by procedure or instrument category would make the dataset more informative and show where current methods struggle most.

The method describes context selection over a video, while inference uses eight-frame clips. Please explain whether context and memory are shared across clips or each clip is processed independently. This would make the temporal scope of the method much clearer.

8 comments

Rebuttal (Part 1/4)

Q1: The paper describes training and test splits, but the validation setup is unclear. Since the choices of (K) and (N_q) are discussed alongside test-set results, please explain how these settings were selected and whether a separate validation split was used.

A1: Thank you for raising this important point. Following the train/test organization of existing surgical RVOS benchmarks such as Ref-EndoVis and UniPVOS, the original LD-RSVIS protocol did not include a separate validation set. Initial settings were empirical, and the final configuration was selected using the reported sensitivity and ablation comparisons on the original test set. We agree that this selection was not separated from final evaluation and should have been stated clearly.

Following the reviewer's suggestion, we additionally reserve 100 videos from the original test pool for validation, leaving 583 videos as a new test set and keeping the 2,853 training videos unchanged. On the 100-video validation set, we compare the three configuration choices: KK, NqN_q, and context fusion. Tables A-C favor K=3K=3, Nq=3N_q=3, and IoU-weighted addition, respectively, consistent with the original configuration.

Table A: Ablation on KK in TCM on the held-out validation set.

Top KK predictionsOverall JJOverall FFOverall J\\&F
K=1K=136.640.038.3
K=3K=338.040.639.3
K=5K=537.939.438.7

Table B: Ablation on NqN_q on the held-out validation set.

Num. of queries NqN_qOverall JJOverall FFOverall J\\&F
Nq=1N_q=136.439.738.1
Nq=3N_q=338.040.639.3
Nq=5N_q=537.139.338.2
Nq=7N_q=736.940.138.5

Table C: Ablation on context fusion in CGS on the held-out validation set.

Context fusion strategyOverall JJOverall FFOverall J\\&F
Addition38.338.638.5
Gated fusion37.839.838.8
IoU-weighted addition38.040.639.3

In the revision, we will identify the original test-set selection, describe how the 100-video validation set and 583-video test set were formed, and add the three validation comparisons for KK, NqN_q, and context fusion while labeling Table 3 with its original test set.

Rebuttal (Part 2/4)

Q2: The overall scores in Table 3 appear to use weights of 0.25, 0.50, and 0.25 for multi-target, single-target, and no-target cases. Please state this rule and explain the weighting. Also, the single-target J&F is 39.6 in Table 3 but 41.2 in Section 5.1.

A2: Thank you for identifying both issues. We clarify the aggregation rule and correct the STE score below.

(a) Aggregation. STE, MTE, and NTE contain different numbers of queries. To account for this imbalance, we use fixed setting weights to control their contributions to Overall, rather than allowing those contributions to be determined solely by query frequency. Specifically, the submitted paper assigns weights of 0.50 to STE and 0.25 each to MTE and NTE:

0.25X_{\mathrm{MTE}}+0.50X_{\mathrm{STE}}+0.25X_{\mathrm{NTE}}, \qquad X\in\\{J,F\\}.$$ These coefficients are predefined evaluation weights, not the exact proportions of queries in the test set. The same weights and aggregation procedure are applied to all baselines and Cascade-RSVIS, providing a consistent basis for comparing methods under this metric. As a supplement, we provide sample-weighted aggregation, in which each video--expression sample has equal weight. For raw sample scores $X_i$, its definition is $$X_{\mathrm{overall}}^{\mathrm{sample}}=\frac{1}{N}\sum_{i=1}^N X_i =\sum_g\frac{N_g}{N}\overline X_g, \quad g\in\\{\mathrm{STE},\mathrm{MTE},\mathrm{NTE}\\},\quad X\in\\{J,F\\}.$$ Here, $N_g$ denotes the number of samples in setting $g$, and $\overline X_g$ is their sample-level mean score. The original test set contains 6,573 STE, 2,732 MTE, and 1,861 NTE samples ($N=11,166$). Table D supplements, rather than replaces, the original setting-weighted results. Cascade-RSVIS obtains 38.2 compared with FlowRVS's 35.0 under this supplementary calculation. **Table D: Comparison with existing methods on the original test set using sample-weighted average.** | Method | Overall $J$ | Overall $F$ | Overall $J\\&F$ | | --- | --- | --- | --- | | VIS-Net | 21.7 | 19.3 | 20.5 | | ReferFormer | 19.9 | 24.9 | 22.4 | | OnlineRefer | 21.3 | 25.4 | 23.4 | | LoSh | 23.7 | 29.7 | 26.7 | | LMPM | 25.3 | 27.3 | 26.3 | | ReSurgSAM2 | 27.1 | 29.9 | 28.5 | | SurgRef | 29.7 | 28.0 | 28.9 | | DsHmp | 24.6 | 33.7 | 29.2 | | SAMWISE | 30.3 | 34.1 | 32.2 | | ReferDINO | 30.7 | 35.5 | 33.1 | | ReferMo | 33.6 | 34.3 | 34.0 | | FlowRVS | 31.5 | 38.5 | 35.0 | | Cascade-RSVIS (ours) | 36.7 | 39.6 | 38.2 | **(b) Metric interpretation.** STE/MTE measure region overlap and boundary quality, whereas NTE measures whether a complete video--expression sample is correctly predicted as empty. For NTE sample $i$, $$A_i=\mathbf{1}\\!\left[\forall t\in\mathcal T_i,\\ \widehat M_{i,t}=\varnothing\right], \qquad J_i=F_i=A_i,$$ where $\widehat M_{i,t}$ is the final binary mask and $\mathcal T_i$ includes all evaluated frames. Clip predictions are assembled before this decision. Thus, Overall combines positive-target segmentation and target-absence handling; it should not be interpreted as a measure of positive-target segmentation quality alone. **(c) Textual inconsistency.** The correct STE scores for Cascade-RSVIS are $J=38.0$, $F=41.2$, and $J\\&F=39.6$. Section 5.1 mistakenly reports the $F$ value as $J\\&F$. The MTE/STE/NTE $J\\&F$ values are 39.8/39.6/30.6. This is a reporting error, not a change to the experiment. In the revision, we will specify the setting weights and whole-video NTE scoring rule, add the sample-weighted comparison, and correct the STE $J\\&F$ value in Section 5.1 to 39.6. > **Q3:** For Table 11, please clarify whether models are trained separately on each benchmark, fine-tuned from LD-RSVIS, or evaluated directly. Since LD-RSVIS also draws from EndoVis17/18, it would help to explain how overlap with the external test videos is excluded. **A3:** Thank you for asking about the external evaluation protocol. For Table 11, models trained by us start from each method's public initialization weights and are trained separately on each benchmark's training set, following its official train/test partition and evaluation protocol. No LD-RSVIS-trained checkpoint is loaded. Thus, these experiments evaluate benchmark-specific training rather than direct transfer from LD-RSVIS. Results taken from the original papers will be attributed accordingly. In the revision, we will specify that the Table 11 models trained by us start from public initialization weights and are trained separately on each external benchmark, without loading LD-RSVIS-trained checkpoints.

Rebuttal (Part 3/4)

Q4: Training all baselines on the same dataset is a good starting point. A short description of how methods originally designed for single-target segmentation handle multiple targets and empty outputs would make the comparison easier to interpret.

A4: Thank you for this suggestion. Each method receives a video and one referring expression and outputs an expression-level binary mask sequence. We adapt supervision and output interpretation to the three settings while retaining each method's prediction mechanism:

  • STE: each expression refers to a single target, and the model is supervised using the corresponding ground-truth mask.
  • MTE: the per-frame GT is the union of all visible referred targets. Direct expression-mask methods are supervised with that union. Methods retaining instance/candidate predictions include multiple matching targets in training matching or target selection, then union the selected masks at inference. They are not restricted to one selected instance.
  • NTE: all evaluated frames have all-zero GT masks and participate in training. Candidate-based methods can reject all candidates; presence-based methods use their presence outputs; direct-mask methods can return all-background masks under their original pixel binarization.

In the revision, we will describe the single-target supervision, multi-target mask union and candidate selection, and empty-target supervision and output handling used to adapt the baselines.

Q5: The independent reannotation study is reassuring. Please clarify whether it also checks the meaning of the expressions, particularly instrument-part descriptions and no-target cases, and briefly describe how ambiguous expressions are resolved.

A5: Thank you for highlighting the importance of expression semantics. The original reannotation study checked masks: its IoU of 0.93 on 50 sequences measures segmentation consistency rather than expression correctness. During construction, the annotation review group first defined and checked the expressions and their referents, after which annotators produced the corresponding temporal masklets.

Following the suggestion, three members of the annotation team additionally reviewed 200 sampled expressions, with 50 each from Whole, Part, MTE, and NTE. They checked the complete corresponding videos for semantic correctness, referent identity, and agreement with the annotation. The review specifically checked parent-instrument/part correspondence, completeness of MTE target sets, and the absence of any visible match throughout NTE sequences. All 200 sampled expressions passed (100%).

In the revision, we will distinguish expression review from mask reannotation and report the 200-expression semantic check, including its four sampled groups and review criteria.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.