- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces DySurg, a new dataset for dynamic surgical workflow analysis, with 470 surgical videos and more than 100 hours of video. Each video includes temporal keystep descriptions and expert commentary. The paper also proposes MSVLR, a multi-scale vision-language framework that uses video titles and expert commentary during training to learn hierarchical workflow representations, while only requiring the video and title at inference.
Critical strengths
The new dataset and task are potentially valuable. Moving beyond a fixed phase/step ontology is an interesting direction for surgical video understanding. DySurg provides video-specific workflow annotations together with expert commentary, which could support research on more flexible and realistic surgical workflow analysis.
Critical weaknesses
1. The clinical validity and reproducibility of the dataset annotations are not sufficiently established.
The dataset is a major contribution, but the clinical validation protocol for the final workflow annotations is unclear. The paper mentions surgeon review for ASR quality control, but does not clearly state how many clinical experts annotated or verified the final keysteps and boundaries. This is particularly important for medical annotations, where the definition and temporal boundaries of a surgical keystep can be subjective and require rigorous expert validation. Ideally, multiple clinical experts should independently and blindly annotate a subset of the data, followed by inter-rater agreement analysis and expert adjudication of disagreements. Without such validation, it is difficult to assess the clinical reliability and reproducibility of the proposed dataset.
2. The paper does not clearly establish that DySurg really requires a dynamic, video-specific workflow formulation rather than a richer fixed step ontology.
The main motivation is that different cases can have different keysteps, but this is currently supported mostly by qualitative examples. Existing step-recognition datasets can already allow steps to be omitted, repeated, or reordered across cases. The paper does not quantify how much the step semantics or workflow structure vary within each surgical category, so it is unclear whether most DySurg annotations could still be mapped to a shared category-level step vocabulary.
Questions for authors
Q1. How were the DySurg annotations clinically validated?
Could the authors provide more details on the number and expertise of annotators, the definition of a valid keystep, whether multiple surgeons independently annotated the same videos, and whether inter-rater agreement or expert adjudication was performed?
Q2. Can most DySurg workflows still be covered by a fixed step ontology within each surgical category?
For example, could the authors show what percentage of keysteps can be mapped to a shared set of canonical steps, and how many steps are truly case-specific? This would help show whether DySurg really requires a dynamic workflow formulation rather than a richer fixed step vocabulary.
Q3. Could the authors separately ablate title alignment and expert-commentary alignment?
The current commentary-alignment ablation removes semantic alignment as a whole, so it is unclear whether the gain mainly comes from the video title, the expert commentary, or the combination of both.
3 comments
Continued response
-
DySurg workflows covered by a fixed step ontology within each surgical category: DySurg requires a dynamic formulation, and a richer fixed step vocabulary would not replace it, because operations in the same category differ substantially in their keysteps.
Figure 1 in the paper shows this within a single category. Both videos in Fig. 1(B) are Upper GI operations, and both are esophagectomies, so they share the category and the overall goal. They differ in technique, as their titles state: one is a minimally invasive esophagectomy with tubularized gastric pull-up, and the other is a laparoscopic and thoracoscopic Ivor-Lewis esophagectomy. Their keysteps differ accordingly.
Video 1: esophagectomy with tubularized gastric pull-up Video 2: Ivor-Lewis esophagectomy P1 Dissection of the esophagus Abdominal phase operative set-up and port placement P2 Mobilization of the esophagus Greater curvature mobilization and Kocher maneuver P3 Laparoscopy, trocar and surgeon position Ligation of left gastric vessels P4 Port placement Dissection of posterior esophagus P5 Patient set-up Mediastinal lymphadenectomy phase P6 Esophagectomy and anastomosis esophagus The two workflows differ in four ways:
- Number of keysteps. Six in the first video and five in the second. Order. The first video begins with the dissection of the esophagus and reaches set-up and port placement in the middle. The second begins with set-up and port placement and dissects the esophagus fourth.
- Granularity. Set-up is three separate keysteps in the first video and a single keystep in the second.
- Content. The anastomosis is a keystep only in the first video. Gastric mobilization with the Kocher maneuver, ligation of the left gastric vessels and mediastinal lymphadenectomy are keysteps only in the second.
The same holds across the dataset. The two colorectal operations in Fig. 6, a liver resection for colorectal metastases and a transanal polyp resection, have no keystep in common. Neither do the two thoracic operations in Fig. 8. Even two videos of the same procedure, the Nissen fundoplications in Fig. 7, are annotated with seven and four keysteps.
Why a richer fixed vocabulary is not enough?
- A DySurg category is a surgical specialty, not a procedure. A category-level vocabulary would have to list in advance every step of every procedure and technique in that specialty, and it would have to be extended whenever a new one appears. Existing fixed ontologies are defined for one procedure each, such as the 12 phases and 46 steps of gastric bypass (App. C).
- A fixed vocabulary also fixes the granularity. In DySurg the granularity follows the operation: set-up is one keystep in one esophagectomy and three in the other.
- A fixed vocabulary cannot name a step outside its list, such as "Ureterolysis, endometriotic nodules excision, and salpingo-oophorectomy" in a thoracic case (Fig. 8) or "Control of bleeding" in a liver resection (Fig. 6).
- Such a vocabulary can be written only after all the videos have been seen. For a new operation it does not exist, whereas the dynamic formulation derives the keysteps from the video and its title alone (Fig. 1(B)).
We agree that some steps recur across videos, port placement for example. The dynamic formulation does not deny this. It removes the requirement that every step be enumerated beforehand.
-
Separately ablate title alignment and expert-commentary alignment: Table 1 in the paper demonstrates results of seperating title alignment and expert-commentary alignment:
- Removing expert commentary (the variant without the middle level, denoted as Middle) results in [email protected]/[email protected]/[email protected] to be 0.35/0.28/0.17.
- However, removing title is not feasible in our framework. Because without title, we have no way to split expert commentary into different scetions (the middle level). That means the expert commentary alignment will disapear together with the title removal.
We thank the reviewer again for the careful reading. We hope these answers and the added analyses resolve the concerns.
Thank you for the detailed response and clarifications. However, my main concern about the clinical validity and reliability of the DySurg dataset remains. The authors acknowledge that no additional independent re-annotation was performed, while the original keystep timestamps and descriptions were further refined. Although the annotations originate from operating surgeons and WebSurg peer review, this does not establish the reliability of the final dataset annotations. Without independent annotations from multiple clinical experts and an assessment of their agreement on keystep labels and temporal boundaries, the quality of the final annotations remains insufficiently verified. Since the dataset is a central contribution of this paper, I will maintain my original rating.
Author response to review comments
Thank you for the review and for recognising the potential of the dataset and the task. We respond to the questions below in detail.
Clinical validity and reproducibility of the dataset annotations: The keysteps come from the operating surgeons. Keysteps in the DySurg dataset are not labels assigned afterwards by third-party annotators. They are based on the keystep information published with each video on WebSurg (Sec. 2). That information is written by the surgeon who performed and narrated the operation, and it is published only after WebSurg's peer review. The operating surgeon is the person best placed to say which steps were performed and where each begins. An annotation with this provenance has stronger clinical validity than a consensus reached afterwards among annotators who did not perform the case. We refined the timestamps and wording of these keysteps and removed entries without surgical content (Sec. 2). The commentary transcripts were separately reviewed by surgeons with expertise in the relevant category (Sec. 2).
Motivation on dynamic, video-specific formulation: In the paper, we present three major reasons that motivates the dynamic, video-specific formulation:
Clinical validation of DySurg: Five points demonstrate the clinical validation of DySurg:
(to be continued)