acceptodds
Under review as a conference paper at ICLR 2027

Analyzing Dynamic Surgical Workflows Through Multi-Scale Vision-Language Reasoning

Abstract

Analyzing surgical workflow is critical for understanding complex procedural dynamics of a surgery. Current works focus on surgical workflow recognition to classify video streams into predetermined workflow sequences, inadequately representing the adaptive nature of clinical surgeries that respond to patient variability and evolving circumstances. We introduce the dynamic surgical workflow reasoning task, which eliminates fixed workflow constraints to address these limitations. Supporting this paradigm shift, we present DySurg (Dynamic Surgical Workflow), a comprehensive dataset containing over 100 hours of long surgical videos in real-world clinical recordings with annotated dynamic workflows across 7 major surgical categories. To enhance reasoning capabilities for this analysis, we propose an commentary-aligned video reasoning framework that constructs top-down visual reasoning sequences modeling surgeons' cognitive processes. Our approach aligns visual embeddings from surgical videos with semantic information extracted from video title and expert commentary, thereby aligning explainability with the visual representations. During inference, our model maps video frames to the semantic space to generate appropriate workflows, without the need of expert commentary. Extensive evaluations on the DySurg dataset demonstrates that our approach significantly outperforms existing large vision-language models (e.g. QWen2.5-VL) and surgical-specific pre-trained models (e.g., SurgVLP) in recognizing dynamic surgical workflows. All code, data, and models will be publicly released after the review process concludes.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Analyzing Dynamic Surgical Workflows Through Multi-Scale Vision-Language Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QtRXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DySurg, a new dataset for dynamic surgical workflow analysis, with 470 surgical videos and more than 100 hours of video. Each video includes temporal keystep descriptions and expert commentary. The paper also proposes MSVLR, a multi-scale vision-language framework that uses video titles and expert commentary during training to learn hierarchical workflow representations, while only requiring the video and title at inference.

Critical strengths

The new dataset and task are potentially valuable. Moving beyond a fixed phase/step ontology is an interesting direction for surgical video understanding. DySurg provides video-specific workflow annotations together with expert commentary, which could support research on more flexible and realistic surgical workflow analysis.

Critical weaknesses

1. The clinical validity and reproducibility of the dataset annotations are not sufficiently established.

The dataset is a major contribution, but the clinical validation protocol for the final workflow annotations is unclear. The paper mentions surgeon review for ASR quality control, but does not clearly state how many clinical experts annotated or verified the final keysteps and boundaries. This is particularly important for medical annotations, where the definition and temporal boundaries of a surgical keystep can be subjective and require rigorous expert validation. Ideally, multiple clinical experts should independently and blindly annotate a subset of the data, followed by inter-rater agreement analysis and expert adjudication of disagreements. Without such validation, it is difficult to assess the clinical reliability and reproducibility of the proposed dataset.

2. The paper does not clearly establish that DySurg really requires a dynamic, video-specific workflow formulation rather than a richer fixed step ontology.

The main motivation is that different cases can have different keysteps, but this is currently supported mostly by qualitative examples. Existing step-recognition datasets can already allow steps to be omitted, repeated, or reordered across cases. The paper does not quantify how much the step semantics or workflow structure vary within each surgical category, so it is unclear whether most DySurg annotations could still be mapped to a shared category-level step vocabulary.

Questions for authors

Q1. How were the DySurg annotations clinically validated?

Could the authors provide more details on the number and expertise of annotators, the definition of a valid keystep, whether multiple surgeons independently annotated the same videos, and whether inter-rater agreement or expert adjudication was performed?

Q2. Can most DySurg workflows still be covered by a fixed step ontology within each surgical category?

For example, could the authors show what percentage of keysteps can be mapped to a shared set of canonical steps, and how many steps are truly case-specific? This would help show whether DySurg really requires a dynamic workflow formulation rather than a richer fixed step vocabulary.

Q3. Could the authors separately ablate title alignment and expert-commentary alignment?

The current commentary-alignment ablation removes semantic alignment as a whole, so it is unclear whether the gain mainly comes from the video title, the expert commentary, or the combination of both.

3 comments
AuthorsOpenReview ↗

Author response to review comments

Thank you for the review and for recognising the potential of the dataset and the task. We respond to the questions below in detail.

  • Clinical validity and reproducibility of the dataset annotations: The keysteps come from the operating surgeons. Keysteps in the DySurg dataset are not labels assigned afterwards by third-party annotators. They are based on the keystep information published with each video on WebSurg (Sec. 2). That information is written by the surgeon who performed and narrated the operation, and it is published only after WebSurg's peer review. The operating surgeon is the person best placed to say which steps were performed and where each begins. An annotation with this provenance has stronger clinical validity than a consensus reached afterwards among annotators who did not perform the case. We refined the timestamps and wording of these keysteps and removed entries without surgical content (Sec. 2). The commentary transcripts were separately reviewed by surgeons with expertise in the relevant category (Sec. 2).

  • Motivation on dynamic, video-specific formulation: In the paper, we present three major reasons that motivates the dynamic, video-specific formulation:

    • A fixed keystep must be enumerated in advance, procedure by procedure. The fixed-keystep datasets we know each cover one procedure: seven phases for cholecystectomy in Cholec80, and 12 phases and 46 steps for gastric bypass (App. C). DySurg spans seven specialties and more than 2,000 distinct procedures. Within thoracic surgery alone, the paper shows a middle lobectomy, a mediastinal lymph node dissection and a combined laparoscopic and thoracoscopic treatment of catamenial pneumothorax (Figs. 4 and 8). A richer fixed keystep for DySurg would mean designing and maintaining a step list for every such procedure. That is the constraint the task removes.
    • Omission, repetition and reordering do not cover new steps. We agree that fixed ontologies allow steps to be omitted, repeated or reordered. They cannot express a step that is not in the list. The catamenial pneumothorax case includes "Ureterolysis, endometriotic nodules excision, and salpingo-oophorectomy" (Fig. 8), which no thoracic step list would contain. The lobectomy of Fig. 4 has a keystep for a second middle-lobe artery, which exists only in patients with that anatomy. The liver resection of Fig. 6 has a keystep for the control of bleeding.
    • A vocabulary built afterwards is not available beforehand. The annotations of a finished dataset can always be mapped to shared canonical steps to some degree. Doing so requires having seen all the cases. At inference on a new operation no such list exists, and the open formulation does not need one. Fixed workflows remain a special case of the formulation, and the same model handles them (Tables 3 and 4).
  • Clinical validation of DySurg: Five points demonstrate the clinical validation of DySurg:

    • Expertise of annotators. The keysteps of each video originate from its operating surgeon and passed WebSurg's peer review.
    • Definition of a valid keystep. A contiguous interval in which the surgeon pursues one procedural sub-goal, such as exposing, dissecting, dividing or reconstructing a named structure. Chapters without a surgical action, such as administrative case labelling, are excluded (Sec. 2).
    • Independent annotation by several surgeons. The source annotation is the operating surgeon's, reviewed by WebSurg's committee. We do not include additional independent re-annotation.

(to be continued)

AuthorsOpenReview ↗

Continued response

  • DySurg workflows covered by a fixed step ontology within each surgical category: DySurg requires a dynamic formulation, and a richer fixed step vocabulary would not replace it, because operations in the same category differ substantially in their keysteps.

    Figure 1 in the paper shows this within a single category. Both videos in Fig. 1(B) are Upper GI operations, and both are esophagectomies, so they share the category and the overall goal. They differ in technique, as their titles state: one is a minimally invasive esophagectomy with tubularized gastric pull-up, and the other is a laparoscopic and thoracoscopic Ivor-Lewis esophagectomy. Their keysteps differ accordingly.

    Video 1: esophagectomy with tubularized gastric pull-upVideo 2: Ivor-Lewis esophagectomy
    P1Dissection of the esophagusAbdominal phase operative set-up and port placement
    P2Mobilization of the esophagusGreater curvature mobilization and Kocher maneuver
    P3Laparoscopy, trocar and surgeon positionLigation of left gastric vessels
    P4Port placementDissection of posterior esophagus
    P5Patient set-upMediastinal lymphadenectomy phase
    P6Esophagectomy and anastomosis esophagus

    The two workflows differ in four ways:

    • Number of keysteps. Six in the first video and five in the second. Order. The first video begins with the dissection of the esophagus and reaches set-up and port placement in the middle. The second begins with set-up and port placement and dissects the esophagus fourth.
    • Granularity. Set-up is three separate keysteps in the first video and a single keystep in the second.
    • Content. The anastomosis is a keystep only in the first video. Gastric mobilization with the Kocher maneuver, ligation of the left gastric vessels and mediastinal lymphadenectomy are keysteps only in the second.

    The same holds across the dataset. The two colorectal operations in Fig. 6, a liver resection for colorectal metastases and a transanal polyp resection, have no keystep in common. Neither do the two thoracic operations in Fig. 8. Even two videos of the same procedure, the Nissen fundoplications in Fig. 7, are annotated with seven and four keysteps.

    Why a richer fixed vocabulary is not enough?

    1. A DySurg category is a surgical specialty, not a procedure. A category-level vocabulary would have to list in advance every step of every procedure and technique in that specialty, and it would have to be extended whenever a new one appears. Existing fixed ontologies are defined for one procedure each, such as the 12 phases and 46 steps of gastric bypass (App. C).
    2. A fixed vocabulary also fixes the granularity. In DySurg the granularity follows the operation: set-up is one keystep in one esophagectomy and three in the other.
    3. A fixed vocabulary cannot name a step outside its list, such as "Ureterolysis, endometriotic nodules excision, and salpingo-oophorectomy" in a thoracic case (Fig. 8) or "Control of bleeding" in a liver resection (Fig. 6).
    4. Such a vocabulary can be written only after all the videos have been seen. For a new operation it does not exist, whereas the dynamic formulation derives the keysteps from the video and its title alone (Fig. 1(B)).

    We agree that some steps recur across videos, port placement for example. The dynamic formulation does not deny this. It removes the requirement that every step be enumerated beforehand.

  • Separately ablate title alignment and expert-commentary alignment: Table 1 in the paper demonstrates results of seperating title alignment and expert-commentary alignment:

    • Removing expert commentary (the variant without the middle level, denoted as Δ\Delta Middle) results in [email protected]/[email protected]/[email protected] to be 0.35/0.28/0.17.
    • However, removing title is not feasible in our framework. Because without title, we have no way to split expert commentary into different scetions (the middle level). That means the expert commentary alignment will disapear together with the title removal.

We thank the reviewer again for the careful reading. We hope these answers and the added analyses resolve the concerns.

Thank you for the detailed response and clarifications. However, my main concern about the clinical validity and reliability of the DySurg dataset remains. The authors acknowledge that no additional independent re-annotation was performed, while the original keystep timestamps and descriptions were further refined. Although the annotations originate from operating surgeons and WebSurg peer review, this does not establish the reliability of the final dataset annotations. Without independent annotations from multiple clinical experts and an assessment of their agreement on keystep labels and temporal boundaries, the quality of the final annotations remains insufficiently verified. Since the dataset is a central contribution of this paper, I will maintain my original rating.

Reviewer_CnWyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes "dynamic surgical workflow reasoning": instead of segmenting a surgical video into a fixed set of phases, it generates a video-specific sequence of keysteps and their textual descriptions, conditioned on the high-level goal given by the video title. To support this task, the authors build DySurg: 470 real surgical videos from WebSurg, over 100 hours, 7 surgical categories, an average of 12.4 minutes and 6.5 keysteps per video, with time-stamped expert commentary. The MSVLR method has three components: a video parser that uses a sliding window and two-level (segment-level and frame-level) boundary discrimination to split the video into keysteps; a commentary alignment module that aligns title/commentary text representations with visual representations via cross-attention and cosine alignment; and a workflow text generation module that uses a LoRA-tuned LLaMA-3.3-70B to generate a description for each keystep from visual features and the title. Commentary is not needed at inference; only the video and title are used. On DySurg, the method reports [email protected]/0.5/0.7 = 0.40/0.32/0.17, outperforming Qwen3-VL, TimeChat, UniVTG, SurgVLP, PeskaVLP, and others; it also reports improvements over SurgVLP on StrasBypass/BernBypass.

Critical strengths
  • Problem level: the task has genuine clinical motivation, and "commentary at training time, not at inference" is a practically valuable setting. A fixed phase classifier cannot express patient-specific steps; using WebSurg teaching commentary as privileged supervision avoids relying on expert narration at inference while providing a richer semantic signal than vision alone. If done rigorously, this idea could advance surgical workflow understanding.
  • Resource level: DySurg is a promising dataset. With 470 videos, over 100 hours, 7 specialties, and time-stamped commentary plus keystep descriptions, it is substantially larger than existing surgical workflow datasets. If the license permits release and the paper provides a reliable annotation-quality analysis, the dataset itself will be valuable to the community.
Critical weaknesses
  • Task definition/evaluation level: the R@IoU matching rule needs to be specified. R@{0.3,0.5,0.7} requires a clear definition of how a predicted segment is matched to a ground-truth segment: is it purely temporal IoU, or does description semantics also participate? If matching is purely temporal, a model with correct boundaries but completely wrong descriptions can still score highly, and the task reduces to boundary detection; if semantic matching is required, the matching algorithm, thresholds, and handling of one-to-many/many-to-one cases all need to be described. In addition, the number of keysteps per video is variable, so Recall needs a defined penalty for predicting too many or too few segments, and trivial baselines (e.g., splitting the video uniformly into kk segments, or emitting a fixed template description) should be reported. Until these are specified, 0.40 is not yet sufficient to support "significantly better than existing methods." That said, this is a common and addressable gap for a new-task paper.
  • Method validation level: baseline adaptation, the caption metric, and some individual numbers need strengthening. In Table 1, ActionFormer scores only 0.02, Qwen3-VL (LoRA-tuned 70B) 0.04, and TimeChat 0.09. These numbers look more like an interface/output-format mismatch than a capability gap; a 70B VLM LoRA-tuned on 400 videos should not be close to zero at [email protected]. I would suggest describing the output format, loss, and training details of these baselines, or using baselines that are closer to temporal grounding. In Table 2, the Cos-scores are SurgVLP 0.8295, PeskaVLP 0.8319, Video-LLM 0.8280, and MSVLR 0.8362—all clustered around 0.83, with differences of 0.004–0.008 and no variance or significance test. A fixed-template caption could plausibly also reach 0.83, so the discriminative power of this metric needs to be demonstrated (e.g., add a template baseline, report variance, or supplement with expert human ratings). In addition, the three SurgVLP rows in Table 3 (StrasBypass) and Table 4 (BernBypass) are identical (0.130/0.106/0.080), and in Table 7 (∆CA), Thoracic has [email protected]=0.03 < [email protected]=0.13, which violates the monotonicity of Recall. These may be copy/paste or typesetting errors, but they affect readers' confidence in the results and should be checked. The average video length also appears as both 12.4 and 13.9 minutes, and the use of SGD with lr=1e-3 for LoRA on LLaMA-3.3-70B should be confirmed.
Questions for authors
  • Could you precisely define the R@IoU matching rule: are predictions and ground truth matched one-to-one by temporal IoU? Does description semantics participate in matching? How are cases with a different number of predicted and ground-truth segments scored? Please also report scores for two trivial baselines: uniform segmentation and fixed-template caption.
  • How were ActionFormer, Qwen3-VL, and TimeChat adapted to this task? Could you provide their output format, loss, whether they use a temporal grounding head, training epochs, and label mapping? Why does a LoRA-tuned 70B VLM score only 0.04 at [email protected]?
  • Could you explain the Thoracic [email protected]=0.03 < [email protected]=0.13 in Table 7 and the identical SurgVLP rows in Tables 3/4? Are these typos or evaluation-script issues? If the latter, please rerun all affected results.
  • The Cos-score differences in Table 2 are only 0.004–0.008. Could you report mean/variance over five seeds and a paired test, and add a fixed-template-caption baseline? If the template baseline also reaches about 0.83, would you consider supplementing this metric with expert human ratings of clinical correctness/completeness?
  • On StrasBypass/BernBypass, MSVLR is trained on the target dataset's training set and then tested, whereas SurgVLP is a pretrained model. Are these protocols comparable? In addition, [email protected] is only 0.13–0.17, far below the usual level of phase recognition in the literature; could you clarify whether the evaluation here is over 12 phases or 46 steps, and what the matching rule is?
  • Commentary alignment uses only a positive cosine loss (1−cos⁡1-\cos), with no negatives or contrastive term. Is there a risk of representation collapse? Could you report the variance of visual/text features or a retrieval metric before and after alignment?
  • The paper says an LLM partitions commentary into "the same number of segments as the visual segments." This number comes from the video parser's prediction; if the prediction is wrong, the commentary is mis-segmented and misaligned. Does training use ground-truth boundaries to segment commentary? Since commentary is not used at inference, how do alignment errors during training affect the visual representation?
  • Is the WebSurg commentary live intraoperative narration or post-edited/recorded teaching voice-over? This directly determines the strength of the claim that "commentary reflects surgeons' cognitive processes"; clarifying it in the paper would help readers interpret the setting correctly.
  • Could you quantify the "dynamic" premise: how large is the edit distance / order variation of keystep sequences across videos of the same procedure? If most keysteps still come from the platform's standard chapters, what is the difference from a fixed workflow?
  • Which is correct: 12.4 or 13.9 minutes as the average video length? What optimizer and learning rate were actually used for LoRA on LLaMA-3.3-70B?
3 comments
AuthorsOpenReview ↗

Author response to review comments

Thank you for the thorough review, the positive assessment of the task and the dataset, and for checking individual numbers. We respond to the questions below in detail.

  • R@IoU definition: Take a video with ground-truth keysteps {g1,⋯ ,gL}\{ g_1, \cdots, g_L \} and predicted segments {p1,⋯ ,pM}\{ p_1, \cdots, p_M \}, the temporal IoU of pp and gg is the length of their intersection divided by the length of their union. A keystep gg is recalled at threshold τ\tau if its best-overlapping prediction has temporal IoU of at least τ\tau. R@τ\tau is the number of recalled keysteps divided by the number of ground-truth keysteps, pooled over all test videos. For remaining sub questions:

    • One-to-one matching? Each keystep is matched to its best-overlapping prediction. MSVLR outputs a partition of the video, so for τ\tau above 0.5 the matching is one-to-one automatically: a segment cannot overlap two disjoint keysteps at more than 0.5 IoU each. At τ\tau of 0.5 or below, one long prediction can in principle recall two keysteps, so we also report strict one-to-one matching below.

    • Do description semantics take part? No, as is standard for this metric. Descriptions are scored on the temporally matched pairs: each predicted segment is paired with the ground-truth keystep it overlaps most (Sec. 3.2), and the cosine score is averaged over these pairs.

    • Different numbers of predicted and ground-truth segments? Extra segments shorten the others and lower their IoU, and missing segments leave keysteps unrecalled.

    • Trivial baseline results are presented below:

      Method[email protected][email protected][email protected]
      Uniform split0.150.130.08
      Fixed template0.120.060.05
      MSVLR (Ours)0.400.320.17
  • Adapting ActionFormer, Qwen3-VL, and TimeChat to the task: Every baseline was trained or fine-tuned on the DySurg training set and scored by the same script as MSVLR. For those baselines, the output format are time intervals (e.g., [t1,t2t_1, t_2]) together with corresponding keystep descriptions; the loss function is the same with our MSVLR (Eq. 7); training epoch is 20; all of them use a temporal grounding head.

  • LoRA-tuned 70B VLM score only 0.04 at [email protected]: The failure has a clear cause. These models are built for query-conditioned grounding or for captioning short clips. Partitioning an operation of 14 minutes on average into consecutive keysteps, in a domain absent from their pre-training, is a different problem. LoRA tuning on 400 videos does not close that gap, and model size does not confer temporal localization. ActionFormer is a class-conditional action detector, and DySurg has no recurring action classes, because every keystep is video-specific.

  • Table 7 results: The Thoracic [email protected] should be 0.23. We apology for the confusion caused by the typo. The updated paper has changed this typo. For Table 3 and 4, there is no typo.

  • StrasBypass and BernBypass results: Protocols are comparable on StrasBypass and BernBypass dataset, since we fine-tune SurgVLP on these two. Regarding remaining questions:

    • Phases or steps. Tables 3 and 4 use the 12 phases. Table 8 uses the 46 steps, on BernBypass only (App. C and D.2).
    • Matching rule. The same as in Q1. It is class-agnostic: predicted segments carry no phase label, and a ground-truth segment is one contiguous occurrence of a phase.
    • Why [email protected] is 0.13–0.17? These numbers cannot be read against phase-recognition accuracy in the literature, for two reasons. First, that accuracy is frame-wise and supervised with the twelve phase labels. Our metric is class-agnostic: the model receives no phase labels and must recover each phase occurrence as a whole segment. Second, a segment-level criterion is much stricter than a frame-level one. A method can label most frames correctly and still miss most segments at IoU 0.5 if it fragments them.
    • The ranking is consistent. MSVLR is ahead of its variant without commentary alignment on StrasBypass (0.148 against 0.129), on BernBypass (0.170 against 0.143) and on the 46 BernBypass steps (0.094 against 0.074). The general-purpose models are near zero on all three.

(to be continued)

AuthorsOpenReview ↗

Continued response

  • Commentary alignment approach: The commentary alignment approach does not have a risk of representation collapse. There are four major reasons:

    • By design. A constant output is not a minimizer of this loss. The text encoder is frozen (Sec. 3.4), so the targets are fixed vectors that differ across videos and segments. A constant output has cosine 1 with at most one of them, and its average loss stays above zero by an amount set by the spread of the targets. Collapse in positive-only objectives arises when both branches are trainable and can move toward each other. Here only the visual branch is trained, so the objective is a regression onto a fixed embedding space, as in feature distillation. The parser features that feed the alignment module are also trained with the GIoU loss (Eq. 6), which a collapsed representation cannot satisfy.

    • Experiment results. The ablation is a direct test. If the aligned representation had collapsed, the alignment loss would carry no information, and removing it would change nothing. Removing it lowers [email protected] from 0.32 to 0.19 (Table 1) and lowers the cosine score in all seven categories (Table 12).

    • The table below reports retrieval metrics before and after alignment.

      Before alignmentAfter alignment
      Video-to-title retrieval, R@1 / R@50.46/0.350.65/0.48
      Segment-to-commentary retrieval, R@1 / R@50.12/0.050.33/0.28
  • The number of segments:

    • Why the commentary is partitioned this way? The DySurg dataset has ground-truth boundaries for keysteps only. The middle level is latent: no annotation exists for intermediate sub-steps, so the commentary cannot be split at such boundaries, and it does not need to be. Partitioning it into the same number of ordered segments as the visual segments gives a one-to-one, order-preserving correspondence between two partitions of the same timeline. Corresponding segments therefore overlap in time even when individual boundaries are imperfect.
    • Errors in the predicted segments. Early in training the parser's middle-level segments are imperfect, and the alignment loss is what improves them. It pulls each visual segment toward a coherent unit of the surgeon's explanation. This is how weak supervision by alignment is meant to work, not a failure mode. The low level is supervised directly by the ground-truth keysteps through the GIoU loss (Eq. 3), which anchors the segmentation throughout.
    • Effect on the visual representation. Commentary enters only through the middle-level loss (Eq. 5 and 6). The visual encoder is frozen (Sec. 3.3), so imperfect pairs can affect only the trainable parser and alignment layers. They cannot affect inference, where no commentary is used (Sec. 3.5). The net effect is positive and measurable: with the alignment the model reaches 0.40 / 0.32 / 0.17, against 0.32 / 0.19 / 0.10 without it (Table 1).
  • WebSurg commentary: Majority of videos are recorded teaching voice-over. By "surgeons' cognitive processes" we mean the top-down reasoning that the surgeon articulates in the commentary, from the objective of the operation to its sub-steps and actions (Sec. 3.2). We do not claim to measure real-time cognition. A voice-over prepared for teaching arguably states this reasoning more completely than live narration does.

  • Quantify the "dynamic" premise: The ''dynamic'' premise is visible in the paper. Keysteps are written per video, not chosen from a list. Fig. 7 shows two laparoscopic Nissen fundoplications with seven and four keysteps. The acute cholecystitis case of Fig. 4 has keysteps for round-ligament suspension and for the dissection of inflammatory adhesions, which a standard cholecystectomy workflow does not contain. For remaining questions:

    • WebSurg has no platform-wide chapter list. The chapters of each video are written by the contributing surgeon for that operation, which is why they differ between videos of the same procedure. We refined their timestamps and wording and removed non-procedural chapters (Sec. 2).
    • The difference from a fixed workflow. A fixed workflow assigns every video to steps enumerated in advance. Here the steps are not known in advance: the model must decide their number, boundaries and content for each video. Even where two videos of a procedure share most steps, the model is not told so.

(to be continued)

AuthorsOpenReview ↗

Continued response

  • Video length: Both numbers are correct, and they describe different stages. The videos as collected average 13.9 minutes, which matches Fig. 2 and the total of over 100 hours. After our cleaning step removes non-surgical and blurred keyframes (Sec. 2), they average 12.4 minutes. The revision states both with these definitions. The LoRA adapters of LLaMA-3.3-70B were trained with AdamW optimizer and learning rate is 0.0002 with linear learning rate schedule.

  • Redistribution permission: As mentioned in the paper, we obtained consent from the WebSurg platform to use its content for research (Sec. 2). The source material is public educational content. WebSurg publishes each video after peer review, and its terms require that commentaries do not identify patients. The footage is endoscopic. Our cleaning step also removes non-surgical frames such as introductory slides (Sec. 2), which may carry case details. Narrators' audio is not part of the release. We will release everything our permission covers. Annotations, splits, video identifiers, preprocessing and evaluation code, and model weights are released openly.

We thank the reviewer again for the careful reading. We hope these answers and the added analyses resolve the concerns.

Reviewer_ZUFwOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies dynamic surgical workflow analysis through video-specific keystep identification and description generation. It introduces DySurg, a dataset of 470 surgical videos across seven categories with temporal annotations and expert commentary. The proposed MSVLR framework combines hierarchical video parsing, visual alignment with titles and commentary during training, and language-model-based description generation. At inference, it requires only a video and its title. Experiments cover DySurg, StrasBypass, and BernBypass, alongside ablations of commentary alignment and intermediate-level processing. I have the following comments.

Critical strengths
  1. A useful dataset for studying variable surgical workflows. DySurg combines 470 surgical videos across seven categories with temporal keystep annotations, textual descriptions, and expert commentary. These complementary annotations support studying both temporal localization and video-specific workflow descriptions. The dataset is the paper’s clearest contribution and provides a useful resource for further research on surgical video understanding.
Critical weaknesses
  1. The central claim of dynamic surgical reasoning is insufficiently validated. The paper motivates its approach through patient-dependent adaptations and unexpected intraoperative events, but the evaluation measures keystep localization and description similarity without directly assessing these capabilities. The DySurg split does not explicitly test held-out workflow variants, while the StrasBypass and BernBypass experiments involve training on each target dataset (Section 4.3). Consequently, the evidence leaves unclear whether the framework captures the adaptive reasoning emphasized in the paper. Evaluation on explicitly identified workflow deviations or unseen step combinations would help substantiate this central claim.

  2. The generated descriptions exhibit substantive mismatches that the evaluation does not adequately capture. In Appendix Figure 6, the liver-resection example produces descriptions involving esophageal mobilization and crural repair, whereas the reference steps concern liver resection and parenchymal transection. This raises a concrete concern about whether the descriptions are grounded in the depicted procedure. Nevertheless, description quality is evaluated only through Sentence-BERT cosine similarity, with scores of 0.8362 for MSVLR and 0.8319 for PeskaVLP (Table 2). Without systematic assessment of incorrect anatomy, actions, and unsupported content, these scores provide limited evidence for reliable workflow descriptions. Expert evaluation and quantified error analysis are needed to resolve this concern.

Questions for authors
  1. What kinds of workflow variation does the DySurg test set cover? Does it include step combinations or orders that do not appear in training? If so, how does the model perform on those cases?

  2. In Appendix Figure 6, the liver-resection example contains descriptions of esophageal mobilization and crural repair. Are these actual model predictions, or is there an error in the figure? If they are predictions, how common are such mismatches in the test set?

  3. Table 2 reports Sentence-BERT scores of 0.8362 for MSVLR and 0.8319 for PeskaVLP. Is this difference statistically reliable? Is there any expert evaluation showing that the higher score corresponds to more accurate descriptions of the anatomy and surgical actions?

4 comments
AuthorsOpenReview ↗

Author response to review comments

Thank you for the careful review and for recognising DySurg as a useful resource. We respond to each weakness and question in turn.

  • Central claim of dynamic surgical reasoning: The task in this paper is to produce, from a video and its title alone, the keysteps of that particular operation: how many there are, where they lie and what each one is (Sec. 2, Eq. 1). The model is given no step list. Every DySurg test video is a different operation with keysteps annotated for that video. A model therefore cannot score by reproducing a memorised sequence, and a fixed-workflow recogniser cannot produce the output at all. The keysteps also encode case-specific decisions, which is the "dynamic" content. In the paper, we have presented results on explicitly identified workflow deviations or unseen step combinations, which are resummarized below for better understanding:

    • The evaluation measures the task as defined. The task is to generate, from a video and its title, the keysteps of that particular operation and a description of each (Sec. 2, Eq. 1). Keystep localization and description quality against video-specific ground truth are exactly these two outputs. The model has no predefined step list to fall back on, so it cannot do well by recognising a fixed sequence.
    • The ground truth carries the adaptations. Patient-dependent adaptation is not a separate label in DySurg. It is what makes one video's keysteps differ from another's. In Fig. 4 (top), an acute cholecystitis case titled "technical variation", the first two keysteps are the suspension of the round ligament for exposure and the dissection of inflammatory adhesions. A fixed cholecystectomy ontology such as the seven phases of Cholec80 would absorb both into a generic preparation phase. MSVLR localizes both and describes them correctly. In the lobectomy of the same figure, the last keystep concerns a second middle-lobe artery, which exists only in patients with that anatomy. Recovering such keysteps is the measurable form of the adaptive behaviour that motivates the paper.
    • The test set consists of held-out workflows. The test videos are different operations from the training videos, and keysteps are annotated per video. The split therefore already tests unseen workflow variants. Fig. 7 shows how far workflows differ even within one procedure: two laparoscopic Nissen fundoplications with seven and four keysteps.
    • The bypass experiments follow the standard protocol. Training on the training split of each dataset and testing on its test split is how these benchmarks are used in prior work (Sec. 4.3). The purpose is to show that the same model also handles fixed workflows, which are a special case of dynamic ones. By "generalization" we meant this breadth across datasets and workflow types, not zero-shot transfer.
  • Evaluating generated descriptions:

    • The main-paper examples show grounded descriptions. Fig. 4 shows outputs of the evaluated model. For the cholecystectomy, all six descriptions name the same structures as the references. For the Nissen fundoplication, three of four match, including the paraphrase of "Crural closure" as "The diaphragmatic crura were approximated". The third example, a long robotic lobectomy with eight keysteps, is harder: the lymph node, artery and vessel-division steps are described correctly and the others are not. We did not select only favourable cases.
    • Why embedding similarity is the appropriate metric. The model produces descriptions of at most ten words (Fig. 3), and the references are short free-text phrases. Correct outputs often share few or no words with the reference, as the paraphrase above shows, so n-gram metrics would penalise correct answers. The measure is not saturated: TimeChat scores 0.4184 where the other methods score above 0.82 (Table 2). It also responds consistently to the model: removing commentary alignment lowers it in all seven categories (Table 12; two-sided sign test, ρ\rho = 0.016).
  • Workflow variation in the DySurg dataset: As presented in the paper, the variation is of three kinds. We reorganize them below for better understanding:

    • Across procedures within a category. The colorectal examples of Fig. 6 are a combined radiofrequency and liver resection for colorectal metastases and a transanal polyp resection. Thoracic procedures shown in Figs. 4 and 8 include a robotic middle lobectomy, a catamenial pneumothorax case and a mediastinal lymph node dissection.
    • Within one procedure. Fig. 7 shows two laparoscopic Nissen fundoplications annotated with seven and four keysteps.
    • Case-specific steps. The acute cholecystitis case of Fig. 4 includes round-ligament suspension and the dissection of inflammatory adhesions. The liver resection of Fig. 6 includes control of bleeding.

    In addition, the test set contains step combinations and orders that do not appear in training.

(to be continued)

AuthorsOpenReview ↗

Continued response

  • Fig. 6 descriptions: Those are predictions from our model. As shown in Table 12, colorectal surgery achieves score of 0.8383 on keystep description generation.

  • Table 2 results: The comparison with PeskaVLP in Table 2 is not where MSVLR's advantage lies, and the paper does not claim a large gain there. The methods differ mainly in localization: MSVLR doubles the recall of both surgical pre-trained models at IoU 0.3 (0.40 against 0.19 and 0.20, Table 1). Table 2 shows that this gain is not bought at the expense of description quality.

We thank the reviewer again for the careful reading. We hope these answers and the added analyses resolve the concerns.

I have read the authors’ response and appreciate their clarifications. I have decided to raise my rating by one point, though I still consider the paper’s novelty limited.

AuthorsOpenReview ↗

We greatly appreciate your time and constructive feedback.

We greatly appreciate your time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.