- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces a surgical video generation framework that learns temporal motion dynamics from optical flow using latent action modeling. During inference, surgical videos are generated using only text prompts, without requiring additional conditioning signals. The paper also introduces a set of metrics to evaluate the quality of generated videos across different aspects. A dataset of 30K surgical action clips, SurgClip-30K, is curated for this study. General video generation models, including Wan, CogVideo, and HunyuanVideo, are adapted as baselines. The proposed method reports improvements across the evaluation metrics on the curated dataset.
Critical strengths
-
The paper uses optical flow to learn motion dynamics from surgical videos and adds this information into a text-conditioned video generation framework. This is an interesting direction, particularly for learning surgical motion without requiring additional conditioning signals during inference. This is innovative although similar approaches have been explored in surgical video generation, such as SurgiFlowVid [1] and SurgSora [2], and in general video generation, such as VideoJAM [3].
-
The collection and curation of SurgClip-30K is appreciated. A dataset containing 30K surgical action clips with corresponding text descriptions could be a useful contribution to surgical video generation research. If the dataset and its annotations are made publicly available, this would benefit future research and improve reproducibility.
Critical weaknesses
-
Lack of downstream task evaluation. One of the main motivations for surgical video generation is its application to surgical education and downstream tasks. However, the paper does not evaluate the generated videos on any downstream tasks. There is already considerable research on surgical video generation [1,2,4,5], and an important challenge is showing the practical utility of such models. The proposed evaluation metrics measure different aspects of video quality, but their relationship to downstream performance remains unclear. An evaluation on tasks such as surgical action recognition, segmentation, instrument tracking, or rare event recognition would help establish whether the generated videos are useful beyond improving generation metrics.
-
Justification for the limitations of downstream metrics. I have to disagree with the statement in Section 2.3 that metrics such as F1-score and Jaccard index are fundamentally unsuitable for evaluating generated surgical videos. Although these metrics do not directly measure perceptual video quality, they can provide useful evidence of downstream utility. For example, errors in tool motion, temporal consistency, or tool-tissue interactions may affect the performance of models trained on generated videos . Previous surgical video generation studies have already shown the use of downstream tasks for evaluation [1,4]. The paper should provide stronger justification for dismissing these metrics and explain how the proposed evaluation metrics relate to practical surgical applications.
-
The proposed flow action decoder is used to learn surgical motion dynamics. However, it is unclear how this differs from existing approaches such as SurgiFlowVid and SurgSora, which also incorporate optical flow cues for motion modeling. The specific technical contribution beyond existing flow-based approaches should be explained more clearly. Furthermore, the paper relies on RAFT for optical flow estimation, but does not discuss whether the performance depends on the choice of flow estimator. An ablation using alternative approaches, such as FlowFormer, would help determine the robustness of the proposed motion representation.
-
Missing comparisons with surgical video generation models. The experimental evaluation primarily compares against general video generation models. However, surgical video generation approaches such as HierSurg and spatial-conditioned models such as Cosmos-H-Predict/Transfer are relevant to the discussion of controllable generation. Although these approaches may require additional conditioning signals, they provide useful references for evaluating motion consistency and surgical scene generation. The paper argues that obtaining auxiliary signals is difficult at scale, but existing pipelines can extract conditioning signals using pretrained models such as SAM. A comparison with relevant surgical generation methods, or a discussion of the trade-offs in annotation cost, scalability, and generation quality, would strengthen the claims regarding the advantages of text-only generation.
-
Limited evaluation of text-only control and tool-tissue interactions. Text prompts provide less explicit spatial information than structural conditioning signals such as semantic maps or instrument trajectories [6]. This is particularly important in surgery, where the correctness of tool motion and tool-tissue interactions is essential. While the proposed method learns motion from optical flow, it is unclear whether the generated videos preserve these interactions under different text prompts. The evaluation would benefit from experiments measuring the consistency of instrument motion, tool-tissue interactions, and controllability. Given the curated dataset, an additional study on imbalanced surgical tasks, such as action recognition in Cholec datasets or instrument recognition in PhaKIR and SurgVU24, could also help show the practical benefit of the generated videos.
-
Reliance on auxiliary models during dataset curation. The paper motivates text-only generation by arguing that collecting auxiliary conditioning signals is difficult for large-scale surgical datasets. However, Section 3.2 uses a YOLO detector and Hulu-Med models to curate action clips and generate text descriptions. While this is different from requiring auxiliary signals during inference, there remains considerable effort i curating the dataset, which can be compared to the effort needed to collect auxillary signals. Given the growth of foundational models in surgery, obtaining such signals have become simpler as before. The paper should inlcude a discussion on this aspect to clearly justify the need for not collecting auxillary signals from pre-trained mdoels.
Minor comment: (Not necessary for the final rating)
- References to the supplementary material should include the corresponding section numbers. This would make it easier for reviewers to locate the relevant experiments and additional details.
References:
- Venkatesh et al., Mitigating Surgical Data Imbalance with Dual-Prediction Video Diffusion Model, ICML 2026
- Chen et al., SurgSora: Object-Aware Diffusion Model for Controllable Surgical Video Generation, MICCAI 2025
- Chefer et al., VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models, ICML 2025
- Sivakumar et al., SG2VID: Scene Graphs Enable Fine-Grained Control for Video Synthesis, MICCAI 2025
- Biagini et al., HieraSurg: Hierarchy-Aware Diffusion Model for Surgical Video Generation, MICCAI 2025
- Sander Dielemann, Wading through the noise: an intuitive look at diffusion models, ICML 2024 talk
Questions for authors
- Could the authors provide evidence that the generated videos improve downstream surgical tasks, such as action recognition, segmentation, or instrument detection? Even a limited data augmentation experiment would help establish the practical utility of the proposed approach.
- What evidence supports the statement in Section 2.3 that F1-score and Jaccard index are fundamentally unsuitable for evaluating generated surgical videos? Have the authors investigated whether the proposed video quality metrics correlate with downstream task performance?
- How was the accuracy of the YOLO and Hulu-Med models assessed during the curation of SurgClip-30K? Were any manual verification procedures performed to identify incorrect action clips or text descriptions?
- What is the main technical difference between the proposed flow action decoder and existing optical flow-based surgical video generation approaches?
- How does the proposed text-only approach compare with surgical video generation methods that use explicit action or structural conditioning? Can the authors provide qualitative or quantitative evidence that tool motion and tool-tissue interactions remain consistent with the text prompts?
3 comments
Response to Reviewer NjBa (2)
W4. Comparison with conditioned surgical video generation models
We thank the reviewer for the suggestion and agree that a discussion of the trade-offs would strengthen our claims. We clarify that a direct quantitative comparison with these methods is difficult for two reasons.
- Most surgical video generation models are not open-sourced. Among the methods below, only Cosmos-H-Surgical has released its weights, and it is a 15.2B-parameter model validated on 80 GB H100 GPUs with control videos as input. Without released checkpoints or the same control videos, these methods cannot be re-run or re-evaluated with SurgMetrics under our benchmark. The model differences in inputs and whether the model is open-sourced or not are listed in the table below:
| Method | Text | Reference frame | Mask / depth / edge | Trajectory / scene graph | Open-source weights |
|---|---|---|---|---|---|
| SurgSora | – | required | – | required (motion cues) | no |
| HieraSurg | phase + triplet labels | required | required (first-frame segmentation) | – | no |
| SG2VID | – | optional | (used to build graphs) | required (scene-graph sequence) | no |
| SurgiFlowVid | required | optional | optional (sparse masks) | – | no |
| Cosmos-H-Surgical-Transfer | requireed | – | required (control video) | – | yes |
| FLAIR (ours) | required | – | – | – | yes |
- Different inference inputs naturally lead to information asymmetry. Structurally conditioned methods receive additional per-sample information at inference, such as a real reference frame, segmentation maps, motion trajectories or scene graphs, which already encode a large part of the target scene layout and motion. While the goal of FLAIR is to make full use of the information within text alone to construct surgical videos, therefore comparing the two settings does not reflect the same task, and it is not possible to construct a fair comparison.
Together, FLAIR targets a different and complementary setting, and its adapter can be combined with structural controls when they are available.
W5 & Q5. Text-only control and tool–tissue interactions
We thank the reviewer for pointing out the importance of verifying tool motion and tool–tissue interaction under text-only control. We provide three complementary pieces of evidence.
(i) Expert action correctness. In our human evaluation, four specialist surgeons independently rated whether the "instrument–action–target" interaction specified by each prompt was correctly executed. FLAIR receives lower interaction-error ratings than LoRA on all three backbones, and the human alignment section in our paper further demonstrates this argument.
(ii) Presence of the prompted instrument. We perform an extra experiment by running a laparoscopic instrument detector on all benchmark generations and comparing FLAIR and LoRA on identical prompts.
| Backbone | Prompts | LoRA | FLAIR | Δ [95% CI] | p |
|---|---|---|---|---|---|
| Wan2.1 | 200 | 0.860 | 0.917 | +0.057 [+0.030, +0.084] | <0.001 |
| Hunyuan-1.5 | 200 | 0.476 | 0.677 | +0.201 [+0.113, +0.286] | <0.001 |
| CogVideoX-2B | 200 | 0.403 | 0.654 | +0.251 [+0.193, +0.311] | <0.001 |
The results show that FLAIR keeps the prompted instrument visible in significantly more frames than LoRA on all three backbones.
W6 & Q3. Reliance on auxiliary models during dataset curation
We agree that dataset curation requires considerable effort. We clarify that this is a one-time, offline cost. We pay it once and release SurgActionClip-30K together with its captions, so users do not need to repeat it. In contrast, inference-time structural conditions must be provided for every generated sample, and they require an existing video to extract from (W4).
Regarding the accuracy of the auxiliary models, the YOLO detector is applied to laparoscopic sources whose geometry matches its training data (e.g., PitVis). On PhaKIR, which is disjoint from the detector's training data, it achieves precision 1.00 / recall 0.69 for clip-level instrument presence (n = 600), so its errors lead to missed instruments rather than hallucinated ones. We also check all Hulu-Med captions against the source annotations:
| Source | Annotated instrument mentioned | Unlisted instrument mentioned |
|---|---|---|
| CholecT50 | 96.7% | – |
| PhaKIR | 100% | 1.1% |
| SurgVU24 | 97.9% | 0.2% |
| PitVis | 82.8% | 0.8% |
We thank Reviewer NjBa again for the constructive review. We hope the above provides clear clarifications for FLAIR.
Comment by reviewer NjBa
I thank the authors for providing a detailed rebuttal and conducting additional experiments to provide clarifications to the raised concerns. I agree with the comments on W2, W5 and W6. The following concerns remain:
-
The results for downstream task looks interesting. I do not understand why non-cholec minority class samples are added to CholecT50 dataset. The idea for the downstream task is to add generated minority class samples to the existing data distribution (which is Cholec) and train a classifier (e.g. X3D, MViT) without using class balancing to measure the effect of the generated samples. It is not a surprising result to see VideoMAE-B performance being so low on real data as the model is not trained using surgical data. The comparison needs to be consistent on what is being measured.
-
According to the authors answer to W3, both surgsora and surgiflowvid already use optical flow fields for conditional generation (pointed out as a novelty imitation by reviewer WyWz). Distilling the flow cues as a text driven generative model cannot be justified without a comparison against these models. It is acceptable that surgiflowvid is not open sourced, but surgsora is available to run experiments.
-
Regarding the comparison with surgical video models, open weights does not have anything to do with comparison against surgical video models. Similar in spirit to how the authors performed LoRa finetuning of different architectures, such experiments are needed for a fair comparison for the surgical video models. Simply, taking the open source weights and inferencing on new dataset does not qualify for a fair comparison. If a set of unified metrics are being proposed with SurgMetrics on new curated dataset, the existing surgical video models with a open source codebase can be fine-tuned for a fair comparison.
Given the current state, I will remain with my initial score for the paper.
Response to Reviewer NjBa
We thank Reviewer NjBa for the review, providing technically grounded feedback. FLAIR learns motion priors from video data and predicts latent actions to improve the scalability of surgical video generation, with only the need for a text prompt without additional visual inputs. We also appreciate the reviewer’s concerns regarding the lack of downstream evaluation, insufficient experiments, and reliance on auxiliary models during dataset creation, with clarification below:
W1 & Q1. Downstream task evaluation
We agree that downstream utility is one of the main motivations of surgical video generation. We perform an extra limited data augmentation experiment using only the existing 200-prompt generations, without any new sampling. We construct an imbalanced five-class CholecT50 action recognition task: the two majority classes (retraction, dissection) keep all real training clips, while each minority class (aspiration, clipping, coagulation) keeps only n ∈ {5, 10, 20} real clips. We then add the generated minority-class videos (~18 clips per action, mostly from non-cholecystectomy procedures) of each model, and train a class-balanced linear probe on frozen VideoMAE-B features. Evaluation uses 5-fold case-grouped cross-validation on real clips, averaged over 5 seeds.
Minority-class recall on held-out real cases (LoRA / FLAIR).
We report the experimental results: used as minority-class training data, FLAIR-generated videos are more useful than LoRA-generated videos in 8 of 9 settings. We also clarify honestly that a portion of FLAIR's data augmentation effectiveness also stems from fine-tuning of LoRA. However, FLAIR's precision in motion gives it a greater advantage over LoRA, especially when n is relatively small.
W2 & Q2. Limitations of downstream metrics
We agree that our statement in Section 2.3 is too strong. We clarify that our intention was to point out that F1-score and Jaccard index do not directly measure the perceptual or surgical plausibility of a generated video, rather than to dismiss them. As the reviewer suggested, they provide complementary evidence of downstream utility when the generated videos are used to train recognition models, which is exactly the setting of our augmentation experiment in W1. We apologize for the inaccurate wording and will revise Section 2.3 to wording like: "These metrics do not directly measure the perceptual or surgical plausibility of generated videos. "
W3 & Q4. Difference from flow-based methods and choice of flow estimator
We clarify that SurgSora uses optical flow as an inference-time control derived from user-specified motion cues on an input frame, and SurgiFlowVid jointly denoises flow with RGB inside a fine-tuned generator. FLAIR instead distills flow into an explicit latent action space (FAE), predicts it from text alone, and injects it into a frozen backbone through a lightweight adapter, which transfers across three different backbones without modifying their weights. We will add SurgiFlowVid and VideoJAM to the related work.
Regarding the flow estimator, we perform an extra robustness experiment by freezing the FAE and replacing only RAFT with other estimators on 348 held-out CholecT50 clips:
Although the flow fields differ (EPE 0.39–0.84 px), the FAE latents still retrieve the same clip with a high accuracy of 97.4% (FlowFormer) and 95.4% (SEA-RAFT) top-1 accuracy. This illustrates that FAE is learning the motion itself instead of the optical patterns within the RAFT.