- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a benchmark for evaluating existing open-source video generation models on medical video generation tasks involving clinical procedures. The benchmark covers 12 medical tasks, ranging from suturing and physical examinations to minimally invasive surgery, with a collection of 900 videos. Five open-source video generation models are evaluated using an initial frame and a text prompt. The generated videos are assessed across different dimensions, including clinical correctness, visual quality, and motion quality. Additionally, a human evaluation study is conducted to assess the quality of the generated videos.
Critical strengths
-
The use of generative video models for creating medical educational content is an important research direction. Rare clinical scenarios and specific procedural techniques can be difficult to capture in real-world settings. Generative video models could help address this limitation, and the paper attempts to investigate their potential for medical education.
-
The collection of 900 videos across 12 different medical tasks is commendable. The inclusion of different clinical scenarios provides a broader evaluation setting than focusing on a single procedure. If the benchmark videos and corresponding annotations are made publicly available, they could serve as a useful resource for the community.
-
Overall, the paper is well written, with figures and illustrations clearly describing the benchmark and dataset statistics. The experimental setup and evaluation dimensions are generally easy to follow.
Critical weaknesses
-
Limited insights from the evaluation. The findings are not entirely surprising, as previous works on surgical video generation have already shown the limitations of general-purpose video generation models in modeling domain-specific spatial and temporal dynamics [1-3]. In particular, SurgVeo [4] has shown that even with detailed and explicit prompts, general-purpose models struggle to generate clinically correct surgical actions (This is cited within this work as well). The present work extends the evaluation to different medical tasks, but it is unclear what additional insights are obtained beyond the existing literature. The paper should better distinguish their findings from those of previous benchmarks and identify whether the different medical tasks reveal new failure patterns.
-
Choice of general-purpose video generation models. The paper evaluates general-purpose video generation models using an initial frame and text prompts containing domain-specific medical terminology. Since these models are not necessarily trained on specialized medical procedures (especially the text encoders), failures in anatomical correctness, tool motion, and temporal consistency are somewhat expected. While evaluating zero-shot capabilities is useful, it is unclear whether this setup alone provides sufficient evidence regarding the suitability of generative models for medical education. The paper should clarify the motivation for selecting only general-purpose models and discuss whether the observed failures arise from limited domain knowledge or limitations in the generation architectures themselves.
-
Lack of domain adaptation and spatial conditioning experiments. The paper does not investigate whether domain adaptation could improve the observed performance. For example, LoRA fine-tuning could provide an initial indication of whether the failures are due to insufficient medical knowledge in the pretrained models. More importantly, recent surgical video generation methods have explored explicit spatial conditioning to improve motion and structural consistency [1,2,5]. Such conditioning is particularly relevant for medical procedures where accurate tool movements and interactions with anatomical structures are essential. Although a comprehensive fine-tuning study may be beyond the scope of this benchmark, an evaluation of at least one domain-adapted or spatially conditioned model would provide a more informative comparison.
-
Unclear relationship between evaluation metrics and educational utility. The paper evaluates different dimensions, including clinical correctness, motion quality, and visual quality. However, the relationship between these metrics and the intended educational applications is not sufficiently established. For example, a surgical resident learning a specific instrument manipulation technique might place greater importance on correct tool motion, while anatomical correctness could be critical for learning procedural anatomy. Different educational objectives may therefore require different evaluation criteria. The paper should clarify how the proposed metrics relate to educational usefulness and whether all evaluation dimensions should be treated equally across the 12 medical tasks.
-
Human evaluation and clinical expertise. The paper includes human evaluation, which is important for assessing generated medical videos. However, the qualifications and experience of the raters need to be clearly established. Were the raters board-certified clinicians, surgical residents, or medical students? How was their expertise matched to the different medical tasks? Since the benchmark covers a diverse range of clinical procedures, the experience of the evaluators could influence the clinical correctness scores. The paper should also clarify how disagreements between raters were handled and whether inter-rater agreement was measured.
-
The paper introduces or uses PostAct and PostCoV, but their implementation is not clear. Are these pretrained models, newly developed metrics, or existing evaluation methods? If pretrained models are used the corresponding references should be provided. This is important for understanding the reliability and reproducibility of the reported evaluation results.
Minor comments (Not necessary for the final rating of the paper)
-
Endora is not a benchmark paper. It introduces an endoscopic video generation method and compares it with existing GAN-based approaches. The related work section should distinguish between video generation methods and benchmark studies. More recent works on surgical video generation and evaluation should also be discussed.
-
It would be helpful to report the parameter count and inference time of each evaluated model. Given the computational requirements of video generation and the use of H200 GPUs, a comparison of computational efficiency would make the benchmark more informative. The authors could also discuss whether techniques such as model distillation or reducing inference steps could improve accessibility. Although real-time generation may not be necessary for all educational applications, reporting these costs would help assess the practicality of deploying these models.
-
Since the models are conditioned on both an initial frame and a text prompt, the paper should clearly distinguish this setup from purely text-to-video generation.
References:
- Biagini et al., HieraSurg: Hierarchy-Aware Diffusion Model for Surgical Video Generation, MICCAI 2025
- Venkatesh et al., Mitigating Surgical Data Imbalance with Dual-Prediction Video Diffusion Model, ICML 2026
- Sun et al., Bora: Biomedical Generalist Video Generation Model, 2024
- Chen et al., How Far Are Surgeons from Surgical World Models?, 2025
- Sivakumar et al., SG2VID: Scene Graphs Enable Fine-Grained Control for Video Synthesis, MICCAI 2025
Questions for authors
-
SurgVeo has already shown that general-purpose video generation models struggle with clinically correct surgical actions despite detailed prompting. What additional insights does the proposed benchmark provide beyond these findings?
-
Why does the evaluation focus only on general-purpose video generation models? Could the authors clarify why domain-adapted or spatially conditioned generation approaches were not considered, particularly for surgical procedures?
-
Would fine-tuning one of the evaluated models using LoRA help distinguish failures caused by limited medical domain knowledge from those caused by the model architecture? Even a small-scale experiment on one or two tasks could provide useful evidence.
-
How does the proposed evaluation dimensions correlate with the intended educational usefulness of the generated videos? Were the clinical raters asked to assess whether the videos were suitable for teaching or learning specific medical procedures?
-
What were the qualifications and clinical experience of the human raters? How were raters assigned to the different medical tasks, and was inter-rater agreement evaluated?