acceptodds
Withdrawn from ICLR 2027

REMEDY: How Far Is Video Generation from Medical Education World Models?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent video generation models produce realistic videos and show potential as a foundation for world models. These advances create opportunities for generating medical teaching demonstrations, which requires both convincing visual quality and precise procedural actions. However, whether current generators can meet these requirements has not been measured. To address this problem, we introduce Readiness Evaluation of Medical Education Demonstration sYnthesis (REMEDY), to our knowledge, the first benchmark for AI-generated medical teaching demonstrations. REMEDY provides 900 first frames from real demonstration videos, covering 12 tasks across four scenarios: operating room, imaging, clinic and bedside, and resuscitation. Five contemporary open-source video generation models produce 4,500 videos from these frames. We combine task-specific clinical checklists with video and motion quality metrics. Evaluation covers four dimensions: clinical action following, clinical profiles, video quality, and motion quality. Our results show that realistic appearance and temporal consistency do not ensure correct clinical actions. Even the most advanced MiniMax-H3 achieves only 28.25% on strict clinical success rate, and fine-grained clinical actions remain challenging. These findings establish a foundation and roadmap for developing future medical education world models.

Primary area:
datasets and benchmarks
Keywords:
Video Generation, Medical Education, Benchmarking, World Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UJzMOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a benchmark for evaluating existing open-source video generation models on medical video generation tasks involving clinical procedures. The benchmark covers 12 medical tasks, ranging from suturing and physical examinations to minimally invasive surgery, with a collection of 900 videos. Five open-source video generation models are evaluated using an initial frame and a text prompt. The generated videos are assessed across different dimensions, including clinical correctness, visual quality, and motion quality. Additionally, a human evaluation study is conducted to assess the quality of the generated videos.

Critical strengths
  1. The use of generative video models for creating medical educational content is an important research direction. Rare clinical scenarios and specific procedural techniques can be difficult to capture in real-world settings. Generative video models could help address this limitation, and the paper attempts to investigate their potential for medical education.

  2. The collection of 900 videos across 12 different medical tasks is commendable. The inclusion of different clinical scenarios provides a broader evaluation setting than focusing on a single procedure. If the benchmark videos and corresponding annotations are made publicly available, they could serve as a useful resource for the community.

  3. Overall, the paper is well written, with figures and illustrations clearly describing the benchmark and dataset statistics. The experimental setup and evaluation dimensions are generally easy to follow.

Critical weaknesses
  1. Limited insights from the evaluation. The findings are not entirely surprising, as previous works on surgical video generation have already shown the limitations of general-purpose video generation models in modeling domain-specific spatial and temporal dynamics [1-3]. In particular, SurgVeo [4] has shown that even with detailed and explicit prompts, general-purpose models struggle to generate clinically correct surgical actions (This is cited within this work as well). The present work extends the evaluation to different medical tasks, but it is unclear what additional insights are obtained beyond the existing literature. The paper should better distinguish their findings from those of previous benchmarks and identify whether the different medical tasks reveal new failure patterns.

  2. Choice of general-purpose video generation models. The paper evaluates general-purpose video generation models using an initial frame and text prompts containing domain-specific medical terminology. Since these models are not necessarily trained on specialized medical procedures (especially the text encoders), failures in anatomical correctness, tool motion, and temporal consistency are somewhat expected. While evaluating zero-shot capabilities is useful, it is unclear whether this setup alone provides sufficient evidence regarding the suitability of generative models for medical education. The paper should clarify the motivation for selecting only general-purpose models and discuss whether the observed failures arise from limited domain knowledge or limitations in the generation architectures themselves.

  3. Lack of domain adaptation and spatial conditioning experiments. The paper does not investigate whether domain adaptation could improve the observed performance. For example, LoRA fine-tuning could provide an initial indication of whether the failures are due to insufficient medical knowledge in the pretrained models. More importantly, recent surgical video generation methods have explored explicit spatial conditioning to improve motion and structural consistency [1,2,5]. Such conditioning is particularly relevant for medical procedures where accurate tool movements and interactions with anatomical structures are essential. Although a comprehensive fine-tuning study may be beyond the scope of this benchmark, an evaluation of at least one domain-adapted or spatially conditioned model would provide a more informative comparison.

  4. Unclear relationship between evaluation metrics and educational utility. The paper evaluates different dimensions, including clinical correctness, motion quality, and visual quality. However, the relationship between these metrics and the intended educational applications is not sufficiently established. For example, a surgical resident learning a specific instrument manipulation technique might place greater importance on correct tool motion, while anatomical correctness could be critical for learning procedural anatomy. Different educational objectives may therefore require different evaluation criteria. The paper should clarify how the proposed metrics relate to educational usefulness and whether all evaluation dimensions should be treated equally across the 12 medical tasks.

  5. Human evaluation and clinical expertise. The paper includes human evaluation, which is important for assessing generated medical videos. However, the qualifications and experience of the raters need to be clearly established. Were the raters board-certified clinicians, surgical residents, or medical students? How was their expertise matched to the different medical tasks? Since the benchmark covers a diverse range of clinical procedures, the experience of the evaluators could influence the clinical correctness scores. The paper should also clarify how disagreements between raters were handled and whether inter-rater agreement was measured.

  6. The paper introduces or uses PostAct and PostCoV, but their implementation is not clear. Are these pretrained models, newly developed metrics, or existing evaluation methods? If pretrained models are used the corresponding references should be provided. This is important for understanding the reliability and reproducibility of the reported evaluation results.

Minor comments (Not necessary for the final rating of the paper)

  1. Endora is not a benchmark paper. It introduces an endoscopic video generation method and compares it with existing GAN-based approaches. The related work section should distinguish between video generation methods and benchmark studies. More recent works on surgical video generation and evaluation should also be discussed.

  2. It would be helpful to report the parameter count and inference time of each evaluated model. Given the computational requirements of video generation and the use of H200 GPUs, a comparison of computational efficiency would make the benchmark more informative. The authors could also discuss whether techniques such as model distillation or reducing inference steps could improve accessibility. Although real-time generation may not be necessary for all educational applications, reporting these costs would help assess the practicality of deploying these models.

  3. Since the models are conditioned on both an initial frame and a text prompt, the paper should clearly distinguish this setup from purely text-to-video generation.

References:

  1. Biagini et al., HieraSurg: Hierarchy-Aware Diffusion Model for Surgical Video Generation, MICCAI 2025
  2. Venkatesh et al., Mitigating Surgical Data Imbalance with Dual-Prediction Video Diffusion Model, ICML 2026
  3. Sun et al., Bora: Biomedical Generalist Video Generation Model, 2024
  4. Chen et al., How Far Are Surgeons from Surgical World Models?, 2025
  5. Sivakumar et al., SG2VID: Scene Graphs Enable Fine-Grained Control for Video Synthesis, MICCAI 2025
Questions for authors
  1. SurgVeo has already shown that general-purpose video generation models struggle with clinically correct surgical actions despite detailed prompting. What additional insights does the proposed benchmark provide beyond these findings?

  2. Why does the evaluation focus only on general-purpose video generation models? Could the authors clarify why domain-adapted or spatially conditioned generation approaches were not considered, particularly for surgical procedures?

  3. Would fine-tuning one of the evaluated models using LoRA help distinguish failures caused by limited medical domain knowledge from those caused by the model architecture? Even a small-scale experiment on one or two tasks could provide useful evidence.

  4. How does the proposed evaluation dimensions correlate with the intended educational usefulness of the generated videos? Were the clinical raters asked to assess whether the videos were suitable for teaching or learning specific medical procedures?

  5. What were the qualifications and clinical experience of the human raters? How were raters assigned to the different medical tasks, and was inter-rater agreement evaluated?

Reviewer_nkv3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces REMEDY, a benchmark for evaluating AI-generated medical teaching videos. It contains 900 clinical cases across 12 medical tasks and evaluates 4,500 videos generated by five video generation models, based on the first frames of the reference videos.

Critical strengths

The paper proposes an interesting idea. Also its shows a common sense (through a benchmark dataset and experiments on 6 human raters): visually realistic generated videos do not necessarily demonstrate correct clinical actions.

Critical weaknesses

The paper's motivation and practical significance are sereverly under-justified. The benchmark is constructed from existing real clinical demonstrations. It remains unclear why synthetic videos are needed for medical education when real demonstrations are already available. More importantly, generated videos introduce potentially misleading procedural errors, particularly in fine-grained clinical actions (as partially demonstrated in the paper's results). What has been shown is a common sense.

Also, the evaluation largely relies on GPT-4o to assess clinical correctness. Although the authors conduct human validation, Table 8 shows substantial disagreement between human and automated ratings, particularly for biomechanics and clinical technique. These discrepancies raise concerns about the reliability of the reported clinical scores. Furthermore, visible motion similarity and checklist satisfaction cannot fully establish the correctness of precise operations.

Questions for authors

What are the occupations of the six human raters?

How long (in hours) is each training session? How many sessions in total?

Reviewer_mrxYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes REMEDY, a benchmark for AI generated medical teaching demonstractions. Covering 12 tasks and 4500 videos are made of the dataset. A set of metrics is proposed for evaluation. Experimental results on 5 opensourced models shows current video world models' performance on the proposed task settings.

Critical strengths

(1) The paper is well-written and easy to follow;

(2) The proposed metrics are covered many aspects both from video quality also domain specific quality.

(3) Human check is involved to help make the evaluation and the data more convinced.

Critical weaknesses

(1) Current analysis is only conducted with the open-sourced models, I am quite curious about the how current close sourced model perform on the proposed benchmark, like seedance. This kind of the experiments help the audience know better about the how the state-of-the-art models perform.

(2) More insight should be derivated from the results: Current results about that most models are not performing well does not bring too many insight or useful message for the development of this field. More fine-degree level analysis on the failure mode is a good idea to conduct, like

  • if the models share the same failure mode or do specific have more specific failures or some specific tasks;
  • what insight could we have from this to improve the performance of current models. etc

(3) The design of the evaluation metrics should be more motivated or more robust:

  • Some of the metrics are use the gpt-4o to get some score from the observed video; I am curious whether gpt-4o is good at this kind of task, does it have good domain knowledge to judge this? Experiments in table 9 is not sufficient to prove this as only GT videos are provided to the models for scoring. The check should be given in a dataset which contains both GT and also bad cases to check the robustness of the gpt-4o to score.
  • Some rule based metrics should also be tested like above to prove the effectiveness of its ability.
  • Why domain specific LLM agents not should be considered for metrics?
  • Since each action have the GT or GT will show the correct operations and the actions, why GT video not provided as the reference for the LLM to score? Let the model to find the difference between the generated video and the GT video is make sense.

(4) Minor: since this work is a benchmark for video generated models, more videos or visualizatin should be provided in the appendix or the supplementary material.

(5) Another related work on surgical video world model that should be discussed:

[1] SurgVista: Long-Horizon Surgical World Modeling with Plausible Instrument-Tissue Dynamics;

[2] SurgSora: Object-Aware Diffusion Model for Controllable Surgical Video Generation

Questions for authors

Refer to the Weakness part; I will raise my score if all the weakness could be addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.