acceptodds
Under review as a conference paper at ICLR 2027

Beyond Masks and Trajectories: Flow-Guided Latent Action Injection for Stable Surgical Video Generation

Abstract

Surgical video generation holds substantial potential for surgical education, simulation, and data augmentation, yet generating surgical videos with realistic and clinically plausible motion remains challenging. Most existing methods rely on auxiliary conditions, such as masks, trajectories, depth, or reference videos, to achieve visually plausible synthesis. Yet, these auxiliary conditions typically require additional manual annotation or specialized acquisition, making it difficult to scale such methods beyond small, curated datasets. This motivates the need for a reference-free architecture capable of generating high-quality surgical video without requiring auxiliary visual conditions at inference time. We propose FLAIR, a Flow-guided Latent Action Injection framework for Reference-free surgical video generation. FLAIR learns action priors from optical flow of real surgical videos, dynamically predicts corresponding latent action representation from an input prompt, and injects it into a frozen base model to generate surgical videos with improved action consistency. We further construct SurgActionClip-30K, the first large-scale surgical vision dataset comprising action-centric segmented clips and structured caption labels, addressing the persistent lack of fine-grained, action-centric surgical datasets. Lastly, we introduce SurgMetrics, the first surgical domain-specific evaluation metrics for quantifying the quality of generated surgical videos, addressing the persistent absence of clinically grounded evaluation standards in this domain. Extensive experiments demonstrate that FLAIR enables generating high-quality surgical videos using text-only inference without auxiliary conditions, and validation in SurgMetrics demonstrates its strength in alignment with human perception compared to traditional metrics.

Primary area:
generative models
Keywords:
Surgical Video Generation, Surgical Vision Dataset, Surgical Video Evaluation, Controllable Video Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NjBaOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a surgical video generation framework that learns temporal motion dynamics from optical flow using latent action modeling. During inference, surgical videos are generated using only text prompts, without requiring additional conditioning signals. The paper also introduces a set of metrics to evaluate the quality of generated videos across different aspects. A dataset of 30K surgical action clips, SurgClip-30K, is curated for this study. General video generation models, including Wan, CogVideo, and HunyuanVideo, are adapted as baselines. The proposed method reports improvements across the evaluation metrics on the curated dataset.

Critical strengths
  1. The paper uses optical flow to learn motion dynamics from surgical videos and adds this information into a text-conditioned video generation framework. This is an interesting direction, particularly for learning surgical motion without requiring additional conditioning signals during inference. This is innovative although similar approaches have been explored in surgical video generation, such as SurgiFlowVid [1] and SurgSora [2], and in general video generation, such as VideoJAM [3].

  2. The collection and curation of SurgClip-30K is appreciated. A dataset containing 30K surgical action clips with corresponding text descriptions could be a useful contribution to surgical video generation research. If the dataset and its annotations are made publicly available, this would benefit future research and improve reproducibility.

Critical weaknesses
  1. Lack of downstream task evaluation. One of the main motivations for surgical video generation is its application to surgical education and downstream tasks. However, the paper does not evaluate the generated videos on any downstream tasks. There is already considerable research on surgical video generation [1,2,4,5], and an important challenge is showing the practical utility of such models. The proposed evaluation metrics measure different aspects of video quality, but their relationship to downstream performance remains unclear. An evaluation on tasks such as surgical action recognition, segmentation, instrument tracking, or rare event recognition would help establish whether the generated videos are useful beyond improving generation metrics.

  2. Justification for the limitations of downstream metrics. I have to disagree with the statement in Section 2.3 that metrics such as F1-score and Jaccard index are fundamentally unsuitable for evaluating generated surgical videos. Although these metrics do not directly measure perceptual video quality, they can provide useful evidence of downstream utility. For example, errors in tool motion, temporal consistency, or tool-tissue interactions may affect the performance of models trained on generated videos . Previous surgical video generation studies have already shown the use of downstream tasks for evaluation [1,4]. The paper should provide stronger justification for dismissing these metrics and explain how the proposed evaluation metrics relate to practical surgical applications.

  3. The proposed flow action decoder is used to learn surgical motion dynamics. However, it is unclear how this differs from existing approaches such as SurgiFlowVid and SurgSora, which also incorporate optical flow cues for motion modeling. The specific technical contribution beyond existing flow-based approaches should be explained more clearly. Furthermore, the paper relies on RAFT for optical flow estimation, but does not discuss whether the performance depends on the choice of flow estimator. An ablation using alternative approaches, such as FlowFormer, would help determine the robustness of the proposed motion representation.

  4. Missing comparisons with surgical video generation models. The experimental evaluation primarily compares against general video generation models. However, surgical video generation approaches such as HierSurg and spatial-conditioned models such as Cosmos-H-Predict/Transfer are relevant to the discussion of controllable generation. Although these approaches may require additional conditioning signals, they provide useful references for evaluating motion consistency and surgical scene generation. The paper argues that obtaining auxiliary signals is difficult at scale, but existing pipelines can extract conditioning signals using pretrained models such as SAM. A comparison with relevant surgical generation methods, or a discussion of the trade-offs in annotation cost, scalability, and generation quality, would strengthen the claims regarding the advantages of text-only generation.

  5. Limited evaluation of text-only control and tool-tissue interactions. Text prompts provide less explicit spatial information than structural conditioning signals such as semantic maps or instrument trajectories [6]. This is particularly important in surgery, where the correctness of tool motion and tool-tissue interactions is essential. While the proposed method learns motion from optical flow, it is unclear whether the generated videos preserve these interactions under different text prompts. The evaluation would benefit from experiments measuring the consistency of instrument motion, tool-tissue interactions, and controllability. Given the curated dataset, an additional study on imbalanced surgical tasks, such as action recognition in Cholec datasets or instrument recognition in PhaKIR and SurgVU24, could also help show the practical benefit of the generated videos.

  6. Reliance on auxiliary models during dataset curation. The paper motivates text-only generation by arguing that collecting auxiliary conditioning signals is difficult for large-scale surgical datasets. However, Section 3.2 uses a YOLO detector and Hulu-Med models to curate action clips and generate text descriptions. While this is different from requiring auxiliary signals during inference, there remains considerable effort i curating the dataset, which can be compared to the effort needed to collect auxillary signals. Given the growth of foundational models in surgery, obtaining such signals have become simpler as before. The paper should inlcude a discussion on this aspect to clearly justify the need for not collecting auxillary signals from pre-trained mdoels.

Minor comment: (Not necessary for the final rating)

  1. References to the supplementary material should include the corresponding section numbers. This would make it easier for reviewers to locate the relevant experiments and additional details.

References:

  1. Venkatesh et al., Mitigating Surgical Data Imbalance with Dual-Prediction Video Diffusion Model, ICML 2026
  2. Chen et al., SurgSora: Object-Aware Diffusion Model for Controllable Surgical Video Generation, MICCAI 2025
  3. Chefer et al., VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models, ICML 2025
  4. Sivakumar et al., SG2VID: Scene Graphs Enable Fine-Grained Control for Video Synthesis, MICCAI 2025
  5. Biagini et al., HieraSurg: Hierarchy-Aware Diffusion Model for Surgical Video Generation, MICCAI 2025
  6. Sander Dielemann, Wading through the noise: an intuitive look at diffusion models, ICML 2024 talk
Questions for authors
  1. Could the authors provide evidence that the generated videos improve downstream surgical tasks, such as action recognition, segmentation, or instrument detection? Even a limited data augmentation experiment would help establish the practical utility of the proposed approach.
  2. What evidence supports the statement in Section 2.3 that F1-score and Jaccard index are fundamentally unsuitable for evaluating generated surgical videos? Have the authors investigated whether the proposed video quality metrics correlate with downstream task performance?
  3. How was the accuracy of the YOLO and Hulu-Med models assessed during the curation of SurgClip-30K? Were any manual verification procedures performed to identify incorrect action clips or text descriptions?
  4. What is the main technical difference between the proposed flow action decoder and existing optical flow-based surgical video generation approaches?
  5. How does the proposed text-only approach compare with surgical video generation methods that use explicit action or structural conditioning? Can the authors provide qualitative or quantitative evidence that tool motion and tool-tissue interactions remain consistent with the text prompts?
3 comments
AuthorsOpenReview ↗

Response to Reviewer NjBa

We thank Reviewer NjBa for the review, providing technically grounded feedback. FLAIR learns motion priors from video data and predicts latent actions to improve the scalability of surgical video generation, with only the need for a text prompt without additional visual inputs. We also appreciate the reviewer’s concerns regarding the lack of downstream evaluation, insufficient experiments, and reliance on auxiliary models during dataset creation, with clarification below:

 

W1 & Q1. Downstream task evaluation

We agree that downstream utility is one of the main motivations of surgical video generation. We perform an extra limited data augmentation experiment using only the existing 200-prompt generations, without any new sampling. We construct an imbalanced five-class CholecT50 action recognition task: the two majority classes (retraction, dissection) keep all real training clips, while each minority class (aspiration, clipping, coagulation) keeps only n ∈ {5, 10, 20} real clips. We then add the generated minority-class videos (~18 clips per action, mostly from non-cholecystectomy procedures) of each model, and train a class-balanced linear probe on frozen VideoMAE-B features. Evaluation uses 5-fold case-grouped cross-validation on real clips, averaged over 5 seeds.

Minority-class recall on held-out real cases (LoRA / FLAIR).

Backbonen = 5n = 10n = 20
Wan2.133.00 / 38.4741.03 / 40.9850.64 / 55.59
Hunyuan-1.532.50 / 38.0135.61 / 39.7954.53 / 56.25
CogVideoX-2B32.49 / 37.2334.70 / 37.4353.80 / 56.52
Real data only13.8929.1447.67

We report the experimental results: used as minority-class training data, FLAIR-generated videos are more useful than LoRA-generated videos in 8 of 9 settings. We also clarify honestly that a portion of FLAIR's data augmentation effectiveness also stems from fine-tuning of LoRA. However, FLAIR's precision in motion gives it a greater advantage over LoRA, especially when n is relatively small.

 

W2 & Q2. Limitations of downstream metrics

We agree that our statement in Section 2.3 is too strong. We clarify that our intention was to point out that F1-score and Jaccard index do not directly measure the perceptual or surgical plausibility of a generated video, rather than to dismiss them. As the reviewer suggested, they provide complementary evidence of downstream utility when the generated videos are used to train recognition models, which is exactly the setting of our augmentation experiment in W1. We apologize for the inaccurate wording and will revise Section 2.3 to wording like: "These metrics do not directly measure the perceptual or surgical plausibility of generated videos. "

 

W3 & Q4. Difference from flow-based methods and choice of flow estimator

We clarify that SurgSora uses optical flow as an inference-time control derived from user-specified motion cues on an input frame, and SurgiFlowVid jointly denoises flow with RGB inside a fine-tuned generator. FLAIR instead distills flow into an explicit latent action space (FAE), predicts it from text alone, and injects it into a frozen backbone through a lightweight adapter, which transfers across three different backbones without modifying their weights. We will add SurgiFlowVid and VideoJAM to the related work.

Regarding the flow estimator, we perform an extra robustness experiment by freezing the FAE and replacing only RAFT with other estimators on 348 held-out CholecT50 clips:

Flow estimatorFlow EPE vs. RAFT (px)Latent cosine vs. RAFTClip retrieval top-1
RAFT-large (paper)01.000100%
FlowFormer0.390.59197.4%
SEA-RAFT-M0.480.53095.4%
RAFT-small0.500.48287.9%
Farnebäck (classical)0.840.41082.5%
Reference level–0.122 (other transition)0.29% (chance)

Although the flow fields differ (EPE 0.39–0.84 px), the FAE latents still retrieve the same clip with a high accuracy of 97.4% (FlowFormer) and 95.4% (SEA-RAFT) top-1 accuracy. This illustrates that FAE is learning the motion itself instead of the optical patterns within the RAFT.

AuthorsOpenReview ↗

Response to Reviewer NjBa (2)

W4. Comparison with conditioned surgical video generation models

We thank the reviewer for the suggestion and agree that a discussion of the trade-offs would strengthen our claims. We clarify that a direct quantitative comparison with these methods is difficult for two reasons.

  1. Most surgical video generation models are not open-sourced. Among the methods below, only Cosmos-H-Surgical has released its weights, and it is a 15.2B-parameter model validated on 80 GB H100 GPUs with control videos as input. Without released checkpoints or the same control videos, these methods cannot be re-run or re-evaluated with SurgMetrics under our benchmark. The model differences in inputs and whether the model is open-sourced or not are listed in the table below:
MethodTextReference frameMask / depth / edgeTrajectory / scene graphOpen-source weights
SurgSora–required–required (motion cues)no
HieraSurgphase + triplet labelsrequiredrequired (first-frame segmentation)–no
SG2VID–optional(used to build graphs)required (scene-graph sequence)no
SurgiFlowVidrequiredoptionaloptional (sparse masks)–no
Cosmos-H-Surgical-Transferrequireed–required (control video)–yes
FLAIR (ours)required–––yes
  1. Different inference inputs naturally lead to information asymmetry. Structurally conditioned methods receive additional per-sample information at inference, such as a real reference frame, segmentation maps, motion trajectories or scene graphs, which already encode a large part of the target scene layout and motion. While the goal of FLAIR is to make full use of the information within text alone to construct surgical videos, therefore comparing the two settings does not reflect the same task, and it is not possible to construct a fair comparison.

Together, FLAIR targets a different and complementary setting, and its adapter can be combined with structural controls when they are available.

 

W5 & Q5. Text-only control and tool–tissue interactions

We thank the reviewer for pointing out the importance of verifying tool motion and tool–tissue interaction under text-only control. We provide three complementary pieces of evidence.

(i) Expert action correctness. In our human evaluation, four specialist surgeons independently rated whether the "instrument–action–target" interaction specified by each prompt was correctly executed. FLAIR receives lower interaction-error ratings than LoRA on all three backbones, and the human alignment section in our paper further demonstrates this argument.

(ii) Presence of the prompted instrument. We perform an extra experiment by running a laparoscopic instrument detector on all benchmark generations and comparing FLAIR and LoRA on identical prompts.

BackbonePromptsLoRAFLAIRΔ [95% CI]p
Wan2.12000.8600.917+0.057 [+0.030, +0.084]<0.001
Hunyuan-1.52000.4760.677+0.201 [+0.113, +0.286]<0.001
CogVideoX-2B2000.4030.654+0.251 [+0.193, +0.311]<0.001

The results show that FLAIR keeps the prompted instrument visible in significantly more frames than LoRA on all three backbones.

 

W6 & Q3. Reliance on auxiliary models during dataset curation

We agree that dataset curation requires considerable effort. We clarify that this is a one-time, offline cost. We pay it once and release SurgActionClip-30K together with its captions, so users do not need to repeat it. In contrast, inference-time structural conditions must be provided for every generated sample, and they require an existing video to extract from (W4).

Regarding the accuracy of the auxiliary models, the YOLO detector is applied to laparoscopic sources whose geometry matches its training data (e.g., PitVis). On PhaKIR, which is disjoint from the detector's training data, it achieves precision 1.00 / recall 0.69 for clip-level instrument presence (n = 600), so its errors lead to missed instruments rather than hallucinated ones. We also check all Hulu-Med captions against the source annotations:

SourceAnnotated instrument mentionedUnlisted instrument mentioned
CholecT5096.7%–
PhaKIR100%1.1%
SurgVU2497.9%0.2%
PitVis82.8%0.8%

We thank Reviewer NjBa again for the constructive review. We hope the above provides clear clarifications for FLAIR.

Reviewer_NjBaOpenReview ↗

Comment by reviewer NjBa

I thank the authors for providing a detailed rebuttal and conducting additional experiments to provide clarifications to the raised concerns. I agree with the comments on W2, W5 and W6. The following concerns remain:

  1. The results for downstream task looks interesting. I do not understand why non-cholec minority class samples are added to CholecT50 dataset. The idea for the downstream task is to add generated minority class samples to the existing data distribution (which is Cholec) and train a classifier (e.g. X3D, MViT) without using class balancing to measure the effect of the generated samples. It is not a surprising result to see VideoMAE-B performance being so low on real data as the model is not trained using surgical data. The comparison needs to be consistent on what is being measured.

  2. According to the authors answer to W3, both surgsora and surgiflowvid already use optical flow fields for conditional generation (pointed out as a novelty imitation by reviewer WyWz). Distilling the flow cues as a text driven generative model cannot be justified without a comparison against these models. It is acceptable that surgiflowvid is not open sourced, but surgsora is available to run experiments.

  3. Regarding the comparison with surgical video models, open weights does not have anything to do with comparison against surgical video models. Similar in spirit to how the authors performed LoRa finetuning of different architectures, such experiments are needed for a fair comparison for the surgical video models. Simply, taking the open source weights and inferencing on new dataset does not qualify for a fair comparison. If a set of unified metrics are being proposed with SurgMetrics on new curated dataset, the existing surgical video models with a open source codebase can be fine-tuned for a fair comparison.

Given the current state, I will remain with my initial score for the paper.

Reviewer_WyYzOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces FLAIR for reference-free surgical video generation, which learns from the optical flow prior for action consistency improvement. This paper also introduces a new SurgActionClip-30k dataset with caption labels. Additionally, this paper provides new metrics SurgMetrics for surgical videos.

Critical strengths
  1. The motivation is clear. It's true that the previous surgical video generation pipelines that depend on masks, trajectories, depth, or reference videos at inference time have a significant drawback.
  2. The introduction of the overall pipeline and method details is clear and easy to follow.
  3. The construction process of the dataset is detailed, with a long summary in the appendix.
Critical weaknesses
  1. The algorithmic novelty is limited. The optical flow as action and the adapter design are from previous works. There are many works that use optical flow as latent actions. And there is already open-source work using optical latent actions for generation conditioning in the general robotics field, e.g. FlowWAM. However, there is almost no discussion and comparison between these methods. I only see the reference Videocontrolnet, and there is even no mention or citation of the original ControlNet and T2IAdapter.
  2. Although the author proposes text-to-action, there is no evaluation of how accurately the text can predict the latent action (optical flow). Without knowing the accuracy of the text-to-action, how can we ensure the improvement is from the latent action instead of other aspects?
  3. There is no comparison with other flow-based control methods, e.g. the Onlyflow. And no comparison between the control modalities, e.g text-to-depth, text-to-edge, text-to-segmentation, etc.
  4. The comparison in Table 1 is not completed, lacking baselines e.g. SurgSora (evaluated in KVLR), and also most of the metrics for Cosos-H-Surgical are missing.
  5. The ablation is limited, there is only a comparison on the FAE latent in the main paper. The authors claim that there will be ablation of losses and weights in the appendix. However, I do not find them. Also, there is no comparison between different versions of CLIPs, and no comparison of different modalities as mentioned above.

Chen Y, Li P, Xu Y, et al. FlowWAM: Optical Flow as a Unified Action Representation for World Action Models[J]. arXiv preprint arXiv:2607.13017, 2026.

Zhang L, Rao A, Agrawala M. Adding conditional control to text-to-image diffusion models[C]//2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023: 3813-3824.

Mou C, Wang X, Xie L, et al. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models[C]//Proceedings of the AAAI conference on artificial intelligence. 2024, 38(5): 4296-4304.

Hu Z, Xu D. Videocontrolnet: A motion-guided video-to-video translation framework by using diffusion model with controlnet[J]. arXiv preprint arXiv:2307.14073, 2023.

Koroglu M, Caselles-Dupré H, Jeanneret G, et al. Onlyflow: Optical flow based motion conditioning for video diffusion models[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE, 2025: 6216-6226.

Chen T, Yang S, Wang J, et al. Surgsora: Object-aware diffusion model for controllable surgical video generation[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2025: 521-531.

Questions for authors
  1. What is the latency, and how do the proposed modules affect the original inference speed?
  2. Why do you use LoRA instead of full finetune, is LoRA suitable for this large amount of data?
3 comments
AuthorsOpenReview ↗

Response to Reviewer WyYz

We thank Reviewer WyYz for the review and for providing valuable advice. FLAIR learns motion priors from video data and predicts latent actions to improve the scalability of surgical video generation, with only the need for a text prompt without additional visual inputs. We also appreciate the reviewer’s concerns regarding the accuracy of prediction from text, insufficient experiments, and questions about latency, with clarification below:

 

A1. Insufficient comparison with related work

We agree that providing comparisons with related flow-based work, such as OnlyFlow in general video generation, would provide stronger quantitative evidence for FLAIR. We perform an extra experiment by evaluating OnlyFlow on the same 200 surgical text prompts used in Table 1. For each prompt, OnlyFlow receives the text prompt, the first frame of an action-matched real surgical video, and RAFT optical flow extracted from that video; FLAIR receives only the text prompt. We evaluate both methods using 16-frame, 512×512 videos at 8 fps. The existing FLAIR videos are temporally sampled and center-cropped to this format without regeneration. We compute V.CLIP, FID, FVD, TI, AF, and the equal-weight mean of the six SurgMetrics.

We report the experimental results:

MethodV.CLIP ↑FID ↓FVD ↓TI ↑AF ↓Mean of six SurgMetrics ↓
OnlyFlow6.933161.7132567.5800.44290.27670.3789
FLAIR8.525111.7271545.1810.74750.13070.1245

From the results, FLAIR achieves a lower mean SurgMetrics score (0.1245 vs. 0.3789), alongside better video quality, temporal consistency, and text-video alignment under this matched short-video evaluation protocol. These results are informative because OnlyFlow is supplied with a real surgical reference video and its optical flow, whereas FLAIR generates from text alone. This provides evidence that FLAIR learns optical flow that is specifically adapted to the surgical field rather than a normal autoencoder.

 

A2. Accuracy of prediction from text

We agree that reporting the accuracy of text-to-action prediction would be valuable. We perform an extra experiment by comparing the mean latent representation predicted from each caption with the corresponding FAE representation extracted from the real video, using cosine similarity. We additionally use wrong-action captions as a control to assess whether the predicted latent captures information specific to the described action.

We report the experimental results:

ComparisonMean cosine similarity
Correct caption ↔ real-video FAE latent0.4524
Wrong-action caption ↔ real-video FAE latent0.3695
Paired improvement+0.0830
95% CI of the paired improvement[+0.0562, +0.1187]

From the results, the correct captions yield higher alignment with the real-video FAE representations than wrong-action captions, with the 95% confidence interval of the paired improvement entirely above zero. This provides evidence that the text-to-action predictor captures action-specific information consistent with the motion representations extracted from real surgical videos.

 

A3. SurgSora, and Cosos-H-surgical

We apologize for the lack of the baseline SurgSora and the incomplete metrics filled in Table 1. However, we would like to explain that this is due to the non-open-source status of the related repository. In their paper, neither of them reports metrics such as TI, AF, or standardized FID/FVD. Given the non-open-source status, we are also unable to reproduce their video or run the related metrics. As a result, certain baselines such as SurgSora are missing, and the rows of Cosmos-H-Surgical are incomplete.

AuthorsOpenReview ↗

Response to Reviewer WyYz(2)

A4. Ablation experiments

We apologize for the missing ablation experiments in the supplementary material. **We performed an extra ablation experiment on “Whether FAE can distinguish surgical actions rather than reconstructing optical flow only”. **

RepresentationEvaluation settingBalanced accuracyChance level
Frozen FAEComplete 80-token sequence, held-out test25.62%12.50%
Frozen FAEMatched four-position protocol16.16%12.50%
Random encoder (same architecture)Matched four-position protocol13.55%12.50%
Low-order optical-flow statisticsMatched four-position protocol13.56%12.50%

The experiment is performed by freezing the FAE and training only a class-balanced linear probe to predict eight surgical actions from its latent representations. The training and held-out test sets are strictly source- and case-disjoint. In the main setting, we temporally average the complete 80-token FAE sequence. We additionally construct a matched four-position setting using tokens at t={9,29,49,69}, and compare the trained FAE against (i) an untrained encoder with the identical architecture and (ii) handcrafted low-order optical-flow statistics, including the mean, standard deviation, quantiles, and coarse spatial flow distribution. We report the experimental results: the complete 80-token FAE achieves 25.62% balanced accuracy on the held-out test set, compared with the 12.50% eight-class chance level (p<10-5). Under the matched four-position protocol, FAE achieves 16.16%, also significantly above chance (p=0.0023), while the random encoder and low-order flow statistics obtain 13.55% and 13.56%, respectively. From the results, it provides evidence that the FAE contains linearly decodable and generalizable information for distinguishing surgical actions. Its higher point estimates over both matched controls further indicate that this action-discriminative information is not adequately explained by random high-dimensional features or simple motion magnitude and direction. We hope the ablation experiment could provide evidence that FAE learns action semantics that would be meaningful for the interpretation of FAE.

 

A5. Latency and SFT

We thank the reviewer for pointing out that reporting the latency of FLAIR inference would be important. We report the latency results: on Wan2.1 over 200 matched prompts under the same hardware and inference configuration, the average end-to-end generation latency is 194.995 s for LoRA and 194.355 s for FLAIR. The paired latency difference (FLAIR minus LoRA) is -0.640 s, with a 95% confidence interval of [-1.400,0.120] s. In addition, the text-to-action latent predictor requires only 8.38 ms on average (median: 8.24 ms; 95th percentile: 9.13 ms), corresponding to less than 0.005%0.005\% of the end-to-end generation time. These results show that the proposed modules introduce negligible computational overhead and do not cause a measurable reduction in the original inference speed.

We also thank the reviewer for raising the question of why we use LoRA instead of full fine-tuning. In our work, FLAIR tends to provide a practical way of improving action consistency in the generated surgical video by making use of optical flow rather than training a surgical video generation foundation model. LoRA provides a lightweight environment for the experiment to be deployed. SurgActionClip-30K also provides valuable dataset captions for unlabeled sourced video, providing the potential for follow-up research work to conduct full fine-tuning by using the datasets.

We thank Reviewer WyYz again for the constructive review. We hope the above provides clear clarifications for FLAIR.

  1. I think the author is avoiding discussing the novelty problem for the optical flow as action design. Although an additional comparison to OnlyFlow is provided, the novelty problem still exists.
  2. There are still no comparison experiments and discussions with T2IAdapter and ControlNet (The most basic baselines in conditioned generation)
  3. The reported cosine similarity is not enough to support the claim of text-to-action accuracy. The author should provide statistical analysis as well as detailed experimental settings ("Wrong-action caption" is ambiguous. I don't even know what the wrong caption is)
  4. I have mentioned text-to-depth, text-to-edge, and text-to-segmentation setups for comparison, but there is no experiment added.
  5. I have mentioned, "The authors claim that there will be ablation of losses and weights in the appendix. ", However, the author seems to have missed this in the previous review. The author should provide all combinations of losses and sensitivity tests for the weights as claimed in the paper.

Overall, the response did not resolve my concerns. I will retain my score.

Reviewer_5Nu8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that surgical video generation shall learn motion priors from video data and predicts latent action ("action-centric"). The FLARE model they developed also uses text prompts. The authors also construct SurgActionClip-30K, an "action-centric" surgical video dataset, and SurgMetrics, a domain-specific video quality evaluator. Human evaluation by four specialist surgeons is performed for evaluating the model.

Critical strengths

The proposed approach can potentially improve the scalability of surgical video generation, as it's only need text prompt without additional visual inputs. The consistent improvements across three backbone models are also encouraging.

SurgActionClip-30K provides a useful collection of surgical video clips with structured descriptions.

Critical weaknesses

The concept and implementation of "action-centric", which the paper extensively argued for its importance, is not new. Nwoye et al. 2022 introduced surgical action triplets, Xu et al. 2026 established a large-scale basic surgical action dataset, and Yeganeh et al. 2024 already explored action-guided surgical video generation. Although FLAIR introduces a text-predicted motion prior, its novelty appears mainly in combining existing action representations with generative model (optical-flow learning and diffusion conditioning).

https://doi.org/10.1016/j.media.2022.102433 https://doi.org/10.48550/arXiv.2603.12787 https://doi.org/10.1007/978-3-031-77610-6_14

Also, the action-centric formulation is not sufficiently validated. The authors use existing phase/step annotations and instrument detection to determine clip boundaries, but neither necessarily corresponds to individual surgical actions. Second, the learned representation may not actually be action-centric. The flow action encoder primarily learns optical-flow reconstruction without explicitly enforcing action semantics. The next-frame prediction experiment (Table 2) shows that the representation contains motion information, but does not demonstrate that it distinguishes surgical actions. Thus, the underlying assumption is questionable. The authors assume that surgical actions occupy compact motion regions with relatively canonical execution patterns. However, the same action may require substantially different movements depending on anatomy and other context.

Questions for authors

Please report the proportion of generated videos excluded by the SurgMetrics domain-distance threshold of 4 and clarify whether the comparisons are performed on identical retained samples.

The supervision targets are defined in [0, 1]. Why some SurgMetrics artifact scores in Table 1 are negative values?

Provide paired statistical analysis of the improvements in Table 1 using the shared 200 prompts, particularly for metrics where FLAIR does not consistently outperform LoRA.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 5Nu8

We thank Reviewer 5Nu8 for the review, providing technically grounded feedback. FLAIR learns motion priors from video data and predicts latent actions to improve the scalability of surgical video generation, with only the need for a text prompt without additional visual inputs. We also appreciate the reviewer’s concerns regarding the uniqueness of action-centric concept, motion that FAE learns, and the experimental results, with clarification below:

 

W1. Uniqueness of action-centric concept

We agree that the concept of introducing action-centric representation is not new. We clarify that we are extending basic action triplets or a collection of action videos into a multimodal and more comprehensive form of representation. Our work goes beyond the three mentioned works in terms of the form of actions and the use of representation.

  1. Compared with Nwoye et al. (2022), their “instrument-verb-target” triplets are mainly used as labels in detection. SurgActionClip-30k incorporates this paradigm structure into richer natural language captions, and treats them as action priors instead of labels.

  2. Compared with Xu et al. (2026), their design goal is action understanding/recognition, and they treat actions as discrete classes, while SurgActionClip-30K's design goal is using action priors for generative motion learning. This is why we are adding structured captions on top of action clips. This enables it to directly establish the training relationship of text semantics -> action clip -> optical-flow motion pattern.

  3. Compared with Yeganeh et al. (2024), FLAIR treats actions in a totally different form. FLAIR learns continuous latent action representations from thousands of action clips, instead of using discrete action semantic conditions for future frame generation.

Together, our action-centric design is distinguished by treating actions as discrete labels or explicit semantic conditions. FLAIR is also among the first to integrate the widely adopted surgical “instrument--verb--target” triplet semantics into a continuous action representation for generative motion modeling.

 

W2. Motion that FAE learns

RepresentationEvaluation settingBalanced accuracyChance level
Frozen FAEComplete 80-token sequence, held-out test25.62%12.50%
Frozen FAEMatched four-position protocol16.16%12.50%
Random encoder (same architecture)Matched four-position protocol13.55%12.50%
Low-order optical-flow statisticsMatched four-position protocol13.56%12.50%

We thank the reviewer for raising the question of whether FAE can distinguish surgical actions beyond reconstructing optical flow. We perform an extra ablation experiment by freezing the FAE and training only a class-balanced linear probe to predict eight surgical actions from its latent representations. The training and held-out test sets are strictly source- and case-disjoint. In the main setting, we temporally average the complete 80-token FAE sequence. We additionally construct a matched four-position setting using tokens at t={9,29,49,69}t=\lbrace 9,29,49,69\rbrace, and compare the trained FAE against (i) an untrained encoder with the identical architecture and (ii) handcrafted low-order optical-flow statistics, including the mean, standard deviation, quantiles, and coarse spatial flow distribution. We report the experimental results: the complete 80-token FAE achieves 25.6225.62% balanced accuracy on the held-out test set, compared with the 12.5012.50% eight-class chance level (p<10−5p<10^{-5}). Under the matched four-position protocol, FAE achieves 16.1616.16%, also significantly above chance (p=0.0023p=0.0023), while the random encoder and low-order flow statistics obtain 13.5513.55% and 13.5613.56%, respectively. From the results, it provides evidence that the FAE contains linearly decodable and generalizable information for distinguishing surgical actions. Its higher point estimates over both matched controls further indicate that this action-discriminative information is not adequately explained by random high-dimensional features or simple motion magnitude and direction. Therefore, it provides evidence that FAE learns action semantics that remain readable across unseen surgical cases and can distinguish surgical actions, rather than merely reconstruct optical flow.

AuthorsOpenReview ↗

Response to Reviewer 5Nu8 (2)

W3. Clarification of experimental result

We thank you for pointing out the issue of threshold 4, and clarify that no generated videos were excluded at the sample level for any model reported in Table 1. All metrics were strictly computed over the identical, full set of 200 prompts. The threshold of 4 is a cohort-level gate. For the three unadapted base models with mean domain distance >4, we remove the entire cohorts from artifact comparisons and show as “-” in the SurgMetrics column in Table 1. For LoRA and FLAIR variants whose cohort-level mean domain distance 4, we retain the mean value of full 200-prompt as comparisons. No selective filtering on individual clips is performed to act as cherry-picking. For all evaluated LoRA and FLAIR variants, the generations are consistently in-domain and retain all 200 samples identically. We apologize for the condensed wording in Section 6.1 and will revise it to state this cohort-level validity criterion.

We agree that in Table 1, some SurgMetrics artifacts scored negative values. We clarify that during training, we set the training goal of the neural network for each sample to be within [0,1]. However, a training goal of [0,1] does not guarantee that the inference output by SurgMetrics would be within the range of [0,1], due to the mathematical nature of neural networks. As a result, the actual output range for each artifact of SurgMetrics is mathematically [-inf, +inf], not restricted, but in practice close to [0,1]. We keep the vanilla output to avoid that by applying a normalization function, for example, the sigmoid function, may compact the output value. We apologize that this design may lead to less standardized results, and we will revise it to a normalized value within the range of [0,1].

 

W4. Reporting paired statistical analysis

We agree that reporting a paired statistical analysis of the improvements in Table 1 would provide stronger quantitative evidence. We perform extra paired tests between LoRA and FLAIR using the same evaluation samples and report the corresponding p-values below:

Paired statistical significance of the results in Table 1.

Backbone (LoRA vs. FLAIR)V.CLIPFIDFVDTIAFJoint 6 SurgMetrics
Wan2.13.79×10−43.79 \times 10^{-4}6.00×10−66.00 \times 10^{-6}9.00×10−69.00 \times 10^{-6}2.16×10−42.16 \times 10^{-4}5.406×10−35.406 \times 10^{-3}<10−6< 10^{-6}
Hunyuan-1.57.99×10−47.99 \times 10^{-4}<10−6< 10^{-6}<10−6< 10^{-6}<10−6< 10^{-6}3.46×10−23.46 \times 10^{-2}<10−6< 10^{-6}
CogVideoX-2B8.27×10−38.27 \times 10^{-3}<10−6< 10^{-6}<10−6< 10^{-6}1.43×10−11.43 \times 10^{-1}4.19×10−24.19 \times 10^{-2}9.70×10−59.70 \times 10^{-5}

The results show that FLAIR yields highly significant improvements in FID, FVD, and the joint six-dimensional SurgMetrics evaluation across all three backbones. The TI improvements are significant on Wan2.1 and Hunyuan-1.5 and remain numerically positive on CogVideoX-2B, while AF is significantly improved across all three backbones.

For V.CLIP, the paired differences are significant, although LoRA consistently outperforms FLAIR on Hunyuan-1.5 and CogVideoX-2B. V.CLIP primarily evaluates global prompt–video semantic similarity, which is already strongly captured by the text-conditioned LoRA backbone. The outperformance of FLAIR by LoRA in V.CLIP score indicates a difference in global embedding similarity in V.CLIP. Yet, as V.CLIP is not specifically pre-trained in the surgical field, the higher value in V.CLIP does not indicate that the video generated by LoRA has a higher alignment with the input text under an endoscopic setup. That is also exactly the reason why we introduce SurgMetrics. Moreover, the absolute V.CLIP gaps remain modest: 8.41 vs. 8.11 on Hunyuan-1.5 and 8.47 vs. 8.27 on CogVideoX-2B, indicating that FLAIR preserves comparable semantic alignment. In contrast, FLAIR is specifically designed to improve action-conditioned dynamics, temporal stability, and surgery-specific motion fidelity. The delivery of its consistent and statistically significant gains in FID, FVD, and the joint SurgMetrics directly measures the primary improvements targeted by our method.

We thank Reviewer N469 again for the constructive review. We hope the above provides clear clarifications for FLAIR.

Thank you for the detailed response and additional experiments. The new linear-probe results provide some evidence that FAE captures action-discriminative information, although the relatively modest performance suggests that its ability to represent distinct surgical actions remains limited. The effort is really appreciated nevertheless.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.