acceptodds
Under review as a conference paper at ICLR 2027

SurgOmni: Towards All-in-One Surgical Video Restoration

16 views

Abstract

Laparoscopic videos are often affected by diverse degradations arising at different stages of the imaging process, making all-in-one surgical restoration necessary for real-world deployment. However, advancing all-in-one surgical restoration remains constrained by three fundamental limitations: (1) existing training datasets either lack exact pixel-level degraded-clean correspondence or rely on simplified synthesis that inadequately models the formation of complex surgical degradations; (2) existing real-world testing datasets often remain image-based and cover only a limited range of degradation types and surgical scenarios, restricting comprehensive evaluation of model performance; and (3) the relationship between all-in-one training degradation composition and real-world testing generalization remains poorly understood. To address these limitations, we introduce SurgOmni, a unified dataset and benchmark suite for all-in-one surgical video restoration. Specifically, we first propose OmniSyn, a 44,000-frame paired training dataset comprising 14 degradation configurations and constructed through a formation-aware synthesis pipeline, where scene- and lens-space degradations are rendered via a physics-based 3D engine with temporal dynamics, while exposure- and illumination-related degradations are modeled through exposure integration and content-aware spatial photometric modeling, respectively. Meanwhile, we further propose OmniBench, a 10,000-frame real-world video testing dataset spanning 6 surgical types, 10 target organs, 14 degradation configurations, and 27 surgical scenes. Extensive training-dataset comparisons first show that models trained on OmniSyn consistently achieve the strongest generalization to real surgical data. Comprehensive benchmarking across diverse restoration methods on OmniBench then reveals distinct strengths and limitations in restoration quality and temporal motion deviation that are obscured by limited real-world evaluation. Beyond performance evaluation, we further investigate how training-data degradation composition affects real-world testing generalization, revealing the relative effectiveness of different composition strategies. These findings provide practical guidance for training-data design and restoration method development for all-in-one surgical restoration.

Primary area:
datasets and benchmarks
Keywords:
All-in-One Surgical Video Restoration, Datasets and Benchmarks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7xc8OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents SurgOmni, a dataset and benchmark suite for all-in-one surgical video restoration. It introduces OmniSyn, a 44,000-frame paired synthetic training dataset covering 14 degradation configurations, and OmniBench, a 10,000-frame real-world evaluation dataset spanning six surgical types.

Critical strengths

High completeness and maturity. The manuscript integrates dataset construction, benchmark evaluation, and systematic analysis into a well-developed study. Detailed reporting and thorough comparisons make it a useful resource for future surgical video restoration research.

Critical weaknesses

Limited methodological novelty. Despite its thorough experiments and potential value to the field, the paper’s contribution lies primarily in its large-scale datasets and comprehensive benchmarking. The synthesis pipeline mainly combines existing rendering software and image-processing techniques, as illustrated in Figure 2, with limited methodological innovation. While the resulting resources and analyses are useful, this limitation moderates my enthusiasm and motivates a Weak Accept recommendation rather than a stronger endorsement.

Questions for authors

Beyond dataset scale and coverage, what methodological innovations distinguish the proposed synthesis pipeline from existing surgical degradation synthesis approaches?

Reviewer_fvKdOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SurgOmni, a dataset and benchmark suite for all-in-one restoration of laparoscopic video. t has three parts:

  1. OmniSyn (training data). A paired synthetic set of 44,000 frames in 440 clips, built from clean clips of CholecTrack20 and M2CAI16-TOOL. It covers five degradation types: smoke, lens haze, lens splash, motion blur and non-uniform illumination. These appear alone and in nine dual or triple combinations, at several severity levels;
  2. OmniBench (real test data). 10,000 frames in 100 real clips. It combines public gallbladder recordings with a new colorectal cohort from one hospital, and covers 6 procedure types, 10 organs and 27 scenes. Each clip is labelled with the degradations it contains; and
  3. Experiments.
Critical strengths
  1. Surgical video is affected by several degradations at once, and paired real data is essentially impossible to collect. A shared training and evaluation resource for all-in-one surgical restoration is useful to the community.
  2. Modelling each degradation at its physical formation stage is a sensible design. Examples: smoke lives in scene space and drifts, while haze and splash stay fixed to the lens. Temporal coherence is preserved, and clips share exact pixel correspondence with their clean sources.
  3. OmniBench is video-based, covers co-occurring degradations, and adds a new colorectal cohort. This broadens the anatomy beyond the usual cholecystectomy data. Splitting public sources at the recording level avoids direct leakage.
  4. Fifteen recent methods across many architecture families are trained under a common protocol and averaged over three runs. The paper also reports efficiency (parameters, compute, latency).
  5. The reversal between synthetic and real results for multi-degradation training is interesting. It is checked with the same metric family on both domains, which rules out a simple metric artefact.
  6. Six experienced surgeons rated synthesis realism in a blinded, randomized study. The authors also commit to releasing the synthesis code, manifests and evaluation scripts.
Critical weaknesses
  1. Real-world evaluation rests only on no-reference image-quality metrics. TOPIQ, MUSIQ and Q-Align were trained on natural images and their validity for surgical scenes is not established. The agreement analysis between these metrics and reference-based metrics is done on synthetic OmniSyn clips and only over 15 method averages, not on real data and not per clip. TOPIQ agreement is only moderate.
  2. Restoration quality on real clips is never rated by humans. The surgeons rated synthesis realism, not restoration quality. Without surgeon judgments on restored real clips, it is unclear whether higher metric scores correspond to clinically better or more faithful restorations. No-reference metrics may reward hallucination. These metrics favor sharp, natural-looking images, which can reward invented texture. This is a particular concern in surgery, where fidelity to the actual tissue matters more than aesthetics.
  3. The motion-deviation metric on real data is hard to interpret. On OmniBench, the degraded input serves as the motion reference. Doing nothing gets a perfect score, and correctly removing moving smoke or splash is penalized. The authors acknowledge this, yet still conclude that "constrained mappings" preserve motion better. Lower deviation may simply mean a method changes the input less.
  4. Downstream consistency is measured on the OmniSyn test set, which favors OmniSyn. OmniSyn-trained models are evaluated in their own training domain, while SurgClean- and LVQ-trained models are out of domain. This is not a fair basis for claiming OmniSyn preserves structure better. The reference is also the frozen model's prediction on the clean frame, not ground-truth annotation.
  5. Source content is confounded with synthesis quality. OmniSyn's clean clips come from the same public collections (CholecTrack20, M2CAI16) as OmniBench's gallbladder subset. The split is at recording level, but camera, hospital, procedure and visual style still match. SurgClean and LVQ come from other sources, so part of OmniSyn's advantage may be domain proximity rather than "formation-aware" modelling.
  6. The key ablation for the main claim is missing. The paper does not apply simpler frame-wise synthesis (for example LVQ-style or standard atmospheric-scattering smoke) to the same clean clips. Without this, the benefit of physics-based, temporally coherent synthesis is not isolated. The authors themselves note that the cross-scale comparison is confounded.
  7. The multi-degradation result partly undercuts the synthesis claim. If formation-aware multi-degradation synthesis were realistic, training on it should help on real multi-degradation clips, but it hurts. Surgeons also rated dual and triple synthesis lowest (around 3.9).
  8. The composition conclusion may also reflect coverage. "Multi" covers only nine combinations and never shows some degradations in isolation. Its very high motion deviation suggests over-aggressive restoration, so the result may reflect coverage and severity mismatch rather than a general principle. The recommendation to prioritize single-degradation data may therefore be specific to this synthesis pipeline.
  9. Conclusions about generative methods rest on two models. EndoIR and RAR, the only two generative methods, perform worst. RAR is a large foundation-model-based method fine-tuned on a small dataset with an unusual optimizer setup, which may simply be poorly adapted. That is not enough to support general advice about generative priors and latent compression.
  10. Training budgets may not be comparable. Each method keeps its own official recipe, with different crop sizes (128 versus 256), losses and optimizers. Iteration counts and total training budgets are not reported, so differences may reflect tuning as much as architecture.
  11. The surgeon realism study is small and possibly biased.
  12. The smoke rendering is adapted from an existing pipeline (STSVD), and the evaluation metrics and protocols are standard. The main contribution is data curation and benchmarking, with limited methodological insight for a general machine-learning venue.
Questions for authors
  1. Can you provide human evaluation of restoration quality on real OmniBench clips? For example, have surgeons rank outputs from several methods, then measure how well TOPIQ, MUSIQ and Q-Align agree with those rankings on real surgical data.
  2. Can you report metric agreement per clip rather than only across 15 method averages, and ideally on real data?
  3. Do the no-reference metrics reward hallucinated detail? Have you checked whether methods that invent texture score higher than methods that faithfully preserve the observed tissue?
  4. On real data, doing nothing gets a perfect motion-deviation score. How do you separate "preserving true motion" from "changing the input less"? Could you report a reference-free temporal consistency measure (for example flicker or warping error between restored frames), or annotate a small set of real clips with motion references?
  5. Downstream consistency is measured on the OmniSyn test set, which is in-domain for OmniSyn-trained models. Can you evaluate it on real clips, for example using predictions on adjacent clean real frames or manual annotations on OmniBench?
  6. Can you run the key ablation: same clean OmniSyn clips, but degraded with a simpler frame-wise synthesis? This would separate the effect of formation-aware synthesis from the effect of the clean source content.
  7. How much of OmniSyn's advantage on the gallbladder subset comes from sharing source collections, cameras and procedures with OmniBench? Does the advantage persist when training on clean clips from unrelated sources?
  8. Why does multi-degradation training hurt on real multi-degradation clips? Could it be due to missing single-degradation exposure, severity mismatch, or over-restoration (suggested by the much higher motion deviation)? Would a mixture with more single-degradation data than multi-degradation data change the picture?
  9. Can you report standard deviations across the three runs and significance tests for the benchmark? How stable are the k-means tiers under small perturbations?
  10. Were training iterations and total compute matched across methods? How sensitive are the rankings to the crop size and training schedule?
  11. Why were recent surgical video desmoking methods and more video restoration methods not included, given the video focus? How was RAR adapted, and could its poor performance be due to fine-tuning on limited data rather than an intrinsic limitation of generative priors?
  12. What is the inter-annotator agreement for the OmniBench degradation labels? How many clips needed clinician adjudication? Do you plan to add severity or frame-level labels?
  13. In the surgeon study, could the real calibration clips and LVQ's lower resolution have biased the ratings? Did you consider including real clips as hidden controls, to measure whether surgeons can tell real from synthetic?
  14. Do the licenses of CholecTrack20 and M2CAI16 allow redistributing derived synthetic videos? Under what terms will the clinical colorectal data be accessible?
  15. Do you plan to add degradations such as noise, compression, defocus, specular highlights and blood on the lens, and procedures from more institutions and camera systems?
Reviewer_NDipOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces 1. a novel dataset for restoration of degraded surgical videos by synthetically generating degradations on real world data, 2. a real world benchmark dataset for evaluating models on the restoration task and 3. an evaluation of the dataset using a variety of prior restoration methods and comparing restoration quality as well as downstream performance against prior datasets. The training dataset consists of videos from public cholecystectomy datasets that are synthetically degraded using 1. physics based rendering pipelines in Blender to simulate splash, haze and smoke and 2. code based synthetic degradation to simulate non-uniform illumination and motion blur. The synthesized data is organized into single- and multi-degradation datasets. To show generalization to real surgical degradation scenarios the authors introduce a new real world benchmark dataset that consists of public and private data. The synthesized training data set is evaluated using 15 prior restoration methods. Restoration quality and downstream segmentation / depth estimation performance is compared against two prior restoration datasets. For evaluation on the real data, learned, no-reference metrics (TOPIQ, MUSIQ, Q-Align) are measured whereas for evaluation on the synthetic held-out test set full-reference metrics (PSNR, SSIM, LPIPS) are reported. To evaluate the temporal behaviour of models trained on their data the authors also report tOF, calculated between the degraded input and the output respectively. The evaluation shows that restoration approaches achieve higher no-reference metric performance on real world data when trained on the proposed data than on prior datasets. The same holds for downstream task consistency on the synthetic test set. The authors argue that the improved performance is due to the exact clean-degradation pairs in the data, higher degradation quality and broader scene coverage. The synthesized data realism is additionally rated via a study done with expert surgeons. Furthermore, the authors ablate on the dataset composition, in particular the amount of multi-degradation data in the training set and report full-reference metrics on the synthetic test set as well as no-reference metrics on the real world data. They show that training on only single-degradation data leads to better real world generalization, even on multi-degradation test data, than training on multi-degradation data and argue that single-degradation data provides a cleaner supervision signal. Finally, comparing restoration methods with each other reveals that generative methods tend to perform worst whereas more structure-focused approaches achieve better restoration quality and optimal transport constrained methods achieve lower tOF scores.

Critical strengths

The synthesis pipeline is novel and yields a video level degradation dataset. The scale of the dataset outmatches that of prior video-based ones and the effects of the data on model performance are extensively evaluated. In particular, the single/multi/mixed comparisons are well budget-matched and the realism study adds a valuable insight on the data quality. The authors also show good generalization from the gallbladder-only training data to real world colorectal data.

Critical weaknesses

The claims of better generalization to real data are supported by relatively weak metrics. The no-reference metrics are designed for non-surgical image data and their correlation with full-reference metrics is only measured on the synthetic data. Also, the relative improvement of these metrics is not visible without a baseline value on the degraded input frames. Similarly, tOF is calculated by comparing the optical flow of the degraded and restored videos, so low values may simply reflect fewer edits rather than better temporal behaviour.

Questions for authors
  1. What are the no-reference metric values in Table 2 for the degraded input data? This baseline would help interpreting the relative improvement better.
  2. How does the input compare to the output in Table 2 e.g. in terms of LPIPS, MAE? These metrics would help interpreting tOF better, i.e. answering the question whether lower tOF actually means better temporal behaviour or just less edits.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.