acceptodds
Withdrawn from ICLR 2027

Medical World Model: Generative Simulation of Long-Horizon Surgery From Untimed Action Descriptions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Surgical descriptions convey a surgeon's intended actions during procedures that can span hours, yet they leave open how those actions unfold and reshape the operative scene over time. Translating such descriptions into sustained visual simulation requires capturing both the progression of interacting actions and their lasting effects on tissue. We introduce SurgWAVE, a world model that generates successive surgical video blocks from an initial observation and untimed action descriptions. The model reconciles correlated descriptions without prescribing a frame-wise action schedule, retains accumulated intervention effects through utility-calibrated recurrent memory, and preserves local structure through training-only spatial distillation. To support this task, we introduce LiRAct, an expert-annotated laparoscopic liver-resection dataset and benchmark comprising 3,333 operative episodes from 18 patients. Experiments on LiRAct and Cholec80 demonstrate state-of-the-art performance in conditional surgical video generation. The code and benchmark will be released upon acceptance.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Surgical Vision, World Model, Video Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HQr3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SurgWAVE, a surgical world model for generating continuous surgical videos from initial observations and untimed action descriptions. The framework integrates correlation-aware signed action conditioning, Contractive Residual State Memory (CReSM), and predicted-state spatial distillation to address action ambiguity, historical state preservation, and spatial consistency. The authors also introduce LiRAct, an expert-annotated laparoscopic liver-resection dataset. Experiments on LiRAct and Cholec80 report improved video generation quality over existing methods, supported by component ablations and expert evaluations.

Critical strengths
  1. The paper addresses a meaningful problem by generating surgical videos from untimed action descriptions without requiring frame-level action schedules or explicit motion trajectories.

  2. The proposed framework combines action conditioning, recurrent memory, and spatial distillation, achieving promising results on LiRAct and Cholec80, supported by quantitative comparisons and surgeon evaluations.

Critical weaknesses
  1. The ablation studies do not adequately isolate the proposed mechanisms, particularly correlation-aware aggregation, spatial distillation, and long-term memory. The interactions among the three modules also remain unclear.

  2. The construction and annotation procedures of LiRAct are insufficiently described. Moreover, temporal action correctness and controllability under modified action descriptions are not systematically evaluated, limiting the evidence for the central claims.

Questions for authors
  1. Could the authors provide further details on the LiRAct construction process, including video segmentation, sample selection, annotation procedures, quality control, and action distribution statistics?

  2. Could the authors provide quantitative comparisons among Independent, Softmax Attention, and Correlated Aggregation, further ablations of the spatial distillation design, and a complete 2³ ablation to examine module contributions and interactions?

  3. Could the authors evaluate CReSM under different KV cache sizes and generation lengths to further demonstrate its effectiveness in preserving historical surgical states?

  4. Could the authors provide further evaluations of action temporal correctness (ordering, duration, and overlap), as well as controlled action-editing experiments to assess responsiveness to modified action descriptions?

Reviewer_bT4SOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies surgical video generation from an initial image and action descriptions without frame-level timing. SurgWAVE extends Wan2.2-TI2V-5B with the LongLive-2.0 block-autoregressive architecture, correlation-aware action conditioning, Contractive Residual State Memory (CReSM), and training-only spatial distillation. The paper introduces LiRAct, which contains 3,333 operative episodes and 5,692 camera-view records from 18 patients. Evaluation covers LiRAct and Cholec80, with an additional expert assessment of three methods on the same 80 cases.

Critical strengths
  1. The task and dataset fit a practical annotation setting. Surgical actions can overlap and recur, while precise event timing is expensive to annotate. LiRAct provides segment-level descriptions for this setting. The patient-disjoint split and the decision to keep paired views within one partition are appropriate. Evaluation uses five patients excluded from the 13-patient adaptation set.
  2. The mechanisms are described in sufficient detail to examine them. The paper specifies the action correction, block-level memory commits, observer Jacobian products, and gradient budgets. The action branch and CReSM add 553,328 and 196,612 parameters, respectively. The derivations also state useful limits on what the mathematical properties establish.
  3. There is evidence for an improvement in the full system. Table 3 includes continued training of the original video generator for 1,000 additional updates without the proposed modules, within the same Wan2.2/LongLive block-causal framework. Its FVD is 337.93, compared with 154.37 for the full model. This is more informative than comparison with the starting checkpoint alone. The expert study adds activity-level evidence, with event support increasing from 63.2% for LingBot-World to 87.0% for SurgWAVE on shared cases.
Critical weaknesses
  1. The component study does not isolate the proposed mechanisms The CReSM comparison also changes attention context. Adding memory to action conditioning and spatial distillation reduces FVD from 215.73 to 154.37 in Table 3. Table 8, however, assigns zero latent-frame attention sink to action/native configurations and one latent frame to CReSM configurations. The 61.36 FVD difference therefore combines recurrent memory with a context change. A comparison under the same sink, including a sink-only control, is needed to attribute this gain. The correlation-aware aggregator has a similar attribution gap. Adding the whole action branch to continued training of the original video generator reduces FVD from 337.93 to 234.53. This also introduces a conditioning path and trainable parameters. Figures 3 and 7 provide qualitative comparisons, and Figure 5 contains numerical diagnostics. However, Section 4.3 and Appendix D explicitly state that no separate cohort-level quantitative comparison of independent and correlation-aware aggregation is reported. Since both variants are implemented, a matched evaluation would directly test the proposed aggregation rule.
  2. The evaluation does not directly test temporal action realization or persistent state Untimed descriptions permit training and use with coarse annotations. Section 3.2 leaves event order to the autoregressive generator, but the evaluation provides little direct evidence about prerequisite order, plausible overlap, or action completion. The original video generator already receives the complete caption. Better visual quality and visible event support alone leave uncertain what temporal reasoning the additional action branch contributes. The long-horizon claim has a related gap. FVD uses 16-frame I3D windows, while reference-based evaluation extends to 15 seconds. These sequences may already involve cache eviction, but neither Table 9 nor the aggregate scores track a specific tissue change through eviction. Section 4.3 acknowledges this limitation. A focused test could generate a shared history containing a tissue change, record when its establishing observations leave the cache, and compare retained, reset, and absent memory. Expert inspection of action order and overlap on the same cases would connect the proposed mechanisms to their intended behavior while allowing multiple valid continuations.
  3. A direct fine-tuning baseline for the original Wan2.2 architecture is missing The introduction motivates sustained generation beyond a local attention window, and Sections 3.1 and 4.1 identify the LongLive-2.0 block-causal architecture. These passages explain its role in sequential generation. The paper gives less justification for choosing this architecture over directly fine-tuning the original Wan2.2-TI2V-5B on surgical videos. Table 7 already defines initialization using Wan2.2 with LongLive-2.0. The continued-training control in Table 3 therefore measures adaptation within that framework, rather than a comparison with the original Wan2.2 architecture. A direct fine-tuning control should use the same initial images and complete captions, patient split, and comparable training budget. At durations supported by both systems, matched resolution and temporal sampling would allow comparison of visual quality, action compatibility, and continuity. Longer rollouts could then test the additional benefit of block-causal generation separately. This would distinguish gains from surgical-domain adaptation from those that require the chosen generation architecture. The proposed modules are also evaluated on only one backbone, so their generality across architectures or model scales remains untested.
  4. The spatial-distillation design lacks comparison with simpler structural supervision Section 3.4 introduces an observer-Jacobian metric, stochastic trace normalization, and constrained auxiliary gradients. Table 3 shows a benefit from adding this package, with FVD decreasing from 234.53 to 215.73. However, the comparison is against omitting distillation. The added value of this construction over direct feature or local-relation matching with the same frozen teacher remains untested. A comparison under comparable training and auxiliary-gradient budgets would help justify the additional machinery. The structural benefit also needs a more specific check. The observer constrains neighboring patch-feature relations, while the intended surgical benefits concern instrument integrity and tool–tissue structure. The expert study assesses instrument integrity and interaction plausibility, but only for the full system. A small paired assessment of the distillation variants, focusing on instrument deformation, boundary errors, and implausible contact, would show which structural failures the proposed objective reduces.
Questions for authors
  1. Can the authors isolate the two proposed mechanisms through matched-sink memory controls and a cohort-level comparison of independent versus correlation-aware aggregation under the same training and sampling conditions?
  2. Can the authors provide a focused evaluation of plausible action order and overlap, together with retention of a tissue change past cache eviction, including a memory-reset control?
  3. What limitation of directly fine-tuning the original Wan2.2-TI2V-5B motivates adopting LongLive-2.0? Can the authors provide this baseline under comparable training and conditioning, with matched evaluation at jointly supported durations?
Reviewer_3bQVOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies surgical video generation conditioned on untimed action descriptions — segment-level sentences that name instruments, activities and (optionally) targets, but carry no frame-aligned onsets or durations. This is a genuinely more realistic input regime than the frame-aligned kinematic/trajectory conditioning used by most surgical world models, and it is the paper's main conceptual contribution. To support the setting, the authors introduce LiRAct, an expert-annotated laparoscopic liver-resection benchmark. The proposed model SurgWAVE (built on Wan2.2-TI2V-5B with a LongLive-2.0 block-causal architecture) contributes three mechanisms: (i) correlation-aware signed action conditioning that aggregates state-dependent flow corrections over correlated descriptions via a closed-form determinantal (DPP-style) marginalization, avoiding the mutual exclusivity a softmax would impose; (ii) CReSM, a bounded, contractive recurrent state that summarizes changes relative to a first-frame anchor and is decoupled from the rolling KV cache; and (iii) predicted-state spatial distillation using the Jacobian of a frozen LingBot-Vision observer with gradient budgeting, used only at training time.

Critical strengths
  1. Well-posed and under-explored problem. "Untimed action descriptions" cleanly captures how surgical reports actually read, and the distinction from frame-aligned control (Fig. 1a) is sharp. It moves surgical world modeling away from interfaces that require instrumentation most hospitals do not have.
  2. Principled and elegant aggregation. The determinantal subset marginalization (Eq. 4–5) is a genuinely nice idea: Gram determinants measure squared key volumes so that dependent action subsets get zero weight, letting correlated actions (retract + dissect) co-contribute rather than compete. The closed form via ½∇_K log Z, the permutation-invariance argument, and the rank-deficiency handling in Appendix G are correct and non-trivial.
  3. Lightweight, well-analyzed memory. CReSM adds only ~197K parameters, keeps every state coordinate in [−1,1] by construction, contracts in its initial state, and commits once per video block so memory advances in video time rather than in denoising-step time — a clean design that sidesteps cache-eviction issues.
Critical weaknesses
  1. W1 — The central "long-horizon" claim is not actually tested. The title, abstract and §1 motivate hours-long procedures, but Table 9 shows the evaluation manifest caps at 15 s (median 263.5 retained frames ≈ 11 s), with 0 records available at 16 s. The authors acknowledge this, but it leaves the headline framing unsubstantiated. More importantly, CReSM's entire motivation is retaining effects beyond the attention window, yet the rolling cache already holds 24 latent frames ≈ 1 s of video while rollouts are ~11 s — so the measured 61.36 FVD gain from M is obtained in a regime where the window is short relative to the rollout but the absolute horizon is still far from "long". There is no horizon-stratified analysis (e.g., FVD at 4/8/12 s buckets, or FVD vs. block index over a longer rollout), which is the single most informative missing experiment and the one that would most directly validate the memory design.

  2. W2 — System-level comparisons carry substantial confounds. Per Table 10, Endora/MOFA-Video/SG2VID are evaluated at 128×128 with 16/25/16 frames, LingBot-World at 832×480 natively 16 fps, and SurgWAVE at 1280×704 and up to 360 frames at 24 fps — all scored by FVD at a common 256-px evaluation resolution. Differences in native resolution, duration, frame rate and output coverage (1,029 vs. 1,310) plausibly account for part of the FVD gap independently of modeling quality. The retained-native-interface choice is defensible and well documented, but a resolution/length-matched subset (or reporting FVD on the subset where all methods share an output grid) would make "state-of-the-art" much harder to argue with. Relatedly: MOFA-Video receives reference-derived optical flow from the recorded future, which is a strictly stronger input — worth stating more prominently when interpreting its 772.06.

  3. W3 — No uncertainty anywhere, and several declared wins are below noise. Single seed (3407), one sample per condition, point estimates only. FVD is known to be high-variance, and the appendix correctly notes that repeated frames, paired views and repeated ratings are correlated. Yet Table 2 claims wins of 0.0023 (background consistency) and 0.0068 (motion smoothness), which cannot be interpreted at this resolution. At minimum I would like mean ± std over 3–5 seeds, or patient-level bootstrap CIs, for the full-vs-best-baseline comparison. Two metrics are also effectively uninformative as reported: dynamic degree saturates at 1.0000 for several methods, and temporal flickering is N/A for SurgWAVE because only 2 static clips are eligible (vs. 859 for Endora) — the latter should probably be dropped or replaced rather than shown as N/A.

Questions for authors
  1. Can you report FVD (and ideally a reference-free consistency metric) as a function of rollout horizon — e.g., the 4/8/12 s buckets of Table 9, or per-block FVD over a longer synthetic rollout than the recorded 15 s? This is the key evidence for the long-horizon and memory claims.
  2. Can you add (a) a budget-matched ablation, (b) leave-one-out ablations of A / D / M, and (c) a cohort-level Independent vs. Correlated FVD?
  3. Can you provide a resolution/length-matched comparison, or at least FVD restricted to a subset with a shared output grid? Do all baselines receive the identical complete caption, or only a short text prompt?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.