acceptodds
Under review as a conference paper at ICLR 2027

SurgCast: Decoding Polar Transitions for Surgical State-Change Forecasting

Abstract

Surgical state-change forecasting predicts changes in activity states within a future window from past observations. Such forecasts can support instrument preparation and intraoperative coordination. Direct categorical decoding leaves transition timing and semantic effects implicit: timing determines whether a change falls within the forecast window, while semantic effects specify which activities are added or removed. We introduce SurgCast, a structured decoder that explicitly models these complementary factors using features from a frozen visual foundation model. TransitionCast parameterizes a learned transition representation in polar coordinates, decoding its magnitude into transition occurrence probabilities over future intervals and its direction into predictions of activity additions and removals. These predictions receive separate supervision and are composed with current-state evidence over the forecast window to construct four-category state-change probabilities. The resulting probabilities are fused with base predictions, after which StateRecast refines the fused logits through an additive residual learned with the first-stage predictor fixed. SurgCast achieves state-of-the-art performance with state mAP scores of 40.33% for surgical actions on CholecT50 and 29.55% for procedural steps on GraSP, exceeding SurgFUTR-TS results by 3.93 and 7.65 percentage points, respectively. On CholecT50, SurgCast improves state mAP over the corresponding baselines by at least 1.67 percentage points across all evaluated backbone types and sizes.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Surgical video understanding, Surgical state-change forecasting, Latent transition modeling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_JZvKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SurgCast, a structured framework for surgical state-change forecasting that predicts how surgical activities evolve from an observed video window to a future window. The task involves predicting four states for each activity of continuity, discontinuity, onset, and background. They are defined according to whether the activity occurs in the observed and future windows. TransitionCast introduces a polar parameterization that decomposes a learned transition representation into magnitude and direction, which are intended to capture transition timing and semantic changes, respectively. A State Composer then integrates these predictions with current-state recognition evidence to construct four-state probabilities. StateRecast refines the predictions through residual logit corrections while keeping the first-stage model frozen. Experiments on CholecT50 and GraSP demonstrate competitive performance.

Critical strengths
  • The paper addresses an important problem in surgical video understanding by explicitly modeling how activities change between observed and future windows. The proposed State Composer is interesting and looks meaningful by incorporating both current-state recognition and temporal transition predictions into a probabilistic manner.

  • The framework operates on frozen visual foundation model features. It looks applicable to different pretrained representations without requiring backbone fine-tuning.

  • Additional experimental results verify the effect of proposed decompositiion from additional model capacity and residual refinement.

  • The analysis clarify how multiple transitions are handled.

Critical weaknesses
  • The experiments evaluated downstream four-state classification performance instead of directly measuring whether the learned representations successfully capture these distinct factors.

  • Several implementation details including the estimation of the orthogonal basis, computation of subspace allocation weights, certain loss coefficients, residual refinement configuration were missing in the intial version of the paper.

  • It was unclear whether the improvements originate from the intended temporal-semantic decomposition or from additional classification flexibility.

  • The improvement on two-branch design looks not significant.

  • Additional temporal analysis indicates the magnitude representation captures transition occurrence intensity and temporal ordering instead of localizing transition time (It might be better to reduce the claim of the paper according to this).

Questions for authors
  • How can the authors demonstrate that the polar parameterization is essential to the improvements?
  • How does the model handle multiple activity transitions within a single forecasting window?
  • What motivates the different subspace aggregation strategies used during training and inference?
  • How robust are the improvements across datasets and forecasting settings?
Reviewer_bgh9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies surgical state-change forecasting: given the video up to time tt, predict whether each action/step in the next 4-second window belongs to continuity, discontinuity, onset, or background. The authors' key observation is that directly decoding frozen VFM features into four classes conflates when a change occurs (timing) with which activities start or end (semantic effect). SurgCast addresses this with: TransitionCast, which parameterizes a learned transition representation in polar coordinates vt=RtUtv_t=R_tU_t, decodes its magnitude into per-second transition intensities via a temporal decoder, and converts them into the probability of at least one change within the window using the survival probability St,ℓ=exp⁡(−∑λ)S_{t,\ell}=\exp(-\sum\lambda); a semantic decoder that maps the direction to per-activity begin/end predictions; a State Composer that combines timing, semantics, and current-state evidence from the recognition head into a four-class distribution, then applies a gated correction to the base logits; and a second-stage StateRecast that learns a zero-initialized additive residual while the first stage is frozen. Experiments on CholecT50 (actions) and GraSP (steps) with frozen DINOv3 report mAP 40.33 / 29.55, with consistent gains across three DINOv3 scales and VideoMAEv2. Overall, this is a well-executed and clearly written paper.

Critical strengths
  • Problem level: the model encodes the probabilistic structure of the task. St,L=exp⁡(−Rt)S_{t,L}=\exp(-R_t) makes the total magnitude directly correspond to the probability of at least one change in the window, and the State Composer turns timing, semantics, and current-state evidence into a valid four-class distribution. This is more structured than a direct four-way head and provides inspectable intermediate quantities for analysis. It matters for my overall assessment because it shows the authors are not merely stacking modules; they are trying to write the task's probabilistic semantics into the model.

  • Method level: matched-backbone comparisons and cross-dataset consistency. Relative to an MLP baseline on the same frozen backbone, the gains are +1.67/+2.21/+2.36 (DINOv3-S/B/L), and +3.36 on VideoMAEv2; Tables 4/5/6/7 separately ablate TC/SR, dir/mag, βdir\beta_{dir}, and KK. This kind of same-backbone comparison is more informative than a single SOTA number, and it is the main reason I consider the paper a weak accept.

Critical weaknesses
  • Method level: the gain from decoupling is not yet cleanly attributed. In Table 5, training direction alone is 0.76 mAP below the baseline (37.21 vs 37.97); direction + magnitude improves over magnitude-only by only 0.25 (39.09 vs 38.84), while the seed standard deviation is about 0.4–0.5, so this difference is currently within noise. Table 6 defines "synergy" as ΔTC−ΔDir−ΔMag\Delta_{TC}-\Delta_{Dir}-\Delta_{Mag}; as βdir\beta_{dir} increases, direction-only drops from 38.07 to 37.21 and synergy rises from 0.01 to 1.01. The authors candidly note that this mainly reflects the decline of direction-only rather than an increase in the joint gain. That observation is honest, but it also means the metric cannot currently support the "decoupling works" claim. Moreover, in Table 7, K=1K=1 already reaches 40.27, essentially the same as K=4K=4 (40.33) and K=8K=8 (40.40), so the orthogonal-subspace design is not a major source of gain either. Taken together, the evidence points to the magnitude/timing branch plus a generic residual correction, rather than the polar structured representation as a whole. This does not reduce the engineering value, but the main narrative could be sharpened.
  • Method level: the distinction between capacity/backbone and algorithmic gain could be clearer. The DINOv3-L MLP baseline already reaches 37.97, above SurgFUTR-TS at 36.4; the real gain of SurgCast over the same-backbone baseline is +2.36, not the +3.93 emphasized in the abstract. The authors are transparent about this in Table 3 and the main text, but the abstract and introduction still make it easy to read the backbone advantage as a method advantage; I would recommend leading with the same-backbone +2.36. In addition, TC+SR adds about 1M parameters over direct decoding, and StateRecast alone already gives +0.89 (Table 4). A parameter-matched control—a larger MLP head with the same parameter count and the same two-stage schedule—would therefore be very informative, as it would separate "structured inductive bias" from "extra capacity."
Questions for authors
  • Could you add a parameter-matched control: a larger MLP head with the same number of parameters as TC+SR, the same two-stage training schedule, and the same frozen features, trained to predict the four classes directly? What is its mAP? This would directly answer whether the gain comes from structure or capacity.
  • Direction-only is below the baseline, and joint is only +0.25 over magnitude-only. Could you report magnitude-only vs. joint with at least five seeds and a paired test (or paired bootstrap)? If the difference is not significant, would you consider narrowing the claim from "polar decoupling" to "timing-aware hazard + residual calibration"? I think this would actually make the paper more robust and convincing.
  • The synergy definition in Table 6 increases mechanically as direction-only degrades. Could you provide evidence that does not depend on this definition—for example, fixing the direction-only training strength and only changing whether it is composed with magnitude—to show that the semantic branch provides independent information?
  • Temporal supervision uses the first change of the entire activity set, while the four labels are per-activity. If an activity onsets at its second change within the window, or different activities start at different seconds, how is a single global change time represented? Could this explain why onset/discontinuity AP remain at ~19/21? A discussion of this limitation would help readers understand the method's applicability.
  • StateRecast is trained on frozen first-stage predictions and uses a consistency loss and a residual penalty. Could you report the distribution of residual norms and the result without LresL_{res}? Is it learning new temporal structure, or mainly recalibrating the confidence of the base logits?
  • In Table 3, the SurgFUTR-TS student is reproduced with only 32/35 training videos (footnote 1). Could you train SurgCast on the same 32 videos for a fully aligned comparison? This would make Table 3 more convincing.
  • What exactly is the "approximately one million trainable parameters over direct decoding"? What is the parameter count of the baseline head? This would help readers assess the capacity difference.
Reviewer_6ageOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SurgCast forecasts activity state changes from past surgical video using frozen visual features. TransitionCast separates transition timing and semantic effects through magnitude and direction, then combines them with recognition evidence to predict four state-change categories. StateRecast further refines these predictions. Experiments on CholecT50 and GraSP show consistent improvements over direct prediction.

Critical strengths
  1. The improvements are supported by comparisons using the same frozen features, multiple backbone sizes, and three random seeds. The gains also hold on reproduced SurgFUTR features, showing that they are not limited to DINOv3.

  2. The State Composer provides a clear way to connect transition predictions with window-level activity states. The component ablations support the usefulness of TransitionCast and StateRecast, both separately and together.

Critical weaknesses
  1. The specific benefit of the polar parameterization remains unclear. The single-subspace variant performs similarly to the default configuration (40.27 versus 40.33 mAP), so the results provide limited evidence for the additional subspace structure. A comparison with ordinary timing and semantic heads, keeping the supervision and composition unchanged, would better establish the value of the polar design.

  2. Transition timing is a central part of the method, but evaluation only reports state classification over a fixed four-second future window. These results establish better state forecasting, while leaving the quality of the predicted first-change timing unclear. A direct timing evaluation would strengthen this part of the contribution.

Questions for authors
  1. What advantage does the polar parameterization provide over ordinary timing and semantic heads with the same supervision and StateComposer? Please also clarify how the subspace allocation weights are computed.

  2. Can you report the quality of the predicted first-change time distribution, for example through held-out likelihood, calibration, or timing error?

Reviewer_pGokOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SurgCast, a structured decoder for surgical state-change forecasting using frozen visual representations. TransitionCast separates transition timing and semantic effects through magnitude and direction, then composes these predictions with recognition evidence into four-state probabilities. A second-stage residual module further refines the predictions. Experiments report improvements on surgical action and procedural-step forecasting.

Critical strengths
  1. A task-relevant structured formulation. Separating when a transition occurs from which activities begin or end provides a reasonable inductive bias. The State Composer explicitly connects these factors to window-level state-change categories, rather than treating them as unrelated auxiliary predictions.

  2. Consistent improvements under matched visual backbones. The method improves over direct decoding across several backbone sizes and representation sources. These comparisons provide evidence that the proposed decoder contributes beyond the choice of a stronger visual encoder.

Critical weaknesses
  1. Unclear necessity of the polar parameterization. Transition type and timing may be strongly related, yet the paper does not sufficiently explain why magnitude and direction provide a particularly suitable factorization. The When–What interpretation is intuitive, but the distinctive benefit of the unit-norm direction constraint and the resulting inductive bias remains unclear.

  2. Insufficiently explained coupling and supervision. Eq. (4) weights subspace semantic predictions by transition intensity, so the aggregated semantic output also depends on temporal allocations. Eq. (8), however, supervises a uniform average. The paper does not sufficiently explain this asymmetry or compare alternative aggregation strategies, leaving the degree and benefit of the claimed decoupling unclear.

  3. Limited isolation and validation of the central design. A conventional two-branch baseline matched for inputs, supervision, capacity, and training budget is missing, and final state AP/mAP does not independently validate timing or semantic predictions. In Table 5, joint training adds only 0.25 points over magnitude alone, while direction alone loses 0.76 points.

Questions for authors
  1. What specific advantage does polar parameterization offer over a conventional timing/semantics two-branch architecture? Could the authors compare them with matched features, auxiliary supervision, capacity, and training budget, and separately isolate the unit-norm direction constraint? This would clarify whether the gains come from the polar representation rather than simply separating two prediction tasks.

  2. Could the authors independently evaluate first-change timing, and report begin/end AP or F1 at the first observed transition? How are these temporal and semantic targets constructed from the original annotations under 1 fps sampling, and how are windows without an observed change handled? Such results would directly test the intended roles of magnitude and direction.

  3. Why does State Composer use intensity-weighted aggregation in Eq. (4), while semantic supervision uses a uniform average in Eq. (8)? Is uniform supervision intended to train all subspaces equally regardless of transition intensity? Have the authors tested intensity-weighted semantic supervision or other aggregation strategies, and how do these choices affect final state forecasting?

1 comment
AuthorsOpenReview ↗

Sincerely Looking Forward to Your Feedback

Dear Reviewer pGok,

We have uploaded a revised version of the paper. Below we briefly point out where it addresses your three main concerns:

  • Necessity of the polar parameterization (Q1, W1, W3). The matched two-branch comparison and the unit-norm ablation from our response (Part 1) have now been moved from Appendix B into Section 4.4 (now Table 6, previously Table 8). Section 1 now clarifies that the two factors are related but can differ in predictability, and Section 3.2 explains why the direction is constrained to unit norm.
  • Independent evaluation of timing and semantics (Q2, W3). The first-change evaluation of the magnitude and direction, together with how the temporal and semantic targets are constructed, is in our response (Part 2). We will present the evaluation at an appropriate place in the paper.
  • Eq. 4 vs. Eq. 8 (Q3, W2). Section 3.4 now explains, where Eq. 8 is introduced, why the semantic supervision uses a uniform average. The comparison with consistent aggregation variants is in our response (Part 3).

Thank you again for your time. If our responses and the revision have addressed your concerns, we would be grateful if you would consider updating your rating. If anything remains unclear, we would be happy to clarify it before the discussion period ends.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.