acceptodds
Under review as a conference paper at ICLR 2027

Controlled Transfer and Sparse Expert Routing for Surgical Foundation Models

Abstract

Adapting surgical workflow recognition to a new procedure requires choosing a visual representation with limited labeled videos. Existing comparisons often change the encoder, temporal model, and training recipe together, leaving it unclear which representations transfer well and when source training reduces annotation needs. We propose SurgFM-Control, a procedure-aware framework that combines timestamp-aligned frozen-feature caching, a common temporal learner, and adaptation to native procedure labels. This design evaluates representation transfer jointly across surgical procedures, label budgets, and visual perturbations while reusing cached features for downstream experiments. Across three datasets, six frozen encoders reveal complementary strengths: VideoMamba achieves 74.4% and 52.9% macro-F1 on Cholec80 and Microanastomosis, respectively, while DINOv2 reaches 68.8% on AutoLaparo. With one labeled target video, source-initialized temporal learning improves macro-F1 by 5.7% points on average over target only training. DINOv2 also retains 88.7% of clean macro-F1 across the evaluated pixel corruptions, while training sample thinning preserves performance more effectively for long phases than for rapid action transitions. These findings establish a practical basis for matching frozen representations and adaptation strategies to a procedure's annotation budget and temporal structure.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Surgical foundation model, surgical video understanding, transfer learning, temporal action segmentation, mixture-of-experts

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_92eJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents SurgFM-Control, a controlled evaluation framework for studying the transferability of frozen visual representations in surgical workflow recognition. The framework aligns encoder features to a common temporal grid and evaluates them using a shared temporal learner across surgical procedures. The study investigates representation performance under multiple conditions, including in-domain recognition, limited annotation budgets, cross-procedure initialization, temporal subsampling, and visual/sequence corruptions.

Critical strengths

This paper provides a commendable alternative perspective for evaluating foundation model representations. It attempts to isolate representation quality through a controlled downstream protocol. Evaluating frozen representations under a shared temporal learner across procedure type, annotation budget, temporal density, and corruption provides a complementary perspective to conventional leaderboard-style comparisons.

Critical weaknesses

1. The sparse expert router is presented as a main contribution, but it is barely discussed in the main paper and does not perform well. “Sparse Expert Routing” appears in the title, but the router is only briefly described in the appendix and is not part of the main paper discussion. More importantly, it does not clearly outperform simple baselines: the uniform ensemble performs best on Cholec80. This makes it unclear why routing is emphasized in the title.

2. The benchmark does not include enough recent surgical foundation models. Most evaluated models are general-purpose or relatively established encoders, such as DINOv2, MAE, VideoMAE, TimeSformer, and VideoMamba. For a paper focused on evaluating surgical foundation representations, more recent surgical-specific models should be included. For example, LemonFM[1] and SurgeNetXL[2] are directly relevant and has already been evaluated on datasets such as Cholec80 and AutoLaparo.

[1] LEMON: A Large Endoscopic MONocular Dataset and Foundation Model for Perception in Surgical Settings. CVPR 2026

[2] Scaling up self-supervised learning for improved surgical foundation models. MedIA 2026

Questions for authors

Could the authors clarify the cross-procedure parameter transfer protocol?

The paper says that all parameters with matching names and shapes are copied from the source model. However, because distinct surgical procedures typically have disjoint label spaces, this strategy risks copying class-specific prediction weights across semantically unrelated targets. To isolate whether performance gains stem from generalizable temporal dynamics rather than arbitrary head initialization, could the authors report an encoder-only transfer baseline (i.e., transferring representations while randomly reinitializing the task-specific classification head)?

2 comments

We thank the reviewer for the feedback. The revision clarifies the contribution, improves the presentation, and makes the transfer protocol explicit.

  1. Routing and presentation. We removed routing from the title and explicitly labeled Appendix D an auxiliary negative result (line 689). The revised discussion acknowledges that the tested router does not outperform the strongest static control (lines 728–733): averaging leads on Cholec80, while routing ties the best expert on microanastomosis. All experts are evaluated, so top-2 weighting provides no encoder or tower computation saving (lines 699–701). We also added Figure 2 to summarize recognition, transfer, and temporal-thinning results (lines 336–339).

  2. Recent surgical foundation models. LemonFM and SurgeNetXL are included. The revised related work now discusses both models and clarifies the scope of the evaluated checkpoint set (lines 130–137). Our framework provides a common protocol for comparing frozen representations, while the reported findings concern the model actually evaluated. LemonFM and SurgeNetXL are not included in the reported experiments as they are new and we do not need to run new experiments at this phase. Their published results cannot be inserted as matched comparisons because downstream heads, dataset splits, and supervision may differ; these distinctions are now explicit in lines 133–137.

  3. Cross-procedure parameter transfer. The experiment measures whether source initialization improves subsequent target learning under the same frozen features and labeled target videos as the target-only control. Compatible temporal-head parameters, including class-dependent layers, are transferred and then adapted to native target labels. No semantic correspondence between procedures’ classes is assumed. This protocol is explicit in lines 253–267.

If “encoder-only transfer” means retaining the frozen pretrained encoder while randomly initializing the entire temporal head, that is the existing target-only baseline (lines 255–257). To address the reviewer’s concern, we now provide the additional control that transfers compatible class-independent parameters while resetting every stage classifier and class-indexed refinement input projection.

DirectionkTarget-onlyAll-compatible initializationClass-dependent layers reset
Cholec80 → AutoLaparo120.6 ± 2.726.2 ± 2.325.3 ± 2.3
Cholec80 → AutoLaparo867.4 ± 1.564.8 ± 1.862.7 ± 1.6
AutoLaparo → Cholec80123.3 ± 3.031.5 ± 2.630.4 ± 2.4
AutoLaparo → Cholec80874.2 ± 1.672.3 ± 1.972.1 ± 1.2

At k=1, resetting class-dependent layers retains macro-F1 gains of 4.7 and 7.1 macro-F1 points over target-only training. The low-label benefit therefore persists without copying class-specific prediction weights, supporting the usefulness of transferred class-independent head parameters.

At k=8, both initialization strategies remain below target-only training. Resetting class-dependent layers does not eliminate negative transfer, reinforcing the budget-dependent operating regime identified in the original analysis (lines 422–426). The original results characterize complete source initialization; this additional ablation separates that finding from the contribution of copied class-dependent layers.

Thank you for the clarifications and additional experiments. I appreciate the improvements made in the revision.

However, I would still like to see direct comparisons with SurgeNetXL and LemonFM under the same evaluation protocol.

Reviewer_TM95OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SurgFM-Control caches features from six public encoders and trains one MS-TCN head for each procedure. The authors describe this as an extension of Spencer et al., who compared encoders on cataract video while keeping an MS-TCN++ head fixed.

Critical strengths

The framework stops asking which encoder is best in general. It asks how that ranking moves when the procedure changes, when labels are scarce, and when the workflow is a long phase or a short action.

Critical weaknesses
  1. Section 3 states that the framework does not require a new backbone, and the temporal head follows an MS-TCN-style design. Spencer et al. is itself a data-efficiency study of frozen foundation models under a fixed temporal head, so the comparison principle and the label-budget axis are theirs. The additions here are more procedures, cross-procedure initialization, temporal density, and synthetic corruption, and I do not see a new method in this paper.
  2. Transfer copies every parameter with a matching name and shape (Section 3.4), so the class layer is copied whenever source and target have the same number of classes, as for Cholec80 and AutoLaparo with seven phases each, even though the labels mean different things. The paper does not separate the effect of the copied class layer from that of the temporal layers. The benefit is also limited to the smallest budget: source initialization gains +5.7 macro-F1 at k=1 but −1.2 on average at k=8, where it wins only 17 of 44 encoder-direction cells (Section 4.4), while the abstract reports only the k=1 figure.
  3. On Cholec80 the router in the title scores 68.3 macro-F1, against 68.8 for the best single expert and 71.7 for averaging. On microanastomosis it ties the best single expert (53.6) and beats averaging (51.8), so, as Appendix D states, the learned gate never dominates both static controls. Every expert is still evaluated, so top-2 sparsity does not reduce computation. Routing, which is half of the title, is therefore not supported as a contribution.
  4. The conclusion says the ranking depends on the procedure, the label budget, and the temporal scale. The tables do name winners per dataset (VideoMamba on Cholec80 and microanastomosis, DINOv2 on AutoLaparo and for pixel robustness), but the paper gives no rule for choosing an encoder for a new procedure before it is labeled.
  5. The visual corruptions are appearance perturbations: Gaussian blur with contrast loss labeled as smoke, saturated white ellipses labeled as specularity, and downsampling. The sequence corruptions drop frames, shift labels, and add Gaussian noise to cached features. The paper itself notes that the smoke model is "not a physical smoke simulator" and that these synthetic perturbations cannot represent clinical variability, so the robustness ranking says little about failures in real surgical video.
Questions for authors
  1. Could the authors report cross-procedure transfer with the class-dependent layers re-initialized, so the effect of the copied temporal layers is separated from that of the copied classifier?
  2. Since the router does not beat the static controls and is not among the three listed contributions, would the authors consider removing it from the title or presenting it explicitly as a negative result?
3 comments

As for my weakness 1, the study by Spencer et al. already includes the controlled comparison of frozen encoders, conducted within a setup that employs a fixed temporal head. My concern is that the core workflow largely reuses existing components; the study’s extensions focus mostly on comparisons across diverse evaluation setups, for example, evaluation protocols, annotation budgets, transfer settings, temporal density, and data corruption scenarios. In this case, the work is more like an empirical study, and the crucial question is whether these additional analyses provide unique and substantive insights going beyond existing comparative frameworks.

We thank the reviewer for the feedback. We confirm and agree that the contribution is an empirical framework and study.

  1. The revised introduction explicitly credits Spencer et al. for fixed-head comparison, feature caching, and the label-budget principle (lines 78–98). Our contribution examines how source initialization and temporal sampling behave across procedures with different native labels and workflow structures. Two findings provide actionable guidance beyond identifying encoder winners:
  • Source initialization is most useful under extremely limited target supervision; its benefit can reverse as more target videos become available (lines 422–426).
  • Training-sequence thinning has substantially different consequences across workflow scales: stride-10 retention is 0.85 on Cholec80 and 0.95 on AutoLaparo, versus 0.56 on microanastomosis, while reducing case coverage substantially degrades performance (lines 404–413). The ablation below further clarifies the first finding: low-label gains persist without copying class-dependent layers, whereas resetting those layers does not eliminate negative transfer at the larger budget. These observations provide guidance on when source initialization is useful and when temporal thinning preserves recognition performance. Figure 2 summarizes the original comparisons (lines 336–339).
  1. Class-dependent transfer and the requested ablation. We agree that matching shapes do not establish semantic correspondence. We now report the requested comparison, resetting all class-dependent layers, including stage classifiers and refinement input projections. | Direction | k | Target-only | All-compatible initialization | Class-dependent layers reset | |---|---:|---:|---:|---:| | Cholec80 → AutoLaparo | 1 | 20.6 ± 2.7 | 26.2 ± 2.3 | 25.3 ± 2.3 | | Cholec80 → AutoLaparo | 8 | 67.4 ± 1.5 | 64.8 ± 1.8 | 62.7 ± 1.6 | | AutoLaparo → Cholec80 | 1 | 23.3 ± 3.0 | 31.5 ± 2.6 | 30.4 ± 2.4 | | AutoLaparo → Cholec80 | 8 | 74.2 ± 1.6 | 72.3 ± 1.9 | 72.1 ± 1.2 |

At k=1, resetting class-dependent layers retains macro-F1 improvements of 4.7 and 7.1 points over target-only training, compared with 5.6 and 8.2 for all-compatible initialization. Thus, the low-label benefit is not explained solely by copying classifiers: source-trained class-independent head parameters retain substantial utility.

At k=8, both initialization strategies remain below target-only training. Resetting class-dependent layers therefore does not remove negative transfer. This strengthens the budget-dependent interpretation while avoiding attribution of the entire effect to copied class-specific weights.

  1. Routing and title. Yes, we accepted this suggestion. Routing has been removed from the title, and Appendix D is explicitly labeled an auxiliary negative result (line 689). The revised discussion states that the tested router does not outperform the strongest static control (lines 728–733): on Cholec80, it achieves 68.3 macro-F1 versus 68.8 for the best expert and 71.7 for averaging; on microanastomosis, it ties the best expert at 53.6 and exceeds averaging at 51.8. Appendix D.1 also states that all experts are evaluated, so sparse weighting provides no encoder or tower computation saving (lines 699–701).

  2. Encoder selection. The framework supports comparison using labeled target validation data; it does not predict the best encoder before target labeling. This distinction is explicit in lines 679–680 and reflected in the limitations and conclusion (lines 515–516 and 533–535).

  3. Robustness. We revised the descriptions to name the actual operations: blur plus intensity attenuation, white-ellipse overlays, label shifts, and generic feature noise (lines 304–310). The retention findings concern these specified perturbations, and the limitations distinguish this evidence from clinical variability (lines 520–522).

Thank you for the detailed response and revisions. I have carefully reviewed the revised manuscript and the authors' responses. I acknowledge the additional ablation studies, as well as the clarifications regarding the routing mechanism and the scope of the claims.

My major concern remains the strength of the empirical contribution. While the added analyses have some merit, they do not offer sufficiently unique or substantive insights to justify the paper's acceptance, especially when compared to existing controlled evaluation frameworks. Consequently, I am maintaining my current score.

Reviewer_L4NgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a novel method for surgical transfer SurgFM-Control, a procedure-aware framework that combines timestamp-aligned frozen-feature caching, a common temporal learner, and adaptation to native procedure labels.

Critical strengths
  • novelty: the paper presents a new transfer learning framework for surgical models called SurgFM control.
  • Thorough experimentation of SrgFM with different models and settings
Critical weaknesses
  • i was not able to figure out how if SurgFM control is compared with other methods around transfer learning?
  • No Code provided.
  • The tables are presentaed but hard to interpret them, maybe use figures?
Questions for authors
  • Is the. method only generalisable to surgical domain? if no then i would like to see some experiments separately with classical ML datasets, if yes then is ICLR a right venue for a paper like this? I suppose MICCAI might be better
  • Is there no comparable method published in CVPR or ICCV for related domain?
1 comment

We thank the reviewer for the feedback.

Comparison with related methods. SurgFM-Control compares six frozen encoders under a common temporal learner and evaluates source-initialized adaptation against target-only training using the same labeled target videos (lines 253–257). The revised introduction clarifies the distinction from Spencer et al.: our framework extends controlled evaluation across procedures, native label spaces, and temporal scales (lines 78–98). We also expanded the discussion of relevant CVPR/ICCV work, including MS-TCN, SKiT, and LemonFM, alongside SurgeNetXL (lines 130–137).

Generality and venue relevance. The framework requires timestamped features and sequence labels rather than surgery-specific architecture, although the experiments concern three surgical procedures. The broader representation-learning question is how transferability changes with supervision and temporal structure. The revised conclusion connects these findings to representation selection using target validation data (lines 526–535).

Code and readability. The supplementary attachment contains the anonymized implementation: README.md, src/, and configs/. The README explains dependencies, execution, and release scope. We also added Figure 2 to summarize encoder performance, transfer gains, and temporal thinning, making the detailed tables easier to interpret (lines 336–339).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.