- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents SurgHiCo, a surgical video–language pretraining framework that builds local, intermediate, and long-horizon representations in three stages. After training a local encoder, it progressively freezes earlier visual modules and trains Transformer composers using captions at corresponding temporal scales. Long-horizon captions are constructed by summarizing and concatenating consecutive intermediate captions from SurgLaVi-β. The framework is evaluated through zero-shot phase recognition and linear probing on five datasets. I have the following comments.
Critical strengths
1、A practical approach to extending frozen surgical video representations. The staged design allows the higher-level modules to train on cached features without repeatedly decoding videos or updating the backbone. Evaluation across five datasets, together with caption- and input-matched mean-pooling controls, provides useful evidence that the complete approach improves over simple aggregation.
Critical weaknesses
1、The experiments do not establish the contribution of the proposed hierarchy. HierVL (CVPR 2023) already learns representations at different temporal scales using self-attention aggregation and language supervision. SurgHiCo introduces a three-level construction and progressively frozen training, but the experiments do not isolate the value of these choices. The main control in Table 2 uses parameter-free mean pooling, while SurgHiCo adds trainable composers, staged optimization, and text adapters. There is no comparison with a non-hierarchical Transformer aggregator of comparable capacity, or an ablation of progressive freezing. The reported improvement therefore leaves the paper’s main methodological claim unresolved.
2、The baseline comparisons leave uncertainty about the reported advantage. Table 1 compares SurgHiCo’s 256-frame input with published results using 16-frame inputs. Section 4.2 acknowledges this difference, but the headline comparison still relies on these unequal settings. The controlled experiments also need a stronger baseline check: the reproduced flat F/M/C model averages 39.47%, compared with 44.04% for the quoted SurgCLIP-β result. The discrepancy is particularly large on StrasBypass, at 16.58% versus 31.24%. The paper does not explain this gap. Until the reproduction is reconciled and the strongest baseline is evaluated under comparable input conditions, the magnitude of the improvement remains difficult to assess.
Questions for authors
1、How does SurgHiCo compare with a single Transformer that aggregates the same 16 local features, using the same caption supervision and a comparable parameter budget?
2、What happens when the hierarchy is trained without progressively freezing earlier visual modules? Is there evidence that freezing contributes to the results independently of the added composers?
3、What accounts for the gap between the reproduced flat F/M/C baseline and the quoted SurgCLIP-β results, particularly on StrasBypass? Can the released SurgCLIP-β checkpoint be evaluated with the same prompts, test videos, and long-context input used for SurgHiCo?
4 comments
Response to Reviewer NBPh: Capacity-Matched Flat Transformer
Comparison with a Capacity-Matched Flat Transformer
To investigate whether the improvements stem from hierarchical modeling rather than increased model capacity, we conducted additional experiments with a capacity-matched flat Transformer.
Specifically, we replace the hierarchical composers (2L Mid + 2L Workflow) with a single 4-layer Transformer that directly aggregates local visual features through global self-attention, without intermediate hierarchical representations. We evaluate two training strategies: Flat-Joint, which jointly optimizes the model using Mid and Workflow supervision, and Flat-Staged, which sequentially applies Mid and Workflow supervision. All methods share the same frozen Stage1 features, caption supervision sources, 16 local features at inference, and evaluation protocol. The trainable parameter counts are nearly identical (28.594M vs. 28.590M; 0.016% difference).
Table R1. Comparison with capacity-matched flat Transformers (video-averaged accuracy, %, seed 42).
| Model | Cholec80 | HeiChole | Bern | Stras | AutoLaparo | Avg |
|---|---|---|---|---|---|---|
| Flat-Joint (4L) | 52.75 | 55.08 | 33.93 | 21.30 | 55.31 | 43.67 |
| Flat-Staged (4L) | 49.86 | 49.14 | 38.07 | 29.08 | 51.87 | 43.60 |
| SurgHiCo (2L+2L) | 60.28 | 63.04 | 39.02 | 30.14 | 59.55 | 50.41 |
As shown in Table R1, SurgHiCo outperforms both flat baselines across all five datasets, improving average accuracy by 6.7 pp over Flat-Joint and 6.8 pp over Flat-Staged. Notably, the two flat variants achieve similar average accuracy despite their different training strategies.
These results show that simply increasing Transformer capacity and applying the same multi-scale language supervision, either jointly or sequentially, does not reproduce SurgHiCo's performance. This provides empirical support for the effectiveness of our progressive hierarchical learning framework beyond model capacity alone.
Response to Reviewer NBPh: Progressive Freezing and SurgCLIP Reproduction
Effect of Progressive Freezing
To isolate the contribution of progressive freezing from the hierarchical architecture, we conducted an additional ablation in which the Mid composer remains trainable during Workflow-level training.
Both variants retain the identical hierarchical architecture (2L Mid + 2L Workflow), Stage2 initialization, caption supervision, and training schedule. The only difference is whether the Mid composer is frozen during Stage3, allowing us to evaluate the effect of freezing independently of the additional composers.
Table R2. Ablation of progressive freezing (video-averaged accuracy, %).
| Model | Cholec80 | HeiChole | Bern | Stras | AutoLaparo | Avg |
|---|---|---|---|---|---|---|
| SurgHiCo (Unfrozen) | 55.89 | 53.54 | 35.21 | 24.06 | 50.49 | 43.84 |
| SurgHiCo (Frozen) | 60.28 | 63.04 | 39.02 | 30.14 | 59.55 | 50.41 |
As shown in Table R2, progressive freezing improves average accuracy from 43.84% to 50.41% (+6.57 pp), with improvements across all five datasets. These results provide direct evidence that freezing contributes to the performance of SurgHiCo beyond the addition of hierarchical composers.
Importantly, our primary motivation remains progressive representation learning across temporal scales. Freezing serves as a complementary training strategy that preserves previously learned intermediate representations while enabling efficient long-horizon training through cached features. We will clarify this distinction in the revised manuscript.
SurgCLIP-β Baseline Reproduction and Fair Comparison
We acknowledge that the discrepancy in baseline reproduction and the difference in temporal input lengths may affect the assessment of SurgHiCo's performance gains. To address both issues, we conducted additional experiments using the officially released SurgCLIP-β checkpoint and matched temporal inputs.
(1) SurgCLIP-β Baseline Reproduction.
During our research, we made multiple attempts to reproduce SurgCLIP-β but were unable to fully recover the performance reported in the original paper. The reproduced results presented in our submission represent the best performance obtained across our reproduction attempts. Nevertheless, a gap remains, and we have not identified its definitive cause.
To avoid relying on our independently reproduced checkpoint, we directly evaluated the officially released SurgCLIP-β checkpoint. Under our downstream evaluation protocol, the released checkpoint achieved 44.12% average accuracy across five datasets, closely matching the 44.04% reported in the original paper.
Notably, on StrasBypass, the released checkpoint achieves 31.01%, closely matching the reported 31.24%, whereas our independently reproduced flat baseline achieves 16.58%. Although the cause of the reproduction gap remains unclear, this confirms that the released checkpoint closely recovers the reported performance under our evaluation protocol.
We further used the released checkpoint as SurgHiCo's frozen Stage1 encoder, while keeping the Stage2 and Stage3 progressive hierarchical training procedure unchanged. This configuration achieved 53.86% average accuracy, compared with 50.41% using our independently trained Stage1 encoder.
These results demonstrate that SurgHiCo's effectiveness does not depend on our independently reproduced Stage1 checkpoint and that the framework can effectively extend the officially released pretrained visual representations.
Response to Reviewer NBPh: Fair Comparison under Matched Long-Context Input
Fair Comparison under Matched Long-Context Input
To further ensure a fair comparison, we evaluated the official SurgCLIP-β checkpoint under the same downstream evaluation protocol, using the same prompts and test videos. We additionally constructed a long-context Mean Pooling baseline to distinguish the benefits of increased temporal coverage from those of hierarchical composition.
Specifically, we extract 16 local visual tokens from a centered 256-second temporal window, with each local clip containing 16 sampled frames (256 frames in total). We compare two aggregation strategies using the official SurgCLIP-β checkpoint:
- SurgCLIP-β + Mean Pooling: Averages the 16 local visual tokens into a single representation.
- SurgHiCo: Applies the proposed hierarchical composition to the same 16 local visual tokens.
For reference, we also report the official checkpoint's original 16-frame evaluation result.
Table R3. Zero-shot comparison using the official SurgCLIP-β checkpoint under different temporal contexts (video-averaged accuracy, %).
| Dataset | Official (16F) | Mean (256F) | SurgHiCo (256F) |
|---|---|---|---|
| Cholec80 | 57.46 | 64.78 | 61.70 |
| HeiChole | 57.72 | 57.27 | 60.50 |
| Bern | 18.12 | 16.98 | 44.50 |
| Stras | 31.01 | 32.12 | 43.20 |
| AutoLaparo | 56.29 | 58.45 | 59.40 |
| Average | 44.12 | 45.92 | 53.86 |
As shown in Table R3, simply extending the temporal input and applying mean pooling improves the average accuracy from 44.12% to 45.92%. However, under the same 256-frame input context, SurgHiCo achieves 53.86%, outperforming Long-context Mean Pooling by 7.94 percentage points.
Notably, SurgHiCo achieves higher accuracy on four out of five datasets, with particularly substantial improvements on BernBypass and StrasBypass. Although Mean Pooling performs better on Cholec80, the overall results demonstrate that the improvements cannot be attributed simply to increased temporal coverage.
Together with the official checkpoint evaluation in our previous response, these experiments demonstrate that SurgHiCo can effectively extend the released SurgCLIP-β representations and achieve substantial improvements under matched long-context evaluation.
We sincerely thank the reviewer for these constructive suggestions, which have helped us strengthen the experimental validation and clarify the contributions of SurgHiCo. We are currently preparing the revised manuscript and incorporating these additional controlled comparisons and ablation results into the corresponding experimental sections.
Response to Reviewer NBPh: Novelty Relative to HierVL
Clarification of Novelty Relative to HierVL
We thank the reviewer for highlighting HierVL as relevant prior work. Although both methods explore hierarchical video–language learning, their research motivations differ. HierVL aims to incorporate long-term activity intent through video-level summaries to enrich local action and global video representations. In contrast, SurgHiCo addresses the limitations of clip-based VLP in representing long-horizon surgical workflows. Surgical procedures exhibit an inherent temporal hierarchy, where individual maneuvers form procedural steps, and steps collectively constitute longer surgical processes. Understanding these workflows requires representations that capture temporal relationships beyond isolated clips.
A related challenge in multi-scale video–language learning is that fine-grained visual details and abstract procedural semantics impose different representational demands. Jointly updating a shared visual encoder with supervision across temporal scales may therefore introduce cross-scale optimization interference.
HierVL addresses the risk of forgetting local representations through alternating training between short- and long-term objectives. SurgHiCo takes a different approach: it progressively freezes previously learned visual modules and employs lightweight composers to explicitly model and combine lower-level visual features into longer-range procedural representations, without repeatedly updating preceding visual modules.
Our additional controlled experiments support this design choice. Under matched trainable parameter budgets, SurgHiCo outperforms flat Transformer aggregation baselines. Furthermore, our freezing ablation shows that preserving the learned Intermediate Composer during long-horizon training achieves higher average accuracy than allowing it to be updated (50.41% vs. 43.84%).
This framework also offers two practical advantages. First, it retains local, intermediate, and long-horizon representations for independent or combined downstream use. Our linear-probing experiments show that combining all three scales outperforms the strongest individual scale across three datasets. Second, once local features are cached, higher-level composers require only a few minutes of additional training, without repeatedly processing raw videos or updating the expensive visual backbone.
The key contribution of SurgHiCo lies in its progressive hierarchical composition and stage-wise freezing strategy, enabling efficient construction, preservation, and reuse of multi-scale video–language representations for long-horizon surgical workflow understanding.