- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents RIVET for surgical triplet recognition, with a focus on rare classes. It uses component–triplet relations, historical prototypes, and spatial queries to refine predictions. The authors also introduce SinusTriplet-95: 324 clips from 15 patients covering 95 triplet classes. On a fixed patient-level split, triplet AP reaches 47.47% with a single model and 50.80% with an ensemble. Results on CholecT45 and the ablation study provide further evidence of the method’s effectiveness.
Critical strengths
-
The paper brings triplet recognition to sinus endoscopy, a useful setting beyond cholecystectomy. Narrow views, occlusion, and similar-looking tissues make this a challenging task.
-
SinusTriplet-95 contributes data from two centers with surgeon annotations for 95 triplet classes. Despite its small patient cohort, it provides a useful starting point for studying rare interactions in sinus surgery.
Critical weaknesses
1. The tail-class split and evaluation coverage need clarification.
Sec. 3.2 defines head and tail classes by frame frequency, with the top K_IVT classes assigned to the head. Appendix A.2 uses a different criterion for evaluation: the 50 classes with at most five annotated segments. What is K_IVT, and how much do the two sets overlap? It is also unclear whether the frequency counts come from the training split only.
The dataset has 15 patients. An exact patient-level 6:2:2 split would mean nine for training and three each for validation and testing. Given that 50 classes have at most five segments in total, their distribution across patients and splits matters. The paper does not give per-class positive patient or segment counts, or explain how classes without test positives are handled in AP.
Only one patient split is reported for SinusTriplet-95, without uncertainty estimates. Many correlated frames from a few patients cannot establish whether the gains hold across patients. Repeated patient splits or patient-level uncertainty estimates would make the results more convincing.
2. The ablations show gains, but leave their source unclear. Several details needed for reproduction are missing.
SR adds both a taxonomy-based reference and a learned class-specific residual b_r(h) in Eq. 3. There is also relation supervision in Eq. 10. Table 2 does not separate these effects or state which supervision terms are retained in each ablation.
For TMB, the paper says κ_q reflects the number and diversity of observations and downweights redundant features, but gives no calculation. The queue holds only eight positive frame features, so nearby frames from one segment could fill it without adding much independent evidence. How is that redundancy measured? A comparison with unweighted prototype matching is also missing. The current results support adding TMB, but do not establish the value of its confidence weighting.
TQ uses global matching alongside local and attended-region signals. Without a matched global-only version, it is hard to tell whether local cues explain the improvement. Fig. 5 has a related attribution problem: it compares CurConMix with full RIVET, which changes several components, so the prediction shifts cannot be assigned specifically to memory refinement.
The implementation description also leaves out the formulas for L_match, L_dn, and L_rank, the definition of κ_q, and values for several loss and query coefficients. Sec. 4.1 refers readers to Sec. 3.6 for these details, but the relevant terms are mostly named rather than fully specified. The authors should also explain whether memory is frozen during validation and testing, and how its state is saved and restored with the selected checkpoint.
3. The annotation protocol is also thin on detail.
Sec. 4.1 says multiple senior surgeons provided the labels, but does not explain whether they reviewed the same segments independently, how disagreements were resolved, or whether agreement was measured. Given the visually similar tissues and ambiguous actions highlighted in the paper, these details matter for assessing label reliability, especially for rare triplets.
Questions for authors
Refer to weaknesses above.
7 comments
Response to Reviewer Comments
2.1 SR: calibration direction, gating, and the contributions of three constituent factors
To identify the sources of SR's gains, we conducted two sets of controlled experiments. First, we compared no SR, calibration in each of the two directions, bidirectional calibration with static gating, and the full model to assess calibration direction and frame-adaptive gating. We then separately removed the structural reference, learnable residual , and relation supervision to assess their individual contributions. The direction and gating experiments use the same fixed patient split and seeds , with all other settings held constant. We report the mean ± sample standard deviation over five runs. All configurations in Table 2 of the paper retain the classification objective in Eq. (9); those containing SR use relation supervision, and those containing TQ use all three query supervision terms.
| Method | AP_I | AP_V | AP_T | AP_IV | AP_IT | AP_IVT | AP_Head | AP_Tail | H |
|---|---|---|---|---|---|---|---|---|---|
| w/o SR (TMB + TQ) | 62.14 ± 0.55 | 60.16 ± 0.60 | 47.55 ± 0.70 | 57.61 ± 1.01 | 43.89 ± 0.66 | 42.46 ± 0.69 | 49.85 ± 0.50 | 35.80 ± 0.90 | 41.67 ± 0.77 |
| Component-to-triplet calibration + static gate | 65.06 ± 0.66 | 61.00 ± 0.08 | 48.68 ± 0.04 | 58.88 ± 0.59 | 45.53 ± 0.54 | 44.80 ± 0.32 | 52.57 ± 0.78 | 37.82 ± 0.16 | 43.99 ± 0.21 |
| Triplet-to-component calibration + static gate | 65.22 ± 0.30 | 59.44 ± 0.44 | 48.30 ± 0.55 | 58.89 ± 0.66 | 43.60 ± 0.61 | 42.46 ± 0.69 | 49.85 ± 0.50 | 35.80 ± 0.90 | 41.67 ± 0.77 |
| Bidirectional calibration + static gate | 66.09 ± 0.13 | 61.23 ± 0.26 | 49.82 ± 0.01 | 61.37 ± 0.72 | 46.41 ± 0.32 | 45.84 ± 0.18 | 53.28 ± 0.11 | 39.14 ± 0.38 | 45.13 ± 0.24 |
| RIVET | 67.25 ± 0.97 | 62.65 ± 0.33 | 50.71 ± 0.42 | 62.94 ± 0.75 | 47.72 ± 0.77 | 47.19 ± 0.43 | 54.69 ± 0.36 | 40.45 ± 0.63 | 46.50 ± 0.48 |
These comparisons address the sources of SR's gains by examining calibration direction and gating separately. With static gating, bidirectional calibration improves AP_IVT from 44.80% to 45.84% over component-to-triplet calibration alone, supporting the additional benefit of information exchange in the reverse direction. Replacing static gating with frame-adaptive gating further increases AP_IVT to 47.19%, a gain of 1.35 percentage points, supporting adaptation of the structural calibration strength to the current frame.
To distinguish the contributions of the three factors, we use full RIVET as the reference and separately set the structural reference term in Eq. (3) of the paper, the output of , or the relation supervision weight to zero. Only one factor is removed at a time, with all other modules, supervision, and training hyperparameters unchanged. When removing the structural reference term, we retain the component–triplet mappings used to construct the relation supervision targets, isolating the contribution of the structural reference residual to prediction.
| Method | AP_I | AP_V | AP_T | AP_IV | AP_IT | AP_IVT | AP_Head | AP_Tail | H |
|---|---|---|---|---|---|---|---|---|---|
| w/o structural reference | 64.19 ± 0.78 | 60.31 ± 0.49 | 47.89 ± 0.67 | 59.42 ± 0.81 | 45.06 ± 0.58 | 43.62 ± 0.77 | 50.99 ± 0.72 | 36.99 ± 1.05 | 42.87 ± 0.85 |
| w/o b_r(h) | 65.57 ± 0.73 | 61.51 ± 0.41 | 49.53 ± 0.56 | 61.10 ± 0.66 | 46.50 ± 0.61 | 45.40 ± 0.76 | 53.15 ± 0.63 | 38.43 ± 0.94 | 44.60 ± 0.82 |
| w/o L_rel | 66.45 ± 0.89 | 61.73 ± 0.46 | 50.07 ± 0.51 | 62.02 ± 0.72 | 47.04 ± 0.66 | 45.99 ± 0.61 | 53.57 ± 0.57 | 39.17 ± 0.87 | 45.25 ± 0.68 |
| RIVET | 67.25 ± 0.97 | 62.65 ± 0.33 | 50.71 ± 0.42 | 62.94 ± 0.75 | 47.72 ± 0.77 | 47.19 ± 0.43 | 54.69 ± 0.36 | 40.45 ± 0.63 | 46.50 ± 0.48 |
Relative to the full model, removing the structural reference, learnable residual , or relation supervision reduces mean AP_IVT by 3.57, 1.79, and 1.20 percentage points, respectively, and mean AP_Tail by 3.46, 2.02, and 1.28 points. These ablations show that each factor provides an additional benefit when the others are retained, with removal of the structural reference causing the largest drop.
Response to Reviewer Comments
2.2 TMB: redundancy estimation and comparisons with unweighted prototypes
The queue of eight features estimates redundancy among recent observations, while the momentum prototype retains information accumulated across batches. The memory branch uses a FIFO queue of positive features with , momentum updates with , and a confidence-weighted residual. The confidence weight is computed as follows:
Let denote the number of occupied slots and the normalized feature of a positive training frame stored in an occupied slot.
\tag{M1}$$ $$\kappa_q= \begin{cases} \operatorname{clip}\left( \dfrac{n_q}{K} \left[0.35+0.65\sqrt{\dfrac{n_{\mathrm{eff},q}}{n_q}}\right], 0,1\right), & n_q>1,\\\\ \dfrac{n_q}{K}, & n_q\le 1. \end{cases} \tag{M2}$$ For $n_q\le1$, we use the boundary case in Eq. (M2): confidence is 0 for an empty queue and $1/8$ for a queue with one occupied slot. The prototype is initialized with the first normalized positive feature and subsequently updated with momentum $m=0.9$ and normalized. The memory residual is zero for classes whose prototypes have not been initialized. Historical matching uses cosine similarity between the current feature and the prototype, with residual coefficient $\lambda_M=0.28$ and temperature $\tau_M=0.07$; $\kappa_q$ scales the correction strength. Eq. (M1) measures feature redundancy using off-diagonal cosine similarities clipped below at zero, and Eq. (M2) also accounts for queue occupancy. Adjacent frames may enter the queue, but high similarity reduces the estimated diversity and correction weight. We compare a simple mean prototype, an unweighted momentum prototype, and full RIVET, keeping all other modules and training settings identical. We use seeds $\{42,123,2024,2025,3407\}$ and report the mean ± sample standard deviation over five runs. The simple mean prototype uses a cumulative arithmetic mean, while the momentum prototype uses $m=0.9$. Both unweighted configurations set $\kappa_q=1$, whereas full RIVET uses the adaptive weight in Eqs. (M1)–(M2): | Method | AP_I | AP_V | AP_T | AP_IV | AP_IT | AP_IVT | AP_Head | AP_Tail | H | | ----------------------------- | ------------:| ------------:| ------------:| ------------:| ------------:| ------------:| ------------:| ------------:| ------------:| | w/o TMB (SR + TQ) | 63.34 ± 0.35 | 60.23 ± 0.51 | 46.35 ± 0.80 | 58.00 ± 0.50 | 45.00 ± 0.64 | 42.93 ± 1.12 | 50.95 ± 0.47 | 35.71 ± 1.94 | 41.97 ± 1.40 | | Simple mean prototype (κ = 1) | 64.28 ± 0.39 | 61.41 ± 0.47 | 47.99 ± 0.25 | 60.07 ± 0.02 | 46.37 ± 0.21 | 43.97 ± 0.78 | 51.78 ± 0.85 | 36.94 ± 0.96 | 43.11 ± 0.83 | | Momentum prototype (κ = 1) | 65.31 ± 0.71 | 61.22 ± 0.44 | 49.92 ± 0.93 | 61.29 ± 0.76 | 46.33 ± 0.12 | 45.58 ± 0.74 | 54.20 ± 0.03 | 37.83 ± 1.39 | 44.55 ± 0.96 | | RIVET | 67.25 ± 0.97 | 62.65 ± 0.33 | 50.71 ± 0.42 | 62.94 ± 0.75 | 47.72 ± 0.77 | 47.19 ± 0.43 | 54.69 ± 0.36 | 40.45 ± 0.63 | 46.50 ± 0.48 | Compared with the unweighted momentum prototype, full RIVET improves mean AP_IVT and AP_Tail by 1.61 and 2.62 percentage points, respectively. Since these configurations differ only in confidence weighting, this comparison supports the additional benefit of adjusting memory correction strength according to observation count and feature diversity, with a larger improvement on tail classes.
Response to Reviewer Comments
Thank you for your careful review and constructive suggestions, and for recognizing the value of surgical triplet recognition in sinus endoscopy and SinusTriplet-95. We have considered each concern, conducted additional experiments, and clarified implementation details. Below, we address class definitions and evaluation coverage, module contributions and reproducibility, and annotation reliability.
1. Class definitions, evaluation coverage, and robustness across patient splits
The model defines the three classes with the highest frame counts as head classes (KIVT=3) and the remaining 92 as tail classes. The 50 extremely rare classes used for tail evaluation are all within these 92 model tail classes. The model’s head–tail partition is based on frame counts across the entire dataset before the frame sampling rates are adjusted. The statement that each of the 50 tail classes has at most five annotated intervals refers to activity intervals across the entire dataset, rather than individual frames. We sample frames at different rates according to each class's interval count: 0.5 FPS for classes with at least 100 intervals, 1 FPS for those with 10–99 intervals, and 5 FPS for those with 1–9 intervals, ensuring at least 500 frames per tail class.
We split the data by patient to prevent leakage of consecutive frames from the same patient across subsets. The fixed 6:2:2 split contains 9 training, 3 validation, and 3 test patients. Each subset covers all 95 triplet classes, including all 50 extremely rare classes, giving 100% class coverage in each subset. Thus, every class has positive examples in training, validation, and testing, and class-wise AP evaluation on this fixed test set does not require handling classes with no test positives. We will clarify these counting conventions and provide further details on dataset construction and frame sampling in the final manuscript.
We have completed experiments with five training seeds on the fixed patient split and experiments across five patient splits to assess training randomness and sensitivity to the patient split, respectively. We use Swin-B (384) as the main configuration and compare all methods using the same splits and seeds. For the fixed-split experiments, we use seeds {42,123,2024,2025,3407} and report the mean ± sample standard deviation over five runs:
For each of the five patient-level 6:2:2 splits, we use seeds {42,123,2024}. For each metric, we first average the three runs within each split, then report the mean and sample standard deviation of the five split averages:
In both experiments, RIVET achieves higher mean AP_IVT and AP_Tail than the comparison methods, supporting its performance advantage across training seeds and patient splits.