acceptodds
Under review as a conference paper at ICLR 2027

RIVET: Multi-Evidence Relational Inference for Rare Surgical Triplets in Sinus Endoscopy

Abstract

Recognizing surgical interactions as instrument–verb–target (IVT) triplets is hindered by long-tailed class distributions and visual ambiguity among triplets that share components. These challenges are particularly pronounced in sinus endoscopy, where narrow operative fields, occlusions, and visually similar tissues obscure discriminative local cues. Here we introduce SinusTriplet-95, a dual-center dataset of 324 videos from 15 patients, comprising 670,381 frames and 2,428 segments annotated by multiple senior attending surgeons across 95 valid IVT triplets. We further propose RIVET (Relational Inference with Visual Evidence for Tail Triplets), which refines long-tailed triplet prediction at inference by integrating structural, historical, and local evidence. Its Dynamic Structural Relation module bidirectionally calibrates component- and triplet-level predictions. To improve the recognition of underrepresented triplets, the Tail-Class Memory Bank stores observations of rare triplet classes across surgical process, while the Tail-Class Prototype Query searches the current frame for local visual evidence associated with each rare triplet candidate. These complementary sources of evidence are integrated through frame-adaptive gating. On SinusTriplet-95, the RIVET ensemble achieves an of 50.80%, surpassing the strongest ensemble baseline by 5.71 percentage points and ranking first among the compared methods across all six AP metrics. Progressive ablations show incremental gains from each evidence source, while class-level analysis shows improvements over the corresponding baselines in 46 of 50 tail classes. Together, SinusTriplet-95 and RIVET provide a benchmark and an evidence-guided framework for surgical interaction recognition in sinus endoscopy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
surgical triplet recognition, long-tailed recognition, sinus endoscopy, multi-evidence verification, prototype learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nHpJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents RIVET for surgical triplet recognition, with a focus on rare classes. It uses component–triplet relations, historical prototypes, and spatial queries to refine predictions. The authors also introduce SinusTriplet-95: 324 clips from 15 patients covering 95 triplet classes. On a fixed patient-level split, triplet AP reaches 47.47% with a single model and 50.80% with an ensemble. Results on CholecT45 and the ablation study provide further evidence of the method’s effectiveness.

Critical strengths
  1. The paper brings triplet recognition to sinus endoscopy, a useful setting beyond cholecystectomy. Narrow views, occlusion, and similar-looking tissues make this a challenging task.

  2. SinusTriplet-95 contributes data from two centers with surgeon annotations for 95 triplet classes. Despite its small patient cohort, it provides a useful starting point for studying rare interactions in sinus surgery.

Critical weaknesses

1. The tail-class split and evaluation coverage need clarification.

Sec. 3.2 defines head and tail classes by frame frequency, with the top K_IVT classes assigned to the head. Appendix A.2 uses a different criterion for evaluation: the 50 classes with at most five annotated segments. What is K_IVT, and how much do the two sets overlap? It is also unclear whether the frequency counts come from the training split only.

The dataset has 15 patients. An exact patient-level 6:2:2 split would mean nine for training and three each for validation and testing. Given that 50 classes have at most five segments in total, their distribution across patients and splits matters. The paper does not give per-class positive patient or segment counts, or explain how classes without test positives are handled in AP.

Only one patient split is reported for SinusTriplet-95, without uncertainty estimates. Many correlated frames from a few patients cannot establish whether the gains hold across patients. Repeated patient splits or patient-level uncertainty estimates would make the results more convincing.

2. The ablations show gains, but leave their source unclear. Several details needed for reproduction are missing.

SR adds both a taxonomy-based reference and a learned class-specific residual b_r(h) in Eq. 3. There is also relation supervision in Eq. 10. Table 2 does not separate these effects or state which supervision terms are retained in each ablation.

For TMB, the paper says κ_q reflects the number and diversity of observations and downweights redundant features, but gives no calculation. The queue holds only eight positive frame features, so nearby frames from one segment could fill it without adding much independent evidence. How is that redundancy measured? A comparison with unweighted prototype matching is also missing. The current results support adding TMB, but do not establish the value of its confidence weighting.

TQ uses global matching alongside local and attended-region signals. Without a matched global-only version, it is hard to tell whether local cues explain the improvement. Fig. 5 has a related attribution problem: it compares CurConMix with full RIVET, which changes several components, so the prediction shifts cannot be assigned specifically to memory refinement.

The implementation description also leaves out the formulas for L_match, L_dn, and L_rank, the definition of κ_q, and values for several loss and query coefficients. Sec. 4.1 refers readers to Sec. 3.6 for these details, but the relevant terms are mostly named rather than fully specified. The authors should also explain whether memory is frozen during validation and testing, and how its state is saved and restored with the selected checkpoint.

3. The annotation protocol is also thin on detail.

Sec. 4.1 says multiple senior surgeons provided the labels, but does not explain whether they reviewed the same segments independently, how disagreements were resolved, or whether agreement was measured. Given the visually similar tissues and ambiguous actions highlighted in the paper, these details matter for assessing label reliability, especially for rare triplets.

Questions for authors

Refer to weaknesses above.

7 comments
AuthorsOpenReview ↗

Response to Reviewer Comments

Thank you for your careful review and constructive suggestions, and for recognizing the value of surgical triplet recognition in sinus endoscopy and SinusTriplet-95. We have considered each concern, conducted additional experiments, and clarified implementation details. Below, we address class definitions and evaluation coverage, module contributions and reproducibility, and annotation reliability.

1. Class definitions, evaluation coverage, and robustness across patient splits

The model defines the three classes with the highest frame counts as head classes (KIVT=3K_{\mathrm{IVT}}=3) and the remaining 92 as tail classes. The 50 extremely rare classes used for tail evaluation are all within these 92 model tail classes. The model’s head–tail partition is based on frame counts across the entire dataset before the frame sampling rates are adjusted. The statement that each of the 50 tail classes has at most five annotated intervals refers to activity intervals across the entire dataset, rather than individual frames. We sample frames at different rates according to each class's interval count: 0.5 FPS for classes with at least 100 intervals, 1 FPS for those with 10–99 intervals, and 5 FPS for those with 1–9 intervals, ensuring at least 500 frames per tail class.

We split the data by patient to prevent leakage of consecutive frames from the same patient across subsets. The fixed 6:2:2 split contains 9 training, 3 validation, and 3 test patients. Each subset covers all 95 triplet classes, including all 50 extremely rare classes, giving 100% class coverage in each subset. Thus, every class has positive examples in training, validation, and testing, and class-wise AP evaluation on this fixed test set does not require handling classes with no test positives. We will clarify these counting conventions and provide further details on dataset construction and frame sampling in the final manuscript.

We have completed experiments with five training seeds on the fixed patient split and experiments across five patient splits to assess training randomness and sensitivity to the patient split, respectively. We use Swin-B (384) as the main configuration and compare all methods using the same splits and seeds. For the fixed-split experiments, we use seeds {42,123,2024,2025,3407}\{42,123,2024,2025,3407\} and report the mean ± sample standard deviation over five runs:

MethodAP_IAP_VAP_TAP_IVAP_ITAP_IVTAP_HeadAP_TailH
TERL63.82 ± 0.8861.66 ± 0.0248.06 ± 0.0759.40 ± 0.3842.44 ± 0.0141.66 ± 0.1949.21 ± 0.1134.86 ± 0.2740.81 ± 0.22
CurConMix61.09 ± 0.9559.00 ± 0.8944.64 ± 0.8657.33 ± 0.2143.61 ± 0.0641.01 ± 0.2547.26 ± 0.0235.39 ± 0.4840.47 ± 0.31
RIVET (Ours)67.25 ± 0.9762.65 ± 0.3350.71 ± 0.4262.94 ± 0.7547.72 ± 0.7747.19 ± 0.4354.69 ± 0.3640.45 ± 0.6346.50 ± 0.48

For each of the five patient-level 6:2:2 splits, we use seeds {42,123,2024}\{42,123,2024\}. For each metric, we first average the three runs within each split, then report the mean and sample standard deviation of the five split averages:

MethodAP_IAP_VAP_TAP_IVAP_ITAP_IVTAP_HeadAP_TailH
TERL62.90 ± 1.7660.00 ± 1.5646.61 ± 1.6259.14 ± 1.9941.14 ± 1.9139.73 ± 1.1846.88 ± 0.7433.30 ± 1.6738.93 ± 1.35
CurConMix60.73 ± 1.9957.15 ± 1.9544.22 ± 1.0355.86 ± 0.2043.51 ± 1.8940.02 ± 2.1146.83 ± 1.7633.88 ± 2.4339.31 ± 2.24
RIVET (Ours)67.32 ± 0.8462.44 ± 2.5849.37 ± 1.0561.68 ± 1.0447.38 ± 1.8546.46 ± 1.8754.81 ± 1.0838.95 ± 2.6245.52 ± 2.11

In both experiments, RIVET achieves higher mean AP_IVT and AP_Tail than the comparison methods, supporting its performance advantage across training seeds and patient splits.

AuthorsOpenReview ↗

Response to Reviewer Comments

2.1 SR: calibration direction, gating, and the contributions of three constituent factors

To identify the sources of SR's gains, we conducted two sets of controlled experiments. First, we compared no SR, calibration in each of the two directions, bidirectional calibration with static gating, and the full model to assess calibration direction and frame-adaptive gating. We then separately removed the structural reference, learnable residual br(h)b_r(h), and relation supervision LrelL_{\mathrm{rel}} to assess their individual contributions. The direction and gating experiments use the same fixed patient split and seeds {42,123,2024,2025,3407}\{42,123,2024,2025,3407\}, with all other settings held constant. We report the mean ± sample standard deviation over five runs. All configurations in Table 2 of the paper retain the classification objective in Eq. (9); those containing SR use relation supervision, and those containing TQ use all three query supervision terms.

MethodAP_IAP_VAP_TAP_IVAP_ITAP_IVTAP_HeadAP_TailH
w/o SR (TMB + TQ)62.14 ± 0.5560.16 ± 0.6047.55 ± 0.7057.61 ± 1.0143.89 ± 0.6642.46 ± 0.6949.85 ± 0.5035.80 ± 0.9041.67 ± 0.77
Component-to-triplet calibration + static gate65.06 ± 0.6661.00 ± 0.0848.68 ± 0.0458.88 ± 0.5945.53 ± 0.5444.80 ± 0.3252.57 ± 0.7837.82 ± 0.1643.99 ± 0.21
Triplet-to-component calibration + static gate65.22 ± 0.3059.44 ± 0.4448.30 ± 0.5558.89 ± 0.6643.60 ± 0.6142.46 ± 0.6949.85 ± 0.5035.80 ± 0.9041.67 ± 0.77
Bidirectional calibration + static gate66.09 ± 0.1361.23 ± 0.2649.82 ± 0.0161.37 ± 0.7246.41 ± 0.3245.84 ± 0.1853.28 ± 0.1139.14 ± 0.3845.13 ± 0.24
RIVET67.25 ± 0.9762.65 ± 0.3350.71 ± 0.4262.94 ± 0.7547.72 ± 0.7747.19 ± 0.4354.69 ± 0.3640.45 ± 0.6346.50 ± 0.48

These comparisons address the sources of SR's gains by examining calibration direction and gating separately. With static gating, bidirectional calibration improves AP_IVT from 44.80% to 45.84% over component-to-triplet calibration alone, supporting the additional benefit of information exchange in the reverse direction. Replacing static gating with frame-adaptive gating further increases AP_IVT to 47.19%, a gain of 1.35 percentage points, supporting adaptation of the structural calibration strength to the current frame.

To distinguish the contributions of the three factors, we use full RIVET as the reference and separately set the structural reference term in Eq. (3) of the paper, the output of br(h)b_r(h), or the relation supervision weight λrel\lambda_{\mathrm{rel}} to zero. Only one factor is removed at a time, with all other modules, supervision, and training hyperparameters unchanged. When removing the structural reference term, we retain the component–triplet mappings used to construct the relation supervision targets, isolating the contribution of the structural reference residual to prediction.

MethodAP_IAP_VAP_TAP_IVAP_ITAP_IVTAP_HeadAP_TailH
w/o structural reference64.19 ± 0.7860.31 ± 0.4947.89 ± 0.6759.42 ± 0.8145.06 ± 0.5843.62 ± 0.7750.99 ± 0.7236.99 ± 1.0542.87 ± 0.85
w/o b_r(h)65.57 ± 0.7361.51 ± 0.4149.53 ± 0.5661.10 ± 0.6646.50 ± 0.6145.40 ± 0.7653.15 ± 0.6338.43 ± 0.9444.60 ± 0.82
w/o L_rel66.45 ± 0.8961.73 ± 0.4650.07 ± 0.5162.02 ± 0.7247.04 ± 0.6645.99 ± 0.6153.57 ± 0.5739.17 ± 0.8745.25 ± 0.68
RIVET67.25 ± 0.9762.65 ± 0.3350.71 ± 0.4262.94 ± 0.7547.72 ± 0.7747.19 ± 0.4354.69 ± 0.3640.45 ± 0.6346.50 ± 0.48

Relative to the full model, removing the structural reference, learnable residual br(h)b_r(h), or relation supervision reduces mean AP_IVT by 3.57, 1.79, and 1.20 percentage points, respectively, and mean AP_Tail by 3.46, 2.02, and 1.28 points. These ablations show that each factor provides an additional benefit when the others are retained, with removal of the structural reference causing the largest drop.

AuthorsOpenReview ↗

Response to Reviewer Comments

2.2 TMB: redundancy estimation and comparisons with unweighted prototypes

The queue of eight features estimates redundancy among recent observations, while the momentum prototype retains information accumulated across batches. The memory branch uses a FIFO queue of positive features with K=8K=8, momentum updates with m=0.9m=0.9, and a confidence-weighted residual. The confidence weight κq\kappa_q is computed as follows:

Let nqn_q denote the number of occupied slots and zq,a\mathbf z_{q,a} the normalized feature of a positive training frame stored in an occupied slot.

sˉq=1nq(nq−1)∑1≤a,b≤nq,;a≠bmax⁡(cos⁡(zq,a,zq,b),0).\bar{s}_q = \frac{1}{n_q(n_q-1)} \sum_{1\le a,b\le n_q,\\;a\ne b} \max\left(\cos(\mathbf{z}_{q,a},\mathbf{z}_{q,b}),0\right).

Dq=clip⁡(1−sˉq,0,1).D_q = \operatorname{clip}(1-\bar{s}_q,0,1).

\tag{M1}$$ $$\kappa_q= \begin{cases} \operatorname{clip}\left( \dfrac{n_q}{K} \left[0.35+0.65\sqrt{\dfrac{n_{\mathrm{eff},q}}{n_q}}\right], 0,1\right), & n_q>1,\\\\ \dfrac{n_q}{K}, & n_q\le 1. \end{cases} \tag{M2}$$ For $n_q\le1$, we use the boundary case in Eq. (M2): confidence is 0 for an empty queue and $1/8$ for a queue with one occupied slot. The prototype is initialized with the first normalized positive feature and subsequently updated with momentum $m=0.9$ and normalized. The memory residual is zero for classes whose prototypes have not been initialized. Historical matching uses cosine similarity between the current feature and the prototype, with residual coefficient $\lambda_M=0.28$ and temperature $\tau_M=0.07$; $\kappa_q$ scales the correction strength. Eq. (M1) measures feature redundancy using off-diagonal cosine similarities clipped below at zero, and Eq. (M2) also accounts for queue occupancy. Adjacent frames may enter the queue, but high similarity reduces the estimated diversity and correction weight. We compare a simple mean prototype, an unweighted momentum prototype, and full RIVET, keeping all other modules and training settings identical. We use seeds $\{42,123,2024,2025,3407\}$ and report the mean ± sample standard deviation over five runs. The simple mean prototype uses a cumulative arithmetic mean, while the momentum prototype uses $m=0.9$. Both unweighted configurations set $\kappa_q=1$, whereas full RIVET uses the adaptive weight in Eqs. (M1)–(M2): | Method | AP_I | AP_V | AP_T | AP_IV | AP_IT | AP_IVT | AP_Head | AP_Tail | H | | ----------------------------- | ------------:| ------------:| ------------:| ------------:| ------------:| ------------:| ------------:| ------------:| ------------:| | w/o TMB (SR + TQ) | 63.34 ± 0.35 | 60.23 ± 0.51 | 46.35 ± 0.80 | 58.00 ± 0.50 | 45.00 ± 0.64 | 42.93 ± 1.12 | 50.95 ± 0.47 | 35.71 ± 1.94 | 41.97 ± 1.40 | | Simple mean prototype (κ = 1) | 64.28 ± 0.39 | 61.41 ± 0.47 | 47.99 ± 0.25 | 60.07 ± 0.02 | 46.37 ± 0.21 | 43.97 ± 0.78 | 51.78 ± 0.85 | 36.94 ± 0.96 | 43.11 ± 0.83 | | Momentum prototype (κ = 1) | 65.31 ± 0.71 | 61.22 ± 0.44 | 49.92 ± 0.93 | 61.29 ± 0.76 | 46.33 ± 0.12 | 45.58 ± 0.74 | 54.20 ± 0.03 | 37.83 ± 1.39 | 44.55 ± 0.96 | | RIVET | 67.25 ± 0.97 | 62.65 ± 0.33 | 50.71 ± 0.42 | 62.94 ± 0.75 | 47.72 ± 0.77 | 47.19 ± 0.43 | 54.69 ± 0.36 | 40.45 ± 0.63 | 46.50 ± 0.48 | Compared with the unweighted momentum prototype, full RIVET improves mean AP_IVT and AP_Tail by 1.61 and 2.62 percentage points, respectively. Since these configurations differ only in confidence weighting, this comparison supports the additional benefit of adjusting memory correction strength according to observation count and feature diversity, with a larger improvement on tail classes.
Reviewer_WSA5OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SinusTriplet-95, a new dataset of sinus endoscopy videos labelled with instrument-verb-target triplets. The label distribution is strongly long-tailed. The paper also proposes a new method that refines triplet predictions at inference time by combining three sources of information: 1) consistency derived from the taxonomy, 2) a memory bank of class prototypes, 3) candidate-specific queries over spatial features. Experiments are conducted on the SinusTriplet-95 and CholecT45 datasets.

Critical strengths
  1. A new benchmark for the sinus endoscopy surgical domain, which is a new domain not covered by previous datasets (for this triplet task). The dataset will be the main contribution of this paper, but only if it is released.

  2. The method overall is intuitive and reasonable, using various inductive biases and human intuitions about the task. The method achieves a reasonable performance gain over recent baselines. The ablation shows that each design adds something.

Critical weaknesses
  1. As the main contribution, the dataset is not promised to be released for public use. So we cannot count it as a contribution.

  2. The dataset has many tail classes that are so rare that they probably have very few examples, especially in the test set. Therefore, the results may not be very stable. The designed methods and the parameter tuning may overfit to these rare classes with limited samples.

  3. The method design is reasonable, but not surprising and somewhat old-fashioned. The method contributes little new knowledge for ICLR. The components are common ideas assembled into a complex pipeline. The paper offers little insight beyond the surgical setting, so the contribution is mainly the dataset. But it is not promised to be released.

Questions for authors

Please refer to the weaknesses. If the authors commit to a public release of the dataset, I am willing to raise my rating.

In addition: For this task, what will the performance be if we directly apply video agents or general-purpose agents? Is it still meaningful to design a complex "hand-crafted" old-fashioned method? Also, broadly for surgical AI, I don't think we should keep working on such specific, narrow tasks; instead, the community should work on more ambitious tasks using more frontier methods.

Reviewer_UyceOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies long-tailed instrument–verb–target (IVT) recognition in sinus endoscopy and introduces RIVET, together with a new dual-center dataset, SinusTriplet-95.

  • The proposed framework combines structural relation refinement, a tail-class memory bank, and class-conditioned local visual queries to improve triplet predictions using compositional relationships, historical features, and local visual evidence.
  • Experiments on SinusTriplet-95 and CholecT45 show promising improvements over existing baselines, particularly for rare triplets.
Critical strengths
  1. Meaningful problem setting.
    • Fine-grained interaction recognition in sinus endoscopy is challenging due to visual ambiguity and severe class imbalance. SinusTriplet-95 provides a useful benchmark for studying this problem.
  2. Reasonable overall design.
    • The three modules address different aspects of the problem. Structural relations help maintain consistency between components and triplets, while the memory bank and local queries provide additional evidence for rare classes. The overall design makes sense for this task.
  3. Encouraging experimental results.
    • RIVET achieves clear improvements over the reported baselines. The ablation results also show consistent gains from the proposed modules, while the additional experiments on CholecT45 provide some evidence of applicability beyond the new dataset.
Critical weaknesses
  1. Limited evidence of experimental robustness.
    • SinusTriplet-95 contains only 15 patients, and the main results are reported on a single fixed patient-level split without variance estimates.
      • Given the extremely small number of examples for many tail classes, it is unclear how stable the reported improvements are across different splits or random seeds.
  2. The novelty of the individual components is somewhat limited.
    • The main ideas behind the three modules are not particularly new. The ablation results show that adding them improves performance, but it is still unclear whether the proposed designs are necessary. For example, comparisons with simpler class prototypes or more straightforward relation modeling approaches would help clarify where the gains come from.
  3. Several important implementation and evaluation details are unclear. For example, the exact computation of memory confidence, the definitions of the query losses, and the training/test coverage of rare classes are not sufficiently specified. These omissions make the method harder to reproduce and the tail-class improvements harder to interpret.
Questions for authors
  1. Could the authors clarify the distribution of rare triplets across the training and test sets, and explain how the memory bank handles classes with very few or no positive training examples?
    • A comparison between confidence-weighted memory and simple class prototypes would also help establish the benefit of the proposed memory design.
  2. please refer to the Weaknesses.
11 comments
AuthorsOpenReview ↗

Response to Reviewer Comments

We thank the reviewer for the careful reading and constructive feedback. To address your concerns about experimental reliability, module design, and implementation details, we have conducted additional experiments and analyses, as detailed below.

1. Patient cohort size and robustness across random seeds and patient splits.

We understand the concern that a cohort of only 15 patients may limit the evaluation. In SinusTriplet-95, the average video duration per patient is approximately four to five times that in CholecT45. After frame sampling, the dataset contains 670,381 frames, approximately 6.6 times the number in CholecT45 (100,863 frames), with annotation coverage exceeding 85%. The statement that each of the 50 tail classes has no more than five instances refers to annotated activity intervals across the entire dataset, rather than individual frames. We use different frame sampling rates according to each class's number of annotated intervals: 0.5 FPS for ≥100 intervals, 1 FPS for 10–99 intervals, and 5 FPS for 1–9 intervals. This ensures that each tail class contains at least 500 frames. To prevent information leakage from consecutive frames of the same patient, we use a patient-level 6:2:2 split, with 9, 3, and 3 patients in the training, validation, and test sets, respectively. All 50 extreme-tail triplet classes, each with no more than five annotated activity intervals, are represented in all three sets, giving 100% class coverage in each set. Although each class has at most five annotated activity intervals across the entire dataset, we ensure that every tail class has at least one such interval in each of the training, validation, and test sets. Thus, no tail class appears only in the test set without positive training examples. We will clarify these counting conventions and provide further details on dataset construction and frame sampling in the final version.

Given the limited number of patients, we further evaluated stability across training seeds and patient splits. Using Swin-B (384) as the main configuration, we ran experiments with five random seeds on a fixed patient split and with five different patient-level 6:2:2 splits. All methods were compared using the same splits and seeds. For the fixed-split experiments and the subsequent ablations, we used seeds {42, 123, 2024, 2025, 3407} and report all metrics as mean ± sample standard deviation. We will release the training code alongside the dataset to support reproducibility.

MethodAP_IAP_VAP_TAP_IVAP_ITAP_IVTAP_HeadAP_TailH
TERL63.82 ± 0.8861.66 ± 0.0248.06 ± 0.0759.40 ± 0.3842.44 ± 0.0141.66 ± 0.1949.21 ± 0.1134.86 ± 0.2740.81 ± 0.22
CurConMix61.09 ± 0.9559.00 ± 0.8944.64 ± 0.8657.33 ± 0.2143.61 ± 0.0641.01 ± 0.2547.26 ± 0.0235.39 ± 0.4840.47 ± 0.31
RIVET (Ours)67.25 ± 0.9762.65 ± 0.3350.71 ± 0.4262.94 ± 0.7547.72 ± 0.7747.19 ± 0.4354.69 ± 0.3640.45 ± 0.6346.50 ± 0.48

For the repeated patient splits, we used the same training seeds {42, 123, 2024} for each split. For each metric, we first average the results over the three seeds within each split, then report the mean and between-split sample standard deviation across the five splits.

MethodAP_IAP_VAP_TAP_IVAP_ITAP_IVTAP_HeadAP_TailH
TERL62.90 ± 1.7660.00 ± 1.5646.61 ± 1.6259.14 ± 1.9941.14 ± 1.9139.73 ± 1.1846.88 ± 0.7433.30 ± 1.6738.93 ± 1.35
CurConMix60.73 ± 1.9957.15 ± 1.9544.22 ± 1.0355.86 ± 0.2043.51 ± 1.8940.02 ± 2.1146.83 ± 1.7633.88 ± 2.4339.31 ± 2.24
RIVET (Ours)67.32 ± 0.8462.44 ± 2.5849.37 ± 1.0561.68 ± 1.0447.38 ± 1.8546.46 ± 1.8754.81 ± 1.0838.95 ± 2.6245.52 ± 2.11

Both sets of experiments show that RIVET maintains a consistent performance advantage across training seeds and patient splits.

AuthorsOpenReview ↗

Response to Reviewer Comments

2.1 Framework novelty and comparisons with simple class prototypes.

We understand the concern that the novelty may lie primarily in combining existing modules. RIVET's core contribution is a prediction refinement framework that integrates structural, historical, and local visual evidence to address scarce evidence and visual ambiguity in long-tailed surgical triplet recognition. Specifically, co-occurrence statistics are difficult to estimate reliably for rare classes, so the structural relation module uses deterministic component–triplet mappings to perform bidirectional soft calibration between component and triplet predictions in the current frame. To provide stable visual references when positive examples are sparse, the tail-class memory bank retains features across training batches and weights historical prototype matches at prediction time by confidence derived from the number and diversity of stored observations. To distinguish triplets that share components, the local query module combines component semantics and classifier weights to construct queries for complete triplet candidates and retrieve discriminative local evidence from the current frame. These mechanisms jointly refine predictions through frame-adaptive gating, bringing compositional relationships, historical references across samples, and local evidence from the current frame into a single inference process. To assess the value of these specific designs over simpler alternatives, we provide controlled comparisons below using simple prototypes, unidirectional structural calibration, and static gating.

To evaluate the memory branch, we varied only the prototype update and confidence weighting strategies, keeping the other modules and training settings fixed. Simple Mean Prototype uses a cumulative arithmetic mean, while Momentum Prototype uses momentum updates (m=0.9). We set κ=1 when confidence weighting is disabled. Results over five random seeds are shown below:

MethodAP_IAP_VAP_TAP_IVAP_ITAP_IVTAP_HeadAP_TailH
w/o TMB (SR + TQ)63.34 ± 0.3560.23 ± 0.5146.35 ± 0.8058.00 ± 0.5045.00 ± 0.6442.93 ± 1.1250.95 ± 0.4735.71 ± 1.9441.97 ± 1.40
Simple mean prototype (κ = 1)64.28 ± 0.3961.41 ± 0.4747.99 ± 0.2560.07 ± 0.0246.37 ± 0.2143.97 ± 0.7851.78 ± 0.8536.94 ± 0.9643.11 ± 0.83
Momentum prototype (κ = 1)65.31 ± 0.7161.22 ± 0.4449.92 ± 0.9361.29 ± 0.7646.33 ± 0.1245.58 ± 0.7454.20 ± 0.0337.83 ± 1.3944.55 ± 0.96
RIVET67.25 ± 0.9762.65 ± 0.3350.71 ± 0.4262.94 ± 0.7547.72 ± 0.7747.19 ± 0.4354.69 ± 0.3640.45 ± 0.6346.50 ± 0.48

Compared with Simple Mean Prototype, full RIVET improves AP_IVT and AP_Tail by 3.22 and 3.51 percentage points, respectively. Compared with the unweighted Momentum Prototype, the improvements are 1.61 and 2.62 percentage points. These results indicate that confidence weighting based on the number and diversity of historical observations further improves the memory branch.

AuthorsOpenReview ↗

Response to Reviewer Comments

2.2 Bidirectional structural calibration and frame-adaptive gating.

To assess the additional benefits of bidirectional calibration and frame-adaptive gating over more direct relation modeling approaches, we conducted the controlled comparisons below. Starting from a baseline without structural calibration and using the same component–triplet mappings, we first compared component-to-triplet, triplet-to-component, and bidirectional calibration under static gating. We then compared static and frame-adaptive gating with bidirectional calibration. Across these experiments, we kept the memory module, local query module, tail-class visual residual gate, and all other training settings fixed, varying only the direction of structural calibration and the gating mechanism for the structural branch. This isolates the effects of bidirectional information exchange and adaptation to individual frames. The results are shown below:

MethodAP_IAP_VAP_TAP_IVAP_ITAP_IVTAP_HeadAP_TailH
w/o SR (TMB + TQ)62.14 ± 0.5560.16 ± 0.6047.55 ± 0.7057.61 ± 1.0143.89 ± 0.6642.46 ± 0.6949.85 ± 0.5035.80 ± 0.9041.67 ± 0.77
Component-to-triplet calibration + static gate65.06 ± 0.6661.00 ± 0.0848.68 ± 0.0458.88 ± 0.5945.53 ± 0.5444.80 ± 0.3252.57 ± 0.7837.82 ± 0.1643.99 ± 0.21
Triplet-to-component calibration + static gate65.22 ± 0.3059.44 ± 0.4448.30 ± 0.5558.89 ± 0.6643.60 ± 0.6142.46 ± 0.6949.85 ± 0.5035.80 ± 0.9041.67 ± 0.77
Bidirectional calibration + static gate66.09 ± 0.1361.23 ± 0.2649.82 ± 0.0161.37 ± 0.7246.41 ± 0.3245.84 ± 0.1853.28 ± 0.1139.14 ± 0.3845.13 ± 0.24
RIVET67.25 ± 0.9762.65 ± 0.3350.71 ± 0.4262.94 ± 0.7547.72 ± 0.7747.19 ± 0.4354.69 ± 0.3640.45 ± 0.6346.50 ± 0.48

Without structural relations, AP_IVT is 42.46%. Unidirectional component-to-triplet calibration raises it to 44.80%, while calibration in the reverse direction provides no improvement. Bidirectional calibration further increases AP_IVT to 45.84%, and adding frame-adaptive gating raises it to 47.19%. These results support the effectiveness of bidirectional structural calibration and frame-adaptive gating.

Comments on the paper

1 comment

Dear Reviewers,

The authors have posted their responses. Please take a look and respond to the authors.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.