- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- Empirical analysis of visual transition structure is interesting. This provides a useful empirical observation motivating the distinction between representing more visual changes and representing more task-relevant changes.
Critical strengths
-
Empirical analysis of visual transition structure is interesting.
-
Integration of predictive learning and semantic alignment is coherent.
-
Empirical results investigate transition dimensionality, spatial grounding, temporal reversal, latent-action capacity, and prediction-loss weighting.
Critical weaknesses
-
The latent-action representation is not convincingly shown to encode meaningful surgical actions. The paper does not convincingly demonstrate that the latent-action space is organized around surgical interaction semantics rather than arbitrary visual changes useful for reconstructing features.
-
Although the paper reports promising performance, the matched ablation results indicate that adding the full LAViF framework does not consistently improve triplet recognition.
Questions for authors
- What evidence demonstrates that the learned latent actions represent meaningful instrument–tissue interaction dynamics rather than arbitrary visual transitions?
We thank the reviewer for the constructive comments.
Question 1 and Weakness 1. To examine whether the latent actions capture instrument–tissue interaction rather than arbitrary visual change, we ran a new experiment: Grad-CAM [1] of the action token alone (Table R1; Appendix A.11 of the revised paper). Its maps fall on the instrument ∪ target region more often than chance on all three encoders; scoring Grad-CAM maps against segmentation masks is a commonly used grounding measure [2, 3]. We do not claim that the token encodes surgical action labels. How it is computed: a scalar is back-propagated to the patch tokens of the labeled (last) frame entering the encoder's last block; weighting the token channels by their patch-averaged gradients gives one relevance value per patch. The scalar is either the ground-truth triplet score with the aggregator's state input detached, so that the gradient reaches the encoder only through the action tokens (via token), or the squared norm of the most recent action token (token norm). IoU: the top-25% patches vs. the instrument ∪ target mask, i.e., the union of the segmentation masks of the instruments and target tissues of the frame's ground-truth triplets (chance: 11.9). Masks: estimated by SASVi [4], whose segmentation model is trained on expert annotations of cholecystectomy anatomy (CholecSeg8k [5]). Result: compared with IDM only (λpred=0; same token, no prediction loss), the token of full LAViF is more focused on these regions on ViT-L (via token: 16.60 vs. 13.47; token norm: 20.40 vs. 10.79) and SurgeNet-L (via token: 16.81 vs. 15.27), and similar on DINOv2-L (15.45 vs. 14.95).
Table R1. Grad-CAM IoU (%) of the action token with the instrument ∪ target region on CholecT50 at da=128 (mean±std over three seeds; chance ≈11.9).
Weakness 2. This is indeed the case at the default da=128. Our analysis in Section 2 of the original paper (Observation 2) shows that the transition structure LAViF encourages, i.e., dominant transition directions that are grounded in instrument–tissue regions, is already present in some encoders, such as DINOv2-L and SurgeNet-L, where LAViF accordingly adds less. Even so, other capacities da improve individual components on these two encoders over No IDM+FDM, e.g., target mAP on DINOv2-L at da=512 (+1.5) and triplet mAP on SurgeNet-L at da=768 (+2.1); see Figure 4(b) and Appendix A.13 of the revised paper.
References
[1] Selvaraju et al., Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization, ICCV 2017.
[2] Chefer et al., Transformer Interpretability Beyond Attention Visualization, CVPR 2021.
[3] Cheng et al., SurgXBench: Explainable Vision-Language Model Benchmark for Surgery, WACV 2026.
[4] Sivakumar et al., SASVi: Segment Any Surgical Video, IJCARS 2025.
[5] Hong et al., CholecSeg8k: A Semantic Segmentation Dataset for Laparoscopic Cholecystectomy Based on Cholec80, arXiv:2012.12453.