acceptodds
Under review as a conference paper at ICLR 2027

Latent-Action-Guided Video-Language Feature Learning for Surgical Instrument–Tissue Interaction Recognition

Abstract

Recognizing instrument–tissue interactions is essential for context-aware surgical AI. Vision-language models offer a natural way to inject semantic structure into surgical representations by aligning video features with textual action descriptions. However, pretrained encoders may lack spatial coherence, while global semantic alignment does not ensure precise spatial and temporal representations. By analyzing frame-to-frame feature changes, we find that semantic alignment increases their dimensionality, but larger increases do not necessarily improve recognition; encoders also differ in how strongly dominant changes localize to interaction regions. Motivated by these findings, we introduce LAViF, which compresses frame-to-frame changes into latent actions and predicts next-frame features during end-to-end video–language alignment. Without additional spatial or motion annotations, LAViF improves the interaction grounding of leading feature changes and temporal-direction sensitivity in our evaluated settings. We further characterize how action capacity and prediction strength affect recognition across encoders and triplet components. Using image encoders without large-scale video pretraining, LAViF achieves competitive recognition with faster inference and smaller INT4 accuracy drops than V-JEPA2/2.1, supporting its deployment potential.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Surgical action recognition, Video representation learning, Vision-language learning

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_Bs5gOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • Empirical analysis of visual transition structure is interesting. This provides a useful empirical observation motivating the distinction between representing more visual changes and representing more task-relevant changes.
Critical strengths
  • Empirical analysis of visual transition structure is interesting.

  • Integration of predictive learning and semantic alignment is coherent.

  • Empirical results investigate transition dimensionality, spatial grounding, temporal reversal, latent-action capacity, and prediction-loss weighting.

Critical weaknesses
  • The latent-action representation is not convincingly shown to encode meaningful surgical actions. The paper does not convincingly demonstrate that the latent-action space is organized around surgical interaction semantics rather than arbitrary visual changes useful for reconstructing features.

  • Although the paper reports promising performance, the matched ablation results indicate that adding the full LAViF framework does not consistently improve triplet recognition.

Questions for authors
  • What evidence demonstrates that the learned latent actions represent meaningful instrument–tissue interaction dynamics rather than arbitrary visual transitions?
1 comment

We thank the reviewer for the constructive comments.

Question 1 and Weakness 1. To examine whether the latent actions capture instrument–tissue interaction rather than arbitrary visual change, we ran a new experiment: Grad-CAM [1] of the action token alone (Table R1; Appendix A.11 of the revised paper). Its maps fall on the instrument ∪ target region more often than chance on all three encoders; scoring Grad-CAM maps against segmentation masks is a commonly used grounding measure [2, 3]. We do not claim that the token encodes surgical action labels. How it is computed: a scalar is back-propagated to the patch tokens of the labeled (last) frame entering the encoder's last block; weighting the token channels by their patch-averaged gradients gives one relevance value per patch. The scalar is either the ground-truth triplet score with the aggregator's state input detached, so that the gradient reaches the encoder only through the action tokens (via token), or the squared norm of the most recent action token (token norm). IoU: the top-25% patches vs. the instrument ∪ target mask, i.e., the union of the segmentation masks of the instruments and target tissues of the frame's ground-truth triplets (chance: 11.9). Masks: estimated by SASVi [4], whose segmentation model is trained on expert annotations of cholecystectomy anatomy (CholecSeg8k [5]). Result: compared with IDM only (λpred=0\lambda_{\text{pred}}=0; same token, no prediction loss), the token of full LAViF is more focused on these regions on ViT-L (via token: 16.60 vs. 13.47; token norm: 20.40 vs. 10.79) and SurgeNet-L (via token: 16.81 vs. 15.27), and similar on DINOv2-L (15.45 vs. 14.95).

Table R1. Grad-CAM IoU (%) of the action token with the instrument ∪ target region on CholecT50 at da=128d_a=128 (mean±std over three seeds; chance ≈11.9).

EncoderVariantToken normVia token
ViT-LIDM only (da=128d_a=128)10.79±2.8813.47±0.94
ViT-LFull LAViF (da=128d_a=128)20.40±1.3916.60±1.24
DINOv2-LIDM only (da=128d_a=128)15.97±1.0414.95±1.35
DINOv2-LFull LAViF (da=128d_a=128)16.76±0.4915.45±1.20
SurgeNet-LIDM only (da=128d_a=128)14.72±1.8315.27±0.91
SurgeNet-LFull LAViF (da=128d_a=128)15.71±2.3016.81±0.49

Weakness 2. This is indeed the case at the default da=128d_a=128. Our analysis in Section 2 of the original paper (Observation 2) shows that the transition structure LAViF encourages, i.e., dominant transition directions that are grounded in instrument–tissue regions, is already present in some encoders, such as DINOv2-L and SurgeNet-L, where LAViF accordingly adds less. Even so, other capacities dad_a improve individual components on these two encoders over No IDM+FDM, e.g., target mAP on DINOv2-L at da=512d_a=512 (+1.5+1.5) and triplet mAP on SurgeNet-L at da=768d_a=768 (+2.1+2.1); see Figure 4(b) and Appendix A.13 of the revised paper.


References

[1] Selvaraju et al., Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization, ICCV 2017.

[2] Chefer et al., Transformer Interpretability Beyond Attention Visualization, CVPR 2021.

[3] Cheng et al., SurgXBench: Explainable Vision-Language Model Benchmark for Surgery, WACV 2026.

[4] Sivakumar et al., SASVi: Segment Any Surgical Video, IJCARS 2025.

[5] Hong et al., CholecSeg8k: A Semantic Segmentation Dataset for Laparoscopic Cholecystectomy Based on Cholec80, arXiv:2012.12453.

Reviewer_XY2YOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a method for surgical action-triplet recognition using video–language alignment. For this, they perform an analysis of frame-to-frame feature transitions across six encoders on CholecT50, which motivates LAViF, which extends an image backbone with a temporal aggregator with a causal inverse-dynamics model, a forward-dynamics model and SIGReg regularization. They evaluate on two public datasets for action-triplet recognition in surgery: CholecT50 and ProstaTD. Results include controlled baselines and an ablation study.

Critical strengths
  • The provided study/comparison is strong as all encoders share the same text encoder, objective and clip setup. The transition-structure diagnostics are a useful tool for determining how semantic alignment reshapes representations. This matters as the paper's main contribution is analytical rather than state-of-the-art accuracy.
  • The run-time analysis/optimization is practically relevant for intraoperative use.
Critical weaknesses
  • The authors main claim is not supported by the baselines. First, table 1 does not include the temporal baselines for the three backbones, ViT and SurgeNet are missing completely, but are in fig. 2a) and table 2, where they seem to match or outperform LAViF on CholecT50 and the performance on ProstaTD is within fold variance. A statistical analysis would be useful here
  • The language component is not really tested, it amounts to text embeddings of fixed class names under closed-set BCE. Semantic and generalization benefits are not tested as there is neither a no zero-shot or held-out-triplet evaluation nor a comparison with a linear classification head
Questions for authors
  • Why are the matched temporal baselines, already reported in Figure 2a and Table 2, omitted from Table 1? Please add them for all backbones and both datasets.
  • What does the no-IDM+FDM baseline score with patch-level SIGReg added? If it matches full LAViF, the latent-action contribution is unsupported.
  • Can you report multiple seeds on CholecT50 for Tables 2-4, and significance tests on both datasets? Were the baselines tuned comparably, given the differing per-component learning rates in Table 5? Does text-based alignment outperform a learned linear head? Minor: A lot of used abbreviations and other terms (e.g. LAViF and Sigreg) are not explained.
3 comments

We thank the reviewer for the constructive comments.

Question 1 and Weakness 1. Yes, we have added them: the main comparison (Table 2 of the revised paper) now lists the eight-frame No IDM+FDM baselines for all three backbones on both datasets; Table 1 had grouped only single-frame and native video encoders. On the main claim: as the reviewer notes, our contribution is analytical rather than state-of-the-art accuracy. We study how reshaping the transition structure, i.e., the spectrum of the transition covariance Σδ\Sigma_\delta, through latent-action training affects the learned features; recognition is one part of this evaluation, next to grounding and reliance on frame order. For encoders that already show the desired structure in their dominant directions (DINOv2-L and SurgeNet-L; Observation 2 in Section 2 of the original paper), LAViF accordingly adds little in recognition, and the original paper already reports this encoder dependence (Section 4.1: ViT-L improves on all four metrics, whereas DINOv2-L and SurgeNet-L remain slightly below their triplet baselines on CholecT50). In triplet mAP, full LAViF is higher on ViT-L (32.65 vs. 29.72) and comparable on DINOv2-L and SurgeNet-L on CholecT50, and higher in mean on all three encoders on ProstaTD (32.82/38.14/38.85 vs. 30.79/36.27/34.81).

Question 2. We trained this new baseline (No IDM+FDM + state/patch SIGReg; Table 3 of the revised paper), also over three seeds. In triplet mAP on CholecT50, it is below full LAViF on ViT-L (31.15 vs. 32.65); on DINOv2-L it is comparable to LAViF at da=128d_a=128 (32.65 vs. 32.25) and below it at da=512d_a=512 (34.06); on SurgeNet-L it is comparable (32.14 vs. 32.40), with lower verb and target mAP (68.22 vs. 71.27 and 47.66 vs. 49.59). On ProstaTD, it is below full LAViF on ViT-L (30.78 vs. 32.82) and comparable on DINOv2-L and SurgeNet-L (37.48 vs. 38.14 and 37.23 vs. 38.85).

Question 3 and Weakness 2. Seeds. Yes, we added them: the ablation, reversal and INT4 tables (Tables 3, 5 and 6 of the revised paper) report mean±std over three seeds on CholecT50. Significance. We report the standard deviation over three seeds (CholecT50) and over the official folds (ProstaTD). Tuning. Yes, the comparison is controlled: all learning rates are listed in Appendix A.5 of the original paper (the table the reviewer refers to), and full LAViF and its baselines use the same rates for all shared modules; only IDM and FDM, which the baselines do not have, have their own. Linear head and language component. Our study is about vision–language models, and we could not run a linear-head variant within the rebuttal period. To test the language component, we added a new zero-shot evaluation, the standard test of text-aligned models [1, 2]: CholecT50-trained models are applied to ProstaTD without retraining (89 triplets, most instruments and anatomy unseen; Table R1, and Table 7 of the revised paper). Accuracy is modest but above random on every component; full LAViF is comparable to or slightly above No IDM+FDM and above No IDM+FDM + state/patch SIGReg. Notably, on all three encoders the best zero-shot triplet mAP is reached at a large capacity (da=1024d_a=1024, 10241024 and 768768), so a larger dad_a appears to help transfer; Appendix A.12 of the revised paper reports the zero-shot results for the full dad_a sweep. Terminology. We now define LAViF and SIGReg at first use.

Table R1. Zero-shot transfer CholecT50→ProstaTD without ProstaTD training: mAP (%) on the 89 ProstaTD triplets, mean±std over the five fold test sets. Full LAViF at the dad_a with the highest zero-shot triplet mAP; best value per encoder in bold. Revised paper: Table 7; all capacities in Appendix A.12.

EncoderVariantIVTIVT
ViT-LFull LAViF (da=1024d_a=1024)9.52±1.8350.14±1.8029.82±1.2227.64±1.43
ViT-LNo IDM+FDM9.02±1.1349.38±2.0727.69±1.2027.01±2.16
ViT-LNo IDM+FDM + state/patch SIGReg8.32±0.8647.64±2.4928.18±1.7827.75±2.06
DINOv2-LFull LAViF (da=1024d_a=1024)10.01±1.0251.09±2.5430.86±1.5229.47±1.85
DINOv2-LNo IDM+FDM9.74±1.1749.59±2.1930.71±1.4530.55±3.84
DINOv2-LNo IDM+FDM + state/patch SIGReg8.44±0.7446.51±2.3827.22±0.7328.97±1.98
SurgeNet-LFull LAViF (da=768d_a=768)13.17±1.0265.72±2.4440.20±1.1735.50±1.78
SurgeNet-LNo IDM+FDM12.80±1.1964.46±3.2340.02±1.1031.94±1.68
SurgeNet-LNo IDM+FDM + state/patch SIGReg11.57±0.7568.62±4.3341.45±1.7231.91±1.03
Random scores4.8637.9221.2619.80

References

[1] Radford et al., Learning Transferable Visual Models From Natural Language Supervision, ICML 2021.

[2] Yuan et al., Learning Multi-Modal Representations by Watching Hundreds of Surgical Video Lectures, Medical Image Analysis, 2025.

Overall, the authors have addressed some of my comments and questions, in particular by adding the matched temporal baselines, the SIGReg-only ablation, and results over multiple seeds. However, the latent-action component only appears to benefit ViT-L consistently. For DINOv2-L and SurgeNet-L, the gains over the SIGReg-only baseline are absent at the default d_a = 128 or fall within the reported variance. For the zero-shot analysis, I would like to know whether the differences between full LAViF and the baselines would hold up under a statistical test, such as a paired test across folds and seeds. The reported gaps are within one standard deviation, and LAViF is shown at its best-performing d_a per encoder. A Linear-head comparison is still missing. Without it, the benefit of the language component over a standard classifier remains unshown. Also my question regarding whether transition grounding or reversal sensitivity predicts recognition within a backbone is still unanswered. I would therefore not update my score

Thank you for your follow-up. We have taken your comments into account and revised our claims as follows.

Contribution. Motivated by the cross-encoder link in Observation 2, we change the transition structure through latent-action training and measure how it affects recognition, grounding and reliance on frame order, where the effect on recognition depends on the encoder. We evaluate these properties side by side, not as within-backbone predictors of recognition.

Statistics and claims. Our experiments run on a shared computing cluster with job queues and time quotas, and during the discussion period we were allocated two A100 GPUs for seven days; one training epoch takes about 40 minutes. Within this time, we prioritized completing three seeds for all variants on CholecT50, so that every result is reported with its seed variability, and we used this variability to calibrate the claims in the revised paper. Running additional seeds and evaluating zero-shot transfer for every seed, which the paired test you suggest requires, was not possible within the remaining time of the discussion period, given the queueing time for further jobs on the cluster. As the zero-shot gains vary across capacities and folds, we have narrowed our zero-shot claim from an improvement over the baselines to a trend that zero-shot transfer tends to benefit from a larger latent-action capacity, and will study it, together with a formal significance analysis, with larger-scale training or pretraining in future work.

Linear head. Our study focuses on visual feature learning, and the text head is shared by all compared variants. Given the same time and compute constraints, we were not able to train the linear-head variants within the discussion period; we have noted this as a limitation and will include the comparison in future work.

Thank you again for your constructive comments, which have helped us state our results and the scope of our claims more precisely in the revision.

Reviewer_ty5oOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents LAViF, a method for surgical instrument–tissue interaction recognition. It uses an inverse dynamics model (IDM) to learn latent actions from changes between video frames and a forward dynamics model (FDM) to predict future visual features. These components are trained together with video–language alignment and regularization. The authors evaluate LAViF on CholecT50 and ProstaTD, looking at recognition performance, spatial grounding, and sensitivity to temporal order. The idea is interesting, but I have some concerns about how the prediction objective is formulated, whether the proposed method really addresses the problems identified in the paper, and whether the experiments provide enough evidence for the claimed benefits.

Critical strengths
  1. The paper focuses on surgical instrument–tissue interaction recognition, where both spatial details and temporal changes are important.

  2. The method combines video–language alignment with latent-action learning, providing an alternative way to incorporate temporal information into surgical video representations.

  3. The paper evaluates LAViF with different visual encoders on two surgical datasets, allowing the authors to examine how the method behaves under different settings.

  4. Beyond recognition accuracy, the paper also analyzes spatial grounding and temporal-direction sensitivity, offering additional perspectives on the learned representations.

Critical weaknesses
  1. The IDM takes δt=st+1−st\delta_t=s_{t+1}-s_t as input to obtain ata_t, while the FDM uses ata_t to predict st+1s_{t+1}. This means that ata_t already contains information from the target state. How does the prediction loss provide a meaningful constraint on learning temporal representations in this setting?

  2. The paper observes that a larger transition space does not necessarily lead to better recognition, and that dominant transition directions differ in their spatial grounding. However, it is unclear why a low-dimensional LA bottleneck and next-state prediction should address these issues. Predictive objectives may also preserve background changes or camera motion, rather than interaction-relevant information.

  3. For DINOv2-L on CholecT50, the reported Triplet mAP is 27.41% in Table 1 but 33.52% for the No IDM+FDM baseline in Table 2. The ProstaTD results also differ slightly (36.01% vs. 36.27%).

  4. The claimed improvement in spatial grounding is not directly supported by the training objective. LAViF does not explicitly optimize instrument or tissue localization. Instead, spatial grounding is evaluated using transition-band IoU after training. Why should next-state prediction and SIGReg lead to better localization of instrument–tissue interactions, rather than simply improve general feature organization? More direct evidence is needed to establish this connection. Also, the ProstaTD analysis uses SAM2-estimated instrument regions only, which does not measure instrument–tissue interaction grounding in the same way as the CholecT50 evaluation.

  5. Temporal-direction sensitivity does not necessarily indicate better temporal understanding. The paper uses the mAP drop under clip reversal as evidence of temporal-direction sensitivity. However, a larger performance drop only shows that the model is more sensitive to reversed inputs; it does not necessarily mean that it better understands the temporal structure of surgical actions. Moreover, the Target reversal drop decreases from 13.0% to 9.7% for ViT-L, contrary to the trend reported for the other components.

  6. What do the learned latent actions actually represent? The LA is inferred from visual feature differences, without direct supervision from actual surgical actions or instrument trajectories. It may encode instrument motion, tissue deformation, camera motion, or other visual changes. The current ablations show that LA dimensionality affects recognition, but they do not establish what information the LA captures.

  7. The use of causal needs further clarification. Both the IDM and FDM use causal masks, but the final state–action aggregator is bidirectional. This is reasonable for offline recognition using a complete clip, but it does not establish that the overall model supports causal or online inference. In particular, ata_t depends on δt=st+1−st\delta_t=s_{t+1}-s_t, which requires the next frame. The authors should clarify the intended inference setting and distinguish causal processing within individual modules from the temporal constraints of the full model.

Questions for authors

Please see the weakness above.

4 comments

We thank the reviewer for the constructive comments.

Weakness 1. That ata_t sees st+1s_{t+1} is intended and common in latent-action models (LAPO [1]; Genie [2]): the loss is not meant to forecast the future, as in a world model, but to make ata_t learn what to retain from the transition. What it retains is characterized by Zhang et al. [3]: for fixed features in a simplified linear setting, the latent action keeps the dominant directions of the transition covariance Σδ\Sigma_\delta, whether they carry action-relevant change or noise. Our analysis in Section 2 of the original paper associates better recognition with dominant directions that are grounded in the interaction regions. We therefore train the encoder jointly, so that, with semantic supervision and with SIGReg preventing collapse, the dominant directions can carry more useful information. Also follow insight from Zhang et al. [3] under end to end training, predictive loss can be minimized by reducing the effective rank of Σδ\Sigma_\delta,
( at every capacity d90d_{90} below No IDM+FDM on all encoders; Figure 4(a)), which may keep the transition space from expanding too much and capture irrelevant variation.

Weakness 2. They help because the encoder is trained jointly: (a) the action token and the features also serve recognition, so semantic supervision can steer the dominant directions toward interaction-relevant change; (b) latent-action training already lowers the effective rank without a bottleneck, which may prevent part of the expansion of the transition space, and different dimensions of dad_a provide different strengths of how concentrated the transition variance is in dominant directions, which differs across encoders (Figure 4(a)). Our new Grad-CAM test places the recognition evidence more on the interaction region than without IDM+FDM (Table 4 of the revised paper).

Weakness 3. The inputs differ: the Table 1 baseline sees one frame (27.41/36.01 on CholecT50/ProstaTD), whereas No IDM+FDM in Table 2 aggregates eight (33.52/36.27); the revised captions state this.

Weakness 4. LAViF indeed has no localization loss, and this is by design (Section 1 of the original paper): surgical data are scarce, so no strong tool exists to localize tissue, and we want the encoder to learn grounding by itself. SIGReg is only a regularizer to prevent collapse [4]. We attribute the improvement mainly to points (a) and (b) of our response to Weakness 2. For the more direct evidence requested, we ran a new experiment: class-conditioned Grad-CAM [5], commonly scored against segmentation masks [6, 7] (Table R1; Table 4 of the revised paper). How it is computed: each ground-truth triplet score is back-propagated to the last-block patch tokens of the labeled frame; weighting the token channels by their patch-averaged gradients gives a per-patch relevance. IoU: the top-25% patches vs. the instrument ∪ target mask, i.e., the union of the masks of the instruments and target tissues of the frame's ground-truth triplets. Masks: estimated by SASVi [8]. Full LAViF exceeds its No IDM+FDM baseline on ViT-L (21.19 vs. 15.37), SurgeNet-L (18.43 vs. 16.35) and DINOv2-L at da=512d_a=512 (19.46 vs. 17.65). Appendix A.11 of the revised paper lists all capacities: LAViF is higher at every capacity on ViT-L and SurgeNet-L, and at da≥512d_a\ge512 on DINOv2-L. ProstaTD: SASVi needs a segmentation model trained for the same procedure, which exists for cholecystectomy (CholecSeg8k [9]) but, to our knowledge, not for prostatectomy, so we use instrument regions only (Appendix A.3 of the original paper).

Table R1. Class-conditioned Grad-CAM IoU (%) with the instrument ∪ target region on CholecT50 (same 5,000 test clips for every model; mean±std over three seeds). All capacities: revised paper, Appendix A.11.

MethodViT-L (da=128d_a=128)DINOv2-L (da=512d_a=512)SurgeNet-L (da=128d_a=128)
Full LAViF21.19±1.2719.46±1.6718.43±0.50
No IDM+FDM15.37±2.8217.65±1.2616.35±0.69
Random patches11.9011.8911.86

(Continued in our next comment, part 2 of 2.)


References

[1] Zhang et al., What Do Latent Action Models Actually Learn?, NeurIPS 2025.

[2] Schmidt & Jiang, Learning to Act without Actions, ICLR 2024.

[3] Bruce et al., Genie: Generative Interactive Environments, ICML 2024.

[4] Maes et al., LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels, arXiv:2603.19312.

[5] Selvaraju et al., Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization, ICCV 2017.

[6] Chefer et al., Transformer Interpretability Beyond Attention Visualization, CVPR 2021.

[7] Cheng et al., SurgXBench: Explainable Vision-Language Model Benchmark for Surgery, WACV 2026.

[8] Sivakumar et al., SASVi: Segment Any Surgical Video, IJCARS 2025.

[9] Hong et al., CholecSeg8k: A Semantic Segmentation Dataset for Laparoscopic Cholecystectomy Based on Cholec80, arXiv:2012.12453.

(continued from our previous comment.)

Weakness 5. Clip reversal is a common test of the temporal sensitivity of video models [10, 11]: it measures how much the predictions rely on frame order. To avoid any misunderstanding, the revised paper now describes this result as reliance on frame order rather than temporal understanding (Sections 1, 2 and 4.2). Regarding the ViT-L target drop, the reported decrease (13.0% to 9.7%) came from a single training run. Over three seeds, it is 10.8±1.6% for full LAViF and 9.9±2.7% for No IDM+FDM (Table 5 of the revised paper), so the decrease does not persist. A direct test would use tasks in which frame order is the only cue, such as distinguishing temporally opposite actions or before/after relations between events; these need expert annotations on medical verbs that CholecT50 and ProstaTD do not provide, so we leave this evaluation to future work

Weakness 6. The latent action encodes the change between frames, drawn preferentially from the instruments and the target tissue. We tested this in a new experiment (Table R2; Appendix A.11 of the revised paper) that applies the Grad-CAM protocol described in our response to Weakness 4 to the action token: we back-propagate either the triplet score with the aggregator's state input detached, so that the gradient passes only through the tokens (via token), or the squared norm of the latest token (token norm). Both maps overlap the instrument ∪ target mask above chance (11.9) on all encoders, and the via-token overlap is higher than for IDM only (same token, no prediction loss) on ViT-L and SurgeNet-L (16.60 vs. 13.47 and 16.81 vs. 15.27). It may also carry camera motion or other visual changes.

Table R2. Grad-CAM IoU (%) of the action token with the instrument ∪ target region on CholecT50 at da=128d_a=128 (mean±std over three seeds; chance ≈11.9).

EncoderVariantToken normVia token
ViT-LIDM only (da=128d_a=128)10.79±2.8813.47±0.94
ViT-LFull LAViF (da=128d_a=128)20.40±1.3916.60±1.24
DINOv2-LIDM only (da=128d_a=128)15.97±1.0414.95±1.35
DINOv2-LFull LAViF (da=128d_a=128)16.76±0.4915.45±1.20
SurgeNet-LIDM only (da=128d_a=128)14.72±1.8315.27±0.91
SurgeNet-LFull LAViF (da=128d_a=128)15.71±2.3016.81±0.49

Weakness 7. The full model uses no future frame: frame tt is classified from clip xt−7,…,xtx_{t-7},\ldots,x_t (Section 2 of the original paper), whose seven transitions the IDM reads, so it can run online. “Causal” refers to the IDM/FDM masks; the aggregator is bidirectional only within this window (Section 3 of the revised paper).


References

[10] Wei et al., Learning and Using the Arrow of Time, CVPR 2018.

[11] Bagad & Zisserman, Chirality in Action: Time-Aware Video Representation Learning by Latent Straightening, NeurIPS 2025.

Thank you for the responses. Some concerns have been addressed, but it remains unclear whether the latent actions capture surgical interactions rather than general visual changes. The Grad-CAM results show better spatial alignment but do not fully explain what the latent actions learn or the role of the prediction objective. I prefer to keep my rating.

Thank you for your follow-up and for pointing out what remains unclear, which we would like to clarify below.

The IDM takes the previous states as input and outputs the action tokens; the states s≤ts_{\le t} and the action token ata_t are then fed to the FDM to predict the next state st+1s_{t+1}, and all of them are finally passed to the aggregator to be aligned with the text features. Intuitively, this design encourages the action token to carry information that is useful for both next-state prediction and semantic alignment. Since the bottleneck on top of it limits how much information it can carry, the action token tends to capture meaningful information rather than irrelevant motion noise. We have also improved Figure 3 in the revised paper for better clarity.

What the latent actions actually learn, however, remains an open question for latent-action models in general: recent analyses show that latent actions can capture both controllable changes and distractors [1, 2]. The existing theory [1] covers only a simplified linear setting, so there is no established metric to determine exactly what information the action token encodes. To interpret what it encodes in our model, Table R2 of our previous response reports Grad-CAM of the action token itself, computed either only through the action tokens (via token) or from the token norm (token norm), rather than class-conditioned Grad-CAM of the whole model. For full LAViF, both maps overlap the instrument–target region above chance on all three encoders, and the overlap is lower in mean without the prediction loss (IDM only). We have added this discussion to Appendix A.11.

Overall, thank you again for your constructive comments, which have helped us clarify the design intuition behind LAViF and the interpretation of its latent actions in the revision. We hope this response addresses your remaining concerns.

References

[1] Zhang et al., What Do Latent Action Models Actually Learn?, NeurIPS 2025.

[2] Nikulin et al., Latent Action Learning Requires Supervision in the Presence of Distractors, ICML 2025.

We hope that the corrected figure and the clarified design intuition resolve the remaining questions, and if so, we would greatly appreciate it if you would consider revisiting your rating.

Reviewer_ujiBOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript analyzes inter-frame feature variations following semantic alignment, offering potentially useful theoretical insights for the community. It proposes a method for improving representations of surgical instrument–tissue interaction recognition through joint training with low-dimensional latent actions and next-state prediction. The evaluation covers recognition, spatial localization, sensitivity to temporal direction, and quantization performance.

Critical strengths
  1. The paper goes beyond reporting recognition scores by examining how training alters the transition spectrum and spatial organization of the learned representations. This is a valuable research question.
  2. The experimental evaluation is reasonably comprehensive, assessing the benefits of the proposed method across multiple backbones on two datasets.
Critical weaknesses
  1. The paper identifies limitations of “frozen pretrained representations” and insufficient local constraints from global semantic alignment. While the visual encoder is explicitly fine-tuned, the additional contribution of latent dynamics modeling beyond end-to-end adaptation could be better established. A matched baseline combining temporal aggregation with patch-level SIGReg, without IDM/FDM, would help distinguish the effects of latent dynamics modeling from those of patch-level regularization.
  2. Figure 5(b) provides limited evidence that LAViF substantially improves local interaction grounding beyond the underlying backbone. The incremental IoU gains over the “No IDM+FDM” baseline are uneven across transition bands, with improvements in some bands accompanied by declines in others. In particular, SurgeNet already exhibits a strong leading-band IoU peak without IDM/FDM, making it difficult to attribute the favorable absolute grounding performance to the proposed method.
  3. As a minor presentation suggestion, Figure 2(a) contains tabular results that could be presented as a separate table. This may make the numerical comparisons easier to reference and improve readability.
Questions for authors

1.In Table 2, at d_a = 128, the “No IDM+FDM” variant achieves higher triplet mAP than the full model on CholecT50 with both DINOv2-L (33.52 vs. 33.13) and SurgeNet-L (33.09 vs. 32.78). These results leave the consistency of its triplet recognition gains unclear. 2. Table 4 reports a decline in recognition performance when video clips are temporally reversed. However, reversing a surgical video does not change the spatial locations of the interactions. It is therefore unclear how this performance drop supports the claims relevant to the target task. The authors should clarify what this experiment is intended to demonstrate and how it relates to surgical interaction recognition and localization.

3 comments

We thank the reviewer for the constructive comments.

Weakness 1. Following this suggestion, we trained this new baseline (No IDM+FDM + state/patch SIGReg; Table R1, and Table 3 of the revised paper). In triplet mAP, full LAViF is higher on ViT-L (32.65 vs. 31.15) and on DINOv2-L at da=512d_a=512 (34.06 vs. 32.65), and comparable on SurgeNet-L (32.40 vs. 32.14), where its verb and target mAP are higher (71.27 vs. 68.22 and 49.59 vs. 47.66).

Table R1. CholecT50 and ProstaTD recognition mAP (%). CholecT50: mean±std over three seeds; ProstaTD: mean±std over five folds. No IDM+FDM + state/patch SIGReg keeps the state and patch-level SIGReg of full LAViF and removes only IDM and FDM. Full tables: revised paper, Tables 2 and 3.

EncoderVariantCholecT50 IVTCholecT50 VCholecT50 TProstaTD IVT
ViT-LFull (da=128d_a=128)32.65±0.5168.45±0.5247.35±0.6232.82±1.29
ViT-LNo IDM+FDM + state/patch SIGReg31.15±1.3065.53±0.3045.93±1.3430.78±1.83
ViT-LNo IDM+FDM29.72±1.8766.44±0.8645.24±0.7530.79±1.21
DINOv2-LFull (da=128d_a=128)32.25±0.5168.34±0.1747.83±0.1638.14±2.61
DINOv2-LFull (da=512d_a=512)34.06±0.3768.42±0.7250.23±0.9037.78±1.91
DINOv2-LNo IDM+FDM + state/patch SIGReg32.65±0.9868.71±0.3747.33±2.6337.48±1.97
DINOv2-LNo IDM+FDM33.62±1.7868.37±1.3348.75±1.2136.27±1.88
SurgeNet-LFull (da=128d_a=128)32.40±0.3771.27±1.2149.59±0.9238.85±2.07
SurgeNet-LNo IDM+FDM + state/patch SIGReg32.14±1.7868.22±2.9247.66±1.8537.23±2.73
SurgeNet-LNo IDM+FDM32.63±1.5869.38±1.5149.20±2.9534.81±2.12

Weakness 2. To provide more direct grounding evidence, we ran a new experiment: class-conditioned Grad-CAM [1] (Table R2; Table 4 of the revised paper), whose overlap with segmentation masks is a commonly used grounding measure [2, 3]. How it is computed: each ground-truth triplet score is back-propagated to the patch tokens of the labeled (last) frame entering the encoder's last block; weighting the token channels by their patch-averaged gradients gives one relevance value per patch. IoU: the top-25% patches vs. the instrument ∪ target mask, i.e., the union of the segmentation masks of the instruments and target tissues of the frame's ground-truth triplets. Masks: estimated by SASVi [4], whose segmentation model is trained on expert cholecystectomy annotations (CholecSeg8k [5]). Against its own No IDM+FDM baseline (so SurgeNet-L's existing peak is not credited to LAViF), full LAViF is higher on ViT-L (21.19 vs. 15.37), SurgeNet-L (18.43 vs. 16.35) and DINOv2-L at da=512d_a=512 (19.46 vs. 17.65). Appendix A.11 of the revised paper lists all capacities: LAViF is higher at every capacity on ViT-L and SurgeNet-L, and at da≥512d_a\ge512 on DINOv2-L.

Table R2. Class-conditioned Grad-CAM IoU (%) with the instrument ∪ target region on CholecT50 (same 5,000 test clips for every model; mean±std over three seeds). All capacities: revised paper, Appendix A.11.

MethodViT-L (da=128d_a=128)DINOv2-L (da=512d_a=512)SurgeNet-L (da=128d_a=128)
Full LAViF21.19±1.2719.46±1.6718.43±0.50
No IDM+FDM15.37±2.8217.65±1.2616.35±0.69
Random patches11.9011.8911.86

Weakness 3. As suggested, we moved these numbers into a separate table (Table 1 of the revised paper).

(Continued in our next comment, part 2 of 2.)


References

[1] Selvaraju et al., Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization, ICCV 2017.

[2] Chefer et al., Transformer Interpretability Beyond Attention Visualization, CVPR 2021.

[3] Cheng et al., SurgXBench: Explainable Vision-Language Model Benchmark for Surgery, WACV 2026.

[4] Sivakumar et al., SASVi: Segment Any Surgical Video, IJCARS 2025.

[5] Hong et al., CholecSeg8k: A Semantic Segmentation Dataset for Laparoscopic Cholecystectomy Based on Cholec80, arXiv:2012.12453.

( continued from our previous comment.)

Question 1. This is indeed the case at da=128d_a=128. Our analysis in Section 2 of the original paper (Observation 2) shows that the transition structure LAViF encourages, i.e., dominant transition directions that are grounded in instrument–tissue regions, is already present in some encoders, such as DINOv2-L and SurgeNet-L, where LAViF accordingly adds less. Even so, other capacities dad_a improve individual components on these two encoders over No IDM+FDM, e.g., target mAP on DINOv2-L at da=512d_a=512 (+1.5+1.5) and triplet mAP on SurgeNet-L at da=768d_a=768 (+2.1+2.1); see Figure 4(b) or Appendix A.13 of the revised paper.

Question 2. Our target task, surgical action triplet recognition, requires both spatial and temporal evidence (Section 1 of the original paper): where instrument and tissue interact, and how they change over time. The reversal test is a complementary experiment for the temporal side, and sensitivity to temporal direction is a common probe of video models [6, 7]: it measures the temporal sensitivity of LAViF, i.e., how much its predictions rely on frame order, and is not a localization test. Reversal keeps every frame, hence the spatial content, and changes only the order, so the mAP drop isolates the use of order, which verbs need: the same irrigator adds fluid when it irrigates and removes it when it aspirates. Over three seeds, the verb drop on ViT-L/DINOv2-L/SurgeNet-L is 13.5/15.3/17.0% for No IDM+FDM and 17.3/17.8/20.8% for full LAViF (Table 5 of the revised paper).


References

[6] Wei et al., Learning and Using the Arrow of Time, CVPR 2018.

[7] Bagad & Zisserman, Chirality in Action: Time-Aware Video Representation Learning by Latent Straightening, NeurIPS 2025.

Thank you again for your thoughtful and constructive review. We appreciate your recognition of our analysis beyond recognition performance. More importantly, your suggestions on strengthening the baseline comparisons and clarifying the presentation of our results have helped us significantly improve the paper. We would like to follow up on two points that may not have been sufficiently clear in our previous response.

  1. Regarding why SIGReg does not directly improve performance (Weakness 1): in LAViF, SIGReg is a regularizer paired with the FDM to prevent the vision encoder from collapsing to trivial solutions. It constrains how the features are distributed, not which visual changes they encode, so by itself it is not expected to make the representation more informative about instrument–tissue interactions. We also trained the suggested matched baseline (Table R1 of our previous response). Overall, adding SIGReg alone has an inconsistent effect: across encoders and components, it raises some scores and lowers others, and on CholecT50 it lowers triplet mAP on two of the three encoders.

  2. Regarding Figure 5(b) (Weakness 2), our original explanation may have caused some confusion. Figure 5(b) was not intended to demonstrate a substantial improvement in grounding, but rather a shift of interaction-region IoU toward the leading transition bands on all three encoders after applying our method, relative to each encoder's own No IDM+FDM baseline. As our original wording may have suggested a stronger claim, we have clarified it in the revision.

If these additional clarifications resolve your concerns, we would be grateful if you could take them into account in your final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.