acceptodds
Under review as a conference paper at ICLR 2027

Shared Representations, Native Actions: Learning Generalizable Surgical Motion from Heterogeneous Kinematics

Abstract

Learning surgical tool-motion representations supports action-centric video understanding and autonomous surgery. Existing surgical video foundation models emphasize appearance and semantics, while video-only latent-action models may capture distractors without physical supervision. We present (Surgical Action Representation through Inverse Dynamics Embedding), a surgical video foundation model pretrained through kinematics-supervised inverse dynamics on heterogeneous robot data. Embodiment-specific heads provide kinematic supervision for a shared visual encoder in each robot’s native action space. An auxiliary tool-centric forward dynamics model connects latent actions to visible instrument motion and enables learning from videos without kinematic annotations. Across OpenH, SAR-RARP50, GraSP, and SurgPose, our frozen encoder outperforms the evaluated visual representation baselines on action regression, action recognition, and point tracking. Few-shot adaptation further transfers action prediction to an unseen robot embodiment. In real-robot experiments, augmenting policy training with model-inferred action labels improves needle-pickup success from 60% to 80% and inter-instrument transfer success from 0% to 30% over the limited-demonstration baseline. These results demonstrate the value of coupled forward–inverse dynamics pretraining for transferable surgical representations and the effectiveness of inferred action labels for downstream control.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Surgical Robot Learning, Motion Representation Learning, Inverse Dynamics Models, Multi-Embodiment Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_78bcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SAR-IDE, a surgical motion representation model trained on heterogeneous robotic surgical data. It uses inverse dynamics with embodiment-specific action heads, together with a tool-centric forward dynamics objective based on instrument masks, point tracks, and optical flow. The model is mainly pretrained on OpenH with synchronized kinematics, with additional intraoperative videos, and is evaluated on action regression, action recognition, point tracking, unseen-embodiment transfer, and downstream robot control.

Critical strengths
  1. The paper studies the problem of learning shared representations across surgical robots with different action spaces. Using embodiment-specific action heads with a shared visual encoder is a reasonable design choice for handling heterogeneous kinematics.

  2. The paper evaluates the learned representation on several motion-related tasks, including action regression, action recognition, point tracking, few-shot transfer to an unseen robot embodiment, and a real-robot policy experiment, providing evidence of transfer across different motion-focused settings.

Critical weaknesses

1.The "surgical video foundation model" claim is broader than the current evaluation and baseline coverage support. Most downstream tasks are closely related to motion or action: action regression, action recognition, point tracking, and robot control. These results support SAR-IDE as a surgical motion/action representation model, but do not yet demonstrate broad surgical video understanding. A foundation-model evaluation should also cover more diverse tasks, such as surgical phase recognition and semantic segmentation. In addition, the comparison with recent surgical foundation models is incomplete. SurgVISTA [1] is cited but not included as an experimental baseline, and its reference should be updated from the arXiv version to the published npj Digital Medicine paper (2026). Also, the recent surgical foundation model LemonFM [2] (CVPR 2026) should also be included in the comparison.

[1] Large-scale self-supervised video foundation model for intelligent surgery. npj Digital Medicine, 2026.

[2] LEMON: A Large Endoscopic MONocular Dataset and Foundation Model for Perception in Surgical Settings. CVPR 2026.

2.Downstream comparisons are confounded by target-domain exposure during pretraining. SAR-IDE uses training videos from SAR-RARP50 and GraSP during pretraining through tool-centric visual supervision, and both datasets are later used for downstream action recognition. The authors exclude the corresponding validation and test videos, however, SAR-IDE still receives additional in-domain exposure that may not be available to the compared frozen encoders. Therefore, the gains on RARP50 and GraSP may reflect both the proposed method and additional target-domain pretraining.

3. The source of the performance gains over external baselines is not fully disentangled. SAR-IDE is initialized from the same V-JEPA 2.1 checkpoint used as one of the main baselines, and is then further pretrained with domain-specific surgical data and tool-centric supervision. Therefore, the improvement over V-JEPA 2.1 may reflect both the proposed learning framework and the additional surgical pretraining recipe. A controlled comparison with matched data, supervision, and training budget would help isolate the contribution of the proposed method. It would also be informative to report a from-scratch SAR-IDE variant to show how much the final performance depends on the V-JEPA 2.1 initialization.

Questions for authors

Questions:

  1. Can the authors evaluate SAR-IDE on broader surgical video tasks, such as phase recognition or semantic segmentation, and compare with recent surgical foundation models such as SurgVISTA and LemonFM?
  2. How much do the RARP50 and GraSP results depend on using their training videos during pretraining? Could the authors report results without this target-domain exposure?
  3. How much of the improvement over V-JEPA 2.1 comes from the proposed method versus additional surgical pretraining? A matched-data comparison would help clarify this.
  4. What is the full computational cost of the SAR-IDE pretraining pipeline, including the generation of instrument masks, point tracks, and optical flow? Reporting the preprocessing cost in addition to the main training cost would help clarify the overall efficiency and practical scalability of the method.

Addressing these questions could lead me to increase my rating.

3 comments

Thank you for identifying the distinction between a general surgical video model and a motion-focused representation model. We address evaluation scope, target-domain exposure, the V-JEPA control, and compute below.

1. Scope across surgical video tasks

Thanks for pointing out the missed SOTA LemonFM. SurgVISTA has not released the model weights so we tested LemonFM on our task and added new step recognition task on PitVQA, and phase recognition on GraSP. We use SAR-IDE trained only on OpenH for phase recognition, so that neither model has seen GraSP videos.

ModelPose regression PSM1 / PSM2 MSE ↓Action Recognition Acc. / F1 ↑Point Tracking ACC@8 / ACC@16 ↑Phase Recognition Acc. / F1 ↑Step Recognition Acc. / F1 ↑
V-JEPA 2.10.227 / 0.43383.55 / 76.6976.55 / 90.3561.57 / 48.3567.23 / 43.85
SurgMotion-G0.211 / 0.43373.03 / 61.9067.72 / 86.8761.74 / 46.3370.87 / 49.66
LemonFM0.285 / 0.44957.89 / 43.2159.99 / 82.3869.23 / 56.7960.05 / 41.99
SAR-IDE0.115 / 0.14491.45 / 89.1884.52 / 95.1168.19 / 57.7471.19 / 51.74

LemonFM is pretrained on 938 h of surgical video, roughly twice ours. Even so, SAR-IDE reaches comparable phase accuracy with the best F1, and outperforms finer-grained step recognition. SAR-IDE also leads by large margins on other tasks, we therefore position SAR-IDE as a surgical motion and action representation rather than a general surgical video foundation model, and will revise the wording accordingly in the final manuscript.

2. Target-domain exposure on RARP50 and GraSP

We evaluated a SAR-IDE checkpoint trained only on OpenH video–kinematics data:

PretrainingRARP50 Acc. ↑RARP50 F1 ↑GraSP mAP ↑
V-JEPA 2.185.0065.5039.85
SAR-IDE86.5966.2743.16
SAR-IDE (OpenH only)85.8666.6445.64

Performance does not decrease significantly when target-domain training videos are removed.

3. Framework contribution vs additional surgical pretraining

We evaluated a V-JEPA 2.1 checkpoint continued on the surgical data with its original objective. Kindly refer to the Answer to Question 1 raised by Reviewer gPys The conclusion is that exposure to more surgical video under the original V-JEPA objective is insufficient to produce the SAR-IDE improvement.

4. Training and preprocessing cost

The main SAR-IDE run uses 32 GPUs for 12,000 optimizer steps and approximately 33 hours of summed wall time. Preprocessing requires approximately 1.35k GPU-hours under our recorded configuration. Since videos and clips are processed independently, its wall-clock time can be substantially reduced through multi-GPU parallelism, while larger per-GPU batch sizes can further shorten required time. Mask, point-track, and optical-flow generation run offline and shard independently by video, with one worker per GPU and modest memory (~3.6 GiB for SAM 3.1, 4.9–7.0 GiB for SEA-RAFT; CoTracker3 fits on a single GPU), so they run on commodity GPUs. Annotations are cached and reused, so this one-time cost adds no overhead to training or inference.

Thank you for the clarifications and additional experiments.

However, my main concern about the surgical video foundation model claim remains. The authors propose narrowing the scope to surgical motion and action representations, but this change is not yet reflected in a revised manuscript. Evaluating a general surgical video foundation model requires broader task coverage than evaluating a surgical domain-specific representation model focused on motion and action.

The new phase recognition evaluation uses GraSP, which contains only 13 surgical videos, and SAR-IDE achieves lower accuracy than LemonFM. Broader evaluations, such as Cholec80 phase recognition, and an important surgical task such as semantic segmentation, are still missing. Without these evaluations, the current evidence remains insufficient to support the broader surgical video foundation model claim.

Thank you for raising this distinction. We agree that broader phase-recognition and semantic-segmentation evaluations would be necessary to support general surgical-video understanding, but this is definitely beyond the scope of this work. Our intended claim is narrower: SAR-IDE learns generalizable representations of surgical instrument motion and action from heterogeneous robot kinematics. To mitigate any potential misunderstanding, we have now reflected this scope in the revised manuscript by replacing “surgical video foundation model” with “surgical action representation model” in the Abstract, Introduction, Contributions, and Conclusion, with the revised terms marked in blue. Thanks for your suggestion!

Accordingly, our main evaluations focus on tasks that directly require instrument-motion or action information: action regression, atomic action recognition, instrument point tracking, few-shot transfer to an unseen embodiment, and downstream policy learning. In contrast, phase recognition describes procedure-level states and can rely strongly on anatomy, appearance, scene context, and temporal workflow, while semantic segmentation primarily evaluates single-frame spatial semantics. These tasks may benefit from motion information, but they do not directly measure whether a representation captures generalizable instrument motion across robot embodiments. We report the GraSP phase-recognition result transparently, but do not use it to claim uniformly stronger general surgical-video understanding. To avoid any misunderstanding of our core motivation of learning transferable motion representations from surgical robotics video, in the revised manuscript, we no longer make the broader surgical video foundation model claim.

Reviewer_gPysOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SAR-IDE continues a V-JEPA 2.1 video backbone with kinematics-supervised inverse dynamics on about 462 hours of Open-H-Embodiment video with synchronized kinematics plus 22.4 hours of intraoperative video without kinematics. Per-embodiment heads regress each robot's native actions, and a tool-centric forward-dynamics decoder predicts future instrument masks, tracks and flow from pseudo-labels. Frozen-representation probes against eleven public encoders report the lowest MedbotWorld translation error (Table 1), the best recognition scores on three datasets (Table 2) and the lowest tracking error at every frame gap (Table 3). A robot study with ten trials per arm uses SAR-IDE to pseudo-label demonstrations for policy training (Fig. 5).

Critical strengths
  1. Broad, same-initialisation, leakage-controlled evidence on three task families. SAR-IDE is initialised from the same checkpoint as the V-JEPA 2.1 baseline, test videos are excluded from pretraining, and each encoder-probe task uses a shared probe. Under these conditions the encoder is the best of the twelve on MedbotWorld translation error (Table 1), on all three recognition sets (Table 2) and at every tracking gap (Table 3), with wide margins on regression and tracking.
  2. A coherent framework for heterogeneous and missing supervision, with an in-framework ablation. The masked joint objective lets kinematics, pseudo-labelled dynamics and kinematics-free video train one encoder. Removing kinematic supervision sharply raises OpenH regression error and lowers recognition accuracy (Table 4).
Critical weaknesses
  1. The named contributions are attribution claims that no experiment isolates. Every baseline is a frozen public checkpoint while SAR-IDE receives a large in-domain training run, and there is no row in which V-JEPA 2.1 is continued on the same data with its own objective. The Table 4 rows are not shown to be data-matched: per-row training data are not stated, and the IDM row cannot use the 22.4 hours of kinematics-free video by construction. The tool-centric FDM distils SAM 3, CoTracker3 and SEA-RAFT outputs that the "Latent FDM" row never sees. There is no shared-head or single-embodiment row, so the title's "native actions" thesis has no controlled evidence, and the alignment pairs in Fig. 13 were "selected for high cosine similarity". Table 8 shows that most of the 30-frame tracking gain, and a large share at shorter gaps, arrives without any kinematics. The representation result is established; the claimed mechanism is not.
  2. Point estimates only, and the abstract headlines the least powered result. No table reports seeds or intervals, and several wins amount to only a few test segments, including the SMARTS accuracy margin over GR00T-H (Table 2). The robot study (Fig. 5) has ten trials per arm, so differences of a few successes are not statistically distinguishable, yet the introduction says the method can "significantly raise the success rate". There is no arm in which another encoder pseudo-labels the same 40 episodes, so the study shows that pseudo-labelling helps, not that SAR-IDE is responsible.
Questions for authors

I would intent to increase the ranking if the authors could sufficient answer my questions.

  1. Please add a V-JEPA 2.1 model continued with its own objective on the same 462 h + 22.4 h with the same compute, and report it in Tables 1-3. How much of the gain over the public V-JEPA 2.1 checkpoint survives?
  2. For each Table 4 row, which data subsets and hours were used, with what training budget, and does Latent FDM see the 22.4 h of kinematics-free video? Can you add a data-matched IDM versus SAR-IDE comparison?
  3. Please report a single shared action head and a single-embodiment-only variant under the Table 4 protocol, evaluated on MedbotWorld as well as the OpenH test splits. If the shared head matches native heads, the title-level claim should be revised.
  4. For the robot study, what are the pickup and transfer rates when the 40 pseudo-labels come from V-JEPA 2.1 or GR00T-H with the same MLP head trained on the same 10 ground-truth episodes?
  5. What is the pseudo-label trajectory error against the available ground truth on those 40 episodes, and does the policy's state input for the pseudo-labelled episodes use ground-truth robot state?
  6. Please report mean and standard deviation over at least three probe seeds and several 5-episode adaptation draws for Table 1 and Figure 3, and bootstrap confidence intervals for Table 2.
  7. All encoder probes use one learning rate and a fixed number of steps, and most baselines land within a narrow band on PSM2. Would per-encoder probe tuning change the ranking?
5 comments

Thank you for the constructive comments. We address the seven questions in order.

1. Continued V-JEPA 2.1

V-JEPA 2.1 continued on our surgical data with its original objective and matched compute:

ModelMedbot PSM1/2 ↓SMARTS PSM1/2 ↓RARP50 Acc/F1 ↑SMARTS Acc/F1 ↑GraSP ↑SurgPose @8/@16 ↑
V-JEPA 2.10.949/0.4550.223/0.42985.00/65.5083.55/76.6939.8576.55/90.35
Continued1.196/0.6720.247/0.46483.41/62.1576.32/61.9840.7978.16/91.67
SAR-IDE0.750/0.4150.115/0.14486.59/66.2791.45/89.1843.1684.51/95.11

Continuation hurts regression and recognition, only marginally helps GraSP and tracking, and stays far below SAR-IDE. Our corpus is orders of magnitude smaller and narrower than the original pretraining data, so the same self-supervised recipe drifts the representation toward this narrow distribution, whereas SAR-IDE's kinematic and tool-centric supervision extracts action-relevant structure from the same data. Additional data and compute therefore do not explain the gain. We will add all experimental results to Sec. 5 accordingly.

2. Data-matched Table 4

VariantOpenH w/ kin.Video w/o kin.
IDM462 h0 h
SAR-IDE (OpenH only)462 h0 h
Latent FDM462 h22.4 h
SAR-IDE462 h22.4 h

Same budget for all rows. SAR-IDE (OpenH only) is data-matched to IDM.

ModelPSM1 ↓PSM2 ↓Rec. Acc ↑Rec. F1 ↑Track@8 ↑Track@16 ↑
IDM0.0720.05586.5386.6582.1694.88
Latent FDM0.0540.04085.9282.9882.9594.94
SAR-IDE (OpenH)0.0470.03988.0588.3683.6494.21
SAR-IDE0.0500.03890.0688.7984.5295.11

Under matched data SAR-IDE still beats IDM, so the gain comes from the objective. The extra 22.4 h further improves recognition and tracking. Latent FDM sees strictly more visual information yet does not improve recognition and shifts attribution away from instruments, consistent with spending capacity on camera motion and tissue deformation. We will add all experimental results to Sec. 5 accordingly.

3. Native vs. shared action heads

Both encoders are pretrained on the Stanford and Turin subsets with the same training budget.

Test embodimentNative ↓Shared ↓
Stanford0.22670.3198
Turin0.07520.0902
Overall0.18810.2613

Both heads reached full convergence, yet the native heads clearly yielded better results, indicating the large variance between two embodiments. Stanford (real dVRK tissue scenes) and Turin (plastic phantoms) are two embodiments in Open-H dataset that differ in visual domain and action space (coordinate frames, scales, control dimensions).

4. Baseline pseudo-label control

The same head on frozen V-JEPA 2.1 features, trained on the same 10 GT episodes, labels the same 40 episodes.

Policy dataPickupTransfer
10 GT60%0%
+ 40 V-JEPA 2.150%10%
+ 40 SAR-IDE80%30%

V-JEPA 2.1 pseudo-labels do not reproduce the gain. With 10 trials per arm the differences are not statistically significant. We will report them as consistent with the label errors below and will remove "significantly". We will add all experiment results to Sec. 5.2.

5. Pseudo-label error and state

Accumulated-trajectory error on the 40 episodes (Trans. mm / Rot. deg / Jaw deg):

ModelPSM1PSM2
V-JEPA 2.17.08/38.89/8.836.73/27.14/7.39
SAR-IDE4.58/35.70/5.842.84/26.99/7.37

The policy takes no robot state (state dropout = 1.0), so no GT state enters training.

6. Probe variability

PSM1/PSM2 error, mean±std over 3 probe seeds (s) or 3 adaptation draws (d):

V-JEPA 2.1SurgMotion-GSAR-IDE (Emb.)SAR-IDE (Lat.)
Medbot, s.935±.013/.451±.004.984±.005/.465±.001.747±.004/.418±.003.673±.003/.393±.002
Medbot, d.903±.014/.456±.018.938±.033/.475±.011.762±.020/.418±.015.685±.015/.408±.015
SMARTS, s.223±.010/.429±.007.213±.002/.433±.004.115±.001/.144±.001.118±.000/.135±.000
SMARTS, d.354±.073/.363±.119.340±.069/.342±.090.162±.026/.136±.030.157±.029/.129±.030

7. Per-encoder tuning

Recognition probes already select per encoder over five LR/WD settings in paper. For regression, we tuned the four strongest encoders over LR {1e-5, 3e-5, 1e-4} and {250–1000} steps by validation loss, and evaluated once on the 65 held-out episodes.

EncoderPSM1 ↓PSM2 ↓Avg ↓
V-JEPA 2.11.1180.4750.797
SurgMotion-G0.9890.4650.727
VideoMAEv21.1010.5590.830
SAR-IDE (Emb.)0.7400.4180.579

MSE. Tuning does not change the main action-regression conclusion.

Reviewer_gPysOpenReview ↗

Reply to the rebuttal

Thanks for the additional experiments. They address most of my questions.

For Q3, I’d still like to see a direct comparison of the encoders. The shared head might have higher action errors on Stanford and Turin simply because it has to handle two different action spaces. That alone doesn’t show that native heads give a better shared encoder, as claimed in the title. Could you run the same frozen-encoder probes (MedbotWorld regression, recognition, and SurgPose tracking) on the shared-head model too, and report the results alongside those for the native-head model?

I’ll finalize my score after reading your reply.

We sincerely appreciate your response! For Q3, apologize for misunderstanding the question, and we would like to clarify 2 points before listing experimental results as requested:

  1. Per-embodiment action heads are not a standalone contribution. We follow the established GR00T-H design, which uses a shared backbone with embodiment-specific action heads to accommodate differences in action dimensions, EEF coordinate conventions, and robot base frames.
  2. “Native actions” refers to supervision in each embodiment’s original action space, not to the use of separate heads. Both the shared-head and native-head variants learn from the same native, unaligned action annotations. Their comparison therefore only evaluates whether embodiment-specific heads improve the shared representation. It does not show that learning from heterogeneous native actions is ineffective.

Here are the added results for the frozen-encoder probe. Both encoders are pretrained on the Stanford and Turin subsets with the same training budget.

Pre-training HeadMedbot PSM1 MSE ↓Medbot PSM2 MSE ↓Recognition Acc. ↑Recognition F1 ↑Tracking Acc@8 ↑Tracking Acc@16 ↑
Native Multiple Heads0.13890.112183.9081.7679.0492.69
Single Shared Head0.14320.111983.6180.6678.2592.20

Native heads perform slightly better overall, but the margins are small. This is expected, since both variants receive the same native-action supervision, which is the core of our method. The head design only determines how that supervision is decoded. We therefore view embodiment-specific heads as an implementation choice following GR00T-H, which remains preferable because it avoids forcing incompatible action dimensions into one output and extends naturally to new embodiments.

To avoid ambiguity, we will remove “native action” from the title and revise the manuscript to clarify that native-action supervision means learning from kinematic annotations in each embodiment’s original action space. Our main claim concerns the transferable representations learned through inverse-dynamics pretraining, rather than the use of embodiment-specific heads.

Reviewer_KkhNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SAR-IDE, a surgical video foundation model pretrained via kinematics-supervised inverse dynamics (IDM) on heterogeneous robot data from the Open-H-Embodiment dataset (~462 hours). Embodiment-specific action heads supervise a shared visual encoder in each robot's native action space, while a tool-centric forward dynamics model (mask, point-track, and optical-flow prediction) regularizes latent actions and enables training on unlabeled videos. Validation includes few-shot action regression on unseen embodiments, action recognition, point tracking, and real-robot needle pickup/transfer, where pseudo-labeled demonstrations raise pickup success from 60% to 80% and transfer success from 0% to 30%.

Critical strengths
  1. A well-motivated and validated solution to heterogeneous kinematic supervision.
  2. Unusually comprehensive empirical validation spanning representation probes to real-robot deployment.
Critical weaknesses
  1. Limited-scale and potentially circular real-robot evaluation. The policy results rest on only 10 trials per condition, so the reported gains (e.g., transfer 0%→30%, i.e., 3 successful trials) are within statistical noise; no confidence intervals or significance tests are reported. More importantly, the pseudo-labels are generated by SAR-IDE itself, which is initialized from the same V-JEPA 2.1 checkpoint used as the strongest baseline—and the action head is trained on the same 10 GT episodes used by the baseline—making it difficult to disentangle whether gains come from the pretrained representations or simply from 4× more training trajectories. A control using pseudo-labels from an off-the-shelf baseline representation (e.g., V-JEPA 2.1 features with the same MLP head) is necessary to attribute the improvement to SAR-IDE specifically. This is consequential because the real-robot experiment is the paper's headline practical claim.
  2. Insufficient theoretical/methodological depth in Section 3. The method section is largely descriptive; key design choices lack justification or analysis: (a) the joint loss (Eq. 4) does not report the values of λ_mask, λ_track, λ_flow, nor any sensitivity analysis—without this, reproducibility and the relative contribution of each FDM branch are unclear; (b) there is no analysis of how the latent action dimension (d_z=512, Q=4 queries) trades off against information loss (the paper itself notes the bottleneck hurts RARP50 recognition but offers no principled treatment); (c) the claim that tool-centric prediction provides "implicit alignment across embodiments" is never formalized or empirically probed beyond visualization. Given that the paper's central contribution is conceptual, the absence of any analysis (even empirical ablations over λ or latent dimension) weakens the scientific contribution. The paper also lacks a Limitations and Future Work discussion, which ICLR reviewers are explicitly asked to weigh.
  3. Heavy reliance on unvetted arXiv sources, undermining the scholarly foundation. A large fraction of cited work—and several load-bearing references, including the key baselines V-JEPA 2.1, DINO-Video-L, SurgMotion-G, GR00T-H, LARY, and the core latent-action model LAPA's strongest successors—are arXiv preprints without peer review. While some are unavoidable in this fast-moving area, the paper would be substantially strengthened by (i) citing peer-reviewed versions where they exist (e.g., EndoViT, EndoSSL, SurgeNet, VideoMAE v2 already have published versions), and (ii) acknowledging that conclusions drawn relative to unreleased/unreviewed baselines (GR00T-H model card as a "method") should be interpreted with caution.
  4. Baseline comparability concerns in the frozen-probe evaluations. The comparisons, while extensive, mix settings in ways that complicate interpretation: SAR-IDE receives 462h of in-domain kinematics-supervised pretraining while most baselines are general-purpose; SMARTS is part of OpenH (some baselines, e.g., GR00T-H, are pretrained on OpenH and thus see related data, while V-JEPA is not); DINO-Video-L/SurgMotion-G use 256×256 inputs while SAR-IDE uses 384×384; and GR00T-H is evaluated via "frozen final VL embeddings" rather than its native interface. The improvements are consistent across tasks, which mitigates this concern, but a controlled comparison (e.g., V-JEPA 2.1 fine-tuned with the same IDM objective and data, to isolate what SAR-IDE adds over its own initialization) is missing.
  5. Missing supplementary evidence materials and reproducibility gaps. For a paper whose central claim is improved motion representation for robot manipulation, the absence of any supplementary video (rollouts, tracking overlays, side-by-side real-robot trials, failure cases) is a notable omission—qualitative dynamics are notoriously hard to judge from static figures alone. Additionally, while the appendix is thorough on data curation, key implementation details (loss weights, FDM decoder size/depth L, prompt-sampling procedure, per-branch validation) are incomplete, and no code/checkpoint release commitment is stated. Given the complexity of the training pipeline, this limits reproducibility.
Questions for authors
  1. Limited-scale and potentially circular real-robot evaluation. The policy results rest on only 10 trials per condition, so the reported gains (e.g., transfer 0%→30%, i.e., 3 successful trials) are within statistical noise; no confidence intervals or significance tests are reported. More importantly, the pseudo-labels are generated by SAR-IDE itself, which is initialized from the same V-JEPA 2.1 checkpoint used as the strongest baseline—and the action head is trained on the same 10 GT episodes used by the baseline—making it difficult to disentangle whether gains come from the pretrained representations or simply from 4× more training trajectories. A control using pseudo-labels from an off-the-shelf baseline representation (e.g., V-JEPA 2.1 features with the same MLP head) is necessary to attribute the improvement to SAR-IDE specifically. This is consequential because the real-robot experiment is the paper's headline practical claim.
  2. Insufficient theoretical/methodological depth in Section 3. The method section is largely descriptive; key design choices lack justification or analysis: (a) the joint loss (Eq. 4) does not report the values of λ_mask, λ_track, λ_flow, nor any sensitivity analysis—without this, reproducibility and the relative contribution of each FDM branch are unclear; (b) there is no analysis of how the latent action dimension (d_z=512, Q=4 queries) trades off against information loss (the paper itself notes the bottleneck hurts RARP50 recognition but offers no principled treatment); (c) the claim that tool-centric prediction provides "implicit alignment across embodiments" is never formalized or empirically probed beyond visualization. Given that the paper's central contribution is conceptual, the absence of any analysis (even empirical ablations over λ or latent dimension) weakens the scientific contribution. The paper also lacks a Limitations and Future Work discussion, which ICLR reviewers are explicitly asked to weigh.
  3. Heavy reliance on unvetted arXiv sources, undermining the scholarly foundation. A large fraction of cited work—and several load-bearing references, including the key baselines V-JEPA 2.1, DINO-Video-L, SurgMotion-G, GR00T-H, LARY, and the core latent-action model LAPA's strongest successors—are arXiv preprints without peer review. While some are unavoidable in this fast-moving area, the paper would be substantially strengthened by (i) citing peer-reviewed versions where they exist (e.g., EndoViT, EndoSSL, SurgeNet, VideoMAE v2 already have published versions), and (ii) acknowledging that conclusions drawn relative to unreleased/unreviewed baselines (GR00T-H model card as a "method") should be interpreted with caution.
  4. Baseline comparability concerns in the frozen-probe evaluations. The comparisons, while extensive, mix settings in ways that complicate interpretation: SAR-IDE receives 462h of in-domain kinematics-supervised pretraining while most baselines are general-purpose; SMARTS is part of OpenH (some baselines, e.g., GR00T-H, are pretrained on OpenH and thus see related data, while V-JEPA is not); DINO-Video-L/SurgMotion-G use 256×256 inputs while SAR-IDE uses 384×384; and GR00T-H is evaluated via "frozen final VL embeddings" rather than its native interface. The improvements are consistent across tasks, which mitigates this concern, but a controlled comparison (e.g., V-JEPA 2.1 fine-tuned with the same IDM objective and data, to isolate what SAR-IDE adds over its own initialization) is missing.
  5. Missing supplementary evidence materials and reproducibility gaps. For a paper whose central claim is improved motion representation for robot manipulation, the absence of any supplementary video (rollouts, tracking overlays, side-by-side real-robot trials, failure cases) is a notable omission—qualitative dynamics are notoriously hard to judge from static figures alone. Additionally, while the appendix is thorough on data curation, key implementation details (loss weights, FDM decoder size/depth L, prompt-sampling procedure, per-branch validation) are incomplete, and no code/checkpoint release commitment is stated. Given the complexity of the training pipeline, this limits reproducibility.
1 comment

Thank you for the constructive comments. We address each point below.

1. Pseudo-label accuracy and real-robot improvement

We trained the same action head on frozen V-JEPA 2.1 features with the same 10 GT episodes and evaluated both models on the same 40 held-out demonstrations. Accumulated-trajectory error (Trans. mm / Rot. deg / Jaw deg):

ModelPSM1PSM2
V-JEPA 2.17.08/38.89/8.836.73/27.14/7.39
SAR-IDE4.58/35.70/5.842.84/26.99/7.37

SAR-IDE yields more accurate pseudo-trajectories on all coordinates. We then trained policies with each set of pseudo-labels, keeping all else unchanged:

Policy dataPickupTransfer
10 GT60%0%
+ 40 V-JEPA 2.150%10%
+ 40 SAR-IDE80%30%

V-JEPA 2.1 pseudo-labels do not reproduce the gain, so it does not come merely from 4× more trajectories. With 10 trials per arm the differences are not statistically significant; we report them as consistent with the label errors above and will remove "significantly".

2. Ablations

All rows use the same probes. Latent FDM and SAR-IDE use identical data and budget; SAR-IDE (OpenH) matches IDM on the 462 h OpenH data without the 22.4 h kinematics-free video.

ModelPSM1 ↓PSM2 ↓Rec. Acc ↑Rec. F1 ↑Track@8 ↑Track@16 ↑
IDM0.0720.05586.5386.6582.1694.88
w/o Kinematics0.9080.84679.6075.9980.4894.06
Latent FDM0.0540.04085.9282.9882.9594.94
SAR-IDE (OpenH)0.0470.03988.0588.3683.6494.21
SAR-IDE0.0500.03890.0688.7984.5295.11

Removing kinematics sharply degrades regression and weakens recognition and tracking, so physical action supervision is central. Latent FDM improves regression over IDM but not recognition, consistent with whole-scene prediction retaining action-irrelevant background changes. SAR-IDE gives the strongest overall transfer, and its gain over IDM remains under matched data.

Task-conditioned attribution supports this:

VariantOn instr. ↑Nearby ↑Peak on instr. ↑Top-5% mass
IDM27.4%54.6%67.0%0.568
Latent FDM21.3%48.2%68.8%0.404
SAR-IDE37.3%65.0%92.0%0.523

SAR-IDE shifts attribution toward the instruments without producing the most concentrated map. We have not swept loss weights or latent dimension, and will narrow "implicit alignment" to a shared observation-space supervision signal across embodiments.

3. Limitations and future work

We will add this section. The IDM-based formulation learns through robot-level action prediction, so it does not capture fine-grained information beyond the action, such as per-instance motion of instruments or tissue. We plan to train VLMs to perform inverse dynamics, aiming for more robust robot state estimation that reflects how surgeons operate, and to evaluate its benefit for other robotics tasks.

4. References

We will cite peer-reviewed versions where available and distinguish peer-reviewed methods from model-card or preprint baselines.

5. Continued V-JEPA 2.1

We continued V-JEPA 2.1 on our surgical data with its original objective and compute matched to SAR-IDE.

ModelMedbot PSM1/2 ↓SMARTS PSM1/2 ↓RARP50 Acc/F1 ↑SMARTS Acc/F1 ↑GraSP ↑SurgPose @8/@16 ↑
V-JEPA 2.10.949/0.4550.223/0.42985.00/65.5083.55/76.6939.8576.55/90.35
Continued1.196/0.6720.247/0.46483.41/62.1576.32/61.9840.7978.16/91.67
SAR-IDE0.750/0.4150.115/0.14486.59/66.2791.45/89.1843.1684.51/95.11

Continuation hurts regression and recognition, only marginally helps GraSP and tracking, and stays far below SAR-IDE. Our corpus is far smaller and narrower than the original pretraining data, so the same recipe drifts the representation toward this distribution, whereas SAR-IDE's supervision extracts action-relevant structure from the same data.

6. Reproducibility

We will add a supplementary video with tracking overlays, side-by-side robot trials, rollouts, and failure cases, and will release code and checkpoints. Details: the mask–tracking and flow decoders each use 8 LAPA-style Transformer blocks (width 1,024, 16 heads) with four latent-action tokens; loss weights for action, mask, tracking, and flow are 1, 0.1, 0.005, and 0.005; we keep up to three instruments visible at both endpoints and sample up to 100 points from their masks, denser near the tips. Training uses 32 GPUs for 12,000 steps (~33 h). Code and model weights will be released upon acceptance

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.