acceptodds
Withdrawn from ICLR 2027

STEP: Surgical Point Tracking from Sparse Endpoints and Pseudo-trajectories

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Surgical point tracking requires persistent correspondence through tissue deformation and occlusion, yet dense trajectory annotations are costly to obtain. We present STEP, a two-stage framework that learns from surgical video and sparse physical endpoints. First, a frozen offline teacher generates candidate trajectories, and temporal reversibility and stereo consistency select dense supervision for a causal student. Second, a robust physical endpoint loss anchors terminal correspondence, while a round-trip loss constrains temporal consistency. A gradient-free prefix exposes training to accumulated drift, while a final differentiable window bounds activation memory. The trained tracker processes a single RGB stream with fixed weights. In STIR-to-VL-SurgPT transfer, temporal–stereo union selection reduces endpoint error by 7.0% relative to unfiltered trajectories. On a post-hoc diagnostic of 29 VL-SurgPT clips excluded from the endpoint-training manifest, STEP reduces endpoint error from 31.52 to 16.80 pixels relative to dense-label fine-tuning. These results support combining curated trajectories with physical observations to train surgical point trackers.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
point tracking, surgical video

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ritPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a framework for adapting point trackers to surgical videos, combining temporal–stereo consistency-based pseudo-label filtering with endpoint-guided streaming training. The approach addresses limited trajectory annotations and long-sequence training constraints. While the application is relevant, my main concerns are the limited differentiation from existing methods and the lack of closely related adaptation baselines.

Critical strengths
  1. The paper addresses a practically important problem: learning reliable surgical point tracking from limited annotations.

  2. The streaming training strategy offers a potentially useful way to incorporate long-sequence supervision without modifying the inference architecture.

Critical weaknesses
  1. Limited methodological novelty beyond existing consistency-based adaptation strategies

My main concern is that the methodological contribution is not sufficiently distinguished from closely related work. Several central components closely resemble existing approaches. SurgTracker [1] already uses a frozen teacher, forward–backward consistency filtering, and student fine-tuning. Endo-TTAP [2] employs staged adaptation with pseudo-labels and final-frame supervision, while MFTIQProb [3] uses stereo consistency to assess tracking reliability, albeit during inference. While none of these works uses exactly the same pipeline, the paper does not make clear what is methodologically new about putting these components together. The main novelty seems to lie in endpoint-guided streaming training, but the paper does not clearly explain how this differs from standard endpoint-supervised fine-tuning. Without a clearer distinction, I am not convinced that the contribution is strong enough for ICLR.

  1. Insufficient comparison with relevant adaptation methods.

The comparison between training with manual trajectories and the proposed approach is useful for studying supervision choices, but it does not establish superiority over existing surgical-domain adaptation methods. SurgTracker [1] is a particularly important missing baseline because it also fine-tunes a pretrained tracker without adding architectural parameters. Endo-TTAP [2] is another relevant surgical tracking baseline, with architectural differences appropriately acknowledged. Without comparisons under matched data and supervision budgets, it is difficult to determine whether the improvements result from the proposed training design, additional supervision, or domain adaptation more generally.

Questions for authors
  1. What is the essential methodological difference between endpoint-guided streaming training and conventional endpoint-supervised fine-tuning, and what evidence demonstrates the benefit of that difference?

  2. How does the method compare with SurgTracker-style adaptation using the same backbone, training data, and supervision budget? Can the authors also provide a comparison with Endo-TTAP?

  3. Why do you reset point memory and query descriptors at each gradient-window boundary instead of carrying over the full state with gradients detached? Does this reset reflect what happens during continuous streaming inference, and is there evidence to support that? Have the authors separately compared initializing from predicted positions with initializing from accurate boundary positions?

Reviewer_kcMDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents STEP, a two-stage framework for surgical point tracking. It first uses temporal and stereo consistency to select trajectories from an offline teacher for training a causal student. It then combines physical endpoint supervision with a round-trip consistency loss, using a gradient-free prefix and a differentiable tail to train on longer clips. At inference, the tracker uses a single RGB stream with fixed weights. Experiments examine trajectory selection, endpoint accuracy, and training configurations on STIR and VL-SurgPT. I have the following comments.

Critical strengths

1、A practical use of complementary supervision. STEP combines teacher-generated trajectories with existing physical endpoint measurements, reducing reliance on dense manual annotation. The STIR-to-VL-SurgPT experiment in Table 3 provides useful evidence for the curation stage: under the same optimization budget, union selection reduces EPE from 15.99 to 14.87 pixels without endpoint adaptation on the target dataset.

Critical weaknesses

1、The endpoint-training results lack a clearly independent test set. Section 5.1 reports that 121 of the 150 evaluation clips also appear in endpoint training. Tables 1 and 5 therefore cannot establish generalization to unseen clips. The remaining 29 clips contain only 73 endpoints and were identified post hoc; Appendix C does not establish procedure-level independence or rule out prior use in model selection. The authors disclose these limitations, but they substantially weaken the evidence for the complete framework. The clean STIR-to-VL-SurgPT experiment validates trajectory curation rather than the endpoint-training stage.

2、The loss comparison does not isolate the benefit of cycle consistency. In Table 7, adding the cycle loss also increases endpoint clips per optimizer step from two to five, the maximum endpoint-clip length from 192 to 256 frames, and the gradient tail from 16 to 24 frames. The EPE reduction from 22.26 to 16.80 pixels in Table 4 therefore compares different training recipes. It remains unclear how much comes from the cycle objective versus greater endpoint exposure and longer gradient propagation. An endpoint-only control with these settings matched is needed to establish the value of combining the two losses.

Questions for authors

1、Can you establish procedure-level separation for the 29 diagnostic clips and clarify whether they influenced checkpoint or hyperparameter selection?

2、Does the joint objective still improve over endpoint-only training when both use five endpoint clips per step, a maximum span of 256 frames, and a 24-frame gradient tail?

3、How does the final endpoint+cycle model perform on intermediate trajectories in the clip-disjoint subset? Appendix D.1 reports intermediate tracking results for the endpoint-only control, leaving unclear whether the final model’s endpoint gains also improve the intervening path.

Reviewer_JuTuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Surgical point tracking is difficult to train due to the absence of dense temporal supervision. The authors propose STEP, a two-stage framework to address this issue. In the first stage, they use pseudo-trajectories generated using frozen AllTracker, and filtered with temporal and stereo consistency checks for fine-tuning a causal student tracker. In the second stage, the tracker is further optimized with endpoint and cycle losses. Exploiting a stereo dataset, the proposed tracks are further filtered for stereo consistency. The second stage exploits datasets where only the start and end points are known. As the for each starting point it is unclear, which of the marked end points in the data set is the corresponding one the authors introduce a set of novel losses. This approach refines the accumulated error. The novel selection and sparse-endpoint training can be applied to multiple existing trackers and consistently improve their tracking results.

The authors train on STIR and evaluate the transfer performance on VL-SurgPT. They do endpoint-guided training (stage-2) on a 603-clip manifest that overlaps with 121 clips from VL-SurgPT, so they also show a post-hoc clip-disjoint evaluation on the remaining 29 clips. Further evaluations are shown with multiple trackers on the full VL-SurgPT benchmark.

The method shows improvement but the general novelty is somewhat limited. The training regime of using a pretrained teacher and selecting specific tracks has been followed by many previous approaches and is now only slightly refined. The endpoint-guided training sounds interesting but is unfortunately not very well described in the given text.

Critical strengths
  • The problem is practically motivated, as obtaining dense trajectories in surgical videos is difficult.
  • The combination of dense pseudo-labelled trajectories and sparse endpoint supervision, in a two-stage fashion, helps in learning better temporal correspondence.
  • The authors show that endpoint-guided training helps multiple trackers.
Critical weaknesses
  • The paper is difficult to read and grasp at some places, specifically Section 4. The structure of this section appears a little disorganized. It should first give a full overview before discussing the details of the individual steps.
  • While the paper provides some interesting experiments, most of the results are mainly quantitative. I am missing an in-depth discussion of the cases in which the approach still fails.
  • The paper has very limited methodological novelty. Training with pseudo-trajectories has already been explored in BootsTAP, CoTracker3, and Track-on-R, while temporal cycle consistency and stereo consistency are well-established techniques for learning and validating correspondences. Moreover, filtering pseudo-trajectories with temporal consistency checks have also been explored in surgical point tracking [1]. The main contribution therefore seems to be a combination of these ideas with sparse endpoint supervision for surgical tracking. It’s unclear what new technical contribution the proposed approach offers.
  • The endpoint-plus-cycle ablation is confounded. As shown in Appendix B, different parameters were used for endpoint-only and endpoint-plus-cycle experiments. Therefore, the improvement can’t be attributed specifically to the cycle consistency objective.
  • The paper reports results on 150 videos of VL-SurgPT. However, since stage-2 training already used data overlapping with 121 videos from SurgPT, the full-set results and insights can’t be trusted. Although the authors show results also on a 29-video disjoint subset, it can still have procedure-level similarities and is a relatively small test set for making any broad claims.
  • In Table 4, the different ablations don’t clearly show the benefit of individual additions. Adding endpoint loss or cycle loss to S1, makes EPE worse, only adding both of them leads to an improvement. Also, it’s not clear if stage 1 is needed.

[1] Bundele, V., Hosseinzadeh, M. and Lensch, H., 2025. You are your best teacher: semi-supervised surgical point tracking with cycle-consistent self-distillation.CVPR 2025 Workshop SynData4CV.

Questions for authors
  • What are the cases that still cannot be tracked? Can they be classified?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.