acceptodds
Under review as a conference paper at ICLR 2027

Dense-Temporal Surgical Video Object Segmentation via Agentic Trajectory Recovery

Abstract

High-frame-rate surgical video object segmentation (VOS) exposes finer-grained intraoperative dynamics over densely sampled sequences. Existing VOS methods often fail to detect unreliable propagation and recover once tracking drift contaminates memory. We propose SurgATR, an Agentic Trajectory Recovery framework for dense-temporal surgical VOS. SurgATR first performs trajectory reflection to identify propagation failures from recent target evolution. Once triggered, an agent directly intervenes in both spatial and temporal tracker states: spatially, it re-anchors the target with corrective prompts; temporally, it rolls back and reconstructs tracker memory from a reliable state to prevent error accumulation. To avoid incorrect recovery toward visually similar distractors, cycle validation further filters identity-inconsistent recoveries, while global Viterbi selection determines a globally coherent trajectory. To support systematic study of dense-temporal surgical VOS, we introduce SurgDVOS, the largest high-frame-rate surgical VOS benchmark, with 330,246 annotated frames and 869,922 pixel-level masks, supporting both mask-guided and referring VOS. Experiments on SurgDVOS and three public natural-video benchmarks show consistent improvements, with SurgATR achieving state-of-the-art performance in mask-guided and referring VOS while demonstrating strong cross-domain generalization.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Surgical Video Object Segmentation, Surgical Scene Understanding, Agentic Recovery

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_jmkWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles the well-known problems of tracking drift and memory contamination in densely sampled surgical video object segmentation. The proposed method, SurgATR, identifies unreliable propagation based on recent trajectories and, when such unreliability is detected, invokes a vision-language agent to perform spatial re-anchoring and memory rollback. Recovered hypotheses are then consolidated through reverse-trajectory validation and a global Viterbi selection. In addition, the authors introduce SurgDVOS, a new dataset comprising 558 clips from 50 procedures, with 330,246 annotated frames and 869,922 instance masks. The evaluation covers both mask-guided and referring segmentation settings on surgical as well as natural videos.

Critical strengths

The dataset could be of lasting value to the community. SurgDVOS provides dense instance trajectories that capture challenging phenomena such as occlusion, reappearance, fast motion, and distractors with similar-looking instruments, while supporting two distinct target-specification settings. Beyond serving as a conventional benchmark for ranking segmentation methods, these annotations could facilitate more systematic research on failure detection and recovery.

The joint design of spatial correction and memory recovery is well motivated and is supported by solid component-level evidence. As shown in Table 6, spatial recovery, memory recovery, and their combination achieve 63.8, 63.4, and 65.0 J&F, respectively, while cycle validation further improves performance to 65.6. These results convincingly demonstrate the complementarity of the two intervention points.

Critical weaknesses

The algorithmic advance over the closest memory-management and multi-hypothesis approaches is not established. SAM2Long [1] already maintains independent memory pathways and performs video-level selection to mitigate error accumulation. SENTRY [2] uses candidate backtracking and cycle-consistent matching to control memory contamination, while Refer-Agent [3] provides a reflection-driven agentic correction framework. Consequently, multiple hypotheses, reflection, cycle validation, and global selection do not individually establish novelty. Retrospective rollback is meaningfully different from preventive memory filtering, but direct comparisons are needed to determine whether it provides a substantial new capability. ReSurgSAM2 [4] is cited but absent from the main comparisons. [1] Ding S, Qian R, Dong X, et al. Sam2long: Enhancing sam 2 for long video segmentation with a training-free memory tree[C]//2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025: 13614-13624. [2] Alansari M, Michael Y, AlMarzouqi H, et al. Sentry: Sam2-enhanced neighbor-aware and temporally reasoned memory for visual tracking[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026: 599-619. [3] Jiang H, Liang T, Zheng W S, et al. Refer-agent: A collaborative multi-agent system with reasoning and reflection for referring video object segmentation[J]. arXiv preprint arXiv:2602.03595, 2026. [4] Liu H, Gao M, Luo X, et al. Resurgsam2: Referring segment anything in surgical video via credible long-term tracking[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2025: 435-445.

It remains unclear whether cycle validation actually tests identity consistency. The current criterion primarily evaluates reverse-trajectory reliability through area, overlap, motion, and presence cues. Consistently tracking a different but similar instrument may also produce plausible temporal evolution. Relative to SENTRY’s explicit matching against target and distractor histories, the proposed validation needs a clearer correspondence to the established target identity. Aggregate J&F improvements and removing cycle validation do not sufficiently distinguish identity correction from generic candidate-quality filtering.

Questions for authors

How does SurgATR compare against SAM2Long, SENTRY, and ReSurgSAM2 when all methods share the same backbone, initialization, and inference budget? It would also help to know which component contributes most to the reported gains—is it the recovery-candidate generation, the retrospective rollback, or the final selection step? An ablation or attribution analysis here would clarify this. With the re-anchoring prompts held fixed, how do the alternative memory strategies perform—e.g., simply retaining the memory, clearing it, selecting reliable historical frames, versus rolling back to a reliable anchor? Such a comparison would directly demonstrate whether the rollback mechanism is independently necessary. In the cycle validation, is the reverse endpoint explicitly compared against the anchor target in terms of mask or identity features? Additionally, I would suggest reporting identity switches, recovery success rate, and recovery delay separately, rather than relying solely on the aggregate J&F metric, as this would give a clearer picture of the recovery behavior. The authors should clarify the training data and action supervision used for the recovery agent, as well as whether the SurgDVOS split is performed at the patient or procedure level to avoid data leakage. It is also unclear what data the agent checkpoint evaluated on natural videos was trained on—please specify the training sources.

Reviewer_9EKjOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SurgDVOS, a densely annotated surgical video dataset, and SurgATR, a framework for recovering tracking failures for video object segmentation. This method checks tracking reliability and uses a VLM agent to correct target localization and restore earlier tracking memory. I like the part the authors not only test the improved performance on their benchmark but also two natural-video benchmarks, and all show improvements on object segmentation.

Critical strengths
  1. The motivation is clear. Tracking errors can contaminate memory thus just correct one frame is not enough, making recovery an important problem in surgical video segmentation.
  2. The method combines relocation with memory rollback. This addresses both the current segmentation error and the stored information that could cause further drift.
  3. They also provide a densely annotated dataset, this is valuable for this field of research.
  4. The ablation study examine the main components separately and support the complementary benefits of spatial correction and memory recovery.
  5. Their evaluation covers both mask-guided and referring segmentation across surgical and natual video benchmarks, providing a more borader evidence beyond their own dataset.
Critical weaknesses
  1. several core implementation details remain unclear, including the reflection function, selection of reliable anchors, how the recovery agents are trained, and how the trajectory scoring is doing. It is hard to understand the entire workflow.
  2. they don't compare the performance across different frame rates, thus it is hard to compare how their method works for the dense temporal sampling dataset over other smapling rate.
Questions for authors
  1. what is the actual training/validation split? "all methods follow the official dataset splits and evaluation protocols", I get confused how many videos covered in the test results reported?
  2. How are the reflection score and the latest reliable anchor determined? Can you provide more details on ur implementation setting?
Reviewer_PyvhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work proposes an approach for video object segmentation approach that refines and recovers errors in segmentations across frames. The approach involves an agentic intervention strategy on encountering errors: upon spatially recovering the errors, it also "temporally recovers" by rolling back to a more reliable memory state. The second contribution of this work is SurgDVOS - a high-frame-rate surgical VOS dataset. The work shows improved performance on both the SurgDVOS dataset, as well as on more general public VOS benchmarks.

Critical strengths
  • The large surgical VOS dataset would be a useful addition for future works to benchmark their methods.
  • The proposed approach shows an extremely strong performance on existing benchmarks, especially on referring video object segmentation.
Critical weaknesses
  • One aspect which is not mentioned in this work is the speed/cost of this method compared with competitor approaches. Due to the nature of refinement and recovery, the approach seems like it may be slow when compared with, e.g. SAM 2. Can we include the time it takes to process videos with this approach.
Questions for authors
  • Was there an exploration on the effect of the base tracker (in this case SAM 2-Small) and the overall quality. Since SAM 2.1-Large shows comparable results on mask-guided VOS, would the agentic trajectory recovery lead to similar improvements in quality if the base model is already quite strong?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.