acceptodds
Withdrawn from ICLR 2027

Coupling trajectories and Masks: An Efficient Sparse-to-Dense Annotation Framework via Bidirectional Enhancement for Event Motion Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Event motion segmentation is bottlenecked by dense pixel-level masks, whose exhaustive annotation is labor-intensive and poorly matched to asynchronous event streams. Existing event-based segmentation approaches are poorly suited as scalable annotation tools: they either expose only discrete frame outputs or require costly mask, optical-flow, or auxiliary-modality supervision, limiting temporal coverage and annotation per interaction. We propose an efficient sparse-to-dense annotation framework that couples trajectories and masks through bidirectional enhancement. In Motion-to-Mask, a learned temporal trajectory field warps events to queried timestamps, forming motion-compensated Images of Warped Events (IWEs) that allow a frozen promptable segmentation model to generate masks from sparse bounding boxes. In Mask-to-Motion, the generated masks provide region-level constraints for mask-aware contrastive trajectory learning, refining the motion estimates used for IWE construction and mask propagation. Because the trajectory field is queryable within each annotation window and the promptable segmentation model propagates masks across query-time IWEs, the framework can produce masks at a denser temporal output rate than the box-prompt rate and transfer them to asynchronous events. Across three datasets, the generated masks enable fine-grained annotation with full or sparse prompts and no pixel-level mask or optical-flow GT; a user study shows up to 389.39% speedup at a 1:3 input-to-output frame-rate ratio across event-data experience levels.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Motion Segmentation and Annotation, Event Camera, Event-wise Annotation Tool

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_s22fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a sparse-to-dense annotation framework for event motion segmentation through bidirectional enhancement between motion estimation and mask generation. The Motion-to-Mask module generates motion-compensated event representations for SAM2-based segmentation, while the Mask-to-Motion module employs mask-aware contrastive learning to refine trajectory estimation. The framework aims to generate dense masks and event-wise annotations from sparse bounding-box prompts without requiring pixel-level mask or optical-flow supervision.

Critical strengths

Practical motivation and application value.

The paper addresses the labor-intensive annotation problem in event motion segmentation. Generating dense annotations from sparse user interactions is a meaningful research direction with practical applications.

Reasonable integration of motion and segmentation.

The proposed bidirectional framework provides an intuitive connection between motion estimation and segmentation. Using motion-compensated representations to improve segmentation and segmentation information to guide motion learning is conceptually appealing.

Comprehensive experimental evaluation.

The paper evaluates the framework across multiple datasets and provides ablation studies, event-wise annotation experiments, and user studies. These results offer empirical support for the overall approach.

Critical weaknesses

Potential error propagation in bidirectional enhancement.

The proposed framework relies on SAM2-generated masks to refine motion estimation, while the quality of these masks depends on the estimated trajectories. This mutual dependency may reinforce incorrect predictions, particularly when the initial motion estimates or segmentation results are unreliable. The paper does not sufficiently establish the robustness of this feedback mechanism.

Questionable assumptions in mask-aware contrastive learning.

The proposed contrastive objective encourages features within the same segmentation region to be similar. However, spatially coherent regions do not necessarily exhibit consistent motion, especially for non-rigid objects or complex motion patterns. The suitability of this region-based supervision for general motion estimation requires stronger justification.

Unclear reliability of automatically generated annotations.

The framework aims to produce dense annotations from sparse human input, but its effectiveness depends on the quality of the estimated trajectories and propagated masks. The paper provides limited discussion of how annotation errors can be detected or corrected when generating pseudo-labels without dense supervision, raising concerns about reliability in practical annotation workflows.

Questions for authors
  • How does the proposed bidirectional enhancement prevent errors in motion estimation and segmentation from reinforcing each other? Is there any mechanism to identify or suppress unreliable pseudo-masks during training?
  • What justifies treating features within the same segmentation region as positive pairs for motion representation learning? How does this assumption hold for non-rigid objects or regions containing diverse motion patterns?
  • How does the framework ensure the reliability of automatically generated dense annotations when no ground-truth masks are available? What happens when incorrect masks are propagated across time?
Reviewer_FHrmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors proposed a sparse-to-dense annotation framework for event-based motion segmentation by coupling trajectory estimation and mask generation. First, in the Motion-to-Mask stage, the paper formulated a motion-compensated Images of Warped Events that used the learned trajectories and employs frozen SAM2 to generate segmentation masks from sparse bounding-box prompts. Then, in the Mask-to-Motion stage, the paper introduced a mask-aware contrastive learning objective to improve trajectory estimation using the generated masks. Experiments on EVIMO2, EVIMO, and DSEC-MOS demonstrate improvements in segmentation quality and annotation efficiency.

Critical strengths
  • The paper addresses a practical annotation bottleneck in event-based vision and provides a coherent framework connecting motion estimation with promptable segmentation.
  • Extensive experiments demonstrate improvements under sparse prompting.
Critical weaknesses
  • The proposed framework primarily combines existing components, including contrast-maximization-based trajectory estimation, SAM2-based segmentation, and InfoNCE-style contrastive learning. Although their integration is meaningful for event annotation, the individual techniques are largely established. The authors should clarify which components constitute the fundamental contributions. In addition, the authors should explain how the proposed bidirectional enhancement differs from conventional iterative pseudo-label refinement or self-training frameworks.
  • The proposed method uses bounding-box prompts and a powerful pretrained segmentation model, while several competing methods operate without prompts or rely on different supervision settings. Although the paper identifies these differences, directly comparing segmentation accuracy across these methods may overstate the proposed approach's advantages. A more controlled comparison under equivalent annotation budgets and segmentation backbones would better establish the contribution of the proposed framework.
  • The user study involves only 20 participants and five samples from EVIMO2. More importantly, participants were provided with ground-truth masks as visual references, which differs substantially from realistic annotation scenarios where such references are unavailable. The reported annotation speedups may therefore not fully reflect practical deployment conditions. The authors should evaluate the framework under more realistic annotation settings, including larger datasets, challenging examples, and correction of inaccurate automatically generated masks.
  • The paper would benefit from additional qualitative visualizations of failure cases, especially under fast motion, occlusion, and multiple independently moving objects.
Questions for authors

Please refer to the weaknesses.

Reviewer_6UZoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a sparse-to-dense annotation framework for event motion segmentation. It uses a continuous-time trajectory predictor to warp events into motion-compensated IWEs, which are then segmented by a frozen SAM2 using sparse bounding-box prompts. The generated masks further supervise trajectory learning through a mask-aware contrastive loss. The framework produces dense segmentation masks and event-wise pseudo-labels without requiring pixel-level mask or optical-flow ground truth.

Critical strengths
  • The paper addresses a practical problem. Reducing the annotation cost of event data is useful, and the proposed workflow could benefit event-based perception research.

  • The mask-guided trajectory learning is an interesting idea. Table 6 shows improvements in both segmentation quality and trajectory estimation after introducing mask-aware contrastive learning.

  • The method is evaluated on three datasets with different prompt densities. The paper also includes a user study to assess annotation efficiency.

Critical weaknesses
  1. It is unclear how the method handles complex motion, especially when both the camera and objects are moving. The trajectory model estimates 2D image motion, which can be caused by either camera ego-motion or independently moving objects. Motion compensation alone does not distinguish these two sources of motion. Although bounding-box prompts can help identify the target objects, it is unclear how reliably the method can recover their complete masks under strong ego-motion, background parallax, or multiple independent motions. Although the method is evaluated on DSEC-MOS, the improvement over SAM2-only is relatively small (+1.1 J&F with full prompts and +0.9 with sparse prompts). The paper does not further analyze performance under different camera-motion magnitudes or complex background motion, leaving its robustness in these scenarios unclear. I think this limitation deserves more discussion.
  2. The experimental results are not sufficiently convincing to support the sparse-to-dense annotation claim. Performance drops substantially with sparse prompts, from 85.42% to 70.04% mIoU on EVIMO2 and from 74.5% to 67.5% J&F on DSEC-MOS. The limited qualitative mask visualizations also make it difficult to assess the actual annotation quality. Moreover, since the framework already relies on manual boxes and motion propagation, why not use available RGB frames with SAM2 for mask initialization and events for temporal propagation? This seems a natural alternative, but no such comparison is provided.
Questions for authors
  1. How does the method handle camera ego-motion when both the camera and objects are moving? Could the authors provide a breakdown of DSEC-MOS performance under different levels of ego-motion, especially with strong background parallax?

  2. In Figure 3, what are the actual temporal intervals between bounding-box prompts at different annotation ratios (10%, 30%, 50%, and 100%)? How many masks are generated per prompt, and what is the output frame rate for each dataset?

  3. Since RGB or intensity frames are available in many event-camera datasets, could the authors compare against a baseline using RGB + SAM2 for mask initialization and the same event-based trajectories for propagation, with the same bounding-box annotation budget?

  4. Could the authors provide qualitative segmentation mask comparisons on all three datasets (EVIMO2, EVIMO, and DSEC-MOS), including SAM2-only, Ours, and ground truth? It would be useful to see failure cases involving multiple moving objects, occlusion, and large motion.

  5. How much does the proposed bidirectional training contribute beyond simply using SAM2-generated pseudo masks? Could the authors compare periodic mask updates against keeping the pseudo masks fixed after the initial trajectory training stage?

  6. Does the trajectory model need to be retrained or optimized for each new dataset? What is the offline training cost, and is this cost included when evaluating the scalability of the annotation workflow?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.