acceptodds
Under review as a conference paper at ICLR 2027

Moving Regions, Not Moving Objects: Segmenting What Actually Moves under Camera Motion

Abstract

Moving object segmentation is conventionally formulated at the sequence and object level, marking an entire instance as moving once it moves anywhere in a video. Recent instantaneous formulations distinguish moving from stationary intervals, but still generate masks at the object level through frozen segmentation foundation models. Because the mask is derived from an instance-level proposal, motion is typically attributed to the entire instance rather than localized to the regions that actually move. Moreover, such instance-level predictions can produce masks even when no part of the instance is actually moving, revealing a more fundamental issue: motion is not an intrinsic property of an instance; it can only be determined by comparing at least two frames. Building on this premise, we introduce **MoRSe**, which jointly interprets two frames to predict a moving-region mask, distinguishing actual scene motion from apparent motion induced by the camera. A single RGB encoder replaces multi-stage foundation stacks, so no optical flow, point tracks, or separate mask generator is needed at inference. We release **MoRSe-5K**, a benchmark of 42 sequences with Static, Dynamic-Static, and Dynamic settings designed to evaluate the model’s ability to distinguish and localize moving and stationary regions under different camera and scene motion conditions. MoRSe outperforms both specialized moving object segmentation methods and dense 4D reconstruction methods with motion-thresholded predictions, while producing the fewest false positives on sequences where nothing moves.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Moving-region segmentation,Motion Segmentation,Camera Motion

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2o3TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper reformulates motion segmentation around an ordered source–target frame pair: a source pixel should be selected when its scene point moves independently of the camera, rather than whenever the containing object moves somewhere in the video. MoRSe uses a UniQuery4R-initialized joint encoder, a shallow convolutional branch, and a coordinate-query decoder that reads target memory through cross-attention. Camera prediction is an auxiliary training objective and does not directly enter mask prediction. Predictions can be unioned over targets, and a SAM 3 readout can expand regions to whole objects for conventional evaluation. The authors introduce MoRSe-5K, with 42 sequences and 5,431 frames, and report improvements in moving-region accuracy and suppression of camera-induced false positives.

Critical strengths
  1. The paper gives a clear operational formulation of local motion: an arm can move while the rest of a person remains still, and the same source image can have different moving regions depending on the target. MoRSe directly predicts these regions without optical flow, point tracks, or a separate mask generator for native region inference. Figures 4 and 6 provide concrete qualitative evidence of local motion and target-dependent responses. The Static setting usefully isolates false positives when only the camera moves. These are useful design and evaluation choices, even though separating scene motion from camera motion is an established problem.

  2. The complete system has substantive empirical support. In Table 1, moving-region J&F is 77.98%, compared with 69.64% for GMOS-S, while Static PER/FER decrease from 0.60%/6.82% to 0.0009%/0.75%. The joint accuracy and false-positive results rule out an explanation based on simply predicting empty masks. Importantly, the paper also evaluates whole-object ground truth on the same frames: MoRSe + SAM 3 achieves 79.32% J&F versus 77.93% for GMOS-S. This does not isolate every source of improvement, but the comparison is not confined to region labels that necessarily penalize instance-level methods. Tables 4 and 9 provide useful component and initialization controls, and the paper appropriately notes that the fusion variants are close within the reported seed variation.

Critical weaknesses
  1. The advantage over explicit camera-motion compensation is not sufficiently explained. A natural alternative is to estimate optical flow, predict the per-pixel flow that a static scene would exhibit under the camera motion, and segment the residual. This requires a spatially varying flow field, not subtraction of one global shift, because camera translation produces depth-dependent parallax. The paper does compare related approaches: RoMo uses flow-based epipolar residuals, and several scene-flow models estimate motion in world coordinates (Section 2.2 and Table 1). Their weaker results support MoRSe, but an epipolar residual is not the same as subtracting a predicted static-scene flow field. It remains unclear which failures of explicit compensation MoRSe overcomes: inaccurate camera/geometry estimates, errors in optical flow, or the conversion of residuals into masks. A focused analysis of the existing baselines, or a direct compensated-flow comparison where those baselines do not test this alternative, would clarify the added capability and its accuracy–cost trade-off. This is an unresolved comparison question, not evidence that a simpler pipeline already matches MoRSe.

  2. Small but genuine motion can be missed entirely, not just segmented with imprecise boundaries. Section 5 and Appendix H acknowledge this failure, and Figure 5 shows partial recovery of a swaying branch and a small animal whose motion is not activated at all. This directly limits the claimed ability to detect local movement independently of camera motion. The examples establish a real failure mode, but not its prevalence. Aggregate scores do not show whether strong static suppression comes with reduced sensitivity to subtle motion, or where the method's reliable operating range ends.

Questions for authors
  1. Corresponding to Weakness 1: Which failures of explicit camera-motion compensation does MoRSe overcome? Please first use the existing geometric baselines to analyze performance under camera rotation and depth-dependent parallax. If they do not directly test the compensated-flow alternative, could you compare against observed optical flow minus a predicted static-scene flow field under matched frame/window inputs? Please state how camera motion and geometry are estimated without ground-truth test information; a homography is appropriate only where its assumptions hold, while general 3D scenes require depth-aware compensation. Report moving-region J&F, Static PER/FER, and end-to-end runtime on the same hardware, including flow and geometry estimation, with thresholds selected without test-label tuning. Analysis of an existing baseline that directly addresses this question would suffice; the goal is to understand the advantage, not to assume that the simpler method must win.

  2. Corresponding to Weakness 2: Could you report detection recall or moving-region J&F by apparent motion magnitude, including the low-displacement cases illustrated in Figure 5? A small threshold sweep, with Static PER/FER reported alongside recall, would help distinguish an adjustable operating-point trade-off from motion that the model does not reliably detect. Please clarify whether adding nearby targets recovers these cases or whether they remain missed under the window protocol.

Reviewer_nG9JOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MoRSe for segmenting independently moving regions under camera motion, with predictions indexed by an ordered source–target frame pair. A geometry-pretrained encoder and shallow convolutional branch provide features for coordinate queries that attend to target-frame representations and directly predict motion masks. Training combines trajectory-derived motion labels, curated real-video masks, and static-scene negatives. The paper also introduces MoRSe-5K, a 42-sequence benchmark covering static, intermittently dynamic, and continuously dynamic scenes, and evaluates temporal aggregation and transfer to conventional moving-object segmentation.

Critical strengths

The distinction between moving regions and whole objects is useful. Detecting a moving limb while suppressing stationary parts can provide more precise information for reconstruction and scene understanding. Explicitly evaluating static scenes under camera motion also addresses an important failure mode.

The reported benchmark improvements are substantial. Table 1 reports moving-region J&F of 77.98 versus 69.64 for GMOS-S, together with markedly lower static-scene false positives.

The paper provides meaningful controls and computational details. The experiments include backbone, convolutional-branch, camera-supervision, and fusion ablations with two-seed variability. The appendix also reports runtime, memory, and window/query-density effects, while acknowledging limitations and weaker conventional MOS results.

Critical weaknesses

The performance advantage is primarily demonstrated on the proposed benchmark. Even with SAM 3 completion, MoRSe remains below GMOS on all three conventional MOS benchmarks, with substantial gaps on FBMS-59 and MoCA. Although the region-versus-object distinction partly explains this difference, the results do not establish a broader performance advantage beyond MoRSe-5K.

Comparisons under matched training conditions are not established. The paper reports controlled reruns and matched temporal windows, but does not clarify whether trainable baselines were retrained using the same training data and moving-region supervision as MoRSe. Consequently, the gains cannot be clearly attributed to the proposed design rather than differences in training data and supervision.

The pooled results do not establish performance across domains and motion regimes. MoRSe-5K contains 21 synthetic and 21 real sequences, but their results are not reported separately. Moreover, Dynamic-Static and Dynamic scores are pooled with frame counts of 1,904 versus 182, leaving the latter only about 8.7% of the weight. The aggregate advantage therefore does not establish consistent gains on real scenes or continuously dynamic sequences.

Questions for authors

How do you explain MoRSe’s lower performance than GMOS on all three conventional MOS benchmarks, even with SAM 3 completion?

Were trainable baselines retrained using the same training data and moving-region supervision as MoRSe, or evaluated using released checkpoints?

Can you provide available results separately for real versus synthetic scenes and Dynamic-Static versus Dynamic settings?

Reviewer_n131OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MoRSe, a pair-conditioned moving-region segmentation method that predicts which pixels in a source frame correspond to independently moving scene points between an ordered source–target frame pair. Unlike conventional moving-object segmentation, which may label an entire object as moving even when only one part moves, the proposed method aims to localize motion at the region level while suppressing apparent motion caused by camera movement.

Critical strengths

1 The author addresses an important limitation of conventional MOS. where an entire object may be labelled as moving even when only a small part is actually moving. 2. The approach is simpler than multi-stage pipelines also it does not require an additional segmentation model during inference. 3 Paper provides the temporal flexibility where target selection sets the granularity without changing the weights. 4 The result for MoRSe-5k in Table-1 supports the claims of the paper.

Critical weaknesses

1 The method struggles to detect small movements between source and target frames. 2 The predicted masks have less accurate boundaries and regions with similar colours or repetitive textures. 3 The maximum aggregation strategy may introduce false positives when more target frames are considered, making the method sensitive to target selection and temporal window size.

Questions for authors

1)Since 92.3% of the training sampling mass comes from synthetic data, how well does MoRSe perform on real-world videos with complex camera motion, motion blur, and lighting changes? 2)The ablation study shows that removing the auxiliary camera head increases the frame error rate.Since the predicted camera pose is not directly used by the mask decoder, could the authors explain how this auxiliary task helps reduce false motion predictions?

Comments on the paper

1 comment
Area_Chair_8fvcOpenReview ↗

Discussion between Authors and Reviewers

Dear Reviewers,

The authors have submitted their rebuttal. Please read it and discuss with the authors via posted messages if necessary.

Note that the author-reviewer discussion phase ends on November 18 at 11:59 pm AoE. It is appreciated if you could share your thoughts as soon as possible, so the authors have enough time to clarify any remaining concerns.

Thank you!

Best, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.