acceptodds
Withdrawn from ICLR 2027

TRACE: Object Motion Editing in Videos with First-Frame Trajectory Guidance

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We study object motion editing in videos, where the goal is to change a target object's trajectory while preserving the remaining content of the original video. Many existing trajectory-controlled video editing methods require users to specify frame-wise control signals, such as dense point tracks or bounding boxes, making motion design cumbersome and error-prone, particularly in videos with camera motion. We introduce TRACE, a framework that allows users to specify the desired motion path in the first frame alone. The key challenge is that a path defined in the first-frame coordinate system does not directly determine the object's image-space trajectory in subsequent frames as the camera moves. To address this mismatch, TRACE employs a two-stage pipeline. First, a cross-view motion transformation module converts the first-frame path into frame-aligned bounding-box trajectories. Second, a motion-conditioned video re-synthesis module removes the object from its original trajectory and regenerates it along the transformed trajectory while preserving non-target video content. Experiments on diverse real-world videos show that TRACE produces more coherent and controllable motion edits than recent image-to-video and video-to-video methods

Primary area:
generative models
Keywords:
video generation, video editing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_doEeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an object motion editing method for videos, named TRACE, which allows users to specify the desired motion path solely in the first frame. Specifically, TRACE employs a two-stage pipeline. Stage 1 utilizes a learnable cross-view motion transformation module to convert the first-frame path into frame-aligned bounding-box trajectories. Stage 2 removes the object from its original trajectory and regenerates it along the new bounding-box trajectories predicted by Stage 1. Extensive experiments on real-world videos demonstrate the effectiveness of the proposed method.

Critical strengths
  1. This paper proposes a first-frame-guided object motion editing framework for videos, which simplifies the user interaction process.
  2. This paper provides extensive demonstration videos in the supplementary material.
Critical weaknesses
  1. Since the method only uses the first frame as the appearance reference for video re-synthesis, it remains unclear how the method preserves identity-specific details and generates natural non-rigid motion under large trajectory changes.
  2. In Stage 1, TRACE leverages a cross-view motion transformation module to predict new bounding-box trajectories. However, under large camera motion, it remains unclear how the method ensures that the predicted trajectories accurately reflect the user's guidance specified in the first frame.
  3. Since video resynthesis relies on the bounding-box trajectories predicted by Stage 1, the robustness of the method to large foreground objects and substantial editing regions is not sufficiently evaluated.
Questions for authors

Please refer to the Critical Weaknesses.

Reviewer_mCMkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TRACE, a framework for object motion editing in videos using first-frame trajectory guidance. Unlike existing methods that require frame-wise motion annotations, TRACE allows users to specify the desired object trajectory using sparse bounding boxes in the first frame. The framework consists of two stages: a cross-view motion transformation module that converts the first-frame path into frame-aligned bounding boxes under camera motion, and a video re-synthesis module that removes the object from its original trajectory and regenerates it along the target path. The latter is built upon Wan 2.1 and trained without paired motion-edited videos. Experiments demonstrate improved motion controllability compared with existing methods.

Critical strengths
  1. The first-frame trajectory guidance provides an intuitive interface for object motion editing under camera movement.

  2. The two-stage framework is well organized, and the experimental results demonstrate comparable video editing quality and controllability.

Critical weaknesses
  1. Sparse bounding-box and trajectory-based motion control have been explored in previous works. The paper could better clarify its methodological contributions beyond the first-frame interaction design and the integration of existing techniques.

  2. Limited evaluation of complex scenarios: Although the paper demonstrates zero-shot multi-object editing, its effectiveness for controlling three or more objects, especially with overlapping trajectories and occlusions, remains unclear.

  3. Insufficient evaluation of motion quality: Some showcase examples appear to exhibit less smooth object motion. The current metrics, such as FVD and Region Containment, do not directly capture motion smoothness or temporal consistency.

Questions for authors
  1. With recent general-purpose video-to-video editing models such as MiniMax-H3 supporting flexible video editing, what are the distinctive advantages of TRACE's multi-stage framework? Could a VLM-based planner generate bounding-box trajectories directly from natural-language instructions, further reducing manual interaction?

  2. Could the authors provide evaluations involving three or more independently controlled objects?

Reviewer_1wx3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TRACE edits object motion in an existing video from sparse boxes drawn in the first-frame view. Stage I predicts frame-aligned target boxes from this path, the first frame, and video point tracks. These boxes and the original object locations are then used to construct a masked video; Stage II jointly restores the vacated background and synthesizes the object along the new path. Stage I uses synthetic paired trajectories, whereas Stage II learns through reconstruction without paired source/motion-edited videos. Evaluation covers trajectory conversion, DAVIS reconstruction, and editing on 70 videos.

Critical strengths
  1. A practical integration with an economical supervision strategy. TRACE automates the conversion of first-frame paths into per-frame controls and jointly handles object removal and re-synthesis. Its role-coded masks allow Stage II to learn from unedited videos rather than paired motion edits (Sec. 3.2). This is useful even though neither trajectory control nor masked regeneration is new by itself.

  2. Encouraging reconstruction and perceptual performance. TRACE obtains the best reported DAVIS SSIM/LPIPS (0.71/0.19; Table 2) and full-pipeline FVD (614.94 versus 772.09 for the strongest baseline on that metric; Table 3). The 52-participant study also favors TRACE in identity preservation and video quality (Fig. 10; Appendix A.8). Together, these results support the system's practical effectiveness.

Critical weaknesses
  1. The novelty boundary and attribution of gains need clarification. ReVideo already edits motion in existing videos, while MotionCanvas already supports start/end/intermediate boxes on an input image and camera-aware conversion into frame-aligned controls. TRACE differs meaningfully by learning this conversion for an existing video and coupling it with joint removal/re-synthesis; however, the first-frame interface alone is not new. Although MotionCanvas is included as a baseline, the shared TRACE-generated box guidance in Fig. 6 does not isolate the benefit of TRACE's conversion over MotionCanvas-style geometric translation. Likewise, the reported Stage II ablation tests smoothing and augmentation, rather than the joint-versus-separate removal/synthesis design (Table 5). The specific technical gains therefore remain insufficiently separated from the overall system improvement.

  2. Evidence for accurate intended motion is weaker than evidence for visual quality. Stage I's quantitative test uses held-out videos from the same synthetic rendering pipeline (Sec. 4.1; Appendix A.6). On this benchmark, the stronger center-depth DA3 baseline nearly matches forward IoU (0.79 versus 0.80), although TRACE has a clear forward mAP advantage (0.91 versus 0.73; Table 6). Real-video editing provides additional evidence, but Region Containment checks generated pixels against TRACE's own predicted boxes, not independently verified target placements; it also gives full credit to an undersized object contained within a box (Appendix A.5). Scores cluster at 0.9822–0.9906 (Table 3), and some human alignment preferences are modest (53.4% against Wan-Move), without reported uncertainty. More direct validation is needed to substantiate accurate camera-aware editing.

Questions for authors
  1. What is the precise methodological advance over MotionCanvas's scene-anchored box interface and geometric translation? With Stage II fixed, how do learned and geometric trajectories compare on changed paths under real camera motion, using independently verified target placements?

  2. Under matched backbone, data, and guidance conditions, how does joint removal/re-synthesis compare with separate removal followed by synthesis? This would help establish the contribution of Stage II's design beyond the existing smoothing/augmentation ablations.

  3. Can you supplement Region Containment with target-location/scale errors and user-study confidence intervals accounting for repeated judgments of participants and videos? Please also reconcile DA3's reverse IoU in Tables 1 and 6 (0.70 versus 0.77 for the apparently corresponding four-corner configuration).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.