- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents CorrRetime, a video-to-video diffusion framework for object-wise temporal retiming under independently controlled object and camera timelines. The key idea is to represent each retiming operation through explicit source-to-target spatiotemporal correspondences, which associate an object state at a specific source time and location with its desired target time and location after camera reprojection. These correspondences, together with the source video, retimed background reference, and object masks, are injected into a pretrained Wan2.1 video diffusion model. The method is trained on synthetic source-target video pairs with known depth, masks, camera poses, and retiming schedules. Evaluation is conducted on a synthetic benchmark with ground-truth retimed videos and on 62 real-world videos using an anchor-based reconstruction protocol, FVD, and a user study. The paper reports improvements over adapted TrajectoryCrafter and ReCamMaster baselines on both synthetic and real-world evaluations.
Critical strengths
- The paper gives a concrete formulation for object-wise retiming by linking each target location to a specific source-time state. This control signal is well matched to the task and can be injected directly into the video diffusion backbone.
- The synthetic setup provides paired ground-truth retimed videos, which makes direct quantitative evaluation possible. The adapted TrajectoryCrafter and ReCamMaster variants use the same backbone and training budget.
Critical weaknesses
- MotionV2V is discussed as a closely related object-level motion editing method, but absent from the experiments. The absence of this comparison leaves the benefit of explicit source-time correspondence insufficiently isolated from a closely related track-based approach.
- The real-world evaluation are computed only at an anchor frame where the object timelines and camera timeline coincide with the observed source frame. These scores provide limited evidence for temporal correctness at the frames where retiming actually changes the source-time state.
Questions for authors
What videos are used as the reference set for the real-world FVD reported in Table 2?
4 comments
Thank you for your response, which substantially addresses my concerns. I noticed that the method explicitly supports temporal schedules that move beyond the source timeline and later re-enter via boundary clipping, but such schedules are excluded from synthetic training. This leaves an important claimed behavior unsupported by the training distribution. Could the authors explain the gap?
Thank you for the follow-up and for noting that our response addressed your concerns. We also thank the reviewer for pointing out that our description of rejection sampling in the appendix (Synthetic Dataset Details) could be misinterpreted; we clarify its scope below.
Clipped schedules are included in training. Rejection sampling applies only to piecewise schedules. Constant-speed schedules are retained even when τ(f) leaves the source range, and the target is rendered by holding the boundary pose, consistent with the clipping in Eq. 2. Because each object and the camera independently use either schedule type, 1,609 of the 3,534 training clips containing piecewise schedules also include an out-of-range constant-speed schedule, and a further 1,565 constant-speed training samples contain clipped segments. Holding at the source boundary is therefore directly supervised.
Re-entry after clipping. Schedules that leave the source range and later re-enter it are not directly included in training. Under Eq. 2, however, re-entry produces the same correspondence conditioning as a pause at the boundary frame followed by forward or reverse motion, since the boundary source frame is repeatedly reprojected to the current camera pose during the clipped interval. Such pause-to-motion transitions occur 454 times across 433 piecewise training samples (276 forward, 178 reverse), with independently retimed cameras, which provides a basis for generalizing to re-entry.
Evaluation. Clipped frames. In our real-video global-retiming evaluation (Table R1), 704 of 5,022 frames (14.0%, across 36 of 62 videos) have clipped source times:
| Frames | Count | Timing error ↓ | Within ±2 frames ↑ | mPSNR ↑ |
|---|---|---|---|---|
| Clipped | 704 | 3.95 | 91.3% | 20.66 |
| Unclipped | 4,318 | 1.28 | 92.4% | 19.40 |
Metrics. Timing error measures the difference between estimated and target source-frame indices. Within ±2 frames denotes the percentage of frames with a timing error of at most two frames. mPSNR measures reconstruction quality over object regions.
Accuracy within two frames and reconstruction quality are maintained on clipped frames. The higher mean timing error (median zero) is driven mainly by two scenes: one with a fast reversed camera, where the background does not follow the requested viewpoint, and one with repetitive motion, where frame-matching timing estimation is ambiguous.
Re-entry. Re-entry is exercised by our anchor-based evaluation: for reversed schedules with anchor 60 (−2×, −3×, −4×), τ(f) exceeds the source range before reversing and re-enters before the anchor frame, which is scored against the captured frame. On these 79 runs, CorrRetime achieves 17.53 dB anchor mPSNR (17.06 dB for TrajectoryCrafter†), compared with 17.71 dB for runs at the same reverse speeds that never leave the source range (anchors 30/45). These results show comparable anchor-frame reconstruction quality for runs with and without re-entry under the evaluated reverse-speed settings. However, since the anchor positions differ and mPSNR is computed over all object regions at the anchor frame, this comparison does not isolate the effect of re-entry or directly measure temporal accuracy during re-entry.
Revision. We will clarify in the appendix that rejection sampling applies only to piecewise schedules, revise the explanation after Eq. 2 to distinguish boundary clipping from re-entry, and add these clipping and re-entry analyses.
Additional analysis: temporal accuracy after re-entry.
To directly evaluate temporal accuracy after re-entry, we analyzed 186 global-retiming runs (62 videos × 3 reverse speeds: −2×, −3×, −4×), all involving source-time clipping followed by re-entry. Unlike our previous anchor-frame evaluation, we evaluated the full sequences using captured-frame ground truth. For each run, we separately analyzed the frames from re-entry up to the anchor frame (11, 7, and 6 frames per run at −2×, −3×, and −4×), clipped frames, and the remaining frames:
| Segment | Frames | Timing error ↓ (median) | Within ±2 frames ↑ | mPSNR ↑ |
|---|---|---|---|---|
| After re-entry | 1,488 | 2.12 (1) | 84.3% | 17.49 |
| Clipped | 2,542 | 1.17 (0) | 97.2% | 20.66 |
| Remaining | 11,036 | 1.90 (1) | 88.4% | 18.27 |
Metrics. Timing error measures the difference between estimated and target source-frame indices. Within ±2 frames denotes the percentage of frames with a timing error of at most two frames. mPSNR measures reconstruction quality over object regions.
The mean per-run timing-error difference between post-re-entry and remaining frames is +0.34 frames (95% scene-bootstrap CI [−0.09, +0.85]), suggesting that temporal accuracy remains reasonably stable after re-entry. Although reconstruction quality is lower (mean per-run mPSNR difference: −0.93 dB), this may partly reflect the difficulty of fast reverse motion and cannot be attributed solely to re-entry.
We will include this frame-level analysis, together with per-speed results, in the appendix of the revised manuscript.
Response to Reviewer AhYY
We thank the reviewer for the helpful comments.
W1. Comparison with MotionV2V.
MotionV2V and Edit-by-Track edit motion through point trajectories. In particular, Edit-by-Track uses paired source–target 3D tracks together with camera poses, so retiming can in principle be expressed by constructing retimed target trajectories.
In contrast, CorrRetime directly controls the timeline of each object and the camera through independent source-time schedules τₖ(f). Each object can start from a different initial source time τₖ(0) and follow a time-varying playback speed sₖ(f), enabling independent acceleration, deceleration, freezing, and reversal.
CorrRetime automatically constructs source–target spatiotemporal correspondences for each output frame using the object state at the selected source time, estimated depth, and camera geometry. Therefore, it does not require persistent point identities across the entire video or manually constructed retimed point trajectories (General Response, Section 3).
A direct quantitative comparison with MotionV2V and Edit-by-Track was not feasible, as neither method had released inference code or model weights as of November 2026 (the MotionV2V GitHub repository hosts only the project page). We instead compare with Go-with-the-Track (GWTT; Namekata et al., SIGGRAPH 2026), a publicly available track-based method.
Under independent object retiming, CorrRetime achieves a substantially lower timing error than GWTT (4.51 vs. 13.65 frames) and also outperforms it in mPSNR (20.05 vs. 19.84 dB), mSSIM (0.627 vs. 0.584), and mLPIPS (0.126 vs. 0.149).
These results suggest that CorrRetime is particularly effective for independent object-wise temporal control (General Response, Table R7).
W2. Real-world evaluation beyond the anchor frame. We extend our real-video evaluation to all 81 frames, rather than evaluating only the anchor frame (General Response, Tables R1 and R2). Under global retiming, where captured frames provide ground truth throughout the video, CorrRetime achieves a mean frame-matching timing error of 1.61 frames, with 92% of output frames matched within two frames of the requested source time, compared with an average error of 20.68 frames without retiming. CorrRetime also achieves the best object-region mPSNR, mSSIM, and mLPIPS (Table R1). Under independent object retiming, we additionally evaluate object-specific timing using geometry-assisted frame matching and reconstruction quality using the captured ground truth for an object retaining the shared schedule. CorrRetime achieves a timing error of 5.24 frames, compared with 19.23 frames without retiming, while obtaining the best reconstruction quality across all three metrics for the locked object (Table R2).
Q. Reference set for real-world FVD. For Table 2, we compute FVD between each method's generated videos and the 62 original captured real-world videos, using I3D features pretrained on Kinetics-400. The original videos serve as the reference distribution for evaluating visual and temporal distributional similarity. Since retimed videos intentionally differ from the original videos in their motion speed and timing, FVD does not directly measure adherence to the requested temporal schedules. We therefore complement it with the frame-level reconstruction and timing evaluations in Tables R1 and R2.